Handle VM shutdown failures and recover partially stopped guests

[borg-codex-bot]

Problem

A reported VM backup aborted after one selected VM was still running after approximately 90 seconds. The log then reported Job aborted by exception: 1 and Borg backup failed (exit 2), although archive creation had not started. The underlying reason why the guest did not shut down is not established by the supplied excerpt.

The current implementation also has a partial-shutdown recovery gap: BackupJob.shutdown_vms() assigns its shutdown result and records recovery state only after the VM manager returns successfully. When some selected VMs stop but another times out, the manager raises SystemExit(1) before returning the stopped-VM list. Consequently, the normal cleanup path may leave VMs already stopped for this job powered off.

Code findings

• runtime/lib/vm_manager.py: both shutdown_all() and shutdown_selected() raise SystemExit(1) on timeout. The return code and captured error output of virsh shutdown are not evaluated.
• runtime/lib/backup_job.py: shutdown results and persistent VM recovery state are recorded only after the complete shutdown succeeds. Generic exception handling turns the failure into exit 2 and can report it as a Borg failure.
• api/wizard_runner.py: VM shutdown precedes Borg archive creation, so this failure should be identified as a preparation failure.

Intended behavior

• Continue to abort before archive creation when required VMs have not shut down. Do not automatically force off guests.
• Use a structured VM shutdown failure with a stable failure code, affected VM names and the configured timeout, replacing the opaque exception: 1 message.
• Report that the backup was not started because VM shutdown failed, consistently in the job log, stored history and notifications.
• Inspect virsh shutdown return codes and relevant output; distinguish command rejection/failure from a guest that remains running until the timeout. Sanitize diagnostics through existing logging conventions.
• Capture the initial state and intended shutdown targets before issuing shutdown requests. Preserve recovery information during partial failure, and recover VMs stopped for this job without starting guests that were already off beforehand.
• Account for guests that finish shutting down around the timeout/cleanup boundary; avoid claiming successful recovery while a requested shutdown remains unresolved.
• Preserve the original shutdown cause if cleanup also fails, and expose any unresolved VM recovery state through the existing recovery mechanism.
• Retain the existing configurable VM shutdown timeout; increasing it alone does not address error reporting or partial recovery.

Validation

Cover both all-VM and selected-VM modes:

• Successful shutdown and restart.
• One guest remains running until timeout; Borg archive creation is not invoked.
• Mixed result: one guest stops and another times out; the stopped guest is recovered.
• A guest already off before the job is not started by cleanup.
• Nonzero virsh shutdown return code, subprocess timeout and execution failure produce useful diagnostics.
• Late shutdown around the timeout boundary and failed restart retain accurate recovery information.
• History and notifications identify the shutdown/preparation failure instead of a Borg archive-creation error.

Scope and release notes

User-visible reliability and diagnostics fix. Preserve the current runtime architecture and backup configuration. Add German/English user-facing messages where applicable and a release-note fragment. Implementation and a verified test-channel candidate will follow separately; this issue records the findings only.