Flaky release gate: TestServerLifecycle/second_release fails on drift from a deliberately-killed scheduled job

[vishr]

TestServerLifecycle/second_release failed once in the v2026.8.9 release run, then passed on a re-run with no code change. Because it runs in Verify release on a real server, a flake there blocks publication until someone re-runs the job by hand.

What happened

Run: https://github.com/labstack/onebox/actions/runs/32674333837 (job Verify release on a real server / End-to-end (Server / lifecycle), first attempt).

--- FAIL: TestServerLifecycle (321.16s)
    --- FAIL: TestServerLifecycle/second_release (16.03s)
        ✗ ob: host drift: timeout-chore runs image , plan saw sha256:9db7b59979c38555a39def84a31fb98b5296952f9e3afd4f6f11f05b07adfab0 — re-plan
        ob deploy --plan ... --approval ... -y failed: exit status 1
    --- FAIL: TestServerLifecycle/rollback_returns_the_previous_release (5.26s)
        ✗ ob: no rollback target: current release 20260823-234411-nogit-deploy-d46c1ed7db76 has no recorded predecessor

The rollback failure is a cascade: the halted deploy never recorded a predecessor, so the next subtest had nothing to roll back to.

Why it is suspicious rather than mysterious

The live image for timeout-chore reads as the empty string, not as a different digest. That is the signature of observing a container that is not in a readable state, rather than of genuine drift.

timeout-chore is the scheduled job the preceding subtest deliberately wedges so systemd terminates it via TimeoutStartSec (e2e/server_test.go:264-282). So by the time second release (:322) runs, the host carries a job container that was killed mid-run. The drift check appears to observe it in a window where docker inspect yields no image, and refuses the deploy.

If that reading is right, the bug is in how live state treats a container that exists but has no readable image — for a job, which is not part of the running application at all — and the fix is either to skip job containers in the drift comparison, to distinguish "no image" from "a different image", or to wait for the killed container to settle.

Evidence it is a race, not a regression

• The five release runs before this one all passed the same job.
• The same commit passed the full server suite locally twice (717s and 730s).
• Re-running the failed job on the identical commit passed every step, including publish.
• The change in that release (fix: budget the drain against the container's own healthcheck #110) touches rolling-deploy drain timing, not job containers, drift observation, or schedule timeouts — though it does alter deploy timing, which is how a latent race would surface.

Why it is worth fixing rather than tolerating

Every occurrence stalls a release and needs a human to re-run it. The failure also looks like a real drift refusal, so the first reading is that the release is unsafe to publish — which costs an investigation each time.

A reproduction likely needs the wedged-job subtest immediately followed by a deploy, which is exactly the ordering the suite already has; the race just usually resolves in time.