[FrostGod]

Summary

If no live supervisor takes over a shared-pool sandbox, the runtime ends it at the 60-second recovery deadline. The gateway kept showing the old status until a supervisor came back. Now the sandbox is reported Stopped once that deadline has passed, and sandbox start runs it again.

Related Issue

No issue. Observed while testing #4864: we froze every supervisor replica for longer than the recovery deadline. The workload was gone, but the sandbox status didn't change until a replica returned.

Changes

• The gateway records when it first sees that an owner's process is gone (expired or restarted, not leaving). It uses the shared DB membership for this, so every gateway replica agrees.
• After the recovery deadline plus MEMBER_TTL (15 s), the pool view marks the route expired. The controller then reports the instance Absent, and the existing Kubernetes path stops the sandbox.
• A takeover starts a new owner term, which clears the expiry.

Status after this change

Scenario	Status
Gateway lost, HA	Ready, no change. Another gateway replica answers.
Gateway lost, non-HA	No status while it's down. Ready when it's back, and the workload keeps running.
Supervisor lost, HA, others healthy	Ready, no change. Another replica takes over within about 24 seconds.
Supervisor lost, non-HA, back within the 60-second deadline	Ready when the restarted replica re-adopts the sandbox, and the workload keeps running. Outages over about 30 seconds show Provisioning until then.
Supervisor lost, non-HA, or every HA replica, out past the deadline	Before: unchanged until a replica returns. After: Stopped about 2.5 minutes after the supervisor died (measured 141 s), without waiting for a replica. sandbox start then runs it fresh.

Testing

• cargo clippy -p openshell-supervisor-control -p openshell-server --all-targets -- -D warnings
• cargo nextest run -p openshell-supervisor-control -p openshell-server -p openshell-driver-kubernetes: 2970 passed
• Unit test added: a gone owner becomes Absent after the deadline, a leaving owner never does, and a takeover clears it
• Manual k3d run with 2 supervisor replicas:
  • All replicas frozen for 20 s: the sandbox stays Ready and the same workload is adopted.
  • All replicas frozen past the deadline: the sandbox is Stopped while they are still frozen, and sandbox start brings it back Ready with a new workload.
  • Existing single-owner freeze and takeover, and deadline expiry with a later thaw, still pass.
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[FrostGod]

/ok to test b3d4aca