Repository navigation
Sandbox stays admission-REJECTED after its middleware registration is restored; Stop/Start then leaves it undeletable #5043
Description
User Story
I use OpenShell v0.1.5 as the sandbox runtime of a self-hosted agent platform I am building (Fort): one gateway as a systemd service on Ubuntu 24.04, Docker driver, a supervisor middleware and gateway interceptors on Unix sockets. I hit this while testing that setup on two VMs.
Problem Statement
When a [[openshell.supervisor.middleware]] registration that existing sandbox policies reference is removed and the gateway is restarted, those sandboxes correctly do not start: admission is CONFIGURATION_ADMISSION_STATE_REJECTED ("Effective middleware configuration is invalid; repair the policy middleware bindings or registered services"). After the registration is restored and the gateway restarted again, the sandbox stays REJECTED: StopSandbox/StartSandbox and a fresh UpdateConfig do not re-admit it. After that Stop/Start attempt, DeleteSandbox fails with FAILED_PRECONDITION "previous driver operation is still pending; retry and deletion are blocked", and provisioning.driverOperationPending stayed true through two more gateway restarts and removal of the exited container the next morning.
The undeletable end state looks like #4409 / #4038; the new part here is that admission is never re-validated once the configuration is fixed, so the only way out we found was deleting the sandbox, and that path was then blocked.
Impact / Why This Matters
A temporary configuration mistake on the gateway (a registration removed by hand, a bad deploy) permanently disables every sandbox that references it, even after the configuration is repaired, and the affected sandboxes cannot be cleaned up through the API.
Acceptance Criteria
- When the referenced middleware registration is present again, the sandbox is re-admitted (at gateway start, on
StartSandbox, or onUpdateConfig). - A sandbox in this state can always be deleted.
Reproduction Steps
- Configure a supervisor middleware registration and create a sandbox whose policy references it; wait for Ready.
- Remove the registration from
gateway.tomland restart the gateway: the sandbox's admission isREJECTED. - Restore the registration and restart the gateway: the sandbox stays
REJECTED. StopSandboxthenStartSandbox, thenDeleteSandbox→FAILED_PRECONDITION"previous driver operation is still pending".
Seen once, on a sandbox that had just been recovered (Stop + Start) after a gateway crash (see #4910 for that symptom).
Environment
- OpenShell: v0.1.5 (bdb956f), gateway release tarball, run as a systemd service (non-root
openshelluser) - OS: Ubuntu 24.04 (arm64 on a Lima VM, kernel 6.8; x86_64 on a GCP VM, kernel 7.0.0-1014-gcp), Docker 29.9.2
- Deployment: single gateway, Docker compute driver, supervisor middleware on HTTPS loopback, two gateway interceptors on Unix sockets (extension auth on)
Logs
phase SANDBOX_PHASE_PROVISIONING, admission CONFIGURATION_ADMISSION_STATE_REJECTED:
"Effective middleware configuration is invalid; repair the policy middleware bindings or registered services"
DeleteSandbox: FailedPrecondition: previous driver operation is still pending; retry and deletion are blocked
Activity