Skip to content

Sandbox stays admission-REJECTED after its middleware registration is restored; Stop/Start then leaves it undeletable #5043

Open

Description

@thenashef

User Story

I use OpenShell v0.1.5 as the sandbox runtime of a self-hosted agent platform I am building (Fort): one gateway as a systemd service on Ubuntu 24.04, Docker driver, a supervisor middleware and gateway interceptors on Unix sockets. I hit this while testing that setup on two VMs.

Problem Statement

When a [[openshell.supervisor.middleware]] registration that existing sandbox policies reference is removed and the gateway is restarted, those sandboxes correctly do not start: admission is CONFIGURATION_ADMISSION_STATE_REJECTED ("Effective middleware configuration is invalid; repair the policy middleware bindings or registered services"). After the registration is restored and the gateway restarted again, the sandbox stays REJECTED: StopSandbox/StartSandbox and a fresh UpdateConfig do not re-admit it. After that Stop/Start attempt, DeleteSandbox fails with FAILED_PRECONDITION "previous driver operation is still pending; retry and deletion are blocked", and provisioning.driverOperationPending stayed true through two more gateway restarts and removal of the exited container the next morning.

The undeletable end state looks like #4409 / #4038; the new part here is that admission is never re-validated once the configuration is fixed, so the only way out we found was deleting the sandbox, and that path was then blocked.

Impact / Why This Matters

A temporary configuration mistake on the gateway (a registration removed by hand, a bad deploy) permanently disables every sandbox that references it, even after the configuration is repaired, and the affected sandboxes cannot be cleaned up through the API.

Acceptance Criteria

  • When the referenced middleware registration is present again, the sandbox is re-admitted (at gateway start, on StartSandbox, or on UpdateConfig).
  • A sandbox in this state can always be deleted.

Reproduction Steps

  1. Configure a supervisor middleware registration and create a sandbox whose policy references it; wait for Ready.
  2. Remove the registration from gateway.toml and restart the gateway: the sandbox's admission is REJECTED.
  3. Restore the registration and restart the gateway: the sandbox stays REJECTED.
  4. StopSandbox then StartSandbox, then DeleteSandbox → FAILED_PRECONDITION "previous driver operation is still pending".

Seen once, on a sandbox that had just been recovered (Stop + Start) after a gateway crash (see #4910 for that symptom).

Environment

  • OpenShell: v0.1.5 (bdb956f), gateway release tarball, run as a systemd service (non-root openshell user)
  • OS: Ubuntu 24.04 (arm64 on a Lima VM, kernel 6.8; x86_64 on a GCP VM, kernel 7.0.0-1014-gcp), Docker 29.9.2
  • Deployment: single gateway, Docker compute driver, supervisor middleware on HTTPS loopback, two gateway interceptors on Unix sockets (extension auth on)

Logs

phase SANDBOX_PHASE_PROVISIONING, admission CONFIGURATION_ADMISSION_STATE_REJECTED:
  "Effective middleware configuration is invalid; repair the policy middleware bindings or registered services"
DeleteSandbox: FailedPrecondition: previous driver operation is still pending; retry and deletion are blocked
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions