Skip to content

ExecSandbox processes keep running after execution_timeout and after the client disconnects #5042

Open

Description

@thenashef

User Story

I use OpenShell v0.1.5 as the sandbox runtime of a self-hosted agent platform I am building (Fort): one gateway as a systemd service on Ubuntu 24.04, Docker driver, a supervisor middleware and gateway interceptors on Unix sockets. I hit this while testing that setup on two VMs.

Problem Statement

A process started with ExecSandbox keeps running after its execution_timeout passes and after the exec client has disconnected. With ExecSandbox{command: ["sh","-c","sleep 600"], execution_timeout: 5s}, the client stream ends after 5 s, but sleep 600 is still running in the workload afterwards (seen 8 s after the timeout).

Impact / Why This Matters

A caller cannot rely on execution_timeout or on cancelling the stream to stop what it started. Orphaned processes keep using the sandbox's resources: in one run a hung exec'd client held about 100 TCP connections, the sandbox's network relay streams (96 per sandbox) stayed busy, and new connections from the sandbox failed for more than 10 minutes. (We saw that long outage once and could not reproduce it once the client process was able to exit.) Today we kill what we start ourselves.

Acceptance Criteria

  • When execution_timeout expires, the exec'd process tree is terminated.
  • When the exec client cancels or disconnects, the process tree is terminated, or the API documents that exec'd processes are detached.

Reproduction Steps

  1. Create any sandbox and wait for Ready.
  2. ExecSandbox with command: ["sh","-c","sleep 600"] and execution_timeout: 5s; let the client stream end.
  3. List the workload's processes from the host, e.g. docker exec -u 0 <workload container> ps -eo pid,etimes,args: sleep 600 is still running.

Environment

  • OpenShell: v0.1.5 (bdb956f), gateway release tarball, run as a systemd service (non-root openshell user)
  • OS: Ubuntu 24.04 (arm64 on a Lima VM, kernel 6.8; x86_64 on a GCP VM, kernel 7.0.0-1014-gcp), Docker 29.9.2
  • Deployment: single gateway, Docker compute driver, supervisor middleware on HTTPS loopback, two gateway interceptors on Unix sockets (extension auth on)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions