feat(cli): add gym sandbox debug

[terrykong]

Top of a 5-PR stack (#2400 → #2401 → #2402 → #2407 → #2403). Review the three below first; this one is the consumer that makes them worth having.

Why

Poking at a sandbox-backed task currently means standing up the whole stack — model server, Ray, agent, rollout — and hoping the failure reproduces. There is no way to ask "what is actually inside that container?" without writing a harness.

What

gym sandbox debug boots the sandbox a server would create for a task — same provider, same image, same SandboxSpec a rollout uses — runs a command or uploaded script inside it, and writes a re-runnable trace. Plus gym sandbox exec / rm for reattaching to a kept sandbox.

Two things need no cluster and no credentials: --list-tasks reads only the dataset, and --dry-run resolves the image and command wrapping offline in milliseconds.

Worked example: a mini-swe-agent-2 container running pip install torch

Point at a server (nothing lands in a config file):

export OPENSANDBOX_DOMAIN=<your-opensandbox-host>:<port>
export OPENSANDBOX_API_KEY=<your-api-key>

If the server runs with auth disabled (api_key = "" plus OPENSANDBOX_INSECURE_SERVER=YES) there is no key to obtain — but the variable must still be set to something, because the config interpolates it and Hydra fails on an unset reference. For a cluster-internal server, port-forward and point OPENSANDBOX_DOMAIN at the local end.

First, what would this task even get? No cluster needed:

AGENT=responses_api_agents/mini_swe_agent_2/configs/mini_swe_agent_2.yaml
PROVIDER=nemo_gym/sandbox/providers/opensandbox/configs/opensandbox.yaml

gym sandbox debug --config $AGENT --config $PROVIDER --task django__django-10973 --dry-run

image     docker.io/swebench/sweb.eval.x86_64.django_1776_django-10973:latest
          via spec_resolver (responses_api_agents.mini_swe_agent_2.sandbox_hooks:spec_for_row)
spec      ttl_s=900.0  ready_timeout_s=1200  cpu=2.0  memory_mib=8192  disk_gib=30  workdir=/testbed
exec      activate_conda=True conda_env=testbed cwd=/testbed user=root
wrapper   responses_api_agents.mini_swe_agent_2.sandbox_hooks:conda_activate_wrap

The image was derived from instance_id + subset — no row field carried it. That answer comes from #2402's hooks; before them it was unreachable without importing ray.

Now actually run the install and time it:

gym sandbox debug --config $AGENT --config $PROVIDER \
    --task django__django-10973 --command "pip install torch" --timeout-total 900

✓ django__django-10973  reason=pass exit=0  67.1s (command 21.0s + overhead 1.3s, boot 44.8s)

gym sandbox debug ... --command "pip install torch" --json | jq '.timing'

{"boot": 44.812, "setup": 0.0, "exec": 22.303, "command": 21.024, "overhead": 1.279, "total": 67.115}

command is execd's measurement from inside the sandbox (#2400); overhead is what the round trip added. Only the first is a property of the task, so it is the number to compare across images, package indexes, or clusters — and here it is a third of the total, with boot dominating because SWE-bench images are large.

Note which torch: this image's testbed env is Python 3.6, so pip resolves to torch 1.10.2 — the last release with cp36 wheels — at about 880 MB, not a multi-GB modern build. Measured against a warm in-cluster package cache the same install runs 21s cold / 13s warm, which is what makes it a useful probe for whether a cache is actually being used.

If the sandbox's egress sidecar is intercepting TLS

A conda-based image will not trust the proxy's CA and pip fails with CERTIFICATE_VERIFY_FAILED. The CA is installed into the system trust store, but conda ships and consults its own at /opt/miniconda3/envs/<env>/ssl/cert.pem — both hold 151 CAs, only the system one has the proxy's. Every SWE-bench image is conda-based, so this is the common case:

--env SSL_CERT_FILE=/opt/opensandbox/mitmproxy-ca-cert.pem \
--env REQUESTS_CA_BUNDLE=/opt/opensandbox/mitmproxy-ca-cert.pem

Debian-style images (python:3.12-slim) read the system store and need none of this. Documented in debugging.mdx.

Not leaking sandboxes

The failure mode of getting this wrong is a leaked, paid-for sandbox, so it is handled deliberately rather than incidentally:

• sandboxes get a short lifetime that renewal (feat(sandbox): let providers renew a sandbox's lifetime #2401) extends while the command runs, instead of a long one nobody remembers to clean up;
• the first SIGINT logs and re-raises so every finally unwinds, and later signals are swallowed — a second Ctrl-C must not orphan what the first was cleaning up;
• an atexit backstop over a WeakSet catches the case where the loop is already gone.

Known limitations

No live streaming. SandboxProvider.exec returns a completed result, so incremental output needs new provider surface. execd does stream over SSE, so this is a follow-up rather than a dead end.

Requesting an egress sidecar. --metadata and --provider-option (added here) are the top layer of the resolution chain, so a spec_resolver's spec can now be extended from the command line — previously only --env could, which left networkPolicy and the istio label unreachable. The provider side is #2407, directly below this PR, so the whole thing works as one command:

gym sandbox debug --config $AGENT --config $PROVIDER --task django__django-10973 \
    --command "pip install --no-input tabulate" \
    --env OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT=true \
    --env SSL_CERT_FILE=/opt/opensandbox/mitmproxy-ca-cert.pem \
    --metadata istio.io/dataplane-mode=none \
    --provider-option 'network_policy={"defaultAction":"allow","egress":[]}'

Verified against a live cell: the wheel was served by the in-cluster cache (cache=MISS on the first run, HIT on the second) with nothing in the container configured to use it.

Tests

260 tests across test_cli_sandbox.py, test_cli_main.py and test_sandbox_hooks.py — resolution at each layer, the precedence chain, --bare, --dry-run, trace shape, classification, cleanup on failure and interrupt, and a resources-server case alongside an agent case so generality is enforced rather than asserted.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-terryk-gym-sandbox-debug.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/about/release-notes
• nemo/gym/infrastructure/sandbox/debugging
• nemo/gym/infrastructure/sandbox
• nemo/gym/infrastructure/sandbox/opensandbox
• nemo/gym/reference/cli-commands
• nemo/gym/about/release-notes

[cmunley1]

Is as_json is only handled for list/dry-run, so gym sandbox debug --json still reaches this rich.print, so the documented --json | jq '.timing' path won’t produce parseable JSON?

[cmunley1]

If run_plan() never calls _renew() before sandbox.exec, can a long --timeout-total outlive the default 900s TTL - should we renew here like gym sandbox exec ?

eg
if you run gym sandbox debug --command "long thing" --timeout-total 1800 the sandbox may expire after default 900s.

[cmunley1]

This sounds like gym sandbox debug can reproduce any sandbox-backed rollout, but envs computed sandbox logic need to opt into sandbox_task - should we either update all the sandbox envs to work, like AnySWE/AnyTerminal, or document this as an opt-in debug contract?