[earakely-scale]

What

A computer env now composes on the put's sandbox provider, by default the env's (get_env_sandbox_provider()). With [sandbox] default = "local" nothing changes: the put builds with this machine's Docker as before. With a VM provider such as modal_vm, each form and arch resolves and builds in a builder VM of its own, which fetches the put's stored files by signed URL, takes the rest inline, pushes straight to the image store, and is always torn down.

Design doc: CUA Env M1.9: Compose on a Sandbox Provider (six choices, all decided: 1a 2a 3a 4a 5b 6 Modal VM). Issue: FD-4734 (absorbs FD-4364).

How

• Builder seam (env/envs/computer/utils/builder.py). One Builder interface with two sides:
  • LocalBuilder is the code that ran before, moved out of composer.py and packages.py.
  • VmBuilder runs the same steps in a VM through exec_script. Its shell steps live in one helper, builder_in_vm.sh, sent to the VM once and run by subcommand (files, link, check, runnable, download, installed, buildx, emulate).
  • The composer stages the build context on the putting machine as before, so the build key is computed there, the same whichever side builds. A VM gets the app and driver files from the store by signed URL, each checked against the sha256 the key used. It gets the generated files and a checkout's protocol source inline, in one tar.
• Which link builds (Builds). Per arch, the first link of the chain that runs it natively, else the first that emulates it.
  • A link builds when its sandboxes are containers on this machine's Docker (ON_THIS_MACHINE with no REACHES_HOST_AT), or when it gives VMs that reach the stores. That means an object store that signs URLs and a registry not on this machine; a VM on this machine reaches them anyway.
  • Each link that can't build says why, and the pair fails at the stage that needed it, naming every link. A pair whose images are reused needs no builder at all.
• Emulation in a VM (5b). The VM mounts its own binfmt_misc, then registers QEMU with a pinned tonistiigi/binfmt; the pair is recorded emulated. On Modal VM's kernel (7.2), a mount inside the privileged container gets an instance of its own, so the host mount is needed first.
• BuildKit in a VM. Modal VM's Docker 29 has no buildx plugin, so RUN --network=none couldn't build. A VM without one gets a pinned buildx release (v0.38.0), checked by sha256.
• Emulation can be turned off. --no-emulate on env build, bundle register and run, and emulate=False on a put, leave an arch no link runs natively skipped instead of emulating it. The reason names each link, the pair isn't a failure (env build exits 0), and it stays pending, so a later put on a provider that runs it natively, or with emulation on, builds it. On Modal VM for real, arm64 was skipped in 1 s and amd64 was ready in 2 min; re-registering with modal_vm,local then built arm64 natively here in 52 s.
• Time limits scale with emulation. A builder VM gets 2 h, or 6 h when it emulates the arch. The resolver gets 1 h, or 3 h under emulation, here or in a VM. Emulation runs some 10 to 20 times slower; the test app's emulated arm64 took 55 min.
• What emulation needs in a VM. A pair that builds emulated says on its progress line how to avoid it: --no-emulate, and adding local when this machine runs the arch. binfmt is pulled through mirror.gcr.io, with the same pinned digest, so Docker Hub's per-IP limit on anonymous pulls can't hit it.
• Images (3a). The VM logs in with the image store's credentials and pushes.
• The index is written over the registry API (registry_api.write_index) instead of docker buildx imagetools create. A put needs no local Docker at all, and the index is OCI unless every manifest is Docker's.
• Recorded, not keyed. A pair the put built records the provider that built it (builder). The key covers composer.py and builder.FORMAT, raised with a builder change that alters what an image holds, instead of the builders' source. So a change to how a build gets there doesn't rebuild every image.
• Failures.
  • A provider's own errors from a VM (a Modal ConflictError when it stops a sandbox, say) become a RuntimeError naming the VM. The pair fails alone instead of the whole compose.
  • A VM that won't terminate is logged and left to its timeout, since what it built is pushed.
• Progress while it runs. A put reports each pair's stages as they happen: where it builds, natively or emulated; resolving; building; pushing; probing. A stage that goes on is reported every 10 min, and each pair ends with its status and time. Lines go to an on_progress callback, and bundle register, run and env build print them without --verbose. This replaces the old on_build hook for computer envs, whose "building with docker" line was wrong for a remote build. A real bundle register --sandbox modal_vm,local:

  envs/kit: composing container/amd64, container/arm64 on ModalVmSandboxProvider or LocalSandboxProvider
  envs/kit: container/arm64: building natively with this machine's Docker
  envs/kit: container/arm64: building the image
  envs/kit: container/amd64: building natively in a ModalVmSandboxProvider VM
  envs/kit: container/amd64: building the image
  envs/kit: container/arm64: pushing the image
  envs/kit: container/arm64: probing on LocalSandboxProvider
  envs/kit: container/arm64: ready (52 s)
  envs/kit: container/amd64: pushing the image
  envs/kit: container/amd64: probing on ModalVmSandboxProvider
  envs/kit: container/amd64: ready (2 min 26 s)
  

• A reused computer env composes what it has left. When run or bundle register reuses an unchanged computer env, it composes the env's pending pairs on its provider, as agent-env env build does, writing no new version. Pending means failed, stopped, or built but unprobed though the provider runs that arch natively. A dry run says what it would compose (envs/kit: would compose container/arm64). bundle register goes on to write when anything is left, instead of stopping at "Nothing to write".
• An app's files under .agentenv/ are refused at put. That folder is the framework's: the app's own server keeps its socket and build settings there, and a build settings file there would reach the app unchecked. The build settings file the composer writes goes to a builder VM inline.
• One Composing carries the sandbox provider, the emulation choice and the progress callback through the puts, AuthoringContext, materialize, run and register.
• Plumbing.
  • compose, pending, ComputerEnv.put and ComputerAppEnv.put take sandbox_provider.
  • AuthoringContext.sandbox_provider and materialize(sandbox_provider=) pass it to bundle writes.
  • --sandbox on agent-env env build and agent-env bundle register. agent-env run --sandbox now composes there too.
  • Gate 3 probes on the put's provider.

Breaking

on_build is gone from ComputerEnv.put, ComputerAppEnv.put, composer.compose and AuthoringContext. Pass on_progress, a callable that takes each progress line (on_progress=lambda line: ...), instead.

Not in this PR

• --no-wait on a put (from the envs design's M4.3 scope): not needed for now, since a put shows its progress.
• Batching the per-file signed fetches; they run 8 at a time.
• Presigning S3 URLs against the regional endpoint (FD-4789). A bucket created minutes ago 307-redirects its global name and breaks the signature; found here, and the harness signs regionally.
• Passing the registry password to a VM over exec stdin instead of in the script text. pull_images and put_from_github work the same way, so all three would change together.

Testing

Unit: 11,929 passed; the one failure is the macOS-only Hermes pipe test, as on main. New tests:

• A fake VmSandbox provider composes the test env with a VM per arch:
  • QEMU is registered only for the emulated arch;
  • driver files are fetched by URL and lock files are stored from the VM;
  • this machine's Docker is never called;
  • a later local put reuses both images by key.
• A chain builds each arch on the link that runs it natively.
• A VM its provider loses fails only its pair and is torn down.
• Each refusal names every link: a store that signs nothing, a provider with no VM, no emulation.
• builder_in_vm.sh's subcommands run for real in bash: plain files listed but not links, runnables marked, links, a checksum mismatch named, a download's sha256. The fake VM runs every other line in bash too, faking only Docker and uploads.
• write_index: a token that pushes; an image that is an index keeps its manifests; Docker vs OCI list.
• env build --sandbox, and materialize passing the provider and progress through.
• A reused env composes its pending pairs on the run's provider, a dry run says so, and nothing is composed once everything is built.
• --no-emulate skips an arch no link runs natively, and a later env build emulates it; VM and resolver limits per arch.
• An app's .agentenv/ file is refused. An app stored before with one is built on a VM with the env's build settings; without the fix it gets the app's own copy.
• Progress: building natively vs emulated, a reused pair building nothing, a heartbeat on a stage that goes on, a failed pair's reason.

Integration, laptop:

• computer_app_test and computer_env_test: 6 passed (6 min), local Docker and localhost:5000.
• New builder_modal_vm_test (slow, remote_sandbox-gated): 1 passed in 7 min. On an amd64 Modal VM it resolves jq and six for arm64 under QEMU, stores a file off the VM through the local store, builds, and reads back what installed.

Remote end to end from the arm64 laptop, with S3, ECR and Modal VM (the CI project's keys, in-process):

Run	Result
Test app, env build --sandbox modal_vm	amd64 built in a VM, pushed and probed on Modal VM: ready in about 1.5 min. arm64 emulated in its own VM: 55 min (32 resolving, 23 building, all under QEMU), unverified.
env build --sandbox modal_vm,local	arm64 reused by key, probed natively on the laptop: ready in 33 s
task run --env-sandbox modal_vm / local	x86_64 / aarch64: every check passes (PPA package, download, install step on the build setting, secret, settings, launch)
Every-install-form app, bundle register --sandbox modal_vm,local	amd64 in a VM, arm64 natively here, about 3 min. Both arches pass: hello=version, a noble-backports package, gh from a repo with a binary keyring, zeek btest from a flat repo (suite="/"), netifaces built from sdist with gcc, jq per arch, run scripts as root and as the session user, an install step.
Same key, either side	That app re-put on modal_vm alone reused both images by key, the laptop-built arm64 included; nothing built
No Docker	A build-setting change put with no docker on PATH and DOCKER_HOST unreachable. amd64 ready, arm64 emulated unverified, lock files staged from S3 by signed URL, 20.5 min.
Teardown	All 12 VMs the logged runs made ended; no teardown warnings in any run

Found by running it, fixed here:

• Modal VM's binfmt_misc mount.
• Its missing buildx.
• A Modal ConflictError (Modal stopped the first arm64 VM 11 min into its resolve) aborted the whole compose.
• An isinstance check on the local provider broke the "ask the provider what it declares" rule.

The stopped VM's cause is unknown: memory isn't it (the rerun sampled about 400 MB of 8 GB), and the rerun completed.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issue was found.

Summary

Computer envs now build on the selected sandbox provider, using local Docker or a separate builder VM for each architecture.

• Computer images build on the put’s sandbox provider, with a VM when needed.
• Build, register, and run commands can choose where computer images are composed.
• The image registry receives a multi-architecture index without local Docker.
• Computer app puts reject files that land in the framework’s .agentenv/ folder.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Put, run, register, or env build] --> B[Select pending pairs]
  B --> C{Image already stored?}
  C -->|Yes| G[Probe on a native provider]
  C -->|No| D[Choose a builder for the architecture]
  D --> E[Resolve packages and build locally or in a VM]
  E --> F[Push image and tear down builder VM]
  F --> G
  G --> H[Record each result and write image index]

Loading

_{Reviews (9) · Last reviewed commit: "test(computer): where and how a pair bui..." · Reviewed by Greptile}