Skip to content

feat(computer): setup and check steps reach inside a computer, and computer app envs - #380

Merged
earakely-scale merged 14 commits into
mainfrom
edgararakelyan/fd-4366-m09-load_artifact-and-run_code-inside-the-computer
Oct 29, 2026
Merged

earakely-scale merged 14 commits into
mainfrom
edgararakelyan/fd-4366-m09-load_artifact-and-run_code-inside-the-computer

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

M0.9 and M1.1 of the CUA env epic (FD-4345): the framework's setup and check steps reach inside a computer, FD-4366, and the computer_app env type, deployable alone as a one-app computer, FD-4359. It follows the reviewed design, CUA Env M0.9 + M1.1: Code in the Computer, and Computer Apps, with its 15 choices confirmed. It builds on #369 (M0.7), now merged, and is rebased onto main (#374).

What's in it

M0.9: steps that reach inside the computer. Every setup and check step on a computer env now goes through its driver, as the session user, inside the desktop session; none runs on the sandbox's host or as root.

  • load_artifact of a file or file universe (choice 1a): a computer env overrides load_file_artifact_universe to send the files as data/add parts (env/envs/computer/utils/inside.py).
    • Each part's bytes come through a read grant when the object store's grants reach the computer and its network allows everything, else inline, so a computer with an allowlist, or none, still loads (choice 2). On local Docker the local store's grant server serves them.
    • Parts go in batches of at most 64 MiB, since the driver holds a call's files in memory; a bigger file goes alone.
    • destination_path is relative to the home, Downloads by default; an absolute path is refused, and the record keeps home-relative paths (choice 3).
  • Environment artifacts (choice 4): load_artifact refuses one aimed at an env whose type has no loader for it (load_environment_artifact, load_environment_universe_artifact), at save and before it loads, so a computer env gets a clear refusal instead of an AttributeError at run. Env itself is unchanged (decided in review, rather than base defaults that refuse). How an environment universe maps onto a computer's apps is designed with M1.4 (FD-4369).
  • run_code with env_id on a computer (choices 1b, 5, 6, 7): a fourth target, chosen when the env's card serves computer-exec/v1 run, so a plugin driver works too. A card serving run but not data/add or data/get is refused, naming what's missing.
    • The runner and input are staged under ~/.agentenv/run_code/<step id>/ with data/add, run with run, the result read back with data/get after the same 50 MiB size check as every target, and the folder removed, since the agent can read the home.
    • input["env"] holds the driver on loopback, http://127.0.0.1:<port> and its /mcp, which code inside the computer reaches with no network. The deploy record's metadata.computer now keeps the driver's port for this.
    • A timeout_seconds that leaves the runner less than 10 s of the card's max_timeout_s is refused, so the runner always stops the script itself; the client waits for the command's timeout plus a margin.
    • Each command has a fresh request_id and is sent once: a call whose connection to the driver breaks fails naming the cause, since a driver that restarted would run it a second time beside the first, which outlives the driver that started it. Staging first ends whatever still runs from the step's folder, so a retried step runs once.
  • run_container_unit_tests_verifier with env_id (choice 8): exclusive with sandbox_name/container_name, it reuses run_code's driver target. Commands run as the session user (user must be left unset), env_vars go into the run's env, and result_paths/reward_path are home-relative (an absolute one is refused) and read with data/get. The step writes user only when it's set, so a worker on an earlier release still reads a container step's user as root.
  • env_outcome_verifier (choice 9): a script whose verify takes an env_url keyword gets the env's address, which the protocol client takes as it is. The shipped examples stop stripping /mcp.

M1.1: computer apps.

  • ComputerAppEnvironment in agentenv_protocol.computer (choices 11, 13, 14): an AgentEnvEnvironment whose class declares what it is to a computer.
    • @computer_app(supports, permissions) declares what it is; its env.toml names it (environment_name) and the computer and driver it deploys alone on.
    • linux = Linux(files, python, services) is how a Linux computer installs it (feat(computer): the computer env type and the composer (container forms) #358's install format); launch = Launch(at, command, window, ready=Ready(...)); a skill as a file (skill = "skill.md") or an @agent_skill method, not both; @install_step(scope=...) and @ready_check.
    • A typed Settings (a SecretStr field is a secret), read as self.settings.
    • Launch, readiness, install steps and data hooks are declared and stored; M1.2, M1.4 and M1.5 act on them. permissions is stored, not checked.
    • The package now loads a driver's names (which need the computer extra) on first use, so an app's code imports with only the plain protocol package.
  • The computer env types share one package: the driver and app env types move into env/envs/computer/ as driver_env.py and app_env.py, beside env.py, and the modules all three share (install, gate, composer, boot, params and inside) move into env/envs/computer/utils/ (decided in review). The env type names don't change, and the plugin-API check reports no break.
  • _declare dispatches by base class: python -m agentenv_protocol.computer._declare reads a driver or an app, and says which; a computer_driver env naming an app's class, or the reverse, is refused.
  • What a class declares is typed (decided in review): agentenv_protocol.computer defines AppDeclaration and DriverDeclaration, which ComputerAppEnvironment.declaration() and _declare build, and which the computer_app and computer_driver envs store and read back as models rather than plain dicts. settings stays the JSON schema app_settings is checked against, and supports the platform predicates drivers already name. The stored JSON is unchanged, so a driver stored by a released version reads as before.
  • The computer_app env type (env/envs/computer/app_env.py): its env.toml takes module, environment_name, and the computer and driver to deploy it alone on, which a bundle builds with it. put reads the class, checks its install, and stores its folder as a file universe.
  • Deploying an app alone (choice 10B): a ComputerAppEnv is a ComputerEnv, so with a computer and a driver it is checked by gate 1 (its platforms and arches against the base's and the driver's, an install for the base's OS), composed and deployed like one, as a one-app computer.
    • The composer installs it under /opt/agentenv/apps/<name>/, built like the driver's layer: its files, a venv of its requirements, and its services as supervisord programs run as the session user, with its ENVIRONMENT_NAME.
    • The deploy record is keyed by the app's env id, and its card is named after it. An app that names no computer is refused at deploy with its own refusal, through the CLI too.
    • No launch, ready checks or data plane run yet. A computer env that lists apps is still refused, until M1.3.
  • app_settings (choice 12): a new env_params key, app_settings: {<app>: {<name>: value}}, checked at preflight against the full JSON schema of each app's Settings (jsonschema, now declared; it already came with mcp), required fields included, a secret (a SecretStr, optional or not) only as a bare secret:/env: reference. At boot each reference is resolved and the values reach the app's services as one JSON variable, AGENTENV_APP_SETTINGS, keyed by app.
  • Secrets on a computer (choice 15): the computers README now says that code anywhere in a computer can read a secret setting, since everything there runs as one user; FD-4612 keeps agents off computer-exec.

Where this differs from the design

  1. How an app's code avoids the computer extra (choice 14): frames.py imports results.py, which imports mcp, so making av and PIL lazy wouldn't have been enough. Instead agentenv_protocol.computer loads the driver, card and recording names on first use, and an app's import loads only app.py. Same outcome; recording.py and frames.py are untouched.
  2. An env target's verifier takes no user (choice 8 allowed the session user's name too): leaving it unset is the one way to run as the session user, so the step needn't look the name up.
  3. The run folder is removed only on a computer: the host, sandbox and agent targets keep theirs, as before.
  4. @computer_app names no computer or driver (choice 11 had the decorator name defaults; decided in review of Greptile's finding): a bundle reads references from env.toml alone, so a computer or driver only the class named was never built. env.toml is the one place for them, as for the app's name.
  5. A computer-exec run is never sent again (choice 6 resent a dropped call with its request_id): the driver starts each command in its own process group, so a driver killed outright leaves the command running, and the restarted one doesn't know the request and runs it again beside the first. A dropped call fails instead, and the step's retry stages afresh.

Review

Greptile's review of ec434c1 raised six threads. Five are fixed in 4df9a68: the verifier on a computer stops when it can't clear the reward file; an optional SecretStr is a secret at save and at boot; an allowlist network loads inline rather than through grants it may not reach; run_code keeps the runner's 10 s inside the computer's command limit; and app_settings are validated against the full schema. The sixth, that a computer and driver named only by @computer_app can't be resolved by a bundle (which reads references from env.toml alone), is decided in 1e5def9: @computer_app no longer names them (difference 4).

Greptile's review of 4df9a68 raised three more, on secrets in that schema check, fixed in a46f63c: a secret is checked only as a reference (or null where it's optional), never against the constraints on its value, which therefore never reaches a message, and a required one left out is named. a46f63c also makes a data/add, data/get or run whose connection to the driver breaks fail naming the env, the call and the cause (the EC2 chaos run below found a bare ReadError: ), and fixes test_artifact_staging.py's import of LocalSandbox, which #374 brought in from before #369 moved it, so main's unit tier fails on it.

Review of 1e5def9 raised seven more concerns. Three are fixed in 7a92051: a run whose connection breaks is never sent again (difference 5); an app deployed with no computer is refused, not an AttributeError; and the verifier writes user only when it's set. Decided to keep as they are: checks on a computer run as the user the agent ran as, with the model key in the verifier's environment; an inline load has no limit but its 64 MiB batches, for now; and ComputerAppEnv stays a ComputerEnv, which FD-4367 (M1.3) notes. A computer on Modal with a real object store is tested below.

Greptile's review of 7a92051 raised one: pkill -f read the step id as a regular expression, so starting read.v1 could end read-v1's runner. Fixed in 334fc5e: the pattern escapes what an extended regular expression gives a meaning to. In Ubuntu 24.04, pgrep -f with the escaped pattern finds only read.v1's runner, and with the old one both.

Greptile's review of c68ef97 raised one more, an inline 1500 limit on quoted stderr, now _ERROR_TAIL_CHARS (c5d3131); the other thread it listed was the secrets finding fixed in a46f63c. Also decided in review: ComputerAppEnv keeps its stored module, which the composer will read when the computer runs the class's own methods (M1.4, M1.5), and the declarations are typed (above).

Testing

  • Unit: 11,507 passed at 1e5def9, and 11,506 at a4cb89a (test_deep_nesting[json] fails on local Python 3.14, on main too, and a gateway timing test passed rerun alone); the fixture app's integration test passed again there. New tests cover the computer's loader (inline and grant parts, batching, a closed network, refused destinations), run_code on a computer (staging, loopback env, the timeout cap, a dropped call resent with its request_id while a new run gets its own, a failed script leaving no folder), the verifier's env_id target, env_url, the environment-artifact refusals, the app class and _declare, the app's bundle folder, its composition, its fit and its app_settings.
  • Integration (tst/integration/computer, laptop arm64, real Docker and the local registry), at 6ace0d5, and again at c68ef97 after the modules moved and at a4cb89a with the typed declarations: 21 passed and 1 skipped (the agent task, which needs a model endpoint). New: the desktop example's files task loads a CSV into Documents, reads it with run_code inside the computer while asking the driver for its card on loopback, and checks it with the verifier's env_id target; a closed-network computer loads a file inline and runs code that reaches its driver; a fixture app, deployed alone through agent-env run, comes up with its card named after it and its service running from a plain-protocol venv on the settings the deploy gave it, a secret resolved at boot.
  • Chaos on the laptop (arm64), 10/10, with a harness for the design's chaos row (each scenario also tears down with nothing failed):
    • a load larger than one batch: 150 MiB in three files lands whole, one data/add per file, through read grants on an open network (0.9 s) and inline on a closed one (1.7 s);
    • the driver killed during data/add: the step fails with RemoteProtocolError: Server disconnected without sending a response; supervisord has the driver back in 3 s, a retry loads every file whole, and no .part file is left in the home;
    • the driver killed during a run_code run (SIGKILL; rerun at 7a92051): the step fails with RemoteProtocolError: Server disconnected without sending a response, naming the env and the call, and isn't sent again; the killed run's runner and script were still going, and the retry ends them, runs once and leaves no staging folder;
    • a run_code past its timeout: reported as timed out after 5 s, with no process of it left running and its folder removed.
  • On EC2 (amd64, c6i.2xlarge): at 4df9a68, tst/integration/computer 21 passed and 1 skipped, and chaos 9/10: the driver killed during data/add failed the step with ReadError: , naming no cause. a46f63c fixes that; at a46f63c, 21 passed and 1 skipped, and chaos 10/10.
  • The agent run (agent-env run desktop --task agent, laptop only, the model key held in-process), at 6ace0d5: passed in 53.4 s. load_artifact put the figures in ~/Downloads through a read grant; Claude Code, given the computer's skill, made Documents/Reports and moved the file into it through the file manager (the prompt took 40.3 s), and the check confirmed it through the driver with env_url. Two sandboxes torn down, nothing left.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issue was found in the changes since the previous review.

Summary

This PR lets setup and check steps run inside computers through their drivers. It also adds computer apps that can deploy alone.

  • Computer task steps can load files and run checks inside the desktop session.
  • Computer apps can be declared, configured, and deployed as one-app computers.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  App[App class] --> Declare[Typed declaration]
  Declare --> Store[Stored app env]
  Store --> Compose[Compose base, driver and app]
  Compose --> Computer[Deployed computer]
  Steps[Setup and check steps] --> Driver[Computer driver]
  Driver --> Computer
  Driver --> Results[Files and check results]
Loading

Reviews (12) · Last reviewed commit: "refactor(computer): type what a driver o..." · Reviewed by Greptile

Comment thread src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py Outdated
Comment thread src/agent_env/env/envs/computer/gate.py Outdated
Comment thread src/agent_env/env/envs/computer_app.py Outdated
Comment thread src/agent_env/env/envs/computer/inside.py Outdated
Comment thread src/agent_env/task_step/task_steps/run_code.py Outdated
Comment thread src/agent_env/env/envs/computer/gate.py Outdated
Comment thread src/agent_env/env/envs/computer/gate.py Outdated
Comment thread src/agent_env/env/envs/computer/gate.py Outdated
Comment thread src/agent_env/env/envs/computer/app_env.py Outdated
earakely-scale and others added 5 commits October 28, 2026 18:41
…ach inside a computer

load_artifact writes files into a computer's home through its driver's data/add; run_code and
run_container_unit_tests_verifier run inside it as the session user through computer-exec/v1;
env_outcome_verifier hands scripts the env's base URL as env_url.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…app computer

ComputerAppEnvironment in agentenv_protocol.computer declares an app to a computer; a computer_app env
reads it at put through _declare, stores its folder, and deployed alone composes it onto its computer
and driver under /opt/agentenv/apps/<name>/, with env_params app_settings handed to it at boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The unit-test verifier on a computer stops when it can't clear the reward file, so one an agent left
  behind a read-only folder isn't graded.
- An optional secret setting (SecretStr | None) is a secret too, at save and at boot.
- Loads read through grants only on a computer whose network allows everything; an allowlist loads inline.
- run_code keeps the runner's 10 s inside the computer's command limit, so a script at the limit is stopped
  by its runner.
- app_settings are checked against the full schema of each app's Settings, required fields included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l names its cause

- A secret app setting is checked only as a reference (or null, where it's optional), never against the
  constraints on its value, which its value then never reaches; a required one left out is named.
- A data/add, data/get or run whose connection to the driver breaks fails naming the env, the call and
  the cause, which httpx often leaves empty (found by the chaos run on EC2: a bare "ReadError: ").
- test_artifact_staging imports LocalSandbox from local_docker, where #369 moved it; main's unit tier has
  failed on it since #374.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale force-pushed the edgararakelyan/fd-4366-m09-load_artifact-and-run_code-inside-the-computer branch from 4df9a68 to a46f63c Compare October 29, 2026 01:47
…does, whoever runs the tests

CodeBuild runs the unit tier as root, which a read-only folder doesn't stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator Author

EC2 rerun at a46f63c (amd64, c6i.2xlarge): tst/integration/computer 21 passed and 1 skipped (the agent task, which needs a model endpoint); chaos 10/10. The driver killed during data/add now fails the step with "env '@local/agentenv-framework/desktop/desktop': the connection to its driver broke during data/add, so some of the files it carried may be written, each whole (…)", and a retry loads every file whole. c01da9e makes the new reward-file test independent of who runs it (CodeBuild runs the unit tier as root, which a read-only folder doesn't stop).

earakely-scale and others added 4 commits October 28, 2026 20:58
….toml alone

@computer_app declares what an app is (supports, permissions); its env.toml names it and the computer and
driver it deploys alone on, which a bundle builds with it. A name only the class gave was never built,
since a bundle reads references from env.toml (decided in review of Greptile's finding).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…puter/

computer_driver.py and computer_app.py move into the computer package as driver.py and app.py, beside
env.py and the install, gate, composer and inside modules all three share. The env types keep their names;
the plugin-API check reports no break.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaving Env as it was

load_artifact refuses an environment artifact aimed at an env whose type has no loader for it, at save and
before it loads, instead of Env growing two methods that only refuse (decided in review). The CLI's hasattr
checks are main's again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… deploy, and old workers read user

- A computer-exec call whose connection breaks fails naming why and is never sent again: a driver that
  restarted would run it a second time beside the first, which outlives the driver that started it.
  run_code's staging ends whatever still runs from the step's folder first, so a retry runs once.
- The computer provider refuses an app that names no computer with its own refusal, not AttributeError.
- run_container_unit_tests_verifier writes user only when it's set, so a release before env_id targets
  still reads a container step's user as root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/task_step/task_steps/run_code.py Outdated
pkill -f reads its pattern as an extended regular expression, so a dot in a step id matched any character and
starting read.v1 could end read-v1's runner. The pattern now escapes what such an expression gives a meaning to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

earakely-scale commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

On Modal (modal_vm), with images in ECR and objects in S3, at 7a92051 (334fc5e changes only the pkill pattern), with the laptop as the Modal client: 10/10.

  • Composition: the desktop computer composes for amd64 on the arm64 laptop, goes to ECR, and gate 3 probes it on modal_vm: ready in 161 s, 71 s when the image is reused.
  • The files steps: deployed in 8–16 s. load_artifact puts the CSV in Documents through an S3 read grant; run_code reads it inside the computer and asks the driver for its card on loopback; the verifier's env_id command passes (score 1.0).
  • A load larger than one batch: 150 MiB in three files, one data/add each, all through S3 read grants, landing whole by sha256 in 7.5 s.
  • The driver killed (SIGKILL) during a run_code run: the step fails naming the env, the call and RemoteProtocolError; the killed run's runner and script were still going; the retry ends them, runs once and leaves no folder.
  • The fixture app alone: composed onto the computer and driver (149 s) and deployed in 18 s; its service runs on the settings the deploy gave it, the env: secret resolved at boot, on a computer whose card is notes.
  • Teardown: clean after every scenario, and all six sandboxes ended.

Two first-run failures were the harness's own and are fixed there: one check counted run_code's inline staging writes as the load's, and the orphan check ran pgrep -f inside sh -c, which matched the shell.

Not in this PR, tracked in FD-4621: agent-env run <bundle> keeps a bundle in this machine's stores, which Modal can't reach, and agent-env bundle register, the route to shared stores, refuses a bundle that names the shipped base computer or driver, as every computer bundle does. So the run above wrote the computer env, and copies of the base computer and driver, under ids of its own.

🤖 Generated with Claude Code

…keep what they share in utils

env/envs/computer/ now holds the three env types, env.py, app_env.py and driver_env.py; the modules they share
(install, gate, composer, boot, params and inside) move to env/envs/computer/utils/. Env type names don't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/task_step/task_steps/verifiers/run_container_unit_tests_verifier.py Outdated
earakely-scale and others added 2 commits October 28, 2026 22:57
…or quotes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agentenv_protocol.computer defines AppDeclaration and DriverDeclaration. ComputerAppEnvironment.declaration() and
_declare build them, and the computer_app and computer_driver envs store and read them back as models, not plain
dicts; the gate, the app's installs and its secret settings read their fields. settings stays the JSON schema
app_settings is checked against, and supports the platform predicates drivers name. The stored JSON is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 2602e1e into main Oct 29, 2026
27 of 28 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-4366-m09-load_artifact-and-run_code-inside-the-computer branch October 29, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant