fix(sandbox): stop creating an empty venv that shadows the image's packages

[apetraru-uipath]

Problem

SandboxConfig.python defaults to a PythonEnvConfig() instance, and setup() gated venv creation on that object alone:

if self.config.python:                    # instance → always truthy
    self._setup_virtualenv()              # `uv venv .venv` → empty, no pip inside
    if self.config.python.env_packages:   # [] → falsy → nothing installed
        self._install_packages()

So every task got a venv with nothing in it.

An empty venv is not neutral. uv venv puts no pip in it, so inside a task image that provisions packages globally the two halves of the toolchain disagree. Measured inside skills-image:latest:

### WITHOUT a venv (post-fix)
import langchain OK -> /usr/local/lib/python3.13/site-packages/langchain/__init__.py

### WITH an empty venv (pre-fix)
which python       : /tmp/probe/.venv/bin/python
which pip          : /usr/local/bin/pip          <- image global; uv venv installs none
pip list langchain : langchain 1.3.14            <- "installed!"
python import      : ModuleNotFoundError: No module named 'langchain'
venv site-packages : _virtualenv.pth _virtualenv.py

venv/bin was also prepended to PATH for the agent and for every run_command criterion, so criteria were graded under that same empty interpreter.

Measured cost

Run 2026-09-10_04-18-49, task skill-agent-guardrail-coded-escalation-smoke (claude-sonnet-5, docker driver):

• The agent finished the guardrail correctly at turn 19.
• Turns 22–31 went to chasing the pip-vs-python contradiction, then uv sync to repair the venv.
• Turns 32–44 went to a uip codedagent init scaffolding detour.
• It hit max_turns: 40 at turn 41 — before writing the app resource into bindings.json.
• Scored 0.80 with correct code; only the weight-3 bindings criterion failed.

The same task passes in 19–21 turns on the host driver, where there is no global environment for the empty venv to contradict.

Fix

Nothing to install means nothing to create. Gate creation on the package list, not on the config object.

Validation

• make verify: format, lint, pyright, 507 lint-rule tests, 4975 tests pass. Three failures in test_reports_stats_nonfinite.py are pre-existing on clean main in this environment ('float' object has no attribute 'numerator') and unrelated.
• New regression test test_default_python_config_creates_no_venv_when_nothing_to_install fails before the change, passes after.
• End-to-end, per the Run Skill Smoke Tests runbook — skill-agent-guardrail-coded-escalation-smoke, claude-code + claude-sonnet-5, --driver tempdir: SUCCESS, score 1.000, 25 turns, all 6 criteria at 1.0, no .venv created.

Blast radius

12 tasks in the skills suite set env_packages and keep their venv unchanged. The task YAMLs that mention .venv do so only as -not -path '*/.venv/*' exclusion filters, which are unaffected by its absence.

Two existing tests asserted the old behavior (a venv from a config that requested no packages) and are updated. _setup_template already runs before venv creation, so the template test's "venv survives the copy" assertion was ordering-redundant.

adopt() is deliberately left alone: it still discovers a venv the agent created itself, so detached grading keeps matching the environment the agent actually ran under.

🤖 Generated with Claude Code

[github-actions]

Claude finished @apetraru-uipath's task in 1m 37s —— View job

---

PR Review in Progress

Todo List:

• Read review guidelines in .github/code_review.md
• Read CLAUDE.md for project conventions
• Examine full PR diff (git diff origin/main...HEAD)
• Review each changed file with full context:
  • docs/TASK_DEFINITION_GUIDE.md
  • src/coder_eval/models/sandbox.py
  • src/coder_eval/sandbox.py
  • tests/test_sandbox.py
  • tests/test_sandbox_templates.py
• Perform cross-file consistency checks
• Analyze what's missing or not covered
• Post comprehensive review feedback

[uipreliga]

Review: coder_eval — pr:162

Scope: pr:162 · branch fix/no-empty-venv-shadowing-image-packages · 9b15512 · 2026-09-10T15:06Z · workflow variant

Change class: complex — changes the semantics of a public task-YAML config field (sandbox.python) and the default sandbox provisioning for every task, so correctness requires reasoning about interpreter/PATH blast radius rather than reading one condition

The codebase is in strong shape — clean style, full type coverage, no security or architecture findings, and a mature custom-lint culture that keeps its own past defects from returning — but this change to the sandbox venv gate (src/coder_eval/sandbox.py:417) left setup and Sandbox.adopt (sandbox.py:359) disagreeing, so the same trajectory can score 0.000 under run and 1.000 under execute + evaluate --in-place, and it removed the harness python-on-PATH guarantee that 13 in-tree run_command criteria still depend on; fix the two verdict-affecting parity gaps and the stale rationale text and the change is sound.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.9 / 10	0	0	0	1	New caller-side gate makes _install_packages' three-clause early-return unreachable (the module's only uncovered line)
2. Type Safety	9.9 / 10	0	0	0	1	New venv gate reads one field of PythonEnvConfig as the whole model's activation signal, so a second field would be silently ignored with no type error
3. Test Health	9 / 10	0	0	2	0	Test coverage does not follow the new venv semantics: the python: null opt-out arm is unexercised, the new tests assert the artifact (no .venv) rather than the effect, and venv creation + install is now reachable only from one network-dependent test
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	8.4 / 10	0	1	1	1	Venv gate desynchronized: setup requires env_packages while Sandbox.adopt still keys on config.python alone, so an agent-created .venv lands on the criterion PATH for an in-place grade but never during run (same trajectory, different verdict) — stale parity comment, and no test pins either half
7. API Surface & Maintainability	10 / 10	0	0	0	0	—
8. Evaluation Harness Quality	8 / 10	0	1	2	0	Removing the default venv drops the harness's python-on-PATH guarantee for run_command criteria: 13 in-tree bare-python criteria now depend on ambient host resolution, and the criterion interpreter/site-packages vary per host and per invocation

Overall Score: 9.4 / 10 · Weakest Axis: Evaluation Harness Quality at 8 / 10
Totals: 🔴 0 · 🟠 2 · 🟡 5 · 🔵 3 across 8 axes.

Blockers

1. [Axis 6] Venv gate desynchronized: setup requires env_packages while Sandbox.adopt still keys on config.python alone, so an agent-created .venv lands on the criterion PATH for an in-place grade but never during run (same trajectory, different verdict) — stale parity comment, and no test pins either half (src/coder_eval/sandbox.py:417) — The gate if self.config.python and self.config.python.env_packages: (sandbox.py:417) leaves self.venv_dir is None for every task whose python: block is present but lists no packages — which is the default (default_factory=PythonEnvConfig) and three in-tree tasks' explicit python: {} (tasks/token_check.yaml:7, tasks/internal/session_resumption.yaml:20, tasks/python_cli_simulated_judged/echo_simulated_judged.yaml:34). Nothing re-discovers a venv after the agent phase: venv_dir is assigned only at sandbox.py:167 (init None), :362 (adopt), :830 (_setup_virtualenv), :1590 (preserve remap) and :1663 (cleanup) — I grepped venv_dir repo-wide and path_utils.py:52 is the only other hit. So when the agent builds its own .venv (the normal uv venv && uv pip install ... move, and now the only way to get one, since uv pip install refuses to run without a virtualenv), _build_run_command_env at sandbox.py:1157-1159 (if self.venv_dir: env["VIRTUAL_ENV"] = ...; env["PATH"] = f"{self._venv_scripts_dir}{os.pathsep}{env['PATH']}") skips both, and every run_command criterion is graded under the ambient interpreter.

Sandbox.adopt still does the opposite, on a gate the PR did not update: if self.config.python: (sandbox.py:359) → candidate = self.sandbox_dir / VENV_DIRNAME; if candidate.is_dir(): self.venv_dir = candidate. So evaluate <run_dir> (in-place is the default for a run dir) and run --resume's regrade_in_place (which calls sandbox.adopt at orchestration/regrade.py:1180) put the agent's .venv/bin on the criterion PATH, while the run that produced the trajectory did not. A criterion like python -c "import pytest" therefore scores 0.000 under run and 1.000 under execute + evaluate --in-place for byte-identical agent output — and run's 0.000 reads as an agent failure (ModuleNotFoundError) with no diagnostic.

The test suite now pins both halves of the contradiction for the same config: tests/test_sandbox_adopt.py:75-82 builds _sandbox(python={"env_packages": []}), creates .venv/bin, and asserts sandbox.venv_dir == ws.resolve() / ".venv" under the docstring "Discovery keeps run_command criteria on the agent's PATH" — the exact property this PR removes from the run path — while the PR's new tests/test_sandbox.py:1337 asserts sandbox.venv_dir is None for that same config. Neither test fails, so nothing guards the parity.

Also note adopt's own rationale at sandbox.py:353-358 is now stale in its premise: it says it is "Gated on config.python for the same reason setup is", which stopped being true at line 417; and it warns that discovery "would also let an agent shadow binaries by writing .venv/bin/ into its own workspace" — which is now precisely what a non-null python: with empty env_packages permits at grade time, since no harness venv exists to have legitimately created that directory.

Fix: pick one PATH and make both paths agree. Either (a) make the run path discover a post-agent .venv under the same condition adopt uses, so the criteria run under the PATH the agent actually built — this is what models/sandbox.py:434 already promises the reader ("Set to null in YAML (or None in Python) to also opt out of adopting a venv the agent created itself", which is only true of the in-place path today); or (b) tighten adopt's gate at sandbox.py:359 to if self.config.python and self.config.python.env_packages: so neither path adopts an agent-written venv, and correct the models/sandbox.py:432-435 description accordingly. Whichever is chosen, add a parity test asserting setup-then-agent-created-.venv and adopt produce the same venv_dir/PATH for the same SandboxConfig, and reconcile tests/test_sandbox_adopt.py:75-82 with tests/test_sandbox.py:1337 so the two can never again assert opposite things.
2. [Axis 8] Removing the default venv drops the harness's python-on-PATH guarantee for run_command criteria: 13 in-tree bare-python criteria now depend on ambient host resolution, and the criterion interpreter/site-packages vary per host and per invocation (src/coder_eval/sandbox.py:417) — The changed line is if self.config.python and self.config.python.env_packages: (sandbox.py:417). Because SandboxConfig.python defaults to a PythonEnvConfig() instance with env_packages: list[str] = ... default_factory=list (models/sandbox.py:429-430, 57), and grep -rn "env_packages" tasks/ experiments/ templates/ returns exactly two hits (tasks/fibonacci_with_template.yaml:16, tasks/inline_starter_example.yaml:13), EVERY other tempdir task now gets venv_dir is None, no VIRTUAL_ENV, and no venv bin/ prepended in _build_run_command_env (sandbox.py:1157-1159: if self.venv_dir: env["VIRTUAL_ENV"] = ...; env["PATH"] = f"{self._venv_scripts_dir}{os.pathsep}{env['PATH']}"). That env is the ONLY one Sandbox.run_command builds (sandbox.py:1261), so it governs every run_command criterion plus pre_run/post_run.

Two consequences, both verified by running the real code at this HEAD:

(a) ISOLATION LEAK. Post-PR, with the repo venv active: sb=Sandbox(SandboxConfig(driver="tempdir"),task_id="probe"); sb.setup(); sb.run_command('python -c "import sys; print(sys.executable)"') -> (0, '/Users/religa/src/coder_eval/.venv/bin/python\n', ''), and python -c "import coder_eval" -> (0, 'leaked ...', ''). Simulating the pre-PR gate (sb.setup(); sb._setup_virtualenv()) on the same machine: sys.executable -> .../coder_eval_probe_old_.../.venv/bin/python and import coder_eval -> ModuleNotFoundError. So criteria that used to run under an isolated per-task interpreter now run under the grader's interpreter and site-packages. A criterion's outcome now depends on how the operator launched coder-eval (uv run / activated venv vs .venv/bin/coder-eval) and on what the grader happens to have installed — the Axis-8 scoring-correctness class (identical agent output, different verdict per host/invocation).

(b) MISSING python. The PR deletes the assertion that the harness supplies an interpreter (tests/test_sandbox.py old lines 64-67 asserted (venv_dir / "bin" / "python").exists()). Bare python is not universally present: env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin sh -c 'command -v python' returns rc=1 on this darwin host (only /usr/bin/python3), as on Debian slim without python-is-python3. Nine in-tree criteria invoke bare python in tasks that declare NO sandbox: block at all (so they took the default instance): tasks/hello_date.yaml:26 command: "python app.py" (tags smoke-pass at :4, and the Windows smoke task at .github/workflows/pr-checks.yml:431), tasks/test_sandbox.yaml:13-15 command: "python --version" with description: "Python should be available in the sandbox." — a criterion whose entire purpose was the guarantee this line removes — tasks/internal/session_resumption.yaml:48 python -c "from farewell import farewell; ..." (whose python: {} at :20 is now inert), tasks/agents/claude_hello_world.yaml:37, tasks/agents/codex_hello_world.yaml:39, tasks/agents/antigravity_hello_world.yaml:36, tasks/agents/codex_string_utils.yaml:50/55/60, tasks/opencode_smoke_test.yaml:35, tasks/pi_smoke_test.yaml:36. On such a host these go from exit 0 to 127 for unchanged agent output.

Fix: keep the fix (an empty venv IS harmful) but restore the interpreter guarantee rather than delegating it to ambient PATH. Options: (1) keep creating the venv but with --system-site-packages so it neither shadows the image's globals nor leaks the grader's, (2) synthesize a minimal bin/ shim dir (already the mechanism mock_path_dirs/record_cli use) that maps python -> the resolved python3, and prepend it in _build_run_command_env, or (3) accept ambient resolution but migrate the nine criteria above to python3 in the same PR and say so in docs/TASK_DEFINITION_GUIDE.md. Harness-loop candidate: a CE rule scanning tasks/ for a run_command/command_pattern invoking bare python — mechanically detectable, and it would have named all nine sites.

Non-blocking, but please consider before merge

1. [Axis 3] Test coverage does not follow the new venv semantics: the python: null opt-out arm is unexercised, the new tests assert the artifact (no .venv) rather than the effect, and venv creation + install is now reachable only from one network-dependent test (src/coder_eval/models/sandbox.py:434) — The PR adds a user-facing behavioral claim to the SandboxConfig.python description: "None in Python) to also opt out of adopting a venv the agent created itself." (models/sandbox.py:434). That opt-out is implemented only by the false arm of if self.config.python: in Sandbox.adopt (sandbox.py:359), and the routed coverage shows that arm is never exercised (359->364 partial branch; tests/test_sandbox_adopt.py covers only python={"env_packages": []} at line 79 and the no-venv case at line 85). This is the repo's most-cited miss shape — new documented behavior with no direct assertion. Add one test to tests/test_sandbox_adopt.py: workspace containing .venv/bin, Sandbox(SandboxConfig(driver="tempdir", python=None)), adopt(ws) -> assert sandbox.venv_dir is None and "VIRTUAL_ENV"/the venv bin dir absent from _build_run_command_env(). Shares a root cause with finding #1 (the adopt-side gate) but is a distinct missing test and a distinct in-scope anchor.
2. [Axis 3] test_template_ignores_venv's only .venv assertion is now dead code — if venv_bin.exists(): (tests/test_sandbox_templates.py:187) can no longer be true (tests/test_sandbox_templates.py:187) — test_template_ignores_venv (tests/test_sandbox_templates.py:161) uses a SandboxConfig with no python: key, so post-PR no .venv is created — the same fact the PR's own rewrite asserts eleven lines earlier in this file ( assert not (sandbox_path / ".venv").exists(), line 40, same default config). Its only .venv assertion is therefore dead: if venv_bin.exists(): (line 187) can no longer be true, so assert not (venv_bin / "python").exists() or (venv_bin / "python").is_symlink() (line 189) never runs and the test's stated purpose ("Test that .venv directory is skipped", line 162) is unguarded — its only live assertion is main.py existing. The comment justifying the conditional is now stale too: # (sandbox creates its own .venv) (line 185). The PR actually makes a STRONGER assertion available: replace lines 184-189 with an unconditional assert not (sandbox_path / ".venv").exists() # template's .venv must not be copied, and none is created, which proves the ignore-pattern behavior outright instead of via a symlink proxy.
3. [Axis 6] Inject-mode containers keep a python/pip split after the PR: the kit venv (/opt/coder-eval/venv, built without --seed) supplies python while pip falls through to the task image's global pip — and since injected images ship no uv, the pre-PR venv.create(with_pip=True) fallback had kept that pair matched (src/coder_eval/sandbox.py:413) — The new comment claims the defect is the sandbox venv: "inside a task image that provisions packages globally, python resolved to the venv (import fails) while pip resolved to the image's global pip (reports the package present)" (sandbox.py:412-414). Removing the sandbox venv closes that on the framework image, whose docker/Dockerfile installs with uv pip install --system (line 93) so python and pip are both the image's global pair.

It does not close it on the injected-runtime path, which docs/DOCKER_ISOLATION.md:130-152 documents as the supported way to keep a task's own base image (the Fedora/dnf example). There, docker/coder_eval_runtime_entrypoint.sh:18 runs export PATH="/opt/coder-eval/venv/bin:/opt/coder-eval/node/bin:${PATH}" before exec coder-eval _run-task-internal, and _build_run_command_env starts from os.environ.copy() (sandbox.py:1154), so every criterion subprocess inherits that prefix. /opt/coder-eval/venv is built by docker/Dockerfile.runtime:47 as uv venv --python ${PYTHON_VERSION} /opt/coder-eval/venv — no --seed, and lines 63-65 install into it via uv pip install --python /opt/coder-eval/venv/bin/python, never seeding pip. So after this PR, in an inject-mode container: python resolves to the kit's venv (holding coder-eval's own deps — pydantic, anthropic — and none of the task image's dnf/pip packages), while pip falls through to the image's global pip. That is the identical contradiction quoted above, one directory up, and the agent still burns its turn budget on it.

The harness neither fails loud nor warns: the criterion simply exits non-zero and is scored as an agent failure. Fix: either seed the kit venv (uv venv --seed, or ship a pip shim inside /opt/coder-eval/venv/bin) so python and pip at least agree, or stop prepending the kit venv to the task-visible PATH in docker/coder_eval_runtime_entrypoint.sh:18 (resolve coder-eval by absolute path instead, as docker/Dockerfile.runtime:88 already does for the sanity check). At minimum, correct the comment at sandbox.py:411-416 so it does not assert a root cause the inject path contradicts, and log a warning when the resolved python lives in a venv the sandbox did not create.
4. [Axis 8] New rationale comment (sandbox.py:415-416) claims the venv bin/ was prepended to the AGENT's PATH; no code path ever did that — only run_command criteria (src/coder_eval/sandbox.py:415) — sandbox.py:415-416 states "and / the venv's bin/ was prepended to PATH for the agent and for every / run_command criterion." The agent half is false. The only site that exports VIRTUAL_ENV or prepends the venv bin/ is _build_run_command_env (sandbox.py:1157-1159), and its sole caller is Sandbox.run_command (sandbox.py:1261) — criteria plus pre_run/post_run. The agent's PATH is built independently: the orchestrator passes only env_path_prepend = [str(p) for p in self.sandbox.resolved_mock_path_dirs] (orchestrator.py:1692), and the agent applies base_env["PATH"] = os.environ["PATH"] then prepends just that list (agents/claude_code_agent.py:780-785). No venv, ever.

The same false mechanism is repeated in the new test docstring (tests/test_sandbox.py:1319-1329: "That empty venv then shadowed a fully provisioned environment: inside a task image with packages installed globally, python resolved to the venv (import fails) while pip resolved to the image's global pip ... Agents burned their turn budget on that contradiction"). That contradiction is real for a run_command/pre_run subprocess, but an agent could not have hit it via PATH; the agent-visible half comes from uv/tooling auto-discovering a .venv in cwd (e.g. uv pip install X targets ./.venv, then the global python cannot import X), which is a different mechanism and is NOT fixed by removing the harness's venv when the agent creates its own.

Fix: correct both texts to name the surface that actually carries the venv (run_command criteria and pre_run/post_run, via _build_run_command_env), and state the agent-side mechanism as cwd .venv auto-discovery if that is the motivating symptom. This is the entire recorded justification for a behavioral gate; leaving it wrong sends the next reader looking at the agent env builder for a defect that lives in the criterion env builder.
5. [Axis 8] Behavior change not rippled to the authoring surfaces: stale venv prose in TASK_DEFINITION_GUIDE (line 519 + python: {} blocks), the published task SKILL.md, the three-state python: semantics, plus a stale adopt comment and test docstring (docs/TASK_DEFINITION_GUIDE.md:524) — The PR edits line 524 to python: # Python env config (a venv is created only if env_packages is non-empty) but leaves line 519 reading "The sandbox block is optional. When omitted, it defaults to driver: \"tempdir\" with standard Python environment." — which is now the opposite of the truth: an omitted block provisions no Python environment at all. Two adjacent lines of the same section now contradict each other.

PythonEnvConfig carries exactly one field (env_packages, models/sandbox.py:52-57), so post-PR python: {} and python: null are indistinguishable on the run path (both skip _setup_virtualenv); they differ only in Sandbox.adopt (sandbox.py:359). The schema therefore has two spellings for "no venv" and none for "a venv without packages", and three in-tree tasks still carry the now-inert knob: tasks/token_check.yaml:7 python: {}, tasks/internal/session_resumption.yaml:20 python: {}, tasks/python_cli_simulated_judged/echo_simulated_judged.yaml:34 python: {}. A reader of those files reasonably concludes the sandbox provisions Python for them; it does not.

Fix: (a) reword line 519 to say the default provisions no venv and that env_packages is what creates one; (b) drop the three dead python: {} blocks (or convert them to python: null and say why); (c) consider making python default to None in parity with node: NodeEnvConfig | None = Field(default=None, ...) (models/sandbox.py:437-439), which would collapse the two equivalent spellings and close the adopt asymmetry in finding #2 at the same time. Ripple also missed: tests/test_glob_paths_in_file_criteria.py:132 still documents ".venv is created inside the root for any task with a python: block", the same claim the PR corrected at guide line 715.

Nits

1. [Axis 1] New caller-side gate makes _install_packages' three-clause early-return unreachable (the module's only uncovered line) (src/coder_eval/sandbox.py:419) — _install_packages() now has exactly one caller (sandbox.py:419, reached only under if self.config.python and self.config.python.env_packages: at 417) and _setup_virtualenv() on the line above unconditionally assigns self.venv_dir = self.sandbox_dir / VENV_DIRNAME (sandbox.py:830), so every clause of its own guard is now unreachable: if not self.config.python or not self.config.python.env_packages or not self.venv_dir: / return (sandbox.py:847-848). The routed coverage run confirms line 848 is uncovered. The guard is not pure dead weight — it also narrows self.config.python for the *self.config.python.env_packages splat at sandbox.py:859 — so replace the silent early return with the invariant it has become, e.g. assert self.config.python and self.config.python.env_packages and self.venv_dir, "caller gates on env_packages; _setup_virtualenv sets venv_dir", mirroring the existing assert scripts_dir is not None # guaranteed by venv_dir guard above on line 852. That keeps the narrowing, removes the unreachable branch, and makes the caller/callee contract explicit.
2. [Axis 2] New venv gate reads one field of PythonEnvConfig as the whole model's activation signal, so a second field would be silently ignored with no type error (src/coder_eval/sandbox.py:417) — The gate is if self.config.python and self.config.python.env_packages: (sandbox.py:417). It substitutes truthiness of ONE field for "this config asks for a Python environment". That is exactly right today, because PythonEnvConfig has exactly one field — env_packages: list[str] = MergeField(strategy="replace", default_factory=list, description="Packages to install") (models/sandbox.py:57) — but the coupling is invisible to the type checker: adding a second, venv-requiring field later (a python_version, a requirements_file, a system_site_packages flag) leaves this gate reading env_packages only, so a task that sets the new field alone gets no venv and pyright reports nothing. Consider putting the predicate on the model where the fields live — e.g. a PythonEnvConfig.wants_venv property (or __bool__) returning bool(self.env_packages) — and gating on if self.config.python and self.config.python.wants_venv:, so a future field is folded in by construction rather than by a reviewer remembering this call site. Same for the parallel predicate at orchestration/regrade.py:299 (if sandbox.python is not None and sandbox.python.env_packages:), which today duplicates the same two-part test. No current bug — PythonEnvConfig is single-field, so the gate and the model agree at this commit.
3. [Axis 6] The comment's uv venv (no pip) premise holds for only one of the two venv shapes _setup_virtualenv can produce, is false on the venv.create(with_pip=True) fallback, and is untested (src/coder_eval/sandbox.py:411) — The new comment states the mechanism as fact: "An empty venv is not neutral: uv venv puts no pip in it" (sandbox.py:411-412). But _setup_virtualenv can produce the opposite shape without saying so: sandbox.py:833-843 runs subprocess.run(["uv", "--version"], check=True, ...) and, on except (subprocess.CalledProcessError, FileNotFoundError):, silently does import venv / venv.create(self.venv_dir, with_pip=True) — a venv that does have pip. Degrading rather than failing is the right call here (a contributor without uv must still be able to run tasks), but the degradation is invisible: there is no logger.warning, nothing is recorded in environment_info, and coverage confirms lines 839-843 are unexercised, so the shape a given host produced is not knowable after the fact.

This matters now because the PR pins its rationale to the no-pip shape. The fix at line 417 is correct under both shapes (an empty venv with pip still shadows the image's global site-packages for python while providing nothing), so the code is fine — the comment is what is shape-specific.

Fix: add a logger.warning("uv unavailable; created %s with stdlib venv (pip seeded)", self.venv_dir) in the except at sandbox.py:839 so the divergence is surfaced, and reword sandbox.py:411-412 to state the shape-independent reason (an empty venv shadows the image's interpreter while providing nothing) rather than a property only the uv path has. The same wording appears in the new test's docstring at tests/test_sandbox.py:1322-1323 ("a uv venv with no packages and no pip in it") and should be reworded with it.

What's Missing

Parallel paths:

• 🟠 _setup_tempdir's gate was tightened to config.python and config.python.env_packages (sandbox.py:417) but the sibling gate in Sandbox.adopt (sandbox.py:359) still keys on config.python alone, so the detached/in-place grading path adopts an agent-written .venv onto the criterion PATH that the run path never had — the two paths must be changed together or one is wrong. (trigger: src/coder_eval/sandbox.py) (restates: Axis 6: Venv gate desynchronized between setup and adopt)
• 🟡 The .venv-is-harness-output assumption baked into the copy/capture machinery was not revisited: _WORKSPACE_CAPTURE_IGNORE lists .venv under "Noise: Python / JS build infra" (sandbox.py:60) with the docstring calling it "sandbox-created bulk" (sandbox.py:1617), so the docker path's capture_to now silently drops an agent-created venv out of artifacts/ — after which evaluate over that artifacts dir adopts nothing while an in-container grade over the live workspace still adopts it. Same stale premise in cli/evaluate_command.py:59,267, orchestration/regrade.py:1112 and isolation/docker_runner.py:1667. (trigger: src/coder_eval/sandbox.py)
• 🟡 The guide got two prose fixes but the published plugin surface authors actually copy did not: plugins/coder-eval/skills/task/SKILL.md:154 still emits python: {} # a venv with no extra packages; add env_packages if needed, which is now false in both halves (no venv is created, and {} is indistinguishable from omitting the block). (trigger: docs/TASK_DEFINITION_GUIDE.md) (restates: Axis 8: Behavior change not rippled to the authoring surfaces)
• 🔵 SandboxConfig.python keeps default_factory=PythonEnvConfig while its now-behaviourally-identical twin node is NodeEnvConfig | None = None (models/sandbox.py:437-439); bringing python to the node default in the same PR would collapse the two spellings of "no venv" and close the adopt asymmetry by construction rather than by a second gate. (trigger: src/coder_eval/models/sandbox.py) (restates: Axis 8: Behavior change not rippled to the authoring surfaces)
• 🔵 The node twin is the precedent this change followed (if self.config.node and self.config.node.env_packages: at sandbox.py:422 with a matching callee guard at :875), so whatever is done about python's now-unreachable callee guard (sandbox.py:847) should be applied to _install_node_packages in the same pass — otherwise the two symmetric gates diverge in style immediately after being made symmetric in behavior. (trigger: src/coder_eval/sandbox.py) (restates: Axis 1: New caller-side gate makes _install_packages' early-return unreachable)

Tests:

• 🟡 No test covers the python: null arm the PR newly documents at models/sandbox.py:434 ("also opt out of adopting a venv the agent created itself") — the false arm of if self.config.python: in adopt (sandbox.py:359) is an unexercised branch across the whole suite. (trigger: src/coder_eval/models/sandbox.py) (restates: Axis 3: Test coverage does not follow the new venv semantics)
• 🟠 No parity test pins setup against adopt for one SandboxConfig: tests/test_sandbox_adopt.py:80 asserts venv_dir == ws/.venv and the PR's new tests/test_sandbox.py:1337 asserts venv_dir is None for the equivalent config, and both pass — the suite now encodes the divergence instead of forbidding it. (trigger: src/coder_eval/sandbox.py) (restates: Axis 6: Venv gate desynchronized between setup and adopt)
• 🟡 The gate's positive arm is now reachable only from test_sandbox_with_packages (tests/test_sandbox.py:482), which shells a real uv pip install requests; the node twin already has the hermetic form (patch.object(sandbox, "_install_node_packages"); mock_install.assert_not_called(), tests/test_node_env_config.py:74-82) — add the mirrored mocked assertion for both arms so the branch is pinned offline. (trigger: tests/test_sandbox.py) (restates: Axis 3: Test coverage does not follow the new venv semantics)
• 🟡 The new test asserts the artifact (not (sandbox_dir / ".venv").exists(), venv_dir is None) but never the effect the change exists for: no assertion that _build_run_command_env() now carries no VIRTUAL_ENV and no .venv/bin PATH prefix — the sibling pattern already used at tests/test_sandbox.py:1313 for REFERENCE_DIR. (trigger: tests/test_sandbox.py) (restates: Axis 3: Test coverage does not follow the new venv semantics)
• 🟠 Nothing tests the motivating scenario end to end. The defect being fixed is image-shaped (a task image that pip installs globally — tasks/samples/skillsbench/*/environment/Dockerfile:9,13), and no test or CI step asserts the post-fix invariant on that path: after setup, python -c "import <globally installed pkg>" must succeed inside such an image. tests/test_image_from_dockerfiles.py is where that guard belongs; without it the fix's own claim is unverified in CI, and the two in-scope unit tests would stay green if the venv came back on the docker path only. (trigger: src/coder_eval/sandbox.py)

Downstream consumers:

• 🟡 Verdicts now depend on which python the grading host happens to expose, yet nothing records it: environment_info stamps command_base_path (orchestrator.py:1807) and installed_tools (:1735) but has no field for the interpreter/venv a criterion resolved, so a host-to-host or run-vs-regrade divergence is unattributable from run.json, the reports or the evalboard. Add a stamp (e.g. criterion_python) with a real writer — CE054 requires one for any key that is read. (trigger: src/coder_eval/sandbox.py)
• 🟡 VIRTUAL_ENV is no longer exported for the default config, and uv consumes that variable: criteria/uipath_eval.py:67 builds uv run uipath eval … through Sandbox.run_command, and any uv run / uv pip install in a run_command criterion previously resolved into <sandbox>/.venv and now resolves (or creates) whatever uv picks. No in-tree task exercises it, so nothing fails locally — the UiPath eval path is an external-suite consumer and the PR does not say what it resolves to now. (trigger: src/coder_eval/sandbox.py)

Daily/nightly:

• 🟠 Blast radius on the production run path is unstated. Only two in-tree tasks declare env_packages (tasks/fibonacci_with_template.yaml:16, tasks/inline_starter_example.yaml:13), so every other nightly row changes the interpreter its run_command criteria resolve — a pass-rate step change with no marker in run.json and no way to separate "the fix worked" from "a criterion lost its interpreter". The PR should state the expected nightly delta and, ideally, land behind one full comparison run. (trigger: src/coder_eval/sandbox.py) (restates: Axis 8: Removing the default venv drops the harness's python-on-PATH guarantee)
• 🟡 The inject-mode / runtime-kit path (docker/Dockerfile.runtime, docker/coder_eval_runtime_entrypoint.sh:17) is the shape converted skillsbench-style suites use, and the PR neither states nor tests what it does there: removing the sandbox venv leaves the unseeded kit venv supplying python while pip falls through to the task image's global pip, i.e. the same contradiction the change is claimed to close. Say explicitly whether inject-mode suites are in or out of scope for this fix. (trigger: src/coder_eval/sandbox.py) (restates: Axis 6: Inject-mode containers keep a python/pip split after the PR)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE057 — one definition for "does this config ask for a language environment". New rule tests/lint/rules/ce057_env_predicate_single_definition.py (class EnvPredicateSingleDefinition, appended to ALL_RULES in tests/lint/runner.py): outside src/coder_eval/models/sandbox.py, forbid (a) a BoolOp that tests <x>.python/<x>.node truthiness/is not None together with <x>.python.env_packages/<x>.node.env_packages, and (b) a bare truthiness/is not None test on <x>.python/<x>.node used as a gate. Both must route through one named predicate on the model (e.g. PythonEnvConfig.wants_venv / SandboxConfig.wants_python_env), so the activation rule lives where the fields live and a future second field is folded in by construction. Six sites duplicate the two-part test today (sandbox.py:417, :422, :847, :875, orchestration/regrade.py:299, :301) and sandbox.py:359 reads only half of it — after this rule that outlier is a lint failure, not a reviewer catch. Note pyright cannot reach this class: self.config.python.env_packages is perfectly typed, so the one-field-stands-for-the-whole-model coupling is invisible to the type checker — which is exactly why it needs a CE rule. As a fixture convention, record that a callee re-testing its caller's named predicate must assert it rather than silently return. Prevents: A6-high venv gate desynchronized between Sandbox.setup (sandbox.py:417) and Sandbox.adopt (sandbox.py:359) — same trajectory, different verdict under run vs evaluate --in-place; A2-low one field of PythonEnvConfig read as the whole model's activation signal (sandbox.py:417); A1-low _install_packages' now-unreachable three-clause early return (sandbox.py:847-848); and it structurally repairs the stale cross-reference comment at sandbox.py:353-358 ("Gated on config.python for the same reason setup is").
• [ce-lint] CE058 — no bare python/pip in a harness-executed command in tasks/. New whole-tree rule wired as a dedicated @pytest.mark.lint test class in tests/test_custom_lint.py (the CE055 shape, since it parses YAML rather than one .py AST): parse every tasks/**/*.yaml, and for each type: run_command command: plus every pre_run/post_run command:, reject a first token of python or pip; require python3, uv run python, or an explicit interpreter path. Scope it to harness-executed strings only — command_executed.command_pattern describes what the AGENT ran (e.g. tasks/early_stop_weighted_low_weight_absorbed.yaml:40) and must stay exempt. It fires on 15 criteria across 13 task files today (tasks/test_sandbox.yaml:13, tasks/hello_date.yaml:26, tasks/internal/session_resumption.yaml:48, tasks/agents/{claude,codex,antigravity}_hello_world*.yaml, tasks/agents/codex_string_utils.yaml:50/55/60, tasks/opencode_smoke_test.yaml:35, tasks/pi_smoke_test.yaml:36, …). Prevents: A8-high removal of the harness python-on-PATH guarantee — bare-python criteria now depend on ambient host resolution, giving 127/ModuleNotFoundError on hosts without python-is-python3 and a grader-interpreter-dependent verdict for identical agent output.
• [ce-lint] CE059 — a venv whose bin/ is exported onto a task-visible PATH must be seeded. New whole-tree lint test class scanning docker/** (Dockerfile*, *.sh): if a path is prepended to PATH by an entrypoint or ENV PATH, then any uv venv creating that path must pass --seed (or a pip shim must be installed into its bin/). Catches docker/Dockerfile.runtime:47 (uv venv --python ${PYTHON_VERSION} /opt/coder-eval/venv, never seeded; installs go in via uv pip install --python … at :62-65) against docker/coder_eval_runtime_entrypoint.sh:17 (export PATH="/opt/coder-eval/venv/bin:…"), which every criterion subprocess inherits through _build_run_command_env's os.environ.copy() (sandbox.py:1154). Prevents: A6-medium inject-mode python/pip split — python resolves to the unseeded kit venv while pip falls through to the task image's global pip, the same contradiction the PR's own comment says it is fixing, one directory up (and on the uv-absent inject path, newly introduced by removing the sandbox venv).
• [ce-lint] CE060 — no inert empty-mapping sub-config in tasks/. New lint test class: reject a {} (or an all-defaults) value for an optional sub-config key in a task YAML — it is indistinguishable from omitting the key, so it reads to the next author as a request the harness honors. Fires on tasks/token_check.yaml:7, tasks/internal/session_resumption.yaml:20 and tasks/python_cli_simulated_judged/echo_simulated_judged.yaml:34 (python: {}). Note the correct fix is deletion, not python: null — null changes grading behavior via Sandbox.adopt's discovery gate. Prevents: A8-medium/A7 python: {} now silently inert on the run path while a reader reasonably concludes the sandbox provisions Python for those three tasks; also surfaces the two-spellings-for-"no venv" schema redundancy.
• [ce-lint] CE061 — venv PATH/VIRTUAL_ENV writes confined to one surface. New rule (or a NoContainerEnvLiteral-shaped SSOT rule): the VIRTUAL_ENV literal and any <venv>/bin-onto-PATH prepend may appear in src/ only inside Sandbox._build_run_command_env (sandbox.py:1157-1159). That makes "which surface carries the venv" answerable by grep and keeps an agent-env builder (agents/*_agent.py::_build_sdk_env) from quietly acquiring one later. Prevents: A8-medium the new rationale comment at sandbox.py:415-416 claiming the venv bin/ was prepended "for the agent" — no code path ever did that (criterion subprocesses only, via criteria/run_command.py:67 and criteria/uipath_eval.py:71); with the surface pinned by a rule the prose has one place to be checked against.
• [ce-lint] CE062 — a test whose every assertion is inside a conditional is vacuous-capable. New rule tests/lint/rules/ce062_no_wholly_conditional_test.py, scoped to tests/ (like CE048): flag a test_* function that contains at least one assert but no assert outside an if/try, so a behavior change can void the test without failing it. # noqa: CE062 for genuine platform/optional-dependency skips (prefer pytest.mark.skipif). Fires on tests/test_sandbox_templates.py:161 (test_template_ignores_venv), whose only .venv assertion sits under if venv_bin.exists(): at line 187 — permanently False post-PR, so line 189 never runs and the test's stated purpose is unguarded. Prevents: A3-medium test_template_ignores_venv's dead assertion and its stale comment at line 185, plus the whole class where a provisioning change silently converts a weak proxy assertion into no assertion.
• [ce-lint] Extend CE030 (doc-schema parity) to the nested sandbox models. tests/lint/doc_schema_parity.py deliberately checks top-level models only (SandboxConfig and friends are excluded — see its module docstring, line 16). Add SandboxConfig, PythonEnvConfig and NodeEnvConfig to the tracked set (with EXEMPT entries for anything genuinely not user-authored), so changing what python:/env_packages MEAN forces the guide to be re-read in the same commit. Prevents: A8-medium/A5 stale authoring prose: docs/TASK_DEFINITION_GUIDE.md:519 ("defaults to driver: \"tempdir\" with standard Python environment") contradicting the line the PR edited at :524, and the un-rippled docstring at tests/test_glob_paths_in_file_criteria.py:132 (".venv is created inside the root for any task with a python: block"), now flatly false.
• [ce-lint] Extend CE005 to narrow provisioning fallbacks. CE005 (no_silent_except) only inspects broad except Exception:/bare handlers, so it never sees except (subprocess.CalledProcessError, FileNotFoundError): at sandbox.py:839, whose body swaps implementation (venv.create(self.venv_dir, with_pip=True)) and produces a materially different artifact shape. Widen the rule (or add a sibling id) to: a handler in src/ whose body invokes an alternate provisioning primitive (venv.create, subprocess.run, shutil.copy*, os.replace) must log at warning or record the divergence in environment_info. Prevents: A6-low the invisible uv-absent degradation at sandbox.py:839-843 — nothing logs it, nothing records it, coverage shows it unexercised, so the venv shape a given host produced (pip-seeded or not) is unknowable after the fact, and the PR pins its written rationale to only one of the two shapes.

Harness improvements (not statically reachable):

• Give the existing run ⟷ execute+evaluate equivalence test an environment-sensitive fixture. tests/test_execute_evaluate_loop.py:77::test_execute_then_evaluate_reaches_the_same_verdict_as_run already encodes the right invariant but its task's criteria do not read the criterion-side environment, so the desync sailed past it. Add a fixture task whose agent stub creates .venv/bin in the workspace and whose criterion is run_command: python -c "import sys; print(sys.prefix)", and assert the SAME verdict and the same resolved prefix across run and execute + evaluate --in-place (and run --resume, which reaches regrade_in_place → sandbox.adopt at orchestration/regrade.py:1180). Why not static: The defect is a divergence between two executed pipelines' criterion environments — it needs a real filesystem, a real agent-created .venv, and two full grading passes to compare; no AST can tell that two differently-spelled gates disagree about a runtime PATH. Prevents: A6-high setup/adopt venv gate desync (identical trajectory scoring 0.000 under run and 1.000 under execute + evaluate --in-place).
• One parameterized setup ⟷ adopt parity table, replacing the two tests that currently assert opposite things. Drive both paths over the same SandboxConfig matrix — python=None, python=PythonEnvConfig() (the default, env_packages=[]), python=PythonEnvConfig(env_packages=[…]) — and assert equal venv_dir and equal _build_run_command_env() PATH/VIRTUAL_ENV for each row. Fold in tests/test_sandbox_adopt.py:80 (asserts venv_dir == ws/.venv for env_packages=[]) and tests/test_sandbox.py:1337 (asserts venv_dir is None for the equivalent config) so the two can never again both pass, and cover the python=None adopt arm (sandbox.py:359->364), which a full-suite branch-coverage run shows is unexercised across the ENTIRE test suite. Why not static: Two code paths agreeing about a derived filesystem path and a composed PATH string is a runtime property; lint can force one predicate (CE057) but not that both paths compute the same environment. Prevents: A6-high the gate desync; A3-medium the unexercised python: null opt-out arm that the PR's own new field description at models/sandbox.py:434 documents as user-facing behavior.
• Add a changed-lines coverage gate to make verify / CI — --cov-branch plus diff-cover (or an equivalent per-diff threshold) on the lines a change touches, on top of the existing global 80%. Why not static: Reachability of sandbox.py:848 after a new caller-side gate is interprocedural, and "which lines did this change touch" is a VCS fact — both are outside any AST rule; only executed-coverage against the diff sees them. Prevents: A1-low the unreachable _install_packages guard (line 848 uncovered, the module's only uncovered line, while the global gate stayed green) and A3-medium the new venv semantics landing with the opt-out arm untested.
• Record the criterion-side interpreter in the row. Persist the resolved python (e.g. shutil.which("python") under the criterion env) and venv_dir into environment_info at grading time — flat scalars such as criterion_python / criterion_venv_dir, so a host-dependent verdict is diagnosable from task.json after the fact and comparable across hosts on the evalboard. CE054 then binds reader to writer. Why not static: The interpreter a criterion actually ran under is a property of the grading host and of how the operator launched coder-eval (uv run vs .venv/bin/coder-eval) — knowable only at runtime, which is precisely what makes the current verdict non-portability invisible. Prevents:
• A host-portability smoke job: run the smoke-pass bucket on a runner where bare python is absent from PATH (a python3-only container, or env -i PATH=/usr/bin:/bin), separate from the actions/setup-python jobs that currently mask the regression (.github/workflows/pr-checks.yml:56, 294, 360, 431, 502). Why not static: CE058 catches the bare-python strings, but only a differently-provisioned host proves the harness no longer supplies an interpreter — the 127-vs-0 outcome depends on the runner image, not on the source. Prevents: A8-high the dropped python-on-PATH guarantee, including tasks/test_sandbox.yaml:13 (python --version, "Python should be available in the sandbox") — the criterion whose entire purpose was the guarantee the PR removes.
• Exercise inject/runtime-kit mode at least once in CI — build a non-Python base image plus the runtime kit (docker/Dockerfile.runtime) and run one task through it, asserting that a criterion's python and pip resolve to the same environment. Why not static: Nothing in the tree runs this documented path today (docs/DOCKER_ISOLATION.md §runtime kit; tests/test_image_from_dockerfiles.py:418-438 only checks the entrypoint path), and the python/pip mismatch only appears once a real image with a global pip is built and a criterion is executed inside it. Prevents: A6-medium the inject-mode python/pip split surviving (and on uv-absent bases, being introduced by) this change, with no test, no warning and no in-tree row to notice it.
• Process note — record the boundary. Two findings in this set are not statically reachable and should be written down as such rather than left implicit: the truthfulness of a mechanism comment (sandbox.py:415-416 claiming the agent's PATH; sandbox.py:353-358 claiming parity with setup) is semantic judgment, and the fallback-shape question at sandbox.py:839 needs a host without uv. CE057 and CE061 shrink both by making the underlying facts single-sited and greppable, but the residue stays a reviewer/test responsibility. Why not static: Comment-to-code agreement over prose is undecidable in general; the repo's precedent (CE026/CE028/CE030/CE033) only reaches surfaces with a machine-derivable source of truth, which a rationale comment does not have. Prevents:

Top 5 Priority Actions

1. Resolve the venv-gate desync that changes a verdict for identical agent output: setup now requires non-empty env_packages (src/coder_eval/sandbox.py:417) while Sandbox.adopt still keys on config.python alone (sandbox.py:359), so an agent-created .venv reaches the criterion PATH via _build_run_command_env (sandbox.py:1157-1159) on the in-place grading path (orchestration/regrade.py:1180) but never during run — pick one PATH (discover post-agent, or tighten adopt's gate) and make both paths agree.
2. Restore an explicit interpreter guarantee for run_command criteria instead of delegating to ambient PATH: with SandboxConfig.python defaulting to an empty PythonEnvConfig (src/coder_eval/models/sandbox.py:429, 57), 13 of 15 bare-python criteria (for example tasks/test_sandbox.yaml:13-15, tasks/hello_date.yaml:26, tasks/agents/claude_hello_world_docker.yaml:45) now run under whatever interpreter and site-packages the grader happens to expose, so add a python->python3 shim dir in _build_run_command_env (or migrate the criteria to python3) and add a CE rule that scans tasks/ for bare python.
3. Close the remaining python/pip split on the injected-runtime container path, where /opt/coder-eval/venv is built without --seed (docker/Dockerfile.runtime:47) and prepended to the task-visible PATH (docker/coder_eval_runtime_entrypoint.sh:17) that every criterion subprocess inherits (sandbox.py:1154) — seed that venv or resolve the CLI by absolute path as docker/Dockerfile.runtime:88 already does.
4. Add the tests that would have caught all of the above: a parity test asserting setup-then-agent-created-.venv and adopt give the same venv_dir/PATH for one SandboxConfig, reconciliation of the two suites that currently assert opposite results for the equivalent config (tests/test_sandbox_adopt.py:75-82 versus tests/test_sandbox.py:1318-1337), coverage of the unexercised python=None opt-out arm (sandbox.py:359), and replacement of the now-dead conditional assertion in tests/test_sandbox_templates.py:187 with an unconditional one.
5. Ripple the behavior change through the surfaces that still describe the old always-create-a-venv world — the false PATH-for-the-agent claim in the new rationale comment (sandbox.py:415-416), adopt's stale "same reason setup is" premise (sandbox.py:353-358), docs/TASK_DEFINITION_GUIDE.md:519, the flatly wrong docstring at tests/test_glob_paths_in_file_criteria.py:132 — and collapse the redundant schema by defaulting python to None (as node already is) and deleting the three misleading python: {} blocks (tasks/token_check.yaml:7, tasks/internal/session_resumption.yaml:20, tasks/python_cli_simulated_judged/echo_simulated_judged.yaml:34).

---

Stats: 0 🔴 · 2 🟠 · 5 🟡 · 3 🔵 across 8 axes reviewed.

[uipreliga]

Fix all bugs and 🚢