Fix release review findings - #2
Merged
mattteufel-e2b merged 6 commits intoSep 8, 2026
Merged
Conversation
- Correct the openai-compatible patch hunk so the upstream anthropic branch of call_llm keeps its return statement. - Reject a dirty vendored upstream checkout at run time, not only in sync-upstream, so shipped validator code always matches the pin. - Fix the result-archive traversal guard: normalize the joined path so ".." members are rejected lexically (filter="data" remains). - Drop gemini-cli from --agent; it speaks the Gemini API, not the OpenAI-compatible transport the adapter wires, and failed silently. - Declare bedrock-mantle.*.api.aws in both packaged policies and match model hosts as patterns so --bedrock-region works outside us-west-2. - Copy the hardened install_codex.sh unconditionally so a mistyped --remote-install-codex path errors instead of shipping upstream's unhardened installer. - Document that network-locked.json needs preloaded agent tooling that this release does not ship. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Regenerate openai-compatible.patch with three lines of context and apply it without --unidiff-zero, so a misplaced hunk fails loudly instead of applying silently at the wrong line. - Refuse agent runs under a policy whose runtime allowlist is only the model host; agent tooling is installed in the task container and such runs previously failed and were graded as eligible model failures. - Drop symlink and hardlink members when extracting result archives so a stray agent symlink neither escapes the artifact root nor fails collection of a completed run. - Validate --network-policy, --patch-file, --remote-smoke, and --remote-install-codex once at CLI entry instead of once per task. - Point the dirty-checkout error at a remedy that works; sync-upstream also refuses dirty trees. - Type RunOptions.agent as codex|openhands, remove the stale gemini test assertion and comment, document supported agents, and assert the two packaged policies share their host lists. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The dirty-checkout remedy ran `git clean -fd` without -C, which would clean the caller's repository; both steps now target the vendored path. - Only scripts/ and projects/ are checked for local changes; those are the trees shipped to the sandbox, and IDE metadata elsewhere is harmless. - Result extraction keeps in-tree hardlinks and symlinks and drops only members the tarfile data filter rejects. - batch evaluates the network policy once before resolving or submitting any task, matching preflight and run. - Type the run kind as a Literal and update the README to say the locked policy is refused up front for agent runs while smoke still works. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
main() requires E2B_API_KEY before dispatching to batch. The test passed locally only because the key was set in the developer shell; set a dummy value so CI exercises the policy gate instead of the key check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Docker Hub allows about 100 anonymous manifest requests per hour per address, so a lock that must resolve all 506 mutable tags before any task can run never completes without an authenticated Docker Hub plan. - `images lock --task <project>/<task>` (repeatable) resolves only the images those tasks need and merges into the existing lock for the same upstream inputs, so the inventory can be walked in slices. - The lock loader no longer rejects a partial lock; every task is still refused at resolution time if its image is absent, with the exact command to add it. Extra, stale, mutable, and provenance-free entries are rejected as before. - The Buildx resolver backs off and retries briefly on a 429 and reports the registry's error text instead of a bare exit status. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Issues are disabled on this repository, so the confirmed reliability findings from the release review that PR #2 leaves alone are tracked in-tree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mattteufel-e2b
marked this pull request as ready for review
September 8, 2026 22:34
mattteufel-e2b
added a commit
that referenced
this pull request
Sep 8, 2026
Land PR #2 on main, remove dead code, rewrite README for customers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1 addressing confirmed findings from the release review, plus a second hardening pass after reviewing this PR itself. Stacked on
codex/cybergym-public-release.Round 1
openai-compatible.patchhunk offset so upstream's anthropiccall_llmbranch keeps itsreturn(was left unreachable after the new branch)._verify_upstreamrejects a dirty vendored checkout, so run/smoke/batch/preflight cannot ship locally edited validator scripts under the pinned commit...members are actually rejected.gemini-clifrom--agent. Gemini CLI speaks Google's protocol and both supported providers are OpenAI-compatible, so it always failed and was graded as a model failure.bedrock-mantle.*.api.awsand the model-host check matches patterns, so--bedrock-regionworks outsideus-west-2.build_code_bundlecopies the hardenedinstall_codex.shunconditionally; a mistyped--remote-install-codexpath raises instead of silently shipping upstream's unhardened installer.network-locked.jsonrequires preloaded agent tooling that this release does not ship.Round 2
--unidiff-zero, so a misplaced hunk fails loudly instead of applying at the wrong line.kind=rununder a policy whose runtime allowlist is only the model host, in both preflight and execution. Such runs previously failed during tooling install and were recorded as eligible model failures that batch reuse never retried.sync-upstreamalso refuses dirty trees).RunOptions.agenttyped ascodex | openhands; stale gemini test assertion and comment removed; README lists supported agents; test asserts both packaged policies share host lists.Round 4 (found by running a task end to end)
images lock --task <project>/<task>(repeatable) now resolves only the images those tasks need and merges into the existing lock. The loader accepts a partial lock; each task is still refused at resolution time if its image is absent, with the exact command to add it. The Buildx resolver backs off briefly on 429 and surfaces the registry's error text.Verification
uv run pytest -q: 66 passed (52 existing + 14 intests/test_review_fixes.py, each written before its fix and observed failing)ruff check,ruff format --check,scripts/check_public_tree.py,actionlintpassuv buildandtwine checkpassb861317with plaingit apply; anthropic branch returnsRelease gate evidence (2026-09-08, this branch at e59fb5f)
PR #1 listed four outstanding gates and asked that they be reviewed before merge. All were exercised live on E2B with this branch:
sync-upstreampinsb861317in CI (both Python jobs and the sdist job) and locallycurl/arvo_66012(0 lookups, image already pinned) andffmpeg/oss-fuzz_368729566(1 lookup)cybergym-e2e-dind:recipe-3b3d5f9d827b70b3built and receipt-verified; FFmpegcybergym-e2e-ffmpeg:recipe-e8010b6665ef25fabuilt in 94 s with the Opus archive checksum and FFmpeg image digest verified;preflight --kind smokeon the FFmpeg task routes to the hot template with a verified tag-to-build receiptsmoke curl/arvo_66012: completed, exit 0,benchmark.status=passed,outcome=ground_truth_passed(upstream S4 validator), 4.3 min end to end, sandboxi2wxj4q2v07fvrimwd9ffrun curl/arvo_66012 --agent codex --provider bedrock --model openai.gpt-5.4: completed, artifacts collected, one Codex turn completed with no API errors,benchmark.status=failedat S1 (a genuine model miss on one attempt), 10.5 min, sandboxibkmt2q3pfnufbi2ytx4uDeferred findings are tracked in
FOLLOW-UPS.md(issues are disabled on this repository).Not changed
🤖 Generated with Claude Code