fix(reward): add resource limits to the untrusted-code grader (refs #71)#111
Open
shin-core wants to merge 1 commit into
Open
fix(reward): add resource limits to the untrusted-code grader (refs #71)#111shin-core wants to merge 1 commit into
shin-core wants to merge 1 commit into
Conversation
tmimmanuel
approved these changes
Jul 12, 2026
tmimmanuel
left a comment
Collaborator
There was a problem hiding this comment.
The patch does what the issue asks for: the grader now runs each candidate in a throwaway home/cwd with a hard allowlist env and adds POSIX rlimits. The new tests cover both the leak PoC and normal grading, so the secret-exposure path is addressed cleanly.
The HOME/secrets leak from mini-router#71 is now fixed on main (isolated HOME/cwd, scrubbed env, python -I), but the grading subprocess still has no resource caps, so an untrusted candidate can exhaust host memory or fork-bomb the eval box within its timeout. Verified: a 4 GiB allocation runs to completion and 'passes' on main. Add _rlimit_preexec() and wire it as preexec_fn: best-effort POSIX setrlimit for address space (2 GiB), CPU (30s), and process count (64), each guarded, returning None (no hook) off POSIX. Purely additive on top of main's sandbox. Append POSIX-gated tests to tests/test_reward_sandbox.py: the memory-bomb candidate is now killed instead of passing; ordinary grading unaffected. Refs mini-router#71 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shin-core
force-pushed
the
sn74-shin-core-sandbox-home
branch
from
July 13, 2026 09:17
5b09e2f to
80d8121
Compare
Contributor
Author
|
Heads-up: the core #71 fix (isolated HOME/scrubbed env/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@tmimmanuel — since you approved this, the core secret-leak fix for #71 landed on main (isolated throwaway HOME/cwd, scrubbed env,
python -I). I've rebased this PR down to only the part that main still lacks: resource limits. It no longer touches the HOME/env logic and does not re-implement what's already merged.What this now adds (purely additive — 73 insertions, 0 deletions)
Main's sandbox closes the exfiltration vector but has no resource caps, so an untrusted candidate can still exhaust host memory or fork-bomb the eval box within its wall-clock timeout.
_rlimit_preexec()wired aspreexec_fnon the grading subprocess: best-effort POSIXsetrlimitfor address space (2 GiB), CPU (30s), and process count (64), each guarded so a platform rejecting one still applies the others; returnsNone(no hook) off POSIX.Tests
POSIX-gated tests appended to main's
tests/test_reward_sandbox.py:Still out of scope (flagged for validator maintainers)
OS-level isolation — network namespace, filesystem confinement, unprivileged user (issue #71 items #2–4) — belongs at the validator host/deploy layer, not a portable code change. The grader should still run on an isolated host with no live credentials.
Refs #71
Notes
A corresponding lab-notebook entry was added to
docs/JOURNAL.mdper repository protocol (AGENTS.md §6).🤖 Generated with Claude Code