Conversation
A task can now declare a hidden per-test oracle the agent never sees: withheld at provision, injected into a throwaway copy of the final repo at grade time, run there so the agent's repo/diff stay untouched. - provision(hidden=...) strips oracle files from the seeded repo. - score.held_out_verdict: copy -> inject oracle -> run fail_to_pass + pass_to_pass; True/False, or None when no D3 oracle is declared. - run_task prefers the D3 verdict and expresses it on the probe axis (pass->0, fail->1), so solved/held_out_passed/gamed/classify all read the held-out signal through one path; D3 tasks run non-strict. Grader-touching (ADR-0011/0040): offline TDD green (incl. an end-to-end gaming-detection test); the global evals.diff validation is a live run.
# Conflicts: # evals/run.py
…tes oracle grading)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked on #107 — the richer held-out oracle for the declared-verification feature: a hidden, per-test partition (ADR-0011 D3) injected only at grade time, so the agent can't special-case a test it never saw.
Motivation
A single
success_probeis one hidden script. ADR-0011 D3 generalizes it to a partitionedfail_to_pass/pass_to_passoracle held out of the agent's repo entirely — the strongest anchor for grading a model that can auto-approve its own contract (ADR-0039): even a gamed self-contract can't touch a test that was never in the workspace.Changes
evals/provision.py:provision(hidden=...)strips oracle files from the seeded scratch repo before the baseline commit — withheld from the agent.evals/score.py:held_out_verdictcopies the final repo to a throwaway, injects the hiddenoraclefiles, runsfail_to_pass+pass_to_passthere —True/False, orNonewhen no D3 oracle is declared. The agent's repo and diff stay untouched.evals/run.py: prefers the D3 verdict and expresses it on the probe axis (pass→exit 0, fail→exit 1), sosolved/held_out_passed/gamed_rate/classifyall read the held-out signal through one path with no downstream changes. D3 tasks run non-strict (graded by the oracle).Testing
tests/test_evals.py:provisionwithholding;held_out_verdict(correct passes, broken fails, no-oracle →None, throwaway isolation); and an end-to-endrun_tasktest where a model that claims done but whose code fails the hidden oracle is classifiedgamed.make checkgreen — 637 passed.python -m evals.diffvalidation is a live run, to be run before the ADRs flip Proposed → Accepted.