[kamui]

Why

The second evaluator audit is declared (#128), and evaluator_audit.py still built the first plan's strata from the previous verdict format, so draw refused the new plan. This ports the tool, so that the sample can be drawn once the last-push cohort is graded.

Stacked on #128.

Scope

• bench/tools/evaluator_audit.py: commands check, draw, claims, second, compare, conclude for the thirteen declared strata and three unit kinds.
  • check runs the three whole-cohort checks (saved credit equals the derivation, a claim with a canonical claim holds its saved label and known problem, no stale grade or missing row), and draw refuses on any failure.
  • draw records each stratum as sampled, whole or empty; a stratum audited whole has no tolerance.
  • claims writes the first grader's claim list (item, kind, claim id, quote) for a sampled batch, and nothing else of the first verdict.
  • compare matches by review, item, claim id and exact quote, as multisets; a missing, added, repeated, renamed or re-quoted answer or review refuses the comparison. It saves nothing about identifies_cause or cause_only.
  • Every operation after the draw rebuilds the sample from the pinned grades and refuses a sample that is not the seeded draw of its populations.
  • conclude verifies every pin, rebuilds the comparison from the sample and the second assessments, and holds only sampled strata to max_errors.
• bench/tools/grade.py, bench/tools/grading_validation.py: a fixed claim list may carry claim ids (ids, parallel to quotes), and the validator holds the answer to them. Lists without ids behave as before.
• bench/tools/calibration.py: STRATA comes from the tool's UNITS.
• docs/evaluator-audit.md, docs/current-grading.md: what the tool does today, and three readings of the plan stated in the doc.

Tradeoffs

• Claim units include claims of reviews that are not admitted. The declared text has no "admitted" for claims; the user was asked on 2026-10-12 and kept the literal reading over admitted-only.
• A known problem is a family with approved eligibility. One awaiting approval is in no stratum and must be saved unresolved. All 67 are approved today.
• A control unit is a review of a task whose saved control status is audited-clean. The provisional state a pending candidate causes does not remove it, so the population does not depend on what the first grader raised.
• A review saved unassessed (its remedy inventory is incomplete) still holds graded claims and credit, so it stays a unit and draw does not wait for it.
• When ids are present, grade.py prepare adds one sentence to the grader's prompt: "Where an item lists ids, give its claims those ids, in that order." It is prompt text outside the pinned template.
• Not built, because the two verdicts do not hold it: the split of trial tasks from the others, the second grader's concerns about the claim list, and queue-level usage.

Blast Radius

The audit tool, and the claim-list path of grade.py prepare and its validator. grade.py and grading_validation.py are runner files, so a grading key prepared before this change fails dispatch with "runner changed after preparation"; no grading is prepared yet. Nothing is graded, so the tool has not run on real batches: on the real records check exits 1 with "189 selected batch(es) have no grade".

Verification

• python3 -m unittest discover -s bench/tools -p "test_evaluator_audit.py": 43 pass. test_calibration.py 21, test_grade.py 66, test_claim_grading.py 28 pass.
• python3 bench/tools/calibration.py check and current_grading.py check: pass.
• bun run test:bench: 934 tests, one failure, test_codex_grading.test_catalog_preserves_native_model_instructions, which also fails on main on this host and passes in CI.
• Reviewed twice by Codex GPT-6.1 Sol (high): seven findings on the first version and three on the fixes, all addressed with a test each. The second pass found no regression in populations, draw order, what reaches the second grader, or the exclusion of the second fact.
• Known gap: sample.json pins the task facts its populations rest on (known problems, impact labels, control status), because a ruling can change them after the draw without changing a grade. An edit that changes those facts and the populations together is not caught until compare pins the sample. Committing sample.json right after the draw closes it; the claim strata do not have the gap, since they come from the pinned grades alone.

Part of #30

[kamui]

Approved (advisory) — no findings.

Reviewed 5286dca. All nine changed files, the full diff, grading input fingerprints, grade projection, provenance checks, record locking and affected validators reviewed. Required independent verification completed. No finding or open question remains. The sample task facts still rely on committing sample.json immediately after drawing, as the PR discloses; no real audit has run because the cohort is ungraded.

Review details

Intent: Port the evaluator audit to verdict contract v2 and the thirteen declared strata, with a fixed claim list for the independent second pass.

Issue fit: The PR delivers the audit-tool portion of #30. Actual regrading, paid second assessments, audit reconciliation and dataset publication remain outside this change. Requirements were taken from the PR text and the declared audit plan.

Reviewed: 5286dca against merge-base 9feb2e1.

Checks:

• Accepted caller evidence from the clean reviewed head: evaluator-audit tests 49, calibration tests 21, grading tests 66 with sandbox escalation, and claim-grading tests 28 passed. Changed-test assertions and fixtures were inspected.
• Accepted exact-head calibration, current-grading and import checks, and diff whitespace validation. Prior sandbox startup failures were environment failures and were superseded by the successful escalated grading run.
• Current CI snapshot: Web and evidence and Client compatibility passed; Grading regressions pending. The prior broad-suite catalog failure is historical and also occurs on main on this host.