feat(grading): declare the second evaluator audit and close the first round

[kamui]

Why

The audit plan of 2026-10-03 names its strata after labels the rubric in force no longer has, and its round was never concluded. A new plan has to be declared before the last-push selection is graded, so that nothing in it can be tuned to results.

The user closed the first round as superseded and selected the replacement on 2026-10-12. The plan was built from two independent proposals (this session's and one by Codex GPT-6 Astra, high), scored blind by a third model. Astra's was the base.

Scope

• bench/grading/rulings/cohort-rebuild-audit.v3.md: the receipt for both answers, with the selected sizes and tolerances.
• bench/grading/current/audits.json: the new plan (seed evaluator-audit-v2-2026-10-12, thirteen strata, at most 640 units) and the first plan with its evidence under superseded. decision-queue.json is regenerated.
• bench/schema/current-audit.schema.json: the control-review unit and the superseded list; the plan's shape moves to $defs/plan so both use it.
• bench/tools/calibration.py: STRATA holds the new names; audit_problems checks each superseded plan's own strata and selection, that each round has its own seed, and that its closure names that seed and is in the saved receipt.
• bench/tools/evaluator_audit.py: draw refuses strata the tool does not build.
• docs/evaluator-audit.md: rewritten for the second plan.
• docs/research/cohort-rebuild-2026-10-05/audit/next-plan/: the proposal as approved, both candidates, the brief, the rubric, the judge's report and the synthesis note.

Tradeoffs

• serious is 100 units with one error allowed, not 60 with none. The limit is the same (4.7% against 4.9%), and a grader with a 0.4% error rate fails 6% of the time instead of 21%. The user was shown both and chose this.
• A stratum audited whole has no tolerance. Scaling the tolerance down would give zero at 59 of 60 units.
• Claims labelled problem have their own stratum. A first draft audited them only through credit; a review showed that a wrong label on a second claim for an already credited problem would sit in no stratum, and the user added the stratum.
• A review of a clean control is alarmed by a refuted or unproven claim. The user fixed this in the plan because the scoring code still names a label the rubric no longer has.
• Claims linked to a saved ruling leave the claim strata. Both graders are handed that label, so agreement on it is not evidence. They get a whole-cohort check instead.
• evaluator_audit.py is not ported here. The port to verdict contract v2 (the three whole-cohort checks, control units, exact matching, the audited-whole rule) is needed before the draw, not before grading, and is its own change.

Blast Radius

Records, the checker and one refusal. No grade exists, so nothing is re-derived. The explorer shows the audit record's new reason. Until the tool is ported, evaluator_audit.py draw stops with a message instead of building the wrong strata.

Verification

• bun run verify:import, verify:current, verify:claims: pass (0 of 604 admitted reviews assessed, 67 of 67 known problems with an approved label).
• bun run test:web: 63 pass.
• bun run test:bench: 904 tests, one failure, test_codex_grading.test_catalog_preserves_native_model_instructions, which also fails on main on this host (local Codex version) and passes in CI.
• The 95% limits in the plan were recomputed from the binomial tail.
• Reviewed by Codex GPT-6.1 Sol (high): four findings, all addressed. Two were checker gaps; two were ruled on by the user (decisions 5 and 6 in the receipt).

Part of #30

[kamui]

Approved (advisory) — no findings.

Reviewed 0e17219. All 18 changed files reviewed, including schemas, tests, receipts, preserved evidence, proposals and affected checker/tool paths. No actionable findings.

Review details

Intent: Declare evaluator audit v2 before grading and close the first round as superseded while preserving its evidence.

Issue fit: Completes the audit declaration and first-round closure portion of #30. Grading, the audit-tool port, paid second assessment and audit conclusion remain separate work.

Reviewed: 0e17219 against merge-base 068e8a6.

Checks:

• Three changed regression tests pass at the reviewed head in disposable sources.
• Exact-head CI passes: 905 grading tests with 18 skips, 63 web tests, import/current/claims checks and client compatibility.
• Integrity checks confirm the old plan and evidence are unchanged, receipt pins match and the selected sample totals 640.
• Independent challenges to preserved evidence, saved decisions and refusal of unsupported draws all hold within the reviewed scope.
• The audit remains unassessed. The verdict-v2 tool port is deliberately deferred; unsupported drawing is refused. Self-review uses COMMENT because no separate reviewing identity is declared.