feat(grading): take the review-code reviews into the published selection and drop three setups

[kamui]

Why

The published selection held four setups on j-trpc-5017 alone. The owner chose on 2026-10-11 to run review-code-sonnet-5-5 on the ten re-cut tasks and take the other three out. The run is recorded (#131). This puts it in the selection, drops claude-builtin-sonnet-5, codex-builtin and codex-builtin-sol-high, and links the new reviews' comments to the saved claims, so the regrade grades the selection that will be shown.

Scope

• bench/scoreboard.current.json, bench/grading/current/inventory.json: one source added, three setups removed. Written by docs/research/last-push-review-code-2026-10-12/selection/selection.py from plan.v1.json, in one all-or-nothing generation that starts from the staged registry and refuses unless that registry has its recorded hash.
• bench/grading/current/claims.json: 19 links (3 equivalent, 16 related) from a blind matcher's output, selection/intake.v1.json, built by packets.py and collect.py. The collector refuses unless every keyed comment appears exactly once in the matcher's output.
• bench/tools/current_grading.py: kept_stops. A stopped last attempt with exactly one diagnosis that names it with a kept cause (skill-timeout, reviewer-stop) resolves its trial with no admitted review. Anything else stays pending.
• src/lib/scoring.ts: a trial resolved without admission shows the trial's own reason. src/lib/fixture.ts gives a resolved trial its last attempt's disposition as the reason, as the inventory does.
• Registry text of review-code-sonnet-5-5, README.md and doc counts, and the CI step that restores the transcripts the export publishes.

	Before	After
Setups	17	14
Scheduled trials	608	631
Admitted reviews	604	623
Batches	189	198
Pending trials	4	4
Trials resolved with no admitted review	0	4
Links from claims to comments	1,042	1,061

Known problems (67), decisions, claims, candidates and rulings on single comments are unchanged. The hero reads 16 tasks, 67 known problems, 5 methods and 6 models.

Tradeoffs

• The four review-code reviews that gave up after a refused command count as trials that delivered nothing. The owner chose that over leaving them pending, which would have kept the setup's recall and reliability unavailable on three pull requests, and over replacing them.
• Nothing is retired. No claim or ruling on a single comment is left without a comment in the selection, and the script stops if one would be. Three links to reviews that left stay as they were, under the owner's earlier rule.
• review-code-sonnet-5-5 has reviews on 11 of the 16 tasks. The default comparison now covers 11 pull requests, up from 1.
• The kept causes are a constant in current_grading.py, held equal to review_queue.KEPT by a test, so the grading queue's pinned file gains no import.
• Two statements recur in the unlinked comments and match no known problem or claim (hono's method-override check, SeaweedFS's readOnly key). Nothing was created for them; a grader raises a candidate if they hold up.

Blast Radius

The selection every later step reads: inventory, export, scorecard, the regrade plan and the audit's populations. No grade exists yet, so nothing is invalidated. The inventory rule changes the state of exactly four trials; every diagnosis in every run was scanned, and only those four carry a kept cause. Scoring values are unchanged; only the reason string of a failed trial differs. The registry text of the other thirteen setups is still stale from the last selection change and is left for a separate cleanup.

Verification

• bun run verify:import, verify:current (0 of 623 admitted reviews assessed), verify:claims, calibration.py check, roster.py (output unchanged): pass.
• current_grading.py status: 631 trials, 627 resolved, 4 pending; 198 batches.
• bun run test:web (64 tests), typecheck, build: pass. bun tools/scorecard.ts --configuration review-code-sonnet-5-5: "29/33 trials admitted, 27 complete, 0 pending, 4 failed; 11/11 PRs."
• bun run test:bench: 974 tests, one failure, test_codex_grading.test_catalog_preserves_native_model_instructions, which also fails on main on this host and passes in CI.
• grade.py preflight --offline: passes for the ten new batches and the changed j-trpc-5017 batch.
• Link intake: 8 Codex GPT-6.1 Sol sessions (high), $0.43 at list price; usage in selection/usage.v1.json.
• Reviewed by Codex GPT-6.1 Sol (high): three findings, all fixed with a test each. It found no defect in the selection, the links, the counts or the scoring values.
• Not done: the delivery tab was not opened in a browser.

Part of #30

[kamui]

Approved (advisory) — no findings.

Reviewed 74ff8ca. All 27 changed files reviewed, including generated registry/inventory changes, all 19 claim links, selection and intake tools, stopped-trial accounting, scoring callers and changed tests. The data-integrity premises were independently challenged. Existing stale descriptions for thirteen other setups remain outside this change.

Review details

Intent: Publish the review-code source on the ten re-cut tasks, remove three setups, and count the four kept stops as resolved trials with no admitted review. No actionable findings remain.

Issue fit: Fits the selection step of #30. Grading, the evaluator audit, and final actual-data verification remain later work.

Reviewed: 74ff8ca against merge-base c6a13d4.

Checks:

• Ran PYTHONPATH=bench/tools python3 -m unittest test_current_grading.KeptStops at the archived head: 8 tests passed.
• CI run 38225556004 passed with the same source tree as the reviewed head: 974 benchmark tests, with 18 skips outside the changed tests; 64 web tests; import, current-evidence and claim checks; build and typecheck. The new selection and rollback tests all passed.
• Independent checks upheld the kept-terminal admission gates and selection preservation/exception rollback. Local reconstruction lacked ignored upstream evidence and the scoring rerun lacked zod; matching CI supplied those checks. Rollback remains bounded to recoverable exceptions with working restore I/O, as the existing Generation helper specifies.
• Checked the full published dataset counts by tracing the exporter and hero count inputs: 16 tasks, 67 known problems, five methods and six models.
• The reported host-specific native-model catalog failure is historical environment evidence; the matching test passed in CI. Offline batch preflight and roster results were assessed by code trace and the supplied report, not rerun.
• The browser delivery view and original private matcher logs were not inspected. All new links were checked against saved source comments and current claims; the scoring reason was traced and covered by the passing test.
• No reviewing app is declared at the pinned base. Publication uses the authenticated author identity with COMMENT and no merge gate.