feat(evals): close the eval suite's calibration gaps and convert other plugins' evals

[kyle-sexton]

Problem

The evals plugin's eval suite reached a VALID run with the plugin scoring 0.99 and 0.31 without (delta +0.68, 95% interval +0.52 to +0.83; 29 of 31 cases at 1.00). The work left gaps in the suite's judge calibration, and a wider conversion task, none of them blocking.

Follow-ups

1. Two judges fail correct answers that their calibration samples don't cover.
   • noise-report-before-posting/gate-before-numbers failed correct answers in four runs. Every one used the shape "run run-validity.py first; on INVALID, post the INVALID line and its reasons". Candidate causes: an absolute script path, a second gate on the noise report, or a trailing partial check.
   • noise-before-gain/not-established failed an answer that opens with "the numbers point the right way", which its rubric names as allowed.
   • Add must-pass samples of these shapes, then recalibrate.
2. Three must-fail samples are never tested, because the reply-only agent will not reproduce them word for word:
   • measurable-criterion four-properties sample 3: the model drops its "launch a medical bot with 'answers are safe'" paragraph as unsafe advice.
   • worktree-guard-stop no-way-around sample 4, and zeros-after-usage-limit check-and-rerun sample 3: both get paraphrased.
   • Rewrite each so it still fails for one reason, with wording the model will repeat, for example a retraction that isn't safety-sensitive.
3. measurable-criterion must-fail sample 2 now misses two properties. It was built to miss only the trial set. Since the Achievable rule changed, its invented "2% measured last month" baseline fails too, so it no longer tests one gap on its own.
4. Convert the other plugins' evals (V7). Only source-control has a claude plugin eval suite (17 cases). 82 plugins carry skill-creator evals.json files only (326 files).

Context

The loop that produced these, and the reasoning behind each, is in the pull request that adds the evals suite and validity gate (link to follow).

[kyle-sexton]

This was generated by AI during triage.

Agent Brief

Type: Feature
Summary: Close the evals plugin suite's judge-calibration gaps, and convert other plugins' skill-creator evals to claude plugin eval suites.

Current behavior:
The evals plugin's own claude plugin eval suite passes its validity gate, but some llm judges have uncovered shapes: the noise-report-before-posting gate-before-numbers judge and the noise-before-gain not-established judge fail correct answers their must-pass samples do not cover; three must-fail samples (measurable-criterion four-properties sample 3, worktree-guard-stop no-way-around sample 4, zeros-after-usage-limit check-and-rerun sample 3) are never exercised because the model will not repeat them word for word; and measurable-criterion must-fail sample 2 now misses two properties instead of one. Most plugins still carry only skill-creator evals.json files.

Desired behavior:

• Each named judge has must-pass samples for the shapes it wrongly failed, and recalibrates at or above the calibration threshold.
• Each named must-fail sample fails for exactly one reason, with wording the model reproduces.
• Conversion of other plugins' evals is scoped and tracked separately (see routing).

Acceptance criteria:

• The judge calibration script passes for every llm grader in the evals suite, including the new samples
• Each rewritten must-fail sample misses exactly one property of its rubric
• A valid suite run (per the validity gate) shows no judge failing an answer its rubric allows

Out of scope:

• Changing grader rubrics' intent
• The run-validity and noise-report scripts

Routing: human-gated (needs decision)

The issue holds two different sizes of work. Follow-ups 1 to 3 are scoped calibration fixes inside one plugin's suite (C3). Follow-up 4, converting other plugins' evals, is a cross-plugin migration across dozens of plugins (C4), and its numbers are stale: code-tidying, discovery, education, improvement and planning now also have case.yaml suites, not only source-control, and the repository holds 341 evals.json files.

Question for the owner: split follow-up 4 into its own item (decomposed per plugin), and let this issue cover follow-ups 1 to 3 only? Recommended: yes. Once split, this issue can be routed agent-ready as work-class: scoped; until then it carries the structural class of its largest part.

Basis: verified, PR #5895 (merged 2026-10-03) names this issue as a follow-up; the named cases and samples exist under plugins/evals/evals/ unchanged since #5895; find plugins -path '*/evals/*' -name case.yaml lists suites in six plugins; find plugins -name evals.json counts 341. Searched docs/out-of-scope/: no matching concept.