fix(evals): methodology answers set success targets without stating their basis

[kyle-sexton]

Problem

In R6 (PR #5895, head 50c2769, run-validity VALID), the measurable-criterion case scored 0.33 with the evals plugin loaded. R5 had scored it 1.00 on the earlier tip. Two of three with-plugin runs failed the calibrated four-properties grader:

• Run 2 set a red-flag escalation target of "at least 99%" with no stated basis and called it a placeholder needing sign-off (judges: FAIL FAIL FAIL).
• Run 3 relied on a target relative to the current baseline, then called that target "a placeholder for you to adjust once you have data" (judges: PASS FAIL FAIL).

The rubric's Achievable property follows Anthropic's "Define your success criteria" page: a target rests on benchmarks, prior experiments, AI research or expert knowledge, and a target stated relative to the current baseline counts. The skill's answers drift from that, either by inventing a number or by withdrawing the basis they just gave.

The PR's commits only respelled this case's calibration samples, which no run reads, so this is model variance the skill does not yet constrain, not a regression.

What to do

1. Read the three R6 with-arm replies (kept traces from the R6 run) beside the methodology hub's success-criteria guidance.
2. Fix the methodology skill's hub so the answer states each target's basis and does not call a grounded target a placeholder. Do not ease the case or its grader.
3. Recalibrate four-properties if its samples change, then confirm with a run that run-validity calls VALID.

Related

• PR feat: adopt eval-design and hillclimbing guidance across six plugins #5895 (R6 in its Verification section)
• feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879: judge calibration gaps. Separate: those are judge errors; this is the skill's answer.

[kyle-sexton]

This was generated by AI during triage.

Triage

Outcome: human-gated (capability blocker, not an open decision). Priority medium. Work class: scoped (C3, bug-fix).

Why a human: the fix can be drafted by anyone, but it is done only when a claude plugin eval run of the evals suite comes back VALID under run-validity.py. That run is paid, is refused inside worktree-isolated sessions (where lanes run), and needs a sandbox backend (WSL2 or macOS; native Windows has none). Step 1 also reads the kept R6 traces, which live on the host that ran R6.

Verified on current main: the methodology hub already states the rule the replies broke. Its "Rewrite this criterion" answer says a target must be grounded the way the Achievable property allows, that a target relative to the current baseline is acceptable, and that placeholders and "set the target later" are not usable. So the gap is not missing guidance: answers still drift from guidance that exists, either inventing an ungrounded number or calling a grounded target a placeholder.
Basis: verified, git show origin/main:plugins/evals/skills/methodology/SKILL.md lines 37-50 (target-number rule); case grader plugins/evals/evals/measurable-criterion/graders/four-properties.md present on main.

Agent Brief

Type: Bug
Summary: Make /evals:methodology answers state the basis of every success target and never call a grounded target a placeholder.

Current behavior:
In R6 (PR #5895), the measurable-criterion case scored 0.33 with the plugin loaded (R5: 1.00). One reply set an "at least 99%" target with no stated basis and called it a placeholder needing sign-off; another gave a baseline-relative target and then called it "a placeholder for you to adjust once you have data". The hub's target-number rule already forbids both, so the model is not applying it.

Desired behavior:
Every target the skill proposes names its basis (measured baseline, prior result, benchmark, expert review, or a quantity relative to the current baseline over a stated trial set). A target grounded that way is presented as the target, not as a placeholder. When no basis exists, the answer says so and uses a baseline-relative target instead of inventing a number.

Key interfaces:

• The methodology hub's target-number rule (the "Is this criterion good?" and "Rewrite this criterion" answers): sharpen or reposition it so replies apply it.
• The four-properties grader on the measurable-criterion case: unchanged unless its samples change, in which case recalibrate it.

Acceptance criteria:

• The three R6 with-arm replies are read beside the hub's guidance and the failure mode is named in the PR.
• A claude plugin eval run of the case set that run-validity.py calls VALID shows measurable-criterion passing four-properties in every with-plugin run.
• No case prompt or grader rubric is loosened.

Out of scope:

• Judge errors tracked in feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879.
• The plugin-eval hub compression in feat(evals): compress plugin-eval/SKILL.md (v2) with a same-day confirm run #5956.