Repository navigation
feat(evals): compress plugin-eval/SKILL.md (v2) with a same-day confirm run #5956
Description
Why
plugins/evals/skills/plugin-eval/SKILL.md is about 6,000 words and sits at 499 lines, one under the skill line cap. Inside an eval run only the hub SKILL.md is readable (reference/ reads are denied), so the hub carries everything, and every request that loads the skill pays for its size. PR #5895 measured a compression (R4 Run B, about 6,080 to 4,080 words) that cut with-arm agent cost per request by about 11% and passed the cost gate. It was reverted because two dropped clauses regressed two cases:
noise-before-gainfell from 0.83 to 0.50 after thenear ceilingline lost "so the suite has almost no room to show a gain".interval-setting-scopefell from 1.00 to 0.67 after the interval-method gotcha lost "and nothing needs checking".
What to do
- Take the R4 Run B patch and restore those two clauses word for word. The patch lives only on the machine that ran it (
r4/B/change.patchin the R4 runner scratchpad), so re-derive it if that copy is gone. - Register the gates before the run, as R3/R4 did: quality, cost and mechanism.
- Run one confirm of the patched skill against a same-day no-change control, both through
run-validity.py. Report numbers only from a VALID run. - Keep or revert on the registered gates.
Lesson for the editor brief
"Keep every directive" does not protect an explanatory clause that a grader depends on, and no mechanical token check catches its loss. Tell the editor to keep the reason clauses, or check each removed clause against the case graders that cite it.
Related
- PR feat: adopt eval-design and hillclimbing guidance across six plugins #5895 (R3 to R6 records in its Verification section)
- feat(evals): close the eval suite's calibration gaps and convert other plugins' evals #5879 (judge calibration gaps, including the recurring
gate-before-numbersjudge miss)
Activity
feat: adopt eval-design and hillclimbing guidance across six plugins (#…
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during triage.
Triage
Outcome: human-gated (capability blocker, not an open decision). Priority medium. Work class: scoped (C3, feature).
Why a human: the work is fully specified, but no autonomous lane can finish it:
claude plugin evalconfirm run plus a same-day control, each checked byrun-validity.py. Lanes run in worktree-isolated sessions, where the Bash guard refuses any command containingeval.Whoever has a WSL2 or macOS host outside worktree isolation, with the cost ceiling set, can run it as written.
Verified on current main: the plugin-eval hub is still 499 lines and about 6,090 words, unchanged since PR #5895.
Basis: verified,
git show origin/main:plugins/evals/skills/plugin-eval/SKILL.md | wc -lw= 499 lines, 6089 words; worktreeevalrefusal and sandbox backend rule in that hub's "Worktree isolation" and "Sandbox backend" sections.Agent Brief
Type: Feature
Summary: Re-apply the plugin-eval hub compression measured in PR #5895 with the two grader-relevant clauses restored, then keep or revert it on gates registered before the run.
Current behavior:
The
/evals:plugin-evalhub sits one line under the skill line cap at about 6,000 words. Inside an eval run only the hub is readable, so every request that loads the skill pays for its full size. The earlier compression (about 6,080 to 4,080 words, about 11% lower with-arm cost) was reverted because two dropped explanatory clauses regressed two cases:noise-before-gain(0.83 to 0.50) andinterval-setting-scope(1.00 to 0.67).Desired behavior:
The hub carries the compressed text with both clauses restored word for word: "so the suite has almost no room to show a gain" on the near-ceiling line, and "and nothing needs checking" on the interval-method gotcha. Re-derive the compression if the R4 patch is gone.
Acceptance criteria:
run-validity.pyas VALID; only numbers from VALID runs are reported.Out of scope: