Repository navigation
Conversation
- 47 `claude plugin eval` cases across 10 skills, discovered from the tree - one neutral fixture every case symlinks to, with `evals/README.md` as the spec - `evals.yaml` runs post-merge, scoped by `--tag` to the changed skills, advisory only - CI pins Claude Code, verifies every case's fixture symlink, runs the hook tests, and spell-checks YAML `create-pr` and `rebase` have no cases: `disable-model-invocation: true` puts them out of reach of any prompt.
A two-arm run of all 47 cases found the suite scoring the harness, not the skills: graders asserting template wording a skill only writes into a file, rubrics a blind judge could not satisfy, and prompts that stopped the skill from firing at all. - Prompts keep the task a task and name the read-only sandbox as a fact; the "do not edit anything, just describe it" phrasing killed routing in 5 of 5 cases that tried it. - Graders assert the vocabulary the reply really carries, with unstable judges converted to regex on the skill's own words. - `evals/README.md` records both traps, plus the ceiling on skills that only speak through a file they write. 21 of 47 cases passed before this; 25 pass now, and the rest are findings about the skills rather than broken cases.
Four cases are about a file the skill produces — the plan's Implementation Phases block, the brainstorm write-up — and were asserting that wording against the chat reply, where it never appears. They now grant `Write` and read the file back with a `files` target, which matches across everything the run wrote and so never has to guess the filename the skill picked. CI and the documented commands pass `--allow-tools Write`; without it those cases score 0 with no failing content grader.
…e one
`target: files` matches the paths a run wrote, not what is inside them, so the
four document cases were asserting template wording against a list of filenames
and failing for a reason that had nothing to do with the skills. They now use
`{ source: file, path: … }` against a path the prompt asks for by name.
Writing a real plan also costs turns: both plan cases were hitting the 12-turn
cap and the 900-second timeout mid-write, which scores 0 with no failing grader
and reads as a content regression. Document cases now run 20 turns and 1500
seconds.
The `llm` grader on a 19k-character plan is gone for the reason the runner
itself prints — judges are noisy on long inputs. `evals/README.md` records all
three traps.
Both drive a triage, a fix, a test and a reduced review inside 12 turns; `hotfix-refuses-to-skip-review-under-pressure` hit the cap mid-run, which scores 0 with no failing grader and reads as a content miss.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Ports the eval suite and the CI checks that guard it from
vgv-ai-flutter-plugin.claude plugin evalcases across 10 skills, each a directory underevals/<skill>/<case>/, discovered from the tree.evals/README.mdis the spec: case format, grader reference, the traps that each cost us a false result, and what the suite cannot reach.evals.yamlruns after a merge tomain, scoped by--tagto the changed skills, with-plugin arm only, advisory. It needs theANTHROPIC_API_KEYsecret.ci.yamlpins the Claude Code version, verifies every case'sfixture.shsymlink, runs the hook test suite, and spell-checks YAML.Runs grant
Read,Glob,GrepandSkill, plusWritefor the four cases whose subject is a document the skill writes — those read the file back with a{ source: file, path: … }target.create-prandrebasehave no cases:disable-model-invocation: trueputs them out of reach of any prompt.Measured state: 24 of 47 cases pass at
--threshold 0.8on a single run each; an earlier two-arm run scored mean Δ +0.28. Reds are three different things and only one of them is a case defect: routing misses the skills are responsible for (7 cases in the last run), judge noise on single runs, and run-to-run variance — cases flip between 1.00 and red across runs of the same suite. That is why the suite is advisory and whyevals/README.mdsays to confirm a red case with--runs 3.The skill fixes these findings justify are in the PR stacked on top of this one.
Type of Change
feat)fix)refactor)docs)ci)chore)🤖 Generated with Claude Code