Add promptfoo-based skill eval suite (Phase 1) - #58
Open
nonrational wants to merge 12 commits into
Open
Conversation
Promptfoo replaces the duplicated per-skill run-evals.mjs runners; evals.json stays the canonical case format, fed to promptfoo by a generator. CI validates structure always and spends model tokens only on PRs touching an evaluated skill. Scoped deliberately apart from Bernard, which compares whole config repos rather than gating one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ten tasks: scaffold the evals package, port the loader, heuristics, prompt builders, and deterministic graders from the bespoke runner, add the claude-code subject provider and promptfoo configs, gate and validate scripts with Make targets, live parity run, then retire the code-comment-register run-evals.mjs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ported from the code-comment-register runner, generalized to any skill and tolerant of case types Phase 1 does not grade yet. Validation checks the cross-skill common schema (required fields, structural constraints) and allows per-skill rubric and trap field naming conventions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes: correct Unicode escapes for smart quotes (‘’ and “”) and restore expected output to use straight quotes in normalize test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Subject prompt templates are ported verbatim from the bespoke runner so the parity check compares like with like. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promptfoo suite reached parity: 11/12 deterministic cases match the old runner's outcomes; the exception (disc-09) was model-response variance against the shared keyword-overlap threshold, with prompts and graders verified byte-identical. Transformation cases now grade via promptfoo llm-rubric, a deliberately different judge protocol than the old runner's bespoke judge; live judge verification awaits an ANTHROPIC_API_KEY environment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The layout snippet and the code use a custom JS provider; the prose still said exec script. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two skills (
code-comment-register,prose-register) shipevals.jsonsuites, but each carries its own ~600-line bespokerun-evals.mjs— near-duplicates of each other — and nothing runs the evals as config changes. Skill behavior that was pinned down by hand can silently regress.Motivation
Skill workflows should be independently testable units: change a skill, run its evals, see whether the behavior you liked still holds. A shared runner is also the foundation for CI (Phase 2: validate-always, path-filtered full runs) without maintaining a per-skill runner copy per suite.
Proposed Solution
A new top-level
evals/npm package with promptfoo as the runner andevals.jsonkept as the canonical authored format (spec:docs/superpowers/specs/2026-08-29-skill-eval-suite-design.md, plan alongside):tests.mjsgenerator readshome/.agents/skills/*/evals.jsonand emits one promptfoo test per case; answer-key fields never enter the subject prompt.providers/subject.mjsspawnsclaude -pwith cwd at the repo root, so the Skill tool and project skills are in the loop; apromptfooconfig.compare.yamladds the skill-hidden baseline condition (successor to the old--compare).llm-rubric(judge pinned toclaude-sonnet-5).bin/validate.mjs(offline, zero tokens) andbin/check-gate.mjs(deterministic cases must all pass; 90% suite floor) behindmake eval-validate/make eval SKILL=<name>/make eval-compare SKILL=<name>.code-comment-registerrunner is deleted;prose-register's stays until Phase 2.36 unit tests (
cd evals && npm test), all offline;make preflightuntouched and green.Feedback
llm-rubricjudge path has not run live — noANTHROPIC_API_KEYin the dev environment. First keyedmake eval SKILL=code-comment-registerverifies it; the failure mode is loud (judge errors gate the run).make eval; worth retuning itsexpected_rule(or the 0.2 threshold) before Phase 2 lets CI block PRs on it.