Problem
Skills are markdown-based prompts with no automated way to verify correctness or catch regressions. During #63, self-review caught 3 SPEC/SKILL misalignments manually — a validation script would have caught them automatically.
Options (in order of effort)
1. Structural validation script (low effort, high ROI)
A script that checks all skills for:
- Valid YAML frontmatter (required fields:
name, description)
- Required sections exist in SKILL.md
- Cross-references (
references/*.md) point to files that actually exist
- Step numbering is sequential after edits
- SPEC.md "In scope" aligns with SKILL.md content
Similar to junior-prod's scripts/quick_validate.py. Can run in CI on any PR that touches skills/.
2. Prompt snapshot testing (medium effort)
Store the assembled prompt for each skill as a snapshot file. When a skill changes, the diff shows exactly what the agent will see differently. No LLM calls — just string comparison. Catches unintended section removal or broken references.
3. BDD-style evals with LLM judge (high effort)
YAML test cases: given (synthetic diff + context), should_find (expected findings), should_not_find (false positives to avoid). Feed the skill prompt + fixture to a model, use a cheaper model as judge. This is Sentry Warden's approach — gold standard but requires LLM invocations and a fixture library.
Recommendation
Start with option 1. It's the highest ROI and would have caught all three findings from the #63 review automatically.
Context
Problem
Skills are markdown-based prompts with no automated way to verify correctness or catch regressions. During #63, self-review caught 3 SPEC/SKILL misalignments manually — a validation script would have caught them automatically.
Options (in order of effort)
1. Structural validation script (low effort, high ROI)
A script that checks all skills for:
name,description)references/*.md) point to files that actually existSimilar to junior-prod's
scripts/quick_validate.py. Can run in CI on any PR that touchesskills/.2. Prompt snapshot testing (medium effort)
Store the assembled prompt for each skill as a snapshot file. When a skill changes, the diff shows exactly what the agent will see differently. No LLM calls — just string comparison. Catches unintended section removal or broken references.
3. BDD-style evals with LLM judge (high effort)
YAML test cases:
given(synthetic diff + context),should_find(expected findings),should_not_find(false positives to avoid). Feed the skill prompt + fixture to a model, use a cheaper model as judge. This is Sentry Warden's approach — gold standard but requires LLM invocations and a fixture library.Recommendation
Start with option 1. It's the highest ROI and would have caught all three findings from the #63 review automatically.
Context