Skip to content

ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
firaen22 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce #89

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion skills/ground-truth-gates/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,18 @@ A generic green test is not proof. A gate is real only if:
scanning nothing (a guard script once did this for the very file its outage
check was written for). A passing scan must also prove its input set is
non-empty — assert the matched count is non-zero; merely printing it is the
same vacuous green if nothing fails on 0. Worker-written guard scripts
same vacuous green if nothing fails on 0. A **substring grader whose match
token can occur in the graded corpus** — scanning prose for a word that the
corpus itself may contain scores the corpus, not the behavior, and unlike
the empty-scan shapes above this one runs correctly over a non-empty input
and still passes every arm. Key on a token the graded material cannot
produce on its own (a structural marker — a heading, a filename, a field
the subject must create), and sanity-check the grader against a known-bad
arm before trusting a clean sweep: a grader that passes an arm you KNOW
failed is the finding, not a formality. Its damage runs opposite to a
vacuous green — it manufactures agreement, so an A/B whose arms all pass
reads as "no effect" and retires a real one. (`unprobed` — contributor
incident as shape; see Provenance.) Worker-written guard scripts
especially: item 2's known-broken run applies before trust, no exemption —
whoever wrote a guard has never seen it fail. (`unprobed` — private
incident as shape; see Provenance.)
Expand Down Expand Up @@ -484,5 +495,13 @@ date-truth" — so the truth evidence lived in the conversation, not the gate;
the fix added anchor-date assertions as a regression guard against a future
bad edit. Ships `unprobed` per the covenant; its probe joins the private
round-5 queue.
The item-3 substring-grader-token-collision shape (2026-07-28) comes from one
downstream consumer's A/B probe: the grader for a "did the arm record the
change?" axis scanned each arm's output file for the word `drift`, which the
fixture's own body text already contained ("a batch that drifts model
mid-run"). Every arm passed, including three that recorded nothing — the run
read as a 4/4 null result and would have retired a rule that did in fact
discriminate 0/3 vs 1/1 once regraded on a port-note heading. Ships
`unprobed` per the covenant.
`template/` scripts are self-contained (Node + bash, zero deps) and were run
green on 2026-07-06 with Node v23; re-verify with `bash template/run-all.sh`.
Loading