ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce - #89
Open
firaen22 wants to merge 1 commit into
Open
ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce#89firaen22 wants to merge 1 commit into
firaen22 wants to merge 1 commit into
Conversation
… corpus cannot produce Adds one fake-pass shape to the item-3 list. Unlike the empty-scan shapes already there, this grader runs correctly over a non-empty input and still passes every arm: it scans prose for a word the graded corpus itself contains, so it scores the corpus rather than the behavior. Its damage runs opposite to a vacuous green. A vacuous green hides a failure; this manufactures agreement, so an A/B whose arms all pass reads as "no effect" and retires a rule that actually worked. Fix is two-part: key on a structural marker the subject must create, and sanity-check the grader against a known-bad arm before trusting a clean sweep. Ships unprobed per the covenant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One fake-pass shape added to
ground-truth-gates§item 3, plus a Provenanceentry.
A substring grader that scans prose for a match token the graded corpus can
itself contain scores the corpus, not the behavior.
Why it isn't covered by the shapes already in item 3
The existing entries — empty glob, moved directory, missing baseline, scanner
that matched zero inputs — all share a signature: the gate ran over nothing.
The prescribed fix follows from that (assert the matched count is non-zero).
This one runs correctly, over a non-empty input, and still passes every arm.
The non-empty assertion does not catch it.
Its failure direction is also inverted from the rest of the list. A vacuous
green hides a failure that exists. This one manufactures agreement: in an
A/B, every arm passes, the run reads as "no effect", and a rule that genuinely
discriminated gets retired on the strength of a grader bug. A false negative
about your own experiment is harder to notice than a false green, because the
null result is the outcome you were already prepared to accept.
Fix is two-part, both cheap:
a field), not a word it might merely mention;
— a grader that passes an arm you know failed is the finding, not a formality.
Provenance
One downstream consumer's A/B probe. The grader for a "did the arm record the
change?" axis scanned each arm's output file for the word
drift— which thefixture's own body text already contained ("a batch that drifts model mid-run").
All four arms passed, including three that recorded nothing. The run read as a
4/4 null result. Regraded on a port-note heading, the same arms split 0/3 bare
vs 1/1 ruled, and the rule under test was real.
Ships
unprobedper the covenant.Verification
git diff --stattouchesskills/ground-truth-gates/SKILL.mdonly.upstream/main(a6ff7d0); staged by explicit pathspec.