Skip to content

ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce - #89

Open
firaen22 wants to merge 1 commit into
F-e-u-e-r:mainfrom
firaen22:grader-token-collision
Open

ground-truth-gates item 3: a substring grader must key on a token its corpus cannot produce#89
firaen22 wants to merge 1 commit into
F-e-u-e-r:mainfrom
firaen22:grader-token-collision

Conversation

@firaen22

Copy link
Copy Markdown
Contributor

What

One fake-pass shape added to ground-truth-gates §item 3, plus a Provenance
entry.

A substring grader that scans prose for a match token the graded corpus can
itself contain scores the corpus, not the behavior.

Why it isn't covered by the shapes already in item 3

The existing entries — empty glob, moved directory, missing baseline, scanner
that matched zero inputs — all share a signature: the gate ran over nothing.
The prescribed fix follows from that (assert the matched count is non-zero).

This one runs correctly, over a non-empty input, and still passes every arm.
The non-empty assertion does not catch it.

Its failure direction is also inverted from the rest of the list. A vacuous
green hides a failure that exists. This one manufactures agreement: in an
A/B, every arm passes, the run reads as "no effect", and a rule that genuinely
discriminated gets retired on the strength of a grader bug. A false negative
about your own experiment is harder to notice than a false green, because the
null result is the outcome you were already prepared to accept.

Fix is two-part, both cheap:

  • key on a structural marker the subject must create (a heading, a filename,
    a field), not a word it might merely mention;
  • sanity-check the grader against a known-bad arm before trusting a clean sweep
    — a grader that passes an arm you know failed is the finding, not a formality.

Provenance

One downstream consumer's A/B probe. The grader for a "did the arm record the
change?" axis scanned each arm's output file for the word drift — which the
fixture's own body text already contained ("a batch that drifts model mid-run").
All four arms passed, including three that recorded nothing. The run read as a
4/4 null result. Regraded on a port-note heading, the same arms split 0/3 bare
vs 1/1 ruled, and the rule under test was real.

Ships unprobed per the covenant.

Verification

  • Scoped: git diff --stat touches skills/ground-truth-gates/SKILL.md only.
  • Additive: word-diff shows zero removed words (the 1 deletion is a split line).
  • All new lines ≤ 80 chars by character count.
  • Non-ASCII limited to the house em-dash.
  • Branched fresh off upstream/main (a6ff7d0); staged by explicit pathspec.

… corpus cannot produce

Adds one fake-pass shape to the item-3 list. Unlike the empty-scan shapes
already there, this grader runs correctly over a non-empty input and still
passes every arm: it scans prose for a word the graded corpus itself contains,
so it scores the corpus rather than the behavior.

Its damage runs opposite to a vacuous green. A vacuous green hides a failure;
this manufactures agreement, so an A/B whose arms all pass reads as "no effect"
and retires a rule that actually worked.

Fix is two-part: key on a structural marker the subject must create, and
sanity-check the grader against a known-bad arm before trusting a clean sweep.

Ships unprobed per the covenant.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant