tooling(differential): a shape may be recorded without pinning a winner - #504
Merged
Conversation
…winner compare.py gains _WATCHED_DIFFS, a sibling of _RECORDED_DIFFS keyed per ledger the same way and held disjoint from it. The two carry different contracts: a shape beside a winner ADJUDICATES a contest (that is _RECORDED_DIFFS, whose rows _CROSS_RULE_WINNERS feeds to classify()), and a shape alone WATCHES a name nothing else watches (a radar name no test names and no contract corpus holds, whose only other watcher is a classification rule -- which asserts a diff is intended, not what it is, so a shape moving inside the rule's `fields` moves silently). Severity follows the row's kind first and the name's tier second. A contest row is fatal on either tier because it carries an argument that a moved shape has made false. A watched row on a contract name fails the run, as an unexplained diff there would; on a radar name it prints under a new non-fatal `MOVED SHAPE (radar)` block, parallel to `UNCLASSIFIED (radar)`. Fatal-on-radar is reserved for the [[never]] shape: a per-name deliberate choice carrying a `why`. A measured snapshot is not one, and the only repair a fired snapshot admits is to re-snapshot, which would make the row a changelog entry in a gate's clothing. recorded_diff_mismatches is unchanged and is called once per roster; main() splits the watched findings by the compared entry's tier. The tier is read off the post-skip entries, an order-None entry deciding where one exists and the first-loaded (contract-first) entry otherwise -- not off order-None entries alone, because a name compared under a declared order ALONE sits in `compared` with no default-order entry, and a watched row on it reports with `measured` None and would have fallen out of both lists and printed nowhere. The departed-name refusal and the NOT CHECKED note read the union of the two dicts; the refusal's repair text is split per roster, naming the _CROSS_RULE_WINNERS partner only for contest rows, since a watched row has none to delete. A name present in both dicts is refused pre-worker, mirroring the pytest-speed guard, because the tool may not assume the suite ran. `assert set(winners) == set(shapes)` in test_every_pinned_winner_has_a_recorded_shape is deliberately KEPT -- the spec's §1 (relax to `<=`) is declined. With a second dict for standalone shapes, a shape here with no winner is not an orphan to tolerate but a row in the wrong dict, i.e. a deleted-winner slip, and the equality is what still catches it. The per-row value checks are factored into _check_shape_rows and applied to both dicts; a new guard asserts the watched roster names every ledger on disk, is disjoint from _RECORDED_DIFFS per ledger, and -- stated on its own -- shares no name with _CROSS_RULE_WINNERS. The dict is EMPTY in this commit. The next one records the measured rows, so the machinery and the data bisect separately: with nothing recorded, nothing can move, and the gate's output at all four baselines is unchanged from before this commit (352 / 247 / 155 / 14 intentional, 0 unexplained, 0 radar unclassified, no finding block). Verified in memory around main(): a watched row corrupted on 'QC MP' (radar) at 2.0.0 prints `MOVED SHAPE (radar)` and exits 0; the same on '선생님' (contract) prints `MOVED SHAPE` and exits 1; one on 'de la Cruz née Vega' (declared-order-only, contract) exits 1 with no KeyError; a name placed in both dicts fails the new guard and a full run refuses it pre-worker. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…9-05 _WATCHED_DIFFS fills: 41 / 36 / 31 / 5 rows per ledger (1.4.0 / 2.0.0 / 2.1.0 / 2.2.0), 113 in all. The 2.0.0 section is 32 sole-watched rows plus the four #501 contests in their own block -- '田中さん 様.', '김민준 박사님', '선생님' (contract tier, so a move on them fails the run) and '田中さん, 様.' (corpus_cjk_tolerated.jsonl, radar, prints). They move to _RECORDED_DIFFS the day #501 argues a winner. Population: 51 distinct sole-watched names, not the 52 a tests/-only scan gives. Four names carry 1.4.0 contest rows in _RECORDED_DIFFS and are excluded there -- 'Bob Jones, author', 'Carod i', 'MD, PHD', 'van ma van'; three return at a 2.x baseline with no contest row, and 'Carod i' diffs under the default order at 1.4.0 only, so it has no row anywhere. 49 of the 51 sit in corpus_issues.jsonl and 3 in corpus.jsonl, 'dr Vincent van Gogh dr' in both. The header carries the population, the recompute (spy on classify() through main(), keep order None) and the two limits (a lower bound; no completeness guard, which must be a run-time NOTE since only a run knows "diffs at some baseline"). Every row was measured by a run before this commit, and every run re-measures it: at all four baselines the gate exits 0 with no MOVED SHAPE, MOVED SHAPE (radar) or NOT CHECKED block, and the intentional counts are unmoved at 352 / 247 / 155 / 14 with 0 unexplained and 0 radar unclassified. Mutation controls, in memory around main(), worktree untouched: 1. 'QC MP' corrupted at 1.4.0 (radar, sole-watched; its only row -- it does not diff at 2.0.0): "MOVED SHAPE (radar) expected_since_1.4.0.toml: 1 watched diff shape(s) disagree with this run", exit 0. 'Smith Dr' at 2.0.0 likewise, exit 0. 2. '선생님' corrupted at 2.0.0 (contract, #501): "MOVED SHAPE expected_since_2.0.0.toml: 1 watched diff shape(s) disagree with this run ... This fails the run because the name is contract tier", exit 1. 3. 'Ph. D., Jr.' corrupted at 2.1.0 (the spec's worst case, 30 silent shapes before this commit): "MOVED SHAPE (radar) expected_since_2.1.0.toml: 1 watched diff shape(s) disagree with this run", exit 0. 4. 'MD, PHD' inserted into the 1.4.0 watched section beside its contest row: a full run refuses pre-worker, before any "baseline:" line -- "1 name(s) sit in both _RECORDED_DIFFS['expected_since_1.4.0.toml'] and _WATCHED_DIFFS['expected_since_1.4.0.toml']" -- and test_the_watched_roster_is_disjoint_and_names_every_ledger fails naming it. The disjointness guard gains the vacuity assertion (`assert any(_WATCHED_DIFFS.values())`), and an all-empty roster is refused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The MOVED SHAPE section of tools/differential/README.md covered one roster and became false for two. It now opens with the two contracts -- a shape beside a winner ADJUDICATES a contest (_RECORDED_DIFFS), a shape alone WATCHES a name no winner is pinned for (_WATCHED_DIFFS), most of those sole-watched and four of them #501's measured contests, whose Case(...) row already pins the parse -- and says the two are disjoint, refused pre-worker and at pytest speed when a name sits in both, and that a watched row MOVES rather than gaining a partner the day a winner is argued. The MOVED SHAPE bullet carries the severity rule: a contest row is fatal on both tiers because it carries an argument; a watched row follows the tier of the DEFAULT-ORDER entry it was measured on -- fatal on a contract name, printed under MOVED SHAPE (radar) on a radar one, since fatal-on-radar is the [[never]] shape and a measured snapshot is not one. 'John Smith, Dr.' is the case that separates the entry's tier from the name's; it carries no row, and the text says so. The departed-name refusal and NOT CHECKED read the union, and the refusal names the _CROSS_RULE_WINNERS partner for contest rows only. The tier paragraph stops describing the 21-of-31 radar overlap as unreconciled history and states the rule, keeping the 2026-09-03 measurement past-tensed with its recompute. The "four checks read the flag differently" paragraph notes that the shape blocks, (radar) included, read no flag. The per-ledger sentence names both dicts. expected_since_1.4.0.toml:435 speaks of a contest row and stays true. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
decisions.md gains the eighth ledger arc, "the watched-shapes arc (2026-09-05, #501)", after the recorded-shapes arc and in its form. Decided: a shape may be recorded without a winner, in a SECOND roster and not by relaxing set(winners) == set(shapes) -- the run cannot see _CROSS_RULE_WINNERS, so a row's kind must be data in the tool, and the equality catches the deleted-winner slip a relaxation would legalize; severity follows the row's kind first and the tier second, on amendment A's three grounds, answering the narrow tier question only and applying #468 rather than reversing it; the tier read is the DEFAULT-ORDER ENTRY's, not the name's ('John Smith, Dr.' the case); the roster's defining property is "no winner pinned", not "nothing else watches" -- false of the four #501 rows, whose Case(...) rows adjudicate the parse; and every sole-watched diff is recorded (109 plus the four contests), not the exposed subset. Found: the spec's 385/71 was gross of #452 AND measured over its first-draft population (the four 1.4.0 contest names in, the #501 four out -- 113 rows by coincidence); over the roster's own 113 rows the recipe gives 390/72 gross, 337/64 net, 44 other-rule (per baseline 105/96/23, 158/137/20, 127/104/1, 0/0/0), and over the 109 sole-watched rows 367/68, 314/60, 35 other-rule; the 8 diffs alone on their rule are the 72 -> 64 drop but carry only 24 of the 53 caught, the other 29 coming from diffs whose rule's other names do not cover its fields; the worst case ('Ph. D., Jr.', 30 at both 2.x) and the 35 other-rule figure are robust across populations; the tests/ scan cannot see compare.py (four 1.4.0 names, and 'Carod i' leaves -- 51 not 52); the population definition's "explained by a ledger rule" clause, which the recipe carried and the definition did not; corpus_shapes.jsonl's row counted entries where the population counts names (14 / 11 / the NOT CHECKED trio reproduce; the spec's "16 entries" did not -- 18 by the recipe given -- and is not carried); 49 + 3 with one overlap. Declined: the relaxation, fatal-on-radar for snapshots, pinning the six #501 winners, a checked-in generator. The measurement paragraph carries 41/36/31/5 = 113, the four mutation controls re-run against the amended data commit with their header lines, 352/247/155/14 unmoved, and the dated quiet-tree count (5 commits, 38+15 lines) with its recompute. The recorded-shapes arc's "ONE roster, TWO checks" bullet gains a one-line supersession pointer (union for the departed-name refusal and NOT CHECKED; the shape comparison runs once per roster); the corpus-tier arc's tier bullet gains one sentence on the two recorded things that outrank the tier on a radar name. mechanisms.md#RECORDED-ROSTERS adds _WATCHED_DIFFS to "Lives in" and one sentence: an expectation whose only consumer is the run that checks it is this entry's pure case. #CROSS-RULE-OUTCOME-PINS says a shape recorded alone pins no winner and lives in the other roster. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…dger glob (#503) Step 1 of the release checklist enumerated three compare.py commands against four ledgers with rules, so `--baseline 2.1.0` -- the one no line reached -- got no dormancy check, no over-declaration check and no recorded-shape check from any releaser, and the "EVERY baseline" sentence above the list was not what the list did. #503 weighed three fixes; this takes the second, a loop over the expected_since_*.toml glob, for the reason compare.py's `--corpus` default is a glob: a baseline that has to be asked for by name is one that stops being run. The loop redirects each run to a file and never pipes it, and the checklist line says why (zsh replaces the exit code with the pipe's). The #503 NOTE block that sat in the checklist is replaced by a two-sentence record: what the gap was, that it closed 2026-09-05, and that THIS bundle armed it before closing it -- the shape gap was "latent" only while expected_since_2.1.0.toml's roster sections were empty, and the data commit two below put 31 watched rows there. The cost sentence stays; it points at the decisions.md timing finding that retired every cost argument in this repo's differential prose. Run once from the worktree root exactly as written, 2026-09-05, all four exiting 0 with no finding block: 1.4.0 corpus: 1113 names; intentional diffs: 352; unexplained: 0; radar unclassified: 0; 247 of 352 changed names are Latin-only 2.0.0 corpus: 1120 names; intentional diffs: 247; unexplained: 0; radar unclassified: 0; 151 of 247 changed names are Latin-only 2.1.0 corpus: 1120 names; intentional diffs: 155; unexplained: 0; radar unclassified: 0; 151 of 155 changed names are Latin-only 2.2.0 corpus: 1120 names; intentional diffs: 14; unexplained: 0; radar unclassified: 0; 14 of 14 changed names are Latin-only Closes #503. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Five sentences the design-docs reviewer's second pass found false or narrower than the code, none of them structural. decisions.md: the supersession pointer in the recorded-shapes arc said the exit code "reads only one" of the two shape-comparison halves; it reads a contest row's mismatch on either tier and a watched row's where its default-order entry is contract tier -- measured, `선생님` corrupted at 2.0.0 exits 1 -- and the reason the comparison runs once per roster is that one call could not tell the two apart. The corpus-tier arc's forward pointer said a watched row fails "on a contract default-order entry"; the tier is the first-loaded entry's where a name has no default-order entry, so the sentence now says so. AGENTS.md: #503 is closed by this branch's merge, not on a date the record can know; the 2.1.0 watched-row count is stated as an invariant with its live expression rather than as a standing digit; and the cost sentence now points at the bullet that actually carries the timing recompute (the rule-order arc's Declined bullet), which the #497 finding it cited says it does not restate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #504 +/- ##
=======================================
Coverage 98.66% 98.66%
=======================================
Files 45 45
Lines 3219 3219
=======================================
Hits 3176 3176
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…cklist
A. tools/differential/compare.py
1. Fail-closed roster sections: main() refuses pre-worker, beside the
`both` refusal, a ledger absent from _RECORDED_DIFFS or _WATCHED_DIFFS,
naming the dict(s), the ledger and the pytest-speed guard; the two
`.get(ledger.name, {})` reads are `[]` after it. Measured on HEAD: the
1.4.0 key deleted from _WATCHED_DIFFS ran 41 rows unchecked, same 375
lines (the baseline: tell's env path differs between any two runs),
exit 0.
2. _CORPUS_TIERS values validated at load: a value outside
contract/radar is refused naming the file and the value; the
`!= "radar"` partition stays as defense in depth and its comment
says the value set is validated upstream.
3. Header corrections: RECOMPUTE excludes test_ledger_guards.py from
the literal set (38/28/27/4 without it, 41/32/31/5 with, recomputed);
fatal-on-radar is reserved for a per-name deliberate choice ([[never]]
or a contest row); "at most one" order-None entry; the _RECORDED_DIFFS
pointer names its rosters; 'Carod i' has no row IN THIS DICT; the
MEASURED bullet is scoped to main() and lists the missing-section
refusal; the "two checks" are named.
4. README parity: the [[never]]-only sentence corrected; the slipped
checklist paragraph rewritten (baselines derive from the ledger glob;
#503 in past tense); the first-file RECOMPUTE says it tiers a NAME
and points at the entry-tier rule for watched rows.
B. tests
5. tests/v2/test_differential.py: _run_main_over (multi-corpus,
order-bearing driver) and _default_order_diff; tests for the
'John Smith, Dr.' shape, the declared-order-only name (parametrized
contract/radar), the bogus tier value, the missing roster section
(parametrized over both dicts), the departed-name refusal with both
lists populated; the `both` refusal test matches "sit in both". No
existing test asserted the old .get behavior.
6. tests/v2/test_ledger_guards.py: _check_shape_rows' docstring says a
watched row is fed to nothing and why the vocabulary is shared; the
_RULE_FIELDS message computes its '_ambiguities' example from the
dicts; the disjointness guard's docstring attributes the key
equality to the sibling rosters' convention and says the run now
agrees; a parametrized negative test for _check_shape_rows under the
_WATCHED_DIFFS label, one case per illegal row kind.
C. docs
7. AGENTS.md step 8: FOUR filename-keyed rosters, _WATCHED_DIFFS added
(in compare.py) with _RECORDED_DIFFS enrolled transitively; the
release note names both non-fatal blocks.
8. decisions.md watched-shapes arc: task numbers replaced by the kind
of review; two FOUND bullets with recipes (the missing-section
swallow; the unpinned tier rule); the first DECIDED bullet notes the
run now holds the key equality too.
Mutant kills, each applied to a copy of compare.py outside the worktree
and run against tests/v2/test_differential.py + test_ledger_guards.py:
M4 order-None entries only -> killed by the declared-order-only
test (both params; KeyError mid-report)
M4b first-loaded always wins -> killed by the 'John Smith, Dr.' test
M1 `== "contract"` fail-open -> EQUIVALENT after A2: no third value
reaches the partition. With the load
check also removed (the pre-A2 tree),
killed by the bogus-tier test.
.get default restored -> killed by the missing-section test
tier value check removed -> killed by the bogus-tier test
Verified: uv run pytest -q 6503 passed, 0 xpass; ruff clean; mypy clean;
the checklist loop four exit 0 with 352/247/155/14, 0/0, no finding block;
the doc subset 696 passed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A diff shape may now be recorded without pinning a winner.
compare._WATCHED_DIFFSis a second roster beside_RECORDED_DIFFS, and every run checks it: 113 measured shapes for the names only a ledger rule watched — 109 sole-watched radar names plus the four #501 contests at 2.0.0. Refs #501 (scoped down, stays open). Closes #503.The two rosters, and what severity follows
_RECORDED_DIFFSkeeps its contract: a shape beside a pinned winner ADJUDICATES a contest, andset(winners) == set(shapes)stays intact._WATCHED_DIFFSis for a shape with no winner pinned — most of its rows are also names nothing else watches (the population the sweep drew), and a contested-but-unadjudicated row (#501) has aCase(...)row that already adjudicates its parse. The two are disjoint per ledger, checked at pytest speed and refused pre-worker by the run.Severity follows the row's kind and the compared entry's tier. A contest row is fatal on both tiers because it carries an argument (
[[never]]'s reasoning: fatal-on-radar is reserved for a per-name deliberate choice). A watched row is a snapshot: it fails the run on a contract-tier default-order entry and prints under a non-fatalMOVED SHAPE (radar)block on a radar one — the tier arc's "watch without promising", applied rather than reversed. The tier read is the default-order entry's, since that is the comparison a recorded shape is measured on:'John Smith, Dr.'is contract only as shape 4 (FAMILY_FIRST) and radar by default order, so a watched row on it would print and not fail while the family-first promise stands.Why a second dict rather than the spec's first idea (relaxing
==):compare.pycannot read_CROSS_RULE_WINNERS, so under one dict the run could not tell a contest row from a snapshot — one repair text and one severity for both — and an orphaned shape would be indistinguishable from a deliberate one.The rows
41 / 36 / 31 / 5 across the four ledgers, measured by spying on
classify()through realmain()runs (recipe in the roster header). Population: radar-only, held by no contract corpus, named by no string literal undertests/outside the guard module, explained by a rule at that baseline, and not already keyed in_RECORDED_DIFFS— the last clause excludes four 1.4.0 names with contest rows, andCarod ithen leaves entirely (51 names, not the 52 atests/-only scan gives). Nothing else changed: no ledger, no corpus, nothing undernameparser/.Motivation, measured over the roster: 390 alternative shapes would classify to the same rule unnoticed (337 once the over-declaration check is credited), 44 more would route to a different rule with nothing pinning it;
'Ph. D., Jr.'alone has 30 silent shapes at each 2.x baseline. The decisions arc carries every figure with its recipe and records where the spec's first-draft numbers were measured on a population it later retracted.Verification
Intentional counts 352 / 247 / 155 / 14 at 1.4.0 / 2.0.0 / 2.1.0 / 2.2.0, unexplained 0, radar-unclassified 0, byte-identical at every commit; no
MOVED SHAPE,MOVED SHAPE (radar)orNOT CHECKEDblock at any baseline, which is the proof the 113 shapes are measured rather than plausible. Five mutation controls in memory aroundmain(), none inert:QC MP@1.4.0 andSmith Dr@2.0.0 →MOVED SHAPE (radar), exit 0;선생님@2.0.0 (contract) →MOVED SHAPE, exit 1;Ph. D., Jr.@2.1.0 → radar block, exit 0;MD, PHDinserted into both dicts → pre-worker refusal and the guard names it. Full suite green; ruff and mypy clean. Each commit went through a spec-conformance and a code-quality review; the design-docs reviewer ran twice on the decisions arc and its first-round findings (population-mismatched figures, a false "union" claim, a missing cross-pointer) are fixed in the amended commit.The release checklist now derives its baselines from the ledger glob (#503): the 31 rows this PR puts at 2.1.0 armed the gap that note had recorded as latent.
Review round (
c669729)A four-agent review of the PR (code, tests, comments, silent failures) found nothing wrong in the tier map or the exit wiring and four things worth fixing, all fixed in one commit: a ledger with NO roster section was invisible to the run (
.get(…, {})— 41 rows unchecked, exit 0, output identical up to the worker path) and is now refused pre-worker beside the both-dicts refusal, as_CORPUS_TIERSrefuses a file with no tier; tier VALUES are validated at load, so the "contract tier" message cannot lie on a typo; the default-order-entry tier rule was unpinned — three mutants survived the suite and all four gates byte-identically — and is now pinned by a two-corpus'John Smith, Dr.'test and a declared-order-only test (mutant table in the commit); and same-PR staleness in the prose — a README paragraph still describing the three-command checklist this PR replaced, AGENTS.md's "THREE filename-keyed rosters" (four now), the release note naming only one of the two non-fatal radar blocks, a RECOMPUTE recipe that omitted the guard-module exclusion (38/28/27/4 run literally), and "fatal-on-radar is reserved for[[never]]" beside a rule that makes contest rows fatal too.🤖 Generated with Claude Code