diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 5d763d69..aafb8e93 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -755,6 +755,39 @@ The fifth ledger arc: the gate gains a column it never had. `initials()` is a de - WHAT IT FOUND, measured 2026-09-01 by patching compare.py's own worker to emit `initials()` and comparing against the tree — and measured on THIS BRANCH'S BASE, before `31623d9` fixed #462, which is the tree the parenthetical describes: there the tree matched 2.2.0 exactly, 0 diffs of any kind. Three clusters of initials-only names, 140 at 1.4.0 and 28 at each of 2.0.0 and 2.1.0. On the SHIPPED tree the same recipe gives 126 at 1.4.0 and 42 / 42 / 14 at 2.0.0 / 2.1.0 / 2.2.0 (re-measured 2026-09-02; the gate itself is the recipe now, see RECOMPUTE). The two readings reconcile through cluster (ii) alone — 126 + 14 = 140, 28 + 14 = 42, 0 + 14 = 14 — the 14 #462 names having moved from the 1.4.0 side to the 2.x side when the fix landed. (i) The R2 readmission, 27 names at 1.4.0, 2.0.0 and 2.1.0 — `Anh Do` → `A. D.`, `Juan van der` → `J. v. d.` — shipped in 2.2.0 with a release note and the R2 entry above but no ledger rule, because none could see it; now `fix(#385/#402)`, a literal list of the 27 (the shape "a part of nothing but particles" is no property of the raw string), plus `de los Santos` under its own `fix(#360)` rule since ITS initials moved when `los` became a particle, not with R2. (ii) The facade dropping a middle `E`/`Y` initial, 14 names at 1.4.0 only, all but one radar tier — the population beside the count, per docs/design/AGENTS.md's population rule, is the names carrying such a letter in the MIDDLE or FAMILY group, which is the only position the bug reached, and there are 14 of them: 14 of 14 in that position moved. The `fix(#462)` rule's regex (`(?:^|[\s,])(?:[EY]|[EeYy]\.)(?=[\s,]|$)`) reaches 18 corpus names, four more than the population, and the other four (`E Anne D`, `E Jones`, `E Maria`, `Y. L.`) carry the letter in the GIVEN group, which has always initialed every word it holds, so nothing moved. The discriminator is the GROUP and not the position: `E Anne D,Leonardo` leads with the E too, but its comma re-roles the run into the family group, and it is one of the 14 that move: #462, a 1.4 parity break shipped in 2.0.0 and recorded nowhere. Fixed in the same PR (R3's bullet below); the 14 then move against the three 2.x baselines instead and carry a `fix(#462)` rule in each. (iii) Per-word grouping, 98 names at 1.4.0 only: 1.4.0 initialed one group per `*_list` element and rendered a multi-word element's letters with the separator and no delimiter — `Juan Velasquez y Garcia` → `J. V G.` at 1.4.0 and `J. V. G.` here, `Abdul Salam Hassan` → `A S. H.` and `A. S. H.` — and the 2.0 facade initials each word. Measure with the FULL name, not with the run alone: `Velasquez y Garcia` and `Abdul Salam` as whole inputs give `V. G.` and `A. S.` at 1.4.0 and here alike, the run being the family and initialed per word either way, so the bare forms this bullet illustrated with until 2026-09-02 reproduced nothing. Strip periods and spaces and all 98 agree with 1.4.0 letter for letter. Changed in 2.0.0, mentioned once in passing (the #408 bullet under R3: "v1's initials GRANULARITY"), never classified. Classified LATE, on purpose: mechanisms.md#FACADE-CONTRACT promises 1.4-warning-free code keeps working except for release-log-classified fixes, and a classification three minors overdue is still owed, so the 2.3.0 log carries a bullet that says the behavior has held since 2.0.0. Four rules rather than one, because the Latin-alternation guard keys a rule to a single copied vocabulary and the 98 copy three (CONJUNCTIONS 66, BOUND_GIVEN_NAMES 19, PARTICLES 11) plus the `Ph. D.` merge (2). The connective rule cannot claim cluster (ii), and it takes BOTH of its exclusions — the case-sensitivity and the trailing whitespace lookahead — to keep it out. Of the 14, 7 carry a bare capital `E` and 7 a dotted `E.`/`e.`: the case-sensitivity excludes the bare capitals, and the alternation's trailing whitespace lookahead excludes the dotted ones whatever their case, a period not being whitespace. Since the fix in `31623d9` those names agree with 1.4.0 and diff only against the 2.x baselines, so neither half is keeping two live clusters apart today. What they do is make any FUTURE facade initials change on these shapes surface as UNEXPLAINED at 1.4.0 rather than be absorbed here as per-word grouping. Gate totals moved from 226 / 205 / 113 / 0 intentional to 352 / 247 / 155 / 14, zero unexplained, exactly the new rules' explained counts. - RECOMPUTE: load `tools/differential/compare.py` by path; build the entries as main() does (`_load_entries`, `_load_shapes`, `_CORPUS_TIERS`, dedup by (name, order), drop order-bearing entries below their shape's minimum); patch `_worker_source` to append `_initials` to both rows; `_run_worker`; compare against the tree's facade and core. TWO RECIPES, and only the second of them needs that patch. (a) The CLASSIFIED TOTAL per baseline is read off the gate itself, no patch and no in-memory run: `compare.py --baseline X` prints a `## issue (N)` heading per rule, and the total is the sum of the headings of the rules whose `fields = ["_initials"]` — at 1.4.0 `fix(#385/#402)` 27, `fix(#360) los` 1 and the four `fix(initials-per-word)` rules 19 / 66 / 11 / 2, summing to 126; at 2.0.0 and at 2.1.0 `fix(#385/#402)` 27, `fix(#360) los` 1 and `fix(#462)` 14, summing to 42; at 2.2.0 `fix(#462)` alone, 14. What the gate does NOT print is the per-name detail — a heading lists at most ten of its names and no diff fields at all, and the UNEXPLAINED and radar blocks are empty on a green tree — so it answers how many and under which rule, never which names or which surface moved. (b) The PER-NAME SPLIT, and the pre-fix 140 / 28, need the in-memory run described above: load compare.py by path, run main()'s comparison loop or hook `diffing`, and keep the names whose diff is exactly `{"_initials"}`. READ THE PATCH STEP AS HISTORY, and it belongs to (b) alone: since `a748862` the shipped worker template emits `_initials` itself, so patching `_worker_source` is what a measurement BELOW that commit needs and re-applying it on this tree changes nothing. The pre-fix 140 / 28 must be taken on the branch's base, before `31623d9`, rather than on this tree. Split "initials only" from "roles and initials"; within the former, lowercase and strip `[.\s]` to separate per-word grouping (equal after stripping) from content changes. The guard's liveness is proved by mutation, not by the green run: with `not diff and` removed from the guard, a roles-and-initials name has its role diff REPLACED by `{_initials}`, which no existing rule declares, and the 1.4.0 run reports 117 of them as no longer classified — 79 unexplained plus 38 radar, from 0 and 0. The same mutation is where the first bullet's churn figures come from: the rules that lose at least one name are the rules strict subset semantics would have had to widen, counted against the rules that explain anything at that baseline (53 of 72, 43 of 66, 36 of 59 at 1.4.0 / 2.0.0 / 2.1.0; re-derived 2026-09-02, the 1.4.0 numerator having been recorded as 43 until then). +### differential-ledger, the rule-order arc (2026-09-02, #382) + +The sixth ledger arc, and the one that closes the question mechanisms.md#LEDGER-RULE-SEPARATION has carried open since #375's reorder mutation: file order settles every contest, and nothing said which contests existed. Mechanics — the TOML shape, the required `why`, the trap that a rule key written below the block joins the exemption — are owned by tools/differential/README.md; these are the decisions. Every figure below was measured 2026-09-02 against the real 1.4.0 wheel or recomputed from the checked-in files, and each carries its recipe. + +Decisions that landed: + +- 2026-09-02 #382 — an order-decided contest must be DECLARED, and narrow-first is the declaration-free default. A pair is a contest when the later rule's `fields` are a strict subset of the earlier one's, some corpus name's `name_regex` reaches both, and some comparison order reaches both; where the EARLIER rule is the wider one it carries a `precedes_narrower` block naming the later rule and saying why it is the classifier of record — usually what it describes that the later one does not, and in the three regex-accident pairs (`fix(#296)`/jr, and the two `fix(cjk-glued-honorific-peel)`/`fix(suffix-routing)` pairs) that the later rule reaches the name through its REGEX rather than by describing it — both `fix(suffix-routing)` patterns are anchored at both ends and open on a bare `\S+` run, which swallows the trailing comma of `Smith,` and matches hangul as readily as Latin, while the prose they carry is scoped to a comma-less two-token Latin name. Only ONE of the three is false outright — `fix(#296)`/jr, where 1.4.0 read `'Smith, Jr.'` as title `'Jr.'`, first `'Smith'`, so the family→suffix movement the jr rule describes never happens on it. The other two are PARTIAL, and symmetrically so: on `김민준씨 Jr.` and on `田中さん II` alike the trailing token really does leave `family` for `suffix` (1.4.0 first `'田中さん'`, last `'II'` → tree last `'田中'`, suffix `'さん, II'`, exactly the movement the numeral rule shows on its own `John V`), and what neither `fix(suffix-routing)` rule can describe is the peel and the segmentation that move `given`. Recomputed 2026-09-02 against the 1.4.0 wheel; the exemptions themselves claim only the scope mismatch, so do not read a falsity claim into the two partial ones. `compare.py` refuses the run, before the worker spawns, where such a pair goes undeclared. Reordering is NOT the alternative fix and the failure message says so: it moves which rule classifies a name and breaks `_CROSS_RULE_WINNERS`. What the check buys is that nothing else in the suite can see the hazard at all: `_CORPUS_CLAIMS` measures each rule alone, the gate total is per-corpus, and `_CROSS_RULE_WINNERS` pins contested outcomes only for names somebody hand-added. +- 2026-09-02 #382 — the check covers NESTED pairs, and nesting is sufficient for an order-decided contest rather than necessary. Two rules whose `fields` merely INTERSECT both admit any diff inside that intersection, so `classify()` hands such a name to whichever is written first exactly as it does for a nested pair, and `order_contests` cannot see it. That class sits outside the check by the reasoning `order_contests`' docstring already states for EQUAL `fields` — neither rule is narrower, so "narrow-first" says nothing about the pair, `precedes_narrower` has no narrower rule to name, and `_CROSS_RULE_WINNERS` stays the instrument — which covers every pair where neither `fields` set contains the other, equal `fields` being its special case. Not oversight, and not free either: measured 2026-09-02 over the four ledgers, counting pairs that share a corpus name and whose `orders` are not disjoint, 11 are strictly nested wide-first (what the check refuses), 40 nest in either direction, 11 have equal `fields`, and 111 have any non-empty intersection — so 60 overlap without nesting or equality. Widening to the general predicate would demand 111 written justifications where the real number is eleven, the same kind of answer this entry already gives below about the 657 figure: a predicate whose roster nobody can write out one justification at a time is not a usable predicate, whichever condition inflated it. The blind spot is worked and filed as [#498](https://github.com/derek73/python-nameparser/issues/498): `fix(#271/#272/#298)` (`{family, given, middle}`) and `fix(cjk-delimited-nickname)` (`{family, given, nickname}`) intersect in `{family, given}` without nesting, three contract-tier corpus names (`マイケル・ジャクソン`, `威廉・莎士比亚`, `高橋・一郎`) diff exactly those two roles against the 1.4.0 wheel, swapping the two rules reattributes all three, none is in `_CROSS_RULE_WINNERS`, and `order_contests` lists the pair in neither arrangement. RECOMPUTE the five figures with `_rule_reach` per ledger over the `corpus*.jsonl` union, classifying each pair by whether `a.fields & b.fields`, `b.fields < a.fields`, `a.fields < b.fields` or `a.fields == b.fields`; #498's body carries the loop. +- 2026-09-02 #382 — an ESCAPE HATCH here, where #452's and #456's were declined, and on the terms #452 set. Both of those bans were free to state: measured at the time, 0 of the 179 rules across the three ledgers then on disk had #456's shape, and #452's fourteen over-declarations (3 of 67 EXPLAINING rules at 1.4.0, 5 of 58 at 2.0.0, 6 of 51 at 2.1.0 — a different and smaller population than #456's 179, which counts every rule) were all narrowed before the check landed, so neither ban had to argue with a rule that was correct as written. #452's entry states the price of that strictness — "the first rule that genuinely needs a wider declaration has to argue for a key the way `dormant` was argued for in #373". Read that as the PROCEDURE it sets, not as a hatch this key opens: `precedes_narrower` is not a wider `fields` declaration and does nothing for an over-declared rule, which still exits the run non-zero. What carries over is the standard of proof, and here it is measured rather than asserted: eleven pairs in `expected_since_1.4.0.toml` are wide-first and every one of them is correct where it sits, so a ban would have had eleven rules to reorder or eleven prose descriptions to falsify. The hatch is narrowed the way `dormant` is: it names ONE rule (a blanket opt-out would be inherited by every narrower rule added later, which is the widening the check exists to refuse), the `why` is required, and a declaration standing over a pair that is no longer contested is refused as loudly as an undeclared contest. +- 2026-09-02 #382 (decided in review) — the vacancy half REFUSES only on a full run and prints a NOTE under `--corpus`. The two checks are not symmetric under a narrowed name set, and the asymmetry is the whole reason: narrowing removes contests, so for the undeclared check `--corpus` is only ever more lenient (it can only under-report, never false-alarm — do not call that fail-closed, which this repo uses for the rosters that REFUSE on a missing entry), while for the vacancy check it INVERTS — a live declaration whose contested names all sit outside the subset reads as vacant. Shipped as a regression and caught in review: as first written, a `--corpus` run against the 1.4 ledger exited 1 and told the contributor to delete exemptions the full gate needs — measured, each of the six corpora run ALONE reports vacancies, 11 of the 11 for `corpus.jsonl`, `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and 8 / 7 / 5 for the other three — after which deleting them would have made the full run refuse with that many undeclared contests. Read the shape and not the digits: the number varies with the subset, and only zero would have been safe. The file already treats `--corpus` differently in two places and this is the third, so the three should be read together rather than made uniform: the corpus-floor roster is SKIPPED entirely under the flag, `over_declared_rules` still FAILS the run and appends a NOTE saying the union it computed is over a subset, and the vacancy check does not fail at all. The strengths differ because the error directions do — only the vacancy check inverts under narrowing. +- 2026-09-02 #382 — TWO name populations, deliberately, rather than one shared function. `main()` must check the corpus it ACTUALLY compares, because `--corpus` narrows it; the unit guard in tests/v2/test_ledger_guards.py must check every corpus on disk, so that a rule added by a later bundle is checked at pytest speed with no baseline wheel. Forcing one function would break `--corpus`. They agree by construction instead: `_entry_name` in tests/v2/_differential_fixtures.py says in its docstring that it mirrors `compare.py`'s `_load_entries`, and both read the same `corpus*.jsonl` glob. +- 2026-09-02 #382 — the recorded negative control is deliberately BLIND to `precedes_narrower`. `_ORDER_EXEMPTION_EFFECT` records the eleven pairs and their name counts from `order_contests`, which never reads a declaration; a control that consulted the mechanism it controls for would measure nothing, and would go green the moment the predicate stopped finding anything at all. Its assertion that not every ledger's list is empty is there for exactly that failure (mechanisms.md#RECORDED-ROSTERS). + +Found rather than decided, and worth as much: + +- **#382 option 3's premise is FALSE, and the crux name is in the corpus.** The issue proposed a mechanical narrow-first check — refuse the wide-first pair, or sort by specificity. `fields`-subset is a proxy for specificity, and it is the wrong one where a wider rule describes a COMPOUND behavior its component rule does not. `马丁·路德·金씨` divides on the nakaguro AND peels its glued hangul honorific: `fix(#272/#308) nakaguro division and a glued hangul honorific in one name` describes what happens to it, `fix(cjk-glued-honorific-peel) glued honorific peels into suffix` describes half of it, and the WIDER rule wins by position, correctly. Narrow-first would have reattributed the name to the rule describing half — #372's defect reintroduced by the check meant to prevent it. The name is contract tier (`corpus_rules.jsonl`) and appears nowhere in `_CROSS_RULE_WINNERS`, so it is exactly the crux #382 was filed over and exactly the name no existing guard was watching. +- **The predicate needed a THIRD key, found in review.** The first implementation read `name_regex` and `fields`. `_entry_matches` narrows by `orders` as well: two rules declaring disjoint `orders` never see the same comparison, so file order decides nothing between them however nested their `fields` are. Omitting it made the detector read different boundaries from the predicate it models — docs/design/AGENTS.md axis 2 — and would have demanded a written justification for a hazard that cannot occur. It changes no figure today and is kept for correctness, not for its yield: measured over the four ledgers, adding the `orders` test removes 2 of 1350 nested pairs and 0 of the 657 wide-first ones. The nearest live shape is in both 2.x ledgers, where `fix(#399) a maiden marker bounds the particle chain that swallowed it` (`orders = ["DEFAULT"]`) and `fix(#399)/feat(#395) a consumed maiden marker leaves the family-first fold no given name` (`["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"]`) have nested `fields` and share a corpus name, `de la Cruz née Vega` — narrow-first today, so nothing reports it either way; invert that nesting and the omission would have demanded a `why` for a comparison that never happens. +- **Nine of the eleven pairs are LATENT, not live**, which is the honest statement of what a static predicate costs and what it buys. Measured against the 1.4.0 wheel, a pair is a live order-decided contest only where some co-matched name's ACTUAL diff is a subset of the narrower rule's `fields`. In nine of eleven the real diff needs a role the narrower rule does not declare, so that rule is ineligible for those names wherever it sits. Only two pairs are live: `fix(comma-family)`/`fix(comma-precomma-family)`, where `John Smith, Mr.` is the one such name, and the compound/peel pair below, where 15 of the 17 co-matched names have such a diff and nine of those are this pair's to decide — the other six go to rules written above both, so the two counts answer different questions and neither is derivable from the other. So the predicate OVER-REPORTS relative to the measured diffs, and the price of that is eleven reasons somebody had to write. What the nine buy is the hazard that would ACTIVATE if a rule's `fields` ever widened — written down before the three later bundles add rules, which is when it is cheap. +- **A whole error class in the first draft of the exemption prose: concluding a RULE's tier from its NAMES'.** Three separate `why` texts did it. Tier is a property of the corpus file a name comes from, and a rule's reach usually spans both tiers. Measured: `fix(cjk-glued-honorific-peel)` explains 17 names, 14 of them CONTRACT tier — say the tier and not the file, even though all 14 happen to sit in `corpus_cjk.jsonl` today: the tier is what the gate acts on and `_CORPUS_TIERS` is where to read it, while a name can join or leave a file without changing either — and deleting the rule sends 12 contract-tier names to UNEXPLAINED (two of the fourteen re-home to `fix(cjk-honorific-suffix)`; the other three of the seventeen are radar) — so the gate does demand that rule, whatever the tier of the names it is contested over. Only `fix(cjk-comma-compound)` is radar-only in the strong sense: 11 explained names, all radar, and its whole 23-name regex reach radar too. Worth recording as a caution and not only as a corrected fact — the review rounds on this prose found six false claims, then one, then two more of this class. Every one was in prose nothing can machine-check, and MOST were caught by re-running the wheel rather than by reading — but not all, which is the part worth keeping: of the first round's six, two came only from reading, one citing "a few rules below" for a quote 52 rules away and one opening "the one live pair of the eleven" while a second exemption in the same file declared itself live too. Distance-in-the-file and prose contradicting prose are the classes no wheel run can reach. +- **One exemption says "least wrong reading available", and that is a finding about the ledger's vocabulary.** `fix(cjk-comma-compound)` outranks `fix(cjk-glued-honorific-peel)` on nine names — 17 reach both regexes, 15 have a diff the peel rule's `{family, given, suffix}` admits, and 6 of those 15 are taken by rules written above both — and its label is untrue of three of the nine. `王先生, V.`, `田中さん, V.` and `김민준씨, V.` show no comma routing and no order flip; their whole diff is `{family, suffix}`, the glued peel alone, against `{family, given, suffix}` on the six genuine compounds. They land on the compound rule because its criterion is "the diff includes `family`", and `family` moves only because the peel took the honorific off it. The order must still stay: the peel rule's prose disclaims comma names, so promoting it would mislabel the six. The rule that would describe them is a family-side twin of `fix(cjk-comma-honorific-peel)`, which covers this shape for a POST-comma given name; writing one was weighed as [#496](https://github.com/derek73/python-nameparser/issues/496) and DECLINED (2026-09-02), and the Declined entry below carries the evidence, the #495 tension and the reopen trigger. Declining it discards no part of this finding: the compound rule's label is still wider than those three names are, which is what this bullet and the exemption's own `why` exist to say. +- **A guard can pin the winner of a contest and still let the recorded diff shape be wrong.** `test_the_recorded_rule_still_wins_each_contested_name` feeds `classify()` the RECORDED shape and never checks that shape against a real comparison, so a wrong shape that still routes to the same rule passes forever. This branch fixed one: `田中さん II` was recorded as diffing `{given, suffix}` and measures `{family, given, suffix}` against the 1.4.0 wheel, and the claim had been copied to four sites. The winner did not move, so every argument resting on it survived — which is why nothing noticed. Filed with the general shape as [#497](https://github.com/derek73/python-nameparser/issues/497). + +The measurement, and how to redo it. Eleven wide-first contests in `expected_since_1.4.0.toml`; 0 in each of the three 2.x ledgers. Six of the eleven are contested over at least one contract-tier name, five only over radar names — the division [#495](https://github.com/derek73/python-nameparser/issues/495) argues from, and it survives a name changing tier even though the two counts do not. The load-bearing contrast is between that eleven and what `fields`-subset alone says: dropped to nesting alone, with no corpus-reach and no `orders` condition, the 1.4 ledger holds 646 nested pairs of which 367 are wide-first, against the eleven contests it actually has — and across all four ledgers, 1350 nested pairs of which 657 are wide-first. Read the ratio and not the digits: the two answers differ by more than an order of magnitude and would still differ by one after any plausible drift, which is the argument — `fields`-subset alone is not a usable predicate, and it is the corpus-reach condition that makes the check something a person can answer eleven times. RECOMPUTE: load `tools/differential/compare.py` by path and call `order_contests(rules, names)` per ledger, with `names` the union of `_load_entries` over the `corpus*.jsonl` glob — 1116 distinct names today, 326 of them reached by a corpus `_CORPUS_TIERS` marks contract. READ THAT ROSTER, do not reconstruct it from a file's name or size: `corpus.jsonl` is the LARGEST corpus at 486 distinct names and it is RADAR, having been demoted by #468 as a v1-test-bank scrape, so the contract tier is the three SMALL files (`corpus_cjk.jsonl` 73, `corpus_rules.jsonl` 248, `corpus_shapes.jsonl` 35) and the split runs 326 / 790 rather than the other way about. Reading `corpus.jsonl` as contract gives exactly 786 / 330, which is the wrong answer this figure attracts — it was measured that way twice before a third recompute caught it, and the gate's own `corpora:` line is the cheap check, since it prints the contract files first; for the contrast, re-run the same combinations keeping only the strict-`fields`-subset test. `undeclared_contests` and `vacant_exemptions` both return empty over every ledger, which is what the unit guard asserts. Note WHICH population each caller reads: `main()` checks the entries the run loaded — 1120 at every baseline, of which 1113 compare at 1.4.0 once the shape-minimum skip runs, since that skip happens after the check — while the unit guard reads every corpus on disk. The arc moved no classification: the gate reports 352 / 247 / 155 / 14 intentional diffs at 1.4.0 / 2.0.0 / 2.1.0 / 2.2.0, 0 unexplained and 0 radar-unclassified at all four, unchanged across the whole branch. + +Declined: + +- Reordering the eleven (2026-09-02) — the obvious way to make the check unnecessary, and wrong twice. Reordering moves which rule classifies a name, which is a behavior change to the ledger and breaks `_CROSS_RULE_WINNERS`; and no pair is even a reorder CANDIDATE, because in all eleven the narrower rule's own prose would be FALSE or MERELY PARTIAL of the co-matched names if it won, and either disqualifies it as classifier-of-record. Partiality is the commoner half and the arc's load-bearing distinction — compound versus component, which the `fix(#400/#274)`/`fix(#400)` exemption calls "the canonical compound-versus-component shape". Read the eleven `why` texts for it: `fix(#400)` "says nothing about a maiden marker"; a widened `fix(comma-family) lone post-comma piece routes to suffix/title` "would take the union and report only half of it"; `fix(cjk-glued-honorific-peel)` says "nothing about a division"; `feat(#273)`'s prose "has nothing to say about" the remainder rejoining as a name; and two of the three regex-accident pairs are partial rather than false, the `fix(suffix-routing)` jr and numeral rules genuinely naming the `{family, suffix}` half of `김민준씨 Jr.` and of `田中さん II` and describing neither the peel nor the segmentation that moves `given`. Falsity is the minority, and one of the eleven says so of itself: the exemption `fix(comma-family) a comma followed only by titles keeps the given/family split` carries over `fix(comma-precomma-family)` calls its own pair "the one where the narrower rule would be actively WRONG rather than merely partial", and adds that the claim "is about THIS name and is deliberately not generalised". `马丁·路德·金씨` is the clearest compound, and `王先生, V.` the reverse case, where the winner is already the least wrong reading available — and stays so, since the family-side twin that would describe it was weighed as #496 and declined (the next bullet). +- A family-side twin of `fix(cjk-comma-honorific-peel)` for a PRE-comma glued honorific (2026-09-02, [#496](https://github.com/derek73/python-nameparser/issues/496), closed as not planned) — the rule that would honestly describe `王先生, V.`, `田中さん, V.` and `김민준씨, V.`, whose diff is the glued peel alone while the label they carry says comma compound (the finding bullet above). Declined on two grounds. FIRST, no gate can demand it: all 17 names the compound/peel pair is contested across measure radar — but by two different warrants, and an audit that assumes one will find the count short. Sixteen sit in `corpus_cjk_tolerated.jsonl`, the file #488 created by demoting the composed comma/Latin-wrapper CJK forms to tolerated input, `김, 민준씨` — the name #382 was filed over — among them. The seventeenth, `Dr 田中さん, V.`, is in `corpus_issues.jsonl` alone: harvested and append-only, radar since #468, and untouched by #488. (`Dr 김민준씨, Jr.` is in both files, and reads radar from either.) Radar means an unmatched diff is reported rather than fatal for want of a RULE; it is not an unconditional never-fatal, since a `[[never]]` exclusion outranks the tier and stays fatal on a radar name — `_CORPUS_TIERS`' own note, and `main()`'s two-reasons comment, which routes an excluded name to `unexplained` rather than to `radar`. No exclusion refuses these, so nothing here can demand the twin. SECOND, [#495](https://github.com/derek73/python-nameparser/issues/495) points the other way over the same corner: it asks whether the radar-only rules already in the ledger still earn their place and records `fix(cjk-comma-compound)` as having zero contract-tier reach, so #496 proposed a SECOND rule for the very names #495 is weighing a first one away from. Fewer rules for this corner is the coherent direction, and that is what makes the decline decisive rather than merely permissive. REOPEN it if a pre-comma glued-honorific name ever reads contract — and watch BOTH routes, because the two files promote by different mechanisms. A `corpus_cjk_tolerated.jsonl` name is promoted by clearing `tolerated` on its case rows, which moves the text into `corpus_cjk.jsonl` (`_CORPUS_TIERS`, and build_cjk_corpus.py's split). `Dr 田中さん, V.` has no case row at all, so there is no flag to clear: it changes tier only by being CHOSEN — a new unmarked row, or a rules.md example — which puts the text in a contract corpus, and the (name, order) dedup loads contract files first and keeps that reading. Read the tier off `_CORPUS_TIERS` either way, never off a hand-built file→tier map (the roster caution above). +- Precise per-name contest detection at differential-run time (2026-09-02) — it would replace the static predicate's over-reporting with the measured nine-of-eleven split, and it needs the pinned-wheel worker pass to do it. That puts the check behind a multi-minute run, so a rule added by a later bundle would go unchecked at pytest speed — which is the whole point of #382, and is the whole of the reason. Do NOT restate this as an error-direction argument: within the NESTED pairs computing real diffs can only remove them, but the predicate is not a superset of the contests, since nesting is sufficient for one and not necessary (the bullet above, and #498). What the static check buys is cheapness and coverage of a rule nobody has run the wheel against, not a guarantee of refusing everything it should. +- Giving the peel rule a predicate the compound rule fails (2026-09-02, #382 option 1) — narrowing `fix(cjk-comma-honorific-peel)`'s `name_regex` to the honorific-bearing shapes, whose stated effect in #382 was that the pair "becomes order-independent and the original contract holds again". Declined because that effect is UNREACHABLE, and measured rather than argued. The two rules ship a BYTE-IDENTICAL `name_regex` (verified 2026-09-02 by comparing the two strings), and `_rule_reach` computes each rule's names from its OWN pattern, so narrowing the peel rule's regex narrows the peel rule's reach and nothing else: the compound rule goes on reaching every one of the names. The general form, which no regex edit escapes — where the earlier rule's `fields` are a SUBSET of the later one's, order-independence requires the LATER rule to stop reaching the name, and an edit to the earlier rule's own regex cannot cause that. Here the peel rule's `{given, suffix}` is a strict subset of the compound rule's `{family, given, suffix, title}`, so every diff the peel rule admits the compound rule admits too. Implemented to check it: narrowing the peel regex to a `GLUED_HONORIFICS` alternation keeps all seven of the co-matched names whose 1.4.0 diff is exactly `{given, suffix}`, and swapping the two rules in the NARROWED ledger still reattributes all seven. So the pair keeps the arrangement the arc settled on, and that is the answer rather than a cost trade: it is narrow-first, the declaration-free default, so it appears in none of the eleven contests `order_contests` reports and owes no `precedes_narrower` block, and all seven names are pinned by name in `_CROSS_RULE_WINNERS` (23 corpus names reach both regexes; the seven are the ones #375's reorder mutation moves). The cost stands as a second reason and not the first: the narrowing would hand-copy more honorific vocabulary into the pattern and grow the `_HONORIFIC_SOURCES` sync-roster surface (mechanisms.md#CURATED-VOCABULARY-ALTERNATION's second half). Option 2's convention plus a check on the wide-first exceptions to it is what the arc took, and option 3 it REFUSED on a false premise (the finding above). One thing this does not settle, and the ledger comment on the peel rule overstates it: `fields` separates the two only for the UNION rows whose diff includes `family`; for those seven it is file order that decides. +- Scoping the check to contract-tier contests only (2026-09-02) — five of the eleven are contested only over radar names, so this would have cut the file's exemptions by nearly half. Declined because `_CROSS_RULE_WINNERS` already pins radar names — measured, most of the names it pins are radar-tier (22 of 33 today) — so the repo would be inconsistent with itself about whether a radar contest matters. Whether the radar-only CJK comma rules still earn their place after #488's demotion is a real question and is filed as #495; it is a question about those rules, not about the check. +- A shared name-population function for both callers (2026-09-02, the spec's own first sketch) — see the two-populations decision above. `main()` and the unit guard must read different populations, so one function would break `--corpus`; they agree by a docstring that names the function it mirrors instead. + ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01) #291 was filed 2026-07-26 out of the 2.0 vocabulary cleanup, with diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 358f95fe..716cc55b 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -77,8 +77,7 @@ Problem shape. A guard needs to know what the answer WAS, so it can detect the a ## LEDGER-RULE-SEPARATION — file order decides, fields narrow by subset -Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest: the narrower rule must be written first. `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). Since #468 there is a THIRD narrowing key: `orders` admits only the comparison orders it lists, the key being optional and its absence the order-blind reading every earlier rule has — a name compared under two orders can move the same roles for opposite reasons, so a rule describing an order-scoped fold would otherwise absorb that fold leaking into the default order (decisions.md#differential-ledger carries the worked case, and the legal set is borrowed from tools/differential/shapes.py rather than copied — plus one member no shape can declare, the `DEFAULT` sentinel naming the comparison run under no declared order, TOML having no null to put in an array). Exclusions take no `orders` and stay order-blind, deliberately. The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 970 of 1120 comparisons (2026-09-01, re-measured the same day after #486 widened the shapes corpus; it read 963 of 1113 before that). What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (four ledgers load today, measured 2026-09-01; the open cycle's carries no rules, so the identity holds trivially there): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). Whether that pair should be separated by a predicate instead of by order is -[#382](https://github.com/derek73/python-nameparser/issues/382). The old #271/#272 +Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest. Narrow-first is the declaration-free DEFAULT, not the contract: a wider rule can be the better classifier where it describes a compound behavior its component rule does not — `马丁·路德·金씨` divides on the nakaguro AND peels its glued honorific, so `fix(#272/#308)` describes it and `fix(cjk-glued-honorific-peel)` describes half of it — which makes `fields`-subset a proxy for specificity and the wrong one there. What IS the contract is that such a pair must be DECLARED: the earlier rule carries a `precedes_narrower` block naming the later one and saying why, and `undeclared_contests` refuses the ledger otherwise (#382). `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). Since #468 there is a THIRD narrowing key: `orders` admits only the comparison orders it lists, the key being optional and its absence the order-blind reading every earlier rule has — a name compared under two orders can move the same roles for opposite reasons, so a rule describing an order-scoped fold would otherwise absorb that fold leaking into the default order (decisions.md#differential-ledger carries the worked case, and the legal set is borrowed from tools/differential/shapes.py rather than copied — plus one member no shape can declare, the `DEFAULT` sentinel naming the comparison run under no declared order, TOML having no null to put in an array). Exclusions take no `orders` and stay order-blind, deliberately. The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 970 of 1120 comparisons (2026-09-01, re-measured the same day after #486 widened the shapes corpus; it read 963 of 1113 before that). What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (four ledgers load today, measured 2026-09-02; the open cycle's carries one rule, `fix(#462)`, and it carries a `name_regex` like every other, so the identity holds there for the same reason and not for want of rules): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). That pair is narrow-first and so declares nothing, and #382 settled what to do about it by declining BOTH predicates on offer. Narrowing the peel rule's own `name_regex` to the honorific-bearing shapes (#382 option 1) cannot make the pair order-independent at all, which is measured and not argued: the two rules ship an IDENTICAL `name_regex`, each rule's reach is computed from its own pattern, and the peel rule's `fields` nest inside the compound rule's — so the compound rule goes on reaching those names and goes on admitting their diffs whatever the peel rule's regex is narrowed to, and it would grow the `_HONORIFIC_SOURCES` sync roster for nothing. What holds the pair is what the arc kept: narrow-first order, with the seven names whose diff both rules admit pinned by name in `_CROSS_RULE_WINNERS`. A mechanical narrow-first sort (option 3) was refused on a false premise: it reattributes names to a rule describing half of what happens to them. So order separates this pair, by the declaration-free narrow-first default, and it is the WIDE-first pairs that must declare themselves (the rule-order arc under decisions.md#differential-ledger). What that check covers is NESTED pairs: two rules whose `fields` merely INTERSECT are decided by file order too, and sit outside it by the same reasoning that leaves EQUAL `fields` outside — neither rule is narrower, so there is nothing for a `precedes_narrower` block to name — with #498 carrying the worked case and the measured size of the class. The old #271/#272 slug taboo is RETIRED (#333): the canonical-rule selector that keyed on those substrings is deliberately deleted — rule authors are free to use them in compound slugs — and the surviving rosters select on their own explicit keys (_HONORIFIC_SOURCES and _LATIN_ALTERNATION_SOURCES by named issue strings, _SPAN_BEARING_RULES by exact leading fix(...) tag). Lives in. tools/differential/compare.py, the expected_since_*.toml ledgers. Reach for it when. A ledger rule's behavior seems to depend on where it sits in the file — it does, and the reorder mutation is the test (run twice in #375; it fails _CROSS_RULE_WINNERS). History: #372 (closed) measured the then-existing fields-only rule owning 1639 of 5257 name×field pairs as filed (2026-08-10); #375/#376 then cut its classifier-of-record share sharply, and the residual pair ownership was read as the last-resort tier working as designed rather than a defect — until #451 retired the shape outright (decisions.md#differential-ledger). #372's two proposed mechanical checks were DECLINED with measurements (see decisions.md#differential-ledger), not left open. ## CANONICAL-VOCABULARY-AT-THE-BOUNDARY — one vocabulary at the comparison diff --git a/tests/v2/test_differential.py b/tests/v2/test_differential.py index 1ee07d1f..1eabbf89 100644 --- a/tests/v2/test_differential.py +++ b/tests/v2/test_differential.py @@ -1058,6 +1058,54 @@ def test_validate_rules_rejects_a_bad_orders_narrowing( "test_ledger.toml") +def test_order_contests_reads_the_orders_narrowing_classify_reads() -> None: + """`orders` is the third narrowing, so the contest predicate has to + read it too: two rules scoped to DISJOINT orders can never claim + the same comparison, whatever their `fields` and regexes say, so + file order decides nothing between them and there is no hazard to + justify. + + The first pair below is the demonstration. fix(b)'s `fields` are a + strict subset of fix(a)'s and both regexes reach 'John Smith', so + the fields-and-reach half of the predicate holds -- and classify() + still routes each order to its own rule, because neither rule is + reachable under the order the other declares. Reporting it would + demand a written exemption for a contest that cannot occur, which + is the detector-disagrees-with-the-predicate failure + docs/design/AGENTS.md axis 2 is about. + + An order-blind rule keeps contesting everything, which is the + second pair: omitting `orders` means claiming every order, so it + overlaps whatever the other rule declares. That is the direction + that must NOT be quietly narrowed away -- every rule in every + shipped ledger is order-blind today, and a skip that swallowed + those would empty the roster while looking like a fix. + """ + names = ["John Smith"] + disjoint = [ + {"issue": "fix(a) x", "name_regex": "Smith", + "fields": ["given", "family"], "orders": ["DEFAULT"]}, + {"issue": "fix(b) y", "name_regex": "Smith", + "fields": ["family"], "orders": ["FAMILY_FIRST"]}] + assert compare.order_contests(disjoint, names) == [] + assert compare.classify("John Smith", {"family"}, disjoint) == "fix(a) x" + assert compare.classify("John Smith", {"family"}, disjoint, + order="FAMILY_FIRST") == "fix(b) y" + + # ... and the same pair overlapping in one order IS a contest, + # which keeps the skip above from passing for the wrong reason + overlapping = [dict(disjoint[0]), + {**disjoint[1], "orders": ["DEFAULT", "FAMILY_FIRST"]}] + assert [c.earlier for c in compare.order_contests(overlapping, names)] \ + == ["fix(a) x"] + + blind = [dict(disjoint[0]), {k: v for k, v in disjoint[1].items() + if k != "orders"}] + assert [(c.earlier, c.later, c.names) + for c in compare.order_contests(blind, names)] \ + == [("fix(a) x", "fix(b) y", ("John Smith",))] + + def test_validate_rules_takes_the_order_names_from_the_shape_inventory( ) -> None: """The legal set is BORROWED, not hand-copied: every order any @@ -1156,6 +1204,34 @@ def test_v2_fields_matches_the_Role_enum() -> None: ({"issue": "x", "fields": ["given"], "dormnat": "typo"}, "unknown key"), # a dormant declaration is not a pass for the rest of the checks ({"issue": "x", "dormant": "reason"}, "neither 'name_regex' nor 'fields'"), + # #382's shape failures. The key is an array-of-tables, so every + # other shape is a rule that reads as an exemption and declares + # none: [] says nothing, a bare table is the single-bracket + # [change.precedes_narrower] slip, and a list of strings is the + # reason written where the table belongs. + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": []}, "not a non-empty list of tables"), + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": {"issue": "y", "why": "r"}}, + "not a non-empty list of tables"), + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": ["y"]}, "not a non-empty list of tables"), + # an entry naming no rule exempts nothing, and 'entry with' is + # load-bearing in the match: a bare "no string 'issue'" would also + # match the RULE-level message and pin nothing here + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": [{"why": "r"}]}, "entry with no string 'issue'"), + # the reason is the whole safeguard, absent as well as blank + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": [{"issue": "y"}]}, "with no 'why'"), + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": [{"issue": "y", "why": 3}]}, "with no 'why'"), + # a rule cannot outrank itself; reported as itself rather than as + # the backwards-pointing case, which would tell the reader the rule + # sits earlier in the file than itself + ({"issue": "x", "name_regex": "Smith", "fields": ["given"], + "precedes_narrower": [{"issue": "x", "why": "r"}]}, + "precedence over ITSELF"), ]) def test_validate_rules_rejects_a_rule_that_would_silently_widen( rule: dict, expect: str) -> None: @@ -1257,6 +1333,252 @@ def test_initials_mixed_with_another_field_is_rejected( "fields": fields}], "ledger.toml") +def _rule(issue: str, **extra: object) -> dict[str, object]: + """A minimal well-formed ledger rule, for the checks below.""" + return {"issue": issue, "name_regex": "x", "fields": ["given"], **extra} + + +def test_an_exemption_naming_an_unknown_rule_is_rejected() -> None: + with pytest.raises(SystemExit, match="names no rule in this ledger"): + compare.validate_rules( + [_rule("fix(a) first", precedes_narrower=[ + {"issue": "fix(ghost) not in this file", "why": "because"}])], + "test_ledger.toml") + + +def test_an_exemption_pointing_backwards_is_rejected() -> None: + """An exemption names the rule it OUTRANKS, which sits later. + + Pointing at an earlier rule describes a pair where the declaring + rule is already the loser, so it protects nothing -- and the + likeliest way to write one is a copy-paste of the wrong issue + string, which would otherwise sit in the file reading as a + justification. + """ + with pytest.raises(SystemExit, match="sits EARLIER"): + compare.validate_rules( + [_rule("fix(a) first"), + _rule("fix(b) second", precedes_narrower=[ + {"issue": "fix(a) first", "why": "because"}])], + "test_ledger.toml") + + +def test_an_exemption_without_a_reason_is_rejected() -> None: + """`dormant`'s precedent: the reason is the whole safeguard.""" + with pytest.raises(SystemExit, match="'why'"): + compare.validate_rules( + [_rule("fix(a) first", precedes_narrower=[ + {"issue": "fix(b) second", "why": " "}]), + _rule("fix(b) second")], + "test_ledger.toml") + + +def test_a_rule_key_misplaced_into_an_exemption_is_rejected() -> None: + """The trap the TOML shape carries. + + `precedes_narrower` is a nested array-of-tables, so once its block + opens EVERY later bare `key = value` in the rule binds to the + exemption instead of the rule. An author appending `orders` after + an exemption silently deletes that rule's order narrowing -- and + the unknown-key check on the rule cannot see it, because the key + never lands in the rule dict at all. This is the same quiet + widening #451 and #456 close, arriving through new syntax rather + than through a misspelling. + """ + with pytest.raises(SystemExit, match="belongs to the RULE"): + compare.validate_rules( + [_rule("fix(a) first", precedes_narrower=[ + {"issue": "fix(b) second", "why": "because", + "orders": ["FAMILY_FIRST"]}]), + _rule("fix(b) second")], + "test_ledger.toml") + + +def test_the_same_rule_exempted_twice_is_rejected() -> None: + """Two reasons for one pair, and no way to tell which is stale. + + A row in the table above cannot carry this: a repeat only reaches + the check once both entries name a rule that exists, which takes a + second rule in the ledger. + """ + with pytest.raises(SystemExit, match="more than once"): + compare.validate_rules( + [_rule("fix(a) first", precedes_narrower=[ + {"issue": "fix(b) second", "why": "a is the compound rule"}, + {"issue": "fix(b) second", "why": "b was reverted"}]), + _rule("fix(b) second")], + "test_ledger.toml") + + +def test_a_well_formed_exemption_is_accepted() -> None: + """The positive control: the four refusals above must not be + refusing every exemption for some unrelated reason.""" + compare.validate_rules( + [_rule("fix(a) first", precedes_narrower=[ + {"issue": "fix(b) second", "why": "a is the compound rule"}]), + _rule("fix(b) second")], + "test_ledger.toml") + + +#: A wide-first TRIO: one regex, and TWO later rules whose `fields` +#: are each a strict subset of the first's, so file order alone decides +#: which of them classify() hands a matching diff to (#382). +#: +#: Three rather than two on purpose, and the two narrow sets are +#: deliberately disjoint from one another -- neither nests in the other, +#: so they contest nothing between themselves and the fixture holds +#: exactly two contests, both owned by `fix(wide)`. That is what makes +#: it possible to declare ONE of them and see the other still reported. +#: On a two-rule fixture, "this rule declares that rule" and "this rule +#: declares something" are the same sentence, and every assertion below +#: would pass against a reader that treated any declaration as a +#: blanket opt-out over every narrower rule -- which is precisely the +#: widening validate_rules' blank-'issue' refusal exists to refuse. The +#: shipped 1.4 ledger already has the shape: two of its rules are the +#: earlier side of two contests each. +_CONTESTED: list[dict[str, object]] = [ + {"issue": "fix(wide) the compound behavior", + "name_regex": "Smith", "fields": ["given", "family", "suffix"]}, + {"issue": "fix(narrow) one half of it", + "name_regex": "Smith", "fields": ["given", "family"]}, + {"issue": "fix(narrow-b) the other half of it", + "name_regex": "Smith", "fields": ["given", "suffix"]}, +] + + +def test_a_wide_first_pair_is_reported_until_it_is_declared() -> None: + """What `precedes_narrower` buys, and only what it buys. + + The declaration is read off the EARLIER rule and names the later + one, so a pair stays on the roster until the rule that wins it + says in writing that it means to. Reading it off the wrong rule + would exempt pairs nobody declared. + + A declaration retires the ONE pair it names and no other, which is + the middle assertion and the reason the fixture carries two + narrower rules: with `fix(narrow)` declared and `fix(narrow-b)` not, + exactly the second pair must still be reported. Read the rule's + declarations as a blanket opt-out -- `if not _declared_over(...)` + in place of the `c.later not in ...` membership test -- and the + reported list goes empty here while every other test in this file + keeps passing. + + The malformed shapes at the end pin CRASH-SAFETY, and that is all + they pin. `_declared_over` reads whatever validate_rules already + accepted plus whatever a test hands it, and a bare string, an entry + with no `issue` at all, and a list of strings must each leave it + returning a set rather than raising. + + They deliberately do not pin the LENIENCE. A stricter reader -- + one demanding a non-blank `why`, say -- passes all three of these, + and could not hide a contest if it wanted to, since declaring less + can only report more. Leniency here is a convenience for callers, + not the safe direction, so there is nothing about it worth pinning. + """ + names = ["Smith, Jr."] + both = [("fix(wide) the compound behavior", "fix(narrow) one half of it"), + ("fix(wide) the compound behavior", + "fix(narrow-b) the other half of it")] + assert [(c.earlier, c.later) for c + in compare.undeclared_contests(_CONTESTED, names)] == both + + # ONE of the two declared: the other must survive. + half = [dict(_CONTESTED[0], precedes_narrower=[ + {"issue": "fix(narrow) one half of it", "why": "wide describes both"}]), + _CONTESTED[1], _CONTESTED[2]] + assert [(c.earlier, c.later) for c + in compare.undeclared_contests(half, names)] == [both[1]] + + declared = [dict(_CONTESTED[0], precedes_narrower=[ + {"issue": "fix(narrow) one half of it", "why": "wide describes both"}, + {"issue": "fix(narrow-b) the other half of it", + "why": "and the other half"}]), + _CONTESTED[1], _CONTESTED[2]] + assert compare.undeclared_contests(declared, names) == [] + + for malformed in ("fix(narrow) one half of it", + [{"why": "a reason, and no rule it is a reason for"}], + ["fix(narrow) one half of it"]): + broken = [dict(_CONTESTED[0], precedes_narrower=malformed), + _CONTESTED[1], _CONTESTED[2]] + assert len(compare.undeclared_contests(broken, names)) == 2 + + +def test_a_pair_whose_regexes_share_no_name_is_no_contest() -> None: + """The shared-name test carries the whole check. + + Drop the shared-name test and the same scan reports 657 wide-first + pairs across the shipped ledgers, against the 11 the full predicate + finds. That gap is the argument and it does not rest on the digits: + an exemption roster in the hundreds, where the real one is eleven, + is a roster nobody writes and nobody reads -- so `fields`-subset is + not a usable predicate on its own. + + To recompute, reimplement order_contests' loop over `_rules(ledger)` + and `_CORPUS_NAMES` for every ledger in `_LEDGERS`, dropping one + condition at a time -- the function itself has no knobs to turn + them off, and a script importing `tests.v2._differential_fixtures` + needs `PYTHONPATH=.`. Mind the basis: 657 is the count with the + `orders` test still in place, and `fields`-subset ALONE -- both + conditions gone, nesting counted in either direction -- reports + 1350, of which the `orders` test removes 2 and none of the 657. + Measured 2026-09-02.""" + apart = [dict(_CONTESTED[0], name_regex="Smith"), + dict(_CONTESTED[1], name_regex="Jones")] + assert compare.order_contests(apart, ["Smith, Jr.", "Jones, Jr."]) == [] + + # The control, inline rather than a pointer at a neighbouring test + # that a rename would silently break: the same fixture with every + # regex reaching one name IS contested, so the empty list above is + # the shared-name test doing work and not the scan having gone + # quiet. Two, because _CONTESTED's wide rule strictly contains both + # of the narrower ones. + assert len(compare.order_contests(_CONTESTED, ["Smith, Jr."])) == 2 + + +def test_an_exemption_for_a_pair_that_is_no_contest_is_vacant() -> None: + """The `dormant`-awake precedent: a narrowing that ends a contest + must not leave a permission nobody re-earned. + + The last block is the one that pins WHICH declaration went vacant. + One rule carrying two declarations, one live and one stale, is the + only arrangement that can tell the real reader from a rule-level + one -- ask `does this rule contest anything` instead of `is THIS + pair contested` and the stale declaration disappears from the + report, silently, on a ledger where two of the shipped rules + already carry two declarations each. + """ + apart = [dict(_CONTESTED[0], name_regex="Smith", precedes_narrower=[ + {"issue": "fix(narrow) one half of it", "why": "stale"}]), + dict(_CONTESTED[1], name_regex="Jones")] + vacancies = compare.vacant_exemptions(apart, ["Smith, Jr.", "Jones, Jr."]) + assert vacancies == [ + ("fix(wide) the compound behavior", "fix(narrow) one half of it")] + # NamedTuple equality is by value, so the assertion above passes + # against a bare 2-tuple and would keep passing if _Vacancy were + # deleted. The field names are what the caller's message reads. + assert vacancies[0].earlier == "fix(wide) the compound behavior" + assert vacancies[0].later == "fix(narrow) one half of it" + + # ... and the live pair, which is what makes the assertion above a + # measurement: a function that simply listed every declaration + # would read identically on the vacant pair alone. + live = [dict(apart[0]), dict(apart[1], name_regex="Smith")] + assert compare.vacant_exemptions(live, ["Smith, Jr.", "Jones, Jr."]) == [] + + # One rule, two declarations, one of each: only the stale one is + # reported. `fix(narrow)` still contests over 'Smith, Jr.', so the + # rule contests SOMETHING -- and `fix(narrow-b)`, pulled away onto + # 'Jones', no longer does. + mixed = [dict(_CONTESTED[0], precedes_narrower=[ + {"issue": "fix(narrow) one half of it", "why": "live"}, + {"issue": "fix(narrow-b) the other half of it", "why": "stale"}]), + _CONTESTED[1], dict(_CONTESTED[2], name_regex="Jones")] + assert compare.vacant_exemptions(mixed, ["Smith, Jr.", "Jones, Jr."]) == [ + ("fix(wide) the compound behavior", + "fix(narrow-b) the other half of it")] + + #: What _run_worker was asked for, so a test can prove main forwarded #: the baseline and the corpus rather than defaults of its own. _WORKER_CALL: dict = {} @@ -1268,7 +1590,9 @@ def _run_main(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ledger_body: str, baseline: str = "1.4.0", baseline_v2: dict | None = None, floor: int | None = 1, - tier: str | None = "contract") -> tuple[int, str]: + tier: str | None = "contract", + corpus_flag: bool = True, + names_every_corpus: bool = False) -> tuple[int, str]: """Drive main() end to end with a faked baseline worker. No uv, no network. The helper exists because every unit test above @@ -1283,6 +1607,24 @@ def _run_main(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ledger_body: str, in order, alongside the fixture's own 'John Smith'. It exists so a test can mix a diffing and a non-diffing name -- the single-name corpus below is structurally incapable of that. + + `corpus_flag=False` drops `--corpus` from argv, so main() globs its + corpora the way a FULL gate run does. It is the only way to reach + main()'s `if not args.corpus` branches from a test, and there are + two of them -- the missing-floor roster and, since #382, the + vacant-exemption refusal, which a partial run must only NOTE. The + fixture corpus is the only file in the patched HERE, so the floor + roster is replaced wholesale rather than added to: left intact it + would name every real corpus as missing. + + `names_every_corpus=True` keeps `--corpus` on argv but replaces the + floor roster wholesale anyway, so the run NAMES every corpus the + roster knows about. That is the case `--corpus` cannot be read as + "a narrowing": the flag is `action="append"`, so a run listing all + six corpora is the full gate wearing a flag, and every check that + softens under narrowing must stay hard here. Meaningless without + `corpus_flag`, which is why the two are separate parameters rather + than one tri-state. """ import json import sys @@ -1347,7 +1689,11 @@ def _fake(v: str, w: bool, n: list[dict]) -> tuple[dict, list[dict]]: # leaves it unregistered, for the test that pins what happens when # a corpus arrives without one. if floor is not None: - monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, floor) + if corpus_flag and not names_every_corpus: + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, floor) + else: + monkeypatch.setattr( + compare, "_CORPUS_FLOORS", {corpus.name: floor}) # The fixture corpus needs a tier like any other. `tier=None` # leaves it unregistered, for the test that pins the fail-closed # roster. @@ -1355,8 +1701,9 @@ def _fake(v: str, w: bool, n: list[dict]) -> tuple[dict, list[dict]]: monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, tier) monkeypatch.setattr(compare, "HERE", tmp_path) monkeypatch.setattr(compare, "_run_worker", _fake) - monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", baseline, - "--corpus", str(corpus)]) + monkeypatch.setattr(sys, "argv", + ["compare.py", "--baseline", baseline] + + (["--corpus", str(corpus)] if corpus_flag else [])) import io import contextlib buf = io.StringIO() @@ -1407,6 +1754,273 @@ def test_main_exits_0_when_every_diff_is_claimed( assert "## claimed (1)" in out +def test_main_checks_contests_against_the_names_it_loaded() -> None: + """The run must refuse the ledger BEFORE the worker pass. + + The unit guard is what catches a new rule at pytest speed; this is + the belt for a run over a --corpus the guard never sees, and it has + to fire early -- after the multi-minute worker pass, the reader has + already paid for the answer. + """ + src = (compare.HERE / "compare.py").read_text(encoding="utf-8") + body = src[src.index("def main("):] + assert "undeclared_contests(" in body and "vacant_exemptions(" in body, ( + "main() does not consult the contest checks") + assert body.index("undeclared_contests(") < body.index("_run_worker("), ( + "main() must refuse an undeclared contest before spawning the " + "worker, not after") + + +#: A wide-first pair over the fixture corpus's own 'John Smith', for +#: the two main() refusals below (#382). Written as ledger text rather +#: than reusing _CONTESTED, because _run_main takes a TOML body. +_CONTESTED_LEDGER = ( + '[[change]]\nissue = "fix(wide) the compound behavior"\n' + 'name_regex = "Smith"\nfields = ["given", "family"]\n' + '\n' + '[[change]]\nissue = "fix(narrow) one half of it"\n' + 'name_regex = "Smith"\nfields = ["family"]\n') + + +def test_main_refuses_an_undeclared_contest_without_running_the_worker( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The source-order assertion above says the call SITES are in the + right order; this says the refusal actually fires, and fires early. + + `_WORKER_CALL` is the instrument for "early": _run_main clears it + and then monkeypatches _run_worker to record into it, so an empty + dict after the SystemExit means the worker was never asked -- which + a source-text index cannot establish, since a call site can sit + ahead of the worker and still be guarded into never running. + """ + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _CONTESTED_LEDGER, _DIFFERS) + message = str(exc.value) + assert "fix(wide) the compound behavior" in message + assert "fix(narrow) one half of it" in message + # the message must send the reader to the declaration, not to the + # reorder that would move which rule classifies a name + assert "precedes_narrower" in message and "do NOT reorder" in message + assert not _WORKER_CALL, ( + "main() spawned the worker before refusing the ledger") + + +def test_a_full_run_refuses_an_undeclared_contest_too( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The same refusal on the run that matters: `corpus_flag=False`. + + `_run_main` defaults to `--corpus`, so every other undeclared-contest + test above reaches main() through the narrowed path -- and `vacant` + right below it genuinely behaves differently there. Making the + undeclared refusal conditional on `args.corpus` the same way, so + that a FULL gate run never refuses a contest nobody declared, + survives the whole suite without this test. It is the mode the CI + gate actually runs in, and it is the one that was untested. + """ + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _CONTESTED_LEDGER, _DIFFERS, + corpus_flag=False) + message = str(exc.value) + assert "fix(wide) the compound behavior" in message + assert "fix(narrow) one half of it" in message + assert not _WORKER_CALL, ( + "main() spawned the worker before refusing the ledger") + + +def test_the_contest_check_reads_names_this_baseline_will_not_compare( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """LOADED, not KEPT: the choice main()'s comment defends, measured. + + main() runs the contest checks over every entry it read, ahead of + the baseline-minimum shape skip -- so an order-bearing name that a + 1.4.0 run will not actually compare still counts toward whether a + ledger is acceptable. The point is that the ledger gets the same + verdict at every baseline; move the block after `kept`, or filter + shape-tagged entries out of `corpus_names`, and it stops. + + Neither mutation is visible to + test_main_checks_contests_against_the_names_it_loaded, which reads + source-text indices and both mutations preserve. Here the ONLY name + the contested pair reaches is a shape-4 (FAMILY_FIRST, minimum + 2.0.0) entry the 1.4.0 run drops, so either mutation leaves the + undeclared contest unreported and the run proceeds to the worker. + """ + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 4}, ensure_ascii=False) + "\n" + + _json.dumps("John Smith") + "\n", encoding="utf-8") + (tmp_path / "expected_since_1.4.0.toml").write_text( + '[[change]]\nissue = "fix(wide) the compound behavior"\n' + 'name_regex = "Ménil"\nfields = ["given", "family", "suffix"]\n' + '\n' + '[[change]]\nissue = "fix(narrow) one half of it"\n' + 'name_regex = "Ménil"\nfields = ["given", "family"]\n', + encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + _WORKER_CALL.clear() + + def _fake(v: str, w: bool, n: list[dict]) -> tuple[dict, list[dict]]: + _WORKER_CALL.update(version=v, want_v2=w) + raise AssertionError("the worker must not run") + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "1.4.0", + "--corpus", str(corpus)]) + with pytest.raises(SystemExit) as exc: + compare.main() + message = str(exc.value) + assert "fix(wide) the compound behavior" in message + assert "fix(narrow) one half of it" in message + assert not _WORKER_CALL + + # The control: the same run at a baseline that DOES compare the + # name refuses identically. Without it the assertion above could + # read as "the check fires", where what it pins is "the check fires + # at a baseline whose comparison never sees this name". + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + (tmp_path / "expected_since_2.0.0.toml").write_text( + (tmp_path / "expected_since_1.4.0.toml").read_text(encoding="utf-8"), + encoding="utf-8") + with pytest.raises(SystemExit) as exc: + compare.main() + assert "fix(wide) the compound behavior" in str(exc.value) + + +def test_the_refusal_asks_for_the_reason_the_ledger_guard_asks_for( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A contributor who follows this message verbatim must not hit a + second refusal. `precedes_narrower` needs an `issue` AND a + non-blank `why` -- validate_rules refuses the entry without one -- + so a message that says only "name the later rule" sends the reader + into a ledger that will not load. The guard message in + tests/v2/test_ledger_guards.py already asks for both halves; the + two are read by the same contributor and must say the same thing. + """ + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _CONTESTED_LEDGER, _DIFFERS) + message = str(exc.value) + assert "what it describes that the later one does not" in message + + +#: The same pair with the exemption declared and the two regexes pulled +#: apart, so the declaration has nothing left to permit (#382). +_VACANT_LEDGER = _CONTESTED_LEDGER.replace( + 'fields = ["given", "family"]\n', + 'fields = ["given", "family"]\n' + '[[change.precedes_narrower]]\n' + 'issue = "fix(narrow) one half of it"\nwhy = "stale"\n' +).replace('name_regex = "Smith"\nfields = ["family"]', + 'name_regex = "Jones"\nfields = ["family"]') + + +def test_main_refuses_a_vacant_exemption_without_running_the_worker( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The other half, on a FULL run -- `corpus_flag=False`, because + that is the only run whose name set can tell a stale declaration + from one this run simply did not reach.""" + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _VACANT_LEDGER, _DIFFERS, + corpus_flag=False) + message = str(exc.value) + assert "fix(narrow) one half of it" in message + assert "Delete the exemption" in message + assert not _WORKER_CALL, ( + "main() spawned the worker before refusing the ledger") + + +def test_main_only_notes_a_vacant_exemption_under_corpus( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The same ledger under `--corpus` must NOTE, never refuse. + + Narrowing the corpus removes contests, and the two checks read that + in opposite directions: fewer contests is fewer things to declare + (fail-closed for `undeclared`), but a live declaration whose names + are outside this run reads as vacant. A refusal here tells a + contributor to delete an exemption the full gate still needs. + """ + code, out = _run_main(tmp_path, monkeypatch, _VACANT_LEDGER, _DIFFERS) + assert "--corpus" in out and "not evidence" in out + assert "Delete the exemption" not in out + # A single value, not `in (0, 1)`: main() has exactly one return + # for the run outcome, so a two-member set is an assertion that + # cannot fail. 1 for reasons orthogonal to the NOTE -- _VACANT_LEDGER + # pulls 'fix(narrow)' onto a regex the fixture corpus has no name + # for (EXPLAINED NOTHING) and leaves 'fix(wide)' declaring a role no + # diff moves (OVER-DECLARED). What this pins is that the vacancy + # RETURNED a verdict rather than raising, and 0 would mean the + # ledger's own defects had gone quiet. + assert code == 1, out + + +def test_naming_every_corpus_refuses_a_vacant_exemption( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """`--corpus` is not the question; the NAME SET is. + + The flag is `action="append"`, so a run that lists every corpus + explicitly compares exactly what the flagless gate compares -- and + a check keyed on `args.corpus` softens for it anyway. Written as + the inversion of the NOTE test right above: same ledger, same + flag, and the only difference is that here the named corpus is the + whole roster. Keying the downgrade on the flag rather than on the + name set leaves a genuinely stale exemption printing a NOTE and + exiting 0. + """ + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _VACANT_LEDGER, _DIFFERS, + names_every_corpus=True) + message = str(exc.value) + assert "Delete the exemption" in message + assert "fix(narrow) one half of it" in message + assert not _WORKER_CALL, ( + "main() spawned the worker before refusing the ledger") + + +def test_the_vacancy_refusal_names_both_of_its_causes( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A vacancy has two reachable causes and the message must not + offer repair advice for only one. The rule may have been narrowed + -- delete the exemption -- or a corpus NAME may have left while the + corpus stayed above its floor, in which case the exemption was + right and the corpus is what regressed. The ledger guard's message + in tests/v2/test_ledger_guards.py already says both; a contributor + reads whichever fires first. + """ + with pytest.raises(SystemExit) as exc: + _run_main(tmp_path, monkeypatch, _VACANT_LEDGER, _DIFFERS, + corpus_flag=False) + assert "or a corpus name left" in str(exc.value) + + +def test_a_corpus_narrowing_does_not_refuse_the_shipped_1_4_ledger( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The regression itself, on the file it was measured against. + + Every one of the six real corpora, run alone against + expected_since_1.4.0.toml, reported vacancies -- 11 of the 11 + exemptions for three of them -- so `--corpus` refused every + narrowing the README documents. A fixture ledger cannot show that: + it would keep passing if the shipped exemptions were deleted, which + is exactly the repair the refusal wrongly asked for. + """ + ledger = (_TOOLS / "expected_since_1.4.0.toml").read_text( + encoding="utf-8") + code, out = _run_main(tmp_path, monkeypatch, ledger, _DIFFERS) + assert "Delete the exemption" not in out + assert "not evidence" in out + # See the neighbouring NOTE test on why this is one value and not + # two. 1 here because the shipped ledger is written for six real + # corpora and this run compares one fixture name, so nearly every + # rule in it reports EXPLAINED NOTHING -- orthogonal to the NOTE, + # and 0 would mean the run had stopped reporting that. + assert code == 1 + + def test_radar_diff_with_no_rule_exits_0_and_is_reported( tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: """The tier split's entire point (#468): a harvested name's diff diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 3f611bfb..e744efea 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -819,8 +819,9 @@ def test_case_shape_ids_exist_in_the_inventory() -> None: # 'Carod i' and '田中さん II' are deliberately NOT probes for the # numeral rule: its regex really does reach both, and rules above it # win them -- 'Carod i' on file order, '田中さん II' on the subset - # test, its diff moving {given, suffix} where the numeral rule - # declares {family, suffix}. _CROSS_RULE_WINNERS pins both instead; + # test, its diff moving {family, given, suffix} where the numeral + # rule declares {family, suffix} and so cannot admit the `given` + # move. _CROSS_RULE_WINNERS pins both instead; # this roster tests the regex, not classify(). "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`": ("Mohamad X Surname", "Smith Jr.", "Donald mc", "Aishwarya Rai"), @@ -2499,7 +2500,9 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: # first is decided by order: 'Carod i' diffs {family, suffix}, # which both rules declare, so nothing but _sorted_rules' # stability inside the name_regex tier keeps it with fix(#397). - # '田中さん II' diffs {given, suffix}, which the numeral rule's + # '田中さん II' diffs {family, given, suffix} -- 1.4 read it + # 'first 田中さん / last II', 2.x reads 'last 田中 / suffix + # さん, II' -- and `given` is the field the numeral rule's # `fields` cannot admit at any position. Both are recorded # because a later edit that moves either rule, or widens the # numeral rule's `fields`, would take one silently -- exactly the @@ -2507,7 +2510,7 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: ("Carod i", ("family", "suffix")): "fix(#397) NOT WANTED: a trailing Catalan/Polish linking " "'i' is read as a generation marker and the family is lost", - ("田中さん II", ("given", "suffix")): + ("田中さん II", ("family", "given", "suffix")): "fix(cjk-glued-honorific-peel) glued honorific peels into " "suffix", # #484's three `_initials` contests with a literal rule on one @@ -3081,3 +3084,153 @@ def test_a_rule_reaching_no_corpus_name_says_why_it_is_kept() -> None: "no rule was examined, so this guard is passing vacuously -- " "every rule declares `dormant`, or (impossible since #451) " "narrows by `fields` alone") + + +#: Every order-decided contest in every shipped ledger, measured with +#: exemptions IGNORED: the earlier rule's issue, the later rule's, and +#: how many corpus names their regexes both reach. +#: +#: The recorded negative control for the guard below (the +#: _EXCLUSION_EFFECT shape AGENTS.md asks of every guard): it is the +#: answer with the mechanism switched off, stored as data. Without it, +#: `precedes_narrower` could be deleted from every rule and the live +#: guard would keep passing if the predicate had quietly stopped +#: finding anything. +#: +#: 11 pairs, all in the 1.4 ledger. They divide by the tier of the +#: names they are contested over -- some reach contract-tier names, +#: the rest only radar -- and #495 argues from that division, which +#: survives a name changing tier even though its two counts there do +#: not. Read a name's tier off `_CORPUS_TIERS` and not off any one +#: demotion: the five radar-only pairs get there by two different +#: warrants, #488's `corpus_cjk_tolerated.jsonl` demotion for most of +#: the CJK names and #468's tier split for every `corpus_issues.jsonl` +#: one -- which is both 'Jr., PhD'/'MD, PHD' pairs whole, and one name +#: of the seventeen the compound/peel pair is contested over. Measured +#: 2026-09-02. A row that MOVES is a finding, not a number to update: +#: re-measure before editing it. +_ORDER_EXEMPTION_EFFECT: dict[str, list[tuple[str, str, int]]] = { + "expected_since_1.4.0.toml": [ + ("fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example", + "fix(comma-precomma-family) pre-comma run reads as family, not given", 2), + ("fix(#296) a credential-only comma string reads a name and its postnominal", + "fix(comma-family) lone post-comma piece routes to suffix/title, not first", 2), + ("fix(#296) a credential-only comma string reads a name and its postnominal", + "fix(comma-precomma-family) pre-comma run reads as family, not given", 2), + ("fix(#296) a lone post-comma credential is a suffix", + "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`", 2), + ("fix(#400/#274) bound-given join and maiden consumption in one name", + "fix(#400) abd joins the word after it as one given name", 1), + ("fix(#411/S2) a declining bound-given join leaves the suffix reading after a family comma", + "fix(#400) abd joins the word after it as one given name", 1), + ("fix(#272/#308) nakaguro division and a glued hangul honorific in one name", + "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 1), + ("fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set", + "feat(#273) typographic nickname delimiters recognized by default", 1), + ("fix(cjk-comma-compound) comma routing compounds with the CJK order flip", + "fix(cjk-glued-honorific-peel) glued honorific peels into suffix", 17), + ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", + "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`", 1), + ("fix(cjk-glued-honorific-peel) glued honorific peels into suffix", + "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`", 1), + ], + "expected_since_2.0.0.toml": [], + "expected_since_2.1.0.toml": [], + "expected_since_2.2.0.toml": [], +} + + +def test_the_recorded_order_contests_are_what_the_ledgers_hold() -> None: + """The negative control: every contest, exemptions ignored. + + A contest is two same-tier rules that overlap on all three of the + keys classify() narrows by: the LATER one's `fields` are a strict + subset of the earlier one's, both regexes reach a common corpus + name, and neither scopes itself to `orders` the other excludes. + There are then diffs both rules admit, and file order alone picks + the winner. + + This roster is deliberately blind to `precedes_narrower`: it + records the hazard, not whether it has been declared away. + """ + compare = load_tool("compare") + assert any(_ORDER_EXEMPTION_EFFECT.values()), ( + "every ledger's contest list is empty, so this control measures " + "nothing: a predicate that had stopped finding anything at all " + "would read exactly the same. That is the inert-measurement " + "class mechanisms.md#RECORDED-ROSTERS is written against. If " + "the last contest genuinely went away, delete this guard and " + "its roster together -- do not leave four empty lists standing " + "in for a measurement.") + assert set(_ORDER_EXEMPTION_EFFECT) == {led.name for led in _LEDGERS}, ( + f"_ORDER_EXEMPTION_EFFECT must name every ledger on disk, with " + f"an explicit empty list for one that genuinely has no contest. " + f"Missing: {sorted({L.name for L in _LEDGERS} - set(_ORDER_EXEMPTION_EFFECT))}; " + f"unknown: {sorted(set(_ORDER_EXEMPTION_EFFECT) - {L.name for L in _LEDGERS})}") + for ledger in _LEDGERS: + rules = _rules(ledger) + # order_contests' shape leniency (see _rule_reach) is a + # BORROWED guarantee: it trusts validate_rules to have run, and + # `_rules()` does not run it. Calling it here turns the borrowed + # guarantee into a local one, so this guard is not measuring a + # ledger nothing has validated. It costs nothing on the shipped + # files -- they already validate -- and fires first on a + # hand-edit that would otherwise be scanned as if well-formed. + compare.validate_rules(rules, ledger.name) + found = [(c.earlier, c.later, len(c.names)) + for c in compare.order_contests(rules, _CORPUS_NAMES)] + assert found == _ORDER_EXEMPTION_EFFECT[ledger.name], ( + f"{ledger.name}: the order-decided contests are no longer " + f"what this roster records.\n found: {found}\n" + f" recorded: {_ORDER_EXEMPTION_EFFECT[ledger.name]}\n" + f"A contest that appeared is a new rule pair whose winner " + f"file order is deciding; one that vanished means a rule " + f"was narrowed or a corpus name left. Re-measure and read " + f"the pair before editing this roster.") + + +def test_every_order_decided_contest_is_declared() -> None: + """Narrow-first is the declaration-free default; wide-first says why. + + Where the later rule's `fields` are a strict subset of the earlier + one's and both regexes reach a common corpus name, file order alone + decides who classifies the diff. That is legal -- a wider rule can + genuinely be the better description, and `马丁·路德·金씨` is the + worked case: it divides on the nakaguro AND peels its honorific, so + `fix(#272/#308)` describes it and `fix(cjk-glued-honorific-peel)` + describes half of it. What is not legal is leaving it unsaid, + because nothing else in the suite can see it: _CORPUS_CLAIMS + measures each rule alone, the gate total is per-corpus, and + _CROSS_RULE_WINNERS pins only names somebody hand-added. + + Do NOT satisfy this by reordering rules -- that moves which rule + classifies a name and breaks _CROSS_RULE_WINNERS. Declare it. + """ + compare = load_tool("compare") + for ledger in _LEDGERS: + rules = _rules(ledger) + # Both scanners below read `precedes_narrower` through + # _declared_over, whose docstring says outright that its shape + # guarantee is BORROWED from validate_rules -- and `_rules()` + # does not validate. Without this call a whitespace-only `why` + # on a real exemption passes here, because _declared_over reads + # the entry as a perfectly good declaration and retires the + # pair. One line converts the borrowed guarantee into a local + # one; the shipped ledgers already validate, so it costs + # nothing. + compare.validate_rules(rules, ledger.name) + undeclared = compare.undeclared_contests(rules, _CORPUS_NAMES) + assert not undeclared, "\n".join( + [f"{ledger.name}: {len(undeclared)} order-decided contest(s) " + f"nobody declared. The EARLIER rule must carry a " + f"[[change.precedes_narrower]] block naming the later one " + f"and saying what it describes that the later one does not:"] + + [f" {c.earlier!r}\n outranks {c.later!r}\n" + f" on {len(c.names)} name(s), e.g. {list(c.names[:3])}" + for c in undeclared]) + vacant = compare.vacant_exemptions(rules, _CORPUS_NAMES) + assert not vacant, ( + f"{ledger.name}: {vacant} declare precedence over a pair " + f"that is no longer contested. A rule was narrowed or a " + f"corpus name left; delete the exemption rather than " + f"leaving a justification for a hazard that is gone") diff --git a/tools/differential/README.md b/tools/differential/README.md index ef56168b..b580a91d 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -551,7 +551,9 @@ argued for in #373. ledger carries a `name_regex`, so they all sit in one tier, the sort is stable, and the order they are written in settles every tie between them. Append a rule to the bottom of a file only after checking that -nothing above it already claims the diff you meant it for. +nothing above it already claims the diff you meant it for. Where the +EARLIER rule of such a tie is the wider one, the pair must say so -- +see "Declaring a wide-first pair" below. `_sorted_rules` still sorts `name_regex` rules ahead of `fields`-only ones, and is now the identity on every ledger that loads. It is kept @@ -592,6 +594,147 @@ never asks whether the reason is still true. Only the dynamic check can catch a `dormant` that quietly stops being true, which is why every ledger with rules needs one. +### Declaring a wide-first pair (`precedes_narrower`) + +File order deciding is safe while the earlier rule is the NARROWER +one, and narrow-first is a default rather than a law (#382). A wider +rule can be the better classifier where it describes a compound +behavior its component rule does not: `马丁·路德·金씨` divides on the +nakaguro AND peels its glued hangul honorific, so `fix(#272/#308)` +describes what happens to the name and +`fix(cjk-glued-honorific-peel)` describes half of it -- and the wider +rule wins, correctly, by sitting first. `fields`-subset is a proxy +for specificity and the wrong one there, which is why the ordering +cannot simply be enforced. + +Such a pair is legal, and must be DECLARED on the rule that WINS it +-- the earlier one, which is the only rule whose word can retire the +contest: + +```toml +[[change]] +issue = "fix(#272/#308) nakaguro division and a glued hangul honorific in one name" +name_regex = "..." +fields = ["family", "given", "middle", "suffix"] +[[change.precedes_narrower]] +issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix" +why = """ +`middle` is the discriminator and the nakaguro is where it comes +from. ... +""" +``` + +DOUBLE brackets: a rule may outrank more than one neighbour, so the +key is an array of tables. Single-bracket +`[change.precedes_narrower]` makes ONE table rather than a list of +them and is refused, as is an empty list -- deleting the key says the +same thing in one place. + +Each block names ONE later rule, by its exact `issue` string, and +gives a `why`. Both are required, and each ban has its reason. The +named rule is the one and only rule this one outranks: a blanket +"may outrank anything narrower" would be inherited by every narrower +rule added afterwards, which is the widening this check exists to +refuse. The `why` is the whole safeguard, as it is for `dormant` -- +`fields` cannot say that a wider rule describes a compound behavior +its component does not, so the reason is the only place that fact can +live, and an exemption nobody had to justify is the one nobody +reviews. `validate_rules` also refuses a target naming no rule in +this ledger, a rule naming ITSELF, a target sitting EARLIER in the +file (the narrower rule of a declared pair is by definition the later +one, so an earlier one is a copy-paste of the wrong issue string), +and the same target twice (one pair takes one exemption, so a repeat +exempts nothing new and means one of the two reasons is stale, with +nothing to tell a reader which). + +**The trap: nothing may follow the block inside a rule.** TOML binds +every later bare `key = value` to the table the last header opened, +so a rule key written BELOW `[[change.precedes_narrower]]` leaves the +rule and joins the exemption. Put the block LAST in the rule. +`validate_rules` rejects any key inside an exemption other than +`issue` and `why` for exactly this reason: an `orders` landing there +deletes the rule's order narrowing, and nothing else would notice. + +**What counts as a contest.** `order_contests` asks the three +questions `_entry_matches` asks, one per narrowing key, and a pair is +a contest only where all three overlap. `fields`: the later rule's +are a STRICT subset of the earlier one's, so every diff fitting the +narrower set passes both rules' subset test. `name_regex`: some +corpus name reaches both. `orders`: some order reaches both -- two +rules scoped to disjoint orders never see the same comparison, so +file order decides nothing between them however nested their `fields` +are, and calling that a contest would demand a justification for a +hazard that cannot occur. No diff is computed, which is what makes +the check cheap enough to run before the worker spawns -- the nesting +supplies the contested shape's EXISTENCE, and computing real diffs +could only ever remove pairs from THIS list, never add one to it. + +**Nesting is SUFFICIENT for a contest and not necessary**, so read +that last sentence as a statement about the nested pairs and not +about contests in general. Two rules whose `fields` merely INTERSECT +both admit any diff inside that intersection, so `classify()` hands +such a name to whichever of them is written first, exactly as it does +for a nested pair -- and this check cannot see it. That class is +outside the check by REASONING and not by oversight, and the +reasoning is the one `order_contests`' docstring gives for EQUAL +`fields`: neither rule is narrower, so "narrow-first" says nothing +about the pair, `precedes_narrower` has no narrower rule to name, and +`_CROSS_RULE_WINNERS` stays the instrument there. That argument +covers every pair where neither `fields` set contains the other, and +equal `fields` is its special case. Measured 2026-09-02 over the four +ledgers -- pairs sharing a corpus name whose `orders` are not disjoint +-- 11 are strictly nested wide-first, which is what this check +refuses; 40 nest in either direction, 11 have equal `fields`, and 111 +have any non-empty intersection, so 60 overlap without nesting or +equality. Read the gap and not the digits, and recompute before +quoting any of them: decisions.md#differential-ledger carries the same +five figures with the recipe, and #498's body the loop. +Widening the predicate to that general case would demand 111 written +justifications where the real number is eleven -- the same argument +decisions.md already makes about the 657 figure, that a predicate +nobody can answer is not a usable one. The worked blind spot is real +and filed as +[#498](https://github.com/derek73/python-nameparser/issues/498): +`fix(#271/#272/#298)` and `fix(cjk-delimited-nickname)` intersect in +{`family`, `given`} without nesting, and swapping them reattributes +three contract-tier names this check never mentions in either +arrangement. + +Two questions, in both tiers, as for `dormant`. Is every contest +DECLARED, and does every declaration still stand over a contest? The +second matters as much: a rule narrowed until it no longer overlaps +its neighbour leaves its exemption behind, and a justification for a +hazard that is gone reads exactly like one for a hazard that is live. +`tests/v2/test_ledger_guards.py` asks both over every corpus on disk +with no baseline wheel, so a rule added by a later bundle is checked +at pytest speed; `compare.py` asks both over the entries the run +actually loaded, before it spawns the worker. Do NOT answer either +failure by reordering: that moves which rule classifies a name and +breaks `_CROSS_RULE_WINNERS`. + +**Under `--corpus` the two checks are NOT symmetric**, which is why +only one of them refuses there. A smaller name set removes contests. +For the undeclared check that can only UNDER-REPORT, never +false-alarm -- fewer contests, fewer things to declare -- so +`--corpus` is only ever more lenient. Not "fail-closed": this file +uses that for the `_CORPUS_TIERS` and floor rosters, which REFUSE on +a missing entry, and a check that errs toward not refusing is the +opposite of one that errs toward refusing. For the vacancy check it +INVERTS: a live declaration whose contested names +all sit outside the subset reads as vacant, and following the advice +would delete an exemption the full gate needs and then fail the full +run for the undeclared contest that reappears. So a vacancy is a hard +failure on a full run and a printed NOTE under `--corpus`. + +Three checks now read the flag differently, and the differences are +deliberate rather than untidy -- read them together before making any +of them uniform. The corpus-floor roster is SKIPPED entirely under +`--corpus`, because narrowing is the point of the flag. +`over_declared_rules` still FAILS the run and appends a NOTE saying +the union it computed is over a subset, so its repair advice is not +followed blindly. The vacancy check does not fail at all, because it +is the only one of the three whose verdict INVERTS under narrowing. + ### Shapes that must never be explained (`[[never]]`) A `[[change]]` rule says "this diff is intended, and here is what diff --git a/tools/differential/compare.py b/tools/differential/compare.py index e04c9159..39c5edc9 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -16,6 +16,7 @@ """ import argparse import importlib.util +import itertools import json import os import re @@ -511,7 +512,9 @@ def _is_latin_only(name: str) -> bool: #: enters only when every role and the ambiguity kinds agree (main()), #: so a rule listing it lists nothing else (validate_rules). _RULE_FIELDS = frozenset((*V2_FIELDS, "_ambiguities", "_initials")) -_RULE_KEYS = frozenset(("issue", "name_regex", "fields", "dormant", "orders")) +_RULE_KEYS = frozenset(( + "issue", "name_regex", "fields", "dormant", "orders", + "precedes_narrower")) #: The `orders` member naming the DEFAULT order -- the comparison whose @@ -717,18 +720,23 @@ def validate_rules(rules: list[dict[str, object]], ledger: str) -> None: about a rule's matching semantics drifting, but about an opt-out carrying a justification someone can review. """ - seen: set[str] = set() - for rule in rules: + # A dict rather than a set because `precedes_narrower` (#382) needs + # each rule's POSITION to check that an exemption points forward. + # The membership test is the same one the dedupe check was written + # with, and a rule's issue is unique by the time this loop ends, so + # the index it records is unambiguous. + positions: dict[str, int] = {} + for k, rule in enumerate(rules): issue = rule.get("issue") if not isinstance(issue, str): continue # the per-rule loop below rejects it with a better message - if issue in seen: + if issue in positions: raise SystemExit( f"{ledger} has two rules sharing the issue {issue!r}. The " f"dormancy check identifies a rule by its issue, so the " f"second would hide behind the first: it could explain " f"nothing and never be reported") - seen.add(issue) + positions[issue] = k for i, rule in enumerate(rules): where = f"{ledger} rule #{i + 1}" issue = rule.get("issue") @@ -778,6 +786,100 @@ def validate_rules(rules: list[dict[str, object]], ledger: str) -> None: f"no shape asks for, so the rule would explain " f"nothing and report as dormant instead of saying " f"the name is wrong") + if "precedes_narrower" in rule: + # #382. Where this rule deliberately outranks a NARROWER one + # it would otherwise lose nothing by yielding to. Legal, and + # never silent: `fields` cannot say that a wider rule + # describes a compound behavior its component rule does + # not, so the reason is the only place that fact can live. + # + # "Sits later in the file" means "loses to this rule" only + # because #451 and #456 force every rule to carry BOTH + # narrowing keys, which leaves one tier and makes + # _sorted_rules the identity on any ledger that validates. + # Relaxing either ban breaks the forward-only check below + # rather than merely widening it: a fields-only rule at + # position 1 could declare precedence over a name_regex rule + # at position 5 and be accepted here, while _sorted_rules + # puts the name_regex rule first and it is the one that + # actually wins. + declared = rule["precedes_narrower"] + if not isinstance(declared, list) or not declared \ + or not all(isinstance(e, dict) for e in declared): + raise SystemExit( + f"{where} has a 'precedes_narrower' that is not a " + f"non-empty list of tables ({declared!r}). Write it " + f"as [[change.precedes_narrower]] blocks under the " + f"rule -- single-bracket [change.precedes_narrower] " + f"makes ONE table rather than a list of them -- and " + f"delete the key rather than leaving an empty one, " + f"which declares nothing") + for entry in declared: + unknown = set(entry) - {"issue", "why"} + if unknown: + raise SystemExit( + f"{where} has {sorted(unknown)} inside a " + f"'precedes_narrower' entry, where only 'issue' " + f"and 'why' belong. If that key belongs to the " + f"RULE, move it ABOVE the " + f"[[change.precedes_narrower]] block: TOML binds " + f"every bare key after that header to the " + f"exemption, so a rule key written below it is " + f"silently dropped from the rule -- an 'orders' " + f"landing here deletes the rule's order " + f"narrowing and nothing else would notice") + target, why = entry.get("issue"), entry.get("why") + if not isinstance(target, str) or not target: + raise SystemExit( + f"{where} has a 'precedes_narrower' entry with " + f"no string 'issue': {entry!r}. An exemption " + f"names the ONE rule it outranks -- a blanket " + f"opt-out would be inherited by every narrower " + f"rule added later, which is the widening this " + f"check exists to refuse") + if not isinstance(why, str) or not why.strip(): + raise SystemExit( + f"{where} declares precedence over {target!r} " + f"with no 'why' ({why!r}). The reason is the " + f"whole safeguard, as it is for 'dormant': an " + f"exemption nobody had to justify is the one " + f"nobody reviews") + if target not in positions: + raise SystemExit( + f"{where} declares precedence over {target!r}, " + f"which names no rule in this ledger. A rule's " + f"issue string is its identity here; a renamed " + f"or deleted rule leaves an exemption that " + f"protects nothing") + if positions[target] == i: + raise SystemExit( + f"{where} declares precedence over ITSELF. An " + f"exemption names the OTHER rule this one " + f"outranks; no rule contests itself, so this is " + f"the declaring rule's own issue string copied " + f"where the narrower rule's belongs") + if positions[target] < i: + raise SystemExit( + f"{where} declares precedence over {target!r}, " + f"which sits EARLIER in the file (rule " + f"#{positions[target] + 1}). An exemption names " + f"the narrower rule this one outranks, and that " + f"rule is by definition the later one -- so this " + f"is a copy-paste of the wrong issue string, " + f"sitting in the file reading as a justification") + # The same copy-paste slip the `fields` duplicate check + # refuses, and worse here: the second entry exempts a pair + # already exempted, so it changes nothing -- but two reasons + # for one pair means one of them is stale, and a reviewer + # reading the ledger cannot tell which. + targets = [e["issue"] for e in declared] + dups = sorted({t for t in targets if targets.count(t) > 1}) + if dups: + raise SystemExit( + f"{where} declares precedence over {dups} more than " + f"once in 'precedes_narrower'. One pair takes one " + f"exemption, so the repeat exempts nothing new; keep " + f"the reason that is still true and delete the rest") has_regex, has_fields = "name_regex" in rule, "fields" in rule if not has_regex and not has_fields: raise SystemExit( @@ -1024,6 +1126,214 @@ def validate_exclusions(entries: list[dict[str, object]], f"'fields' to exclude any diff on a matching name") +class _Contest(NamedTuple): + """Two rules that file order alone separates (#382). + + `earlier` outranks `later` purely by position: there are diffs + both rules admit, so `classify()` returns the first one it reaches. + """ + #: issue of the earlier, WIDER rule -- the one that wins today + earlier: str + #: issue of the later, NARROWER rule + later: str + #: corpus names both `name_regex`es reach, sorted + names: tuple[str, ...] + + +class _Reach(NamedTuple): + """What one rule may claim, on each key _entry_matches narrows by.""" + issue: str + fields: frozenset[str] + #: corpus names its `name_regex` reaches + names: frozenset[str] + #: the orders it admits, or None when it declares none and so + #: admits every order. _legal_orders() IS the set of every order, + #: and substituting it here would still be wrong: _entry_matches + #: reads an absent `orders` off the key's ABSENCE rather than off a + #: member list, so on a rule list validate_rules never saw -- the + #: hand-built ones the tests pass in -- the two readings diverge. A + #: rule with `orders = ["MADE_UP"]` intersects _legal_orders() to + #: the empty set and would stop being a contest, where classify() + #: would happily run it against a real comparison and it IS one. + orders: frozenset[str] | None + + +def _rule_reach(rules: list[dict[str, object]], + names: list[str]) -> list[_Reach | None]: + """Per rule: its issue, and what it may claim on each narrowing key. + + None for a rule this check cannot reason about -- a missing or + mistyped `name_regex` or `fields`, or an empty `fields`. Every one + of those is already refused by validate_rules with a better + message; skipping rather than raising keeps this function usable on + the hand-built rule lists the tests pass it, and an empty `fields` + is skipped for a second reason: the empty set is a strict subset of + every other, so admitting it would report a contest against every + rule in the file. + + A mistyped `orders` is read as ABSENT rather than skipping the + rule, which is what _entry_matches does with one: it tests + `isinstance(orders, list)` and so returns a non-list rule to + claiming every order. validate_rules rejects that shape too, and + an empty `orders` besides -- so a frozenset here is never empty, + and "declares orders" always means a real restriction. + """ + out: list[_Reach | None] = [] + for rule in rules: + pattern, fields = rule.get("name_regex"), rule.get("fields") + if (not isinstance(pattern, str) or not isinstance(fields, list) + or not fields or not all(isinstance(f, str) for f in fields)): + out.append(None) + continue + orders = rule.get("orders") + matcher = re.compile(pattern) + out.append(_Reach( + str(rule.get("issue", "")), frozenset(fields), + frozenset(n for n in names if matcher.search(n)), + frozenset(orders) if isinstance(orders, list) else None)) + return out + + +def order_contests(rules: list[dict[str, object]], + names: list[str]) -> list[_Contest]: + """Every pair whose winner file order decides, exemptions IGNORED (#382). + + The predicate needs no diff shapes and that is what makes it cheap. + It asks the three questions _entry_matches asks, one per narrowing + key, and a pair is a contest only where all three overlap: + + `fields` -- where the later rule's are a STRICT subset of the + earlier one's, every diff D fitting the narrower set passes both + rules' subset test. The nesting supplies the contested shape's + EXISTENCE, which is why no diff has to be computed: doing that + properly would need the pinned-wheel worker pass, and could only + ever remove pairs from this list, never add one. + + `name_regex` -- some corpus name must reach both, which the corpus + supplies. + + `orders` -- some order must reach both. Two rules scoped to + disjoint orders never see the same comparison, so file order + decides nothing between them however nested their `fields` are, + and calling that a contest would demand a justification for a + hazard that cannot occur. A rule declaring no `orders` is + order-blind and overlaps every other, which is every rule in every + shipped ledger today. + + Equal `fields` are deliberately not a contest: neither rule is + narrower, so "narrow-first" says nothing about the pair and + _CROSS_RULE_WINNERS stays the instrument there. + + Read `precedes_narrower` through undeclared_contests, not here. + This function is what the recorded negative control measures, and a + control that consulted the mechanism it controls for measures + nothing. + """ + reach = _rule_reach(rules, names) + found: list[_Contest] = [] + for i, j in itertools.combinations(range(len(rules)), 2): + a, b = reach[i], reach[j] + if a is None or b is None: + continue + if not b.fields < a.fields: + continue + if a.orders is not None and b.orders is not None \ + and not a.orders & b.orders: + continue + shared = a.names & b.names + if shared: + found.append(_Contest(a.issue, b.issue, tuple(sorted(shared)))) + return found + + +def _declared_over(rule: dict[str, object]) -> frozenset[str]: + """Issues this rule declares precedence over (#382). + + Shape is validate_rules' business and this reader TRUSTS that it + ran: main() validates every ledger before reaching any of this, and + test_validate_rules_accepts_the_shipped_ledgers covers the files on + disk. The leniency exists so the function stays usable on the + hand-built rule lists the tests pass it. It is tempting to write + it up as a safety property; it is not one. Of the shapes + validate_rules refuses, four are refused toward REPORTING the + contest -- a non-list value, an EMPTY list, an entry that is not a + table, and an entry whose `issue` is not a string -- since each + leaves that entry out of the set and a smaller set declares less. + Two go the other way. An entry naming a real rule with a missing or + blank `why` reads here as a perfectly good declaration and retires + the pair; so does one whose `issue` is the EMPTY string, which + validate_rules refuses as a blanket opt-out but the + `isinstance(str)` test here admits -- and which retires the pair + against any rule whose own issue reads as "", the shape + `str(rule.get("issue", ""))` gives an issue-less hand-built rule. + The blank `why` is the likeliest hand-edit slip in a ledger, and + nothing in this function catches either of the two. + + So the guarantee is borrowed, not intrinsic. Reading strictly here + would be no less safe -- a stricter reader declares LESS and so can + only report MORE -- and the reason not to is convenience for + callers, which is a much smaller claim than "the safe direction". + """ + declared = rule.get("precedes_narrower") + if not isinstance(declared, list): + return frozenset() + return frozenset( + e["issue"] for e in declared + if isinstance(e, dict) and isinstance(e.get("issue"), str)) + + +def undeclared_contests(rules: list[dict[str, object]], + names: list[str]) -> list[_Contest]: + """Contests whose earlier rule does not declare the later one (#382). + + The declaration is read off the rule that WINS the pair, which is + the earlier one: an exemption is that rule saying it means to + outrank its narrower neighbour, so it is the only rule whose word + can retire the pair. + + `by_issue` is last-wins, so two rules sharing an issue string would + let a declaration on the second copy retire a contest the first + copy owns. validate_rules refuses duplicate issues, which is the + only reason that is unreachable -- the same borrowed guarantee + main()'s `rules_by_issue` leans on, and the same one _declared_over + leans on for shape. + """ + by_issue = {str(r.get("issue", "")): r for r in rules} + return [c for c in order_contests(rules, names) + if c.later not in _declared_over(by_issue.get(c.earlier, {}))] + + +class _Vacancy(NamedTuple): + """An exemption whose pair stopped being a contest (#382). + + Named rather than a bare pair for _Dormancy's reason: the caller + formats these into a message, and `v.earlier`/`v.later` says which + end is which where `v[0]`/`v[1]` would not. + """ + #: issue of the rule carrying the declaration + earlier: str + #: issue it declares precedence over + later: str + + +def vacant_exemptions(rules: list[dict[str, object]], + names: list[str]) -> list[_Vacancy]: + """Declared precedences over a pair that is NOT a contest (#382). + + A rule narrowed until it no longer overlaps its neighbour leaves + its exemption behind, and the file then carries a justification for + a hazard that is gone -- indistinguishable, to a reader, from one + that is live. Same shape as `dormant`'s awake check: the ledger + states a condition, and the harness refuses to let it go on + standing after the condition stops holding. + """ + live = {(c.earlier, c.later) for c in order_contests(rules, names)} + return [_Vacancy(str(rule.get("issue", "")), later) + for rule in rules + for later in sorted(_declared_over(rule)) + if (str(rule.get("issue", "")), later) not in live] + + def _entry_matches(rule: dict[str, object], name: str, diff_fields: set[str], order: str | None = None) -> bool: """Does this entry's narrowing admit this diff? @@ -1417,8 +1727,15 @@ def main() -> int: # while printing a summary that reads exactly like a full one. The # floors already name every corpus that is supposed to exist, so # ask them. Skipped under --corpus, where narrowing is the point. + # + # The same question, asked of the names rather than of the flag, + # answers "is this run over the FULL corpus" for the vacancy check + # below -- which is what that check needs, and not the same as + # "was --corpus omitted": the flag is `action="append"`, so naming + # every corpus explicitly narrows nothing. + missing = sorted(set(_CORPUS_FLOORS) - {p.name for p in paths}) + full_corpus = not missing if not args.corpus: - missing = sorted(set(_CORPUS_FLOORS) - {p.name for p in paths}) if missing: raise SystemExit( f"corpus files named in _CORPUS_FLOORS are not on disk: " @@ -1495,6 +1812,103 @@ def main() -> int: for e in entries: by_key.setdefault((e["name"], e.get("order")), e) entries = list(by_key.values()) + # Order checks here rather than beside validate_rules, which runs + # before any corpus is read: whether two rules CONTEST a diff is a + # question about NAMES -- both regexes have to reach one -- and the + # names arrive at this line. Before the worker pass, deliberately: + # a ledger refused after the multi-minute wait is a ledger refused + # too late (#382). + # + # The names are the LOADED entries, not the corpus*.jsonl glob the + # unit guard in tests/v2/test_ledger_guards.py reads: `--corpus` + # narrows what this run compares, and a run is judged on the names + # it read, while the guard judges every corpus on disk. + # + # LOADED, precisely -- this sits ahead of the baseline-minimum + # shape skip below, so `corpus_names` holds names an old baseline + # will not actually compare (1120 here against the 1113 the 1.4.0 + # run reports; the 7 are order-bearing shape-4/5 names). Kept ahead + # of it on purpose: the check then asks the same question at every + # baseline, as the unit guard does, and moving it after `kept` + # would make a ledger's acceptability depend on which release it is + # being compared against. + # + # THE TWO CHECKS READ A SMALLER NAME SET IN OPPOSITE DIRECTIONS, + # which is the whole reason only one of them refuses below. Dropping + # names can only remove contests. For `undeclared` that can only + # UNDER-REPORT, never false-alarm: fewer contests is fewer pairs + # anyone owes a declaration, so a partial run is strictly more + # lenient and can never invent a refusal. (Not "fail-closed" -- + # this file uses that term above for the _CORPUS_FLOORS and + # _CORPUS_TIERS rosters, which REFUSE on a missing entry, and a + # check that errs toward not refusing is the opposite of that.) + # For `vacant` it INVERTS -- a live declaration whose contested + # names are outside this run reads exactly like a stale one. Measured: every one of the six corpora, + # run alone against expected_since_1.4.0.toml, reports vacancies -- + # 11 of the 11 exemptions for corpus.jsonl, corpus_cjk.jsonl and + # corpus_shapes.jsonl, and 8, 7 and 5 for the other three. So a + # partial run NOTES that count and does not act on it. + # + # THREE CHECKS READ THE NARROWING AT THREE DIFFERENT STRENGTHS, and + # the differences are the point rather than an inconsistency to + # tidy. The corpus-floor roster above is SKIPPED entirely, because + # narrowing is what the flag is for. over_declared_rules still + # FAILS the run -- `overwide` feeds the exit code on every run -- + # and only appends a NOTE that the union it computed is over a + # subset, so its repair advice is not followed blindly. `vacant` + # alone does not fail, because it is the only one of the three + # whose VERDICT inverts under narrowing rather than merely its + # evidence. Do not fold the two branches below back into one + # shape, and do not level the three checks onto one strength: an + # earlier draft of `vacant` refused under `--corpus` and told the + # contributor to delete legitimate exemptions. + # + # `full_corpus`, not `args.corpus`: the question the inversion + # turns on is whether this run read every corpus, and `--corpus` is + # `action="append"`, so a run naming all six of them narrows + # nothing and must refuse a stale exemption exactly as a flagless + # run does. A genuine subset still only NOTEs, which is the whole + # of the argument above. + # + # `rules` here is _sorted_rules' output, which is intentional and + # harmless: since #451 every rule carries a name_regex, so the sort + # is the identity on every ledger that loads and positions are + # unchanged. Verified against all four shipped ledgers -- element + # identity, not just equality -- at the time of writing. + corpus_names = [str(e["name"]) for e in entries] + undeclared = undeclared_contests(rules, corpus_names) + if undeclared: + raise SystemExit("\n".join( + [f"{ledger.name} has {len(undeclared)} order-decided " + f"contest(s) nobody declared. Where the later rule's " + f"'fields' are a strict subset of the earlier one's and " + f"both regexes reach one name, file order alone picks the " + f"winner. Declare it on the EARLIER rule with a " + f"[[change.precedes_narrower]] block naming the later one " + f"and saying what it describes that the later one does " + f"not -- do NOT reorder, which moves which rule classifies " + f"a name and breaks _CROSS_RULE_WINNERS:"] + + [f" {c.earlier!r}\n outranks {c.later!r}\n" + f" on {len(c.names)} name(s), e.g. {list(c.names[:3])}" + for c in undeclared])) + vacant = vacant_exemptions(rules, corpus_names) + if vacant and not full_corpus: + print(f"NOTE: this run used --corpus, and over that SUBSET " + f"{len(vacant)} exemption(s) in {ledger.name} declare " + f"precedence over a pair nothing here contests. That " + f"count is not evidence of a stale exemption -- narrowing " + f"removes contests, so a declaration the full gate needs " + f"reads the same way. Re-run without --corpus before " + f"touching any of them.\n") + elif vacant: + raise SystemExit("\n".join( + [f"{ledger.name} carries {len(vacant)} exemption(s) over a " + f"pair that is not contested over the full corpus. A rule " + f"was narrowed or a corpus name left. Delete the exemption " + f"-- a justification for a hazard that is gone reads " + f"exactly like one for a hazard that is live:"] + + [f" {v.earlier!r}\n declares precedence over {v.later!r}" + for v in vacant])) # an ORDER-BEARING entry must never reach a worker whose baseline # cannot honor it (no Policy below 2.0.0) -- skip it and say so, # rather than shrink the comparison silently. An order-NONE diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 03ae1e37..50707d3b 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -12,8 +12,45 @@ # each entry's `name_regex`/`fields` as tight as the diff allows. # FILE ORDER IS LOAD-BEARING: every rule carries a `name_regex`, so # they all sit in one tier, _sorted_rules is stable, and file order -# decides every tie. Write the narrower rule first, and see the note -# above the four fix(suffix-routing) rules at the end of this file. +# decides every tie. Write the narrower rule first -- that is the +# DEFAULT and not a law: a wider rule that describes a compound +# behavior its component rule does not is the better classifier and +# stays where it is, saying so in a [[change.precedes_narrower]] block +# naming the rule it outranks (#382; eleven rules below carry one, and +# tools/differential/README.md's "Declaring a wide-first pair" owns +# the mechanics). Do NOT reorder to satisfy the check -- that moves +# which rule classifies a name and breaks _CROSS_RULE_WINNERS. See +# also the note above the four fix(suffix-routing) rules at the end of +# this file. +# +# RECOMPUTE, once for all eleven. Those `why` texts quote counts that +# drift -- "explains N names", the contract/radar splits inside them, +# the 17/15/9/6 breakdown on the compound/peel pair, and how many +# contract names go UNEXPLAINED if a rule is deleted. NONE is pinned +# by a test, so re-measure before editing any of them rather than +# copying the neighbouring digits (docs/design/AGENTS.md: a count +# that drifts carries its recompute; a stale one here has already +# reached a filed issue once). +# 1. `uv run python tools/differential/compare.py --baseline 1.4.0` +# prints `## (N)` per rule. N IS the explains-count; the +# roster under it truncates at ten names, the heading does not. +# 2. For a tier split, take each explained name's corpus file from +# the tools/differential/corpus*.jsonl glob and its tier from +# compare.py's _CORPUS_TIERS -- read that map, never rebuild it +# from a file's name or size. corpus.jsonl is the LARGEST corpus +# and it is RADAR; a name in both tiers reads contract, the +# dedup loading contract files first. +# 3. For "N names this pair is contested over", call +# order_contests(rules, names) with `names` the union of +# _load_entries over that same glob. +# 4. For "deleting the rule sends N contract names to UNEXPLAINED", +# delete the rule and re-run step 1. +# Do NOT substitute driving classify() over a rule's regex reach at +# its declared `fields`: that answers what the rule COULD claim, not +# what it explains. Measured 2026-09-02 over the rules named in the +# eleven exemptions, reach runs from equal to the explained count up +# to forty times it -- 288 against 8 for the lone post-comma routing +# rule, which is the rule a wrong count reached a filed issue on. [[change]] issue = "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots" @@ -301,6 +338,38 @@ issue = "fix(comma-family) a comma followed only by titles keeps the given/famil name_regex = "(?i)^john\\s+smith,\\s*mr\\.?(\\s+jr\\.?)?$" fields = ["given", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(comma-precomma-family) pre-comma run reads as family, not given" +why = """ +LIVE -- one of the two live pairs in this file, and the one where the +narrower rule would be actively WRONG rather than merely partial. +'John Smith, Mr.' diffs {given, family}, which the precomma rule +admits, so nothing but file order keeps it here. + +The discriminator is what becomes of the pre-comma run. Measured, 2.x +CREATES a split across it where 1.4 had none: 1.4 read first 'John +Smith' with no family at all, and the tree reads given 'John', family +'Smith'. So the precomma rule's stated behaviour -- the pre-comma run +reads as family -- is half true of this name and no more: 'Smith' +becomes the family, 'John' does not. This rule describes the split; +that one describes the whole run moving, which is not what happens +here. + +That claim is about THIS name and is deliberately not generalised to +the precomma rule's other names. 'Dr. Do Van Johnson, MD' is one of +its seven and its pre-comma run splits too, so "that rule only ever +explains whole-run moves" would be false -- which is the shape of +argument to distrust here, since nothing recomputes a claim about +another rule's names. + +'John Smith, Mr. Jr.' sits beside it latently: {given, family, +suffix} is outside the precomma rule's fields at any position. +_CROSS_RULE_WINNERS has no row for either name. It pins the sibling +decision instead -- ('Bob Jones, author', ('family','given')) to the +unsuffixed C1 rule above, on the note that "the rule written for that +shape is ahead of the precomma merge in the file" -- and this rule is +that rule's suffixed twin.""" + [[change]] issue = "fix(#296) a dropped prenominal takes the name position it occupies" # 'Do Quang Minh': 'do' left TITLES -- a postnominal (D.O.) and a @@ -343,6 +412,48 @@ issue = "fix(#296) a credential-only comma string reads a name and its postnomin name_regex = "(?i)^[a-z]{2,3}\\.?,\\s*phd$" fields = ["title", "given", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(comma-family) lone post-comma piece routes to suffix/title, not first" +why = """ +LATENT. Both names move {title, given, family, suffix}, and the +routing rule declares no `family`, so it is ineligible for them +wherever it sits -- file order decides nothing here today. What the +declaration records is the hazard that wakes if that rule's `fields` +ever grow `family`, and its own comment disclaims precisely that +growth ("family/`last` is unchanged either way -- pre-comma is +definitionally family") while the family move is the whole of the +credential reading here: 'MD' stops being a first name and becomes the +one-word name. This rule is the routing rule PLUS the pre-comma merge +on a single string, so a widened routing rule would take the union and +report only half of it. _CROSS_RULE_WINNERS pins ('MD, PHD', +('family','given','suffix','title')) here. Both names sit in +corpus_issues.jsonl alone -- radar since #468's tier split, and +untouched by #488, which created corpus_cjk_tolerated.jsonl and holds +neither name -- so neither can demand a rule any more and nothing +fatal turns on this pair. The tier reaches the NAMES, not both rules: +measured, this rule explains two names and both are radar, which makes +it a candidate for #495; the routing rule explains eight of which six +are contract, and is not one.""" + +[[change.precedes_narrower]] +issue = "fix(comma-precomma-family) pre-comma run reads as family, not given" +why = """ +LATENT by the same arithmetic and for a different reason: a four-field +diff cannot fit {given, family} at any position, so the precomma rule +is ineligible for these two wherever it sits. The hazard here is a +rule that is this one's PRE-COMMA HALF alone. 'MD' going given -> +family really is its shape, so a reader widening it would have a +plausible-looking case -- and it would then explain a credential-only +string as a name with a listing comma, losing the postnominal reading +that moves the title and the suffix in the same breath. A string that +is nothing but credentials is not a name plus a comma. Radar tier +since #468 on both names -- corpus_issues.jsonl, which #488 did not +touch -- so this pair is watched rather than enforced, and again only +the wider rule is #495's business. +Measured, the precomma rule explains seven names of which three are +contract ('Berg, abdul vd', 'Smith, Dr.', 'Smith, de Mesnil Jean'), so +it outlives the radar tier whatever #495 decides about this one.""" + [[change]] issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" # 'Smith, Ph. D. Jr.' and the #325 rows it leads ('Smith, Ph. D. MD', @@ -413,12 +524,37 @@ fields = ["given", "title", "suffix"] [[change]] issue = "fix(comma-precomma-family) pre-comma run reads as family, not given" -# 'Bob Jones, author' / 'MD, PHD': the post-comma piece is a title on +# 'Smith, Dr.' / 'Smith, Prof.': the post-comma piece is a title on # BOTH sides -- it does not move, and no suffix moves either. What # changes is the pre-comma run: 1.4 read it as `first`, 2.x reads it # as `last`, because pre-comma is definitionally family. Diff is # exactly {given, family}. # +# This comment opened on 'Bob Jones, author' / 'MD, PHD' until #382's +# sweep, and the rule explains neither any more. Since #296's audit +# 'Bob Jones, author' KEEPS its pre-comma split (first 'Bob Jones' -> +# 'Bob', last '' -> 'Jones') and goes to `fix(comma-family) a comma +# followed only by titles keeps the given/family split`, and 'MD, PHD' +# reads as a one-word name plus a postnominal -- a four-field diff +# this rule's fields cannot admit -- and goes to `fix(#296) a +# credential-only comma string reads a name and its postnominal`. Both +# handovers are pinned in _CROSS_RULE_WINNERS and both winners are +# written above this rule, so nothing about this rule changed; only +# the example did. Nothing recomputes a name in a comment, which is +# how this one stayed wrong through two audits. +# +# The seven it does explain are not all the pure shape, and `fields` +# is what the rule actually claims. 'Smith, Ph. D.' and 'Ph. D., Jr.' +# are the pure move; 'Dr. Do Van Johnson, MD' has its pre-comma run +# SPLIT into given/family rather than move whole; 'Berg, abdul vd' +# hands a trailing particle to the family; 'Smith, de Mesnil Jean' +# re-orders the family run with `first` empty on both sides. Measured, +# all seven move `family` and six move `given` with it; the seventh is +# 'Smith, de Mesnil Jean', which moves `family` ALONE -- `first` is +# empty on both sides, so there is no given name for it to move. No +# title and no suffix moves in any of the seven. So the fields below +# are the boundary and the title above names the majority reading. +# # Its own rule because these have nothing to do with suffix routing: # they were falling to the fields-only fix(suffix-routing) catch-all # (deleted in #451), on a rule whose every other name moves a trailing @@ -452,6 +588,25 @@ issue = "fix(#296) a lone post-comma credential is a suffix" name_regex = "(?i)^[a-z]+,\\s*[a-z]{2,6}\\.?$" fields = ["title", "given", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`" +why = """ +a REGEX ACCIDENT, not two descriptions competing. The jr rule asks for +a leading run of non-space characters, and that run swallows the +trailing comma of 'Smith,' -- which is the only way a comma name ever +reaches a rule whose prose scopes it to the comma-LESS two-token name. +Nothing that rule says is true of 'Smith, Jr.' or 'Kim, Jr.'. + +LATENT as well: both move {title, given, family, suffix} against the +jr rule's {family, suffix}. The jr rule's OWN comment already argues +this from the other side and names these two names -- "what keeps them +there is `fields`, not file order... Order is the second line of +defence, not the first". This entry is that second line written down: +a widening of the jr rule's `fields` would spend the fields argument +and leave nothing but the accident holding the names, and an accident +is a worse reason to hold a name than a description is. +_CROSS_RULE_WINNERS pins both names to this rule.""" + [[change]] issue = "fix(#400/#274) bound-given join and maiden consumption in one name" # 'abd Allah Smith nee Jones': two behaviours meeting in one name, so @@ -469,6 +624,24 @@ issue = "fix(#400/#274) bound-given join and maiden consumption in one name" name_regex = "(?i)^abd(\\s+\\S+){2}\\s+n[eé]e\\s" fields = ["given", "middle", "family", "maiden"] +[[change.precedes_narrower]] +issue = "fix(#400) abd joins the word after it as one given name" +why = """ +the canonical compound-versus-component shape. fix(#400) describes the +join alone -- 'abd' plus the word after it, with a family name left +behind -- and says nothing about a maiden marker, which is where the +other two roles in this name come from. + +LATENT: the measured diff is {given, middle, family, maiden}, and +`family` and `maiden` are outside fix(#400)'s {given, middle} at any +position, so the join rule cannot take this name wherever it sits. +The declaration is therefore about a future widening rather than +about today: teach fix(#400) anything about markers and it would +quietly absorb a four-role change and label it a bound-given join. +No _CROSS_RULE_WINNERS row for this name; the reason rests on the two +rules' own prose, which is unambiguous here -- one names two +behaviours meeting, the other one behaviour.""" + [[change]] issue = "fix(#411/S2) a declining bound-given join leaves the suffix reading after a family comma" # 'Berg, abd nee Jones': `abd` is the one shipped word in BOTH the @@ -488,6 +661,23 @@ issue = "fix(#411/S2) a declining bound-given join leaves the suffix reading aft name_regex = "(?i),\\s*abd\\s+n[eé]e\\b" fields = ["given", "middle", "suffix", "maiden"] +[[change.precedes_narrower]] +issue = "fix(#400) abd joins the word after it as one given name" +why = """ +this rule describes the join DECLINING, which is the negation of what +fix(#400) describes, and then describes what a declining join leaves +in the given slot after a family comma: S2's suffix reading, with +'abd' ending up in `suffix` and the name having no given name at all. +Two rules cannot both be right about the same word, and the one +asserting that it joins is not the one to explain a name where it did +not. + +LATENT: {given, middle, suffix, maiden} is two roles wider than +fix(#400)'s {given, middle}, so the subset test excludes it at any +position -- this rule's comment already reads the same arithmetic as +why neither fix(#411) nor fix(#274) claims the name. No +_CROSS_RULE_WINNERS row; the prose carries it.""" + [[change]] issue = "fix(#411) the bound-given reserve stops counting words the maiden name takes" # 'van der Berg, abdul nee Jones': P5 reserves a name word so the join @@ -557,6 +747,23 @@ issue = "fix(#272/#308) nakaguro division and a glued hangul honorific in one na name_regex = "^\\S*\u00b7\\S*\uc528$" fields = ["given", "middle", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix" +why = """ +`middle` is the discriminator and the nakaguro is where it comes from. +This name divides into given/middle/family AND peels its glued \uc528, and +this rule is the one describing both; the peel rule describes the peel +and scopes itself to a name with no comma anywhere and no space before +the honorific, saying nothing about a division. Claiming the name +there would be a rule claiming a diff it does not describe (#372). + +LATENT: {given, middle, family, suffix} is outside the peel rule's +{family, given, suffix} at any position, so what is declared is the +hazard of that rule acquiring `middle` -- which is exactly the role +the CJK order flip hands it, so the widening is a plausible one. +Contract tier (corpus_rules.jsonl), and not in _CROSS_RULE_WINNERS: +the roster never had this name added.""" + [[change]] issue = "fix(emoji-boundary) an emoji inside a token divides it" # 'John😀Smith': 1.4 dropped the emoji and read one token @@ -616,6 +823,24 @@ issue = "fix(nickname-typographic-pairs) two typographic quote spans read as one name_regex = "\u201e[^\u201c]+\u201c\\s+\\S+\\s+\u201c[^\u201d]+\u201d" fields = ["given", "middle", "family", "nickname"] +[[change.precedes_narrower]] +issue = "feat(#273) typographic nickname delimiters recognized by default" +why = """ +feat(#273) is single-span RECOGNITION: a smart-quoted nickname stops +leaking into `middle` as literal text, which is the {middle, nickname} +diff its own names carry ('Hans \u201eHansi\u201c M\u00fcller', 'John \u201cJack\u201d +Kennedy'). This name has two spans of different pairs, and the reason +it needs a rule of its own is the REMAINDER REJOINING as a name once +both come out -- that is what moves `given` and `family`, and #273's +prose has nothing to say about it. + +LATENT: {given, middle, family, nickname} is outside {middle, +nickname} at any position. The hazard declared is #273 being widened +toward the multi-span shape, which this rule's own comment already +guards from the other side ("A single-span regression must not be +absorbed here") -- this declaration is that guard read backwards. No +_CROSS_RULE_WINNERS row.""" + [[change]] issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" # 'Vega, Juan de la': the Dutch alphabetized listing moves the @@ -968,12 +1193,22 @@ issue = "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma gi # move no family, so claiming them there would be the same field-shape # coincidence #372 was filed over, relocated rather than fixed. # -# `fields` is what separates the two, not file order. The union rows -# ('Dr 김민준씨, V.', '田中さん, PhD') diff with `family` in the mix, -# which is outside this rule's [given, suffix], so they cannot match -# here however early it sits and still fall through to the compound -# rule. Written above it anyway, so the narrower rule is also the -# earlier one and the tie never has to be reasoned about. +# `fields` separates the two for the UNION rows ONLY, and file order +# decides the rest -- do not read the first half as the whole. The +# union rows ('Dr 김민준씨, V.', '田中さん, PhD') diff with `family` in +# the mix, which is outside this rule's [given, suffix], so they +# cannot match here however early it sits and still fall through to +# the compound rule. But the seven names this rule EXPLAINS diff +# exactly {given, suffix}, which both rules' `fields` admit, and the +# two regexes are byte-identical -- so file order alone holds them +# here, and swapping the two rules reattributes all seven (measured +# 2026-09-02; docs/design/decisions.md#differential-ledger's Declined +# entry on #382 option 1 carries the measurement and why narrowing +# the regex cannot change it). This rule is narrow-first, the +# declaration-free default, so the pair is no `order_contests` +# contest and owes no [[change.precedes_narrower]] block -- which is +# not the same as order deciding nothing. What pins the seven is +# _CROSS_RULE_WINNERS, by name. # # The regex is the same comma-plus-classified-codepoint pair the # compound rule carries, hand-copied from _SCRIPT_RANGES and pinned by @@ -1053,6 +1288,63 @@ issue = "fix(cjk-comma-compound) comma routing compounds with the CJK order flip name_regex = "(?s)(?=.*,)(?=.*[\\u3005-\\u3006\\u3040-\\u309F\\u30A0-\\u30FF\\u3400-\\u4DBF\\u4E00-\\u9FFF\\uF900-\\uFAFF\\uAC00-\\uD7A3\\uFF65-\\uFF65])" fields = ["given", "family", "title", "suffix"] +[[change.precedes_narrower]] +issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix" +why = """ +LIVE, and the least comfortable of the declarations in this file. +Seventeen names reach both regexes and FIFTEEN have a diff the peel +rule's {family, given, suffix} admits -- but six of those fifteen are +taken by rules written above both, and so are not this pair's to +decide: '田中, 太郎さん', '김, 민준씨', '김, 민준씨 (Jimmy)', +'선생님, J.씨' and '이, J.씨' go to fix(cjk-comma-honorific-peel), and +'田中さん, Dr.' to fix(#271/#272/#298). That leaves NINE this rule +holds and the peel rule could take, and position alone is what keeps +those nine here. Of the two names outside the fifteen, '田中さん, PhD' +lands here as well but moves `title` too, so the peel rule is +ineligible for it wherever it sits, and '田中さん, 太郎' does not diff +at all. All three counts are classify() over the seventeen, recomputed +rather than carried over. + +The order stays because the peel rule scopes itself +to a name with NO COMMA ANYWHERE, and promoting it would put that +label on '田中さん, 様.' and '김민준씨, J.씨', which do route a +post-comma piece and are the comma compound this rule is named for. + +What must NOT be claimed is that all nine are compounds. Measured +against the wheel, '王先生, V.', '田中さん, V.' and '김민준씨, V.' show +no comma routing and no order flip at all: 'V.' is the given name on +both sides, the pre-comma family is one unsegmented run, and the whole +diff is the glued peel. They arrive here because this rule's criterion +is "the diff includes `family`", and `family` moves only because the +peel took the honorific off it -- so for those three the rule's own +label is wider than the name is. The rule that would describe them, +fix(cjk-comma-honorific-peel) above, covers this shape for a +POST-comma given name and has no family-side twin. Writing one was +weighed as #496 and DECLINED: the three are radar (the paragraph +below), so no gate can demand the rule, and #495 asks whether the +radar-only rules already here still earn their place -- fewer rules +for this corner, not a second one. So holding them here is the least +wrong of the readings available, and this is the entry that says so +out loud rather than letting the rule name imply otherwise. + +All seventeen contested names are radar tier, by two warrants and +not one: sixteen sit in corpus_cjk_tolerated.jsonl, demoted there by +#488, and the seventeenth, 'Dr 田中さん, V.', is in corpus_issues.jsonl +alone -- harvested, append-only, and radar since #468. So nothing +fatal turns on who wins this particular contest: no rule here can be +demanded on their account, because an unmatched diff on a radar name +is reported rather than failed for want of a rule. Not never-fatal +unconditionally, though -- a [[never]] exclusion outranks the tier +(see _CORPUS_TIERS) and none refuses these. That is a fact about the +NAMES and it does not carry to the rules. Measured by deleting each +rule and re-classifying the corpus -- without fix(cjk-comma-compound) +the run still reports 0 unexplained, so it really is radar-only and is +the candidate #495 weighs; without the peel rule TWELVE contract-tier +names go UNEXPLAINED ('Andersonさん', '王先生' and '김민준씨' among +them) and the run fails. Fourteen of the seventeen names the peel rule +explains are contract, which is why #495 already records it as a rule +to keep rather than a deletion candidate.""" + [[change]] issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix" # '김민준씨' -> 김/민준/씨, '田中さん' -> 田中/さん, 'Andersonさん' -> @@ -1087,6 +1379,47 @@ issue = "fix(cjk-glued-honorific-peel) glued honorific peels into suffix" name_regex = "(?<=[^\\s,])(?:박사님|선생님|교수님|박사|씨|님|先生|女士|小姐|教授|様|さん|さま|くん|ちゃん)\\.?(?=$|[ ,])" fields = ["given", "family", "suffix"] +[[change.precedes_narrower]] +issue = "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`" +why = """ +a regex accident, and a script-blind one: the numeral rule opens on a +run of non-space characters, which matches kana as readily as Latin, +while its prose is about a two-token LATIN name with a trailing +numeral and no comma in it. This rule's own pattern already names +'田中さん II' -- it closes on a token-boundary lookahead rather than an +end anchor precisely because a Latin suffix can follow the honorific. + +LATENT: measured, the diff is {family, given, suffix} (1.4 read first +'田中さん', last 'II'; 2.x reads last '田中', suffix 'さん, II'), and +`given` is outside the numeral rule's {family, suffix} at any +position. _CROSS_RULE_WINNERS pins the name here. '田中さん II' is +radar tier since #488, so nothing in the gate turns on this contest. +The demotion candidate #495 weighs is the NUMERAL rule -- measured, +both names it explains ('John V', 'Mohamad X') are radar -- and not +this one, which explains fourteen contract-tier names.""" + +[[change.precedes_narrower]] +issue = "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`" +why = """ +the same script-blindness one rule further down: the jr rule's leading +non-space run matches hangul as readily as Latin, while the rule is +written for 'Smith Jr.' -- one Latin name word, one suffix word, no +comma. The trailing 'Jr.' does move here -- 1.4 read it as the family, +the tree reads suffix '씨, Jr.' -- and that half of the diff is the +{family, suffix} the jr rule is named for. What that rule cannot +describe is the OTHER token: the glued 씨 peels off '김민준씨' and +segments what is left into given '민준', family '김'. `given` is that +segmentation, it is this rule's subject, and it is where the diff +leaves the jr rule's reach. + +LATENT, the diff being {family, given, suffix} against the jr rule's +{family, suffix}. _CROSS_RULE_WINNERS pins the name here, on a shape +measured against the wheel -- the corroboration its structural twin +'田中さん II' did not have until #382 re-measured it. The NAME is radar +tier since #488, so nothing fatal turns on it, but neither RULE is a +#495 candidate: the jr rule explains the contract-tier 'Smith Jr.', +and this one fourteen contract names.""" + [[change]] issue = "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip" # '王小明 先生', '김민준 씨', '田中 太郎 様': #307 ships the spaced CJK @@ -2061,8 +2394,9 @@ fields = ["family", "suffix"] # reaches 'Carod i' and '田中さん II', and _CROSS_RULE_WINNERS pins # both -- but only the first is a real contest: measured by moving the # numeral rule to the front of the file, 'Carod i' changes hands and -# '田中さん II' does not, because its diff moves {given, suffix} and -# these four declare {family, suffix}. File order decides one, the +# '田中さん II' does not, because its diff moves {family, given, +# suffix} and these four declare {family, suffix}, so the `given` move +# is outside them wherever they sit. File order decides one, the # subset test the other, and the roster does not care which. [[change]] @@ -2113,8 +2447,9 @@ issue = "fix(suffix-routing) a two-token name ending in a roman numeral keeps it # 'Carod i' is fix(#397)'s NOT-WANTED rule above -- a real contest, # same fields and both regexes matching, decided by nothing but file # order -- and '田中さん II' is fix(cjk-glued-honorific-peel)'s, where -# the diff moves {given, suffix} and this rule's `fields` cannot admit -# it whatever the order. _CROSS_RULE_WINNERS pins both. That reach is +# the diff moves {family, given, suffix} and this rule's `fields` +# cannot admit the `given` move whatever the order. +# _CROSS_RULE_WINNERS pins both. That reach is # real, so neither is a _MUST_NOT_MATCH probe here: that roster tests # the regex, not classify(). name_regex = "(?i)^\\S+\\s+(X|IX|IV|V?I{1,3}|V)$" diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index e28ec3fb..f60f7148 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -5,7 +5,12 @@ # `fields` and no regex, #456 banned the reverse -- so a rule here # narrows by name AND by role, never by just one. Every # rule therefore sits in one tier, _sorted_rules is stable, and FILE -# ORDER decides every tie: write the narrower rule first. +# ORDER decides every tie: write the narrower rule first. That is the +# declaration-free DEFAULT rather than a law -- a wider rule that +# describes a compound behavior its component rule does not may sit +# first and declare it in a [[change.precedes_narrower]] block naming +# the rule it outranks (#382). Never reorder to satisfy that check; +# see tools/differential/README.md, "Declaring a wide-first pair". # # This file was opened empty the day 2.1.0 shipped (AGENTS.md release # step 8) and stayed that way through #354, #358 and #361, none of