diff --git a/AGENTS.md b/AGENTS.md index b7f8feef..e210185e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Three committed contributor docs carry the parser's normative rules and their re **Counting claims.** A bare count in prose is either an assertion or a liability, keyed by who observes its staleness: asserted counts (a test holds the number) fail CI at change time — the useful kind; dated snapshots ("51 sites at spec time") cannot go stale; standing present-tense prose counts are the forbidden class — promote to an assertion, add a date, or state the invariant and let a test count. After changing how many times something runs, sweep for counts, not for the thing's name. -**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the differential gate's classified summary, be verified against rules.md examples, or -- for a view the gate cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. The gate compares the seven role fields and `_ambiguities`, so `capitalized()`, `initials()` and any future render view are invisible to it (decisions.md#R4, #R3) and the first two sources cannot reach a claim about one: a gate run is byte-identical across the change, and an example line witnesses an output without counting anything. A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. Per-rule ledger toml comments asserting PARSER behavior cite rule IDs under the excerpt discipline; free prose is for ledger mechanics only (owned by tools/differential/README.md). +**Release-log claims.** Quantified or universal behavior claims in release bullets must come from the differential gate's classified summary, be verified against rules.md examples, or -- for a view the gate cannot see -- carry a recompute recipe stored with the design entry the bullet cites; never write one from memory. The classified summary covers the CONTRACT tier plus whatever radar diffs a rule classifies; a radar corpus's unmatched diffs are listed under UNCLASSIFIED (radar) and are not in it, so a claim quantified from the summary alone is silent about them. The gate compares the seven role fields and `_ambiguities`, so `capitalized()`, `initials()` and any future render view are invisible to it (decisions.md#R4, #R3) and the first two sources cannot reach a claim about one: a gate run is byte-identical across the change, and an example line witnesses an output without counting anything. A recipe names the corpus files, the policy sweep, and -- the part that is easy to omit and fatal -- THE COMPARATOR, which must be something the shipped tree is not: #408's first recipe said to compare `initials()` against a folded-first partition, which is what `initials()` now IS, so it reproduced 0 where the bullet claimed 660 and was the only stated provenance for the number. Run the recipe as written before shipping the bullet. Cross-version numbers (a released wheel, the pre-change tree) are dated snapshots under Counting claims, since nothing in the repository re-runs them. Per-rule ledger toml comments asserting PARSER behavior cite rule IDs under the excerpt discipline; free prose is for ledger mechanics only (owned by tools/differential/README.md). **Working on docs/design/ has its own AGENTS.md.** `docs/design/AGENTS.md` carries the landing-a-design distillation checklist, the primary-source review rule, the dated-count convention, and the review axes. Claude Code loads it automatically when a session reads or edits anything under docs/design/; if your tool does not do nested discovery, read it yourself before touching those files or reviewing a change to them. @@ -110,6 +110,11 @@ uv run sphinx-build -b html docs dist/docs # code with the pipe's, so a failing run reads as a passing one. The # classified summary it prints is the source for the release notes' behavior # claims, including the count of changed names that are Latin-only. +# Exit 0 no longer means every diff is classified: since the tier split +# (#468) a radar corpus's unmatched diffs print under UNCLASSIFIED (radar) +# and cannot fail the run. Read that block. A radar diff worth a release +# note gets promoted (a cases.py row plus a shape tag) or classified with a +# rule BEFORE the log is drafted, not after. # 2. Clear PRE_RELEASE in nameparser/_version.py — it carries 'dev' through the # cycle (see step 9), so releasing is setting it to ''. VERSION should already # be the version you are shipping; bump it here only if step 9 was skipped. diff --git a/docs/customize.rst b/docs/customize.rst index 8e4010fa..397a618b 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -433,7 +433,7 @@ particle *ends* the name there is nothing ahead of it to join, and what it belongs to is decided by what the writing says rather than by the word. Two things say it, and both amount to someone stating that the family name came first — a family comma, and a declared family-first -order — so a Dutch listing reads the same either way: +order: .. doctest:: @@ -442,10 +442,41 @@ order — so a Dutch listing reads the same either way: >>> family_first.parse("Jong Anke de").family # the order says so 'de Jong' -``FAMILY_FIRST`` is the only order this arises under, because it is the -only one that puts a trailing piece in the *middle*, where a particle -means nothing. ``FAMILY_FIRST_GIVEN_LAST`` puts it in the given slot, -where your own declaration says it is the given name, so it stays one: +That pair is not a coincidence but one shape written two ways: form 4 +(``Title Family Given Middle Middle [Particle] [, Suffix]``) is form 2 +(``Family [Suffix], Title Given (Nickname) Middle Middle[,] Suffix [, +Suffix]``) with the comma removed and the family folded inline — +titles included, so a title that form 2 writes after the comma leads +the name in form 4 instead. If your records are family-first without +commas, ``Policy(name_order=FAMILY_FIRST)`` reads them the way the +comma format is already read. Measured over the whole particle +vocabulary — every particle nameparser ships, crossed with three +families and three given names, 630 generated pairs in all — 603 of +630 agree (2026-08-30); the executable form of this correspondence is +``tests/v2/test_order_correspondence.py``. + +Three limits keep that statement honest. + +The correspondence covers one shape written two ways, not +comma-deletion in general: a name whose shape changes when the comma +is removed — a title or suffix crossing to a different position — +parses as the shape it becomes, not as a disagreeing reading of form +2. Where the trailing word is both particle and suffix vocabulary, +the two writings read it differently, and it is that ASYMMETRY rather +than a precedence that breaks the correspondence. The particle +attachment outranks the suffix reading on the comma side alone — +that is the scope the rule is stated in — so +``parse("Ménil, Christophe vd")`` reads family ``vd Ménil``, while +``family_first.parse("Ménil Christophe vd")`` reads family ``Ménil`` +and suffix ``vd``. A listing ending in one of those three words +therefore does not correspond between the two writings — this is the +whole of the 27 disagreeing pairs, not scatter. And ``FAMILY_FIRST`` is the only order the correspondence +reaches at all, because it is the only one that puts a trailing piece +in the *middle*, where a particle means nothing; ``FAMILY_FIRST_GIVEN_LAST`` +puts it in the given slot, where your own declaration already says +it is the given name, and no comma format writes the given name +last, so form 5 has no comma twin to correspond to in the first +place: .. doctest:: diff --git a/docs/design/decisions.md b/docs/design/decisions.md index f16badd4..8412bc02 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -32,7 +32,7 @@ the 2026-08-16 entries below. The survivor is the degenerate bare - 2026-08 #367 — titles are transparent to the fold: "Sir de Mesnil" now reads like "de Mesnil". Fixed by removing the title→particle chain in grouping, not by touching this rule. - 2026-08-16 (order-precedence keystone; #364, #365, #368) — the stage split is the decision, and the three issues are one question seen from three angles. GROUPING is vocabulary's job and is order-independent: a particle joins forward through consecutive particles and stops at the first non-particle, and no name_order moves that stopping point. ASSIGNMENT is name_order's job: groups take roles by the declared order. The bugs existed because the implementation runs the two in the opposite dependency — `assign` hands out positions from `_effective_order` and `post_rules` then inspects a fixed list of ROLES, so P1's fold sites and P2's chain had coverage that varied with the declared order by accident. Consequences, each recorded in its own right below: the fold takes only the particle's own group (#364); the middle position needs no third site once grouping is order-independent (#365); and #368 reverses. - 2026-08-16 #364 — the fold takes the particle RUN and the ONE name word it attaches to, not every remaining word. - "de Mesnil Jean" is family="de Mesnil" plus given="Juan". Run, + "de Mesnil Jean" is family="de Mesnil" plus given="Jean". Run, not particle: "de la Vega" is two particles onto one word, and an earlier wording here said "the particle", which its own example contradicted (rule-vs-decision-record review). Nothing ever argued for "takes everything"; it was the shape of v1's handle_non_first_name_prefix, not a decision. Measured before deciding, over every name in the three differential corpora with NO string prefilter: exactly ONE family holds words beyond its particle's group — "de Mesnil Garcia". @@ -66,7 +66,7 @@ Declined: "not a title or a prefix". - 2026-08-16 — deleting P4 so a leading particle chains and is then positioned, which is the only way to make "de Mesnil Jean" vary by declared order. It avoids given="de" (the group would be [de Mesnil]) but breaks "de la Vega": measured, a single group is - positioned by the declared order — "Cher" reads given under GIVEN_FIRST — so "de la Vega" would read given="de la Vega" unless a further rule forced a particle-headed group into the family. Add that rule and [de Mesnil][Juan] yields the #364 reading anyway, so the deletion buys nothing and costs P4. + positioned by the declared order — "Cher" reads given under GIVEN_FIRST — so "de la Vega" would read given="de la Vega" unless a further rule forced a particle-headed group into the family. Add that rule and [de Mesnil][Jean] yields the #364 reading anyway, so the deletion buys nothing and costs P4. Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which particles count as never-given (the criterion is settled at decisions.md#vocabulary-collisions; the 39-member application is not). @@ -129,7 +129,7 @@ Open: [#380](https://github.com/derek73/python-nameparser/issues/380) covers "Be Scope the invariant to that SHAPE as well: 219 of the 304 comma-bearing corpus names parse differently with the comma simply removed, because a comma also delimits titles and suffixes. (An earlier draft said "181 of 247", which reproduces under no definition of "comma name" — the 304 matches the count the 2026-08-29 entry above already records.) An asymmetry falls out of it rather than being carved: shape 2 places the given name BEFORE the middles, so it corresponds to FAMILY_FIRST only. There is no comma format that puts the given name last, so FAMILY_FIRST_GIVEN_LAST has no twin to inherit an equivalence from. That is why the trailing particle is one order's business (#469). ONE EXAMPLE LINE WAS DELETED RATHER THAN UPDATED: `"Nguyen Thi Van" family-first → middle="Van"`. The name is written in FAMILY_FIRST_GIVEN_LAST; its FAMILY_FIRST reading is wrong by construction, so the line pinned a value nobody wants and nothing should depend on. Recorded here because the general question — whether a rules.md example should declare its format, and whether the doc test should refuse to run one outside it — is #470 and is NOT decided by this change. - REACH: 30 of 6606 parses move — this tree's 1101-name corpus × three `name_order` values × `middle_as_family` off and on — over twelve names, NONE under the default order. Recompute by parsing the four `tools/differential/corpus*.jsonl` files of this tree, deduped, under each of the six policies, and comparing the seven role fields against a checkout of the parent commit reading the SAME corpus files; the comparator is the pre-change code, never a construction of this rule. Dated snapshot, measured 2026-08-30. Note the corpus grew from master's 1094 by this change's own example lines, which is why the count must name the tree it was taken on. + REACH: 30 of 6606 parses move — this tree's 1101-name corpus × three `name_order` values × `middle_as_family` off and on — over twelve names, NONE under the default order. Recompute by parsing the `tools/differential/corpus*.jsonl` files of this tree, deduped, under each of the six policies, and comparing the seven role fields against a checkout of the parent commit reading the SAME corpus files; the comparator is the pre-change code, never a construction of this rule. Dated snapshot, measured 2026-08-30. Note the corpus grew from master's 1094 by this change's own example lines, which is why the count must name the tree it was taken on. The differential gate cannot see any of it: `compare.py` parses every name as `HumanName(name)` and sweeps no policy, so no non-default `name_order` behavior has ever been compared across versions. Output is byte-identical at all three baselines apart from the added corpus names, none of which diffs. WHAT REVIEW FOUND, because the corpus could not. Four agents on PR #472 found the site firing outside the order it was reasoned from, in two directions, and the differential corpus reported neither: 52 default-order names (a conjunction stops a particle's forward chain and leaves it standing in a middle — `Maria Luisa y de la Cruz` gave family 'de la Cruz' where master gives 'la Cruz') and 366 FAMILY_FIRST_GIVEN_LAST folds of a particle that does not end the name (`de Anke van y` gave family 'van de Anke' with the given name still behind it). Of 1099 corpus names only 6 hold a conjunction-then-particle adjacency and none in this shape, so the REACH recipe honestly reported 0 default-order movers throughout. The count was right and the population was blind; "none under the default order" was a property of the corpus, not of the code. Both are now `· boundary` examples in rules.md so the doc runner holds them. The repair is one clause — the run must END the name — and it subsumes the order test a first repair added: measured over 542,592 generated parses, an explicit `state.order[0] is Role.FAMILY` never decides anything the ends-the-name test has not already decided, so it is not in the code. What IS kept is `state.order is not None`, which is what keeps this site off the family-comma path (0 co-firings with P6's comma site over ~892,000 parses). @@ -687,6 +687,20 @@ Declined: - `explains = N` per rule (2026-08-28, #452's own first sketch) — catches a rule explaining fewer NAMES even while its fields stay exercised, which the field check does not. Declined on maintenance cost rather than merit: 179 numbers across the three ledgers, every one of them liable to move whenever a corpus does rather than only when a rule does. Measured on the precedent — #414's harvest moved 17 of the 30 `_CORPUS_CLAIMS` entries that existed then and added 14 more, parsing the roster at `7a10689^` and `7a10689`. (That roster's own comment said "Twenty of these moved" and was wrong; measured three ways — that commit, the whole of PR #415, and the backtick-harvest commit after it — the figure is 17 every time, and the comment now says so with the recipe.) The field check needs no number and cannot go stale. This is the design to revisit if a name-count shrink ever bites. - Extending it to `'Donald mc'` (2026-08-27) — `mc` is `SUFFIX_ACRONYMS` AND `PARTICLES`, the collision C-ii decided for `vd`, and C-ii reaches this shape only through P6's attachment, which is scoped to the comma form. The authority is rules.md#S2's Accepted — "an unambiguous suffix is consumed even when that leaves no family name at all" — and NOT P6's scope note, which was this entry's first citation and does not support it. P6 promises the comma-less shapes "keep their positional reading", and for the words that are both particle and suffix vocabulary it does not hold: measured, `'Jong de'` and `'Smith do'` keep theirs (family 'de', family 'do') while `'Donald mc'` and `'Smith vd'` lose the family entirely to `suffix`. So `'Donald mc'` is out of #451's scope on S2's statement, and P6's own Accepted clause is falsified by two of the three words it sweeps in — its only example, `"Jong Anke de"`, is `de`-shaped, which is why no test catches it. Recorded here rather than fixed; it is a rules.md defect, not a ledger one. +### differential-ledger, the corpus-tier arc (2026-09-01, #468/#469) + +The fourth ledger arc, and the first to change what the gate is FOR rather than what a rule may say. Mechanics — the tier table, the promotion recipe, the shape key, the `corpora:` line — are owned by tools/differential/README.md and not restated here; these are the decisions behind them. Counts are this session's, each recomputable by running `uv run python tools/differential/compare.py` (add `--baseline 1.4.0` for the skip figures) over the checked-in corpora and reading its `corpora:` and `corpus:` lines. + +Decisions that landed: + +- 2026-09-01 #468 — every corpus carries a TIER, and only the CONTRACT tier can fail the run. `corpus.jsonl` (scraped from v1's test banks) and `corpus_issues.jsonl` (harvested from the tracker) became RADAR: their diffs still classify against the ledger, so release notes keep their grouping, but an unmatched one prints under `UNCLASSIFIED (radar)` and the run still exits 0. `corpus_cjk.jsonl`, `corpus_rules.jsonl` and `corpus_shapes.jsonl` keep the full discipline. The reason is a VOCABULARY gap and not a volume one: the gate demanded a rule-precision explanation for names nobody chose, and the honest explanation for many of them was "this string exercises nothing" — which no ledger key can say: a rule means "this diff is intended and here is why", an exclusion means "this shape must never be explained", and neither is the sentence wanted. A rule invented to say it would be the #372 shape by construction, since the only thing it could narrow on is which file the name happens to sit in. `_CORPUS_TIERS` in compare.py is the roster and is fail-closed like `_CORPUS_FLOORS`: a corpus added without a tier is a hard error, so the split forces a decision when a corpus arrives rather than defaulting one. +- 2026-09-01 #468 — nothing is DELETED to quiet the gate; promotion is the answer to "then how does a radar name ever bind again". A name enters the contract by being CHOSEN — a tests/v2/cases.py row plus a shape tag, which puts it in `corpus_shapes.jsonl` at the next regeneration. Deleting the string was the alternative and is worse in both directions: it loses the radar signal, and it makes the gate's own population editable by whoever finds the gate inconvenient, which is the one edit a differential gate must not make cheap. A meaningless radar string costs one parse and one report line against a comparison of 1113 names today. +- 2026-09-01 (decided in review) — a `[[never]]` exclusion OUTRANKS the tier: a diff it refuses reports UNEXPLAINED and exits 1 on either tier. Not a carve-out but the tier's own principle applied — the split governs names nobody looked at, and an exclusion is chosen exactly as a rule is, someone having written its `why` and its `examples`. It is also what the shipped ledgers already promise, and the measurement is what settled it: the 1.4.0 ledger's trailing-`Ph. D.` exclusion, whose `why` says "a diff here is a regression", reaches exactly three corpus names (`John Smith Ph. D.`, `John Smith, Ph. D.`, `John Smith, Ph.D.`) and every one of them lives in `corpus.jsonl` or `corpus_issues.jsonl` alone. Reading the tier as outranking the exclusion would have retired that entry outright, silently, in the commit that split the corpora. Recompute by matching each `[[never]]` entry's `name_regex` over the corpus files and reading which files the matches come from. +- 2026-09-01 (decided in review) — a ledger RULE may narrow by comparison order, through a new `orders` key; it is optional, and its absence is the order-blind behavior every rule written before shape-tagged entries has. Forced by the first order-scoped rule anyone wrote: `feat(#395)`'s fold on `de la Cruz Juan Carlos` moves `{family, given, middle}` under the two family-first orders, and the most plausible regression of that very change — the fold leaking into the DEFAULT order — moves those same three roles on that same string, which `corpus_rules.jsonl` carries as a bare string and therefore compares. Order-blind, the rule describing the feature would have absorbed the regression of the feature and called it intentional: #372's failure mode reached from a new direction, and on the one name best placed to hide it. The legal set is BORROWED from shapes.py rather than hand-copied (`_legal_orders`), the same call build_cjk_corpus.py makes for the script table — an order no shape declares is an order no comparison runs under, so a rule scoped to a typo could only ever be dormant. Amended 2026-09-01 (decided in review): the legal set gains a `DEFAULT` SENTINEL, and the two legacy rules that needed it were scoped. Found when the reviewer traced the absorption running the OTHER way from the one above — the order-blind `fix(#399) a maiden marker bounds the particle chain that swallowed it` rule reaches `de la Cruz née Vega`, which `corpus_shapes.jsonl` compares under both family-first orders, and it sorts AHEAD of the two rules written for that name, so a family-first-only `{family, maiden}` regression there would have classified as intentional. The fix a rule wants is `orders`, and no rule could say "default order only": the members are constant NAMES, the default order is the absence of one, and TOML has no null inside an array. So `"DEFAULT"` names it. `fix(#399) …that swallowed it` took `["DEFAULT"]` in both 2.x ledgers and `fix(#296) … 'John Smith, Dr.' keeps its split and its title` took `["DEFAULT", "FAMILY_FIRST"]`, those being the comparisons each one's diffs actually come from. Order-blind rules stay legal — every ledger written before shape tags is full of them — so the absorption is made VISIBLE rather than illegal: a run prints an `ORDER-BLIND` block for each (issue, name, order) where a rule with no `orders` key explained an order-bearing diff, informational and outside the exit code. Recompute by running the gate at each of the four baselines and reading that block; it is empty at all four today. +- 2026-09-01 (decided in review) — EXCLUSIONS stay order-blind, and `validate_exclusions` rejects an `orders` key as unknown. The dormancy arc's monotone-refusal reasoning is the whole argument: an exclusion only ever removes a name from classification, so the widest thing an over-wide one can do is make a name report UNEXPLAINED and fail the run. There is no silent direction to protect against, and the narrowing a rule needs is the one an exclusion must not have — "never explain this, but only under FAMILY_FIRST" is a promise with a hole in it. +- 2026-09-01 #469 — "some names are only supported under one order" is answered by the SHAPE rather than by a ledger exception. shapes.py records, per input shape, the `name_order` the shape is an input FOR and the oldest baseline whose worker can honor that order; `compare.py` parses a tagged entry under that order on both sides, compares it on the v2 surface alone (the facade is the v1-compat surface, and a family-first name is not a v1 contract), and leaves out an entry whose run predates its `min_baseline` — reported as `skipped N names tagged shape(s) [...]` and counted in the `corpora:` line, so a shrunken comparison is never silent. Structural, and that is the point: nothing has to be written into any ledger to say a family-first name is not a default-order promise. Today's arithmetic, all of it recomputable from the checked-in files and one gate run: 13 shape entries, of which 4 dedupe into names already compared under the default order, for 1113 comparisons; shapes 4 and 5 declare `min_baseline` 2.0.0, so 7 entries are skipped at `--baseline 1.4.0` and 1106 names compare there. +- 2026-09-01 #469 — the CJK arrangement is deliberately OUTSIDE the shape inventory. Whether an unspaced CJK name is a third family-first shape is the open question on #469, and `corpus_cjk.jsonl` — contract-tier already, generated from the case table already — covers that ground meanwhile. Tagging it in would have to ANSWER the question first, since a shape row cannot be written without a `name_order` and a `min_baseline` for it, and the inventory is a table of arrangements rather than where that argument belongs. + ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01) #291 was filed 2026-07-26 out of the 2.0 vocabulary cleanup, with @@ -756,7 +770,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea - 2026-08-30 #408 — DONE: `initials()` orders folded words first, as every other view of the same parse already did. O3's fold and P6's attachment both TAG rather than move a token (mechanisms.md#FOLDED_TAG — spans cannot reorder), and `_types._text_for` partitions on that tag for every role it renders, so the `family` field reads `van der` where the tokens sit `der ... van`. `initials()` walked `tokens_for(role)` in written order and never asked, so the two views disagreed about one parse: `parse("der, y van")` gave family `van der` and initials `y. d. v.`, and it gives `y. v. d.` now. This is mechanisms.md#RENDER-HONORS-THE-PARSE's second shape, filed as its worked instance, so the design is not re-argued here — only what the measurement added to it. WHAT THE MEASUREMENT ADDED, and it reframes the issue from a consistency fix to a parity restoration. v1 had this feature as `CONSTANTS.middle_name_as_last` (`_config_shim` maps it to `Policy.middle_as_family`) and had NO general `name_order`, so only the two default-order policies have a v1 reference at all. Measured on the released 1.4.0 wheel over the 1094-name deduped corpus, from a script run outside the worktree that asserts `nameparser.__version__` first: at the default policy 1 name moves and agrees with 1.4.0 neither before nor after; under `middle_as_family` 71 move, of which **54 come into exact 1.4.0 agreement and NOT ONE moves away from it**. Worked pair: `Doe, Dr. John A.` gives `J. A. D.` at 1.4.0, gave `J. D. A.` here, and gives `J. A. D.` again; `Brundridge, Contessa A` likewise `C. A. B.` / `C. B. A.` / `C. A. B.`. The facade had been giving the folded answer all along — `HumanName.initials()` reads `first_list`/`middle_list`/`last_list`, and `_facade._list_for` prepends the carriers exactly as `_text_for` does — so the core view was simultaneously out of step with the field beside it, with the facade above it, and with 1.4.0 behind it. A view that stopped honoring a record had been reproducing a v1 bug v1 did not have. THE 17 THAT AGREE WITH 1.4.0 NEITHER WAY, recorded because the obvious explanation is wrong — and the FIRST REPLACEMENT for it was wrong the same way, corrected by this branch's docs review before merge. It is not that their parse already differs: only 2 of the 17 parse differently from 1.4.0 under this policy, `Smith, John V.` (#432's dotted numeral) and `남궁민수 지훈` (the unspaced-hangul surname split and the script order, rules.md#W1). The other 15 have SEVEN BYTE-IDENTICAL FIELDS at 1.4.0 and here — `der, y van` INCLUDED, which an earlier draft of this bullet filed with the two above by carrying over a measurement taken at the OTHER policy, where its parse does differ. Of those 15, 14 differ from 1.4.0 only in v1's initials GRANULARITY — v1 initials one group per `*_list` element, so a run its parse joined into one element (`Velasquez y Garcia`, `abdul salam`) yields `V G.` where 2.x, one token per word, yields `V. G.` — and strip the periods and spaces and all 14 come into 1.4.0's letter ORDER after this change, none having been in it before (`JVGQ` → `JQVG`). The 15th is `der, y van`, and it is worth naming rather than lumping: its fields match 1.4.0 exactly and its initials do not, because 1.4.0 contributes NOTHING for a family that is all particles (`van der` → `y.`) where 2.x contributes its words. That is R3's own adjudicated clause and decisions.md#R2's territory — R3 carries `"Juan van der" → initials="J. v. d."` against 1.4.0's `J.` — so this one name is a divergence this document has already decided in favor of, not an unexplained residue. So the parity claim is stronger than "54 of 71 and the rest unexplained": 68 of the 71 come into 1.4.0's ordering, 54 of them exactly, one more is R2's decided divergence, and 2 have a parse difference that predates this change. - BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. RECOMPUTE, and the COMPARATOR is the half that has to be stated, because the obvious phrasing of it measures nothing: parse the four `tools/differential/corpus*.jsonl` files deduped under each of the six policies and compare `initials()` against THE PRE-CHANGE RENDERING — the same contributing tokens left in WRITTEN order. What this sentence said until 2026-08-30 was "against a folded-first partition of `tokens_for(role)`", which is what `initials()` now IS: that recipe compares the shipped implementation with itself and yields 0 differing parses, the inert-measurement shape (mechanisms.md's Verification shapes section) written into the very sentence meant to prevent it, and the only recompute path the repository offered for docs/release_log.rst's counts as well. CONTRIBUTING is the other half of the recipe and it comes FIRST: apply this view's own skip filter — outside the given group a token tagged `particle` or `conjunction` contributes nothing unless it also carries UNJOINED_TAG — and compare the two orders only over what survives it. Leave the filter out and every skipped word reads as a difference instead: 1546 rather than 660 against written order, and 1018 rather than 0 against the folded partition. Re-measured this way on 2026-08-30 and reproducing the counts above: 660 of 6564 parses, 1 / 71 at the default order, 1 / 294 and 1 / 292 under the two family-first orders. + BLAST RADIUS by policy, over the same corpus × three `name_order` values × `middle_as_family` off/on (6564 parses, 660 move). Default order: 1 / 71. FAMILY_FIRST: 1 / 294. FAMILY_FIRST_GIVEN_LAST: 1 / 292. STATE THE POPULATION BESIDE THE COUNT, per this file's own rule: the shape an order change can be observed in is a contributing folded token standing AFTER a contributing unfolded one of the same role, and at the default order the corpus holds exactly 1 such name without `middle_as_family` and 72 with it. So the movers are 1 of 1 and 71 of 72 — a near-total hit rate over a corpus that carries one instance of P6's half, not a blast radius near zero. (The one non-mover is `Chang, Andy C I`, where the folded `C` and the family `Chang` share a letter and the reorder is invisible in initials.) Nine corpus names carry the tag at all at the default order; the other eight have a single contributing family token, which makes the partition the identity. The `middle_as_family` half is where the reach is, which follows from the producers: O3 folds every middle word, while P6's attachment moves initials only where the attached run is READMITTED by the unjoined mark — a family that is all particles — so it reaches one corpus name in each order and the same one. RECOMPUTE, and the COMPARATOR is the half that has to be stated, because the obvious phrasing of it measures nothing: parse the `tools/differential/corpus*.jsonl` files deduped under each of the six policies and compare `initials()` against THE PRE-CHANGE RENDERING — the same contributing tokens left in WRITTEN order. What this sentence said until 2026-08-30 was "against a folded-first partition of `tokens_for(role)`", which is what `initials()` now IS: that recipe compares the shipped implementation with itself and yields 0 differing parses, the inert-measurement shape (mechanisms.md's Verification shapes section) written into the very sentence meant to prevent it, and the only recompute path the repository offered for docs/release_log.rst's counts as well. CONTRIBUTING is the other half of the recipe and it comes FIRST: apply this view's own skip filter — outside the given group a token tagged `particle` or `conjunction` contributes nothing unless it also carries UNJOINED_TAG — and compare the two orders only over what survives it. Leave the filter out and every skipped word reads as a difference instead: 1546 rather than 660 against written order, and 1018 rather than 0 against the folded partition. Re-measured this way on 2026-08-30 and reproducing the counts above: 660 of 6564 parses, 1 / 71 at the default order, 1 / 294 and 1 / 292 under the two family-first orders. THE VERIFICATION LIMIT, and it is decisions.md#R4's, restated because a plausible reading of "one name moves at the default policy" is that the ledger owes it a rule. It does not, and could not: `tools/differential/compare.py` compares the seven role fields plus `_ambiguities`, and `initials()` is not among them — the same blindness recorded for `capitalized()` under R4. Measured rather than reasoned: the gate was run at all three baselines on the pre-change tree (exported with `git archive HEAD` into a scratch directory, so nothing in the working tree moved) and on the changed one, and the two outputs are byte-identical apart from the tree path and the baseline-worker temp path — 1094 corpus names, 229 / 194 / 102 intentional diffs, `unexplained: 0`, and 67 / 58 / 51 headings with identical per-heading counts, at 1.4.0 / 2.0.0 / 2.1.0. NO LEDGER RULE WAS ADDED, and none was needed. What stands in its place is `rules.md#R3`'s example line and `tests/v2/test_render.py`, both mutation-checked. WHY R3 GAINED A SENTENCE rather than leaning on R1. R1 already says folded family words "render before the rest of the family wherever they stood in the string" — but its subject is "Every FIELD", and `initials()` is not a field: R3 calls it "this view", and it takes a format spec and two separators rather than being read as an attribute. So R1 does not reach it by its own words, and #408 is precisely what leaving that to inference costs. R3's new clause states the order in R3's own voice, carries the O3 example line (`"Hassan, Mohamad Ahmad Ali"` under `middle_as_family` → `initials="M. A. A. H."`, reusing R1's own input so `corpus_rules.jsonl` is unchanged at 241 names), and R1 and R3 now name each other in `interacts:` — the field says which order, the view says it follows the field. P6's half gets no example line, for the same reason R1 carries none: the only input that witnesses it is `der, y van`, whose PARSE is itself contested, so pinning its initials normatively would pin a value another open question can move. It is pinned in the unit test instead, which is where today's-behavior pins belong. THE PARTITION IS PER ROLE, not scoped to FAMILY, and this is a deliberate choice rather than a copied line. Both producers re-role to FAMILY, so FAMILY is the only role a parse can put the tag on today and the GIVEN and MIDDLE arms are unreachable. They are there because `_text_for` partitions for every role it renders and the two views must not diverge again — the same uniformity `_post_rules`' own UNJOINED_TAG loop takes for its three roles, "for uniformity with the rule, not because all three are observable". A producer that ever folded into another part would otherwise reopen #408 there with nothing to fail. `test_initials_folds_in_every_role_it_renders` pins it from a hand-built name, since no input string can. Mutation-checked four ways — scoping the partition to FAMILY, and skipping it for GIVEN, for MIDDLE, for FAMILY — and each fails that test. It took THREE drafts of the fixture to make that true, and the two misses are the same miss one role apart. The first carried two GIVEN tokens and ONE MIDDLE: a one-element partition is the identity, so the MIDDLE arm was asserted and unpinned, and skipping the partition for MIDDLE passed the entire suite. The second fixed MIDDLE and carried NO FAMILY token at all, in a test named for every role it renders — so skipping the partition for FAMILY passed this test, and was caught only by its siblings and by R3's example line, neither of which is about per-role application. A zero-element group is the identity too, and reads even less like a gap than a one-element one, which is why the second miss survived a review that had just named the first. Two tokens per role is what closes it, and the general form is carried as mechanisms.md#TWO-ELEMENT-GROUPS rather than left here: a test written to pin a partition, a sort or a dedup needs at least two elements in every group it claims to cover, or the claim rides on a no-op. @@ -767,7 +781,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea - 2026-08-29 — this is a DELIBERATE DEVIATION from 1.4.0, not a parity restoration, said plainly because the surrounding branch work is mostly restorations and a reader will otherwise assume this is one. Measured on the released 1.4.0 wheel: `HumanName('ANH DO').capitalize()` leaves `'Anh do'` and `HumanName('anh van do').capitalize()` leaves `'Anh van do'`; both are `'Anh Do'` and `'Anh Van Do'` on this branch. There was nothing to restore — v1 lowercased on vocabulary membership alone and had no notion of a particle with nothing to join, the mark being a 2.x invention. - 2026-08-29 — the ACCEPTED COST, agreed before the change rather than discovered after it: a degenerate family that is nothing but particles capitalizes too, so `juan van der` renders `Juan Van Der` where 1.4.0 gave `Juan van der`. This is the same trade rules.md#R2 already took for `family_base` — those words ARE the base, so they are name words, and a parser that called them name words for sorting and particles for display would be the incoherent option. Blast radius, measured by running both predicates in one process, and stated over BOTH populations because the neighbouring bullets in this entry measure over 1094 and a bare "4 and 23" would read as measured there too. Over the 1090 names the corpora carried before this change: 4 names move under a plain `capitalized()` — `ANH DO`, `vai la`, `van ma van`, and `der, y van`, which becomes `y Van Der` and keeps its lowercase conjunction — and 23 under `force=True`, the extra 19 being mixed-case inputs the gate holds back (rules.md#R5). Over today's 1094 it is 5 and 24: this change's own R4 example rows added `anh van do`, and that name is itself a mover. - 2026-08-29 — THE VERIFICATION LIMIT, which matters more than the change. `tools/differential/compare.py` compares the seven role fields plus `_ambiguities` and nothing else, so `capitalized()` is not a compared field and NO ledger run at any baseline can see this deviation: the corpus counts and the three intentional-diff totals hold across it precisely because the harness is blind to it, and a green run is evidence about roles, not about case. mechanisms.md#FACADE-CONTRACT promises that v1-visible changes are "release-log-classified fixes, each verified in the differential ledger". The first half is kept — docs/release_log.rst carries the 2.2.0 bullet — and the SECOND HALF CANNOT BE, for this change or for any other that moves only case. Standing in its place: rules.md#R4's own example rows (`ANH DO`, `anh van do`, and the `juan de la vega` boundary), rules.md#R5's, and the v1-facade tests in tests/test_capitalization.py. The ones that pin the fix are mutation-checked against deletion of the tag consult; the `juan de la vega` control passes that mutation BY DESIGN, which is what makes it a control rather than a witness. Read the silence as a SCOPE and not as a gap: the harness compares parses, case repair is a render view, and nothing is owed on the tracker for it — the next bullet is why that is a decision rather than a deferral. -- 2026-08-29 — WHY CASE IS NOT A COMPARED SURFACE, settled here rather than left open, because "add a column" is the obvious answer to the bullet above and it is the wrong instrument. What the harness is FOR is parsing behavior — tools/differential/README.md says to run it "when touching parsing behavior, and before cutting a release" — and what earns it its two-process cost is that a role is EMERGENT: the eight stages fold over one name and each can move what the next sees, so the reach of a vocabulary or rule edit is not derivable by reading the diff and has to be measured over a corpus instead. Case repair is the opposite kind of thing. `_render._cap_word` is a pure function of `(word, role, tags, lexicon)` with four branches — the particle/conjunction lowercasing, the exceptions map, the Mac/Mc convention, and plain `capitalize` — sitting behind R5's single gate, so an edit to it is legible: you can see what you touched. The measurement agrees with the reasoning. Over the 1094 corpus names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the four corpus files deduped, and diffing the two maps; the count was the same over the 1090 names the corpora carried before this change). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. Isolating the residue means comparing case only where the fields agree, a diff-of-diffs, to reach a class that a unit test reaches directly. So the compared surfaces stay the seven roles plus `_ambiguities`, by design and not by omission, and case is pinned where case belongs: the `rules.md#R4` and `#R5` example rows, and tests/test_capitalization.py. +- 2026-08-29 — WHY CASE IS NOT A COMPARED SURFACE, settled here rather than left open, because "add a column" is the obvious answer to the bullet above and it is the wrong instrument. What the harness is FOR is parsing behavior — tools/differential/README.md says to run it "when touching parsing behavior, and before cutting a release" — and what earns it its two-process cost is that a role is EMERGENT: the eight stages fold over one name and each can move what the next sees, so the reach of a vocabulary or rule edit is not derivable by reading the diff and has to be measured over a corpus instead. Case repair is the opposite kind of thing. `_render._cap_word` is a pure function of `(word, role, tags, lexicon)` with four branches — the particle/conjunction lowercasing, the exceptions map, the Mac/Mc convention, and plain `capitalize` — sitting behind R5's single gate, so an edit to it is legible: you can see what you touched. The measurement agrees with the reasoning. Over the 1094 corpus names the 1.4.0 and 2.1.0 wheels ALREADY disagree on `capitalize(force=True)` for 139 of them (recompute by running `HumanName(n).capitalize(force=True)` under each wheel over the corpus files deduped, and diffing the two maps; the count was the same over the 1090 names the corpora carried before this change). Nearly every one is an echo of a parse diff the field comparison already classifies — a token that changed role capitalizes differently — so a column would re-report the ledger's own explained diffs in a vocabulary its rules cannot match, and would arrive pre-failed by 139 entries. Isolating the residue means comparing case only where the fields agree, a diff-of-diffs, to reach a class that a unit test reaches directly. So the compared surfaces stay the seven roles plus `_ambiguities`, by design and not by omission, and case is pinned where case belongs: the `rules.md#R4` and `#R5` example rows, and tests/test_capitalization.py. - 2026-08-29 — DEBT DISCHARGED, the obligation the last bullet of decisions.md#R5 left this commit; recorded here so the pointer resolves rather than dangling. The FALSEHOOD: R4 promised repair "vocabulary exceptions (McDonald) included" flatly, which `str(parse('Juan Mcdonald').capitalized())` — `'Juan Mcdonald'` — falsifies, the gate refusing before any vocabulary is consulted. The statement now scopes that promise to where repair acts at all and hands the where to R5. A SECOND error in the same eight words turned up while rewriting them, and is worth recording because it had survived unread for as long as the first: McDonald is not a vocabulary exception. The shipped exceptions are five entries — II, III, IV, M.D., Ph.D. (recompute with `Lexicon.default().capitalization_exceptions_map`) — and `Mc`/`Mac` is a separate convention applied to any name shaped like one, so the old parenthetical illustrated the promise with the one mechanism it was not about. The statement now names an exception the vocabulary actually records (Ph.D.) and the Mac/Mc convention separately. The AMBIGUITY: "an already-correct name comes back unchanged" is true read as idempotence and reads as false under bearer-correctness, and the repair was to say WHICH is meant — a name already written the way repair would write it comes back unchanged — plus one clause refusing the inference a reader might otherwise draw, that a spelling its bearer chose is not spared for having been chosen. That clause is deliberately a REFUSAL and not a carve-out: sparing deliberately single-cased names is behavior the parser does not have and that R5's rationale declines on the record, so `str(parse('bell hooks').capitalized())` is still `'Bell Hooks'`. R4's boundary moved too: the old `"Juan McDonald" → capitalized="Juan McDonald"` row was inert — it passes with R5's gate deleted — so it became a `capitalized_forced=` row, which asserts what repair APPLIES rather than what the gate keeps and fails when the Mac/Mc convention is removed (measured by neutering it in one process: `'Juan Mcdonald'`). The boundary is now `juan de la vega`, a working particle run R4's new clause deliberately does not reach. - 2026-08-29 — the CONJUNCTION CARVE-OUT, stated in rules.md#R4 rather than left to a code comment, and the reason the comment gave was false. `_cap_word`'s conjunction conjunct is deliberately NOT gated on the unjoined mark, and the behavior is right: R3 already carries the carve-out for initials ("A CONJUNCTION never initials, so a base that is one contributes nothing even then"), and R4 was the half of the pair that carried it nowhere. What the comment claimed instead was an impossibility — `particles` and `conjunctions` are disjoint, so no conjunction token can ever carry the mark — and disjointness is a property of the SHIPPED vocabulary while both sets are public, configurable API. Measured: under `Lexicon.default().add(particles={'y'})`, `anh y van` parses to an all-particle family whose `y` carries `conjunction`, `particle` and the mark at once, and repairs to `Anh y Van`; gating the conjunct too, in one process, gives `Anh Y Van`. So the clause decides something and is not a no-op. It gets NO example row, and that is a limit of the document rather than a choice: the runner parses with the default vocabulary, over which `particles ∩ conjunctions` is empty in the default lexicon and in all four locale packs, so no input string can witness it. `der, y van` is the nearest thing and does not witness it either — the `y` there is a GIVEN word standing beside the all-particle family, not inside it (tests/test_capitalization.py says so at the row). @@ -789,7 +803,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea - 2026-08-29 — WHY R5'S STATEMENT WAS REWORDED RATHER THAN GIVEN A THIRD EXAMPLE, recorded at length because the wrong turn was taken twice and the reasoning is reusable. R5 first said a mixed-case name is returned untouched "whether or not its casing is correct". That phrasing reads as a disjunction with two branches, and invites the question of which example witnesses the already-correct one. Nothing can: for a mixed-case name that repair would not alter, kept and repaired are the SAME STRING by construction, so no example line distinguishes a parser with the gate from one without it. The unwitnessable branch is a property of the claim, not a gap in our choice of names — and the bullet above had already said so ("an already-correct input cannot witness a rule whose subject is what gets kept REGARDLESS of correctness") before two review rounds pushed past it. Two rows were tried and both withdrawn. `"Juan McDonald" → capitalized="Juan McDonald"` was INERT (mechanisms.md's inert-measurement class): it passes with R5's gate deleted, which is the whole failure shape that class names. `"Vincent Van Gogh" → capitalized="Vincent Van Gogh"` did discriminate — measured by comparing the shipped call against gate-off behavior, which the forced call reproduces exactly, since deleting the gate is what `force` already does: shipped `'Vincent Van Gogh'`, gate-off `'Vincent van Gogh'`, the particle rule lowercasing `Van` the moment repair runs — but it was REDUNDANT, not complementary. It asserts the same proposition as the `Shirley Maclaine` row above it: repair would change this name, and the gate keeps it anyway. The two differ only in which repair rule would have fired, and which rule repair applies is R4's subject, not R5's. And the frame it was chosen under was itself wrong, which is the more useful half of the lesson (reframed 2026-08-29 on Derek's correction, while R4 was being reworked). The row was picked to be bearer-correct — a name whose casing the bearer would endorse — and that frame drags the document into per-name arguments about whose spelling is right: this one needed `Vincent van Gogh` to be the correct spelling, which P6's own examples contradict, and the tussenvoegsel convention behind them is amended later on this branch anyway. CORRECTNESS DOES NOT ENTER INTO IT. Mixed case is the writer making an explicit choice, and repair defers to that choice rather than judging it — a name kept is not a name endorsed. Read that way the withdrawn row needed no claim about Dutch orthography at all, and the reason it needed one is exactly that the frame was wrong. The fix was to stop claiming the unwitnessable branch: the statement now says a mixed-case name is kept and that whether its casing is right does not enter into the decision, which `Shirley Maclaine` — casing wrong, kept — witnesses whole. GENERAL LESSON, and the reason this is long: when an example cannot be found for half a rule, suspect the STATEMENT before suspecting the example set. A phrasing that promises more branches than the behavior has will absorb inert examples indefinitely, each one looking like progress. Caution for the commit that reworks R4 and will choose its own mixed-case rows: avoid a name whose family base is wholly particle vocabulary (`Anh Van Do`, base `Van Do`), because the #407 work changes how those capitalize and would neuter such a row silently; `gogh` and `vega` are in neither `particles` nor `particles_ambiguous`, `van` and `do` are in both. - 2026-08-29 — the override has TWO routes, and the rule states neither, by design. Per-call is the obvious one. The second is a v1 Constants attribute the facade still honors, `force_mixed_case_capitalization` (nameparser/_facade.py resolves it when no per-call value is given; docs/release_log.rst records it as "still honored through the facade"). Measured today: with that attribute set True, `HumanName('Shirley Maclaine')` then a bare `.capitalize()` — no argument at all — LEAVES `'Shirley MacLaine'` (the v1 call mutates in place and returns None, so the name is read back with `str()`). An earlier draft of R5 said "only an explicit request to repair regardless overrides that", which that measurement falsifies; the statement now says repair was asked for anyway, without saying by what route (it read `repair regardless of how the name is cased was asked for` until 2026-08-29, when that phrasing turned out to have a second reading -- see the override bullet below), because rules.md is implementation-free by its own preamble. - 2026-08-29 — `capitalized_forced` is a test-side pseudo-field, not a parser field. The route the EXAMPLES use is the per-call argument to `capitalized()`, and an argument is not a policy, locale or extras gate — the only three things rules.md's grammar admits in an example's annotation slot — so it cannot ride that slot, and the doc runner grew a resolver branch instead. (The facade attribute above is a second route to the same behavior, not a second thing to assert; the core takes the argument only.) The R5 block's example lines are asserted by the suite like any others. They do enlarge one file: `corpus_rules.jsonl` is generated from this document's examples, and `Shirley Maclaine` had not been among them, so regenerating adds it (235 lines, from 234). Whether that moves the DIFFERENTIAL population is a separate question with a per-name answer, and the general answer is that a rules.md example CAN move it: the harness dedupes across corpus files, so a new example name costs a population slot exactly when no other corpus already carries that same string. `Shirley Maclaine` was already in `corpus.jsonl`, so it costs nothing and the population holds at 1090 — measured, not assumed. The row withdrawn above is the counter-case, and worth keeping for it: `Vincent Van Gogh` appears in no other corpus (`corpus.jsonl` and `corpus_issues.jsonl` carry `Vincent van Gogh`, a DIFFERENT string that does not dedupe against it), and while it was in the block the population read 1091. Neither name needed a ledger rule, which is a measurement rather than a consequence of the above: the gate exits 0 with 0 unexplained at all three baselines either way. -- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16 — and the two directions are dropped for DIFFERENT reasons, which an earlier draft of this bullet ran together under the first. UPPER: the facade's default render spec omits the maiden name, so `str()` cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name — `maiden` is `y Jones` forced against `Y Jones` uppercased and every other role is byte-identical). LOWER: the two the facade drops are `John van der J. V` and `abdul V Smith`, and the maiden field is EMPTY in both directions for both, so the omitted role explains neither. What hides them is that `str()` CONCATENATES adjacent roles, so a token that crosses a role BOUNDARY and moves nothing else is invisible in the joined string: `John van der J. V` is family `van der J. V` forced against family `van der J.` plus suffix `V` lowercased, and `abdul V Smith` is given `Abdul V` forced against given `Abdul` plus middle `V` lowercased — same seven roles rendered, same string joined. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the four corpus files deduped and diffing. +- 2026-08-29 — WHAT THE OVERRIDE DOES AND DOES NOT PROMISE, from Derek's framing of R5 and then measured, because the framing implies a property that is ALMOST true and the gap is the useful part. The framing first, and it supersedes the correctness talk elsewhere in this entry: mixed case is the writer making an explicit choice, and repair defers to that choice instead of judging it. Nothing is being called correct or incorrect — a name kept is a name whose writer said something about it, and a name repaired is one whose writer did not. The property that seems to follow is that asking for repair REGARDLESS should ignore the given casing entirely, so one name repairs to one string however it was written. MEASURED over the 1094 corpus names and it does NOT hold. On the v2 core — `parse(n).capitalized(force=True)` against `parse(n.upper()).capitalized()`, rendering all seven roles — the two differ for 63 names, and the lowercase direction for 18. Naming the surface matters here, and BOTH counts move with the surface, which an earlier draft of this bullet got half right: through `HumanName` and `str()`, which is what tests/test_capitalization.py uses, the counts are 62 and 16 — and the two directions are dropped for DIFFERENT reasons, which an earlier draft of this bullet ran together under the first. UPPER: the facade's default render spec omits the maiden name, so `str()` cannot see `Jane van der Berg née y Jones` (the whole of the upper difference, and it is a conjunction inside the maiden name — `maiden` is `y Jones` forced against `Y Jones` uppercased and every other role is byte-identical). LOWER: the two the facade drops are `John van der J. V` and `abdul V Smith`, and the maiden field is EMPTY in both directions for both, so the omitted role explains neither. What hides them is that `str()` CONCATENATES adjacent roles, so a token that crosses a role BOUNDARY and moves nothing else is invisible in the joined string: `John van der J. V` is family `van der J. V` forced against family `van der J.` plus suffix `V` lowercased, and `abdul V Smith` is given `Abdul V` forced against given `Abdul` plus middle `V` lowercased — same seven roles rendered, same string joined. Re-derived 2026-08-29 on this branch, and named per name so the next reader can check the arithmetic: core 63/18, facade 62/16. Two things about those numbers are the opposite of what one would guess. UPPERCASE IS THE WORSE DIRECTION, not the clean one. And the misses are not merely the pipeline's case-sensitivity leaking in: of the 63, only 25 move a role at all, and the other 38 parse byte-identically and diverge inside the repair (25 and 37 through the facade, the missing name being one of the byte-identical ones). The mechanism is v1's initial carve-out — a conjunction is not lowercased where it is written initial-shaped, and initial-shaped means one CAPITAL letter — so uppercasing turns every one-letter conjunction into an initial (`Velasquez y Garcia, Dr. Juan Q.` forced keeps `y`; the same name uppercased then repaired gives `Y`, with a byte-identical partition either side -- the comma form is the one to cite here, the space-written `Dr. Juan Q. Velasquez y Garcia` being a member of the 25 whose roles DO move), and lowercasing turns a middle initial `E` into the Italian conjunction. 1.4.0 does the same (`JUAN Y GARCIA` capitalizes to `Juan Y Garcia`), so this is inherited, and it is recorded here rather than fixed here. Recompute both directions by running the two forms over the corpus files deduped and diffing. - 2026-08-29 — and therefore NOT stated in rules.md, which is a deliberate choice rather than an oversight. The document's examples are keyed on input STRINGS, so any statement of the property invites exactly the test that falsifies it — re-case the input, expect the same output — and the counterexamples are already in the corpora. The property is true of the repair given a parse, and rules.md speaks input-to-output; a rule stating it would be over-broad in the one direction a reader would check. What R5's statement says is enough for the promise that IS kept: a mixed-case name is kept unless repair was asked for anyway. That clause was REWORDED for this, and the reword is the whole point rather than a tidy-up. It read `unless repair regardless of how the name is cased was asked for`, which carries two readings -- the intended one, that the request overrides the keeping, and a second one, that the repair disregards the input's casing, which is this property in nearly this bullet's own words. A reader taking the second reading would run the re-casing test predicted above, land on `Velasquez y Garcia, Dr. Juan Q.` (in the corpus today), and conclude the RULE is wrong when only the phrasing was. Nine words, and they asserted the thing the paragraph exists to deny. The property is pinned in tests/test_capitalization.py instead, over names carrying no single-letter word whose class case decides, with `juan y garcia` beside it as the recorded exception. R5's example block gains `"SHIRLEY MACLAINE" → capitalized="Shirley MacLaine"` from this work, and it earns its place on its own ground rather than as half of a convergence pair: it is the only row in the block that fails when the gate is narrowed to lowercase-only, every other row passing that mutation. Measured three ways — gate deleted (passes, so it does not witness the gate's existence), gate narrowed to accept only all-lowercase (FAILS, and alone in the block), Mac/Mc convention deleted (fails, with the other two rows). Until it was added, R5 stated that repair acts on a name written entirely in one case and witnessed only the lowercase half of it. That lowercase half is still `"juan mcdonald"`, which is byte for byte an R4 row as well, and the duplication is deliberate rather than an editing slip: the two rules make different claims about the same line — R4 that the repair honors the Mac/Mc convention, R5 that an all-lowercase name is acted on at all — and dropping it from R5 would leave the gate's lowercase half unwitnessed inside the rule that states the gate. Five other rows already sit under two rules apiece for the same reason (P5/P6 twice, P5/O5, N3/M4, W1/W3). - 2026-08-29 — DEBT this extraction leaves, named so the next commit inherits an obligation rather than a rediscovery. Pulling the gate out into R5 leaves R4 carrying ONE falsehood and ONE ambiguity — different defects wanting different repairs, and `interacts: R5` carries neither, the field being advisory. FALSE: R4 promises repair "vocabulary exceptions (McDonald) included", but `str(parse('Juan Mcdonald').capitalized())` is `'Juan Mcdonald'` — the gate refuses before any vocabulary is consulted, and only `str(parse('Juan Mcdonald').capitalized(force=True))`, `'Juan McDonald'`, reaches the exception. R4 needs its promise scoped to names the gate admits. AMBIGUOUS, not false: R4's "an already-correct name comes back unchanged" means correct by the repair's own conventions, i.e. idempotence, and under that meaning it is true; a reader hears correct as the bearer writes it, and under THAT meaning `str(parse('bell hooks').capitalized())` — `'Bell Hooks'` — looks like a counterexample. It is not one, because `bell hooks` is not already-correct in R4's sense. What R4 owes is a disambiguation of "correct", NOT a narrowing to spare deliberately single-cased names: that would be new behavior, and R5's own rationale declines it on the ground that single case leaves the repair nothing to read. Also for that commit, and inert as things stand: R4's boundary row `"Juan McDonald" → capitalized="Juan McDonald"` passes with R5's gate deleted, exactly like the R5 row that was withdrawn above; rewriting it to `capitalized_forced=` makes it discriminate for R4's own subject but still witnesses nothing about the already-correct question. This commit adds R5 and touches R4 only on its pointer line, leaving both defects as found rather than half-fixed by a commit whose subject is something else. @@ -836,7 +850,7 @@ Declined: a test pinning the agent's axis headings against AGENTS.md's list, whi Declined: #473's "0 of 216 → 216 of 216" framing — but not re-argued here, because decisions.md#P6 already records that number's provenance (a script slicing `sorted(particles)[:14]`, which excluded exactly do, mc and vd), already names it axis 2 in its purest form, and already closes "Recompute over the WHOLE vocabulary, never a sample." Cite that entry for why the figure cannot be trusted; a second account of it in this file is the duplication decisions.md#review-agent-single-source was written to stop. #473's companion "72 of 216 — the ambiguous-particle half" measures a DIFFERENT partition of the vocabulary from the one above and must not be read as agreeing with it: the ambiguous particles are the 37 that are not never-given, and decisions.md#P6 records that #466 failed because a never-given test at the new site would have excluded all 37, `van` among them — not because of the three particle-and-suffix words. Two partitions, two different failures. -Open: #469 — whether the invariant gets an executable form alongside the family-first input-shape documentation. +Resolved: #469's invariant landed as `tests/v2/test_order_correspondence.py` — a generated parametrized sweep over a small deliberately-forked vocabulary (never-given/ambiguous/multi-token particles, an ASCII and a non-ASCII family, doubled middles), plus a second, non-parametrized test sweeping the entire shipped particle vocabulary (`nameparser.config.particles.PARTICLES`) in the minimal pair and asserting the disagreeing set is exactly `{do, mc, vd}` — the negative control this entry's Recompute bullet describes, now executable rather than a recipe to rerun by hand. ### 3-0-reevaluations — decisions shaped by the v1 shim diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index decf36f2..132e633d 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -77,7 +77,7 @@ Problem shape. A guard needs to know what the answer WAS, so it can detect the a ## LEDGER-RULE-SEPARATION — file order decides, fields narrow by subset -Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest: the narrower rule must be written first. `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 941 of 1090 corpus names. What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (measured on all three): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). Whether that pair should be separated by a predicate instead of by order is +Problem shape. Two differential-ledger rules claim overlapping names. Contract statement. Every ledger rule must carry a `name_regex` — since #451 `validate_rules` REJECTS a rule with `fields` and no `name_regex`, and one with neither was already rejected — so every rule sits in ONE tier, the sort is stable, and FILE ORDER decides every contest: the narrower rule must be written first. `fields` narrows a rule by subset; it does not separate rules by sorting. Narrowing by subset is not the whole contract: since #452 a rule's `fields` must EQUAL the union of the diffs it explains, and `compare.py` reports OVER-DECLARED and exits non-zero otherwise — a declared role no diff moves is not inert, it lets the rule keep claiming a name whose diff SHRINKS into the excess (decisions.md#differential-ledger). Since #468 there is a THIRD narrowing key: `orders` admits only the comparison orders it lists, the key being optional and its absence the order-blind reading every earlier rule has — a name compared under two orders can move the same roles for opposite reasons, so a rule describing an order-scoped fold would otherwise absorb that fold leaking into the default order (decisions.md#differential-ledger carries the worked case, and the legal set is borrowed from tools/differential/shapes.py rather than copied — plus one member no shape can declare, the `DEFAULT` sentinel naming the comparison run under no declared order, TOML having no null to put in an array). Exclusions take no `orders` and stay order-blind, deliberately. The ban ends the SHAPE and not the property it enabled: a required `name_regex` bounds nothing by itself, since the only width check is the sentinel probe — measured, `[a-z]` validates and reaches 963 of 1113 comparisons (2026-09-01). What changed is that such a rule now carries a `_CORPUS_CLAIMS` reach and digest, so its breadth is visible once at recording time rather than never (#452). The two-tier sort in `_sorted_rules` is KEPT although the ban makes it the identity on every ledger that loads (four ledgers load today, measured 2026-09-01; the open cycle's carries no rules, so the identity holds trivially there): it is the defence for a reader that does not call `validate_rules` first — a future tool, a REPL, a test fixture — and its docstring in tools/differential/compare.py says so. How it works. Detail is owned by tools/differential/README.md. The file-order clause is measured, not theoretical: in the 1.4 ledger the comma-honorific-peel rule's fields are a strict subset of the comma-compound rule's, both carry a name_regex, and a pure reorder reattributes seven names — caught by _CROSS_RULE_WINNERS and by nothing else in the suite (#375's mutation). Whether that pair should be separated by a predicate instead of by order is [#382](https://github.com/derek73/python-nameparser/issues/382). The old #271/#272 slug taboo is RETIRED (#333): the canonical-rule selector that keyed on those substrings is deliberately deleted — rule authors are free to use them in compound slugs — and the surviving rosters select on their own explicit keys (_HONORIFIC_SOURCES and _LATIN_ALTERNATION_SOURCES by named issue strings, _SPAN_BEARING_RULES by exact leading fix(...) tag). Lives in. tools/differential/compare.py, the expected_since_*.toml ledgers. Reach for it when. A ledger rule's behavior seems to depend on where it sits in the file — it does, and the reorder mutation is the test (run twice in #375; it fails _CROSS_RULE_WINNERS). History: #372 (closed) measured the then-existing fields-only rule owning 1639 of 5257 name×field pairs as filed (2026-08-10); #375/#376 then cut its classifier-of-record share sharply, and the residual pair ownership was read as the last-resort tier working as designed rather than a defect — until #451 retired the shape outright (decisions.md#differential-ledger). #372's two proposed mechanical checks were DECLINED with measurements (see decisions.md#differential-ledger), not left open. diff --git a/docs/design/rules.md b/docs/design/rules.md index 6b45b16e..2ebfbe46 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -496,6 +496,13 @@ P6. Rationale: a particle ending the name has nothing to link (vd, mc), which the join declines and the attachment then takes. "Berg, abdul van" → given="abdul van" "Berg, abdul vd" → family="vd Berg" + The comma site above and the no-comma family-first site earlier + in this rule are checked against each other directly, as a pair, + rather than only against their own examples: tests/v2/test_order_correspondence.py + generates both writings and asserts they parse alike, plus a + negative-control sweep pinning the disagreeing set the precedence + bullet above names. A change that breaks one side of that pair + should expect that test, not this file, to say so first. history: decisions.md#P6 · interacts: A1, C1, P1, S2, P5 · implemented: nameparser/_pipeline/_post_rules.py ## Suffixes: generational & credentials (S) diff --git a/docs/release_log.rst b/docs/release_log.rst index 75fbf50e..18625eb6 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -80,7 +80,7 @@ Release Log - **Add abd to BOUND_GIVEN_NAMES, so the spellings that write the article as its own word join like the others do:** ``"abd Allah Smith"`` was given ``abd``, middle ``Allah`` and is now given ``abd Allah``. ``abdul``, ``abdel`` and ``abdal`` were already there, and the Arabic-script ``عبد`` has covered the same word since 2.0, so only the Latin spelling was short. The word is also the postnominal ABD ("All But Dissertation") and stays in ``SUFFIX_ACRONYMS``: position tells the two readings apart, so ``"Jane Smith ABD"``, ``"Jane Smith, ABD"`` and ``"Jane Smith A.B.D."`` all still read the credential as a suffix (#400) - - **Change how far a leading never-given particle takes the surname when a family-first name_order is declared.** ``Policy(name_order=FAMILY_FIRST)`` read ``"de Mesnil Jean"`` as family ``de Mesnil Jean`` -- the whole name -- and now reads family ``de Mesnil``, given ``Juan``. The default order is unchanged, deliberately: with no order declared nothing marks where the surname ends, and a particle followed by several words really can be all surname (``von Bergen Wessels``); a caller who means family ``de la Vega`` plus given ``Juan`` there writes the comma. The stop cannot land inside a conjunction-joined run or a bound given-name pair: ``"de la Vega y Santos Juan"`` reads family ``de la Vega y Santos``, ``"ibn Awf abdul Rahman"`` given ``abdul Rahman``. Where two or more words are left over, the two family-first orders differ from each other for the first time: ``"de la Cruz Juan Carlos"`` reads given ``Juan``, middle ``Carlos`` under ``FAMILY_FIRST`` and the reverse under ``FAMILY_FIRST_GIVEN_LAST``. See :doc:`customize`, and the ``P1`` entry of ``docs/design/decisions.md`` for the reasoning (closes #395) + - **Change how far a leading never-given particle takes the surname when a family-first name_order is declared.** ``Policy(name_order=FAMILY_FIRST)`` read ``"de Mesnil Jean"`` as family ``de Mesnil Jean`` -- the whole name -- and now reads family ``de Mesnil``, given ``Jean``. The default order is unchanged, deliberately: with no order declared nothing marks where the surname ends, and a particle followed by several words really can be all surname (``von Bergen Wessels``); a caller who means family ``de la Vega`` plus given ``Juan`` there writes the comma. The stop cannot land inside a conjunction-joined run or a bound given-name pair: ``"de la Vega y Santos Juan"`` reads family ``de la Vega y Santos``, ``"ibn Awf abdul Rahman"`` given ``abdul Rahman``. Where two or more words are left over, the two family-first orders differ from each other for the first time: ``"de la Cruz Juan Carlos"`` reads given ``Juan``, middle ``Carlos`` under ``FAMILY_FIRST`` and the reverse under ``FAMILY_FIRST_GIVEN_LAST``. See :doc:`customize`, and the ``P1`` entry of ``docs/design/decisions.md`` for the reasoning (closes #395) - **Change the detail text of a PARTICLE_OR_GIVEN ambiguity to name the role the leading particle was actually given.** It said "read as a given name" under every ``name_order``, which is false under ``Policy(name_order=FAMILY_FIRST)`` -- there ``"Van Johnson"`` reads family ``Van``, given ``Johnson``, and the report described the reading not taken. It now ends "read as a family name" in that case. The ``kind`` is unchanged and stays ``PARTICLE_OR_GIVEN``; only the human-readable text moved, and default-order output is identical (#355) diff --git a/docs/usage.rst b/docs/usage.rst index fc766fe0..060e85b0 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -51,10 +51,20 @@ given-then-family: >>> parse("John Doe, Jr.").family # form 3 'Doe' -For family-first input *without* a comma — common outside Europe — set -``name_order``; see :doc:`customize`. Names written in Han or Hangul, -and Japanese names written in kanji and kana, are the exception that -needs no setting at all: see `East Asian names`_ below. +Two more arrangements apply only when ``name_order`` declares +family-first input — common outside Europe; see :doc:`customize`: + +4. ``Title Family Given Middle Middle [Particle] [, Suffix]`` (``FAMILY_FIRST``) +5. ``Title Family Middle Middle Given [, Suffix]`` (``FAMILY_FIRST_GIVEN_LAST``) + +A trailing particle earns a slot in form 4 alone because it is +displaced from the family name it belongs to; form 5's trailing word +is the given name by the caller's declaration, so there is nothing +there to reinterpret. + +Names written in Han or Hangul, and Japanese names written in kanji +and kana, are the exception that needs no setting at all: see `East +Asian names`_ below. Words that attach to their neighbors -------------------------------------- diff --git a/tests/test_capitalization.py b/tests/test_capitalization.py index bdea44d7..63b456ff 100644 --- a/tests/test_capitalization.py +++ b/tests/test_capitalization.py @@ -228,7 +228,7 @@ def test_capitalize_all_particle_family_beside_a_conjunction(self) -> None: # `Jane van der Berg nee y Jones`, whose conjunction sits in the # MAIDEN name: `str(HumanName)` renders the default spec, and # that spec omits the field. Recompute by running both forms over - # the four corpus files deduped and diffing, on whichever surface + # the corpus files deduped and diffing, on whichever surface # you name. # # The mechanism is v1's initial carve-out, taken in the PARSE diff --git a/tests/v2/_differential_fixtures.py b/tests/v2/_differential_fixtures.py index 8d4e45f2..b2cc26a3 100644 --- a/tests/v2/_differential_fixtures.py +++ b/tests/v2/_differential_fixtures.py @@ -29,6 +29,17 @@ #: release -- see AGENTS.md's release step 8. _LEDGERS = sorted(_TOOLS.glob("expected_since_*.toml")) + +def _entry_name(raw: object) -> str: + """A corpus line's name, whichever format the line uses. Mirrors + compare.py's _load_entries: a line is a bare JSON string or an + object carrying "name".""" + if isinstance(raw, str): + return raw + assert isinstance(raw, dict) and isinstance(raw.get("name"), str), raw + return raw["name"] + + #: Every name the harness classifies, deduplicated. The ledgers exist #: to explain diffs on THESE strings and no others, so "what does this #: rule claim?" is answerable here without parsing anything -- a plain @@ -40,7 +51,7 @@ #: actually matters; a rule cannot widen its corpus reach and still #: answer this one the same way, however it is spelled. _CORPUS_NAMES = sorted({ - json.loads(line) + _entry_name(json.loads(line)) for path in sorted(_TOOLS.glob("corpus*.jsonl")) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()}) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index b9c9a867..8a73f280 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -34,9 +34,23 @@ from dataclasses import dataclass -from nameparser import Policy -from nameparser._policy import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, - PatronymicRule) +from nameparser import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, + Policy) +# Not in nameparser.__all__: _order_repr renders a name_order for an +# error message, and _SCRIPT_RANGES/_script_matcher build the same +# borrowed predicate build_cjk_corpus.py uses to find CJK text. +from nameparser._policy import (PatronymicRule, _SCRIPT_RANGES, _order_repr, + _script_matcher) + +#: mirrors tools/differential/shapes.py's SHAPES keys; +#: test_case_shape_ids_exist_in_the_inventory (test_ledger_guards.py) +#: holds the two equal, since this file cannot import tools/. +_SHAPE_IDS = frozenset({1, 2, 3, 4, 5}) + +#: Whether a text carries a codepoint the parser's script table +#: classifies -- built once, same idiom as build_cjk_corpus.py's +#: module-scope `_has_cjk`. +_has_cjk = _script_matcher(*_SCRIPT_RANGES) @dataclass(frozen=True) @@ -51,11 +65,67 @@ class Case: classification: str = "parity" ambiguities: tuple[str, ...] = () # expected AmbiguityKind values notes: str = "" + #: input-shape id from tools/differential/shapes.py (#468/#469). + #: Tagging a row admits its text to the differential's CONTRACT + #: corpus (see tools/differential/shapes.py, projected into + #: corpus_shapes.jsonl by build_shapes_corpus.py beside it) under + #: the shape's name_order. Optional: a row + #: exercising a policy fork rather than an input shape stays + #: untagged. + shape: int | None = None def __post_init__(self) -> None: if self.policy is not None and self.locale is not None: raise ValueError( f"{self.id}: policy and locale are mutually exclusive") + # A shape CARRIES its order (the 2026-09-01 corpus design), so + # a tag that disagrees with the row's own policy would admit a + # name to the corpus under an order the row never asserted. + if self.shape is not None: + if self.shape not in _SHAPE_IDS: + raise ValueError(f"{self.id}: unknown shape {self.shape}") + # A locale carries an order too (script_orders), but as a + # LOOKUP this table cannot see -- cases.py stays + # import-light and stores only the locale CODE. Faking + # "declared" as GIVEN_FIRST for a locale row would let a + # tag validate against an order nobody here can name. + if self.locale is not None: + raise ValueError( + f"{self.id}: a shape tag needs the row's own " + f"policy; a locale carries an order this table " + f"cannot see") + # corpus_cjk.jsonl already claims this ground: _has_cjk is + # the same predicate build_cjk_corpus.py selects with, so + # a shape tag would double-admit the text (shapes 1-5 are + # the Latin-order arrangements; CJK is deliberately absent + # from shapes.py, #469's open question). Order alone + # cannot stand in for this check -- DEFAULT_SCRIPT_ORDERS + # forces HAN/HANGUL/HIRAGANA to FAMILY_FIRST but leaves + # KATAKANA unmapped, so a pure-katakana text can carry a + # GIVEN_FIRST name_order and still be CJK ground, not a + # shape. + if _has_cjk(self.text): + raise ValueError( + f"{self.id}: shape {self.shape} cannot tag CJK " + f"text; that ground is corpus_cjk.jsonl's, and " + f"whether a family-first CJK shape exists is " + f"#469's open question") + declared = (self.policy.name_order if self.policy is not None + else GIVEN_FIRST) + wanted = {4: FAMILY_FIRST, 5: FAMILY_FIRST_GIVEN_LAST}.get( + self.shape, GIVEN_FIRST) + if declared != wanted: + if self.policy is not None: + declared_desc = ( + f"the row's policy declares {_order_repr(declared)}") + else: + declared_desc = ("this row declares no policy, so it " + "is GIVEN_FIRST") + raise ValueError( + f"{self.id}: shape {self.shape} implies name_order " + f"{_order_repr(wanted)}, but {declared_desc}; add " + f"policy=Policy(name_order={_order_repr(wanted)}) or " + f"drop the tag") _ES = Policy(patronymic_rules=frozenset({PatronymicRule.EAST_SLAVIC})) @@ -63,21 +133,24 @@ def __post_init__(self) -> None: _SD = Policy(extra_suffix_delimiters=frozenset({" - "})) CASES: tuple[Case, ...] = ( - Case("plain", "John Smith", {"given": "John", "family": "Smith"}), + Case("plain", "John Smith", {"given": "John", "family": "Smith"}, + shape=1), Case("family_comma", "Smith, John", - {"given": "John", "family": "Smith"}), + {"given": "John", "family": "Smith"}, shape=2), Case("suffix_comma", "John Smith, PhD", - {"given": "John", "family": "Smith", "suffix": "PhD"}), + {"given": "John", "family": "Smith", "suffix": "PhD"}, shape=3), Case("bound_given_pairwise_only", "Salem, Abdul Rahman Ahmed", {"given": "Abdul Rahman", "middle": "Ahmed", "family": "Salem"}, notes="the bound-given join is PAIRWISE (one merge, v1 " - "parity): the third piece stays a middle name"), + "parity): the third piece stays a middle name", + shape=2), Case("family_comma_three_part_trailing_strict", "Smith, John V, Jr.", {"given": "John", "middle": "V", "family": "Smith", "suffix": "Jr."}, notes="the lenient trailing test applies only to TWO-part " "names; a third comma part makes the trailing token a " - "middle initial (v1 parity, pinned live 2026-07-17)"), + "middle initial (v1 parity, pinned live 2026-07-17)", + shape=2), Case("triple_trailing_commas", "Doe,,,", {"family": "Doe"}, notes="one trailing comma is cosmetic; the rest are " @@ -307,7 +380,8 @@ def __post_init__(self) -> None: "stops at 'Cruz' because the declared order says what " "follows the family is not more surname. It reaches " "'Cruz' THROUGH ambiguous 'la', which is the chain a " - "stop keyed on never-given membership would break"), + "stop keyed on never-given membership would break", + shape=4), Case("leading_never_given_particle_two_leftovers_" "family_first_given_last", "de la Cruz Juan Carlos", @@ -321,7 +395,8 @@ def __post_init__(self) -> None: "PR #394 put the placing in grouping, its review found " "the whole suite passed with name_order discarded from " "it; on this branch the same mutation fails three " - "tests, this row among them"), + "tests, this row among them", + shape=5), # The Dutch alphabetized listing: "Beethoven, Ludwig van" is how # "Ludwig van Beethoven" is filed, the tussenvoegsel moved behind # the given name but belonging to the surname (#379). @@ -336,7 +411,8 @@ def __post_init__(self) -> None: "'Beethoven', which is what #130 asked for. The " "textbook-correct Dutch listing reports the fork all " "the same (#405): the parser cannot tell it from " - "'Nguyen, Thi Van', which is the same string shape"), + "'Nguyen, Thi Van', which is the same string shape", + shape=2), Case("tussenvoegsel_multiword", "Berg, Jan van der", {"given": "Jan", "family": "van der Berg"}, classification="fix(#379)", @@ -1058,7 +1134,8 @@ def __post_init__(self) -> None: "leftover given slot, so this read given 'née' / middle " "'Vega'. Consumed and dropped, it never reaches the " "placement. No given name at all is the right answer " - "for family-plus-maiden input"), + "for family-plus-maiden input", + shape=4), Case("maiden_marker_leaves_family_all_particles", "Jane de la née Jones", {"given": "Jane", "family": "de la", "maiden": "Jones"}, @@ -1156,7 +1233,8 @@ def __post_init__(self) -> None: "leaves no leftover to distribute, so the reading that " "distinguishes them has nothing to work on. Before " "#399 they differed -- given 'Vega' middle 'née' here " - "against given 'née' middle 'Vega' under FAMILY_FIRST"), + "against given 'née' middle 'Vega' under FAMILY_FIRST", + shape=5), Case("connective_join_never_reaches_a_taken_marker", "Jane van der Berg née y Jones", {"given": "Jane", "family": "van der Berg", @@ -2528,19 +2606,42 @@ def __post_init__(self) -> None: Case("family_comma_no_name_word_family_first", "de Mesnil Jean, Dr.", {"title": "Dr.", "given": "Jean", "family": "de Mesnil"}, policy=Policy(name_order=FAMILY_FIRST), - notes="as 'de Mesnil Jean' reads under the same order; master " - "read it through the suffix-comma route ('dr' was " - "suffix vocabulary) and got the fold that way"), + classification="feat(#395)", + notes="core-only: name_order has no v1 spelling, so 'parity' " + "(this field's default) could never have been true of " + "this row. The differential compares it at 2.0.0 and " + "2.1.0 under its own order instead, where it moves: " + "#395's fold takes 'de Mesnil' where both read family " + "'de', and the #296 half turns the post-comma 'Dr.' " + "from a suffix into a title. As 'de Mesnil Jean' reads " + "under the same order; master read it through the " + "suffix-comma route ('dr' was suffix vocabulary) and " + "got the fold that way", + shape=4), Case("family_comma_no_name_word_family_first_given_last", "de la Cruz Juan Carlos, Dr.", {"title": "Dr.", "given": "Carlos", "middle": "Juan", "family": "de la Cruz"}, - policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST)), + policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST), + classification="feat(#395)", + notes="core-only: name_order has no v1 spelling. The row " + "above with two leftovers instead of one, which is the " + "only shape in which the two family-first orders can " + "disagree about the distribution -- 'Juan' is the " + "middle name here and the given name there", + shape=5), Case("family_comma_no_name_word_family_first_plain", "John Smith, Dr.", {"title": "Dr.", "given": "Smith", "family": "John"}, policy=Policy(name_order=FAMILY_FIRST), - notes="the declared order applies to the pre-comma name as it " - "does to 'John Smith' alone -- deliberate"), + classification="fix(#296)", + notes="core-only: name_order has no v1 spelling, but the " + "CHANGE here is the vocabulary half alone -- no " + "particle, so nothing folds, and the diff against 2.0.0 " + "is the {title, suffix} move 'dr' leaving the suffix " + "sets produces under every order. The declared order " + "applies to the pre-comma name as it does to 'John " + "Smith' alone -- deliberate", + shape=4), Case("title_word_trailing_is_not_a_title", "John Smith Prof.", {"given": "John", "middle": "Smith", "family": "Prof."}, notes="the pre-existing behavior the audit_dr_trailing and " diff --git a/tests/v2/conftest.py b/tests/v2/conftest.py index 0c2ab973..436d23bb 100644 --- a/tests/v2/conftest.py +++ b/tests/v2/conftest.py @@ -12,6 +12,8 @@ import json from pathlib import Path +from ._differential_fixtures import _entry_name + def differential_corpus() -> list[str]: """Every name the differential harness compares, from all of its @@ -30,7 +32,7 @@ def differential_corpus() -> list[str]: corpus_dir = Path(__file__).parents[2] / "tools" / "differential" paths = sorted(corpus_dir.glob("corpus*.jsonl")) assert paths, f"no corpus*.jsonl in {corpus_dir}" - names = [json.loads(line) + names = [_entry_name(json.loads(line)) for path in paths for line in path.read_text().splitlines() if line.strip()] return list(dict.fromkeys(names)) diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 82f61153..165fa0f7 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -1,5 +1,7 @@ """Core runner over the shared case table. The facade runner (migration plan) consumes the same CASES.""" +from typing import Any + import pytest from nameparser import Parser, Policy, Role, locales, parser_for @@ -70,3 +72,41 @@ def test_the_family_partitions_into_particles_and_base( (pn.family_particles + " " + pn.family_base).split()), ( f"{case.text!r}: family={pn.family!r} is not partitioned by " f"particles={pn.family_particles!r} + base={pn.family_base!r}") + + +#: Case.__post_init__'s shape checks, each probed for the one message it +#: alone raises. A row here is a Case that must fail to construct, not +#: one that ever joins CASES -- unlike test_case above, this exercises +#: the dataclass's own validation rather than the parser. +@pytest.mark.parametrize("kwargs, match", [ + pytest.param( + dict(text="Beethoven, Ludwig van", shape=2, locale="nl_NL"), + "needs the row's own policy", + id="shape-plus-locale-has-no-order-to-check"), + pytest.param( + dict(text="田中さん, Jr. Ph. D.", shape=1), + "cannot tag CJK text", + id="cjk-text-is-corpus-cjk-jsonl-ground-not-a-shape"), + pytest.param( + dict(text="John Smith", shape=4), + "declares no policy", + id="family-first-shape-needs-a-family-first-policy"), + pytest.param( + dict(text="John Smith", shape=4, + policy=Policy(name_order=FAMILY_FIRST_GIVEN_LAST)), + "the row's policy declares FAMILY_FIRST_GIVEN_LAST", + id="shape-order-disagrees-with-the-rows-own-policy"), + pytest.param( + dict(text="John Smith", shape=6), + "unknown shape", + id="shape-id-outside-the-inventory"), + pytest.param( + dict(text="田中太郎", shape=4, + policy=Policy(name_order=FAMILY_FIRST)), + "cannot tag CJK text", + id="cjk-refusal-survives-a-matching-order"), +]) +def test_case_construction_rejects_a_bad_shape_tag( + kwargs: dict[str, Any], match: str) -> None: + with pytest.raises(ValueError, match=match): + Case(id="probe", expect={}, **kwargs) diff --git a/tests/v2/test_differential.py b/tests/v2/test_differential.py index c8583b54..b7e92079 100644 --- a/tests/v2/test_differential.py +++ b/tests/v2/test_differential.py @@ -14,9 +14,11 @@ import pytest -from ._differential_fixtures import _TOOLS, load_tool +from ._differential_fixtures import ( + _LEDGERS, _TOOLS, _claimed, _rules, load_tool) compare = load_tool("compare") +shapes = load_tool("shapes") def test_parse_version_pads_a_short_release_to_three_parts() -> None: @@ -115,6 +117,448 @@ def test_worker_source_gates_the_v2_import_on_the_baseline() -> None: assert "WANT_V2 = True" in compare._worker_source("2.0.0", want_v2=True) +def test_every_shape_orders_resolve_and_bound_sanely() -> None: + """The inventory's two contracts: an `order` is a public constant + name on the installed tree that Policy actually accepts as a + name_order -- hasattr alone would admit "HumanName" or any other + real attribute, failing only at runtime inside the worker -- and a + shape with an order cannot claim a pre-2.0 baseline -- Policy + shipped in 2.0.0, so an earlier min_baseline would send an order + to a worker with no Policy to apply it. + + `_parse_version(shape.min_baseline)` is called for EVERY shape, + not just ordered ones, so a typo'd min on shapes 1-3 (whose order + is None and so skips the >= (2, 0, 0) check) is still caught -- + an unparsable string raises SystemExit on its own. + """ + import nameparser + from nameparser import Policy + for sid, shape in shapes.SHAPES.items(): + parsed_min = compare._parse_version(shape.min_baseline) + if shape.order is not None: + assert hasattr(nameparser, shape.order), (sid, shape.order) + Policy(name_order=getattr(nameparser, shape.order)) + assert parsed_min >= (2, 0, 0), sid + + +def test_worker_reads_entry_objects_and_applies_an_order() -> None: + """The template must parse {"name","order"} lines and build the + order's parser; rendering is checked textually the way + test_worker_source_carries_the_requested_pin does.""" + src = compare._worker_source("2.2.0", want_v2=True) + assert '"order"' in src and "getattr(nameparser, order)" in src + + +def test_entries_below_their_shapes_min_baseline_are_skipped( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A 1.4 worker must never see an order it cannot honor; the + skip is printed so a shrunken comparison is never silent.""" + import contextlib + import io + import json as _json + import sys + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": "Ménil Christophe du", "shape": 4}, + ensure_ascii=False) + "\n" + + _json.dumps("John Smith") + "\n", encoding="utf-8") + (tmp_path / "expected_since_1.4.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + sent: dict = {} + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + sent["entries"] = list(entries) + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"facade": {"title": "", "first": "John", "middle": "", + "last": "Smith", "suffix": "", "nickname": "", + "maiden": ""}}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "1.4.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert code == 0 + assert [e["name"] for e in sent["entries"]] == ["John Smith"] + out = buf.getvalue() + assert ("skipped 1 name tagged shape(s) [4]: baseline 1.4.0 predates " + "their minimum (2.0.0)") in out + assert "corpus_x.jsonl (2, 1 skipped)" in out + + +def _tree_v2_row(name: str, order: str) -> dict: + """The tree's own v2 reading of `name` under `order`, built the + same way main()'s tree side and the worker template's _v2_row both + build it. Used to fabricate a baseline row that agrees (or, with a + field mutated, disagrees) with the tree, without needing a real + baseline wheel.""" + from nameparser import Parser, Policy + import nameparser as _np + p = Parser(policy=Policy(name_order=getattr(_np, order))).parse(name) + row = {f: (getattr(p, f, "") or "") for f in compare.V2_FIELDS} + row["_ambiguities"] = sorted( + {a.kind.name for a in getattr(p, "ambiguities", ())}) + return row + + +def test_order_bearing_entry_reaches_the_worker_and_compares_on_v2_alone( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Mutating the shape resolution to always leave e["order"] = None + must not leave this suite green: a shape-4 entry must reach the + worker with order == "FAMILY_FIRST" (not None), and a baseline row + that carries no "facade" key at all -- exactly what an + order-bearing worker row looks like -- must diff against nothing + but the v2 fields, proving the branch that skips the facade + comparison for these rows actually runs rather than crashing or + silently defaulting to the facade path.""" + import contextlib + import io + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 4}, ensure_ascii=False) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + v2_row = _tree_v2_row(name, "FAMILY_FIRST") + sent: dict = {} + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + sent["entries"] = list(entries) + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"v2": v2_row}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert sent["entries"][0]["order"] == "FAMILY_FIRST" + assert code == 0 + assert "UNEXPLAINED" not in buf.getvalue() + + +@pytest.mark.parametrize("tier,header,want_code", [ + ("contract", "UNEXPLAINED", 1), + # radar tier: same order tag, but never fatal (#468) -- exit 0 + ("radar", "UNCLASSIFIED (radar)", 0), +]) +def test_order_bearing_diff_row_tags_its_order_and_hides_v2_only( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + tier: str, header: str, want_code: int) -> None: + """Both the UNEXPLAINED and UNCLASSIFIED (radar) headers must say + which order produced a diff -- a family-first regression and a + default-order regression on the same name are otherwise + indistinguishable in the report. Both draw the tag from the same + _order_tag helper, so this parametrization pins BOTH call sites: + a test covering one header alone says nothing about the other. + And "[v2 surface only]" + means "the facade was compared and agreed", which is false for an + order-bearing row: its facade was never consulted, so the tag must + not appear on either header.""" + import contextlib + import io + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 4}, ensure_ascii=False) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, tier) + monkeypatch.setattr(compare, "HERE", tmp_path) + v2_row = _tree_v2_row(name, "FAMILY_FIRST") + v2_row["family"] = v2_row["family"] + "X" # force a diff on `family` + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"v2": v2_row}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert code == want_code + out = buf.getvalue() + assert f"{header} {name!r}" in out and "[order: FAMILY_FIRST]" in out + assert "[v2 surface only]" not in out + + +def test_classified_order_bearing_diff_tags_its_order( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The `## issue` block carries the order tag too. Without it, one + string compared under two orders lists twice under the same issue + with nothing telling the two lines apart -- and the release note + written from that block would claim a default-order change the + run never made.""" + import contextlib + import io + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 4}, ensure_ascii=False) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text( + '[[change]]\nissue = "claimed"\nname_regex = "Ménil"\n' + 'fields = ["family"]\n', encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + v2_row = _tree_v2_row(name, "FAMILY_FIRST") + v2_row["family"] = v2_row["family"] + "X" # force a diff on `family` + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"v2": v2_row}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + out = buf.getvalue() + assert code == 0 + assert "## claimed (1)" in out + assert f" {name!r} [order: FAMILY_FIRST]" in out + + +def _order_bearing_run( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + ledger_body: str) -> tuple[int, str]: + """One shape-4 (FAMILY_FIRST) entry whose v2 reading disagrees on + `family`, run against `ledger_body`. The order-bearing sibling of + _run_main, which can only build default-order comparisons.""" + import contextlib + import io + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text( + _json.dumps({"name": name, "shape": 4}, ensure_ascii=False) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text( + ledger_body, encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + v2_row = _tree_v2_row(name, "FAMILY_FIRST") + v2_row["family"] = v2_row["family"] + "X" # force a diff on `family` + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"v2": v2_row}]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + return code, buf.getvalue() + + +def test_an_order_blind_rule_absorbing_an_order_bearing_diff_is_reported( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The leak this notice makes visible runs the OTHER way from the + one `orders` was added for: a legacy rule carries no `orders`, so + it claims a family-first diff its author never considered, and an + order-only regression there reports as an intentional change. The + notice is informational -- order-blind rules stay legal, and a + ledger full of them is what every baseline before shape tags + has -- so it must NOT move the exit code.""" + code, out = _order_bearing_run( + tmp_path, monkeypatch, + '[[change]]\nissue = "blind"\nname_regex = "Ménil"\n' + 'fields = ["family"]\n') + assert code == 0 + assert "ORDER-BLIND" in out + assert "'blind'" in out and "FAMILY_FIRST" in out + + +def test_one_string_tagged_with_two_shapes_is_two_comparisons( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The dedup key is (name, order), not name. Regressing it to the + name alone drops the second reading silently: the run compares one + of the two orders, prints a corpus count one smaller, and exits 0 + -- and nothing above this pins it, because every other order test + uses a single-shape corpus where the two keys agree.""" + import contextlib + import io + import json as _json + import sys + name = "Ménil Christophe du" + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text("".join( + _json.dumps({"name": name, "shape": s}, ensure_ascii=False) + "\n" + for s in (4, 5)), encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + sent: dict = {} + + def _fake(v: str, w: bool, + entries: list[dict[str, object]]) -> tuple[dict, list[dict]]: + sent["entries"] = list(entries) + return ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"v2": _tree_v2_row(name, str(e["order"]))} + for e in entries]) + + monkeypatch.setattr(compare, "_run_worker", _fake) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert code == 0 + assert [e["name"] for e in sent["entries"]] == [name, name] + assert [e["order"] for e in sent["entries"]] == [ + "FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"] + assert "corpus: 2 names" in buf.getvalue() + + +def test_every_order_scoped_rule_declines_the_default_order() -> None: + """Swept over the shipped ledgers rather than pinned per rule: a + rule whose `orders` omits "DEFAULT" is saying its diffs come from + declared orders alone, and the whole value of that statement is + that the default-order reading of the same string stays somebody + else's business -- unclaimed, and so still able to report + UNEXPLAINED. + + Asked over the names the rule's own regex reaches, so it is the + rule's real population and not a fixture's. Both directions, since + a narrowing that declined everything would pass the first half.""" + checked = 0 + for ledger in _LEDGERS: + for rule in _rules(ledger): + orders = rule.get("orders") + if not isinstance(orders, list) or "DEFAULT" in orders: + continue + fields = set(rule["fields"]) + examples = _claimed(rule["name_regex"]) + assert examples, ( + f"{ledger.name}: {rule['issue']!r} is order-scoped but its " + f"regex reaches no corpus name, so this sweep would check " + f"it vacuously") + for example in examples: + assert not compare._entry_matches( + rule, example, fields, None), ( + f"{ledger.name}: {rule['issue']!r} scopes to {orders} " + f"yet claims the DEFAULT-order diff on {example!r}") + assert any(compare._entry_matches(rule, example, fields, o) + for o in orders), ( + f"{ledger.name}: {rule['issue']!r} claims nothing under " + f"any order it lists, on {example!r}") + checked += 1 + assert checked >= 4, ( + f"only {checked} order-scoped rules were swept; the shipped 2.x " + f"ledgers carry four each, so this pin is passing vacuously") + + +def test_the_worker_reads_an_order_bearing_line_as_the_tree_does() -> None: + """The generated worker's order branch, RUN rather than compiled. + test_worker_source_compiles proves it parses and + test_run_worker_sends_the_name_and_resolved_order_on_the_wire + proves what goes in; between them the branch that resolves an order + constant, builds a Parser for it and emits a v2-only row was + covered by nothing that executes it. + + exec'd in-process on purpose: the template's `import nameparser` + then resolves to this checkout, so the row it emits is the tree's + own reading and can be compared against the tree's own Parser. + That is exactly the equality a real run depends on -- the worker + and main()'s tree side must build the same row from the same parse + -- and it is what makes a diff mean a behavior change rather than + a protocol one.""" + import contextlib + import io + import json as _json + import sys + import nameparser + name = "Ménil Christophe du" + source = compare._worker_source(nameparser.__version__, want_v2=True) + stdin = io.StringIO(_json.dumps( + {"name": name, "order": "FAMILY_FIRST"}, ensure_ascii=False) + "\n") + real_stdin, buf = sys.stdin, io.StringIO() + try: + sys.stdin = stdin + with contextlib.redirect_stdout(buf): + exec(compile(source, "baseline_worker.py", "exec"), {}) + finally: + sys.stdin = real_stdin + lines = buf.getvalue().splitlines() + # the version tell is the first line, always -- a reader that + # forgets it compares the tell against a parse and sees nothing + tell, row = (_json.loads(line) for line in lines) + assert tell["__version__"] == nameparser.__version__ + assert row == {"v2": _tree_v2_row(name, "FAMILY_FIRST")} + # the facade is never consulted for an order-bearing entry, so the + # key must be absent rather than empty: main() branches on it + assert "facade" not in row + + +def test_dormancy_diagnoses_a_reverted_scoped_rule_as_reverted() -> None: + """An order-scoped rule that stops explaining anything has had its + behavior reverted, and must say so even when another rule explains + the SAME NAME under the default order. Read order-blind, the scoped + rule matches that default-order diff, sees the other rule win it + and reports `shadowed` -- which sends someone to delete a rule that + is not redundant, while the family-first behavior it described + stays gone.""" + rules = [{"issue": "fix(default)", "name_regex": "^de la Cruz$", + "fields": ["family"], "orders": ["DEFAULT"]}, + {"issue": "feat(scoped)", "name_regex": "^de la Cruz$", + "fields": ["family"], "orders": ["FAMILY_FIRST"]}] + # only the default-order comparison diffs: the family-first one the + # scoped rule describes has been reverted + report = compare.dormant_rules( + rules, {"fix(default)"}, [("de la Cruz", {"family"}, None)]) + assert [d.issue for d in report.undeclared] == ["feat(scoped)"] + assert report.undeclared[0].kind == "reverted" + assert report.undeclared[0].detail == "" + + +def test_a_scoped_rule_absorbing_its_own_order_is_not_reported( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Scoping is the fix the notice asks for, so a scoped rule must + print nothing -- otherwise the block is noise on every ledger that + took the advice, and a reader stops reading it.""" + code, out = _order_bearing_run( + tmp_path, monkeypatch, + '[[change]]\nissue = "scoped"\nname_regex = "Ménil"\n' + 'fields = ["family"]\norders = ["FAMILY_FIRST"]\n') + assert code == 0 + assert "## scoped (1)" in out + assert "ORDER-BLIND" not in out + + _WHEEL = "/Users/x/.cache/uv/environments-v2/w/lib/python3.11/" \ "site-packages/nameparser/__init__.py" @@ -327,6 +771,102 @@ def test_classify_declines_a_diff_touching_a_field_the_rule_omits() -> None: assert compare.classify("x", {"given", "suffix"}, rules) is None +def test_classify_scopes_a_rule_to_the_orders_it_declares() -> None: + """`orders` is the third narrowing, and the one a name compared + twice needs: the SAME string and the SAME moved roles mean + different things under a declared family-first order (the #395 + fold, intended) and under the default order (that fold leaking + where it must not). Order-blind, one rule claims both and the + leak reports as an intentional change. + + Both directions are pinned. A rule with `orders` must decline the + default-order diff -- comparison order None is never a member, + since the members are constant NAMES -- and must still claim the + diff under each order it lists. A rule without the key stays + order-blind, which is what every rule written before shape-tagged + entries existed relies on.""" + scoped = [{"issue": "family-first only", "name_regex": "Cruz", + "fields": ["family"], + "orders": ["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"]}] + assert compare.classify("de la Cruz", {"family"}, scoped) is None + assert compare.classify("de la Cruz", {"family"}, scoped, + order="FAMILY_FIRST") == "family-first only" + assert compare.classify( + "de la Cruz", {"family"}, scoped, + order="FAMILY_FIRST_GIVEN_LAST") == "family-first only" + blind = [{"issue": "any order", "name_regex": "Cruz", + "fields": ["family"]}] + assert compare.classify("de la Cruz", {"family"}, blind) == "any order" + assert compare.classify("de la Cruz", {"family"}, blind, + order="FAMILY_FIRST") == "any order" + + +def test_the_default_order_sentinel_scopes_a_rule_to_the_default_order( + ) -> None: + """"DEFAULT" is a sentinel and not an order constant: no shape + declares it, because it names the absence of a declared order. It + exists because TOML has no null inside an array, so a rule that + explains only default-order diffs had no way to say so and had to + stay order-blind -- which is the leak running the other way from + the one `orders` was added for.""" + scoped = [{"issue": "default only", "name_regex": "Cruz", + "fields": ["family"], "orders": ["DEFAULT"]}] + assert compare.classify("de la Cruz", {"family"}, + scoped) == "default only" + assert compare.classify("de la Cruz", {"family"}, scoped, + order="FAMILY_FIRST") is None + both = [{"issue": "default and one order", "name_regex": "Cruz", + "fields": ["family"], "orders": ["DEFAULT", "FAMILY_FIRST"]}] + assert compare.classify("de la Cruz", {"family"}, + both) == "default and one order" + assert compare.classify("de la Cruz", {"family"}, both, + order="FAMILY_FIRST") == "default and one order" + assert compare.classify("de la Cruz", {"family"}, both, + order="FAMILY_FIRST_GIVEN_LAST") is None + + +@pytest.mark.parametrize("orders,message", [ + (["NO_SUCH_ORDER"], "shapes.py declares for no shape"), + ([], "empty 'orders'"), + ("FAMILY_FIRST", "not a list of strings"), + ([1], "not a list of strings"), +]) +def test_validate_rules_rejects_a_bad_orders_narrowing( + orders: object, message: str) -> None: + """Each way an `orders` can stop meaning what its author wrote. + The two type failures are the dangerous direction -- _entry_matches + ignores a non-list, so a mistyped key silently returns the rule to + claiming every order, which is the scoping it was added to undo -- + and the other two can only ever match nothing, which is loud but + reads as a dormant rule rather than as a typo.""" + with pytest.raises(SystemExit, match=message): + compare.validate_rules( + [{"issue": "fix(x) scoped", "name_regex": "Smith", + "fields": ["given"], "orders": orders}], + "test_ledger.toml") + + +def test_validate_rules_takes_the_order_names_from_the_shape_inventory( + ) -> None: + """The legal set is BORROWED, not hand-copied: every order any + shape declares is legal and nothing else is, so an order added to + shapes.py is usable in a rule the same day, and one removed stops + validating without anyone remembering a second list. An order no + shape declares is an order no comparison runs under, so a rule + naming it could only ever be dormant. + + "DEFAULT" is the one member shapes.py does not supply, and cannot: + it names the absence of a declared order, which is not a shape.""" + assert compare._legal_orders() == { + shape.order for shape in shapes.SHAPES.values() + if shape.order is not None} | {"DEFAULT"} + for order in compare._legal_orders(): + compare.validate_rules( + [{"issue": "fix(x) scoped", "name_regex": "Smith", + "fields": ["given"], "orders": [order]}], + "test_ledger.toml") + + def test_v2_fields_matches_the_Role_enum() -> None: """AGENTS.md: the seven roles are 'defined once and derived everywhere'. compare.py cannot import Role into the WORKER (that @@ -475,7 +1015,8 @@ def _run_main(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ledger_body: str, extra: list[tuple[str, dict]] | None = None, baseline: str = "1.4.0", baseline_v2: dict | None = None, - floor: int | None = 1) -> tuple[int, str]: + floor: int | None = 1, + tier: str | None = "contract") -> tuple[int, str]: """Drive main() end to end with a faked baseline worker. No uv, no network. The helper exists because every unit test above @@ -506,8 +1047,8 @@ def _run_main(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ledger_body: str, rows.append({"facade": facade}) _WORKER_CALL.clear() - def _fake(v: str, w: bool, n: list[str]) -> tuple[dict, list[dict]]: - _WORKER_CALL.update(version=v, want_v2=w, names=list(n)) + def _fake(v: str, w: bool, n: list[dict]) -> tuple[dict, list[dict]]: + _WORKER_CALL.update(version=v, want_v2=w, names=[e["name"] for e in n]) return ({"__version__": v, "__file__": "/wheel/nameparser/__init__.py"}, rows) @@ -516,6 +1057,11 @@ def _fake(v: str, w: bool, n: list[str]) -> tuple[dict, list[dict]]: # a corpus arrives without one. if floor is not None: monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, floor) + # The fixture corpus needs a tier like any other. `tier=None` + # leaves it unregistered, for the test that pins the fail-closed + # roster. + if tier is not None: + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, tier) monkeypatch.setattr(compare, "HERE", tmp_path) monkeypatch.setattr(compare, "_run_worker", _fake) monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", baseline, @@ -570,6 +1116,241 @@ def test_main_exits_0_when_every_diff_is_claimed( assert "## claimed (1)" in out +def test_radar_diff_with_no_rule_exits_0_and_is_reported( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The tier split's entire point (#468): a harvested name's diff + is shown, not owed a ledger rule. The heading is pinned because it + is what a release reader greps for. + + An empty ledger, not the usual 'unrelated'/ZZZ decoy rule: that + decoy matches no diffing name in a single-name corpus and so is + itself EXPLAINED NOTHING (dormant_rules' "reverted" case) -- + orthogonal to the tier split and would fail this run for a reason + that has nothing to do with what it is pinning.""" + code, out = _run_main(tmp_path, monkeypatch, "", _DIFFERS, tier="radar") + assert code == 0 + assert "UNCLASSIFIED (radar) 'John Smith'" in out + assert "family:" in out + assert "UNEXPLAINED" not in out + + +def test_radar_diff_matching_a_rule_still_classifies( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Radar names keep feeding the release-note grouping, and a rule + explaining only radar diffs is NOT dormant -- exit 0 with no + EXPLAINED NOTHING block is the pin for both at once.""" + code, out = _run_main( + tmp_path, monkeypatch, + '[[change]]\nissue = "claimed"\nname_regex = "Smith"\nfields = ["family"]\n', + _DIFFERS, tier="radar") + assert code == 0 + assert "## claimed (1)" in out + assert "EXPLAINED NOTHING" not in out + + +def test_contract_diff_still_fails_under_the_tier_roster( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The split must not loosen the tier that keeps the old promise.""" + code, out = _run_main( + tmp_path, monkeypatch, + '[[change]]\nissue = "unrelated"\nname_regex = "ZZZ"\nfields = ["family"]\n', + _DIFFERS, tier="contract") + assert code == 1 + assert "UNEXPLAINED 'John Smith'" in out + + +def test_radar_name_refused_by_an_exclusion_still_fails( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A [[never]] entry is chosen -- someone wrote its `why` and its + `examples` -- so it belongs to the contract even when the name it + refuses sits in a radar file. classify() returns None for both + 'no rule matched' and 'an exclusion refused this', and only the + first is the tier split's business: an excluded shape must stay + UNEXPLAINED and exit 1 on every tier, matching what + validate_exclusions' docstring and every 1.4.0 `why` promise.""" + code, out = _run_main( + tmp_path, monkeypatch, + '[[never]]\nwhy = "test exclusion"\nname_regex = "Smith"\n' + 'examples = ["John Smith"]\n', + _DIFFERS, tier="radar") + assert code == 1 + assert "UNEXPLAINED 'John Smith'" in out + assert "UNCLASSIFIED" not in out + + +def test_a_corpus_without_a_tier_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Fail-closed like _CORPUS_FLOORS: a new corpus must choose.""" + with pytest.raises(SystemExit, match="_CORPUS_TIERS"): + _run_main(tmp_path, monkeypatch, "", _DIFFERS, tier=None) + + +def test_object_corpus_lines_are_read_and_labels_printed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A corpus line may be {"name": ..., "tests": [...]} -- the + label-bearing format corpus.jsonl ships in (both shapes are legal + on any corpus file). The name is compared and the labels ride into + the radar report, which is what they are for.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "tests": ["test_two_word_name"]}) + "\n", + encoding="utf-8") + (tmp_path / "expected_since_1.4.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "radar") + monkeypatch.setattr(compare, "HERE", tmp_path) + monkeypatch.setattr( + compare, "_run_worker", + lambda v, w, entries: ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"facade": _DIFFERS}])) + import contextlib + import io + import sys + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "1.4.0", + "--corpus", str(corpus)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + assert code == 0 + assert "test_two_word_name" in buf.getvalue() + + +def test_a_malformed_tests_label_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Caught at load time, not report time: a 'tests' label is read + only when printing the radar block, after the multi-minute worker + pass, so a bad one left unchecked would crash there instead -- + the failure mode validate_rules' compile-at-startup paragraph + exists to avoid.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "tests": "not_a_list"}) + "\n", + encoding="utf-8") + with pytest.raises(SystemExit, match="'tests' must be a list"): + compare._load_entries(corpus) + + +def test_a_misspelled_corpus_key_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """'shpae' is not 'shape', and an ignored key is a narrowing that + silently did not happen: the line compares under the default order + while its author believes they declared a family-first one. The + message names the FILE, like every other loader error here, because + five corpora are read in one run.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "shpae": 4}) + "\n", encoding="utf-8") + with pytest.raises(SystemExit, match=r"corpus_x\.jsonl.*'shpae'"): + compare._load_entries(corpus) + + +def test_a_corpus_line_writing_a_computed_key_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """'order' is the key the WIRE protocol documents, so it is the one + a corpus author is likeliest to write by hand -- and main() computes + it from `shape` and overwrites whatever the line said. Rejected + rather than obeyed: honoring it would let a corpus line name an + order no shape declares, which is the check `orders` rules get.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "order": "FAMILY_FIRST"}) + "\n", + encoding="utf-8") + with pytest.raises(SystemExit, match=r"corpus_x\.jsonl.*'order'"): + compare._load_entries(corpus) + + +@pytest.mark.parametrize("shape", [True, "4"]) +def test_a_malformed_shape_id_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + shape: object) -> None: + """`true` is the dangerous one: bool is an int subclass and + hash(True) == hash(1), so an unchecked {"shape": true} resolves + against shapes.py's entry 1 and the line is compared under THAT + shape's order -- a wrong comparison that reports as a passing one. + The string spelling is the honest typo beside it.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "shape": shape}) + "\n", encoding="utf-8") + with pytest.raises(SystemExit, match="'shape' must be an int"): + compare._load_entries(corpus) + + +def test_a_corpus_line_that_is_neither_shape_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """A bare number, or an object with no string 'name', carries no + name to compare. Skipping it would shrink the comparison by one + name and print a summary that reads exactly like a full run.""" + import json as _json + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps({"nmae": "John Smith"}) + "\n", + encoding="utf-8") + with pytest.raises(SystemExit, match="neither a JSON string"): + compare._load_entries(corpus) + + +def test_a_shape_id_shapes_py_does_not_define_is_a_hard_error( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """_load_entries checks the TYPE; only main() has shapes.py loaded, + so resolvability is its check. Unchecked, the entry would compare + under whatever `.get` returned rather than the order it declared.""" + import json as _json + import sys + corpus = tmp_path / "corpus_x.jsonl" + corpus.write_text(_json.dumps( + {"name": "John Smith", "shape": 9999}) + "\n", encoding="utf-8") + (tmp_path / "expected_since_2.0.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, corpus.name, 1) + monkeypatch.setitem(compare._CORPUS_TIERS, corpus.name, "contract") + monkeypatch.setattr(compare, "HERE", tmp_path) + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "2.0.0", + "--corpus", str(corpus)]) + with pytest.raises(SystemExit, match="shapes.py does not define"): + compare.main() + + +def test_cross_tier_dedup_keeps_the_contract_reading( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """'John Smith' really does sit in both corpus.jsonl (radar) and + corpus_rules.jsonl (contract). Nothing above the dedup itself + would catch a regression here, so this is the one guard that pins + contract-first loading rather than just describing it: an + unmatched diff on the shared name must still be UNEXPLAINED and + fail the run, whichever way the sort key is written.""" + import json as _json + radar_file = tmp_path / "corpus.jsonl" + contract_file = tmp_path / "corpus_rules.jsonl" + radar_file.write_text(_json.dumps("John Smith") + "\n", encoding="utf-8") + contract_file.write_text( + _json.dumps("John Smith") + "\n", encoding="utf-8") + (tmp_path / "expected_since_1.4.0.toml").write_text("", encoding="utf-8") + monkeypatch.setitem(compare._CORPUS_FLOORS, radar_file.name, 1) + monkeypatch.setitem(compare._CORPUS_FLOORS, contract_file.name, 1) + monkeypatch.setattr(compare, "HERE", tmp_path) + monkeypatch.setattr( + compare, "_run_worker", + lambda v, w, entries: ({"__version__": v, + "__file__": "/wheel/nameparser/__init__.py"}, + [{"facade": _DIFFERS}])) + import contextlib + import io + import sys + monkeypatch.setattr(sys, "argv", ["compare.py", "--baseline", "1.4.0", + "--corpus", str(radar_file), + "--corpus", str(contract_file)]) + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + code = compare.main() + out = buf.getvalue() + assert code == 1 + assert "UNEXPLAINED 'John Smith'" in out + + def test_main_validates_the_ledger_before_running_anything( tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: """validate_rules has its own tests; this pins that main CALLS it. @@ -799,7 +1580,7 @@ def test_run_worker_strips_the_import_path_overrides_from_the_child( with an unproved call site.""" monkeypatch.setenv("PYTHONPATH", "/shadow") _fake_popen(monkeypatch, f"{_TELL}\n{_ROW}\n") - compare._run_worker("1.4.0", False, ["John Smith"]) + compare._run_worker("1.4.0", False, [{"name": "John Smith"}]) env = _FakePopen.last["env"] assert "PYTHONPATH" not in env and "PYTHONHOME" not in env @@ -808,14 +1589,14 @@ def test_run_worker_aborts_on_a_nonzero_exit( monkeypatch: pytest.MonkeyPatch) -> None: _fake_popen(monkeypatch, "", rc=3) with pytest.raises(SystemExit, match="exited 3"): - compare._run_worker("1.4.0", False, ["John Smith"]) + compare._run_worker("1.4.0", False, [{"name": "John Smith"}]) def test_run_worker_aborts_on_empty_output( monkeypatch: pytest.MonkeyPatch) -> None: _fake_popen(monkeypatch, "") with pytest.raises(SystemExit, match="not even a version tell"): - compare._run_worker("1.4.0", False, ["John Smith"]) + compare._run_worker("1.4.0", False, [{"name": "John Smith"}]) def test_run_worker_aborts_when_fewer_results_than_names( @@ -823,8 +1604,9 @@ def test_run_worker_aborts_when_fewer_results_than_names( """The guard behind main's zip(), which truncates silently. This is the comparing-fewer-names-than-you-think failure.""" _fake_popen(monkeypatch, f"{_TELL}\n{_ROW}\n") - with pytest.raises(SystemExit, match="1 results for 2 corpus names"): - compare._run_worker("1.4.0", False, ["John Smith", "Jane Doe"]) + with pytest.raises(SystemExit, match="1 results for 2 corpus entries"): + compare._run_worker("1.4.0", False, + [{"name": "John Smith"}, {"name": "Jane Doe"}]) def test_run_worker_checks_the_tell_before_returning_results( @@ -833,7 +1615,36 @@ def test_run_worker_checks_the_tell_before_returning_results( '"__file__": "/wheel/nameparser/__init__.py"}') _fake_popen(monkeypatch, f"{wrong}\n{_ROW}\n") with pytest.raises(SystemExit, match="not the requested"): - compare._run_worker("1.4.0", False, ["John Smith"]) + compare._run_worker("1.4.0", False, [{"name": "John Smith"}]) + + +def test_run_worker_sends_the_name_and_resolved_order_on_the_wire( + monkeypatch: pytest.MonkeyPatch) -> None: + """The wire format is the whole contract between compare.py and the + generated worker; nothing else pins it, so a resolution bug -- e.g. + forgetting to set e["order"] before calling this -- would leave + every other test in this file green while the worker silently + received the wrong order for every name. _FakePopen.communicate + already records the payload it was given; this reads it back.""" + _fake_popen(monkeypatch, f"{_TELL}\n{_ROW}\n{_ROW}\n") + compare._run_worker( + "1.4.0", False, + [{"name": "John Smith", "order": None}, + {"name": "Ménil Christophe du", "order": "FAMILY_FIRST"}]) + lines = _FakePopen.last["stdin"].splitlines() + assert lines[0] == '{"name": "John Smith", "order": null}' + assert lines[1] == \ + '{"name": "Ménil Christophe du", "order": "FAMILY_FIRST"}' + + +@pytest.mark.parametrize("want_v2", [True, False]) +def test_worker_source_compiles(want_v2: bool) -> None: + """A syntax error in the rendered template currently surfaces only + as 'worker exited 1' after a multi-minute uv install; this catches + it at test time instead, for both renderings (WANT_V2 gates a + def-inside-if that is easy to misindent).""" + compile(compare._worker_source("2.2.0", want_v2=want_v2), + "", "exec") @pytest.mark.parametrize("rel", [ @@ -970,6 +1781,16 @@ def test_a_floor_names_a_corpus_that_exists() -> None: assert set(compare._CORPUS_FLOORS) <= on_disk +def test_every_corpus_with_a_floor_also_has_a_tier() -> None: + """The two rosters are meant to name the same files. A corpus in + one but not the other reopens the vanished-file hole the floors + were added to close: main() only checks _CORPUS_FLOORS' keys + against the files on disk (see the 'missing' check above the + loading loop), so a file present in _CORPUS_TIERS alone, or in + _CORPUS_FLOORS alone, would not be caught there.""" + assert set(compare._CORPUS_TIERS) == set(compare._CORPUS_FLOORS) + + def test_main_aborts_on_a_truncated_corpus( tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: """A corpus below its floor must stop the run, not shrink it.""" @@ -1130,7 +1951,7 @@ def test_dormant_rules_reports_a_rule_whose_behavior_vanished() -> None: rules = [{"issue": "fix(a)", "name_regex": "Smith", "fields": ["given"]}, {"issue": "fix(b)", "name_regex": "Jones", "fields": ["given"]}] report = compare.dormant_rules( - rules, {"fix(a)"}, [("John Smith", {"given"})]) + rules, {"fix(a)"}, [("John Smith", {"given"}, None)]) assert report.awake == () assert [d.issue for d in report.undeclared] == ["fix(b)"] assert report.undeclared[0].kind == "reverted" @@ -1146,7 +1967,7 @@ def test_dormant_rules_names_the_rule_that_shadows_one() -> None: {"issue": "fix(narrow)", "name_regex": "John Smith", "fields": ["given"]}] report = compare.dormant_rules( - rules, {"fix(broad)"}, [("John Smith", {"given"})]) + rules, {"fix(broad)"}, [("John Smith", {"given"}, None)]) assert [d.issue for d in report.undeclared] == ["fix(narrow)"] assert report.undeclared[0].kind == "shadowed" assert report.undeclared[0].detail == "fix(broad)" @@ -1159,7 +1980,7 @@ def test_dormant_rules_distinguishes_an_excluded_shape() -> None: rules = [{"issue": "fix(a)", "name_regex": "Smith", "fields": ["given"]}] never = [{"why": "protected", "name_regex": "Smith"}] report = compare.dormant_rules( - rules, set(), [("John Smith", {"given"})], never) + rules, set(), [("John Smith", {"given"}, None)], never) assert [d.issue for d in report.undeclared] == ["fix(a)"] assert report.undeclared[0].kind == "excluded" @@ -1177,7 +1998,7 @@ def test_dormant_rules_reports_a_declared_rule_that_woke_up() -> None: this tree checks both directions or it checks nothing.""" rules = [{"issue": "fix(a)", "fields": ["given"], "dormant": "was idle"}] report = compare.dormant_rules( - rules, {"fix(a)"}, [("John Smith", {"given"})]) + rules, {"fix(a)"}, [("John Smith", {"given"}, None)]) assert report.awake == ("fix(a)",) assert report.undeclared == () @@ -1192,8 +2013,8 @@ def test_dormant_rules_names_the_shadower_that_does_the_shadowing() -> None: "fields": ["given"]}] report = compare.dormant_rules( rules, {"fix(a)", "fix(z)"}, - [("Alpha One", {"given"}), ("Zeta One", {"given"}), - ("Zeta Two", {"given"}), ("Zeta Three", {"given"})]) + [("Alpha One", {"given"}, None), ("Zeta One", {"given"}, None), + ("Zeta Two", {"given"}, None), ("Zeta Three", {"given"}, None)]) assert [d.issue for d in report.undeclared] == ["fix(idle)"] assert report.undeclared[0].detail == "fix(z)" @@ -1212,7 +2033,7 @@ def test_dormant_rules_sorts_before_diagnosing() -> None: {"issue": "specific", "name_regex": "Smith", "fields": ["given"]}] report = compare.dormant_rules( - rules, {"specific"}, [("John Smith", {"given"})]) + rules, {"specific"}, [("John Smith", {"given"}, None)]) assert [d.issue for d in report.undeclared] == ["broad"] assert report.undeclared[0].detail == "specific" diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index 8edf2ae9..43d225a9 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -612,6 +612,46 @@ def test_cjk_corpus_matches_the_case_table() -> None: "`uv run python tools/differential/build_cjk_corpus.py`") +def test_shapes_corpus_matches_the_case_table() -> None: + """corpus_shapes.jsonl is GENERATED from the shape-tagged case + rows -- the same promise test_cjk_corpus_matches_the_case_table + makes one function up, for the tag predicate instead of the + codepoint one: a row tagged without regenerating fails HERE + instead of silently keeping the contract tier narrower than the + table says it is.""" + module = load_tool("build_shapes_corpus") + checked_in = [json.loads(line) for line in + (_TOOLS / "corpus_shapes.jsonl") + .read_text(encoding="utf-8").splitlines() if line.strip()] + assert checked_in == module.selected(), ( + "corpus_shapes.jsonl is stale: regenerate with " + "`uv run python tools/differential/build_shapes_corpus.py`") + + +def test_case_shape_ids_exist_in_the_inventory() -> None: + """cases.py cannot import tools/, so Case.__post_init__ validates + every shape tag against its own hand copy of the inventory's id + set (cases._SHAPE_IDS). This is the cross-file half, in two parts. + The equality holds the copy itself honest: a shape added to or + removed from shapes.py without updating cases._SHAPE_IDS fails + here, whichever side changed. The subset check catches a + different drift -- a row tagged under a stale _SHAPE_IDS before + this test last ran -- so it stays even though __post_init__ would + refuse the same tag today.""" + from tests.v2 import cases + shapes = load_tool("shapes") + tagged = {c.shape for c in cases.CASES if c.shape is not None} + assert tagged <= set(shapes.SHAPES), ( + f"tests/v2/cases.py tags shape id(s) " + f"{sorted(tagged - set(shapes.SHAPES))} that are not in " + f"tools/differential/shapes.py's SHAPES; add the shape there " + f"or fix the tag in cases.py") + assert cases._SHAPE_IDS == set(shapes.SHAPES), ( + f"cases._SHAPE_IDS {sorted(cases._SHAPE_IDS)} != shapes.py's " + f"{sorted(shapes.SHAPES)}; they are hand copies of one " + f"inventory -- update whichever side is stale") + + #: Names each rule MUST NOT match, keyed by a substring of its `issue`. #: #: A wall, not a change detector, and that is the point. _CORPUS_CLAIMS @@ -1345,7 +1385,8 @@ def _carries(name: str, vocabulary: frozenset[str]) -> bool: from nameparser import DEFAULT_NICKNAME_DELIMITERS as D from nameparser.config.maiden_markers import MAIDEN_MARKERS as V from nameparser._lexicon import _normalize -names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()} +def _n(x): return x if isinstance(x, str) else x['name'] +names = {_n(json.loads(l)) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()} strip = ''.join({c for p in D for c in p}) sub = lambda n: any(e in n for e in V if not e.isascii()) print(sum(not {_normalize(t.strip(strip)) for t in n.split()} & V and sub(n) for n in names), @@ -1446,17 +1487,26 @@ class _Claim(NamedTuple): #: still fails. Unreadable by design -- the count above is what a #: reviewer reads, and the failure message prints what moved. digest: str + #: the comparison orders it narrows by, sorted; None when it carries + #: no `orders` key at all. The three above are all blind to the + #: third narrowing: deleting `orders` from a shipped rule widens it + #: to every order while its regex reaches the same names, narrows by + #: the same roles, and digests identically. No default, so a new + #: entry has to state which one it is. + orders: tuple[str, ...] | None def _claim(rule: dict) -> _Claim: regex = rule.get("name_regex") names = (_claimed(regex) if isinstance(regex, str) else list(_CORPUS_NAMES)) fields = rule.get("fields") + orders = rule.get("orders") return _Claim( names=len(names), roles=tuple(sorted(fields)) if isinstance(fields, list) else (), digest=hashlib.sha256( - "\n".join(names).encode("utf-8")).hexdigest()[:12]) + "\n".join(names).encode("utf-8")).hexdigest()[:12], + orders=tuple(sorted(orders)) if isinstance(orders, list) else None) #: How many corpus names each rule's name_regex matches, per ledger. @@ -1467,9 +1517,13 @@ def _claim(rule: dict) -> _Claim: #: a rule outside whichever category the last fix had covered. The #: categories are the test's, not the ledger's. What every rule shares #: is what it claims: how much corpus its regex reaches, which roles -#: it narrows by, and WHICH names those are. Scoped to the corpus -- +#: it narrows by, which comparison ORDERS it admits, and WHICH names +#: those are. The orders entry closes a widening none of the other +#: three can see: deleting `orders` from a shipped rule returns it to +#: claiming every order while its reach, its roles and its digest all +#: stand still. Scoped to the corpus -- #: and only there -- a widening cannot change what a rule explains -#: without moving one of the three. Restoring 'born' moves none of +#: without moving one of the four. Restoring 'born' moves none of #: them, because no corpus name contains it; the member guards catch #: that, which is why this does not replace them. #: @@ -1507,140 +1561,140 @@ def _claim(rule: dict) -> _Claim: _CORPUS_CLAIMS: dict[str, dict[str, _Claim]] = { "expected_since_1.4.0.toml": { "fix(A2) content-free input names nobody, so every role empties": - _Claim(5, ('given',), "1af8d718688b"), + _Claim(5, ('given',), "1af8d718688b", None), "fix(#335) a marker-led clause leaves the one name word its bare reading": - _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b"), + _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b", None), "fix(#434) a multi-word maiden marker takes the maiden name": - _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef", None), "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": - _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283", None), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": - _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), + _Claim(5, ('maiden', 'nickname'), "a419f74143e3", None), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": - _Claim(3, ('family', 'given'), "24d6223e472f"), + _Claim(3, ('family', 'given'), "24d6223e472f", None), "fix(#410) the maiden flavor, where 1.4.0 read the marker as a middle name": - _Claim(1, ('family', 'given', 'maiden', 'middle'), "309e39fc2475"), + _Claim(1, ('family', 'given', 'maiden', 'middle'), "309e39fc2475", None), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": - _Claim(1, ('middle', 'suffix'), "e9f282da0d0f"), + _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(108, ('family', 'given', 'middle'), "9a814f70c2dc"), + _Claim(108, ('family', 'given', 'middle'), "9a814f70c2dc", None), "fix(#274) maiden markers consumed": - _Claim(31, ('family', 'maiden', 'middle'), "67e2280be79d"), + _Claim(32, ('family', 'maiden', 'middle'), "06d199ceb249", None), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": - _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8"), + _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8", None), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(13, ('family', 'middle'), "973617235cda"), + _Claim(13, ('family', 'middle'), "973617235cda", None), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(2, ('family', 'suffix'), "ec0d45289dc1"), + _Claim(2, ('family', 'suffix'), "ec0d45289dc1", None), # 279 -> 280 with #371, and the growth is corpus, not behavior: # that PR added `Ph. D., John` as a rules.md example, so the # regex matches one more corpus name. The name does not diff at # this baseline (v1 and HEAD both read first 'John', last # 'Ph. D.'), so nothing was absorbed. "fix(comma-family) lone post-comma piece routes to suffix/title, not first": - _Claim(280, ('given', 'suffix', 'title'), "c85e12fa5d66"), + _Claim(284, ('given', 'suffix', 'title'), "8b046fbdb1a1", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": - _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": - _Claim(2, ('family', 'given', 'suffix'), "a3cfff4e78f4"), + _Claim(2, ('family', 'given', 'suffix'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('given', 'middle', 'title'), "263d5957cfc1"), + _Claim(3, ('given', 'middle', 'title'), "263d5957cfc1", None), "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), "fix(#296) a credential-only comma string reads a name and its postnominal": - _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee"), + _Claim(2, ('family', 'given', 'suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444"), + _Claim(18, ('family', 'given', 'suffix', 'title'), "1f79efa10444", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": - _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": - _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + _Claim(1, ('suffix', 'title'), "f025c5f70a4e", None), "fix(#367) an inferred title no longer displaces a leading particle either": - _Claim(1, ('family', 'given'), "d8ee9cd5da5f"), + _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(comma-precomma-family) pre-comma run reads as family, not given": - _Claim(280, ('family', 'given'), "c85e12fa5d66"), + _Claim(284, ('family', 'given'), "8b046fbdb1a1", None), "fix(#342) NOT WANTED: a bare trailing 'Rai' is read as a post-nominal suffix and the family is lost": - _Claim(1, ('family', 'suffix'), "694fd06a2e9a"), + _Claim(1, ('family', 'suffix'), "694fd06a2e9a", None), "fix(#397) NOT WANTED: a trailing Catalan/Polish linking 'i' is read as a generation marker and the family is lost": - _Claim(1, ('family', 'suffix'), "498602f3cfd0"), + _Claim(1, ('family', 'suffix'), "498602f3cfd0", None), "fix(suffix-delimiter-rendering) no-space delimiter core token kept whole": - _Claim(0, ('suffix',), "e3b0c44298fc"), + _Claim(0, ('suffix',), "e3b0c44298fc", None), "ambiguous-surname-acronym data change: parenthesized (MA)/(DO) now stays nickname": - _Claim(0, ('nickname', 'suffix'), "e3b0c44298fc"), + _Claim(0, ('nickname', 'suffix'), "e3b0c44298fc", None), "feat(#269) Arabic بن prefix chains onto family (non-Latin new-recognition)": - _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d"), + _Claim(2, ('family', 'middle'), "3e2b5c6d1f4d", None), "feat(#273) typographic nickname delimiters recognized by default": - _Claim(8, ('middle', 'nickname'), "968bd4162257"), + _Claim(8, ('middle', 'nickname'), "968bd4162257", None), "fix(cjk-delimited-nickname) delimiter recognition compounds with the CJK order flip": - _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608"), + _Claim(6, ('family', 'given', 'nickname'), "ae1dffa01608", None), "fix(cjk-fullwidth-paren-nickname) fullwidth-parenthesis recognition compounds with the CJK order flip": - _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7"), + _Claim(1, ('family', 'given', 'middle', 'nickname'), "cf370e856ae7", None), "fix(cjk-comma-honorific-peel) glued honorific peels off a post-comma given name": - _Claim(23, ('given', 'suffix'), "344de804e2c6"), + _Claim(23, ('given', 'suffix'), "344de804e2c6", None), "fix(cjk-comma-compound) comma routing compounds with the CJK order flip": - _Claim(23, ('family', 'given', 'suffix', 'title'), "344de804e2c6"), + _Claim(23, ('family', 'given', 'suffix', 'title'), "344de804e2c6", None), "fix(cjk-glued-honorific-peel) glued honorific peels into suffix": - _Claim(37, ('family', 'given', 'suffix'), "719c31233502"), + _Claim(37, ('family', 'given', 'suffix'), "719c31233502", None), "fix(cjk-honorific-suffix) postnominal honorifics recognized, compounding with the CJK order flip": - _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745"), + _Claim(19, ('family', 'given', 'middle', 'suffix'), "aa475ddd4745", None), "feat(#269) non-Latin titles/conjunctions recognized": - _Claim(4, ('given', 'middle', 'title'), "e86eeb13eeb2"), + _Claim(4, ('given', 'middle', 'title'), "e86eeb13eeb2", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": - _Claim(1, ('family', 'given'), "ca7b37af6cf8"), + _Claim(1, ('family', 'given'), "ca7b37af6cf8", None), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(3, ('family', 'given'), "724967a4a117"), + _Claim(3, ('family', 'given'), "724967a4a117", None), "fix(#400) abd joins the word after it as one given name": - _Claim(11, ('given', 'middle'), "1eaed91fc574"), + _Claim(11, ('given', 'middle'), "1eaed91fc574", None), "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": - _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122", None), "fix(emoji-boundary) an emoji inside a token divides it": - _Claim(1, ('family', 'given'), "efa60ca42d4a"), + _Claim(1, ('family', 'given'), "efa60ca42d4a", None), "fix(nickname-typographic-pairs) two typographic quote spans read as one nickname set": - _Claim(1, ('family', 'given', 'middle', 'nickname'), "3cf566c78800"), + _Claim(1, ('family', 'given', 'middle', 'nickname'), "3cf566c78800", None), "fix(#411) the bound-given reserve stops counting words the maiden name takes": - _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613"), + _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613", None), "fix(#400/#274) bound-given join and maiden consumption in one name": - _Claim(1, ('family', 'given', 'maiden', 'middle'), "6bed6d349342"), + _Claim(1, ('family', 'given', 'maiden', 'middle'), "6bed6d349342", None), "fix(#411/S2) a declining bound-given join leaves the suffix reading after a family comma": - _Claim(1, ('given', 'maiden', 'middle', 'suffix'), "0f8ed9db0a32"), + _Claim(1, ('given', 'maiden', 'middle', 'suffix'), "0f8ed9db0a32", None), "fix(#418) the connective carve-out counts the name the maiden clause leaves behind": - _Claim(1, ('family', 'given', 'maiden', 'middle'), "7923e6d3c5a7"), + _Claim(1, ('family', 'given', 'maiden', 'middle'), "7923e6d3c5a7", None), "fix(credential-pair-order) a split credential and a suffix render in written order": - _Claim(1, ('suffix',), "6f6eef764248"), + _Claim(1, ('suffix',), "6f6eef764248", None), "fix(#369) a given-name title licenses the bound given-name join with one word to spare": - _Claim(3, ('family', 'given'), "724be3e6b926"), + _Claim(3, ('family', 'given'), "724be3e6b926", None), "fix(#401) the bound-given reserve counts the trailing numeral assign reads as the suffix": - _Claim(2, ('family', 'given', 'suffix'), "5713d4c0bd68"), + _Claim(2, ('family', 'given', 'suffix'), "5713d4c0bd68", None), "fix(#421) the bound-given join never absorbs a suffix piece": - _Claim(1, ('given', 'middle'), "9523e518e6ec"), + _Claim(1, ('given', 'middle'), "9523e518e6ec", None), "fix(#421) the bound-given join never absorbs a split credential": - _Claim(1, ('given', 'middle'), "228abe0f32ef"), + _Claim(1, ('given', 'middle'), "228abe0f32ef", None), "fix(#425) accepted: a bare ambiguous acronym the peel does not take joins as a name word": - _Claim(1, ('given', 'middle'), "2010cc79a34d"), + _Claim(1, ('given', 'middle'), "2010cc79a34d", None), "fix(#424) the particle chain stops before the trailing numeral": - _Claim(1, ('family', 'suffix'), "2c99162bc9cf"), + _Claim(1, ('family', 'suffix'), "2c99162bc9cf", None), "fix(#424/#445) accepted: the maiden walk keeps a bare acronym, and the lone name word is the family": - _Claim(1, ('family', 'given', 'maiden', 'middle', 'suffix'), "f2c6cd2e3001"), + _Claim(1, ('family', 'given', 'maiden', 'middle', 'suffix'), "f2c6cd2e3001", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": - _Claim(1, ('family', 'given'), "42b69cf1b320"), + _Claim(1, ('family', 'given'), "42b69cf1b320", None), "fix(#424) accepted: the maiden walk keeps the numeral an initial before the marker vetoes": - _Claim(1, ('family', 'maiden', 'middle', 'suffix'), "08c0158c8d3f"), + _Claim(1, ('family', 'maiden', 'middle', 'suffix'), "08c0158c8d3f", None), "fix(#424) accepted: the chain keeps an acronym assign will not peel behind a title-and-particle word": - _Claim(1, ('family', 'given'), "faa4bedda537"), + _Claim(1, ('family', 'given'), "faa4bedda537", None), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('family', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('family', 'suffix'), "5b3a743f9e35", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": - _Claim(1, ('family', 'middle', 'suffix'), "a564b97f7162"), + _Claim(1, ('family', 'middle', 'suffix'), "a564b97f7162", None), "fix(#360) ste moved into the never-given particles with mc": - _Claim(1, ('family', 'given'), "e62caedec864"), + _Claim(1, ('family', 'given'), "e62caedec864", None), "fix(#360) mc moved into the never-given particles, so it folds into the family": - _Claim(1, ('family', 'given'), "ee4339908f4d"), + _Claim(1, ('family', 'given'), "ee4339908f4d", None), "fix(#367) a title no longer displaces a leading never-given particle": - _Claim(1, ('family', 'given'), "db724fb9c779"), + _Claim(1, ('family', 'given'), "db724fb9c779", None), "fix(#445) a maiden marker makes the lone name word the family": - _Claim(7, ('family', 'given', 'maiden', 'middle'), "3de9ef12b4a8"), + _Claim(7, ('family', 'given', 'maiden', 'middle'), "3de9ef12b4a8", None), "fix(N3) a nickname-led name with a trailing suffix keeps the suffix in `suffix`": - _Claim(1, ('family', 'suffix'), "570f265a2f46"), + _Claim(1, ('family', 'suffix'), "570f265a2f46", None), # #451's four replacements for the fields-only catch-all, whose # own entry recorded the whole 1090-name corpus -- a rule with no # name_regex reaches everything, so its reach could never move and @@ -1649,240 +1703,256 @@ def _claim(rule: dict) -> _Claim: # take the surplus, which is why the four are last in the file; # the acronym and M.A. rules reach exactly what they explain. "fix(suffix-routing) a two-token name ending in a roman numeral keeps it in `suffix`": - _Claim(4, ('family', 'suffix'), "fc52089dfa8e"), + _Claim(4, ('family', 'suffix'), "fc52089dfa8e", None), "fix(suffix-routing) a two-token name ending in the suffix word jr keeps it in `suffix`": - _Claim(5, ('family', 'suffix'), "602e2d83a23b"), + _Claim(5, ('family', 'suffix'), "602e2d83a23b", None), "fix(suffix-routing) a two-token name ending in a credential acronym keeps it in `suffix`": - _Claim(2, ('family', 'suffix'), "ed72c9672214"), + _Claim(2, ('family', 'suffix'), "ed72c9672214", None), "fix(suffix-routing) the dotted M.A. spelling reads as a credential (ma-do)": - _Claim(1, ('family', 'suffix'), "17379620526b"), + _Claim(1, ('family', 'suffix'), "17379620526b", None), }, "expected_since_2.0.0.toml": { "fix(#335) a marker-led clause leaves the one name word its bare reading": - _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b"), + _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b", None), "fix(#434) a multi-word maiden marker takes the maiden name": - _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef", None), "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": - _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283", None), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": - _Claim(5, ('maiden', 'nickname'), "a419f74143e3"), + _Claim(5, ('maiden', 'nickname'), "a419f74143e3", None), "fix(#335) a marker-led bracketed clause reads as the maiden name, compounding with the CJK order flip": - _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7"), + _Claim(1, ('family', 'given', 'maiden', 'nickname'), "cf370e856ae7", None), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": - _Claim(4, ('family', 'given'), "da1dd1473145"), + _Claim(4, ('family', 'given'), "da1dd1473145", None), "fix(#430) a credential run does not end at the roman numeral describing it": - _Claim(2, ('given', 'suffix'), "3c8fa6bc827a"), + _Claim(2, ('given', 'suffix'), "3c8fa6bc827a", None), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": - _Claim(1, ('middle', 'suffix'), "e9f282da0d0f"), + _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), "fix(#429) a wholly-credential segment after a one-word family renders as one entry": - _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), + _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe", None), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda"), + _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda", None), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": - _Claim(108, ('_ambiguities', 'family', 'given', 'middle'), "9a814f70c2dc"), + _Claim(108, ('_ambiguities', 'family', 'given', 'middle'), "9a814f70c2dc", None), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": - _Claim(37, ('family', 'given', 'suffix'), "719c31233502"), + _Claim(37, ('family', 'given', 'suffix'), "719c31233502", None), "fix(#307/#308/#320) spaced CJK postnominal honorific routed to suffix": - _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2"), + _Claim(16, ('family', 'given', 'middle', 'suffix'), "6d390e518bd2", None), "fix(#309) 旧姓 maiden marker consumed, compounding with the CJK order flip": - _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8"), + _Claim(5, ('family', 'given', 'maiden', 'middle'), "bc0e10dd7ec8", None), "fix(#272) nakaguro inside delimited content renders as a space, compounding with the CJK order flip": - _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23"), + _Claim(1, ('family', 'given', 'nickname'), "d4069d459f23", None), "fix(#298) 间隔号 division changes the comma reading, sending the credential from title to suffix": - _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb"), + _Claim(1, ('family', 'given', 'suffix', 'title'), "1d45596e6fdb", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": - _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), + _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8", None), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(3, ('family', 'given'), "724967a4a117"), + _Claim(3, ('family', 'given'), "724967a4a117", None), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1"), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1", None), "fix(#399) a maiden marker bounds the particle chain that swallowed it": - _Claim(5, ('family', 'maiden'), "15ec75a89f07"), + _Claim(6, ('family', 'maiden'), "89e1f4afdd2a", ('DEFAULT',)), "fix(#360) mc moved into the never-given particles, so it folds into the family": - _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d", None), "fix(#400) abd joins the word after it as one given name": - _Claim(11, ('given', 'middle'), "1eaed91fc574"), + _Claim(11, ('given', 'middle'), "1eaed91fc574", None), "fix(#367) a title no longer displaces a leading never-given particle": - _Claim(1, ('family', 'given'), "db724fb9c779"), + _Claim(1, ('family', 'given'), "db724fb9c779", None), "fix(#272/#308) nakaguro division and a glued hangul honorific in one name": - _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122"), + _Claim(1, ('family', 'given', 'middle', 'suffix'), "2fbf1a94f122", None), "fix(#411) the bound-given reserve stops counting words the maiden name takes": - _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613"), + _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613", None), "fix(#412) a connective join no longer absorbs the maiden marker beside it": - _Claim(2, ('family', 'maiden'), "51c0eb36b5c5"), + _Claim(2, ('family', 'maiden'), "51c0eb36b5c5", None), "fix(#418) the connective carve-out counts the name the maiden clause leaves behind": - _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7"), + _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7", None), "fix(#418) accepted: a suffix word inside the maiden name ends it, connective or not": - _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a"), + _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a", None), "fix(#369) a given-name title licenses the bound given-name join with one word to spare": - _Claim(3, ('family', 'given'), "724be3e6b926"), + _Claim(3, ('family', 'given'), "724be3e6b926", None), "fix(#401) the bound-given reserve counts the trailing numeral assign reads as the suffix": - _Claim(2, ('family', 'given'), "5713d4c0bd68"), + _Claim(2, ('family', 'given'), "5713d4c0bd68", None), "fix(#421) the bound-given join never absorbs a suffix piece": - _Claim(1, ('given', 'middle'), "9523e518e6ec"), + _Claim(1, ('given', 'middle'), "9523e518e6ec", None), "fix(#421) the bound-given join never absorbs a split credential": - _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), + _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef", None), "fix(#425) the bound-given reserve runs assign's peel over the joined view": - _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), + _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e", None), "fix(#424) the particle chain stops before the trailing numeral": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf", None), "fix(#424) the particle chain stops before a bare acronym with words to spare": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b", None), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": - _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": - _Claim(2, ('family', 'given'), "a3cfff4e78f4"), + _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1"), + _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), "fix(#296) a credential-only comma string reads a name and its postnominal": - _Claim(2, ('suffix', 'title'), "3f983ff71dee"), + _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('suffix', 'title'), "1f79efa10444"), + _Claim(18, ('suffix', 'title'), "1f79efa10444", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": - _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": - _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + _Claim(1, ('suffix', 'title'), "f025c5f70a4e", None), "fix(#296) a glued honorific before a lone credential: the credential is the postnominal": - _Claim(1, ('family', 'suffix', 'title'), "01bf2bd3f895"), + _Claim(1, ('family', 'suffix', 'title'), "01bf2bd3f895", None), "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title": - _Claim(1, ('family', 'given'), "faa2c70fc49e"), + _Claim(1, ('family', 'given'), "faa2c70fc49e", None), "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": - _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65"), + _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65", ('DEFAULT', 'FAMILY_FIRST')), "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4", None), "fix(#367) an inferred title no longer displaces a leading particle either": - _Claim(1, ('family', 'given'), "d8ee9cd5da5f"), + _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": - _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), + _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": - _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320"), + _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320", None), "fix(#424/#445) the maiden walk stops before the trailing numeral, and the lone name word is the family": - _Claim(1, ('_ambiguities', 'family', 'given', 'maiden', 'suffix'), "cbe5bdd97317"), + _Claim(1, ('_ambiguities', 'family', 'given', 'maiden', 'suffix'), "cbe5bdd97317", None), "fix(#445) a maiden marker makes the lone name word the family": - _Claim(6, ('family', 'given'), "f521c94c79fc"), + _Claim(6, ('family', 'given'), "f521c94c79fc", None), "fix(#445) the lone name word beside a marker a connective join no longer absorbs": - _Claim(1, ('family', 'given', 'maiden', 'middle'), "52544a41dd62"), + _Claim(1, ('family', 'given', 'maiden', 'middle'), "52544a41dd62", None), "fix(#424) a marker followed only by the numeral is just a word": - _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071"), + _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071", None), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": - _Claim(1, ('_ambiguities',), "81cf02ffdb33"), + _Claim(1, ('_ambiguities',), "81cf02ffdb33", None), "fix(#360) ste moved into the never-given particles with mc": - _Claim(1, ('_ambiguities', 'family', 'given'), "e62caedec864"), + _Claim(1, ('_ambiguities', 'family', 'given'), "e62caedec864", None), "fix(#399) a maiden marker bounds the particle chain: the geb. spelling": - _Claim(1, ('family', 'maiden'), "2150936a8c55"), + _Claim(1, ('family', 'maiden'), "2150936a8c55", None), "fix(#371) a suffix never begins a name: the Ph./D. merge declines at the head": - _Claim(4, ('family', 'given', 'middle', 'suffix', 'title'), "1425d85a2d86"), + _Claim(4, ('family', 'given', 'middle', 'suffix', 'title'), "1425d85a2d86", None), + "feat(#395) a leading never-given particle bounds the family under a declared family-first order": + _Claim(1, ('family', 'given', 'middle'), "47dcff268731", ('FAMILY_FIRST', 'FAMILY_FIRST_GIVEN_LAST')), + "fix(#399)/feat(#395) a consumed maiden marker leaves the family-first fold no given name": + _Claim(1, ('family', 'given', 'maiden'), "504eb466e347", ('FAMILY_FIRST', 'FAMILY_FIRST_GIVEN_LAST')), + "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold": + _Claim(1, ('family', 'given', 'middle', 'suffix', 'title'), "fc6bc9e605e1", ('FAMILY_FIRST',)), + "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold, the given-last spelling": + _Claim(1, ('family', 'given', 'middle', 'suffix', 'title'), "3e43a2be022e", ('FAMILY_FIRST_GIVEN_LAST',)), }, "expected_since_2.2.0.toml": {}, # open cycle, no rules yet "expected_since_2.1.0.toml": { "fix(#371) a suffix never begins a name: the Ph./D. merge declines at the head": - _Claim(4, ('family', 'given', 'middle', 'suffix', 'title'), "1425d85a2d86"), + _Claim(4, ('family', 'given', 'middle', 'suffix', 'title'), "1425d85a2d86", None), "fix(#335) a marker-led clause leaves the one name word its bare reading": - _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b"), + _Claim(1, ('maiden', 'nickname'), "c09cc7dba88b", None), "fix(#434) a multi-word maiden marker takes the maiden name": - _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef"), + _Claim(1, ('family', 'maiden', 'middle'), "c428798fc6ef", None), "fix(#434) a multi-word marker leads a bracketed clause to the maiden name": - _Claim(1, ('maiden', 'nickname'), "0b3ef183f283"), + _Claim(1, ('maiden', 'nickname'), "0b3ef183f283", None), "fix(#335) a marker-led bracketed clause reads as the maiden name whatever pair encloses it": - _Claim(6, ('maiden', 'nickname'), "d0e857deddb2"), + _Claim(6, ('maiden', 'nickname'), "d0e857deddb2", None), "fix(#410) a title and one name word name the family, whatever annotation stands beside it": - _Claim(4, ('family', 'given'), "da1dd1473145"), + _Claim(4, ('family', 'given'), "da1dd1473145", None), "fix(#430) a credential run does not end at the roman numeral describing it": - _Claim(2, ('given', 'suffix'), "3c8fa6bc827a"), + _Claim(2, ('given', 'suffix'), "3c8fa6bc827a", None), "fix(#432) a dotted numeral behind a name is a middle initial, not the generation": - _Claim(1, ('middle', 'suffix'), "e9f282da0d0f"), + _Claim(1, ('middle', 'suffix'), "e9f282da0d0f", None), "fix(#429) a wholly-credential segment after a one-word family renders as one entry": - _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe"), + _Claim(1, ('suffix', 'title'), "9e0b9e8d5cbe", None), "fix(#379) a tussenvoegsel after a family comma attaches to the family": - _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda"), + _Claim(13, ('_ambiguities', 'family', 'middle'), "973617235cda", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle": - _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8"), + _Claim(1, ('_ambiguities', 'family', 'given'), "ca7b37af6cf8", None), "fix(#367) a title no longer displaces a leading particle out of the leading position": - _Claim(3, ('family', 'given'), "724967a4a117"), + _Claim(3, ('family', 'given'), "724967a4a117", None), "fix(#380) a trailing vd after a family comma is the tussenvoegsel, not a post-nominal": - _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1"), + _Claim(2, ('_ambiguities', 'family', 'suffix'), "ec0d45289dc1", None), "fix(#399) a maiden marker bounds the particle chain that swallowed it": - _Claim(5, ('family', 'maiden'), "15ec75a89f07"), + _Claim(6, ('family', 'maiden'), "89e1f4afdd2a", ('DEFAULT',)), "fix(#360) mc moved into the never-given particles, so it folds into the family": - _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d"), + _Claim(1, ('_ambiguities', 'family', 'given'), "ee4339908f4d", None), "fix(#400) abd joins the word after it as one given name": - _Claim(11, ('given', 'middle'), "1eaed91fc574"), + _Claim(11, ('given', 'middle'), "1eaed91fc574", None), "fix(#367) a title no longer displaces a leading never-given particle": - _Claim(1, ('family', 'given'), "db724fb9c779"), + _Claim(1, ('family', 'given'), "db724fb9c779", None), "fix(#411) the bound-given reserve stops counting words the maiden name takes": - _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613"), + _Claim(1, ('given', 'maiden', 'middle'), "7515923c9613", None), "fix(#412) a connective join no longer absorbs the maiden marker beside it": - _Claim(2, ('family', 'maiden'), "51c0eb36b5c5"), + _Claim(2, ('family', 'maiden'), "51c0eb36b5c5", None), "fix(#418) the connective carve-out counts the name the maiden clause leaves behind": - _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7"), + _Claim(1, ('family', 'given', 'middle'), "7923e6d3c5a7", None), "fix(#418) accepted: a suffix word inside the maiden name ends it, connective or not": - _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a"), + _Claim(1, ('family', 'maiden', 'middle'), "bedc18423d2a", None), "fix(#369) a given-name title licenses the bound given-name join with one word to spare": - _Claim(3, ('family', 'given'), "724be3e6b926"), + _Claim(3, ('family', 'given'), "724be3e6b926", None), "fix(#401) the bound-given reserve counts the trailing numeral assign reads as the suffix": - _Claim(2, ('family', 'given'), "5713d4c0bd68"), + _Claim(2, ('family', 'given'), "5713d4c0bd68", None), "fix(#421) the bound-given join never absorbs a suffix piece": - _Claim(1, ('given', 'middle'), "9523e518e6ec"), + _Claim(1, ('given', 'middle'), "9523e518e6ec", None), "fix(#421) the bound-given join never absorbs a split credential": - _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef"), + _Claim(1, ('given', 'middle', 'suffix'), "228abe0f32ef", None), "fix(#425) the bound-given reserve runs assign's peel over the joined view": - _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e"), + _Claim(2, ('family', 'given', 'suffix'), "ef1ab03b617e", None), "fix(#424) the particle chain stops before the trailing numeral": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "2c99162bc9cf", None), "fix(#424) the particle chain stops before a bare acronym with words to spare": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "3e3aae6a5b4b", None), "fix(#424) a title-led chain before the numeral is the one name piece": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "5b3a743f9e35", None), "fix(comma-family) a comma followed only by titles keeps the given/family split": - _Claim(2, ('family', 'given'), "5bd9c6d96c38"), + _Claim(2, ('family', 'given'), "5bd9c6d96c38", None), "fix(comma-family) a comma followed only by titles keeps the given/family split, the C1 example": - _Claim(2, ('family', 'given'), "a3cfff4e78f4"), + _Claim(2, ('family', 'given'), "a3cfff4e78f4", None), "fix(#296) a dropped prenominal takes the name position it occupies": - _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1"), + _Claim(3, ('_ambiguities', 'given', 'middle', 'title'), "263d5957cfc1", None), "fix(#296) dr is not postnominal vocabulary, so a trailing Dr. is a name word": - _Claim(9, ('family', 'middle', 'suffix'), "8d6e9a1b43c0"), + _Claim(11, ('family', 'middle', 'suffix'), "b9cfc0d88bf6", None), "fix(#296) a credential-only comma string reads a name and its postnominal": - _Claim(2, ('suffix', 'title'), "3f983ff71dee"), + _Claim(2, ('suffix', 'title'), "3f983ff71dee", None), "fix(#296) a lone post-comma credential is a suffix": - _Claim(18, ('suffix', 'title'), "1f79efa10444"), + _Claim(18, ('suffix', 'title'), "1f79efa10444", None), "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes": - _Claim(6, ('given', 'suffix', 'title'), "7911e0158337"), + _Claim(6, ('given', 'suffix', 'title'), "7911e0158337", None), "fix(#325) a credential run across a second comma reads as suffixes": - _Claim(1, ('suffix', 'title'), "f025c5f70a4e"), + _Claim(1, ('suffix', 'title'), "f025c5f70a4e", None), "fix(#296) a glued honorific before a lone credential: the credential is the postnominal": - _Claim(1, ('suffix', 'title'), "01bf2bd3f895"), + _Claim(1, ('suffix', 'title'), "01bf2bd3f895", None), "fix(#296) do is a name, so it no longer stops the leading-particle scan as a title": - _Claim(1, ('family', 'given'), "faa2c70fc49e"), + _Claim(1, ('family', 'given'), "faa2c70fc49e", None), "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps its split and its title": - _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65"), + _Claim(2, ('_ambiguities', 'suffix', 'title'), "34d3d96adb65", ('DEFAULT', 'FAMILY_FIRST')), "fix(#296) an ambiguous acronym counts as a suffix only when written with its periods": - _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4"), + _Claim(1, ('_ambiguities', 'family', 'suffix'), "e13b3c769de4", None), "fix(#367) an inferred title no longer displaces a leading particle either": - _Claim(1, ('family', 'given'), "d8ee9cd5da5f"), + _Claim(1, ('family', 'given'), "d8ee9cd5da5f", None), "fix(#424) accepted: a particle of the suffix vocabulary opening the trailing run is a suffix piece": - _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162"), + _Claim(1, ('_ambiguities', 'family', 'middle', 'suffix'), "a564b97f7162", None), "fix(#424) an unlisted abbreviation is as transparent as a listed title to the leading particle, the P4 example": - _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320"), + _Claim(1, ('_ambiguities', 'family', 'given'), "42b69cf1b320", None), "fix(#424/#445) the maiden walk stops before the trailing numeral, and the lone name word is the family": - _Claim(1, ('_ambiguities', 'family', 'given', 'maiden', 'suffix'), "cbe5bdd97317"), + _Claim(1, ('_ambiguities', 'family', 'given', 'maiden', 'suffix'), "cbe5bdd97317", None), "fix(#445) a maiden marker makes the lone name word the family": - _Claim(6, ('family', 'given'), "f521c94c79fc"), + _Claim(6, ('family', 'given'), "f521c94c79fc", None), "fix(#445) the lone name word beside a marker a connective join no longer absorbs": - _Claim(1, ('family', 'given', 'maiden', 'middle'), "52544a41dd62"), + _Claim(1, ('family', 'given', 'maiden', 'middle'), "52544a41dd62", None), "fix(#424) a marker followed only by the numeral is just a word": - _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071"), + _Claim(1, ('_ambiguities', 'family', 'maiden', 'middle', 'suffix'), "aaf53040b071", None), "fix(#369) the bound given-name join takes a particle-and-bound word, so no fork is reported": - _Claim(1, ('_ambiguities',), "81cf02ffdb33"), + _Claim(1, ('_ambiguities',), "81cf02ffdb33", None), "fix(#360) ste moved into the never-given particles with mc": - _Claim(1, ('_ambiguities', 'family', 'given'), "e62caedec864"), + _Claim(1, ('_ambiguities', 'family', 'given'), "e62caedec864", None), "fix(#399) a maiden marker bounds the particle chain: the geb. spelling": - _Claim(1, ('family', 'maiden'), "2150936a8c55"), + _Claim(1, ('family', 'maiden'), "2150936a8c55", None), "fix(#399) a maiden marker bounds the particle chain: a native-script marker": - _Claim(1, ('family', 'maiden'), "f016cc61ca43"), + _Claim(1, ('family', 'maiden'), "f016cc61ca43", None), + "feat(#395) a leading never-given particle bounds the family under a declared family-first order": + _Claim(1, ('family', 'given', 'middle'), "47dcff268731", ('FAMILY_FIRST', 'FAMILY_FIRST_GIVEN_LAST')), + "fix(#399)/feat(#395) a consumed maiden marker leaves the family-first fold no given name": + _Claim(1, ('family', 'given', 'maiden'), "504eb466e347", ('FAMILY_FIRST', 'FAMILY_FIRST_GIVEN_LAST')), + "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold": + _Claim(1, ('family', 'given', 'middle', 'suffix', 'title'), "fc6bc9e605e1", ('FAMILY_FIRST',)), + "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold, the given-last spelling": + _Claim(1, ('family', 'given', 'middle', 'suffix', 'title'), "3e43a2be022e", ('FAMILY_FIRST_GIVEN_LAST',)), }, } @@ -1974,6 +2044,12 @@ def test_every_rule_claims_the_recorded_share_of_the_corpus() -> None: #: The diff shapes are measured against the 1.4.0 wheel, not guessed. #: Re-measure rather than adjust them if a parser change moves one: #: a diff shape that shifted is a finding, not a number to update. +#: +#: Blind spot since `orders` (#468): every row is recomputed at +#: comparison order None, so a contest that exists only under a +#: declared order is not recorded here -- true today because no +#: recorded winner is order-scoped, and it stops being true the day one +#: is. _CROSS_RULE_WINNERS: dict[str, dict[tuple[str, tuple[str, ...]], str]] = { "expected_since_2.2.0.toml": {}, # open cycle, no rules and so no contest "expected_since_1.4.0.toml": { @@ -2177,6 +2253,64 @@ def test_the_recorded_rule_still_wins_each_contested_name() -> None: f"was the odd one out.") +def test_the_family_first_fold_is_not_explained_under_the_default_order( + ) -> None: + """The hazard the `orders` key exists for, pinned against the + shipped 2.x ledgers rather than a fixture. + + 'de la Cruz Juan Carlos' is compared three times: twice from + corpus_shapes.jsonl under the two family-first orders, where #395's + fold is the intended reading, and once from corpus_rules.jsonl + under the DEFAULT order, where rules.md#P1 says the whole string is + the family. If the fold ever leaked into the default order it would + move {family, given, middle} -- the same three roles, on the same + string -- so an order-blind rule would claim the leak, label it + feat(#395), and exit 0. That is #372's failure mode aimed at the + most plausible regression of the very change the rule describes. + + Both directions again: declining the default-order diff is the + point, but a rule that declined its OWN orders too would be + silently dormant and this pin would pass on a broken ledger. + + The default-order assertion runs on EVERY ledger, before the + scoped-rule search decides whether there is anything else to check. + Guarding it behind that search is what made an earlier draft weak: + dropping `orders` from one 2.x ledger takes that ledger through the + `continue` while the other keeps `checked` above zero, and the + suite stays green with one ledger absorbing the leak. On a ledger + whose rules do not reach this name at all -- 1.4.0, the open cycle + -- the assertion is trivially true, which costs nothing. + """ + compare = load_tool("compare") + name, moved = "de la Cruz Juan Carlos", {"family", "given", "middle"} + checked = 0 + for ledger in _LEDGERS: + rules = compare._sorted_rules(_rules(ledger)) + never = _exclusions(ledger) + assert compare.classify(name, moved, rules, never) is None, ( + f"{ledger.name}: a DEFAULT-order diff moving {sorted(moved)} " + f"on {name!r} is explained by " + f"{compare.classify(name, moved, rules, never)!r}. That is " + f"the family-first fold firing where P1 forbids it, and the " + f"rule describing the fold must not absorb it -- scope it " + f"with `orders`") + scoped = [r for r in rules + if isinstance(r.get("orders"), list) + and re.search(str(r["name_regex"]), name) + and moved <= set(r["fields"])] + for rule in scoped: + for order in rule["orders"]: + assert compare.classify( + name, moved, rules, never, order) == rule["issue"], ( + f"{ledger.name}: {rule['issue']!r} declares order " + f"{order} and does not explain its own diff under " + f"it; the rule is dormant and the pin above is " + f"passing for the wrong reason") + checked += 1 + assert checked, ("no orders-scoped rule reaches this name in any " + "ledger, so this pin is vacuous") + + class _Excluded(NamedTuple): """What a [[never]] entry silences, in the two dimensions that can change under it.""" @@ -2220,6 +2354,12 @@ class _Excluded(NamedTuple): #: answers those subsets is invisible here. A rule reaching a #: protected READING is caught; a rule shadowed by an existing one on #: every subset it claims is not. +#: +#: A second, newer limit: `absorbed_by` is recomputed at comparison +#: order None, so an absorption that happens only under a declared +#: order is invisible to it -- true today because no exclusion +#: protects a shape that reads differently under one, and something to +#: revisit the day one does. _EXCLUSION_EFFECT: dict[str, _Excluded] = { "(?i)^(?!\\s*ph\\.)(?![^\\s,]+\\s*,\\s*ph\\.\\s*d\\.\\s*$)(?![\\u0000-\\u024f]*\\b(?:jr|sr|ii|iii|iv)\\.?\\s+ph\\.\\s*d\\.\\s*$)[\\u0000-\\u024f]*\\bph\\.\\s*d\\.\\s*$": _Excluded(3, "5a12a8117651", diff --git a/tests/v2/test_order_correspondence.py b/tests/v2/test_order_correspondence.py new file mode 100644 index 00000000..5fa1f781 --- /dev/null +++ b/tests/v2/test_order_correspondence.py @@ -0,0 +1,206 @@ +"""The family-first/comma correspondence, as an executable invariant +(#469). + +Form 4 (Title Family Given Middle Middle [Particle] [, Suffix] under +FAMILY_FIRST) is form 2 (Family [Suffix], Title Given (Nickname) +Middle Middle[,] Suffix [, Suffix]) with the comma removed and the +family inline -- the two notations quoted verbatim from +tools/differential/shapes.py. The pairs are GENERATED and asserted +equal, which consults no vocabulary and no rule -- so it catches a +rule that is WRONG, which review cannot. + +Provenance (decisions.md#P6, the 2026-08-30 #467 entry): over ALL 70 +particles x 3 families x 3 givens = 630 generated names, 0 of 630 +agreed before the #467 change and 603 of 630 after. The 27 failures +are the particle-and-suffix trio -- vd, mc, do -- a real limit and +not noise (rules.md#P6, restated as a third bullet below); this +module's own parametrized sweep below excludes that trio on purpose +and covers 2 x 1 x 3 x 4 x 2 x 2 pairs of its own -- families, +givens, middles, particles, titles, suffixes -- a SEPARATE, smaller +count that must never be quoted as the 630 measurement or vice versa. +Written as the product rather than as a figure because the figure was +the thing that went stale: the tuples below are what decides it. decisions.md +#two-input-invariants recomputes the same correspondence a second +way -- a 6x6 spread of families and givens per particle rather than +3x3 -- and gets 67 of 70 particles agreeing on all 36 of their pairs, +the same trio failing all 36; its Recompute recipe is what this +file's negative-control test below runs. #466, the rejected +predecessor, has no comparable number on record; its defects are +qualitative -- it lost a given name outright on "van Berg Jan de" +and promoted a post-nominal into the given slot on "Berg Jan Jr. +de" (decisions.md#P6). An earlier draft of this module cited +"0/216 -> 216/216", which decisions.md#P6 explicitly retracts: that +number came from a script slicing the vocabulary +(`sorted(particles)[:14]`), which happened to exclude exactly vd, mc +and do -- the project's own example of a detector inheriting the +design's blind spot (decisions.md#P6 says: "A FIRST DRAFT OF THIS +ENTRY CLAIMED 216 of 216 ... Recompute over the WHOLE vocabulary, +never a sample"). + +Three limits, load-bearing (from #469 and #467). All three are now +restated in customize.rst's family-first section, alongside the +corrected 603/630 measurement: + +- One shape written two ways, NOT comma-deletion in general. A name + whose SHAPE changes when the comma goes (a title moving mid-name) + is a different shape, not a counterexample, so no such pair is + generated here. +- Form 5 has no comma twin: no comma format puts the given name + last, which is why FAMILY_FIRST_GIVEN_LAST appears nowhere in this + file. +- Where a word is both particle and suffix vocabulary, the two + writings read it differently and the correspondence genuinely + fails. It is the ASYMMETRY that breaks it, not a precedence + applying to both sides: the attachment outranks the suffix reading + on the COMMA side, which is the scope rules.md#P6 states it in, so + `Ménil, Christophe vd` reads family `vd Ménil` while the + family-first `Ménil Christophe vd` reads family `Ménil` and suffix + `vd`. Not a bug, but rules.md#P6 stating it plainly: + "the word is BOTH a particle and suffix vocabulary, this + attachment outranks the suffix reading (S2): a trailing + abbreviation after a family comma is the tussenvoegsel far more + often than the decoration it collides with" -- and decisions.md + #P6 calling it "a real limit, not noise". The parametrized + vocabulary below deliberately excludes vd/mc/do; the negative + control test owns them instead. + +A mass failure here -- most or all pairs turning red, rather than a +handful -- means the correspondence itself has narrowed and rules.md +#P6 needs amending alongside this file, not that the vocabulary +needs shrinking to get back to green. +""" +import itertools +import unicodedata +from collections.abc import Iterator + +import pytest + +from nameparser import FAMILY_FIRST, ParsedName, Parser, Policy, Role, parse +from nameparser.config.particles import PARTICLES as _SHIPPED_PARTICLES + +_FF = Parser(policy=Policy(name_order=FAMILY_FIRST)) +_ROLES = tuple(r.value for r in Role) # test_cases.py's idiom; declaration + # order is canonical + + +def _reading(p: ParsedName) -> dict[str, object]: + """The comparator shape tools/differential/compare.py uses: the + seven role fields (never None -- ParsedName's role properties are + typed `-> str`) plus the ambiguity-kind SET. decisions.md#P6 + (#405) records that this design's own repaired failure was + exactly an ambiguity ASYMMETRY between two writings of one name, + so a role-only comparison would miss the most fragile part of the + correspondence. A set and not a sorted multiset, mirroring + compare.py's own `sorted({a.kind.name for a in ...})` exactly -- + two occurrences of one kind are not a second thing to agree on.""" + out: dict[str, object] = {r: getattr(p, r) for r in _ROLES} + out["_ambiguities"] = sorted({a.kind.name for a in p.ambiguities}) + return out + + +FAMILIES = ("Ménil", "Jong") # Ménil pins the non-ASCII case +GIVENS = ("Christophe",) +MIDDLES = (None, "Marie", "Marie Louise") # shape 4's notation names + # Middle Middle; "Marie Louise" + # is the one pair generating two +# never-given (de), ambiguous (van), and a multi-token run (van der). +# The multi-token member is not decoration: a last-particle-only walk +# once survived the whole suite on single-token fixtures alone, and +# tests/v2/pipeline/test_post_rules.py::test_a_multi_token_run_is_taken_whole +# is the pin that now owns that shape at the rule layer. +PARTICLES = (None, "de", "van", "van der") +TITLES = (None, "Dr.") +SUFFIXES = (None, "Jr.") + + +def _slug(s: str) -> str: + """Turn an inline spelling into a bare pytest id: no quoting needed + to select one case. Folded to ASCII -- pytest itself backslash- + escapes a non-ASCII character in an id, which would make that + claim false for Menil's accent.""" + ascii_s = unicodedata.normalize("NFKD", s).encode("ascii", "ignore") + return ascii_s.decode("ascii").replace(" ", "_").translate( + str.maketrans("", "", ".,")) + + +def _pairs() -> Iterator[tuple[str, str]]: + """(comma spelling, family-first spelling) for every combination. + The title moves position between the writings -- post-comma in + form 2, leading in form 4 -- which is the part of the + correspondence review keeps getting wrong, so it is generated + rather than sampled.""" + for fam, giv, mid, part, title, suf in itertools.product( + FAMILIES, GIVENS, MIDDLES, PARTICLES, TITLES, SUFFIXES): + tail = " ".join(p for p in (giv, mid, part) if p) + t = f"{title} " if title else "" + comma = f"{fam}, {t}{tail}" + (f", {suf}" if suf else "") + inline = f"{t}{fam} {tail}" + (f", {suf}" if suf else "") + yield comma, inline + + +_PAIRS = list(_pairs()) +_IDS = [_slug(inline) for _, inline in _PAIRS] + + +@pytest.mark.parametrize("comma, inline", _PAIRS, ids=_IDS) +def test_form_4_parses_as_its_comma_twin(comma: str, inline: str) -> None: + a, b = parse(comma), _FF.parse(inline) + got_a, got_b = _reading(a), _reading(b) + assert got_a == got_b, ( + f"{comma!r} (default) and {inline!r} (FAMILY_FIRST) are one " + f"shape written two ways and must agree") + + +# The recorded negative control (decisions.md#P6, decisions.md#two- +# input-invariants, rules.md#P6): the ONLY members of the shipped +# particle vocabulary where the correspondence fails are the words +# that are both particle and suffix vocabulary. Swept over the entire +# shipped vocabulary -- nameparser.config.particles.PARTICLES, the +# same source the invariant's own measurements drew from, never a +# hand copy -- rather than the small parametrized set above, so this +# is the test that would notice if a fourth word joined the trio. +# +# mechanisms.md#RECORDED-ROSTERS: "never re-derive the expectation +# from the same inputs the check reads, because a derivation from the +# same data always agrees with itself" -- a recorded literal, kept by +# hand and never computed from the vocabulary intersection below. +_EXPECTED_DISAGREEING_PARTICLES = frozenset({"do", "mc", "vd"}) + + +def test_only_the_particle_suffix_trio_breaks_the_correspondence() -> None: + # rules.md#P6 states the exception this sweep measures: "the word + # is BOTH a particle and suffix vocabulary, this attachment + # outranks the suffix reading (S2)" + # + # The shape is the minimal "Family Given Particle" -- one family, + # one given -- rather than decisions.md#two-input-invariants' 6x6 + # spread, because that entry's own Recompute bullet says a spread + # only matters to rule out families/givens that are THEMSELVES + # particle vocabulary (the precondition asserted below); once + # that is excluded, the failures partition on the particle word + # alone; one point suffices. + fam, giv = "Ménil", "Christophe" + # mechanisms.md#VOCABULARY-OVERLAP-AS-PRECONDITION: "assert the + # intersection as a precondition" -- inverted from the mechanism's + # usual shape: what has to hold here is the intersection being + # EMPTY, not populated. decisions.md#two-input-invariants records + # that a particle-vocabulary family or given measures a different + # shape and reports 0 of 70 rather than 67 of 70. + assert not ({fam.lower(), giv.lower()} & _SHIPPED_PARTICLES), ( + f"{fam!r} or {giv!r} joined the shipped particle vocabulary; " + f"this sweep measures the Family Given Particle shape and " + f"needs a family/given that is not itself particle vocabulary " + f"-- pick another pair") + disagreeing = set() + for p in _SHIPPED_PARTICLES: + comma = f"{fam}, {giv} {p}" + inline = f"{fam} {giv} {p}" + got_a = _reading(parse(comma)) + got_b = _reading(_FF.parse(inline)) + if got_a != got_b: + disagreeing.add(p) + assert disagreeing == _EXPECTED_DISAGREEING_PARTICLES, ( + f"the particle vocabulary's disagreeing set changed: " + f"got {sorted(disagreeing)}, expected " + f"{sorted(_EXPECTED_DISAGREEING_PARTICLES)} -- this is a finding, " + f"not a fixture to update blindly") diff --git a/tools/differential/README.md b/tools/differential/README.md index 2d859e72..9226c3ae 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -27,17 +27,23 @@ uv run python tools/differential/compare.py # bare = DEFAULT_BASELINE, the las ``` `compare.py` spawns the worker as a subprocess, feeds it every corpus -name as a line of JSON, and diffs the two sides field by field. Every -diff is checked against that baseline's ledger: +entry as a line of JSON -- `{"name": ..., "order": ...}`, where `order` +names the `name_order` constant the entry is parsed under on both +sides and is `null` for an entry carrying no order -- and diffs the two +sides field by field. Every diff is checked against that baseline's +ledger: - Matches a rule -> counted as an intentional, classified change. -- Matches no rule -> printed under `UNEXPLAINED` and the run exits 1. +- Matches no rule -> reported, and what happens next depends on the + name's tier (see below): on a CONTRACT corpus it prints under + `UNEXPLAINED` and the run exits 1; on a RADAR corpus it prints under + `UNCLASSIFIED (radar)` and the run keeps exiting 0. -An unexplained diff means either a real parity bug (fix it, don't -allowlist it) or a known change whose ledger rule needs widening. The -run must exit 0 at every baseline you claim before a release; the -classified summary it prints is the source for the "Behavior Changes" -section of `docs/release_log.rst`. +An unexplained diff on the contract tier means either a real parity +bug (fix it, don't allowlist it) or a known change whose ledger rule +needs widening. The run must exit 0 at every baseline you claim before +a release; the classified summary it prints is the source for the +"Behavior Changes" section of `docs/release_log.rst`. ## Baselines @@ -173,18 +179,73 @@ Why this is easy to miss when probing by hand: from the repo root, the checkout wins and the trap does not reproduce. Only the script invocation shows it. -## The three corpora +## The corpora and their tiers `compare.py` reads **every** `corpus*.jsonl` beside it by default (deduped), because a corpus you have to ask for by name is a corpus that stops being run. Pass `--corpus PATH` (repeatable) to narrow it. +The run's `corpora:` line names each file with its entry count, and +with `(N, K skipped)` where K entries were left out of this baseline's +comparison (see the shape key below). Each file also needs an entry in +`_CORPUS_FLOORS`, a minimum set a little under its real size: a corpus +that silently shrinks to nothing would otherwise report a green run. + +| File | Source | Tier | Blind to | +|---|---|---|---| +| `corpus.jsonl` | v1's own test suite at a pinned ref | radar | anything 2.0 added — v1's authors had no reason to test a typographic nickname delimiter or a Cyrillic title | +| `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | radar | anything nobody ever reported | +| `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py`, via `build_cjk_corpus.py` (#295) | contract | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | +| `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | contract | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | +| `corpus_shapes.jsonl` | shape-tagged rows of `tests/v2/cases.py`, via `build_shapes_corpus.py` (#468) | contract | anything no one has tagged a row for | + +Since the v2.3 tier split (#468), a corpus is CONTRACT or RADAR -- +the roster is `_CORPUS_TIERS` in compare.py, fail-closed like the +floors. Contract corpora hold names someone chose, and an unmatched +diff on one is UNEXPLAINED and fails the run. Radar corpora hold the +scraped and harvested names: their diffs still classify against the +ledger, so intended changes keep their release-note grouping, but an +unmatched radar diff prints under UNCLASSIFIED (radar) and cannot +fail the run or demand a ledger rule. Nothing is deleted to keep the +gate quiet -- a meaningless string in radar costs one parse and a +report line. To promote a radar name, give it a tests/v2/cases.py row +and a shape tag: it enters the contract by being chosen. A +`[[never]]` exclusion outranks the tier either way: it was chosen too +-- someone wrote its `why` and its `examples` -- so a name it refuses +stays UNEXPLAINED and fails the run even when the name itself sits in +a radar file. + +A corpus line is a bare JSON string or an object carrying `name` and, +optionally, `tests` or `shape` -- the input-shape id from +`tools/differential/shapes.py`, which is where each shape's notation, +the `name_order` it is an input shape FOR, and the oldest baseline +whose worker can honor that order are written down. `compare.py` +resolves every entry's shape through that table: an entry whose shape +declares an order is parsed under that order on both sides and +compared on the v2 surface alone (the facade is the v1-compat surface, +and a family-first name is not a v1 contract), and an entry whose run +predates its shape's `min_baseline` is left out of the comparison -- +reported as `skipped N names tagged shape(s) [...]` and counted in the +`corpora:` line, so a shrunken comparison is never silent. That a +family-first name is not compared under the default order is +structural, not a ledger exception: its shape says what the name is an +input FOR. + +`corpus_shapes.jsonl` is the corpus of those entries: every distinct +(shape, name) pair from the shape-tagged rows of the case table, so a +name tagged under two orders is two entries and two comparisons. +Regenerate it after tagging or editing a tagged row; `--coverage` +prints names-per-shape instead of writing, which is the "which shapes, +how many names deep" answer: -| File | Source | Blind to | -|---|---|---| -| `corpus.jsonl` | v1's own test suite at a pinned ref | anything 2.0 added — v1's authors had no reason to test a typographic nickname delimiter or a Cyrillic title | -| `corpus_issues.jsonl` | name-like strings harvested from the GitHub issue tracker | anything nobody ever reported | -| `corpus_cjk.jsonl` | the CJK-bearing rows of `tests/v2/cases.py`, via `build_cjk_corpus.py` (#295) | anything the case table itself missed — it re-witnesses reviewed expectations at the baseline boundary rather than discovering new shapes | -| `corpus_rules.jsonl` | every example in `docs/design/rules.md`, via `build_rules_corpus.py` (#414) | anything the rules doc has no example for — it re-witnesses the normative examples at the baseline boundary | +``` +uv run python tools/differential/build_shapes_corpus.py +uv run python tools/differential/build_shapes_corpus.py --coverage +``` + +It is pinned exactly as the CJK corpus is -- `tests/v2/test_ledger_guards.py` +holds the checked-in file equal to the generator's selection, so a row +tagged without regenerating fails the suite instead of leaving the +contract tier narrower than the case table says it is. They are deliberately separate rather than merged: `corpus.jsonl` is reproducible forever from an immutable git ref, while the issue @@ -226,7 +287,7 @@ the moment a ` Ph. D.` name appeared. Backticks prompted a second look at prose, and the same change added two screens neither branch had: `:` joins the structural characters (it -appears in no name across all four corpora, and accounts for three +appears in no name across all five corpora, and accounts for three error messages and a PyPI trove classifier), and a short list of English function words rejects capitalized sentences the character screen cannot see — `What this gate does not cover` is well-formed as a phrase. That list @@ -267,15 +328,45 @@ operation on this worktree's own log; it does not check out, stash, or otherwise mutate anything. The AST extraction over-collects on purpose (string literals passed as -`HumanName(...)`'s first argument, plus string members of module-level -list/dict/tuple banks that contain a space) -- more candidate strings -is more coverage, and the corpus is deduplicated. Obvious non-names -(strings containing `{`, `@`, or a backslash -- format placeholders, -decorator/email-shaped fixtures, escape sequences) are dropped. +`HumanName(...)`'s first argument, plus string members of list/dict/ +tuple banks that contain a space) -- more candidate strings is more +coverage, and the corpus is deduplicated. Obvious non-names (strings +containing `{`, `@`, or a backslash -- format placeholders, decorator/ +email-shaped fixtures, escape sequences) are dropped. + +Each line is `{"name": ..., "tests": [...]}`: `tests` is the sorted +labels the name appeared under at the pinned ref -- the shape context +(`test_title_with_conjunction` and kin) the original bare-string scrape +kept the string but threw away. A `HumanName(...)` call is labelled by +its nearest enclosing test method, falling back to the source filename +(e.g. `test_titles.py`) for a call at module scope; a list/dict/tuple +bank is labelled by its variable name, prefixed `bank:` so it reads +apart from a test method at a glance (`bank:` when the +assignment target isn't a plain name). Labels merge across files: two +files sharing a method name (e.g. two `test_basic_parsing`s) produce +one merged label set on any name both contribute, which is accepted -- +the merged labels still describe the same string. `compare.py` +surfaces these as `[v1: ...]` tags on radar rows. + +Regenerate the corpus only (a) at the same ref, as a format-only +enrichment, with the name set proven identical to the previous file by +set comparison, or (b) if the v1 test banks are revisited again at a +still-earlier point in history; otherwise leave the checked-in file +alone so the harness stays comparable run to run. For (a): -Regenerate the corpus only if the v1 test banks are revisited again at -a still-earlier point in history; otherwise leave the checked-in file -alone so the harness stays comparable run to run. +``` +uv run python tools/differential/build_corpus.py --ref 2d5d8c2 > /tmp/corpus-new.jsonl +uv run python -c " +import json, subprocess +def _n(x): return x if isinstance(x, str) else x['name'] +old = {_n(json.loads(l)) for l in subprocess.run( + ['git', 'show', 'HEAD:tools/differential/corpus.jsonl'], + capture_output=True, text=True, check=True).stdout.splitlines() if l.strip()} +new = {_n(json.loads(l)) for l in open('/tmp/corpus-new.jsonl', encoding='utf-8') if l.strip()} +if old != new: raise SystemExit(sorted(old ^ new)) +print(f'identical: {len(old)} names')" +cp /tmp/corpus-new.jsonl tools/differential/corpus.jsonl +``` `corpus_issues.jsonl` is built by `build_issues_corpus.py` from the issue tracker (`gh issue list --state all`), taking `HumanName("...")` @@ -315,6 +406,46 @@ a rule. Note the two bans are each other's obvious wrong answer -- deleting `fields` to silence an over-declaration failure lands on #456's, and adding `fields` while dropping the regex lands on #451's. +**`orders` is the optional third narrowing** (#468). A rule may carry +`orders = ["FAMILY_FIRST", ...]` -- public order-constant names, taken +from the ones `shapes.py` declares, so `validate_rules` rejects a name +no shape asks for rather than letting the rule sit dormant. A rule +carrying it matches only diffs from comparisons run under one of those +orders. Omit the key and the rule is order-blind, which is what every +rule written before shape-tagged entries existed is. + +`"DEFAULT"` is the one member `shapes.py` does not supply, and cannot: +it names the comparison run under no declared order (`order` is +`None`), which is the absence of a shape rather than one of them. It +exists because TOML has no null to put inside an array, so without it +a rule explaining only default-order diffs had no way to SAY so and +had to stay order-blind -- and an order-blind rule leaks the other +way from the leak the key was added for, absorbing an order-bearing +diff on any name its regex happens to reach. `orders = ["DEFAULT"]` +is a default-order-only rule; `orders = ["DEFAULT", "FAMILY_FIRST"]` +is a rule that genuinely explains both and stops there. + +A run PRINTS the order-blind absorptions it sees: when a rule with no +`orders` key explains a diff from an order-bearing comparison, the +report carries an `ORDER-BLIND` block naming the issue, the name and +the order. It is informational and outside the exit code -- order-blind +rules stay legal -- but the absorption is no longer invisible. + +It exists because a name can now be compared more than once, and the +two diffs can move the SAME roles for opposite reasons. +`de la Cruz Juan Carlos` is compared under both family-first orders +from `corpus_shapes.jsonl`, where #395's fold reads family +`de la Cruz` and distributes the leftovers, and under the default +order from `corpus_rules.jsonl`, where rules.md#P1 says the whole +string is the family. If that fold ever leaked into the default order +it would move `{family, given, middle}` -- exactly what the +`feat(#395)` rule declares -- so an order-blind rule would claim the +leak, label it intentional and exit 0: #372's failure mode aimed at +the most plausible regression of the very change the rule describes. +Exclusions stay order-blind for now; refusal is monotone, so the worst +an over-wide one can do is make a name report `UNEXPLAINED`, which is +loud. + **That closes the SHAPE, not the property.** A required `name_regex` is not a bound on how much a rule reaches: the only width check is the sentinel probe, which rejects a pattern matching all four of @@ -511,7 +642,8 @@ import glob, json from nameparser import Parser from nameparser._lexicon import _normalize L = Parser().lexicon -names = [json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()] +def _n(x): return x if isinstance(x, str) else x['name'] +names = [_n(json.loads(l)) for f in glob.glob('tools/differential/corpus*.jsonl') for l in open(f, encoding='utf-8') if l.strip()] for s in ('maiden_markers', 'honorific_tails'): v = getattr(L, s) longest = max(e.count(' ') + 1 for e in v) diff --git a/tools/differential/build_corpus.py b/tools/differential/build_corpus.py index 2c2613b8..a4f39246 100644 --- a/tools/differential/build_corpus.py +++ b/tools/differential/build_corpus.py @@ -1,7 +1,13 @@ """Extract every plausible test-name string from the v1 test suite: string literals passed as the first argument to HumanName(...) calls, -plus string elements/keys of module-level list/dict banks. Dev tooling --- over-collection is fine, the comparator just parses more names. +plus string elements/keys of list/dict banks. Dev tooling -- over- +collection is fine, the comparator just parses more names. Output is +one JSON object per line, `{"name": ..., "tests": [...]}`, where +`tests` is the sorted v1 test method names (bare) or bank variable +names (prefixed `bank:`) the string appeared in at the pinned ref -- +the shape context the original scrape kept the string but threw away. +A `HumanName(...)` call is labelled by its NEAREST enclosing function, +falling back to the source filename for one at module scope. PROVENANCE: the checked-in tools/differential/corpus.jsonl was built against commit 2d5d8c2 ("Trim constant-factor waste on the tokenize @@ -36,23 +42,55 @@ def _is_name_like(value: str) -> bool: return not any(ch in value for ch in _NOISE_CHARS) -def _strings_from(tree: ast.Module) -> set[str]: - found: set[str] = set() - for node in ast.walk(tree): +def _labelled_strings_from( + tree: ast.Module, module_label: str) -> dict[str, set[str]]: + """name -> the v1 test methods (or bank variables) it appeared in. + This is the metadata the original scrape discarded: v1's test + names say what shape each name exercised + (test_title_with_conjunction), and #468's whole complaint is that + nothing recorded it. + + Single recursive walk, tracking the NEAREST enclosing + FunctionDef/AsyncFunctionDef as the current label (falling back to + `module_label`, the source filename, for a HumanName(...) call at + module scope) -- ast.walk's flat iteration cannot express nearest- + enclosing, and doing two separate module-wide sweeps double-labels + a call inside a nested function with both the inner and outer + names. The bank branch is untouched by this label: a list/dict + assignment is labelled by its own target name (prefixed `bank:` to + read apart from a test method at a glance) regardless of what + function, if any, encloses it. + """ + found: dict[str, set[str]] = {} + + def _add(value: str, label: str) -> None: + found.setdefault(value, set()).add(label) + + def _walk(node: ast.AST, label: str) -> None: if (isinstance(node, ast.Call) and getattr(node.func, "id", getattr( node.func, "attr", "")) == "HumanName" and node.args and isinstance(node.args[0], ast.Constant) and isinstance(node.args[0].value, str)): - found.add(node.args[0].value) + _add(node.args[0].value, label) if isinstance(node, ast.Assign) and isinstance( node.value, (ast.List, ast.Tuple, ast.Dict)): + target_name = getattr(node.targets[0], "id", None) + bank_label = f"bank:{target_name}" if target_name \ + else "bank:" for c in ast.walk(node.value): if isinstance(c, ast.Constant) and isinstance(c.value, str) \ and " " in c.value and "\n" not in c.value: - found.add(c.value) - return {n for n in found if _is_name_like(n)} + _add(c.value, bank_label) + for child in ast.iter_child_nodes(node): + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)): + _walk(child, child.name) + else: + _walk(child, label) + + _walk(tree, module_label) + return {n: labels for n, labels in found.items() if _is_name_like(n)} def _working_tree_sources() -> dict[str, str]: @@ -89,17 +127,19 @@ def main() -> None: args = ap.parse_args() sources = _ref_sources(args.ref) if args.ref else _working_tree_sources() - names: set[str] = set() + names: dict[str, set[str]] = {} for filename, text in sources.items(): try: tree = ast.parse(text, filename=filename) except SyntaxError as e: print(f"skipping {filename}: {e}", file=sys.stderr) continue - names |= _strings_from(tree) + for value, labels in _labelled_strings_from(tree, filename).items(): + names.setdefault(value, set()).update(labels) for name in sorted(names): - print(json.dumps(name, ensure_ascii=False)) + print(json.dumps({"name": name, "tests": sorted(names[name])}, + ensure_ascii=False)) print(f"{len(names)} names", file=sys.stderr) diff --git a/tools/differential/build_issues_corpus.py b/tools/differential/build_issues_corpus.py index 3f97d41d..b78234a4 100644 --- a/tools/differential/build_issues_corpus.py +++ b/tools/differential/build_issues_corpus.py @@ -76,7 +76,7 @@ # 'TypeError: a bytes-like object is required'. None of '*#~;_:' # appears in a name; the colon alone accounts for three error messages # and a PyPI trove classifier, and none of the six characters occurs in -# any name across the four corpora. The one existing entry this drops, +# any name across all five corpora. The one existing entry this drops, # 'St. ___', is a placeholder, not a name. _NOT_A_NAME = set('{}<>=/\\|*#~;_:') diff --git a/tools/differential/build_rules_corpus.py b/tools/differential/build_rules_corpus.py index 6e711026..62006c00 100644 --- a/tools/differential/build_rules_corpus.py +++ b/tools/differential/build_rules_corpus.py @@ -1,17 +1,12 @@ """Regenerate corpus_rules.jsonl from the examples in docs/design/rules.md. -The fourth corpus, with the fourth provenance (#414). The other three -each have a blind spot the others cover, and this one closes a blind -spot none of them was built to have: - -- corpus.jsonl comes from v1's own test suite at a pinned git ref, so - it is structurally blind to anything 2.0 added. -- corpus_issues.jsonl comes from what users reported, which is the - adversarial half. -- corpus_cjk.jsonl comes from the CJK rows of the case table, because - neither of the above can carry an unspaced CJK name. -- this file comes from the NORMATIVE RULES -- the names the project - has written down as defining its behavior. +A corpus with its own provenance (#414). Each corpus covers a blind +spot the others have -- the roster, with each file's source and the +blind spot it leaves, is the table in tools/differential/README.md. +This one comes from the NORMATIVE RULES: the names the project has +written down as defining its behavior, which is a blind spot no +corpus built from test suites, issue reports or case rows was made +to cover. Why the rules doc needs a corpus when its examples are already tests. `test_rules_doc.py` executes every example, which pins them harder diff --git a/tools/differential/build_shapes_corpus.py b/tools/differential/build_shapes_corpus.py new file mode 100644 index 00000000..0e60a398 --- /dev/null +++ b/tools/differential/build_shapes_corpus.py @@ -0,0 +1,72 @@ +"""Regenerate corpus_shapes.jsonl from the shape-tagged rows of the +case table. + +The contract corpus of the v2.3 tier design (#468): every entry here +was CHOSEN -- a row someone wrote with reviewed expected values -- +and the shape id records, once, at admission, what the name is for. +The corpus_cjk pattern generalized: same source of truth (cases.py), +same pinned projection (tests/v2/test_ledger_guards.py holds the +checked-in file equal to this module's selection), different +predicate -- explicit tags instead of codepoint content. + +compare.py resolves each entry's shape to its name_order through +tools/differential/shapes.py and compares family-first entries under +that order, on the v2 surface, at baselines >= the shape's minimum. + +Regenerate after tagging or editing tagged case rows: + + uv run python tools/differential/build_shapes_corpus.py + +Pass --coverage to print names-per-shape instead of writing, which is +the "which shapes, how many names deep" answer #468 asked for. +""" +from __future__ import annotations + +import argparse +import json +import sys +from collections import Counter +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parents[1] +sys.path.insert(0, str(ROOT)) + +from tests.v2.cases import CASES # noqa: E402 + +OUT = HERE / "corpus_shapes.jsonl" + + +def _pairs() -> list[tuple[int, str]]: + """Every distinct (shape, text) pair from tagged rows, sorted for + a deterministic file. A text tagged with TWO shapes (the + family-first divergence pairs) is two pairs: two orders are two + comparisons.""" + return sorted({(case.shape, case.text) for case in CASES + if case.shape is not None}) + + +def selected() -> list[dict[str, object]]: + """The corpus lines, in file order.""" + return [{"name": text, "shape": shape} for shape, text in _pairs()] + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--coverage", action="store_true", + help="print names-per-shape and exit") + args = ap.parse_args() + if args.coverage: + by_shape = Counter(shape for shape, _ in _pairs()) + for shape in sorted(by_shape): + print(f"shape {shape}: {by_shape[shape]} names") + return + rows = selected() + with OUT.open("w", encoding="utf-8") as fh: + for row in rows: + fh.write(json.dumps(row, ensure_ascii=False) + "\n") + print(f"wrote {len(rows)} entries to {OUT.name}") + + +if __name__ == "__main__": + main() diff --git a/tools/differential/compare.py b/tools/differential/compare.py index a2a749c8..af6b092d 100644 --- a/tools/differential/compare.py +++ b/tools/differential/compare.py @@ -1,6 +1,10 @@ """Differential harness: a released baseline vs -the working tree over the corpora. Every diff must classify against -that baseline's ledger or the run fails. +the working tree over the corpora. Every diff on a CONTRACT corpus +must classify against that baseline's ledger or the run fails; a +RADAR corpus reports its unmatched diffs instead of failing on them. +A `[[never]]` exclusion is fatal on EITHER tier -- it was chosen, the +same way a rule was, so it belongs to the contract regardless of +which corpus the name it refuses happens to sit in. uv run python tools/differential/compare.py [--baseline VERSION] @@ -11,6 +15,7 @@ exit code with tail's, so a failing run reads as a passing one. """ import argparse +import importlib.util import json import os import re @@ -39,11 +44,13 @@ _V1_TO_ROLE = {"first": "given", "last": "family"} #: An unclassified diff, carrying BOTH surfaces' before/after: -#: (name, old_facade, new_facade, old_v2, new_v2). Both halves are kept -#: because a diff can exist on the v2 surface alone, and a report that -#: named such a diff without showing it would be unactionable. +#: (name, old_facade, new_facade, old_v2, new_v2, order). Both halves +#: are kept because a diff can exist on the v2 surface alone, and a +#: report that named such a diff without showing it would be +#: unactionable. `order` rides along so the report can say which +#: order produced the diff -- None for the default order. _Unexplained = tuple[str, dict[str, str], dict[str, str], - dict[str, object], dict[str, object]] + dict[str, object], dict[str, object], str | None] def _parse_version(text: str) -> tuple[int, int, int]: @@ -101,6 +108,23 @@ def _allowlist_for(version: str) -> Path: return path +_SHAPES_PATH = Path(__file__).resolve().parent / "shapes.py" + + +def _load_shapes() -> dict: + """SHAPES from shapes.py beside this script, loaded by path so it + works both run-as-script and under tests' load_tool. Resolved + against the script's own directory, not HERE: shapes.py is code, + not corpus data -- a --corpus override or a test's HERE patch must + not change which inventory loads.""" + spec = importlib.util.spec_from_file_location( + "differential_shapes", _SHAPES_PATH) + assert spec is not None and spec.loader is not None, _SHAPES_PATH + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module.SHAPES + + def _sorted_rules(rules: list[dict[str, object]]) -> list[dict[str, object]]: """Most-specific-first: a name_regex rule outranks a fields-only rule wherever both match, so file order does not decide BETWEEN THE @@ -140,11 +164,18 @@ def _sorted_rules(rules: list[dict[str, object]]) -> list[dict[str, object]]: there, not a copy of this. Writes a VERSION TELL as its first stdout line, then one result per -input name. The tell exists because the alternative failure is +input entry. The tell exists because the alternative failure is invisible: a worker that silently resolved to the checkout answers every query as the working tree while the run is labelled with the baseline, so every diff vanishes and the run reports parity -- the precise opposite of the truth. + +Input lines are {"name": ..., "order": null | ""}. An order +names a public nameparser constant; only workers whose baseline +supports it are ever sent one (compare.py skips below the shape's +min_baseline), and an order-bearing entry is compared on the v2 +surface alone -- the facade is the v1-compat surface, and a +family-first name is not a v1 contract. """ import json import sys @@ -162,22 +193,43 @@ def _sorted_rules(rules: list[dict[str, object]]) -> list[dict[str, object]]: "__file__": nameparser.__file__}), flush=True) if WANT_V2: - from nameparser import parse + from nameparser import Parser, Policy, parse + _parsers = {} + + def _parser_for(order): + if order not in _parsers: + _parsers[order] = Parser(policy=Policy( + name_order=getattr(nameparser, order))) + return _parsers[order] + + +def _v2_row(p): + # must stay identical to main()'s tree-side row (compare.py) -- + # duplicated rather than shared across the process boundary + row = {f: (getattr(p, f, "") or "") for f in V2_FIELDS} + row["_ambiguities"] = sorted( + {a.kind.name for a in getattr(p, "ambiguities", ())}) + return row + for line in sys.stdin: line = line.strip() if not line: continue - name = json.loads(line) + entry = json.loads(line) + name = entry["name"] + order = entry.get("order") + if order is not None: + # never reaches a worker without WANT_V2; see the docstring + print(json.dumps( + {"v2": _v2_row(_parser_for(order).parse(name))}, + ensure_ascii=False), flush=True) + continue row = {"facade": {k: v or "" for k, v in HumanName(name).as_dict().items() if k in V1_FIELDS}} if WANT_V2: - p = parse(name) - v2 = {f: (getattr(p, f, "") or "") for f in V2_FIELDS} - v2["_ambiguities"] = sorted( - {a.kind.name for a in getattr(p, "ambiguities", ())}) - row["v2"] = v2 + row["v2"] = _v2_row(parse(name)) print(json.dumps(row, ensure_ascii=False), flush=True) ''' @@ -283,7 +335,7 @@ def _check_tree(module_file: str) -> Path: def _run_worker(version: str, want_v2: bool, - names: list[str]) -> tuple[dict[str, str], list[dict]]: + entries: list[dict]) -> tuple[dict[str, str], list[dict]]: """Run the baseline worker from a temp dir OUTSIDE the worktree. The placement is the safety mechanism, not plumbing. uv reads @@ -301,8 +353,10 @@ def _run_worker(version: str, want_v2: bool, ["uv", "run", "--no-project", str(script)], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, cwd=tmp, env=_worker_env()) - payload = "".join(json.dumps(n, ensure_ascii=False) + "\n" - for n in names) + payload = "".join( + json.dumps({"name": e["name"], "order": e.get("order")}, + ensure_ascii=False) + "\n" + for e in entries) out, _ = proc.communicate(payload) # hard checks, not asserts: -O must not turn a crashed worker into # a truncated-but-green comparison @@ -317,10 +371,10 @@ def _run_worker(version: str, want_v2: bool, tell = json.loads(lines[0]) _check_tell(tell, version) results = [json.loads(x) for x in lines[1:]] - if len(results) != len(names): + if len(results) != len(entries): raise SystemExit( - f"worker returned {len(results)} results for {len(names)} " - f"corpus names; comparison aborted") + f"worker returned {len(results)} results for {len(entries)} " + f"corpus entries; comparison aborted") return tell, results @@ -331,6 +385,52 @@ def _canonical_field(field: str) -> str: return _V1_TO_ROLE.get(field, field) +def _order_tag(order: str | None) -> str: + """The ` [order: X]` header suffix, or "" for a default-order + entry. Shared by the UNEXPLAINED and UNCLASSIFIED (radar) headers + in main() -- one copy for the same reason _print_field_diffs is: + two copies can drift.""" + return f" [order: {order}]" if order is not None else "" + + +def _print_field_diffs(old_facade: dict[str, str], new: dict[str, str], + old_v2: dict[str, object], + new_v2: dict[str, object], + order: str | None = None) -> None: + """Print each moved field under one name, Role's, whichever + surface(s) moved it. Shared by the UNEXPLAINED and UNCLASSIFIED + (radar) blocks in main() -- one copy rather than two that can + drift apart, the same reason _entry_matches is "one predicate + rather than three copies". + + Role's names, not the facade's: both report blocks exist to be + turned into a ledger rule, and a rule naming the facade's `first` + is rejected by validate_rules at startup. Both surfaces are + walked, and a field is reported once even when both moved, since + one rule covers it. + + `order` is None for a default-order entry, else the order both + surfaces were read under. The "[v2 surface only]" tag means "the + facade was compared and agreed" -- untrue for an order-bearing + entry, whose facade is never consulted at all (main() passes empty + dicts for it), so the tag is suppressed there rather than printed + with a meaning it does not have. + """ + seen: set[str] = set() + for f in FIELDS: + if old_facade.get(f, "") != new.get(f, ""): + seen.add(_canonical_field(f)) + print(f" {_canonical_field(f)}: " + f"{old_facade.get(f, '')!r} -> {new.get(f, '')!r}") + for f in (*V2_FIELDS, "_ambiguities"): + if old_v2.get(f, "") != new_v2.get(f, "") \ + and _canonical_field(f) not in seen: + tag = "" if order is not None else " [v2 surface only]" + print(f" {_canonical_field(f)}: " + f"{old_v2.get(f, '')!r} -> {new_v2.get(f, '')!r}" + f"{tag}") + + def _is_latin_only(name: str) -> bool: """Every character below U+0250 -- Latin, ASCII punctuation and Latin-1 accents. @@ -348,7 +448,39 @@ def _is_latin_only(name: str) -> bool: #: ambiguity entry is legal and load-bearing -- a SEGMENTATION-only diff #: is facade-identical, so this is the one name that can classify it. _RULE_FIELDS = frozenset((*V2_FIELDS, "_ambiguities")) -_RULE_KEYS = frozenset(("issue", "name_regex", "fields", "dormant")) +_RULE_KEYS = frozenset(("issue", "name_regex", "fields", "dormant", "orders")) + + +#: The `orders` member naming the DEFAULT order -- the comparison whose +#: `order` is None, run under no declared name_order at all. A sentinel +#: rather than a constant name because there is no constant to borrow: +#: the default order is the absence of one, no shape declares it, and +#: TOML has no null to put inside an array. Without it a rule that +#: explains only default-order diffs cannot say so and has to stay +#: order-blind, which is the leak running the OTHER way from the one +#: `orders` was added for: an order-blind rule sorted ahead of the +#: scoped ones absorbs a family-first regression on a name its regex +#: happens to reach. +_DEFAULT_ORDER = "DEFAULT" + + +def _legal_orders() -> frozenset[str]: + """The names a rule's `orders` may carry: every order shapes.py's + inventory declares, plus the "DEFAULT" sentinel. + + The constants are borrowed rather than hand-copied, the same way + build_cjk_corpus.py borrows the script table: an order no shape + declares is an order no comparison can run under, so a rule scoped + to it could only ever be dormant, and a typo in one would be a rule + that silently explains nothing. + + "DEFAULT" is the one member that cannot be borrowed, since it names + the absence of a declared order rather than a shape. + """ + return frozenset(shape.order for shape in _load_shapes().values() + if shape.order is not None) | {_DEFAULT_ORDER} + + #: Probe names for the over-match check, chosen to share no script, no #: vocabulary and no punctuation. A `name_regex` matching ALL of them is #: not targeting a behavior family, it is matching everything -- and @@ -373,7 +505,33 @@ def _is_latin_only(name: str) -> bool: "corpus.jsonl": 480, # 486 today, from v1's banks at a pinned ref "corpus_cjk.jsonl": 95, # 98 today, generated from the case table "corpus_issues.jsonl": 370, # 381 today, harvested and append-only - "corpus_rules.jsonl": 150, # 241 today, generated from rules.md + "corpus_rules.jsonl": 150, # 252 today, generated from rules.md + "corpus_shapes.jsonl": 11, # 13 today, generated from shape-tagged + # case rows +} + +#: Tier per corpus file, fail-closed like the floors above. CONTRACT +#: corpora hold names someone chose -- an unmatched diff on one is +#: UNEXPLAINED and fails the run, today's discipline. RADAR corpora +#: hold scraped/harvested names (#468): their diffs still classify +#: against the ledger (release notes want the grouping) but an +#: unmatched one prints under UNCLASSIFIED (radar) and cannot fail +#: the run or demand a rule. Promotion is a cases.py row plus a +#: shape tag -- a name enters the contract by being chosen. +#: +#: A `[[never]]` exclusion is the same kind of choice, and stays fatal +#: on a name in a radar file for exactly that reason: someone wrote +#: the entry, its `why`, and its `examples`, so the shape it refuses +#: was chosen the same way a rule is -- unlike the rest of a radar +#: file, which nobody has looked at name by name. The tier split +#: governs names nobody chose; a [[never]] entry is the opposite of +#: that, so it outranks the tier the name happens to sit in. +_CORPUS_TIERS = { + "corpus.jsonl": "radar", + "corpus_cjk.jsonl": "contract", + "corpus_issues.jsonl": "radar", + "corpus_rules.jsonl": "contract", + "corpus_shapes.jsonl": "contract", } @@ -464,6 +622,32 @@ def validate_rules(rules: list[dict[str, object]], ledger: str) -> None: f"is expected to explain nothing, and the reason is " f"the whole safeguard -- an exemption nobody can " f"justify means the rule should be deleted instead") + if "orders" in rule: + orders = rule["orders"] + if not isinstance(orders, list) \ + or not all(isinstance(o, str) for o in orders): + raise SystemExit( + f"{where} has an 'orders' that is not a list of " + f"strings ({orders!r}); _entry_matches would ignore " + f"it and the rule would go back to claiming a diff " + f"under EVERY order, which is the scoping this key " + f"exists to undo") + if not orders: + raise SystemExit( + f"{where} has an empty 'orders', which no comparison " + f"can be run under -- a rule that can never match. " + f"Omit the key to stay order-blind") + legal = _legal_orders() + bad = sorted(set(orders) - legal) + if bad: + raise SystemExit( + f"{where} names {bad} in 'orders', which shapes.py " + f"declares for no shape and which is not the " + f"{_DEFAULT_ORDER!r} sentinel; expected from " + f"{sorted(legal)}. No comparison runs under an order " + f"no shape asks for, so the rule would explain " + f"nothing and report as dormant instead of saying " + f"the name is wrong") has_regex, has_fields = "name_regex" in rule, "fields" in rule if not has_regex and not has_fields: raise SystemExit( @@ -657,19 +841,40 @@ def validate_exclusions(entries: list[dict[str, object]], def _entry_matches(rule: dict[str, object], name: str, - diff_fields: set[str]) -> bool: + diff_fields: set[str], order: str | None = None) -> bool: """Does this entry's narrowing admit this diff? Called twice in classify() -- once for exclusions, once for rules -- - and again in dormant_rules(). All three narrow on the same two keys, + and again in dormant_rules(). All three narrow on the same keys, and the dormancy diagnosis is only meaningful if it asks the question classify asks, so there is one predicate rather than three copies of it. - A non-str `name_regex` or non-list `fields` is IGNORED rather than - rejected here: validate_rules and validate_exclusions reject both at - startup, and duplicating that judgement in the hot path would put the - two in a position to disagree. + `order` is the name_order the COMPARISON ran under (None = the + default order), and a rule carrying `orders` admits only the orders + it lists. A comparison under the default order is matched by the + "DEFAULT" sentinel, there being no constant to name and no null to + put in a TOML array. Without that narrowing a rule is order-blind, + which is what every rule written before shape-tagged entries + existed is: the key is optional and its absence is today's + behavior -- and the absence is not free, since an order-blind rule + reaching an order-bearing name absorbs that name's order-only + regressions (main() prints an ORDER-BLIND notice where it sees + that happen). It matters + because a name can be compared twice, once per order, and the two + diffs can have the SAME fields for opposite reasons -- the + feat(#395) fold moving {family, given, middle} under a declared + family-first order is intended, and the same three roles moving on + the same string under the DEFAULT order would be that fold leaking + where it must not, which an order-blind rule would absorb and call + intentional (#372's failure mode, on the most plausible regression + of the very change the rule describes). + + A non-str `name_regex`, non-list `fields` or non-list `orders` is + IGNORED rather than rejected here: validate_rules and + validate_exclusions reject them at startup, and duplicating that + judgement in the hot path would put the two in a position to + disagree. """ name_regex = rule.get("name_regex") if isinstance(name_regex, str) and not re.search(name_regex, name): @@ -677,12 +882,17 @@ def _entry_matches(rule: dict[str, object], name: str, fields = rule.get("fields") if isinstance(fields, list) and not diff_fields <= set(fields): return False + orders = rule.get("orders") + if isinstance(orders, list) \ + and (_DEFAULT_ORDER if order is None else order) not in orders: + return False return True def classify(name: str, diff_fields: set[str], rules: list[dict[str, object]], - exclusions: list[dict[str, object]] | None = None) -> str | None: + exclusions: list[dict[str, object]] | None = None, + order: str | None = None) -> str | None: """Which rule explains this diff, or None if nothing does. Exclusions are consulted FIRST and win outright. They are the @@ -704,12 +914,21 @@ def classify(name: str, diff_fields: set[str], name whose parens mark a nickname to one rule and a suffix to another stays classifiable on the reading the exclusion is not about. + + `order` is the name_order this comparison ran under and narrows the + RULES alone: exclusions have no `orders` key (validate_exclusions + rejects one as unknown), so they stay order-blind, deliberately. + An over-wide exclusion is loud rather than silent -- refusal is + monotone, so the widest thing it can do is make a name report + UNEXPLAINED and fail the run -- and no exclusion on the books + protects a shape that reads differently under a declared order. The + key can be given to them the day one does. """ for entry in exclusions or (): if _entry_matches(entry, name, diff_fields): return None for rule in rules: - if _entry_matches(rule, name, diff_fields): + if _entry_matches(rule, name, diff_fields, order): return rule["issue"] # type: ignore[return-value] return None @@ -750,7 +969,7 @@ class _Dormancy(NamedTuple): def dormant_rules(rules: list[dict[str, object]], explained: set[str], - diffing: list[tuple[str, set[str]]], + diffing: list[tuple[str, set[str], str | None]], exclusions: list[dict[str, object]] | None = None, ) -> _Dormancy: """Which rules explained nothing, and which kind of nothing. @@ -789,11 +1008,16 @@ def dormant_rules(rules: list[dict[str, object]], explained: set[str], continue if declared: continue - matched = [(n, d) for n, d in diffing - if _entry_matches(rule, n, d)] + # each diff carries the ORDER its comparison ran under, so a + # rule scoped by `orders` is asked the question classify asks + # it: a rule that would claim a name only under FAMILY_FIRST is + # not "shadowed" by whatever explains that name's default-order + # diff + matched = [(n, d, o) for n, d, o in diffing + if _entry_matches(rule, n, d, o)] winners = Counter( - c for c in (classify(n, d, ordered, exclusions) - for n, d in matched) if c is not None) + c for c in (classify(n, d, ordered, exclusions, o) + for n, d, o in matched) if c is not None) if not matched: undeclared.append(_Dormant(issue, "reverted", "")) elif winners: @@ -890,6 +1114,86 @@ def over_declared_rules( return tuple(found) +def _load_entries(path: Path) -> list[dict[str, object]]: + """Corpus lines as entry dicts. A line is either a bare JSON + string (the original format) or an object with a "name" plus + optional metadata -- "tests" labels from build_corpus.py, and a + "shape" id from build_shapes_corpus.py (#469). Tolerating both + means compare.py itself never needs a flag day across its five + corpus files: corpus.jsonl and corpus_shapes.jsonl carry object + lines, the other three are still bare strings, and both shapes + stay legal everywhere a corpus line is read. + + A "tests" label is read only when the radar block prints, well + after the multi-minute worker pass, so a malformed one left + unchecked would crash there rather than here -- exactly what + validate_rules' compile-at-startup paragraph exists to prevent. + + "shape" is checked for a different hazard. main() resolves it + against shapes.py into the "order" the worker protocol sends, and + that loop runs BEFORE the worker, so a bad id is not a late crash: + it is a wrong comparison that reports as a passing one. `true` + passes isinstance(shape, int) and hash(True) == hash(1), so an + unchecked one resolves against shapes.py's entry 1 and the line is + compared under that shape's order, silently. Only the TYPE is + checked here; an unresolvable id is main()'s to catch, since only + it has shapes.py loaded. + + Unknown keys are rejected the way validate_rules rejects them, and + for the same reason: a misspelled key is not ignored, it drops the + narrowing the line meant to declare, and the line then compares + under the default order with nothing saying so. "order", "tier" and + "file" are rejected rather than obeyed -- the comparison computes + all three per entry and overwrites whatever a line said, so a line + writing one would be silently discarded. "order" in particular is + the key the WIRE protocol documents, which makes it the one a + corpus author is likeliest to reach for. + """ + allowed = {"name", "tests", "shape"} + entries: list[dict[str, object]] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + if not line.strip(): + continue + raw = json.loads(line) + if isinstance(raw, str): + entries.append({"name": raw}) + elif isinstance(raw, dict) and isinstance(raw.get("name"), str): + unknown = sorted(set(raw) - allowed) + if unknown: + raise SystemExit( + f"{path.name}: a corpus line has unknown key(s) " + f"{unknown}; expected only {sorted(allowed)}: " + f"{line!r}. A misspelled key is not ignored -- it " + f"drops the narrowing the line declares, and the " + f"name is then compared under the default order " + f"with nothing saying so. 'order', 'tier' and " + f"'file' are computed by the comparison itself, so " + f"a line writing one would be overwritten") + tests = raw.get("tests") + if tests is not None and ( + not isinstance(tests, list) + or not all(isinstance(t, str) for t in tests)): + raise SystemExit( + f"{path.name}: a corpus line's 'tests' must be a " + f"list of strings, not {tests!r}: {line!r}") + shape = raw.get("shape") + # bool is an int subclass in Python, and hash(True) == hash(1) + # -- unexcluded, {"shape": true} would pass isinstance(shape, + # int) and silently resolve to shape 1's order + if shape is not None and ( + not isinstance(shape, int) or isinstance(shape, bool)): + raise SystemExit( + f"{path.name}: a corpus line's 'shape' must be an " + f"int naming a shapes.py entry, not {shape!r}: " + f"{line!r}") + entries.append(dict(raw)) + else: + raise SystemExit( + f"{path.name}: corpus line is neither a JSON string " + f"nor an object with a string 'name': {line!r}") + return entries + + def main() -> int: ap = argparse.ArgumentParser() # Every corpus by default: they have different blind spots (see @@ -937,12 +1241,23 @@ def main() -> int: f"{missing}. A corpus that vanishes shrinks the " f"comparison silently -- restore it, or drop its floor " f"if it is meant to be gone") + # Contract files load FIRST so the (name, order) dedup below keeps + # the contract reading of a string both tiers hold. + paths = sorted(paths, key=lambda p: ( + _CORPUS_TIERS.get(p.name) != "contract", p.name)) per_file = {} - corpus = [] + entries: list[dict[str, object]] = [] for path in paths: - names = [json.loads(line) - for line in path.read_text().splitlines() if line.strip()] - if not names: + tier = _CORPUS_TIERS.get(path.name) + if tier is None: + raise SystemExit( + f"{path.name} has no entry in _CORPUS_TIERS. Every " + f"corpus must choose: 'contract' (an unmatched diff " + f"fails the run) or 'radar' (an unmatched diff is " + f"reported and cannot fail). A default here would let " + f"a new corpus pick one by accident") + file_entries = _load_entries(path) + if not file_entries: raise SystemExit(f"{path.name} is empty; comparison aborted") floor = _CORPUS_FLOORS.get(path.name) if floor is None: @@ -951,21 +1266,86 @@ def main() -> int: f"a little under its size: without a floor a corpus can " f"shrink to a handful of names and the run still exits " f"0, having compared far less than it reports") - if len(names) < floor: + if len(file_entries) < floor: + raise SystemExit( + f"{path.name} holds {len(file_entries)} names, below its " + f"floor of {floor} -- it has shrunk or been truncated. " + f"The run would still exit 0 while comparing a fraction " + f"of what it claims. Restore the file, or lower the " + f"floor deliberately if names were removed on purpose") + per_file[path.name] = len(file_entries) + for e in file_entries: + e["tier"] = tier + e["file"] = path.name + entries.extend(file_entries) + # resolve each entry's optional "shape" to the "order" the worker + # protocol actually sends -- a public nameparser constant name, or + # None for the default order + shapes_by_id = _load_shapes() + for e in entries: + shape = e.get("shape") + if shape is None: + e["order"] = None + continue + if shape not in shapes_by_id: raise SystemExit( - f"{path.name} holds {len(names)} names, below its floor " - f"of {floor} -- it has shrunk or been truncated. The run " - f"would still exit 0 while comparing a fraction of what " - f"it claims. Restore the file, or lower the floor " - f"deliberately if names were removed on purpose") - per_file[path.name] = len(names) - corpus.extend(names) - # dedupe across files, keeping first-seen order stable for output - corpus = list(dict.fromkeys(corpus)) + f"corpus entry {e['name']!r} declares shape {shape!r}, " + f"which shapes.py does not define") + e["order"] = shapes_by_id[shape].order + # dedupe on (name, order): the same string tagged with two shapes + # is two comparisons, not a duplicate -- each order is compared + # under its own reading. First-seen wins, and contract files were + # loaded first. The min-baseline skip below reads the SURVIVOR's + # shape, which is safe only while min_baseline is a function of + # order alone (true today, since every order-bearing shape's + # minimum is 2.0.0) -- a future order-None shape carrying a higher + # minimum than an order-bearing duplicate would make survival, and + # so the skip decision, depend on which file loaded first. + by_key: dict[tuple[str, str | None], dict[str, object]] = {} + for e in entries: + by_key.setdefault((e["name"], e.get("order")), e) + entries = list(by_key.values()) + # an order-bearing entry must never reach a worker whose baseline + # cannot honor it (no Policy below 2.0.0) -- skip it and say so, + # rather than shrink the comparison silently. Skips are also + # counted PER FILE: per_file above records pre-skip counts, so a + # shapes corpus fully skipped at an old baseline would otherwise + # print at full size while contributing nothing. + kept = [] + dropped = 0 + dropped_by_file: dict[str, int] = {} + dropped_shape_ids: set[int] = set() + dropped_minimums: set[str] = set() + for e in entries: + shape = e.get("shape") + if shape is not None and _parse_version(baseline) \ + < _parse_version(shapes_by_id[shape].min_baseline): + dropped += 1 + dropped_by_file[e["file"]] = dropped_by_file.get(e["file"], 0) + 1 + dropped_shape_ids.add(shape) + dropped_minimums.add(shapes_by_id[shape].min_baseline) + continue + kept.append(e) + if dropped: + ids = ", ".join(str(i) for i in sorted(dropped_shape_ids)) + minimums = ", ".join(sorted(dropped_minimums, key=_parse_version)) + print(f"skipped {dropped} name{'s' if dropped > 1 else ''} " + f"tagged shape(s) [{ids}]: baseline {baseline} predates " + f"their minimum ({minimums})") + entries = kept + corpus = [e["name"] for e in entries] # per-file counts, not just the total: a corpus that shrinks or - # vanishes is only visible if its own number is printed - print("corpora: " + ", ".join(f"{name} ({n})" - for name, n in per_file.items())) + # vanishes is only visible if its own number is printed. A file + # with skips ALSO prints its skip count, not just its final size -- + # otherwise a shapes corpus skipped to zero reads as a corpus that + # was simply never that large. N is pre-dedup and K counts only + # the baseline-minimum skip, so N - K is NOT "how many from this + # file were compared" -- an entry the cross-file dedup dropped is + # in neither number. + print("corpora: " + ", ".join( + f"{name} ({n}, {dropped_by_file[name]} skipped)" + if dropped_by_file.get(name) else f"{name} ({n})" + for name, n in per_file.items())) # The tree is checked BEFORE the worker runs. It depends on nothing # the worker produces, and validate_rules' own reasoning applies: a @@ -978,13 +1358,29 @@ def main() -> int: want_v2 = "v2" in surfaces if want_v2: - from nameparser import parse - tell, old_rows = _run_worker(baseline, want_v2, corpus) + from nameparser import Parser, Policy, parse + tree_parsers: dict[str, object] = {} + + def _tree_parse(name: str, order: str | None) -> object: + if order is None: + return parse(name) + if order not in tree_parsers: + tree_parsers[order] = Parser(policy=Policy( + name_order=getattr(nameparser, order))) + return tree_parsers[order].parse(name) + + tell, old_rows = _run_worker(baseline, want_v2, entries) print(f"baseline: nameparser {tell['__version__']} ({tell['__file__']})") - by_issue: dict[str, list[str]] = {} + #: (name, order) pairs, not names: one string compared under two + #: orders is two entries, and the report renders the order tag from + #: the pair. Rendering at print time rather than storing the line + #: keeps the bare name available to the Latin-only stat below, + #: which would otherwise need a second list to drift out of step + #: with this one. + by_issue: dict[str, list[tuple[str, str | None]]] = {} #: the union of the diffs each rule explained, for over_declared_rules. #: Kept beside by_issue rather than inside it: the summary printout - #: and `changed` both read by_issue as a list of names. + #: reads by_issue as a list of pairs. roles_by_issue: dict[str, set[str]] = {} # BOTH surfaces' old/new are retained, not just the facade's. A diff # can exist on the v2 surface alone -- an _ambiguities-only change is @@ -993,47 +1389,105 @@ def main() -> int: # facade dicts would print such a name under UNEXPLAINED with no # field lines under it at all: a failure nobody can act on. unexplained: list[_Unexplained] = [] - # every name that diffed, with its diff, so dormant_rules can ask - # which rule WOULD have claimed one that no rule did - diffing: list[tuple[str, set[str]]] = [] - for name, old in zip(corpus, old_rows): - new = {k: v or "" for k, v in HumanName(name).as_dict().items() - if k in FIELDS} - # canonicalized on the way in: the ledger speaks Role's names, - # and the facade is the surface whose vocabulary differs - diff = {_canonical_field(f) for f in FIELDS - if old["facade"].get(f, "") != new.get(f, "")} + # radar-tier equivalent of unexplained: reported, never fatal (#468) + radar: list[tuple[dict, _Unexplained]] = [] + # every name that diffed, with its diff AND the order its + # comparison ran under, so dormant_rules can ask which rule WOULD + # have claimed one that no rule did -- the same question classify + # asks, order included + diffing: list[tuple[str, set[str], str | None]] = [] + #: (issue, name, order) for each diff an order-blind rule explained + #: under a declared order. Informational, never fatal. + order_blind: list[tuple[str, str, str]] = [] + #: classify() returns an issue; the notice needs the rule behind it. + #: Keyed by issue because validate_rules has already refused two + #: rules sharing one. + rules_by_issue = {str(r["issue"]): r for r in rules} + for entry, old in zip(entries, old_rows): + name = entry["name"] + order = entry.get("order") + if order is None: + new = {k: v or "" for k, v in HumanName(name).as_dict().items() + if k in FIELDS} + # canonicalized on the way in: the ledger speaks Role's + # names, and the facade is the surface whose vocabulary + # differs + diff = {_canonical_field(f) for f in FIELDS + if old["facade"].get(f, "") != new.get(f, "")} + else: + # order-bearing entries are compared on the v2 surface + # alone -- the facade is the v1-compat surface, and a + # family-first name is not a v1 contract + new = {} + diff = set() new_v2: dict[str, object] = {} if want_v2: - p = parse(name) + p = _tree_parse(name, order) + # must stay identical to the worker template's _v2_row + # (_WORKER_TEMPLATE, this file) -- duplicated rather than + # shared across the process boundary new_v2 = {f: (getattr(p, f, "") or "") for f in V2_FIELDS} new_v2["_ambiguities"] = sorted( {a.kind.name for a in getattr(p, "ambiguities", ())}) diff |= {_canonical_field(f) for f in (*V2_FIELDS, "_ambiguities") - if old["v2"].get(f, "") != new_v2.get(f, "")} + if old.get("v2", {}).get(f, "") != new_v2.get(f, "")} if not diff: continue - diffing.append((name, diff)) - issue = classify(name, diff, rules, exclusions) + diffing.append((name, diff, order)) + issue = classify(name, diff, rules, exclusions, order) if issue is None: - unexplained.append( - (name, old["facade"], new, old.get("v2", {}), new_v2)) + row = (name, old.get("facade", {}), new, old.get("v2", {}), + new_v2, order) + # classify() returns None for two different reasons: no + # rule matched, or a [[never]] entry refused the name -- + # and only the first belongs to the tier split. An + # exclusion was chosen (see _CORPUS_TIERS), so it is fatal + # on a radar name exactly as it is on a contract one. + excluded = any(_entry_matches(x, name, diff) + for x in exclusions) + if entry["tier"] == "radar" and not excluded: + radar.append((entry, row)) + else: + unexplained.append(row) else: - by_issue.setdefault(issue, []).append(name) + by_issue.setdefault(issue, []).append((name, order)) roles_by_issue.setdefault(issue, set()).update(diff) - - changed = [n for names in by_issue.values() for n in names] \ - + [row[0] for row in unexplained] + if order is not None and "orders" not in rules_by_issue[issue]: + order_blind.append((issue, name, order)) + + # the bare halves of by_issue's pairs: _is_latin_only reads the + # string as a name, so it must never see the rendered order tag. A + # string compared under two orders counts twice here, accepted. + changed = [n for pairs in by_issue.values() for n, _ in pairs] \ + + [row[0] for row in unexplained] \ + + [e["name"] for e, _ in radar] latin = sum(1 for n in changed if _is_latin_only(n)) print(f"corpus: {len(corpus)} names; " f"intentional diffs: {sum(map(len, by_issue.values()))}; " f"unexplained: {len(unexplained)}; " + f"radar unclassified: {len(radar)}; " f"{latin} of {len(changed)} changed names are Latin-only\n") for issue, names in sorted(by_issue.items()): print(f"## {issue} ({len(names)})") - for n in names[:10]: - print(f" {n!r}") + for n, o in names[:10]: + print(f" {n!r}{_order_tag(o)}") + print() + # Informational, and deliberately outside the exit code: an + # order-blind rule is legal, and every rule written before shape + # tags is one. What the block buys is that the absorption stops + # being invisible -- a rule sorted ahead of the scoped ones can + # reach an order-bearing name its author never considered, and an + # order-only regression on that name would then classify as an + # intentional change. + if order_blind: + print("ORDER-BLIND (informational, not in the exit code): a rule " + "carrying no `orders` key explained a diff from an " + "order-bearing comparison. Consider scoping it with " + "`orders` -- including the \"DEFAULT\" sentinel if it " + "explains default-order diffs too.\n") + for issue, name, tagged in order_blind: + print(f" {issue!r} explained {name!r}{_order_tag(tagged)}") print() dormancy = dormant_rules(rules, set(by_issue), diffing, exclusions) for dormant in dormancy.undeclared: @@ -1066,30 +1520,27 @@ def main() -> int: if args.corpus else "")) if overwide: print() - if unexplained: + # the radar rows below print the same Role-named field lines, so + # the legend belongs to both blocks or the radar reader is told + # nothing about the vocabulary they are reading + if unexplained or radar: print("Field names below are Role's, matching what a ledger " "`fields` rule must say.\n") - for name, old_facade, new, old_v2, new_v2 in unexplained: - print(f"UNEXPLAINED {name!r}") - # Role's names, not the facade's: this block exists to be turned - # into a ledger rule, and a rule naming the facade's `first` - # is rejected by validate_rules at startup. (Before that guard - # existed it parsed, validated and silently never matched -- - # which is why the label printed here has to be the label a - # rule needs.) Both surfaces are walked, and a field is - # reported once even when both moved, since one rule covers it. - seen: set[str] = set() - for f in FIELDS: - if old_facade.get(f, "") != new.get(f, ""): - seen.add(_canonical_field(f)) - print(f" {_canonical_field(f)}: " - f"{old_facade.get(f, '')!r} -> {new.get(f, '')!r}") - for f in (*V2_FIELDS, "_ambiguities"): - if old_v2.get(f, "") != new_v2.get(f, "") \ - and _canonical_field(f) not in seen: - print(f" {_canonical_field(f)}: " - f"{old_v2.get(f, '')!r} -> {new_v2.get(f, '')!r}" - f" [v2 surface only]") + for name, old_facade, new, old_v2, new_v2, order in unexplained: + # the order tag distinguishes a family-first regression from a + # default-order one on the same name -- otherwise indistinguishable + # in the report + print(f"UNEXPLAINED {name!r}{_order_tag(order)}") + _print_field_diffs(old_facade, new, old_v2, new_v2, order) + if radar: + print("\nRadar tier (scraped/harvested names, #468): shown, " + "never blocking. Promote a name that matters via a " + "cases.py row + shape tag.\n") + for entry, (name, old_facade, new, old_v2, new_v2, order) in radar: + labels = entry.get("tests") + tag = f" [v1: {', '.join(labels)}]" if labels else "" + print(f"UNCLASSIFIED (radar) {name!r}{tag}{_order_tag(order)}") + _print_field_diffs(old_facade, new, old_v2, new_v2, order) # A rule explaining nothing is as much a broken contract as an # unexplained diff: both mean the ledger no longer describes what the # code does. A rule explaining LESS than it declares is the third diff --git a/tools/differential/corpus.jsonl b/tools/differential/corpus.jsonl index b057779b..fa34a003 100644 --- a/tools/differential/corpus.jsonl +++ b/tools/differential/corpus.jsonl @@ -1,486 +1,486 @@ -"" -"\"Rick\" Edmonds" -"()" -"," -", John" -"A.B. Vajpayee" -"Abdul Salam Ahmed Salem, MD" -"Abdul Salam Hassan, MD" -"Abdul Salam, MD" -"Abu Bakr Al Baghdadi, MD" -"Adolph D" -"Ahmad ben Husain" -"Al Arnold Gore, Jr." -"Alex Ben Johnson" -"Alex van Johnson" -"Aliyev Vusal Said oglu" -"Alois von und zu Liechtenstein" -"Alois von und zu und von Liechtenstein" -"Amy E Maid" -"Amy E. Maid" -"Andrew" -"Andrew Boris Petersen" -"Andrew Perkins \"MBA\"" -"Andrew Perkins 'MBA'" -"Andrew Perkins (JD)" -"Andrew Perkins (M.D)" -"Andrew Perkins (MBA)" -"Andrew Perkins (Mgr.)" -"Andrew Perkins (XYZ)" -"Andrew Perkins, Jr., Col. (Ret)" -"Andrews, M.D." -"Anh Do" -"Annette Charlotte Freiherrin von und zu der Tann-Rathsamhausen" -"Assoc Dean of Chemistry Robert Johns" -"Baker (Johnson), Jenny" -"Baker (Jr.), Jenny" -"Ben Alex Johnson" -"Ben Johnson" -"Benjamin \"Ben\" Franklin" -"Benjamin \"Big Ben\" Franklin" -"Benjamin 'Ben' Franklin" -"Benjamin (Ben) Franklin" -"Benjamin (Big Ben) Franklin" -"Benjamin [Ben] Franklin" -"Bob Dole" -"Brian Andrew O'connor" -"Brian O'connor" -"Buca di Beppo" -"Cardinal Secretary of State Hillary Clinton" -"Chancellor Jane Smith" -"Chang, Andy C I" -"Charles van der van der Berg" -"Chemistry Jane Smith" -"Chief Judge J. Leon Holmes" -"Chief Judge Sharon Lovelace Blackburn" -"Clarke, Kenneth, Q.C. M.P." -"Coach" -"Col. (Ret.) Smith" -"DE MESNIL" -"DR DOE" -"Dame Mary" -"Dean Ms Hon Solo" -"Del Toro" -"Della Reese" -"Designated Judge David A. Ezra" -"Di Caprio" -"Doctor, Jane E." -"Doe, Dr. John" -"Doe, Dr. John A." -"Doe, Dr. John A. III" -"Doe, Dr. John A. Jr." -"Doe, Dr. John A. Kenneth" -"Doe, Dr. John A. Kenneth III" -"Doe, Dr. John A. Kenneth Jr." -"Doe, Dr. John III" -"Doe, Dr. John P., CLU, CFP, LUTC" -"Doe, Dr. John, Jr." -"Doe, John" -"Doe, John A." -"Doe, John A. III" -"Doe, John A. Kenneth" -"Doe, John A. Kenneth III" -"Doe, John A. Kenneth, Jr." -"Doe, John A., III" -"Doe, John A., Jr." -"Doe, John III" -"Doe, John Msc.Ed." -"Doe, John jr., MD" -"Doe, John, Jr." -"Doe, John, MD PhD - FACS Fellow" -"Doe, John,, Jr." -"Doe, John,, Jr.,, III" -"Doe, John. A. Kenneth" -"Doe, John. A. Kenneth III" -"Doe, John. A. Kenneth, Jr." -"Doe, Lt. Gen. John A. Kenneth IV" -"Doe, Lt.Gov. John" -"Doe, Mary - Kate, RN" -"Doe, Rev. John A. Jr." -"Doe, Rev. John A., V, Jr." -"Doe, Rev. John V, Jr." -"Doe,, Jr." -"Doe-Ray, Dr. John P., CLU, CFP, LUTC" -"Doe-Ray, Hon. Barrington P. Jr." -"Doe-Ray, Hon. Barrington P. Jr., CFP, LUTC" -"Donovan McNabb-Smith" -"Dr King Jr" -"Dr Martin Luther King, Jr." -"Dr. Abdul Salam Hassan, MD" -"Dr. John A. Doe" -"Dr. John A. Doe III" -"Dr. John A. Doe, Jr." -"Dr. John A. Kenneth Doe" -"Dr. John A. Kenneth Doe III" -"Dr. John A. Kenneth Doe, Jr." -"Dr. John Doe" -"Dr. John Doe III" -"Dr. John Doe, Jr." -"Dr. John P. Doe-Ray, CLU" -"Dr. John P. Doe-Ray, CLU, CFP, LUTC" -"Dr. John Smith" -"Dr. Juan Q. Velasquez y Garcia" -"Dr. Juan Q. Velasquez y Garcia III" -"Dr. Juan Q. Velasquez y Garcia, Jr." -"Dr. Juan Q. Xavier Velasquez y Garcia" -"Dr. Juan Q. Xavier Velasquez y Garcia III" -"Dr. Juan Q. Xavier Velasquez y Garcia, Jr." -"Dr. Juan Q. Xavier de la Vega" -"Dr. Juan Q. Xavier de la Vega III" -"Dr. Juan Q. Xavier de la Vega, Jr." -"Dr. Juan Q. Xavier de la dos Vega III" -"Dr. Juan Q. Xavier de la dos Vega, III" -"Dr. Juan Q. de la Vega" -"Dr. Juan Q. de la Vega III" -"Dr. Juan Q. de la Vega, Jr." -"Dr. Juan Velasquez y Garcia" -"Dr. Juan Velasquez y Garcia III" -"Dr. Juan Velasquez y Garcia, Jr." -"Dr. Juan de la Vega" -"Dr. Juan de la Vega III" -"Dr. Juan de la Vega, Jr." -"Dr. Martin Luther King Jr." -"Dr. Williams" -"Dr. abdul salam ahmed salem" -"Dra. Andréia da Silva" -"E.T. Smith" -"E.T. Smith, II" -"Esq Jane Smith" -"Foo. John Smith" -"Foo. Xyz. John Smith" -"Franklin Washington, Jr. MD" -"Franklin, Benjamin (Ben)" -"Franklin, Benjamin (Ben), Jr." -"Frau Anna Müller" -"Fritz Freiherr und von Bar" -"Frøken Jensen" -"GREGORY HOUSE M.D." -"Gunny de Mesnil" -"Harietta Keopuolani Nahi'ena'ena" -"Harrieta Keōpūolani Nāhiʻenaʻena" -"Her Majesty Queen Elizabeth" -"Herr Klaus Schmidt" -"Herr Schmidt" -"His Excellency Lord Duncan" -"Hon Solo" -"Hon. Barrington P. Doe-Ray, Jr." -"Hon. Charles J. Siragusa" -"Hon. Marian W. Payson" -"Honorable Judge Susan Russ Walker" -"Honorable Judge Terry F. Moorer" -"Honorable Judge W. Harold Albritton, III" -"Honorable Terry F. Moorer" -"Honorable W. Harold Albritton, III" -"Ivanov Ivan Ivanovich" -"J. Smith" -"JEFFREY (JD) BRICKEN" -"JOHN DOE" -"JOHN DOE PHD" -"JOHN DOE PHD MD" -"JOHN SMITH" -"JOSÉ GARCÍA" -"Jack Ma" -"Jack Ma Jr" -"Jane Doctor" -"Jane Doe" -"Jane Mac Beth" -"Jane Smith" -"Jean de Mesnil" -"Jenny \"JJ\" Baker (Johnson)" -"Jenny (Johnson) Baker" -"Jenny Baker" -"Jenny Baker (Johnson)" -"Jill St. John" -"Joao da Silva do Amaral de Souza" -"Joe Dentist D.D.S." -"Joe Franklin Jr" -"John & Jane" -"John A. Doe" -"John A. Doe III" -"John A. Doe, Jr" -"John A. Doe, Jr." -"John A. Kenneth Doe" -"John A. Kenneth Doe III" -"John A. Kenneth Doe, Jr." -"John Doe" -"John Doe III" -"John Doe MD PhD" -"John Doe Msc.Ed." -"John Doe jr., MD" -"John Doe, CLU, CFP, LUTC" -"John Doe, Jr." -"John Doe, Jr.,," -"John Doe, MD - PhD - FACS" -"John Doe, MD - PhD, FACS" -"John Doe, MD, PhD" -"John Doe, MD-PhD-" -"John Doe, Msc.Ed." -"John E Smith" -"John Edgar Casey Williams III" -"John Jones (Google Docs)" -"John Jones (Google Docs), Jr. (Unknown)" -"John Jones (Unknown)" -"John King" -"John Major. Smith" -"John P. Doe, CLU, CFP, LUTC" -"John Q. Smith" -"John Smith" -"John Smith Jr" -"John Smith Ph. D." -"John Smith VI" -"John Smith, Ph. D." -"John Smith, V Jr." -"John Smith, V MD" -"John W. Ingram, V" -"John W. Smith, I" -"John Williams" -"John and Jane Aznar y Lopez" -"John and Jane Smith" -"John e Smith" -"John e Smith III" -"John e Smith, III" -"John y Jane" -"Jon Dough and" -"Jon Dough and of" -"Jose Aznar y Lopez" -"Juan Q. Velasquez y Garcia" -"Juan Q. Velasquez y Garcia III" -"Juan Q. Velasquez y Garcia, Jr." -"Juan Q. Xavier Velasquez y Garcia" -"Juan Q. Xavier Velasquez y Garcia III" -"Juan Q. Xavier Velasquez y Garcia, Jr." -"Juan Q. Xavier de la Vega" -"Juan Q. Xavier de la Vega III" -"Juan Q. Xavier de la Vega, Jr." -"Juan Q. de la Vega" -"Juan Q. de la Vega III" -"Juan Q. de la Vega, Jr." -"Juan Velasquez y Garcia" -"Juan Velasquez y Garcia III" -"Juan Velasquez y Garcia, Jr." -"Juan de la Vega" -"Juan de la Vega III" -"Juan de la Vega, Jr." -"Juan de la de la Vega" -"Juan de la de la Vega Jr." -"Juan de la de la de la Vega" -"Juan del Sur" -"Judge C Lynwood Smith, Jr" -"Judge G. Thomas Eisele" -"Judge James M. Moody" -"Kenneth Clarke Q.C." -"Kenneth Clarke Q.C., M.P." -"Kenneth Clarke QC MP" -"King Henry" -"King John Alexander V" -"King John V." -"LT. GEN. JOHN A. KENNETH DOE IV" -"La'tanya O'connor" -"Larry James Johnson I" -"Larry V I" -"Lon (Jr.) Williams" -"Lord God Almighty" -"Lord and of the Universe" -"Lord of the Universe" -"Lord of the Universe and Associate Supreme Queen of the World Lisa Simpson" -"Lord of the Universe and Supreme King of the World Lisa Simpson" -"Lt. Gen. John A. Kenneth Doe IV" -"Lt. Gen. John A. Kenneth Doe, Jr." -"Lt.Gen. John A. Kenneth Doe IV" -"Lt.Gov. John Doe" -"Lt.Gov. juan e garcia" -"Ma III, Jack Jr" -"Ma, Jack" -"Mac Miller" -"Mag-Judge Harwell G Davis, III" -"Mag. Judge Byron G. Cudmore" -"Magistrate Judge John F. Forster, Jr" -"Magistrate-Judge Elizabeth Todd Campbell" -"Maid Marion" -"Maier, Amy I, Jr." -"Maier, Amy Lauren I" -"Major. Dona Smith" -"Major. John Smith, Jr." -"Mari' Aube'" -"Mevrouw Anna de Vries" -"Mike van der Velt" -"Mohamad Ahmad Ali Hassan" -"Mohamad Ali Khalil" -"Monsieur Jean Dupont" -"Mr. & Mrs. John Smith" -"Mr. Van Nguyen" -"Mr. and Mrs. John Smith" -"Mr. and Mrs. John and Jane Smith" -"Ms Hon Solo" -"Naomi Wambui Ng'ang'a" -"Nguyen, Van" -"No1. John Smith" -"None Smith" -"Nonez Smith" -"O'B. John Smith" -"Q R" -"Queen Elizabeth" -"RONALD MACDONALD" -"RONALD MCDONALD" -"Rafael Sousa dos Anjos" -"Rev Andrews" -"Rev John A. Kenneth Doe" -"Rev John A. Kenneth Doe III (Kenny)" -"Rev. John A. Kenneth Doe" -"Rt. Hon. Paul E. Mary" -"Sam Smith 😊" -"Secretary of State Hillary Clinton" -"Senator \"Rick\" Edmonds" -"Senior Judge Charles R. Butler, Jr" -"Senior Judge Harold D. Vietor" -"Senior Judge Virgil Pittman" -"Señor Carlos García" -"Señora María García" -"Shirley Maclaine" -"Signor Marco Rossi" -"Sir Gerald" -"Smith Jr., John" -"Smith van der" -"Smith, Dr. John" -"Smith, E.T., Jr." -"Smith, J.R." -"Smith, John" -"Smith, John I" -"Smith, John V" -"Smith, John e, III, Jr" -"Smith, MD - PhD - FACS" -"Smith, Major. John" -"Smith,John" -"Sr US District Judge Richard G Kopf" -"Srta. Andréia da Silva" -"Steven Hardman, RN - CRNA" -"Te Awanui-a-Rangi Black" -"The Lord of the Universe" -"The Right Hon. the President of the Queen's Bench Division" -"The Rt Hon John Jones" -"Title First Middle Middle Last, Jr." -"U. S. Grant" -"U.S. District Judge Marc Thomas Treadwell" -"US Magistrate Judge T Michael Putnam" -"US Magistrate-Judge Elizabeth E Campbell" -"VINCENT VAN GOGH" -"Va'apu'u Vitale" -"Van Jeremy Johnson" -"Van Johnson" -"Van Nguyen" -"Vega, Juan de la" -"Velasquez y Garcia, Dr. Juan" -"Velasquez y Garcia, Dr. Juan III" -"Velasquez y Garcia, Dr. Juan Q." -"Velasquez y Garcia, Dr. Juan Q. III" -"Velasquez y Garcia, Dr. Juan Q. Xavier" -"Velasquez y Garcia, Dr. Juan Q. Xavier III" -"Velasquez y Garcia, Dr. Juan Q. Xavier, Jr." -"Velasquez y Garcia, Dr. Juan Q., Jr." -"Velasquez y Garcia, Dr. Juan, Jr." -"Velasquez y Garcia, Juan" -"Velasquez y Garcia, Juan III" -"Velasquez y Garcia, Juan Q." -"Velasquez y Garcia, Juan Q. III" -"Velasquez y Garcia, Juan Q. Xavier" -"Velasquez y Garcia, Juan Q. Xavier III" -"Velasquez y Garcia, Juan Q. Xavier, Jr." -"Velasquez y Garcia, Juan Q., Jr." -"Velasquez y Garcia, Juan, Jr." -"Vincent van Gogh" -"Vincent van Gogh van Beethoven" -"Washington Jr. MD, Franklin" -"Xyz. (Bud) Smith" -"Yin Le" -"Yin a Le" -"Zephyrmark Jane Smith" -"abdul" -"abdul salam" -"abdul salam ahmed salem" -"abdul salam ahmed salem jr" -"abdul salam jr" -"abdul salam salem" -"abdulsalam ahmed salem" -"abu bakr al baghdadi" -"ahmed abu bakr" -"and Jon Dough" -"and van Buren" -"bob v. de la macdole-eisenhower phd" -"de" -"de Mesnil" -"de Mesnil Garcia" -"de Mesnil Jr." -"de la Vega" -"de la Vega, Dr. Juan" -"de la Vega, Dr. Juan III" -"de la Vega, Dr. Juan Q." -"de la Vega, Dr. Juan Q. III" -"de la Vega, Dr. Juan Q. Xavier" -"de la Vega, Dr. Juan Q. Xavier III" -"de la Vega, Dr. Juan Q. Xavier, Jr." -"de la Vega, Dr. Juan Q., Jr." -"de la Vega, Dr. Juan, Jr." -"de la Vega, Juan" -"de la Vega, Juan III" -"de la Vega, Juan Q." -"de la Vega, Juan Q. III" -"de la Vega, Juan Q. Xavier" -"de la Vega, Juan Q. Xavier III" -"de la Vega, Juan Q. Xavier, Jr." -"de la Vega, Juan Q., Jr." -"de la Vega, Juan, Jr." -"de la Véña, Jüan" -"de la dos Vega, Dr. Juan Q. Xavier III" -"de la vega, dr. juan Q. xavier III" -"donovan mcnabb-smith" -"dos Santos" -"dr Vincent James van Gogh dr" -"dr Vincent van Gogh dr" -"dr Vincent van der Gogh dr" -"dr. ben alex johnson III" -"dr. john p. doe-Ray, CLU, CFP, LUTC" -"dr. juan de la vega jr." -"e and e" -"e j smith" -"joao da silva do amaral de souza" -"john doe" -"john e jones" -"john e jones, III" -"john e. smith" -"john smith" -"johnny y" -"jones, john e" -"juan garcia III" -"juan q. xavier velasquez y garcia iii" -"larry james edward johnson v" -"lt. gen. john a. kenneth doe iv" -"mack johnson" -"matthëus schmidt" -"part1 of The part2 of the part3 and part4" -"part1 of and The part2 of the part3 And part4" -"pennie von bergen wessels" -"pennie von bergen wessels III" -"pennie von bergen wessels M.D." -"pennie von bergen wessels MD, III" -"pennie von bergen wessels, III" -"salem, abdul" -"salem, abdul salam" -"salem, abdul salam ahmed" -"scott e. werner" -"señora María García" -"test" -"the and Jon Dough" -"vai la" -"van nguyen" -"von Braun" -"von bergen wessels MD, pennie" -"von bergen wessels MD, pennie III" -"von bergen wessels, pennie III" -"von bergen wessels, pennie MD" -"xyz. John Smith" -"سلمان،" -"سلمان، محمد" -"‏John‏ Smith" -"‏محمد بن سلمان‏" -"∫≜⩕ Smith 😊" -"∫≜⩕ Smith😊" +{"name": "", "tests": ["test_can_add_global_nickname_delimiter", "test_can_change_global_constants", "test_empty_attribute_default", "test_empty_attribute_on_instance", "test_empty_parses_share_a_comparison_key", "test_initials_all_empty_returns_empty_attribute_default", "test_initials_separator_custom_value", "test_initials_separator_kwarg_multiword_part", "test_instance_owned_constants_not_mutated_by_parsing", "test_instances_can_have_own_constants", "test_is_rootname_consistent_with_is_prefix", "test_is_rootname_consistent_with_is_title", "test_iterating_empty_name_yields_nothing", "test_len", "test_none_empty_attribute_string_formatting"]} +{"name": "\"Rick\" Edmonds", "tests": ["test_nickname_and_last_name"]} +{"name": "()", "tests": ["test_empty_parses_share_a_comparison_key"]} +{"name": ",", "tests": ["test_degenerate_comma_input_leaves_no_empty_pieces", "test_empty_parses_share_a_comparison_key"]} +{"name": ", John", "tests": ["test_degenerate_comma_input_leaves_no_empty_pieces"]} +{"name": "A.B. Vajpayee", "tests": ["bank:TEST_NAMES"]} +{"name": "Abdul Salam Ahmed Salem, MD", "tests": ["test_suffix_comma_join_with_middle"]} +{"name": "Abdul Salam Hassan, MD", "tests": ["test_suffix_comma_join"]} +{"name": "Abdul Salam, MD", "tests": ["test_suffix_comma_guard_two_tokens_no_join"]} +{"name": "Abu Bakr Al Baghdadi, MD", "tests": ["test_suffix_comma_abu_bakr_al_baghdadi"]} +{"name": "Adolph D", "tests": ["test_phd_conflict"]} +{"name": "Ahmad ben Husain", "tests": ["test_ben_as_conjunction"]} +{"name": "Al Arnold Gore, Jr.", "tests": ["test33"]} +{"name": "Alex Ben Johnson", "tests": ["test_ben_as_middle_name"]} +{"name": "Alex van Johnson", "tests": ["test_initials_with_prefix"]} +{"name": "Aliyev Vusal Said oglu", "tests": ["test_reversed_order_unchanged"]} +{"name": "Alois von und zu Liechtenstein", "tests": ["test_conjunction_bridges_prefix_chain"]} +{"name": "Alois von und zu und von Liechtenstein", "tests": ["test_conjunction_bridges_prefix_chain_with_multiple_conjunctions"]} +{"name": "Amy E Maid", "tests": ["test_last_name_is_also_title"]} +{"name": "Amy E. Maid", "tests": ["bank:TEST_NAMES"]} +{"name": "Andrew", "tests": ["test_first_name"]} +{"name": "Andrew Boris Petersen", "tests": ["test_initials", "test_initials_list"]} +{"name": "Andrew Perkins \"MBA\"", "tests": ["test_suffix_in_double_quotes"]} +{"name": "Andrew Perkins 'MBA'", "tests": ["test_suffix_in_single_quotes"]} +{"name": "Andrew Perkins (JD)", "tests": ["test_suffix_acronyms_ambiguous_removal_routes_to_suffix"]} +{"name": "Andrew Perkins (M.D)", "tests": ["test_acronym_suffix_with_internal_periods_in_parenthesis"]} +{"name": "Andrew Perkins (MBA)", "tests": ["test_acronym_suffix_in_parenthesis"]} +{"name": "Andrew Perkins (Mgr.)", "tests": ["test_period_terminated_content_in_parenthesis_not_forced_either_way"]} +{"name": "Andrew Perkins (XYZ)", "tests": ["test_suffix_acronyms_ambiguous_custom_entry_stays_nickname"]} +{"name": "Andrew Perkins, Jr., Col. (Ret)", "tests": ["test_suffix_in_parenthesis_with_other_suffixes"]} +{"name": "Andrews, M.D.", "tests": ["test_assume_suffix_title_and_one_other_name_is_last_name"]} +{"name": "Anh Do", "tests": ["test_do_guard_surname_equals_prefix_word"]} +{"name": "Annette Charlotte Freiherrin von und zu der Tann-Rathsamhausen", "tests": ["test_conjunction_bridges_prefix_chain_with_leading_title"]} +{"name": "Assoc Dean of Chemistry Robert Johns", "tests": ["test_add_multiple_arguments"]} +{"name": "Baker (Johnson), Jenny", "tests": ["test_maiden_appears_in_as_dict_via_routing", "test_maiden_name_in_parenthesis_with_comma", "test_maiden_off_by_default_parenthesis_still_routes_to_nickname", "test_routing_same_delimiter_to_both_buckets_nickname_wins"]} +{"name": "Baker (Jr.), Jenny", "tests": ["test_suffix_shaped_content_in_maiden_bucket_stays_in_place"]} +{"name": "Ben Alex Johnson", "tests": ["test_ben_as_first_name_with_middle_name"]} +{"name": "Ben Johnson", "tests": ["test_ben_as_first_name"]} +{"name": "Benjamin \"Ben\" Franklin", "tests": ["bank:TEST_NAMES", "test_nickname_in_double_quotes"]} +{"name": "Benjamin \"Big Ben\" Franklin", "tests": ["test_two_words_in_quotes"]} +{"name": "Benjamin 'Ben' Franklin", "tests": ["test_nickname_in_single_quotes"]} +{"name": "Benjamin (Ben) Franklin", "tests": ["bank:TEST_NAMES", "test_nickname_in_parenthesis"]} +{"name": "Benjamin (Big Ben) Franklin", "tests": ["test_two_word_nickname_in_parenthesis"]} +{"name": "Benjamin [Ben] Franklin", "tests": ["test_overriding_builtin_regex_still_affects_nickname_parsing"]} +{"name": "Bob Dole", "tests": ["test_maiden_key_always_in_as_dict"]} +{"name": "Brian Andrew O'connor", "tests": ["test_single_quotes_on_first_name_not_treated_as_nickname"]} +{"name": "Brian O'connor", "tests": ["bank:TEST_NAMES"]} +{"name": "Buca di Beppo", "tests": ["bank:TEST_NAMES", "test127"]} +{"name": "Cardinal Secretary of State Hillary Clinton", "tests": ["test_compound_title_with_conjunction"]} +{"name": "Chancellor Jane Smith", "tests": ["test_constants_subclass_instance_is_used"]} +{"name": "Chang, Andy C I", "tests": ["test_roman_numeral_i_after_single_initial_lastname_comma_format"]} +{"name": "Charles van der van der Berg", "tests": ["test_repeated_prefix_chain_van_der"]} +{"name": "Chemistry Jane Smith", "tests": ["test_constants_kwarg_elements_are_normalized"]} +{"name": "Chief Judge J. Leon Holmes", "tests": ["bank:TEST_NAMES"]} +{"name": "Chief Judge Sharon Lovelace Blackburn", "tests": ["bank:TEST_NAMES"]} +{"name": "Clarke, Kenneth, Q.C. M.P.", "tests": ["test_suffix_containing_periods_lastname_comma_format"]} +{"name": "Coach", "tests": ["test_title_is_title"]} +{"name": "Col. (Ret.) Smith", "tests": ["test_suffix_in_parenthesis_with_period"]} +{"name": "DE MESNIL", "tests": ["test_leading_non_first_name_prefix_case_insensitive"]} +{"name": "DR DOE", "tests": ["test_capitalize_title_and_last_only_no_spurious_tokens"]} +{"name": "Dame Mary", "tests": ["test_dame_exception_to_first_name_rule"]} +{"name": "Dean Ms Hon Solo", "tests": ["test_chain_multiple_arguments"]} +{"name": "Del Toro", "tests": ["test_leading_del_is_unchanged"]} +{"name": "Della Reese", "tests": ["test_leading_della_is_unchanged"]} +{"name": "Designated Judge David A. Ezra", "tests": ["bank:TEST_NAMES"]} +{"name": "Di Caprio", "tests": ["test_leading_di_is_unchanged"]} +{"name": "Doctor, Jane E.", "tests": ["bank:TEST_NAMES"]} +{"name": "Doe, Dr. John", "tests": ["bank:TEST_NAMES", "test22"]} +{"name": "Doe, Dr. John A.", "tests": ["bank:TEST_NAMES", "test28"]} +{"name": "Doe, Dr. John A. III", "tests": ["bank:TEST_NAMES", "test30"]} +{"name": "Doe, Dr. John A. Jr.", "tests": ["bank:TEST_NAMES", "test29"]} +{"name": "Doe, Dr. John A. Kenneth", "tests": ["bank:TEST_NAMES", "test35"]} +{"name": "Doe, Dr. John A. Kenneth III", "tests": ["bank:TEST_NAMES", "test37"]} +{"name": "Doe, Dr. John A. Kenneth Jr.", "tests": ["bank:TEST_NAMES", "test36"]} +{"name": "Doe, Dr. John III", "tests": ["bank:TEST_NAMES", "test24"]} +{"name": "Doe, Dr. John P., CLU, CFP, LUTC", "tests": ["test_comparison"]} +{"name": "Doe, Dr. John, Jr.", "tests": ["bank:TEST_NAMES", "test23"]} +{"name": "Doe, John", "tests": ["bank:TEST_NAMES", "test4"]} +{"name": "Doe, John A.", "tests": ["bank:TEST_NAMES", "test10", "test_initials_delimiter_empty_string_kwarg", "test_initials_format_empty_string_kwarg"]} +{"name": "Doe, John A. III", "tests": ["bank:TEST_NAMES"]} +{"name": "Doe, John A. Kenneth", "tests": ["bank:TEST_NAMES", "test_initials_separator_constants_multi_part_middle", "test_initials_separator_empty_multi_part_middle", "test_initials_separator_kwarg"]} +{"name": "Doe, John A. Kenneth III", "tests": ["bank:TEST_NAMES"]} +{"name": "Doe, John A. Kenneth, Jr.", "tests": ["bank:TEST_NAMES", "test_initials_complex_name", "test_initials_delimiter", "test_initials_delimiter_constants", "test_initials_format", "test_initials_format_constants", "test_initials_list_complex_name"]} +{"name": "Doe, John A., III", "tests": ["test12"]} +{"name": "Doe, John A., Jr.", "tests": ["bank:TEST_NAMES", "test11"]} +{"name": "Doe, John III", "tests": ["bank:TEST_NAMES", "test6"]} +{"name": "Doe, John Msc.Ed.", "tests": ["test_suffix_with_periods_with_lastname_comma"]} +{"name": "Doe, John jr., MD", "tests": ["test_suffix_with_double_comma_format"]} +{"name": "Doe, John, Jr.", "tests": ["bank:TEST_NAMES", "test5"]} +{"name": "Doe, John, MD PhD - FACS Fellow", "tests": ["test_suffix_delimiter_detection_with_multi_word_side"]} +{"name": "Doe, John,, Jr.", "tests": ["test_empty_comma_segment_does_not_drop_following_suffix"]} +{"name": "Doe, John,, Jr.,, III", "tests": ["test_empty_comma_segment_does_not_drop_following_suffix"]} +{"name": "Doe, John. A. Kenneth", "tests": ["test16"]} +{"name": "Doe, John. A. Kenneth III", "tests": ["test18"]} +{"name": "Doe, John. A. Kenneth, Jr.", "tests": ["test17"]} +{"name": "Doe, Lt. Gen. John A. Kenneth IV", "tests": ["bank:TEST_NAMES", "test_two_part_title_with_lastname_comma"]} +{"name": "Doe, Lt.Gov. John", "tests": ["test_title_with_periods_lastname_comma"]} +{"name": "Doe, Mary - Kate, RN", "tests": ["test_suffix_delimiter_inverted_format_not_misparsed"]} +{"name": "Doe, Rev. John A. Jr.", "tests": ["bank:TEST_NAMES", "test126"]} +{"name": "Doe, Rev. John A., V, Jr.", "tests": ["test_possible_conflict_with_suffix_that_could_be_initial"]} +{"name": "Doe, Rev. John V, Jr.", "tests": ["test_possible_conflict_with_middle_initial_that_could_be_suffix"]} +{"name": "Doe,, Jr.", "tests": ["test_degenerate_comma_input_leaves_no_empty_pieces"]} +{"name": "Doe-Ray, Dr. John P., CLU, CFP, LUTC", "tests": ["bank:TEST_NAMES", "test113", "test_comparison", "test_comparison_case_insensitive", "test_hash_matches_case_insensitive_equality", "test_len", "test_slice", "test_slice_getitem_deprecation_names_issue"]} +{"name": "Doe-Ray, Hon. Barrington P. Jr.", "tests": ["bank:TEST_NAMES"]} +{"name": "Doe-Ray, Hon. Barrington P. Jr., CFP, LUTC", "tests": ["bank:TEST_NAMES", "test116"]} +{"name": "Donovan McNabb-Smith", "tests": ["bank:TEST_NAMES"]} +{"name": "Dr King Jr", "tests": ["test_king"]} +{"name": "Dr Martin Luther King, Jr.", "tests": ["test_last_name_is_also_title_with_comma"]} +{"name": "Dr. Abdul Salam Hassan, MD", "tests": ["test_suffix_comma_title_kept_prefix_joins"]} +{"name": "Dr. John A. Doe", "tests": ["bank:TEST_NAMES", "test25"]} +{"name": "Dr. John A. Doe III", "tests": ["bank:TEST_NAMES", "test27"]} +{"name": "Dr. John A. Doe, Jr.", "tests": ["bank:TEST_NAMES", "test26", "test_str_fallback_without_string_format"]} +{"name": "Dr. John A. Kenneth Doe", "tests": ["bank:TEST_NAMES", "test31"]} +{"name": "Dr. John A. Kenneth Doe III", "tests": ["bank:TEST_NAMES", "test34"]} +{"name": "Dr. John A. Kenneth Doe, Jr.", "tests": ["bank:TEST_NAMES", "test32", "test_getitem", "test_setitem", "test_setitem_emits_deprecation_warning_naming_attribute_assignment", "test_setitem_invalid_key_raises_keyerror"]} +{"name": "Dr. John Doe", "tests": ["bank:TEST_NAMES", "test19"]} +{"name": "Dr. John Doe III", "tests": ["bank:TEST_NAMES", "test21"]} +{"name": "Dr. John Doe, Jr.", "tests": ["bank:TEST_NAMES", "test20"]} +{"name": "Dr. John P. Doe-Ray, CLU", "tests": ["test_name_instance_deepcopy"]} +{"name": "Dr. John P. Doe-Ray, CLU, CFP, LUTC", "tests": ["bank:TEST_NAMES", "test112", "test_comparison"]} +{"name": "Dr. John Smith", "tests": ["test_known_title_with_period_still_a_title"]} +{"name": "Dr. Juan Q. Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test80"]} +{"name": "Dr. Juan Q. Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test82"]} +{"name": "Dr. Juan Q. Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test81"]} +{"name": "Dr. Juan Q. Xavier Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test104"]} +{"name": "Dr. Juan Q. Xavier Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test106"]} +{"name": "Dr. Juan Q. Xavier Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test105"]} +{"name": "Dr. Juan Q. Xavier de la Vega", "tests": ["bank:TEST_NAMES", "test92"]} +{"name": "Dr. Juan Q. Xavier de la Vega III", "tests": ["bank:TEST_NAMES", "test94", "test_comparison_key_case_insensitive_across_formats", "test_comparison_key_components", "test_matches_str_is_semantic_not_textual"]} +{"name": "Dr. Juan Q. Xavier de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test93"]} +{"name": "Dr. Juan Q. Xavier de la dos Vega III", "tests": ["test_three_conjunctions"]} +{"name": "Dr. Juan Q. Xavier de la dos Vega, III", "tests": ["test_comma_three_conjunctions"]} +{"name": "Dr. Juan Q. de la Vega", "tests": ["bank:TEST_NAMES", "test74"]} +{"name": "Dr. Juan Q. de la Vega III", "tests": ["bank:TEST_NAMES", "test76"]} +{"name": "Dr. Juan Q. de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test75"]} +{"name": "Dr. Juan Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test56"]} +{"name": "Dr. Juan Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test58"]} +{"name": "Dr. Juan Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test57"]} +{"name": "Dr. Juan de la Vega", "tests": ["bank:TEST_NAMES", "test50"]} +{"name": "Dr. Juan de la Vega III", "tests": ["bank:TEST_NAMES", "test52"]} +{"name": "Dr. Juan de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test51"]} +{"name": "Dr. Martin Luther King Jr.", "tests": ["test_last_name_is_also_title_no_comma"]} +{"name": "Dr. Williams", "tests": ["test_given_names_attribute_empty"]} +{"name": "Dr. abdul salam ahmed salem", "tests": ["test_title_kept_prefix_joins"]} +{"name": "Dra. Andréia da Silva", "tests": ["bank:TEST_NAMES", "test_da_silva_title_plus_prefix"]} +{"name": "E.T. Smith", "tests": ["bank:TEST_NAMES", "test_leading_internal_period_abbreviation_excluded", "test_two_initials_conflict_with_conjunction"]} +{"name": "E.T. Smith, II", "tests": ["bank:TEST_NAMES"]} +{"name": "Esq Jane Smith", "tests": ["test_set_manager_operators_accept_lists"]} +{"name": "Foo. John Smith", "tests": ["test_leading_period_abbreviation_unknown_word"]} +{"name": "Foo. Xyz. John Smith", "tests": ["test_leading_period_abbreviation_chained"]} +{"name": "Franklin Washington, Jr. MD", "tests": ["test_two_suffixes_suffix_comma_format"]} +{"name": "Franklin, Benjamin (Ben)", "tests": ["test_nickname_in_parenthesis_with_comma"]} +{"name": "Franklin, Benjamin (Ben), Jr.", "tests": ["test_nickname_in_parenthesis_with_comma_and_suffix"]} +{"name": "Frau Anna Müller", "tests": ["bank:TEST_NAMES"]} +{"name": "Fritz Freiherr und von Bar", "tests": ["test_conjunction_bridges_word_that_is_both_title_and_prefix"]} +{"name": "Frøken Jensen", "tests": ["test_frøken_non_ascii_title"]} +{"name": "GREGORY HOUSE M.D.", "tests": ["test_capitalize_suffix_acronym_with_dots"]} +{"name": "Gunny de Mesnil", "tests": ["test_non_first_name_prefix_with_custom_title"]} +{"name": "Harietta Keopuolani Nahi'ena'ena", "tests": ["test_single_quotes_not_treated_as_nickname_Hawaiian_example"]} +{"name": "Harrieta Keōpūolani Nāhiʻenaʻena", "tests": ["test_okina_inside_name_not_treated_as_nickname"]} +{"name": "Her Majesty Queen Elizabeth", "tests": ["test_conjunction_in_an_address_with_a_first_name_title"]} +{"name": "Herr Klaus Schmidt", "tests": ["bank:TEST_NAMES", "test_herr_title_with_first_name"]} +{"name": "Herr Schmidt", "tests": ["test_herr_title_not_first_name"]} +{"name": "His Excellency Lord Duncan", "tests": ["test_conjunction_in_an_address_with_a_title"]} +{"name": "Hon Solo", "tests": ["test_remove_title"]} +{"name": "Hon. Barrington P. Doe-Ray, Jr.", "tests": ["bank:TEST_NAMES", "test115"]} +{"name": "Hon. Charles J. Siragusa", "tests": ["bank:TEST_NAMES"]} +{"name": "Hon. Marian W. Payson", "tests": ["bank:TEST_NAMES"]} +{"name": "Honorable Judge Susan Russ Walker", "tests": ["bank:TEST_NAMES"]} +{"name": "Honorable Judge Terry F. Moorer", "tests": ["bank:TEST_NAMES"]} +{"name": "Honorable Judge W. Harold Albritton, III", "tests": ["bank:TEST_NAMES"]} +{"name": "Honorable Terry F. Moorer", "tests": ["bank:TEST_NAMES"]} +{"name": "Honorable W. Harold Albritton, III", "tests": ["bank:TEST_NAMES"]} +{"name": "Ivanov Ivan Ivanovich", "tests": ["test_canonical_unchanged"]} +{"name": "J. Smith", "tests": ["test_leading_single_letter_initial_excluded"]} +{"name": "JEFFREY (JD) BRICKEN", "tests": ["test_ambiguous_suffix_acronym_in_parenthesis_stays_nickname"]} +{"name": "JOHN DOE", "tests": ["test_capitalize_empty_suffix_produces_no_spurious_tokens"]} +{"name": "JOHN DOE PHD", "tests": ["test_capitalize_single_suffix_still_works"]} +{"name": "JOHN DOE PHD MD", "tests": ["test_capitalize_multiple_suffixes_still_split_correctly"]} +{"name": "JOHN SMITH", "tests": ["bank:names", "test_comparison_key_usable_for_dedup", "test_matches_humanname_operand"]} +{"name": "JOSÉ GARCÍA", "tests": ["test_matches_non_ascii_case_insensitive"]} +{"name": "Jack Ma", "tests": ["test_potential_suffix_that_is_also_last_name"]} +{"name": "Jack Ma Jr", "tests": ["test_potential_suffix_that_is_also_last_name_with_suffix"]} +{"name": "Jane Doctor", "tests": ["bank:TEST_NAMES", "test_last_name_also_prefix"]} +{"name": "Jane Doe", "tests": ["test_bytes_full_name_emits_deprecation_warning", "test_capitalize_empty_attributes_produce_no_spurious_tokens", "test_capitalize_force_empty_middle_produces_no_leading_space_in_surnames"]} +{"name": "Jane Mac Beth", "tests": ["test_mac_with_spaces"]} +{"name": "Jane Smith", "tests": ["bank:names", "test_comparison_key_usable_for_dedup", "test_matches_humanname_operand", "test_not_equal_operator"]} +{"name": "Jean de Mesnil", "tests": ["test_non_leading_prefix_is_unchanged"]} +{"name": "Jenny \"JJ\" Baker (Johnson)", "tests": ["test_quotes_still_nickname_when_parens_routed_to_maiden"]} +{"name": "Jenny (Johnson) Baker", "tests": ["test_unresolvable_string_sentinel_raises"]} +{"name": "Jenny Baker", "tests": ["test_maiden_appears_in_as_dict_when_populated", "test_maiden_appears_in_slice", "test_maiden_assignment_and_property", "test_maiden_defaults_empty"]} +{"name": "Jenny Baker (Johnson)", "tests": ["test_maiden_name_in_parenthesis_no_comma"]} +{"name": "Jill St. John", "tests": ["test_prefix_with_period"]} +{"name": "Joao da Silva do Amaral de Souza", "tests": ["test_portuguese_prefixes"]} +{"name": "Joe Dentist D.D.S.", "tests": ["test_suffix_with_periods"]} +{"name": "Joe Franklin Jr", "tests": ["test_suffix"]} +{"name": "John & Jane", "tests": ["test_ampersand_conjunction_short_name_no_titles"]} +{"name": "John A. Doe", "tests": ["bank:TEST_NAMES", "test7"]} +{"name": "John A. Doe III", "tests": ["bank:TEST_NAMES", "test9"]} +{"name": "John A. Doe, Jr", "tests": ["test8"]} +{"name": "John A. Doe, Jr.", "tests": ["bank:TEST_NAMES"]} +{"name": "John A. Kenneth Doe", "tests": ["bank:TEST_NAMES", "test13"]} +{"name": "John A. Kenneth Doe III", "tests": ["bank:TEST_NAMES", "test15"]} +{"name": "John A. Kenneth Doe, Jr.", "tests": ["bank:TEST_NAMES", "test14", "test_assign_list_to_attribute", "test_assignment_to_attribute", "test_assignment_to_full_name"]} +{"name": "John Doe", "tests": ["bank:TEST_NAMES", "test1", "test_assigning_constants_class_after_construction_raises_with_hint", "test_assigning_invalid_constants_after_construction_raises", "test_assigning_none_to_constants_after_construction_builds_new_instance", "test_assignment_filters_empty_tokens", "test_constants_class_instead_of_instance_raises_with_hint", "test_constants_invalid_type_raises_typeerror", "test_deepcopy_default_name_preserves_singleton_identity", "test_explicit_none_still_produces_a_working_private_config", "test_explicit_none_warning_names_both_replacements", "test_explicit_none_warns_on_c_setter", "test_explicit_none_warns_on_construction", "test_explicit_none_warns_on_positional_argument", "test_explicit_own_constants_instance_does_not_warn", "test_initials_empty_part_with_none_default_not_literal_none", "test_initials_simple_name", "test_instance_is_not_its_own_iterator", "test_iteration_restarts_after_break", "test_iterators_are_independent", "test_len", "test_len_during_iteration", "test_omitted_constants_argument_does_not_warn", "test_pickle_default_name_preserves_singleton_identity", "test_shallow_copy_default_name_preserves_singleton_identity", "test_str_default_behavior_unchanged", "test_string_format_empty_string_kwarg"]} +{"name": "John Doe III", "tests": ["bank:TEST_NAMES", "test3"]} +{"name": "John Doe MD PhD", "tests": ["test_suffix_delimiter_no_effect_without_comma"]} +{"name": "John Doe Msc.Ed.", "tests": ["test_multiple_letter_suffix_with_periods"]} +{"name": "John Doe jr., MD", "tests": ["test_suffix_with_single_comma_format"]} +{"name": "John Doe, CLU, CFP, LUTC", "tests": ["bank:TEST_NAMES", "test110"]} +{"name": "John Doe, Jr.", "tests": ["bank:TEST_NAMES", "test2"]} +{"name": "John Doe, Jr.,,", "tests": ["test_suffix_comma_empty_segment_not_added_to_suffix_list"]} +{"name": "John Doe, MD - PhD - FACS", "tests": ["test_suffix_delimiter_multiple"]} +{"name": "John Doe, MD - PhD, FACS", "tests": ["test_suffix_delimiter_expands_each_comma_segment"]} +{"name": "John Doe, MD, PhD", "tests": ["test_suffix_delimiter_comma_space_is_noop"]} +{"name": "John Doe, MD-PhD-", "tests": ["test_suffix_delimiter_trailing_delimiter_ignored"]} +{"name": "John Doe, Msc.Ed.", "tests": ["test_suffix_with_periods_with_comma"]} +{"name": "John E Smith", "tests": ["bank:TEST_NAMES", "test_uppercase_middle_initial_conflict_with_conjunction"]} +{"name": "John Edgar Casey Williams III", "tests": ["test_given_names_attribute", "test_given_names_list_attribute", "test_surnames_attribute", "test_surnames_list_attribute"]} +{"name": "John Jones (Google Docs)", "tests": ["bank:TEST_NAMES"]} +{"name": "John Jones (Google Docs), Jr. (Unknown)", "tests": ["test_duplicate_parenthesis_are_removed_from_name"]} +{"name": "John Jones (Unknown)", "tests": ["test_parenthesis_are_removed_from_name"]} +{"name": "John King", "tests": ["test_last_name_is_also_title3"]} +{"name": "John Major. Smith", "tests": ["test_period_abbreviation_after_first_name_stays_middle"]} +{"name": "John P. Doe, CLU, CFP, LUTC", "tests": ["bank:TEST_NAMES", "test111"]} +{"name": "John Q. Smith", "tests": ["test_middle_initial_with_period_unaffected"]} +{"name": "John Smith", "tests": ["bank:names", "test_comparison_key_independent_of_string_format", "test_comparison_key_usable_for_dedup", "test_empty_field_drops_surrounding_whitespace", "test_eq_emits_deprecation_warning", "test_hash_emits_deprecation_warning", "test_hash_matches_case_insensitive_equality", "test_matches_humanname_operand", "test_matches_rejects_other_types", "test_multiple_empty_fields_before_comma", "test_new_comparison_api_does_not_warn", "test_no_prefix", "test_not_equal_operator", "test_str_full_name_does_not_warn", "test_unparsable_attribute_removed"]} +{"name": "John Smith Jr", "tests": ["test_empty_field_present_suffix_unaffected"]} +{"name": "John Smith Ph. D.", "tests": ["test_phd_extracted_without_comma"]} +{"name": "John Smith VI", "tests": ["test_roman_numeral_suffix_not_in_suffix_list"]} +{"name": "John Smith, Ph. D.", "tests": ["test_phd_with_erroneous_space"]} +{"name": "John Smith, V Jr.", "tests": ["test_two_suffix_not_acronyms_suffix_comma_format"]} +{"name": "John Smith, V MD", "tests": ["test_suffix_not_acronym_then_acronym_suffix_comma_format"]} +{"name": "John W. Ingram, V", "tests": ["test_roman_numeral_v_suffix_comma_format"]} +{"name": "John W. Smith, I", "tests": ["test_roman_numeral_i_suffix_comma_format"]} +{"name": "John Williams", "tests": ["test_get_full_name_attribute_references_internal_lists", "test_given_names_attribute_first_only"]} +{"name": "John and Jane Aznar y Lopez", "tests": ["test_couples_names_with_conjunction_lastname"]} +{"name": "John and Jane Smith", "tests": ["bank:TEST_NAMES", "test_couples_names"]} +{"name": "John e Smith", "tests": ["bank:TEST_NAMES", "test_lowercase_middle_initial_conflict_with_conjunction"]} +{"name": "John e Smith III", "tests": ["test_lowercase_middle_initial_and_nocomma_suffix_conflict_with_conjunction"]} +{"name": "John e Smith, III", "tests": ["test_lowercase_middle_initial_and_suffix_conflict_with_conjunction"]} +{"name": "John y Jane", "tests": ["test_single_char_alpha_conjunction_still_treated_as_initial_when_short"]} +{"name": "Jon Dough and", "tests": ["test_ends_with_conjunction"]} +{"name": "Jon Dough and of", "tests": ["test_ends_with_two_conjunctions"]} +{"name": "Jose Aznar y Lopez", "tests": ["bank:TEST_NAMES", "test_last_name_with_conjunction"]} +{"name": "Juan Q. Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test68"]} +{"name": "Juan Q. Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test70"]} +{"name": "Juan Q. Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test69"]} +{"name": "Juan Q. Xavier Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test98"]} +{"name": "Juan Q. Xavier Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test100"]} +{"name": "Juan Q. Xavier Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test99"]} +{"name": "Juan Q. Xavier de la Vega", "tests": ["bank:TEST_NAMES", "test86"]} +{"name": "Juan Q. Xavier de la Vega III", "tests": ["bank:TEST_NAMES", "test88"]} +{"name": "Juan Q. Xavier de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test87"]} +{"name": "Juan Q. de la Vega", "tests": ["bank:TEST_NAMES", "test62"]} +{"name": "Juan Q. de la Vega III", "tests": ["bank:TEST_NAMES", "test64"]} +{"name": "Juan Q. de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test63"]} +{"name": "Juan Velasquez y Garcia", "tests": ["bank:TEST_NAMES", "test44"]} +{"name": "Juan Velasquez y Garcia III", "tests": ["bank:TEST_NAMES", "test46"]} +{"name": "Juan Velasquez y Garcia, Jr.", "tests": ["bank:TEST_NAMES", "test45"]} +{"name": "Juan de la Vega", "tests": ["bank:TEST_NAMES", "test38", "test_de_la_vega_multiword_prefix"]} +{"name": "Juan de la Vega III", "tests": ["bank:TEST_NAMES", "test40"]} +{"name": "Juan de la Vega, Jr.", "tests": ["bank:TEST_NAMES", "test39"]} +{"name": "Juan de la de la Vega", "tests": ["test_repeated_prefix_chain_de_la"]} +{"name": "Juan de la de la Vega Jr.", "tests": ["test_repeated_prefix_chain_followed_by_suffix"]} +{"name": "Juan de la de la de la Vega", "tests": ["test_triple_repeated_prefix_chain"]} +{"name": "Juan del Sur", "tests": ["test_prefix"]} +{"name": "Judge C Lynwood Smith, Jr", "tests": ["bank:TEST_NAMES"]} +{"name": "Judge G. Thomas Eisele", "tests": ["bank:TEST_NAMES"]} +{"name": "Judge James M. Moody", "tests": ["bank:TEST_NAMES"]} +{"name": "Kenneth Clarke Q.C.", "tests": ["test_suffix_containing_periods"]} +{"name": "Kenneth Clarke Q.C., M.P.", "tests": ["test_suffix_containing_periods_suffix_comma_format"]} +{"name": "Kenneth Clarke QC MP", "tests": ["test_two_suffixes"]} +{"name": "King Henry", "tests": ["test_king_exception_to_first_name_rule"]} +{"name": "King John Alexander V", "tests": ["test_title_with_three_part_name_last_initial_is_suffix_uppercase_no_period"]} +{"name": "King John V.", "tests": ["test_title_with_last_initial_is_suffix"]} +{"name": "LT. GEN. JOHN A. KENNETH DOE IV", "tests": ["test_capitalize_title_to_lower"]} +{"name": "La'tanya O'connor", "tests": ["test_single_quotes_on_both_name_not_treated_as_nickname"]} +{"name": "Larry James Johnson I", "tests": ["test_four_name_parts_with_suffix_that_could_be_initial_uppercase_no_period"]} +{"name": "Larry V I", "tests": ["test_roman_numeral_initials"]} +{"name": "Lon (Jr.) Williams", "tests": ["test_suffix_in_parenthesis_mid_name"]} +{"name": "Lord God Almighty", "tests": ["test119"]} +{"name": "Lord and of the Universe", "tests": ["test_triple_conjunction_on_title"]} +{"name": "Lord of the Universe", "tests": ["test_double_conjunction_on_title"]} +{"name": "Lord of the Universe and Associate Supreme Queen of the World Lisa Simpson", "tests": ["test_multiple_conjunctions_on_multiple_titles"]} +{"name": "Lord of the Universe and Supreme King of the World Lisa Simpson", "tests": ["bank:TEST_NAMES"]} +{"name": "Lt. Gen. John A. Kenneth Doe IV", "tests": ["bank:TEST_NAMES", "test_two_part_title"]} +{"name": "Lt. Gen. John A. Kenneth Doe, Jr.", "tests": ["bank:TEST_NAMES", "test_two_part_title_with_suffix_comma"]} +{"name": "Lt.Gen. John A. Kenneth Doe IV", "tests": ["test_two_title_parts_separated_by_periods"]} +{"name": "Lt.Gov. John Doe", "tests": ["test_derivations_reset_between_parses_of_same_instance", "test_title_with_periods"]} +{"name": "Lt.Gov. juan e garcia", "tests": ["test_title_with_periods_and_single_letter_middle_name"]} +{"name": "Ma III, Jack Jr", "tests": ["test_potential_suffix_that_is_also_last_name_with_suffix_comma"]} +{"name": "Ma, Jack", "tests": ["test_potential_suffix_that_is_also_last_name_comma"]} +{"name": "Mac Miller", "tests": ["test_mac_as_first_name"]} +{"name": "Mag-Judge Harwell G Davis, III", "tests": ["bank:TEST_NAMES", "test_chained_hyphenated_title_with_comma_suffix"]} +{"name": "Mag. Judge Byron G. Cudmore", "tests": ["bank:TEST_NAMES"]} +{"name": "Magistrate Judge John F. Forster, Jr", "tests": ["bank:TEST_NAMES"]} +{"name": "Magistrate-Judge Elizabeth Todd Campbell", "tests": ["bank:TEST_NAMES"]} +{"name": "Maid Marion", "tests": ["bank:TEST_NAMES"]} +{"name": "Maier, Amy I, Jr.", "tests": ["test_roman_numeral_i_with_explicit_suffix_comma_known_limitation"]} +{"name": "Maier, Amy Lauren I", "tests": ["test_roman_numeral_i_lastname_comma_format"]} +{"name": "Major. Dona Smith", "tests": ["test_leading_period_abbreviation_is_title"]} +{"name": "Major. John Smith, Jr.", "tests": ["test_leading_period_abbreviation_suffix_comma"]} +{"name": "Mari' Aube'", "tests": ["test_single_quotes_on_end_of_last_name_not_treated_as_nickname"]} +{"name": "Mevrouw Anna de Vries", "tests": ["bank:TEST_NAMES"]} +{"name": "Mike van der Velt", "tests": ["test_multiple_prefixes"]} +{"name": "Mohamad Ahmad Ali Hassan", "tests": ["test_default_constants_unaffected"]} +{"name": "Mohamad Ali Khalil", "tests": ["test_mohamad_unchanged"]} +{"name": "Monsieur Jean Dupont", "tests": ["bank:TEST_NAMES"]} +{"name": "Mr. & Mrs. John Smith", "tests": ["test_couple_titles_ampersand_conjunction"]} +{"name": "Mr. Van Nguyen", "tests": ["test_first_name_is_prefix_if_three_parts"]} +{"name": "Mr. and Mrs. John Smith", "tests": ["bank:TEST_NAMES"]} +{"name": "Mr. and Mrs. John and Jane Smith", "tests": ["test_couple_titles"]} +{"name": "Ms Hon Solo", "tests": ["test_clear_removes_all_entries", "test_remove_multiple_arguments"]} +{"name": "Naomi Wambui Ng'ang'a", "tests": ["test_single_quotes_not_treated_as_nickname_Kenyan_example"]} +{"name": "Nguyen, Van", "tests": ["test_first_name_is_not_prefix_if_only_two_parts_comma"]} +{"name": "No1. John Smith", "tests": ["test_leading_period_abbreviation_excludes_digits"]} +{"name": "None Smith", "tests": ["test_name_none_as_literal_name_with_none_empty_attribute_default"]} +{"name": "Nonez Smith", "tests": ["test_name_containing_none_substring_with_none_empty_attribute_default"]} +{"name": "O'B. John Smith", "tests": ["test_leading_period_abbreviation_excludes_apostrophe"]} +{"name": "Q R", "tests": ["bank:"]} +{"name": "Queen Elizabeth", "tests": ["test_queen_exception_to_first_name_rule"]} +{"name": "RONALD MACDONALD", "tests": ["test_downcasing_mac"]} +{"name": "RONALD MCDONALD", "tests": ["test_downcasing_mc"]} +{"name": "Rafael Sousa dos Anjos", "tests": ["test_portuguese_dos"]} +{"name": "Rev Andrews", "tests": ["test_assume_title_and_one_other_name_is_last_name"]} +{"name": "Rev John A. Kenneth Doe", "tests": ["bank:TEST_NAMES", "test125"]} +{"name": "Rev John A. Kenneth Doe III (Kenny)", "tests": ["test_formating_of_nicknames_in_middle", "test_formating_of_nicknames_with_double_quotes", "test_formating_of_nicknames_with_parenthesis", "test_formating_of_nicknames_with_single_quotes", "test_formating_removing_keys_from_format_string", "test_formating_removing_pieces_from_name_buckets", "test_formatting_constants_attribute", "test_formatting_init_argument", "test_quote_nickname_formating"]} +{"name": "Rev. John A. Kenneth Doe", "tests": ["bank:TEST_NAMES", "test124"]} +{"name": "Rt. Hon. Paul E. Mary", "tests": ["bank:TEST_NAMES", "test117"]} +{"name": "Sam Smith 😊", "tests": ["test_remove_emojis"]} +{"name": "Secretary of State Hillary Clinton", "tests": ["test_title_with_conjunction"]} +{"name": "Senator \"Rick\" Edmonds", "tests": ["test_nickname_and_last_name_with_title"]} +{"name": "Senior Judge Charles R. Butler, Jr", "tests": ["bank:TEST_NAMES"]} +{"name": "Senior Judge Harold D. Vietor", "tests": ["bank:TEST_NAMES"]} +{"name": "Senior Judge Virgil Pittman", "tests": ["bank:TEST_NAMES"]} +{"name": "Señor Carlos García", "tests": ["bank:TEST_NAMES"]} +{"name": "Señora María García", "tests": ["bank:TEST_NAMES", "test_señora_non_ascii_title"]} +{"name": "Shirley Maclaine", "tests": ["test_capitalize_name_and_force_mixed_case_capitalization_constants_attributes", "test_force_capitalization", "test_force_mixed_case_capitalization_constants_attribute", "test_no_change_to_mixed_chase"]} +{"name": "Signor Marco Rossi", "tests": ["bank:TEST_NAMES"]} +{"name": "Sir Gerald", "tests": ["bank:TEST_NAMES", "test_sir_exception_to_first_name_rule"]} +{"name": "Smith Jr., John", "tests": ["test_suffix_in_lastname_part_of_lastname_comma_format"]} +{"name": "Smith van der", "tests": ["test_all_particles_guard"]} +{"name": "Smith, Dr. John", "tests": ["test_name_instance_deepcopy_isolates_instance_config", "test_name_instance_pickle_preserves_instance_config", "test_pickle_instance_config_name_preserves_own_config"]} +{"name": "Smith, E.T., Jr.", "tests": ["bank:TEST_NAMES"]} +{"name": "Smith, J.R.", "tests": ["test_initials_also_suffix"]} +{"name": "Smith, John", "tests": ["bank:names", "test_comparison_key_usable_for_dedup", "test_custom_regexes_without_commas_key_does_not_shatter_name"]} +{"name": "Smith, John I", "tests": ["test_roman_numeral_i_no_middle_lastname_comma_format"]} +{"name": "Smith, John V", "tests": ["test_roman_numeral_v_lastname_comma_format"]} +{"name": "Smith, John e, III, Jr", "tests": ["test_lowercase_middle_initial_comma_lastname_and_suffix_conflict_with_conjunction"]} +{"name": "Smith, MD - PhD - FACS", "tests": ["test_suffix_delimiter_no_effect_when_not_suffix_comma"]} +{"name": "Smith, Major. John", "tests": ["test_leading_period_abbreviation_lastname_comma"]} +{"name": "Smith,John", "tests": ["test_fullwidth_comma_splits_lastname_format"]} +{"name": "Sr US District Judge Richard G Kopf", "tests": ["bank:TEST_NAMES"]} +{"name": "Srta. Andréia da Silva", "tests": ["bank:TEST_NAMES"]} +{"name": "Steven Hardman, RN - CRNA", "tests": ["test_suffix_delimiter_basic", "test_suffix_delimiter_constants_level", "test_suffix_delimiter_kwarg_accepted", "test_suffix_delimiter_none_by_default_known_limitation"]} +{"name": "Te Awanui-a-Rangi Black", "tests": ["test_add_title"]} +{"name": "The Lord of the Universe", "tests": ["test_conjunction_before_title"]} +{"name": "The Right Hon. the President of the Queen's Bench Division", "tests": ["test_title_multiple_titles_with_apostrophe_s"]} +{"name": "The Rt Hon John Jones", "tests": ["test_title_starts_with_conjunction"]} +{"name": "Title First Middle Middle Last, Jr.", "tests": ["test_name_instance_pickle"]} +{"name": "U. S. Grant", "tests": ["test_conflict_with_chained_title_first_name_initial"]} +{"name": "U.S. District Judge Marc Thomas Treadwell", "tests": ["bank:TEST_NAMES", "test_chained_title_first_name_title_is_initials"]} +{"name": "US Magistrate Judge T Michael Putnam", "tests": ["bank:TEST_NAMES", "test_chained_title_first_name_initial_with_no_period"]} +{"name": "US Magistrate-Judge Elizabeth E Campbell", "tests": ["test_chained_hyphenated_title"]} +{"name": "VINCENT VAN GOGH", "tests": ["test_case_insensitive_prefix_detection"]} +{"name": "Va'apu'u Vitale", "tests": ["test_single_quotes_not_treated_as_nickname_Samoan_example"]} +{"name": "Van Jeremy Johnson", "tests": ["test_initials_with_prefix_firstname", "test_prefix_is_first_name_with_middle_name"]} +{"name": "Van Johnson", "tests": ["test_leading_van_is_unchanged", "test_prefix_is_first_name"]} +{"name": "Van Nguyen", "tests": ["test_first_name_is_not_prefix_if_only_two_parts"]} +{"name": "Vega, Juan de la", "tests": ["test_initials_middle_name_all_prefixes"]} +{"name": "Velasquez y Garcia, Dr. Juan", "tests": ["bank:TEST_NAMES", "test59"]} +{"name": "Velasquez y Garcia, Dr. Juan III", "tests": ["bank:TEST_NAMES", "test61"]} +{"name": "Velasquez y Garcia, Dr. Juan Q.", "tests": ["bank:TEST_NAMES", "test83"]} +{"name": "Velasquez y Garcia, Dr. Juan Q. III", "tests": ["bank:TEST_NAMES", "test85"]} +{"name": "Velasquez y Garcia, Dr. Juan Q. Xavier", "tests": ["bank:TEST_NAMES", "test107"]} +{"name": "Velasquez y Garcia, Dr. Juan Q. Xavier III", "tests": ["bank:TEST_NAMES", "test109"]} +{"name": "Velasquez y Garcia, Dr. Juan Q. Xavier, Jr.", "tests": ["bank:TEST_NAMES", "test108"]} +{"name": "Velasquez y Garcia, Dr. Juan Q., Jr.", "tests": ["bank:TEST_NAMES", "test84"]} +{"name": "Velasquez y Garcia, Dr. Juan, Jr.", "tests": ["bank:TEST_NAMES", "test60"]} +{"name": "Velasquez y Garcia, Juan", "tests": ["bank:TEST_NAMES", "test47"]} +{"name": "Velasquez y Garcia, Juan III", "tests": ["bank:TEST_NAMES", "test49"]} +{"name": "Velasquez y Garcia, Juan Q.", "tests": ["bank:TEST_NAMES", "test71"]} +{"name": "Velasquez y Garcia, Juan Q. III", "tests": ["bank:TEST_NAMES", "test73"]} +{"name": "Velasquez y Garcia, Juan Q. Xavier", "tests": ["bank:TEST_NAMES", "test101"]} +{"name": "Velasquez y Garcia, Juan Q. Xavier III", "tests": ["bank:TEST_NAMES", "test103"]} +{"name": "Velasquez y Garcia, Juan Q. Xavier, Jr.", "tests": ["bank:TEST_NAMES", "test102"]} +{"name": "Velasquez y Garcia, Juan Q., Jr.", "tests": ["bank:TEST_NAMES", "test72"]} +{"name": "Velasquez y Garcia, Juan, Jr.", "tests": ["bank:TEST_NAMES", "test48"]} +{"name": "Vincent van Gogh", "tests": ["test_van_gogh_last_base", "test_van_gogh_last_base_list", "test_van_gogh_last_prefixes", "test_van_gogh_last_prefixes_list"]} +{"name": "Vincent van Gogh van Beethoven", "tests": ["test_2_same_prefixes_in_the_name"]} +{"name": "Washington Jr. MD, Franklin", "tests": ["test_two_suffixes_lastname_comma_format"]} +{"name": "Xyz. (Bud) Smith", "tests": ["test_leading_period_abbreviation_with_nickname"]} +{"name": "Yin Le", "tests": ["test_le_as_last_name"]} +{"name": "Yin a Le", "tests": ["test_le_as_last_name_with_middle_initial"]} +{"name": "Zephyrmark Jane Smith", "tests": ["test_matches_humanname_operand_keeps_its_own_parse", "test_matches_parses_str_with_instance_constants"]} +{"name": "abdul", "tests": ["test_prefix_alone_no_join"]} +{"name": "abdul salam", "tests": ["test_no_comma_guard_two_tokens_no_join"]} +{"name": "abdul salam ahmed salem", "tests": ["test_no_comma_basic_join", "test_opt_out_via_clear"]} +{"name": "abdul salam ahmed salem jr", "tests": ["test_suffix_kept_prefix_joins"]} +{"name": "abdul salam jr", "tests": ["test_no_comma_guard_suffix_not_swallowed"]} +{"name": "abdul salam salem", "tests": ["test_no_comma_three_tokens_no_middle"]} +{"name": "abdulsalam ahmed salem", "tests": ["test_single_token_already_joined_unchanged"]} +{"name": "abu bakr al baghdadi", "tests": ["test_abu_bakr_al_baghdadi"]} +{"name": "ahmed abu bakr", "tests": ["test_mid_name_prefix_becomes_last_prefix"]} +{"name": "and Jon Dough", "tests": ["test_starts_with_conjunction"]} +{"name": "and van Buren", "tests": ["test_conjunction_prefix_merge_at_start_stays_first_name"]} +{"name": "bob v. de la macdole-eisenhower phd", "tests": ["test_capitalize_name_constants_attribute"]} +{"name": "de", "tests": ["test_bare_non_first_name_prefix_guard"]} +{"name": "de Mesnil", "tests": ["test_leading_non_first_name_prefix_de", "test_leading_non_first_name_prefix_derived_props", "test_leading_non_first_name_prefix_with_patronymic_name_order"]} +{"name": "de Mesnil Garcia", "tests": ["test_leading_non_first_name_prefix_with_middle_name_as_last"]} +{"name": "de Mesnil Jr.", "tests": ["test_leading_non_first_name_prefix_with_suffix"]} +{"name": "de la Vega", "tests": ["test_leading_non_first_name_prefix_chain"]} +{"name": "de la Vega, Dr. Juan", "tests": ["bank:TEST_NAMES", "test53"]} +{"name": "de la Vega, Dr. Juan III", "tests": ["bank:TEST_NAMES", "test55"]} +{"name": "de la Vega, Dr. Juan Q.", "tests": ["bank:TEST_NAMES", "test77"]} +{"name": "de la Vega, Dr. Juan Q. III", "tests": ["bank:TEST_NAMES", "test79"]} +{"name": "de la Vega, Dr. Juan Q. Xavier", "tests": ["bank:TEST_NAMES", "test95"]} +{"name": "de la Vega, Dr. Juan Q. Xavier III", "tests": ["bank:TEST_NAMES", "test97"]} +{"name": "de la Vega, Dr. Juan Q. Xavier, Jr.", "tests": ["bank:TEST_NAMES", "test96"]} +{"name": "de la Vega, Dr. Juan Q., Jr.", "tests": ["bank:TEST_NAMES", "test78"]} +{"name": "de la Vega, Dr. Juan, Jr.", "tests": ["bank:TEST_NAMES", "test54"]} +{"name": "de la Vega, Juan", "tests": ["bank:TEST_NAMES", "test41"]} +{"name": "de la Vega, Juan III", "tests": ["bank:TEST_NAMES", "test43"]} +{"name": "de la Vega, Juan Q.", "tests": ["bank:TEST_NAMES", "test65"]} +{"name": "de la Vega, Juan Q. III", "tests": ["bank:TEST_NAMES", "test67"]} +{"name": "de la Vega, Juan Q. Xavier", "tests": ["bank:TEST_NAMES", "test89"]} +{"name": "de la Vega, Juan Q. Xavier III", "tests": ["bank:TEST_NAMES", "test91"]} +{"name": "de la Vega, Juan Q. Xavier, Jr.", "tests": ["bank:TEST_NAMES", "test90"]} +{"name": "de la Vega, Juan Q., Jr.", "tests": ["bank:TEST_NAMES", "test66"]} +{"name": "de la Vega, Juan, Jr.", "tests": ["bank:TEST_NAMES", "test42"]} +{"name": "de la Véña, Jüan", "tests": ["test_string_output", "test_utf8"]} +{"name": "de la dos Vega, Dr. Juan Q. Xavier III", "tests": ["test_lastname_three_conjunctions"]} +{"name": "de la vega, dr. juan Q. xavier III", "tests": ["test_comparison_key_case_insensitive_across_formats"]} +{"name": "donovan mcnabb-smith", "tests": ["test_capitalization_with_Mac_as_hyphenated_names"]} +{"name": "dos Santos", "tests": ["test_leading_non_first_name_prefix_dos"]} +{"name": "dr Vincent James van Gogh dr", "tests": ["test_title_before_and_after_prefixed_last_name_with_middle"]} +{"name": "dr Vincent van Gogh dr", "tests": ["test_title_before_and_after_prefixed_last_name"]} +{"name": "dr Vincent van der Gogh dr", "tests": ["test_suffix_token_collision_with_two_word_prefix"]} +{"name": "dr. ben alex johnson III", "tests": ["bank:TEST_NAMES"]} +{"name": "dr. john p. doe-Ray, CLU, CFP, LUTC", "tests": ["test_comparison_case_insensitive", "test_hash_matches_case_insensitive_equality"]} +{"name": "dr. juan de la vega jr.", "tests": ["test_unpickle_legacy_state_without_derived_sets"]} +{"name": "e and e", "tests": ["test_name_is_conjunctions"]} +{"name": "e j smith", "tests": ["test_lowercase_first_initial_conflict_with_conjunction"]} +{"name": "joao da silva do amaral de souza", "tests": ["test_portuguese_prefixes"]} +{"name": "john doe", "tests": ["test_capitalize_empty_middle_produces_no_leading_space_in_surnames"]} +{"name": "john e jones", "tests": ["bank:TEST_NAMES"]} +{"name": "john e jones, III", "tests": ["bank:TEST_NAMES"]} +{"name": "john e. smith", "tests": ["test_lowercase_middle_initial_with_period_conflict_with_conjunction"]} +{"name": "john smith", "tests": ["test_eq_emits_deprecation_warning", "test_not_equal_operator"]} +{"name": "johnny y", "tests": ["test_conjunction_names"]} +{"name": "jones, john e", "tests": ["bank:TEST_NAMES"]} +{"name": "juan garcia III", "tests": ["test_capitalization_exception_for_already_capitalized_III_KNOWN_FAILURE"]} +{"name": "juan q. xavier velasquez y garcia iii", "tests": ["test_capitalization_exception_for_III"]} +{"name": "larry james edward johnson v", "tests": ["test_four_name_parts_with_suffix_that_could_be_initial_lowercase_no_period"]} +{"name": "lt. gen. john a. kenneth doe iv", "tests": ["test_capitalize_title"]} +{"name": "mack johnson", "tests": ["test_short_names_with_mac"]} +{"name": "matthëus schmidt", "tests": ["test_capitalize_diacritics"]} +{"name": "part1 of The part2 of the part3 and part4", "tests": ["test_multiple_conjunctions"]} +{"name": "part1 of and The part2 of the part3 And part4", "tests": ["test_multiple_conjunctions2"]} +{"name": "pennie von bergen wessels", "tests": ["test_prefix_before_two_part_last_name", "test_von_bergen_wessels"]} +{"name": "pennie von bergen wessels III", "tests": ["test_prefix_before_two_part_last_name_with_suffix"]} +{"name": "pennie von bergen wessels M.D.", "tests": ["test_prefix_before_two_part_last_name_with_acronym_suffix"]} +{"name": "pennie von bergen wessels MD, III", "tests": ["test_title_two_part_last_name_with_suffix_in_first_part"]} +{"name": "pennie von bergen wessels, III", "tests": ["test_two_part_last_name_with_suffix_comma"]} +{"name": "salem, abdul", "tests": ["test_lastname_comma_prefix_only_no_join"]} +{"name": "salem, abdul salam", "tests": ["test_lastname_comma_join"]} +{"name": "salem, abdul salam ahmed", "tests": ["test_lastname_comma_join_with_middle"]} +{"name": "scott e. werner", "tests": ["test_capitization_middle_initial_is_also_a_conjunction"]} +{"name": "señora María García", "tests": ["test_señora_lowercase_non_ascii_title"]} +{"name": "test", "tests": ["test_is_bound_first_name_false", "test_is_bound_first_name_true"]} +{"name": "the and Jon Dough", "tests": ["test_starts_with_two_conjunctions"]} +{"name": "vai la", "tests": ["test_prefix_names"]} +{"name": "van nguyen", "tests": ["test_capitalize_prefix_clash_on_first_name"]} +{"name": "von Braun", "tests": ["test_leading_von_is_unchanged"]} +{"name": "von bergen wessels MD, pennie", "tests": ["test_comma_two_part_last_name_with_suffix_in_first_part"]} +{"name": "von bergen wessels MD, pennie III", "tests": ["test_last_name_two_part_last_name_with_two_suffixes"]} +{"name": "von bergen wessels, pennie III", "tests": ["test_two_part_last_name_with_suffix"]} +{"name": "von bergen wessels, pennie MD", "tests": ["test_comma_two_part_last_name_with_acronym_suffix"]} +{"name": "xyz. John Smith", "tests": ["test_leading_period_abbreviation_case_insensitive"]} +{"name": "سلمان،", "tests": ["test_trailing_arabic_comma_stripped"]} +{"name": "سلمان، محمد", "tests": ["test_arabic_comma_does_not_pollute_output", "test_arabic_comma_splits_lastname_format"]} +{"name": "‏John‏ Smith", "tests": ["test_keep_bidi_control_chars"]} +{"name": "‏محمد بن سلمان‏", "tests": ["test_bidi_stripped_name_compares_equal"]} +{"name": "∫≜⩕ Smith 😊", "tests": ["test_keep_non_emojis"]} +{"name": "∫≜⩕ Smith😊", "tests": ["test_keep_emojis"]} diff --git a/tools/differential/corpus_shapes.jsonl b/tools/differential/corpus_shapes.jsonl new file mode 100644 index 00000000..3e4f9b3a --- /dev/null +++ b/tools/differential/corpus_shapes.jsonl @@ -0,0 +1,13 @@ +{"name": "John Smith", "shape": 1} +{"name": "Beethoven, Ludwig van", "shape": 2} +{"name": "Salem, Abdul Rahman Ahmed", "shape": 2} +{"name": "Smith, John", "shape": 2} +{"name": "Smith, John V, Jr.", "shape": 2} +{"name": "John Smith, PhD", "shape": 3} +{"name": "John Smith, Dr.", "shape": 4} +{"name": "de Mesnil Jean, Dr.", "shape": 4} +{"name": "de la Cruz Juan Carlos", "shape": 4} +{"name": "de la Cruz née Vega", "shape": 4} +{"name": "de la Cruz Juan Carlos", "shape": 5} +{"name": "de la Cruz Juan Carlos, Dr.", "shape": 5} +{"name": "de la Cruz née Vega", "shape": 5} diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index 7a2a9cf1..706bd973 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -1713,7 +1713,8 @@ why = "feat(#273) recognizes TYPOGRAPHIC nickname delimiters; the ASCII pairs we # # uv run python -c " # import glob, json, re -# names = {json.loads(l) for f in glob.glob('tools/differential/corpus*.jsonl') +# def _n(x): return x if isinstance(x, str) else x['name'] +# names = {_n(json.loads(l)) for f in glob.glob('tools/differential/corpus*.jsonl') # for l in open(f, encoding='utf-8') if l.strip()} # hit = [n for n in names if re.search(r'[(\"\']', n)] # only = [n for n in hit if not re.search(r'[(\"]', n) diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index 2d2043f4..8dc167da 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -267,6 +267,13 @@ issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" # per rule. See the 2.1 ledger's copy for the argument. name_regex = "(?i)^(?![^\\n]*,)[^\\n]*\\b(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)\\b.*?\\sn[e\u00e9]e\\s+\\S+$" fields = ["family", "maiden"] +# Scoped to the DEFAULT order: every diff this rule explains is a +# default-order one. The regex reaches 'de la Cruz n\u00e9e Vega', which +# corpus_shapes.jsonl tags under both family-first orders, and this +# rule sorts ahead of the two rules written for that name -- so +# order-blind it would absorb a family-first regression there and call +# it intentional. +orders = ["DEFAULT"] [[change]] issue = "fix(#360) mc moved into the never-given particles, so it folds into the family" @@ -607,8 +614,21 @@ issue = "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps # is suffix by position) and moves only by gaining the # COMMA_STRUCTURE report C2 gives a third part that is not suffix # words. +# +# The two-part spelling is ALSO compared under FAMILY_FIRST, as shape +# 4 of corpus_shapes.jsonl: the declared order applies to the +# pre-comma name on both sides, so the split does not move and the +# diff is the same {title, suffix} one from the same cause. That is +# why the name lists twice in this rule's block, once order-tagged -- +# and why this rule's `orders` names exactly those two: unlike the +# four rules at the foot of this file it genuinely explains the +# default-order diff too, and unlike an order-blind rule it stops +# there. FAMILY_FIRST_GIVEN_LAST is not listed because no shape tags +# this string under it, so a diff arriving there would be one nobody +# has looked at. name_regex = "(?i)^john,?\\s+smith,\\s*dr\\.?$" fields = ["title", "suffix", "_ambiguities"] +orders = ["DEFAULT", "FAMILY_FIRST"] [[change]] issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" @@ -1272,3 +1292,91 @@ issue = "fix(#371) a suffix never begins a name: the Ph./D. merge declines at th # on one. name_regex = "(?i)^Ph\\. D\\.( John Smith| Van Johnson|, Jr\\.)?$" fields = ["title", "given", "middle", "family", "suffix"] + +# -- the four rules below are the shape-tagged contract corpus's +# arrival (#468): corpus_shapes.jsonl compares each tagged name under +# its shape's declared order from baseline 2.0.0 up, so these are +# comparisons no run before it made, and every one of them reads +# [order: ...] in the report. The same four stand in +# expected_since_2.1.0.toml -- both baselines predate the 2.2 cycle +# where #395 and #399 landed, and a ledger's rules are its own +# baseline's history. + +[[change]] +issue = "feat(#395) a leading never-given particle bounds the family under a declared family-first order" +# 'de la Cruz Juan Carlos', compared under FAMILY_FIRST and under +# FAMILY_FIRST_GIVEN_LAST -- one rule for both, since the fold moves +# the same three roles either way. This baseline read family 'de', +# given 'la Cruz Juan Carlos': the leading chain took the particle +# alone and everything after it was one given name. #395 lets the +# chain reach 'Cruz' THROUGH ambiguous 'la' and stop there, because a +# declared family-first order says what follows the family is not more +# surname; the leftovers then distribute by the order -- given 'Juan' +# middle 'Carlos' under FAMILY_FIRST, the reverse under +# FAMILY_FIRST_GIVEN_LAST. The DEFAULT order is untouched, which is +# why the untagged sibling case row reads family 'de la Cruz Juan +# Carlos' at every baseline: with no order declared, 'Juan Carlos' +# really can be more surname. +name_regex = "^de la Cruz Juan Carlos$" +fields = ["family", "given", "middle"] +# Scoped, and this is the rule the key exists for: the same string is +# also compared under the DEFAULT order (corpus_rules.jsonl carries +# it), where the fold must NOT fire -- and if it leaked there it would +# move these same three roles. Order-blind, this rule would absorb +# that regression and call it intentional. +orders = ["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"] + +[[change]] +issue = "fix(#399)/feat(#395) a consumed maiden marker leaves the family-first fold no given name" +# 'de la Cruz née Vega' under both family-first orders. Two changes +# compound: #395 bounds the family at 'Cruz' (this baseline read +# family 'de'), and #399 consumes the marker before anything can place +# it, so 'Vega' is the maiden name rather than a leftover competing +# for the given slot. What is left is no given name at all, the right +# answer for family-plus-maiden input -- and the two orders AGREE +# after it, having nothing to distribute. +# +# Its own rule rather than a widening of `fix(#399) a maiden marker +# bounds the particle chain that swallowed it` above: that rule +# already reaches this name by regex and declines it on `fields` +# (family+maiden, no `given`), and widening it would pre-excuse a lost +# given name on the names it explains today. +name_regex = "^de la Cruz née Vega$" +fields = ["family", "given", "maiden"] +# both family-first orders diff identically here, the marker having +# left them nothing to distribute; the default-order reading of this +# string is another rule's business +orders = ["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"] + +[[change]] +issue = "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold" +# 'de Mesnil Jean, Dr.' under FAMILY_FIRST. Two changes compound, and +# the case row carries no tag of its own (cases.py's default), so the +# rule names them: with 'dr' out of the suffix vocabulary (#296) the +# comma is followed by nothing but titles, so the pre-comma run is +# read as a name under the declared order -- and #395's fold then +# takes it. suffix 'Dr.' -> title 'Dr.', family 'de' -> 'de Mesnil', +# given 'Mesnil' middle 'Jean' -> given 'Jean'. +# +# The default-order sibling 'John Smith, Dr.' is NOT here: no +# particle, so nothing folds, and its title/suffix move rides under +# `fix(#296) ... keeps its split and its title` above, whose block now +# lists that name twice -- once tagged [order: FAMILY_FIRST]. +name_regex = "^de Mesnil Jean, Dr\\.$" +fields = ["title", "given", "middle", "family", "suffix"] +orders = ["FAMILY_FIRST"] + +[[change]] +issue = "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold, the given-last spelling" +# 'de la Cruz Juan Carlos, Dr.' under FAMILY_FIRST_GIVEN_LAST: the +# rule above's story with two leftovers instead of one, which is the +# only shape in which the two family-first orders can disagree about +# the distribution. suffix 'Dr.' -> title 'Dr.', family 'de' -> 'de la +# Cruz', given 'la Cruz Juan Carlos' -> middle 'Juan' given 'Carlos'. +# +# A separate rule rather than an alternation with the one above: these +# are two chosen names, and a literal-anchored regex per name is what +# keeps a rule from reaching a string nobody wrote down. +name_regex = "^de la Cruz Juan Carlos, Dr\\.$" +fields = ["title", "given", "middle", "family", "suffix"] +orders = ["FAMILY_FIRST_GIVEN_LAST"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 5e067b72..e6d31875 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -303,8 +303,21 @@ issue = "fix(#296) dr is not postnominal vocabulary, so 'John Smith, Dr.' keeps # is suffix by position) and moves only by gaining the # COMMA_STRUCTURE report C2 gives a third part that is not suffix # words. +# +# The two-part spelling is ALSO compared under FAMILY_FIRST, as shape +# 4 of corpus_shapes.jsonl: the declared order applies to the +# pre-comma name on both sides, so the split does not move and the +# diff is the same {title, suffix} one from the same cause. That is +# why the name lists twice in this rule's block, once order-tagged -- +# and why this rule's `orders` names exactly those two: unlike the +# four rules at the foot of this file it genuinely explains the +# default-order diff too, and unlike an order-blind rule it stops +# there. FAMILY_FIRST_GIVEN_LAST is not listed because no shape tags +# this string under it, so a diff arriving there would be one nobody +# has looked at. name_regex = "(?i)^john,?\\s+smith,\\s*dr\\.?$" fields = ["title", "suffix", "_ambiguities"] +orders = ["DEFAULT", "FAMILY_FIRST"] [[change]] issue = "fix(#325) a split credential followed by another suffix after a one-word family comma reads as suffixes" @@ -628,8 +641,10 @@ issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" # kept both the marker and the maiden name. rules.md#M2 states the # stop; decisions.md#M2 records it. # -# Fields are {family, maiden}: the given name does not move, and no -# corpus name puts this shape under a non-default name_order. +# Fields are {family, maiden}: the given name does not move. The +# `orders` key below says the rest of it -- this shape IS compared +# under a declared order, on a name this regex reaches, and the rule +# has nothing to say about that reading. # # The rule requires a PARTICLE ahead of the marker because that IS the # defect -- a marker with no particle in front of it was always @@ -663,6 +678,13 @@ issue = "fix(#399) a maiden marker bounds the particle chain that swallowed it" # arrive UNEXPLAINED rather than be absorbed. name_regex = "(?i)^(?![^\\n]*,)[^\\n]*\\b(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)\\b.*?\\sn[e\u00e9]e\\s+\\S+$" fields = ["family", "maiden"] +# Scoped to the DEFAULT order: every diff this rule explains is a +# default-order one. The regex reaches 'de la Cruz n\u00e9e Vega', which +# corpus_shapes.jsonl tags under both family-first orders, and this +# rule sorts ahead of the two rules written for that name -- so +# order-blind it would absorb a family-first regression there and call +# it intentional. +orders = ["DEFAULT"] [[change]] issue = "fix(#360) mc moved into the never-given particles, so it folds into the family" @@ -1214,3 +1236,91 @@ issue = "fix(#371) a suffix never begins a name: the Ph./D. merge declines at th # on one. name_regex = "(?i)^Ph\\. D\\.( John Smith| Van Johnson|, Jr\\.)?$" fields = ["title", "given", "middle", "family", "suffix"] + +# -- the four rules below are the shape-tagged contract corpus's +# arrival (#468): corpus_shapes.jsonl compares each tagged name under +# its shape's declared order from baseline 2.0.0 up, so these are +# comparisons no run before it made, and every one of them reads +# [order: ...] in the report. The same four stand in +# expected_since_2.0.0.toml -- both baselines predate the 2.2 cycle +# where #395 and #399 landed, and a ledger's rules are its own +# baseline's history. + +[[change]] +issue = "feat(#395) a leading never-given particle bounds the family under a declared family-first order" +# 'de la Cruz Juan Carlos', compared under FAMILY_FIRST and under +# FAMILY_FIRST_GIVEN_LAST -- one rule for both, since the fold moves +# the same three roles either way. This baseline read family 'de', +# given 'la Cruz Juan Carlos': the leading chain took the particle +# alone and everything after it was one given name. #395 lets the +# chain reach 'Cruz' THROUGH ambiguous 'la' and stop there, because a +# declared family-first order says what follows the family is not more +# surname; the leftovers then distribute by the order -- given 'Juan' +# middle 'Carlos' under FAMILY_FIRST, the reverse under +# FAMILY_FIRST_GIVEN_LAST. The DEFAULT order is untouched, which is +# why the untagged sibling case row reads family 'de la Cruz Juan +# Carlos' at every baseline: with no order declared, 'Juan Carlos' +# really can be more surname. +name_regex = "^de la Cruz Juan Carlos$" +fields = ["family", "given", "middle"] +# Scoped, and this is the rule the key exists for: the same string is +# also compared under the DEFAULT order (corpus_rules.jsonl carries +# it), where the fold must NOT fire -- and if it leaked there it would +# move these same three roles. Order-blind, this rule would absorb +# that regression and call it intentional. +orders = ["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"] + +[[change]] +issue = "fix(#399)/feat(#395) a consumed maiden marker leaves the family-first fold no given name" +# 'de la Cruz née Vega' under both family-first orders. Two changes +# compound: #395 bounds the family at 'Cruz' (this baseline read +# family 'de'), and #399 consumes the marker before anything can place +# it, so 'Vega' is the maiden name rather than a leftover competing +# for the given slot. What is left is no given name at all, the right +# answer for family-plus-maiden input -- and the two orders AGREE +# after it, having nothing to distribute. +# +# Its own rule rather than a widening of `fix(#399) a maiden marker +# bounds the particle chain that swallowed it` above: that rule +# already reaches this name by regex and declines it on `fields` +# (family+maiden, no `given`), and widening it would pre-excuse a lost +# given name on the names it explains today. +name_regex = "^de la Cruz née Vega$" +fields = ["family", "given", "maiden"] +# both family-first orders diff identically here, the marker having +# left them nothing to distribute; the default-order reading of this +# string is another rule's business +orders = ["FAMILY_FIRST", "FAMILY_FIRST_GIVEN_LAST"] + +[[change]] +issue = "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold" +# 'de Mesnil Jean, Dr.' under FAMILY_FIRST. Two changes compound, and +# the case row carries no tag of its own (cases.py's default), so the +# rule names them: with 'dr' out of the suffix vocabulary (#296) the +# comma is followed by nothing but titles, so the pre-comma run is +# read as a name under the declared order -- and #395's fold then +# takes it. suffix 'Dr.' -> title 'Dr.', family 'de' -> 'de Mesnil', +# given 'Mesnil' middle 'Jean' -> given 'Jean'. +# +# The default-order sibling 'John Smith, Dr.' is NOT here: no +# particle, so nothing folds, and its title/suffix move rides under +# `fix(#296) ... keeps its split and its title` above, whose block now +# lists that name twice -- once tagged [order: FAMILY_FIRST]. +name_regex = "^de Mesnil Jean, Dr\\.$" +fields = ["title", "given", "middle", "family", "suffix"] +orders = ["FAMILY_FIRST"] + +[[change]] +issue = "feat(#395)/fix(#296) a comma followed only by a title leaves the pre-comma name to the declared order's fold, the given-last spelling" +# 'de la Cruz Juan Carlos, Dr.' under FAMILY_FIRST_GIVEN_LAST: the +# rule above's story with two leftovers instead of one, which is the +# only shape in which the two family-first orders can disagree about +# the distribution. suffix 'Dr.' -> title 'Dr.', family 'de' -> 'de la +# Cruz', given 'la Cruz Juan Carlos' -> middle 'Juan' given 'Carlos'. +# +# A separate rule rather than an alternation with the one above: these +# are two chosen names, and a literal-anchored regex per name is what +# keeps a rule from reaching a string nobody wrote down. +name_regex = "^de la Cruz Juan Carlos, Dr\\.$" +fields = ["title", "given", "middle", "family", "suffix"] +orders = ["FAMILY_FIRST_GIVEN_LAST"] diff --git a/tools/differential/shapes.py b/tools/differential/shapes.py new file mode 100644 index 00000000..5d7294d7 --- /dev/null +++ b/tools/differential/shapes.py @@ -0,0 +1,56 @@ +"""The input-shape inventory (#469): each shape's notation, the +name_order it is an input shape FOR, and the oldest baseline whose +worker can honor that order. + +Shapes 1-3 are docs/usage.rst's three given-first arrangements. +Shapes 4-5 are the family-first arrangements: their names are only +supported UNDER their order, so the differential compares them under +it and simply not at all below min_baseline -- "we don't care what +they do in other orders" is structural, not a ledger exception +(name_order and both constants shipped in 2.0.0). + +The notations are canonical SKELETONS, not exhaustive grammars: which +bucket a written word lands in (a lone post-comma credential reading +as Suffix, a trailing particle joining the Family) is the parser's +vocabulary question, not this table's. Tagging a case row with a +shape asserts that the row instantiates the shape's ARRANGEMENT under +its declared order -- not that every word-level reading the notation +could admit is pinned here. + +The CJK arrangement is deliberately absent: whether it is a third +family-first shape is #469's open question, and corpus_cjk.jsonl +covers that ground meanwhile. + +`order` is the PUBLIC constant name on the nameparser package, as a +string, because the consumer that matters is the generated baseline +worker: it imports a released wheel and resolves the name with +getattr, so a misspelling fails loudly there and this module needs no +nameparser import at all. +""" +from typing import NamedTuple + + +class Shape(NamedTuple): + order: str | None # public constant name on nameparser, None = default + notation: str + min_baseline: str # oldest baseline that supports the order + + +SHAPES: dict[int, Shape] = { + 1: Shape(None, + 'Title Given "Nickname" Middle Middle Family Suffix', + "1.4.0"), + 2: Shape(None, + "Family [Suffix], Title Given (Nickname) Middle Middle[,] " + "Suffix [, Suffix]", + "1.4.0"), + 3: Shape(None, + "Title Given Middle Family [Suffix], Suffix [, Suffix]", + "1.4.0"), + 4: Shape("FAMILY_FIRST", + "Title Family Given Middle Middle [Particle] [, Suffix]", + "2.0.0"), + 5: Shape("FAMILY_FIRST_GIVEN_LAST", + "Title Family Middle Middle Given [, Suffix]", + "2.0.0"), +}