nameparser 2.0 implementation#288
Draft
derek73 wants to merge 255 commits into
Draft
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Validate that all replace() field values are str (user error if None) - Append missing-role synthetics in canonical Role order, not kwargs order - Unquote return type annotation (postponed annotations in effect) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Casefolded seven-component tuple in canonical Role order for dedup, dict keys, and sorting. Semantic layer comparison; __eq__ remains strict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds tests/v2/test_layering.py to mechanically enforce the conventions
doc's import layering and the public v2 export surface. Appends the v2
core types (Span, Role, Token, Ambiguity, AmbiguityKind, ParsedName,
Lexicon, Policy, PolicyPatch, PatronymicRule, UNSET, GIVEN_FIRST,
FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, Locale) to nameparser/__init__.py
alongside the existing v1 HumanName export.
Turns on component-flag mypy strictness for the four v2 modules and
check_untyped_defs for tests/v2, then fixes the resulting test-only
type mismatches by using the precise constructed types (Span(...),
frozenset({...}), tuple-of-pairs) where the literal type didn't matter
to the test, and adding narrow # type: ignore[arg-type] comments only
where a test deliberately exercises runtime coercion/validation of an
intentionally mismatched static type (e.g. Token span/Ambiguity kind
coercion tests, PatronymicRule string coercion, dict aliasing test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pickle.dumps(Lexicon.default()) raised TypeError because the default slots-dataclass pickle path serializes the _cap_map MappingProxyType. Ship every other slot and rebuild the proxy from the canonical capitalization_exceptions tuple on load. Parser (Plan 3) is picklable by construction per the core spec, and a Parser holds a Lexicon. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
requires-python is >=3.11 on this branch, so the 3.10 job can no longer install the package (uv sync either fails or silently substitutes a managed 3.11). The rest of #257 (typing_extensions, classifiers) stays tracked on the issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With requires-python at >=3.11, Self imports directly from typing and the conditional dependency can never activate; ruff (UP035/UP036) flags the dead version block once the floor is raised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v1 suite is fully annotated (since #250 put tests/ under mypy), so pyproject carries no per-file ANN ignores; the new tests/v2 modules must be annotated too or 'ruff check' fails in CI. Also drops an unused import ruff flagged (F401). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A bare startswith('nameparser._types') would also admit a future
sibling like nameparser._types_helpers. Entries ending in '.' stay
pure prefixes (subpackage contents); anything else now means that
exact module or its submodules. Also carries this file's ANN
annotations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing tied the 'particle' string hard-coded in _text_for callers to the published STABLE_TAGS constant; assert the linkage in the derived-view test until Plan 3's tag-emission contract tests land. Also carries this file's ANN annotations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lexicon(titles={'Dr.': 'Doctor'}) silently kept only the keys -- the
lone quiet coercion on an otherwise fail-loud surface, and a dict here
almost always means the field was confused with
capitalization_exceptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
repr(Lexicon.empty().add(titles={'zqx'})) rendered as default() plus
ten '-N' deltas -- technically bounded, but the wrong story for the
documented build-from-empty() power-user path. Compare against both
named constructors and render the smaller diff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pure move: constructors ahead of dunders, __or__ grouped with the dunders, then properties, then the editing methods (with _edit at the head of its section per the documented exception). No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These three had no executable example anywhere in the docs, despite being the features 2.0 is pitched on. Ambiguity and AmbiguityKind appeared in zero code blocks project-wide; Token.span only in prose; name_order (#270) as a single table row. The only working examples of tokens_for(), Token and Role lived in migrate.rst -- a page a reader new to nameparser never opens. usage.rst gains two sections: "When the parser had to guess" shows the ambiguities loop on "Van Johnson" and the StrEnum affordance (kind == "particle-or-given"), which nothing previously revealed, plus the practical use: a non-empty ambiguities is a routing signal for human review. "Tokens and spans" makes the provenance claim executable -- iterate tokens, then slice original[span.start:span.end] to recover the source text. concepts.rst asserted the (0, 3) offsets in prose; now they run. customize.rst gains a name_order example contrasting the default reading of "Nguyen Van Minh" with FAMILY_FIRST, and pins the documented comma override in a second block. Every block runs under sphinx-build -b doctest in CI, so a behavior change fails the build rather than rotting quietly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three vocabulary features were explained in prose and never
demonstrated, so the reader saw the principle without its instance.
suffix_acronyms_ambiguous had thirteen lines of abstract prose ("they
narrow how an existing entry is read when it appears alone") and no
code. The shipped ma/do case makes it immediate: "Jack Ma" keeps its
family name, "Jack M.A." reads as a suffix. That example existed only
in the release log.
capitalization_exceptions showed a constructor call with no output, so
the payoff was invisible. Now chained end to end -- "jane smith dds"
capitalizing to "Jane Smith Dds" by default, "Jane Smith DDS" with the
exception added. This doubles as the only example of capitalized()'s
lexicon argument, which appeared nowhere.
Writing that example exposed a trap in the surrounding prose: the
suggested dataclasses.replace(lex, capitalization_exceptions=(("phd",
"PhD"),)) REPLACES the default pairs rather than extending them,
dropping ii/iii/iv/md -- a reader following it literally gets "John
Smith Phd". The example uses tuple(default...) + (...) and says why.
Lexicon.remove() and | were both named in prose and never shown; add a
line each. remove() uses "sir" rather than "dr", because "Dr." with a
period is recognized structurally as an abbreviation and still parses
as a title after removal -- which would have made a confusing example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Actionable knowledge had accumulated in release_log.rst, which readers consult once at upgrade time and never search for how-to. Relocate it to the task-oriented pages. migrate.rst "Before you upgrade" gains: - The pre-1.3.0 pickle constraint, called out as an ordering hazard. A Constants blob written by 1.2.x must be re-pickled ON 1.3/1.4; after upgrading, the code that could rewrite it is gone. Every other migration step is fixable afterwards, which is why this one cannot live in a changelog. - A replacement table for the warned removals (decode() for bytes, set(manager), discard(), Constants()/CONSTANTS.copy(), plain attribute assignment, .get()). The section previously assumed a clean -W error::DeprecationWarning run and offered nothing to the readers who most need it -- those who didn't get one. - A pointer to tools/differential/ for diffing 1.4 vs 2.0 over the reader's own corpus, noting it is source-repo tooling, not installed. migrate.rst also inlines the casefold example (STRASSE/Straße) and the two behavior-diff shapes worth grepping fixtures for, both of which previously redirected to the release log. locales.rst gains "What works without a pack". The page listed two packs, so a reader with Arabic or Hebrew data concluded there was no support -- while the five-script default vocabulary from #269 handles it out of the box and was documented nowhere but the changelog. Also records which entries are deliberately held back and how to add them. Writing that last example surfaced the reason to document it: adding an honorific to given_name_titles ALONE silently does nothing, because the field is a marker over titles rather than its own vocabulary. The doctest caught it; the text now says so. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lexicon guarded two of its three marker-over-base relationships: particles_ambiguous <= particles and suffix_acronyms_ambiguous <= suffix_acronyms both raised ValueError, while given_name_titles <= titles was unchecked. An entry present only in given_name_titles is never consulted by any rule, so the misconfiguration did nothing and said nothing -- the silent-degradation failure mode #256 was deprecated for. Found while writing the locales.rst example: the doctest failed because the honorific was added to given_name_titles alone. Fold all three checks into one _SUBSET_FIELDS loop; the messages are unchanged. v1 compatibility is preserved by intersecting at the shim boundary. v1 treats titles and first_name_titles as independent sets and only consults first_name_titles for a word already recognized as a title, so an entry missing from titles was unreachable in v1 too -- dropping it changes no v1 behavior while satisfying the invariant. Verified against a live 1.4.0 with a word absent from the defaults: first_name_titles alone yields no title on both 1.4 and the 2.0 facade; both fields yield the title on both. Without this the guard broke six compat tests, since Constants(titles=[...]) replaces titles wholesale while first_name_titles keeps its defaults. (An earlier check of the same question used "sheikh", which is already in the default TITLES, so it proved nothing -- the retest with a novel word reversed the conclusion.) Docs: customize.rst states the subset rule for all three pairs, and its remove() example moves from "sir" to "professor" -- "sir" is itself a given_name_title, so removing it now correctly raises. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The message stated the constraint and left the remedy to be inferred:
"given_name_titles must be a subset of titles; not in titles: سيد".
Since adding to both fields is the only thing the caller can have
meant, the error should say so.
given_name_titles marks a subset of titles; not in titles: سيد.
Add them to titles as well — add(titles={...}, given_name_titles={...})
Considered auto-adding to the base field instead, and rejected it for
2.0: a typo in a marker field would silently mint new vocabulary
(add(given_name_titles={"profesor"}) inventing a title), the same
inference would have to extend to the other two marker pairs and to
remove()'s cascade, and the direction is one-way -- strict now can
relax in 2.1, loose now can never tighten. The message carries most of
the ergonomic benefit at none of that cost.
Parametrized over all three marker pairs, via constructor lambdas
rather than **kwargs: Lexicon's signature is heterogeneous
(capitalization_exceptions is pair-valued), so dynamic kwargs fail
mypy, and tests/ is type-checked here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nameparser/config/prefixes.py asserts two invariants on its own data:
NON_FIRST_NAME_PREFIXES stays a subset of PREFIXES, and stays disjoint
from BOUND_FIRST_NAMES. Only the first had a 2.0 counterpart --
particles_ambiguous <= particles is checked on every Lexicon, while the
disjointness rule guarded nameparser's shipped vocabulary and nothing a
caller supplied.
In 2.0's complement model the second reads as
bound_given_names & particles <= particles_ambiguous
"a bound given name that is also a particle must be one that may start a
given name". A particle declared never-to-start-a-given-name cannot also
bind one; before this, the contradiction resolved by discarding the
never-given declaration at leading position, with no signal.
Note the import-time assert only protects the shipped data, and vanishes
under python -O. This check runs on every Lexicon.
v1 compatibility, as with given_name_titles: v1 has no runtime guard, so
a caller can add a never-given prefix to bound_first_names, and 1.4 then
lets the bound rule win -- "dos Santos Silva" parses first="dos Santos".
The shim reproduces that by promoting such a word to may-be-given rather
than raising on a config v1 accepted. Verified byte-identical to a live
1.4.0 for both the leading and non-leading positions, and pinned by
test_snapshot_keeps_a_bound_never_given_prefix_parseable -- the hole had
no coverage, since no existing test built that combination.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
prefixes.py asserts its own invariants at import so a bad edit fails
immediately rather than drifting until a test happens to catch it. The
other data modules had no such guard, and two of them had already
drifted.
Data fixes:
- 'esq' was in both SUFFIX_ACRONYMS and SUFFIX_NOT_ACRONYMS. Those are
the two halves of one distinction, not overlapping categories.
_classify checks suffix_words first, so the acronym membership was
unreachable -- removing it changes 0 of the 486 differential-corpus
parses. Present in v1's data too; inherited, not introduced.
- TITLES contained 'actor ' and 'television ' with trailing spaces, so
"'actor' in TITLES" was False. Parsing was unaffected because both
Lexicon and v1's SetManager normalize on ingest, which is exactly what
let the typo survive. Also inherited from v1.
Assertions added:
- suffixes.py: SUFFIX_ACRONYMS_AMBIGUOUS <= SUFFIX_ACRONYMS, and
SUFFIX_ACRONYMS disjoint from SUFFIX_NOT_ACRONYMS.
- titles.py: FIRST_NAME_TITLES <= TITLES. True by construction today
(TITLES = FIRST_NAME_TITLES | {...}), so it pins that against a
future edit making TITLES standalone.
- All six vocabulary modules plus CAPITALIZATION_EXCEPTIONS' keys:
entries stored lowercase and whitespace-free. This is the check that
found the TITLES typo.
Deliberately NOT asserted: cross-category disjointness. TITLES and
SUFFIX_ACRONYMS share 15 entries (lt, md, phd, cpt, ...) and every one
is a legitimate homograph resolved by position, as are TITLES/PREFIXES
and SUFFIX_ACRONYMS/PREFIXES.
The predicates use .strip().lower() rather than _lexicon._normalize:
config/ is the lower layer and _lexicon imports from it, so importing
back would invert the layering. Each guard was verified to fire on a
mutated copy of its own data -- an assertion that cannot fail is worth
nothing.
Note `assert` is stripped under `python -O`. These protect the shipped
data during development; Lexicon re-checks the relationships at
construction, which is what protects a caller's own vocabulary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
esq and the TITLES trailing spaces do not change any parse, so the entry says so plainly -- but membership tests against the exported constants do change, and "'actor' in TITLES" flipping from False to True is the kind of thing someone's code could depend on either way. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comparing v1.4's docs against 2.0's found content that was lost rather
than superseded. These are the corrections and the surprises.
locales.rst promised "a pack never changes a name it doesn't declare",
which reads as reassurance about names from other traditions. A pack
declares a SHAPE, not a language: parser_for(locales.RU) reads "David
Michael Abramovich" as given=Michael, family=David. v1 warned about
this in bold; the rewrite replaced the warning with the implementer's
view, in which the pack is behaving correctly. Restore the warning with
the worked case, and make the promise sentence say what "declare"
covers.
customize.rst gains the wholesale-clear recipe. v1's one-liner
(titles.clear()) is now a two-part operation, because emptying titles
alone orphans given_name_titles and raises. It also over-promised: an
empty vocabulary does not switch titles off, since a leading
period-abbreviation is structural.
usage.rst gains three behaviors, all verified intact in 2.0 and
documented nowhere:
- delimited content that is a known suffix, or ends in a period, reads
as a suffix, not a nickname ("Andrew Perkins (MBA)"); and the
exception to that, an ambiguous acronym standing alone, which keeps
the nickname reading ("JEFFREY (JD) BRICKEN"). customize.rst teaches
suffix_acronyms_ambiguous only via the periods rule (Jack Ma / M.A.),
a different disambiguator, so this was unpredictable from the docs.
- leading period-abbreviation titles, with the three bounds that keep
them from swallowing ordinary names. This is what makes unfamiliar
ranks work with no configuration, and it explains the customize.rst
caveat above.
- bool(name) to distinguish "nothing parsed" from "parsed to
something". usage.rst said parsing never raises and stopped there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
concepts.rst described the data model — string becomes tokens, tokens carry roles, fields are views — but never the decision procedure. v1's README did: a vocabulary layer claims words for what they are, wherever they sit, and a positional layer assigns whatever is left by where it sits. That framing is what customize.rst's Lexicon/Policy split rests on, so without it "which container does my setting go in?" is answerable only by consulting a table. It also carries three things v1 said and 2.0 had dropped: that an unrecognized word still gets a sensible role (the positional layer recognizes nothing), that title/suffix are really pre-nominal and post-nominal — "Dr." is a title before a name and a suffix after — and that parsing is deterministic, with no model and no training data, so a wrong parse is wrong reproducibly and can be fixed by configuration. usage.rst gains v1 README's "Supported Name Structures". Comma order was demonstrated once inside a matches() example and described once in a name_order table cell, so a reader could not learn that "Doe, John" works without stumbling onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1's customize.rst carried ~109 doctest lines; the 2.0 rewrite kept 31. Prose survived better than code did, leaving six capabilities described but never demonstrated. Each of these exists and works; only the example was missing. - bound_given_names had exactly one mention in all of 2.0's docs, a rename row in migrate.rst. That is the entire Arabic given-name feature, invisible to anyone not migrating from v1. Show the default, adding one, and emptying the set to switch it off; note the shipped set now includes Arabic-script spellings, not just transliterations. - particles_ambiguous gets the before/after that makes the flip concrete: "van Gogh" keeps given='van' because van may be a given name, "de Mesnil" has no given name at all because de may not. That second rule — a name starting with a never-given particle is entirely surname — was documented nowhere. - extra_suffix_delimiters, additive nickname delimiters, and the strip flags each get the unconfigured parse first. Seeing "Jane Smith, RN - CRNA" come back as given='RN', family='Jane Smith' is the reason a reader goes looking for the setting; the table row alone never conveyed that. The additive-delimiter example also warns about replacing the defaults, the same trap already flagged for capitalization_exceptions. - usage.rst gains v1's sorting recipe: family_base is the key you want for alphabetizing, since it drops the particles a phone book ignores. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1's customize.rst used task-shaped headings — "Removing a Title", "Don't Remove Emojis", "Bound First Names" — so a reader scanning for their problem found it. The 2.0 rewrite named its three sections after the architecture instead (Vocabulary / Behavior / Presentation), which teaches the model better but leaves nothing to scan: the word "emoji" appeared nowhere outside a table cell, and the Policy section ran 150 lines without a subheading. Keep the three architectural sections and add task-shaped subheadings under them, so the page works for both reading and lookup. All eight terms a reader would search for (emoji, title, order, suffix, nickname, maiden, case, lexicon) now appear in a heading. Also fixes a seam left by the previous commit: the original maiden/nickname precedence paragraph had ended up orphaned after the strip-flag examples, several screens below the doctest it explains, and its closing sentence duplicated the additive-delimiter example added alongside it. Moved back to introduce its own example, duplicate dropped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The three patterns were lifted from v1's README verbatim, v1 vocabulary included, so a page teaching given/family described its input shapes in first/middle/last. That undercuts the rename the API is built on. Restoring the names also recovered detail the first pass had trimmed: v1's forms distinguish a comma that separates the family name (form 2, family-first) from a comma that only sets off suffixes (form 3, still given-then-family). That distinction is the whole reason both forms are listed, so it is now stated and pinned with a third doctest. The nickname and pre-comma suffix slots are back in the patterns too, the latter matching the "Doe Jr., John" example already there. Checked the rest of the 2.0 docs for the same slip; this was the only occurrence outside migrate.rst, where v1 names are the subject. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Input shapes" says where the fields sit; nothing said which words merge into one field rather than standing alone. Between them those are most of what decides a parse, so the new section sits next to it. Organized by which field the word pulls into rather than by what kind of word it is. The fields are a closed list the reader has just learned — seven of them — whereas the word kinds (particles, bound given names, conjunctions, maiden markers, title chains, suffix runs) are an open-ended taxonomy with no natural stopping point, which is what makes the set feel incomprehensible when listed that way. The one genuinely position-sensitive behavior gets its own paragraph instead of a mostly-empty column: a particle at the start of a name has no surname to attach to yet, so it either becomes the given name or turns the whole name into a surname, depending on whether it can double as a given name. Explaining that by cause turns two arbitrary-looking outcomes into one rule with two branches. Table examples are prose, not doctests, so CI cannot catch a wrong one — each was verified by hand against the parser. The conjunction and leading-particle claims are pinned as doctests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The default output printed three things, two of them unlabeled ParsedName reprs — the parse and the capitalized view. For input that is already correctly cased those two are byte-identical, so the CLI looked like it was printing the same thing twice, and there was no way to tell which block was which. The third line was already labeled "Initials:", so the inconsistency was internal to the command. Label all three. usage.rst gains an intro to that section framing the CLI as the quick way to check how a string parses — trying a variation of a name that came out wrong, or a locale pack before wiring it into code — plus the real output, which the section previously showed only for --json. The family_base example gains its reason for existing. #130 asked for the surname split specifically for tussenvoegsels, the Dutch particle that directories file a name *under* rather than *by*. Naming the convention gives the searchable term, and the contrast doctest shows what the split avoids: sorting on family files van Gogh under "v" and de la Vega under "d". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The page has roughly tripled — 414 lines across 13 sections — since it was named. "Getting started" now sets the wrong expectation in both directions: a newcomer expects something short, and a reader returning to look up how to sort by surname or what ambiguities does never thinks to check a page called Getting started. "Using the parser" also aligns the title with the URL it has always had (usage.html), and echoes v1's "Using the HumanName Parser" for readers coming back. Dropping HumanName from it on purpose: 2.0's entry point is parse(), and naming the compatibility layer in the title of the main tour would point newcomers at the wrong API. The README already showed the strain — it linked "Getting started" and then glossed it as "the full tour". Gloss updated too; its list of contents predated input shapes, attaching words, ambiguities and tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Three containers, three concerns" never said what was in three containers, so a reader scanning the page could not tell the section was about configuration. The chime is memorable once you have read it and invisible before that. "Configuration lives in three containers" adds the subject, keeps the compact subject-verb shape of its neighbor "Two layers decide the roles", and echoes the section's own opening sentence. Deliberately not converting concepts.rst headings to the task-shaped style used in customize.rst -- one page explains, the other is looked things up in. Also fixes the page intro, which still enumerated four ideas and promised "these four ideas". Adding the two-layer section earlier today made it five; the enumeration never mentioned how tokens get their roles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
022fd07 removed 'esq' from SUFFIX_ACRONYMS because it was also in SUFFIX_NOT_ACRONYMS, and added an assert requiring the two sets to be disjoint. Both were wrong: the sets are matched with DIFFERENT normalization. The word test strips only edge periods; the acronym test strips all of them. So "Esq" matches only through the word set and "E.S.Q." only through the acronym set, and an entry in both is covering two spellings rather than repeating itself. The removal was a v1-compat regression that also lost the family name: "John Smith E.S.Q." 1.4.0: suffix='E.S.Q.' last='Smith' 022fd07: suffix='' family='E.S.Q.' The 486-name differential corpus missed it because no corpus name uses the multi-dot spelling, and the reasoning I checked instead -- "the word branch is tested first, so the acronym membership is unreachable" -- was true only for the spellings the word branch can normalize. Pinned by a case-table row so the two-spelling coverage cannot be "cleaned up" again. Replaced the bad assert with one that holds and has a real payoff: SUFFIX_ACRONYMS_AMBIGUOUS must not intersect SUFFIX_NOT_ACRONYMS. suffix_as_written ORs the two branches, so a word membership silently bypasses the period gate the ambiguous set exists to impose -- listing 'ma' in suffix_words would make "Jack Ma" lose its family name. Release log corrected: that bullet claimed parse output was unchanged for both data edits. It was true only for the TITLES trailing spaces. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
d8feb3e added given_name_titles <= titles to _SUBSET_FIELDS. That is the wrong shape for the field: _post_rules space-joins a multi-token title run and looks the JOINED string up, so "grand duke" is a legitimate entry whose WORDS are titles while the phrase itself is not. The invariant made multi-word honorifics inexpressible in the 2.0 API, and the matching shim intersection silently dropped them, reclassifying the given name: Constants: titles += {grand, duke}, first_name_titles += {grand duke} "Grand Duke John" 1.4.0: first='John' d8feb3e: last='John' Check the words instead. The original footgun -- add(given_name_titles= {"sheikh"}) doing nothing -- still raises, because 'sheikh'.split() is not in titles either. The shim now drops only entries with a non-title word, which are the ones v1 could not reach. Two more fixes to the same validation block: - _SUBSET_FIELDS carried one rationale ("an orphan is never consulted") for all its pairs, and it was false for both survivors: an orphan particles_ambiguous entry makes _assign emit a spurious ambiguity, and an orphan suffix_acronyms_ambiguous entry makes _vocab treat the word as a period-gated suffix. Each pair now carries its own reason, and the reason appears in the error. - The remedy was "Add them to {base} as well", which inverts the intent of a caller who was REMOVING from the base -- telling them to add back what they just removed. Now names both directions. And adds the invariant that guard should have been: an ambiguous suffix acronym must not also be a suffix word. suffix_as_written ORs the two branches, so the word membership bypasses the period gate the ambiguous set exists to impose -- add(suffix_words={"ma"}) was accepted and made "Jack Ma" lose its family name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
__setstate__ checked the field LAYOUT and then hand-assigned every value, so a state dict could load a Lexicon that no constructor would produce: invariants unchecked, entries unnormalized, a plain set on a frozenset field. The layout guard exists to catch version skew, but the likeliest skew is same-name/flipped-meaning rather than a changed field list -- particles_ambiguous inverted sense between v1's never-given set and v2's may-be-given set without changing its name. A 2.0-alpha pickle carrying the old sense passed the name check and loaded silently inverted. Call __post_init__ after restoring state. That re-runs normalization and all four invariant checks, and rebuilds _cap_map for free -- the hand-rolled rebuild it replaces was already duplicating that line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
020e5ad fixed .get() being swallowed by _RegexesProxy.__getattr__ but stopped at the one method that had a deprecation promise attached. items(), values() and len() had the identical failure -- v1's regexes was a dict subclass, this proxy is not, and the catch-all __getattr__ claims every unimplemented mapping name as a regex lookup: CONSTANTS.regexes.items() -> AttributeError: no regex named 'items' That is the exact confusing message the get() fix was written to eliminate, and items()/values() are the next most likely v1 idioms after get(). Spell out the read half of the mapping surface, and say in the comment why a proxy has to. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
182249a's comment claimed the particles_ambiguous union "reproduces" v1 for a config that puts a word in both bound_first_names and non_first_name_prefixes. It reproduces two of three shapes. v1's join has a reserve_last guard, so with only two pieces it does not fire: "dos Santos" 1.4.0: last='dos Santos' here: given='dos' v1's rule is piece-count dependent; a static vocabulary set cannot express that, so the deviation is irreducible without reintroducing the raise on a config v1 accepted. Narrow the claim to what the code does and pin the divergent case as a deviation, so it is recorded rather than discovered later as a bug. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each verified against a live 1.4.0. - release_log: assigning empty_attribute_default raises AttributeError, not TypeError. - release_log + migrate: only the mapping manager's AttributeError lists the known keys; the regexes proxy says "no regex named 'typo'". The two pages had also disagreed with each other. - migrate: v1's CONSTANTS.regexes.typo returned EMPTY_REGEX, not None. None was the capitalization_exceptions behavior. - README + migrate: "four removals ... which all raise on contact" was false for one of the four. A subclass overriding a parsing hook gets a DeprecationWarning and parsing proceeds -- and DeprecationWarning is suppressed by default outside __main__, so it is the opposite of raising on contact. This matters: it is the one pre-upgrade check item a reader could reasonably skip. - release_log: the "Smith, Dr." example claimed a title was being routed to first in 1.x. It was not -- 1.4.0 already gave title='Dr.'. What moved is the pre-comma piece, first -> family. The bullet now describes the change that happened instead of one that didn't. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Test gaps:
- __setstate__ now rejects a bare str component list. A str is
iterable, so the per-entry rebuild split its CHARACTERS and produced
a plausible-looking wrong name ("John" -> first "J o h n") rather
than failing at the load site. Only reachable from foreign or
hand-built state, since v1 stored lists.
- Added a test that restores a FOREIGN state dict carrying a multi-word
suffix entry. The parametrized round-trips all go through v2's own
__getstate__, which is self-consistent by construction, so none of
them exercised the 1.4-produced-blob case the fix exists for. It
passes as written; it was simply untested.
- Added a test importing all seven guarded config modules. An
import-time assert only runs if its module is imported, and
maiden_markers is lazy -- its guard fired only incidentally, whenever
some other test happened to build a default Lexicon.
Docs and comments:
- locales.rst: a pack's Lexicon fragment is validated standalone,
before the union onto the base, so a fragment that marks a word must
also carry it. Latent today (both shipped packs are policy-only) but
a trap for the next pack author.
- titles.py: the normalization comment was past tense about a bug fixed
in the same commit, so a later reader would find the claim false and
not know if it was stale. Restated as the hypothetical, which reads
correctly forever.
- _config_shim.py: name the drift-guard test instead of describing it.
Not changed: the "spec §2" citations flagged as dangling (the specs
directory is gitignored). They predate this branch and appear four
times in _facade.py; fixing one would be inconsistent, and fixing all
is its own change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A second review of the remediation found the per-word rule was wrong
too, in the opposite direction from the whole-entry rule it replaced.
Both are now gone.
The lookup key is the space-joined run of Role.TITLE tokens, built by
the PARSE rather than drawn from this vocabulary. It can contain a
multi-word TITLES member ("chargé d'affaires" is one, and the per-word
rule rejected it), and it can contain a conjunction, because 'and' in
"Sir and Dame Alex" is tagged Role.TITLE. No static relation over these
sets decides reachability. Two attempts each rejected working configs,
while the condition they guarded -- an entry nothing consults -- is
inert. Guarding a harmless case cost three broken configurations, so
the guard goes.
The shim now TRANSLATES rather than filters. v1 joins the raw title run
and applies lc(), which strips only the whole string's edge periods, so
an interior word keeps its own ("lt. col"); v2 normalizes per token and
then joins ("lt col"). Re-folding per word converts between them.
Filtering had silently swapped given and family for every multi-word
honorific containing an abbreviation or a conjunction. Verified against
a live 1.4.0: "Grand Duke John", "Lt. Col. Smith" and "Sir and Dame
Alex" now all match.
Also fixed, all from the same review:
- _normalize now strips to a fixed point. strip().strip(".") left
periods-around-whitespace half done ('. a .' -> ' a ' -> 'a'), so any
value re-normalized later changed under its owner -- which the
unpickle revalidation had just made reachable.
- __setstate__ RAISES on unnormalized state instead of quietly
correcting it. Silently rewriting caller data on load would have made
it a fourth undocumented transformation.
- The suffix_acronyms_ambiguous/suffix_words invariant kept, but the
shim now subtracts, so a v1-legal config translates instead of
raising at parse time naming a field (suffix_words) that does not
exist on v1's Constants.
- The facade's setstate guard covers Mapping and bytes and checks entry
types, not just bare str. A dict silently loaded its keys.
- _RegexesProxy gained copy(), and reports dict mutators as unsupported
rather than as missing regexes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The translation added in 23ee08e handled the reachable cases correctly but widened the degenerate ones, in both directions. v1's lookup key is lc(" ".join(title_list)) -- always single-spaced, never empty. So: - An entry holding a whitespace run ("grand duke") could never match there. Translating it collapsed the run and started matching, turning an inert entry into a behavior change. - An entry that is all periods (".", "..") folds to "" under per-word normalization, which _normset rejects -- so a config 1.4.0 accepts and ignores raised ValueError at parse time, naming a field that does not exist on v1's Constants. The same complaint 23ee08e leveled at the suffix_words error, reintroduced two lines away. Filter to entries v1's key shape can produce, then drop any that fold away. Both verified against a live 1.4.0. Two comment corrections from the same review: - the suffix_acronyms_ambiguous subtraction claimed parity with v1; it trades a raise for the two-piece deviation already documented below it ("Jack Ma" is last='Ma' in v1, suffix='Ma' here), so say that rather than claiming equivalence. - __setstate__ said "not written by nameparser" about state that an earlier commit on this branch could legitimately have written, before _normalize converged. Now "not written by this version". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subtraction in 23ee08e traded a raise for something worse. It dropped the word from suffix_acronyms_ambiguous, which ungates it -- and v1 does NOT ungate. v1's is_suffix already accepts an ambiguous acronym through the acronym branch, and reserve_last keeps it as the surname, so adding it to suffix_not_acronyms is INERT there: c.suffix_not_acronyms.add("ma"); "Jack Ma" 1.4.0: first='Jack' last='Ma' (same as unconfigured) 23ee08e: first='Jack' last='' suffix='Ma' A person with no family name, silently -- from a change whose stated purpose was avoiding a loud error. Subtract from suffix_words instead, which reproduces v1's inertness. The test that was supposed to cover this used "John Smith jd", where v1 yields suffix='jd' with or without the config, so it passed on a case the translation does not decide. Rewritten to "Jack Ma", where it does. Also from the same review: - The justification for removing the given_name_titles guard cited "chargé d'affaires" as a multi-word TITLES member that would be rejected. Neither version ever tags it -- v2 builds its key per token, so a space-bearing vocabulary entry cannot appear in one. The conjunction case ("sir and dame", where 'and' is tagged Role.TITLE) is real and sufficient on its own; the false premise is struck from the comment and from the test that pinned it. - A stale comment still pointed at the per-word check deleted in the same commit. - The unpickle drift check covered ten fields and silently re-canonicalized the eleventh; capitalization_exceptions now raises like the rest. Every configuration broken across this session's four attempts now matches a live 1.4.0: grand duke, lt. col, sir and dame, ma-as-word, all-period entry, and E.S.Q. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Four rounds of review found the same class of mistake repeatedly: inferring an invariant from data that looked inconsistent, then editing the data or config to fit. Two of the notes below are anti-cleanup warnings for exactly the fences that got removed. Gotchas: - Comparing against v1 needs PYTHONSAFEPATH=1 and a directory outside the worktree, or --isolated still imports the branch as "v1" and every comparison reports parity. This produces false confidence rather than an error, which is the worst failure mode a verification method can have. - 'esq' is in both suffix sets on purpose; the branches normalize differently, so it is coverage of two spellings, not duplication. Deleting it lost the family name for "John Smith E.S.Q.". - Lexicon.given_name_titles is deliberately unvalidated, with the reason: a conjunction inside a title run is itself tagged Role.TITLE, so no static relation over the vocabulary decides reachability. Two guards, three broken configurations. - _normalize must reach a fixed point, now that storage and match-time share one fold and unpickling re-validates. 2.0 conventions: - Invariants guard harm, not no-ops. The criterion that would have prevented most of the above: construct the config it forbids and check what actually breaks. - The shim translates; it never raises on a config v1 accepted and never silently changes the parse. Documents all four transformations with their v1-reachability arguments, and the testing rule that would have caught the last bug -- test the case the translation DECIDES, not one where both branches agree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1 reads "John Smith MA" as suffix='MA' and "Jack MA" as last='MA'.
That is not inconsistency — it is evidence-weighing. With only two
pieces, "one of them is a credential" is the less likely reading, so
the ambiguous acronym stays the surname; with three, peeling it still
leaves a given and a family name, so the credential reading wins.
2.0 had generalized the period gate over every position, making a bare
ambiguous acronym never a suffix, so "John Smith MA" came out
family='MA', middle='Smith'. That was an unclassified divergence, and
the wrong call: MA after a full name really is more likely a degree.
Root cause is the same shape as the rest of this session's bugs. In v1
suffix_acronyms_ambiguous has exactly ONE use site -- parse_nicknames'
delimited-content escape, deciding whether "(JD)" is a nickname or a
suffix. It is a delimiter-path disambiguator, not a vocabulary
property, and 2.0 promoted it to a universal rule.
Narrow it to what it can decide: peel a bare ambiguous acronym only
when at least two name pieces remain. This is v1's reserve_last
restricted to the ambiguous set -- 2.0 still peels UNambiguous suffixes
when nothing is left ("Smith PhD" -> suffix, a classified fix),
because there the vocabulary is not in doubt.
Verified against a live 1.4.0: 9 of 10 shapes now identical, the tenth
being that classified PhD fix. Four case-table rows pin both halves of
the rule, and the release-log bullet now describes it instead of
claiming periods are the only path.
Case rows use uppercase "MA" where a suffix is expected, so the
intended reading is legible without running the parser. (Case itself is
not yet an input to the decision; it could reasonably become one.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
particles_ambiguous emitted PARTICLE_OR_GIVEN; suffix_acronyms_ambiguous
emitted nothing, for the structurally identical situation -- a word in
an ambiguous vocabulary where the parser picks one of two readings.
"parse('derek gulbranson MA').ambiguities" was empty even though the
parser had just chosen credential over surname.
Two kinds now emit:
SUFFIX_OR_FAMILY (new) at the peel in _assign, in BOTH directions --
"John Smith MA" reports that MA was read as a suffix, "Jack MA" that it
was read as the family name. Both are guesses.
SUFFIX_OR_NICKNAME (reserved since the core API landed) at classify,
for delimited content the vocabulary cannot settle: "(MBA)" escapes to
suffix on vocabulary alone, so "(JD)" keeping the nickname reading was
a silent coin-flip. Emitted at classify rather than at the escape in
extract, which runs before tokenize and so has no token index to
point at.
The emission has to be at the DECISION site, not on tag presence. In
"Joao da Silva do Amaral de Souza" the token 'do' carries
vocab:suffix-ambiguous but sits mid-name and is never at the peel
boundary -- no choice was made, so nothing is reported. Same reasoning
excludes "Ma, Jack", where a comma fixes the family before the question
arises: that mirrors the existing rule that PARTICLE_OR_GIVEN is not
emitted on the FAMILY_COMMA path. An ambiguity is a property of a
decision, not of a token.
Corpus impact is ~1%: 5 of 486 names.
test_every_ambiguity_kind_has_a_registered_trigger caught the new kind
immediately and the case table's exact-ambiguities assertion caught all
six affected rows -- both guards did their job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation branch for the 2.0 design — see the RFC in #285 and the umbrella issue #284 for the design discussion and feedback questions. Implementation learnings that change the design will land as amendment commits on #285, per the plan described there.
Progress
Span,Role,Token,Ambiguity/AmbiguityKind,ParsedName;Lexicon,Policy/PolicyPatch/apply_patch,Locale. Frozen, slotted, hashable, picklable dataclasses with eager fail-loud validation; ~110 dedicated tests undertests/v2/.render()/initials()/capitalized()/__str__)Parser/parse()/parser_for()/matches()+ shared case tableHumanNamefacade +CONSTANTSshim (existing test corpus as regression harness) — full v1 suite (1,240+ tests) reconciled and passing against the facade; differential harness (tools/differential/) verifies 1.4-on-PyPI vs the facade over 486 corpus names with one classified diffru,tr_az) — shipped with the non-interference gate, Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269 default vocabulary, and CLI--localeNotes on what's here so far
TypeError, well-typed bad value →ValueError, enum lookups stayValueErrorper stdlib precedent.tests/v2/test_layering.py), and the seven-field canonical order derives fromRoledeclaration order everywhere — no ordering is ever stated twice.nameparser/configdata modules through 2.x, including the newmaiden_markers.py(Recognize maiden-name markers like "née" and "geb." (Jane Smith née Jones) #274 — marker words like "née"/"geb."; decisions recorded on the issue).🤖 Generated with Claude Code