Skip to content

Commit af450cd

Browse files
derek73claude
andauthored
Give the conjunctions, bound given names and birth markers their languages (#751) (#780)
* Give the conjunctions, bound given names and birth markers their languages (#751) The last 44 words in und.toml but bon and dí move to the files of the languages whose names they occur in, by the record each one carries: the #269/#397 conjunctions to en, fr, de, es, pt, it, ca, pl, ru, bg, uk, el and ar, with & in mul; the bound given names to ar; the birth markers by the #274 module docstring, with z domu in pl and 旧姓 in ja. ca.toml and nn.toml are new. No compiled set moves; the snapshot matches. The vetting comments become fields: notes and issues on the connectives, the y and Cyrillic may_be_initial keep-outs become exclusions, abd's comment is its note in ar.toml, and roz, removed from the birth markers on 2026-08-26, is an exclusion in cs.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Put nee beside née in fr.toml, record their gender in en.toml, and make f. a refusal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the README's total entry count, which every added entry staled Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d4fa368 commit af450cd

26 files changed

Lines changed: 268 additions & 265 deletions

‎docs/design/decisions/vocabulary-exclusions.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,10 @@
11
# vocabulary-exclusions — words kept out of a vocabulary set, and contested memberships
22

3-
The `Excluded:` and `Open:` blocks keyed by a vocabulary set rather than a rule (the entry conventions in README.md). Until #713 they sat in decisions.md after whichever entry happened to precede them, so the split filed them into the P3, W2, P5 and R4 entries; they moved here together, in their old order, so a wordlist editor finds every set's prohibitions in one place. Since 2026-10-18 a word-level refusal can be data instead — an `excluded` key on the word's entry in nameparser/vocab/*.toml (decisions/vocabulary-model) — and a block shrinks to its class-level part as its words move (the README's exception to append-only); the honorific_tails block below was the first, the particle block the second (#749), and the two title blocks the third (#750). `uv run python tools/vocab/query.py --excluded <kind>` lists the moved ones.
3+
The `Excluded:` and `Open:` blocks keyed by a vocabulary set rather than a rule (the entry conventions in README.md). Until #713 they sat in decisions.md after whichever entry happened to precede them, so the split filed them into the P3, W2, P5 and R4 entries; they moved here together, in their old order, so a wordlist editor finds every set's prohibitions in one place. Since 2026-10-18 a word-level refusal can be data instead — an `excluded` key on the word's entry in nameparser/vocab/*.toml (decisions/vocabulary-model) — and a block shrinks to its class-level part as its words move (the README's exception to append-only); the honorific_tails block below was the first, the particle block the second (#749), the two title blocks the third (#750), and the case-mask and single-letter-connective refusals and two of the maiden-marker ones (`roz`, `f.`) the fourth (#751). `uv run python tools/vocab/query.py --excluded <kind>` lists the moved ones, a refused property by its name (`--excluded may_be_initial`).
44

55
Excluded (Lexicon.conjunctions_ambiguous, the conjunction readings marked `may_be_initial` — an entry here reads as an INITIAL in a name written wholly in one case):
66

7-
- `y` — the commonest Hispanic double-surname link and this library's oldest fixture ("Velasquez y Garcia"); marking it would read the surname link of every lowercase Hispanic record as a middle initial. A caller who wants the Dutch reading adds it (2026-09-13, #383/#479).
8-
- Cyrillic `и`, `і`, `й` — #267 blessed their joining and this change narrows nothing about it; marking one would break the reading #267 adjudicated, in the one-case spellings where nothing else decides (2026-09-13, #383/#479).
7+
- The per-word refusals are data since #751 (2026-10-20): `y` in es.toml — the commonest Hispanic double-surname link and this library's oldest fixture, "Velasquez y Garcia" — and the Cyrillic `и`, `і` and `й` in ru.toml and uk.toml, whose joining #267 blessed, each carry `excluded = { may_be_initial = … }` on their conjunction reading with the reason (2026-09-13, #383/#479). `uv run python tools/vocab/query.py --excluded may_be_initial` lists them; a contributor who marks one meets the reason as a build error.
98
- `و` and any other caseless letter — inert rather than harmful, and excluded so nobody adds one believing it does something: the fork requires a token whose `upper()` and `lower()` differ, so a caseless member is never consulted (2026-09-13).
109
- Multi-letter entries (`and`, `та`, `και`) and `&` — the fork tests a single CHARACTER, so an entry longer than one letter is silently inert. The vocabulary build refuses `may_be_initial` on anything but a cased single letter (tools/vocab/build.py; an import-time assert in the conjunctions module until 2026-10-18) (2026-09-13).
1110

@@ -74,9 +73,9 @@ Excluded (MAIDEN_MARKERS, the birth_marker readings):
7473
#274 flagging "is nee safe as a default (it's also a rare
7574
surname)" as open; the question resolved silently with the shipped set. Recorded here because the included risky member deserves its analysis as much as the excluded ones; M1's (Nee) boundary covers only the enclosure path, not this marker path.
7675
- "born" — never shipped: a release-log drafting invention, caught by the 2.0 milestone audit and corrected (5ccf9f3). Recorded so nobody "restores" it; if ever proposed for real, Max Born is the counterexample to analyze.
77-
- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in.
76+
- Scandinavian "f." — collides with the initial F.; only the full participles (født/fødd/född) are safe. Data since #751 (2026-10-20): `f` in da.toml carries `excluded = { birth_marker = … }`. Czech masculine "rozený" awaits the same vetting, on its own merits rather than following the feminine form in.
7877
- Polish "z domu" — INCLUDED 2026-08-26 as the set's first phrase entry, and the contrast to draw against the roz removal below rather than against any included word. Its words fail C-i separately and it passes as a pair; the argument is in the #434 entry under #M2. The point to carry away is that the entry below could not have been rescued the same way: roz is a whole marker as written, so there is no longer unit to move the claim onto.
79-
- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes.
78+
- Czech/Slovak "roz." — SHIPPED through 2.1 and removed 2026-08-26 (Derek's call). Roz is an ordinary English diminutive of Rosalind, and matching is whole-token, case-folded and period-insensitive, so "Roz", "roz" and "roz." are one string to this set. The collision is not theoretical and it is not new: on the bare path, which predates the 2.2 work entirely, "Rosalind Roz Smith" read maiden "Smith" and NO family name at all, and "Rosalind Roz Jones Smith" read maiden "Jones Smith", because M2 hands the marker every word after it. Grounded in #vocabulary-collisions C-i — a word belongs in its set's ambiguous subset iff it is borne as an ordinary name IN THE POSITION THE VOCABULARY CLAIM ACTS ON, and under uncertainty default to ambiguous. Roz is borne exactly there: the word after a given name, which is the only position M2's claim acts on. What C-i cannot do for this set is the marking: MAIDEN_MARKERS has no ambiguous subset, and no code path consults one, so the only two expressions of C-i available here are ship and do not ship. That is why the remedy is removal rather than an ambiguous marking, and it is worth knowing before someone proposes marking the next collision instead. The full participle rozená stays and is unambiguous; a Czech caller who needs the abbreviation adds it to their own Lexicon, which is the same answer this section gives for every locale-specific vocabulary. The cost is accepted and stated: "Anna Nováková roz. Svobodová" now reads middle "Nováková roz.", family "Svobodová" — which is how 1.4.0 read it — and "Rosalind Roz Smith" is back to 1.4.0's given/middle/family. Pinned by tests/v2/cases.py::diminutive_that_was_a_marker_keeps_the_family and ::full_participle_marker_still_consumes. The refusal is data since #751 (2026-10-20): `roz` in cs.toml carries `excluded = { birth_marker = … }` with its reason, measured again that day (`--excluded birth_marker`).
8079

8180
Excluded (CAPITALIZATION_EXCEPTIONS — meng, edd, lac, ded, left out of the 2026-09-24 masks, #459):
8281

‎docs/design/decisions/vocabulary-model.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,3 +81,9 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
8181
- **Comments became fields.** und.toml's block over the ambiguous credentials was class-level — the criterion is decisions/suffix-acronym-collisions' and decisions/vocabulary-collisions' C-i, the reading rules/S4 and S6, the bracket escape rules/S1 — and is gone, its per-word facts already notes on `ms`, `sa`, `se`, `om`, `mc`, `vd`, `meng` and `lac`; `ba` gained the note that earned its marking. The block over the case masks became `excluded = { cased = … }` on `meng`, `edd`, `lac` and `ded`, each with the rendering a mask would cost (decisions/vocabulary-exclusions' CAPITALIZATION_EXCEPTIONS block keeps the class-level part), and the credential half of `abd`'s comment became `expands` and a `note` on its en.toml reading.
8282
- **`mc` sorts as a credential now.** An entry sorts under its earliest kind and `credential` precedes `clan`, so en.toml's `mc` left `mac`'s side for the credential block, its `same_as` kept.
8383
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind credential --file en --count`, `--kind credential --file de`, `--kind suffix --file und --count`, `--excluded cased`.
84+
85+
- **2026-10-20 #751 — the conjunctions, bound given names and birth markers given their languages (part 2 of #751).** No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. The 44 entries still in `und.toml` on 85ac4167 other than `bon` and `dí` left it, so `und.toml` holds only those two particles of unrecorded origin, and the comment at its top says why (#766 asks whether they stay at all).
86+
- **Each went where its own record put it.** The conjunctions follow the #269 and #397 issues and the names their comments gave: `and`, `of` and `the` to en.toml, `et` to fr.toml, `und` to de.toml, `y` to es.toml, `e` to pt.toml and it.toml, `i` to ca.toml and pl.toml, `и` to ru.toml and bg.toml, `і`, `й` and `та` to uk.toml, `και` to el.toml, `و` to ar.toml, and `&` to mul.toml, a symbol no language owns. The bound given names, Latin transliterations and Arabic script alike, went to ar.toml, as #749 and #750 sent a transliteration to its source language; `abu`, `أبو` and `ابو` join the teknonymic readings already there. The birth markers follow the old module docstring (#274): `née`, `né` and the unaccented `nee` to fr.toml, which the docstring calls French, and to en.toml, English writing them as French does; `geb` and `geborene` to de.toml and `geboren` to nl.toml, which the docstring calls Dutch; `rozená` to cs.toml; `født` to da.toml and nb.toml, the Nynorsk `fødd` to nn.toml and `född` to sv.toml; the five Russian spellings to ru.toml; `z domu` to pl.toml (#434); `旧姓` to ja.toml (#309). Two files are new: ca and nn.
87+
- **Comments became fields.** The block over the single-letter connectives (#383, #479, #397) was class-level, held by decisions/P8 and rules/P3, and is gone; its per-word facts are notes on `e` and `i`, and its keep-outs, `y` and the Cyrillic `и`, `і` and `й`, are `excluded = { may_be_initial = … }` (decisions/vocabulary-exclusions' conjunctions_ambiguous block keeps the class-level part). The #269 blocks over the Cyrillic, Greek and Arabic connectives and the Arabic-script bound given names became an `issue` on each and notes on `і`, `و` and `عبد`; `abd`'s comment became its note in ar.toml, beside the credential note part 1 left in en.toml. Spellings carry `same_as` where one word has several (`abdel` and `abdal` of `abdul`, `abou` of `abu`, `й` of `і`, `nee` of `née`, the е spellings of the ё ones), and gender is recorded where the form marks it (`née` and `né`, `geborene`, `rozená`, the Russian participles).
88+
- **Two keep-outs became data that had been prose.** The Scandinavian abbreviation `f.`, which collides with the initial F., is an exclusion in da.toml. `roz`, removed from the birth markers on 2026-08-26, is an exclusion in cs.toml with its reason, measured again: with it listed, `Rosalind Roz Smith` reads maiden 'Smith' and takes Rosalind for the family, where the 2026-08-26 record says it left no family name at all.
89+
- **Recompute** rather than quoting: `uv run python tools/vocab/query.py --file und`, `--kind conjunction`, `--kind bound_given --file ar --count`, `--kind birth_marker`, `--excluded may_be_initial`, `--excluded birth_marker`.

‎docs/design/vocabulary-model-rfc.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Four phases, in a fixed order:
2222

2323
| Phase | What | Changes parsing? | State |
2424
|---|---|---|---|
25-
| 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); classifying the words still in `und.toml` is #749–#751 |
25+
| 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin |
2626
| 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752 |
2727
| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 |
2828
| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 |

‎nameparser/vocab/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ A file is named for a BCP 47 language tag in canonical case: `fr.toml`, `pt-BR.t
1212

1313
Two tags are not languages:
1414

15-
- **`und`** (undetermined) holds the words not yet assigned a language. A few are still here (46 of 1,931 on 2026-10-20, after #751 gave the credentials their languages): the conjunctions, the bound given names, the birth markers and two particles, converted mechanically from the Python lists they used to be; assigning a word to a language means moving its entry to that language's file. The comments above an entry here are the vetting notes that stood above the word in the Python module, in their old order; classifying a category is what turns them into fields.
15+
- **`und`** (undetermined) holds the words no language file fits: two particles of unrecorded origin, `bon` and `dí` (since 2026-10-20, when #751 finished what #749 and #750 began, giving every other word its languages). The comment at the top of the file says why they are there; a word whose language is found moves to that language's file.
1616
- **`mul`** (multiple) holds words shared on purpose, such as the Roman numerals.
1717

1818
The script is read from the word, not from the file: `sr.toml` holds `mlađi` and `млађи` side by side. A script subtag (`zh-Hant`) is for a difference the text cannot show.
@@ -91,7 +91,7 @@ Text is compared by codepoint. The texts are already normalized, so this is the
9191

9292
The build checks the order and never rewrites a file, because a comment above an entry belongs to it and must move with it, which a TOML reader cannot see. When the build says an entry belongs somewhere else, move it with its comment. The order is checked last, so any other refusal (an exclusion's reason, a `same_as` naming no word) is reported first. The same check refuses a `same_as` chain that loops, in every file.
9393

94-
`und.toml` is exempt. Its entries keep the order of the Python lists they were converted from, and its vetting comments still point at their neighbours ("Same word as عبد below"); a word leaving it for a language file takes its place in that file's order. After moving entries out, re-read every comment that stays behind: a pointer to a neighbour ("above", "below", "the three above") may now name a block that left, and nothing checks it. #769's reviews found two such pointers in comments the PR had already rewritten ("the three above" over the Devanagari civil set, "the reason the Devanagari block above gives" over the Bengali one).
94+
`und.toml` is exempt, holding no language's words; a word leaving it for a language file takes its place in that file's order.
9595

9696
## Kinds
9797

‎nameparser/vocab/ar.toml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,7 @@ issue = 269
445445
expands = "father of"
446446
gender = "m"
447447
note = "\"Abu Bakr\" -- Abu reads as the given name; also a bound given-name word, so it joins forward (P5)"
448+
[word.bound_given]
448449

449450
[[word]]
450451
text = "أبو"
@@ -454,6 +455,8 @@ issue = 269
454455
expands = "father of: abu, hamza spelling"
455456
gender = "m"
456457
note = "Left ambiguous like its Latin transliteration 'abu': \"Abu Bakr\" reads Abu as a given name"
458+
[word.bound_given]
459+
issue = 269
457460

458461
[[word]]
459462
text = "ابو"
@@ -464,6 +467,9 @@ expands = "father of: abu, hamza-less spelling"
464467
same_as = "أبو"
465468
gender = "m"
466469
note = "left ambiguous as أبو is, both spellings vetted together (#269)"
470+
[word.bound_given]
471+
issue = 269
472+
same_as = "أبو"
467473

468474
[[word]]
469475
text = "آل"
@@ -472,3 +478,49 @@ may_be_given = false
472478
issue = 269
473479
expands = "family of: \"Al Saud\""
474480
note = "Distinct from the article ال, excluded in this file; a clan prefix, never a bare given name."
481+
482+
[[word]]
483+
text = "و"
484+
[word.conjunction]
485+
issue = 269
486+
note = "formal script attaches و to the following word (وفاطمة), so a standalone و is informal spacing, common in real data; single-character, so the single-letter carve-out protects short names"
487+
488+
[[word]]
489+
text = "abd"
490+
[word.bound_given]
491+
expands = "servant of"
492+
note = "the bare transliteration, which abdul/abdel/abdal do not match: 'abd Allah Smith' and 'Abd al-Rahman Smith' read given 'abd Allah' and 'Abd al-Rahman'; the three-token 'abd al rahman smith' is not joined, al being a particle that chains forward. Also the credential ABD in en.toml; position decides at the two ends but not in a family comma's given slot (decisions/P5)"
493+
494+
[[word]]
495+
text = "abdul"
496+
bound_given = { expands = "servant of the" }
497+
498+
[[word]]
499+
text = "abdal"
500+
bound_given = { expands = "servant of the", same_as = "abdul" }
501+
502+
[[word]]
503+
text = "abdel"
504+
bound_given = { expands = "servant of the", same_as = "abdul" }
505+
506+
[[word]]
507+
text = "abou"
508+
bound_given = { expands = "father of", same_as = "abu" }
509+
510+
[[word]]
511+
text = "umm"
512+
bound_given = { expands = "mother of" }
513+
514+
[[word]]
515+
text = "أم"
516+
bound_given = { note = "\"umm\" (mother of), hamza spelling", issue = 269 }
517+
518+
[[word]]
519+
text = "ام"
520+
bound_given = { note = "\"umm\", hamza-less spelling", issue = 269, same_as = "أم" }
521+
522+
[[word]]
523+
text = "عبد"
524+
[word.bound_given]
525+
note = "\"abd\" (servant of): the script writes Abdul Rahman as two words, عبد + الرحمن, the article attaching to the word after it, so عبد alone covers the abdul/abdel/abdal spellings -- عبد الرحمن -> given \"عبد الرحمن\""
526+
issue = 269

‎nameparser/vocab/bg.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,7 @@ generation = { issue = 661 }
5555
[[word]]
5656
text = "старши"
5757
generation = { issue = 661 }
58+
59+
[[word]]
60+
text = "и"
61+
conjunction = {}

‎nameparser/vocab/ca.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
schema = 1
2+
3+
[[word]]
4+
text = "i"
5+
[word.conjunction]
6+
may_be_initial = true
7+
issue = [397, 721]
8+
note = "the Catalan and Polish surname link (Carod i Rovira, Kowalski i Nowak); a bare I initial is as common as a bare E, so marked like e. Also the Roman numeral I, so it joins only with a name word on each side (rules/P3)"

0 commit comments

Comments
 (0)