Repository navigation
feat(crossref): strip company legal forms and honorifics in entity dedup (#116) - #122
Merged
Merged
Conversation
JordanCoin
marked this pull request as draft
October 21, 2026 15:43
Owner
Author
|
Held as draft for Jordan's decision: rigour pulls in pyicu, which has no wheels and needs system ICU + pkg-config (build failed here until PKG_CONFIG_PATH pointed at Homebrew icu4c). That is a rough install on Windows and on the $300 laptop the mission targets. Options: (a) merge as an opt-in extra with the ICU caveat documented, (b) drop rigour and vendor a short company-form list (the useful part is data, about 40 suffixes), (c) check whether an older rigour pinned to normality 2.x avoids pyicu. Fallback path is unchanged either way. |
New openfoia/matching.py dedup_key(name, entity_type). With the optional `match` extra (rigour>=2.5.0, also in `all`), organizations go through rigour.names.normalize_name + remove_org_types and persons through remove_person_prefixes + normalize_name, so "Clearview AI Inc.", "Clearview AI, Ltd" and "CLEARVIEW AI LIMITED" collapse to one crossref lookup. The legacy six-suffix strip still runs on rigour's output because rigour keeps "co"; a name that is only a legal form keeps the legacy key. Without rigour the key is byte-identical to the old inline logic in _deduplicate_entities. rigour depends on normality, which requires pyicu (sdist only, builds against system ICU), so the install-extras hint names libicu-dev / icu4c + pkg-config. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
…igour (#116) rigour pulls in pyicu, which has no wheels and needs a system ICU library, breaking the $300-laptop and Windows installs. dedup_key is now stdlib only: NFKC + casefold, punctuation to spaces, then strip one trailing legal form (63 entries, multi-token first) for organizations or leading honorifics for persons. A name that is only a legal form keeps its key. No transliteration and no expansion (AG does not match Aktiengesellschaft); nomenklatura is phase 2. Removes the match extra and its install-extras entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y
JordanCoin
force-pushed
the
feat/crossref-rigour
branch
from
October 23, 2026 17:57
14795e4 to
207e26a
Compare
JordanCoin
changed the title
JordanCoin
marked this pull request as ready for review
October 23, 2026 17:57
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #116.
Crossref entity dedup now collapses company legal forms and person honorifics with a vendored, stdlib-only normalizer in
openfoia/matching.py. No new dependency (rigour was dropped: it needs pyicu, which has no wheels and requires a system ICU library, breaking low-end and Windows installs).dedup_key(name, entity_type):Known limits: no transliteration (Cyrillic and Latin spellings do not merge) and no expansion ("Siemens AG" does not match "Siemens Aktiengesellschaft"). Fuzzy matching via nomenklatura stays phase 2.
Tests:
tests/test_matching.pycovers the Clearview acceptance case, Petrobras S.A./SA, Siemens AG vs Aktiengesellschaft (not merged), Dr. Jane Doe, bare "Inc.", same text across person/org (still merges, unchanged), and list ordering.🤖 Generated with Claude Code
https://claude.ai/code/session_01Wdqj51cPquyTRKmoJLQV6y