fix(domains): GTIN float rewrite, EIDR check character, ambiguous dates with time, healthcare tz - #353
Open
kevincostner17 wants to merge 1 commit into
Open
fix(domains): GTIN float rewrite, EIDR check character, ambiguous dates with time, healthcare tz#353kevincostner17 wants to merge 1 commit into
kevincostner17 wants to merge 1 commit into
Conversation
…es, tz compares Retail (#229): a GTIN column with a blank cell loads from CSV as float64, so _gtin_text rendered "4012345678901.0" and GS1-002 flagged it. The strip_nondigits repair then kept the "0" from ".0" and could write a different, mod-10-valid GTIN-14 ("40123456789010"). Integral float cells are now rendered as integer text for validation and repair, and the repair skips non-integral numbers and any text with a "." followed by digits, so such values are only flagged. Media (#259): eidr_check_char implemented the pure ISO 7064 MOD 37-2 system, but the EIDR ID Format spec uses the hybrid MOD 37,36 system over 0-9A-Z. All published EIDR IDs failed MD-C002. It now uses MOD 37,36, and "*" is no longer accepted as a check character. The known-answer test used an ID produced by the old algorithm; it now uses ten published IDs (four registry IDs and six examples from the EIDR ID Format document). CONTRIBUTING_DOMAINS.md is corrected. Finance (#260): the FIN-003 ambiguity guard was anchored at end of string, so "03/04/2024 09:30" skipped it and was parsed month-first. The guard now matches the numeric D/M/Y prefix with any trailing time, so such values are left unresolved like their date-only form. Healthcare (#233 part 2): deceased-after-birth, age-range, encounter-duration and the shared ge_date (end >= start) check compared tz-aware FHIR dateTime values with naive dates or mixed offsets and raised TypeError. They now parse with utc=True (naive values read as UTC) and use pd.Timestamp.now(tz="UTC"). Closes #229 Closes #259 Closes #260 Refs #233 (part 2: healthcare validator)
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes four domain-validator bugs.
4012345678901was checked as"4012345678901.0". Thestrip_nondigitsrepair then rewrote it to40123456789010, a different GTIN-14 that still passes the mod-10 check..followed by digits, so those values are flagged but never rewritten.eidr_check_charused pure ISO 7064 MOD 37-2 (with*). The EIDR ID Format spec uses the hybrid MOD 37,36 system over 0-9A-Z, so every real EIDR ID failed MD-C002.*is no longer accepted.CONTRIBUTING_DOMAINS.mdis corrected.03/04/2024 09:30bypassed it and was parsed month-first.ge_datechecks compared offset-aware FHIRdateTimevalues with naive dates or mixed offsets, and raisedTypeError.utc=True(naive values are read as UTC) and usepd.Timestamp.now(tz="UTC").Behaviour changes
*check character is always rejected.Tests
tests/domains/test_retail.py:tests/domains/test_media.py:*are rejectedtests/domains/test_spec_conformance.py: the published EIDR ID in row 0 is no longer flagged (MD-C002 rows are[1, 2, 3]).tests/domains/test_finance.py: an ambiguous date with a time stays unresolved; an unambiguous date with a time is still repaired.tests/domains/test_healthcare.py:deceasedDateTimeVerification
ruff check .: all checks passedmypy src/freshdata: no issues found in 202 source files-m "not online and not large"), py3.12 / pandas 2.3.3: 4262 passed, 6 skippedmainand passes on this branch, on both lanes.Closes #229
Closes #259
Closes #260
Refs #233 (part 2: healthcare validator)