fix(domains): read an integral float code column as integer text - #488
Merged
Merged
Conversation
A numeric code column with one blank cell loads from CSV as float64, so
Series.astype("string") renders 10000266 as "10000266.0" and the trailing 0
reads as an extra digit. Every row of a perfectly valid column then fails a
digit-only pattern:
GS1-008 gpc_brick_code, int64 -> passed, 0 violations, trust 0.25
GS1-008 gpc_brick_code, float64 -> FAILED, 3 violations, trust 0.0625
Same codes, same rule, four-fold trust drop, decided by whether one cell
happened to be blank. Both regex rules in the repo were affected: GS1-008
([0-9]{8}) and FIN-008 ([A-Za-z0-9]{4,12}).
The repository had already settled the intended behaviour. retail/validator.py
carried _integral_float_text for exactly this case on the GTIN checks, and its
docstring describes this same CSV-blank-cell scenario -- the shared rule engine
simply never used it. The helper now lives in domains/base.py as
integral_float_text, _check_regex applies it, and retail imports it rather than
keeping a second copy.
Deliberately narrow: only an integral, finite float is rewritten. A genuine
decimal keeps its fraction, NaN and ±inf pass through (int(nan) would raise),
and text, integers and None are untouched -- so zero-padded strings keep their
padding and a real violation is still reported. Verified: a frame of genuinely
invalid codes still fails with all four rows flagged.
Three assertions in the new test file fail on main, across both affected rules.
Also updates test_domain_validation_lane.py, which pinned this defect as
current behaviour when #482 found it. That test is a tripwire for the fix and
the fix duly tripped it, so it now asserts the repaired behaviour with the
history kept in its docstring. This is the second time a pinned-defect test has
had to be flipped by the fix that resolved it (see #485); worth checking for
others before landing a behaviour change.
Full suite 7107 passed / 0 failed, coverage 94.94%; gauntlet gates all pass;
ruff clean repo-wide.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A numeric code column with one blank cell loads from CSV as
float64, soSeries.astype("string")renders10000266as"10000266.0"and the trailing0reads as an extra digit. Every row of a perfectly valid column then fails a digit-only pattern.Same codes, same rule, four-fold trust drop, decided by whether one cell happened to be blank. Both regex rules in the repo were affected: GS1-008 (
[0-9]{8}) and FIN-008 ([A-Za-z0-9]{4,12}).Found by the Phase 8 domain lane (#482) and reproduced independently before fixing.
The intended behaviour was already settled
retail/validator.pycarried_integral_float_textfor exactly this case on the GTIN checks, and its docstring describes this same CSV-blank-cell scenario. The shared rule engine simply never used it.The helper now lives in
domains/base.pyasintegral_float_text,_check_regexapplies it, and retail imports it rather than keeping a second copy.Deliberately narrow
Only an integral, finite float is rewritten:
10000266.0"10000266"12.512.5— a real decimal keeps its fractionnan,±infint(nan)would raise"0250""0250"— text untouched, padding survives7,NoneSo a real violation is still reported: verified that a frame of genuinely invalid codes still fails with all four rows flagged.
A note for reviewers
This also updates
test_domain_validation_lane.py, which pinned this defect as current behaviour when #482 found it. That test was a tripwire for the fix, and the fix duly tripped it — it now asserts the repaired behaviour, with the history kept in its docstring.This is the second time a pinned-defect test has had to be flipped by the fix that resolved it (see #485). Worth grepping for
DEFECT/FINDING (Sbefore landing any behaviour change.Verification
main, across both affected rules.python -m benchmarks.gauntlet run --check: all gates passed.ruff check .clean repo-wide.