fix: positional writes for duplicate row labels, UTC date bounds, honest join-key scoring - #383
Merged
Merged
Conversation
… honest join-key scoring clean_text (#231 part 2): changes were collected in a dict keyed by row label and written back with .loc, so rows sharing a label (e.g. after pd.concat) all received the last row's cleaned value. Changes are now collected by position and written with iloc; the report still lists the original row labels. validate_fields (#231 part 3): suspect cells were read with series.loc[label], which returns a Series on a duplicated label and made _check_value raise "truth value of a Series is ambiguous". Each column is now validated on a reset_index(drop=True) copy, and positions are mapped back to the original labels in issues and the normalization audit. validate_fields date bounds (#233 part 3): an offset-aware value compared with a naive min/max bound (or the reverse) raised TypeError. Both sides are now compared in UTC, with a naive side taken as UTC. The vectorized prescreen parses with utc=True for the same semantics, so mixed offsets no longer knock every cell onto the slow path. suggest_join_keys (#231 part 4, #272, #273): - Cells were read with frame.at[label, col], which returns a Series on a duplicated label, so exact matches were missed. Rows are now addressed by position (blocking, scoring and ambiguity resolution) and candidates keep the original labels. - Values were compared through str(), so two missing keys ('nan' == 'nan') scored a confident 1.0 match. A field now scores 0.0 when either value is None/NaN/NaT/pd.NA/"", and missing values are excluded from exact-key overlap. - An int key column and a NaN-promoted float column never agreed ('101' vs '101.0'). Key text now renders finite integral floats below 2**53 without the trailing .0, in overlap, blocking and similarity. Closes #272 Closes #273 Refs #231 (parts 2-4) Refs #233 (part 3)
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This was referenced Sep 15, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.loc, so rows sharing a label (e.g. afterpd.concat) all received one row's cleaned value. Changes are now collected by position and written withiloc; the report still lists original labels.series.loc[label]returned a Series and_check_valueraised. Each column is validated on areset_index(drop=True)copy and positions are mapped back to the original labels in issues and the normalization audit (covers suspect-cell, rare-category, outlier and column-consensus checks).min_value/max_value(or the reverse) raisedTypeError. Both sides are compared in UTC, with a naive side taken as UTC; the vectorized prescreen parses withutc=Truefor the same semantics.pd.NA/""; missing values are excluded from exact-key overlap..0, in overlap, blocking and similarity, so101and101.0agree.Behaviour notes: date bounds are evaluated in UTC;
""no longer counts toward exact-key overlap; int and float blocking columns now share blocks.Tests
New module
tests/test_index_alignment_and_join_keys.py:pd.concat-style index) give the same results as a unique-index control forclean_text,validate_fields(identifier, numeric outlier, rare category, no-spec consensus) andsuggest_join_keys(with blocking and ambiguity);_key_textedge cases (inf, NaN, 2**53, float32, bool).All issue repros fail on main before this change and pass with it.
Verification
ruff check .— cleanmypy src/freshdata— no issues (202 files)Closes #272
Closes #273
Refs #231 (parts 2-4)
Refs #233 (part 3)