fix(contracts): mixed-format datetimes, polars diff_schema, robust baseline keys, label and tz handling - #388
Merged
Conversation
…seline keys, label and tz handling - min_datetime/max_datetime: on pandas >= 2 string columns are parsed with format="mixed" (keeping the utc=True fallback for mixed offsets), so values in a second date format are no longer coerced to NaT and silently dropped. Non-null values that still fail to parse are reported as a warning-level contract.unparseable_datetime finding with a count and examples. - diff_schema converts its input with to_pandas, so polars frames no longer raise AttributeError. - compare_to_baseline(key=...) matches keys with pandas Index difference/intersection (sort=False) and aligns with .loc. A NaN key is treated as one key value, consistent with drop_duplicates, so identical frames report no added/removed records, and mixed int/str keys no longer raise TypeError from sorted(). - Non-string column labels: a shared _resolve_label maps a declared column name to the frame's label (exact label first, then a unique str() match, ValueError when ambiguous). Used by the contract checks, compound_unique, distribution drift, key/event_time lookups and diff_schema, so contract column 0 or "0" finds integer label 0 and absent columns stay "missing" findings. - Baseline datetime range: min/max timestamps are compared in UTC (naive taken as UTC), and a drift.timezone_change warning is emitted when exactly one of baseline/current is tz-aware, instead of raising TypeError. - build_baseline, compare_to_baseline, enforce_contract and diff_schema raise ValueError naming duplicated column labels (or labels that collide when converted to str), instead of crashing on frame[col] returning a DataFrame or silently overwriting a column profile. Closes #242 Closes #274 Closes #276 Refs #232 (part 4) Refs #233 (part 5) Refs #265 (part 2)
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This was referenced Sep 15, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Input-robustness fixes for the contract / baseline entry points in
enterprise/contracts.py:min_datetime/max_datetimeare parsed withformat="mixed"(theutc=Truefallback for mixed offsets is kept), so values in a second date format are no longer coerced toNaTand silently dropped. Non-null values that still fail to parse are reported as a warning-levelcontract.unparseable_datetimefinding with a count and examples. pandas 1.5 parsing is unchanged.diff_schemaon polars (diff_schema raises AttributeError on polars DataFrames #274): the input is converted withto_pandasfirst.Index.difference/Index.intersection(sort=False) and aligned with.loc. ANaNkey is treated as one key value, consistent withdrop_duplicates, so identical frames reportadded=0 removed=0 changed=0; mixed int/str keys no longer raiseTypeError._resolve_labelmaps a declared column name to the frame's label (exact label first, then a uniquestr()match,ValueErrorif ambiguous). Used by column contracts,strict_columns,compound_unique, distribution drift, key/event_timelookups anddiff_schema, so contract column0or"0"finds integer label0, and absent columns still produce missing-column findings.drift.timezone_changewarning is emitted when exactly one side is tz-aware, instead of raisingTypeError.build_baseline,compare_to_baseline,enforce_contractanddiff_schemaraise aValueErrornaming the duplicated labels (or labels that collide when converted tostr, which would otherwise overwrite each other's profile).Behaviour changes
ValueErrorin these entry points, and through them infd.validate(suite=...),fd.clean(contract=...)and the enterprise drift step.Tests
New module
tests/test_contracts_input_robustness.py(33 tests) with each issue's reproduction plus edge cases: unparseable examples, mixed time zones without FutureWarning, polarsdiff_schema, NaN and composite NaN keys, mixed int/str keys, integer labels across enforce/diff/baseline/compound-unique/key, ambiguous label resolution, timezone added and dropped, and duplicate labels across all entry points includingfd.validate.Verification
ruff check .: clean;mypy src/freshdata: no issues (202 files)Closes #242
Closes #274
Closes #276
Refs #232 (part 4)
Refs #233 (part 5)
Refs #265 (part 2)