Skip to content

fix(contracts): mixed-format datetimes, polars diff_schema, robust baseline keys, label and tz handling - #388

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/contracts-input-robustness
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/contracts-input-robustness

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Input-robustness fixes for the contract / baseline entry points in enterprise/contracts.py:

Behaviour changes

  • Frames with duplicate (or string-colliding) column labels now raise ValueError in these entry points, and through them in fd.validate(suite=...), fd.clean(contract=...) and the enterprise drift step.
  • On pandas 2, mixed-format date strings can now fail datetime-bound contracts that previously passed silently.

Tests

New module tests/test_contracts_input_robustness.py (33 tests) with each issue's reproduction plus edge cases: unparseable examples, mixed time zones without FutureWarning, polars diff_schema, NaN and composite NaN keys, mixed int/str keys, integer labels across enforce/diff/baseline/compound-unique/key, ambiguous label resolution, timezone added and dropped, and duplicate labels across all entry points including fd.validate.

Verification

  • ruff check .: clean; mypy src/freshdata: no issues (202 files)
  • Python 3.12 / pandas 2.3.3: 5073 passed, 6 skipped
  • Python 3.9 / pandas 1.5.3: 5069 passed, 10 skipped

Closes #242
Closes #274
Closes #276
Refs #232 (part 4)
Refs #233 (part 5)
Refs #265 (part 2)

…seline keys, label and tz handling

- min_datetime/max_datetime: on pandas >= 2 string columns are parsed with
  format="mixed" (keeping the utc=True fallback for mixed offsets), so values
  in a second date format are no longer coerced to NaT and silently dropped.
  Non-null values that still fail to parse are reported as a warning-level
  contract.unparseable_datetime finding with a count and examples.
- diff_schema converts its input with to_pandas, so polars frames no longer
  raise AttributeError.
- compare_to_baseline(key=...) matches keys with pandas Index
  difference/intersection (sort=False) and aligns with .loc. A NaN key is
  treated as one key value, consistent with drop_duplicates, so identical
  frames report no added/removed records, and mixed int/str keys no longer
  raise TypeError from sorted().
- Non-string column labels: a shared _resolve_label maps a declared column
  name to the frame's label (exact label first, then a unique str() match,
  ValueError when ambiguous). Used by the contract checks, compound_unique,
  distribution drift, key/event_time lookups and diff_schema, so contract
  column 0 or "0" finds integer label 0 and absent columns stay "missing"
  findings.
- Baseline datetime range: min/max timestamps are compared in UTC (naive
  taken as UTC), and a drift.timezone_change warning is emitted when exactly
  one of baseline/current is tz-aware, instead of raising TypeError.
- build_baseline, compare_to_baseline, enforce_contract and diff_schema raise
  ValueError naming duplicated column labels (or labels that collide when
  converted to str), instead of crashing on frame[col] returning a DataFrame
  or silently overwriting a column profile.

Closes #242
Closes #274
Closes #276
Refs #232 (part 4)
Refs #233 (part 5)
Refs #265 (part 2)
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: c51fe15b-fe48-4f85-b172-4d480b398d82


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant