fix(dtypes): dayfirst=True no longer corrupts ISO-8601 dates

[kevincostner17]

The defect

fd.clean(pd.DataFrame({"v": ["2021-01-05", "2021-02-11", "2021-03-09"]}), dayfirst=True)["v"]

input	before	after
2021-01-05	2021-05-01	2021-01-05
2021-02-11	2021-11-02	2021-02-11
2021-03-09	2021-09-03	2021-03-09

Month and day swapped, with no warning, no report entry and no coercion record. Silent semantic corruption through a documented public kwarg, on the most standard date format there is.

dayfirst exists to resolve ambiguous short dates such as 05/12/2021. An ISO-8601 date is YYYY-MM-DD by definition and has no ambiguity to resolve, so this was not a defensible reading of the input.

Why it was easy to miss

The corruption was data-dependent. It was silent only while every day was <= 12. A day >= 13 made the guessed format fail on that value, which dropped the parse share below datetime_threshold and triggered the mixed-format retry that produced the correct reading.

Nineteen consecutive January dates, identical data, identical dayfirst=True:

config	result
default threshold	correct
datetime_threshold=0.5	2021-01-01, 2021-02-01, 2021-03-01 … + 7 NaT

So the same column read correctly or incorrectly depending on values it happened to contain, or on an unrelated knob. A user validating on a sample containing a day >= 13 would see correct output and ship — and the pipeline would then corrupt a month where every day was <= 12.

Only the top-level dayfirst=True kwarg was affected. dayfirst="auto" and both semantic_context routes were already correct — which is why this was never caught, since those are the routes previously verified.

Cause and fix

pandas 2 infers one format for a whole column from its first value, and under dayfirst=True reads an ISO date as %Y-%d-%m:

pd.to_datetime(pd.Series(["2021-01-05"]), errors="coerce", dayfirst=True)                  # 2021-05-01
pd.to_datetime(pd.Series(["2021-01-05"]), errors="coerce", dayfirst=True, format="mixed")  # 2021-01-05

_parse_datetime now passes format="mixed" when dayfirst is set, so each value is read by its own shape.

pandas 1.x was never affected — verified directly on 1.5.3, which infers per value and returns 2021-01-05. It also has no format="mixed", hence the guard on PANDAS_MAJOR >= 2. This is a pandas 2 format-inference regression that FreshData inherited.

Verification

• Proved the tests fail without the fix rather than assuming: reverting src/ gives 3 failed, 5 passed.
• With the fix: 8 passed on py3.12/pandas 2.3.3 and on py3.9/pandas 1.5.3.
• Full suite: 7356 passed, 22 skipped, 0 failed, coverage 95.01%.
• ruff check . clean.

dayfirst still does its actual job: 05/12/2021, 06/11/2021 → 2021-12-05, 2021-11-06 with dayfirst=True, and 2021-05-12, 2021-06-11 with dayfirst=False. A column mixing ISO and slash forms now reads each by its own shape.

Compatibility impact

Output changes for ISO-8601 columns cleaned with dayfirst=True on pandas 2 — from a wrong reading to the correct one. dayfirst behaviour on genuinely ambiguous slash dates is unchanged, as are the dayfirst="auto" and semantic_context routes. CHANGELOG entry added under [Unreleased] / Fixed stating this.

Provenance

Found by mutation testing on steps/dtypes.py — not as a mutant, but while constructing adversarial inputs to kill one. Searching for inputs that distinguish two near-identical implementations is also a search for inputs the implementation mishandles.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e43350a3-b4be-4281-9508-799e339dd227

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)