test(dtypes): cover the protection and boolean paths, and pin three crashes

[kevincostner17]

What

Two of four mutation lanes on steps/dtypes.py — the module that decides when a column's type is silently rewritten. 22 tests.

The filter that matters

Every candidate mutant was confirmed against the full 7370-test suite before a test was written, not just the 130-test targeted subset. That removed 6 of 18 candidates — a third of what a subset-only run would have reported as gaps.

This corrects how earlier numbers in this series were framed: a surviving mutant is a candidate, not a finding, until it survives the whole suite. The headline "dtypes.py is at 26.5%" was subset-relative and overstated the gap.

Lane	Candidates	Killed by full suite (not gaps)	Real gaps	Outcome
protection / leading-zero	10	5	5	4 killed by new tests, 1 proven equivalent
boolean / regex / NA-fill	8	1	7	6 killed, 2 proven equivalent

Where the real gaps were

refine_numeric_after_semantic was reachable from no existing test at all — only cleaner.py calls it, and only after an applied numeric semantic repair. Two gaps live there:

• a StringDtype column is never an object dtype, so flipping or to and makes the function skip every such column;
• flipping the leading-zero guard to or makes the retry convert nothing, ever, so the preserve_leading_zeros=False opt-out silently stops working.

The rest: the boolean-vocabulary subset test (<= → < stops a column whose values are exactly the 8-word vocabulary from being detected), the NA-fill on match masks, the literal (non-regex) decimal replacement — where a regex-metacharacter decimal separator turns "1|5" into ".1.|.5." — and the dateish screen answering no when it has nothing to inspect (a yes makes to_datetime turn a bare 42 into 1970-01-01T00:00:00.000000042).

Three equivalent mutants, proven by execution

• int64 fit guard — unreachable: _finalize_numeric consults _fits_int64 only after (nonnull % 1 == 0).all() holds, and that is false for every non-finite value (inf % 1 is nan). sys.settrace over 22 adversarial series records zero hits on the guard's return line while recording many on the function; the test asserts reached >= 10 so the proof cannot go vacuous. A counterexample was hunted first — masked Float64 would break it if inf % 1 produced pd.NA — and pandas keeps NaN.
• two mask-normalisation comparisons — no-ops on pandas 2.3.3: exhaustive enumeration of every mask pattern over {True, False, NA} for lengths 0–5 against 6 target series = 2,562 configurations, 1,806 carrying NA, zero divergences, plus a dual-evaluation census over 81,439 executions.

Both are scoped to pandas 2.3.3 / py3.12 and say so in their docstrings. On pandas < 2.0 a nullable mask containing NA raised on indexing, so the normalisation these mutants skip is what keeps the code version-proof.

Three crashes pinned (not fixed)

Found while building adversarial inputs — none is a mutant. All reproduce from fd.clean on default settings; all verified independently.

1. Duplicate index labels break the formatted-number rescue (dtypes.py:213, .loc[labels] expands each repeated label to every matching row). A duplicate index alone is harmless; it needs a duplicate index plus a formatted-number rescue plus ~20+ rows. Both natural routes fail at realistic scale:

fd.clean(pd.concat([a, b]))                         # ValueError
fd.clean(pd.concat([a, b]).reset_index(drop=True))  # fine
fd.clean(pd.read_csv(path, index_col="key"))        # ValueError when key repeats

2. A complex value beside text crashes the numeric finalizer (nonnull % 1 == 0). All-complex is fine — infer_dtype reports "complex" and the column is left alone; one complex beside text makes it "mixed" and it reaches the finalizer.

3. Outside this module, the same duplicate-index frame with a coerced casualty crashes engine/missing.py:110. Not pinned here (different module), but it means the index-uniqueness assumption wants auditing across the engine rather than patching twice.

One test deliberately does not go through fd.clean

The complex-crash test asserts _finalize_numeric directly. The end-to-end route is order-dependent: the same fd.clean call raises in a fresh process but not after certain other work in the same process, so an fd.clean-level pytest.raises was genuinely flaky under pytest-randomly (seeds 5 and 6 failed; 1–4 and 7–8 passed).

Hypotheses tested and rejected: unrestored monkeypatch or tracer (both restored, verified); global RNG (same input raised on 40/40 seeds); the _finalize_numeric calls themselves; importing the test module. The trigger is constructing the adversarial pool, whose helper is pure. infer_dtype is stable ("mixed" throughout) and _finalize_numeric is stable (always raises), so the divergence is upstream in clean's routing. Mechanism not identified — likely a pandas/numpy caching detail, but that is a conjecture and is labelled as one.

A companion test records the order dependence and asserts only what holds in either state: the frame comes back unchanged, or it raises — never a silently coerced column that has lost the complex value. Verified across 12 orderings including both formerly-failing seeds.

Scope

Tests only — no src/ change. No behaviour change, no changelog entry, no compatibility impact.

Verification

• 22 new tests across 12 random orderings — all pass
• Full suite, py3.12 — 7370 passed, 22 skipped, 0 failed, coverage 95.05%
• ruff check . clean

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9053ea48-8130-47b2-9f3e-f7c103d14de0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}