fix(dtypes,strings): never abort cleaning on an unusual object cell

[kevincostner17]

Summary

Three crashes on the default fd.clean path, found by fuzzing. Each aborts the whole call on data that arrives routinely from databases, JSON and spreadsheets.

• Closes fd.clean raises UnicodeDecodeError on an object column containing non-UTF-8 bytes (pandas 2.x) #447 — non-UTF-8 bytes raise UnicodeDecodeError on pandas 2
• Closes fd.clean raises TypeError 'boolean value of NA is ambiguous' on an object column holding pd.NA  #448 — pd.NA beside a list or dict raises TypeError: boolean value of NA is ambiguous
• Closes fd.clean raises TypeError('Need to pass bool-like values') on an object column mixing booleans  #451 — booleans mixed with pd.NaT raise TypeError: Need to pass bool-like values

They share one assumption: that every cell of an object column is scalar text.

Root cause

• fd.clean raises UnicodeDecodeError on an object column containing non-UTF-8 bytes (pandas 2.x) #447: dtype inference inspects a sample with astype("string"). pandas 2 decodes bytes when casting to StringDtype, so one non-UTF-8 byte aborts the cast — a BLOB read straight out of a database is enough. pandas 1.5 left such a cell untouched, so the two versions disagreed.
• fd.clean raises TypeError 'boolean value of NA is ambiguous' on an object column holding pd.NA  #448: the strip and case passes counted repairs with stripped.ne(s). That flex comparison hands object columns to NumPy, which calls bool() on pd.NA != pd.NA and raises. A list or dict cell keeps the column object-dtype, which is exactly how JSON-derived data looks, so [pd.NA, []] was enough. Every API that runs the pipeline was affected: clean, explain_clean, compare_clean and clean_enterprise.
• fd.clean raises TypeError('Need to pass bool-like values') on an object column mixing booleans  #451: BooleanArray accepts only None/NaN as a missing value, so a pd.NaT raised even though the caller had already treated it as missing. It only triggered when the frame had another column, because that changed dtype-inference order and made the single-column case escape.

Behaviour change

• Bytes cells: text inspection falls back to a view that keeps str cells and ignores everything else, so an undecodable cell is left exactly as it was, on both pandas versions. ASCII bytes are unaffected.
• Container cells: repair counts are computed only over positions holding a real str. Lists, dicts, sets and tuples are never rewritten and never counted, and the reported repair count no longer includes cells the pipeline did not touch.
• Booleans with NaT: missing cells are normalized to NaN before the boolean cast, so the real booleans still convert and the missing cells stay missing.

Default-output changes

None for frames that already cleaned successfully.

• Frames that previously raised now return a cleaned frame. The unusual cell is passed through untouched, and the rest of the column is cleaned as usual.
• Repair counts in the report are unchanged for columns of scalar text. They only differ where a column also holds a container cell, which previously raised.
• Golden suites are unchanged (tests/test_golden.py passes untouched).

Tests

• tests/test_dtypes.py: an undecodable bytes cell survives while the rest of the frame cleans; ASCII bytes columns are unchanged; booleans mixed with NaT clean both with and without a second column (the inference-order trigger); ordinary boolean columns still convert.
• tests/test_strings.py: pd.NA beside a list cell cleans; list, dict, set and tuple cells are passed through while ordinary text is still stripped; repair counts exclude untouched container cells.

Verification

• ruff check and mypy src/freshdata: clean.
• All three campaign repros exit 0 on py3.12 / pandas 2.3.3 and py3.9 / pandas 1.5.3; each fails on main.
• tests/test_dtypes.py, tests/test_strings.py and tests/test_golden.py: 111 passed on py3.12, 108 passed / 3 skipped on py3.9.
• Full not online and not large lanes: py3.12 6514 passed, 17 skipped; py3.9 6485 passed, 21 skipped.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 119c48bc-62d4-4a1c-9fb3-3c16d08d214f

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)