fix(text): normalize ArrowDtype string and categorical columns

[kevincostner17]

Summary

pd.ArrowDtype(pa.string()) columns (pandas ≥ 2) now go through whitespace stripping, sentinel normalization, fix_dtypes and profile issue detection, just like the same data as string[pyarrow].

• _is_stringlike_dtype also accepts Arrow string, large_string and string_view.
• Type inference runs on a string[pyarrow] view, because Arrow-backed parse results don't implement the arithmetic the numeric check uses. Without the view, simply marking these columns as text crashes fix_dtypes with NotImplementedError: mod not implemented.

Categorical columns with text categories are now repaired, and they keep the categorical dtype and the ordered flag. This follows the choice made for #212.

• Values are normalized the same way as the equivalent object column, so the counts match.
• Categories that become equal merge (for example " a " and "a"), and sentinel categories are removed.
• fix_dtypes still leaves categoricals alone, and fd.profile reports their whitespace and sentinel issues without suggesting a dtype.

fd.clean_text now picks its default columns with the same text-dtype check. docs/cleaning-engine.md notes which column types are repaired.

Issue repro on this branch

Python 3.12 / pandas 2.3.3 (the Arrow case is pandas ≥ 2 only; categorical output is the same on pandas 1.5.3):

arrow: ['a', <NA>, '3', '4']
string[pyarrow]: ['a', <NA>, '3', '4']
categorical: category ['a', nan, 'b', nan]
object: ['a', nan, 'b', nan]
profile categorical: ['1 value(s) with surrounding whitespace', '2 sentinel value(s) meaning missing']

Tests

• tests/test_strings.py:
  • Arrow and string[pyarrow] give the same values and actions for text, numeric-looking and date-looking columns.
  • A categorical keeps its dtype and ordered flag, gets merged categories and correct counts, and ends up with the same values as the object column.
• tests/test_profile.py: profile issues for categorical and Arrow string columns.

Local lane pytest -m "not online and not large" on Python 3.12 (numpy 2.5) and Python 3.9 (pandas 1.5.3) passes, except the five sandbox tests that also fail locally on main (they need an installed package). ruff check . and mypy src/freshdata pass.

Closes #212

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: ad980a21-eb5c-4431-8a37-7af9452e437e

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)