fix(polars): keep integers exact when a polars column holds nulls

[kevincostner17]

Closes #444.

pl.DataFrame.to_pandas() renders an integer column that has nulls as float64, because a NumPy integer array cannot hold one. That silently rounds every value a float64 cannot represent.

This is not limited to engine="polars". Every public entry point reads a polars source through adapters.polars.to_pandas(), so the rounding happens on the default engine too:

df = pl.DataFrame({"v": pl.Series([2**53 + 1, None, 7], dtype=pl.Int64), "k": [1.0, 2.0, 3.0]})
fd.clean(df, verbose=False)["v"].to_list()
# before: [9007199254740992, None, 7]   <- input was 9007199254740993
# after:  [9007199254740993, None, 7]

A UInt64 beyond 2**63 loses considerably more.

The fix: integer columns that hold nulls are rebuilt from the raw integers plus a null mask, giving the pandas nullable dtype of the same width (Int64, UInt64, Int32, …) — lossless, and what the same data already looked like when passed in as pandas. Integer columns without nulls round-trip exactly today and are left untouched (int64 stays int64). The native polars engine now converts its result through the same adapter instead of calling frame.to_pandas() directly.

Default-output change: a polars input whose integer column holds nulls now cleans as a nullable integer column instead of float64. This is the one behaviour change and it is the point of the fix — it belongs in the 2.2.0 tally.

Out of scope, filed separately as #465: native ingestion also drops extension dtypes without changing any value (Int64-without-nulls → int64, Float64 → float64, string → object). This PR fixes only the case where data is lost.

Verification — py3.12/pandas 2.3.3/polars 1.42.1 and py3.9/pandas 1.5.3/polars 1.36.1:

• 5 new tests in tests/test_polars_adapter.py; all 5 fail on main
• exact values for Int64/UInt64/Int32 with nulls, on the adapter, the default engine and the native polars engine
• full suite green on both: 6559 passed / 21 skipped (py3.12), 6588 passed / 17 skipped (py3.9); ruff and mypy clean

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: ef760778-fee9-4d47-a71b-6aeb380fe826

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)