fix(polars): NaN as missing, finite outlier fences, LazyFrame inputs

[kevincostner17]

Summary

#200: float NaN counted as a value. The Polars engine counted only nulls. So when a native Polars DataFrame/LazyFrame, an Arrow table or Polars-written Parquet held float NaN, empty rows and columns were kept and missing_before was wrong.

• Native sources now read float NaN as null at ingestion. pl.from_pandas already did this for pandas input.
• Outlier fences are computed over finite values only, matching pandas drop_infinite, so ±inf no longer skews the quantiles and still gets flagged or clipped.
• Visible change: Polars output for a native Polars/Arrow source now shows null where it showed NaN, the same as for pandas input.

#203: pl.LazyFrame sources.

• They crashed whenever the run needed the pandas fallback (cannot materialize source of type LazyFrame), and the default path rejected them (expected a pandas or polars DataFrame).
• The fallback now collects the LazyFrame, and fd.clean(lazyframe) cleans it like a DataFrame and returns a LazyFrame.
• fallback_policy="error" still raises before anything is collected.

Issue repros on this branch

Same output on Python 3.12 and 3.9:

#200 pandas ['a', 's'] 2 5 [('drop_empty_columns', 1), ('drop_empty_rows', 1)]
#200 polars ['a', 's'] 2 5 [('drop_empty_columns', 1), ('drop_empty_rows', 1)]
#203 {} LazyFrame
#203 {'engine': 'auto'} DataFrame
#203 {'engine': 'polars', 'strategy': 'balanced'} DataFrame
#203 {'engine': 'polars', 'strategy': 'conservative'} DataFrame

Tests

• tests/test_execution/test_polars_engine.py:
  • NaN parity with pandas for DataFrame, LazyFrame and Arrow sources, and for Polars-written Parquet
  • inf excluded from clip fences, matching pandas
  • LazyFrame through the default path, engine="auto", and Polars balanced/conservative; fallback_policy="error" still raises
• tests/test_polars_adapter.py: a LazyFrame round trip on the default path.

Local lane pytest -m "not online and not large" on Python 3.12 and 3.9 passes, except the five sandbox tests that also fail locally on main. Targeted engine tests re-run after rebasing onto #220. ruff check . and mypy src/freshdata pass.

Note: this and the DuckDB parity PR both touch execution/backends/_polars.py, in separate hunks. Whichever merges second gets rebased.

Closes #200
Closes #203

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 05004ec7-7662-47a9-9730-50351933a637

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)