fix(duckdb): exact metadata scan, all-empty columns, ordered dedup, Unicode whitespace

[kevincostner17]

Summary

#199: inf/NaN crash and inexact missing_before.

• MetadataScanner.from_duckdb no longer runs SUMMARIZE twice. That call raised STDDEV_SAMP is out of range on any inf/NaN float column, and it rebuilt missing_before from a rounded null percentage.
• It now reads column types from DESCRIBE and runs one exact COUNT/approx_count_distinct query, counting float NaN as missing the way pandas does.
• The SQL pipeline reads real NaN (from Arrow, Parquet or Polars sources) as NULL.
• Outlier fences are computed from finite values only, matching pandas drop_infinite.
• A non-finite fill value is written as a cast literal; a bare inf would be parsed as a column name.

#201: dropping all-empty columns.

• When every column was empty, DuckDB used to record the drop but run SELECT *, so every column came back.
• It now returns a zero-column frame that keeps the row count, as the pandas pipeline does.
• The Polars engine had the same parity gap (a zero-column Polars frame has height 0) and is fixed the same way.
• Recorded as a backend_differences entry:
  • polars/arrow output, which cannot hold rows without columns
  • output_format="duckdb" relations, which cannot have zero columns and so keep them

#202: full-row dedup order.

• SELECT DISTINCT returned rows in arbitrary order and ignored duplicate_keep.
• A scan-order row_number() ordinal now keeps the first or last occurrence and restores input order: QUALIFY … ORDER BY ordinal.
• The sort only runs when rows are actually removed.

#204: Unicode whitespace.

• DuckDB stripped with RE2 \s, which is ASCII-only and misses \v. Polars stripped with Rust's whitespace set, which misses \x1c–\x1f.
• Both now strip exactly the str.isspace set (_util.PY_WHITESPACE, checked against Python by a test), so they match the pandas engine.

Tests

tests/test_execution/test_engine_parity_edge_cases.py, run on pandas, Polars and DuckDB:

• every PY_WHITESPACE character, compared against str.strip()
• inf does not crash; mean-impute with inf matches pandas; clip uses finite fences
• Arrow NaN counts as missing
• missing_before is exact on 100,003 rows
• all-empty columns match pandas, with polars/arrow output and native relation cases disclosed
• dedup keep="first"/"last" matches pandas order on 5 and 20k rows, including a Parquet source, with duckdb_threads=4

tests/test_execution/test_metadata_scanner.py adds exact counts with inf/NaN.

Verification

• Full lane pytest -m "not online and not large" on Python 3.12 (numpy 2.5) and 3.9 (pandas 1.5.3): all pass except the five sandbox tests that also fail locally on main (they need an installed package). Targeted engine tests re-run after rebasing onto fix(text): normalize ArrowDtype string and categorical columns #220.
• Row-order check on the declared floor: Python 3.9 venv with duckdb==0.10.0 runs the parity, metadata and DuckDB engine tests: 42 passed.
• ruff check . and mypy src/freshdata pass.

Closes #199
Closes #201
Closes #202
Closes #204

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a5d198ae-f58d-4ee3-85cf-40aeb5d2ec95

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)