fix(csv): preserve leading-zero ID columns on read; schema-stable atomic stream writer

[kevincostner17]

Builds on #351 (merged); rebased onto main.

Summary

Leading zeros in CSV input (#228). freshdata clean, freshdata stream and fd.clean_csv read CSV with pandas type inference, so 02134 became 2134 before preserve_leading_zeros could act. A new internal helper, freshdata._csv_io.leading_zero_dtypes, reads a bounded sample (10,000 rows; the first chunk when streaming) as text and returns {column: str} for numeric-looking columns with zero-padded values, using the dtype step's existing _has_leading_zero_ids detector. All three read paths use it, and streaming passes the same mapping to every chunk. The scan is skipped when read_csv_kwargs sets dtype/converters or preserve_leading_zeros=False. Decimal (0.25), negative, all-zero and scientific-notation columns stay numeric.

Schema-stable, atomic stream output (#248, part a). The first batch now fixes the output layout. Later CSV batches are reindexed to its columns, so a missing flag column is written empty instead of shifting values. Later Parquet tables are cast (safe=True) to its schema. An unexpected new column or a failed cast raises a clear ValueError. Output is written to a sibling <output>.partial and moved onto the output path with os.replace on success, or deleted on failure, so a failed run leaves neither a truncated file nor a .partial behind.

Behaviour changes:

• Zero-padded numeric CSV columns now load as text (non-padded numeric columns are unchanged); a zero-padded column with blanks keeps unpadded values as text (123, not 123.0).
• freshdata stream exits 1 without writing output when a later batch adds columns or cannot be cast to the first Parquet schema.
• Padding that first appears after the scanned sample is not detected.

Tests

• tests/test_csv_leading_zeros.py: CLI clean, multi-chunk CLI stream (and that every chunk gets the same dtype mapping), and fd.clean_csv keep "02134"/"007"; unpadded numeric columns stay numeric; explicit dtype/converters and preserve_leading_zeros=False skip the scan; read_csv_kwargs (e.g. sep) are honoured by the scan.
• tests/test_stream_writer_schema.py (pyarrow via importorskip): dtype-flipping batches give a well-formed CSV and a readable Parquet with the first schema; a new column (CSV and Parquet) and an uncastable batch raise and leave no output and no .partial (an existing output is untouched); the freshdata stream output breaks when chunk dtypes change: misaligned CSV, partial Parquet #248 CLI reproductions now succeed with no .partial left.

Verification

• ruff check .: passed
• mypy src/freshdata: no issues
• pytest -m "not online and not large" on the rebased branch: Python 3.12 / pandas 2.3.3: 5146 passed, 13 skipped; Python 3.9 / pandas 1.5.3: 5142 passed, 17 skipped

Closes #228
Refs #248 (part a)

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 255fde4c-65bf-4fcd-a00c-8d947f287e7f

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)