fix: route every pd.to_numeric call through a shared exponent-overflow guard

[kevincostner17]

Summary

Builds on the merged #407. pandas < 3 can segfault in pd.to_numeric on text that starts with a scientific-notation token whose exponent overflows a C int, even when trailing text follows (for example "81e3104049863b72"). #407 guarded dtype inference only. About 30 other direct pd.to_numeric calls in src/ could still pass such text to the parser and crash the process: domain validators, fieldcheck, CSV leading-zero detection, time-series scoring, MissForest, semantic checks and learning.

• New helper. freshdata._numeric.safe_to_numeric(values, **kwargs) keeps unsafe cells away from pandas and otherwise forwards to pd.to_numeric unchanged (errors=, downcast=, dtype_backend=; index, name and dtype preserved). It accepts a Series, Index, array-like or scalar.
• Which cells are guarded. Only cells whose leading exponent has ten or more significant digits, the smallest size that can overflow pandas' C int exponent accumulator.
  • Every shorter exponent parses exactly as pandas parses it, including subnormal and out-of-range values.
  • Guarded cells become missing with errors="coerce", raise pandas' own Unable to parse string error with errors="raise", and come back unchanged with errors="ignore".
• Cost. Numeric, boolean and datetime dtypes skip the check. Text is screened as one joined string, and only a column containing an exponent marker followed by ten digits pays for the per-cell check.
• Shared bound. The guard pieces move into _numeric.py, and steps/dtypes.py imports them, so dtype inference uses the same bound.
• Call sites. 29 call sites are migrated, plus the dtypes guard. Calls on provably numeric dtypes stay direct: streaming/_state.py, streaming/_drift.py, imputation/missforest.py _features, and steps/memory.py.

Behaviour change: dtype inference now keeps valid subnormal and underflow values (e.g. "4.9e-324", "1e-310", and "5e-400" → 0.0). The previous guard masked any exponent beyond ±308 and dropped them to missing.

Root cause

pandas < 3 precise_xstrtod accumulates the exponent digits in a C int (n = n * 10 + digit, up to 17 digits). It adds that to a mantissa adjustment with no overflow check, and does so before it rejects trailing text. See pandas-dev/pandas#62617, #63089 and #63167; the fix is in pandas 3.0 via pandas-dev/pandas#62741. This repo pins pandas<3. #407 fixed the prefix match in dtype inference; this PR applies the guard to every other call site and narrows it to exponents that can actually overflow.

Tests

tests/test_numeric.py:

• Parity with pd.to_numeric in every errors= mode and with downcast, across Series, Index, list, ndarray and scalars. Inputs are 4,000 seeded hex tokens plus subnormal, underflow, out-of-range and 9- vs 10-digit exponent tokens. The raw pandas baseline runs in a child interpreter, so a missed crash token fails the test instead of killing the run.
• The exponent bound. 1e999999999 and 1e0000000001 are not guarded. 1e1000000000, 1e2147483648 and a 5000-digit exponent are.
  • The one difference from pandas is 1e-1000000000. It has a ten-digit exponent, so it is guarded, while pandas underflows it to 0.0. A test records this.
• Edge cases. object, string, bytes, categorical, NaN/None, empty input, numeric passthrough, downcast, dtype_backend (pandas ≥ 2), invalid arguments and 2-D input.
• Crash-token tripwire. A tripwire on pd.to_numeric checks that coerce, raise and ignore never hand an overflowing exponent to pandas.
• Public APIs. A child process (-X faulthandler, with a timeout) runs fd.validate_fields, run_domain(..., "finance") and fd.clean on crash tokens, with the same tripwire.
• Static guard. It parses src/freshdata and fails on any to_numeric reference outside _numeric.py, the call sites left for a follow-up, and the justified numeric-only sites.

tests/test_dtypes.py:

• The fix(dtypes): mask leading exponent overflows before pandas parses numbers #407 parity baseline runs in a child interpreter.
• Tests that asserted the old ±308 masking now assert parity with pandas.
• A new end-to-end test checks that dtype inference keeps subnormal and underflow values.

Verification

• ruff: all checks passed. mypy src/freshdata: no issues (204 files).
• pytest -m "not online and not large":
  • py3.12 / pandas 2.3.3: 5875 passed, 14 skipped
  • py3.9 / pandas 1.5.3: 5846 passed, 18 skipped
  • Changed test files run 5× on each interpreter, all green.
• Raw pandas vs safe_to_numeric, identical on pandas 2.3.3 and 1.5.3:

Token	pandas coerce	helper coerce	pandas raise	helper raise
4.9e-324	5e-324	5e-324	5e-324	5e-324
1e-320	1e-320	1e-320	1e-320	1e-320
1e-310	1e-310	1e-310	1e-310	1e-310
5e-400	0.0	0.0	0.0	0.0
2.2e-308	2.2e-308	2.2e-308	2.2e-308	2.2e-308
1e309 / 1e400 / -1e400	nan	nan	ValueError	ValueError
7e123456789 / 1e999999999	nan	nan	ValueError	ValueError
1e0000000001	10.0	10.0	10.0	10.0
1e2147483648	(crashes pandas < 3)	nan	(crashes pandas < 3)	ValueError

• Benchmark, 1M rows. Object-column timings are medians of 15 interleaved runs.

Column	Call	pandas 2.3.3	pandas 1.5.3
Object, no e	raw pd.to_numeric	147 ms	136 ms
Object, no e	safe_to_numeric	164 ms	156 ms
Object, no e	dtype-inference guard, before → after	231 → 162 ms	199 → 156 ms
float64	raw and helper	0.01 ms	0.02 ms

On a text column, the helper adds about 17 ms per 1M rows. That is the fixed cost of the joined-string screen. Float columns cost nothing.

• Against the unmigrated code, the public-API tripwire fired for validate_fields and the finance domain validator.

Remaining call sites

Left for a follow-up PR:

• src/freshdata/enterprise/contracts.py:779
• src/freshdata/enterprise/contracts.py:877
• src/freshdata/enterprise/contracts.py:948
• src/freshdata/enterprise/contracts.py:1793

When migrating these, remove their entries from _DEFERRED in tests/test_numeric.py.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 7473c5ad-8bee-4f8e-a147-055c20cbd6ef

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)