fix(dtypes): mask leading exponent overflows before pandas parses numbers - #407
Merged
Merged
Conversation
…bers pandas < 3 reads the exponent digits of a numeric cell into a C int with no overflow check, before it rejects trailing text (pandas-dev/pandas#62617, fixed in pandas 3.0 by pandas-dev/pandas#62741). A cell that merely starts with such a token - e.g. the hash-masked value "81e3104049863b72" - can segfault pd.to_numeric(errors="coerce"). The existing guard only matched whole-cell scientific notation, so hash-like tokens slipped through; because masked values are random per run, CI crashed intermittently in _to_numeric_or_none. Match the scientific-notation prefix (and bytes cells, which pandas parses with the same C code) instead of the whole cell. Results are unchanged: pandas already coerces every newly masked cell to NaN.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
pandas < 3 reads the exponent digits of a numeric string into a C int with no overflow check, before it rejects trailing text (pandas-dev/pandas#62617, #63089, #63167; fixed in pandas 3.0 by pandas-dev/pandas#62741). #407 masked such cells in dtype inference only; about 30 other pd.to_numeric calls (domain validators, fieldcheck, CSV leading-zero detection, time-series scoring, MissForest, semantic checks, learning) still handed hash-like text such as "81e3104049863b72" straight to the parser and could crash the process. Add freshdata._numeric.safe_to_numeric, which keeps those cells away from pandas and otherwise forwards to pd.to_numeric unchanged (errors=, downcast=, dtype_backend=, index, name and dtype). A cell is guarded only when its leading exponent has ten or more significant digits, the smallest size that can overflow the C int accumulator; every shorter exponent, including subnormal and out-of-range values, is parsed exactly as pandas parses it. Numeric, boolean and datetime inputs skip the check; text is screened as one joined string, so clean columns stay close to free. The guard pieces move there and dtypes.py imports them, so dtype inference uses the same bound and keeps valid subnormal and underflow values. Calls on provably numeric dtypes stay direct, and a static test fails when a new unguarded call appears.
kevincostner17
added a commit
that referenced
this pull request
Sep 15, 2026
pandas < 3 reads the exponent digits of a numeric string into a C int with no overflow check, before it rejects trailing text (pandas-dev/pandas#62617, #63089, #63167; fixed in pandas 3.0 by pandas-dev/pandas#62741). #407 masked such cells in dtype inference only; about 30 other pd.to_numeric calls (domain validators, fieldcheck, CSV leading-zero detection, time-series scoring, MissForest, semantic checks, learning) still handed hash-like text such as "81e3104049863b72" straight to the parser and could crash the process. Add freshdata._numeric.safe_to_numeric, which keeps those cells away from pandas and otherwise forwards to pd.to_numeric unchanged (errors=, downcast=, dtype_backend=, index, name and dtype). A cell is guarded only when its leading exponent has ten or more significant digits, the smallest size that can overflow the C int accumulator; every shorter exponent, including subnormal and out-of-range values, is parsed exactly as pandas parses it. Numeric, boolean and datetime inputs skip the check; text is screened as one joined string, so clean columns stay close to free. The guard pieces move there and dtypes.py imports them, so dtype inference uses the same bound and keeps valid subnormal and underflow values. Calls on provably numeric dtypes stay direct, and a static test fails when a new unguarded call appears.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the intermittent
Fatal Python error: Segmentation faultin CI insidepd.to_numeric, called fromsteps/dtypes.py_to_numeric_or_none.The numeric-parse guard now masks any cell that starts with a scientific-notation token whose exponent is outside the float range. Before, it only masked cells that were entirely scientific notation. Bytes cells are covered too. Parsed results are unchanged.
Root cause
pandas < 3 reads exponent digits into a C
intwithout an overflow check (precise_xstrtod), and it does this before rejecting trailing text. See pandas-dev/pandas#62617, #63089 and #63167; the fix landed only in pandas 3.0 (pandas-dev/pandas#62741). So a hash-masked value such as81e3104049863b72segfaultspd.to_numeric(errors="coerce"). Masked values are random on each run, so only some runs produce such a token.Deterministic repro on Linux x86_64 with pandas 2.3.3. It is identical on numpy 2.4.4 and 2.5.3; pandas 3.0.5 is unaffected.
The existing guard matched the whole cell, so tokens with trailing characters reached pandas. Five CI crashes match this cause:
A dependency bound can't fix this: every pandas allowed by
pandas>=1.5,<3has the bug.Follow-up: other direct
pd.to_numeric(errors="coerce")call sites (domain validators, contracts, streaming, MissForest, semantic) don't go through this guard yet. Routing them through one shared helper will be a separate PR.Tests
fd.cleanon the crashing tokens, so a regression fails one test instead of killing the run. On Linux with pandas 2.3.3 it fails with returncode -11 against main'sdtypes.py, and passes with this change.pd.to_numeric, for both object and string dtype.Verification
test_dtypes+test_experimental_ai_copilot+test_enterprise_cli: 89 passed, 1 skippedpytest -m "not online and not large":ruff check .: cleanmypy src/freshdata: no issues