Skip to content

Bound CSV recovery offsets across chunks - #4

Merged
osipovartem merged 1 commit into
embucket-parquet-59.2.0-bounded-skipfrom
csv-invalid-record-handler-bounds
Sep 17, 2026
Merged

osipovartem merged 1 commit into
embucket-parquet-59.2.0-bounded-skipfrom
csv-invalid-record-handler-bounds

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • make the malformed-record excess-field limit stable for the lifetime of one CSV record
  • cap the offsets slice passed to csv-core even when normal batch preallocation is larger
  • use checked arithmetic for recovery offset preallocation
  • add a regression test that feeds one oversized malformed record across many 17-byte chunks

Why

The initial recovery implementation derived its limit from the current offsets buffer on every decode call. A single record split across many input chunks could therefore grow the limit repeatedly. This follow-up enforces the documented hard bound across chunk boundaries.

Tests

  • cargo +1.95.0 test -p arrow-csv --lib --offline (74 passed)
  • cargo +1.95.0 clippy -p arrow-csv --all-targets --offline --no-deps -- -D warnings

@osipovartem
osipovartem merged commit 763e450 into embucket-parquet-59.2.0-bounded-skip Sep 17, 2026
22 of 26 checks passed
@github-actions github-actions Bot added the arrow label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant