Skip to content

Add recoverable CSV record error handler - #3

Merged
osipovartem merged 1 commit into
embucket-parquet-59.2.0-bounded-skipfrom
csv-invalid-record-handler
Sep 17, 2026
Merged

osipovartem merged 1 commit into
embucket-parquet-59.2.0-bounded-skipfrom
csv-invalid-record-handler

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 17, 2026

Copy link
Copy Markdown

Summary

  • add an opt-in CsvRecordErrorHandler to the Arrow CSV reader
  • report line, byte offset, expected/actual field counts, and original record bytes
  • skip malformed field-count records only when a handler is configured
  • preserve the existing allocation-free strict decoder path when no handler is configured
  • bound recovery-only field-offset growth

Why

DataFusion consumers implementing Snowflake-compatible COPY ... VALIDATION_MODE need to report malformed CSV rows and continue scanning later valid rows. Arrow currently aborts on the first field-count mismatch and discards the source record metadata.

Performance

The default path dispatches once per input chunk and then executes the existing strict decoder loop unchanged. Raw record retention and recovery allocations are enabled only when a handler is explicitly configured.

Tests

cargo +1.95.0 test -p arrow-csv --lib --offline

73 tests passed, including chunk-boundary recovery for both extra and missing fields.

@osipovartem

Copy link
Copy Markdown
Author

Performance evidence: Criterion on the existing 4-column, 4,096-row CSV reader benchmark measured 329.91 us before and 333.16 us after (+0.96%). Criterion classified the result as within the noise threshold. The no-handler path retains the original strict decoder loop; recovery allocations remain opt-in.

@osipovartem
osipovartem merged commit 4aee61a into embucket-parquet-59.2.0-bounded-skip Sep 17, 2026
24 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant