Skip to content

Expose recoverable CSV record diagnostics - #101

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
csv-record-error-handler
Sep 17, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
csv-record-error-handler

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

  • expose Arrow's opt-in CsvRecordErrorHandler through CsvFormat and CsvSource
  • preserve the callback across physical plan construction and projection pushdown
  • disable file range repartitioning only when recoverable diagnostics are enabled, keeping file-relative line and byte offsets correct
  • reject protobuf serialization instead of silently dropping the process-local callback
  • pin the DataFusion fork to the matching Embucket Arrow revision

Depends on Embucket/arrow-rs#3.

Performance

The handler is optional and defaults to None. Normal CSV scans retain existing repartitioning and Arrow's strict decoder fast path. Record retention and recovery work occur only for consumers that explicitly install a handler.

Tests

cargo +1.95.0 test -p datafusion-datasource-csv --all-features --offline
cargo +1.95.0 clippy -p datafusion-datasource-csv --all-targets --all-features --offline -- -D warnings

The integration test covers valid rows before/after extra- and missing-field records, exact callback offsets, and diagnostic-mode repartitioning behavior.

@osipovartem
osipovartem merged commit 7346b82 into embucket-sync-df55.0.0 Sep 17, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant