Skip to content

Add vectorized Iceberg v3 row lineage reads - #77

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
row-lineage-read-df55
Sep 15, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
row-lineage-read-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • inherit Iceberg v3 first_row_id values from data-manifest metadata before snapshot filtering
  • expose opt-in _row_id and _last_updated_sequence_number columns from DataFusionTable
  • resolve physically stored lineage fields by reserved Iceberg field ID and coalesce them with manifest-derived values
  • keep the normal scan path unchanged when lineage columns are disabled

Performance

  • lineage synthesis uses vectorized Arrow kernels (add, validity masks, and zip)
  • no per-row ScalarValue conversion or expression evaluation
  • Parquet reads include lineage dependencies only when the caller explicitly enables the columns
  • filters on synthesized lineage columns stay above the scan; unrelated filters retain Parquet pushdown

Validation

  • cargo +1.95.0 test -p iceberg-rust --lib --offline table::tests:: -- --nocapture
  • cargo +1.95.0 test -p datafusion_iceberg --lib --offline -- --nocapture (71 passed)
  • cargo +1.95.0 clippy -p iceberg-rust-spec -p iceberg-rust -p datafusion_iceberg --all-targets --offline -- -D warnings

The integration test creates an Iceberg v3 table, writes rows through DataFusion, reads both lineage columns through a projected Parquet scan, and verifies a filter on synthesized _row_id.

This is the read-side foundation for a correct row-level changelog implementation. It does not claim to implement a full changelog scan or row-lineage-preserving copy-on-write rewrites.

@osipovartem
osipovartem merged commit 6a0dce9 into embucket-sync-df55.0.0 Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant