Skip to content

Preserve changelog events across rewrite snapshots - #79

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
incremental-changelog-multi-rewrite-df55
Sep 15, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
incremental-changelog-multi-rewrite-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • walk every Iceberg snapshot in a changed-data scan range instead of relying on the final snapshot manifest list
  • scan only manifests written for each snapshot sequence and expose the event snapshot sequence as a partition metadata column
  • keep the ordinary table-scan path unchanged and allocate changelog ordering state only when scan_changed_data_files is enabled

Why

A final snapshot can omit file events from intermediate copy-on-write rewrites. Consumers need those events and their commit order to reduce a multi-snapshot changelog to Snowflake CHANGES(INFORMATION => DEFAULT) semantics.

Verification

  • cargo +1.94.0 test -p datafusion_iceberg --lib test_changed_file_scan -- --nocapture
  • cargo +1.94.0 test -p datafusion_iceberg --lib test_v3_row_lineage_columns_are_synthesized_by_scan -- --nocapture
  • cargo +1.94.0 check -p datafusion_iceberg
  • cargo +1.94.0 clippy -p iceberg-rust -p datafusion_iceberg --lib -- -D warnings

@osipovartem
osipovartem merged commit 1fffa3b into embucket-sync-df55.0.0 Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant