Skip to content

Write and preserve Iceberg v3 row lineage - #88

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
iceberg-v3-lineage-write
Sep 22, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
iceberg-v3-lineage-write

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 22, 2026

Copy link
Copy Markdown

Summary

  • add shared Iceberg v3 row-lineage field definitions and an opt-in Parquet writer
  • preserve inherited first_row_id values when overwrite transactions rewrite existing manifests
  • reserve row-ID ranges for new files and advance next_row_id atomically with the snapshot commit
  • keep the existing Iceberg v2 writer and manifest paths unchanged

Correctness

The writer emits _row_id and _last_updated_sequence_number using the Iceberg reserved field IDs. During copy-on-write, surviving rows retain their physical row IDs, updated rows can inherit the new sequence number, and inserted files receive non-overlapping row-ID ranges.

Validation

  • cargo +1.95.0 test -p iceberg-rust v3_ --lib
  • cargo +1.95.0 clippy -p iceberg-rust -p datafusion_iceberg --all-targets -- -D warnings
  • cargo +1.95.0 fmt --check

This is the storage-layer prerequisite for Rustice Iceberg v3 copy-on-write UPDATE, DELETE, and MERGE.

@osipovartem
osipovartem merged commit 8e521c0 into embucket-sync-df55.0.0 Sep 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant