Skip to content

Read Iceberg v3 Puffin deletion vectors - #87

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
iceberg-v3-deletion-vector-read
Sep 22, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
iceberg-v3-deletion-vector-read

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 22, 2026

Copy link
Copy Markdown

Summary

  • decode Iceberg v3 deletion-vector-v1 blobs with framing, CRC, roaring-directory, and cardinality validation
  • add PUFFIN manifest file-format support
  • load only active-table DV blob ranges with bounded concurrent object-store requests
  • apply compact roaring vectors through a batch DataFusion predicate without materializing delete rows
  • preserve the existing v2 position-delete path and tables without deletes

Performance

  • one exact ranged GET per relevant DV blob; no whole-Puffin read
  • query-pruned data files do not trigger DV reads
  • one file-path lookup per record batch and one roaring membership check per row
  • no ScalarValue conversion or (path, position) row expansion

Validation

  • cargo +1.95.0 test -p iceberg-rust-spec deletion_vector --lib
  • cargo +1.95.0 test -p datafusion_iceberg deletion_vector_filter --lib
  • cargo +1.95.0 test -p datafusion_iceberg --test position_delete
  • cargo +1.95.0 clippy -p iceberg-rust-spec -p datafusion_iceberg --lib --tests -- -D warnings
  • live Snowflake/Horizon fixture: Snowflake created a managed Iceberg v3 MOR table, inserted 100,000 rows, and deleted IDs 2/50000/99999; Rustice returned the matching 99,997-row aggregate and affected rows
  • Rustice EXPLAIN showed iceberg_deletion_vector_keep(__data_file_path, __iceberg_file_row_position), confirming the Snowflake-generated Puffin DV path

The offline integration test creates an Iceberg v3 table, commits a Puffin deletion vector for row positions 1 and 3, and verifies SQL returns only rows 10, 30, and 50.

@osipovartem
osipovartem merged commit d9ec90d into embucket-sync-df55.0.0 Sep 22, 2026
1 of 2 checks passed
@osipovartem
osipovartem deleted the iceberg-v3-deletion-vector-read branch September 22, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant