Skip to content

Avoid position delete metadata column collisions - #89

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/position-delete-column-collisions-df55
Sep 23, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/position-delete-column-collisions-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • port the position-delete metadata-column collision fix to the DataFusion 55 branch
  • identify internal data-file path, row-position, and sequence-number columns by physical index instead of public names
  • generate collision-safe internal names when valid user columns use Iceberg metadata-like names
  • preserve the df55 Iceberg v3 deletion-vector and row-lineage paths
  • keep hidden partition columns and public manifest metadata correctly projected
  • preserve PartitionedFile value ordering when the internal data path and public manifest path are both present

This is the df55 counterpart of the upstream contribution in JanKaul#412, adapted to the fork's v3 row-lineage and deletion-vector implementation.

Performance

The added name selection and index remapping happen once during scan planning. V2 joins and v3 deletion-vector filters continue to use native DataFusion physical expressions; no row-wise conversion, interpreted evaluation, sorting, or materialization was added.

Validation

  • cargo +1.95.0 test -p datafusion_iceberg --test position_delete (v2 position deletes and v3 Puffin deletion vectors)
  • cargo +1.95.0 test -p datafusion_iceberg position_deletes_include_equal_data_sequence_numbers --lib
  • cargo +1.95.0 clippy -p datafusion_iceberg --all-targets -- -D warnings
  • cargo +1.95.0 fmt --all -- --check
  • git diff --check

The integration test covers a partitioned table with colliding user column names, reordered projections, public manifest metadata, v2 position deletes, and subsequent equality deletes.

Independent review

An independent read-only review initially found a manifest/data-file path ordering defect in the df55 adaptation. The ordering was fixed and the test was strengthened to assert the exact manifest path and deleted rows through the affected provider configuration. Final review verdict: APPROVED, with no blocking or non-blocking findings.

@osipovartem
osipovartem force-pushed the fix/position-delete-column-collisions-df55 branch from 38d2e9f to 1335780 Compare September 23, 2026 09:12
@osipovartem

Copy link
Copy Markdown
Author

Rebased onto embucket-sync-df55.0.0 after #83 merged. The rebase was clean and preserved the reviewed diff.

Post-rebase validation:

  • cargo +1.95.0 test -p datafusion_iceberg --test position_delete (2 passed: v2 position deletes and v3 Puffin deletion vectors)
  • cargo +1.95.0 test -p datafusion_iceberg position_deletes_include_equal_data_sequence_numbers --lib (1 passed)
  • cargo +1.95.0 clippy -p datafusion_iceberg --all-targets -- -D warnings
  • cargo +1.95.0 fmt --all -- --check
  • git diff --check

@osipovartem
osipovartem merged commit 22c5181 into embucket-sync-df55.0.0 Sep 23, 2026
2 checks passed
@osipovartem
osipovartem deleted the fix/position-delete-column-collisions-df55 branch September 23, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant