Skip to content

Add incremental changed data file scans - #78

Merged
osipovartem merged 2 commits into
embucket-sync-df55.0.0from
incremental-changelog-df55
Sep 15, 2026
Merged

osipovartem merged 2 commits into
embucket-sync-df55.0.0from
incremental-changelog-df55

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 15, 2026

Copy link
Copy Markdown

Summary

  • add an opt-in scan_changed_data_files mode to DataFusionTableConfig
  • select changed data-file events by snapshot provenance across an exclusive-start/inclusive-end range
  • normalize an EXISTING entry whose original add snapshot is inside the range to an ADDED event
  • retain DELETED entries based on their delete snapshot even when their data sequence predates the range
  • expose each event as the partition-constant __iceberg_change_file_status column
  • inherit missing manifest-entry snapshot IDs according to the Iceberg specification

Performance and correctness

  • normal current-state scans keep their existing path
  • range membership and status normalization run once per manifest entry, never per row
  • only manifests in the requested snapshot range and their relevant data files are read
  • change status is a per-file partition constant
  • equality and position delete files are not misreported as changed data rows

Validation

  • full datafusion_iceberg suite: 72 passed
  • strict clippy for iceberg-rust and datafusion_iceberg with -D warnings
  • Iceberg v3 integration test covers a range spanning two consecutive append snapshots and verifies all rows, inherited row IDs, update sequence numbers, and ADDED statuses
  • Rustice end-to-end CHANGES range test passes against this revision

Rustice uses this read-side primitive to classify Snowflake-compatible CHANGES(INFORMATION => DEFAULT) rows. Row-level classification remains outside this library PR.

@osipovartem
osipovartem changed the base branch from row-lineage-read-df55 to embucket-sync-df55.0.0 September 15, 2026 13:22
@osipovartem
osipovartem merged commit c3ddc7b into embucket-sync-df55.0.0 Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant