Skip to content

Complete V3 row ID assignment and inheritance - #83

Open
osipovartem wants to merge 3 commits into
embucket-sync-df55.0.0from
row-lineage-assignment-df55
Open

osipovartem wants to merge 3 commits into
embucket-sync-df55.0.0from
row-lineage-assignment-df55

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 18, 2026

Copy link
Copy Markdown

Summary

  • complete Iceberg v3 row-ID assignment and inheritance across V1, V2, and V3 manifest-list upgrades
  • decode historical manifest lists from their embedded writer schema
  • hydrate counters missing from V1 manifests before V2/V3 serialization
  • preserve partition summaries across schema/spec evolution, missing snapshot schema_id, and expired historical snapshots
  • restrict metadata promotion to Iceberg-legal integer, floating-point, and same-scale decimal widening
  • retain compatible lower/upper bounds while discarding incompatible optional metrics
  • route partition INT -> LONG and FLOAT -> DOUBLE evolution through the explicit Iceberg-promotion path while keeping generic casts strict

Performance

  • V1 counter hydration and manifest inspection use bounded concurrency of eight tasks
  • candidate partition decoders are cached by partition-spec and preferred-schema identity
  • unchanged values use equality/no-allocation fast paths; conversion occurs only for actual widening
  • no per-row evaluation or change to the vectorized Arrow/DataFusion scan path is introduced

Validation

  • manifest-list specification tests pass, including V1-to-V2 and V1-to-V3 upgrade coverage
  • all 172 active iceberg-rust-spec tests pass
  • all 10 transaction operation tests pass
  • focused bounds-promotion and historical-schema/spec regressions pass
  • cargo +1.94.0 clippy -p iceberg-rust -p iceberg-rust-spec --all-targets -- -D warnings
  • git diff --check
  • independent read-only review covering correctness, regressions, performance, API compatibility, and missing tests: APPROVED

The implementation preserves the existing DF55 manifest cache and vectorized scan path.

@osipovartem
osipovartem force-pushed the row-lineage-assignment-df55 branch from e8b0f97 to 0a42fe6 Compare September 18, 2026 12:22
@osipovartem
osipovartem force-pushed the row-lineage-assignment-df55 branch from 0a42fe6 to 3eb2e5e Compare September 23, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant