Skip to content

Map renamed Iceberg columns by field ID - #84

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
iceberg-field-id-read-df55
Sep 22, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
iceberg-field-id-read-df55

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • resolve renamed Iceberg columns by their stable Parquet/Iceberg field ID instead of the physical file name
  • preserve the logical field name, metadata, and nullability returned to DataFusion
  • apply the same field-ID lookup to renamed Variant columns
  • pre-index physical fields once per file-schema adapter

Performance

  • unchanged names retain the existing DefaultPhysicalExprAdapter fast path
  • same-type renames use a zero-copy expression that returns the original Arrow array
  • type casts are created only when the physical and logical data types differ
  • field-ID lookup is O(1) from a per-adapter map; there is no per-row conversion

Why

Iceberg column renames preserve field IDs and do not rewrite existing Parquet files. The previous adapter resolved ordinary columns by name, causing renamed non-null columns to fail as physically missing.

Validation

  • cargo +1.95.0 test -p datafusion_iceberg --lib --locked (83/83)
  • cargo +1.95.0 clippy -p datafusion_iceberg --lib --tests --locked -- -D warnings
  • the focused test asserts Arc::ptr_eq between the physical and renamed output arrays

@osipovartem
osipovartem force-pushed the iceberg-field-id-read-df55 branch 2 times, most recently from 6b0ab7a to 4d6900b Compare September 22, 2026 10:47
@osipovartem
osipovartem force-pushed the iceberg-field-id-read-df55 branch from 4d6900b to 51a3823 Compare September 22, 2026 10:47
@osipovartem
osipovartem merged commit 119d9b6 into embucket-sync-df55.0.0 Sep 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant