ORC: Fill initial defaults for absent scalar columns on read by cbb330 · Pull Request #252 · linkedin/iceberg

cbb330 · 2026-06-30T00:02:46Z

Summary

Fills a scalar column's initial-default when the field is absent from an ORC data file whose embedded Iceberg field IDs are trustworthy. Support is intentionally limited to the generic reader and the Spark 3.1 row reader. Scalar fields may be top-level or nested inside structs, map values, or list elements.

The table-level read-time overlay is the master gate. Tables that are not opted in never carry initialDefault in their in-memory schema and follow the unchanged read path. The default is never persisted to table metadata.

This is an adapted backport for Iceberg 1.2's positional ORC reader. It uses the default semantics from the newer implementation and the positional idToConstant pattern previously deployed in LinkedIn Iceberg #76. It does not backport the newer id-binding reader.

Review annotation legend

Inline annotations classify each change by provenance:

EXACT BACKPORT — mechanically equivalent code from a cited source, with only compatibility edits.
ADAPTED BACKPORT — behavior or test scenarios are ported, but implementation differs for this release line.
PRIOR ART — a proven design or invariant is reused in newly written code.
NET NEW — no hardened implementation was copied; review relies on local reasoning, containment, and tests.

These labels describe provenance, not correctness.

Activation invariant

A default is applied exactly when:

the field declares initialDefault;
the field is absent from the data file; and
the file carries trustworthy embedded Iceberg field IDs.

There is no ORC-level enable flag. OrcIterable derives file trust directly from ORCSchemaUtil.hasIds(fileSchema). Id-less/name-mapped files fail closed and read NULL.

How it works

buildOrcProjection(..., hasTrustedIds) omits a field satisfying the activation invariant.
The reader's record() step reads the value from initialDefault and adds it to the per-file idToConstant map.
The existing positional constant path fills the value without consuming an ORC column vector. The hot row-assembly loop is unchanged.

idToConstant is the transport required by the 1.2 positional reader; initialDefault remains the semantic source of the value.

Supported routing

Opted-in tables must use one of the wired readers:

Generic GenericOrcReader
Spark 3.1 row SparkOrcReader

Vectorized ORC, Flink, MR, and other Spark versions are not wired. The table-level overlay must not route opted-in tables to those paths until their reader support is added.

Deferred work and limitations

Predicate pushdown: filtering on an absent defaulted column is not supported and can fail or mis-prune. Avoid such filters until the follow-up is complete.
Vectorized ORC: deferred because it is disabled by default and rarely used in the fleet. Opted-in tables must remain on the row path.
Other engines and Spark versions: Flink and Spark 2.4/3.2/3.3 are not wired.
Parquet: this release line has no Parquet default-fill support.
Write defaults: writeDefault is not handled.
Non-Java engines: Trino, Presto, and Hive apply defaults independently and are unaffected.

Testing Done

TestBuildOrcProjection: trusted-identity omission, untrusted-identity null synthesis, required fields with defaults, nested omission, and empty nested structs.
TestOrcDefaultValues: generic end-to-end fill, default-only projection, representative scalar types, required fields, nested struct/map/list scenarios, empty and null parent structs, id-less fail-closed behavior, and preservation of present values and explicit NULLs.
TestSparkOrcReaderDefaults: declared default fill, absent field without a default remains NULL, and nested declared default fill.
Focused ORC and Spark 3.1 ORC reader tests passed locally.
ORC RevAPI, formatting, and lint checks passed locally.
CI remains responsible for full module suites.

Fill a field's initial-default as a per-file constant when its column is absent from an ORC data file, for the generic, Spark (2.4/3.1/3.2/3.3, row + vectorized) and Flink (1.14/1.15/1.16) readers. buildOrcProjection omits an absent top-level scalar field that declares an initial-default; each reader's record() step then injects the converted default into idToConstant so the existing partition-constant path fills it for every row (consuming no column vector). The omit is gated on applyDefaults, which OrcIterable sets to hasIds(fileSchema): id-less (name-mapped/migrated) files synthesize a null column instead and read NULL, so a default is never name-matched onto a file lacking embedded field ids. Scope is top-level scalar defaults; nested defaults are synthesized as null (read NULL) and remain a follow-up. Present-column binding is unchanged (ORC name-evolution + positional assembly), so existing reads do not regress. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Inline provenance annotations as requested, so the split is visible per-hunk.

Legend: 🟣 PRIOR ART (LI fork #76, same positional reader generation) · 🔵 BACKPORTED (decision/semantics carried from my apache/main ORC PR, re-implemented for the positional reader) · 🟢 NET NEW (did not exist on main or in #76).

Mental model: #76 = the how (omit + idToConstant inject on a positional reader); main = the what/why (top-level scalar scope, hasIds gate, NULL fallbacks); the shared idToConstantWithDefaults helper + all-engine wiring = the net-new glue.

cbb330

Stamping the remaining files so no file is left to interpretation. These are the per-version copies of the spots already annotated on the v3.3 / v1.16 representatives, plus the tests. Same legend: 🟣 PRIOR ART (#76) · 🔵 BACKPORTED (from apache/main) · 🟢 NET NEW.

Add an opt-in applyColumnDefaults flag on the ORC read builder (off by default). Only the row SparkOrcReader (Spark 3.1) and generic GenericOrcReader read paths enable it; OrcIterable then omits an absent top-level scalar default only when opted in (and the file has ids), so the reader can inject it via idToConstant. Every other ORC reader leaves the flag off, keeping the column synthesized as a null column (reads NULL) and therefore positionally aligned -- so it cannot misalign. Defers vectorized ORC and Flink defaults to a follow-up, and drops the Spark 2.4/3.2/3.3 wiring (those versions read NULL, unchanged). Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Re-stamped provenance for the now-scoped file set (generic + Spark 3.1 row + the opt-in gate). Supersedes my earlier two reviews, which were partly anchored to files that have since been reverted.

Legend: 🟣 PRIOR ART (LI fork #76, positional reader) · 🔵 BACKPORTED (decision/semantics from my apache/main ORC PR) · 🟢 NET NEW (did not exist on main or #76 — includes the opt-in gate introduced in this revision).

…th-default Add typed-default cases (boolean/int/long/float/double/string/decimal) and a required-field-with-default case (end-to-end + projection-level), confirming the fill reuses the partition-constant channel correctly across scalar types and that an absent required field with a default is omitted+filled rather than rejected by the required-missing check. Co-authored-by: Cursor <cursoragent@cursor.com>

Drop the top-level-only gate in buildOrcProjection so an absent scalar field declaring an initial-default is omitted (and filled via idToConstant) at any nesting level, not just the top level. The reader path already keys idToConstant by global field id and consults it at every struct level, so no reader change is needed. Add a safeguard so a non-root struct is never emitted empty (which ORC can reject): if every subfield of a nested struct is absent + defaulted, keep one synthesized null column (it reads NULL). The root struct may still be empty (select-only-default). Nested fill scenarios (struct/map/list) are ported from upstream BaseFormatModelTests. Co-authored-by: Cursor <cursoragent@cursor.com>

…ed structs) ORC accepts an empty nested read struct (as it already does for the empty root struct in select-only-default), so the "keep one synthesized null column" safeguard was unnecessary and introduced a surprising case where one subfield read NULL instead of its default. Remove it: an absent scalar default is now omitted and filled at any nesting level with no exceptions, including when every subfield of a nested struct is absent + defaulted (covered by a new end-to-end test). Also drops the now-unused topLevel parameter from buildOrcProjection. Co-authored-by: Cursor <cursoragent@cursor.com>

Add a row with a null nested struct to testNestedStructScalarDefault: a null parent struct stays null on read (the absent-only default is not fabricated), locking the positional reader's null-struct branch. Upstream's nested-default tests only cover this incidentally via random data, and via the id-binding reader rather than this branch. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Provenance re-stamp for the current state (now includes nested-scalar defaults + the opt-in gate). Supersedes earlier reviews.

Legend: 🟣 PRIOR ART (LI fork #76, positional reader) · 🔵 BACKPORTED (decision/semantics from my apache/main ORC PR) · 🟢 NET NEW (did not exist on main or #76 — chiefly the opt-in gate and the single shared helper).

Remove the reader-level enable flag so declared initial defaults are applied only when the field is absent and the file carries trustworthy IDs. Add regression coverage for id-less files, present values, and nested Spark rows. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Provenance annotations refreshed for the schema-driven implementation. Labels describe code origin, not correctness.

cbb330 · 2026-07-13T02:41:09Z

+
+  private static boolean isOmittableDefault(
+      Types.NestedField field, boolean hasTrustedIds, Map<Integer, OrcField> mapping) {
+    return field.initialDefault() != null && !mapping.containsKey(field.fieldId()) && hasTrustedIds;


[ADAPTED BACKPORT + PRIOR ART] This is the complete activation invariant: the field declares initialDefault, the field ID is absent from the file mapping, and the file carries trustworthy embedded IDs. The omission behavior follows the newer implementation; transporting the value through idToConstant follows #76 for this positional reader generation.

cbb330 · 2026-07-13T02:41:09Z

+   * @param convertConstant converts an internal default value to the engine's in-memory form
+   * @return {@code idToConstant} unchanged when no defaults apply, otherwise a new merged map
+   */
+  public static Map<Integer, ?> idToConstantWithDefaults(


[NET NEW + PRIOR ART] Shared engine-neutral adapter around the #76 constant mechanism. It reads the value from initialDefault, injects only fields omitted after trusted absence detection, preserves existing partition constants, and leaves engine-specific conversion to the caller.

cbb330 · 2026-07-13T02:41:09Z

+    if (hasTrustedIds) {
+      // Embedded IDs make field identity trustworthy, so absent fields that declare an initial
+      // default can be omitted from the projection and filled by a default-aware reader.
+      readOrcSchema = ORCSchemaUtil.buildOrcProjection(schema, fileSchema, hasTrustedIds);


[ADAPTED BACKPORT] File identity is derived from the existing embedded-ID check. Trusted files may omit absent defaulted fields; id-less/name-mapped files take the separate fail-closed path and synthesize NULL. There is no reader-level enable flag.

cbb330 · 2026-07-13T02:41:09Z

+      return GenericOrcReaders.struct(
+          fields,
+          expected,
+          ORCSchemaUtil.idToConstantWithDefaults(


[NET NEW BREADTH + PRIOR ART] #76 covered Spark only. This applies the same positional constant mechanism to the generic reader, using its existing partition-constant converter.

cbb330 · 2026-07-13T02:41:09Z

+      return SparkOrcValueReaders.struct(
+          fields,
+          expected,
+          ORCSchemaUtil.idToConstantWithDefaults(


[PRIOR ART: #76] Reuses the deployed positional invariant: an absent field becomes a per-file constant, and the unchanged StructReader consumes no ORC vector for that field. The source value is now initialDefault.

cbb330 · 2026-07-13T02:41:09Z

  }

-  protected static Object convertConstant(Type type, Object value) {
+  public static Object convertConstant(Type type, Object value) {


[EXACT BACKPORT: #76] Widens convertConstant for cross-package access from the Spark ORC visitor, matching the visibility change used by #76 on this reader generation.

cbb330 · 2026-07-13T02:41:09Z

          Sets.union(constantFieldIds, metadataFieldIds);
      Schema schemaWithoutConstantAndMetadataFields =
          TypeUtil.selectNot(expectedSchema, constantAndMetadataFieldIds);
+      // Follow-up: wire initial-default constants into VectorizedSparkOrcReaders. Tables whose


[DOCUMENTED LIMITATION] Vectorized ORC is intentionally deferred. The table-level overlay must route opted-in tables to the supported row reader until this path can materialize declared defaults.

cbb330 · 2026-07-13T02:41:09Z

  }
+
+  @Test
+  public void testTopLevelScalarDefaultOmittedWhenFileIdentityIsTrustworthy() {


[ADAPTED BACKPORT] Projection scenarios are adapted from the newer behavior for this positional reader: trusted absence omits, untrusted identity synthesizes NULL, and nested/required defaults follow the same invariant.

cbb330 · 2026-07-13T02:41:09Z

+  }
+
+  @Test
+  public void testReadDoesNotApplyDefaultToIdLessFile() throws IOException {


[NET NEW SAFETY COVERAGE] End-to-end proof that an id-less file remains NULL even when the in-memory field declares a default. The adjacent test also proves present values and explicit NULLs are never replaced.

cbb330 · 2026-07-13T02:41:09Z

+  }
+
+  @Test
+  public void testRowReadFillsDeclaredDefault() throws IOException {


[ADAPTED BACKPORT + NET NEW COVERAGE] Adapts the Spark row-fill scenario and proves the field declaration is the trigger: declared absent defaults fill, absent fields without a default remain NULL, and nested declared defaults fill through the same path.

Keep the pre-existing two-argument projection call for files with embedded IDs, introducing the explicit trust argument only for the id-less name-mapped branch. Co-authored-by: Cursor <cursoragent@cursor.com>

Preserve the existing two-argument projection behavior for all callers and activate default omission only in the embedded-ID branch. This leaves the id-less name-mapped path unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove unrelated filter commentary so OrcIterable changes only at the trusted-ID projection call. Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions Bot added SPARK ORC FLINK labels Jun 30, 2026

cbb330 commented Jun 30, 2026

View reviewed changes

github-actions Bot added DATA and removed FLINK labels Jun 30, 2026

cbb330 commented Jun 30, 2026

View reviewed changes

cbb330 and others added 4 commits June 30, 2026 15:33

cbb330 commented Jun 30, 2026

View reviewed changes

cbb330 changed the title ~~ORC: Apply top-level scalar initial-default on read (absent-only fill)~~ ORC: Fill initial defaults for absent scalar columns on read Jul 13, 2026

github-actions Bot removed the DATA label Jul 13, 2026

cbb330 commented Jul 13, 2026

View reviewed changes

cbb330 and others added 3 commits July 13, 2026 16:36

ORC defaults: preserve existing trusted-ID projection path

2f13368

Keep the pre-existing two-argument projection call for files with embedded IDs, introducing the explicit trust argument only for the id-less name-mapped branch. Co-authored-by: Cursor <cursoragent@cursor.com>

ORC defaults: keep iterable diff minimal

d3474e3

Remove unrelated filter commentary so OrcIterable changes only at the trusted-ID projection call. Co-authored-by: Cursor <cursoragent@cursor.com>

Uh oh!

Conversation

cbb330 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review annotation legend

Activation invariant

How it works

Supported routing

Deferred work and limitations

Testing Done

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cbb330 commented Jun 30, 2026 •

edited

Loading