feat(avro): apply column default values when reading missing fields

[huan233usc]

What

Part 3 of 4 of Iceberg v3 column default-value support (POC #731), built on the
schema layer (#746) and the Parquet read path (#792).

When a column is present in the read (table) schema but absent from an Avro data
file — because the column was added after those rows were written — fill it with
the column's v3 initial-default instead of null.

Changes

• Avro projection (avro_schema_util.cc): when a field is missing from the
  file and carries an initial-default, project it as
  FieldProjection::Kind::kDefault, mirroring the generic / Parquet paths.
• Avro decode (avro_data_util.cc, avro_direct_decoder.cc): materialize
  the kDefault branch through an Avro-local AppendDefaultToBuilder helper.
  It reuses the shared ToArrowScalar conversion, while keeping Avro's
  row-by-row ArrayBuilder append behavior out of the shared Arrow utility.

Tests

• avro_data_test: AppendDefaultToBuilder appends a value and casts to the
  builder type; AppendDatumToBuilder fills missing required and optional
  default fields.
• avro_test: end-to-end — write an Avro file with an old schema, then read it
  through ReaderFactoryRegistry with an evolved schema carrying defaults
  (ReadMissingFieldsWithDefaults).

Stack

1. feat(schema): represent, serialize and validate v3 column default values (1/4) #746 — schema: represent / serialize / validate (merged)
2. feat(parquet): apply column default values when reading missing fields (2/4) #792 — read path: Parquet (merged)
3. this PR — read path: Avro
4. schema evolution: addColumn / updateColumnDefault (feat: support v3 column default values in UpdateSchema (3/4) #793)

Draft while the earlier PRs in the stack settle.

[Copilot]

Pull request overview

Adds Avro read-path support for Iceberg v3 initial-default column defaults, ensuring that when projected (table) schema fields are missing from older Avro files, readers materialize the specified defaults instead of null.

Changes:

• Avro schema projection: project missing fields with initial-default as FieldProjection::Kind::kDefault.
• Avro decode paths (generic + direct): materialize kDefault by appending the literal default into Arrow builders via a new Avro-local AppendDefaultToBuilder.
• Tests: add unit coverage for default appends and an end-to-end Avro read test exercising schema evolution with defaults.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/iceberg/avro/avro_schema_util.cc	Projects missing fields with initial-default as kDefault.
src/iceberg/avro/avro_data_util.cc	Implements and uses AppendDefaultToBuilder to materialize kDefault during generic Avro decoding.
src/iceberg/avro/avro_direct_decoder.cc	Materializes kDefault during direct decoding into Arrow builders.
src/iceberg/avro/avro_data_util_internal.h	Exposes AppendDefaultToBuilder for reuse across Avro decode implementations.
src/iceberg/test/avro_data_test.cc	Adds unit tests covering default materialization behavior.
src/iceberg/test/avro_test.cc	Adds end-to-end test for reading missing fields with defaults using an evolved schema.

[wgtmac]

Could this follow the ParquetExtraAttributes pattern and become AvroExtraAttributes with a default_scalar member? FieldProjection has a single attributes slot, so separate subclasses for future Avro attributes would not compose. A single Avro-specific container would also let the append path use checked_cast instead of a per-row dynamic_cast.

[huan233usc]

Done — renamed to AvroExtraAttributes with a default_scalar member, mirroring ParquetExtraAttributes. Since Avro now has a single attributes container, the append path uses checked_cast instead of a per-row dynamic_cast, and future Avro attributes can be added to the same struct rather than as competing subclasses.

[wgtmac]

attributes != nullptr does not guarantee that default_scalar is prepared now that this container may hold other Avro attributes. If another attribute creates the container first, this skips preparation and the append path falls back to rebuilding the scalar per row. Please reuse/cast the existing AvroExtraAttributes, check default_scalar instead, and add a test with a pre-existing empty container.