Skip to content

Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643

Description

@andygrove

What is the problem the feature request solves?

The native Iceberg write eligibility gate (CometIcebergNativeWrite, documented in iceberg-writes.md) is an allowlist, and each restriction it enforces was added conservatively in #5298 so the writer could land incrementally. None of the restrictions has an issue saying whether it is permanent or what would lift it, so there is no way to tell how much of the fallback surface is intended.

Current restrictions, each of which makes a write fall back to iceberg-java:

  • format version 3 or newer (MinUnsupportedFormatVersion)
  • any encryption.* property
  • write.parquet.bloom-filter-max-bytes and per-column bloom filter enablement
  • write.parquet.page-version=v2
  • write.parquet.shred-variants=true (also depends on [Variant] Support whole-value VariantType in native Parquet writes #5433 / the Variant epic [EPIC] Native Variant Support #5438)
  • any unvetted write.parquet.* key, and any parquet.* table property (including parquet.enable.dictionary)
  • any parquet.* key in the session Hadoop configuration (other than parquet.hadoop.vectored.io.enabled)
  • non-default write.parquet.row-group-check-min-record-count / max-record-count
  • write.object-storage.enabled=true and write.location-provider.impl
  • io-impl set, or a FileIO outside the recognized class hierarchy
  • data location schemes other than file, memory, s3, s3a, gs (oss falls back because oss.* properties are not forwarded; see Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637 for gs)

Describe the potential solution

For each restriction, record one of three decisions in this issue and in iceberg-writes.md:

  • Lift: the native path already reproduces iceberg-java's outcome, or can with a bounded change; file a follow-up and link it.
  • Keep, pending upstream: needs an iceberg-rust or parquet-rs feature; link the upstream issue.
  • Keep, permanent: intentionally out of scope (for example encryption); say why.

The ordering should follow how often the setting appears in real tables. Object-storage layout (write.object-storage.enabled) and bloom filters are the two most likely to be hit by production tables and should be decided first.

Additional context

Part of the native Iceberg writes epic, #5649. Related: #5298 (introduced the allowlist), #5361 (native writer).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions