Describe the bug
The native Iceberg write eligibility path supports gs:// locations and NativeConfig.extractObjectStoreOptions extracts the corresponding fs.gs.* settings from the Hadoop configuration.
However, CometIcebergNativeWrite then passes those extracted settings through:
CometIcebergNativeScan.hadoopToIcebergS3Properties(...)
which translates S3/S3A configuration only.
For a gs:// location, the extracted fs.gs.* settings are therefore not preserved when building the native GCS FileIO.
This is particularly problematic for tables using HadoopFileIO, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the Hadoop Configuration rather than FileIO.properties().
The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration.
Depending on the environment, the native writer could:
- fail even though the JVM writer succeeds;
- use a different GCS endpoint; or
- resolve a different ambient/default identity than the one configured for the Hadoop filesystem.
Native writes should fail closed rather than silently changing storage identity or endpoint configuration.
Steps to reproduce
No response
Expected behavior
The native writer must either:
- faithfully translate the relevant Hadoop GCS configuration into the native GCS backend configuration; or
- fail closed and fall back to the JVM Iceberg writer for
HadoopFileIO + gs://.
Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination.
A table using an actual GCSFileIO whose properties already provide supported native gcs.* settings can remain eligible.
Additional context
Found during follow-up review of #5361.
A simple fs.gs.* → gcs.* key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation.
The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer.
Describe the bug
The native Iceberg write eligibility path supports
gs://locations andNativeConfig.extractObjectStoreOptionsextracts the correspondingfs.gs.*settings from the Hadoop configuration.However,
CometIcebergNativeWritethen passes those extracted settings through:CometIcebergNativeScan.hadoopToIcebergS3Properties(...)which translates S3/S3A configuration only.
For a
gs://location, the extractedfs.gs.*settings are therefore not preserved when building the native GCS FileIO.This is particularly problematic for tables using
HadoopFileIO, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the HadoopConfigurationrather thanFileIO.properties().The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration.
Depending on the environment, the native writer could:
Native writes should fail closed rather than silently changing storage identity or endpoint configuration.
Steps to reproduce
No response
Expected behavior
The native writer must either:
HadoopFileIO + gs://.Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination.
A table using an actual
GCSFileIOwhose properties already provide supported nativegcs.*settings can remain eligible.Additional context
Found during follow-up review of #5361.
A simple
fs.gs.*→gcs.*key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation.The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer.