What is the problem the feature request solves?
The only performance numbers for native Iceberg writes are the ad hoc measurements in the #5361 description (2.9x unpartitioned, 5.5x partitioned fanout, from a harness in a personal fork). There is no benchmark in the repository, so a regression in the writer, in the shuffle feeding it, or in the JVM metrics rebuild would not be noticed, and the partitioned number was measured with write.distribution-mode=none rather than the default plan shape.
Describe the potential solution
Add a Spark benchmark (the org.apache.spark.sql.benchmark framework used by CometReadBenchmark and the shuffle benchmark) that measures INSERT INTO ... SELECT into an Iceberg table for:
Each case runs Spark, Comet scan only, and Comet scan plus native write, so the writer's contribution is isolated the way the #5361 table did. Wire it into the benchmark targets in the Makefile and document how to run it. Follow-on: run it on the EC2 benchmark runner and keep results with the other benchmark files.
Keep the Spark benchmark framework's warmup and minimum-time floors in mind when sizing the data (see the shuffle benchmark work in #5388 for how that dominated wall clock).
Additional context
Part of the native Iceberg writes epic, #5649. Related: #5361, #3595 (fused sort plus write, which this benchmark would measure).
What is the problem the feature request solves?
The only performance numbers for native Iceberg writes are the ad hoc measurements in the #5361 description (2.9x unpartitioned, 5.5x partitioned fanout, from a harness in a personal fork). There is no benchmark in the repository, so a regression in the writer, in the shuffle feeding it, or in the JVM metrics rebuild would not be noticed, and the partitioned number was measured with
write.distribution-mode=nonerather than the default plan shape.Describe the potential solution
Add a Spark benchmark (the
org.apache.spark.sql.benchmarkframework used byCometReadBenchmarkand the shuffle benchmark) that measuresINSERT INTO ... SELECTinto an Iceberg table for:write.spark.fanout.enabled=trueDELETE(rewrite of affected files)Each case runs Spark, Comet scan only, and Comet scan plus native write, so the writer's contribution is isolated the way the #5361 table did. Wire it into the benchmark targets in the
Makefileand document how to run it. Follow-on: run it on the EC2 benchmark runner and keep results with the other benchmark files.Keep the Spark benchmark framework's warmup and minimum-time floors in mind when sizing the data (see the shuffle benchmark work in #5388 for how that dominated wall clock).
Additional context
Part of the native Iceberg writes epic, #5649. Related: #5361, #3595 (fused sort plus write, which this benchmark would measure).