Skip to content

feat: accelerate Iceberg RewriteDataFiles reads via Comet native scan#4251

Open
jordepic wants to merge 1 commit intoapache:mainfrom
jordepic:main
Open

feat: accelerate Iceberg RewriteDataFiles reads via Comet native scan#4251
jordepic wants to merge 1 commit intoapache:mainfrom
jordepic:main

Conversation

@jordepic
Copy link
Copy Markdown

@jordepic jordepic commented May 6, 2026

Which issue does this PR close?

Closes #4250.

Rationale for this change

A large number of query resources are devoted across the industry to rewriting data files using spark procedures for iceberg tables. Using native code here where possible can significantly speed up this process!

What changes are included in this PR?

Detect spark scans (SparkStagedScan) that are created during the RewriteDataFilesSparkAction and replace them with comet scans. Extract their associated tasks and pass in the lack of filter (see SparkStagedScan line 50 in the apache iceberg project).

Note that some things are NOT included in this PR:
Datafusion-comet does not yet support writing iceberg data. Once it does, we can "comet-ify" this whole pipeline!

How are these changes tested?

We write two tests to inspect the spark plan associated with rewriting data files and ensure that the operators get replaced. Before this change is merged I can also try to run it locally and pick up some benchmarks for table compactions (on tables that are only data files, and those with delete files associated).

I have also added tests to ensure that the compaction works, not just ensuring operator replacement, but also runtime compaction correctness on both bin packing and sorting!

@jordepic jordepic force-pushed the main branch 3 times, most recently from 352861a to f1366a1 Compare May 7, 2026 02:18
@mbutrovich
Copy link
Copy Markdown
Contributor

Spark 3.4 failed for a reflection access. IIRC we use an older version of Iceberg there, so signatures might have changed. We might need different reflection logic for older versions of Iceberg. I had to do that somewhere else in the reflection class, but can't recall what right now.

@mbutrovich
Copy link
Copy Markdown
Contributor

Thanks @jordepic! I will take a proper pass through this today.

@mbutrovich mbutrovich self-requested a review May 7, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Iceberg "Rewrite Data Files Procedure"

2 participants