datafusion_iceberg: parquet row-filter pushdown for wide scans with a narrow predicate - #75
Merged
Merged
Conversation
…ly for wide scans (>= 8 projected columns) with a narrow predicate (<= 2 columns)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ParquetSourceis created withwith_pushdown_filters(true)only when the scan projects at least 8 columns and its pushed predicates reference at most 2 columns. Everything else keeps DataFusion's default (off).Why
Row-filter pushdown (predicate evaluated inside the parquet decoder, late materialisation of the remaining columns, TopK / join dynamic filters reaching the scan) pays off when the scan is wide and the predicate narrow — the ClickBench
SELECT * … WHERE url LIKE … ORDER BY t LIMIT nshape: on SPCSCPU_X64_M, ClickBench 100M Q24 went 38 → 5 s. Enabled globally it made TPC-H narrow scans 30 % slower (row filters lose to the vectorisedFilterExecon non-selective date/decimal predicates), so it is gated per scan.Requires the
parquet59.2.0 decoder fix inEmbucket/arrow-rs#embucket-parquet-59.2.0(DeltaBitPackskipfor Snowflake's mini-block sizes); rustice pins both together.Measured in the interactive-scenario autoresearch (rustice
benchmark-results/autoresearch-interactive-2026-09-11/REPORT.md).🤖 Generated with Claude Code
https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT