datafusion_iceberg: pure pushdown heuristic with unit tests - #76
Merged
Merged
Conversation
…ic and cover it Move the wide-scan / narrow-predicate decision out of `table_scan` into a pure module-level `use_parquet_row_filter_pushdown(projected_columns, filter_columns)` with named thresholds (`WIDE_SCAN_MIN_PROJECTED_COLUMNS = 8`, `NARROW_PREDICATE_MAX_COLUMNS = 2`) and the rationale in its doc comment. The call site now passes `requested_projection.len()` and the deduplicated filter-column count; behavior is unchanged. Add `pushdown_heuristic_tests` covering the boundaries (no filter columns is never pushed down; 8+1, 8+2, 100+2 on; 8+3, 7+1, 7+2 off) and a test that builds the filter-column set the way the call site does from `Expr` filters and asserts `column_refs` deduplicates repeated columns across predicates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extracts the parquet row-filter pushdown decision from
table_scan(#75) into a pure, documented helperuse_parquet_row_filter_pushdown(projected_columns, filter_columns)with named thresholds (WIDE_SCAN_MIN_PROJECTED_COLUMNS = 8,NARROW_PREDICATE_MAX_COLUMNS = 2); the wide-scan / narrow-predicate rationale moved into its doc comment. No behaviour change: the call site passes the same projection width and deduplicated filter-column count.Tests
pushdown_heuristic_tests: no filter columns is never pushed down; 8+1, 8+2, 100+2 enable; 8+3, 7+1, 7+2, 0+1 disable; and a test building the filter-column set exactly as the call site does fromExprfilters, assertingcolumn_refsdeduplicates repeated columns.cargo test -p datafusion_iceberg --lib: 66 passed;make fmt/make clippyclean.Pinned by Embucket/rustice.
🤖 Generated with Claude Code
https://claude.ai/code/session_012m5Yx6yEpZsacqAhZotgkT