Repository navigation
Conversation
…oups. Separate GETs for each filter dominate cold reads from object storage. `input_format_parquet_bloom_filter_coalesce_max_bytes` merges contiguous filters, and reads a small file once when that is cheaper. Co-authored-by: Cursor <cursoragent@cursor.com>
… buffer. The bloom-filter header that starts the read is reset as soon as it is parsed, so charging that handle let the scheduler treat a live buffer as free. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>
That field is optional. Filters that only store an offset were left out of coalescing, even though their length is already derived later. Related: #2526 Co-authored-by: Cursor <cursoragent@cursor.com>
…or a whole-file read. Co-authored-by: Cursor <cursoragent@cursor.com>
…compatibility` values restore separate reads. Co-authored-by: Cursor <cursoragent@cursor.com>
…is below `bytes_per_read_task`. Co-authored-by: Cursor <cursoragent@cursor.com>
…and the filters after it stay inside `input_format_parquet_bloom_filter_coalesce_max_bytes`. Co-authored-by: Cursor <cursoragent@cursor.com>
….6 so older `compatibility` values restore separate reads, and count only the bloom ranges `Reader` registers. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 356269bf74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | `input_format_parquet_filter_push_down` | When reading Parquet files, skip whole row groups based on the WHERE/PREWHERE expressions and min/max statistics in the Parquet metadata. | `1` | | ||
| | `input_format_parquet_bloom_filter_push_down` | When reading Parquet files, skip whole row groups based on the WHERE expressions and bloom filter in the Parquet metadata. | `0` | | ||
| | `input_format_parquet_dictionary_filter_push_down` | When reading Parquet files (with reader v3), skip whole row groups based on the WHERE/PREWHERE expressions and the dictionary page contents, for equality and `IN` conditions, when all data pages of a column chunk are dictionary-encoded. The value is the maximum dictionary page size (in bytes) for which this optimization is applied; set to `0` to disable. Takes precedence over the bloom filter when both are available. | `1048576` | | ||
| | `input_format_parquet_bloom_filter_coalesce_max_bytes` | Max bytes of one read when loading Parquet bloom filters. Contiguous filters are read together. A file not larger than this, with many filters, is read once. 0 keeps one read per bloom filter. | `67108864` | |
There was a problem hiding this comment.
This line is inside the page's AUTOGENERATED_START/AUTOGENERATED_END region, so editing it in this feature commit will fail the documentation check unless the PR is the dedicated autogeneration change. Keep the corresponding source-registration update in ParquetV3BlockInputFormat.cpp and let the documentation workflow regenerate this file.
AGENTS.md reference: docs/reference/AGENTS.md:L1-L3
Useful? React with 👍 / 👎.
| const size_t prospective_start = std::min(start_offset, r.start); | ||
| const size_t prospective_end = std::max(end_offset, r.end); | ||
| const bool end_past_cap = same_group && prospective_end - prospective_start > span_limit; | ||
| if ((initial_req->coalesce_group != 0 && !same_group) || |
There was a problem hiding this comment.
When an ungrouped request starts first, this condition does not stop it from absorbing an adjacent grouped bloom-filter header because it checks only whether initial_req belongs to a group. This is reachable when a row group without an applicable bloom filter advances to index or data prefetching while another row group's bloom-header work remains queued; the bloom read then follows ordinary bytes_per_read_task rules and can exceed input_format_parquet_bloom_filter_coalesce_max_bytes (especially when configured below 4 MiB). Treat a group mismatch as a boundary whenever either request has a nonzero group, symmetrically in both scan directions.
Useful? React with 👍 / 👎.
This branch has not been deployed
1 participant
Separate GETs for each filter dominate cold reads from object storage.
input_format_parquet_bloom_filter_coalesce_max_bytesmerges contiguous filters, and reads a small file once when that is cheaper.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Read Parquet bloom filters in one request when a file has many row groups
Documentation entry for user-facing changes
ClickHouse can build Bloom-filter for Iceberg tables for queries with 'WHERE' condition not on key columns. For this ClickHouse reads metainfo for each row group from source files. In initial implementation it use single read operation for each row group, which is converted to separate GET request when data files are on object storage like S3. For large table it can produce thousands of requests. In customer test first request takes near 500 seconds, while second (when data are in local cache) less than 4 seconds.
This PR introduced reading metainfo for multiple row groups in single request, when possible. It can increase total traffic, but decrease number of requests, which can reduce time for query execution.
CI/CD Options
Exclude tests:
Regression jobs to run: