Preserve Parquet reader progress and add opt-in bounded range cache - #80
Merged
Vedin merged 2 commits intoSep 15, 2026
Merged
Conversation
Port the selected immutable-cache changes onto current row-lineage main. Keep range caching disabled by default and cold metadata readers independently pollable. Fresh DF55.1 integration library Clippy, format, and all 28 actual-module cache tests pass; full repository CI remains required on the PR.
Preserve all assertions and runtime code. Rust 1.95 library and actual-module test Clippy with -D warnings, format, and all 28 cache tests pass on the resolved DF55.1 integration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parent has paused it must not hold up another input reading the same file.
Contention now performs an independent read rather than waiting on that
reader's gate. Add a bypass metric and a deterministic paused-loader regression.
(
ICEBERG_PARQUET_DATA_CACHE_MB=0). Keys include store-qualified file identity,file size and exact range. Misses preserve batched I/O; retained ranges own
compact buffers, with entry/key overhead included in the bound.
and the intentional duplicate-cold-I/O tradeoff. Preserve current row-lineage
code and existing metadata cache defaults.
Verification
cargo fmt --all -- --check: PASS.-D warnings: PASS on DataFusion 55.1.0 withthe Rustice Arrow integration. Cargo metadata verifies a single 31-package
DF55.1 closure (no mixed DF55.0/55.1 packages).
wrapper compiles the two real source files, while the library Clippy run checks
their real reader-factory integration. The wrapper avoids unrelated bundled
DuckDB/Python test dependencies; this is not a claim that full repository CI
has run locally. The PR's full CI remains required.
-D warnings:PASS. The first CI run caught two
single_range_in_vec_initlints in testsetup; ranges are now constructed explicitly without changing assertions or
allowing the lint. Runtime code is unchanged by this follow-up.
and after the local gate (
iceberg55-r9, Rust 1.95.0).The selected implementation was exercised during the earlier E01–E21 campaign
on TPC-H SF100 and ClickBench100m, locally and on SPCS. Those are historical
end-to-end results, not an isolated cache speedup and not qualification of the
current-main integration. A 1 GiB range-cache trial regressed its screen, so the
range cache remains opt-in. Fresh integrated benchmarks are a separate Rustice
merge gate. No raw research logs or result workbooks are included in this PR.
Limits
The footer cache trades possible duplicate cold reads for progress; retained
metadata stays bounded, but in-flight/query-owned memory needs separate headroom.
The range cache does not merge overlapping ranges or deduplicate concurrent
misses. Same-URI/same-size mutable files are not supported by its identity model.
No query-result caching, new runtime dependencies, detached background work, or
test skips are introduced.