fix(object_store): make get_ranges coalescing configurable - #8062
Open
Xuanwo wants to merge 1 commit into
Open
Conversation
Member
Author
|
@comphead, could you please retest this PR with the same HDFS/DataFusion Comet workload from #7380 and #7383? Please compare at least:
The most useful results would be task wall-clock time and cluster memory usage. I am keeping the PR in draft until we understand whether the configurable gap avoids the earlier memory regression while improving the HDFS read path. |
Xuanwo
marked this pull request as ready for review
August 13, 2026 05:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Follow-up to #7380 and #7383.
Rationale for this change
object_store_opendal::get_rangescurrently executes onereader.read(range)for every requested range. The reads are buffered concurrently, but nearby ranges still cause separate backend reads.Reader::fetchalready provides range coalescing and concurrent execution. However, the earlier HDFS investigation in #7380 found that unconditional 1 MiB coalescing could approximately double cluster memory usage because small returnedBytesslices retained large coalesced backing buffers. The integration therefore needs a configurable coalescing threshold instead of choosing between always coalescing and never coalescing.What changes are included in this PR?
get_rangesthroughReader::fetchwith the existing concurrency of 8.object_store::OBJECT_STORE_COALESCE_DEFAULT(1 MiB).OpendalStore::with_get_ranges_gapso users can tune the threshold or set it to0.gap = 0a valid OpenDAL reader setting. It disables merging ranges separated by bytes while still merging overlapping or adjacent ranges.Are there any user-facing changes?
Yes.
get_rangesnow coalesces nearby ranges by default. HDFS users whose workloads are sensitive to retained coalesced buffers can construct the store with.with_get_ranges_gap(0)or choose an intermediate threshold.Validation
cargo test --manifest-path integrations/object_store/Cargo.tomlcargo clippy --manifest-path integrations/object_store/Cargo.toml --all-targets --all-features -- -D warningscargo test -p opendal-core -p opendal-layer-timeout test_merge_rangescargo clippy -p opendal-core -p opendal-layer-timeout --all-targets -- -D warnings./scripts/workspace.py cargo fmt -- --checkA real HDFS/DataFusion Comet cluster retest is pending.
AI Usage Statement
OpenAI Codex (GPT-5) assisted with implementation, tests, and PR preparation.