Skip to content

[core] Build row range indexes directly from bitmap ranges - #9882

Draft
jerry-024 wants to merge 1 commit into
apache:masterfrom
jerry-024:optimize-row-range-from-bitmap
Draft

jerry-024 wants to merge 1 commit into
apache:masterfrom
jerry-024:optimize-row-range-from-bitmap

Conversation

@jerry-024

@jerry-024 jerry-024 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

DE scan planning converts global-index matches into row ranges, then calls RowRangeIndex.create, which copies, sorts and merges those ranges again. For non-negative row IDs, the bitmap conversion already produces ordered, disjoint ranges. Reprocessing them adds allocations and traversal overhead, especially when an index query returns many fragmented ranges.

Add RowRangeIndex.fromBitmap and use it in DataEvolutionBatchScan to build the range index directly from the converted ranges. Mixed-sign bitmaps retain the existing signed sort-and-merge path because bitmap iteration uses unsigned order. The general create(List<Range>) API keeps its existing behavior.

Tests

Added an equivalence test covering empty and singleton bitmaps, sparse and contiguous ranges, 32-bit boundaries, mixed signs and long-value boundaries. It checks range queries against the existing construction path and verifies that subsequent bitmap mutations do not change the constructed index.

Local validation with JDK 11:

  • Passed RowRangeIndexTest (2 tests) and RoaringNavigableMap64Test (8 tests).
  • Passed DataEvolutionBatchScanTest (6 tests) and DataEvolutionFileStoreScanTest#testIntersectsRowRanges (1 test).
  • Passed Spotless checks.
  • BtreeGlobalIndexTableTest#testBTreeGlobalIndexWithCoreScan failed during index creation with NoSuchMethodError: ManifestAvroReader.<init>(SeekableInputStream). This end-to-end check remains unverified.

@jerry-024
jerry-024 marked this pull request as draft September 16, 2026 09:01
@jerry-024 jerry-024 changed the title [core] Avoid redundant sorting when building row range indexes [core] Build row range indexes directly from bitmap ranges Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant