Repository navigation
perf(reader): select a filtered chunk's rows through a Rust-style Mask (#602) - #607
Merged
Merged
Conversation
#602) The filtered scan's selection is now a Mask, as Rust's vortex_mask::Mask: AllTrue, AllFalse, or Values holding the bitmap, whose sorted indices and runs of consecutive rows are built on first use and cached, so every column of the chunk reuses them (Rust's OnceLock fields). The copy picks as Rust's filter does at vortex-jni 0.89.0: runs in bulk when the mask is dense and its runs average at least 8 rows (MIN_SLICES_AVERAGE_RUN_LENGTH), a typed per-row gather otherwise. - RowSelection.toMask: one walk sets a bit per selected row; a bitmap selection with no residual test hands its words over with no walk. - Mask.limit truncates the bitmap, as Mask::limit does. - An all-true mask returns the column itself instead of copying it. - A filtered chunk past Integer.MAX_VALUE rows, or a fixed-size list whose element count would pass it, fails as VortexException; a fixed-size list of zero elements selects no elements. - Public surface: Mask, its three variants, fromBitmap, length, trueCount and limit; the index and run forms stay package-private. JavaVsJniFilterBenchmark.javaFilterClose, ohlc.vortex, -f 3, ops/s (the 0.001 label selects about 11% of rows on this file): selectivity int[] (main) Mask alloc/op int[] -> Mask 0.001 6729 ± 303 7130 ± 607 196 KB -> 81 KB 0.01 6806 ± 113 6843 ± 560 196 KB -> 81 KB 0.1 5326 ± 44 5860 ± 50 363 KB -> 118 KB 1.0 2458 ± 103 2347 ± 135 1247 KB -> 718 KB Long.compress/expand were considered for the bit walks and rejected: on this Apple M5 C2 inlines them as Java (no BEXT/BDEP without SVE2), while Long.bitCount and numberOfTrailingZeros, used here, are intrinsics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #606 (#602): the filtered scan's selection becomes a Rust-style
Mask(bitmap with cached indices and runs), and the copy picks runs or gather as Rust's filter does at vortex-jni 0.89.0. Benchmark table in the commit message: ties or beats theint[]version, with 1.7–3× less allocation../mvnw verifygreen (3065 integration tests).🤖 Generated with Claude Code
https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x