Repository navigation
perf(reader): gather dict and window folds straight off their segments - #600
Merged
Merged
Conversation
The two-column taxi read (ParquetVsVortexReadBenchmark, fare_amount + PULocationID) spent its time in per-row virtual calls: - a vortex.dict whose pool is ALP-encoded re-ran the ALP multiply for every row it gathered; a floating pool is now decoded once, as Rust's dict canonicalizes its values before taking. Integer pools stay lazy: FoR costs an add a row, and decoding them shifted the gathers' type profile enough to cost the TPC-DS store_sales scan 3.5%. - DictDoubleArray.fold and DictIntArray.materialize read a flat pool and flat u8/u16 codes off their segments instead of through getShort and getDouble/getInt. - OffsetIntArray and OffsetDoubleArray, the scan windows over a shared flat, fold the window off its segment instead of inheriting the per-row default. ParquetVsVortexReadBenchmark.vortexReadMultiColumn, M5, Zulu 25.0.2, 3 forks: 156.7 -> 162.0 ops/s (+3.4%). RealFileScanBenchmark on all seven real files, base and change interleaved, 2 forks: every file within noise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x
dfa1
force-pushed
the
perf/dict-flat-gather
branch
from
October 24, 2026 15:41
daf87e6 to
27aa66d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiling
ParquetVsVortexReadBenchmark.vortexReadMultiColumnshowed per-row virtual calls dominating: an ALP-encoded dict pool re-decoded on every gathered row, and the dict and scan-window folds went throughgetShort/getDouble/getIntper row.store_sales3.5%.DictDoubleArray.fold,DictIntArray.materialize,OffsetIntArray.fold,OffsetDoubleArray.foldread flat segments directly.Two-column taxi read, 3 forks: 156.7 → 162.0 ops/s (+3.4%), still behind Parquet batch (221).
RealFileScanBenchmarkA/B on all seven real files, interleaved, 2 forks: all within noise (bi-corporationsandstore_salesre-checked at 3 forks).New unit tests compare each fast path against the per-row path with an order-sensitive fold operator. Full
./mvnw verifypasses locally.🤖 Generated with Claude Code
https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x