Repository navigation
bench: compare Vortex with single-threaded Hardwood reads - #618
Merged
Merged
Conversation
Hardwood decodes pages on a pool sized to the machine, while the Vortex
reads scan on the calling thread, so parquetRead and
parquetReadMultiColumn measured several cores against one. On an
18-core M5 Max they kept about 2.9 and 4.4 cores busy and spent 8.5 ms
and 15.8 ms of CPU per scan, against Vortex's 3.8 ms and 6.2 ms.
parquetReadSingleThread and parquetReadMultiColumnSingleThread run the
same batch loops (moved into shared helpers) with
HardwoodContext.create(1). The default variants stay, as Hardwood's
out-of-the-box wall time.
3 forks x 5 x 5 s, ops/s:
default pool 1 thread Vortex
1 col trip_distance 349.9 +- 1.5 224.5 +- 14.0 288.6 +- 8.8 1.29x
2 cols fare+PULoc 319.9 +- 3.3 135.2 +- 3.5 191.9 +- 6.4 1.42x
All six read methods return the same sums as a null-aware row oracle.
docs/explanation.md now reports these numbers instead of "Vortex trails
Parquet batch, 0.71x", and DocsConsistencyTest allowlists HardwoodContext,
which it cites.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYuxQ3bvrntYDhPY1oD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ParquetVsVortexReadBenchmarkcompared several cores against one: Hardwood decodes pages on a pool sized to the machine, while the Vortex reads scan on the calling thread. On an 18-core M5 Max the Parquet batch reads kept about 2.9 and 4.4 cores busy, spending 8.5 ms and 15.8 ms of CPU per scan against Vortex's 3.8 ms and 6.2 ms.parquetReadSingleThreadandparquetReadMultiColumnSingleThreadrun the same batch loops (now shared helpers) withHardwoodContext.create(1). The default variants stay, as Hardwood's out-of-the-box wall time.3 forks × 5 × 5 s, ops/s:
trip_distancefare_amount,PULocationIDAll six read methods return the same sums as a null-aware row oracle (10,827,308.40 and 546,062,826.76), so every side does the full work.
docs/explanation.mdreports these numbers instead of "Vortex trails Parquet batch, 0.71×";DocsConsistencyTestallowlistsHardwoodContext, which the docs now cite. Benchmark and docs only, so no CHANGELOG entry.🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYuxQ3bvrntYDhPY1oD