Repository navigation
fix(reader)!: ArrayStats mirrors Rust's array stats, including min/max precision - #581
Merged
Merged
Conversation
dfa1
force-pushed
the
array-stats-rust-parity
branch
from
October 20, 2026 06:44
d259169 to
4f6c578
Compare
ArrayStats.fromFbs dropped is_sorted, is_strict_sorted, is_constant, uncompressed_size_in_bytes and nan_count, and the record carried a trueCount the wire has no field for (Rust has no TrueCount stat: a bool column's true count is its sum). Rust writes is_sorted/is_strict_sorted into flat array stats, so Java read them as unknown. The record now has one component per field of the FlatBuffer ArrayStats table (Rust's Stat enum), in schema order; the 7-arg constructor is gone. min_precision/max_precision are still not read: an inexact bound is still a valid bound, so pruning stays correct. Cross-check: vortex-jni writes an ascending id column, and every flat data segment's peeked stats report isSorted/isStrictSorted true. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nan_count is u64 on the wire and surfaces as a signed Long, so `> 0` read a count >= 2^63 as "no NaN" and let a float chunk's max prune a filter that a NaN row matches. Rust sees a huge count and keeps the chunk; `!= 0` matches it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
array-stats-rust-parity
branch
from
October 20, 2026 06:51
4f6c578 to
63875b0
Compare
buildArrayFlatBuffer never set min_precision/max_precision, so they took the FlatBuffer default Inexact and Rust read every Java-written bound as inexact. The encoders' min/max are the exact extremes (Java never truncates flat stats), so write Precision::Exact as Rust's flat writer does for an untruncated stat. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ArrayStats ignored min_precision/max_precision, so a truncated bound (Rust's flat writer cuts a string min/max past 64 bytes, a legacy vortex.stats table flags it in *_is_truncated) read as the exact extreme. VortexReader.columnStats fell back to those flat stats for string columns, and Calcite answered SQL MIN/MAX from them. - ArrayStats gains minExact/maxExact. fromFbs reads precision as Rust's from_flatbuffer does: absent is Inexact, a value other than 0/1 is a corrupted field (VortexException), read only when its bound is present. - Zone tables: vortex.zoned MIN/MAX are exact, legacy MIN/MAX inexact where *_is_truncated is set, bounded_min/bounded_max never exact. - columnStats reports only exact min/max (null otherwise); Calcite's zone fold answers MIN/MAX only from exact stats. - An inexact bound still brackets the values, so range pruning keeps using it; only the min == max == v cases (Neq prune, Calcite Eq IN / Neq OUT) now require both bounds exact. vortex-jni writes no flat min/max on its default path (probed), so no Rust file produces an inexact flat bound to cross-check against; the cross-check is the Java-written >64-byte key column that vortex-jni filters, whose columnStats must be the full keys. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Oct 20, 2026
dfa1
added a commit
that referenced
this pull request
Oct 20, 2026
- CHANGELOG: #581 merged after the 0.18.0 section was cut but ships in it, so its three entries move from Unreleased into 0.18.0; the release is dated 2026-10-20; the jdbc:vortex: driver, the release's headline, leads Added. - README: a short section on querying .vortex files from IntelliJ or DBeaver with vortex-jdbc-driver, linking to the how-to. - how-to: the driver's query tips cover this release's dictionary filters and ORDER BY ... LIMIT. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits, one bug each (rebase-merge, don't squash):
ArrayStatsmirrors Rust's array stats table.fromFbsdroppedis_sorted,is_strict_sorted,is_constant,uncompressed_size_in_bytesandnan_count, and the record had atrueCountwith no wire field (Rust has noTrueCountstat; a bool column's true count is itssum). Rust writesis_sorted/is_strict_sortedinto flat array stats, so these now read back instead ofnull. Breaking:trueCount()and the 7-arg constructor are gone.nan_countsignedness. The count is u64, so pruning's> 0read a value ≥ 2^63 as "no NaN". It now uses!= 0, as Rust does. Only a corrupt file can trigger this, and there is no test for it.Exact. The precision was never written, so Rust read every Java-written bound asInexact(the FlatBuffer default).ArrayStatsgainsminExact/maxExact, read the way Rust'sfrom_flatbufferreads them: absent meansInexact, and any value other than 0/1 throwsVortexException. Zone tables:vortex.zonedMIN/MAX are exact, a legacy*_is_truncatedflag makes them inexact, andbounded_*values are never exact.columnStats()reports only exact min/max, and Calcite answersMIN/MAXonly from exact stats. Range pruning still uses inexact bounds, which is safe; only themin == max == vcases require both bounds exact.vortex-jni writes no flat min/max on its default path (checked by writing a file and reading its stats back), so no Rust-written file produces an inexact flat bound to cross-check against.
Tests:
VortexInspectorIntegrationTest#peek_readsSortednessRustRecordsInFlatArrayStats: Rust's sorted flags read back. It fails ifis_sortedis dropped.JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds: vortex-jni filters a Java-written column of keys longer than 64 bytes, and Java'scolumnStats()returns the full keys. It fails without the writer fix.Full
./mvnw verifypasses locally (9198 tests).🤖 Generated with Claude Code