Skip to content

fix(reader)!: ArrayStats mirrors Rust's array stats, including min/max precision - #581

Merged
dfa1 merged 4 commits into
mainfrom
array-stats-rust-parity
Oct 20, 2026
Merged

dfa1 merged 4 commits into
mainfrom
array-stats-rust-parity

Conversation

@dfa1

@dfa1 dfa1 commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Four commits, one bug each (rebase-merge, don't squash):

  1. ArrayStats mirrors Rust's array stats table. fromFbs dropped is_sorted, is_strict_sorted, is_constant, uncompressed_size_in_bytes and nan_count, and the record had a trueCount with no wire field (Rust has no TrueCount stat; a bool column's true count is its sum). Rust writes is_sorted/is_strict_sorted into flat array stats, so these now read back instead of null. Breaking: trueCount() and the 7-arg constructor are gone.
  2. nan_count signedness. The count is u64, so pruning's > 0 read a value ≥ 2^63 as "no NaN". It now uses != 0, as Rust does. Only a corrupt file can trigger this, and there is no test for it.
  3. Writer: mark flat min/max Exact. The precision was never written, so Rust read every Java-written bound as Inexact (the FlatBuffer default).
  4. Reader: honor min/max precision. ArrayStats gains minExact/maxExact, read the way Rust's from_flatbuffer reads them: absent means Inexact, and any value other than 0/1 throws VortexException. Zone tables: vortex.zoned MIN/MAX are exact, a legacy *_is_truncated flag makes them inexact, and bounded_* values are never exact. columnStats() reports only exact min/max, and Calcite answers MIN/MAX only from exact stats. Range pruning still uses inexact bounds, which is safe; only the min == max == v cases require both bounds exact.

vortex-jni writes no flat min/max on its default path (checked by writing a file and reading its stats back), so no Rust-written file produces an inexact flat bound to cross-check against.

Tests:

  • Unit tests for every stat field (false/0 values, which equal the FlatBuffer defaults) and for exact, absent and corrupt precision.
  • vortex-jni cross-check VortexInspectorIntegrationTest#peek_readsSortednessRustRecordsInFlatArrayStats: Rust's sorted flags read back. It fails if is_sorted is dropped.
  • vortex-jni cross-check JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds: vortex-jni filters a Java-written column of keys longer than 64 bytes, and Java's columnStats() returns the full keys. It fails without the writer fix.

Full ./mvnw verify passes locally (9198 tests).

🤖 Generated with Claude Code

@dfa1
dfa1 force-pushed the array-stats-rust-parity branch from d259169 to 4f6c578 Compare October 20, 2026 06:44
dfa1 and others added 2 commits October 20, 2026 08:49
ArrayStats.fromFbs dropped is_sorted, is_strict_sorted, is_constant,
uncompressed_size_in_bytes and nan_count, and the record carried a
trueCount the wire has no field for (Rust has no TrueCount stat: a bool
column's true count is its sum). Rust writes is_sorted/is_strict_sorted
into flat array stats, so Java read them as unknown.

The record now has one component per field of the FlatBuffer ArrayStats
table (Rust's Stat enum), in schema order; the 7-arg constructor is gone.
min_precision/max_precision are still not read: an inexact bound is still
a valid bound, so pruning stays correct.

Cross-check: vortex-jni writes an ascending id column, and every flat data
segment's peeked stats report isSorted/isStrictSorted true.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nan_count is u64 on the wire and surfaces as a signed Long, so `> 0` read
a count >= 2^63 as "no NaN" and let a float chunk's max prune a filter
that a NaN row matches. Rust sees a huge count and keeps the chunk;
`!= 0` matches it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1
dfa1 force-pushed the array-stats-rust-parity branch from 4f6c578 to 63875b0 Compare October 20, 2026 06:51
dfa1 and others added 2 commits October 20, 2026 09:03
buildArrayFlatBuffer never set min_precision/max_precision, so they took
the FlatBuffer default Inexact and Rust read every Java-written bound as
inexact. The encoders' min/max are the exact extremes (Java never
truncates flat stats), so write Precision::Exact as Rust's flat writer
does for an untruncated stat.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ArrayStats ignored min_precision/max_precision, so a truncated bound
(Rust's flat writer cuts a string min/max past 64 bytes, a legacy
vortex.stats table flags it in *_is_truncated) read as the exact extreme.
VortexReader.columnStats fell back to those flat stats for string columns,
and Calcite answered SQL MIN/MAX from them.

- ArrayStats gains minExact/maxExact. fromFbs reads precision as Rust's
  from_flatbuffer does: absent is Inexact, a value other than 0/1 is a
  corrupted field (VortexException), read only when its bound is present.
- Zone tables: vortex.zoned MIN/MAX are exact, legacy MIN/MAX inexact
  where *_is_truncated is set, bounded_min/bounded_max never exact.
- columnStats reports only exact min/max (null otherwise); Calcite's
  zone fold answers MIN/MAX only from exact stats.
- An inexact bound still brackets the values, so range pruning keeps
  using it; only the min == max == v cases (Neq prune, Calcite Eq IN /
  Neq OUT) now require both bounds exact.

vortex-jni writes no flat min/max on its default path (probed), so no
Rust file produces an inexact flat bound to cross-check against; the
cross-check is the Java-written >64-byte key column that vortex-jni
filters, whose columnStats must be the full keys.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dfa1 dfa1 changed the title fix(reader)!: ArrayStats mirrors Rust's array stats table fix(reader)!: ArrayStats mirrors Rust's array stats, including min/max precision Oct 20, 2026
@dfa1
dfa1 merged commit 39ad904 into main Oct 20, 2026
9 checks passed
@dfa1
dfa1 deleted the array-stats-rust-parity branch October 20, 2026 07:56
dfa1 added a commit that referenced this pull request Oct 20, 2026
- CHANGELOG: #581 merged after the 0.18.0 section was cut but ships in it,
  so its three entries move from Unreleased into 0.18.0; the release is
  dated 2026-10-20; the jdbc:vortex: driver, the release's headline, leads
  Added.
- README: a short section on querying .vortex files from IntelliJ or
  DBeaver with vortex-jdbc-driver, linking to the how-to.
- how-to: the driver's query tips cover this release's dictionary filters
  and ORDER BY ... LIMIT.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant