Repository navigation
Writer: flat array stats should match the ones Rust keeps per encoding node #583
Description
Follow-up to #581. vortex-java and vortex-jni write different stats into flat segments' ArrayNodes.
What Rust does
FlatLayoutStrategy doesn't compute stats; it keeps the ones already cached on the array (flat/writer.rs: "flat layouts do not force compute stats during write, they only retain pre-computed stats"). It serializes them on every node of the tree. So what ends up in a node depends on what the compressor computed while picking a scheme, and on which encodings pass those stats on to their output array (statistics().inherit* / with_stats_set in encodings/fastlanes/src/bitpacking/array/bitpack_compress/{global,blocked}.rs, vortex-btrblocks/src/schemes/integer/{bitpacking,for_}.rs, vortex-btrblocks/src/schemes/float/alp.rs, encodings/sequence/src/array.rs, encodings/runend/src/array.rs, encodings/zstd/src/zstd_buffers.rs, vortex-array/src/executor.rs).
Observed (same 20k-row data through both writers, all stats Exact)
| root encoding (Rust) | Rust writes | Java writes |
|---|---|---|
vortex.primitive, fastlanes.bitpacked |
min, max, is_constant=false, null_count | min, max, null_count |
vortex.sequence |
is_sorted, is_strict_sorted | min, max, null_count |
vortex.alp, vortex.alprd, vortex.fsst, vortex.constant |
nothing | min, max, null_count |
vortex.bool |
is_constant | min, max, null_count |
| child nodes (ALP's sequence, FSST offsets, validity bool, dict codes) | often stats (sorted flags, min/max/sum/null_count) | never: Java writes stats on the root only |
Zone-map table segments differ too. Rust's struct children carry stats and aren't masked; Java wraps each field in vortex.masked and writes no child stats.
Plan
- Port the inheritance rules per encoding from the Rust source (not from the table above), applied through Java's cascade to every node, root and children.
- Cross-check: the same data through vortex-jni and vortex-java, asserting identical per-node stats (encoding, stat set, values, precision) for each column shape: sorted/random/small-range ints, random/sorted floats, low- and high-cardinality strings, bool, nullable, constant.
Expected consequences (intended, per the parity directive)
- Java-written string columns lose their flat min/max (FSST and dict values keep none in Rust). So
VortexReader.columnStats()string min/max and the SQLMIN/MAXstats shortcut go away for them, as already happens for Rust-written files.JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredUtf8_prunesOnTruncatedBounds(and the emoji test above it) then expectnullmin/max. - Flat-path pruning with zone maps disabled mostly stops for non-primitive encodings.
- Numeric
columnStatsare unaffected, because they come from the zone table. VortexWriter.buildArrayFlatBuffer's comment saysnull_countis always written "matching the Rust writer". vortex-jni no longer does that for every encoding, so the comment and the IS NULL pruning it enables need checking.
The zone-table segment shape (masked fields, child stats) is a separate gap and could be split out.
Activity