compatibility.md

Compatibility

Tested against the Rust reference implementation v0.88.0. For the rest of the API surface (reader, writer, scan, CLI), see reference.md.

Read-only deployment

The reader and inspector modules carry no transitive dependency on the writer module. A consumer that only needs to read Vortex files can depend on a strict subset:

<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-reader</artifactId>
  <version>0.18.0</version>
</dependency>

<!-- optional: inspector for layout-tree introspection -->
<dependency>
  <groupId>io.github.dfa1.vortex</groupId>
  <artifactId>vortex-inspector</artifactId>
  <version>0.18.0</version>
</dependency>

The zstd binding is optional on both sides: files using vortex.zstd need io.github.dfa1.zstd:zstd plus io.github.dfa1.zstd:zstd-platform on the classpath (versions pinned by the vortex BOM); without them, touching a vortex.zstd segment fails with a VortexException that names the two artifacts. All other encodings are pure Java.

./mvnw -pl core,reader,inspector verify builds the read-only artifact set without the writer module on the classpath. ReadRegistry.loadAll() registers only the built-in decoders in reader; no encoder class is loaded.

Known wire-format gaps

Item	Introduced	Java status
DType::Union (fbs.DType.Type.Union = 12)	Rust 0.71.0	⚠️ Read: DType.Union (Rust's shape rules) and the vortex.union sparse union array (UnionArray). Not written: vortex.union is in no Rust edition. No interop test: vortex-jni's Arrow bridge rejects unions, so no Rust producer exists.
vortex.onpair experimental string encoding	Rust 0.74.0	✅ Read and written. In core2026.08.1, so default cascading writes offer it, as Rust's default compressor does; the trained dictionary is valid for Rust but not byte-identical to Rust's (Rust's sampling RNG is not portable).
vortex.variant arbitrary nested objects	Rust (vortex.parquet.variant)	⚠️ Java reads vortex.parquet.variant, the Apache Variant binary encoding vortex-jni's default writer uses for Arrow arrow.parquet.variant columns, as a struct of per-row metadata/value binaries; it does not interpret the Variant binary itself. Java writes variant columns of typed scalar values only (constant / chunked-of-constants core, optional shredded child), via vortex.variant; writing vortex.parquet.variant is not implemented (ADR 0014).
Arrow extension array import affecting Variant shape	Rust 0.74.0 (#8125)	Untested against the currently pinned v0.88.0 fixtures; #8125 not yet re-verified.
vortex.dict layout over a values pool that is neither VarBin- nor primitive-shaped (e.g. a dict-encoded vortex.uuid, whose storage is FixedSizeList(U8, 16))	Not written by Rust: its dict layout admits only Primitive | Utf8 | Binary (dict_layout_supported), and its dict compressor schemes only integers, floats and strings	⚠️ Unreachable from Rust- or Java-written files. Every type vortex-jni writes reads back exactly, dict-encoded or not (DictAllTypesInteropIntegrationTest, which also fails if a Rust bump starts dict-encoding another type). A foreign file with such a pool fails with VortexException("unsupported dict values shape: …").
Duplicate struct field names	Rust writer rejects ("StructLayout must have unique field names"); Rust reader tolerates foreign files (first-match access)	⚠️ Deliberate divergence on read: Java rejects such files with VortexException("duplicate field name in file schema") instead of tolerating them — the name-keyed Chunk API cannot represent both columns, and silent column loss is worse than a loud failure on a file the reference writer refuses to produce. Java's writer mirrors the Rust writer's rejection.
Blank / control-character field names	Wire-legal; the Rust writer produces "" and whitespace-only names. NUL (U+0000) additionally aborts the Rust toolchain: Arrow FFI schema export hits a panic-cannot-unwind in arrow-rs (ffi_stream::get_schema) and SIGABRTs the process (measured against vortex-jni 0.75.0)	⚠️ Deliberate strictness BOTH ways: vortex-java's writer refuses blank and control-character field names (IllegalArgumentException), and its reader rejects files carrying them (VortexException naming the producing pipeline as the likely bug) — the JSON-""-key principle: wire-legal is a floor, not a policy. Printable names of any shape ($-runs, spaces inside, emoji) are legal and round-trip intact both directions (measured; pinned by ColumnNameEdgeCasesIntegrationTest).

SIMD kernels and the Vector API

Bulk loops (min/max, sums, run counting, widening, bit-packing, ...) live behind SimdOperations. Two implementations exist: the default, plain Java loops shaped for C2's auto-vectorizer, and one written with the incubating Vector API, selected only when the JVM is launched with --add-modules jdk.incubator.vector (omitting the flag is the opt-out; the JVM prints WARNING: Using incubator modules once). Both give Rust's results, pinned by a differential test: floating-point sums are sequential (lane-wise accumulation would change the last bits), and the I64 sum sums in vector lanes only when no prefix can overflow (every element in [-2^k, 2^k) with n * 2^k <= 2^63), otherwise it runs Rust's checked add per element, so [MAX, 1, -1] is dropped in both. F16 sum and min/max are scalar in both.

Separate from the Vector API, found while comparing against the Rust source:

• Float min/max statistics order -0.0 before 0.0 and skip NaN, as Rust's total_compare with skip_nans does.
• Float filters order values totally, as vortex-jni does: -0.0 < 0.0, and NaN equals itself and is the maximum (so NaN >= 0.0 matches, where IEEE says it does not). Zone-map pruning follows: a float zone without a nan_count (the legacy vortex.stats table) may hold a NaN, so it is never pruned on its maximum. Pinned by JavaWritesRustReadsIntegrationTest#javaWriter_jniReader_filteredFloats_matchJni (#518).
• U64 min/max and the dense-counter span in ArrayStats are computed in unsigned order, as Rust's typed_int_stats does.
• The zoned float Sum skips NaN, as Rust's skip_nans() binding does; an all-NaN zone sums to 0.0 (#523).

Real-world conformance: the Raincloud corpus

Cross-implementation conformance against Raincloud (tracked in #205): 430 curated public datasets whose .vortex files are written by the Vortex Python bindings (vortex-data 0.86.1, pinned via Raincloud v0.3.0) with a Parquet sibling built from the same Arrow table. RaincloudConformanceIntegrationTest reads each hydrated .vortex with vortex-java, exports CSV, and diffs it against the Parquet sibling read by hardwood (the oracle) — a zero-diff proves every value survived the Python-write / Java-read boundary.

scripts/hydrate-raincloud-corpus.sh --max-mb 200      # cache → mirror → local build
./mvnw verify -pl integration -am -Dvortex.it.excludedGroups= -Dit.test=RaincloudConformanceIntegrationTest

Per-slug status lives in integration/src/test/resources/raincloud/expected-status.csv (ok must pass; gap:<issue> must still fail, so a fix flips the entry in the same change; untriaged runs and reports without failing the build; missing_auth marks a slug that can't even be hydrated without credentials this environment lacks — Kaggle, gated HuggingFace, etc. — distinguishing "blocked on a credential" from plain untriaged). A scheduled workflow (raincloud-conformance.yml) hydrates a size-capped subset weekly. Current triage — 252 ok, 0 known gaps, 33 missing_auth, 145 untriaged (mostly large datasets and the tpcgen-rs-* family, which needs tpcgen-cli to build). On the v0.3.0 corpus, 172 hydrated datasets match their Parquet sibling exactly. Every gap found so far is fixed (#206–#211, #215–#217, #221, #225 RunEnd run-values validity, #226 Sparse null fill / nullable patches, #257 CsvExporter FixedSizeList/List support, #259 conformance-test pipe deadlock, #261 oracle repeated/list column support, plus two bugs the widened oracle then caught: CsvExporter list offsets assumed I32/I64 only (any integer width is wire-legal, mirroring VarBinArray), and ScanIterator could not slice a shared ListArray/FixedSizeListArray spanning several split windows). The v0.3.0 bump found three more, all fixed: vortex.bool bit offsets (#429), CSV export of decimals (#430) and nulls in nullable decimal_byte_parts columns (#431).

Encodings

Encoding ID	Decoder	Encoder	Decode	Encode	Notes
vortex.primitive	PrimitiveEncodingDecoder	PrimitiveEncodingEncoder	✅	✅	All PType (I8–I64, U8–U64, F16, F32, F64)
vortex.bool	BoolEncodingDecoder	BoolEncodingEncoder	✅	✅	Bool (bit-packed)
vortex.null	NullEncodingDecoder	NullEncodingEncoder	✅	✅	Null
vortex.bytebool	ByteBoolEncodingDecoder	ByteBoolEncodingEncoder	✅	✅	Bool (byte-per-element)
vortex.zigzag	ZigZagEncodingDecoder	ZigZagEncodingEncoder	✅	✅	Signed integer PTypes
vortex.constant	ConstantEncodingDecoder	ConstantEncodingEncoder	✅	✅	Primitive, Utf8, Binary, Bool, Null, Decimal, Extension; the cascade writes it for Primitive, Bool and Utf8
vortex.ext	ExtEncodingDecoder	ExtEncodingEncoder	✅	✅	Extension
vortex.runend	RunEndEncodingDecoder	RunEndEncodingEncoder	✅	✅	Primitive, Utf8/Binary, Bool
vortex.varbin	VarBinEncodingDecoder	VarBinEncodingEncoder	✅	✅	Utf8, Binary
vortex.varbinview	VarBinViewEncodingDecoder	VarBinViewEncodingEncoder	✅	✅	Utf8, Binary
vortex.alp	AlpEncodingDecoder	AlpEncodingEncoder	✅	✅	F64, F32
vortex.alprd	AlpRdEncodingDecoder	AlpRdEncodingEncoder	✅	✅	F64, F32
vortex.dict	DictEncodingDecoder	DictEncodingEncoder	✅	✅	Primitive, Utf8/Binary. Writes DictMetadata + children [codes, values], both cascaded (a float pool goes through ALP). With globalDict on (the default) the writer's data cascade never dictionary-encodes integers: Rust's file writer excludes IntDictScheme there and dictionary-encodes suitable columns once, as a layout. With it off (the CSV and Parquet importers stream), integer chunks keep their per-chunk dictionary. Decode also accepts the pre-0.14.5 Java shape (1-byte metadata, [values, codes]) so older Java-written files stay readable; that shape was never readable by Rust
vortex.sparse	SparseEncodingDecoder	SparseEncodingEncoder	✅	✅	Primitive, Bool, Utf8/Binary (decode); Primitive (encode)
vortex.sequence	SequenceEncodingDecoder	SequenceEncodingEncoder	✅	✅	Primitive (encode: integer only, as Rust; decode also float)
vortex.struct	StructEncodingDecoder	StructEncodingEncoder	✅	✅	Struct
vortex.chunked	ChunkedEncodingDecoder	ChunkedEncodingEncoder	✅	✅	Primitive + Struct concat
vortex.fsst	FsstEncodingDecoder	FsstEncodingEncoder	✅	✅	Utf8, Binary
vortex.list	ListEncodingDecoder	ListEncodingEncoder	✅	✅	A nullable list's validity is its optional third child, as in Rust
vortex.listview	ListViewEncodingDecoder	ListViewEncodingEncoder	✅	✅	Validity lives in the encoding's own fourth child slot, not under a vortex.masked wrapper. Read: both shapes decode. Write: only vortex.map's entries child takes this path — a plain nullable DType.List column still routes through vortex.masked + vortex.list
vortex.map	MapEncodingDecoder	MapEncodingEncoder	✅	✅	Single entries child, a bare vortex.listview of non-nullable {key, value} structs (any other entries encoding is rejected, as in Rust); map validity is delegated to that child's validity slot. Single-flat columns only: neither ChunkedArrayCombiner nor ScanIterator's row-window slicing has a MapArray arm yet, so a chunked map column or a split-window scan fails as VortexException. Pre-existing gap shared with vortex.listview; tracked separately
vortex.fixed_size_list	FixedSizeListEncodingDecoder	FixedSizeListEncodingEncoder	✅	✅
vortex.zstd	ZstdEncodingDecoder	ZstdEncodingEncoder	✅	✅	Primitive, Utf8, Binary
vortex.masked	MaskedEncodingDecoder	MaskedEncodingEncoder	✅	✅	NullableData carrier; inner values cascade (Dict/FSST/ALP/…) when cascading is enabled; Utf8/Primitive still compete on measured size even at depth 0, else first-match FixedSizeList. A nullable Utf8 column whose per-chunk dict is smaller skips the mask entirely: null is one more dict entry, invalid in the pool's validity, with non-nullable codes, as Rust's dict builder writes it
vortex.decimal	DecimalEncodingDecoder	DecimalEncodingEncoder	✅	✅	Precision 1–76 (i8 … i256 storage by precision), negative scales. Written from BigDecimal[]
vortex.decimal_byte_parts	DecimalBytePartsEncodingDecoder	DecimalBytePartsEncodingEncoder	✅	✅	Write: precision ≤ 18 only (one i64 mantissa, no low parts); competes in the cascade
vortex.datetimeparts	DateTimePartsEncodingDecoder	DateTimePartsEncodingEncoder	✅	✅
vortex.pco	PcoEncodingDecoder	PcoEncodingEncoder	✅	✅	Decode: all modes. Encode: Classic + Consecutive (orders 1–7) delta + IntMult, in Rust's chunk/page layout and byte-identical to pco 1.0.1 for those; FloatMult/FloatQuant and Lookback deferred
fastlanes.bitpacked	BitpackedEncodingDecoder	BitpackedEncodingEncoder	✅	✅	Unsigned integer PTypes
fastlanes.delta	DeltaEncodingDecoder	DeltaEncodingEncoder	✅	✅	Integer PTypes. In no edition: cascading writes offer it (bases/deltas cascaded, as Rust's DeltaScheme) only with WriteOptions.withoutEditions()
fastlanes.for	FrameOfReferenceEncodingDecoder	FrameOfReferenceEncodingEncoder	✅	✅	Integer PTypes
fastlanes.rle	RleEncodingDecoder	RleEncodingEncoder	✅	✅	Chunk-based RLE. Integers and floats (Rust's int and float RLE schemes); float runs compare raw bits, so -0.0 and NaN payloads round-trip. Cascades values/indices/offsets
vortex.patched	PatchedEncodingDecoder	PatchedEncodingEncoder	✅	✅	Primitive PTypes; base + chunked patches (1024-elem blocks)
vortex.variant	VariantEncodingDecoder	VariantEncodingEncoder	✅	✅	Canonical container; constant / chunked-of-constants core + optional shredded child. Typed-scalar values only — nested objects need parquet.variant (ADR 0014)
vortex.union	UnionEncodingDecoder	—	✅	❌	Read only: sparse union, U8 type ids (nullable iff the union is) + one row-aligned child per variant; in no edition, so never written
vortex.parquet.variant	ParquetVariantEncodingDecoder	—	✅	❌	Read only: decodes to a struct {metadata, value?, typed_value?} of Apache Variant binaries (Arrow's arrow.parquet.variant storage shape), masked by row validity. Writing is not implemented; Java writes typed-scalar variants through vortex.variant (ADR 0014)
vortex.zstd_buffers	ZstdBuffersEncodingDecoder	—	✅	❌	Read only (zstd2026.02.0, opt-in in Rust): each buffer of any wrapped encoding is Zstd-decompressed at its declared alignment, then the inner encoding decodes with its own children. Needs the optional zstd binding, like vortex.zstd. Interop-tested against a Rust-written fixture (scripts/fixtures/zstd-buffers)
vortex.onpair	OnPairEncodingDecoder	OnPairEncodingEncoder	✅	✅	Utf8, Binary; core2026.08.1, a cascade candidate under the default edition (competes with FSST, as in Rust)

Decode shape

Per ADR 0010 and ADR 0012, each decoder falls into one of three shapes:

• Zero-copy — output is a view over the memory-mapped file (or a wrapper over child arrays). No arena allocation, no per-element copy.
• Lazy — output is a LazyXxxArray / ChunkedXxxArray record that holds the encoded child plus the transform parameters. Per-row getXxx(i) applies the transform on demand. No output buffer is allocated unless a caller explicitly materializes via Array.materialize(arena).
• Materialized — output is a buffer allocated from ctx.arena() populated during decode(). Required for decompression-style encodings (Bitpacked, Pco, Zstd, etc.) where reading element i would require decoding a window.

Encoding ID	Now	Target	Notes
vortex.primitive	Zero-copy	Zero-copy	mmap slice
vortex.bool	Zero-copy	Zero-copy	mmap slice (bit-packed)
vortex.null	n/a	n/a	no per-row data
vortex.bytebool	Zero-copy	Zero-copy	LazyByteBoolArray — mmap slice read byte-per-row; bitmap on materialize
vortex.zigzag	Lazy	Lazy	LazyZigZagXxxArray (I8/I16/I32/I64); broadcast → LazyConstantXxxArray, ADR 0010 + 0015
vortex.constant	Lazy	Lazy	LazyConstantXxxArray (primitive + bool + decimal) + VarBinConstantArray (Utf8/Binary); per-row broadcast, no buffer, ADR 0015
vortex.ext	Zero-copy	Zero-copy	wraps storage
vortex.runend	Lazy	Lazy	LazyRunEndXxxArray (primitive + bool) + VarBinRunEndArray (Utf8/Binary); run resolved per access, no expansion, ADR 0015
vortex.varbin	Zero-copy	Zero-copy	bytes + offsets slices
vortex.varbinview	Lazy	Lazy	VarBinViewArray — keeps views + data buffers as mmap slices
vortex.alp	Lazy	Lazy	LazyAlpXxxArray; broadcast → LazyConstantXxxArray; patched stays Materialized, ADR 0010 + 0015
vortex.alprd	Lazy	Lazy	LazyAlpRdDoubleArray/LazyAlpRdFloatArray — left/right + patches on access
vortex.dict	Lazy	Lazy	DictXxxArray (numeric) + VarBinDictArray (string), ADR 0012. The vortex.dict layout resolves the same way for any VarBin-shaped values pool — a StringView pool, or any VarBin-backed extension dtype — normalizing a pool no longer than its codes whole and a longer one only up to the entries the codes reach, never the rows; a vortex.constant pool collapses to VarBinConstantArray
vortex.sparse	Lazy	Lazy	LazySparseXxxArray (primitive + bool) + VarBinSparseArray (Utf8/Binary); fill broadcast, patch resolved per access; patch-free range → VarBinConstantArray, ADR 0015
vortex.sequence	Lazy	Lazy	LazySequenceXxxArray; base + i * multiplier per access, no buffer, ADR 0015
vortex.struct	Zero-copy	Zero-copy	StructArray wraps fields
vortex.chunked	Lazy	Lazy	ChunkedXxxArray (primitive/Bool) + VarBinChunkedArray (Utf8/Binary), ADR 0012
vortex.fsst	Lazy	Lazy	LazyFsstVarBinArray — per-row code range is independent, so getBytes(i) decompresses only row i; getByteLength/forEachByteLength read the uncompressed-lengths child directly, no decompression at all, ADR 0026
vortex.list	Lazy	Lazy	ListArray wraps elements + offsets children; shape inherits from child; a validity child yields a MaskedArray over it
vortex.listview	Lazy	Lazy	ListViewArray wraps elements + offsets + sizes children; a validity child yields a MaskedArray over it
vortex.map	Lazy	Lazy	MapArray wraps the entries child (a ListViewArray, or a MaskedArray over one when the map is nullable)
vortex.fixed_size_list	Lazy	Lazy	FixedSizeListArray wraps flat elements child; no per-row alloc
vortex.zstd	Materialized	Materialized	block decompression
vortex.masked	Zero-copy	Zero-copy	wraps inner + validity
vortex.decimal	Lazy	Lazy	LazyDecimalArray — BigDecimal materialized per row on getDecimal(i)
vortex.decimal_byte_parts	Lazy	Lazy	LazyDecimalBytePartsArray — reassembles byte parts on access
vortex.datetimeparts	Lazy	Lazy	LazyDateTimePartsLongArray — reassembles parts on access
vortex.pco	Materialized	Materialized	range-encoded decompression
fastlanes.bitpacked	Materialized	Materialized	window unpacks bits
fastlanes.delta	Materialized	Materialized	cumulative sum requires sequential decode; output is one arena segment at the ptype's width, and only the chunks the row window touches are reconstructed
fastlanes.for	Lazy	Lazy	LazyForXxxArray (I8/U8/I16/U16/I32/U32/I64/U64), ADR 0010 + 0015
fastlanes.rle	Lazy	Lazy	LazyRleXxxArray reads run values and the index table straight from their mmapped segments; validity → OffsetBoolArray; empty → LazyConstantXxxArray, ADR 0015
vortex.patched	Materialized	Materialized	inner is full base + chunked patches (1024-elem blocks, lane-window-sorted); per-row access requires 2 laneOffsets reads + binary search inside the chunk window, so eager scatter wins for full scans
vortex.variant	Lazy	Lazy	container wraps constant/chunked core (inner-typed) + optional shredded child
vortex.onpair	Materialized	Materialized	codes walked in order into one VarBinOffsetArray
vortex.union	Lazy	Lazy	UnionArray over its decoded type ids and variant children

Decompression-style encodings (Bitpacked / Pco / Zstd / Delta) stay Materialized by design — element-at-i requires decoding a window, so they must allocate output (ADR 0010). Their output can itself be wrapped in a 1:1 lazy transform (e.g. ALP over Bitpacked produces LazyAlp(MaterializedXxx)). Fsst is the one exception: its per-row code range is independent of every other row, so it stays Lazy instead (ADR 0026).

A scan filtered on a vortex.dict layout column skips the chunks of a dictionary none of whose values satisfies the filter, when a null row cannot satisfy it either, deciding from the values without reading a code. Rust evaluates a dictionary filter over the values too, but always reads the codes: results are the same, but a corrupt codes buffer under such a filter goes unreported where Rust fails the scan. This is a deliberate divergence, for WHERE col = 'x' with x absent from the dictionary. The row test Compute.matcher(RowFilter) builds for a string dictionary follows Rust: the filter is evaluated once per dictionary value per scan, and u8/u16 codes are mapped to the selected rows in one pass, Rust's values.take(codes).

Unknown encodings

Files containing unrecognized encoding IDs throw VortexException by default. The message names which edition the id belongs to, or says the id is unknown to every edition if it belongs to none. Opt in to passthrough mode to read such files without failing:

ReadRegistry registry = ReadRegistry.builder()
        .registerDefaults()
        .allowUnknown()
        .build();
try (VortexReader vf = VortexReader.open(path, registry)) {
    // columns with unknown encodings are returned as UnknownArray
}

Editions

Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of the wire format. WriteOptions.defaults() targets the latest frozen core edition; see the reference doc for the writer/reader integration.

Edition	Family	Adds
core2025.05.0	core	23 baseline: fastlanes.bitpacked/for, vortex.alp/alprd/bool/bytebool/chunked/constant/datetimeparts/decimal/decimal_byte_parts/dict/ext/fsst/list/null/primitive/runend/sparse/struct/varbin/varbinview/zigzag
core2025.06.0	core	vortex.pco, vortex.sequence, vortex.zstd
core2025.10.0	core	fastlanes.rle, vortex.fixed_size_list, vortex.listview, vortex.masked
core2026.08.0	core	no encoding; Rust adds the vortex.zoned layout, which the writer emits from this edition on (older targets get vortex.stats)
core2026.08.1	core	vortex.onpair
core2026.08.2	core	vortex.map
core2026.08.3	core	vortex.variant, vortex.parquet.variant (read only) — default write target
preview2026.08.0	preview	nothing yet
zstd2026.02.0	zstd	vortex.zstd_buffers (read only; buffer-level Zstd for GPU decode; declared by Rust's vortex-zstd plugin, opt-in)

Mirrors Rust's vortex-edition declarations at the pinned vortex-jni release (0.88.0); EditionCatalogParityIntegrationTest fails the build on drift. core editions are frozen with a forever read-compatibility guarantee; preview holds opt-in components not yet adopted by core; zstd is declared by Rust's vortex-zstd plugin crate rather than the core declarations, and is never enabled by default. Zone maps follow the edition too. A write targeting core2026.08.0 or later (the default) emits Rust's vortex.zoned with fixed 8192-row zones and the aggregate set vortex-jni writes; earlier targets emit the legacy vortex.stats, one zone per writeChunk batch. vortex-java readers from 0.12.1 read vortex.zoned; before this release they placed a dictionary column's vortex.zoned zones by its code chunks rather than the stride, which can mis-attribute zone stats when the chunk count happens to equal the zone count, so read new files with this release or later.

Per-column encoder choice (WriteOptions.withColumnEncoding) mirrors Rust's WriteStrategyBuilder::with_field_writer with a compressor restricted to chosen schemes, but is narrower: a Rust field writer replaces the column's whole layout strategy, while vortex-java keeps the column's zone map and chunking and only restricts its encodings.

fastlanes.delta and vortex.patched are in no edition: as in Rust, only a write with the guard off (WriteOptions.withoutEditions()) may emit them. Rust editions also gate layouts, extension dtypes and zone-map aggregates; vortex-java's catalog models array encodings only. (Upstream additionally records each edition's minimum Rust Vortex release; that number belongs to a different implementation's release train and isn't surfaced in any vortex-java API.)

Extension types

Extension dtypes wrap a primitive storage array with a logical-id tag plus optional metadata. The Rust catalog lives in vortex-array/src/extension/; each subdir below names a canonical extension id and its on-disk shape.

Read-side extensions live in io.github.dfa1.vortex.reader.extension (write-side encoders implement ExtensionEncoder in the writer module). Both are registered on their registry, the built-ins by registerDefaults(), as Rust registers an ExtVTable on the session. Each spec extension is a singleton with typed decode methods on the concrete impl — grab the singleton directly:

DType.Extension dtype = (DType.Extension) schema.field(ColumnName.of("birthdays"));
List<LocalDate> values = DateExtensionDecoder.INSTANCE.decodeAll(dtype, chunk.column(ColumnName.of("birthdays")));

End-to-end round-trip — write a List<LocalDate>, read it back:

var schema = DType.structBuilder()
        .field(ColumnName.of("birthdays"), DateExtensionDecoder.INSTANCE.dtype(false))
        .build();
writer.writeChunk(c -> c.put(ColumnName.of("birthdays"), dates));              // Collection auto-routed

try (var iter = reader.scan(ScanOptions.all());
     Chunk chunk = iter.next()) {
    List<LocalDate> back = chunk.as(ColumnName.of("birthdays"), LocalDate.class);
}

Chunk.as(name, Class) decodes through the ExtensionDecoder registered for the column's id (LocalDate ↔ vortex.date, LocalTime ↔ vortex.time, Instant ↔ vortex.timestamp, UUID ↔ vortex.uuid, plus any registered third-party type). Both sides run the registered validate(DType.Extension) — Rust's validate_dtype — on every extension dtype, at VortexWriter.create and at open, and on the dtypes embedded in Variant metadata. The four built-ins apply Rust's deserialize_metadata + validate_dtype rules (ExtensionId.WellKnown#validate): vortex.date Days → I32, Milliseconds → I64; vortex.time s/ms → I32, μs/ns → I64; vortex.timestamp metadata ≥ 3 bytes with a UTF-8 zone and I64 storage; vortex.uuid 0–1 metadata bytes (a known version) over FixedSizeList(non-null U8, 16). An extension dtype must carry its metadata field, empty when there is none, as Rust requires; the writer always emits it.

ExtensionId is a sealed WellKnown/Custom type like EncodingId: the spec ids (VORTEX_DATE, VORTEX_TIME, VORTEX_TIMESTAMP, VORTEX_UUID) plus any third-party id. The ids the vortex-jni session registers by default are ExtensionId.WellKnown: besides the four above, vortex.json and the vortex.st.* geometries (wkb, point, linestring, polygon, multipoint, multilinestring, multipolygon, box). A file with one of them opens without a decoder and the column reads as its storage — JSON text as Utf8, WKB as Binary, geometries as GeoArrow coordinate structs and lists — validated with Rust's storage rules (the spatial CRS metadata is not decoded). Java has no JSON or geometry domain type, so Chunk.as refuses them.

As in Rust, opening a file with any other extension id that has no registered decoder fails (VortexException; Rust: "No such DType extension ID"), unless ReadRegistry.Builder#allowUnknown() is set — then the id round-trips verbatim through DType.Extension's raw String field and the column reads as its storage array, Rust's ForeignExtDType. vortex-jni 0.88.0 exposes no allow_unknown, so it cannot read a file with a custom extension id at all. A blank id, or one with a control character, fails the read with VortexException (Rust accepts any string).

Extension id	Impl	Storage	Metadata	Round-trip
vortex.date	DateExtension	I32 days (Arrow Date32) or I64 milliseconds (Date64) since 1970-01-01	1 byte: TimeUnit (Days or Milliseconds)	✅ (writes Days)
vortex.time	TimeExtension	I32 (s/ms) or I64 (μs/ns) since midnight	1 byte: TimeUnit	✅
vortex.timestamp	TimestampExtension	I64 epoch count in the recorded TimeUnit	unit byte + u16 LE tz_len + UTF-8 tz	✅
vortex.uuid	UuidExtension	FixedSizeList(Primitive(U8), 16)	none	✅
vortex.json	none (storage)	Utf8	empty	storage
vortex.st.*	none (storage)	Binary (wkb) or GeoArrow coordinate Struct/List	spatial protobuf (not decoded)	storage
custom ids	registered decoder	whatever the column declares	opaque bytes	✅ when registered; storage with allowUnknown()

TimeUnit (see extension/datetime/unit.rs) encodes precision in the first metadata byte:

Value	Unit
0	Nanoseconds
1	Microseconds
2	Milliseconds
3	Seconds
4	Days

For unsupported extension ids the inspector falls back to a placeholder cell (<GenericArray ext<vortex.X>>); the underlying storage array still decodes correctly via the primitive accessors, callers just have to format the value themselves.

S3 Fixture Status (v0.88.0)

Note: the oracle round-trip suite is pinned to v0.88.0/ (current Rust release). The bucket reuses identical fixture file names across versions but rewrites the bytes, so the local cache is version-keyed; bump the RustFixtures.VERSION constant in the integration tests and refresh this section when a newer set is published.

Cross-language round-trips tested against Rust-written fixture files hosted at s3://vortex-compat-fixtures/v0.88.0/arrays/.

Fixture	Status
primitives.vortex	✅
alp.vortex	✅
bitpacked.vortex	✅
booleans.vortex	✅
constant.vortex	✅
for.vortex	✅
fsst.vortex	✅
runend.vortex	✅
sequence.vortex	✅
varbin.vortex	✅
struct_nested.vortex	✅
null.vortex	✅
bytebool.vortex	✅
zigzag.vortex	✅
datetime.vortex	✅
dict.vortex	✅
sparse.vortex	✅
varbinview.vortex	✅
chunked.vortex	✅
rle.vortex	✅
alprd.vortex	✅
decimal.vortex	✅
decimal_byte_parts.vortex	✅
datetimeparts.vortex	✅
list.vortex	✅
listview.vortex	✅
fixed_size_list.vortex	✅
zstd.vortex	✅
tpch_lineitem.compact.vortex	✅
tpch_lineitem.regular.vortex	✅
tpch_orders.compact.vortex	✅
tpch_orders.regular.vortex	✅
pco.vortex	✅
clickbench_hits_5k.compact.vortex	✅
clickbench_hits_5k.regular.vortex	✅
masked.vortex	❓
patched.vortex	❓
variant.vortex	✅
map.vortex	✅