How-to guides

Task-oriented recipes. Each section solves one concrete goal. For API details (classes, methods, operator tables), see reference.md. For the design rationale behind the iterator lifecycle, see explanation.md#memory-model.

Build the CLI

Build the fat jar once; reuse it for every CLI recipe below:

./mvnw package -pl cli -am -DskipTests
java -jar cli/target/vortex-cli-*-all.jar <subcommand> [args]

For the full subcommand list, see reference.md#cli.

Count rows

API:

var total = new java.util.concurrent.atomic.AtomicLong();
try (VortexReader vf = VortexReader.open(Path.of("data.vortex"));
     var iter = vf.scan(ScanOptions.all())) {
    iter.forEachRemaining(c -> total.addAndGet(c.rowCount()));
}
System.out.println(total.get());

CLI:

java -jar cli/target/vortex-cli-*-all.jar count data.vortex

Inspect file structure

API:

try (VortexReader vf = VortexReader.open(Path.of("data.vortex"))) {
    System.out.println(vf.dtype());   // column names and types
    System.out.println(vf.layout());  // layout tree (Struct → Chunked → Flat …)
}

CLI:

# column names and types
java -jar cli/target/vortex-cli-*-all.jar schema data.vortex

# full layout tree with encoding IDs, row counts, buffer sizes
java -jar cli/target/vortex-cli-*-all.jar inspect data.vortex

# per-column min/max statistics
java -jar cli/target/vortex-cli-*-all.jar stats data.vortex

Explore in the browser

explore is the data-first view, for asking what is in a file rather than how it is stored. By default (--serve) it serves a page on a local port and opens it in your browser, and runs until you press Ctrl-C.

java -jar cli/target/vortex-cli-*-all.jar explore data.vortex
# serving http://127.0.0.1:49605/b3a1f2.../ - press Ctrl-C to stop

To publish or send the page instead, --html analyzes every column up front and writes one self-contained file, nothing to fetch and no server needed:

java -jar cli/target/vortex-cli-*-all.jar explore --html report.html data.vortex
# written: report.html  (19 columns analyzed, 111.4 KB)

The page lists every column with its share of nulls, its min and max, and a sparkline of each zone's min-max band from the first row to the last. The sparkline shows whether a column is sorted (a rising band), clustered or random, which is also how well a filter on it prunes zones. Everything comes from the statistics the file already records and no data is decoded, so a 2.3 GB, 100-million-row file opens in well under a second. A mean appears only for files that record a sum: Rust's zone map and array stats carry none.

Analyze on a column (or Analyze all) reads its values once and adds, under its row, the non-null count, the total and mean of a numeric column, the number of distinct values, a string column's byte lengths, and how the values are spread:

a column with at most 30 distinct values (a status, a country, a flag) shows each value with its count and share
a numeric, date or timestamp column with more shows a 30-bin histogram between its min and max
a string column with more shows its 10 most frequent values, and past 10,000 distinct values none

This one decodes the column: on a 2.3 GB, 100-million-row file each column takes one to two seconds. Columns are analyzed in parallel, one per core, so Analyze all on that file's nine columns takes about as long as the slowest one: 2.2 s on 10 cores against 9-11 s one after another.

The page is reachable only from this machine (127.0.0.1) and only under the random path printed at start-up. The file can also be an http(s):// URL, as for inspect; analyzing a column then downloads it.

Inspect as a web page

inspect --html writes a single self-contained HTML page to stdout — no scripts, no CDN, nothing to install. It shows where each column's bytes physically sit in the file, what share of the file each column costs, and the per-chunk row ranges, min/max and sizes behind those totals.

java -jar cli/target/vortex-cli-*-all.jar inspect --html data.vortex > report.html
open report.html

The file can also be remote. Any http(s):// URL whose server answers HTTP Range requests works, S3 included, and nothing is downloaded up front. This is one of Rust's compat fixtures on S3:

java -jar cli/target/vortex-cli-*-all.jar inspect --html \
    https://vortex-compat-fixtures.s3.amazonaws.com/v0.88.0/arrays/tpch_orders.compact.vortex \
    > report.html

The report fetches every data segment to read its encoding and stats, one Range request each, so a remote file costs one round trip per segment: this 22-segment, 325 KB file takes about 11 s from S3 against well under a second locally. For a large remote file, download it first.

Each schema row names the encoding a reader dispatches on for that column (fastlanes.for, vortex.fsst, vortex.dict, …) and, for fixed-width columns, how much smaller it is on disk than unencoded — 400,000 rows of I64 is 3.1 MB raw against 781 KB stored, so 4.0x. Variable-width columns claim no ratio: what a utf8 column would occupy unencoded depends on its contents, which the footer does not record.

Columns are colored in a fixed palette order that is validated for color-vision deficiency in both light and dark mode; past the eighth column the strip falls back to a neutral fill and the schema table names every column regardless. Click a column or a chunk to expand its detail table.

Inspect interactively (TUI)

For files where the static inspect output is too dense, the tui subcommand opens an interactive terminal browser. The layout tree is loaded lazily — per-array statistics, dictionary entries, hex previews, and decoded data are fetched on demand as you navigate.

# local file
java -jar cli/target/vortex-cli-*-all.jar tui data.vortex

# remote file (HTTP range requests)
java -jar cli/target/vortex-cli-*-all.jar tui https://example.com/data.vortex

A loading bar prints to stderr while metadata is read, then the screen splits into a tree pane on the left and a details pane on the right:

 data.vortex                                                                    
 v struct  (3000000 rows)                                                       
     v timestamp: vortex.zoned  (3000000 rows, stats)              | encoding: vortex.zoned
         > vortex.chunked  (3000000 rows)                          | rows:     3000000
     > symbol: vortex.dict  (3000000 rows)                         | min:      1700000000000
     > price: vortex.alp  (3000000 rows, stats)                    | max:      1700002999000
       volume: fastlanes.bitpacked  (3000000 rows)                 |
                                                                   | bit width: 21
                                                                   | offsets:   8 segments
                                                                   |
                                                                   | preview (hex):
                                                                   |   0x00f0c2e9b3 8c01...
 ↑↓ nav   →/Enter expand   ← collapse   q quit

Keymap:

Key	Action
`↑` / `↓`	Move selection one row
`PgUp` / `PgDn`	Jump 10 rows
`Home` / `End`	Jump to first / last visible row
`→`	Expand node
`←`	Collapse node
`Enter`	Toggle expand / collapse
`q` / `Q` / `Esc`	Quit

Tree markers:

Marker	Meaning
`>`	Collapsed (has children)
`v`	Expanded
(none)	Leaf node

The , stats suffix on a row indicates the node carries zone-map statistics (min / max per chunk) — selecting it shows the values in the details pane. vortex.dict nodes show their dictionary entries; flat numeric leaves show a hex preview of the encoded buffer plus decoded data.

Windows: Git Bash / MinTTY. The TUI calls GetConsoleMode on stdio, which only works on a real Windows console handle. Git Bash and other MinTTY-based shells pipe stdio through the terminal emulator, so the console APIs fail and the TUI aborts with a winpty pointer in the error message. Two options:

# wrap with winpty (ships with Git for Windows)
winpty java -jar vortex-cli-*-all.jar tui data.vortex

# or switch to a terminal that attaches a real console: Windows Terminal,
# PowerShell, or cmd.exe

inspect (static, non-interactive) works in any shell since it does not toggle terminal modes.

Project columns

API:

ScanOptions opts = ScanOptions.all().withColumns(ColumnName.of("symbol"), ColumnName.of("price"));

try (VortexReader vf = VortexReader.open(Path.of("trades.vortex"));
     var iter = vf.scan(opts)) {
    while (iter.hasNext()) {
        try (var chunk = iter.next()) {
            // chunk.columns() contains only "symbol" and "price"
        }
    }
}

CLI:

java -jar cli/target/vortex-cli-*-all.jar select trades.vortex symbol price

Add --where (repeatable, all conditions must hold) to keep only some rows. It takes the same expressions as filter, and the filtered column does not have to be printed. An interval is two conditions:

java -jar cli/target/vortex-cli-*-all.jar select trades.vortex ts price \
    --where "symbol = BTCUSDT" --where "ts >= 1718452800000" --where "ts < 1718456400000"

Filter rows

API:

ScanFilter filter = ScanFilter.gte(ColumnName.of("volume"), 1_000_000);
ScanOptions opts = ScanOptions.all().withFilter(filter);

try (VortexReader vf = VortexReader.open(Path.of("trades.vortex"));
     var iter = vf.scan(opts)) {
    while (iter.hasNext()) {
        try (var chunk = iter.next()) {
            // only rows where volume >= 1_000_000
        }
    }
}

Combine filters with and():

ScanFilter filter = ScanFilter.gte(ColumnName.of("volume"), 1_000_000)
    .and(ScanFilter.lte(ColumnName.of("price"), 200.0));

For the supported predicate set and CLI operator syntax, see reference.md#scanfilter and reference.md#filter-expression-syntax.

CLI:

java -jar cli/target/vortex-cli-*-all.jar filter trades.vortex "volume >= 1000000"

filter prints every column of the matching rows. To print only some columns, or to combine several conditions, use select ... --where (see Project columns).

Time a CLI command

Add --timing to any vortex subcommand to see how long the operation took. The time goes to stderr, so the data on stdout is untouched, and it leaves out the JVM's start-up:

java -jar cli/target/vortex-cli-*-all.jar count trades.vortex --timing

4216320
elapsed: 33.7 ms

It works in a pipeline too, where only the data is passed on and the time still appears on your terminal:

java -jar cli/target/vortex-cli-*-all.jar select trades.vortex ts price \
    --where "symbol = BTCUSDT" --timing | head -3

Compare it with time java -jar ..., which also counts the JVM's own start-up (about 25 to 30 ms here).

Preview the first N rows

API:

ScanOptions opts = ScanOptions.all().withLimit(ScanLimit.of(10));

try (VortexReader vf = VortexReader.open(Path.of("data.vortex"));
     var iter = vf.scan(opts)) {
    while (iter.hasNext()) {
        try (var chunk = iter.next()) {
            // at most 10 rows total across all chunks
        }
    }
}

CLI:

# export first 10 rows to CSV
java -jar cli/target/vortex-cli-*-all.jar export data.vortex | head -n 11   # 1 header + 10 rows

Convert Parquet to Vortex

API:

import io.github.dfa1.vortex.parquet.ParquetImporter;

ParquetImporter.importParquet(
    Path.of("data.parquet"),
    Path.of("data.vortex")
);

Project specific columns during conversion:

import io.github.dfa1.vortex.parquet.ImportOptions;

ImportOptions opts = ImportOptions.defaults()
    .withColumns(List.of("trip_distance", "fare_amount"));

ParquetImporter.importParquet(Path.of("data.parquet"), Path.of("data.vortex"), opts);

From a remote Parquet file over HTTP(S), fetched entirely through targeted Range requests — no full-file download occurs:

ParquetImporter.importParquet(URI.create("https://example.com/data.parquet"), Path.of("data.vortex"));

CLI:

# output defaults to <input>.vortex
java -jar cli/target/vortex-cli-*-all.jar import data.parquet

# explicit output path
java -jar cli/target/vortex-cli-*-all.jar import data.parquet out.vortex

# remote source — Parquet only, output path required or derived from the URL's file name
java -jar cli/target/vortex-cli-*-all.jar import https://example.com/data.parquet out.vortex

Convert Vortex to Parquet

Flat schemas only (Bool, Primitive including F16, Utf8, Binary, vortex.timestamp); a Struct/List/Map top-level column throws UnsupportedOperationException.

API:

import io.github.dfa1.vortex.parquet.ParquetExporter;

ParquetExporter.exportParquet(
    Path.of("data.vortex"),
    Path.of("data.parquet")
);

Project specific columns during conversion:

import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.parquet.ExportOptions;

ExportOptions opts = ExportOptions.defaults()
    .withColumns(List.of(ColumnName.of("trip_distance"), ColumnName.of("fare_amount")));

ParquetExporter.exportParquet(Path.of("data.vortex"), Path.of("data.parquet"), opts);

From an already-open handle — a local VortexReader or a remote VortexHttpReader — without an intervening local copy; the handle isn't closed here, the caller keeps ownership of its lifecycle:

import io.github.dfa1.vortex.reader.VortexHttpReader;

try (var vortex = VortexHttpReader.open(URI.create("https://example.com/data.vortex"))) {
    ParquetExporter.exportParquet(vortex, Path.of("data.parquet"));
}

CLI:

# dispatches on the output extension — same `export` subcommand CSV export uses
java -jar cli/target/vortex-cli-*-all.jar export data.vortex out.parquet

# remote source — Parquet output only, and the output path must be given explicitly
java -jar cli/target/vortex-cli-*-all.jar export https://example.com/data.vortex out.parquet

Convert CSV to Vortex

API:

import io.github.dfa1.vortex.csv.CsvImporter;

CsvImporter.importCsv(Path.of("data.csv"), Path.of("data.vortex"));

From a remote CSV file over HTTP(S). CSV is read front to back in one streaming pass, so the response body is consumed directly — no Range requests, no local temp file:

CsvImporter.importCsv(URI.create("https://example.com/data.csv"), Path.of("data.vortex"));

CLI (types are inferred from the data):

java -jar cli/target/vortex-cli-*-all.jar import data.csv
# writes data.vortex, prints size savings

# remote source
java -jar cli/target/vortex-cli-*-all.jar import https://example.com/data.csv out.vortex

# straight to Parquet — chains CSV -> temp Vortex -> Parquet internally, local or remote source
java -jar cli/target/vortex-cli-*-all.jar import data.csv out.parquet
java -jar cli/target/vortex-cli-*-all.jar import https://example.com/data.csv out.parquet

Text-heavy files (long, repetitive strings such as names, descriptions, accession lists) and numbers with structure that bit-packing cannot use (positions sorted within groups) compress better with --compact, Rust's compact preset (with_compact): Zstandard competes for strings and binary, and Pco for integers and floats, next to the default encodings, each winning a chunk only where it makes it smaller:

java -jar cli/target/vortex-cli-*-all.jar import --compact data.csv

On the ClinVar variant summary (4.6M rows, 43 columns, mostly free text) the default import wrote 392 MB and --compact 251 MB, in 16 s instead of 13 s. Both encodings are slower to write and to decode than the structural ones, so scans of those columns are slower. It is off by default, as in Rust. In Java, WriteOptions#withCompact does the same, and WriteOptions#withColumnEncoding forces the encodings of one column.

Import from a JDBC source

API:

import io.github.dfa1.vortex.jdbc.JdbcImporter;

import java.sql.Connection;
import java.sql.DriverManager;

try (Connection conn = DriverManager.getConnection("jdbc:postgresql://localhost/mydb", "user", "pw")) {
    JdbcImporter.importTable(conn, "trades", Path.of("trades.vortex"));
}

An arbitrary query instead of a whole table:

try (Connection conn = DriverManager.getConnection("jdbc:postgresql://localhost/mydb", "user", "pw")) {
    JdbcImporter.importQuery(conn, "SELECT * FROM trades WHERE volume > 1000000", Path.of("trades.vortex"));
}

Tune the driver fetch size, Vortex chunk size, write options, and progress reporting:

import io.github.dfa1.vortex.jdbc.JdbcImportOptions;

JdbcImportOptions opts = JdbcImportOptions.defaults()
    .withFetchSize(50_000)
    .withProgressListener((rowsDone, rowsTotal) -> System.out.println("imported " + rowsDone + " rows"));

try (Connection conn = DriverManager.getConnection("jdbc:postgresql://localhost/mydb", "user", "pw")) {
    JdbcImporter.importQuery(conn, "SELECT * FROM trades", Path.of("trades.vortex"), opts);
}

The schema is derived entirely from ResultSetMetaData — no separate schema step needed. Columns mapping to vortex.date/vortex.time/vortex.timestamp/vortex.uuid round-trip through the matching JDBC getter; other SQL types map to the closest DType.Primitive/Utf8.

Start the CLI faster (JDK 25 AOT cache)

A short CLI command spends most of its time warming up the JVM: loading classes, then running code that the JIT has not compiled yet. On JDK 25 you can record that work once in an ahead-of-time cache (JEP 483, with the one-step creation of JEP 514 and the method profiles of JEP 515) and reuse it. Build the cache by running a representative command once:

java -XX:AOTCacheOutput=vortex-cli.aot \
     -jar cli/target/vortex-cli-*-all.jar select trades.vortex ts price --where "symbol = BTCUSDT"

Then pass it to later runs:

java -XX:AOTCache=vortex-cli.aot \
     -jar cli/target/vortex-cli-*-all.jar select trades.vortex ts price --where "symbol = BTCUSDT" --timing

Measured with --timing on a 127 MB, 4.2-million-row file (five runs each, JDK 25.0.2):

Command	default	with the AOT cache
`count`	33 ms	13 ms
`select … --where` (one hour of one pair)	148 ms	73 ms

The whole process for the second command went from 0.18 s to 0.10 s. The cache is about 15 to 17 MB. Notes:

Rebuild the cache every time you rebuild the JAR. In our tests the JVM did not check the cache against the JAR it was given: after the JAR at the same path was replaced by a different build, the old cache kept supplying its own classes, and the command ran the old code, with no message. Delete the cache whenever the JAR changes, and do not ship or share one. To see where classes come from, run with -Xlog:class+load and look for source: shared objects file against a file: path.
A missing cache file is not fatal. The JVM logs an error and runs normally.
It only helps short runs. A long import or export spends its time in hot, compiled code, so the cache gains little there. We did not measure that.
The training run matters. The cache holds what the training run used, so train it on the commands you run. Our numbers come from a cache trained on the very command we then timed, which is the best case.
-XX:TieredStopAtLevel=1 (C1 only) takes the query down further, to about 64 ms, but gives up the optimizing compiler, so use it only for short commands.

Query a Vortex file with SQL (Calcite)

API:

import io.github.dfa1.vortex.calcite.VortexCalcite;

import java.sql.ResultSet;
import java.sql.Statement;

try (Connection conn = VortexCalcite.connect("vtx", Map.of("ohlc", Path.of("ohlc.vortex")));
     Statement st = conn.createStatement();
     ResultSet rs = st.executeQuery("select symbol, avg(price) from vtx.ohlc group by symbol")) {
    while (rs.next()) {
        System.out.println(rs.getString(1) + ": " + rs.getDouble(2));
    }
}

Map<String, Path> tables = Map.of(
    "ohlc", Path.of("ohlc.vortex"),
    "trades", Path.of("trades.vortex")
);
try (Connection conn = VortexCalcite.connect("vtx", tables)) {
    // select * from vtx.ohlc join vtx.trades on ...
}

Or point at a directory — every *.vortex file in it becomes a table named after the file, and the columns come from each file's own schema:

try (Connection conn = VortexCalcite.connect(Path.of("/data/vortex"))) {
    // select * from vtx.ohlc   (ohlc.vortex)
}

Whole-table min/max/sum aggregates are answered straight from zone-map statistics — no data segment is decoded — on files with the legacy vortex.stats zone map (written with WriteOptions.defaults().withEdition(Editions.CORE_2025_10_0)). Files with the default vortex.zoned zone map answer them through a scan. Column names that collide with SQL reserved words (close, open, value, year, …) work unquoted; the handful that open a typed literal (date, time, timestamp, interval) still need double quotes: select "date" from vtx.ohlc. See reference.md#calcite-sql-adapter for the full lexical/parser policy.

Query Vortex files from an SQL tool (IntelliJ, DBeaver)

vortex-jdbc-driver is a JDBC driver: one self-contained jar (Calcite and the Vortex modules inside), so a tool that only takes a driver jar and a URL can browse and query .vortex files. It is an embedded engine like DuckDB or SQLite: the query runs inside the tool's JVM, which must be JDK 25 or later. It is read-only.

Set it up in IntelliJ

Get the jar. Download vortex-jdbc-driver-<version>-all.jar from Maven Central, or build it: ./mvnw package -pl jdbc-driver -am -DskipTests leaves it in jdbc-driver/target/. Use the -all jar, not the plain one.
Add the driver. Open the Database tool window, then + → Driver. Name it Vortex. Under Driver Files click + → Custom JARs and pick the jar. Set Class to io.github.dfa1.vortex.jdbc.driver.VortexDriver (it appears in the dropdown once the jar is added). Apply.
Add a data source. + → Data Source → Vortex. Set the URL (see below), leave user and password empty, and press Test Connection.
Browse. In the schema list tick vtx. Each .vortex file is a table, with its columns and types. Double-click a table for the data grid, or open a query console on the data source and run SQL.

DBeaver is the same: Database → Driver Manager → New with the same jar and class, then a connection with the URL.

The URL

jdbc:vortex:/data/vortex
jdbc:vortex:/data/vortex?schema=lake&table.trades=/other/trades.vortex

Every *.vortex file in the directory becomes a table named after the file (ohlc.vortex is vtx.ohlc); columns come from the file, so there is nothing to declare. schema renames vtx, and table.<name>=<path> adds a file from elsewhere or overrides a discovered one. Any other key is an error, so a typo cannot silently expose no tables. Values are URL-decoded.

The directory is scanned when the connection opens, so a file added later needs a reconnect (and a Synchronize in the Database tool window).

Writing queries

select symbol, count(*), max(high) from vtx.ohlc group by symbol order by 2 desc;
select close, open from vtx.ohlc where "date" > 20000 limit 100;

Quote with double quotes, not back-ticks. Identifiers are case-sensitive, and `x` is not an identifier. date, time, timestamp and interval still need double quotes even as column names; other reserved words such as close, open and value work unquoted. IntelliJ and DBeaver generate "name" themselves, so opening a table or sorting its grid works as is.
Filters and LIMIT are cheap. A WHERE on a clustered column skips whole chunks using the files' zone maps, and the data grid only reads the page it shows.
Low-cardinality string columns filter fastest. A WHERE symbol = 'AAPL' (or IN, <>, a range) on a dictionary-encoded column is decided once per distinct value, then only the matching rows are read; a value missing from the dictionary returns at once, without reading the rows.
ORDER BY ... LIMIT does not sort the table. The first rows are kept while scanning, so a top-10 over millions of rows costs about one pass over the sort column.
A sort or grouping that returns thousands of values is handed to Calcite's own engine instead of being answered from the file's statistics. The result is the same, only slower. A huge result in the grid is still paged.

Troubleshooting

Symptom	Cause
`class file has wrong version` or `UnsupportedClassVersionError`	The JDBC process runs on a JDK older than 25. Set the project or IDE SDK to 25+.
`invalid Vortex URL ... unknown key`	A misspelled option; the message lists the keys that exist.
`Object 'OHLC' not found ... did you mean 'ohlc'?`	Names are case-sensitive; write `ohlc` as the file is named.
Lexical error ... Encountered: "`"	A back-ticked identifier, accepted before 0.19; use double quotes.

On JDK 25 the first query prints a sun.misc.Unsafe warning from protobuf, a Calcite dependency; it is harmless.

Query a Vortex file with DuckDB

DuckDB reads Vortex files through its vortex extension, which is built from the Rust reference implementation (the vortex-duckdb crate). Files written by vortex-java are read back by it, so you can use it for ad-hoc SQL without writing any Java:

INSTALL vortex;
LOAD vortex;

SELECT open_time, close
FROM read_vortex('trades.vortex')
WHERE symbol = 'BTCUSDT'
  AND open_time >= 1718452800000 AND open_time < 1718456400000
ORDER BY open_time;

String literals are quoted here, unlike in the vortex CLI's --where. As a check of the round trip, hash every value of every column in the original CSV and in the Vortex file and compare the sums (this was done with DuckDB 1.5.6 on a 4,216,320-row, 13-column file):

SELECT count(*), sum(hash(symbol)::HUGEINT), sum(hash(close)::HUGEINT) /* ... one per column */
FROM read_csv('data.csv', header = true);          -- and the same over read_vortex('data.vortex')

Export to CSV

CLI:

# all columns
java -jar cli/target/vortex-cli-*-all.jar export data.vortex > out.csv

# specific columns
java -jar cli/target/vortex-cli-*-all.jar select data.vortex col1 col2 > out.csv

# filtered rows
java -jar cli/target/vortex-cli-*-all.jar filter data.vortex "price >= 100" > out.csv

Write and read a Map column

DType.Map has no dedicated write-side value type. Physically, a map column's vortex.map node has exactly one child — entries, a ListView<Struct{key, value}> — so you hand writeChunk the same shape you'd hand a plain ListView<Struct> column: a ListViewData whose elements is a StructData of a keys array and a values array. See explanation.md#map-column-layout for why the wire format looks like this and how the two independent nullability slots (map row vs. entry value) work.

Write:

import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.writer.encode.ListViewData;
import io.github.dfa1.vortex.writer.encode.NullableData;
import io.github.dfa1.vortex.writer.encode.StructData;

// map<utf8, i64?> — non-nullable string keys, nullable long values
DType.Map mapType = new DType.Map(DType.UTF8, DType.I64.asNullable(), false, false);
DType.Struct schema = new DType.Struct(List.of(ColumnName.of("attrs")), List.of(mapType), false);

// 3 rows: {a:1, b:2}, {} (empty map), {c:null}
String[] keys = {"a", "b", "c"};
long[] values = {1L, 2L, 0L};                     // placeholder at the null entry
boolean[] valueValidity = {true, true, false};    // per-entry value validity
StructData entryStructs = new StructData(List.of(keys, new NullableData(values, valueValidity)));

int[] offsets = {0, 2, 2};   // row i's entries start at entryStructs[offsets[i]]
int[] sizes = {2, 0, 1};     // row i has sizes[i] entries
ListViewData column = new ListViewData(entryStructs, offsets, sizes, 3);

try (var ch = FileChannel.open(Path.of("attrs.vortex"), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     var writer = VortexWriter.create(ch, schema, WriteOptions.defaults())) {
    writer.writeChunk(Map.of(ColumnName.of("attrs"), column));
}

A nullable map row (as opposed to a nullable value inside a present map) wraps the whole ListViewData in NullableData instead — mapType.asNullable() in the schema, and new NullableData(column, new boolean[]{true, false, true}) in place of column above.

Read:

import io.github.dfa1.vortex.reader.array.IntArray;
import io.github.dfa1.vortex.reader.array.ListViewArray;
import io.github.dfa1.vortex.reader.array.MapArray;
import io.github.dfa1.vortex.reader.array.MaskedArray;
import io.github.dfa1.vortex.reader.array.StructArray;
import io.github.dfa1.vortex.reader.array.VarBinArray;

try (var reader = VortexReader.open(Path.of("attrs.vortex"));
     var iter = reader.scan(ScanOptions.all())) {
    while (iter.hasNext()) {
        try (var chunk = iter.next()) {
            MapArray map = chunk.column(ColumnName.of("attrs"));

            // If the map itself is nullable, entries() is a MaskedArray; unwrap it first.
            var entries = map.entries() instanceof MaskedArray masked
                    ? (ListViewArray) masked.inner() : (ListViewArray) map.entries();
            StructArray entryStructs = (StructArray) entries.elements();
            VarBinArray keys = (VarBinArray) entryStructs.field(ColumnName.of("key"));
            var values = entryStructs.field(ColumnName.of("value")); // MaskedArray, since the value type is nullable here
            // A file written by vortex-java's own writer always emits I32 offsets/sizes; a file
            // from another producer (e.g. the Rust reference) may pick a narrower or wider integer
            // width, so switch on the concrete Array subtype there instead of casting to IntArray.
            IntArray offsets = (IntArray) entries.offsets();
            IntArray sizes = (IntArray) entries.sizes();

            for (long row = 0; row < map.length(); row++) {
                long start = offsets.getInt(row);
                long end = start + sizes.getInt(row);
                for (long i = start; i < end; i++) {
                    // keys.getBytes(i) / values at index i are this row's i-th {key, value} pair
                }
            }
        }
    }
}

ScanOptions.all()/CLI inspect show vortex.map in a file's layout tree as a vortex.listview child under the vortex.map node — see docs/reference.md#core-types for DType.Map's full field list (keyType, valueType, keysSorted, nullable) and entriesDtype().

Choose a column's encodings

When a column's shape is known in advance — a fixed-precision decimal, an enum — restrict its encoder selection to the encodings that fit. The column then skips candidates that cannot win, which is most of a cascading write's cost: one million two-decimal prices write about 3.5× faster with ALP/FoR/bit-packing as the only candidates, to the same bytes.

import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.EncodingId;
import io.github.dfa1.vortex.writer.ColumnEncoding;

DType.Struct schema = new DType.Struct(List.of(ColumnName.of("price")), List.of(DType.F64), false);
double[] prices = {19.99, 4.50, 120.00};

WriteOptions options = WriteOptions.defaults().withColumnEncoding(ColumnName.of("price"),
        ColumnEncoding.candidates(EncodingId.VORTEX_ALP, EncodingId.FASTLANES_FOR,
                EncodingId.FASTLANES_BITPACKED));

try (var ch = FileChannel.open(Path.of("prices.vortex"), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     var writer = VortexWriter.create(ch, schema, options)) {
    writer.writeChunk(chunk -> chunk.put(ColumnName.of("price"), prices));
}

The listed encoders are the column's whole candidate set: ALP's integers can only be frame-of-reference- or bit-packed here, and anything no candidate improves on stays in its canonical encoding. Other columns keep the default cascade. Encodings are named by EncodingId; a custom encoding registered on the writer's WriteRegistry is named by its id the same way, and an id the writer cannot encode fails VortexWriter.create.

Register a custom encoding (write side)

EncodingId is a sealed WellKnown/Custom type — EncodingId.Custom lets a third party mint its own wire id (e.g. "acme.xor64") without touching vortex-java itself. A custom EncodingEncoder writes it; the matching EncodingDecoder, registered separately in the reader module, reads it back — writer and reader never share code, only the wire id.

Write side (encoder + registration + write, in one flow):

import io.github.dfa1.vortex.core.io.VortexFormat;
import io.github.dfa1.vortex.core.model.ColumnName;
import io.github.dfa1.vortex.core.model.DType;
import io.github.dfa1.vortex.core.model.EncodingId;
import io.github.dfa1.vortex.core.model.PType;
import io.github.dfa1.vortex.writer.encode.EncodeContext;
import io.github.dfa1.vortex.writer.encode.EncodeResult;
import io.github.dfa1.vortex.writer.encode.EncodedBuffer;
import io.github.dfa1.vortex.writer.encode.EncodingEncoder;
import io.github.dfa1.vortex.writer.WriteRegistry;

import java.lang.foreign.MemorySegment;

final class XorI64EncodingEncoder implements EncodingEncoder {
    private static final EncodingId ID = new EncodingId.Custom("acme.xor64");
    private static final long KEY = 0xA5A5_A5A5_A5A5_A5A5L;

    @Override
    public EncodingId encodingId() {
        return ID;
    }

    @Override
    public boolean accepts(DType dtype) {
        return dtype instanceof DType.Primitive p && p.ptype() == PType.I64;
    }

    @Override
    public EncodeResult encode(DType dtype, Object data, EncodeContext ctx) {
        long[] values = (long[]) data;
        MemorySegment seg = ctx.arena().allocate((long) values.length * 8, 8);
        for (int i = 0; i < values.length; i++) {
            seg.setAtIndex(VortexFormat.LE_LONG, i, values[i] ^ KEY);
        }
        // Each buffer declares its element type's alignment — 8 bytes for i64 here
        return EncodeResult.simple(ID, EncodedBuffer.of(seg, PType.I64), null, null);
    }
}

DType.Struct schema = new DType.Struct(List.of(ColumnName.of("id")), List.of(DType.I64), false);

// a registry containing only this encoder, at cascade depth 0 so every I64 column takes the
// first accepting encoder; under the default cascade it must measure smaller than the canonical
// vortex.primitive fallback to be picked
WriteRegistry registry = WriteRegistry.builder().register(new XorI64EncodingEncoder()).build();

try (var ch = FileChannel.open(Path.of("data.vortex"), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     var writer = VortexWriter.create(ch, schema, WriteOptions.cascading(0), registry)) {
    writer.writeChunk(chunk -> chunk.put(ColumnName.of("id"), new long[]{1, 2, 3}));
}

Every buffer an encoder emits is an EncodedBuffer carrying its element alignment — EncodedBuffer.of(seg, ptype) for typed values, EncodedBuffer.bytes(seg) for bitmaps, strings and opaque payloads. The Rust reader holds a buffer to exactly that alignment: less fails array construction, more aborts any row-range read that slices the buffer.

To mix a custom encoder into the normal cascade competition instead of a single-encoder registry, start from WriteRegistry.builder().registerDefaults() and override EncodingEncoder#expectedRatio(DType, ArrayAndStats, EncodeContext) to steer selection, rather than relying on accepts(DType) alone: return Estimate.SKIP, Estimate.ratio(double) (an estimate that competes without encoding) or Estimate.COMPLETE (measure on a sample). Declare the stats it reads in statsOptions(); ArrayAndStats#stats() computes them on first use and shares the scan with every other candidate.

Read side (matching decoder, registered on ReadRegistry — see reference.md#encoding-registry):

import io.github.dfa1.vortex.core.io.VortexFormat;
import io.github.dfa1.vortex.core.model.EncodingId;
import io.github.dfa1.vortex.reader.ReadRegistry;
import io.github.dfa1.vortex.reader.array.Array;
import io.github.dfa1.vortex.reader.array.MaterializedLongArray;
import io.github.dfa1.vortex.reader.decode.DecodeContext;
import io.github.dfa1.vortex.reader.decode.EncodingDecoder;

import java.lang.foreign.MemorySegment;

final class XorI64EncodingDecoder implements EncodingDecoder {
    private static final EncodingId ID = new EncodingId.Custom("acme.xor64");
    private static final long KEY = 0xA5A5_A5A5_A5A5_A5A5L;

    @Override
    public EncodingId encodingId() {
        return ID;
    }

    @Override
    public Array decode(DecodeContext ctx) {
        MemorySegment buf = ctx.buffer(0);
        long n = ctx.rowCount();
        MemorySegment out = ctx.arena().allocate(n * 8, 8);
        for (long i = 0; i < n; i++) {
            out.setAtIndex(VortexFormat.LE_LONG, i, buf.getAtIndex(VortexFormat.LE_LONG, i) ^ KEY);
        }
        return new MaterializedLongArray(ctx.dtype(), n, out);
    }
}

ReadRegistry readRegistry = ReadRegistry.builder()
    .registerDefaults()
    .register(new XorI64EncodingDecoder())
    .build();

try (VortexReader vf = VortexReader.open(Path.of("data.vortex"), readRegistry)) {
    // "id" decodes through XorI64EncodingDecoder
}

Add a custom extension type

An extension type gives a storage dtype a logical meaning, as vortex.timestamp does for I64. ExtensionId is a sealed WellKnown/Custom type like EncodingId, so a third party names its own id, writes domain values through an ExtensionEncoder and reads them back through an ExtensionDecoder — the Java side of Rust's ExtVTable. Both sides may override validate (Rust's validate_dtype), run on every dtype with that id when the writer is created and when a file is opened.

The example is a money column with one currency per column: amounts are I64 minor units (cents for EUR), and the ISO 4217 code is the extension metadata, as a timestamp keeps its zone. Never store money as a float.

Write side:

import io.github.dfa1.vortex.core.error.VortexException;
import io.github.dfa1.vortex.core.model.ExtensionId;
import io.github.dfa1.vortex.writer.ExtensionEncoder;

import java.lang.foreign.ValueLayout;
import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
import java.util.Collection;
import java.util.Currency;

final class MoneyEncoder implements ExtensionEncoder {
    static final ExtensionId ID = ExtensionId.parse("acme.money");

    static DType.Extension dtype(Currency currency) {
        byte[] code = currency.getCurrencyCode().getBytes(StandardCharsets.US_ASCII);
        return new DType.Extension(ID.id(), DType.I64, MemorySegment.ofArray(code), false);
    }

    @Override
    public ExtensionId extensionId() {
        return ID;
    }

    @Override
    public DType.Extension dtype(boolean nullable) {
        return dtype(Currency.getInstance("EUR"));
    }

    @Override
    public void validate(DType.Extension dtype) {
        if (!dtype.storageDType().equals(DType.I64)) {
            throw new VortexException("acme.money storage must be non-nullable I64");
        }
        currency(dtype); // rejects a code that is not ISO 4217
    }

    @Override
    public Object encodeAll(DType.Extension dtype, Collection<?> values) {
        int digits = currency(dtype).getDefaultFractionDigits();
        long[] minorUnits = new long[values.size()];
        int i = 0;
        for (Object value : values) {
            // throws on an amount finer than the currency's minor unit (e.g. 0.001 EUR)
            minorUnits[i++] = ((BigDecimal) value).movePointRight(digits).longValueExact();
        }
        return minorUnits;
    }

    static Currency currency(DType.Extension dtype) {
        return Currency.getInstance(new String(dtype.metadata().toArray(ValueLayout.JAVA_BYTE),
                StandardCharsets.US_ASCII));
    }
}

DType.Struct schema = new DType.Struct(List.of(ColumnName.of("price")),
        List.of(MoneyEncoder.dtype(Currency.getInstance("EUR"))), false);
WriteRegistry registry = WriteRegistry.builder().registerDefaults().register(new MoneyEncoder()).build();

try (var ch = FileChannel.open(Path.of("prices.vortex"), StandardOpenOption.CREATE, StandardOpenOption.WRITE);
     var writer = VortexWriter.create(ch, schema, WriteOptions.defaults(), registry)) {
    writer.writeChunk(chunk -> chunk.put(ColumnName.of("price"),
            List.of(new BigDecimal("9.99"), new BigDecimal("120.50"))));
}

Read side:

import io.github.dfa1.vortex.reader.Chunk;
import io.github.dfa1.vortex.reader.array.LongArray;
import io.github.dfa1.vortex.reader.extension.ExtensionDecoder;

import java.util.ArrayList;

final class MoneyDecoder implements ExtensionDecoder<BigDecimal> {
    static final ExtensionId ID = ExtensionId.parse("acme.money");

    @Override
    public ExtensionId extensionId() {
        return ID;
    }

    @Override
    public DType.Extension dtype(boolean nullable) {
        byte[] code = "EUR".getBytes(StandardCharsets.US_ASCII);
        return new DType.Extension(ID.id(), DType.I64, MemorySegment.ofArray(code), false);
    }

    @Override
    public Class<BigDecimal> domainType() {
        return BigDecimal.class;
    }

    @Override
    public List<BigDecimal> decodeAll(DType.Extension dtype, Array storage) {
        Currency currency = Currency.getInstance(new String(dtype.metadata().toArray(ValueLayout.JAVA_BYTE),
                StandardCharsets.US_ASCII));
        LongArray minorUnits = (LongArray) storage;
        List<BigDecimal> out = new ArrayList<>();
        for (long i = 0; i < minorUnits.length(); i++) {
            out.add(BigDecimal.valueOf(minorUnits.getLong(i), currency.getDefaultFractionDigits()));
        }
        return out;
    }
}

ReadRegistry readRegistry = ReadRegistry.builder().registerDefaults().register(new MoneyDecoder()).build();

try (VortexReader vf = VortexReader.open(Path.of("prices.vortex"), readRegistry);
     var iter = vf.scan(ScanOptions.all())) {
    while (iter.hasNext()) {
        try (Chunk chunk = iter.next()) {
            List<BigDecimal> prices = chunk.as(ColumnName.of("price"), BigDecimal.class); // 9.99, 120.50
        }
    }
}

validate checks the dtype, not the values: per-value rules belong in encodeAll. Filters and zone maps see the stored I64. A reader without the decoder — including the Rust reference — refuses the file, as Rust does for an extension id its session has no vtable for; with ReadRegistry.Builder#allowUnknown() the column reads as its storage array. Writing a Collection for an extension column needs a registered encoder; the storage array itself (long[] here) can always be written. Auto-routing a Collection applies to top-level columns only.

Read files with unknown encodings

By default, a file containing an unrecognized encoding ID throws VortexException. Use allowUnknown() to read the file anyway — columns with unknown encodings are returned as UnknownArray (opaque, not decodable, but the rest of the file is readable):

import io.github.dfa1.vortex.reader.ReadRegistry;
import io.github.dfa1.vortex.reader.array.UnknownArray;

ReadRegistry registry = ReadRegistry.builder()
        .registerDefaults()
        .allowUnknown()
        .build();

try (VortexReader vf = VortexReader.open(Path.of("future.vortex"), registry);
     var iter = vf.scan(ScanOptions.all())) {
    while (iter.hasNext()) {
        try (var chunk = iter.next()) {
            chunk.columns().forEach((name, column) -> {
                if (column.array() instanceof UnknownArray u) {
                    System.out.println(name + ": unknown encoding " + u.encodingId());
                }
            });
        }
    }
}

Navigation Menu

Repository navigation

how-to.md

how-to.md

How-to guides

Build the CLI

Count rows

Inspect file structure

Explore in the browser

Inspect as a web page

Inspect interactively (TUI)

Project columns

Filter rows

Time a CLI command

Preview the first N rows

Convert Parquet to Vortex

Convert Vortex to Parquet

Convert CSV to Vortex

Import from a JDBC source

Start the CLI faster (JDK 25 AOT cache)

Query a Vortex file with SQL (Calcite)

Query Vortex files from an SQL tool (IntelliJ, DBeaver)

Set it up in IntelliJ

The URL

Writing queries

Troubleshooting

Query a Vortex file with DuckDB

Export to CSV

Write and read a Map column

Choose a column's encodings

Register a custom encoding (write side)

Add a custom extension type

Read files with unknown encodings

FilesExpand file tree

how-to.md

Latest commit

History

how-to.md

File metadata and controls

How-to guides

Build the CLI

Count rows

Inspect file structure

Explore in the browser

Inspect as a web page

Inspect interactively (TUI)

Project columns

Filter rows

Time a CLI command

Preview the first N rows

Convert Parquet to Vortex

Convert Vortex to Parquet

Convert CSV to Vortex

Import from a JDBC source

Start the CLI faster (JDK 25 AOT cache)

Query a Vortex file with SQL (Calcite)

Query Vortex files from an SQL tool (IntelliJ, DBeaver)

Set it up in IntelliJ

The URL

Writing queries

Troubleshooting

Query a Vortex file with DuckDB

Export to CSV

Write and read a Map column

Choose a column's encodings

Register a custom encoding (write side)

Add a custom extension type

Read files with unknown encodings