From 92c9cdcb025967f32d65193fcf371fd56779fa7f Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Tue, 20 Oct 2026 18:42:34 +0200 Subject: [PATCH] docs: prepare the 0.18.0 changelog and document the JDBC driver - CHANGELOG: #581 merged after the 0.18.0 section was cut but ships in it, so its three entries move from Unreleased into 0.18.0; the release is dated 2026-10-20; the jdbc:vortex: driver, the release's headline, leads Added. - README: a short section on querying .vortex files from IntelliJ or DBeaver with vortex-jdbc-driver, linking to the how-to. - how-to: the driver's query tips cover this release's dictionary filters and ORDER BY ... LIMIT. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 16 ++++++---------- README.md | 22 ++++++++++++++++++++++ docs/how-to.md | 5 +++++ 3 files changed, 33 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 046e1f756..0d22d3299 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,21 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Changed -- **Breaking:** `ArrayStats` has one component per field of Rust's array stats, so `minExact`, `maxExact`, `isSorted`, `isStrictSorted`, `isConstant`, `uncompressedSizeInBytes` and `nanCount` are read from the file and `trueCount` is gone (a bool column's true count is its `sum`). ([#581](https://github.com/dfa1/vortex-java/pull/581)) - -### Fixed -- Files written by vortex-java mark their array min/max as exact, which Rust read as inexact bounds. ([#581](https://github.com/dfa1/vortex-java/pull/581)) -- `VortexReader.columnStats()` and SQL `MIN`/`MAX` no longer report a truncated (inexact) bound as a column's minimum or maximum. ([#581](https://github.com/dfa1/vortex-java/pull/581)) - -## [0.18.0] — 2026-10-19 +## [0.18.0] — 2026-10-20 ### Added -- `Compute.matcher` returns a `RowSelection`, whose `nextSelected(long)` jumps to the next selected row instead of testing each one. ([#580](https://github.com/dfa1/vortex-java/pull/580)) -- `Compute.matcher(RowFilter)` gives the row test of each chunk of one scan, evaluating a dictionary the chunks share once. ([#576](https://github.com/dfa1/vortex-java/pull/576)) - `vortex-jdbc-driver` is a `jdbc:vortex:` JDBC driver in one self-contained jar, so IntelliJ, DBeaver and other SQL tools can query a directory of `.vortex` files. ([#566](https://github.com/dfa1/vortex-java/issues/566)) - `VortexSchemaFactory` builds the Calcite schema from a `jdbc:calcite:model=...` URL, so SQL tools can query a directory of `.vortex` files without Java code. ([#566](https://github.com/dfa1/vortex-java/issues/566)) - `VortexCalcite.connect(Path)` and `VortexCalciteOptions` expose every `.vortex` file in a directory as a SQL table. ([#566](https://github.com/dfa1/vortex-java/issues/566)) +- `Compute.matcher` returns a `RowSelection`, whose `nextSelected(long)` jumps to the next selected row instead of testing each one. ([#580](https://github.com/dfa1/vortex-java/pull/580)) +- `Compute.matcher(RowFilter)` gives the row test of each chunk of one scan, evaluating a dictionary the chunks share once. ([#576](https://github.com/dfa1/vortex-java/pull/576)) - `DecimalArray.getUnscaledLong(long)` reads a decimal's unscaled value without building a `BigDecimal`. ([#563](https://github.com/dfa1/vortex-java/pull/563)) - `Compute.groupedAggregate(Chunk, ColumnName, RowFilter, ColumnName)` folds an aggregate per value of a dictionary-encoded string, boolean or integer column. ([#555](https://github.com/dfa1/vortex-java/pull/555)) - `Chunk.slice(offset, length)` gives a zero-copy view of a row range. ([#550](https://github.com/dfa1/vortex-java/pull/550)) @@ -32,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Custom extension types: register an `ExtensionDecoder` on `ReadRegistry` and an `ExtensionEncoder` on `WriteRegistry`, with a `validate` hook on both sides, and `Chunk.as` decodes any registered type. ([#530](https://github.com/dfa1/vortex-java/issues/530)) ### Changed +- **Breaking:** `ArrayStats` has one component per field of Rust's array stats, so `minExact`, `maxExact`, `isSorted`, `isStrictSorted`, `isConstant`, `uncompressedSizeInBytes` and `nanCount` are read from the file and `trueCount` is gone (a bool column's true count is its `sum`). ([#581](https://github.com/dfa1/vortex-java/pull/581)) - SQL `WHERE` on a dictionary-encoded string column visits only the matching rows, up to a third faster for a selective `=` or `IN`. ([#580](https://github.com/dfa1/vortex-java/pull/580)) - SQL `WHERE col = 'x'` on a dictionary-encoded string column is about twice as fast, and skips the column's rows entirely when `x` is not in the dictionary. ([#576](https://github.com/dfa1/vortex-java/pull/576)) - SQL `SELECT DISTINCT` of one dictionary-encoded string column with `ORDER BY ... LIMIT` is about a third faster: it ranks the dictionary values the rows use instead of every row. ([#574](https://github.com/dfa1/vortex-java/pull/574)) @@ -65,6 +59,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Breaking:** `ExtensionId` is a sealed `WellKnown`/`Custom` type like `EncodingId`, so third-party extension ids can be named, and `ExtensionId.parse` returns the id instead of an `Optional`. ([#530](https://github.com/dfa1/vortex-java/issues/530)) ### Fixed +- Files written by vortex-java mark their array min/max as exact, which Rust read as inexact bounds. ([#581](https://github.com/dfa1/vortex-java/pull/581)) +- `VortexReader.columnStats()` and SQL `MIN`/`MAX` no longer report a truncated (inexact) bound as a column's minimum or maximum. ([#581](https://github.com/dfa1/vortex-java/pull/581)) - A negative code in a nullable dictionary-encoded column fails with `VortexException` instead of a raw `IndexOutOfBoundsException`. ([#576](https://github.com/dfa1/vortex-java/pull/576)) - The CLI's terminal UI runs on JDK 28, whose arenas no longer over-align `allocate(size)`. ([8858286](https://github.com/dfa1/vortex-java/commit/8858286a1addb5ffca289c42e6106c7395564a57)) - vortex-jni no longer aborts the JVM when it filters a string column vortex-java compressed with FSST, whose symbol table could hold two symbols Rust hashes to one slot, and FSST columns are about 5% smaller. ([#573](https://github.com/dfa1/vortex-java/pull/573)) diff --git a/README.md b/README.md index 0d686eb82..cf92b776e 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ and against 172 real-world [Raincloud](docs/compatibility.md#real-world-conforma - JVM analytics engines and OLAP systems - Anyone who wants mmap-backed, zero-copy columnar reads without native-library management +- Anyone who wants to query `.vortex` files with SQL from IntelliJ, DBeaver or another JDBC tool - **Small footprint.** The Rust reference's JNI bindings (`vortex-jni`) are a 75 MB jar (native binaries for four platforms) that pulls in Arrow, Netty, Guava and RoaringBitmap. vortex-java's reader is ~800 KB of plain Java (`vortex-reader` + `vortex-core` + @@ -90,6 +91,27 @@ try (var ch = FileChannel.open(Path.of("data/example.vortex"), > `IllegalArgumentException` immediately. Missing columns surface as > `IllegalStateException` when the lambda returns. +### Query with SQL from IntelliJ or DBeaver + +`vortex-jdbc-driver` is a read-only JDBC driver in one self-contained jar, so SQL tools can browse and +query `.vortex` files with no code. Download `vortex-jdbc-driver-0.18.0-all.jar` from +[Maven Central](https://central.sonatype.com/artifact/io.github.dfa1.vortex/vortex-jdbc-driver), add +it as a driver with class `io.github.dfa1.vortex.jdbc.driver.VortexDriver`, and connect to a directory: + +``` +jdbc:vortex:/data/vortex +``` + +Every `*.vortex` file in it is a table of schema `vtx`, named after the file: + +```sql +select symbol, count(*), max(high) from vtx.ohlc group by symbol order by 2 desc; +``` + +The tool must run on JDK 25 or later. Setup steps for IntelliJ and DBeaver, URL options and +troubleshooting are in the +[how-to](docs/how-to.md#query-vortex-files-from-an-sql-tool-intellij-dbeaver). + For more examples — projection, filtering, custom encodings, and the CLI — see the [tutorial](docs/tutorial.md). diff --git a/docs/how-to.md b/docs/how-to.md index 74b8e7ea3..d509df861 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -643,6 +643,11 @@ select close, open from vtx.ohlc where `date` > 20000 limit 100; other reserved words such as `close`, `open` and `value` work unquoted. - **Filters and `LIMIT` are cheap.** A `WHERE` on a clustered column skips whole chunks using the files' zone maps, and the data grid only reads the page it shows. +- **Low-cardinality string columns filter fastest.** A `WHERE symbol = 'AAPL'` (or `IN`, `<>`, a + range) on a dictionary-encoded column is decided once per distinct value, then only the matching + rows are read; a value missing from the dictionary returns at once, without reading the rows. +- **`ORDER BY ... LIMIT` does not sort the table.** The first rows are kept while scanning, so a + top-10 over millions of rows costs about one pass over the sort column. - **A sort or grouping that returns thousands of values is handed to Calcite's own engine** instead of being answered from the file's statistics. The result is the same, only slower. A huge result in the grid is still paged.