diff --git a/README.md b/README.md index c8e9f22..65e6580 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin ## What works today -`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time, and `load` for building a whole database out of columns and an edge list. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. Both module formats, typed separately. +`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time, and `load` for building a whole database out of columns and an edge list. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. `columnar`, for a result read down its columns as the buffers themselves rather than across its rows as objects. Both module formats, typed separately. Build it with `npm run build`, and run the suite with `npm test`. Nothing is published yet, so `npm i zudb` is not a thing you can type at anybody's terminal, but everything it will do is built and installed on every run of the release workflow. @@ -186,6 +186,60 @@ The frame belongs to the connection it was registered on and goes when that conn Registering the same name again replaces what it stands for, columns and all. Registering over a table the database already holds is refused, since a statement naming it would mean the stored one. A frame with no rows is a table to match on and answers nothing, because a frame knows its columns without being told by a row. A null anywhere is refused by column and row, since a property that is null is one no row of this engine holds, and registering inside a transaction is refused because a frame is registered on the session, which is the thing the transaction is running on. +## Reading a result as columns + +`query` builds an object a row and a JavaScript value a cell, which is what a program reading a hundred rows wants and the wrong shape for a million. `columnar` runs the same statement and hands back the buffers instead: + +```ts +const read = await conn.columnar(`MATCH (p:person) RETURN p.age AS age`); +read.rows; // 1000000 +read.columns[0].values; // a BigInt64Array of every age, and not one object +``` + +The buffers are the engine's own, moved rather than read: the pointer V8 is given is the pointer the engine filled, and the allocation is freed when the typed array is collected. So a column of a million integers crosses the boundary as a pointer and a length. On this machine, with `npm run bench:columnar` over a million rows: + +``` +one integer column, columnar 38.4 ms 38 ns/row +one integer column, rows 243.1 ms 243 ns/row +a string column, columnar 50.3 ms 50 ns/row +a string column, rows 262.0 ms 262 ns/row +three columns, columnar 75.8 ms 76 ns/row +three columns, rows 632.2 ms 632 ns/row +``` + +Walking what came back costs the same either way, at about 14 ns a row for a sum over the buffer and the same over the rows, which is worth saying because it is where the win is not. V8 reads a property of a small object about as fast as an element of a typed array. What it cannot do is make a million of those objects for nothing, and that is the whole of the six to eight times above. + +Every column says what it is, and reading one is a switch on `type` rather than a series of tests for what is there. `values` carries everything of a fixed width: a `BigInt64Array` of integers, nanoseconds or months, a `Float64Array` of floats, an `Int32Array` of days, and for booleans a `Uint8Array` of one bit a row, least significant bit first. A string column has `data`, the bytes of every string end to end, and `offsets`, one more than there are rows, so row `i` is `data.subarray(offsets[i], offsets[i + 1])`. `validity` is one bit a row again, set meaning the row has a value, and it is null when nothing in the column is, so the common case costs a reader nothing to skip. `unit` says whether a cell counts days, nanoseconds or months, and `zone` is the minutes east of UTC a column of zoned times was written with. + +That layout is Arrow's, which is the point of it. `apache-arrow` wraps a buffer of this shape without copying it, so a table is eleven lines and no dependency of this package: + +```ts +// with apache-arrow installed, and nothing in zudb importing it +const table = new Table( + Object.fromEntries( + read.columns.map((column) => [ + column.name, + new Vector([ + makeData({ + type: arrow(column), // Int64, Float64, Bool, Utf8, DateDay, TimestampNanosecond + length: column.length, + nullCount: column.nulls, + nullBitmap: column.validity ?? undefined, + data: column.data ?? column.values, + valueOffsets: column.offsets ?? undefined, + }), + ]), + ]), + ), +); +``` + +The same memory is on both sides of that: `table.getChild("age").data[0].values` is the array the engine filled, not a copy of it. The recipe is printed here rather than shipped because a client that hands out an Arrow object has to agree with one version of Arrow forever, and a client that hands out the bytes agrees with all of them. It is run in the test suite, so it is checked rather than believed. + +Two things are not buffers, and both are named by the type rather than found out by looking. A column of nodes, rels, paths, lists or records has no fixed width cell, so it arrives as `items`, holding the same JavaScript values `query` would have made. A column of nothing but nulls has a length and nothing else, because there is nothing to put in a buffer. A column that mixes two types is refused, naming the column and the row that did it, since a columnar result holds one type per column and a column that quietly became strings is worse than one that would not build. + +`bigIntMode` says nothing here. A columnar read has one physical layout per type and an INT64 column is 64 bit cells however a caller would rather read one, which is the difference between a buffer and a value. The mode still decides what is inside `items`, where this client is making objects anyway. + ## Asking for numbers instead of bigints `bigIntMode` says how INT64 is spelled on the way out. It goes on one statement, or on a connection for all of them, and a statement on a connection that named one may still name the other: @@ -278,7 +332,7 @@ typedoc rather than api-documenter, which would have been the obvious pick since Anything outside that table has no binary and no source build to fall back on, so the install resolves nothing and the first `require` says so. The browser and the platforms nobody builds for are what the WASM target answers, later. -`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the appender, `npm run bench:load` for building a database out of columns, and `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows. +`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the appender, `npm run bench:load` for building a database out of columns, `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows, and `npm run bench:columnar` for a result read down its columns against the same result read across its rows. ## Still to come diff --git a/bench/columnar.mjs b/bench/columnar.mjs new file mode 100644 index 0000000..468b642 --- /dev/null +++ b/bench/columnar.mjs @@ -0,0 +1,126 @@ +// What a result costs read down its columns against read across its +// rows. +// +// The two calls run the same statement and differ only in what they +// build out of the answer: `query` makes an object a row and a value a +// cell, and `columnar` moves one buffer a column. So the gap between +// the two lines of a pair is the cost of making JavaScript values, which +// is what this is measuring and the only reason the second call exists. +// +// The last block is what a caller does next. A sum over a typed array +// against a sum over an array of objects is the honest comparison, +// because a program that asked for a million rows is going to walk them, +// and the buffer is quicker to walk as well as quicker to make. +// +// Run it against a release build, for the reason bench/query.mjs gives. +// +// npm run build && npm run bench:columnar + +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { connect } from 'zudb' + +const ROWS = Number(process.env.ZU_BENCH_ROWS ?? 1_000_000) +const REPEATS = Number(process.env.ZU_BENCH_REPEATS ?? 5) + +const dir = await mkdtemp(join(tmpdir(), 'zu-bench-columnar-')) +const conn = await connect(join(dir, 'bench.zu1')) + +await conn.exec("INSERT (p:person {uid: 1, score: 1.5, name: 'n1'})") +{ + const rows = await conn.appender('person') + for (let ix = 2; ix <= ROWS; ix++) rows.appendRow([BigInt(ix), ix / 3, `n${ix}`]) + await rows.close() +} + +/// The fastest of `REPEATS` runs, in milliseconds, after one warmup. +/// +/// The fastest for the reason bench/query.mjs gives: everything that +/// makes a run slower than the work itself is something that happened to +/// it rather than something about it. +async function time(run) { + await run() + let best = Infinity + for (let round = 0; round < REPEATS; round++) { + const started = performance.now() + await run() + best = Math.min(best, performance.now() - started) + } + return best +} + +function report(name, ms) { + const each = (ms * 1e6) / ROWS + console.log( + `${name.padEnd(30)} ${ms.toFixed(1).padStart(8)} ms ${Math.round(each).toString().padStart(6)} ns/row`, + ) +} + +const cases = [ + { + name: 'one integer column, columnar', + run: () => conn.columnar('MATCH (p:person) RETURN p.uid AS uid'), + }, + { + name: 'one integer column, rows', + run: () => conn.query('MATCH (p:person) RETURN p.uid AS uid'), + }, + { + name: 'a float column, columnar', + run: () => conn.columnar('MATCH (p:person) RETURN p.score AS score'), + }, + { + name: 'a float column, rows', + run: () => conn.query('MATCH (p:person) RETURN p.score AS score'), + }, + { + name: 'a string column, columnar', + run: () => conn.columnar('MATCH (p:person) RETURN p.name AS name'), + }, + { + name: 'a string column, rows', + run: () => conn.query('MATCH (p:person) RETURN p.name AS name'), + }, + { + name: 'three columns, columnar', + run: () => + conn.columnar('MATCH (p:person) RETURN p.uid AS uid, p.score AS score, p.name AS name'), + }, + { + name: 'three columns, rows', + run: () => conn.query('MATCH (p:person) RETURN p.uid AS uid, p.score AS score, p.name AS name'), + }, +] + +console.log(`reading ${ROWS} rows, fastest of ${REPEATS}`) +for (const { name, run } of cases) report(name, await time(run)) + +// What the caller does with what they were handed. The statement is not +// timed here: both sides already have the whole answer and the question +// is what walking it costs. +const read = await conn.columnar('MATCH (p:person) RETURN p.uid AS uid') +const rows = await conn.query('MATCH (p:person) RETURN p.uid AS uid') + +console.log('') +console.log('summing what came back') +report( + 'over the buffer', + await time(async () => { + let total = 0n + for (const value of read.columns[0].values) total += value + return total + }), +) +report( + 'over the rows', + await time(async () => { + let total = 0n + for (const row of rows) total += row.uid + return total + }), +) + +await conn.close() +await rm(dir, { recursive: true, force: true }) diff --git a/binding.d.cts b/binding.d.cts index 6e6da73..f09f4f2 100644 --- a/binding.d.cts +++ b/binding.d.cts @@ -290,6 +290,93 @@ export interface ZuNotice { readonly docUrl: string } +/** + * What a column of a columnar read turned out to hold. + * + * Narrower than the type the statement declared, because the question + * here is which buffer arrived: a time with an offset and a time + * without are the same 64 bit cells, and the offset rides beside as + * `zone`. `value` is the fallback for what no fixed width cell covers, + * which is nodes, rels, paths, lists and records, and `null` is a + * column that held nothing else. + */ +export type ZuColumnType = + | 'null' + | 'bool' + | 'int' + | 'float' + | 'string' + | 'date' + | 'time' + | 'datetime' + | 'duration' + | 'value' + +/** + * One column of a result, as the buffer holding it. + * + * Every field is present on every column and holds null where it does + * not apply, so reading one is a switch on `type` rather than a series + * of tests for what is there. Which field carries the values follows + * from the type: `values` for everything of a fixed width, `data` and + * `offsets` for strings, `items` for what no buffer covers, and none of + * them for a column of nulls. + * + * The buffers are the engine's own, handed over rather than copied, and + * they are laid out the way Arrow lays them out: values end to end, a + * boolean as one bit a row, a string column as its bytes and `length + + * 1` offsets into them, where row `i` spans `offsets[i]` to `offsets[i + * + 1]`. + */ +export interface ZuColumn { + readonly name: string + readonly type: ZuColumnType + readonly length: number + /** + * The cells, for a column of a fixed width: `BigInt64Array` for + * integers, nanoseconds and months, `Float64Array` for floats, + * `Int32Array` for days, and a `Uint8Array` of packed bits for + * booleans, least significant bit first. + */ + readonly values: BigInt64Array | Float64Array | Int32Array | Uint8Array | null + /** The bytes of every string end to end, for a string column. */ + readonly data: Uint8Array | null + /** + * `length + 1` offsets into `data`, for a string column. Narrow until + * the bytes pass what a 32 bit offset addresses, which is the + * difference Arrow calls Utf8 against LargeUtf8. + */ + readonly offsets: Int32Array | BigInt64Array | null + /** The values themselves, for a column of type `value`. */ + readonly items: ZuValue[] | null + /** + * One bit a row, least significant bit first, set meaning the row has + * a value. Null when every row has one, which is the common case and + * the one where a reader gets to skip the test. + */ + readonly validity: Uint8Array | null + /** How many rows are null, which is zero when `validity` is null. */ + readonly nulls: number + /** What one cell counts: `days`, `nanos` or `months`. */ + readonly unit: 'days' | 'nanos' | 'months' | null + /** Minutes east of UTC, for a column of zoned times or datetimes. */ + readonly zone: number | null +} + +/** + * A whole result read down its columns. + * + * `rows` is every column's length, and is the answer for a statement + * that projected nothing at all. `gqlstatus` and `notices` are the + * statement's, exactly as they are on the rows. + */ +export interface ZuColumnar { + readonly rows: number + readonly columns: ZuColumn[] + readonly gqlstatus: string + readonly notices: ZuNotice[] +} + /** * The rows a statement gave back. * @@ -707,6 +794,18 @@ export declare class Connection { * reads still costs a row object per row on the way out. */ exec(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise + /** + * Runs one statement and gives back its columns rather than its + * rows. + * + * The same statement as [`Connection::query`], read down instead + * of across: what comes back is one buffer a column, in the layout + * Arrow already uses, and no object a row. That is the way out for + * anything that is going to be counted, plotted or handed to a + * dataframe, and it is the way out that does not build a million + * JavaScript values on the way. + */ + columnar(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise /** * Runs one statement and gives back a cursor over its rows. * diff --git a/etc/zudb.api.md b/etc/zudb.api.md index b2f0199..5e483ad 100644 --- a/etc/zudb.api.md +++ b/etc/zudb.api.md @@ -28,6 +28,7 @@ export function connect(path: string, options?: ConnectOptions | undefined | nul export class Connection { appender(table: string): Promise close(): void + columnar(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise cursor(statement: string, params?: Record | null, options?: ZuStreamOptions | null): ZuCursor dispose(): Promise exec(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise @@ -99,6 +100,49 @@ export interface ZuBatch> extends Array { // @public export type ZuBigIntMode = 'bigint' | 'number' +// @public +export interface ZuColumn { + readonly data: Uint8Array | null + readonly items: ZuValue[] | null + // (undocumented) + readonly length: number + // (undocumented) + readonly name: string + readonly nulls: number + readonly offsets: Int32Array | BigInt64Array | null + // (undocumented) + readonly type: ZuColumnType + readonly unit: 'days' | 'nanos' | 'months' | null + readonly validity: Uint8Array | null + readonly values: BigInt64Array | Float64Array | Int32Array | Uint8Array | null + readonly zone: number | null +} + +// @public +export interface ZuColumnar { + // (undocumented) + readonly columns: ZuColumn[] + // (undocumented) + readonly gqlstatus: string + // (undocumented) + readonly notices: ZuNotice[] + // (undocumented) + readonly rows: number +} + +// @public +export type ZuColumnType = +| 'null' +| 'bool' +| 'int' +| 'float' +| 'string' +| 'date' +| 'time' +| 'datetime' +| 'duration' +| 'value' + // @public export class ZuCursor { cancel(): Promise diff --git a/package.json b/package.json index 8a7a645..fc9f2f0 100644 --- a/package.json +++ b/package.json @@ -72,6 +72,7 @@ "reference": "node tools/reference.mjs reference", "bench": "node bench/query.mjs", "bench:append": "node bench/append.mjs", + "bench:columnar": "node bench/columnar.mjs", "bench:load": "node bench/load.mjs", "bench:register": "node bench/register.mjs", "bench:temporal": "node --harmony-temporal bench/query.mjs" diff --git a/src/columns.rs b/src/columns.rs new file mode 100644 index 0000000..514c59f --- /dev/null +++ b/src/columns.rs @@ -0,0 +1,305 @@ +//! A result read down its columns instead of across its rows. +//! +//! ```js +//! const read = await conn.columnar('MATCH (p:person) RETURN p.age AS age') +//! read.columns[0].values // a BigInt64Array of every age, and no objects +//! ``` +//! +//! `query` builds an object per row and a JavaScript value per cell, +//! which is what a program reading a hundred rows wants and is the +//! wrong shape for a million: every value costs an allocation and a +//! write barrier, and everything a caller is likely to do next with a +//! million of them wants columns anyway. Arrow is columns, so is every +//! dataframe, so is every plotting library worth the name, and so is +//! the typed array a numeric loop reads. +//! +//! What comes back is the buffers themselves, in the layout Arrow +//! already uses: values end to end, one bit a row of validity where +//! anything is null, strings as bytes and offsets. `zu::query::column` +//! builds them in the engine, in two passes over the rows, and this +//! module hands each one to the runtime as an external typed array. The +//! `Vec` is moved rather than read: the pointer V8 is given is the +//! pointer the engine filled, and the allocation is freed when the +//! typed array is collected. So a column of a million integers crosses +//! the boundary as a pointer and a length. +//! +//! The layout is Arrow's because that is the layout with readers. +//! `apache-arrow` wraps a buffer of this shape without copying it, and +//! the README says how in the ten lines it takes. This package still +//! does not depend on that one, which is the whole reason the answer is +//! buffers rather than a `Table`: a client that hands out an Arrow +//! object has to agree with one version of Arrow forever, and a client +//! that hands out the bytes agrees with all of them. +//! +//! Two things are not buffers. A column of nodes, rels, paths, lists or +//! records has no fixed width cell, so it arrives as `items`, the same +//! JavaScript values `query` would have made, and a column of nothing +//! but nulls has a length and nothing else, because there is nothing to +//! put in a buffer. Both are named by the column's `type` rather than +//! found out by looking. +//! +//! `bigIntMode` says nothing here. A columnar read has one physical +//! layout per type and an INT64 column is 64 bit cells whatever a +//! caller would rather read one cell as, which is the difference +//! between a buffer and a value: the mode applies to the values inside +//! `items`, where this client is making objects anyway. + +use napi::bindgen_prelude::*; +use napi::{Env, ScopedTask}; +use zudb::DiagnosticRecord; +use zudb::query::Value; +use zudb::query::column::{ColumnData, ColumnType, Columns, Offsets, Validity}; + +use crate::conn::{Failure, QueryTask, notices}; +use crate::value::{Shape, to_js}; + +/// One statement, read as columns. +pub struct ColumnsTask(pub(crate) QueryTask); + +/// What one column turned out to hold, owned here rather than borrowed +/// from the result, so that the result is free to go before any of this +/// reaches the runtime. +enum Held { + /// A column of nulls, which has a length and no buffer. + Empty, + /// One bit a row, least significant bit first. + Bits(Vec), + Int(Vec), + Float(Vec), + Days(Vec), + Nanos(Vec), + Months(Vec), + Str { + bytes: Vec, + offsets: Offsets, + }, + /// The values themselves, for what no buffer covers. + Items(Vec), +} + +/// One column: what it is called, what it holds, and the rows that have +/// a value. +struct Out { + name: String, + /// What a reader calls this type, which is a smaller vocabulary + /// than the engine's because the physical layout is the question. + kind: &'static str, + /// What a fixed width cell counts, where counting is what it does. + unit: Option<&'static str>, + /// Minutes east of UTC, for a column of zoned times or datetimes. + zone: Option, + held: Held, + validity: Option, + len: usize, +} + +/// A whole result, read down its columns. +pub struct Read { + rows: usize, + columns: Vec, + gqlstatus: &'static str, + notices: Vec, +} + +impl ColumnsTask { + fn run(&mut self) -> std::result::Result<(Read, Shape), Failure> { + let (result, shape) = self.0.run()?; + // The borrow of the result ends with this block, and everything + // that leaves it is owned, so the rows are freed here rather + // than held across the hop back to the runtime thread. + let read = { + let columns = result + .columnar() + .map_err(|mixed| Failure::Usage(mixed.to_string()))?; + taken(columns, result.status().code(), result.notices.clone()) + }; + Ok((read, shape)) + } +} + +/// The engine's columns, with every buffer moved out of them and every +/// borrowed value cloned. +fn taken(columns: Columns<'_>, gqlstatus: &'static str, notices: Vec) -> Read { + let Columns { columns, rows } = columns; + let columns = columns + .into_iter() + .map(|column| Out { + name: column.name.to_string(), + kind: kind(&column.ty), + unit: unit(&column.ty), + zone: match column.ty { + ColumnType::ZonedTime { offset } | ColumnType::ZonedDatetime { offset } => { + Some(offset as i32) + } + _ => None, + }, + held: match column.data { + ColumnData::Null => Held::Empty, + ColumnData::Bool { bits } => Held::Bits(bits), + ColumnData::Int(values) => Held::Int(values), + ColumnData::Float(values) => Held::Float(values), + ColumnData::Days(values) => Held::Days(values), + ColumnData::Nanos(values) => Held::Nanos(values), + ColumnData::Months(values) => Held::Months(values), + ColumnData::Str(column) => Held::Str { + bytes: column.bytes, + offsets: column.offsets, + }, + // The one arm that copies, and the one arm whose values + // are objects at the other end anyway. + ColumnData::Complex(values) => { + Held::Items(values.into_iter().cloned().collect::>()) + } + }, + validity: column.validity, + len: column.len, + }) + .collect(); + Read { + rows, + columns, + gqlstatus, + notices, + } +} + +/// What a reader calls this column's type. +/// +/// Narrower than [`ColumnType`] on purpose: what a caller has to branch +/// on is which buffer they were handed, and a time with an offset and a +/// time without are the same 64 bit cells. The offset rides beside as +/// `zone`, where it can be read by the caller who cares and ignored by +/// the loop that does not. +fn kind(ty: &ColumnType) -> &'static str { + match ty { + ColumnType::Null => "null", + ColumnType::Bool => "bool", + ColumnType::Int => "int", + ColumnType::Float => "float", + ColumnType::Str => "string", + ColumnType::Date => "date", + ColumnType::LocalTime | ColumnType::ZonedTime { .. } => "time", + ColumnType::LocalDatetime | ColumnType::ZonedDatetime { .. } => "datetime", + ColumnType::YearMonth | ColumnType::DayTime => "duration", + _ => "value", + } +} + +/// What one cell of this column counts. +/// +/// A duration is months or nanoseconds and never both, which is the +/// engine's rule and the one thing a reader of the buffer cannot work +/// out from the numbers in it. +fn unit(ty: &ColumnType) -> Option<&'static str> { + match ty { + ColumnType::Date => Some("days"), + ColumnType::LocalTime + | ColumnType::ZonedTime { .. } + | ColumnType::LocalDatetime + | ColumnType::ZonedDatetime { .. } + | ColumnType::DayTime => Some("nanos"), + ColumnType::YearMonth => Some("months"), + _ => None, + } +} + +impl<'task> ScopedTask<'task> for ColumnsTask { + type Output = std::result::Result<(Read, Shape), Failure>; + type JsValue = Object<'task>; + + fn compute(&mut self) -> Result { + Ok(self.run()) + } + + fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { + let (read, shape) = output.map_err(|failure| self.0.failed(env, failure))?; + let mut array = env.create_array(read.columns.len() as u32)?; + for (ix, column) in read.columns.into_iter().enumerate() { + array.set(ix as u32, described(env, column, &shape)?)?; + } + let mut object = Object::new(env)?; + object.set("rows", read.rows as f64)?; + object.set("columns", array)?; + object.set("gqlstatus", read.gqlstatus)?; + object.set("notices", notices(env, &read.notices)?)?; + Ok(object) + } + + fn finally(mut self, env: Env) -> Result<()> { + self.0.release(&env) + } +} + +/// One column as the object a caller reads. +/// +/// Every field is present on every column, holding null where it does +/// not apply, because a shape that changes by type is a shape a program +/// has to test before it can read, and the whole point of `type` is +/// that the test is one string comparison. +fn described<'env>(env: &'env Env, column: Out, shape: &Shape) -> Result> { + let Out { + name, + kind, + unit, + zone, + held, + validity, + len, + } = column; + let mut object = Object::new(env)?; + object.set("name", name.as_str())?; + object.set("type", kind)?; + object.set("length", len as f64)?; + object.set("unit", unit)?; + object.set("zone", zone)?; + + // The three that hold the values, one of which is not null. + let (mut values, mut data, mut offsets, mut items) = (None, None, None, None); + match held { + Held::Empty => {} + Held::Bits(bits) => values = Some(Uint8Array::new(bits).into_unknown(env)?), + Held::Int(v) | Held::Nanos(v) | Held::Months(v) => { + values = Some(BigInt64Array::new(v).into_unknown(env)?) + } + Held::Float(v) => values = Some(Float64Array::new(v).into_unknown(env)?), + Held::Days(v) => values = Some(Int32Array::new(v).into_unknown(env)?), + Held::Str { bytes, offsets: at } => { + data = Some(Uint8Array::new(bytes)); + offsets = Some(match at { + // Narrow until the bytes pass what a 32 bit offset + // addresses, which is what Arrow calls Utf8 against + // LargeUtf8 and what a reader has to know before it + // reads one. + Offsets::I32(o) => Int32Array::new(o).into_unknown(env)?, + Offsets::I64(o) => BigInt64Array::new(o).into_unknown(env)?, + }); + } + Held::Items(held) => { + let mut array = env.create_array(held.len() as u32)?; + for (ix, value) in held.iter().enumerate() { + array.set(ix as u32, to_js(env, name.as_str(), value, shape)?)?; + } + items = Some(array); + } + } + object.set("values", values)?; + object.set("data", data)?; + object.set("offsets", offsets)?; + object.set("items", items)?; + + // Absent when every row has a value, which is the common case and + // the one where a reader gets to skip the test entirely. Present + // means at least one null, so a caller never has to count to find + // out whether the bits are worth attaching. + match validity { + Some(Validity { bits, nulls, .. }) if nulls > 0 => { + object.set("validity", Uint8Array::new(bits))?; + object.set("nulls", nulls as f64)?; + } + _ => { + object.set("validity", Null)?; + object.set("nulls", 0.0)?; + } + } + Ok(object) +} diff --git a/src/conn.rs b/src/conn.rs index b1304ad..a17e7a8 100644 --- a/src/conn.rs +++ b/src/conn.rs @@ -34,6 +34,7 @@ use zudb::{Config, Database, DiagnosticRecord, Interrupt, ZuError}; use crate::append::OpenTask; use crate::cancel::Watch; +use crate::columns::ColumnsTask; use crate::error::{aborted, raise, usage}; use crate::register::{self, RegisterTask, RegisteredTask, UnregisterTask}; use crate::stream::{self, Started, ZuCursor}; @@ -549,6 +550,29 @@ impl Connection { AsyncTask::new(ExecTask(self.task(env, statement, params, options))) } + /// Runs one statement and gives back its columns rather than its + /// rows. + /// + /// The same statement as [`Connection::query`], read down instead + /// of across: what comes back is one buffer a column, in the layout + /// Arrow already uses, and no object a row. That is the way out for + /// anything that is going to be counted, plotted or handed to a + /// dataframe, and it is the way out that does not build a million + /// JavaScript values on the way. + #[napi( + ts_args_type = "statement: string, params?: Record | null, options?: ZuStatementOptions | null", + ts_return_type = "Promise" + )] + pub fn columnar( + &self, + env: &Env, + statement: Unknown<'_>, + params: Option>, + options: Option>, + ) -> AsyncTask { + AsyncTask::new(ColumnsTask(self.task(env, statement, params, options))) + } + /// Runs one statement and gives back a cursor over its rows. /// /// The pull underneath `stream`, which is what a program uses. The @@ -1052,7 +1076,7 @@ impl QueryTask { /// The names are read while the lock is held, because a catalog /// borrowed from the connection cannot outlive it and a result that /// names its tables has to carry them. - fn run(&mut self) -> std::result::Result<(QueryResult, Shape), Failure> { + pub(crate) fn run(&mut self) -> std::result::Result<(QueryResult, Shape), Failure> { if let Some(message) = self.refused.take() { return Err(Failure::Usage(message)); } @@ -1093,12 +1117,12 @@ impl QueryTask { } /// The exception this rejects the caller's promise with. - fn failed(&self, env: &Env, failure: Failure) -> Error { + pub(crate) fn failed(&self, env: &Env, failure: Failure) -> Error { failed(env, failure, self.watch.as_ref()) } /// Takes the listener back off the signal, whatever happened. - fn release(&mut self, env: &Env) -> Result<()> { + pub(crate) fn release(&mut self, env: &Env) -> Result<()> { match self.watch.take() { Some(watch) => watch.release(env), None => Ok(()), diff --git a/src/lib.rs b/src/lib.rs index ca9e455..a62c2a5 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -19,6 +19,7 @@ mod append; mod buffer; mod cancel; +mod columns; mod conn; mod error; mod frame; diff --git a/test/columnar.test.mjs b/test/columnar.test.mjs new file mode 100644 index 0000000..bc4679e --- /dev/null +++ b/test/columnar.test.mjs @@ -0,0 +1,457 @@ +// A result read down its columns. +// +// The buffers are the engine's own and they cross the boundary as a +// pointer and a length, so what these assert is both halves of that: +// that the numbers in them are the numbers the statement produced, and +// that the layout is the one every columnar reader already knows, down +// to which bit of which byte a boolean sits in. + +import assert from 'node:assert/strict' +import test from 'node:test' + +import { + Bool, + DateDay, + DurationNanosecond, + Float64, + Int64, + LargeUtf8, + makeData, + Table, + TimeNanosecond, + TimestampNanosecond, + Utf8, + Vector, +} from 'apache-arrow' + +import { fresh, isZuError, twoPeople } from './helper.mjs' + +// The columns by name, since a test asks about one of them and the +// order they were projected in is asserted where it is the question. +function named(read) { + return Object.fromEntries(read.columns.map((column) => [column.name, column])) +} + +// A string column read back out of its bytes and offsets, which is the +// walk a caller writes once and the one thing about the layout worth +// showing in full. +function strings(column) { + const text = new TextDecoder() + const out = [] + for (let row = 0; row < column.length; row += 1) { + const from = Number(column.offsets[row]) + const to = Number(column.offsets[row + 1]) + out.push(text.decode(column.data.subarray(from, to))) + } + return out +} + +// Whether row `at` has a value, which is one bit of one byte and the +// same test every columnar format uses. +function valid(column, at) { + return column.validity === null || (column.validity[at >> 3] & (1 << (at & 7))) !== 0 +} + +test('a column of integers is a BigInt64Array of the values', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + + assert.equal(read.rows, 2) + assert.equal(read.columns.length, 1) + const [id] = read.columns + assert.equal(id.name, 'id') + assert.equal(id.type, 'int') + assert.equal(id.length, 2) + assert.ok(id.values instanceof BigInt64Array) + assert.deepEqual([...id.values], [1n, 2n]) + + // Nothing else applies, and every one of them is present and null + // rather than missing, so reading a column is a switch on its type. + assert.equal(id.data, null) + assert.equal(id.offsets, null) + assert.equal(id.items, null) + assert.equal(id.validity, null) + assert.equal(id.nulls, 0) + assert.equal(id.unit, null) + assert.equal(id.zone, null) +}) + +test('the columns come back in the order the statement projected them', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.name AS name, p.id AS id') + assert.deepEqual(read.columns.map((column) => column.name), ['name', 'id']) +}) + +test('a column of strings is bytes and offsets into them', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.name AS name') + + const [name] = read.columns + assert.equal(name.type, 'string') + assert.equal(name.values, null) + assert.ok(name.data instanceof Uint8Array) + // One offset more than there are rows: the last one closes the last + // string, which is what lets row `i` be `offsets[i]` to `offsets[i+1]` + // without a length beside it. + assert.ok(name.offsets instanceof Int32Array) + assert.equal(name.offsets.length, name.length + 1) + assert.deepEqual([...name.offsets], [0, 3, 6]) + assert.deepEqual(strings(name), ['ada', 'zoe']) +}) + +test('a column of floats is a Float64Array', async (t) => { + const { conn } = await fresh(t) + await conn.exec("INSERT (m:measure {id: 1, ratio: 1.5})") + await conn.exec("INSERT (m:measure {id: 2, ratio: -0.25})") + const read = await conn.columnar('MATCH (m:measure) RETURN m.ratio AS ratio') + + const [ratio] = read.columns + assert.equal(ratio.type, 'float') + assert.ok(ratio.values instanceof Float64Array) + assert.deepEqual([...ratio.values], [1.5, -0.25]) +}) + +test('a column of booleans is one bit a row, least significant first', async (t) => { + const { conn } = await fresh(t) + const yes = [true, false, true, false, true, true, false, true, false, true] + for (const [ix, hot] of yes.entries()) { + await conn.exec(`INSERT (f:flag {id: ${ix + 1}, hot: ${hot}})`) + } + const read = await conn.columnar('MATCH (f:flag) RETURN f.hot AS hot') + + const [hot] = read.columns + assert.equal(hot.type, 'bool') + assert.ok(hot.values instanceof Uint8Array) + // Ten rows are two bytes, and the second holds two bits of value and + // six of nothing. + assert.equal(hot.values.length, 2) + const bits = yes.map((_, at) => (hot.values[at >> 3] & (1 << (at & 7))) !== 0) + assert.deepEqual(bits, yes) +}) + +test('a temporal column says what its cells count', async (t) => { + const { conn } = await fresh(t) + await conn.exec( + "INSERT (e:event {id: 1, on: DATE '2024-01-01', at: LOCAL DATETIME '2024-01-02T03:04:05', " + + "took: DURATION 'PT1H'})", + ) + const read = await conn.columnar( + 'MATCH (e:event) RETURN e.on AS on, e.at AS at, e.took AS took', + ) + const { on, at, took } = named(read) + + assert.equal(on.type, 'date') + assert.equal(on.unit, 'days') + assert.ok(on.values instanceof Int32Array) + assert.deepEqual([...on.values], [19_723]) + + assert.equal(at.type, 'datetime') + assert.equal(at.unit, 'nanos') + assert.deepEqual([...at.values], [1_704_164_645_000_000_000n]) + + assert.equal(took.type, 'duration') + assert.equal(took.unit, 'nanos') + assert.deepEqual([...took.values], [3_600_000_000_000n]) +}) + +test('a duration of months counts months rather than nanoseconds', async (t) => { + const { conn } = await fresh(t) + const read = await conn.columnar("RETURN DURATION 'P14M' AS every") + + const [every] = read.columns + assert.equal(every.type, 'duration') + assert.equal(every.unit, 'months') + assert.deepEqual([...every.values], [14n]) +}) + +test('a zoned column carries its offset beside the cells', async (t) => { + const { conn } = await fresh(t) + const read = await conn.columnar( + "RETURN ZONED TIME '03:04:05+02:00' AS t, ZONED DATETIME '2024-01-02T03:04:05+02:00' AS d", + ) + const { t: at, d } = named(read) + + // The type is the physical one, since a time with an offset and a + // time without are the same 64 bit cells, and the offset is the + // minutes east of UTC the column was written with. + assert.equal(at.type, 'time') + assert.equal(at.zone, 120) + assert.equal(d.type, 'datetime') + assert.equal(d.zone, 120) +}) + +test('a null row keeps its cell and clears its bit', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar( + 'MATCH (p:person) RETURN CASE WHEN p.id = 1 THEN p.id ELSE null END AS maybe', + ) + + const [maybe] = read.columns + assert.equal(maybe.type, 'int') + assert.equal(maybe.nulls, 1) + assert.ok(maybe.validity instanceof Uint8Array) + // The null row still occupies its cell, holding the type's zero, + // which is what lets the buffer be strided and moved rather than + // rebuilt. + assert.deepEqual([...maybe.values], [1n, 0n]) + assert.equal(valid(maybe, 0), true) + assert.equal(valid(maybe, 1), false) +}) + +test('a column with nothing null has no bitmap at all', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + // Present would mean at least one null, so a caller never has to + // count to find out whether the bits are worth attaching. + assert.equal(read.columns[0].validity, null) + assert.equal(read.columns[0].nulls, 0) +}) + +test('a column of nothing but nulls has a length and no buffer', async (t) => { + const { conn } = await fresh(t) + const read = await conn.columnar('RETURN null AS nothing') + + const [nothing] = read.columns + assert.equal(nothing.type, 'null') + assert.equal(nothing.length, 1) + assert.equal(nothing.values, null) + assert.equal(nothing.items, null) +}) + +test('what no buffer covers arrives as the values themselves', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar( + 'MATCH (p:person) RETURN p AS who, [p.id, p.id] AS pair, {name: p.name} AS record', + ) + const { who, pair, record } = named(read) + + assert.equal(who.type, 'value') + assert.equal(who.values, null) + assert.equal(who.items.length, 2) + assert.equal(who.items[0].table, 'person') + assert.equal(who.items[0].offset, 0n) + + assert.equal(pair.type, 'value') + assert.deepEqual(pair.items[0], [1n, 1n]) + + assert.equal(record.type, 'value') + assert.deepEqual(record.items[1], { name: 'zoe' }) +}) + +test('a statement that matched nothing is columns of no rows', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar("MATCH (p:person) WHERE p.name = 'nobody' RETURN p.id AS id") + + assert.equal(read.rows, 0) + assert.equal(read.columns.length, 1) + assert.equal(read.columns[0].length, 0) + // Nothing settled the type, so it is the type of nothing, which is + // the one every columnar format has for exactly this. + assert.equal(read.columns[0].type, 'null') +}) + +test('a statement that projects nothing has no columns and says so', async (t) => { + const { conn } = await fresh(t) + const read = await conn.columnar("INSERT (p:person {id: 1, name: 'ada'})") + + assert.equal(read.columns.length, 0) + assert.equal(read.rows, 0) + // 00001 is the standard's own way of saying the statement completed + // and had no result to give back. + assert.equal(read.gqlstatus, '00001') +}) + +test('the status and the notices ride beside the columns', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + assert.equal(read.gqlstatus, '00000') + assert.deepEqual(read.notices, []) +}) + +test('parameters bind the same way they do for rows', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar( + 'MATCH (p:person) WHERE p.name = $name RETURN p.id AS id', + { name: 'zoe' }, + ) + assert.deepEqual([...read.columns[0].values], [2n]) +}) + +test('a column that mixes two types is refused, with the row that did it', async (t) => { + const { conn } = await twoPeople(t) + const caught = await conn + .columnar('MATCH (p:person) RETURN CASE WHEN p.id = 1 THEN p.id ELSE p.name END AS mixed') + .then(() => null, (err) => err) + + // A column holds one type and a caller told which column and which + // row can act on it, while one told neither goes looking through a + // million rows by hand. + assert.ok(isZuError(caught, 'ZuUsageError')) + assert.match(caught.message, /column 'mixed' mixes integers and strings at row 1/) +}) + +test('the spelling of an integer is not a question a buffer answers', async (t) => { + const { conn } = await twoPeople(t, { bigIntMode: 'number' }) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + + // A columnar read has one physical layout per type, so the mode that + // decides how a value is spelled has nothing to decide here. It still + // decides inside `items`, where this client is making objects anyway. + assert.ok(read.columns[0].values instanceof BigInt64Array) + assert.deepEqual([...read.columns[0].values], [1n, 2n]) +}) + +test('a statement can be stopped by a signal like any other', async (t) => { + const { conn } = await twoPeople(t) + const caught = await conn + .columnar('MATCH (p:person) RETURN p.id AS id', null, { signal: AbortSignal.abort() }) + .then(() => null, (err) => err) + + assert.equal(caught.name, 'AbortError') + // The connection is left exactly as it was, which is what makes a + // stopped statement a stopped statement rather than a broken one. + assert.equal((await conn.columnar('MATCH (p:person) RETURN p.id AS id')).rows, 2) +}) + +test('a closed connection refuses the call as a rejection', async (t) => { + const { conn } = await twoPeople(t) + await conn.close() + const caught = await conn + .columnar('MATCH (p:person) RETURN p.id AS id') + .then(() => null, (err) => err) + assert.ok(isZuError(caught, 'ZuUsageError')) +}) + +test('a statement that is not a string is refused inside the promise', async (t) => { + const { conn } = await fresh(t) + const caught = await conn.columnar(42).then(() => null, (err) => err) + assert.ok(isZuError(caught, 'ZuUsageError')) + assert.match(caught.message, /the statement is a Number/) +}) + +test('the buffers are handed over rather than shared, so two reads are two buffers', async (t) => { + const { conn } = await twoPeople(t) + const first = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + const second = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + + // Writing into one is writing into a buffer nothing else is reading, + // which is what makes handing the memory over safe: the engine freed + // its side of it when the array was made. + first.columns[0].values[0] = 99n + assert.deepEqual([...second.columns[0].values], [1n, 2n]) +}) + +test('a million rows come back down one buffer and the loop stays free', async (t) => { + const { conn } = await fresh(t) + await conn.exec("INSERT (n:number {id: 1, at: 1})") + const rows = 1_000_000 + const appender = await conn.appender('number') + for (let at = 2; at <= rows; at += 1) appender.appendRow([BigInt(at), BigInt(at)]) + await appender.close() + + let ticks = 0 + const timer = setInterval(() => (ticks += 1), 1) + const read = await conn.columnar('MATCH (n:number) RETURN n.at AS at') + clearInterval(timer) + + assert.equal(read.rows, rows) + assert.equal(read.columns[0].values.length, rows) + assert.equal(read.columns[0].values[rows - 1], BigInt(rows)) + // The whole read is on the threadpool, so the timer kept firing + // throughout it rather than queueing behind it. + assert.ok(ticks > 20, `the event loop ticked ${ticks} times`) +}) + +// The types every fixed-width column maps to, which is the whole of +// what a caller has to write to make the buffers an Arrow table. +const ARROW = { + int: () => new Int64(), + float: () => new Float64(), + bool: () => new Bool(), + string: (column) => (column.offsets instanceof Int32Array ? new Utf8() : new LargeUtf8()), + date: () => new DateDay(), + time: () => new TimeNanosecond(), + datetime: () => new TimestampNanosecond(), + duration: () => new DurationNanosecond(), +} + +// The recipe the README prints, kept here so that it is run rather than +// believed. Nothing in the package imports `apache-arrow`, and this is +// what that costs a caller who wants one: eleven lines, once. +function tableOf(read) { + const columns = {} + for (const column of read.columns) { + columns[column.name] = new Vector([ + makeData({ + type: ARROW[column.type](column), + length: column.length, + nullCount: column.nulls, + nullBitmap: column.validity ?? undefined, + data: column.data ?? column.values, + valueOffsets: column.offsets ?? undefined, + }), + ]) + } + return new Table(columns) +} + +test('the columns become an Arrow table without being copied', async (t) => { + const { conn } = await fresh(t) + await conn.exec( + "INSERT (e:event {id: 1, name: 'ada', ratio: 1.5, hot: true, on: DATE '2024-01-01', " + + "took: DURATION 'PT1H'})", + ) + await conn.exec( + "INSERT (e:event {id: 2, name: 'zoe', ratio: 2.5, hot: false, on: DATE '2024-02-01', " + + "took: DURATION 'PT2H'})", + ) + const read = await conn.columnar( + 'MATCH (e:event) RETURN e.id AS id, e.name AS name, e.ratio AS ratio, e.hot AS hot, ' + + 'e.on AS on, e.took AS took', + ) + const table = tableOf(read) + + assert.equal(table.numRows, 2) + assert.deepEqual( + table.schema.fields.map((field) => `${field.name}:${field.type}`), + [ + 'id:Int64', + 'name:Utf8', + 'ratio:Float64', + 'hot:Bool', + 'on:Date32', + 'took:Duration', + ], + ) + assert.equal(table.getChild('id').get(1), 2n) + assert.equal(table.getChild('name').get(0), 'ada') + assert.equal(table.getChild('hot').get(1), false) + + // The same memory on both sides, which is the whole claim: the array + // Arrow reads is the array the engine filled and not a copy of it. + assert.equal(table.getChild('id').data[0].values, read.columns[0].values) +}) + +test('an Arrow table built this way keeps the nulls it was given', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar( + 'MATCH (p:person) RETURN CASE WHEN p.id = 1 THEN p.id ELSE null END AS maybe', + ) + const table = tableOf(read) + + assert.equal(table.getChild('maybe').get(0), 1n) + assert.equal(table.getChild('maybe').get(1), null) + assert.equal(table.getChild('maybe').nullCount, 1) +}) + +test('the columns go straight back in as a frame', async (t) => { + const { conn } = await twoPeople(t) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + + // The buffer that came out is a column a statement can match on, + // which is what makes the two halves of this one shape: nothing is + // decoded on the way out and nothing is encoded on the way back. + assert.equal(await conn.register('ids', { id: read.columns[0].values }), 2) + const found = await conn.query('MATCH (i:ids) WHERE i.id > 1 RETURN i.id AS id') + assert.deepEqual(found.map((row) => row.id), [2n]) +}) diff --git a/test/types/cjs.cts b/test/types/cjs.cts index 01e3734..d32ccfd 100644 --- a/test/types/cjs.cts +++ b/test/types/cjs.cts @@ -9,6 +9,7 @@ import { ZuTimestamp, type ZuAppendValue, type ZuArrowTable, + type ZuColumnType, type ZuLoadOptions, type ZuParam, type ZuStream, @@ -122,3 +123,20 @@ export async function ids(path: string): Promise { } return out } + +export async function shapes(path: string): Promise { + const conn = await connect(path, { readOnly: true }) + const read = await conn.columnar('MATCH (p:person) RETURN p.id AS id, p.name AS name') + + // A string column is its bytes and its offsets, and both are nullable + // on every column, so reading one without asking what it is does not + // compile. + const text = new TextDecoder() + for (const column of read.columns) { + if (column.type !== 'string' || column.data === null || column.offsets === null) continue + text.decode(column.data.subarray(Number(column.offsets[0]), Number(column.offsets[1]))) + } + + await conn.close() + return read.columns.map((column) => column.type) +} diff --git a/test/types/esm.mts b/test/types/esm.mts index 6791796..7571485 100644 --- a/test/types/esm.mts +++ b/test/types/esm.mts @@ -15,6 +15,8 @@ import { type ZuEdges, type ZuFrame, type ZuFrameColumn, + type ZuColumn, + type ZuColumnar, type ZuLoadStats, type ZuPlainDate, type ZuRows, @@ -224,3 +226,24 @@ export function width(value: ZuValue): number { if (Array.isArray(value)) return value.length return 0 } + +export async function totals(path: string): Promise { + await using conn = await connect(path, { readOnly: true }) + + // The buffers themselves, so summing them is a loop over a typed + // array and not over a million objects. Which field carries the + // values follows from the type, and narrowing on `type` is what makes + // `values` a `BigInt64Array` here rather than the union it starts as. + const read: ZuColumnar = await conn.columnar('MATCH (p:person) RETURN p.id AS id') + const column: ZuColumn = read.columns[0]! + if (column.type !== 'int') throw new Error('the projection changed shape') + + let total = 0n + for (const value of column.values as BigInt64Array) total += value + + // The counts beside them are numbers, which is the one place a count + // in this package is not a bigint, and the status is the statement's. + const rows: number = read.rows + if (read.gqlstatus !== '00000') throw new Error(read.gqlstatus) + return total + BigInt(rows) + BigInt(column.nulls) +} diff --git a/types/header.d.ts b/types/header.d.ts index 127f4bc..e33f109 100644 --- a/types/header.d.ts +++ b/types/header.d.ts @@ -290,6 +290,93 @@ export interface ZuNotice { readonly docUrl: string } +/** + * What a column of a columnar read turned out to hold. + * + * Narrower than the type the statement declared, because the question + * here is which buffer arrived: a time with an offset and a time + * without are the same 64 bit cells, and the offset rides beside as + * `zone`. `value` is the fallback for what no fixed width cell covers, + * which is nodes, rels, paths, lists and records, and `null` is a + * column that held nothing else. + */ +export type ZuColumnType = + | 'null' + | 'bool' + | 'int' + | 'float' + | 'string' + | 'date' + | 'time' + | 'datetime' + | 'duration' + | 'value' + +/** + * One column of a result, as the buffer holding it. + * + * Every field is present on every column and holds null where it does + * not apply, so reading one is a switch on `type` rather than a series + * of tests for what is there. Which field carries the values follows + * from the type: `values` for everything of a fixed width, `data` and + * `offsets` for strings, `items` for what no buffer covers, and none of + * them for a column of nulls. + * + * The buffers are the engine's own, handed over rather than copied, and + * they are laid out the way Arrow lays them out: values end to end, a + * boolean as one bit a row, a string column as its bytes and `length + + * 1` offsets into them, where row `i` spans `offsets[i]` to `offsets[i + * + 1]`. + */ +export interface ZuColumn { + readonly name: string + readonly type: ZuColumnType + readonly length: number + /** + * The cells, for a column of a fixed width: `BigInt64Array` for + * integers, nanoseconds and months, `Float64Array` for floats, + * `Int32Array` for days, and a `Uint8Array` of packed bits for + * booleans, least significant bit first. + */ + readonly values: BigInt64Array | Float64Array | Int32Array | Uint8Array | null + /** The bytes of every string end to end, for a string column. */ + readonly data: Uint8Array | null + /** + * `length + 1` offsets into `data`, for a string column. Narrow until + * the bytes pass what a 32 bit offset addresses, which is the + * difference Arrow calls Utf8 against LargeUtf8. + */ + readonly offsets: Int32Array | BigInt64Array | null + /** The values themselves, for a column of type `value`. */ + readonly items: ZuValue[] | null + /** + * One bit a row, least significant bit first, set meaning the row has + * a value. Null when every row has one, which is the common case and + * the one where a reader gets to skip the test. + */ + readonly validity: Uint8Array | null + /** How many rows are null, which is zero when `validity` is null. */ + readonly nulls: number + /** What one cell counts: `days`, `nanos` or `months`. */ + readonly unit: 'days' | 'nanos' | 'months' | null + /** Minutes east of UTC, for a column of zoned times or datetimes. */ + readonly zone: number | null +} + +/** + * A whole result read down its columns. + * + * `rows` is every column's length, and is the answer for a statement + * that projected nothing at all. `gqlstatus` and `notices` are the + * statement's, exactly as they are on the rows. + */ +export interface ZuColumnar { + readonly rows: number + readonly columns: ZuColumn[] + readonly gqlstatus: string + readonly notices: ZuNotice[] +} + /** * The rows a statement gave back. *