diff --git a/README.md b/README.md index 65e6580..343ea61 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin ## What works today -`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time, and `load` for building a whole database out of columns and an edge list. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. `columnar`, for a result read down its columns as the buffers themselves rather than across its rows as objects. Both module formats, typed separately. +`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time, and `load` for building a whole database out of columns and an edge list. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. `columnar`, for a result read down its columns as the buffers themselves rather than across its rows as objects. Prepared statements, compiled at the line that asked and run as often as wanted, and `explain` and `profile`, as a tree a program walks and as the listing a person reads. Both module formats, typed separately. Build it with `npm run build`, and run the suite with `npm test`. Nothing is published yet, so `npm i zudb` is not a thing you can type at anybody's terminal, but everything it will do is built and installed on every run of the release workflow. @@ -240,6 +240,66 @@ Two things are not buffers, and both are named by the type rather than found out `bigIntMode` says nothing here. A columnar read has one physical layout per type and an INT64 column is 64 bit cells however a caller would rather read one, which is the difference between a buffer and a value. The mode still decides what is inside `items`, where this client is making objects anyway. +## Preparing a statement + +`conn.prepare` compiles a statement now and hands back something that runs it later, as often as it is asked to, with different values bound each time: + +```ts +await using find = await conn.prepare(`MATCH (p:person) WHERE p.name = $name RETURN p.id AS id`); +find.params; // ["name"], which the statement asked for and nobody had to read out of it +const ada = await find.query<{ id: bigint }>({ name: "ada" }); +const zoe = await find.query<{ id: bigint }>({ name: "zoe" }); +``` + +It answers the same three ways a connection does. `query` gives rows, `exec` gives nothing and is for a statement written to change something, and `columnar` gives the buffers. Each takes the bindings and the same options a statement takes, so a signal and a `bigIntMode` go on the run rather than on the prepare, since which run a caller wants to stop is a property of that run. + +What this is not is a speedup, and it is worth saying so here rather than letting a reader assume the thing every other client's documentation says. A driver prepares to save a round trip to a server, and there is no server and no round trip here. The engine already caches the plan for a statement by its text, so the second `conn.query` of the same string is not compiled a second time either. On this machine, with `npm run bench:prepared` over 100 rows and 5000 runs: + +``` +prepared, bound per run 13756 ns each +the same text, bound per run 13812 ns each +a new text per run 21209 ns each +``` + +The first two are the same number, and that is the honest result. The third is the one to read: a statement whose text is different every run, which is what a program that pastes its values into the string is writing, pays the compile every time, and the roughly 7 microseconds between it and the other two is the size of that mistake. `prepare` and `close` together cost 10 microseconds, which is that same compile bought once. + +So what preparing buys is two things, and neither of them is throughput. The compile happens at the line that asked for it, at startup, where a statement that does not compile fails on the way up rather than on the first request that needed it. And the names come back: `params` is what the statement wants bound, in the order the engine found them, which is how a layer that binds from a record knows what to look for. `statement` is the text it was given back, and `closed` says whether it still holds anything. + +A prepared statement holds an id on the connection that made it, so closing it gives that back. `await using` is the intended scoping, `close()` is there for callers who cannot use the syntax, closing twice is not an error, and every run after the close is refused with the reason. One whose connection closed first is refused too, saying the connection is closed, because the session that was holding the id went with it. There is no `stream` on a prepared statement, deliberately: the engine's streaming path takes a text rather than a pinned id, so a streamed prepared statement would be this client quietly running the text again behind the caller, and a method that does not do what its name says is worse than one that is not there. + +## Seeing what a statement will do + +`explain` compiles a statement and answers the plan without running it: + +```ts +const plan = await conn.explain(`MATCH (p:person) WHERE p.name = $name RETURN p.id AS id`); +plan.columns; // ["id"] +plan.params; // ["name"] +plan.root.op; // "Project" +plan.root.children[0].detail; // "p.name = $name" +console.log(plan.text); +// Project p.id AS id +// Filter p.name = $name +// ScanNodes p: person +``` + +It comes back twice on purpose. `root` is the tree, for a program: every operator carries `op`, the `detail` it works on, the `binds` it introduces, the `tables` it touches, and its `children`, so a test can assert that a scan became an index seek without matching on a string. `text` is the engine's own listing, for a person, and it is the engine's rather than this client's rendering of the tree so the two cannot drift apart from one release to the next. An operator inside a bracket says which one it is in, `Optional`, `Semi`, `Anti` or `Mark`, and `name` is what the listing calls it, which for an expand inside an optional match is `OptionalExpand` while `op` stays `Expand`. + +`explain` takes no parameters, and that is not an oversight. A plan is chosen from the shape of the statement, and the values are bound when it runs, so a plan asked for with values would suggest that the values changed it. `scalars` is the other half of that: a query written where a value belongs gets a plan of its own, and `reads` says which variables of the query around it that plan reads, which is the whole difference between a subquery that runs once and one that runs once a row. + +`profile` runs the statement and answers what the operators actually did: + +```ts +const run = await conn.profile(`MATCH (p:person) RETURN p.name AS name`, { since: 1990 }); +const scan = run.stages[0].ops.find((op) => op.op === "Scan"); +scan.rows; // what it really produced +scan.estimate; // what the optimizer thought, or null if it had nothing to say +scan.qerror; // the two divided, the way the literature writes it +run.nanos; // every stage added up +``` + +A profile takes bindings, since it is a run. Every count is a `number` and not a `bigint`, which is the one place in this client an integer is spelled as a double on purpose: nothing a profile counts, not rows, not pulls, not nanoseconds of a statement anybody waited for, comes anywhere near 2^53, and a caller doing arithmetic on a measurement should not have to convert first. `pulls` is how many times the operator above asked, `rows` is how many it answered, `flat` is the same count with vectors unpacked, `bound` is the upper bound the optimizer had, and `qerror` is null where an estimate was. A statement that writes is refused rather than profiled, saying so, because a profile that also inserted two rows is a measurement that changed the thing measured. + ## Asking for numbers instead of bigints `bigIntMode` says how INT64 is spelled on the way out. It goes on one statement, or on a connection for all of them, and a statement on a connection that named one may still name the other: @@ -332,7 +392,7 @@ typedoc rather than api-documenter, which would have been the obvious pick since Anything outside that table has no binary and no source build to fall back on, so the install resolves nothing and the first `require` says so. The browser and the platforms nobody builds for are what the WASM target answers, later. -`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the appender, `npm run bench:load` for building a database out of columns, `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows, and `npm run bench:columnar` for a result read down its columns against the same result read across its rows. +`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the appender, `npm run bench:load` for building a database out of columns, `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows, `npm run bench:columnar` for a result read down its columns against the same result read across its rows, and `npm run bench:prepared` for a prepared statement against the same text run again and against a text that is new every time, which is the one of these whose interesting number is that the first two are equal. ## Still to come diff --git a/bench/prepared.mjs b/bench/prepared.mjs new file mode 100644 index 0000000..3fa696a --- /dev/null +++ b/bench/prepared.mjs @@ -0,0 +1,132 @@ +// What preparing a statement costs and what it saves. +// +// The answer is not the one a driver would give, and this exists to +// print that rather than to hide it. A driver prepares to save a round +// trip; there is no round trip here, and the engine caches the plan for +// a statement by its text, so the second `conn.query` of the same string +// is already not being compiled a second time. The two lines of the +// first pair should therefore land close together, and if `prepared` is +// a shade behind that is the id being looked up and the text cloned. +// +// The line worth reading is the third: a statement whose text is +// different every time, which is what a program that pastes its values +// into the string is doing. That one pays the whole compile per run, and +// the gap between it and the other two is the size of the mistake. +// +// The last block is the prepare itself, which is the compile a program +// pays once at startup so that no request pays it. +// +// Run it against a release build, for the reason bench/query.mjs gives. +// +// npm run build && npm run bench:prepared + +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { connect } from 'zudb' + +const ROWS = Number(process.env.ZU_BENCH_ROWS ?? 10_000) +const RUNS = Number(process.env.ZU_BENCH_RUNS ?? 2_000) +const REPEATS = Number(process.env.ZU_BENCH_REPEATS ?? 5) + +const dir = await mkdtemp(join(tmpdir(), 'zu-bench-prepared-')) +const conn = await connect(join(dir, 'bench.zu1')) + +await conn.exec("INSERT (p:person {uid: 1, name: 'n1'})") +{ + const rows = await conn.appender('person') + for (let ix = 2; ix <= ROWS; ix++) rows.appendRow([BigInt(ix), `n${ix}`]) + await rows.close() +} + +/// The fastest of `REPEATS` runs, in milliseconds, after one warmup. +async function time(run) { + await run() + let best = Infinity + for (let round = 0; round < REPEATS; round++) { + const started = performance.now() + await run() + best = Math.min(best, performance.now() - started) + } + return best +} + +function report(name, ms, each) { + console.log( + `${name.padEnd(34)} ${ms.toFixed(1).padStart(8)} ms ${Math.round((ms * 1e6) / each) + .toString() + .padStart(7)} ns each`, + ) +} + +const find = await conn.prepare('MATCH (p:person) WHERE p.name = $name RETURN p.uid AS uid') + +/// A number that never repeats, for the case that wants a statement the +/// plan cache has never seen. +let stamp = 0 + +const cases = [ + { + name: 'prepared, bound per run', + run: async () => { + for (let ix = 0; ix < RUNS; ix++) await find.query({ name: `n${(ix % ROWS) + 1}` }) + }, + }, + { + name: 'the same text, bound per run', + run: async () => { + for (let ix = 0; ix < RUNS; ix++) + await conn.query('MATCH (p:person) WHERE p.name = $name RETURN p.uid AS uid', { + name: `n${(ix % ROWS) + 1}`, + }) + }, + }, + { + name: 'a new text per run', + // The alias carries a counter that never repeats, so every one of + // these is a text the plan cache has not seen. A statement that + // pastes its values in rather than binding them is only this slow + // once the values stop repeating, which in a program serving + // requests is immediately. + run: async () => { + for (let ix = 0; ix < RUNS; ix++) + await conn.query( + `MATCH (p:person) WHERE p.name = 'n${(ix % ROWS) + 1}' RETURN p.uid AS uid${stamp++}`, + ) + }, + }, +] + +console.log(`${RUNS} runs over ${ROWS} rows, fastest of ${REPEATS}`) +for (const { name, run } of cases) report(name, await time(run), RUNS) + +// The compile, which is what a program pays at startup so that no +// request pays it. Every prepare is closed again, since a bench that +// leaked two thousand of them would be measuring the map they went into. +console.log('') +console.log('preparing itself') +report( + 'prepare and close', + await time(async () => { + for (let ix = 0; ix < 100; ix++) { + const statement = await conn.prepare( + 'MATCH (p:person) WHERE p.name = $name RETURN p.uid AS uid', + ) + await statement.close() + } + }), + 100, +) +report( + 'explain', + await time(async () => { + for (let ix = 0; ix < 100; ix++) + await conn.explain('MATCH (p:person) WHERE p.name = $name RETURN p.uid AS uid') + }), + 100, +) + +await find.close() +await conn.close() +await rm(dir, { recursive: true, force: true }) diff --git a/binding.cjs b/binding.cjs index 37ad67f..2dc28d7 100644 --- a/binding.cjs +++ b/binding.cjs @@ -702,6 +702,7 @@ if (!nativeBinding) { module.exports = nativeBinding module.exports.Appender = nativeBinding.Appender module.exports.Connection = nativeBinding.Connection +module.exports.Prepared = nativeBinding.Prepared module.exports.Transaction = nativeBinding.Transaction module.exports.ZuCursor = nativeBinding.ZuCursor module.exports.ZuDate = nativeBinding.ZuDate diff --git a/binding.d.cts b/binding.d.cts index f09f4f2..4058d86 100644 --- a/binding.d.cts +++ b/binding.d.cts @@ -432,6 +432,143 @@ export interface ZuSummary { readonly notices: ZuNotice[] } +/** + * One operator of a plan, and everything under it. + * + * The tree runs the way the rows do: a parent pulls from its children, + * so the leaves are the scans and the root is whatever the statement + * ends with. + */ +export interface ZuPlanNode { + /** The operator: `Scan`, `Expand`, `Filter`, `Project` and the rest. */ + readonly op: string + /** + * What the listing calls it, which is `op` with the bracket in front + * of it where there is one, so an OPTIONAL MATCH expand is an + * `Expand` named `OptionalExpand`. + */ + readonly name: string + /** The bracket this operator is inside, and null for a plain match. */ + readonly bracket: 'Optional' | 'Semi' | 'Anti' | 'Mark' | null + /** + * What it is working on, written the way the statement wrote it: the + * tables a scan reads, the pattern an expand walks, the predicate a + * filter asks. Empty where the operator has nothing to name. + */ + readonly detail: string + /** The variables it introduces, in the order it binds them. */ + readonly binds: string[] + /** + * The tables it touches: node tables for a scan, rel tables for an + * expand, both for an insert, and none anywhere else. + */ + readonly tables: string[] + readonly children: ZuPlanNode[] +} + +/** + * A query written where a value belongs, planned on its own. + * + * `reads` is what it reads from the query around it, and empty is the + * whole test for whether it runs once: a subquery that reads nothing + * answers the same value for every row, and one that reads a name runs + * per row. `exists` is true where what was written around it asks only + * whether it answered a row. + */ +export interface ZuScalarPlan { + readonly reads: string[] + readonly exists: boolean + readonly plan: ZuPlan +} + +/** + * What a statement would do, without doing it. + * + * A tree and a rendering of it. `text` is what the engine prints, so a + * listing logged from Node is the listing the shell shows, and the tree + * is for the questions a program asks: which tables were touched, how + * deep the expands go, whether the scan reached an index. + */ +export interface ZuPlan { + /** + * The top operator, and null for the plan with no operators at all, + * which is the one row a statement with no clauses runs over. + */ + readonly root: ZuPlanNode | null + /** The columns the statement answers with, in the order it wrote them. */ + readonly columns: string[] + /** The parameters it wants, without the `$` they are written with. */ + readonly params: string[] + /** What compiling it raised, which is empty for most statements. */ + readonly notes: string[] + readonly scalars: ZuScalarPlan[] + /** The listing, indented, as `EXPLAIN` prints it. */ + readonly text: string +} + +/** + * One operator of a profiled run, and what the counters saw of it. + */ +export interface ZuOp { + readonly op: string + readonly detail: string + /** How many chunks it produced. */ + readonly pulls: number + /** Values produced across every pull. Over `pulls` that is the + * average vector length, which is the factorization statistic. */ + readonly rows: number + /** + * The rows those values stand for with the factorization multiplied + * out. On a chain it is `rows`, and on a star it is the product over + * every vector still unflat beside this one, which is the count the + * optimizer was estimating. + */ + readonly flat: number + /** + * What the optimizer expected, and null for the operators that pass + * their input through rather than producing rows of their own. + */ + readonly estimate: number | null + /** The most rows the optimizer's ceiling allowed, where the + * statistics were there to set one. */ + readonly bound: number | null + /** Self time in nanoseconds, with the children's excluded. */ + readonly nanos: number + /** + * How wrong the estimate was: `max(estimate/rows, rows/estimate)`, + * both floored at one row. An operator the optimizer got right is 1, + * and null wherever `estimate` is. + */ + readonly qerror: number | null +} + +/** + * One stage of a profiled run: the operators bottom-up and the sink + * that took their rows. + */ +export interface ZuStage { + readonly sink: string + /** How many rows the sink was handed. */ + readonly rows: number + /** Wall time of the whole stage in nanoseconds, sink included. */ + readonly nanos: number + readonly ops: ZuOp[] +} + +/** + * What a statement did, with the counters on. + * + * The rows are not here: a profile is about the run rather than the + * answer, and keeping both would make the measurement pay for the thing + * it is measuring. `text` is the listing `EXPLAIN ANALYZE` prints. + */ +export interface ZuProfile { + readonly stages: ZuStage[] + /** Every stage end to end, in nanoseconds. */ + readonly nanos: number + readonly text: string +} + /** * What a streamed statement takes beside its parameters. */ @@ -806,6 +943,75 @@ export declare class Connection { * JavaScript values on the way. */ columnar(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise + /** + * Compiles a statement, pins it, and hands back something that + * runs it. + * + * ```js + * await using find = await conn.prepare('MATCH (p:person) WHERE p.id = $id RETURN p.name AS name') + * for (const id of ids) console.log(await find.query({ id })) + * ``` + * + * What this buys is not what it buys in a driver. A driver prepares + * to save a round trip and this database has none, and the plan for + * a statement is cached by its text either way, so a loop that + * writes the same query twice is not compiling it twice whether or + * not anybody prepared it. What preparing buys here is the compile + * happening at the line that asked for it, so a statement that does + * not compile is a failure at startup rather than on the first + * request that used it, and the parameter names coming back, so a + * program can bind by what the statement actually wants rather than + * by what somebody remembered writing. + * + * The id maps back to the text rather than to a plan, so a prepared + * statement crossing a catalog change recompiles instead of running + * a plan for a table that has since changed shape. + */ + prepare(statement: string): Promise + /** + * The plan this connection would run for a statement, without + * running it. + * + * ```js + * const plan = await conn.explain('MATCH (p:person) WHERE p.id = $id RETURN p.name AS name') + * console.log(plan.text) + * ``` + * + * A tree and a rendering of it, because the two questions a plan + * gets asked want different things: a person reading it wants the + * listing, and a program asking whether the scan reached an index + * or which tables were touched wants operators it can walk. They + * are one plan printed two ways rather than two answers that can + * drift, since `text` is what the engine renders from the same + * tree. + * + * No parameters, because a plan does not depend on the values bound + * to it: it depends on the names, and those are in `params`. A + * statement that does not compile fails here, which is most of why + * this is worth calling. + */ + explain(statement: string): Promise + /** + * Runs a statement with the counters on and gives back what they + * saw, rather than the rows. + * + * ```js + * const run = await conn.profile('MATCH (p:person)-[:knows]->(q) RETURN q.name AS name') + * console.log(run.text) + * ``` + * + * This is the call for a statement that is slower than its plan + * says it should be. Every operator carries what it really + * produced beside what the optimizer expected, and `qerror` is the + * ratio between them, so the operator whose estimate was wrong is + * the one to look at and `nanos` says whether being wrong cost + * anything. + * + * It costs the execution, since the way to find out what a + * statement does is to do it. A statement that writes is refused, + * because profiling it would apply the write. + */ + profile(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise /** * Runs one statement and gives back a cursor over its rows. * @@ -835,6 +1041,62 @@ export declare class Connection { dispose(): Promise } +/** + * A statement the connection has compiled and pinned. + * + * Take one with `Connection.prepare`, run it as often as you like, and + * close it. It runs the same three ways a connection does, `query`, + * `exec` and `columnar`, because a prepared statement is a statement + * and the way a caller wants its answer is not a decision the prepare + * should have made. + * + * There is no `stream`. A stream is the engine's `run_streaming`, which + * takes the text of a statement rather than a pinned id, so a streamed + * prepared statement would be this client re-running the text behind + * the caller's back and calling it prepared. `conn.stream(sql, params)` + * is that, honestly spelled. + */ +export declare class Prepared { + /** The statement, as it was written. */ + get statement(): string + /** + * The names this statement wants, in the order the binder assigned + * them, and without the `$` they are written with. + * + * Empty for a statement that takes none. A name in here that the + * caller does not bind is a failure at the run rather than a null, + * which is the engine's rule and the one worth relying on. + */ + get params(): Array + /** Whether this prepared statement has been closed. */ + get closed(): boolean + /** Runs it and gives back its rows. */ + query>(params?: Record | null, options?: ZuStatementOptions | null): Promise> + /** Runs it for its effect and gives back nothing. */ + exec(params?: Record | null, options?: ZuStatementOptions | null): Promise + /** Runs it and gives back its columns rather than its rows. */ + columnar(params?: Record | null, options?: ZuStatementOptions | null): Promise + /** + * Gives the id back to the session. + * + * Closing twice does nothing the second time, and closing one whose + * connection has already gone does nothing at all, because the + * session that was holding it went with it. Both of those are what + * makes an explicit close safe to write inside a block that an + * `await using` is also going to leave. + */ + close(): Promise + /** + * The close `await using` calls, which is the intended way to scope + * a prepared statement. + * + * It is also reachable as `Symbol.asyncDispose`, which is what + * `await using` actually looks for and which [`wire_disposal`] puts + * on every prepared statement as it is made. + */ + dispose(): Promise +} + /** * A transaction that has been started and not yet ended. * diff --git a/etc/zudb.api.md b/etc/zudb.api.md index 5e483ad..851fcbb 100644 --- a/etc/zudb.api.md +++ b/etc/zudb.api.md @@ -32,9 +32,12 @@ export class Connection { cursor(statement: string, params?: Record | null, options?: ZuStreamOptions | null): ZuCursor dispose(): Promise exec(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise + explain(statement: string): Promise get inTransaction(): boolean get open(): boolean get path(): string + prepare(statement: string): Promise + profile(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise query>(statement: string, params?: Record | null, options?: ZuStatementOptions | null): Promise> get readOnly(): boolean register(name: string, data: ZuFrame): Promise @@ -58,6 +61,18 @@ export function isZuError(value: unknown): value is ZuError // @public export function load(path: string, options: ZuLoadOptions): Promise +// @public +export class Prepared { + close(): Promise + get closed(): boolean + columnar(params?: Record | null, options?: ZuStatementOptions | null): Promise + dispose(): Promise + exec(params?: Record | null, options?: ZuStatementOptions | null): Promise + get params(): Array + query>(params?: Record | null, options?: ZuStatementOptions | null): Promise> + get statement(): string +} + // @public export class Transaction { commit(): Promise @@ -261,6 +276,21 @@ export interface ZuNotice { readonly message: string } +// @public +export interface ZuOp { + readonly bound: number | null + // (undocumented) + readonly detail: string + readonly estimate: number | null + readonly flat: number + readonly nanos: number + // (undocumented) + readonly op: string + readonly pulls: number + readonly qerror: number | null + readonly rows: number +} + // @public export type ZuParam = | null @@ -306,6 +336,38 @@ export type ZuPlainTime = typeof globalThis extends { ? Value : unknown +// @public +export interface ZuPlan { + readonly columns: string[] + readonly notes: string[] + readonly params: string[] + readonly root: ZuPlanNode | null + // (undocumented) + readonly scalars: ZuScalarPlan[] + readonly text: string +} + +// @public +export interface ZuPlanNode { + readonly binds: string[] + readonly bracket: 'Optional' | 'Semi' | 'Anti' | 'Mark' | null + // (undocumented) + readonly children: ZuPlanNode[] + readonly detail: string + readonly name: string + readonly op: string + readonly tables: string[] +} + +// @public +export interface ZuProfile { + readonly nanos: number + // (undocumented) + readonly stages: ZuStage[] + // (undocumented) + readonly text: string +} + // @public export class ZuRel { // (undocumented) @@ -329,6 +391,26 @@ export interface ZuRows> extends Array { readonly notices: ZuNotice[] } +// @public +export interface ZuScalarPlan { + // (undocumented) + readonly exists: boolean + // (undocumented) + readonly plan: ZuPlan + // (undocumented) + readonly reads: string[] +} + +// @public +export interface ZuStage { + readonly nanos: number + // (undocumented) + readonly ops: ZuOp[] + readonly rows: number + // (undocumented) + readonly sink: string +} + // @public export interface ZuStatementOptions { readonly bigIntMode?: ZuBigIntMode diff --git a/package.json b/package.json index fc9f2f0..094a98b 100644 --- a/package.json +++ b/package.json @@ -74,6 +74,7 @@ "bench:append": "node bench/append.mjs", "bench:columnar": "node bench/columnar.mjs", "bench:load": "node bench/load.mjs", + "bench:prepared": "node bench/prepared.mjs", "bench:register": "node bench/register.mjs", "bench:temporal": "node --harmony-temporal bench/query.mjs" }, diff --git a/src/conn.rs b/src/conn.rs index a17e7a8..607a7cf 100644 --- a/src/conn.rs +++ b/src/conn.rs @@ -36,6 +36,8 @@ use crate::append::OpenTask; use crate::cancel::Watch; use crate::columns::ColumnsTask; use crate::error::{aborted, raise, usage}; +use crate::plan::{PlanTask, ProfileTask}; +use crate::prepared::PrepareTask; use crate::register::{self, RegisterTask, RegisteredTask, UnregisterTask}; use crate::stream::{self, Started, ZuCursor}; use crate::temporal; @@ -573,6 +575,136 @@ impl Connection { AsyncTask::new(ColumnsTask(self.task(env, statement, params, options))) } + /// Compiles a statement, pins it, and hands back something that + /// runs it. + /// + /// ```js + /// await using find = await conn.prepare('MATCH (p:person) WHERE p.id = $id RETURN p.name AS name') + /// for (const id of ids) console.log(await find.query({ id })) + /// ``` + /// + /// What this buys is not what it buys in a driver. A driver prepares + /// to save a round trip and this database has none, and the plan for + /// a statement is cached by its text either way, so a loop that + /// writes the same query twice is not compiling it twice whether or + /// not anybody prepared it. What preparing buys here is the compile + /// happening at the line that asked for it, so a statement that does + /// not compile is a failure at startup rather than on the first + /// request that used it, and the parameter names coming back, so a + /// program can bind by what the statement actually wants rather than + /// by what somebody remembered writing. + /// + /// The id maps back to the text rather than to a plan, so a prepared + /// statement crossing a catalog change recompiles instead of running + /// a plan for a table that has since changed shape. + #[napi( + ts_args_type = "statement: string", + ts_return_type = "Promise" + )] + pub fn prepare(&self, statement: Unknown<'_>) -> AsyncTask { + let named = match self.alive.load(Ordering::Acquire) { + true => text(&statement, "statement"), + false => Err(CLOSED.to_string()), + }; + AsyncTask::new(PrepareTask::new( + self.handles(), + self.interrupt.clone(), + self.spelling, + named.as_deref().unwrap_or_default().to_string(), + named.err(), + )) + } + + /// The plan this connection would run for a statement, without + /// running it. + /// + /// ```js + /// const plan = await conn.explain('MATCH (p:person) WHERE p.id = $id RETURN p.name AS name') + /// console.log(plan.text) + /// ``` + /// + /// A tree and a rendering of it, because the two questions a plan + /// gets asked want different things: a person reading it wants the + /// listing, and a program asking whether the scan reached an index + /// or which tables were touched wants operators it can walk. They + /// are one plan printed two ways rather than two answers that can + /// drift, since `text` is what the engine renders from the same + /// tree. + /// + /// No parameters, because a plan does not depend on the values bound + /// to it: it depends on the names, and those are in `params`. A + /// statement that does not compile fails here, which is most of why + /// this is worth calling. + #[napi(ts_args_type = "statement: string", ts_return_type = "Promise")] + pub fn explain(&self, statement: Unknown<'_>) -> AsyncTask { + let named = match self.alive.load(Ordering::Acquire) { + true => text(&statement, "statement"), + false => Err(CLOSED.to_string()), + }; + AsyncTask::new(PlanTask::new( + self.handles(), + named.as_deref().unwrap_or_default().to_string(), + named.err(), + )) + } + + /// Runs a statement with the counters on and gives back what they + /// saw, rather than the rows. + /// + /// ```js + /// const run = await conn.profile('MATCH (p:person)-[:knows]->(q) RETURN q.name AS name') + /// console.log(run.text) + /// ``` + /// + /// This is the call for a statement that is slower than its plan + /// says it should be. Every operator carries what it really + /// produced beside what the optimizer expected, and `qerror` is the + /// ratio between them, so the operator whose estimate was wrong is + /// the one to look at and `nanos` says whether being wrong cost + /// anything. + /// + /// It costs the execution, since the way to find out what a + /// statement does is to do it. A statement that writes is refused, + /// because profiling it would apply the write. + #[napi( + ts_args_type = "statement: string, params?: Record | null, options?: ZuStatementOptions | null", + ts_return_type = "Promise" + )] + pub fn profile( + &self, + env: &Env, + statement: Unknown<'_>, + params: Option>, + options: Option>, + ) -> AsyncTask { + // Read here rather than on the threadpool thread, for the reason + // every other statement's are: reading a JavaScript value is + // something only the thread that owns the runtime may do, and so + // is adding the listener the signal is watched through. + let bound = if self.alive.load(Ordering::Acquire) { + text(&statement, "statement").and_then(|statement| { + Ok(( + statement, + bind(env, params)?, + watch(env, options, self.interrupt.clone())?, + )) + }) + } else { + Err(CLOSED.to_string()) + }; + let (statement, params, watch, refused) = match bound { + Ok((statement, params, watch)) => (statement, params, watch, None), + Err(message) => (String::new(), Vec::new(), None, Some(message)), + }; + AsyncTask::new(ProfileTask::new( + self.handles(), + statement, + params, + watch, + refused, + )) + } + /// Runs one statement and gives back a cursor over its rows. /// /// The pull underneath `stream`, which is what a program uses. The @@ -680,15 +812,22 @@ impl Connection { Some(message), ), }; - QueryTask { - inner: Arc::clone(&self.inner), - alive: Arc::clone(&self.alive), - in_txn: Arc::clone(&self.in_txn), - statement, + QueryTask::new( + self.handles(), + Source::Text(statement), params, spelling, watch, refused, + ) + } + + /// The three handles a call on this connection runs against. + pub(crate) fn handles(&self) -> Handles { + Handles { + inner: Arc::clone(&self.inner), + alive: Arc::clone(&self.alive), + in_txn: Arc::clone(&self.in_txn), } } @@ -952,7 +1091,10 @@ fn flag(options: Option<&Object<'_>>, name: &str) -> std::result::Result>, connection: Ints) -> std::result::Result { +pub(crate) fn int_mode( + options: Option<&Object<'_>>, + connection: Ints, +) -> std::result::Result { let Some(options) = options else { return Ok(connection); }; @@ -1052,6 +1194,32 @@ pub(crate) fn bind( Ok(bound) } +/// What a task runs: the text of a statement, or the id a prepared one +/// was pinned under. +/// +/// One enum rather than two tasks, because everything after the call +/// that starts a statement is the same either way: the same lock, the +/// same signal, the same rows, the same columns. So `query`, `exec` and +/// `columnar` are written once and a prepared statement reaches all +/// three by handing them an id instead of a string. +pub(crate) enum Source { + Text(String), + Prepared(u64), +} + +/// The three handles every call on a connection runs against. +/// +/// They are counted rather than borrowed because the threadpool thread +/// a statement runs on takes a share of each: a prepared statement, an +/// appender and a task all outlive the call that made them in the type +/// system even though none of them does in fact. +#[derive(Clone)] +pub(crate) struct Handles { + pub(crate) inner: Arc>>, + pub(crate) alive: Arc, + pub(crate) in_txn: Arc, +} + pub struct QueryTask { inner: Arc>>, /// Whether the connection is still open, which is what tells an @@ -1060,7 +1228,7 @@ pub struct QueryTask { /// Where this statement writes whether the connection is inside a /// transaction now that it has run. in_txn: Arc, - statement: String, + source: Source, params: Vec<(String, Value)>, /// How this statement spells the values it gives back. spelling: Spelling, @@ -1071,6 +1239,27 @@ pub struct QueryTask { } impl QueryTask { + /// One statement, ready to run or ready to say why it will not. + pub(crate) fn new( + handles: Handles, + source: Source, + params: Vec<(String, Value)>, + spelling: Spelling, + watch: Option, + refused: Option, + ) -> QueryTask { + QueryTask { + inner: handles.inner, + alive: handles.alive, + in_txn: handles.in_txn, + source, + params, + spelling, + watch, + refused, + } + } + /// Runs the statement, with the names of the tables it saw. /// /// The names are read while the lock is held, because a catalog @@ -1080,8 +1269,8 @@ impl QueryTask { if let Some(message) = self.refused.take() { return Err(Failure::Usage(message)); } - let (statement, params, spelling, watch) = - (&self.statement, &self.params, self.spelling, &self.watch); + let (source, params, spelling, watch) = + (&self.source, &self.params, self.spelling, &self.watch); with(&self.inner, &self.alive, &self.in_txn, move |conn| { // From here the connection is this statement's, so this is // where a signal can start stopping it and where it stops @@ -1099,7 +1288,14 @@ impl QueryTask { .map(|(name, value)| (name.as_str(), value.clone())) .collect(); let shape = Shape::of(conn.session_mut().catalog(), spelling); - let result = conn.query_with(statement, ¶ms); + let result = match source { + Source::Text(statement) => conn.query_with(statement, ¶ms), + // The id maps back to the text the session pinned, so a + // catalog change between the prepare and this recompiles + // rather than running a plan that describes a table that + // has since changed shape. + Source::Prepared(id) => conn.execute_prepared(*id, ¶ms), + }; if let Some(watch) = watch { watch.leave(); // An interrupt is the engine's answer to somebody having @@ -1217,7 +1413,7 @@ pub(crate) fn beside(env: &Env, name: &str, value: T) -> Result< } /// The same statement with the rows thrown away. -pub struct ExecTask(QueryTask); +pub struct ExecTask(pub(crate) QueryTask); impl<'task> ScopedTask<'task> for ExecTask { type Output = std::result::Result<(), Failure>; diff --git a/src/lib.rs b/src/lib.rs index a62c2a5..0c73ad1 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -24,6 +24,8 @@ mod conn; mod error; mod frame; mod load; +mod plan; +mod prepared; mod register; mod stream; mod temporal; diff --git a/src/plan.rs b/src/plan.rs new file mode 100644 index 0000000..5d3a83c --- /dev/null +++ b/src/plan.rs @@ -0,0 +1,281 @@ +//! What a statement would do, and what it did. +//! +//! ```js +//! console.log((await conn.explain('MATCH (p:person) RETURN p.id AS id')).text) +//! console.log((await conn.profile('MATCH (p:person) RETURN p.id AS id')).text) +//! ``` +//! +//! Two calls with one shape between them. `explain` compiles and stops, +//! so it costs the compile and answers the operators the statement would +//! have run. `profile` runs it with the counters on and answers the same +//! operators with what each of them really did: how many times it was +//! pulled, how many rows it produced, how many the optimizer thought it +//! would, and where the wall clock went. +//! +//! ## Why both a tree and a string +//! +//! The two questions a plan gets asked want different answers. A person +//! looking at a slow query wants the listing, indented, the way every +//! database prints one, and building that out of a tree in JavaScript is +//! twenty lines nobody should write twice. A program asking whether the +//! scan reached an index, how deep the expands go, or which tables a +//! statement touches wants operators it can walk. +//! +//! So both are here, and `text` is the engine's own rendering of the +//! same tree rather than something this client assembles, which is what +//! keeps the two from drifting: the listing a Node program prints is +//! character for character the listing the shell prints. +//! +//! ## The numbers +//! +//! Counts are `number` and not `bigint`, which is the one place this +//! client spells an integer as a double on purpose. Nothing a profile +//! counts comes near 2^53: a statement that pulled nine quadrillion rows +//! is not a statement anybody is profiling. Times are nanoseconds, as +//! integers, for the same reason and with a lot more room. +//! +//! `estimate` and `bound` are null where the optimizer had nothing to +//! say, which is the operators that pass their input through rather than +//! producing rows of their own, and `qerror` is null wherever `estimate` +//! is. Where it is a number it is `max(estimate/rows, rows/estimate)` +//! with both sides floored at one row, so an operator the optimizer got +//! right is 1 and the one to look at is the one furthest from it. + +use napi::bindgen_prelude::*; +use napi::{Env, ScopedTask}; +use zudb::query::Value; +use zudb::{OpProfile, PlanNode, Profile, QueryPlan, StageProfile, ZuError}; + +use crate::cancel::Watch; +use crate::conn::{Failure, Handles, failed, with}; + +/// Compiling a statement to see what it would do. +pub struct PlanTask { + handles: Handles, + statement: String, + /// Why this is not going to run, when it is not. + refused: Option, +} + +impl PlanTask { + pub(crate) fn new(handles: Handles, statement: String, refused: Option) -> PlanTask { + PlanTask { + handles, + statement, + refused, + } + } +} + +impl<'task> ScopedTask<'task> for PlanTask { + type Output = std::result::Result; + type JsValue = Object<'task>; + + fn compute(&mut self) -> Result { + if let Some(message) = self.refused.take() { + return Ok(Err(Failure::Usage(message))); + } + let statement = self.statement.clone(); + let handles = &self.handles; + Ok(with( + &handles.inner, + &handles.alive, + &handles.in_txn, + |conn| Ok(conn.explain_plan(&statement)?), + )) + } + + fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { + let plan = output.map_err(|failure| failed(env, failure, None))?; + planned(env, &plan) + } +} + +/// Running one with the counters on. +pub struct ProfileTask { + handles: Handles, + statement: String, + params: Vec<(String, Value)>, + /// The signal watching this run, when the caller gave one. + watch: Option, + refused: Option, +} + +impl ProfileTask { + pub(crate) fn new( + handles: Handles, + statement: String, + params: Vec<(String, Value)>, + watch: Option, + refused: Option, + ) -> ProfileTask { + ProfileTask { + handles, + statement, + params, + watch, + refused, + } + } +} + +impl<'task> ScopedTask<'task> for ProfileTask { + type Output = std::result::Result; + type JsValue = Object<'task>; + + fn compute(&mut self) -> Result { + if let Some(message) = self.refused.take() { + return Ok(Err(Failure::Usage(message))); + } + let (statement, params, watch) = (&self.statement, &self.params, &self.watch); + let handles = &self.handles; + Ok(with( + &handles.inner, + &handles.alive, + &handles.in_txn, + move |conn| { + // A profile is a run, so it is stoppable exactly the way + // a run is: the signal is entered when the connection + // becomes this call's and left when it stops being. + if let Some(watch) = watch + && !watch.enter() + { + watch.leave(); + return Err(Failure::Aborted); + } + let params: Vec<(&str, Value)> = params + .iter() + .map(|(name, value)| (name.as_str(), value.clone())) + .collect(); + let out = conn.profile(statement, ¶ms); + if let Some(watch) = watch { + watch.leave(); + if watch.asked() && matches!(out, Err(ZuError::Interrupted)) { + return Err(Failure::Aborted); + } + } + Ok(out?) + }, + )) + } + + fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { + let profile = output.map_err(|failure| failed(env, failure, self.watch.as_ref()))?; + profiled(env, &profile) + } + + fn finally(mut self, env: Env) -> Result<()> { + match self.watch.take() { + Some(watch) => watch.release(&env), + None => Ok(()), + } + } +} + +/// A whole plan as the object a caller reads. +fn planned<'env>(env: &'env Env, plan: &QueryPlan) -> Result> { + let mut object = Object::new(env)?; + object.set( + "root", + match &plan.root { + Some(root) => Some(operator(env, root)?), + None => None, + }, + )?; + object.set("columns", strings(env, &plan.columns)?)?; + object.set("params", strings(env, &plan.params)?)?; + object.set("notes", strings(env, &plan.notes)?)?; + let mut scalars = env.create_array(plan.scalars.len() as u32)?; + for (ix, scalar) in plan.scalars.iter().enumerate() { + let mut held = Object::new(env)?; + held.set("reads", strings(env, &scalar.reads)?)?; + held.set("exists", scalar.exists)?; + held.set("plan", planned(env, &scalar.plan)?)?; + scalars.set(ix as u32, held)?; + } + object.set("scalars", scalars)?; + // Rendered by the engine rather than assembled here, so the listing + // a program prints is the listing the shell prints. + object.set("text", plan.render())?; + Ok(object) +} + +/// One operator and everything under it. +/// +/// `op` is the operator and `name` is what the listing calls it, which +/// differ only where the operator sits inside a bracket: an OPTIONAL +/// MATCH expand is an `Expand` named `OptionalExpand`. A program +/// grouping by operator wants the first and a program printing wants the +/// second, so both are here and neither has to be derived. +fn operator<'env>(env: &'env Env, node: &PlanNode) -> Result> { + let mut object = Object::new(env)?; + object.set("op", node.op)?; + object.set("name", node.name())?; + object.set("bracket", node.bracket.as_ref().map(|b| b.prefix()))?; + object.set("detail", node.detail.as_str())?; + object.set("binds", strings(env, &node.binds)?)?; + object.set("tables", strings(env, &node.tables)?)?; + let mut children = env.create_array(node.children.len() as u32)?; + for (ix, child) in node.children.iter().enumerate() { + children.set(ix as u32, operator(env, child)?)?; + } + object.set("children", children)?; + Ok(object) +} + +/// A whole profile as the object a caller reads. +fn profiled<'env>(env: &'env Env, profile: &Profile) -> Result> { + let mut stages = env.create_array(profile.stages.len() as u32)?; + for (ix, stage) in profile.stages.iter().enumerate() { + stages.set(ix as u32, staged(env, stage)?)?; + } + let mut object = Object::new(env)?; + object.set("stages", stages)?; + // The stages end to end, which is what a caller comparing two runs + // reaches for and the one number the engine's own listing does not + // print on a line of its own. + object.set( + "nanos", + profile.stages.iter().map(|stage| stage.nanos).sum::() as f64, + )?; + object.set("text", profile.render())?; + Ok(object) +} + +/// One stage: the operators bottom-up and the sink that took their rows. +fn staged<'env>(env: &'env Env, stage: &StageProfile) -> Result> { + let mut ops = env.create_array(stage.ops.len() as u32)?; + for (ix, op) in stage.ops.iter().enumerate() { + ops.set(ix as u32, counted(env, op)?)?; + } + let mut object = Object::new(env)?; + object.set("sink", stage.sink.as_str())?; + object.set("rows", stage.out_rows as f64)?; + object.set("nanos", stage.nanos as f64)?; + object.set("ops", ops)?; + Ok(object) +} + +/// One operator and what the counters saw of it. +fn counted<'env>(env: &'env Env, op: &OpProfile) -> Result> { + let mut object = Object::new(env)?; + object.set("op", op.kind)?; + object.set("detail", op.detail.as_str())?; + object.set("pulls", op.pulls as f64)?; + object.set("rows", op.rows as f64)?; + // What the vectors stand for with the factorization multiplied out, + // which is the count to compare against an estimate: on a chain it + // is `rows` and on a star it is the product of the vectors beside + // this one, and the optimizer was estimating the latter. + object.set("flat", op.flat as f64)?; + object.set("estimate", op.est)?; + object.set("bound", op.bnd)?; + object.set("nanos", op.nanos as f64)?; + object.set("qerror", op.qerror())?; + Ok(object) +} + +/// A list of names, which is most of what a plan is made of. +fn strings<'env>(env: &'env Env, held: &[String]) -> Result> { + Array::from_ref_vec_string(env, held) +} diff --git a/src/prepared.rs b/src/prepared.rs new file mode 100644 index 0000000..fbcabe5 --- /dev/null +++ b/src/prepared.rs @@ -0,0 +1,339 @@ +//! A statement compiled once and run many times. +//! +//! ```js +//! await using find = await conn.prepare('MATCH (p:person) WHERE p.id = $id RETURN p.name AS name') +//! for (const id of ids) console.log(await find.query({ id })) +//! ``` +//! +//! ## What this is for, since it is not what it is for in a driver +//! +//! A driver prepares to save a round trip: the text goes to the server +//! once and the values go every time. There is no server here and no +//! round trip to save, and the engine caches the plan for a statement by +//! its text, so the second `conn.query` of the same string is already +//! not compiling it a second time. A benchmark that expects preparing to +//! be faster than not preparing is going to be disappointed, and +//! `bench/prepared.mjs` prints the two side by side rather than +//! pretending otherwise. +//! +//! What it is for is the two things the plan cache cannot do. The +//! compile happens at the call that asked for it, so a statement with a +//! typo in it fails when the program starts rather than on the first +//! request that reached it, which is the difference between a deploy +//! that fails and a deploy that pages somebody. And the parameter names +//! come back, in the order the binder assigned them, so a program can +//! bind what the statement actually wants rather than what somebody +//! remembered writing. +//! +//! ## What is pinned +//! +//! The id maps back to the text of the statement rather than to a plan, +//! so a prepared statement that outlives a catalog change recompiles +//! instead of running a plan describing a table that has since changed +//! shape. That is the engine's decision and it is the right one: a +//! prepared statement in a long-lived process is exactly the thing most +//! likely to be holding a stale plan. +//! +//! A prepared statement belongs to the connection that made it and dies +//! with it. Closing one gives its id back to the session; leaving one +//! open leaks a string until the connection closes, which is why this +//! answers `Symbol.asyncDispose` and why `await using` is how the +//! examples spell it. + +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; + +use napi::bindgen_prelude::*; +use napi::{Env, ScopedTask}; +use napi_derive::napi; +use zudb::Interrupt; + +use crate::columns::ColumnsTask; +use crate::conn::{ + CLOSED, ExecTask, Failure, Handles, QueryTask, Source, bind, failed, int_mode, watch, + wire_disposal, with, +}; +use crate::value::Spelling; + +/// A statement the connection has compiled and pinned. +/// +/// Take one with `Connection.prepare`, run it as often as you like, and +/// close it. It runs the same three ways a connection does, `query`, +/// `exec` and `columnar`, because a prepared statement is a statement +/// and the way a caller wants its answer is not a decision the prepare +/// should have made. +/// +/// There is no `stream`. A stream is the engine's `run_streaming`, which +/// takes the text of a statement rather than a pinned id, so a streamed +/// prepared statement would be this client re-running the text behind +/// the caller's back and calling it prepared. `conn.stream(sql, params)` +/// is that, honestly spelled. +#[napi] +pub struct Prepared { + /// The same handles every statement on this connection uses, rather + /// than a reference to the JavaScript object, so a prepared + /// statement whose `Connection` was collected still has somewhere to + /// run. + handles: Handles, + /// The word this statement reads at every boundary, for a caller who + /// passes a signal to one of the runs. + interrupt: Interrupt, + /// How this statement spells the values it gives back, taken from + /// the connection when it was prepared. A single run may say + /// otherwise about the integers, the same way a single query may. + spelling: Spelling, + statement: String, + /// The names the binder assigned, in the order it assigned them. + params: Vec, + /// The id the session pinned this under. + id: u64, + /// Whether it is still pinned, kept beside the connection's lock + /// rather than inside it, so asking costs nothing and never queues + /// behind the statement being asked about. + open: Arc, +} + +#[napi] +impl Prepared { + /// The statement, as it was written. + #[napi(getter)] + pub fn statement(&self) -> String { + self.statement.clone() + } + + /// The names this statement wants, in the order the binder assigned + /// them, and without the `$` they are written with. + /// + /// Empty for a statement that takes none. A name in here that the + /// caller does not bind is a failure at the run rather than a null, + /// which is the engine's rule and the one worth relying on. + #[napi(getter)] + pub fn params(&self) -> Vec { + self.params.clone() + } + + /// Whether this prepared statement has been closed. + #[napi(getter)] + pub fn closed(&self) -> bool { + !self.open.load(Ordering::Acquire) + } + + /// Runs it and gives back its rows. + #[napi( + ts_generic_types = "Row = Record", + ts_args_type = "params?: Record | null, options?: ZuStatementOptions | null", + ts_return_type = "Promise>" + )] + pub fn query( + &self, + env: &Env, + params: Option>, + options: Option>, + ) -> AsyncTask { + AsyncTask::new(self.task(env, params, options)) + } + + /// Runs it for its effect and gives back nothing. + #[napi( + ts_args_type = "params?: Record | null, options?: ZuStatementOptions | null", + ts_return_type = "Promise" + )] + pub fn exec( + &self, + env: &Env, + params: Option>, + options: Option>, + ) -> AsyncTask { + AsyncTask::new(ExecTask(self.task(env, params, options))) + } + + /// Runs it and gives back its columns rather than its rows. + #[napi( + ts_args_type = "params?: Record | null, options?: ZuStatementOptions | null", + ts_return_type = "Promise" + )] + pub fn columnar( + &self, + env: &Env, + params: Option>, + options: Option>, + ) -> AsyncTask { + AsyncTask::new(ColumnsTask(self.task(env, params, options))) + } + + /// Gives the id back to the session. + /// + /// Closing twice does nothing the second time, and closing one whose + /// connection has already gone does nothing at all, because the + /// session that was holding it went with it. Both of those are what + /// makes an explicit close safe to write inside a block that an + /// `await using` is also going to leave. + #[napi(ts_return_type = "Promise")] + pub fn close(&self) -> AsyncTask { + AsyncTask::new(CloseTask { + handles: self.handles.clone(), + id: self.id, + open: Arc::clone(&self.open), + }) + } + + /// The close `await using` calls, which is the intended way to scope + /// a prepared statement. + /// + /// It is also reachable as `Symbol.asyncDispose`, which is what + /// `await using` actually looks for and which [`wire_disposal`] puts + /// on every prepared statement as it is made. + #[napi(ts_return_type = "Promise")] + pub fn dispose(&self) -> AsyncTask { + self.close() + } + + /// The task one run of this statement is, whether or not it is going + /// to work. + /// + /// The same [`QueryTask`] a `conn.query` builds, carrying an id + /// instead of a string, which is what lets a prepared statement have + /// all three shapes of answer without any of the three being written + /// twice. + fn task( + &self, + env: &Env, + params: Option>, + options: Option>, + ) -> QueryTask { + // The parameters are read here, on the thread that owns the + // runtime, because reading a JavaScript value is something no + // other thread may do. So is adding the listener the signal is + // watched through. + let bound = if !self.open.load(Ordering::Acquire) { + Err(FINISHED.to_string()) + } else if !self.handles.alive.load(Ordering::Acquire) { + Err(CLOSED.to_string()) + } else { + int_mode(options.as_ref(), self.spelling.ints).and_then(|ints| { + Ok(( + bind(env, params)?, + Spelling { + ints, + ..self.spelling + }, + watch(env, options, self.interrupt.clone())?, + )) + }) + }; + let (params, spelling, watch, refused) = match bound { + Ok((params, spelling, watch)) => (params, spelling, watch, None), + Err(message) => (Vec::new(), self.spelling, None, Some(message)), + }; + QueryTask::new( + self.handles.clone(), + Source::Prepared(self.id), + params, + spelling, + watch, + refused, + ) + } +} + +/// Compiling one, which is the call that finds out whether the statement +/// is a statement at all. +pub struct PrepareTask { + handles: Handles, + interrupt: Interrupt, + spelling: Spelling, + statement: String, + /// Why this is not going to run, when it is not. + refused: Option, +} + +impl PrepareTask { + pub(crate) fn new( + handles: Handles, + interrupt: Interrupt, + spelling: Spelling, + statement: String, + refused: Option, + ) -> PrepareTask { + PrepareTask { + handles, + interrupt, + spelling, + statement, + refused, + } + } +} + +impl<'task> ScopedTask<'task> for PrepareTask { + type Output = std::result::Result<(u64, Vec), Failure>; + type JsValue = ClassInstance<'task, Prepared>; + + fn compute(&mut self) -> Result { + if let Some(message) = self.refused.take() { + return Ok(Err(Failure::Usage(message))); + } + let statement = self.statement.clone(); + let handles = &self.handles; + Ok(with( + &handles.inner, + &handles.alive, + &handles.in_txn, + |conn| Ok(conn.prepare(&statement)?), + )) + } + + fn resolve(&mut self, env: &'task Env, output: Self::Output) -> Result { + let (id, params) = output.map_err(|failure| failed(env, failure, None))?; + let mut instance = Prepared { + handles: self.handles.clone(), + interrupt: self.interrupt.clone(), + spelling: self.spelling, + statement: self.statement.clone(), + params, + id, + open: Arc::new(AtomicBool::new(true)), + } + .into_instance(env)?; + wire_disposal(env, &mut instance, "dispose")?; + Ok(instance) + } +} + +/// Giving one back, off the event loop because it takes the connection's +/// lock and a statement may be holding it. +pub struct CloseTask { + handles: Handles, + id: u64, + open: Arc, +} + +impl<'task> ScopedTask<'task> for CloseTask { + type Output = (); + type JsValue = (); + + fn compute(&mut self) -> Result { + if !self.open.swap(false, Ordering::AcqRel) { + return Ok(()); + } + let handles = &self.handles; + // A connection that has already closed took the session and the + // statement with it, so there is nothing to give back and + // nothing to complain about. Every other reason the connection + // cannot be had, a poisoned lock or a stream still reading, is + // the same: the id outlives none of them. + let _ = with(&handles.inner, &handles.alive, &handles.in_txn, |conn| { + conn.close_prepared(self.id); + Ok(()) + }); + Ok(()) + } + + fn resolve(&mut self, _env: &'task Env, _output: Self::Output) -> Result { + Ok(()) + } +} + +/// What a prepared statement that has already been closed says. +const FINISHED: &str = "this prepared statement is closed, and a closed one has given its \ + statement back to the connection"; diff --git a/test/exports.test.mjs b/test/exports.test.mjs index 83edef9..fdb348e 100644 --- a/test/exports.test.mjs +++ b/test/exports.test.mjs @@ -25,6 +25,7 @@ const SURFACE = [ 'Connection', 'Transaction', 'Appender', + 'Prepared', 'ZuStream', 'ZuCursor', 'ZuDate', diff --git a/test/plan.test.mjs b/test/plan.test.mjs new file mode 100644 index 0000000..ef92b37 --- /dev/null +++ b/test/plan.test.mjs @@ -0,0 +1,309 @@ +// What a statement would do, and what it did. +// +// A plan is the engine's and this client only carries it, so what these +// assert is that the carrying is faithful: every operator, in the shape +// the tree had, with the fields that mean something and null where the +// engine had nothing to say. The listing is asserted against the tree it +// was rendered from rather than against a string written here, because a +// test that pins the exact words would fail every time the optimizer +// learns to print one better. + +import assert from 'node:assert/strict' +import test from 'node:test' + +import { fresh, isZuError, twoPeople } from './helper.mjs' + +const BY_NAME = 'MATCH (p:person) WHERE p.name = $name RETURN p.id AS id' + +// Every operator of a plan, depth first, which is the order the listing +// prints them in. +function operators(node) { + return node === null ? [] : [node, ...node.children.flatMap(operators)] +} + +// A database with an edge in it, since a plan is only interesting once +// there is something to expand. +async function twoPeopleWhoKnow(t) { + const made = await twoPeople(t) + await made.conn.exec('MATCH (a:person), (b:person) INSERT (a)-[:knows]->(b)') + return made +} + +test('a plan is the tree of operators the statement would run', async (t) => { + const { conn } = await twoPeople(t) + + const plan = await conn.explain(BY_NAME) + + assert.deepEqual( + operators(plan.root).map((op) => op.op), + ['Project', 'Filter', 'ScanNodes'], + ) + assert.deepEqual(plan.columns, ['id']) + assert.deepEqual(plan.params, ['name']) + assert.deepEqual(plan.notes, []) + assert.deepEqual(plan.scalars, []) +}) + +test('an operator carries what it works on, what it binds and what it touches', async (t) => { + const { conn } = await twoPeople(t) + + const plan = await conn.explain(BY_NAME) + const [project, filter, scan] = operators(plan.root) + + assert.equal(project.detail, 'p.id AS id') + assert.deepEqual(project.binds, ['id']) + assert.deepEqual(project.tables, []) + + assert.equal(filter.detail, 'p.name = $name') + assert.deepEqual(filter.binds, []) + + assert.equal(scan.detail, 'p: person') + assert.deepEqual(scan.binds, ['p']) + assert.deepEqual(scan.tables, ['person']) + assert.deepEqual(scan.children, []) +}) + +test('an operator inside a bracket is named for the bracket and is not it', async (t) => { + const { conn } = await twoPeopleWhoKnow(t) + + const plan = await conn.explain( + 'MATCH (a:person) OPTIONAL MATCH (a)-[:knows]->(b:person) RETURN a.name AS a, b.name AS b', + ) + const expand = operators(plan.root).find((op) => op.op === 'Expand') + + assert.equal(expand.op, 'Expand') + assert.equal(expand.name, 'OptionalExpand') + assert.equal(expand.bracket, 'Optional') + assert.deepEqual(expand.tables, ['knows']) +}) + +test('an operator outside a bracket has none, and is named for itself', async (t) => { + const { conn } = await twoPeopleWhoKnow(t) + + const plan = await conn.explain('MATCH (a:person)-[:knows]->(b:person) RETURN a.name AS a') + const expand = operators(plan.root).find((op) => op.op === 'Expand') + + assert.equal(expand.name, 'Expand') + assert.equal(expand.bracket, null) +}) + +test('the text is the listing, and it is the tree', async (t) => { + const { conn } = await twoPeople(t) + + const plan = await conn.explain(BY_NAME) + + assert.equal(plan.text, 'Project p.id AS id\n Filter p.name = $name\n ScanNodes p: person\n') + // Written twice on purpose: the listing is what a person reads and + // the tree is what a program walks, and this is the one assertion + // that says they describe the same plan. + assert.deepEqual( + plan.text + .trimEnd() + .split('\n') + .map((line) => line.trim().split(' ')[0]), + operators(plan.root).map((op) => op.name), + ) +}) + +test('a query written where a value belongs is a plan of its own', async (t) => { + const { conn } = await twoPeople(t) + + const plan = await conn.explain( + 'MATCH (p:person) RETURN VALUE { MATCH (q:person) WHERE q.name = p.name RETURN q.id LIMIT 1 } AS v', + ) + + assert.equal(plan.scalars.length, 1) + const [scalar] = plan.scalars + // It reads a name from the query around it, which is the whole test + // for whether it runs once or once a row. + assert.deepEqual(scalar.reads, ['p']) + assert.equal(scalar.exists, false) + assert.equal(scalar.plan.root.op, 'Limit') + assert.ok(scalar.plan.text.includes('ScanNodes q: person')) +}) + +test('a subquery that reads nothing runs once and says so by reading nothing', async (t) => { + const { conn } = await twoPeople(t) + + const plan = await conn.explain( + 'MATCH (p:person) RETURN VALUE { MATCH (q:person) RETURN q.id LIMIT 1 } AS v', + ) + + assert.deepEqual(plan.scalars[0].reads, []) + assert.ok(plan.text.includes('(once)')) +}) + +test('explaining does not run the statement', async (t) => { + const { conn } = await twoPeople(t) + + await conn.explain("INSERT (p:person {id: 3, name: 'ida'})") + + const rows = await conn.query('MATCH (p:person) RETURN count(*) AS n') + assert.equal(rows[0].n, 2n) +}) + +test('a statement that does not compile fails at the explain', async (t) => { + const { conn } = await twoPeople(t) + + await assert.rejects( + () => conn.explain('MATCH ('), + (err) => isZuError(err, 'ZuSyntaxError'), + ) +}) + +test('explaining on a closed connection is refused', async (t) => { + const { conn } = await fresh(t) + conn.close() + + await assert.rejects( + () => conn.explain('MATCH (p:person) RETURN p.name AS name'), + (err) => isZuError(err, 'ZuUsageError') && err.message.includes('the connection is closed'), + ) +}) + +test('a profile is what the operators really did', async (t) => { + const { conn } = await twoPeople(t) + + const run = await conn.profile('MATCH (p:person) RETURN p.name AS name') + + assert.equal(run.stages.length, 1) + const [stage] = run.stages + assert.equal(stage.sink, 'Project') + assert.equal(stage.rows, 2) + assert.ok(stage.nanos > 0) + assert.deepEqual( + stage.ops.map((op) => op.op), + ['Source', 'Scan'], + ) + + const scan = stage.ops.find((op) => op.op === 'Scan') + assert.equal(scan.detail, 'p: person') + assert.equal(scan.pulls, 1) + assert.equal(scan.rows, 2) + assert.equal(scan.flat, 2) + assert.equal(scan.estimate, 2) + // The optimizer was right about a table it has the statistics for, + // which is what a q-error of one means. + assert.equal(scan.qerror, 1) + assert.ok(scan.nanos > 0) +}) + +test('an operator the optimizer has nothing to say about carries nulls', async (t) => { + const { conn } = await twoPeople(t) + + const run = await conn.profile('MATCH (p:person) RETURN p.name AS name') + const source = run.stages[0].ops.find((op) => op.op === 'Source') + + assert.equal(source.estimate, null) + assert.equal(source.bound, null) + assert.equal(source.qerror, null) +}) + +test('the profile totals its stages and prints them', async (t) => { + const { conn } = await twoPeople(t) + + const run = await conn.profile('MATCH (p:person) RETURN p.name AS name') + + assert.equal( + run.nanos, + run.stages.reduce((total, stage) => total + stage.nanos, 0), + ) + assert.ok(run.text.startsWith('stage 1: Project')) + assert.ok(run.text.includes('Scan p: person')) +}) + +test('the counts are numbers and the times are whole nanoseconds', async (t) => { + const { conn } = await twoPeople(t) + + const run = await conn.profile('MATCH (p:person) RETURN p.name AS name') + + for (const stage of run.stages) { + assert.equal(typeof stage.rows, 'number') + assert.equal(typeof stage.nanos, 'number') + assert.equal(stage.nanos % 1, 0) + for (const op of stage.ops) { + for (const field of ['pulls', 'rows', 'flat', 'nanos']) { + assert.equal(typeof op[field], 'number', `${op.op}.${field}`) + assert.equal(op[field] % 1, 0, `${op.op}.${field}`) + } + } + } +}) + +test('a profile binds its parameters', async (t) => { + const { conn } = await twoPeople(t) + + const run = await conn.profile(BY_NAME, { name: 'ada' }) + const filter = run.stages[0].ops.find((op) => op.op === 'Filter') + + assert.equal(filter.detail, 'p.name = $name') + assert.equal(run.stages[0].rows, 1) +}) + +test('a profile that binds nothing fails the way the statement would', async (t) => { + const { conn } = await twoPeople(t) + + await assert.rejects( + () => conn.profile(BY_NAME), + (err) => isZuError(err, 'ZuSyntaxError') && err.message.includes('$name'), + ) +}) + +test('an expand shows up as its own operator with the rows it walked', async (t) => { + const { conn } = await twoPeopleWhoKnow(t) + + const run = await conn.profile( + 'MATCH (a:person)-[:knows]->(b:person) RETURN a.name AS a, b.name AS b', + ) + const expand = run.stages[0].ops.find((op) => op.op === 'Expand') + + assert.ok(expand.detail.includes('knows')) + assert.equal(run.stages[0].rows, 4) +}) + +test('a statement that writes is refused rather than profiled', async (t) => { + const { conn } = await twoPeople(t) + + await assert.rejects( + () => conn.profile("INSERT (p:person {id: 3, name: 'ida'})"), + (err) => err.message.includes('profiling a statement that writes'), + ) + + const rows = await conn.query('MATCH (p:person) RETURN count(*) AS n') + assert.equal(rows[0].n, 2n) +}) + +test('a signal stops a profile', async (t) => { + const { conn } = await twoPeople(t) + + const control = new AbortController() + control.abort(new Error('changed my mind')) + + await assert.rejects( + () => conn.profile('MATCH (p:person) RETURN p.name AS name', null, { signal: control.signal }), + (err) => err.message === 'changed my mind', + ) + // The connection is still usable, which is what says the signal took + // the statement rather than the connection with it. + assert.equal((await conn.query('MATCH (p:person) RETURN p.name AS name')).length, 2) +}) + +test('profiling on a closed connection is refused', async (t) => { + const { conn } = await fresh(t) + conn.close() + + await assert.rejects( + () => conn.profile('MATCH (p:person) RETURN p.name AS name'), + (err) => isZuError(err, 'ZuUsageError') && err.message.includes('the connection is closed'), + ) +}) + +test('a statement that is not a string is refused by both calls', async (t) => { + const { conn } = await fresh(t) + + const refused = (err) => + isZuError(err, 'ZuUsageError') && + err.message === 'the statement is a Number, and a statement is a string' + await assert.rejects(() => conn.explain(42), refused) + await assert.rejects(() => conn.profile(42), refused) +}) diff --git a/test/prepared.test.mjs b/test/prepared.test.mjs new file mode 100644 index 0000000..6d6e487 --- /dev/null +++ b/test/prepared.test.mjs @@ -0,0 +1,226 @@ +// A statement compiled once and run many times. +// +// The interesting part is not that it answers rows, which is the same +// answer `conn.query` gives and is asserted here mostly so that the +// three ways of asking are known to be the same three. It is the +// lifetime: what a prepared statement is before it is closed, what it +// says after, what happens to one whose connection went first, and +// whether the names it reports are the names the statement wants. + +import assert from 'node:assert/strict' +import test from 'node:test' + +import { connect, Prepared } from 'zudb' + +import { fresh, isZuError, twoPeople } from './helper.mjs' + +const BY_NAME = 'MATCH (p:person) WHERE p.name = $name RETURN p.id AS id' + +test('a prepared statement reports its text and the names it wants', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + + assert.ok(find instanceof Prepared) + assert.equal(find.statement, BY_NAME) + assert.deepEqual(find.params, ['name']) + assert.equal(find.closed, false) +}) + +test('a statement that takes no parameters reports none', async (t) => { + const { conn } = await twoPeople(t) + + await using all = await conn.prepare('MATCH (p:person) RETURN p.name AS name') + + assert.deepEqual(all.params, []) +}) + +test('it runs as often as it is asked to, with different bindings', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + + assert.deepEqual([...(await find.query({ name: 'ada' }))], [{ id: 1n }]) + assert.deepEqual([...(await find.query({ name: 'zoe' }))], [{ id: 2n }]) + assert.deepEqual([...(await find.query({ name: 'nobody' }))], []) + assert.deepEqual([...(await find.query({ name: 'ada' }))], [{ id: 1n }]) +}) + +test('the rows carry the same three properties a query gives back', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + const rows = await find.query({ name: 'ada' }) + + assert.deepEqual(rows.columns, ['id']) + assert.equal(rows.gqlstatus, '00000') + assert.deepEqual(rows.notices, []) +}) + +test('exec runs it and answers nothing', async (t) => { + const { conn } = await twoPeople(t) + + await using insert = await conn.prepare('INSERT (p:person {id: 3, name: $name})') + + assert.equal(await insert.exec({ name: 'ida' }), undefined) + assert.equal(await insert.exec({ name: 'eve' }), undefined) + + const rows = await conn.query('MATCH (p:person) RETURN p.name AS name') + assert.deepEqual( + rows.map((row) => row.name), + ['ada', 'zoe', 'ida', 'eve'], + ) +}) + +test('columnar reads it down its columns', async (t) => { + const { conn } = await twoPeople(t) + + await using all = await conn.prepare('MATCH (p:person) RETURN p.id AS id') + const read = await all.columnar() + + assert.equal(read.rows, 2) + assert.equal(read.columns[0].type, 'int') + assert.deepEqual([...read.columns[0].values], [1n, 2n]) +}) + +test('a statement that does not compile fails at the prepare', async (t) => { + const { conn } = await twoPeople(t) + + await assert.rejects( + () => conn.prepare('MATCH ('), + (err) => isZuError(err, 'ZuSyntaxError'), + ) +}) + +test('a name the caller did not bind fails at the run', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + + await assert.rejects( + () => find.query(), + (err) => isZuError(err, 'ZuSyntaxError') && err.message.includes('$name'), + ) + // And the statement is still there to be run properly, since nothing + // about a missing binding is about the statement. + assert.deepEqual([...(await find.query({ name: 'ada' }))], [{ id: 1n }]) +}) + +test('closing it twice is not an error', async (t) => { + const { conn } = await twoPeople(t) + + const find = await conn.prepare(BY_NAME) + await find.close() + assert.equal(find.closed, true) + await find.close() + assert.equal(find.closed, true) +}) + +test('a closed prepared statement says so at every one of the three runs', async (t) => { + const { conn } = await twoPeople(t) + + const find = await conn.prepare(BY_NAME) + await find.close() + + const closed = (err) => isZuError(err, 'ZuUsageError') && err.message.includes('closed') + await assert.rejects(() => find.query({ name: 'ada' }), closed) + await assert.rejects(() => find.exec({ name: 'ada' }), closed) + await assert.rejects(() => find.columnar({ name: 'ada' }), closed) +}) + +test('await using closes it at the end of the block', async (t) => { + const { conn } = await twoPeople(t) + + let held + { + await using find = await conn.prepare(BY_NAME) + held = find + assert.equal(held.closed, false) + } + assert.equal(held.closed, true) +}) + +test('a prepared statement whose connection closed says the connection is closed', async (t) => { + const { conn } = await twoPeople(t) + + const find = await conn.prepare(BY_NAME) + conn.close() + + await assert.rejects( + () => find.query({ name: 'ada' }), + (err) => isZuError(err, 'ZuUsageError') && err.message.includes('the connection is closed'), + ) + // Closing it is still fine, and still does nothing: the session that + // was holding the id went when the connection did. + await find.close() + assert.equal(find.closed, true) +}) + +test('a read-only connection prepares and runs a statement that reads', async (t) => { + const { conn, path } = await twoPeople(t) + conn.close() + + const reader = await connect(path, { readOnly: true }) + t.after(() => reader.close()) + await using all = await reader.prepare('MATCH (p:person) RETURN p.name AS name') + + assert.equal((await all.query()).length, 2) +}) + +test('a signal stops a run of a prepared statement', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + const control = new AbortController() + control.abort(new Error('changed my mind')) + + await assert.rejects( + () => find.query({ name: 'ada' }, { signal: control.signal }), + (err) => err.message === 'changed my mind', + ) + assert.deepEqual([...(await find.query({ name: 'ada' }))], [{ id: 1n }]) +}) + +test('bigIntMode on a run says how that run spells its integers', async (t) => { + const { conn } = await twoPeople(t) + + await using find = await conn.prepare(BY_NAME) + + assert.deepEqual([...(await find.query({ name: 'ada' }, { bigIntMode: 'number' }))], [{ id: 1 }]) + assert.deepEqual([...(await find.query({ name: 'ada' }))], [{ id: 1n }]) +}) + +test('a connection prepares as many statements as it likes', async (t) => { + const { conn } = await twoPeople(t) + + const prepared = await Promise.all([ + conn.prepare(BY_NAME), + conn.prepare('MATCH (p:person) RETURN count(*) AS n'), + conn.prepare('MATCH (p:person) RETURN p.name AS name'), + ]) + + assert.deepEqual([...(await prepared[0].query({ name: 'zoe' }))], [{ id: 2n }]) + assert.equal((await prepared[1].query())[0].n, 2n) + assert.equal((await prepared[2].query()).length, 2) + + for (const statement of prepared) await statement.close() +}) + +test('preparing on a closed connection is refused', async (t) => { + const { conn } = await fresh(t) + conn.close() + + await assert.rejects( + () => conn.prepare('MATCH (p:person) RETURN p.name AS name'), + (err) => isZuError(err, 'ZuUsageError') && err.message.includes('the connection is closed'), + ) +}) + +test('a statement that is not a string is refused with what arrived', async (t) => { + const { conn } = await fresh(t) + + await assert.rejects( + () => conn.prepare(42), + (err) => isZuError(err, 'ZuUsageError') && err.message === 'the statement is a Number, and a statement is a string', + ) +}) diff --git a/test/types/cjs.cts b/test/types/cjs.cts index d32ccfd..73c8082 100644 --- a/test/types/cjs.cts +++ b/test/types/cjs.cts @@ -12,6 +12,7 @@ import { type ZuColumnType, type ZuLoadOptions, type ZuParam, + type ZuProfile, type ZuStream, type ZuTransactionOptions, } from 'zudb' @@ -140,3 +141,30 @@ export async function shapes(path: string): Promise { await conn.close() return read.columns.map((column) => column.type) } + +export async function measured(path: string, name: string): Promise { + const conn = await connect(path, { readOnly: true }) + try { + const run: ZuProfile = await conn.profile( + 'MATCH (p:person) WHERE p.name = $name RETURN p.id AS id', + { name }, + { signal: AbortSignal.timeout(50) }, + ) + + // Every count here is a number and not a bigint, which is the one + // place this package spells an integer as a double on purpose, so + // adding them up needs no conversion anywhere. + let rows = 0 + for (const stage of run.stages) { + rows += stage.rows + for (const op of stage.ops) { + // The estimate and the q-error are null where the optimizer had + // nothing to say, so using one without asking does not compile. + if (op.estimate !== null && op.qerror !== null) rows += op.qerror > 10 ? 1 : 0 + } + } + return rows + run.nanos + } finally { + conn.close() + } +} diff --git a/test/types/esm.mts b/test/types/esm.mts index 7571485..a596f10 100644 --- a/test/types/esm.mts +++ b/test/types/esm.mts @@ -17,8 +17,11 @@ import { type ZuFrameColumn, type ZuColumn, type ZuColumnar, + type Prepared, type ZuLoadStats, type ZuPlainDate, + type ZuPlan, + type ZuPlanNode, type ZuRows, type ZuStream, type ZuSummary, @@ -247,3 +250,37 @@ export async function totals(path: string): Promise { if (read.gqlstatus !== '00000') throw new Error(read.gqlstatus) return total + BigInt(rows) + BigInt(column.nulls) } + +export async function repeated(path: string, names: string[]): Promise { + await using conn = await connect(path, { readOnly: true }) + + // A prepared statement is disposable too, and the row type goes on + // the run rather than on the prepare, because one statement answers + // whatever the projection says and the projection is in the text. + await using find: Prepared = await conn.prepare( + 'MATCH (p:person) WHERE p.name = $name RETURN p.id AS id', + ) + + // The names it wants, which is the half of a prepared statement that + // is not just a faster `query`. + const wanted: string[] = find.params + if (wanted.length !== 1) throw new Error('the statement changed shape') + + const out: bigint[] = [] + for (const name of names) { + const rows = await find.query<{ id: bigint }>({ name }) + for (const row of rows) out.push(row.id) + } + return out +} + +export async function scans(path: string): Promise { + await using conn = await connect(path, { readOnly: true }) + const plan: ZuPlan = await conn.explain('MATCH (p:person) RETURN p.id AS id') + + // The root is nullable, since a statement can compile to no operator + // at all, so walking the tree without asking does not compile. + const walk = (node: ZuPlanNode): boolean => + node.op === 'ScanNodes' || node.children.some(walk) + return plan.root === null ? false : walk(plan.root) +} diff --git a/types/header.d.ts b/types/header.d.ts index e33f109..ac20380 100644 --- a/types/header.d.ts +++ b/types/header.d.ts @@ -432,6 +432,143 @@ export interface ZuSummary { readonly notices: ZuNotice[] } +/** + * One operator of a plan, and everything under it. + * + * The tree runs the way the rows do: a parent pulls from its children, + * so the leaves are the scans and the root is whatever the statement + * ends with. + */ +export interface ZuPlanNode { + /** The operator: `Scan`, `Expand`, `Filter`, `Project` and the rest. */ + readonly op: string + /** + * What the listing calls it, which is `op` with the bracket in front + * of it where there is one, so an OPTIONAL MATCH expand is an + * `Expand` named `OptionalExpand`. + */ + readonly name: string + /** The bracket this operator is inside, and null for a plain match. */ + readonly bracket: 'Optional' | 'Semi' | 'Anti' | 'Mark' | null + /** + * What it is working on, written the way the statement wrote it: the + * tables a scan reads, the pattern an expand walks, the predicate a + * filter asks. Empty where the operator has nothing to name. + */ + readonly detail: string + /** The variables it introduces, in the order it binds them. */ + readonly binds: string[] + /** + * The tables it touches: node tables for a scan, rel tables for an + * expand, both for an insert, and none anywhere else. + */ + readonly tables: string[] + readonly children: ZuPlanNode[] +} + +/** + * A query written where a value belongs, planned on its own. + * + * `reads` is what it reads from the query around it, and empty is the + * whole test for whether it runs once: a subquery that reads nothing + * answers the same value for every row, and one that reads a name runs + * per row. `exists` is true where what was written around it asks only + * whether it answered a row. + */ +export interface ZuScalarPlan { + readonly reads: string[] + readonly exists: boolean + readonly plan: ZuPlan +} + +/** + * What a statement would do, without doing it. + * + * A tree and a rendering of it. `text` is what the engine prints, so a + * listing logged from Node is the listing the shell shows, and the tree + * is for the questions a program asks: which tables were touched, how + * deep the expands go, whether the scan reached an index. + */ +export interface ZuPlan { + /** + * The top operator, and null for the plan with no operators at all, + * which is the one row a statement with no clauses runs over. + */ + readonly root: ZuPlanNode | null + /** The columns the statement answers with, in the order it wrote them. */ + readonly columns: string[] + /** The parameters it wants, without the `$` they are written with. */ + readonly params: string[] + /** What compiling it raised, which is empty for most statements. */ + readonly notes: string[] + readonly scalars: ZuScalarPlan[] + /** The listing, indented, as `EXPLAIN` prints it. */ + readonly text: string +} + +/** + * One operator of a profiled run, and what the counters saw of it. + */ +export interface ZuOp { + readonly op: string + readonly detail: string + /** How many chunks it produced. */ + readonly pulls: number + /** Values produced across every pull. Over `pulls` that is the + * average vector length, which is the factorization statistic. */ + readonly rows: number + /** + * The rows those values stand for with the factorization multiplied + * out. On a chain it is `rows`, and on a star it is the product over + * every vector still unflat beside this one, which is the count the + * optimizer was estimating. + */ + readonly flat: number + /** + * What the optimizer expected, and null for the operators that pass + * their input through rather than producing rows of their own. + */ + readonly estimate: number | null + /** The most rows the optimizer's ceiling allowed, where the + * statistics were there to set one. */ + readonly bound: number | null + /** Self time in nanoseconds, with the children's excluded. */ + readonly nanos: number + /** + * How wrong the estimate was: `max(estimate/rows, rows/estimate)`, + * both floored at one row. An operator the optimizer got right is 1, + * and null wherever `estimate` is. + */ + readonly qerror: number | null +} + +/** + * One stage of a profiled run: the operators bottom-up and the sink + * that took their rows. + */ +export interface ZuStage { + readonly sink: string + /** How many rows the sink was handed. */ + readonly rows: number + /** Wall time of the whole stage in nanoseconds, sink included. */ + readonly nanos: number + readonly ops: ZuOp[] +} + +/** + * What a statement did, with the counters on. + * + * The rows are not here: a profile is about the run rather than the + * answer, and keeping both would make the measurement pay for the thing + * it is measuring. `text` is the listing `EXPLAIN ANALYZE` prints. + */ +export interface ZuProfile { + readonly stages: ZuStage[] + /** Every stage end to end, in nanoseconds. */ + readonly nanos: number + readonly text: string +} + /** * What a streamed statement takes beside its parameters. */ diff --git a/zudb.cjs b/zudb.cjs index 0212986..c527f1a 100644 --- a/zudb.cjs +++ b/zudb.cjs @@ -166,6 +166,7 @@ module.exports = { Connection: binding.Connection, Transaction: binding.Transaction, Appender: binding.Appender, + Prepared: binding.Prepared, ZuStream, // The pull underneath a stream, which `conn.cursor(...)` hands back // and almost nobody should be holding. It is here because it is in diff --git a/zudb.d.cts b/zudb.d.cts index f22764e..677c6bf 100644 --- a/zudb.d.cts +++ b/zudb.d.cts @@ -53,6 +53,16 @@ declare module './binding.cjs' { * caller who meant exactly that. */ interface Appender extends AsyncDisposable {} + + /** + * The disposal of a prepared statement, declared here for the same + * reason the other three are. + * + * It closes, which gives the statement back to the connection. There + * is nothing to undo and nothing to write, so unlike the other three + * this one has only the one thing it could mean. + */ + interface Prepared extends AsyncDisposable {} } /** diff --git a/zudb.mjs b/zudb.mjs index 3167250..0b39cbe 100644 --- a/zudb.mjs +++ b/zudb.mjs @@ -24,6 +24,7 @@ export const isZuError = zudb.isZuError export const Connection = zudb.Connection export const Transaction = zudb.Transaction export const Appender = zudb.Appender +export const Prepared = zudb.Prepared export const ZuStream = zudb.ZuStream export const ZuCursor = zudb.ZuCursor export const ZuDate = zudb.ZuDate