Register frames, so a statement can match on columns a program already holds

README.md

@@ -36,7 +36,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin
 
 ## What works today
 
-`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time. Both module formats, typed separately.
+`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. Both module formats, typed separately.
 
 Build it with `npm run build`, and run the suite with `npm test`. Nothing is published yet, so `npm i zudb` is not a thing you can type at anybody's terminal, but everything it will do is built and installed on every run of the release workflow.
 
@@ -124,6 +124,28 @@ Two more things are worth knowing before a load. A flush issued while one is sti
 
 A rel table has no property columns. A row of one is the two ends of an edge, as offsets into the tables it runs between, so `conn.appender("knows")` takes two columns and the flush checks that both rows are there before it writes anything. That check is here rather than the engine's, because the engine's comes after the write is durable.
 
+## Matching on columns a program already has
+
+Columns a program is already holding become something a statement can match on, under a name the program picks.
+
+```ts
+await conn.register("people", table);
+const rows = await conn.query(`MATCH (p:people) WHERE p.age > 40 RETURN p.name AS name`);
+await conn.unregister("people");
+```
+
+An Arrow table goes in, which is what `apache-arrow` and everything built on it hands out, and so does an object of column name to typed array for a caller with none of that installed. `apache-arrow` is not a dependency of this package and is not imported by it: a table is recognized by its shape, so any library that speaks that shape takes the same path.
+
+Nothing is copied. What the engine is told is where each column is, how wide its values are and what they mean, and a statement that names the frame builds vectors pointing straight at the caller's arrays. So registering costs what describing the columns costs and not what the rows cost: on this machine a frame of a million rows registers in 26 microseconds and one of ten rows in 34, both of which are mostly the promise, since the round trip on its own is 16.
+
+The one column that is walked is a string column, and it is walked once. Every offset is checked at registration so that reading the frame afterwards cannot fail, which is 1.2 ms for a million strings. Two other things copy and both are said rather than hidden: a table that arrived as several record batches is concatenated into one, because a column of a frame is one run of bytes and two batches are two of them, and a column given as a plain array is read into a buffer of this client's own, because an array holds JavaScript values rather than numbers and there is nothing in it to point at. That last one is the expensive way in at 127 ns a row, and it is there so that a caller with an array is not stuck rather than because it is the way to do this.
+
+Because it is not a copy, a registered frame is a view and not a snapshot. Write into the typed array behind it and the next statement answers what is there now, which is the thing to know about the call and the reason it is worth having. Reading one is as fast as reading a table of the database and faster where the database has to decode: over a million rows here, summing an integer column takes 1.3 ms against a stored table's 1.6, and finding one row by a string takes 2.2 ms against 5.8.
+
+The frame belongs to the connection it was registered on and goes when that connection does. Nothing is written to the file, so another program opening the same database has never heard of it, and nothing writes to it either: a statement that inserts into or deletes from a registered name is refused with the reason, because that memory is the caller's array. `unregister(name)` takes the name away and hands the arrays back, which is not always that instant, since a statement still reading the frame holds it until it ends. `registered()` says what is registered here, and it is a method rather than a getter because it takes the connection's lock like everything else and nothing here blocks the event loop.
+
+Registering the same name again replaces what it stands for, columns and all. Registering over a table the database already holds is refused, since a statement naming it would mean the stored one. A frame with no rows is a table to match on and answers nothing, because a frame knows its columns without being told by a row. A null anywhere is refused by column and row, since a property that is null is one no row of this engine holds, and registering inside a transaction is refused because a frame is registered on the session, which is the thing the transaction is running on.
+
 ## Asking for numbers instead of bigints
 
 `bigIntMode` says how INT64 is spelled on the way out. It goes on one statement, or on a connection for all of them, and a statement on a connection that named one may still name the other:
@@ -216,7 +238,7 @@ typedoc rather than api-documenter, which would have been the obvious pick since
 
 Anything outside that table has no binary and no source build to fall back on, so the install resolves nothing and the first `require` says so. The browser and the platforms nobody builds for are what the WASM target answers, later.
 
-`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it.
+`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the load path and `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows.
 
 ## Still to come
 
@@ -232,7 +254,7 @@ Bun and Deno in CI, and the WASM build for the browser.
 | Browser and edge | `zudb/wasm` | read-mostly, over OPFS or HTTP range requests |
 | Electron | the same binary | N-API is ABI-stable across Electron versions, so no per-Electron rebuild |
 
-`apache-arrow` is an optional peer dependency behind the `zudb/arrow` entry point, so the base package stays small.
+`apache-arrow` is a dev dependency and not a dependency, and it is one so that the tests can build the tables the register path reads. Nothing in the package imports it, so a caller who never registers a frame never installs it.
 
 One binary serves all three, because N-API is the ABI all three implement, and the whole suite runs on each of them in CI rather than the other two being assumed from Node passing. `npm run test:bun` and `npm run test:deno` run it locally. What the three do not agree on is what a native error carries: V8 writes a `stack` when the error is made and JavaScriptCore writes none at all through N-API, so this client writes the header line itself when it finds none, non-enumerably, and `err.stack` starts with the condition's name on all of them.
 

bench/register.mjs

@@ -0,0 +1,158 @@
+// What registering a frame costs, and what reading one costs after.
+//
+// The claim the call makes is that nothing is copied, so the first block
+// here is the one that has to hold: registering ten rows and registering
+// ten million should cost the same, because what happens is that the
+// engine is told where the columns are. A line in that block that scales
+// with the rows is a line that copied them.
+//
+// Three cases do copy and all three are here rather than hidden. A string
+// column is walked once, to check every offset at registration so that
+// reading it afterwards cannot fail. A table that arrived as several
+// batches is concatenated, because a column of a frame is one run of
+// bytes and two batches are two of them. A plain JavaScript array is read
+// into a buffer of this client's own, because an array holds values
+// rather than numbers and there is nothing in it to point at.
+//
+// The second block is the reason to register at all: a statement reading
+// a frame against the same statement reading a table of the database.
+//
+// Run it against a release build, for the reason bench/query.mjs gives.
+//
+//   npm run build && npm run bench:register
+
+import { mkdtemp, rm } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+
+import { Table, Utf8, tableFromArrays, vectorFromArray } from 'apache-arrow'
+import { connect } from 'zudb'
+
+const ROWS = Number(process.env.ZU_BENCH_ROWS ?? 1_000_000)
+const REPEATS = Number(process.env.ZU_BENCH_REPEATS ?? 5)
+// The small frame, for the pair of lines that says registration does not
+// scale with the rows. Ten is small enough that anything proportional to
+// the rows disappears from it.
+const FEW = 10
+
+const dir = await mkdtemp(join(tmpdir(), 'zu-bench-register-'))
+
+const uid = BigInt64Array.from({ length: ROWS }, (_, ix) => BigInt(ix))
+const score = Float64Array.from({ length: ROWS }, (_, ix) => ix / 3)
+const name = Array.from({ length: ROWS }, (_, ix) => `n${ix}`)
+const plain = Array.from({ length: ROWS }, (_, ix) => ix)
+
+const wide = { uid, score }
+const small = { uid: uid.subarray(0, FEW), score: score.subarray(0, FEW) }
+const words = tableFromArrays({ name: vectorFromArray(name, new Utf8()) })
+const arrow = tableFromArrays({ uid, score })
+const halves = (() => {
+  const cut = ROWS >> 1
+  const first = tableFromArrays({ uid: uid.slice(0, cut) }).batches[0]
+  const second = tableFromArrays({ uid: uid.slice(cut) }).batches[0]
+  return new Table([first, second])
+})()
+
+let counter = 0
+
+/// A connection with nothing in it.
+async function blank() {
+  return await connect(join(dir, `bench-${counter++}.zu1`))
+}
+
+/// The fastest of `REPEATS` runs, in milliseconds, after one warmup.
+///
+/// The fastest for the reason bench/query.mjs gives: everything that
+/// makes a run slower than the work itself is something that happened to
+/// it rather than something about it.
+async function time(conn, run) {
+  await run(conn)
+  let best = Infinity
+  for (let round = 0; round < REPEATS; round++) {
+    const started = performance.now()
+    await run(conn)
+    best = Math.min(best, performance.now() - started)
+  }
+  return best
+}
+
+/// One registration, and what it costs is one promise round trip as well
+/// as the work, because nothing here runs on the thread that called it.
+/// The rounds after the first replace the name rather than taking it
+/// away, which is the same description being built again and is what a
+/// program rerunning the same cell does anyway.
+function once(frame) {
+  return async (conn) => {
+    await conn.register('frame', frame)
+  }
+}
+
+const registering = [
+  // The floor, which is a call that takes the lock and answers a list of
+  // one name. Every line under it carries the same round trip, so this
+  // is what to subtract before believing any of them.
+  { name: 'the round trip alone', rows: 1, run: (conn) => conn.registered() },
+  { name: `typed arrays, ${FEW} rows`, rows: FEW, run: once(small) },
+  { name: 'typed arrays', rows: ROWS, run: once(wide) },
+  { name: 'arrow table', rows: ROWS, run: once(arrow) },
+  { name: 'arrow, two batches', rows: ROWS, run: once(halves) },
+  { name: 'arrow strings', rows: ROWS, run: once(words) },
+  { name: 'plain array', rows: ROWS, run: once({ n: plain }) },
+]
+
+const conn = await blank()
+console.log(`registering ${ROWS} rows, fastest of ${REPEATS}`)
+for (const { name, rows, run } of registering) {
+  const ms = await time(conn, run)
+  if (rows === 1) {
+    console.log(`${name.padEnd(24)} ${ms.toFixed(3).padStart(9)} ms`)
+    continue
+  }
+  const each = (ms * 1e6) / rows
+  const scale = rows === ROWS ? '' : ` (over ${rows})`
+  console.log(
+    `${name.padEnd(24)} ${ms.toFixed(3).padStart(9)} ms  ${Math.round(each).toString().padStart(7)} ns/row${scale}`,
+  )
+}
+conn.close()
+
+// The same rows twice, once as a frame the caller holds and once as a
+// table the database holds, so that the two lines of each pair are the
+// same statement over the same values.
+const stored = await blank()
+{
+  await stored.exec("INSERT (p:person {uid: 0, name: 'n0'})")
+  const rows = await stored.appender('person')
+  for (let ix = 1; ix < ROWS; ix++) rows.appendRow([uid[ix], name[ix]])
+  await rows.close()
+}
+await stored.register('frame', {
+  uid,
+  name,
+})
+
+const hunted = `n${ROWS - 1}`
+const reading = [
+  {
+    name: 'sum an integer column',
+    frame: 'MATCH (p:frame) RETURN sum(p.uid) AS total',
+    table: 'MATCH (p:person) RETURN sum(p.uid) AS total',
+  },
+  {
+    name: 'find a row by string',
+    frame: `MATCH (p:frame) WHERE p.name = '${hunted}' RETURN p.uid AS uid`,
+    table: `MATCH (p:person) WHERE p.name = '${hunted}' RETURN p.uid AS uid`,
+  },
+]
+
+console.log(`\nreading ${ROWS} rows, fastest of ${REPEATS}`)
+for (const { name, frame, table } of reading) {
+  const asFrame = await time(stored, (conn) => conn.query(frame))
+  const asTable = await time(stored, (conn) => conn.query(table))
+  console.log(
+    `${name.padEnd(24)} ${asFrame.toFixed(3).padStart(9)} ms frame  ${asTable.toFixed(3).padStart(9)} ms table`,
+  )
+}
+
+stored.close()
+await rm(dir, { recursive: true, force: true })

binding.d.cts

@@ -151,6 +151,76 @@ export type ZuAppendValue =
   | ZuDuration
   | ZuTemporalValue
 
+/**
+ * One value of a registered frame's column, when the column is written
+ * as a plain array.
+ *
+ * The same values an appender takes, without the bytes: a column of
+ * BYTES is a column no statement can read back yet, so registering one
+ * would be naming data the caller cannot get at. There is no `null`
+ * either, for the reason there is none in a row of an appender.
+ */
+export type ZuFrameValue =
+  | boolean
+  | number
+  | bigint
+  | string
+  | ZuDate
+  | ZuTime
+  | ZuTimestamp
+  | ZuDuration
+  | ZuTemporalValue
+
+/**
+ * One column of a registered frame.
+ *
+ * A typed array is the shape that costs nothing: the engine reads it
+ * where it lies and no byte of it is copied. A plain array is read into
+ * buffers of this client's own, because an array holds values of the
+ * runtime rather than numbers, and its first value settles what the
+ * column holds.
+ */
+export type ZuFrameColumn =
+  | Int8Array
+  | Uint8Array
+  | Uint8ClampedArray
+  | Int16Array
+  | Uint16Array
+  | Int32Array
+  | Uint32Array
+  | Float32Array
+  | Float64Array
+  | BigInt64Array
+  | BigUint64Array
+  | readonly ZuFrameValue[]
+
+/**
+ * An Arrow table or record batch, described by its shape rather than by
+ * its class.
+ *
+ * Structural on purpose. `apache-arrow` is not a dependency of this
+ * client and should not have to be: recognizing a table by the two
+ * things every version of it has means a caller's copy of that library
+ * and this client's are never two copies of one package disagreeing
+ * about `instanceof`, and it means anything else that speaks the same
+ * shape works too.
+ */
+export interface ZuArrowTable {
+  readonly schema: { readonly fields: readonly { readonly name: string }[] }
+  getChildAt(index: number): unknown
+}
+
+/**
+ * Columns the caller already holds, ready to be registered under a name.
+ *
+ * An Arrow table, or an object of column name to values. Both are read
+ * where they lie wherever there is one run of bytes to read: the two
+ * cases that copy are an Arrow column that arrived in several chunks,
+ * which is concatenated once, and a plain JavaScript array, which was
+ * never a column of numbers to begin with.
+ */
+export type ZuFrame = ZuArrowTable | Record<string, ZuFrameColumn>
+
 /**
  * A walk through the graph: nodes and edges, alternating, a node at
  * each end.
@@ -531,6 +601,53 @@ export declare class Connection {
    * million rows later.
    */
   appender(table: string): Promise<Appender>
+  /**
+   * Registers columns the caller already holds as a table called
+   * `name`, and answers how many rows it has.
+   *
+   * The zero-copy way in. Nothing is read into the database: the
+   * engine is told where the caller's buffers are, and a statement
+   * that matches the name scans them where they lie, so registering
+   * ten million rows costs a description of their columns rather than
+   * ten million writes.
+   *
+   * ```js
+   * await conn.register('people', arrow.tableFromArrays({ id, name }))
+   * const rows = await conn.query('MATCH (p:people) RETURN p.name AS name')
+   * ```
+   *
+   * An Arrow table or record batch, which is what `apache-arrow` and
+   * everything built on it hands out, or an object of column name to
+   * values. The values of that object are typed arrays where the
+   * caller has them, which is the zero-copy shape, and plain arrays
+   * where they do not, which is read into buffers of this client's
+   * own because an array holds values of the runtime rather than
+   * numbers.
+   *
+   * A frame is a view and not a snapshot: write into the array behind
+   * it and the next statement answers what is there now. It belongs
+   * to this connection, is never written to the database, and no
+   * other program opening the same file sees it. Nothing writes to
+   * one either, so a statement that inserts into a registered name is
+   * refused with the reason.
+   */
+  register(name: string, data: ZuFrame): Promise<number>
+  /**
+   * Takes a registered frame's name away and gives the bytes back.
+   *
+   * The bytes go when the last statement reading them lets go, which
+   * is usually now and is never before: a frame a running statement
+   * is still scanning is held until it ends.
+   */
+  unregister(name: string): Promise<void>
+  /**
+   * The names frames are registered under on this connection, sorted.
+   *
+   * A method rather than a getter, and asynchronous like everything
+   * else here, because reading them takes the connection's lock and
+   * nothing on this class waits on the event loop.
+   */
+  registered(): Promise<string[]>
   /**
    * Runs one statement and gives back its rows.
    *