Load a whole database out of columns and an edge list

README.md

@@ -36,7 +36,7 @@ The rows are an array, so iterating them is `for (const row of rows)` and nothin
 
 ## What works today
 
-`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. Both module formats, typed separately.
+`connect`, `query`, `exec`, `stream`, `close`, `dispose` and `await using`. Named parameters both ways, including lists, records and nesting. Every scalar the engine has, plus nodes, edges and paths with their tables named rather than numbered, and `ZuDate`, `ZuTime`, `ZuTimestamp` and `ZuDuration`, with `{ temporal: true }` and `toTemporal()` for the runtimes that have `Temporal`. Read-only connections, memory and thread limits. `bigIntMode`, per statement or per connection. An `AbortSignal` on any statement. The full error surface above, and `isZuError` to recognize it. Streaming, as an async iterable, as batches and as a Web Stream. Transactions, with `inTransaction` on the connection. An appender, for loading rows a batch at a time, and `load` for building a whole database out of columns and an edge list. Registered frames, so an Arrow table or an object of typed arrays is something a statement can match on without the rows being copied. Both module formats, typed separately.
 
 Build it with `npm run build`, and run the suite with `npm test`. Nothing is published yet, so `npm i zudb` is not a thing you can type at anybody's terminal, but everything it will do is built and installed on every run of the release workflow.
 
@@ -124,6 +124,46 @@ Two more things are worth knowing before a load. A flush issued while one is sti
 
 A rel table has no property columns. A row of one is the two ends of an edge, as offsets into the tables it runs between, so `conn.appender("knows")` takes two columns and the flush checks that both rows are there before it writes anything. That check is here rather than the engine's, because the engine's comes after the write is durable.
 
+## Building a graph out of columns and an edge list
+
+An appender writes rows into a table that already exists, and no statement makes a rel table, so neither of them is a way to a graph with edges in it. `load` is the other shape and the one the C ABI's loader has: a table's columns whole, an edge list whole, one file written once.
+
+```ts
+import { load } from "zudb";
+
+const uid = new BigInt64Array([1n, 2n, 3n]);
+const name = ["ada", "grace", "kay"];
+
+const stats = await load("social.zu1", {
+  nodes: "person",
+  rels: "knows",
+  columns: { uid, name },
+  edges: [
+    [0, 1],
+    [1, 2],
+  ],
+});
+console.log(stats); // { nodes: 3, rels: 2, columns: 2 }
+```
+
+It is a function rather than a method because there is no connection yet: the file it writes is the file a program connects to afterwards. The path must not exist, since a load builds a database rather than adding to one, and a path that already holds one is a caller who meant a different path. What comes back is what went in, as `{ nodes, rels, columns }`.
+
+Edges name rows by position, counting from zero in the order the columns were written, because at load time a row has no other name. They go in as pairs, `[[0, 1], [1, 2]]`, or as a flat `Int32Array` or `Uint32Array` of two elements an edge for a program that built them in memory and would rather not make a million small arrays to hand them over. The same edge twice is one edge, and an edge naming a row the table has not got is refused rather than written, because a builder handed one would either invent the row or lose the edge.
+
+A column goes in as an array of values or as a typed array. The first value of an array settles what the column holds and every value after it has to agree, which is the appender's rule, with the same one widening: `[1, 2, 2.5]` is a column of floats. A typed array is read as the numbers it already holds, which is one pass over memory rather than a runtime call per value, and every integer width lands as the INT64 the store keeps. On this machine, with `npm run bench:load` over a million rows:
+
+```
+one column, typed array             51.3 ms      51 ns/row
+one column, plain array             92.2 ms      92 ns/row
+two columns, with names           1060.1 ms    1060 ns/row
+two columns and an edge each      1096.9 ms    1097 ns/row
+the appender, for contrast        2124.5 ms    2125 ns/row
+```
+
+The last line is the same two columns through the appender, which is the closest comparison there is: a load is about twice as quick and is the only one of the two that can write the edges. The strings are where the rest of the time goes, which is the store encoding them rather than anything on this side of the boundary.
+
+Everything the caller passed is read on the thread that owns the runtime, because that is the only thread allowed to read a JavaScript value, and everything after that runs on the threadpool: the edges are sorted, the graph is built, and every column is encoded and written to disk. So the event loop is free for the whole of the expensive part, which on a load is all of it.
+
 ## Matching on columns a program already has
 
 Columns a program is already holding become something a statement can match on, under a name the program picks.
@@ -238,7 +278,7 @@ typedoc rather than api-documenter, which would have been the obvious pick since
 
 Anything outside that table has no binary and no source build to fall back on, so the install resolves nothing and the first `require` says so. The browser and the platforms nobody builds for are what the WASM target answers, later.
 
-`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the load path and `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows.
+`npm run bench` measures what this package adds to the engine, which is a row object and one JavaScript value per column: the same scan with the rows dropped is the floor, and the difference between the two is what the boundary costs. Run it against a release build, since a debug build of the engine moves the floor by an order of magnitude and not the rest of it. `npm run bench:append` does the same for the appender, `npm run bench:load` for building a database out of columns, and `npm run bench:register` for registered frames, where what is being watched is that the registration does not scale with the rows.
 
 ## Still to come
 

bench/load.mjs

@@ -0,0 +1,110 @@
+// What building a database out of columns costs.
+//
+// `load` is the other way in, and it is the only way to a graph with
+// edges in it. The appender writes rows into a table that already
+// exists; this writes the file, so the numbers here are not the
+// appender's numbers with a different name on them and the first two
+// lines are what says so.
+//
+// The third and fourth are about the way the columns were handed over.
+// A typed array is read as the numbers it already holds, which is one
+// pass over memory, and an ordinary array is read a value at a time
+// through the runtime. The gap between them is the whole argument for
+// reaching for a typed array on a load this size.
+//
+// Run it against a release build, for the reason bench/query.mjs gives.
+//
+//   npm run build && npm run bench:load
+
+import { mkdtemp, rm } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+
+import { connect, load } from 'zudb'
+
+const ROWS = Number(process.env.ZU_BENCH_ROWS ?? 1_000_000)
+const REPEATS = Number(process.env.ZU_BENCH_REPEATS ?? 3)
+
+const dir = await mkdtemp(join(tmpdir(), 'zu-bench-load-'))
+
+const wide = BigInt64Array.from({ length: ROWS }, (_, ix) => BigInt(ix))
+const plain = Array.from({ length: ROWS }, (_, ix) => ix)
+const name = Array.from({ length: ROWS }, (_, ix) => `n${ix}`)
+// Out of order on purpose, because sorting them is half the work of
+// building the graph and an edge list a program produced by walking
+// something else is never in order.
+const edges = new Uint32Array(ROWS * 2)
+for (let ix = 0; ix < ROWS; ix++) {
+  edges[ix * 2] = ix
+  edges[ix * 2 + 1] = (ix * 7 + 1) % ROWS
+}
+
+let counter = 0
+
+/// The fastest of `REPEATS` runs, in milliseconds, after one warmup.
+///
+/// A fresh path every time, because a load will not write over a
+/// database that is there and because a second load into a warm page
+/// cache is not the load being measured.
+async function time(run) {
+  await run(join(dir, `warm-${counter++}.zu1`))
+  let best = Infinity
+  for (let round = 0; round < REPEATS; round++) {
+    const path = join(dir, `bench-${counter++}.zu1`)
+    const started = performance.now()
+    await run(path)
+    best = Math.min(best, performance.now() - started)
+  }
+  return best
+}
+
+const cases = [
+  {
+    // One column of whole numbers and nothing else, which is the floor:
+    // the file, the table and one run of words.
+    name: 'one column, typed array',
+    run: (path) => load(path, { nodes: 'person', columns: { uid: wide } }),
+  },
+  {
+    // The same column written as an ordinary array, which is a runtime
+    // call per value on the way in.
+    name: 'one column, plain array',
+    run: (path) => load(path, { nodes: 'person', columns: { uid: plain } }),
+  },
+  {
+    // A column of strings, which is where the bytes are.
+    name: 'two columns, with names',
+    run: (path) => load(path, { nodes: 'person', columns: { uid: wide, name } }),
+  },
+  {
+    // The graph: the same columns and an edge per row, sorted and built
+    // into the CSRs a pattern walks.
+    name: 'two columns and an edge each',
+    run: (path) => load(path, { nodes: 'person', rels: 'knows', columns: { uid: wide, name }, edges }),
+  },
+  {
+    // The other way in, for the one case both can do: rows into a table
+    // that a first insert declared. No edges, since no statement makes
+    // a rel table.
+    name: 'the appender, for contrast',
+    run: async (path) => {
+      const conn = await connect(path)
+      await conn.exec("INSERT (p:person {uid: 0, name: 'n0'})")
+      const rows = await conn.appender('person')
+      for (let ix = 1; ix < ROWS; ix++) rows.appendRow([wide[ix], name[ix]])
+      await rows.close()
+      conn.close()
+    },
+  },
+]
+
+console.log(`${ROWS} rows, fastest of ${REPEATS}`)
+for (const { name, run } of cases) {
+  const ms = await time(run)
+  const each = (ms * 1e6) / ROWS
+  console.log(
+    `${name.padEnd(30)} ${ms.toFixed(1).padStart(9)} ms  ${Math.round(each).toString().padStart(6)} ns/row`,
+  )
+}
+
+await rm(dir, { recursive: true, force: true })

binding.cjs

@@ -712,4 +712,5 @@ module.exports.ZuTime = nativeBinding.ZuTime
 module.exports.ZuTimestamp = nativeBinding.ZuTimestamp
 module.exports.abiVersion = nativeBinding.abiVersion
 module.exports.connect = nativeBinding.connect
+module.exports.load = nativeBinding.load
 module.exports.version = nativeBinding.version

binding.d.cts

@@ -221,6 +221,48 @@ export interface ZuArrowTable {
  */
 export type ZuFrame = ZuArrowTable | Record<string, ZuFrameColumn>
 
+/**
+ * An edge list, as the pairs of row numbers it is.
+ *
+ * Rows are numbered from zero in the order their columns were written,
+ * because at load time a row has no other name. The flat spelling is
+ * there for a program that built its edges in memory and would rather
+ * not make a million two element arrays to hand them over: two elements
+ * an edge, read in one pass.
+ */
+export type ZuEdges = readonly (readonly [number, number])[] | Int32Array | Uint32Array
+
+/**
+ * What a load writes, and how much of it.
+ */
+export interface ZuLoadOptions {
+  /** The node table, which gets a row per element of every column. */
+  readonly nodes: string
+  /** The rel table holding the edges between those rows. `rel` by default. */
+  readonly rels?: string
+  /** The node table's properties, as column name to values. */
+  readonly columns?: Readonly<Record<string, ZuFrameColumn>>
+  /** The edges, as pairs of row numbers. */
+  readonly edges?: ZuEdges | null
+  /**
+   * How many rows the node table has.
+   *
+   * Read off the columns when there are any, so this is for the load
+   * that writes a graph with no properties at all, and a check on the
+   * columns when both are given.
+   */
+  readonly rows?: number
+}
+
+/**
+ * What went into a load.
+ */
+export interface ZuLoadStats {
+  readonly nodes: number
+  readonly rels: number
+  readonly columns: number
+}
+
 /**
  * A walk through the graph: nodes and edges, alternating, a node at
  * each end.
@@ -1024,5 +1066,8 @@ export interface ConnectOptions {
   temporal?: boolean
 }
 
+/** Writes a new database at `path` and answers what went into it. */
+export declare function load(path: string, options: ZuLoadOptions): Promise<ZuLoadStats>
+
 /** The version of the client. */
 export declare function version(): string

etc/zudb.api.md

@@ -54,6 +54,9 @@ export interface ConnectOptions {
 // @public
 export function isZuError(value: unknown): value is ZuError
 
+// @public
+export function load(path: string, options: ZuLoadOptions): Promise<ZuLoadStats>
+
 // @public
 export class Transaction {
     commit(): Promise<void>
@@ -123,6 +126,9 @@ export class ZuDuration {
     toTemporal(): ZuTemporalDuration
 }
 
+// @public
+export type ZuEdges = readonly (readonly [number, number])[] | Int32Array | Uint32Array
+
 // @public
 export interface ZuError extends Error {
     readonly code?: string
@@ -171,6 +177,25 @@ export type ZuFrameValue =
 | ZuDuration
 | ZuTemporalValue
 
+// @public
+export interface ZuLoadOptions {
+    readonly columns?: Readonly<Record<string, ZuFrameColumn>>
+    readonly edges?: ZuEdges | null
+    readonly nodes: string
+    readonly rels?: string
+    readonly rows?: number
+}
+
+// @public
+export interface ZuLoadStats {
+    // (undocumented)
+    readonly columns: number
+    // (undocumented)
+    readonly nodes: number
+    // (undocumented)
+    readonly rels: number
+}
+
 // @public
 export class ZuNode {
     // (undocumented)

package.json

@@ -72,6 +72,7 @@
     "reference": "node tools/reference.mjs reference",
     "bench": "node bench/query.mjs",
     "bench:append": "node bench/append.mjs",
+    "bench:load": "node bench/load.mjs",
     "bench:register": "node bench/register.mjs",
     "bench:temporal": "node --harmony-temporal bench/query.mjs"
   },

src/conn.rs

@@ -956,7 +956,7 @@ fn int_mode(options: Option<&Object<'_>>, connection: Ints) -> std::result::Resu
 /// from a method whose every other failure is a rejection is a method
 /// callers have to wrap twice. So the argument arrives unread and this
 /// is what reads it.
-fn text(value: &Unknown<'_>, what: &str) -> std::result::Result<String, String> {
+pub(crate) fn text(value: &Unknown<'_>, what: &str) -> std::result::Result<String, String> {
     match value.get_type().map_err(|err| err.reason)? {
         ValueType::String => String::from_unknown(*value).map_err(|err| err.reason),
         other => Err(format!(

src/lib.rs

@@ -22,6 +22,7 @@ mod cancel;
 mod conn;
 mod error;
 mod frame;
+mod load;
 mod register;
 mod stream;
 mod temporal;