Load a whole database out of columns and an edge list

[tamnd]

An appender writes rows into a table that already exists, and no statement makes a rel table, so neither of them is a way to a graph with edges in it. load(path, options) is the other shape and the one the C ABI's loader has: a table's columns whole, an edge list whole, one file written once.

const stats = await load("social.zu1", {
  nodes: "person",
  rels: "knows",
  columns: { uid, name },
  edges: [[0, 1], [1, 2]],
});

It is a function rather than a method because there is no connection yet: the file it writes is the file a program connects to afterwards. The path must not exist, since a load builds a database rather than adding to one, and a path that already holds one is a caller who meant a different path. What comes back is what went in, as { nodes, rels, columns }.

Edges name rows by position, counting from zero, because at load time a row has no other name. They go in as pairs, or as a flat Int32Array or Uint32Array for a program that built them in memory and would rather not make a million small arrays to hand them over. The same edge twice is one edge, and an edge naming a row the table has not got is refused rather than written, because a builder handed one would either invent the row or lose the edge.

A column goes in as an array of values or as a typed array. The first value of an array settles what the column holds and every value after it has to agree, which is the appender's rule, with the same one widening. A typed array is read as the numbers it already holds, which is one pass over memory rather than a runtime call per value, and every integer width lands as the INT64 the store keeps.

Everything the caller passed is read on the thread that owns the runtime, because that is the only thread allowed to read a JavaScript value, and everything after that runs on the threadpool: the edges are sorted, the graph is built, and every column is encoded and written to disk. So the event loop is free for the whole of the expensive part, which on a load is all of it.

Over a million rows on this machine, from npm run bench:load:

one column, typed array             51.3 ms      51 ns/row
one column, plain array             92.2 ms      92 ns/row
two columns, with names           1060.1 ms    1060 ns/row
two columns and an edge each      1096.9 ms    1097 ns/row
the appender, for contrast        2124.5 ms    2125 ns/row

The last line is the same two columns through the appender, which is the closest comparison there is: a load is about twice as quick and is the only one of the two that can write the edges.

41 tests in test/load.test.mjs, covering the stats, row order, the edges as a pattern walk, a rel with its table/src/dst/ord, a two hop path, no edges, no columns, the default rel name, a duplicate edge, a flat edge list, every typed array width, a column of every value kind, widening, refusing to write over an existing database, fifteen bad option objects each also asserting no file was made, nine mixed type columns, and a liveness check over 200k rows. Part of the milestone item that has the TypeScript client reaching the Python one.