Read a result down its columns

[tamnd]

query builds an object a row and a JavaScript value a cell, which is what a program reading a hundred rows wants and the wrong shape for a million. conn.columnar(...) runs the same statement and hands back the buffers instead.

const read = await conn.columnar(`MATCH (p:person) RETURN p.age AS age`);
read.rows; // 1000000
read.columns[0].values; // a BigInt64Array of every age, and not one object

The buffers are the engine's own, moved rather than read: the pointer V8 is given is the pointer zu::query::column filled, and the allocation is freed when the typed array is collected. So a column of a million integers crosses the boundary as a pointer and a length. From npm run bench:columnar over a million rows:

one integer column, columnar       38.4 ms      38 ns/row
one integer column, rows          243.1 ms     243 ns/row
a float column, columnar           31.2 ms      31 ns/row
a float column, rows              236.2 ms     236 ns/row
a string column, columnar          50.3 ms      50 ns/row
a string column, rows             262.0 ms     262 ns/row
three columns, columnar            75.8 ms      76 ns/row
three columns, rows               632.2 ms     632 ns/row

Walking what came back costs the same either way, at about 14 ns a row for a sum over the buffer and the same over the rows, which is in the README because it is where the win is not. V8 reads a property of a small object about as fast as an element of a typed array. What it cannot do is make a million of those objects for nothing, and that is the whole of the six to eight times above.

Every column says what it is, so reading one is a switch on type rather than a series of tests for what is there. values carries everything of a fixed width, a string column has data and offsets, validity is one bit a row and is null when nothing in the column is, unit says whether a cell counts days, nanoseconds or months, and zone is the minutes east of UTC a column of zoned times was written with.

That layout is Arrow's, which is the point of it. apache-arrow wraps a buffer of this shape without copying it, so a table is eleven lines and no dependency of this package, and the README prints them rather than shipping them: a client that hands out an Arrow object has to agree with one version of Arrow forever, and a client that hands out the bytes agrees with all of them. The recipe is run in the test suite, including the assertion that table.getChild("id").data[0].values is the same array the engine filled.

Two things are not buffers and both are named by the type rather than found out by looking. A column of nodes, rels, paths, lists or records has no fixed width cell, so it arrives as items, holding the same values query would have made, and a column of nothing but nulls has a length and nothing else. A column that mixes two types is refused, naming the column and the row that did it.

bigIntMode says nothing here, since a columnar read has one physical layout per type. It still decides what is inside items, where this client is making objects anyway.

26 tests in test/columnar.test.mjs: every type and its buffer, the bit order of a boolean column and of a validity bitmap, a null row keeping its cell, a column of nulls, a result of no rows, a statement that projects nothing, the mixed column refusal, parameters, a signal, a closed connection, the Arrow recipe with and without nulls, the columns going straight back in as a registered frame, and a million rows read while the event loop keeps ticking.

This is the last piece of the milestone item that has the TypeScript client reaching the Python one.