Add zu2, a hash index over a hybrid log - #411
Open
tamnd wants to merge 162 commits into
Open
Conversation
Owner
Author
|
Pushed the C API as a second commit on this branch, because the crate has no consumer without it. graph-bench drives every engine it measures with query text over a process boundary, and zu2 has neither a query language nor a server, so the traversal numbers in 07-benchmarks.md come from a Rust bench rather than from the harness the rival engines are measured by. That is the gap this closes.
The traversals are the point. A hop in zu2 is an indexed load, so a host that called back across the boundary once per vertex would be measuring the boundary. Two things worth reading in the diff rather than taking on trust: The buffers are copies and the header says so instead of implying otherwise. A neighbour list is pinned only while the epoch is announced and the epoch ends when the call returns, so a returned pointer into the storage would point into a block a writer may already have replaced. The field order in Tests: 15 in Both smoke runs print |
zu1 reads a point key with a full label scan and commits with a fold plus an fsync, so a read is linear in the table size and a durable commit caps at 266 a second whatever the device can do. Neither is a tuning problem, so this is a second engine rather than another pass over the first one. zu2 is a new crate that depends on nothing in zu1 and that nothing in zu1 depends on, so both build into the same binary and a benchmark can put them side by side on one host. That was the point of keeping it separate. The record plane is FASTER's shape brought up to F2: a hash index with cacheline buckets, a 14 bit tag and the collision chain living in the log, over a hybrid log whose tail is in memory and whose body is on the file. Records are seqlocks so an update inside the mutable region takes no latch. Epochs retire pages. Compaction is lookup based and punches holes rather than rewriting, on all three platforms. The graph plane sits beside it on the same log. Vertex ids are dense, a neighbourhood is one cacheline with ten inline neighbours and a sorted doubling block past that, and there is one version cell per vertex rather than one per neighbour, which is what the 2025 survey measures the keyed edge designs at 4.1 to 8.9x CSR memory for. Two benches come with it, both in process against rusqlite so there is no harness in the middle. ycsb runs eight sqlite configurations so the comparison is against sqlite's best rather than sqlite's default, and it measures storage as bytes the device is holding against sqlite in two shapes and against zu1. traverse runs both engines through the same generic traversal code against a WITHOUT ROWID edge table keyed on (src, dst), which is the best shape sqlite has for it. Numbers from five machines are in Spec/2064g/zu2/07-benchmarks.md, along with the two defects the benchmark had before it produced them.
zu2 has no query language, so nothing outside this repo can reach it. graph-bench drives every engine it measures with query text and a process boundary, and neither is a shape zu2 has, which is why the graph numbers in Spec/2064g/zu2/07-benchmarks.md come from a Rust bench and not from the harness every other engine is measured by. This is the surface that fixes that: records, vertices and edges, and the traversals a host would otherwise write as a loop over hops. The traversals are the part that matters. A hop in zu2 is an indexed load, and a host calling back across the boundary once per vertex would spend more on the boundary than on the hop, so zu2_khop, zu2_reach and zu2_triangles run the whole walk inside one call and inside one announced epoch, and hand back only the answer. Buffers a call hands out belong to the session and are good until the next call on it. They are copies, which the header says plainly: a neighbour list is pinned only while the epoch is announced and the epoch ends when the call returns, so a pointer that outlived the call would point into a block a writer is free to replace. Concurrency is checked rather than left undefined. A session carries a flag a call takes on the way in, and a second thread gets ZU2_MISUSE_CONCURRENT instead of a buffer being written under it. The status numbers are libzu's where the two libraries overlap, so a host that links both carries one table. The field order in Zu2Session is load bearing and the comment says so. Fields drop in declaration order and the session holds an epoch slot it hands back on drop, so the Arc that keeps the database alive has to be the last field. With it first, closing a database before its session freed the Db and then wrote the slot back into it, which showed up as an index into an empty slot table. Tested from Rust through the extern functions, because what is being tested is the boundary, and from C through the header, because a hand-written header can disagree with the library about a layout and nothing in Rust would notice. The C smoke test links against both the static archive and the shared library.
The graph-bench micro suite asks four questions this could not answer. Variable length expansion is `-[:EDGE*1..3]->`, which is a reach with a depth on it, and zu2_reach had no depth and counted the seed as one of its own answers. Shortest path had nothing at all. Two of the queries are undirected, and the engine keeps an out list and an in list and has no third list for an edge with no arrow on it. So: ZU2_BOTH, taken by every call that takes a direction and merging the two lists rather than summing them, since a pair of vertices that point at each other is one neighbour. zu2_reach grows a max_depth and drops the seed from its own answer, which is the difference between the question a host asks and a walk of the component. zu2_shortest is one breadth first search that stops on arrival and reports not found as an answer rather than an error. The undirected reach can walk back along the edge it arrived on, so a seed with any neighbour is two hops from itself. That is reachability over an undirected graph and it is not Cypher's rule about a path using a relationship twice, and the header says which one it is so an adapter does not map the wrong query onto it. 17 Rust tests and the C smoke test against both the archive and the shared library.
A durable commit in zu2 was a write past the end of the file followed by fdatasync, so the barrier had an inode size change and an extent allocation to commit as well as the bytes. crates/zu2/examples/appendcost.rs measures what that costs against a write into blocks the file already owns: 4.1x on server1, 4.6x on server2 and 1.9x on server3. So the log now keeps a megabyte allocated past its write frontier. fallocate without KEEP_SIZE on Linux, F_PREALLOCATE and a set_len on macOS, and a set_len on Windows, where SetFileValidData is the only allocation call and it hands out whatever was on the disk, so a database has no business asking for it. A filesystem that refuses is asked once and then left alone. The reservation is not data and does not get counted as data. Db::disk_bytes gives it back before it measures, the log gives it back when it closes, and the storage numbers are what they were. A file that carries one into a crash costs nothing either: recovery now resumes at the end of the last record it accepted rather than where its scan stopped, so an append lands after the records instead of after the blocks nobody wrote. Four tests, one per place the reservation has to be invisible.
The ycsb sweep cannot settle whether provisioning helped: on machines sitting at load average 28 the sqlite rows themselves move by 3x between two runs of the same binary, so a 2x move in a zu2 row is inside the noise. This gives the log a reservation size instead of a constant, with zero meaning never reserve, and rewrites the probe to open two databases in one process and alternate rounds between them. A burst of load from something else on the machine then lands on both sides rather than on whichever one was unlucky, and the async row is the control: it never touches the file on the commit path, so if it moves the machine moved. appendcost grows a fourth shape, fallocated, between sparse and allocated. fallocate hands out unwritten extents and a filesystem that converts one on the first write is still making a metadata change there, so this is what says whether reserving blocks buys the allocated shape's speed or only part of it. On this laptop, which is APFS and where a sync is not a device barrier, the reservation is worth 1.26x.
The fallocated shape settles the question the last commit asked. On server2 a growing file does 160 durable writes a second, a file that was fallocated does 104, and a file that was written and synced first does 1207. On server1 it is 54, 69 and 237. So asking for the space buys almost nothing on ext4: fallocate hands out unwritten extents and the first write to one is still a metadata change that fdatasync has to commit, which is the same cost in a different place. So the log writes the reservation. A megabyte of zeros and one sync per megabyte of log, above the frontier only, since the bytes below it are about to be written anyway. Zeros because page padding and holes both already read as zeros, so recovery treats the unused part of a reservation as the end of the log without knowing that reservations exist. On this laptop the reservation goes from being worth 1.26x to 1.81x on the durable probe. The new test is the one that would have caught the earlier version: a file whose blocks add up to less than its length has holes in it, and holes are exactly what asking for the space leaves behind.
A flush that has a megabyte of records behind it is a load, not a commit. It pays one inode update for a megabyte of data it was going to write anyway, so the reservation buys it nothing, and writing the zeros first would double what it puts on the device. The reservation is for the small durable commit, where the metadata is most of the cost, so that is the only case that now pays for it.
The terminology table has one entry for this and zu2 was ignoring it: a node is never a vertex on one page and a node on the next. The lint only reads prose, so the smallest fix would have been the doc comments, but leaving zu2_add_vertex next to a comment about adding a node is the inconsistency it exists to prevent. So the identifiers move too, which is cheap now and expensive once anything outside graph-bench links against the C API: zu2_add_node, zu2_node_of, zu2_nodes, max_nodes, KIND_NODE, NodeOutOfRange. Also the last prose hit on the branch, a bench comment that said query planners where the table says optimizer.
The end to end sweep on server2 says the durable rows went up by three to four times and the async update row went down by about half, from 93889 and 142793 to 62728 and 55120 operations a second at one thread. That is the reservation being paid for by the wrong thread. In async mode nobody is waiting for the flusher, so the metadata cost of growing the file is not on anyone's critical path, and the flusher wakes every millisecond with tens of kilobytes behind it, which is under the bulk rule, so it was writing a megabyte of zeros for every megabyte of records: twice the device traffic to save a cost nobody was paying. So the flusher does not provision and a commit does. The async path is now the same code it was before any of this.
Log::make_durable waited for global epoch quiescence before every device write. That answers a much larger question than the one it asks: it waits for every session to leave whatever it was doing, readers included, so a commit cost what the other threads happened to be doing rather than what the device charges. Each session now publishes the lowest address it may be writing, in the same cacheline as its epoch, and the flusher writes below the lowest of those. A session that is only reading publishes nothing. An appender publishes before it claims tail space, so a flusher whose tail read saw the claim also sees the frontier, and one that did not is bounded by the tail instead. An in-place rewrite has no claim to carry it, so that one is a sequenced store of the frontier followed by a sequenced load of the flush target, against the mirror image on the flusher's side: either the flusher waits for the rewrite or the rewrite stands down and appends. crates/zu2/examples/commitwait.rs measures it as a ratio, which is what survives a machine that gets busy halfway through: one writer alone against the same writer with seven readers beside it. On the laptop that was 1.87x and is now 1.18x. The regression test is the one that hangs rather than fails: the reader in it does not stand down until the commit has already returned.
Two probes and two counters, because the story in 07 section 6.2 turned out to be wrong and the only way to find that was to measure the parts separately. examples/epochwait.rs times the two ways of answering "is everything below me complete?" with no log and no device underneath. Waiting for the epoch to turn over costs 4 ns with nobody else running and 2182 ns with eight sessions announcing and standing down as fast as they can. Reading the write frontiers costs 1 ns and 405 ns. So the new way is 5x to 29x cheaper and both of them are microseconds, which is nothing next to an fsync, so this is not what made durable commits slow. examples/commitwait.rs and the two counters say what did. Log::syncs and Log::commits are device writes and durable commits since the log was opened, and the ratio is whether group commit is grouping. On the laptop with five writers it is 1.34, so a commit is very nearly paying the device on its own and the leader arrangement is buying almost nothing. That is the reason durable throughput barely moves with thread count, not the epoch wait.
This was referenced Aug 19, 2026
A copy is appended before the compare and swap that would publish it, so a copy that loses that race to a concurrent update stays on the log above the record that beat it with nobody pointing at it. It was also given a fresh version, so it looked newer by both of the things a replay could go on, and the replay installed it. Every key read correctly in memory and thousands of them came back a round or two behind after a reopen (#436). The copy now carries the version of the record it copies, which is what it always was, and the replay installs a record only when its version is at least the version of the record the index already reaches for that key. Not greater than: a pass that copies a copy leaves two records with the same version and the higher address is the one that survives the punch below it. The narrow test builds the shape by hand rather than racing for it, so it fails without the fix every time instead of one run in three. The compaction tests gain the workload it came out of, passes from one thread while four others write, and the value length alternates by round so that an update outside the mutable window appends instead of rewriting in place, which is the #435 fix and is what made this frequent enough to see. The same race on edges is not fixed and is #437.
The tail allocator moves to the next page when a record does not fit in what is left of the current one, so a page can end with eight, sixteen or twenty four bytes belonging to no record. The scan walked to that address and read a header out of it, which is past the end of the page allocation: key_len is at offset sixteen and the checksum at twenty four. Next door is usually zeros, so the scan read it as page padding and moved on and nothing showed. When it was not, the scan took the garbage for a record, failed the checksum and stopped, losing every record above that page. Three of thirty six parallel hammer runs, and address sanitizer says heap-buffer-overflow on the header read. Check the room first and treat a gap too small for a header the way padding is already treated. compact::compact had this right in its loop condition, so this was the one scan of the two that did not.
The leader held the flush lock across its own barrier, so a thread
arriving during a sync could not register and could not learn it was
already durable except by winning the lock. Whichever thread the
platform handed the lock back to kept it, and everybody else waited.
Measured on the laptop at eight writers: 12338 queued committers sat
an average of 1254 bytes above the durable boundary, which is one
record, and the fast path before the lock never hit once in 18000
commits.
So the barrier moves out of the lock. A committer that finds nobody at
the device claims it, computes its target and writes with the lock
released. A committer that finds somebody there waits on a condition
instead, and the leader wakes every waiter at once when it publishes.
A leader takes everything appended so far, so most of the threads it
wakes find themselves durable and return without a barrier.
The flush state splits in two for this: Device holds what the writer
owns while it writes, Flushing holds who is allowed to write. The
flusher joins the same protocol and stands down when a commit is
already at the device, rather than queueing behind it.
commitwait on the laptop, ten cores, median of five rounds, both
measured on this machine rather than carried over:
writers 1 2 4 8 16 32
before ratio 1.00 1.00 1.00 1.04 1.05 1.35
op/s 49134 48225 46895 48100 49729 57506
after ratio 1.00 1.33 2.26 4.04 7.87 14.04
op/s 47333 66459 101186 153849 240763 302491
Thirty two writers were worth 1.17x one writer and are now worth
6.39x. The servers and gamingpc are next, and their fsync is a real
barrier where the laptop's is 20 us, so the ratio matters more there
than it does here.
compact::keep_edge decides an edge record by asking the adjacency what it looks like now, and Session::edge puts its record on the log before it touches the adjacency, because an edge in memory that is not on the log is a lost write. So a pass reading between a writer's two halves was told an edge was present, appended an add copy above the remove record that was already down, and the next open brought the edge back. The version rule that fixes this for keyed records (#436) does not reach here. Edges are not keyed, so a replay has no per edge version to compare against, and building one would need a map the size of the edge set on a scan whose whole point is that it carries no state but the index it is rebuilding. So close the window instead. A writer holds Graph::order_edges across its append and its apply, a pass holds the same one across its question and its copy, and neither can see the other with one half done. The lock is striped a thousand ways over the source node, and it is deliberately not the seqlock in the node entry: that one is taken by every reader of the node, and holding it across a log append that can allocate a page would stall the readers of a hub for the length of the append. Readers never come near this one. It fixes a second thing that had not been written down. Without an order, two writers on the same edge could append in one order and apply in the other, and memory and the log would disagree from then on. The other direction in #437, emitting the adjacency a node holds now rather than deciding records one at a time, was dropped because it writes the whole neighbourhood of a hub on every pass that touches any one of its edges, and because it needs a lock across the snapshot and the appends anyway. The test builds the window rather than racing for it, which is the rule that came out of #436. a_pass_cannot_copy_an_edge_a_writer_has_already _removed puts a remove record on the log by hand, leaves the adjacency untouched, and only then lets a pass look, which is exactly the state Session::edge is in between its two lines. Without the order the reopened graph holds the edge the writer removed. Cost, traverse bench edge insert, 100000 nodes at out degree 8, the two binaries alternated three times on gamingpc with nothing else running: async, 1 thread before 7149k 6873k 6641k after 6722k 7565k 7463k durable, 8 threads before 12488 12399 12718 after 11451 12706 12767 Means are 6888k against 7250k and 12535 against 12308, so the two agree to within three percent on a machine whose own spread is five, which is what an uncontended mutex next to a log append and an adjacency update should look like. The laptop was run first and said nothing at all: its spread between repeats of one binary was two to three times. Also formats a test the previous commit left unformatted.
The go-ycsb sweep sizes the zu2 index off the record count, and at 20000 records over 8192 buckets the adapter's storage line reported 19991 index entries rather than 20000. Nine keys short, deterministic, and absent when the same load ran at the default table size. It is the displacement the index was designed to do. A bucket is eight slots with no overflow pointer, so the ninth key to land in one takes an entry over and points its previous at what the entry held, and the chain lives in the log. A displaced key stops owning an entry, so occupancy stops counting it. 20000 keys over 8192 buckets is a mean of 2.44 and the Poisson tail past eight is about ten keys, which is the size of the shortfall. Nothing said that out loud, so these tests do. Every key reads back at the bench sizing and at one single bucket, a delete under crowding takes only its own key, and an update under crowding is seen by the next read.
`Db::compact` loops until another pass would not pay for itself, and the test for that was whether a pass found everything it read still live. The reasoning was that such a pass has just put its copies at the oldest end of the log, so the next one would read those same records and copy them again. True as far as it goes, and it terminates, but it is the wrong test in both directions. It stops early when the log opens with a live block, because the first pass over that block copies everything and the loop ends with the dead records above it untouched. It does not stop at all when the live and dead records are interleaved, which is what a random update workload leaves. Every pass then has a dead record in it, the test never fires, and the loop walks its own copies up the address space. The go-ycsb sweep is what showed it: 250000 records loaded and 250000 random updates over them, then one `compact()`, and the log spent 7304 MiB of addresses across 136 passes to reclaim a 571 MiB file down to 251 MiB. It took 26.8 seconds to do it. The ceiling is now clamped to the tail as it stands when the call is made. Nothing above that address is the call's to reclaim: it is either a writer's record or one of this loop's own copies. The region only shrinks from the bottom, the loop ends when `begin` walks up to meet the clamp, and a pass that finds nothing left to read still ends it. The same shape now copies 258 MiB across 9 passes in 1.4 seconds and reclaims the file to the same 251 MiB, so it is 28 times less writing for the same result. The new test is the one that failed on the old code: it scatters the garbage rather than leaving it in a block, which is the shape the old rule could not see, and it bounds the copying at twice the live set.
keep_edge kept a remove record whenever the edge was absent from the adjacency, which for a removed edge is permanent, so the record matched on every pass and was copied forward on every pass. A graph that churns edges paid 48 bytes a deleted edge, forever, and paid to copy them again on each pass after that. A remove is never live and the reason is the prefix property everything else in the pass rests on. A region is always a prefix, so a record above it replays after everything in it. If the adjacency has the edge the remove is stale. If it does not, the last edge record for the pair on the whole log is a remove, and that remove is either this one or one above the region. One above does the job alone. This one being the last means the pair has no add above the region, the rest of its history is in the region about to be punched, and dropping it leaves the pair with no record anywhere, which replays as absent, which is what the adjacency says. Measured on the ring churn in the new test: 8000 added then removed edges left 383568 bytes on the log before, and nothing after, against the same workload with no edges in it. Fixes #452
install scans the eight slots and then acts on what it saw. A slot mid-insert holds a tentative claim, which names a tag and no address, so the scan cannot tell which key it is for and walks past it. That makes the scan report a key missing when it is being inserted right now, and both paths that create an entry act on the report: the empty path takes the first free slot, the full path takes over tag % SLOTS. claim_lost only closes the window while the other claim is still visible, and the interleaving that gets through is the one where it is not. A claims the last free slot for k, B walks past the claim and finds the bucket full, A finishes and stores a real entry for k, B takes over a slot and writes a second one. Two entries for one key is not a lost record, both chains hold it, but a lookup answers from the lower slot and a replay answers from the higher version, and after that interleaving those are not always the same record. A probe on a one bucket index with two racers measured 258 duplicates in 20000 pairs and 6 of them left memory pointing at the older of the two records, which is a value that reads back one way now and the other way after a reopen. So a scan that walked past a claim does not get to create an entry. It goes round the loop again, and on the retry the claim has either resolved, in which case the key is found and this is an update, or it is still there, in which case round again. Reads are untouched: a read that walks past a claim is reading a key that was not there when it started. And the claim comes off when the append between claiming a slot and storing the entry fails. A slot left tentative is one nothing can look through and nothing can reuse, and under the retry above it is also one every insert for that tag waits on forever. StorageFull is not hypothetical, this laptop has produced it. The grouping ratio test takes the best of six attempts rather than three. It failed once in a full suite run on a busy machine and passed twenty times on its own. The bar of 3.0 is not the fragile part, the number of chances at it is. Fixes #454
Session::edge appended the record and then applied it, which is the right order and is what makes an edge durable, but Graph::apply is also where the ids are checked. So an edge the graph had no room for was refused after its record was already down. The caller got NodeOutOfRange and every reason to think nothing had happened, and then replay_edge hit the same record on the next open, got the same error, and carried it out of Db::open. Ten nodes, one good edge, one add_edge(0, 99999) against a max_nodes of 1 << 14, and the file could never be opened again. The ids are now checked before the append, which covers the whole reachable case: ids from add_node are bounded by the capacity already, so an out of range id can only come from a caller that made one up. There is a second way into the same replay error that needs no bad call: reopening a file with a smaller max_nodes than the run that wrote it. Every edge in it is fine and the options are the problem. Dropping the edges would open a database that has quietly lost part of its graph, so that still refuses, but it refuses with GraphTooSmall naming the number that would open the file instead of repeating what the write path says. replay_node already skipped this case, so the two halves of recovery no longer disagree about a record they cannot hold. Graph::allocate is a compare and swap now rather than a fetch and add with the check after it. Adding first and looking after moved the counter whether or not the id was usable, so a caller that kept asking past the end walked nodes() up past capacity() and it reported nodes that cannot exist. Fixes #455
Fixes #456. The tail allocator claimed its bytes and then found out the page was past the end of the page table, so a refused append left the tail in a page that cannot exist. Both a flush and a durable commit take the tail as their target, so from the first refusal on the database could not be made durable at all, and under Async that is every record since the last flush.
Fixes #458. A page that leaves the table is queued for a deferred free, and the free only runs once the epoch it was queued in has passed. The bump that would let it pass was only ever done by compaction, so an eviction with compaction off freed nothing, and Log::drop used the same epoch test and leaked whatever was queued after the last bump.
Fixes #459. Epochs::claim panicked when every slot was taken, and across the C API that is an abort, so go-ycsb at more threads than the default 128 died with no message. The claim is fallible now, with Error::NoSessions and a ZU2_NO_SESSIONS status, and zu2_options carries a sessions field so a host can say how many threads it has. Flushing and compaction claim from two slots the host cannot take, so sizing sessions for the workers is the right sizing and a compaction pass is never what runs out.
Part of #396, Z9. A displaced key costs a log dereference per lookup, so a table four times too small turns a point read into four random reads and the whole argument for a hash index over a hybrid log goes with it. The table doubles at half full instead, which is before displacement starts rather than after. The split is FASTER's: bucket b of the old table feeds exactly b and b + old_len, because the index is the low bits of the hash and doubling looks at one more of them. What is different here is that the entry has no spare bit to say which side it belongs to, so the split reads one record per entry and takes the side from the key, and a foreign entry names a chain of more than one key and goes to both. Splitting per key rather than per entry is what makes the doubling relieve crowding, so a split walks the chains and gives every key it finds its own entry, with a bounded walk and a copy the bucket over whole fallback that is always correct and never helps. Migration is per bucket and lazy: an operation drains the bucket its key came from before it touches the new table, and the flusher drains what the traffic did not. The grower publishes the migration before the new table, so an operation that saw the new table also sees the migration, then waits for the operations already running and only then opens the migration for draining. An operation that finds one unopened leaves its epoch and comes back, which is what keeps it from waiting on itself. Fixes #462 along the way, because growth is what made it happen without anyone asking. An index entry names the head of a chain and everything behind it, and the links were written under whatever table that run had. A scan filling a differently shaped table and installing on top of an entry its record does not point into drops every key that entry reached, and nothing in the file is wrong to show for it. So the scan repairs the link: a record about to take an entry over that does not already point at what the entry holds gets its previous rewritten and its checksum refreshed, and the pages that changed go back to the file before anything can evict them. Index::presize counts the records before the scan installs anything and builds the table for them, which keeps the repair to the few percent that were displaced under the old table and also stops a reopen starting crowded. #463 is the torn write hole in the write back. Fixes #466, which the doubling is also what brought out. The foreign bit is a statement about the entry and not about the chain: cleared means this entry answers for the key at its head record and for nothing else. It is not a claim that the chain holds one key, because a split names records out of a chain without taking the chain apart and the records under a placed one stay where they were, other keys and all. The read path did not read the bit at all, so an update to one key could find itself through the entry of another key with the same fourteen bit tag, swing that entry, and bury the other key under a head that is not its own. The next split then read the cleared bit as licence to stop at the head and the buried key was gone with no error anywhere. chain_find and recover::chain_version now both stop at the head for an entry without the bit, so a lookup, a write and a reopen all read a chain the same way. The split also deduplicates by version rather than by first sighting, because a key can be in more than one chain of a bucket with an older copy displaced under somebody else, and the walk has to place the newest of them. Db::index_resizing so a test that counts entries can wait for the table to stop being replaced under it, and tests/resizing.rs for the protocol from the outside: nothing lost under writers, nothing stale to a reader crossing a doubling, and no key given an entry on both sides of a split.
scan_borrowed tells its caller per record whether the value will outlive the call, and the caller keeps a pointer to the ones it was told will. The test was on the address alone, cold or above the read only boundary, and it missed the third answer locate can give: a record whose log page has been evicted is read into the session's one scratch buffer, which the next record overwrites. Every row of a scan then came back pointing at the same bytes. A reopen is what makes it show. The planes are restored before the pages are, so the first scans of a run are served out of pages the warmer has not read back yet. go-ycsb workload E at ten thousand records failed the scan integrity check every time, at a thousand it never did, and the pairs the C API handed back had seventy nine ascending keys and one value pointer. locate now says which of the two places the bytes are in, since the caller cannot work it out from the address and the answer is only true of the moment the load happened, newest_record carries it through, and both stability tests gain it. Reads had the same hole and it did not bite: a read hands out one slice and the contract ends at the next call, and until then the scratch still holds the right record. The test has to keep the pointers rather than copy the bytes inside the callback, which is what every Rust caller in the tree does and why nothing here caught this. Fixes #751
A walk that failed released its own protection on the way out, so after an error zu2_scan holds nothing. It was still taking one of the two branches: setting scan_held, which has the next enter give back a protection this session does not own, or calling scan_release, which does the same thing a call earlier. Both land on a slot that is already idle, so neither is a fault today, but they are claims the code should not be making and zu2_read has had the order the other way round all along. Found auditing the borrowed scan lifecycle after #751. The rest of that audit came out clean: zu2_session_close releases a held protection without going through enter, so a session closed in the middle of reading a scan does not leave a floor under reclamation, and Slotted::drop puts the slot back at FREE with parked cleared, so a session dropped rather than closed cannot either.
Three gaps, all found going through the header after #751. zu2_read said the buffer is the session's. Since #738 that is false for most reads on a warm database: a record below the read only boundary whose page is in memory is handed over where it lies. The lifetime was right and the sentence in front of it was not, and a caller reasoning from it would conclude the bytes are heap and survive a compaction. Neither zu2_read nor zu2_scan said what a live pointer costs. The bytes are good until the next call because the session keeps its epoch announced, which is a floor under reclamation for as long as it is out. A host that scans and then goes away pins the log, the fix is to make another call, and neither of those was discoverable from the header. Slotted::park says it correctly on the Rust side and now the C side says it too. zu2_session_error stated no lifetime at all, where zu2_db_error two lines above states one, and it did not say it is NUL terminated or empty on success either. Same answer as the db version in all three. The other four pointer returning entry points were already right. zu2_neighbours, zu2_khop and zu2_reach copy into the session's buffer, which they say, and being copies they carry no lease. zu2_version is static. #753.
Every read and every scan hands the caller a pointer into memory the library owns, and 6302578 wrote down how long each one is good for and what ends it. Nothing was checking that the library agrees. The Rust tests hold Rust borrows, so the compiler settles these questions before they run and they cannot ask the one a C host actually asks, which is whether a pointer handed back three calls ago still points at what it did. The scan is the one that matters. A scan hands back an array of pairs and says all of them are good at once, and that is only true if the library did not answer two of them out of the same buffer. It did, until 58102a8: a record whose page had been evicted was read into the session's single scratch buffer and handed out as if it were resident, so the next such record in the same walk overwrote the one before it. This is #751 in the shape the bug had. Getting that shape needs a database bigger than the memory it is allowed, which is why it writes 30 MiB against memory_pages of five and scans from the first key. A small database walks nothing but resident records and would pass either way, which is most of why nothing caught #751. Checked against the code it is meant to catch. With crates/zu2/src/db.rs reverted to 58102a8~1 it fails on both of the checks that are about aliasing, wrong == 0 and distinct > 0: all two hundred values came back at one address carrying the last record's bytes. At head it passes, and it passes under -fsanitize=address,undefined too. Also covers the read bound across calls on the db handle rather than the session, that a miss leaves nothing behind, that the session error buffer is empty and NUL-terminated when nothing has failed, that the lease a borrowing scan leaves is ended by the next call rather than hanging it, and that closing a session with a lease still held releases it, checked by compacting afterwards. Not run by CI, same as readfloor.c: it writes 30 MiB. The header comment has the two command lines. Refs #753
A record reaches the tier by surviving a lap of the log without being written, and reading it back is a pread of a device. A decompress of a few microseconds sits under a cost that is already there, so the tier is where a coder pays and the hot log is not. zstd at level 3, values only, and only when the value is at least 256 bytes and the frame comes back smaller than what it was given, which is what keeps this safe on data that arrives compressed already. Nothing in the format moves. The frame goes where the value went and its length goes in value_len, so size_of(key_len, value_len) still describes the bytes on the device and every walk, offset and crc in the tier keeps working. What says a value is a frame is a flag bit on kind rather than a kind of its own: a kind of its own would have to replace the record's real kind, and KIND_VERTEX is a keyed value that settles into the tier, so replacing it would lose the thing that tells a replay to restore the node id. Three places a cold value leaves the tier and all three expand it: a load, a cold pass copying a survivor forward, and the rehome a reopen's scan does. The last two matter because a survivor can go to the hot log, and nothing up there expands anything. Options::compress_cold turns it off for the A/B, and the option only says what an append does. A record carries its own flag, so turning it off leaves everything already written readable, which is what a setting on a durable format has to do. cold_value_bytes is what it bought, counted where a record enters the tier: the values it took and the bytes it wrote for them. Reclaimed records are in both, which makes the ratio a property of the data rather than of when it is asked for. a_stale_tier_gives_its_blocks_back had eight rounds and a fixed shape, and that shape stopped working once the same records took 2.4 MiB rather than 3.1. A tier under a page is not compacted and a pass that reaches nothing puts the bar a page higher, so the smaller tier never got looked at and the test read its new arrivals as growth. It is a loop now, and the fixture is sized in bytes the tier holds rather than in records. Refs #725.
…e was The first cut of the cold coder called zstd::bulk::compress and zstd::bulk::decompress, which build a context per call and, on the decode side, allocate a buffer of whatever bound they are handed. The bound available was a page, so every read of a compressed cold record built a context and allocated four MiB. The end to end half of benches/compress.rs has what that cost: a cold read of 1.17 us became one of 34.94 us, against a decode that takes 1.8 us on its own. The context is now one per thread, built on the first compressed record that thread meets and kept, and the buffers a read needs are thread local too, so an expansion allocates nothing after the first. Same for the encoder on the append side. A cold read is 4.71 us against a plain one at 0.86 on this laptop, where the file is in page cache and the plain read is not a device read at all. On a device the comparison the gate asks for is 1.8 us of decode against a pread of tens of microseconds. Sizing the decode buffer needs the value's length, and a zstd frame carries the content size only when the encoder was told it in advance, which neither the bulk compressor nor the free function does. So four bytes of length go in front of the frame. That is the only thing in the record that is not a frame, and it is exact rather than a bound. The frame is built in a scratch buffer and copied out behind the length because compress_to_buffer writes from the start of what it is given rather than appending to it, so a prefix written first is a prefix the coder overwrites. benches/compress.rs asks the question of the engine as well as of the coder now: the same database built with the coder and without it, the file and the read side by side. Refs #725.
Eviction runs where a thread opens a page, and it can only drop a page whose bytes are already on the device. A burst of async appends outruns the flusher, so the loop stops at the first page that is not flushed yet, which is correct while the burst is going on. What is not correct is what happens next: the last thing to open a page is the last append, so when the flusher catches up a moment later there is nobody left to finish the job, and a database that has stopped writing sits above its bound for as long as it stays open. That is the shape of every run in this series. A load, and then a read phase that appends nothing. The test that goes with this writes 80 MiB under a bound of two pages and then stops: 20 pages resident, 8 MiB asked for and 80 MiB held. So the maintainer evicts after each flush, which is the thread that just made those pages evictable. 20 pages becomes 3, and 3 is the bound plus the page the log ends in, which is the engine's existing reading of memory_pages and what an_evicted_page_gives_its_memory_back asserts. The head moves with fetch_max rather than a store now, because a writer opening a page and the maintainer finishing the job behind it can be in the eviction loop at the same time and the slower of two stores would put the floor back down. Refs #636.
Every resident page in zu2 is anonymous heap memory. A page below the mutable window is read only, flushed, and byte for byte identical to the region of the file underneath it, so the private copy of it is memory the kernel can do nothing with but swap, while the same bytes sit in its own cache where it could have dropped them. Measured on server2 at a million records, workload e, loadavg 23.7 to 35.9: zu2 puts the records on the device in 1090.7 MiB against lmdb's 2486.0, and holds 1152.6 MiB of anonymous memory against lmdb's 17.6. The device column is a win and the memory column is not a tuning problem, it is the kind of memory. So Options::map_settled, off by default. The maintainer converts a page after the flush that settles it: below the read only boundary, below the flushed frontier, and already resident, each of which is load bearing and each of which has its reason next to the check. A cursor rather than a scan, because the conversion is one way and a log with no eviction floor would otherwise be walked from the head every time. The two kinds have to be told apart at exactly one place, the free, where one is a dealloc and the other a munmap, and the low bit of the slot carries which. A heap page is 64 byte aligned and a mapping is aligned to whatever the kernel maps at, so the bit is free in both, and page_ptr masks it off so everything above treats the two as the same thing, which they are for every purpose except that free. The mapping is PROT_READ on purpose. Nothing writes to a page below the window, so a mapping that faults on a write says loudly in a test what would otherwise be a quiet corruption of the file. No SIGBUS to worry about: the only call that shortens the file is trim_tail, which never goes below the write frontier, and a mapped page is always below it. A hole punched by compaction reads back as zeros through a mapping exactly as it does through the pread it replaces, so that path is unchanged. The whole zu2 suite passes with the default flipped on, compaction, recovery, damage and crash suites included, and passes with it off. The new test asserts the split rather than the total, reads every record back through a mapping, and then compacts beside it so a pass that reclaims a mapped page has to give it back the right way. Off by default because the read of a mapped page is a fault where the read of a heap page is a load, and what that costs has not been measured on any host of record. zu2_options gains map_settled so the A/B is one flag. #757.
zu2_resident_pages says how many pages are held and says nothing about what kind of memory they are, which after #757 is the part that decides what the number costs the system. Mapped pages are page cache the kernel can drop and fault back; the remainder is heap it can only swap.
evict_settled ran remap_settled first and evict_behind second, so with both a memory bound and map_settled set, a page that had settled below the floor was mapped by the first half of the call and unmapped by the second half of the same call. Two syscalls a page, a mapping that never served a read, and page cache warmed only to be dropped. A bulk load is the worst case for it, because the flusher settles pages in bursts and every page in a burst lands below the floor at once. Eviction first means the remap never looks below the floor, since it already starts at max(cursor, head). With no bound evict_behind returns at its first line, so the common path is unchanged, and retire_pages stays behind the same condition it was behind before so an unbounded database still does no epoch work on a flush that changed nothing. This does not settle whether the bound should apply to a mapped page at all. A mapping costs no anonymous memory, so evicting one gives back nothing the kernel was not already free to take and buys a pread on the next read, and a page still transits the band between the floor and the read-only boundary on a steady write. That is a design question and it goes on its own issue.
remap_settled held self.allocating across the mmap. That is the lock a writer takes to open a page, so on a bulk load, where the maintainer has a burst of settled pages to convert, every one of them put a syscall in the way of the threads doing the writing. The mapping does not have to be made atomically with the swap, it only has to be published atomically. So the slot is read, the mapping is made, and only the publish happens under the lock, guarded by a check that the slot still holds what it held when the mapping was made. If it does not, somebody evicted or replaced the page in between, and the mapping goes back with a straight munmap rather than through the epoch, because no reader ever saw the pointer. The common case where there is nothing to convert now takes no lock at all, which matters because this runs after every flush.
Reviewing the mapping against the rest of the engine turned up one thing that is safe today and safe by accident. A settled page is only read from, which is what makes PROT_READ right, but recovery does write into a page it did not just append to: it repairs a hash chain through RecordRef::relink, straight through a raw pointer into the page, and its own safety note says the page is one that can be written back. Nothing goes wrong because recover::replay finishes before the flusher is spawned and the flusher is the only caller of evict_settled. That is an ordering in Db::open that reads like housekeeping and is not. Moving the spawn up, or giving a live compaction pass the ability to rewrite a record header in place, would put a repair and a mapping on the same page, so both ends now say so. No behaviour change. The symptom if somebody breaks it is a fault and not a wrong byte, which is what PROT_READ buys over a writable mapping, and that is written down too.
memory_pages counted every resident page, so a settled page held as a read only mapping of the file counted the same as a page of heap. That is backwards: the mapped page costs the process nothing it owns, the kernel drops it whenever the frame is wanted and faults it back if it is not, and the heap page can only be swapped. With a bound set, the two options together threw out the free page to make room for the expensive one, so #757 bought nothing for exactly the databases that needed it. The bound is now a count of anonymous pages, kept as a counter rather than a walk because eviction asks on every page it considers. Eviction steps over a mapped victim and moves the floor past it, which leaves pages below the floor that are still resident. The read path was already written for that: Log::resident says in its own comment that the head boundary is deliberately not consulted and the page pointer is the test, nothing outside log.rs reads head at all, and the other two writers of it only push it up to a compaction floor that has made the addresses unreachable rather than merely cold. evict_settled maps before it evicts now, the other way round from #757. The reason for the old order was that a mapped page still counted against the bound, so the remap and the eviction in one call undid each other. That cannot happen now, and a settled page should be offered the cheaper of the two ways out of anonymous memory first. Also adds zu2_anonymous_pages, so a caller reads one counter instead of subtracting two page table walks taken a moment apart. The ycsb driver was doing the subtraction and printing 1.8e13 MiB when it lost the race. #759
The distance rule got this for free. It evicted up to memory_pages behind the tail, and the constructor clamps memory_pages to at least mutable_pages plus one, so however small the bound was the floor could not reach the window. A count has no such geometry. Pages the flusher has not caught up with, or a page warmed at startup, can put the count over the bound while everything below the window is already gone, and the loop would then take the window itself, which is where in place updates are written. The cursor stops below the read only boundary now. The startup warmer asks for the anonymous count rather than the resident one, which is what the bound is about and is also a load where the other is a walk of the page table, taken once a page by every warm worker. #759
concurrent_sessions_keep_their_own_keys_under_compaction failed about once in eight loaded full suite runs and never in isolation. The fallback it has, compact once if the threads never triggered a pass, was not enough on its own. A pass may only touch pages below compact::ceiling, which is the flushed frontier and the read only boundary whichever is lower, and this test runs async. On a loaded machine the flusher is behind at the moment the threads stop, so the ceiling sits below the page the log starts in, the pass is allowed to read nothing, and the counter stays at zero on a run where everything worked. It syncs first now, which hands the pass the whole log up to the mutable window, the same thing a quiet machine was handing it by luck. The other test named in the issue is not changed, because what failed in it was never established: the assertion text was not in the output that was kept and it has not reproduced since, including under eight spinners. Its assertion prints the span and what the padding filled it to now, so the next failure says what it saw. Eight loaded release full suite runs clean. #763
The gate is the first option on #736 and it was already there in `compact_slice`: survey the region, work out how much of it is live, and skip the pass when the span is already inside the target rather than copying a region that has nothing to reclaim. Nothing held it in place, though, so a later change could take it out and every test would still pass. This is the test that fails when it does. A load of forty thousand records with no updates over a `compact_below` of eight megabytes, which is a log that runs well past the threshold with nothing dead in it. It asserts both halves: that the survey ran at all, so the run says something about what the survey decided rather than about a maintainer that never woke, and that the cold span stayed at zero. Checked the way a regression test should be checked, by breaking the thing it guards. With the survey short circuited the test fails with "the maintainer never surveyed, so this run says nothing about what the survey decided: 1 passes, 4194096 migrated", which is a whole page of live records copied into the tier for no reason.
#767 lists four planes of anonymous memory and says the first thing to do is measure them on a host rather than count them off the source. Three of the four were already reportable: the log pages through zu2_anonymous_pages since #759, the scan plane through zu2_ordered_bytes, and the mutable window is inside the page count by construction. The index was not, so the largest remaining line was the one nobody could ask a running database about. Index::bytes is maintained rather than counted. Counting means reading the live pointer and the migration pointer, and a migration is only a valid reference inside an epoch, so a metric that counted would either take an epoch to answer or would read a pointer it has no right to. The counter moves where the memory does instead: up when a table is installed, up again at a doubling for the new table and the state array, and down inside the closure the retirement defers. That last one is the reason the counter is an Arc. Decrementing at the unlink instead would be a line of code shorter and would report the drained table gone while it is still held, which for a memory figure is the wrong direction to be wrong in. A bucket is eight 8 byte entries in a repr(align(64)) struct, so the resting figure is exact rather than an estimate, and tests/resizing.rs asserts exactly that: buckets times the cacheline at rest, strictly more than that while a doubling is in flight, and back to exact once the drain has finished. The middle assertion is sampled during the load rather than waited for, because a drain under a single writer finishes in the writes that follow it and a loop that waited there would be waiting for itself.
#769. `remap_settled` walked from a high water mark, so a page the kernel refused a mapping for was passed once and never looked at again. The comment said a refusal costs one syscall a page and not a retry loop, which is right for a platform with no mapping call and wrong for a kernel that said no because the process was at `vm.max_map_count`. That is pressure that lifts, and stepping over it turned it into a database that stops mapping anything for the rest of the run and reports the count from before the refusal, which reads as a database that settled rather than one that gave up. The two are separate numbers now. `remap_from` is still the high water mark and still only goes up, and `remap_retry` holds the lowest page a refusal landed on, which is where the next walk starts. The walk itself carries on past a refusal, so the pages above a refused one are still mapped on the call that refused it, and the rewalk that follows costs no syscall because a mapped slot is recognised and stepped over. `remap_refused` counts them, exposed as `Db::remap_refused`, and is zero on every healthy run and on every run with the option off. No test. Forcing the refusal means putting the process at `vm.max_map_count`, which needs either a fault injection hook in the mapping path or a test that raises a resource limit and hopes nothing else in the process wanted a mapping, and neither is worth writing before #768 changes what a mapping is here. Said so on the issue rather than left implied.
#768. `remap_settled` mapped each 4 MiB page with a null address hint, so the kernel chose where each one went, and it hands addresses out in the opposite order to the file offsets the log maps in. Two pages that are neighbours in the file were therefore never neighbours in memory, the kernel could not merge their mappings, and the process ended up holding one region of address space per page. Measured on server1 with a small C program, 512 pages mapped a page at a time added 512 lines to /proc/self/maps and the same 512 into a reservation added one. Two things that costs. Every fault walks the region tree, and that fault is the one map_settled traded a pread for, so the option was paying for itself in the same place it was trying to save. And vm.max_map_count is 65530 by default, which is 256 GiB of log, past which mmap refuses for everything in the process and not only for this. So the log takes the address space once at open, PROT_NONE, anonymous and MAP_NORESERVE, which is a promise from the kernel that nothing else lands in the range rather than a request for memory. A page goes in at page % reserved_pages with MAP_FIXED and comes out as PROT_NONE again rather than through munmap, because a munmap would take the range out of the reservation and let the next unrelated mmap in the process land in the middle of it. The ring is the log's span cap plus two pages, so two live pages can never want the same address. That is an invariant living in another function, though, and the cost of it being wrong is a MAP_FIXED over a mapping somebody is reading, so map_page checks the one page that could collide before it maps anything and refuses if it is live. A refusal costs a page that stays on the heap. A refused reservation, map_settled off, and a platform with no mmap all end up in the same place: a null hint and a munmap, which is what happened before this. tests/mapping.rs is new and is deliberately not behind a cfg. A test file behind a crate level cfg is not type checked on the machine it is written on, which is how a Linux only test goes stale without anybody finding out. The region counts come from /proc/self/maps and are skipped where there is nowhere to ask; everything else, including that the data reads back through the mappings and that ten opens and closes give the regions back, runs everywhere. The region halves of those two tests have not run on Linux yet. Every host is mid rival benchmark and a compile is load I am not putting on a published number, so it is queued behind them rather than claimed.
zu2_remap_refused, so a run can say whether map_settled did what it was asked or quietly stopped. The A/B this is for is the one queued on server3, where a mapped arm that reports refusals is measuring something other than mapping. #769.
The reservation from #768 is 256 GiB at the default max_pages and shows up in VSZ, which reads as a leak to anybody who has not been told it is PROT_NONE and MAP_NORESERVE. RSS is unchanged and RSS is what every memory number is measured in. Said in both the Rust option and the C header, including that a host which refuses the reservation gets the mapping without it rather than no mapping at all.
The header held the key length and the height as a `u32` each. The height cannot exceed `MAX_HEIGHT`, which is twenty, so it needs five bits and was using thirty two. Packing the two together gets four bytes back, and because the arena rounds a node up to eight it usually gets another four with them: a height one node over a thirty three byte key was 8 + 8 + 33 = 49 rounding to 56, and is now 4 + 33 = 37 rounding to 40 with the links at 48. The header cannot just shrink where it was, because the links sat behind it and would have stopped being eight aligned. Moving them to the front would have kept the alignment and does not work: the header would then sit at `8 * height` and there is no way to read the height without already knowing it. So the links go last, after the key and after enough padding to get back to eight, and the header stays at offset zero where the key length it holds is what says where the padding ends. Weighted by the geometric height distribution the mean node over a go-ycsb key goes from 57.8 bytes to 50.6, so the plane goes from 58.7 bytes a key to about 51.5. The walk is unchanged: same dereferences, same comparisons, one fewer word in front of the key. Nothing on the walk ever asks a node its height, since a node reached at a level has more levels than that by construction. The height is in the header anyway now that the length had bits to spare, so `link` uses it for a debug assertion, which is the first time that invariant has been checked rather than argued. The test loads sixty thousand thirty three byte keys and asserts the plane fits in three of the arena's one megabyte chunks. Proved by widening `HEADER_LEN` back to eight, which is the old arithmetic exactly: it fails with four megabytes. See #772 and #767.
`memory_pages` says it is pages of anonymous memory the log may hold. Under `Durability::Async` it was not a bound on anything while the writing was going on. A page can only be evicted once its bytes are durable, the writer never waits for the device, so the flusher never caught up and the log held the load. At a bound of two pages with 8 KiB values, 80 MiB of writes peaked at 48 MiB held and 320 MiB peaked at 240: four times the load for five times the overshoot. Under `Durable` the same load held three pages, because there the commit is its own back pressure. So the append path gets the back pressure the durable path had for free. An allocation that finds the count over the bound wakes the flusher, takes whatever is already durable, and waits on the condition variable a landed device write signals. The same load now peaks at three pages. Three things keep that safe. It runs before the session publishes its write frontier, so a writer waiting for a flush is never waiting for itself. Every wait has a timeout and the whole thing gives up after ten milliseconds and lets the writer through, because holding memory over the bound for a moment is a worse database and a writer that cannot be woken is not a database at all. And the engine's own sessions are not throttled, since compaction has to copy a record to the tail before it can reclaim the page the record was in, and holding a pass because memory is tight would hold the one thing that makes memory less tight. The default is `usize::MAX`, which returns on the first branch, so a database with no bound pays a predictable test per record and nothing else. The cost where it does bite is real and it is the point: the load above went from 668 ms to 1.25 s on a laptop, which is a bound of 8 MiB forcing a device write every 8 MiB. That is what a bound means. What it costs on a host that publishes numbers has to be measured there, and #775 says so. See #775 and #767.
A mapped page goes back to PROT_NONE inside the reservation rather than through munmap, and the kernel keeps the pages that are still mapped as one region only while they are contiguous. The measurement on server1 that showed 512 pages collapsing to one region also showed that dropping every other one splits it back into 512, so the collapse is a property of the drop policy and not of the design. The policy is that the two places which drop a mapped page both take them from the bottom of the run: reclaim_to takes from..upto, which is a prefix by construction, and evict_behind walks the head boundary up. Neither was saying so anywhere, which is how a read bit or a per page policy would quietly take the region count back. See #768.
`remap_settled` maps a settled page outside the allocation lock and publishes it under the lock, so a page can be evicted in between and the mapping is given back without ever having been published. That giveback was a plain `munmap`, and `map_page` maps inside the #768 reservation whenever there is one, so it took the range out of the reservation and handed it to the process. The next unrelated `mmap` anywhere in the process is then free to land in the hole, and the next time the log maps that page it goes in with `MAP_FIXED` and takes that mapping away from whoever made it, silently. Every other site already dispatched on `within_reservation`. This one did not. Now it does, the same two ways back `release_page` has. The race is not theoretical: a run under a four page memory bound with `map_settled` on lost nine mappings in a quarter of a second, which is nine holes. `Db::remap_lost` counts them, and the new test in tests/mapping.rs uses that count to say the run reached the path before it asserts the region count did not follow it. Closes #776.
#769 left the retry untested because forcing a refusal means putting the process at `vm.max_map_count`, and it said the test belonged with #768 since that changes what a mapping is here. #768 is in, so here it is. `file::refuse_mappings` arms a count of mapping attempts to fail, which is the one behaviour in this path a kernel decides and a test otherwise cannot reach. It is not behind a cargo feature: the cost unarmed is one relaxed compare on a path that runs once per 4 MiB page on the flusher thread, and a hook that only compiles under a feature nobody runs is a hook that quietly stops compiling. The test is its own binary because the count is process wide. Map some pages, arm a wall and write 48 MiB past it so a dozen pages settle refused, then take the wall down and write 1.6 MiB, which is less than a page and so can settle nothing of its own. Every page mapped after that is one from under the wall. The sizes are what make it a test rather than a shape. With the retry put back to a high water mark it goes 2 mapped, 12 refusals, 3 mapped and fails; with the retry it goes 2 mapped, 41 refusals, 15 mapped. Closes #769.
tests/racy.rs runs four writers, an evictor and a compactor over a log that laps, and it had never run with map_settled on. That is the one shape where a page a reader is inside can be unmapped by somebody else rather than freed by an epoch that knows about the reader, and where a mapped page has to go back with munmap and a heap page with dealloc from the same call sites. It is also the code #776 was just found in. The run is now a function taking Options with two tests over it. Nothing new to assert: a mapping given back early is a SIGSEGV or a SIGBUS, not a value that comes back wrong. A million operations a thread makes 383 mappings under all three and passes. Db::remap_made counts mappings published, because mapped_pages is a gauge and cannot answer the one question the test has to ask. A run of this shape ends with the gauge at zero however much it mapped, since the last compaction takes the pages back, and a first draft of the test read that zero and concluded map_settled did nothing under a page bound. It does: 142 mappings with the bound against 150 without, at four hundred thousand operations a thread. The bound is not what drives the churn in #759 on this shape, the compaction is.
A seqlock reader takes the sequence, reads the data, and takes the sequence again, and the second read only means something if the data was read before it. An acquire load orders what comes after it, not what came before, so the plain loads of the neighbours or of the value were free to be taken after the validating load: the check reads a version from before the writer started, the data comes back from the middle of what the writer did, the versions match and the torn read is the answer. Both readers had the hole, `Neighbourhood::read` and `Record::read_value`. An acquire fence between the two is the `smp_rmb` the protocol wants. It is nothing on x86 and a `dmb ishld` on aarch64, on a path that already does two atomic loads, and the validating load drops to relaxed because the fence is what orders it now. `tests/graphracy.rs` is what caught it and is new here for its own sake: nothing was writing edges from more than one thread, which left the plane's one moving part uncovered. Four threads write edges into their own runs of node ids, so each model is exact, while a compaction runs under them, and each thread also reads another thread's hub, where sorted, no duplicates and every id inside the owner's run has to hold whatever the writer is doing. At 200000 operations a thread on an aarch64 laptop it fails 3 to 4 runs in 20 without the fence, always with one adjacent duplicate, which is the state `insert_sorted` leaves halfway through its memmove. With the fence, 30 runs in a row green. #782
`compact::keep_edge` decides an add record is live by asking whether
the adjacency has the edge. Nothing asks whether the adjacency got it
from this record or from the two hundred below it, so every one of them
is live and every pass copies every one of them forward. The live set
was a record per edge operation rather than per edge, over a graph that
was not growing at all, which is what every MERGE shaped write and
every loader run twice does.
`tests/edgedup.rs` is the measurement. One edge, then the same edge
again a hundred thousand times, compacting throughout so the span is
what a pass could not get rid of:
span after one edge 128, after a hundred thousand re-adds 4800152
48 bytes an operation that nothing would ever reclaim. With the write
side fix the same run reads 128.
The fix is that a write which changes nothing does not append. Under
the node's edge order lock an add of an edge that is there and a remove
of one that is not are both no-ops in memory, so they return without
touching the log. Under that lock and inside the epoch, so the question
and the append are one step against another writer on the same node and
reading the neighbours cannot meet a block a grow is retiring.
This is also what `tests/graphracy.rs` was hitting: four threads over
1024 nodes ended in `LogFull { span: 19, max: 16 }` with under 5 MB of
live padding, because the rest was duplicate add records. It goes back
to 16 pages here, and twelve runs of 200000 operations a thread are
green.
The pass side is still open on #784. A log written by a build without
this keeps its duplicates, and dropping them needs the pass to remember
which pairs it has already copied.
#784
Everywhere else in the file a neighbourhood has one writer and many readers, because each thread owns a run of node ids. The inward neighbourhood of a node several threads link to has none of that: four threads write it at once, through four different edge order stripes, so the only thing keeping them off each other is the neighbourhood's own lock, and nothing was testing that. Every thread links its own nodes to one sink, so the model is still exact, since no two threads ever write the same neighbour. During the run the sink has to come back sorted, without a repeat and made of ids that are nodes; at rest it has to be the union of what the four threads say they wrote, and its degree has to agree with its length. Its id sits above every run, which is what lets a thread's expected out list keep the sink on the end and stay sorted. Fifteen runs at 200000 operations a thread, green, sink degree in the thousands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #396. This is Z1 through Z7 in one commit, because the crate is new and splitting a new crate into seven PRs that each fail to compile is not a review, it is a formality.
Why a second engine
The Y program fixed the measured YCSB defects one at a time inside zu1 and it was worth doing, but it does not change the two curves. A zu1 point read is a full label scan, so it is linear in the table size across a factor of 200 in record count with no knee anywhere. A zu1 commit is a fold plus an fsync, and a 3.75 ms fsync with one writer caps a durable commit rate at 266 a second whatever the device can do. The first is fixed only by an index and the second only by a different commit protocol.
crates/zu2depends on nothing inzu-zu1and nothing inzu-zu1depends on it, so both engines build and run in the same binary and the benches put them side by side on one host.What is in it
The record plane is FASTER (SIGMOD 2018) brought up to F2 (PVLDB 18(12):4910-4923, 2025). A hash index with cacheline buckets, a 14 bit tag and the collision chain in the log rather than in overflow buckets. A hybrid log with an in-memory tail and a file-backed body. Records as seqlocks, so an update inside the mutable region takes no latch. Epoch protected reclamation. Lookup based compaction that punches holes rather than rewriting, with a Linux, a macOS and a Windows implementation.
The graph plane is on the same log and in the same commit. Dense vertex ids, a neighbourhood as one cacheline with ten inline neighbours and a sorted doubling block past that, and one version cell per vertex rather than one per neighbour, which is the thing the 2025 survey (arXiv 2502.10959) measures the keyed edge designs at 4.1 to 8.9x CSR memory for.
Two benches, both in process against rusqlite so there is no harness in the middle.
ycsbruns eight sqlite configurations so the comparison is against sqlite's best rather than sqlite's default, and measures storage as bytes the device is holding.traversedrives both engines through the same generic traversal code against aWITHOUT ROWIDedge table keyed on(src, dst).32 unit tests, 8 end to end tests, 6 compaction tests. Clippy clean on
--all-targets.Two benchmark defects fixed before any of these numbers
The update phase derived each value from its key, so it rewrote every record with the bytes it already held, and sqlite's
btreeOverwriteContentskips dirtying a page when the new payload is byte identical. The update column was measuring amemcmp. On server2, 200 single threaded updates went from 545 op/s with 10 fsyncs to 23 op/s with 810 fsyncs in DELETE mode, and from 3902 op/s with 10 fsyncs to 115 op/s with 213 fsyncs in WAL. Every value now carries a revision no earlier write of that record used.PRAGMA synchronousran beforePRAGMA busy_timeout, which crashed every multithreaded phase withdatabase is lockedonce the updates started doing real work. The timeout is now the first statement on every connection in both benches.Both benches also measure and print the filesystem's own durable write floor before any row, so a row that claims to wait for the device and beats the floor is visible as such.
Numbers
Five machines on 2026-08-19: an Apple silicon laptop, gamingpc (i9-13900K, Windows 11, LLM services stopped, 0.8 percent busy before the run), and three EPYC Ubuntu VMs. Full tables in
Spec/2064g/zu2/07-benchmarks.md, raw logs inSpec/2064g/zu2/runs/2026-08-19/.YCSB, best zu2 against best sqlite at the same durability, commit does not wait:
Traversal, zu2 over sqlite. Every cell of the full table is over 10x on every machine at every thread count, and the lowest is 19.7x. gamingpc at 32 threads: degree 41213x, 1 hop 1392x, BFS 3004x, triangles 2326x. The 90/10 traverse and insert mixed row, which is the one that carries the transactional graph claim, is 82.6x on the worst machine and 221x on gamingpc.
Storage, 243.9 MiB of keys and values measured as bytes the device is holding: zu2 compacted 1.03x raw on every machine, sqlite rowid 1.37x, sqlite
WITHOUT ROWID4.58x, zu1 1.01x.What this does not do
zu2 loses the durable write, on every Linux machine, by two to ten times on ycsb and up to thirty seven times on graph insert. Two causes and both are in 07 section 6. The log file is never preallocated, so every
fdatasynccommits an inode size change too, andexamples/appendcost.rsmeasures that at 1.9x to 4.6x on the three Linux machines. Andmake_durablewaits for global epoch quiescence before every device write, so durable throughput barely moves with thread count. That is Z15 and it is next. Nothing here should be read as a durability win.Only sqlite has been run. postgres, duckdb, mongodb, neo4j and ladybug are Z8. Compaction runs once at the end of a phase rather than continuously, so nothing here shows it keeps up with a sustained write rate. Recovery is in neither bench.