feat(engine): log key changes at commit and write pages at checkpoints

[venkat1701]

Stacked on #85.

Why

Reads were ahead of HyperGraphDB after #83 to #85. Small writes were the one place HStore still lost: in the #85 runs, a commit that changed one value wrote 14.9 KiB and took 214 µs at p50, against 10.1 KiB and 121 µs for HyperGraphDB. Every commit copied the path of each tree it touched into new pages and wrote them before returning, and the dead pages it left behind were what compaction spent its time on.

What changes

Commits log keys, checkpoints write pages.

• A commit now writes one Changes record to the log, holding the keys it changed in every tree. Nested trees, such as an edge's members inside its catalog record, are logged the same way. The roots it publishes point at the new nodes in memory.
• A tree that changed in more than 4,096 places, such as a bulk load, is written to pages at commit time and logged as a reference to them, as before.
• Checkpoints write the pages:
  • Outside the commit lock, a checkpoint writes the nodes of the current generation that no earlier checkpoint wrote, and syncs them.
  • It then moves the reference counts from the last checkpointed generation to this one, publishes the catalog, and swaps the written pages into the head. Only the swap holds the commit lock, so the catalog publish and the log sync no longer stall commits.
• Recovery replays the Changes records onto the last checkpoint.

Written nodes are stamped.

• Each node a checkpoint writes is stamped with its page and the checkpoint attempt.
• The next checkpoint reuses that page instead of writing it again.
• Ref.same treats the node as that page once the checkpoint has published, so a transaction that started before the swap sees no conflict. Two small leaves that would be stored inline compare by content.

Reference counts change only at checkpoints.

• Counts move only between two fully written generations.
• Pages a commit does write (spills, and large trees logged by reference) wait as provisional pages until a checkpoint counts them or marks them dead.
• Pages found with no count when the database opens wait for the first checkpoint the same way.

Memory is capped.

• write_buffer_mb (default 64) limits the heap held by unwritten nodes, counted from the head.
• The maintenance pass checkpoints at half of it. A commit that finds the buffer full wakes that pass and waits.

Smaller changes.

• A commit batch syncs the segment files only if one of its commits wrote pages.
• The page budget counts the nodes a transaction adds.
• Compaction can seal an active segment that is mostly dead. With far less written, a 256 MiB segment can take a long time to fill.

Storage format goes to 7.

What gets worse

• History across a restart. After a restart, time travel reaches only the last checkpoint and the commits replayed from the log; history older than that is no longer kept. A branch merge needs its fork generation, so a branch forked before the last checkpoint can't be merged after a restart. Before this change, a merge already failed once the fork generation fell out of history_limit. Keeping branch fork generations across restarts is a possible follow-up.
• Pages are released one checkpoint later. A dead page is now released only after the checkpoint that finds it dead, and after the retained history has moved past that checkpoint.
• Crash recovery takes longer. Replaying key changes costs more CPU than pointing the roots at pages that were already written. recover went from 90 ms on perf: warm reads up fairly and visit incident edges without records #85 to 171 ms, so HStore now trails HyperGraphDB there (123 ms), where it led before. docs/benchmarks.md lists it with the other losses.

Testing

• Full suite: 206 tests pass, three times in a row on the final code. I ran the suite on each of the first six commits on its own. All passed except one run of 60fc871, which had a single failure; that log was overwritten before I looked at it, so I don't know which test it was, and two reruns of that commit were clean. The recovery fix was only tested together with the commits after it, which change docs alone.
• Change log replay: StorageEngine.verifyChangeLog replays every commit still in memory from the log and compares it with the generation the commit published, entry by entry and through nested trees.
  • ChangeLogTest runs it over random engine work: set and ordered edges, relabels, deletes, bulk and aborted loads, branches and request ids.
  • ChangeLogReplayTest runs it over random database work: property writes with validity, property and JSON index backfills, documents, embeddings, deletes, loads and branches.
  • Every commit replays exactly. Breaking the applier on purpose (skipping removes, nested changes, or large trees) fails the check.
• Recovery fix: after a restart with commits to replay, history used to start one generation past the checkpoint, so the checkpointed generation itself couldn't be read. Reading the code, main has the same gap, but I haven't run a test on main.
• New DeferredWritesTest:
  • Small commits: 300 single inserts write no pages, and the next checkpoint writes fewer than 60.
  • Bulk loads: the checkpoint after a load writes almost nothing, because the load's pages are reused.
  • Crashes during a checkpoint: a crash at each of CHECKPOINT_PAGES, CHECKPOINT_CATALOG and CHECKPOINT_JOURNAL, while another thread keeps committing, loses no acknowledged commit and leaves the counts exact.
  • Full write buffer: a full buffer makes commits wait for a checkpoint.
  • Restart: after a restart, both the checkpointed and the replayed generations are readable.
• Breaking it on purpose: I disabled, one at a time, replaying Changes on recovery, provisional pages, the swap into the head, and page reuse. Each one fails tests; page reuse needed the bulk-load test added for it.
• Updated tests:
  • Tests that churned with commits alone now checkpoint each round, since commits no longer leave dead pages.
  • Tests that read record shapes or history checkpoint first.
  • The missing-pages recovery test uses bulk loads, now the only commits that write pages.
  • Compaction tests set a long maintenance interval, so the background pass doesn't compact first.
• Native image:
  • After loading examples/clinical-claims.hql, hstore check verifies the trees and the counts.
  • After a SIGKILL partway through a 20,000-insert load, reopening replays 1,090 commits and the counts match.
  • COMPACT; and check pass afterwards.

Numbers

HStore against HyperGraphDB, scale 1, async, 64 MiB caches, the median of 3 runs each. These are from a laptop under load, so only the ratios mean much; one HyperGraphDB run took 16 minutes instead of 90 s, so the machine was disturbed during it. The Azure rerun is still to do.

Workload	HStore	HyperGraphDB	Ratio
latency.commit	9,173 ops/s	4,739 ops/s	1.94×
latency.commit p50 / p99 / p99.9	92 µs / 267 µs / 408 µs	104 µs / 1.74 ms / 7.19 ms
write.update	60,357 ops/s	4,249 ops/s	14.2×
mixed.write p99	21.4 ms	77.1 ms	3.6×
ingest.nodes	31,091 ops/s	19,069 ops/s	1.63×
ingest.edges	24,721 ops/s	20,097 ops/s	1.23×
churn.delete	17,491 ops/s	2,774 ops/s	6.3×
disk after load	91.3 MiB	440.2 MiB	4.8×
disk.churned	121.2 MiB	330.9 MiB	2.7×
disk.compacted	90.8 MiB	247.1 MiB	2.7×
recover	171 ms	123 ms	0.72×

Bytes written per operation, HStore against HyperGraphDB: latency.commit 649 B against 9.6 KiB, write.update 221 B against 9.9 KiB, mixed.write 350 B against 26.4 KiB. Every read workload is still ahead of HyperGraphDB. Some read medians are lower than in the #85 runs, for example read.members 3.46M to 2.98M per second and read.incidence.parallel 27.3M to 13.3M, though one run of the latter reached 26.0M. I haven't run the two branches back to back, so I can't tell yet whether that is the machine or this change.

The same comparison against this branch's base (#85), HStore only. The #85 column is the median of its two complete benchmark runs:

	before	after
latency.commit	4,082 ops/s, 14.9 KiB per commit	9,173 ops/s, 649 B per commit
latency.commit p99	679 µs	267 µs
write.update	1.9 KiB per update	221 B per update
disk after load	145 MiB	91 MiB
disk.churned	297 MiB	121 MiB
disk.compacted	66 MiB	91 MiB
reopen	236 ms	149 ms
recover	90 ms	171 ms

disk.compacted is higher than before. My guess is that dead pages are now released one checkpoint later, after the retained history has moved past it, so some are still in the files when the measurement is taken. I haven't confirmed that yet.