[venkat1701]

Refs #46

What a commit waited on

The last comment on #46 says a checkpoint publishes the catalog and syncs the WAL while holding the commit lock. On main that is only partly true. catalog.publish, the reference journal, the feed sync and applyJournal don't take any lock a commit needs. Two things do:

1. WriteAheadLog.sync() held the WAL lock for the whole fsync. The checkpoint calls it after appending its Checkpoint record. A writer appends to the WAL while it holds the commit lock, so the first writer to arrive waited for the fsync with the commit lock in hand, and every other writer waited behind it on the commit lock. Group commit under durability = sync had the same problem: appends waited for the leader's fsync.
2. TransactionManager.checkpointed held the commit lock while it walked the head generation to swap the checkpoint's pages in (rebind). The walk visits every pending node the checkpoint did not write.

truncateBefore also deleted old WAL segment files under the WAL lock.

Changes

• WriteAheadLog.sync() takes a separate sync lock, reads and clears dirty and picks the active segment under the WAL lock, then forces after releasing it. Appends carry on during the force. Syncs queue on the sync lock, so a sync() that finds nothing dirty only returns after the force that covered its records has finished. If the log rolls during a force, the roll has already forced and closed that segment, so the failed force on the closed file is not an error. Any other failure sets dirty again.
• truncateBefore removes the segments from the list under the lock and deletes the files after.
• Materializer.Rebinding replaces Materializer.rebound. It remembers its result for every pending node it visits. checkpointed runs one pass over the head without the lock, then takes the commit lock, drains the group committer as before, and runs a second pass over the head as it is now. The second pass looks up what the first already did and only walks nodes that commits added in between.

The checkpoint steps run in the same order as before, and nothing written to disk changed.

Commits landing between the steps

Recovery reads the WAL from the catalog's checkpointLsn, which is the end of the target generation's Commit record. The journal is tagged with the same LSN.

• A commit appended while the checkpoint forces the WAL lands after the Checkpoint record, so after the LSN. Recovery replays it if it reached the disk. This sync never promised it would: under async nothing does, and under sync the commit's own group commit forces again, after waiting on the sync lock.
• truncateBefore(lsn) runs after the catalog with that LSN is durable, same as before. Only the file deletion moved.
• The rebind is in memory only. A commit that lands between the two passes builds on the head that is not yet rebound. The second pass runs under the commit lock over the head that includes that commit, so the commit is kept. A node that a commit stamps with the new epoch after the first pass saw it stays pending, and the next checkpoint reuses its page instead of writing it again.

New crash points: WAL_FORCE (lock released, before the force) and CHECKPOINT_REBIND (between the two passes).

Tests

CheckpointConcurrencyTest:

• commits from another thread while a checkpoint is held at WAL_FORCE or CHECKPOINT_REBIND. The commit must finish within 5 s and still be there after the rebind and after a reopen. Both cases fail on main's locking (moving the force back inside the WAL lock, or taking the commit lock before the first pass, makes the commit wait for the checkpoint).
• crashes a checkpoint at CHECKPOINT_CATALOG, CHECKPOINT_REBIND, WAL_FORCE and CHECKPOINT_JOURNAL while three writers commit under sync durability, reopens twice, and checks that every acknowledged commit is back, that nothing else came back except commits that were in flight, and the reference counts. A crash at WAL_FORCE can also land in a commit batch the checkpoint drains, so there the writers may fail too.
• runs four writers through back-to-back checkpoints, each checking that its commit is visible right after it returns, then compares every edge with what was acknowledged, verifies the counts and the change log, and reopens. Installing the first pass's result instead of running the second pass fails this test and the crash tests.

WriteAheadLogTest.aSyncWhoseSegmentRollsAwayDuringTheForceSucceeds rolls the log from inside the WAL_FORCE hook. Without the check for a rolled segment the sync throws.

Measurements

Probe: a 20,000-edge store (4 members each), durability = async, background checkpoints off. One or four writer threads commit single-member inserts into random edges in a loop while the main thread runs 30 checkpoints back to back. JFR records every jdk.ThreadPark on the writer threads with stack traces. For each checkpoint the probe takes the longest park that overlaps it, split into parks inside TransactionManager.commit (the commit lock) and inside WriteAheadLog (the WAL lock). Ten rounds per variant, main and branch interleaved, so 300 checkpoints each. Same probe class against the two engine jars.

The laptop was running other builds the whole time (load average between 30 and 80), so treat these as a rough picture, not as latency numbers.

Longest wait per checkpoint, one writer, in ms:

	p10	p50	p90	p99	max	checkpoints with no wait
WAL lock, main	0	11.11	25.01	67.71	158.89	38 of 300
WAL lock, branch	0	0	5.79	16.10	126.34	168 of 300
commit lock, main	0	1.69	13.93	84.55	256.13	41 of 300
commit lock, branch	0	0.67	10.36	73.07	282.63	98 of 300
either lock, main	4.17	12.34	31.69	96.27	256.13	14 of 300
either lock, branch	0	2.67	12.23	99.23	282.63	59 of 300

With four writers the WAL lock wait drops the same way (p50 10.73 to 0.07 ms, p90 29.00 to 6.21 ms). The commit lock wait does not move (p50 124 ms on both). With four writers on a machine this busy, a writer mostly waits for the other writers, and the probe can't tell those waits apart from waits on the checkpoint. The worst commit latency per checkpoint did not change in a way I can separate from the noise on either writer count; at the tail it is CPU scheduling, not locks.

Nothing on disk changed, so bytes written per commit are the same as main.

Left open

• This does not close Small commits are slow and occasionally stall for seconds #46. The per-commit write and allocation cost from the last comment is untouched, and the mixed writer p99 needs its own profile. A writer can also still wait for a checkpoint through the write buffer (awaitRoom), which this does not change.
• Step 9 still drains the group committer under the commit lock. Under durability = sync that drain can wait for a commit batch's fsync. Dropping the drain needs a different way to keep current from running ahead of a commit that is not yet durable, so I left it.
• SegmentFile.force clears its dirty flag before forcing, so two concurrent pages.sync() calls can let the second return before the first force finishes. That is on main too and is not touched here.
• Merging after fix(engine): keep at most a fixed number of segment files open #94 or feat(engine): start segments at 8 MiB and double them as the store grows #95 gives a one-line conflict in the StorageEngine constructor: they change the PageStore.open call on the line above the WriteAheadLog.open call this branch changes. Keep both edits.