Make optimistic commit lock buckets and validation policy configurable

[kriszyp]

⊙ Problem

Every writable optimistic database opens with RocksDB's default 2^20 commit lock buckets. On Linux x64 that is about 40 MiB of mutexes per database, allocated whether the database is busy or empty. Ten empty databases measured 401 MiB of anonymous memory. Processes that open many databases (Harper opens one per schema database, plus system databases) pay that per database. Neither the bucket count nor RocksDB's other validation policy, kValidateSerial, which needs no buckets at all, could be configured.

Closes #894.

❓ Your call: as a rebase-maintenance pass, this task's job was to get the already-agreed design onto current main and resolve what independent review rounds surfaced — not to revisit the requirement itself. The requirement (configurable bucket count + validation policy, private pools) was decided in the linked issue conversation and is unchanged by this rebase; nothing in-flight supersedes it (see the #902 note under Related PRs below — it has since merged directly into this branch, additive to this PR rather than a replacement for it).

💡 Solution

Two process-wide settings, read when an optimistic database is opened. Both follow the same rules: they apply to optimistic databases opened after the call, databases already open keep what they opened with, nothing freezes, and pessimistic, read-only and secondary opens ignore them.

• RocksDatabase.config({ occLockBuckets }): the size of each database's private bucket pool under parallel validation. The default is now 65,536 (about 2.5 MiB per database on Linux x64) instead of RocksDB's 2^20 (about 40 MiB). That is a behavior change: memory per optimistic database drops by about 37 MiB.
  • The default can be small because async commits to one database already run one at a time on that database's commit lane, so the lane never waits on itself. Only commitSync() callers on the same database, or concurrent libuv commits under ROCKSDB_JS_COMMIT_THREAD=0, can wait on its buckets.
  • 65,536 recovers the measured 64-key concurrent-commitSync() case (sweep below). Workloads with 1,000-key commitSync() mixes still lose up to about a third of their throughput and can raise the count to about 2^20.
  • The legacy ROCKSDB_JS_COMMIT_THREAD=0 mode has no lane, so its async commits contend too. With 4 workers, 65,536 buckets gave 11,474 commits/s against 12,762 at 2^20 for 64-key transactions (about −10%), and 350 against 505 for 1,000-key transactions (about −31%). Three runs each, with non-overlapping ranges. The README says so.
  • The range is 16..16,777,216.
• RocksDatabase.config({ occValidation: 'parallel' | 'serial' }): maps to OptimisticTransactionDBOptions::validate_policy. The default stays 'parallel'. Under 'serial', RocksDB runs the conflict check inside the write group and allocates no bucket pool: OptimisticTransactionDBImpl's constructor builds one only for kValidateParallel (utilities/transactions/optimistic_transaction_db_impl.h). Measured at 0.2 MiB for three databases configured with 2^20 buckets.
  • When to use 'serial': nearly all writes to a database are async transaction commits through the commit lane, with few concurrent commitSync() calls or non-transactional writes on that database. It also suits large transactions and processes that open many databases.
  • Benefits: about 8–12% more throughput, with lower p95 latency and CPU per commit, at 1,000 or more keys per transaction. No bucket memory. Because the check and the write are one step, a column-family drop cannot land between them (defense in depth beyond Defer physical column-family drops behind admitted commits #850's claim protocol), and a non-transactional write that lands after validation can no longer be silently overwritten.
  • Cost: RocksDB never batches a serial-validated commit with another writer, so concurrent synchronous writers on the same database lose substantial throughput: 30–55% fewer commits per second than parallel validation with 4,096 buckets.

Harper will set occValidation: 'serial' (and its own bucket count) in a companion Harper PR. rocksdb-js keeps a general-purpose default.

⚖️ Alternatives

• One process-wide shared pool (shared_lock_buckets, this PR's first revision). This is removed. Buckets are held through DB::Write(), so commits in unrelated databases contend on hash collisions, and a database in a write stall blocks every database whose keys collide with its pending commit. Measured with four databases and one worker each: −40% commits/s at 1,000 keys per transaction and −58% at 4,096 keys, against private pools.
• A 4,096-bucket default (this PR's second revision). It saves more memory (about 160 KiB per database), but concurrent 64-key commitSync() mixes lost about 40% of their throughput.
• Keeping RocksDB's 2^20 default. It changes nothing for existing users but keeps about 40 MiB per database, which the commit lane makes unnecessary.
• A per-open option. Not added. A path's first opener would decide for every later handle (the descriptor is process-global), and nothing needs per-database sizing yet. It can be added later without breaking anything.

🔧 Changes

• src/binding/database/db_descriptor.cpp: pass OptimisticTransactionDBOptions{ occ_lock_buckets, validate_policy } to OptimisticTransactionDB::Open. validate_policy is now assigned explicitly in both branches (serial or parallel) rather than relying on RocksDB's header default for the parallel case, so a future vendored default change can't silently flip behavior.
• src/binding/database/db_settings.{h,cpp}: atomic settings for the bucket count (default 65,536) and the validation policy (default parallel). config() validates both before applying either: occLockBuckets must be an integer in 16..16,777,216, and occValidation must be the string 'parallel' or 'serial' (TypeError for a non-string, RangeError for any other string). The shared pool, its mutex and the freeze are removed. The two stores are now applied last, after every other field's validation, so a later field's thrown error (negative blockCacheSize/writeBufferManagerSize, invalid lifecycleWaitSeconds, a second verificationTableEntries change) leaves the occ settings untouched.

  ❓ Your call: blockCacheSize and verificationTableEntries already silently ignore a wrong-type value instead of throwing (confirmed empirically: config({ blockCacheSize: 'garbage' }) neither throws nor applies anything) — this is pre-existing Config() behavior, shared by both fields, and not something this diff touches. A second round of rebase maintenance found a sharper variant reachable through this PR's own new fields: a throwing getter on verificationTableEntries (e.g. config({ occLockBuckets: 1, get verificationTableEntries() { throw ... } })) left a pending JS exception the function never checked, so config() threw to the caller but still applied occLockBuckets/occValidation/commitThreads — contradicting this very function's own "stored last" comment. Fixed by checking napi_is_exception_pending() once, immediately before the three stores, so it holds regardless of which field is last. The broader non-throwing wrong-type silent-ignore above is unchanged and still out of this PR's scope.

• src/load-binding.ts, README.md: document both settings, their memory cost, and when to use each. The README also states a window that main already has under parallel validation: a non-transactional putSync() takes no bucket, so one that lands between a commit's conflict check and its write is overwritten without ERR_BUSY.
• AGENTS.md invariant 24: under serial validation, the write-group check also refuses a dropped family. It is defense in depth only, because the claim protocol still applies under both policies and is the only protection for pessimistic commits.
• src/binding/database/commit_worker.h: correct the comment that said optimistic validation does not block. Parallel validation takes bucket mutexes, sorted acquisition keeps that deadlock-free, and with private pools only commitSync() callers contend with the lane.
• src/binding/database/DESIGN.md, DESIGN.md: record both policies, why the default pool is small, and why the pool must stay private.
• benchmark/occ-commit-contention.mts: a standalone benchmark (same convention as benchmark/mutex-contention.mts). It covers one database with many workers, four databases, async plus commitSync(), real conflicts, and the memory of ten empty databases. --lib compares builds, and --occ-lock-buckets and --occ-validation set the two configs.

✅ Verification

• test/occ-lock-buckets.test.ts (child-process fixture test/fixtures/fork-occ-lock-buckets.mts):
  • validation of both settings;
  • 1,024-key commits with 16 buckets, sync and async;
  • the same key in different databases and column families does not conflict;
  • true conflicts are still rejected for both commit forms, under both policies;
  • a lane-versus-commitSync() counter loses no updates, under both policies;
  • Linux memory bounds (skipped, rather than failing, when /proc/self/smaps_rollup is unreadable or lacks an Anonymous: line — reviewer request):
    • ten empty databases stay under 60 MiB at the default (26 MiB measured);
    • a 2^20 configuration grows memory, which proves the count reaches native code;
    • serial validation with 2^20 configured stays under 30 MiB (0.2 MiB measured).
• The worker-backed fixture modes load their worker by a cwd-relative path, because import() of an absolute Windows path failed on every Windows leg.
• test/occ-serial-drop.test.ts reruns Defer physical column-family drops behind admitted commits #850's suites (drop-deferred-reclamation, drop, txn-close-commit-uaf; 60 tests) under occValidation: 'serial'. A NODE_OPTIONS=--import preload reaches the suites' spawned fixtures. They pass on macOS and Linux with unchanged expectations and error messages.
• The cross-model pre-push review (Codex, Gemini, Cursor Composer, plus Harper-domain adjudication) ran on the private-pool and settings rounds, and its findings are addressed. One remains as a nit: the memory assertions, which are the only proof that the settings reach native code, run on Linux only.
• Fails on base: with the old Open overload restored, the default, configured and serial memory tests fail (401, 120 and 120 MiB against bounds of 60, 30 and 30).
• An lldb breakpoint confirms that occValidation: 'serial' reaches RocksDB's serial path (DBImpl::WriteWithCallback), and 'parallel' reaches CommitWithParallelValidate.
• macOS: pnpm check, pnpm test (76 files, 1,108 passed, 10 skipped) and pnpm test:native (299 passed). Linux: the two OCC test files. The full Linux, Windows, Bun and Deno runs are left to CI.
• Rebase onto main (2026-10-06): resolved a DESIGN.md add/add conflict across the 13 replayed commits (merged both PRs' index entries; kept this branch's own later wording revisions for the src/binding/database/DESIGN.md line). Post-rebase: full production build, pnpm check, the full JS suite (1,124 passed, 10 skipped), native GoogleTest (309 passed) and stress tests (9 passed) all green on Linux. A second, delta pre-push review round on the rebased head retired the config-atomicity finding (now fixed, see 🔧 Changes) and found no new actionable issues; the remaining open findings are the pre-existing ones already covered above and in ⚖️ Alternatives (default-bucket-count tradeoff, Linux-only memory assertions, Bun/Deno serial-drop skip, standalone benchmark not wired into pnpm bench).
• Second rebase onto main (2026-10-06, same day): main had advanced 13 more commits; this branch's own tip had also grown by one — #902, a PR based on this branch rather than main, had merged into it directly. Same DESIGN.md add/add conflict pattern, now hit 4 times across the 16 replayed commits; resolved the same way each time (merge both sides' index lines). git range-diff against the old head confirmed every commit replayed cleanly with no silent content changes outside those hunks. Full build, pnpm check, the full JS suite (1,151 passed, 10 skipped), native GoogleTest (315 passed) and stress tests (9 passed) all green on Linux. Two further delta pre-push review rounds found one genuine, PR-scoped bug the two prior rounds missed: a throwing getter on verificationTableEntries (not just a wrong type) left a pending exception the atomicity fix above didn't check, so config() threw to the caller but still silently applied occLockBuckets/occValidation/commitThreads (see the 🔧 Changes Your call note). Fixed, and empirically verified: a database opened right after the throwing call and burst with 50 commits now reaches the real default thread ceiling rather than the leaked value. Verdict moved from CHANGES to COMMENTS; converged on the same pre-existing/previously-adjudicated findings as the first rebase round.
• Benchmarks on Linux, i7-12700H (20 threads), pinned to the P-cores (0–11), Node 24.21, RocksDB 11.8.1.
  • Method: medians of 3 runs with the variant order rotated each run, a 4 s window, 8 in-flight async transactions per worker, disableWAL: true unless noted.
  • The box was shared with other work (1-minute load median 5.8, range 2.3–14.1), so only differences with non-overlapping ranges are claimed.
  • Variants: A = main (private 2^20, parallel); B = private 4,096, parallel; B64k = private 65,536, parallel (the new default, from the sweep run); C = serial; D = shared 2^20.

scenario	A	B	B64k	C	D
10 empty DBs, anonymous MiB	401	2.5	26 (test)	1.2	41
1 DB, 4 async workers, 64 keys, commits/s	5,460	5,098	–	5,882	4,697
1 DB, 4 async workers, 1,000 keys	343	353	–	383	343
1 DB, 8 async workers, 1,000 keys	319	322	–	359	317
1 DB, 4 async workers, 1,000 keys, WAL on	296	298	–	324	292
4 DBs, 1 worker each, 1,000 keys	827	758	–	846	496
4 DBs, 1 worker each, 4,096 keys	185	185	–	200	77
1 DB, 2 async + 2 commitSync, 64 keys (async / sync)	4,721 / 4,419	3,014 / 3,148	4,733 / 4,462	1,680 / 2,190	4,642 / 4,314
1 DB, 4 async + 4 commitSync, 1,000 keys (async / sync)	105 / 366	66 / 235	85 / 212	65 / 254	99 / 331

• Async commits to one database through the lane alone: no measurable difference between A and B at any size, so the bucket count does not matter there. C is 8–12% faster at 1,000+ keys with non-overlapping ranges, and uses 4–7% less CPU per commit.
• Real conflicts (4 workers, overlapping key sets): conflicts per commit and throughput match across A, B, C and D, and the counter checksum found no lost updates in any run.
• Bucket sweep with one build, 2 async + 2 commitSync() workers, 64 keys, async commits/s: 4,096 → 2,913, 16,384 → 3,695, 65,536 → 4,733, 2^20 → 4,999 (main: 4,948). At 1,000 keys only 2^20 recovers. The 2^20 run of this build matches main, so the code itself adds no overhead.

Stop verification-table slot collisions from answering FRESH for another key (#901) fixed the pre-existing VT collision bug behind this PR's intermittent verification-table.test.ts failure, and has merged. Run a database's async commits on up to four concurrent commit threads (#902) was based on this PR's branch (not main) and has since merged into it — its commit is now this branch's own tip, so the single-lane contention numbers above are superseded by that PR's own measurements wherever the two disagree. A Harper companion PR (to come) sets occValidation: 'serial'.

Related PRs: #901 independent, #902 overlaps (merged into this branch), #890 independent, #778 independent, #742 independent, #767 independent, #842 independent, #900 independent, #117 independent, #88 independent
Complexity: low

🤖 Generated with Claude Code

_{Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=12; full=5 @ 9ed9cad}

_{Review-Attention: study ~20m (critical: transaction.cpp, transaction_log_store.cpp +2; decisions: default-commit-threads, occ-bucket-default, settings-read-at-open, expose-serial-policy, do-less-alternative) @ 9ed9cad}

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	22.20K ops/sec	45.05	43.35	651.323	0.115	110,979
🥈 rocksdb	2	10.72K ops/sec	93.32	89.61	24,327.978	0.987	53,580

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.39K ops/sec	35.23	34.12	518.812	0.102	141,940
🥈 rocksdb	2	10.97K ops/sec	91.12	87.51	3,076.225	0.131	54,870

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.60K ops/sec	40.65	36.38	1,960.579	0.290	123,010
🥈 rocksdb	2	15.34K ops/sec	65.19	56.71	1,053.123	0.118	76,704

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	432.22 ops/sec	2,313.624	45.40	64,471.105	14.12	870
🥈 lmdb	2	26.65 ops/sec	37,520.402	436.327	1,200,701.783	136.169	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	35.02K ops/sec	28.56	13.24	21,055.337	0.857	175,084
🥈 lmdb	2	441.27 ops/sec	2,266.208	85.90	33,936.508	1.71	2,207

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	621.78K ops/sec	1.61	1.38	4,724.891	0.214	3,108,916
🥈 lmdb	2	477.13K ops/sec	2.10	1.14	9,890.641	0.572	2,385,669

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	824.66 ops/sec	1,212.62	1,056.709	2,936.539	0.333	1,650
🥈 lmdb	2	1.14 ops/sec	877,018.444	799,198.729	1,068,807.324	6.02	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	20.86K ops/sec	47.93	30.36	20,501.999	2.07	41,729
🥈 lmdb	2	804.04 ops/sec	1,243.725	80.51	11,579.814	5.42	1,609

---

Results from commit 9b6d038

[gemini-code-assist]

Code Review

This pull request introduces process-wide optimistic commit lock buckets (occLockBuckets) shared across all writable optimistic databases and worker threads, optimizing memory usage. Feedback on the changes suggests explicitly casting the double configuration value to uint32_t in db_settings.cpp to prevent compiler warnings, and wrapping the /proc/self/smaps_rollup read in the test fixture in a try/catch block to handle environments where the file is inaccessible.