From 24446c4ffc9d615b4cc326100521b5861c817148 Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Tue, 4 Aug 2026 08:04:27 -0700 Subject: [PATCH 1/4] docs: record the 2026-07-29 clean-slate review Thirteen reviewers over disjoint slices of the workspace, forbidden from reading ISSUES.md and docs/reviews/ so the pass would be genuinely clean-slate, then an adversarial verification pass instructed to refute. 140 findings: 5 BUG, 64 DESIGN, 57 SMELL, 14 NIT. The document records its own reliability caveat, which is the reason to keep it: a 1-in-137 refutation rate is the rubber-stamping failure mode, so the five BUG-severity findings were re-verified by hand from the source rather than taken on a verifier's word. Every other review in docs/reviews/ is tracked; this one was left untracked when its findings were filed as issues #102-#126. --- docs/reviews/review-20260729-183138.md | 1415 ++++++++++++++++++++++++ 1 file changed, 1415 insertions(+) create mode 100644 docs/reviews/review-20260729-183138.md diff --git a/docs/reviews/review-20260729-183138.md b/docs/reviews/review-20260729-183138.md new file mode 100644 index 0000000..da80eb9 --- /dev/null +++ b/docs/reviews/review-20260729-183138.md @@ -0,0 +1,1415 @@ +# Code Review — Chisel +Date: 2026-07-29T18:31:38Z +Reviewer: Claude Code (clean-slate adversarial pass, 13 independent reviewers + adversarial verification) +Commit: d87e67033cb6ab407185b09a6b2e2ea44fe12cea +Prior review: docs/reviews/review-20260702-001902.md (deliberately not read by the reviewers; used only for the delta section) + +Baseline at review time: `cargo fmt --check` clean, `cargo clippy --workspace -- -D warnings` clean, `cargo test` **681 passed / 0 failed**. Every finding below is something the toolchain cannot see. + +**Tally:** 5 BUG · 64 DESIGN · 57 SMELL · 14 NIT — 140 findings (137 from the subsystem reviewers plus 4 from the completeness critic, minus 1 refuted during verification) + +--- + +## Method, and how much to trust this + +Thirteen reviewers each took a disjoint slice of the workspace with instructions to assume nothing works and that every comment is wrong until checked. They were **forbidden from reading `ISSUES.md` and `docs/reviews/`** so the pass would be genuinely clean-slate. Each dimension's findings then went to a separate agent instructed to *refute* them, with "when uncertain, prefer REFUTED". + +That produced 137 raw findings, 1 outright refutation, and 27 severity adjustments (mostly downgrades). **A 1-in-137 refutation rate is itself a warning sign** — it is the rubber-stamping failure mode this codebase's previous review also hit. So the five BUG-severity findings were re-verified by me personally, from the source, not taken on an agent's word: + +| BUG | How I verified it independently | +|---|---| +| `Chisel::open` destroys sub-page files | **Reproduced** in a probe crate outside the repo: 71-byte text file → `Ok(Chisel)` returned despite `create_if_missing:false`, file became 16384 bytes, `"IMPORTAN"` → `"LSHC\x01\x00\x01\x00"` | +| Unbounded Argon2 `m_cost` from disk | Read argon2 0.5.3 source: `params.rs:49` `MAX_M_COST: u32 = u32::MAX` with comment *"we don't need to check `MAX_M_COST`"*; `lib.rs:230` is an infallible `vec![Block::default(); block_count()]`. Grepped chisel: no bound anywhere | +| `file_size_bytes` ignores stride | Read `page_io.rs:216-227` (`set_stride` re-seeds `cached_page_count = len/stride`) against `lib.rs:895` (`* PAGE_SIZE`); the correct sibling at `recovery.rs:433` uses `.stride()` | +| Bench cell discovery walks wrong depth | Enumerated both trees: real Criterion output is depth **5** (`row/mode/size/new/sample.json`), fixtures are depth **4**, code walks exactly 4 | +| SQLite cold-read times a journal conversion | Read `flush_for_snapshot` (sets `journal_mode=DELETE`) against the timed `mode.open()` → `open_file` (sets `WAL`) | + +I also personally confirmed GAP-1 (below) and the README/`Tag` mismatch. Findings *not* marked as personally verified rest on one reviewer plus one adversarial verifier — treat them as strong leads, not established fact. + +One process note: a verifier agent violated the read-only instruction and wrote a probe test into `src/transaction/tests.rs`. I ran it (it showed savepoint cursor *restoration working*, not a bug — so I did not file it), then reverted. The working tree is as you left it. + +--- + +## Executive summary + +1. **`Chisel::open` silently destroys any existing file smaller than one page, and ignores `create_if_missing: false` while doing it** (`src/lib.rs:366` vs `:384`). Two different "does this file exist" predicates disagree for lengths 1..8191. **This is a regression introduced five commits ago by the I143 fix itself** (`04534c0`): that commit moved the create-vs-open decision to `io.page_count()? > 0` to close a lock race, but left the `create_if_missing` gate on `metadata.len() > 0`. Reproduced end-to-end. Fix by deriving both decisions from one post-lock predicate. +2. **A hostile or bit-rotted database file can hang or abort the process before any version check** (`src/superblock/crypto_header.rs:110`). Argon2 `m_cost`/`t_cost` are parsed straight out of the superblock and handed to the KDF with no bound, and argon2 0.5.3 enforces no ceiling — up to 4 TiB of infallible allocation, retried for up to 8 key slots. This is the same trust boundary the project already hardened twice (I144 stride, ct_len); the key slots were missed. +3. **`swift/` is 974 MB of untracked *and* unignored build output on `main`** — `.o`, `.d`, `.swiftdeps`, `.build/`, `.xcbuild/`, and a three-architecture `Chisel.xcframework`, with **zero `.swift` sources** in it. The actual binding sources live on the `design/swift-binding` branch. So the risk is not data loss, it is that `git add swift/` (the obvious response to `?? swift/`) commits nearly a gigabyte of artifacts. It needs a `.gitignore` rule, not a commit. +4. **The benchmark harness reports nothing and compares unfairly, and its tests cannot catch either.** `discover_cells` walks depth 4 while Criterion writes at depth 5, so every micro-grid timing cell renders as an em-dash — and the test fixtures were built at depth 4, matching the bug, so the suite stays green. Separately, SQLite's cold-read cells time a DELETE→WAL journal conversion the harness itself created, inside the timed region. +5. **Documentation has drifted far enough to be actively dangerous, not merely stale.** `ARCHITECTURE.md:347` documents a `compact()` that does not exist and whose contract (remapping slot indices) would break the handle table's load-bearing slot-stability invariant; `README.md:325` puts `Poisoned` in the fatal tier while `is_fatal()` deliberately excludes it, so the documented recovery loop never reopens; the README's tag examples do not compile (`Tag` is a `NonZeroU32` newtype, examples pass bare `42`) and no doctest covers README. + +--- + +## Delta from prior work + +The reviewers worked blind to `ISSUES.md`; a separate pass then annotated their output against it. Of 137 findings: **58 genuinely new, 33 already recorded, 8 regressions, 3 previously declined/deferred** (left in, marked, not re-litigated). + +Regressions — recorded as fixed, still present: + +| Finding | Regression of | Note | +|---|---|---| +| PUBLIC-API-1 / SUPERBLOCK-RECOVERY-4 | I143 | The I143 fix introduced it. Verified against `git show 04534c0` | +| TXN-COMMIT-4 | I130 | `ARCHITECTURE.md` still documents `DataPage::compact()` | +| FREEMAP-10 | I122, I130 | `ARCHITECTURE.md` still names `DefragOptions::max_pages` and `compact()` | +| TXN-COMMIT-2 / FREEMAP-2 / HANDLES-INDEX-5 | I118 (doc side) | `ARCHITECTURE.md:597` still says handle-table COW bypasses the freemap — the exact page leak I118 fixed | +| BENCH-8 | I95 | Regression diff still divides by a zero baseline | +| PRIOR-1 | I146 | Closed on the premise README was the only site; `src/page.rs:107` still says the write-gate is deferred | + +Previously declined or deferred — listed for completeness only: **CRYPTO-1** (revoked credential still decrypts via the sibling superblock slot — declined per design spec §9), **FREEMAP-3**, **PRIOR-1**. + +--- + +## BUG-severity findings + +### SUPERBLOCK-RECOVERY-1 — Argon2 cost parameters are read verbatim out of the untrusted superblock and fed to the KDF with no bound, before any version gate +**Location:** `src/superblock/crypto_header.rs:110, src/transaction/recovery.rs:649, src/crypto/mod.rs:185` · **Category:** security · **Status:** KNOWN as I150 (duplicate of SECURITY-SWEEP-1) + +**What the code does.** `KeySlot::read_from` parses the cost parameters straight out of the page with zero validation: `m_cost: u32::from_le_bytes(slot[2..6].try_into().unwrap())` (crypto_header.rs:110). `open_existing` then hands them to the KDF for every active slot: `let kek = match derive_kek(key, kdf, &slot.salt, &slot.argon2)` (recovery.rs:649), which does `Params::new(params.m_cost, params.t_cost, params.p_cost, Some(32))` and runs `a2.hash_password_into(...)` (crypto/mod.rs:185-194). `m_cost` is in KiB and the argon2 crate's only ceiling is MAX_M_COST (0x0FFFFFFF ≈ 256 GiB). Nothing between the disk bytes and the allocation checks the value. This runs at recovery.rs:342, BEFORE the format-version gate at recovery.rs:384 and before the page-size gate at recovery.rs:406. + +**Why it is a problem.** Open a crafted or bit-rotted `.chsl` with a passphrase: set the superblock's crypto header algorithm=1, slot 0 state=1, kdf_id=2, m_cost=0x0FFFFFFF, and re-stamp the XXH3 page checksum (the code elsewhere explicitly assumes an attacker can do exactly this — see the ct_len bound at src/superblock/mod.rs:446-453 and the I144 stride bound at src/transaction/recovery.rs:258-269). `Chisel::open` then tries to allocate ~256 GiB inside Argon2 and aborts on allocation failure (or hangs for minutes), instead of returning a typed error. All 8 slots are tried in turn, so the cost multiplies by 8. This is the same trust boundary the project already hardened twice; the key-slot parameters were missed. + +**Direction of a fix.** Bound `m_cost`/`t_cost`/`p_cost` at parse time in `KeySlot::read_from` (or reject in `unwrap_first_matching_slot` before `derive_kek`) against a sane ceiling — e.g. m_cost <= 1 GiB, t_cost <= 16, p_cost <= 8 — and treat an out-of-range slot the same way an unknown `kdf_id` is treated today (`continue`, i.e. non-matching). + +
Adversarial verification + +Survives attack, and the finding is if anything understated. crypto_header.rs:110-112 does read m_cost/t_cost/p_cost as bare u32::from_le_bytes with no range check; KeySlot::read_from is the only parse site and CryptoHeader::deserialize (145-165) adds nothing. recovery.rs:342 `let dek = unwrap_first_matching_slot(header, k)?` runs inside the `(Some(header), Some(k))` arm at line 339 — i.e. after select() (which only checks XXH3 + MAGIC + superblock_count range, see superblock/mod.rs:477 validate) and BEFORE the major gate at 384 and the page-size gate at 406, exactly as claimed. unwrap_first_matching_slot (637-665) validates kdf_id (`_ => continue`) but passes `&slot.argon2` through verbatim to derive_kek, which does Params::new(m_cost, t_cost, p_cost, Some(32)) at crypto/mod.rs:185 and a2.hash_password_into at 193. I checked the pinned dependency: argon2 0.5.3 (Cargo.lock), and params.rs:49 has `MAX_M_COST: u32 = u32::MAX` with an explicit comment that no upper check is needed — so the ceiling is 4 TiB, not the 256 GiB the finding assumed. lib.rs:230 is `let mut blocks = vec![Block::default(); self.params.block_count()]` (Block = 1 KiB), an infallible alloc that aborts the process on failure; MAX_T_COST is also u32::MAX (params.rs:58), so a forged t_cost is an unbounded hang. Reachable through the public Chisel::open on a file whose XXH3 the attacker re-stamps — the identical trust boundary the repo already hardened twice (superblock/mod.rs:446-453 ct_len, recovery.rs:265-269 I144 stride), and I144 was shipped as BUG severity in #89, so BUG is the project's own taxonomy for this class. Multiplied by up to 8 active slots since a failed derive just `continue`s. + +
+ +### CRYPTO-5 — `file_size_bytes` and `Stats.file_size_bytes` multiply stride-unit page counts by PAGE_SIZE, under-reporting every encrypted database's file size +**Location:** `src/lib.rs:895, src/lib.rs:859, src/stats.rs:23` · **Category:** correctness + +**What the code does.** `Chisel::file_size_bytes` does `let page_count = self.txm.file_page_count()?; Ok(page_count.saturating_mul(PAGE_SIZE as u64))` (lib.rs:894-895), and `Stats` does the same: `file_size_bytes: page_count.saturating_mul(PAGE_SIZE as u64)` (lib.rs:859). But `PageIo::page_count` is documented as "the number of whole stride-units (pages) in the file ... re-seeded by `set_stride()` when the stride changes" (page_io.rs:510-517), and an encrypted DB runs at `ENC_PAGE_SIZE` = 8232 from birth (`cache.io_mut().set_stride(crate::crypto::ENC_PAGE_SIZE)`, recovery.rs:67 and 296). The doc on the field promises the true size: "Raw size of the database file on disk" (stats.rs:23); README.md:282 says "Physical size of the database file in bytes". `open_existing` gets this right in the sibling code path and even names the hazard: "multiply by the CURRENT stride, not the hardcoded PAGE_SIZE ... reporting them as PAGE_SIZE bytes would understate an encrypted file's true size" (recovery.rs:427-433, using `cache.io_mut().stride()`). + +**Why it is a problem.** For an encrypted database of P pages the file is P*8232 bytes but both public accessors report P*8192 — a 0.49% understatement that grows without bound in absolute terms (at 1 M pages: 8.19 GB reported vs 8.23 GB actual, a 40 MB error). Any caller using `file_size_bytes()` for disk-quota accounting, backup sizing, or a free-space precondition gets a value that is silently too small on exactly the databases where the extra bytes exist. + +**Direction of a fix.** Multiply by the live stride instead of `PAGE_SIZE` in both sites — expose the stride through `TransactionManager` the same way `open_existing` already reaches it (`cache.io_mut().stride()`), and add a test asserting `file_size_bytes()` equals `std::fs::metadata(path).len()` for an encrypted DB. + +
Adversarial verification + +Every cited line checks out. lib.rs:893-896 `pub fn file_size_bytes(&self) -> Result { let page_count = self.txm.file_page_count()?; Ok(page_count.saturating_mul(PAGE_SIZE as u64)) }` and lib.rs:859 `file_size_bytes: page_count.saturating_mul(PAGE_SIZE as u64)`. `PageIo::set_stride` (page_io.rs:216-227) re-seeds `cached_page_count = len / stride`, and `page_count()` (page_io.rs:521) just returns that cache — so on an encrypted DB, which is at ENC_PAGE_SIZE from birth (recovery.rs:67 and 296), the count is in 8232-byte units and multiplying by 8192 under-reports the real file length by 0.49%. The promise is a true file size, not a formula: stats.rs:23 'Raw size of the database file on disk' and README.md:282 'Physical size of the database file in bytes'. The adjacent correct sibling is real too — recovery.rs:427-433/446 deliberately uses `cache.io_mut().stride()` with a comment naming this exact hazard, which makes the two accessors an inconsistency inside the same crate. Encrypted DBs are reachable from the public API via `Options::encryption_key`, so this is a wrong value returned from public accessors on a supported configuration. BUG stands. + +
+ +### BENCH-1 — discover_cells walks the wrong depth and never finds a real Criterion sample.json — every micro-grid timing cell renders as an em-dash +**Location:** `bench/src/summary/discover.rs:148-157, bench/src/summary/discover.rs:174` · **Category:** correctness · **Status:** NEW + +**What the code does.** The walk is bounded to exactly depth 4 with the comment "Each leaf is at depth 4 (criterion_dir/row/mode/size/sample.json)": `for entry in walkdir::WalkDir::new(criterion_dir).min_depth(4).max_depth(4)` … `if entry.file_name() != "sample.json" { continue; }`. Criterion 0.5 actually writes `///new/sample.json` — depth 5. Verified on this repo's real tree: `find target/criterion -maxdepth 4 -name sample.json` → 0 hits; `-maxdepth 5` → 2 hits. The only fixture that exercises this code (bench/tests/fixtures/criterion/allocate-1pertx/chisel-strict/32B/sample.json) omits the `new/` level, so the tests pass against a layout Criterion never produces. + +**Why it is a problem.** I ran the committed `summarize` binary against the real criterion dir: `Wrote 165 cells + 0 scenarios`, and every micro-grid cell in summary.md is `—` (all 165 cells came from aux_metrics.jsonl via the Step-3 leftover path with `timing: None`). `copy_raw_archive` (unbounded depth) in the same run correctly archived `raw/smoke/chisel-strict/256B/new/sample.json`, proving the file is present and only the depth filter is wrong. Net effect: summary.md and results.json report no p50/p95/p99 for any micro-grid cell, and the failure is silent — `NoCellsFound` never fires because the aux file supplies cells. + +**Direction of a fix.** Drop the depth bounds (or use min_depth(5).max_depth(5)) and select on the leaf's parent directory name being `new` so the `base/` copy isn't double-counted. Replace the synthetic fixture tree with one that has the `new/` level so the test actually covers the real layout. + +
Adversarial verification + +discover.rs:148-150 is verbatim `.min_depth(4).max_depth(4)` with the comment "Each leaf is at depth 4 (criterion_dir/row/mode/size/sample.json)", and line 161 `if entry.file_name() != "sample.json" { continue; }`. Ran the check on this repo's real tree: `find target/criterion -maxdepth 4 -name sample.json` -> 0, `-maxdepth 5` -> 2 (target/criterion/smoke/chisel-strict/256B/{new,base}/sample.json). Criterion's `new/` level is real and the depth filter excludes it. The three committed fixtures (bench/tests/fixtures/criterion/**) all omit `new/`, so the tests validate a layout Criterion never writes. The silent-failure path is confirmed too: summarize.rs defaults `--criterion target/criterion`, and discover.rs Step 3 emits every leftover aux entry with `timing: None`, so `NoCellsFound` (only fires when sample_paths_seen==0 AND cells.is_empty()) cannot trigger while aux_metrics.jsonl has its 165 lines. No guard anywhere restores the timing. Note: bench/results/ currently contains only aux_metrics.jsonl (no committed summary.md), so the "every cell renders as em-dash in the committed summary" phrasing is about a regenerated run, not a checked-in artifact — the code defect itself is exactly as described. + +
+ +### BENCH-3 — SQLite's cold-read cells time a journal_mode DELETE→WAL conversion that the harness itself created — cross-engine unfairness inside the timed region +**Location:** `bench/benches/micro_grid.rs:126-157 (run_cold_read_cell; engine open at :146-147 inside the routine), bench/src/sqlite_engine.rs:36-41 (unconditional `PRAGMA journal_mode = WAL;` on every open), :265-277 (flush_for_snapshot leaves DELETE mode), bench/src/runner.rs:399` · **Category:** performance · **Status:** NEW + +**What the code does.** `populate_snapshot` ends with `engine.flush_for_snapshot()?`, whose SQLite override does `PRAGMA journal_mode = DELETE` and asserts the result: "if new_mode != \"delete\" { return Err(...) }". The snapshot file is therefore left in rollback-journal mode. `run_cold_read_cell` then puts engine construction *inside* the timed routine — "// Cold-read cell-runner — row 4 only. Engine open is INSIDE the timed / routine" — and `SqliteEngine::open_file` unconditionally executes `PRAGMA journal_mode = WAL;` on every open. Chisel and redb do no equivalent conversion on open. + +**Why it is a problem.** Every cold-read iteration for sqlite-strict and sqlite-unsafe pays a full journal-mode conversion (exclusive lock, database-header rewrite, fsync — under `PRAGMA fullfsync=ON` on macOS that is a F_FULLFSYNC, milliseconds) on top of the single 1-op read the row claims to measure (`group.throughput(Throughput::Elements(1))`, workload = `gen_read_random(..., 1)`). The reported sqlite cold-read latency is inflated by orders of magnitude relative to chisel/redb, and the inflation is a pure harness artifact created by `flush_for_snapshot`, not a property of SQLite. + +**Direction of a fix.** Restore WAL mode on the snapshot after `flush_for_snapshot` (reopen-and-flip in `populate_snapshot`, outside any timed region), or have the cold-read runner pre-convert the working copy in setup so the timed `open` sees a WAL-mode file like a real deployment would. + +
Adversarial verification + +Mechanism confirmed: populate_snapshot ends with `engine.flush_for_snapshot()` (runner.rs:399); the SQLite impl runs `PRAGMA journal_mode = DELETE` and errors unless it returns "delete", so every sqlite snapshot file is left in rollback-journal mode. run_cold_read_cell's setup closure only copies the file; `mode.open(...)` is inside the timed routine (the doc comment states this explicitly), and SqliteEngine::open_file unconditionally issues `PRAGMA journal_mode = WAL;`. Chisel/redb do no equivalent. Workload is one op (gen_read_random(..., 1)), so the conversion dominates. Two rationale corrections: (1) at conversion time fullfsync is NOT yet on — open_file's execute_batch order is cache_size + journal_mode=WAL first, then `PRAGMA synchronous`, then `PRAGMA fullfsync = ON`, so the conversion's sync is an ordinary fsync at SQLite's default synchronous=FULL, not F_FULLFSYNC; (2) "orders of magnitude" is asserted, not measured — a header rewrite + fsync + -wal/-shm creation is realistically a small multiple of a one-row cold read, not 100x. Severity kept at BUG: the harness's product is the number, and the cold-read row reports a cross-engine comparison inflated by work the harness created for itself. + +
+ +### PUBLIC-API-1 — `Chisel::open` creates a database over an existing non-empty file even when `create_if_missing` is false, destroying its contents +**Location:** `src/lib.rs:366, src/lib.rs:384, src/lib.rs:393` · **Category:** correctness · **Status:** REGRESSION of I143 + +**What the code does.** Two different "does it exist" tests disagree. The gate uses byte length: `let file_exists = path.exists() && std::fs::metadata(path).map(|m| m.len() > 0).unwrap_or(false);` followed by `if !file_exists && !options.create_if_missing { return Err(ChiselError::FileNotFound); }`. The create-vs-open decision uses page count: `let existed = io.page_count()? > 0;` … `let txm = if existed { open_existing(...) } else { create_new(...) };`. `PageIo::page_count()` is `cached_page_count`, seeded at src/page_io.rs:154 as `initial_len / PAGE_SIZE as u64`, so ANY file of 1..8191 bytes yields 0. `PageIo::open` uses `.create(true).truncate(false)` (src/page_io.rs:138-143) on the read-write path, so the existing file is opened, and `create_new` then writes N superblock slots over it. + +**Why it is a problem.** Verified empirically against the built crate from a downstream binary: writing a 60-byte text file and calling `Chisel::open(path, Options::default().create_if_missing(false))` returned `Ok(Chisel)` and the file grew 60 → 16384 bytes with the original bytes gone. Two failures at once: (a) `create_if_missing: false` created a database, violating the option's documented contract; (b) an arbitrary user file under 8 KB (a mistyped path, a config file, or a database whose very first superblock write was torn mid-create) is silently overwritten instead of surfacing `FileNotFound`/`CorruptSuperblock`. The docstring at src/lib.rs:332-337 also claims the check "deliberately treats a zero-length file as nonexistent", while the code treats every sub-page file as nonexistent. + +**Direction of a fix.** Derive both decisions from the same post-lock observation: after taking the flock, treat `page_count() == 0` as "absent" and re-apply the `create_if_missing` gate there (return `FileNotFound` when it is false), so the pre-lock stat is only an optimization. A non-empty file that is too short to hold a superblock should never reach `create_new`. + +
Adversarial verification + +Reproduced from a downstream probe crate compiled against this repo. Wrote a 53-byte text file, called Chisel::open(path, Options::default().create_if_missing(false)): returned Ok, and the file went 53 -> 16384 bytes with the first bytes now the superblock magic (76,83,72,67 = 'LSHC'). Code path verified: src/lib.rs `let file_exists = path.exists() && metadata(..).map(|m| m.len() > 0)` gates only create_if_missing (true for a 53-byte file, so the FileNotFound gate is skipped), while `let existed = io.page_count()? > 0` (I143 comment block) decides create-vs-open; PageIo::open seeds cached_page_count = initial_len / PAGE_SIZE (src/page_io.rs ~154) so any 1..8191-byte file yields 0. PageIo::open uses .create(true).truncate(false) on the read-write path so the existing file is opened and create_new writes N superblock slots over it. create_new (src/transaction/recovery.rs:32) has no emptiness guard — only an assert on superblock_count. Both halves of the finding hold: create_if_missing:false creates, and a sub-page user file is silently destroyed. + +
+ +### GAP-1 — `rollback_to` never rewinds `FreemapRecycle` (BUG candidate, filed DESIGN) +**Location:** `src/transaction/savepoints.rs:65-107` vs `src/transaction/lifecycle.rs:301` + +**What the code does.** I confirmed this asymmetry directly. `rollback_inner` (full rollback) calls `self.freemap.rollback()` at `lifecycle.rs:301` with a comment explaining the per-stream reasoning. `rollback_to_inner` restores the cache (`truncate(watermark)`), `current_roots`, membership-index depth, handle-table depth, packer state, the savepoint stack, and `txn_freed_pages` — and never touches `FreemapRecycle` at all. Grepping lines 64-135 of `savepoints.rs` for `freemap`/`structural` returns only an unrelated comment about `txn_freed_pages`. + +**Why it is a problem.** `truncate(watermark)` destroys every page above the savepoint watermark, but `structural_superseded` / `structural_reuse` / the session set still describe those pages. This is the one rewind path that leaves the recycle streams describing pages that no longer exist. + +**Direction of a fix.** Give `FreemapRecycle` a watermark-aware rewind and call it from `rollback_to_inner`, or snapshot the recycle state into `Savepoint` alongside `live_slots`/`insert_cursor`. Whether this reaches corruption depends on how the streams are consumed at commit — worth tracing before choosing a severity. + +*This finding came from the completeness critic and went through no verifier; the asymmetry is mine, verified by reading both paths.* + +--- + +## Findings by category + +### correctness (15) + +#### `DESIGN` SECURITY-SWEEP-6 — txn_counter is adopted unvalidated from the superblock, so the "structurally unreachable" expect in commit is reachable from a hostile file +**src/transaction/commit.rs:117-120 (rationale comment 111-116); src/transaction/keys.rs:65-68 (identical construction); src/transaction/recovery.rs:533 `txn_counter: sb.txn_counter`; deserialize at src/superblock/mod.rs:525** · *NEW* + +`open_existing` adopts the counter verbatim: `txn_counter: sb.txn_counter` (recovery.rs:533), and `Superblock::deserialize` reads it with no bound (`txn_counter: u64::from_le_bytes(buf[8..16].try_into().unwrap())`, superblock/mod.rs:525) — `validate()` checks only checksum, magic, and `superblock_count` range. Commit then does `*ctx.txn_counter = ctx.txn_counter.checked_add(1).expect("txn_counter overflowed u64 (2^64 commits) — unreachable")` (commit.rs:117-120). The comment justifying the `expect` over a typed error reads: "Overflow needs 2^64 commits, so it is structurally unreachable; a dedicated fatal error variant for an impossible event would be speculative public surface." `rewrite_crypto_header_inner` carries the identical construction and identical claim (keys.rs:66-69). + +**Why:** The unreachability argument assumes the counter can only ever be produced by this binary's own increments; it is in fact read from a file the threat model says the attacker controls. Craft a superblock with `txn_counter = u64::MAX`, valid magic, `superblock_count = 2`, `page_size = 8192`, `format_version = pack(1,1)`, `total_pages <= page_count`, and a recomputed XXH3 checksum. `Chisel::open` accepts it; the first `begin()` + `commit()` then panics on the `expect` rather than returning a typed error. The panic unwinds through `TransactionManager::commit`, so unlike the abort cases it is at least catchable, but it bypasses the documented poison-and-reopen contract in lib.rs:295-304 and, in the PyO3 binding, surfaces to Python as a `PanicException` rather than one of the mapped error classes. + +**Fix:** Either bound `txn_counter` at open (a database whose counter is within a few of u64::MAX is corrupt by any reasonable measure — reject it as `CorruptSuperblock` alongside the existing `superblock_count` range check in `validate`), or convert the two `expect` sites into a fatal typed error. Whichever is chosen, correct the "structurally unreachable" rationale in commit.rs:111-116 and keys.rs:64-65, which is the sentence a future maintainer will trust. + +#### `DESIGN` SUPERBLOCK-RECOVERY-4 — Any database file shorter than one page is treated as nonexistent and silently overwritten by `create_new`, contradicting the documented zero-length rule +**src/lib.rs:384, src/lib.rs:332, src/transaction/recovery.rs:91** · *REGRESSION of I143 (same defect as PUBLIC-API-1)* + +The doc on `Chisel::open` states the rule as a zero-length test: "The \"exists\" check deliberately treats a zero-length file as nonexistent" (lib.rs:332-334). The post-lock decision that actually routes create-vs-open is `let existed = io.page_count()? > 0;` (lib.rs:384), and `page_count()` is floor division by the stride (`self.cached_page_count`, seeded from `len / stride`, page_io.rs:521 and page_io.rs:226). So the real threshold is `len >= 8192`, not `len > 0`. When it resolves false, `TransactionManager::create_new` runs and writes fresh superblock slots over the file: `cache.io_mut().write_page(i as u64, &buf)?` (recovery.rs:91). + +**Why:** A database file truncated by an external tool (a failed `cp`, a filesystem event, a partial restore) to any length in 1..8191 bytes is not reported as corrupt — it is silently reinitialized as an empty database, destroying the surviving bytes at page 0 and returning a working, empty handle to the caller. The documented fatal errors for a damaged file (`CorruptSuperblock`, `FileSizeMismatch`) are never reached, and the operator gets no signal that data was lost. + +**Fix:** Route the create-vs-open decision on the post-lock file LENGTH being exactly zero, not on `page_count() > 0`; a nonzero-but-sub-page file should reach `open_existing` and fail as `CorruptSuperblock`. + +#### `DESIGN` HANDLES-INDEX-3 — HandleTable::insert's eager depth bump is unwound at only one of its three call sites; the other two rely on a prose-only "cannot grow" argument +**src/transaction/freemap.rs:629 (`ht_insert`) and src/transaction/mutate.rs:99 (`update_inner` — calls `handle_table_insert_candidate`, NOT `handle_table.insert`); the eager bump is handle_table.rs:521; the only restore is staging.rs:153 inside `abort_allocate_prepare`, called from `allocate_inner` at staging.rs:315** + +`HandleTable::grow` performs `self.depth += 1;` (handle_table.rs:521) after its own alloc succeeds but *before* `insert_recursive`'s fallible COW runs, so a failure after a successful grow leaves `handle_table.depth` one level deeper than the root that is actually installed. `staging.rs:75-77` documents the hazard ("`HandleTable::insert` may `grow`, which bumps the in-memory descent depth eagerly — the caller captures and restores that depth on the prepare-abort path") and `abort_allocate_prepare` restores it (`staging.rs:153` `self.handle_table.set_depth(saved_depth);`). The other two callers do not: `ht_insert` (src/transaction/freemap.rs:629-658) calls `self.handle_table.insert(...)` and on `result?` (:654) returns without touching depth; `update_inner` (src/transaction/mutate.rs:98-107) discards the error after only releasing the inline slot. `update_inner`'s justification is prose at mutate.rs:52-54: "`update` replaces an EXISTING handle, so handle_table.insert never grows (handle < capacity) — no in-memory depth save is needed"; `ht_insert` states no justification at all. + +**Why:** Both unguarded callers happen to be safe today only because their handle was just resolved by `lookup_live` (read.rs:42-48), whose `find_leaf` capacity guard (handle_table.rs:695) implies `handle < capacity()` and therefore that `while handle >= self.capacity()` at handle_table.rs:253 never fires. That is an inter-module argument held together entirely by a comment: the moment any caller passes a not-yet-resolved handle to `ht_insert` (the natural way to add a second entry-mutating API next to `set_client_byte`), a `CacheFull` after a grow leaves depth deeper than the installed root, and every subsequent `lookup` mis-descends and returns `InvalidHandle` for committed handles — the documented I99 failure mode, reached without any rollback. + +**Fix:** Make the invariant structural instead of prose: either have `HandleTable::insert` restore `self.depth` itself on the error path (bump depth only after `insert_recursive` returns Ok, or snapshot-and-restore inside `insert`), or route all three callers through the single `handle_table_insert_candidate` + save/restore-depth helper. + +#### `DESIGN` BENCH-2 — freemap_churn's "in-memory" benchmark is file-backed — the Criterion report label is wrong +**bench/benches/freemap_churn.rs:114 (doc), :140 (BenchmarkId::new("in-memory", label)), :145-157 (tempfile setup), :81-83 (churn_cycle doc)** + +The doc comment reads "Bench group 1: DELETE-CHURN-FLAT (in-memory engine)" and the benchmark id is `BenchmarkId::new("in-memory", label)`, but the setup opens a real file: "// Use a tempfile-backed engine since in-memory / // Vec opens need special handling in Chisel." followed by `let tf = NamedTempFile::new()…; let mut db = Chisel::open(tf.path(), …)`. `churn_cycle`'s doc reinforces the false claim: "= 2 commits = 6 fsyncs (in-memory: 6 no-op fsyncs)". + +**Why:** Criterion emits a row literally named `freemap-churn-flat/in-memory/500x256B` whose numbers include real file writes and real F_FULLFSYNC calls per commit. Anyone reading the report (or a later reviewer comparing it against the file-backed group 2) will attribute file+fsync cost to pure CPU/memory work — off by the entire fsync tax the harness elsewhere (I92, EngineMode::ChiselMemory) goes out of its way to isolate. The stated justification is also false: `Chisel::open_in_memory_with_options` is public and is already used by `ChiselEngine::open_in_memory` (bench/src/chisel_engine.rs:51-58). + +**Fix:** Either switch the setup to `Chisel::open_in_memory_with_options` (matching the label) or rename the benchmark id and doc to `chisel-file`, and delete the "in-memory: 6 no-op fsyncs" clause from `churn_cycle`. + +#### `DESIGN` BENCH-10 — The noise gate reports PASS when it has no variance data at all (--runs 0 or 1, or an empty cell set) +**bench/src/bin/noise_gate.rs:32-34, bench/src/bin/noise_gate.rs:72, bench/src/bin/noise_gate.rs:154-155, bench/src/noise_gate/report.rs:29-31, bench/src/noise_gate/cov.rs:22-30** · *KNOWN as I95* + +`runs` is taken verbatim from argv (`#[arg(long, default_value = "5")] runs: usize`) and drives `for run_idx in 0..cli.runs`. `compute_cov` returns `Cov { mean, stddev: 0.0, cov: 0.0 }` for a single sample ("Return 0.0 stddev so the noise gate doesn't false-fail"), and `passes = throughput.cov <= cli.throughput_threshold && p99_latency_ns.cov <= cli.p99_threshold`. The verdict is `self.cells.iter().all(|c| c.passes)`, which is `true` for an empty `cells` vector. + +**Why:** `chisel-bench-noise-gate --provider x --instance-type y --runs 0` executes no benchmark, collects no samples, and exits 0 printing "Noise gate PASSED" over a "0 / 0 cells under threshold" report. `--runs 1` is worse: it runs once, every COV is a hard-coded 0.0, and every cell passes — a candidate machine is qualified for the dedicated bench fleet on evidence that cannot detect noise by construction. This is the one tool whose entire job is to refuse bad hardware. + +**Fix:** Reject `runs < 2` at argv-parse time (a COV needs at least two samples) and make `all_pass()` false when `cells` is empty, so an empty or single-run result is a FAIL rather than a vacuous PASS. + +#### `DESIGN` TXN-COMMIT-1 — rollback_to restores a non-None insert cursor, so slot packing resumes into a below-watermark page while a savepoint is still active — the exact case the design says it forbids +**src/transaction/savepoints.rs:98, src/transaction/packing.rs:119** · *KNOWN as I151* + +`savepoint_inner` snapshots the cursor BEFORE clearing it: `let (live_slots, insert_cursor) = self.packer.snapshot(); self.packer.clear_cursor();` (savepoints.rs:32-33). `rollback_to_inner` then puts that snapshot back — `let snap = (self.savepoints[idx].live_slots.clone(), self.savepoints[idx].insert_cursor); self.packer.restore(snap);` (savepoints.rs:98-102) — and immediately afterwards keeps the savepoint on the stack: `self.savepoints.truncate(idx + 1);` (savepoints.rs:103). `SlotPacker::insert` takes the cursor branch unconditionally: `if let Some(cursor_page_id) = self.insert_cursor {` (packing.rs:119), guarded only by the comment `// The cursor only exists when packing is enabled (savepoints empty), so this branch implicitly respects the "no packing under savepoints" rule.` (packing.rs:115-118). `packing_enabled` (packing.rs:160) gates only INSTALLING a new cursor, never USING an existing one. + +**Why:** Concrete sequence: `begin(); allocate(v1)` — savepoints is empty so packing_enabled is true and the cursor becomes page P (packing.rs:160-161). `savepoint("s")` snapshots cursor=Some(P) and clears it. `rollback_to("s")` restores cursor=Some(P) while savepoint "s" is still on the stack. The next `allocate(v2)` now appends a slot to P — and P's id is BELOW the savepoint watermark, so a second `rollback_to("s")` neither truncates it (savepoints.rs:75 only calls `truncate(watermark)`) nor discards it (unlike `rollback_inner`, which calls `cache.discard_all_dirty()` first, lifecycle.rs:265). The slot bytes and the advanced free_start/free_end survive the rewind as permanent dead weight; repeated savepoint/rollback_to/insert cycles keep consuming P until it is full. Reads stay correct (DataPage::insert is append-only, live_slots is restored from the snapshot), so this is a bounded space leak rather than corruption — but it silently violates the invariant that packing.rs:27-30, savepoints.rs:31, freemap.rs:26-28 and page_cache.rs:688-693 ("Savepoint-bearing transactions disable freemap reuse ... so there are no dirty reused-id pages to worry about") all rely on. No test in src/transaction/tests.rs allocates after a rollback_to; `rollback_to_savepoint_truncates_to_savepoint_watermark` (tests.rs:345) stops at the rewind. + +**Fix:** Have `rollback_to_inner` clear the cursor instead of restoring it (call `packer.clear_cursor()` after `restore`, or drop `insert_cursor` from the `Savepoint` record entirely and always restore `None`). Alternatively gate the cursor branch in `SlotPacker::insert` on `packing_enabled` so the caller's savepoint state is authoritative in one place. + +#### `DESIGN` FREEMAP-1 — The freemap allocation hint is advanced by allocate_first but never rewound on rollback, permanently stranding free pages below it +**src/transaction/freemap.rs:137, src/transaction/freemap.rs:522, src/freemap_tree.rs:479** · *NEW* + +`FreemapRecycle::hint` is documented as safe to leave untracked: "Deliberately NOT transactionally tracked: a too-low hint only costs a wasted left-to-right scan, never correctness ... so it needs no begin/rollback snapshotting — it PERSISTS across transactions" (src/transaction/freemap.rs:137-141). `FreeMapTree::allocate_first` advances it on every claim: "*hint = found;" (src/freemap_tree.rs:479). `FreemapRecycle::rollback` clears `structural_superseded`, `structural_reuse` and `session_owned` but does NOT touch `hint` (src/transaction/freemap.rs:522-526), and `begin` explicitly leaves it: "`hint` is NOT reset — it persists across transactions (a stale hint only costs a scan)" (src/transaction/freemap.rs:487-488). The only thing that ever lowers it is `self.hint = self.hint.min(id);` in `mark_free_committed_path` (src/transaction/freemap.rs:286), which runs only at commit (persist) or during the defrag orphan sweep. The doc's own argument covers only the too-LOW direction; five lines earlier the code admits the other direction is not benign: "a too-high hint would start the next scan above `id` and never reuse it" (src/transaction/freemap.rs:283-284). + +**Why:** Committed bitmap holds free ids {10, 20, 30}. A transaction allocates twice via `cow_alloc` -> `allocate_first`: it gets 10 (hint=10) then 20 (hint=20). The caller then rolls back. `rollback_inner` restores `current_roots` from `committed_roots`, so both bits are free again in the committed tree — but `hint` stays at 20. Every subsequent `allocate_first` calls `scan_from(cache, 20)`, so id 10 is never handed out again; the allocator extends the file instead. Because `mark_free_committed_path` is the only thing that pulls the hint back, the stranded range is recovered only if a page at an id <= 10 is later freed, or the database is reopened (`FreemapRecycle::new` seeds hint=0). A long-lived session with rollbacks and only high-id frees leaks reusable space monotonically. + +**Fix:** Snapshot `hint` at `begin` and restore it in `rollback` (one u64, no on-disk change), or simply reset `hint = 0` in `rollback`. Either way, correct the field doc so it no longer claims the untracked hint "never" costs correctness — state the too-high direction and how it is bounded. + +#### `DESIGN` FREEMAP-9 — defrag's empty-handle-table fast path skips the orphan sweep on a premise that the allocate-abort path falsifies +**src/defrag.rs:201, src/transaction/staging.rs:139** · *NEW* + +defrag returns before step 7 when the handle-table root is unset — `if txm.current_handle_table_root_page() == PAGE_ID_NONE { return Ok(stats); }` (src/defrag.rs:205-207) — justified by: "the handle-table root materialises permanently on the first `allocate` and never reverts to PAGE_ID_NONE, so an empty handle table implies no allocation has ever succeeded — therefore no freemap tree exists" (src/defrag.rs:200-205). `abort_allocate_prepare` documents the direct contradiction: it restores "`current_roots.handle_table_page` — reverts to the pre-allocate value, undoing any lazy `ensure_handle_table` materialization (empty DB goes back to `PAGE_ID_NONE`)" (src/transaction/staging.rs:135-139), and on the same path releases the inline value's data slot via `release_data_slot`, which pushes the drained page onto `txn_freed_pages` (src/transaction/packing.rs:298-302). A commit after that operational failure runs `persist`, whose lazy-create branch materializes the tree: `if tree.root == PAGE_ID_NONE { ... tree = FreeMapTree::create(cache, &mut extend)?; }` (src/transaction/freemap.rs:275-278). + +**Why:** Concrete state: a fresh database; `allocate` fails non-fatally (e.g. the CacheFull the fault injector models at src/transaction/staging.rs:88-91) after `ensure_handle_table` and the inline insert; `abort_allocate_prepare` reverts the handle-table root to PAGE_ID_NONE and frees the data page; the caller commits rather than rolling back. The committed superblock now has `root_handle_table_page == PAGE_ID_NONE` and a real `root_freemap_page`. Every future `defrag()` on that database returns at src/defrag.rs:205 and never runs `reclaim_freemap_orphans`, so any freemap page a later crash strands is unreclaimable by the only mechanism that reclaims them. The stated proof ("therefore no freemap tree exists") is what makes the shortcut look safe. + +**Fix:** Gate the fast path on the freemap root as well — skip step 7 only when `current_roots.freemap_page == PAGE_ID_NONE` (which `reclaim_orphans` already checks at src/transaction/freemap.rs:410) — and drop the 'never reverts' claim from the comment. + +#### `SMELL` SUPERBLOCK-RECOVERY-7 — A forged crypto-header stride is a hard error on the intact-page-0 path and silently ignored on the torn-page-0 path, and the trailing comment misstates which value was applied +**src/transaction/recovery.rs:296, src/transaction/recovery.rs:265, src/transaction/recovery.rs:350** + +The anchor path validates the advertised stride and refuses anything else: `if stride != crate::crypto::ENC_PAGE_SIZE { return Err(ChiselError::CorruptSuperblock { defects: Vec::new() }); }` (recovery.rs:265-269). The torn-slot-0 fallback instead hardcodes the stride and accepts the candidates on the mere PRESENCE of a header, never comparing the sibling's advertised value: `cache.io_mut().set_stride(crate::crypto::ENC_PAGE_SIZE); ... if encrypted_stride(&enc_candidates).is_some() { candidates = enc_candidates; }` (recovery.rs:296-299). The comment 50 lines later then claims "The IO stride was already switched to header.stride during the bootstrap read above" (recovery.rs:350-351), which is only true on the anchor path. + +**Why:** Two identical inputs get two different verdicts purely on whether page 0 happens to be torn: a file whose header advertises stride 12345 is rejected as `CorruptSuperblock` when page 0 is intact, and opened normally (at 8232) when page 0 is torn. Neither outcome is unsafe today, but the divergence means the I144 guard is not actually a guard on the header field — it is a guard on one of two code paths — and the comment tells the next maintainer the winner's `header.stride` was honored when in the fallback it was never consulted at all. + +**Fix:** Validate the winner's `header.stride == ENC_PAGE_SIZE` once, after selection, on both paths (the anchor check then becomes a fast pre-check), and reword the comment at recovery.rs:350 to say the stride is the constant `ENC_PAGE_SIZE`, not the header's value. + +#### `SMELL` SUPERBLOCK-RECOVERY-8 — `Spillway::slot_size()` is dead code whose doc names a caller that does not exist +**src/spillway.rs:214, src/spillway.rs:212, src/page_cache.rs:522** · *NEW* + +The method is annotated `#[allow(dead_code)]` yet its doc asserts a live consumer: "On-disk slot size in bytes: `SLOT_HEADER_SIZE + payload_size`. Used by Task 3.3/3.4 to size drain buffers for encrypted DBs." (spillway.rs:211-216). Grepping the tree, `slot_size()` has no call sites at all — the drain path it names sizes its buffer from the crypto constant directly: `let unit: [u8; ENC_PAGE_SIZE] = blob.as_slice().try_into()` (page_cache.rs:522). The sibling `SLOT_SIZE` const (spillway.rs:60) is likewise doc'd as used "by callers that pass PAGE_SIZE as payload_size" but appears only in this file's own `#[cfg(test)]` module. + +**Why:** The `#[allow(dead_code)]` plus a confident "used by" doc makes an unused accessor look load-bearing, so nobody deletes it and a reader hunting the drain sizing logic goes to `slot_size()` and finds it is not what the drain uses. The two size sources (this method's `SLOT_HEADER_SIZE + payload_size` and the drain's `ENC_PAGE_SIZE`) can drift with no compiler complaint precisely because the method is never called. + +**Fix:** Delete `slot_size()` (and `SLOT_SIZE` if only tests use it, moving it into the test module), or make the drain path call it so the doc becomes true and there is a single sizing source. + +#### `SMELL` HANDLES-INDEX-7 — MAX_DEPTH is enforced only on the recovery walk, not on the growth loop, which does not terminate for handle == u64::MAX +**src/handle_table.rs:253, src/handle_table.rs:77, src/membership_index.rs:243** · *KNOWN as I147* + +The header asserts a hard bound: handle_table.rs:75-81 — "capacity(5) ≈ 5.7e17 < u64::MAX < capacity(6), so a tree keyed by u64 handles is never deeper than 6 (a handle in (capacity(5), u64::MAX] forces one final grow to depth 6, whose capacity saturates to u64::MAX). Any spine claiming a deeper tree is corrupt". `MAX_DEPTH` is referenced only inside `recover_depth` (:448). The growth loop is `while handle >= self.capacity() { current_root = self.grow(...)?; }` (:253) and `capacity()` uses `saturating_mul` (:483), so at depth >= 6 it returns `u64::MAX`. For `handle == u64::MAX` the predicate `u64::MAX >= u64::MAX` stays true after every grow, so the loop keeps calling `grow` — allocating one page and incrementing `self.depth` per iteration — until `alloc` errors. `RadixU64::insert` has the identical loop at membership_index.rs:243. Note that `find_leaf` explicitly reasons about this saturation case (handle_table.rs:688-693, the `cap != u64::MAX` clause) while the growth loop does not. + +**Why:** With `handle == u64::MAX` the tree is driven past the depth the module documents as impossible-unless-corrupt, and the only thing that stops it is an allocation failure after having extended the file. Unreachable today — `next_handle` starts at 1 and is only ever `+= 1` (staging.rs:325), so reaching u64::MAX needs ~1.8e19 committed allocations, and no public API lets a caller name an arbitrary handle for insert. The defect is that the stated depth invariant is enforced nowhere on the write path, so the header's "never deeper than 6" is a claim about arithmetic, not about the code. + +**Fix:** Bound the loop explicitly — `while self.depth < MAX_DEPTH && handle >= self.capacity()` in both radices — and let the (now impossible) leftover case fall out as a typed error rather than an unbounded grow; that also makes MAX_DEPTH a real invariant instead of a recovery-only constant. + +#### `SMELL` CRYPTO-4 — The crypto-header `algorithm` byte is written but never validated; any nonzero value is treated as XChaCha20-Poly1305 +**src/superblock/crypto_header.rs:146, src/transaction/recovery.rs:265, src/transaction/recovery.rs:612** + +`CryptoHeader::deserialize` gates only on zero: `let algorithm = buf[CRYPTO_HEADER_OFFSET]; if algorithm == 0 { return None; }` (crypto_header.rs:146-149) and then returns `Some(CryptoHeader { algorithm, ... })` for any other value. The declared contract is narrower: "Algorithm id stored in the header. 0 means \"no encryption\" ... the only supported nonzero value today is 1 = XChaCha20-Poly1305" (crypto_header.rs:43-44). `open_existing` validates the sibling field — `if stride != crate::crypto::ENC_PAGE_SIZE { return Err(ChiselError::CorruptSuperblock ... ) }` (recovery.rs:265-269) — but never compares `algorithm` against `ALGO_XCHACHA20POLY1305`. A grep of the whole tree shows `ALGO_XCHACHA20POLY1305` appears only at write sites (recovery.rs:613, plus tests); it is never read back for comparison. + +**Why:** A file stamped `algorithm = 2` (a future AES-GCM or a corrupted/forged byte) is accepted as encrypted and every page is fed to `PageCipher::open`, which is hardcoded to XChaCha20-Poly1305. If a future format ever uses a second algorithm id, today's binary will not reject those files with a clean `UnsupportedFormatVersion`; it will attempt XChaCha20 decryption and surface `DecryptionFailed`, which `is_fatal()` (error.rs:226) — poisoning the handle and presenting an algorithm-mismatch as data corruption. The field exists precisely to prevent this and is dead on the read path. + +**Fix:** In `open_existing`, next to the existing stride check, reject `header.algorithm != ALGO_XCHACHA20POLY1305` with a typed error (`UnsupportedFormatVersion` or a dedicated variant), before any `PageCipher` is constructed. + +#### `SMELL` PAGE-IO-3 — claim_page's I20 guard is spillway-blind: is_dirty() ignores spilled pages and claim_page never calls spillway.forget, so a stale spilled blob can overwrite the claimed page at flush +**src/page_cache.rs:926-932 (is_dirty doc + body), src/page_cache.rs:800-836 (claim_page)** · *NEW* + +`claim_page` enforces its safety invariant with `debug_assert!(!self.is_dirty(page_id), "claim_page called on a dirty page ...")` and then does only `self.entries.remove(&page_id);` before inserting the fresh zeroed dirty entry — with the justification "Remove any pre-existing entry so a stale cached copy from a prior reader doesn't leak into the new transaction's view." But `is_dirty` is `self.entries.get(&page_id).is_some_and(|e| e.dirty)` — it only consults `entries`. A page that was evicted to the spillway is by definition dirty (load_page's own comment, src/page_cache.rs:967-968: "a resident page is by definition dirty") yet reports `false`, and `claim_page` never calls `spw.forget(page_id)`. Its doc additionally claims "Used by the transaction layer to reason about whether a page is safe to drop at savepoint/rollback boundaries" — grep shows `is_dirty` has zero callers outside this file's own debug_assert and its tests. + +**Why:** If any allocator ever hands `cow_alloc`/`structural_extend` an id that is spillway-resident, the debug_assert stays silent and the stale copy survives in the spillway. flush() then writes the NEW claimed content in Phase 1a and, in Phase 1b, copies the OLD spilled blob over it (`self.io.write_page_unit(page_id, &pt)?`, src/page_cache.rs:539) — and the Vacant-only re-insert at src/page_cache.rs:553 leaves the cache holding the new bytes marked clean, so the divergence is invisible until the page is evicted and cold-read. The invariant the assert exists to protect is exactly the one it cannot check. + +**Fix:** Make `is_dirty` return true when the page is spillway-resident (`self.spillway.as_ref().is_some_and(|s| s.is_resident(page_id))`), so the I20 debug_assert covers the spilled case, and have `claim_page` call `spw.forget(page_id)` alongside `entries.remove`. Also delete the false "used by the transaction layer" sentence from is_dirty's doc. + +#### `SMELL` PAGE-IO-7 — flush() clears per-entry dirty flags one at a time but zeroes dirty_count in bulk, leaving the counter over-counting on any mid-loop error +**src/page_cache.rs:443-451, src/page_cache.rs:1093-1096** · *NEW* + +Phase 1a clears the flag per page — `self.entries.get_mut(&page_id).unwrap().dirty = false;` — but the counter is only reset after the whole loop: `self.dirty_count = 0;` (src/page_cache.rs:451). The design note at src/page_cache.rs:416-418 defends this ("dirty_count is reset to zero here, not decremented per page, to survive any future reordering of the loop body"), but the loop body contains a `?` at src/page_cache.rs:444. An I/O error on the Nth page returns with N-1 entries already flagged clean while `dirty_count` still counts them. `untrack_dirty`'s I116 doc reasons only about the opposite direction — "`saturating_sub` makes a (buggy) `entries`/`dirty_count` desync degrade to ..." — which does nothing for an over-count. + +**Why:** After a failed flush, `dirty_count > (number of entries with dirty == true)`. `evict_clean_to_cap`'s early-out `if self.dirty_count == self.entries.len() { break; }` can then fire while clean, evictable entries exist, so the cache stops enforcing `max_pages` and `maybe_evict` Phase B starts spilling or raising CacheFull on a cache that had room. Today this is masked only because a flush error poisons the manager (the same argument the durability-window comment makes at src/page_cache.rs:397-412) — it is one poison-model relaxation away from being live. + +**Fix:** Decrement `dirty_count` in the same statement that clears each entry's flag, or move the flag clears into a second pass that runs only after every write succeeded. Either way the counter and the flags stop diverging on the error path. + +#### `NIT` PAGE-IO-6 — The stamp-checksum-before-write invariant is stated only in prose; flush() writes whatever bytes it finds with no assertion +**src/page_cache.rs:443-445 (flush Phase 1a)** · *NEW* + +`stamp_checksum`'s doc states the invariant: "Must be called after every mutation and before the page is handed to page_io for writing — otherwise the next read will see a stale checksum and treat the page as corrupt." Nothing enforces it. flush() Phase 1a copies and writes unconditionally: `let plaintext: [u8; PAGE_SIZE] = *self.entries.get(&page_id).unwrap().buf; self.write_sealed(page_id, &plaintext)?;` and `write_sealed` either seals or forwards to `write_page_unit` — neither computes or checks the trailing 8 bytes. Contrast `claim_page`, which does guard its analogous prose invariant with a `debug_assert!` (src/page_cache.rs:808-811). + +**Why:** A page-type module (or a future one) that mutates a buffer via `get_mut` and forgets to restamp writes a self-inconsistent page. The write and the fsync both succeed, commit succeeds, and the damage surfaces arbitrarily later as `ChecksumMismatch { page_id }` on the first cold read after eviction — which `load_page`'s doc declares fatal ("signals the database is broken") and which poisons the manager. Diagnosis then starts at a read site with no connection to the module that skipped the stamp. + +**Fix:** Add `debug_assert!(page::verify_checksum(&plaintext), "flush: page {page_id} written without a stamped checksum")` in flush's Phase 1a loop (and the same in the Phase 1b plaintext drain branch). Zero release cost, and it fails at the offending write instead of hours later at an unrelated read. + +### security (9) + +#### `DESIGN` SECURITY-SWEEP-1 — Argon2id cost parameters are read verbatim from the untrusted superblock and drive an unbounded allocation and unbounded CPU during Chisel::open +**src/transaction/recovery.rs:649 and src/superblock/crypto_header.rs:209 (both exact); derive_kek at src/crypto/mod.rs:185/193 (exact)** · *KNOWN as I150* + +Every key-slot's Argon2 cost params are parsed straight out of the on-disk crypto header with no range check and then handed to the KDF. `crypto_header.rs:110-112` reads them: `m_cost: u32::from_le_bytes(slot[2..6].try_into().unwrap()), t_cost: ..., p_cost: ...`. The open path passes them through unmodified — `recovery.rs:649`: `let kek = match derive_kek(key, kdf, &slot.salt, &slot.argon2)` — as does the key-management path, `crypto_header.rs:209`: `let kek = match crypto::derive_kek(key, kdf, &slot.salt, &slot.argon2)`. `derive_kek` then does `let p = Params::new(params.m_cost, params.t_cost, params.p_cost, Some(32))` (crypto/mod.rs:185) followed by `a2.hash_password_into(ikm, salt, okm.as_mut())` (crypto/mod.rs:193). In argon2 0.5.3, `Params::MAX_M_COST = u32::MAX` and `MAX_T_COST = u32::MAX` (params.rs:49,58 — the crate comments say "we don't need to check MAX_M_COST, since it's u32::MAX"), and `hash_password_into` does `let mut blocks = vec![Block::default(); self.params.block_count()]` (lib.rs:230), one 1 KiB Block per m_cost unit. Only `m_cost < MIN_M_COST` (8) is rejected. Nothing in Chisel clamps these values. + +**Why:** An attacker who supplies a .chsl file (the stated threat model: attacker controls the on-disk bytes) sets slot 0 to `state=1, kdf_id=2, m_cost=0xFFFFFFFF, t_cost=0xFFFFFFFF` and recomputes the XXH3 page checksum, which is non-cryptographic and publicly computable. When the victim calls `Chisel::open(path, Options::default().encryption_key(k))`, `unwrap_first_matching_slot` reaches `derive_kek` and argon2 tries to allocate 4 TiB of Blocks: `handle_alloc_error` -> process abort, uncatchable, bypassing the poison model entirely. A milder m_cost (e.g. 8_000_000 = 8 GiB) OOM-kills the host instead; a large t_cost hangs the process indefinitely. All 8 slots are tried in sequence, multiplying the cost 8x. The AAD binding in `KeySlot::aad()` does NOT help — it authenticates the params only AFTER the expensive KDF has already run. The KDF is selected from the slot's `kdf_id`, not from the caller's key variant (documented deliberately at crypto/mod.rs:144-150), so a victim using `Key::Raw` is equally exposed. The same reachability exists via `Chisel::add_key`/`rotate_key`/`remove_key`, which call `CryptoHeader::unlock`. + +**Fix:** Clamp the on-disk Argon2 params before they reach `derive_kek`: reject (skip the slot, or fail the open with `InvalidEncryptionKey`) any slot whose `m_cost` exceeds a sane ceiling — a few hundred MiB, well above the 19456 KiB OWASP default this crate writes — and likewise bound `t_cost`/`p_cost`. Do the validation in one place, ideally inside `derive_kek` itself so both `unwrap_first_matching_slot` and `CryptoHeader::unlock` inherit it. + +#### `DESIGN` SECURITY-SWEEP-2 — Overflow::read sizes a Vec from a disk-controlled u64 length; a forged total_length on an Overflow-typed page aborts the process from Chisel::read +**src/overflow.rs:167 and src/overflow.rs:183 (both exact; the overstated comment is at 178-182)** · *KNOWN as I149* + +`Overflow::read` takes the value length straight off the page: `u64::from_le_bytes(buf[16..24].try_into().unwrap()) as usize` (overflow.rs:167), rejects only `total_length == 0`, and then does `let mut result = Vec::with_capacity(total_length);` (overflow.rs:183) — before the chain-walk loop, whose `pages_visited >= max_pages` guard is the only other bound. The comment immediately above claims safety: "the wrong-type and zero-length guards above run BEFORE this allocation, so an untrusted `total_length` (e.g. a stale handle pointing at a non-overflow page whose bytes 16..24 read as u64::MAX) can never drive a speculative giant allocation here." That reasoning only covers a page whose type byte is NOT 0x03. It does not cover a page that genuinely carries `buf[0] == PageType::Overflow` and a forged length — exactly what an attacker with byte-level control writes. + +**Why:** Craft a file where a live handle-table entry has `HandleFlags::Overflow` pointing at page N, and page N has `buf[0] = 0x03`, `buf[16..24] = u64::MAX`, with the XXH3 checksum recomputed (XXH3 is non-cryptographic — page.rs:228-230 says so explicitly). `Chisel::read(handle)` -> `read_inner` -> `Overflow::read(&mut cache, entry.page_id)` (src/transaction/read.rs:148) -> `Vec::with_capacity(usize::MAX)`. On 64-bit this either panics with "capacity overflow" or, for a large-but-representable length, calls the allocator and aborts via `handle_alloc_error`. The abort path terminates the process without going through the poison model, so a public read API on an untrusted file is a denial of service. The module's own test at overflow.rs:471-497 acknowledges this exact failure shape ("without the first-page guard, read aborts on the allocation rather than returning a clean CorruptPage") but only pins the wrong-page-type variant. + +**Fix:** Bound `total_length` against something the file can actually contain before allocating — e.g. reject any `total_length > cache.file_page_count() * OVERFLOW_PAYLOAD` as `CorruptPage` — or drop the pre-reservation entirely and let `result` grow as the chain is walked (the `pages_visited >= max_pages` guard already caps the real work). Update the comment at overflow.rs:178-182, which currently asserts a guarantee the code does not provide. + +#### `DESIGN` SECURITY-SWEEP-3 — freemap_depth is loaded from the superblock without the MAX_DEPTH validation every other radix tree applies, giving unbounded recursion and unbounded allocation on a hostile file +**src/transaction/recovery.rs:471 (exact); scan_from src/freemap_tree.rs:486, scan_node 494 (recursive call at 525); reachable_pages 546 / collect_reachable 557; cow_descend depth loop ~306** · *NEW* + +`open_existing` copies the raw field through: `freemap_depth: sb.freemap_depth` (recovery.rs:471). `Superblock::deserialize` reads it as `freemap_depth: u32::from_le_bytes(buf[FREEMAP_DEPTH_OFFSET..+4]...)` (superblock/mod.rs:538) and `validate()` (superblock/mod.rs:238-255) checks only checksum, magic, and `superblock_count` range. `FreemapRecycle::take_tree` then builds the working tree with it verbatim: `FreeMapTree::from_roots(roots.freemap_page, roots.freemap_depth)` (transaction/freemap.rs:200). Nothing anywhere compares it to `const MAX_DEPTH: u32 = 5` (freemap_tree.rs:63). This is the odd one out: `HandleTable::recover_depth` caps the walk (`if depth > MAX_DEPTH { return Err(CorruptPage) }`, handle_table.rs:448-450), `RadixU64::recover_depth` does the same (membership_index.rs:551-553), and every `MembershipIndex` entry point re-checks `if inner_root != 0 && inner_depth > MAX_DEPTH` (membership_index.rs:631, 672, 728, 755, 792). The freemap comment at freemap_tree.rs:57-63 claims the protection exists — "A spine or stored depth claiming deeper is corrupt; capacity() saturates so a bad on-disk depth fails closed (rejects the descent)" — but saturation only prevents arithmetic overflow; it rejects nothing. + +**Why:** Two distinct process-killing paths from an attacker-supplied file. (1) Stack overflow: `scan_node` recurses once per level — `self.scan_node(cache, child, level - 1, child_base, lo)` (freemap_tree.rs:525) — starting at `self.depth` (freemap_tree.rs:487). A superblock with `freemap_depth = 500_000` plus one FreeMapInterior page whose child pointer 0 points at itself makes `allocate_first` (reached by any allocation that consults the freemap) recurse 500k frames -> stack overflow -> SIGSEGV, which Rust cannot catch and which bypasses the poison model. `collect_reachable` (freemap_tree.rs:581), reached from `Chisel::defrag` via `reclaim_freemap_orphans`, has the identical shape. (2) Unbounded work and file growth: `cow_descend` loops `for level in (1..=self.depth).rev()` (freemap_tree.rs:306) and calls `self.child_span(level)` inside, which itself loops `level` times (freemap_tree.rs:184-190) — O(depth^2), roughly 10^19 operations at depth = u32::MAX — while calling `extend(cache)` to materialize a fresh page on each absent child, extending the file without bound. This fires on the ordinary commit path (`persist` -> `mark_free_committed_path`, transaction/freemap.rs:280). + +**Fix:** Validate `freemap_depth <= freemap_tree::MAX_DEPTH` in `open_existing` right beside the existing `page_size` / format-version gates, returning `CorruptSuperblock`, and correct the freemap_tree.rs:57-63 comment so it no longer claims a rejection the code does not perform. A belt-and-braces `if self.depth > MAX_DEPTH { return Err(CorruptPage) }` at the top of `scan_from` / `reachable_pages` / `cow_descend` would match the defense-in-depth style the other two radixes already use. + +#### `DESIGN` SECURITY-SWEEP-4 — The database file and the predictable .spillway sidecar are created with default umask permissions and no O_NOFOLLOW +**src/page_io.rs:138-143 and src/spillway.rs:133-139 (both exact); the spill trigger is PageCache::ensure_spillway at src/page_cache.rs:1219, called from maybe_evict at 1201 — not 1251** · *NEW* + +`PageIo::open` creates the database with `OpenOptions::new().read(true).write(true).create(true).truncate(false).open(path)` (page_io.rs:138-143) — no `OpenOptionsExt::mode`, so the file lands at 0666 & ~umask, typically 0644 (world-readable). `Spillway::open_file` does the same for the sidecar, and additionally truncates: `OpenOptions::new().read(true).write(true).create(true).truncate(true).open(&path)` (spillway.rs:133-139), where `path` is deterministically `db_path + ".spillway"` (spillway.rs:122-123). Neither call sets a restrictive mode and neither uses `O_NOFOLLOW` or an `openat`-based creation. The module doc at spillway.rs:11-13 says "Any pre-existing content is garbage from a crashed prior process and unconditionally discarded" — it does not consider the case where the pre-existing entry is a symlink planted by another local user. + +**Why:** For a plaintext database every user value is world-readable on a shared host, which is a surprising default for a storage engine that ships at-rest encryption. Worse, because the sidecar path is fully predictable from the database path, a local attacker who can create entries in the database's directory can pre-place `.spillway` as a symlink to any file the database owner can write; the first spill (which happens automatically under cache pressure, mid-transaction, from `ensure_spillway` at page_cache.rs:1251) then follows the link and truncates the victim's file to zero. The encrypted case limits the confidentiality exposure to ciphertext, but the symlink-truncation hazard is unaffected by encryption. + +**Fix:** Set an explicit mode on both creations via `std::os::unix::fs::OpenOptionsExt::mode(0o600)` (mode applies only when the file is newly created, so reopening an existing DB is unaffected), and add `custom_flags(libc::O_NOFOLLOW)` to the spillway open so a symlinked sidecar fails rather than being followed. Document the chosen mode in the `PageIo::open` doc comment so it is part of the stated contract. + +#### `DESIGN` SUPERBLOCK-RECOVERY-3 — The crypto-header `algorithm` field is written but never validated; any nonzero byte is decrypted as XChaCha20-Poly1305 +**src/superblock/crypto_header.rs:145, src/transaction/recovery.rs:339** + +`CryptoHeader::deserialize` treats every nonzero byte as "encrypted": `let algorithm = buf[CRYPTO_HEADER_OFFSET]; if algorithm == 0 { return None; }` (crypto_header.rs:146-149) — the value is stored on the struct but never compared to `ALGO_XCHACHA20POLY1305`. Grepping the whole tree, `ALGO_XCHACHA20POLY1305` appears only at its definition (crypto_header.rs:45), in the create path (recovery.rs:613), and in tests; there is no read-side comparison anywhere. `open_existing` then unconditionally constructs the one cipher it knows: `let cipher = crate::crypto::PageCipher::new(dek);` (recovery.rs:343). + +**Why:** A file written by a future binary with `algorithm = 2` (a different page cipher) opens in this binary. The DEK unwrap is a separate XChaCha20 primitive that does not depend on the page algorithm, so `unwrap_first_matching_slot` succeeds and the DB opens; every subsequent page read then feeds the wrong cipher and surfaces as `DecryptionFailed`, and — combined with SUPERBLOCK-RECOVERY-2 — the file may be writable, letting this binary write XChaCha20 pages into a file whose header says otherwise. Even in the benign case the user is told `InvalidEncryptionKey` ("your key is wrong") when the truth is "unsupported algorithm". + +**Fix:** Reject any `algorithm` other than `ALGO_XCHACHA20POLY1305` in `CryptoHeader::deserialize` (or at the open-time gate) with a distinct typed error, the same way `superblock_count` is range-checked in `validate`. + +#### `DESIGN` CRYPTO-1 — A revoked credential still decrypts the live database: the pre-revocation key-slot table survives verbatim in the sibling superblock slot, and the DEK never changes +**src/transaction/keys.rs:98-113 (write path), src/transaction/commit.rs:161-175 (same single-slot rule), src/superblock/mod.rs:410 (header written cleartext)** · *DECLINED per design spec §9 + DEFERRED per I142* + +`rewrite_crypto_header_inner` writes the new key-slot table into exactly ONE superblock slot: `let inactive = self.txn_counter % self.superblock_count as u64;` (keys.rs:98) then `cache.io_mut().write_page_unit(inactive, &unit)?;` (keys.rs:107). The other N-1 slots are never touched. The module header states the reason this is cheap: "the per-DB DEK never changes, so there is no re-encryption" (keys.rs:5). `remove_key` only does `new_header.slots[idx] = crate::superblock::KeySlot::EMPTY;` (keys.rs:216) in the in-memory copy that goes into that one slot. The key-slot table is stored in CLEARTEXT at bytes 332..1356 of every superblock page (`crypto_header.rs:13`, `serialize_into` at crypto_header.rs:133), and its wrap is authenticated only by `KeySlot::aad()`, which contains nothing tying it to a superblock generation. The public contract says the opposite: "Revoke the credential `key`. After this returns, `key` no longer opens the database" (lib.rs:984-985) and "After this returns, `old` no longer opens the database" (lib.rs:970-971). + +**Why:** Concrete sequence on a default N=2 database: create (slots 0,1 seeded, txn_counter=1) → one commit (counter 2 → slot 0) → `remove_key(old)` or `rotate_key(old,new)` (counter 3 → slot 1). Slot 1 now holds the post-revocation table; slot 0 still holds the PRE-revocation table with `old`'s wrapped_dek/wrap_tag/salt/wrap_nonce intact. `Chisel::open` picks the highest counter and correctly refuses `old`, but an adversary holding the revoked credential and mere READ access to the current file can parse slot 0's 128-byte record at offset 332, run `derive_kek` + `unwrap_dek` with `slot.aad()`, and recover the DEK — which is the SAME DEK that still seals every current data page and superblock body. No tampering, no older file image, no rollback needed. THEORY.md:164 documents "no rollback/replay resistance" for an attacker who substitutes a *wholly older image*; this is different — the stale credential lives inside the current file. The residue survives until N-1 further commits happen to overwrite every sibling slot, which for an idle database is never. + +**Fix:** Bind each key-slot's wrap to a per-header generation counter (add it to `KeySlot::aad()` and bump it on every `rewrite_crypto_header`), so a spliced-in older slot fails authentication. Independently, have `rewrite_crypto_header` overwrite the key-slot region of ALL N slots (write the surviving slots' pages with the new table before the fsync) so no stale wrapped DEK remains readable. At minimum, document in `lib.rs::remove_key`/`rotate_key` and README that revocation is not cryptographic erasure and that full re-keying requires DEK rotation. + +#### `DESIGN` SWIFT-6 — `ChiselKey` and `Options` derive `Debug`, so a `{:?}` prints the raw passphrase / key bytes the rest of the crate is careful to zeroize +**chisel-ffi/src/types.rs:71, chisel-ffi/src/types.rs:110** · *NEW* + +`#[derive(uniffi::Enum, Clone, Debug)] pub enum ChiselKey { Raw { bytes: Vec }, Passphrase { phrase: String } }` (types.rs:71-75), and `Options` — which embeds `pub encryption_key: Option` (types.rs:128) — is likewise `#[derive(uniffi::Record, Clone, Debug)]` (types.rs:110). Both derive `Debug` on the plaintext secret. This sits directly against the crate's stated key-handling discipline two lines below: `into_engine` wraps everything in `Zeroizing` (types.rs:83-89: `chisel::Key::Raw(Zeroizing::new(bytes))` / `chisel::Key::Passphrase(Zeroizing::new(phrase))`), and the root `Cargo.toml` pins `zeroize = "~1.8"` specifically to "wipe key material on drop". The `Clone` derive has the same problem in the other direction: a cloned `ChiselKey` holds a plain `Vec`/`String` that is dropped without wiping. + +**Why:** No call site formats these today, so nothing leaks right now — but the derives are a standing trap. Any future `tracing::debug!(?options)`, `assert_eq!` failure message, `#[derive(Debug)]` on an enclosing struct, or `unwrap()` on a `Result<_, Options>` prints the user's passphrase verbatim into logs or a crash report. On iOS those destinations (os_log, crash reporters) are routinely uploaded off-device. + +**Fix:** Hand-write `Debug` for `ChiselKey` (and therefore for `Options`) to print `ChiselKey::Raw { .. }` / `ChiselKey::Passphrase { .. }` with the payload redacted, and drop `Clone` from `ChiselKey` unless a call site needs it. + +#### `SMELL` SECURITY-SWEEP-7 — The cargo-audit CI job's rationale comment understates the production dependency surface by seven crates, five of them cryptographic +**.github/workflows/ci.yml:60-64 (the sentence is on lines 61-63; line 59 is blank)** + +The comment justifying the scope of the supply-chain gate reads: "Currently the root crate ships with only `xxhash-rust` and `libc` as production deps — both well-maintained — so the practical risk surface is small." The root `[dependencies]` block in Cargo.toml now lists nine: `xxhash-rust`, `libc`, `rustc-hash`, `chacha20poly1305`, `argon2`, `hkdf`, `sha2`, `zeroize` (pinned `~1.8`), and `getrandom`, plus a floor-pinned transitive `base64ct = "~1.6"`. The Cargo.toml comment on the crypto block correctly notes "they are unconditional deps (the seal/open code is always compiled)". + +**Why:** The comment is the recorded reason for how much supply-chain tooling this repo runs, and it is now false in the direction that matters: the crate's audit surface grew to include the entire RustCrypto AEAD/KDF stack, and two of those deps (`zeroize`, `base64ct`) are deliberately held below their current releases to preserve the 1.82 MSRV — pins that will age into advisory exposure and that this comment gives no reason to revisit. A reader deciding whether `cargo audit` alone is sufficient (there is no deny.toml, so there is no license or duplicate-version or ban gating) will make that call from a two-dep picture that has not been true since encryption landed. + +**Fix:** Update the comment to reflect the current nine-dependency surface and call out the two MSRV-motivated downgrade pins as items to re-check when the floor moves past 1.85. Separately worth a decision: whether `cargo deny` should join `cargo audit` now that the tree pulls a full crypto stack. + +#### `NIT` CRYPTO-8 — `open_body` strips the Zeroizing wrapper the layer below it deliberately added, so decrypted superblock plaintext is handed to the allocator unscrubbed +**src/crypto/mod.rs:382, src/crypto/mod.rs:224, src/superblock/mod.rs:456** · *NEW* + +`open_detached` documents its return type as a security measure: "The returned buffer is Zeroizing so the decrypted plaintext (key material) is wiped on drop rather than handed back to the allocator un-scrubbed" (crypto/mod.rs:224-226), and returns `Result>, CryptoError>`. `PageCipher::open_body` immediately undoes it: `open_detached(self.dek.as_bytes(), nonce, aad, ct, tag).map(|z| z.to_vec())` (crypto/mod.rs:382) — `to_vec()` allocates a fresh non-zeroizing `Vec` and copies the plaintext into it; the `Zeroizing` original is dropped and wiped, the copy is not. The consumer keeps it alive: `let body = cipher.open_body(&aad, &nonce, &tag, ct)?; self.load_body(&body);` (superblock/mod.rs:456-457), where `body` is a plain `Vec` freed unscrubbed at end of scope. + +**Why:** Every `Chisel::open` of an encrypted database leaves a 300-byte heap allocation containing the decrypted superblock body — root page pointers, `total_pages`, `next_handle`, `freemap_depth`, and the full `named_roots` table (user-chosen names, which THEORY.md:160 explicitly identifies as "real user data" and the stated reason the body is encrypted at all) — in freed, un-wiped heap memory for the remaining lifetime of the process. That is the exact failure mode the `Zeroizing` return type was introduced to prevent, defeated one call later. It also makes the `open_detached` comment false for the only variable-length caller. + +**Fix:** Change `open_body`'s signature to return `Zeroizing>` (or take a `&mut` output buffer the caller owns as `Zeroizing`) and let `decrypt_body` bind it as such — `load_body` already takes `&[u8]`, so it needs no change. If the wrapper genuinely is not wanted here, delete the claim from `open_detached`'s doc instead of leaving both. + +### unsafe (1) + +#### `DESIGN` SECURITY-SWEEP-5 — The const assertions guarding the cross-crate &[Identifier] -> &[Handle] transmute cannot detect removal of #[repr(transparent)], contrary to their own comment +**src/handle.rs:29-43 (comment 29-33, asserts 34-43); bench/src/chisel_engine.rs:98-99** + +`bench/src/chisel_engine.rs:98-99` performs a cross-crate reinterpret: `let handles: &[chisel::Handle] = unsafe { std::slice::from_raw_parts(ids.as_ptr() as *const chisel::Handle, ids.len()) };` with a SAFETY note resting on "Identifier and chisel::Handle are both #[repr(transparent)] over u64". The guard for the `Handle` side lives in src/handle.rs, whose comment claims: "Pin the layout the bench adapter's `&[u64]` -> `&[Handle]` transmute depends on... With this, dropping `#[repr(transparent)]` or adding a field fails THIS crate's build with a clear message, instead of turning the cross-crate transmute into silent UB the bench can't detect." The assertions themselves are only `assert!(core::mem::size_of::() == core::mem::size_of::())` and the matching `align_of` check (handle.rs:35-42). + +**Why:** A `#[repr(Rust)] struct Handle(u64)` — i.e. the type with `#[repr(transparent)]` deleted — still has `size_of == 8` and `align_of == 8`, so both assertions pass and the build succeeds. The stated tripwire therefore does not exist for the exact edit it names first. A maintainer who reads this comment and deletes `#[repr(transparent)]` (say, while adding a niche or a derive that seems to conflict) gets a green build in both crates while `bench/src/chisel_engine.rs:99` silently becomes UB, since `repr(Rust)` layout is explicitly unspecified and not guaranteed to match `u64`. The comment is the whole safety argument for an `unsafe` block that crosses a crate boundary, and it overstates what the code enforces. + +**Fix:** Either make the guarantee real — add `const _: () = { let _: u64 = unsafe { core::mem::transmute(Handle(0)) }; };` or a `#[deny]`-style trait bound that only `repr(transparent)` satisfies — or, simpler and lazier, delete the unsafe transmute in the bench adapter and collect into a `Vec`; the bench comment itself says the only benefit is avoiding one allocation per `delete_many` call, which is far below the fsync floor it is measuring. At minimum, correct the comment to say the asserts catch an added field but not the loss of `repr(transparent)`. + +### error-handling (6) + +#### `DESIGN` BENCH-8 — Regression diff divides by the baseline value with no zero guard — a zero baseline yields inf or NaN percentages +**bench/src/diff/compare.rs:205-218** · *KNOWN as I95* + +`let delta_pct = match metric { Metric::Throughput => (bv - pv) / bv * 100.0, Metric::P50 | Metric::P95 | Metric::P99 => (pv - bv) / bv * 100.0 };` followed by `if delta_pct > metric.threshold_pct() { Regressed { pct: delta_pct, .. } } else if delta_pct < 0.0 { Improved } else { Unchanged }`. Nothing checks `bv != 0.0`. Zero baselines are producible upstream: runner.rs:606-610 sets `throughput_ops_per_sec = 0.0` when `total_ns == 0`, and runner.rs:600-602 defaults every percentile to `percentile_linear_interp(...).unwrap_or(0.0)` for an empty op list. + +**Why:** With `bv == 0.0` and `pv > 0.0` the p-metric branch yields `+inf`, which passes `delta_pct > threshold` and is reported as `Regressed { pct: inf }`, rendering as "inf%" in the PR comment and inflating `regression_count`. With `bv == 0.0` and `pv == 0.0` the result is NaN, which fails both comparisons and is silently classified `Unchanged` — a degenerate cell is reported as healthy. + +**Fix:** Treat a zero (or non-finite) baseline as its own status alongside BaselineMissing/PrMissing rather than dividing, and assert `delta_pct.is_finite()` before classifying. + +#### `DESIGN` PUBLIC-API-6 — `begin`'s `# Errors` omits `ReadOnlyMode` and the key-management methods omit `TransactionInProgress` — the two operational errors those calls actually raise +**src/lib.rs:497, src/lib.rs:961, src/transaction/lifecycle.rs:83, src/transaction/keys.rs:40** · *NEW* + +`Chisel::begin` documents only `TransactionAlreadyActive` (src/lib.rs:497-498), but `begin_inner` checks read-only FIRST: `if self.cache.borrow().io().is_read_only() { return Err(ChiselError::ReadOnlyMode); }` (src/transaction/lifecycle.rs:83-85). `add_key`/`rotate_key`/`remove_key` document `EncryptionNotSupported`/`InvalidEncryptionKey`/`NoFreeKeySlot`/`LastKeySlot` (src/lib.rs:961-965, 975-979, 989-994) but all three route through `rewrite_crypto_header`, which returns `ChiselError::TransactionInProgress` when `self.active_txn` (src/transaction/keys.rs:40-42). + +**Why:** `ReadOnlyMode` from `begin()` is the *only* way two documented features surface at the API: `Options::read_only` (src/lib.rs:124-125 says it "only suppresses writes at the application layer" without naming an error) and the I29 minor-version write-gate, which silently calls `force_read_only()` on an otherwise-successful open (src/transaction/recovery.rs:421-423). Verified: opening an existing DB with `read_only(true)` and calling `begin()` returns `Err(ReadOnlyMode)`. A caller matching only the documented variants hits the `_ =>` arm for the single most likely error on a read-only handle. Likewise a caller who calls `add_key` inside a transaction gets an undocumented variant. + +**Fix:** Add `ReadOnlyMode` to `begin`'s `# Errors` (noting both causes: `Options::read_only` and a newer-minor file forced read-only) and `TransactionInProgress` to the three key-management methods' `# Errors`. + +#### `SMELL` PAGE-IO-8 — page_count() is infallible after I123 but still returns Result, and PageCache::new's doc explains a failure mode that cannot occur +**src/page_io.rs:521-523, src/page_cache.rs:173-175, src/page_cache.rs:187, src/page_cache.rs:667-670** + +`page_count` is now a pure Cell read — its own doc says so ("I123 (ISSUES.md, 2026-06-21): takes `&self` — pure Cell read, no syscall") and the body is `Ok(self.cached_page_count.get())`, which can never be Err. Yet `PageCache::new` still documents a recovery strategy for that impossible error: "`unwrap_or(0)` on page_count failure is a tradeoff: we'd rather construct a usable cache and surface the underlying I/O error on the next real operation than fail the constructor", implemented as `let next_page_id = io.page_count().unwrap_or(0);`. `PageCache::file_page_count` likewise propagates a `Result` that is always `Ok`. + +**Why:** Every caller of page_count and file_page_count pays a `?` or an `unwrap_or` for an unreachable branch, and the constructor doc actively misleads: a maintainer reading it believes a fallback-to-zero page count is a live risk at open time and may add defensive handling (or a test) for a state the type system already precludes. The dead `unwrap_or(0)` also silently hides a real regression if page_count is ever made fallible again. + +**Fix:** Change `PageIo::page_count` to return `u64`, drop the `unwrap_or(0)` and its doc paragraph, and let `file_page_count` return `u64` too. If the Result is being kept as a deliberate API hedge, say that explicitly the way `set_cache_max_bytes` does (src/page_cache.rs:856-860) instead of describing a failure that cannot happen. + +#### `SMELL` PAGE-IO-9 — read_page panics in release on a non-PAGE_SIZE stride while its sibling write_page returns a typed error +**src/page_io.rs:420-431 (read_page); comment at src/transaction/recovery.rs:96** + +`read_page` guards the stride with a debug-only assertion and then does an unchecked length-coupled copy: `debug_assert_eq!(self.stride, PAGE_SIZE, "read_page called on an encrypted stride; use read_page_unit"); let blob = self.read_page_unit(page_id)?; let mut buf = [0u8; PAGE_SIZE]; buf.copy_from_slice(&blob);`. In a release build the debug_assert is compiled out and `blob` is `stride` bytes, so `copy_from_slice` panics ("source slice length (8232) does not match destination slice length (8192)"). The sibling `write_page` has the same debug_assert but degrades gracefully in release, because `write_page_unit` returns `ChiselError::IoError("page unit length {} != stride {}")` (src/page_io.rs:370-376). Callers in src/transaction/recovery.rs even document the write side's behaviour as a panic ("write_page would panic (stride assert)", src/transaction/recovery.rs:98) — true only in debug. + +**Why:** Today this is unreachable because the only `read_page` caller is the pre-`set_stride` bootstrap at src/transaction/recovery.rs:254 and `page_io` is `pub(crate)` (src/lib.rs:52). But the two functions advertise identical preconditions with opposite release-mode failure modes, so a future caller that reaches `read_page` after `set_stride(ENC_PAGE_SIZE)` aborts the process on an encrypted database instead of returning an error the poison model can handle — and their tests would pass in debug, where the assert fires loudly and looks like adequate coverage. + +**Fix:** Make `read_page` return a typed error on a stride mismatch (mirror write_page_unit's length check) rather than relying on a debug-only assert plus an implicit `copy_from_slice` panic, and correct the recovery.rs comment to say "returns IoError (and trips a debug_assert)". + +#### `SMELL` PUBLIC-API-7 — A corrupt encrypted superblock body is reported as the operational `InvalidEncryptionKey`, indistinguishable from a wrong passphrase +**src/transaction/recovery.rs:348** + +After the key slot has already authenticated the DEK (`unwrap_first_matching_slot` succeeded at src/transaction/recovery.rs:342), the sealed body is opened with `sb.decrypt_body(&cipher, raw).map_err(|_| ChiselError::InvalidEncryptionKey)?`. The comment two lines above states the opposite of what the mapping says: "A tag failure here means corruption, not a wrong key (the slot already authenticated the DEK), so map to InvalidEncryptionKey rather than poisoning." + +**Why:** At this point the supplied credential is proven correct — the only way `decrypt_body` can fail is a damaged or tampered superblock body. Reporting `InvalidEncryptionKey` tells the operator "wrong passphrase or raw key" (src/error.rs:320-323) and classifies as non-fatal (`is_fatal() == false`, src/error.rs:214), so an automated caller retries credentials or falls through to its "user typo" path while the file is actually corrupt. There is no way for a caller to tell the two apart, and the failure is unrecoverable by retrying keys. + +**Fix:** Return a distinct error for a post-unwrap body failure — `DecryptionFailed { page_id }` (already fatal and already meaning "AEAD failure on data we located") or `CorruptSuperblock` — so the caller can distinguish "try another credential" from "restore from backup". + +#### `NIT` TXN-COMMIT-7 — update_inner silently no-ops on a Deleted old entry while delete_inner escalates the identical impossible state to a fatal CorruptPage +**src/transaction/mutate.rs:136, src/transaction/mutate.rs:234** · *NEW* + +`update_inner` obtains its entry via `let entry = self.lookup_live(handle)?;` (mutate.rs:38), and `lookup_live` maps a tombstone to `InvalidHandle` (read.rs:42-48, via `handle_table::lookup` returning `None` for `HandleFlags::Deleted`, handle_table.rs:214-219). The arm `HandleFlags::Deleted => OldRelease::Nothing,` (mutate.rs:136) is therefore unreachable, and it handles the impossible state by doing nothing and proceeding to install the new entry. `delete_inner` reaches the same impossible state through the same `handle_table::delete` / `ok_or(InvalidHandle)` funnel and treats it as a broken cross-module contract: `return Err(ChiselError::CorruptPage { page_id: entry.page_id });` with an eight-line I45 rationale (mutate.rs:234-246). + +**Why:** If the liveness funnel ever drifts — e.g. a future `lookup_live` variant that returns tombstones, or a direct `handle_table.lookup` call substituted for it — `delete` would poison the manager while `update` would quietly install a new HandleEntry over a tombstone and retire nothing, resurrecting a deleted handle. Two opposite responses to one invariant violation, with only one of them documented as intentional. + +**Fix:** Make `update_inner`'s Deleted arm mirror `delete_inner`'s: return the typed `CorruptPage` with the same I45-style note, or restructure both to share one `expect_live_entry` helper so the impossible-state policy lives in a single place. + +### api-design (12) + +#### `DESIGN` SUPERBLOCK-RECOVERY-2 — The I29 write-gate compares an encrypted DB's MINOR against the PLAINTEXT minor constant, so it fails open on the very next encrypted-format bump +**src/transaction/recovery.rs:421, src/page.rs:114, src/page.rs:121** · *KNOWN as I152* + +The MAJOR gate correctly branches on encryption: `let expected_major = if sb.encryption.is_some() { page::FORMAT_MAJOR_VERSION_ENCRYPTED } else { page::FORMAT_MAJOR_VERSION };` (recovery.rs:379-383). The MINOR write-gate 38 lines later does not: `if page::format_minor(sb.format_version) > page::FORMAT_MINOR_VERSION { cache.io_mut().force_read_only(); }` (recovery.rs:421). `FORMAT_MINOR_VERSION` is the plaintext series (1, page.rs:114); the encrypted series is `FORMAT_MINOR_VERSION_ENCRYPTED` (0), documented at page.rs:121-123 as "The encrypted format carries its own minor series, independent of plaintext." + +**Why:** The first encrypted-format minor bump — a file stamped (2, 1) by a newer binary — evaluates `format_minor(0x0002_0001) = 1 > FORMAT_MINOR_VERSION = 1` as FALSE. The gate does not fire, the file opens READ-WRITE, and this binary commits superblocks stamped at its own (2, 0), silently dropping every field the newer minor added. That is precisely the data-loss scenario the gate's own comment (recovery.rs:414-419) says it exists to prevent. It fails in the safe direction today only because the encrypted minor happens to be 0. + +**Fix:** Compute the expected minor from the same branch that computes `expected_major` (encrypted → `FORMAT_MINOR_VERSION_ENCRYPTED`, plaintext → `FORMAT_MINOR_VERSION`) and compare against that, so the two independent minor series are never cross-compared. + +#### `DESIGN` PUBLIC-API-8 — `Argon2Params` accepts unusable cost values and only fails deep inside `open`, misreported as `InvalidEncryptionKey` on a database being created +**src/lib.rs:270, src/crypto/mod.rs:102, src/transaction/recovery.rs:593, src/error.rs:397** · *KNOWN as I154* + +`Argon2Params` is a plain public struct with three unvalidated `u32` fields (src/crypto/mod.rs:102-107) and `Options::argon2_params` accepts any value (src/lib.rs:273-276). On create, `build_create_cipher` calls `derive_kek(key, kdf, &salt, ¶ms)?` (src/transaction/recovery.rs:593); `Params::new` rejects out-of-range costs with `CryptoError::Kdf` (src/crypto/mod.rs:185-186), and the blanket `impl From for ChiselError` maps every crypto error to `ChiselError::InvalidEncryptionKey` (src/error.rs:397-399). + +**Why:** Verified from a downstream binary: `Chisel::open(fresh_path, Options::default().encryption_key(Key::Passphrase(..)).argon2_params(Argon2Params { m_cost: 0, t_cost: 1, p_cost: 1 }))` returns `Err(InvalidEncryptionKey)`. On a *create* there is no key slot to mismatch, so the documented meaning of that variant — "key unwraps no key slot" (src/lib.rs:348-349) and "wrong passphrase or raw key" (src/error.rs:321-322) — is impossible; the actual cause (a rejected cost parameter) is nowhere in the error. The same blanket mapping also hides `CryptoError::BadKeyLength` from an empty `Key::Raw`. + +**Fix:** Validate cost parameters at the boundary — either a checked constructor (`Argon2Params::new(m,t,p) -> Result<..>`) or a range check in `Chisel::open` next to the existing `superblock_count` check — and raise a distinct operational variant (or reuse `InvalidSuperblockCount`-style typed reporting) instead of collapsing KDF-parameter failures into `InvalidEncryptionKey`. + +#### `DESIGN` PYTHON-1 — A finished Transaction object still mutates the engine — data methods bypass the `finished` guard and write into the NEXT transaction +**python/src/transaction.rs:135** + +`commit()` and `rollback()` check the one-shot guard (`if self.finished.load(Ordering::SeqCst) { return Err(already_finished_err()); }`, lines 117 and 128), but every one of the 20 data methods delegates unconditionally: `fn allocate(&self, py: Python<'_>, value: &Bound<'_, PyAny>) -> PyResult { self.db.bind(py).borrow().allocate(value) }` (135-137), and identically for update/delete/delete_many/allocate_tagged/set_client_byte/delete_tagged/delete_with_tag/set_root_name/clear_root_name/savepoint. Verified against the installed extension: `tx1 = db.transaction(); tx1.commit(); with db.transaction() as tx2: tx1.allocate(b'leaked-into-tx2')` succeeds and the value is committed by tx2. + +**Why:** The guard's stated purpose is to make 'called the wrong object' bugs visible (errors.rs:146-152, 'Pre-I22 these calls silently succeeded; returning an explicit error makes "called the wrong object" bugs visible'). For the read-only .commit()/.rollback() drive it does; for the methods that actually WRITE DATA it does not. A stale `tx` handle held past its block silently injects writes into whatever unrelated transaction happens to be open — the write commits with the other transaction's work and rolls back with it. `tx1.rollback()` afterwards raises AlreadyFinishedError, so the caller gets an error only after the data has already landed in the wrong unit of work. + +**Fix:** Check `finished` at the top of the data methods too (one shared `fn check_live(&self) -> PyResult<()>` called before each delegation), so a post-finish `tx.allocate(...)` raises AlreadyFinishedError instead of writing. Add a pytest for the stale-object case, which the suite currently has no coverage of. + +#### `DESIGN` PYTHON-3 — One-shot guard makes a savepoint name permanently unusable after rollback_to, and README documents the opposite behaviour in the same bullet list +**python/src/savepoint.rs:105** + +`rollback_to` sets `finished` before driving the engine (savepoint.rs:106-110), so after one call: a second `sp.rollback_to()` raises AlreadyFinishedError, `sp.release()` raises AlreadyFinishedError, and — because the engine kept the mark (see PYTHON-2) — `tx.savepoint(same_name)` raises DuplicateSavepointError. All three verified against the installed extension. README.md:90 says "`sp.rollback_to()` undoes changes back to the savepoint and leaves it on the stack so you can try again", and two lines later says a second `.rollback_to()` raises AlreadyFinishedError. README.md:85-86's example comments "sp still open" then "sp is released by __exit__ on normal exit" — after an explicit rollback_to the __exit__ short-circuits and never releases. + +**Why:** An engine capability the engine documents ('can be rolled back to again') is unreachable from Python, and the savepoint name is burned for the remainder of the transaction with no operation that can free it: release() is blocked by the Python guard and re-creation is blocked by the engine. A user following README:90's 'so you can try again' gets AlreadyFinishedError on the second attempt and then cannot re-mark the same name either. + +**Fix:** Decide one semantic and make code, README and comments agree: either let `rollback_to` be repeatable (do not set `finished` on rollback_to; keep it only for release), or keep the one-shot guard and rewrite README.md:85-90 to say the savepoint object is consumed and the name is retired for the transaction. + +#### `DESIGN` PYTHON-7 — All binding exception classes carry `__module__ = "_chisel"`, which is not importable — Chisel exceptions cannot be pickled +**python/src/errors.rs:83** + +Every exception is declared with `create_exception!(_chisel, X, Y)` (errors.rs:97-172), which sets `__module__` to `"_chisel"`, and `build_io_error_class` hard-codes the same value: `namespace.set_item("__module__", "_chisel")?;` (errors.rs:84). The real importable path is `chisel._chisel` — which the pyclasses get right: `#[pyclass(name = "Chisel", module = "chisel._chisel")]` (db.rs:89), likewise PyTransaction/PySavepoint/PyDrainInsertion. Verified on the installed extension: `chisel.PoisonedError.__module__ == '_chisel'` while `chisel.Chisel.__module__ == 'chisel._chisel'`, and `pickle.dumps(chisel.PoisonedError('x'))` raises `PicklingError: Can't pickle : import of module '_chisel' failed`. + +**Why:** An exception raised in a `concurrent.futures.ProcessPoolExecutor` / `multiprocessing` worker is pickled to be re-raised in the parent; with an unimportable `__module__` the pickle fails and the original ChiselError is replaced by a PicklingError, destroying the two-tier operational/fatal contract exactly where it matters (a background writer process reporting a FatalError). It also makes tracebacks and `repr()` name a module the user cannot import. + +**Fix:** Pass the full dotted path everywhere: `create_exception!(chisel._chisel, X, Y)` (or set `__module__` explicitly after creation) and change errors.rs:84 to `"chisel._chisel"`, matching the pyclass declarations. Add a test that `pickle.loads(pickle.dumps(exc))` round-trips at least one operational and one fatal class. + +#### `DESIGN` SWIFT-1 — Swift `Tag` is a bare `UInt32` with no zero-rejection, and a zero tag can never surface as a `ChiselError` +**swift/Sources/Chisel/Generated/chisel_ffi.swift:2545, swift/Sources/Chisel/ChiselDatabase.swift:152, chisel-ffi/src/types.rs:38, chisel-ffi/src/error.rs:32** · *NEW* + +UniFFI lowers the custom type to a plain alias: `public typealias Tag = UInt32` (Generated:2545), and the lowering does no validation — `public static func lower(_ value: Tag) -> UInt32 { return FfiConverterUInt32.lower(value) }` (Generated:2563-2565). The hand-written wrapper forwards unchecked: `public func allocateTagged(_ value: Data, tag: Tag) throws -> Handle { try native.allocateTagged(value: value, tag: tag) }` (ChiselDatabase.swift:152). Rejection happens only inside the Rust lift, per the comment at types.rs:38-42: "try_lift rejects 0 via the engine's real `TryFrom for Tag` (Err(ZeroTagError) on zero)". But `ZeroTagError` (src/handle.rs:118) is NOT a `chisel::ChiselError`, so `impl From for ChiselError` (error.rs:193) never sees it, and the FFI `ChiselError` enum (error.rs:32-141) has no zero-tag/invalid-tag case at all. Meanwhile swift/README.md:130 states "Every throwing call raises `ChiselError`". The Python binding, by contrast, validates explicitly: `chisel::Tag::new(tag).ok_or_else(|| pyo3::exceptions::PyValueError::new_err("tag must be non-zero"))` (python/src/db.rs:85-87). + +**Why:** `try txn.allocateTagged(Data("x".utf8), tag: 0)` compiles (Tag is UInt32) and cannot fail with any `ChiselError` — the type system makes that impossible. The call fails somewhere below the typed-error surface, so a caller writing the documented `catch let error as ChiselError { switch error.category ... }` (swift/README.md:133-147) does not match, and an opaque non-ChiselError escapes to the top of the app. Same for `handlesWithTag(0)`, `deleteTagged(h, tag: 0)`, `deleteWithTag(0, max:)`. PARITY.md:16 claims "No known gaps" against a Python surface that raises a documented, catchable error for this exact input. + +**Fix:** Validate at the Swift boundary or in `NativeDatabase`: reject tag 0 before it reaches the lift and map it onto a typed error. Either add an `InvalidTag { message }` case to the FFI `ChiselError` and check `tag == 0` in the `chisel-ffi` methods, or introduce a Swift `Tag` struct with a failable/throwing initializer instead of `typealias Tag = UInt32`. + +#### `DESIGN` SWIFT-3 — `savepoint { }` never releases the savepoint on the throw path, so the name leaks for the rest of the transaction +**swift/Sources/Chisel/ChiselDatabase.swift:184, src/transaction/savepoints.rs:103** · *NEW* + +The error path is `catch { try? native.rollbackTo(name: name); throw error }` (ChiselDatabase.swift:191-194) — `release` is called only on the success path (line 189). The engine deliberately keeps the mark after a rollback: `self.savepoints.truncate(idx + 1);` (src/transaction/savepoints.rs:103), documented at savepoints.rs:48-49 as "The named savepoint itself remains on the stack and can be rolled back to again or released." + +**Why:** Retrying a savepoint-scoped operation inside one transaction fails on the second attempt: `for _ in 0..<2 { try? txn.savepoint("attempt") { _ in throw Boom() } }` — the first call leaves "attempt" on the stack, so the second `native.savepoint(name:)` hits `savepoint_inner`'s duplicate guard (savepoints.rs:23-25) and throws `DuplicateSavepoint`, which the caller reads as a bug in its own retry loop rather than as leaked scope state. The whole stated value of the closure form (PARITY.md:42-44: "one transaction, always resolved one way or the other" as a "structural guarantee") does not hold for savepoint scopes on the failure path. + +**Fix:** Release the name after rolling back to it in the catch arm, so the scope is symmetric: `try? native.rollbackTo(name: name); try? native.release(name: name)`. Add a test that runs the same `savepoint(name)` scope twice in one transaction with the first body throwing. + +#### `DESIGN` SWIFT-4 — `Transaction` is documented as a non-escaping façade but nothing enforces it, and the `AlreadyFinished` guard reserved for exactly this is never raised +**swift/Sources/Chisel/ChiselDatabase.swift:131, swift/Sources/Chisel/ChiselStore.swift:198, chisel-ffi/src/error.rs:81** · *NEW* + +`Transaction` is a plain `public struct` holding `let native: NativeDatabase` (ChiselDatabase.swift:138-139), documented as "non-escaping façade valid only inside a `transaction { }` body" (ChiselDatabase.swift:10-11) and "not escapable (don't store this past the closure ...)" (ChiselDatabase.swift:132-134). Nothing implements that: a struct can be copied out of a non-escaping closure into a captured `var`, and `ChiselStore.transaction` widens the hole further — its body is `@escaping @Sendable (Transaction) throws -> R` (ChiselStore.swift:199). The FFI reserves a variant for the missing guard: `// Ditto: reserved for the transaction/savepoint wrapper's one-shot "finished" guard, mirroring Python's `AlreadyFinishedError`` / `AlreadyFinished { message: String }` (error.rs:81-86), echoed at Extensions.swift:69-70 ("`.AlreadyFinished` is reserved for a finished handle"). Grepping the whole binding, no code path ever constructs it — the only references are the enum declaration, the `message()`/`ffi_name` match arms, the Swift `category`/`message` arms, and `ErrorTests.swift:30`. + +**Why:** `var escaped: Transaction?; try db.transaction { txn in escaped = txn }` compiles today. Calling `escaped!.allocate(...)` later, while a DIFFERENT transaction is open, does not error — `NativeDatabase.with_db` (database.rs:43-50) just takes the lock and dispatches to whatever transaction the engine currently has open, so a write intended for the finished transaction silently joins and commits with the new one. Python guards this with a per-object `finished: AtomicBool` and raises `AlreadyFinishedError` (python/src/savepoint.rs:43, 91-111); Swift has the error case but no guard. + +**Fix:** Make `Transaction`/`Savepoint` reference types carrying a one-shot `finished` flag that `transaction`/`savepoint` sets on exit, and throw `.AlreadyFinished` from every forwarder once set — the Python shape the FFI variant was already reserved for. Either that, or delete the unreachable `AlreadyFinished` variant so it stops advertising a guard that does not exist. + +#### `DESIGN` SWIFT-5 — `ChiselDatabase(path: URL?)` accepts any URL and uses `url.path` with no file-URL check +**swift/Sources/Chisel/ChiselDatabase.swift:49, swift/Sources/Chisel/ChiselStore.swift:37** · *NEW* + +`public init(path: URL?, options: Options = Options()) throws { self.native = try NativeDatabase(path: path?.path, options: options) }` (ChiselDatabase.swift:49-51). There is no `guard path.isFileURL` and no use of `URL(fileURLWithPath:)`; `ChiselStore.init` (ChiselStore.swift:37-43) funnels through the same initializer. No Swift test ever passes a non-nil path — every test in `swift/Tests/ChiselTests/` calls `ChiselDatabase(path: nil)` / `ChiselStore(path: nil)` (DatabaseTests.swift:7,17,32; StoreTests.swift:7; ConcurrencyTests.swift:14), so this conversion has zero Swift-side coverage. + +**Why:** `URL.path` on a non-file URL silently yields the path component with the scheme and host discarded. `try ChiselDatabase(path: URL(string: "https://example.com/prod.db")!)` creates and flocks `/prod.db` on the local filesystem; `URL(string: "data.db")!` (no scheme) yields `"data.db"`, creating the database relative to the process's cwd — which on iOS is not a writable app-container location. Both succeed silently rather than failing at the boundary. + +**Fix:** Either take a `String` path, or `guard path.isFileURL else { throw ... }` at the top of `ChiselDatabase.init` and document that the URL must be a file URL. Add one Swift test that opens a file-backed database in a temp directory and reads back a committed value. + +#### `DESIGN` GAP-4 — `sb_identity_aad`'s reserved bytes invite a change that would silently make every existing encrypted database undecryptable, with no version gate and no warning +**src/superblock/mod.rs:341, src/superblock/mod.rs:333, src/transaction/recovery.rs:348** · *unverified* + +`sb_identity_aad` returns a fixed `[u8; 24]` filled with magic, format_version, txn_counter and superblock_count, ending with `// bytes 20..24 are reserved (zero) for future AAD fields.` (src/superblock/mod.rs:341-342). That array is the AAD for the sealed superblock body (`seal_body` / `open_body`, :415 and :455) and, per its own doc at :324-327, for the key-slot DEK wraps. Nothing versions it: the AAD carries no scheme byte, no branch on `format_version`, and no test pins its bytes against a stored fixture. + +**Why:** The comment reads as an extension point but is a format trap. Populating bytes 20..24 — the exact edit the comment invites — changes the AAD for every previously written superblock, so `open_body` returns `CryptoError::Auth` on every existing encrypted database. That surfaces to the user as `InvalidEncryptionKey`, i.e. indistinguishable from a wrong passphrase (src/transaction/recovery.rs:348), so an upgraded binary reports every correct passphrase as wrong rather than reporting a format break. The encrypted-format MINOR gate cannot catch it either — it compares against the plaintext minor constant (recovery.rs:421), so an encrypted-minor bump would not fire. Nothing in types, asserts, or tests enforces "the AAD layout is frozen for format MAJOR 2"; only prose that says the opposite. + +**Fix:** Either delete the "reserved for future AAD fields" invitation and state that the 24-byte layout is frozen for MAJOR=2 (a new field requires a MAJOR bump plus a rewrap pass), or add a fixture test that pins the exact 24 AAD bytes for a known superblock so any edit fails loudly at test time rather than silently at a customer's `open`. + +#### `SMELL` PUBLIC-API-9 — The public `Key` type can only be constructed through `zeroize::Zeroizing`, which the crate neither re-exports nor wraps in a constructor +**src/crypto/mod.rs:35, src/lib.rs:86, Cargo.toml:83** · *NEW* + +`pub enum Key { Raw(Zeroizing>), Passphrase(Zeroizing) }` (src/crypto/mod.rs:35-38) is re-exported as `chisel::Key` (src/lib.rs:86), but it has no constructor (`Key::raw(&[u8])` / `Key::passphrase(&str)`) and `lib.rs` has no `pub use zeroize`. The crate pins `zeroize = "~1.8"` (Cargo.toml:83); README.md:239 tells users to `use zeroize::Zeroizing;`. + +**Why:** Reproduced while writing a downstream probe: `chisel::Key::Raw(...)` fails to compile with E0433 (`use of unresolved module or unlinked crate zeroize`) until the consumer adds their own `zeroize` dependency whose resolved version must unify with chisel's tilde pin. The entire encryption API — `Options::encryption_key`, `add_key`, `rotate_key`, `remove_key` — is unreachable without that extra, undeclared-in-the-API dependency, and a future bump of chisel's `zeroize` major would break every caller's construction site without any change to chisel's own signatures. + +**Fix:** Add inherent constructors (`Key::raw(impl Into>)`, `Key::passphrase(impl Into)`) that wrap in `Zeroizing` internally, so the third-party type never appears in a construction path; optionally `pub use zeroize::Zeroizing` as well for callers who already hold one. + +#### `NIT` FREEMAP-6 — cow_descend enforces its in-range precondition only in prose; find_leaf's bounds guard has no counterpart on the write path +**src/freemap_tree.rs:308, src/freemap_tree.rs:360, src/freemap_tree.rs:218** · *NEW* + +The read path guards the child slot index: "Defense-in-depth: a validated depth keeps child_idx in bounds, but guard the slot index so a corrupt one reads as \"absent\" rather than slicing past the page. if child_idx >= PTRS_PER_INTERIOR { return Ok(None); }" (src/freemap_tree.rs:216-220), and rejects out-of-reach ids up front via `if cap != u64::MAX && id >= cap` (src/freemap_tree.rs:201). The COW write path has neither. `cow_descend` computes `let child_idx = (remaining / span) as usize;` (src/freemap_tree.rs:308) and passes it straight to `read_child` / `write_child`, which index `DATA_PAGE_HEADER_SIZE + index * PTR_SIZE` with no bound (src/freemap_tree.rs:66, :71). At depth 0 the loop never runs and the leaf op is applied to `remaining % LEAF_CAPACITY` (src/freemap_tree.rs:360), silently aliasing an out-of-range id onto some other page's bit. The only protection is prose: "Assumes `id` is in range (callers grow first if needed)" (src/freemap_tree.rs:279-280) and "Assumes `id` is in range; use `mark_free_growing` when it may exceed capacity" (src/freemap_tree.rs:407-408). + +**Why:** Not reachable today: the only production entry is `mark_free_growing`, which grows first (src/freemap_tree.rs:454-457), and `clear_bit` is only called by `allocate_first` on an id `scan_from` just proved free. But `mark_free` is `pub` on a `pub(crate)` type, so one new caller that forgets `_growing` gets, at depth 0, a silently corrupted bitmap (id 65344+7 clears/sets the bit for page 7 — a live page marked free, i.e. a double-hand-out), and at depth 1 with id >= LEAF_CAPACITY*1021 a `child_idx` of 1021+ that makes `write_child` slice past the 8192-byte buffer and panic. The invariant that keeps this safe lives only in a doc comment, while the identical invariant on the read path is enforced in code. + +**Fix:** Either make `mark_free` private and expose only `mark_free_growing`, or have `cow_descend` return `CorruptPage`/a typed error when `id >= self.capacity()` or `child_idx >= PTRS_PER_INTERIOR`, mirroring `find_leaf`'s guard. + +### ownership (1) + +#### `SMELL` TXN-COMMIT-8 — reclaim_freemap_orphans documents an active-transaction requirement it never enforces, while mutating current_roots and the structural supersede stream +**src/transaction/freemap.rs:362 (doc is on FreemapRecycle::reclaim_orphans) / src/transaction/freemap.rs:663 (unguarded TM wrapper reclaim_freemap_orphans)** + +freemap.rs:362-363: "Requires an active transaction (called by defrag)." `pub(crate) fn reclaim_freemap_orphans(&mut self)` (freemap.rs:663-683) checks only `savepoints.is_empty()` and the poison flag; there is no `if !self.active_txn { return Err(NoActiveTransaction) }`. It passes `&mut self.current_roots` into `reclaim_orphans`, which calls `mark_free_committed_path` — COWing committed freemap pages, pushing the superseded (still committed-LIVE) ids into `structural_superseded` via `put_tree` (freemap.rs:210-216, :472-474), and advancing `current_roots.freemap_page`. The sole caller does guard: `if !txm.is_active() { return Err(ChiselError::NoActiveTransaction); }` (defrag.rs:184-186). + +**Why:** The prose invariant is the only thing standing between this function and a durable-freemap hazard, and it is enforced in a different module. Called with no transaction, `current_roots` diverges from `committed_roots` with no commit path to promote it, and `structural_superseded` fills with pages the last-durable superblock still references — the exact condition `reclaim_orphans`' own doc (freemap.rs:368-377) says would produce "silent durable freemap corruption". Today it is defused only because `begin()` resets `current_roots` (lifecycle.rs:89) and `FreemapRecycle::begin` clears `structural_superseded` (freemap.rs:492); a second caller, or any reordering of those two lines, removes the accident that makes it safe. + +**Fix:** Move the active-transaction check into `reclaim_freemap_orphans` itself, alongside the existing savepoint gate, and drop the now-redundant guard in `defrag` (or keep it for the better error message). + +### performance (2) + +#### `DESIGN` HANDLES-INDEX-2 — Neither radix has the freemap's session-COW dedup: every mutation re-COWs the whole root-to-leaf path, so one transaction's file high-water mark grows linearly with its mutation count +**src/handle_table.rs:552, src/handle_table.rs:567, src/membership_index.rs:262, src/transaction/freemap.rs:48** · *NEW* + +`HandleTable::insert_recursive` opens unconditionally with `let new_page = alloc(cache)?;` (handle_table.rs:552) and then `freed.push(page_id);` (:567); `RadixU64::insert_recursive` does the same at membership_index.rs:262-266. There is no "already COW'd this transaction" check at either site. The allocator behind them, `cow_alloc` (src/transaction/freemap.rs:48-67), reuses only pages that are free in the *committed* freemap bitmap (`tree.allocate_first`) and otherwise falls through to `cache.new_page()`; pages this transaction superseded sit in `txn_freed_pages` and do not reach the bitmap until commit. The freemap tree solves exactly this and documents why: src/freemap_tree.rs:100-112 — "SUBSEQUENT mutations that re-touch a page THIS handle already created mutate it IN PLACE ... Without this, `persist_freemap`'s loop over many frees ... would re-COW the leaf once per id, extending and superseding a fresh page each time and turning the committed-free reclamation into unbounded file growth." Neither handle_table.rs nor membership_index.rs has an equivalent of `session_owned`. + +**Why:** N allocations in a single transaction on a depth-d handle table allocate N*(d+1) pages and queue N*(d+1) supersedes; a tagged allocate adds the outer and inner membership spines on top (roughly 4-6 fresh pages per tagged insert at shallow depth). Once the committed free pool is exhausted every one of those extends the file, and none is reclaimable until commit. A 1M-row bulk load in one transaction therefore grows the file by millions of pages that are all dirtied, all flushed by `cache.flush()` in commit phase 1, and only then marked free — the exact "unbounded file growth" the freemap comment names as the reason its dedup exists. The handle_table.rs:236 promise ("lets the handle table reach a bounded steady-state page count") is true only across transactions and reads as if it also bounded within-transaction churn. + +**Fix:** Give both radices the same treatment as `FreeMapTree.session_owned`: thread a per-transaction set of page ids this transaction already materialized/COW'd into `insert_recursive`/`delete_recursive`, and mutate in place (no alloc, no `freed.push`) on a re-touch. The manager already owns and swaps such a set for the freemap, so the plumbing pattern exists. At minimum, correct handle_table.rs:236 to say the bound is across commits, not within one. + +#### `DESIGN` BENCH-4 — Engine teardown (SQLite's close-time WAL checkpoint, tempfile unlink) runs inside every snapshot-restore and cold-read timed region +**bench/benches/micro_grid.rs:94-103, bench/benches/micro_grid.rs:146-155, bench/tests/runner_smoke.rs:47-55** · *NEW* + +The timed routines take the setup tuple by value and return `()`: `|(mut engine, _working)| { drive_workload_with_tx_granularity(&mut *engine, workload, ops_per_tx, snapshot_ids); }`. Criterion 0.5's `iter_batched` with `BatchSize::PerIteration` runs `let start = self.measurement.start(); let output = routine(input); let end = self.measurement.end(start);` (criterion-0.5.1/src/bencher.rs:247-257), so anything the routine drops is dropped between start and end — here the `Box` and the `NamedTempFile`. + +**Why:** For sqlite-strict/unsafe, closing the last connection triggers an automatic WAL checkpoint of everything the iteration wrote — for `allocate-1000pertx` that is up to 8 MB of frames checkpointed inside the measured span, charged to SQLite alone. For every mode it also includes a `NamedTempFile` unlink of a file up to ~25 MB, whose cost varies by filesystem. None of that is the operation the row is named after, and the contamination is asymmetric across engines. + +**Fix:** Return the tuple from the routine (`|(mut engine, working)| { drive(...); (engine, working) }`) so Criterion drops it after `measurement.end`, or switch to `iter_batched_ref` where the input outlives the timed span. + +### tests (7) + +#### `DESIGN` TESTS-CI-5 — No test covers whether a key revoked by rotate_key/remove_key stays revoked after a torn superblock slot +**src/transaction/keys.rs:98, src/transaction/keys.rs:107, tests/encryption_keys.rs:96, tests/encryption_open.rs:286** · *KNOWN as a 2026-07-02 open question* + +`rewrite_crypto_header_inner` writes the new key-slot table into exactly one slot: `let inactive = self.txn_counter % self.superblock_count as u64;` (keys.rs:98) followed by a single `cache.io_mut().write_page_unit(inactive, &unit)?` (keys.rs:107). The other N-1 slots keep the PREVIOUS key-slot table, which still lists the revoked credential as active and still wraps the same unchanged DEK (keys.rs:88 — "the DEK inside `cipher` is unchanged"). tests/encryption_keys.rs `rotate_key_revokes_old_and_admits_new` and `remove_key_leaves_others_working` only assert revocation via a clean reopen; no test corrupts the winning slot after a rotation. Meanwhile tests/encryption_open.rs:286 `torn_slot_0_encrypted_db_recovers_via_sibling` proves the engine will happily fall back to a stale sibling slot. + +**Why:** Zero slot 0 after `rotate_key(old, new)` on an N=2 database (exactly the byte pattern that test already writes at encryption_open.rs:310-314) and `Superblock::select` picks the surviving slot, whose counter is one lower and whose crypto header still has the OLD key active. The old key then unwraps the same DEK and opens the database with full read access. That is a revocation that a single torn write — or an attacker with write access to 8 KB of the file — silently undoes, and nothing in the suite would notice the day someone changes slot-selection or header-write behavior. + +**Fix:** Add a test that rotates a key, zeroes the winning slot, and asserts the OLD key is still refused (or, if fallback-admits-old-key is the accepted design, assert that outcome explicitly so the exposure is documented rather than latent). The stronger implementation fix is to write the new crypto header to all N slots before returning from rotate_key/remove_key. + +#### `DESIGN` SWIFT-7 — The Swift test suite never exercises close, poison, file-backed open, encryption, or key management — only the in-memory happy path +**swift/Tests/ChiselTests/DatabaseTests.swift:1, swift/Tests/ChiselTests/StoreTests.swift:1** · *NEW* + +The whole Swift suite is 6 files / ~156 lines. It covers: `libraryVersion()` non-empty (SmokeTests.swift:6), commit and rollback-on-throw in memory (DatabaseTests.swift:6-27), stats/counters/defrag reachability (DatabaseTests.swift:31-39), one async round-trip (StoreTests.swift:6-13), 100-task serialization (ConcurrencyTests.swift:13-33), pure-enum error classification with hand-constructed values (ErrorTests.swift), and one Keychain lookup miss (KeychainTests.swift:12-19). Nothing calls `ChiselDatabase.close()` / `ChiselStore.close()`, `isPoisoned`, `addKey`/`rotateKey`/`removeKey`, `setCacheMaxBytes`/`setSpillwayMaxBytes`/`setDrainInsertion`, `savepoint { }`, or opens with a non-nil path or an `encryptionKey`. Note the equivalents DO exist on the Rust side (`use_after_close_is_closed_error`, `double_close_is_closed_error`, `savepoint_rollback_to`, `encrypted_round_trip_wrong_key_rejected` at database.rs:291-374) — it is precisely the hand-written Swift wrapper layer that is untested. + +**Why:** The bugs this dimension is most exposed to all live in the untested layer: the savepoint scope leak (SWIFT-3), the `URL.path` conversion (SWIFT-5), and the `isPoisoned`-after-close semantics (SWIFT-8) are each one Swift test away from being caught and are all currently invisible to `swift test`, which swift.yml:54-56 labels the "Hard gate". + +**Fix:** Add Swift tests for: open/commit/reopen against a temp-directory file URL; `close()` then a second `close()` expecting `.Closed`; `isPoisoned` after `close()`; a passphrase-encrypted round trip plus a wrong-key reopen; and a `savepoint { }` scope whose body throws. + +#### `SMELL` TESTS-CI-2 — counters_snapshot_does_not_mutate_when_engine_continues is a tautology that can never fail +**tests/counters.rs:108-124** · *NEW* + +The test does `let snap = db.counters().unwrap(); let snap_copy = snap.clone();`, performs a commit, then `assert_eq!(snap, snap_copy, "ChiselCounters is a snapshot, not a live view")`. `ChiselCounters` (src/stats.rs:81-88) is `#[derive(Debug, Clone, Default, PartialEq, Eq)]` over four plain `u64` fields — no `Cell`, no reference, no interior mutability. + +**Why:** `snap_copy` is a bitwise copy of `snap` made before any work; nothing in the language could make two owned `u64`-only structs diverge. If `counters()` were changed tomorrow to return a live view (e.g. `Rc>` fields, or a borrow of the engine's counters), this test would still pass, because the test only compares the snapshot against a clone of itself rather than against a freshly-read `db.counters()`. The test occupies the slot reserved for the snapshot contract while verifying nothing. + +**Fix:** Compare the pre-work snapshot against the value of the field it should differ from: take `snap` before, do the commit, then assert `snap.fsync_calls < db.counters().unwrap().fsync_calls` AND `snap.fsync_calls == snap_copy.fsync_calls`. The second half is only meaningful once the first half proves the engine actually moved. + +#### `SMELL` TESTS-CI-9 — The named-root suite is duplicated between src/recovery_tests.rs and tests/error_and_format.rs, contradicting recovery_tests.rs's own stated reason for existing +**src/recovery_tests.rs:867-1035 — five of the six duplicate, not six** · *NEW* + +src/recovery_tests.rs:1-7 states the file "Lives in src/ rather than tests/ because the I35 pub→pub(crate) reshape locks these internals down; integration tests no longer have access." But `test_named_roots_survive_commit_and_reopen` (:868), `test_named_roots_revert_on_rollback` (:893), `test_named_roots_revert_on_rollback_to_savepoint` (:918), `test_named_roots_validation_errors` (:945), `test_named_roots_table_full` (:981) and `test_named_roots_clear_is_idempotent` (:1023) touch nothing but `Chisel`, `ChiselError` and `crate::Handle` — all public. Every one has a near-identical twin in tests/error_and_format.rs:141-292 (`test_named_root_empty_name_rejected`, `_too_long_rejected`, `_max_length_accepted`, `_embedded_nul_rejected`, `_table_full`, `_clear_missing_name_is_noop`, `_rollback_reverts_set`, `test_named_root_survives_commit_and_reopen`). + +**Why:** Two suites assert the same contract with different names and different hardcoded bounds — recovery_tests.rs:993 hardcodes the literal `8` (with a comment warning it must move if NAMED_ROOT_COUNT changes) while tests/error_and_format.rs:214 uses the exported `NAMED_ROOT_COUNT`. A change to NAMED_ROOT_COUNT silently stops testing the boundary in one file while the other keeps working, so the duplicate provides false reassurance rather than redundancy. It also stretches recovery_tests.rs to 1652 lines of "crash recovery" that is substantially not about crash recovery. + +**Fix:** Delete the six named-root tests from src/recovery_tests.rs; the public-API versions in tests/error_and_format.rs already cover them and use the exported constant. Keep recovery_tests.rs to the tests that genuinely need `Superblock`, `PageType`, and `page::stamp_checksum`. + +#### `SMELL` TESTS-CI-11 — test_defrag_respects_max_values asserts values_moved <= 2, so a defrag that moves nothing passes +**tests/defrag.rs:218, tests/defrag.rs:220** + +After deleting 48 of 50 values, the test runs `defrag(DefragOptions::default().sparse_threshold(0.25).max_values(2))` and asserts only `result.values_moved <= 2` with the message "max_values=2 should cap values_moved to 2, got {}". The two surviving handles are then read back, which passes whether or not defrag relocated anything. + +**Why:** The message states the expectation is 2, but the assertion accepts 0, 1 or 2. A regression in `max_values` handling that turns the cap into "stop before the first move" (an off-by-one on the budget check — `remaining == 0` evaluated before rather than after the decrement) makes `values_moved == 0` and this test still passes. The sibling test `test_defrag_skips_dense_pages` (tests/defrag.rs:124-129) asserts `values_moved == 0` for the do-nothing case, so the suite cannot distinguish "cap respected" from "cap broke defrag entirely". + +**Fix:** Assert `values_moved == 2` — the setup is deterministic (50 values × 200 bytes, 48 deleted, threshold 0.25), exactly as the sibling `test_defrag_reclaims_space_after_deletes` already asserts exact counts at tests/defrag.rs:66-74. + +#### `SMELL` PYTHON-11 — Nothing in the Python suite would fail if `drain_insertion` were dropped on the floor +**python/tests/test_runtime_config.py:51** + +`test_open_with_drain_insertion_lru_tail` and `..._mru` (tests/test_open.py:111-124) do `with chisel.open(str(tmp_db), drain_insertion=...) as db: assert db is not None`; `test_open_in_memory_with_spillway_and_drain` (:127-134) is the same shape; `test_set_drain_insertion_accepts_both_variants` (tests/test_runtime_config.py:66-72) only checks that a following allocate does not raise. Deleting `.drain_insertion(drain_insertion.into())` from the builder chain (db.rs:265) or making `set_drain_insertion` (db.rs:601-603) a no-op keeps every one of them green. The sibling kwargs are not in this state: `read_only` got an end-to-end assertion for exactly this reason (tests/test_open.py:43-57, 'a dropped/inverted read_only kwarg ... would be caught'), and `spillway_max_bytes` is pinned by `stats().spillway_max_bytes == spillway_max` (tests/test_stats_defrag.py:31-45). + +**Why:** The kwarg is plumbed through two conversion layers (`PyDrainInsertion` -> `From` impl -> `Options`/`set_drain_insertion`); a transposed match arm in `impl From for chisel::DrainInsertion` (db.rs:141-148) would silently invert the cache-drain policy for every Python user with no test failing anywhere in the suite. + +**Fix:** Assert an observable difference — e.g. drive a spillway overflow under each policy and compare `counters().cache_hits` on a re-read of the drained pages — or, at minimum, expose the effective policy on `stats()` and assert the round-trip. + +#### `NIT` TESTS-CI-1 — error_and_format.rs claims exhaustive ChiselError coverage but tests 19 of 30 variants; the "compile error" safety net it describes does not exist +**tests/error_and_format.rs:7-8 and :81-83 (comments only); the real guard is src/error.rs:524-637** · *NEW* + +The file header asserts "is_fatal() classification of every ChiselError variant" and "Display output is non-empty for every variant", and the Display test's body says "We enumerate every variant explicitly so a newly added variant forces a compile error in the match below, reminding the author to add a Display arm." There is no `match` in that test — it is a `let variants: Vec = vec![...]` literal, which a new variant never breaks. `ChiselError` has 30 variants; the Display list carries 19. Uncovered: CacheFull, SpillwayFull, TransactionInProgress, TagMismatch, UnsupportedPageSize, NoEncryptionKey, InvalidEncryptionKey, EncryptionNotSupported, NoFreeKeySlot, LastKeySlot, DecryptionFailed. `test_is_fatal_storage_integrity_variants_are_fatal` omits `UnsupportedPageSize` and `DecryptionFailed`, both of which `is_fatal` (src/error.rs:225-226) classifies as fatal; the non-fatal test omits CacheFull, SpillwayFull, TransactionInProgress, TagMismatch and every encryption variant. + +**Why:** Concretely: drop `ChiselError::DecryptionFailed { .. }` from the `matches!` in src/error.rs:214-227 and the whole suite still passes. A page whose AEAD tag fails under the correct session DEK — i.e. on-disk tampering — would then be classified operational, so `TransactionManager` would not poison and the caller would keep issuing reads against a store known to be corrupt. The same silent hole exists for `UnsupportedPageSize`. The header comment is worse than no comment: a maintainer adding a variant reads "forces a compile error" and does not add a case. + +**Fix:** Make the enumeration real: build the variant list inside a function that takes `&ChiselError` and `match`es exhaustively (an `#[allow(unreachable_patterns)]`-free match on the crate-internal type, or a `fn all_variants() -> Vec` fed by a match on a marker enum), and add the 11 missing variants plus the two missing is_fatal classifications. Failing that, delete the "forces a compile error" claim so nobody relies on it. + +### ci (7) + +#### `DESIGN` BENCH-9 — The bench workflow has no continue-on-error anywhere despite documenting itself as non-blocking and "fail gracefully" on fork PRs +**.github/workflows/bench.yml:3-12, .github/workflows/bench.yml:100-106** · *NEW* + +The header states "Never blocks merge — signal, not gate" and, for fork PRs, "`${{ secrets.GITHUB_TOKEN }}` is read-only for fork PRs and the comment-post step will fail gracefully." No step in the file carries `continue-on-error: true`, and the "Create or update PR comment" step (`peter-evans/create-or-update-comment@v5`) is a plain required step. The same claim is baked into the report body itself (bench/src/diff/render.rs:262 "Never blocks merge — signal, not gate"). + +**Why:** On a fork PR the comment step gets HTTP 403 (the `permissions: pull-requests: write` block cannot grant write to a fork's read-only token), the step fails, and the whole Bench job goes red — the opposite of "fail gracefully". The same happens on any harness flake, e.g. the intermittent documented at bench/src/sqlite_engine.rs:245-252 ("the bench harness still failed ~1 in 5 full --quick runs"), turning a report-only benchmark into a red check on unrelated PRs. + +**Fix:** Mark the benchmark, diff, and comment steps `continue-on-error: true` (and/or guard the comment step with `if: github.event.pull_request.head.repo.full_name == github.repository`) so the documented "signal, not gate" property is enforced by the workflow rather than only asserted in a comment. + +#### `DESIGN` TESTS-CI-6 — The Rust test/clippy/fmt jobs run only on ubuntu-latest, yet tests encode macOS-specific behavior that CI never exercises +**.github/workflows/ci.yml:14, .github/workflows/ci.yml:29, .github/workflows/ci.yml:49, tests/options_validation.rs:286** · *NEW* + +`test`, `clippy`, and `fmt` are all `runs-on: ubuntu-latest` with no matrix (ci.yml:14, 29, 49). The only multi-OS job is `python` (ci.yml:135, `os: [ubuntu-latest, macos-latest]`), which builds the PyO3 wheel and runs `pytest` — it never runs `cargo test`. tests/options_validation.rs:285-288 documents platform divergence the suite deliberately works around: "Pre-create with size > 0 so the \"zero-length => create_new\" branch doesn't interfere with the lock test on macOS, which sometimes reports ENOENT before the lock is attempted on a missing path." + +**Why:** The engine's platform-sensitive surface is exactly the part CI does not cover: `libc::flock` (src/page_io.rs:263), `File::sync_all` fsync semantics, and the open/create ordering the comment above calls out as differing on macOS. A regression in `PageIo::open`'s create/lock ordering that only misbehaves on macOS ships green, and would only surface through the Python job's indirect coverage — which does not assert `LockFailed`, `FileNotFound`, or the zero-length-file path at all. The project ships macOS wheels (wheels.yml:52) and a macOS/iOS Swift binding, so macOS is a first-class target. + +**Fix:** Add `strategy: matrix: os: [ubuntu-latest, macos-latest]` to the `test` job (clippy/fmt can stay single-platform; they are host-independent). + +#### `DESIGN` TESTS-CI-7 — bench.yml documents the fork-PR comment step as failing "gracefully", but it has no continue-on-error and will turn the whole Bench job red +**.github/workflows/bench.yml:8, .github/workflows/bench.yml:4, .github/workflows/bench.yml:100** · *NEW* + +The header comment says: "`${{ secrets.GITHUB_TOKEN }}` is read-only for fork PRs and the comment-post step will fail gracefully" (bench.yml:6-11), under a banner reading "Never blocks merge — signal, not gate" (bench.yml:4). The step in question, `Create or update PR comment` using `peter-evans/create-or-update-comment@v5` (bench.yml:100-106), carries no `continue-on-error:` and no `if:` guard, unlike the artifact upload immediately below it which does set `if: always()` (bench.yml:123) precisely because earlier steps can fail. + +**Why:** On a fork PR the token cannot write issue comments regardless of the job's `permissions: pull-requests: write` (bench.yml:29); the action fails the step with "Resource not accessible by integration", the step failure fails the job, and the Bench check goes red on every fork contribution. That is not "graceful" — and if Bench is ever added to the branch's required checks, the documented "never blocks merge" property is false. The `if: always()` on the next step shows the author expected step failures but did not neutralize the one they described as harmless. + +**Fix:** Add `continue-on-error: true` to the find-comment and create-or-update-comment steps (or gate them on `github.event.pull_request.head.repo.full_name == github.repository`), so the comparison still runs and uploads artifacts on fork PRs without reddening the check. + +#### `DESIGN` SWIFT-2 — 974 MB of Swift build artifacts sit untracked AND unignored on `main`, because the branch's `.gitignore` rules were never merged +**.gitignore:1, swift/** · *NEW* + +`git status --short` on `main` reports exactly one entry: `?? swift/`. `du -sh swift/` is 974M, containing `swift/.build/`, `swift/.xcbuild/`, `swift/Chisel.xcframework/` (three static libraries: `libchisel_ffi-ios.a`, `libchisel_ffi-ios-sim.a`, `libchisel_ffi-macos.a`), plus loose `ChiselStore.o`, `ChiselDatabase.o`, `Keychain.o`, `*.d`, `*.swiftdeps` at the top level — and NO sources. `main`'s `.gitignore` is 10 lines (`.superpowers/`, `.worktrees/`, `/target`, `.keys`) with no swift entry. The branch `design/swift-binding` DOES carry the rules (`git show design/swift-binding:.gitignore` adds `swift/Chisel.xcframework/`, `swift/.build/`, `swift/.xcbuild/`, `swift/*.o`, `swift/*.d`, `swift/*.swiftdeps`), but that branch is unmerged, so on `main` none of them apply. + +**Why:** On `main` today, `git add -A` (or any IDE "stage all") commits ~974 MB of binary build output, including three prebuilt static archives, into git history — effectively unrecoverable without a history rewrite. Separately, the entire Swift/UniFFI binding (chisel-ffi/ + swift/Sources + swift/Tests, ~2000 hand-written lines) exists only on an unmerged branch, so the only thing on `main` is the artifacts. + +**Fix:** Land the branch's `.gitignore` additions on `main` independently of the binding itself, and delete the stale `swift/*.o` / `*.d` / `*.swiftdeps` files from the working tree (they are stray top-level SwiftPM outputs, not part of `.build/`). + +#### `SMELL` TESTS-CI-3 — ci.yml's audit job carries a 14-line "Permissions notes" comment describing a permissions: block and an annotation-posting action that the job no longer has +**.github/workflows/ci.yml:65-72 (comment) vs :77-84 (audit job)** + +Lines 66-72 read: "# Permissions notes: # * `pull-requests: write` lets the action post per-line annotations on PR runs. # * `checks: write` is needed on push-to-main runs — without it the action's annotation-posting step trips \"Resource not accessible by integration\"...". The `audit:` job that follows (lines 80-89) declares **no** `permissions:` key at all, and lines 85-86 record that `rustsec/audit-check` was replaced by a bare `run: cargo audit`, so there is no action and no annotation-posting step left to need those scopes. + +**Why:** A maintainer tightening token scope reads this block, believes `audit` depends on `pull-requests: write` / `checks: write`, and either grants them repo-wide (a real over-permission on a job that compiles and runs `cargo install cargo-audit --locked`, i.e. arbitrary third-party build scripts) or spends time chasing a permission failure that cannot occur. It also masks the fact that neither ci.yml nor wheels.yml declares any `permissions:` block, so every job silently inherits the repository default token scope. + +**Fix:** Delete the stale Permissions notes block, and add an explicit top-level `permissions: contents: read` to ci.yml and wheels.yml (bench.yml already scopes per-job), overriding only where a job genuinely needs more. + +#### `SMELL` GAP-3 — The release workflow ties nothing to the tag it fires on: no version check across the three manifests, and the sdist is uploaded without ever being built from +**.github/workflows/wheels.yml:5, .github/workflows/wheels.yml:100, .github/workflows/wheels.yml:115, python/pyproject.toml:7, Cargo.toml:36** · *unverified* + +`wheels.yml` fires on `tags: ["v*"]` (:5). The gate job runs `cargo test --release` and `cargo audit` (:24, :32); the wheel job runs `CIBW_TEST_COMMAND: pytest {package}/tests` against each built wheel. No step compares the tag to the versions in `Cargo.toml:36` (`0.1.0`), `python/Cargo.toml` (`0.1.0`), or `python/pyproject.toml:7` (`0.1.0`). Separately, the `sdist` job (:100-120) runs `maturin sdist -o ../dist` and uploads the tarball (:117-120) with no step that installs or compiles it. + +**Why:** Tagging `v0.2.0` today produces and uploads wheels and an sdist whose internal version is `0.1.0` — silently mislabeled release artifacts, with no CHANGELOG in the repo to catch the discrepancy by eye. And the sdist is the one artifact whose build is genuinely fragile here: `python/Cargo.toml` has `chisel = { path = ".." }`, a path dependency that resolves outside the sdist's own directory, so an sdist that fails to compile at `pip install` time on a user's machine ships green because CI never attempts it (the wheel job's `CIBW_TEST_COMMAND` exercises wheels only). + +**Fix:** Add a step to the `cargo-test-gate` job asserting `${GITHUB_REF_NAME#v}` equals the version in all three manifests, and add a verification step to the `sdist` job that does `pip install dist/*.tar.gz && pytest tests` in a clean venv before upload. + +#### `NIT` TESTS-CI-8 — No CI job measures test coverage, so the suite's blind spots are invisible +**.github/workflows/ci.yml:12, .github/workflows/ci.yml:129** · *NEW* + +The complete job inventory across all three workflows is: ci.yml — `test` (cargo build + cargo test, debug, ubuntu), `clippy` (`cargo clippy --workspace -- -D warnings`), `fmt` (`cargo fmt -- --check`), `audit` (`cargo audit`), `msrv` (`cargo build -p chisel` on 1.82), `python` (maturin + pytest, 2 OS × 2 Python); bench.yml — `bench` (report-only PR comment); wheels.yml — `cargo-test-gate` (`cargo test --release` + `cargo audit`), `wheels`, `sdist`. None of them runs `cargo llvm-cov`, `cargo tarpaulin`, or any equivalent, and no coverage artifact or badge is produced. + +**Why:** The gaps this review found are exactly the kind a coverage run surfaces mechanically: 11 ChiselError variants with no Display test (TESTS-CI-1), the `Fault::FailReadPage` arm at src/page_io.rs:290 exercised only from one unit test, and every `#[cfg(test)]` fault path that integration tests cannot reach. Without a coverage number, "is this path tested?" is answered by grep and hope, and a refactor that deletes the last caller of a branch produces no signal. + +**Fix:** Add a non-gating `coverage` job running `cargo llvm-cov --workspace --lcov` and uploading the report as an artifact (report-only, same posture as bench). Gate later, once a baseline exists. + +### docs-vs-reality (34) + +#### `DESIGN` HANDLES-INDEX-1 — "Handles are never reused within a database's lifetime" is false: next_handle rewinds on rollback and on crash recovery, and there is no generation/epoch to detect it +**src/handle_table.rs:48, src/lib.rs:581, ARCHITECTURE.md:531, src/transaction/staging.rs:234, src/transaction/lifecycle.rs:269** · *NEW* + +Three independent places state an absolute no-reuse guarantee. `src/handle_table.rs:48`: "handles are monotonic from `next_handle` and never reused, starting at 1". `src/lib.rs:581`: "they are never reused within a database's lifetime and are stable". `ARCHITECTURE.md:531`: "never reused within a database's lifetime, even after delete ... This permanent-burn policy makes handles safe to embed in long-lived references without a stale handle later pointing at unrelated data". But `next_handle` is a field of `Roots` (`src/transaction/mod.rs:104`), minted at `src/transaction/staging.rs:234` (`let handle = self.current_roots.next_handle;`) and bumped at `staging.rs:325`, and every rewind path restores it wholesale: `lifecycle.rs:269` `self.current_roots = self.committed_roots.clone();`, `savepoints.rs:77` `self.current_roots = self.savepoints[idx].roots.clone();`, and `recovery.rs:472` `next_handle: sb.next_handle` on open. `HandleEntry` (handle_table.rs:134-147) carries page_id/slot_index/flags/tag/client_byte — no generation or epoch field — so a re-minted id is indistinguishable from the original. The repo's own test asserts the reuse: `src/recovery_tests.rs:353-356` `assert_eq!(hc, hb, "next_handle must have rewound to state A on recovery")`, with the comment at :345 "next_handle rewound to A, so the new handle reuses B's id". + +**Why:** `begin(); let h = allocate(A)?; rollback(); begin(); let h2 = allocate(B)?;` yields `h2 == h`. A caller that recorded `h` outside the database (a log line, an in-memory cache, an external index, a named root written by another process's copy of the id) and later calls `read(h)` gets B's bytes with no error — a logical use-after-free at the handle level, which is exactly the failure mode ARCHITECTURE.md:531 claims the permanent-burn policy prevents. The same holds after `rollback_to(savepoint)` and after crash recovery (the test above). The code's behavior is defensible (a rolled-back allocate never happened), so the docs are the wrong side: the guarantee holds only for handles that reached a durable commit and were then deleted, not "within a database's lifetime". + +**Fix:** Scope the claim in all three places to committed handles: monotonic and never reused *once committed*; ids minted inside a transaction that is rolled back (or lost to crash recovery) are re-minted. If callers genuinely need embed-anywhere safety, that requires a generation counter in `HandleEntry` (a format change) — say so explicitly rather than implying it already holds. + +#### `DESIGN` BENCH-5 — Grid cells and three whole rows are skipped on a "Chisel cache hard ceiling" rationale that the enabled spillway removed +**bench/benches/micro_grid.rs:50-54, bench/benches/micro_grid.rs:173-175, bench/benches/micro_grid.rs:300-302, bench/benches/micro_grid.rs:412-416** + +micro_grid.rs says "Cells where `ops_per_tx * size_bytes` exceeds this are skipped to avoid Chisel's CacheFull at large 1000-per-tx writes (cache hard ceiling is ~16 MB…)" and skips with "continue; // skip cells too large to fit in Chisel's cache", and `micro_grid()` disables two rows entirely: "update-1000pertx and delete-1000pertx skipped: … exceeding Chisel's 2048-page cache ceiling. The cells are not measurable under default cache settings". The same crate states the opposite: runner.rs:330-338 "Bench engines now run with the spillway enabled at the production-default scale (1024 × cache budget) … so there is no strict cache-page ceiling on transaction size", and chisel_engine.rs:38-40 does set `.spillway_max_bytes(cache_max_bytes * 1024)`. runner.rs's own test `populate_snapshot_chisel_large_size_chunks` allocates 24 MiB through a 2 MB cache and passes. + +**Why:** Coverage is silently lost on an invalidated premise: allocate-1000pertx and update-1000pertx drop the 16KB/128KB/1MB columns, and update-1000pertx / delete-1000pertx are never registered at all. The grid that is supposed to characterise Chisel under large transactions measures nothing there, and the comments will lead the next maintainer to re-derive the same wrong ceiling. + +**Fix:** Re-run the excluded cells against the spillway-enabled engine; if they now complete, delete `TX_BUDGET_BYTES` and the skips and re-register the two rows. If a real limit remains, restate the comments in terms of the spillway budget instead of the removed 2048-page cache cap. + +#### `DESIGN` BENCH-6 — Module docs claim a 270-cell, 9-row micro grid; the code registers 6 rows and emits 165 cells +**bench/benches/micro_grid.rs:1-2, bench/benches/micro_grid.rs:407-417, bench/src/runner.rs:405-407** + +micro_grid.rs opens with "Bench binary: the 270-cell micro grid. Iterates EngineMode::ALL × SIZES × the 9 row groups", and runner.rs documents `capture_aux_metrics_snapshot_restore` as covering "the snapshot-restore-style rows (8 of 9 rows: rows 1, 2, 4–9)". `micro_grid()` actually calls six row builders (allocate-1pertx, allocate-1000pertx, read-warm, read-cold, update-1pertx, delete-1pertx). The committed bench/results/aux_metrics.jsonl has 165 lines, and discover.rs:258 independently says "165 cells × 2 small JSON files". + +**Why:** A reader sizing runtime, reviewing coverage, or checking that a run completed will compare against 270 cells / 9 rows and conclude 105 cells silently failed. `seed_for` still carries live arms for "update-1000pertx" and "delete-1000pertx" that nothing calls, reinforcing the illusion that those rows run. + +**Fix:** Update both headers to the six rows / 165 cells actually registered (or re-enable the missing rows per BENCH-5 and make the number true), and drop the dead `seed_for` arms if the rows stay disabled. + +#### `DESIGN` BENCH-7 — cross-engine.md's methodology footer claims "a single fsync per commit" for every engine; Chisel performs three +**bench/src/summary/render_cross_engine.rs:145-147** · *NEW* + +The footer rendered into the published comparison document states: "Each engine takes / a single fsync per commit through the disk write cache". Chisel's commit protocol is three fsyncs — src/lib.rs:503 "Commit the active transaction. Performs three fsyncs before", src/transaction/lifecycle.rs:121 "This is the FIRST of the three fsyncs", and the bench's own runner.rs:878-879 comment says "Commits executed (3 protocol fsyncs each)". + +**Why:** This renderer's module doc says the output is "suitable for the README and 1.0 release notes", so the wrong claim propagates into public performance material. It understates Chisel's per-commit durability cost by 3× and invites the reader to conclude the three engines' commit paths are equivalent when Chisel is doing strictly more syscall work for the same throughput number. + +**Fix:** State the per-engine commit protocol honestly in the footer (Chisel: three fsyncs; redb Durability::Immediate; SQLite WAL + synchronous=FULL + fullfsync), since that asymmetry is precisely what the throughput table is being read against. + +#### `DESIGN` PUBLIC-API-2 — README classifies `Poisoned` as a fatal error and tells callers to branch on `is_fatal()`, which returns false for it — its own recovery example never reopens +**README.md:325, README.md:329, README.md:337, src/error.rs:214** · *NEW* + +README.md:325 lists `Poisoned` in the **Fatal errors** group ("Drop the handle and reopen"), and README.md:329 says "Use `ChiselError::is_fatal()` to classify at runtime." But `is_fatal()`'s `matches!` at src/error.rs:215-227 deliberately omits `Poisoned`, and src/error.rs:206-208 documents that "`Poisoned` itself is NOT fatal by this definition"; the exhaustiveness test at src/error.rs:554-556 pins it in the Operational block. + +**Why:** The README's own recovery snippet (README.md:336-343) is `Err(e) if e.is_fatal() => { drop(db); reopen }` / `Err(e) => return Err(e), // operational`. A caller who copies it and later sees `ChiselError::Poisoned` (which every call on a poisoned handle returns, per the same README section) takes the operational arm, keeps the dead handle, and every subsequent call returns `Poisoned` forever — the exact non-recovery the poison model exists to prevent. + +**Fix:** Pick one story and state it in both places: either move `Poisoned` out of the README's fatal list and tell callers to match `Poisoned` explicitly alongside `is_fatal()`, or add a separate classifier (e.g. `needs_reopen()`) that is true for the fatal set *and* `Poisoned`, and point the README at that. + +#### `DESIGN` PUBLIC-API-3 — README's Rust examples do not compile against the current public API (private `defrag` path, renamed field, `#[non_exhaustive]` literals, u32/u64 vs `Tag`/`Handle`) +**README.md:175, README.md:180, README.md:191, README.md:195, README.md:293, README.md:264** · *NEW* + +Compiling the README snippets verbatim from a downstream crate yields 5 errors. `use chisel::defrag::DefragOptions;` (README.md:175) — `defrag` is `pub(crate) mod defrag;` (src/lib.rs:41); the type is re-exported as `chisel::DefragOptions` (src/lib.rs:72). `DefragOptions { sparse_threshold: 0.25, max_pages: 0 }` (README.md:178-181) — the field is `max_values` (src/defrag.rs:80) and the struct is `#[non_exhaustive]` (src/defrag.rs:76). `db.allocate_tagged(b"row-a", 42)?` / `assert_eq!(db.tag(a)?, 42)` / `db.handles_with_tag(42)?` (README.md:191-196) — the signatures take `Tag` and return `Option`/`Handle` (src/lib.rs:609, 618, 655). `let options = Options { … };` (README.md:293) — `Options` is `#[non_exhaustive]` (src/lib.rs:139), which the very next paragraph then says. The API table at README.md:264-270 still advertises `u64` handles and a `u32` tag "`0` if untagged". + +**Why:** The README is the entry point for every new downstream user; each snippet is a compile error, and nothing catches it because the README is not wired into a doctest (`lib.rs` has no `#![doc = include_str!("../README.md")]`). A user following the tag example writes `42` where a `Tag` is required and cannot tell from the docs that `Tag::new(42)` and `Option` are the real vocabulary. + +**Fix:** Update the snippets to the post-I120/I126 signatures and builder-based construction, then wire README.md in as a doctest (`#![doc = include_str!("../README.md")]` with `no_run` fences where a real file is needed) so the compiler enforces it from now on. + +#### `DESIGN` PUBLIC-API-5 — `get_root_name`'s `# Errors` says "Only on poisoning" but it returns `InvalidRootName`; `clear_root_name` omits it too +**src/lib.rs:794, src/lib.rs:804, src/transaction/named_roots.rs:87** · *NEW* + +`Chisel::get_root_name`'s doc reads "# Errors — Only on poisoning — an unbound `name` returns `Ok(None)`" (src/lib.rs:794-795) and `clear_root_name`'s reads "# Errors — `NoActiveTransaction` if no transaction is open" (src/lib.rs:803-804). Both delegate to inner functions that call `Self::encode_root_name(name)?` (src/transaction/named_roots.rs:87 and :114), which returns `ChiselError::InvalidRootName` for an empty name, a name longer than 24 bytes, or a name containing NUL (src/transaction/named_roots.rs:28-33). + +**Why:** Confirmed from a downstream binary: `db.get_root_name("")` returns `Err(InvalidRootName)` and `db.clear_root_name("")` (inside a transaction) returns `Err(InvalidRootName)`. A caller who trusts "only on poisoning" and treats any `Err` from a lookup as a fatal, drop-and-reopen condition will tear down a perfectly healthy handle because of a 25-byte name. The crate enables `#![warn(clippy::missing_errors_doc)]` (src/lib.rs:27), which guarantees the section exists but not that it is complete — making these omissions easy to trust. + +**Fix:** Add `InvalidRootName` to both `# Errors` sections (it is the same validation `set_root_name` already documents), or drop the validation on the read/clear paths so an unrepresentable name simply cannot match and returns `Ok(None)`/`Ok(())`. + +#### `DESIGN` PYTHON-4 — python/README.md Tags section documents the pre-I126 tag API: tag 0 and an int-valued tag() that no longer exist +**python/README.md:131** + +README.md:131-134: "Tag `0` is the \"untagged\" sentinel: it is never indexed, so `handles_with_tag(0)` is always empty -- use plain `allocate()` for untagged values." README.md:144: "assert tx.tag(h) == 42 # 0 if untagged". The binding rejects tag 0 outright — `require_tag` (python/src/db.rs:84-87) returns `PyValueError::new_err("tag must be non-zero")` for every tagged method — and `tag()` returns `Option` (db.rs:415-420), i.e. Python `None`. Verified: `db.handles_with_tag(0)` raises `ValueError: tag must be non-zero`; `db.tag(h)` on an untagged handle returns `None`. python/chisel/__init__.py:3-7 documents the correct rules, so the two docs contradict each other. + +**Why:** A user writing the documented defensive pattern `if db.handles_with_tag(0): ...` gets an unhandled ValueError, and `if tx.tag(h) == 0:` silently never matches (None != 0) so untagged handles are misclassified as tagged. The README is the wrong side here — the code and __init__.py agree. + +**Fix:** Rewrite README.md:131-134 and the inline comment at :144 to match __init__.py's docstring: tags are >= 1, tag 0 raises ValueError on every tagged method, `tag()` returns None for untagged handles. + +#### `DESIGN` PYTHON-5 — README defrag example uses a `max_pages` kwarg that does not exist and denies that Transaction.defrag exists +**python/README.md:242** + +README.md:242: `result = db.defrag(chisel.DefragOptions(sparse_threshold=0.25, max_pages=0))`. The dataclass field is `max_values` (python/chisel/__init__.py:119-120), and the binding reads exactly `sparse_threshold` and `max_values` (python/src/db.rs:518-519). Verified: `chisel.DefragOptions(sparse_threshold=0.25, max_pages=0)` raises `TypeError: DefragOptions.__init__() got an unexpected keyword argument 'max_pages'. Did you mean 'max_values'?`. README.md:246 compounds it by pointing at "`DefragOptions.max_pages`'s docstring", which does not exist. README.md:240 also asserts "defrag lives on the Chisel object, not the Transaction object", but `PyTransaction::defrag` exists (python/src/transaction.rs:211-214) and is tested (tests/test_stats_defrag.py::test_transaction_defrag_mid_tx). + +**Why:** The only defrag example in the binding's documentation cannot be run: copy-pasting it raises TypeError before reaching the engine. The `max_pages` name also does not appear anywhere in the codebase, so a reader cannot map it to the real knob without reading the Rust source. + +**Fix:** Change the example and the surrounding prose to `max_values`, drop the 'legacy carry-over / see max_pages's docstring' sentence, and delete the incorrect 'defrag lives on the Chisel object, not the Transaction object' claim (or restate it as 'available on both'). + +#### `DESIGN` PYTHON-6 — The type stub is named chisel/chisel.pyi, so PEP 561 resolves it to a nonexistent module `chisel.chisel` and no checker ever reads it +**python/chisel/chisel.pyi:7** + +The stub file is `python/chisel/chisel.pyi` and its header asserts "Why this file lives alongside __init__.py rather than at the package root: the stubs describe the `chisel` package namespace as users see it. The `py.typed` marker next to this file signals PEP 561 inline-typed package so type checkers pick these up." PEP 561 resolves stubs by module path: inside package `chisel`, the file `chisel.pyi` is the stub for module `chisel.chisel`, which does not exist; the stub for the package itself must be `chisel/__init__.pyi`. `py.typed` marks the package inline-typed, which sends checkers to `chisel/__init__.py` — whose entire public surface comes from `from chisel._chisel import (...)`, a compiled module with no stub of its own. The file is nonetheless shipped into the wheel (pyproject.toml:31). No CI job runs mypy or pyright (.github/workflows/ci.yml has cargo test/clippy/fmt and pytest only), so the misnaming is unverifiable in-tree. + +**Why:** Every type declaration in the 244-line stub — the whole `Chisel`/`Transaction`/`Savepoint` API, the exception hierarchy, `IoError.errno`/`.kind` — is inert. A user type-checking `chisel.open('db').allocate(b'x')` gets 'Cannot find implementation or library stub for module named chisel._chisel' and no signatures, despite the package advertising py.typed. Stub/impl drift also accumulates undetected (e.g. `__exit__` is declared `-> None` in the stub while the implementation returns `bool`). + +**Fix:** Rename to `python/chisel/__init__.pyi` (updating the pyproject include and the header comment), or move the declarations into a `chisel/_chisel.pyi` stub for the extension module. Add a mypy/pyright job to the python CI so the stub is actually exercised. + +#### `DESIGN` PYTHON-8 — README 'Thread safety' contradicts the shipped concurrency test and omits that the GIL is held across every blocking engine call +**python/README.md:403** + +README.md:403: "A `Chisel` instance is **not** safe for concurrent use from multiple threads ... two threads must never call into the same `Chisel` at the same time." The suite asserts the opposite for reads: `test_two_thread_mutex_contention` (tests/test_exception_contract.py:229-283) runs two threads doing 200×8 concurrent `db.read()` calls on one handle and asserts no error, no corruption, no poison. That is sound because per-op methods never release the GIL — `with_inner_io`/`with_inner_mut_io` take no `Python` token and call the engine with the GIL held (db.rs:636-672) — plus the `Mutex>`. The genuine per-op consequence is documented only in a Rust comment the user never sees: "long-running engine calls (e.g. a large commit's fsync, a big defrag) will block ALL other Python threads in this process" (db.rs:643-645). The README never mentions the GIL. + +**Why:** The two statements cannot both guide a user: the README forbids what the test suite certifies, so a reader cannot tell whether a concurrent read is undefined behaviour or supported. Meanwhile the property that actually bites — every commit's three fsyncs stall every other thread in the process, since only `open()` releases the GIL (db.rs:276) — is undocumented for users, who will reasonably assume a Rust extension releases the GIL around blocking I/O. + +**Fix:** Rewrite the Thread safety section to state what holds: calls are serialized (GIL + Mutex) so concurrent calls cannot corrupt memory, but the single-writer transaction state is shared, so two threads must not interleave transactions; and the GIL is held for the duration of every engine call, so long commits/defrags block all Python threads. + +#### `DESIGN` PYTHON-9 — python/README.md documents the encryption and key-rotation ERRORS but never documents the feature, the `encryption_key` kwarg, or add_key/rotate_key/remove_key +**python/README.md:279** + +The `open()` signature block (README.md:277-288) lists path, cache_max_bytes, spillway_max_bytes, drain_insertion, create_if_missing, read_only, superblock_count — and stops. `encryption_key` is a real parameter (python/src/db.rs:176, :194, :222-240) and `add_key`/`rotate_key`/`remove_key` are real methods (db.rs:555-578), all covered by tests (tests/test_encryption.py, tests/test_encryption_keys.py). Their names appear in the README exactly once each, inside the error tables (README.md:352-356: NoEncryptionKeyError / InvalidEncryptionKeyError / EncryptionNotSupportedError / NoFreeKeySlotError / LastKeySlotError). There is no `## Encryption` section (section list: Status, Install, Quick start, Transactions, Savepoints, Values, Handles, Tags, Client byte, Named roots, Stats and defrag, Engine counters, Opening a database, Errors, Recovery, Thread safety, In-memory mode, On-disk format compatibility, Design). + +**Why:** The binding's only user-facing documentation makes an entire shipped subsystem undiscoverable: a reader learns that `NoFreeKeySlotError` fires when 'all 8 key-slot table entries are in use' without ever being told the API that fills them, or that passing a `str` means passphrase and `bytes` means a raw key (db.rs:63-77). Combined with the inert type stub (PYTHON-6), there is no in-tree source a user can read to discover these methods short of the Rust source. + +**Fix:** Add an Encryption section covering the `encryption_key` kwarg (bytes = raw key, str = passphrase), add it to the `open()` signature block, and document add_key/rotate_key/remove_key with the 8-slot limit and the between-transactions restriction. + +#### `DESIGN` DOCS-COMMENTS-1 — ARCHITECTURE.md documents a `DataPage::compact()` and a `SLOT_FLAG_DEAD` constant that do not exist anywhere in the codebase +**ARCHITECTURE.md:347** + +ARCHITECTURE.md asserts in four places that data pages have a compaction primitive and a dead-slot flag. Line 347: "Each slot directory entry is 6 bytes: 2-byte data offset + 2-byte length + 2-byte flags (`SLOT_FLAG_LIVE = 0x0001`, `SLOT_FLAG_DEAD = 0x0000`) ... `compact()` reclaims dead slots and returns an old→new index mapping; the transaction layer is responsible for rewriting any handle-table entries that reference a compacted page." Line 583: "freed slots become tombstones until `compact()` reclaims the space. Compact is invoked by `defrag()` (R3)". Line 711 glossary: "**Slot tombstone** — a slot directory entry with `SLOT_FLAG_DEAD`. Reclaimed by `compact()`, not reused by `insert()`." In the code, `src/data_page.rs` defines only `const SLOT_FLAG_LIVE: u16 = 0x0001;` (line 53) and the methods `init_page`, `slot_count`, `free_space`, `validate_header`, `insert`, `read`, `used_space`, `iter_live`, `read_slot_entry`. `grep -rn "compact\|SLOT_FLAG_DEAD" src/` returns no definition of either. Nothing ever clears a slot's LIVE flag: `SlotPacker::release` (src/transaction/packing.rs:181) only decrements an in-memory counter. The code's own header says the opposite of ARCHITECTURE — src/data_page.rs:32-34: "Dead slots are NOT reused by insert(); the transaction layer frees whole pages rather than compacting individual pages." + +**Why:** A maintainer reading ARCHITECTURE believes deleted values leave a `SLOT_FLAG_DEAD` marker on the page and that `defrag()` calls `compact()`. Both are false: after `delete(h)`, the data-page slot keeps `SLOT_FLAG_LIVE` and its payload bytes; only the handle-table entry is tombstoned. Concretely, `DataPage::used_space` and `DataPage::iter_live` (src/data_page.rs:227, 254) count and return those orphaned slots as live, so anyone who wrote page-density or leak-detection logic on the documented model would compute wrong occupancy. It also hides that deleted inline values are not scrubbed from the page, which matters for the encryption threat model. + +**Fix:** Rewrite the data-page section of ARCHITECTURE to describe the actual model: the slot directory is append-only, `SLOT_FLAG_LIVE` is the only flag ever written, liveness is tracked out-of-band by `SlotPacker`'s live-slot counts, and reclamation is whole-page (a page whose live count hits zero is pushed to `txn_freed_pages`) rather than intra-page compaction. Drop `compact()` and `SLOT_FLAG_DEAD` from the module table, the layout section, the slot-packing section, and the glossary. + +#### `DESIGN` DOCS-COMMENTS-2 — `DefragOptions::max_pages` does not exist — the field is `max_values`; both README examples and the ARCHITECTURE "legacy name" note are wrong and neither example compiles/runs +**ARCHITECTURE.md:609** + +`src/defrag.rs:76-81` declares `pub struct DefragOptions { pub sparse_threshold: f64, pub max_values: usize }` and the setter at line 100 is `pub fn max_values(mut self, cap: usize) -> Self`. The Python dataclass matches: `python/chisel/__init__.py:120` is `max_values: int = 0`, and the binding reads it as `obj.getattr("max_values")` (python/src/db.rs:519). But README.md:178-181 shows `db.defrag(DefragOptions { sparse_threshold: 0.25, max_pages: 0, })?;`, python/README.md:242 shows `chisel.DefragOptions(sparse_threshold=0.25, max_pages=0)`, and ARCHITECTURE.md:609 states "The cap parameter (`DefragOptions::max_pages`) bounds the number of *values* relocated in one pass, despite the legacy name (kept for API stability; see C4 in ISSUES.md)." python/README.md:246 further points the reader at "`DefragOptions.max_pages`'s docstring", which does not exist. + +**Why:** A user copying README.md:174-183 gets a compile error (`struct DefragOptions has no field named max_pages`); a user copying python/README.md:242 gets `TypeError: DefragOptions.__init__() got an unexpected keyword argument 'max_pages'` from the frozen dataclass. ARCHITECTURE's "legacy name kept for API stability" note is doubly misleading: it asserts a rename was deliberately avoided when the rename already happened, so a maintainer tidying the API would think the current name is `max_pages` and "fix" the wrong side. + +**Fix:** Rename `max_pages` → `max_values` in all three documents, drop the "despite the legacy name / kept for API stability" sentence from ARCHITECTURE.md:609 and the parallel parenthetical in python/README.md:246, and re-check both examples against the actual constructors (the Rust one also needs the chained-setter form, since `DefragOptions` is `#[non_exhaustive]`). + +#### `DESIGN` DOCS-COMMENTS-4 — ARCHITECTURE's freemap-reclamation section is written against a removed `allocate_data_page` and claims handle-table COW pages bypass the freemap and "always extend" +**ARCHITECTURE.md:597** + +ARCHITECTURE.md:587 says "`allocate_data_page` tries `FreeMapTree::allocate_first` first, falls back to `cache.new_page`", line 595 says "(`allocate_data_page` checks `savepoints.is_empty()`)", and line 597 says "Overflow pages and handle-table COW pages do *not* go through `allocate_data_page` (they call `cache.new_page` directly and always extend)". No function named `allocate_data_page` exists — `grep -rn allocate_data_page src/` returns only three historical mentions inside comments (src/transaction/freemap.rs:28, src/transaction/packing.rs:78, src/transaction/packing.rs:271, all phrased "formerly" / "historical"). The live allocator is the free function `cow_alloc` (src/transaction/freemap.rs:48) reached via `FreemapRecycle::cow_alloc_into` (line 223). Handle-table COW pages go through it: src/transaction/staging.rs:50 and src/transaction/mutate.rs:198 build `let mut alloc = |c: &mut PageCache| self.freemap.cow_alloc_into(c, &mut tree, reuse);` and hand that closure to `HandleTable::insert`/`delete`. src/handle_table.rs:25-31 confirms: "Page allocation goes through an `alloc` closure the caller injects (the transaction layer's freemap-aware `cow_alloc`), so superseded pages from a prior committed transaction are reused before the file is extended." src/freemap.rs:7-11 says the same. The savepoint gate is in `insert_into_data_page` (src/transaction/packing.rs:282), not in any `allocate_data_page`. + +**Why:** Line 597 states the exact opposite of the current behaviour, and that behaviour is the fix for a real page leak (handle-table COW pages that always extended grew the file one page per mutation). A maintainer auditing reclamation against ARCHITECTURE would conclude the handle table is *expected* to leak COW pages and would not treat unbounded file growth as a bug; conversely someone "restoring" the documented behaviour would reintroduce the leak. The three references to a nonexistent `allocate_data_page` also send any reader grepping for it to a dead end. + +**Fix:** Rewrite ARCHITECTURE's "Freemap reclamation" section against the current call graph: `cow_alloc` / `FreemapRecycle::cow_alloc_into` is the single freemap-aware allocator, used by data-page inserts (via `insert_into_data_page`), handle-table COW, and membership-index COW alike; the `savepoints.is_empty()` reuse gate lives at those call sites; only overflow-chain pages still call `cache.new_page` directly and always extend. + +#### `DESIGN` DOCS-COMMENTS-5 — README classifies `Poisoned` as a fatal error and tells callers to use `is_fatal()`, but `is_fatal(Poisoned)` is false — the README's own recovery snippet routes a poisoned handle into the "operational, keep going" arm +**README.md:325** + +README.md:323-325 lists under "**Fatal errors** — storage integrity is in question. Drop the handle and reopen.": "`IoError`, `ChecksumMismatch`, `CorruptSuperblock`, `FileSizeMismatch`, `LockFailed`, `UnsupportedFormatVersion`, `UnsupportedPageSize`, `CorruptPage`, `InvalidPageId`, `DecryptionFailed`, `Poisoned`." Line 329: "Use `ChiselError::is_fatal()` to classify at runtime." But `ChiselError::is_fatal` (src/error.rs:214-228) does not include `Poisoned` in its `matches!` list, and the in-tree classification test spells the intent out — src/error.rs:554-556: "Poisoned is operational by is_fatal()'s definition: the manager is already dead, so re-seeing it must not re-poison." README's recovery example at lines 336-344 then does `Err(e) if e.is_fatal() => { drop(db); db = Chisel::open(...)?; }` with a fallthrough `Err(e) => return Err(e), // operational — handle per your caller's policy`. + +**Why:** A caller who implements README's documented policy and receives `ChiselError::Poisoned` (which every call returns once the handle is dead — README.md:333) falls into the *operational* arm, keeps the dead handle, and every subsequent call returns `Poisoned` again: an unrecoverable loop that the README told them how to build. The two statements cannot both be followed — the classification table and the `is_fatal()` instruction disagree about the one variant a caller is most likely to see after a failure. + +**Fix:** Pick one side and make the docs say it. Either move `Poisoned` out of README's fatal list into a third "terminal — already poisoned" note explaining that `is_fatal()` deliberately returns false for it, and change the recovery snippet to match on `ChiselError::Poisoned` in addition to `e.is_fatal()`; or add `Poisoned` to `is_fatal()` and update the src/error.rs test's rationale. The doc-only fix is the smaller change and matches the code's stated reasoning. + +#### `DESIGN` DOCS-COMMENTS-6 — `Chisel::commit`'s `# Errors` section lists `CacheFull`/`SpillwayFull` as operational, but commit poisons the handle on *any* error +**src/lib.rs:527** + +src/lib.rs:526-530 documents commit as: "# Errors — `NoActiveTransaction` if none is open. Operationally, `CacheFull` or `SpillwayFull` if the transaction's working set exceeds the cache / spillway caps. A failure inside the fsync/superblock protocol is fatal and poisons the handle". The type-level contract at src/lib.rs:306-316 defines what a `# Errors` list means: "Each method's own `# Errors` section therefore lists only the *operational* (recoverable, non-poisoning) errors specific to that call ... Operational errors leave the handle usable: fix the condition (or `rollback`) and continue." The implementation contradicts this — src/transaction/lifecycle.rs:194-198: `let result = self.commit_inner(); if result.is_err() { self.poisoned.set(true); } result`, with the comment at lines 180-190 stating "Special poison policy for commit: we refuse BOTH operational and fatal errors that arise after the commit protocol has started ... once cache.flush() has run, any subsequent error — even an otherwise operational one — leaves the manager in a partial-commit state". Only the pre-protocol `NoActiveTransaction` check (line 191) escapes. + +**Why:** A caller who catches `CacheFull` from `commit()` and follows the documented recovery ("fix the condition or `rollback`, and continue") calls `db.rollback()` on an already-poisoned handle and gets `ChiselError::Poisoned`, with no doc anywhere telling them that a `CacheFull` *from commit specifically* is terminal. The blast radius is the whole "operational vs fatal" contract the crate advertises: this is the one method where the contract is inverted, and the doc says nothing about it. + +**Fix:** Amend `Chisel::commit`'s `# Errors` to state that `NoActiveTransaction` is the only non-poisoning error it can return, and that every other error — including `CacheFull` and `SpillwayFull` — poisons the handle because the commit protocol has already begun. Mirror the note in README's "Poison model" section, which currently only mentions "a failed commit-protocol fsync". + +#### `DESIGN` TXN-COMMIT-2 — ARCHITECTURE.md states handle-table COW pages bypass the freemap allocator and always extend — the implementation does the exact opposite, and that routing is load-bearing +**ARCHITECTURE.md:597, ARCHITECTURE.md:587, ARCHITECTURE.md:595** · *REGRESSION of I118's doc side* + +ARCHITECTURE.md:597 says: "Overflow pages and handle-table COW pages do *not* go through `allocate_data_page` (they call `cache.new_page` directly and always extend)". The code routes handle-table AND membership-index COW allocation through the freemap-aware allocator: `let mut alloc = |c: &mut PageCache| self.freemap.cow_alloc_into(c, &mut tree, reuse);` in `handle_table_insert_candidate` (staging.rs:97), `membership_insert_candidate` (staging.rs:50), `membership_remove_candidate` (staging.rs:172), `delete_inner` (mutate.rs:198) and `ht_insert` (freemap.rs:638). freemap.rs:45-47 states the point explicitly: "Routing handle-table and membership COW allocation through here — rather than the monotonic `new_page` — is what lets those structures reach a bounded steady-state page count instead of leaking one page per mutation." Only overflow still calls `new_page` directly (freemap.rs:616-619). Separately, the function `allocate_data_page` named at ARCHITECTURE.md:587, :595 and :597 no longer exists anywhere in src/ (it is now `insert_into_data_page` + `freemap::cow_alloc`; packing.rs:78 and freemap.rs:28 both call it "historical"/"formerly"). + +**Why:** The doc is the ARCHITECTURE reference for the freemap-reclamation design. A maintainer trusting :597 would conclude the handle table has unbounded page growth per mutation and would either (a) "fix" it by adding freemap routing that already exists — duplicating the take_tree/put_tree dance and double-draining `pending_superseded` — or (b) reason about the I18 allocate-before-merge crash invariant while believing HT COW targets can never come from the committed bitmap, which is precisely the case I18 exists to bound. :595's "one carve-out for data allocation" understates it too: `reuse = self.savepoints.is_empty()` is evaluated at five call sites and gates ALL freemap reuse, not just data pages. + +**Fix:** Rewrite :597 to say overflow pages alone bypass the freemap allocator; state that handle-table and membership-index COW pages allocate through `freemap::cow_alloc` and that this is what bounds their steady-state page count. Replace the three `allocate_data_page` references with `insert_into_data_page` / `freemap::cow_alloc`, and reword :595 to say the savepoint gate disables reuse for every allocation site. + +#### `DESIGN` TXN-COMMIT-3 — ARCHITECTURE.md describes overflow values as occupying a data-page slot holding a chain-head pointer; no data page or slot is involved at all +**ARCHITECTURE.md:581** · *NEW* + +ARCHITECTURE.md:581: "Larger values get an overflow chain; the slot directory entry then points at the first chain page id with `HandleFlags::Overflow` set, and the data-page slot contains the chain head pointer rather than the value itself." The code allocates no data page and no slot for an overflow value: `let first_page = { ... Overflow::write(&mut cache, value)? }; HandleEntry { page_id: first_page, slot_index: 0, flags: HandleFlags::Overflow, ... }` (staging.rs:242-253, mirrored in mutate.rs:69-80). `HandleFlags` is a field of the handle-table `HandleEntry` (handle_table.rs:145-149), not of a data-page slot-directory entry — data_page.rs has only `SLOT_FLAG_LIVE` (data_page.rs:53). mod.rs:70-72 states the real model: "Larger values are written to an overflow chain and referenced by a single HandleEntry with HandleFlags::Overflow." + +**Why:** The R1 live-slot accounting depends on overflow entries owning NO slot: `open_existing` counts only `if entry.flags == HandleFlags::Live` (recovery.rs:522), and both `update_inner` and `delete_inner` release the chain via `txn_freed_pages` while deliberately NOT calling `release_data_slot` (mutate.rs:148, mutate.rs:295). A maintainer following :581 would add slot accounting for the phantom overflow slot — either decrementing a count that was never incremented (so `SlotPacker::release` under-counts and a still-referenced page is pushed to `txn_freed_pages`, i.e. a freed-but-referenced page) or freeing `entry.page_id` as a data page in addition to the chain walk. + +**Fix:** State that an overflow value's `HandleEntry` points directly at the first chain page with `slot_index = 0`, that no data page or slot-directory entry is allocated for it, and that overflow entries therefore contribute nothing to the R1 live-slot map. + +#### `DESIGN` TXN-COMMIT-4 — ARCHITECTURE.md documents a `DataPage::compact()` API and an old→new slot-index remapping contract that do not exist in the codebase +**ARCHITECTURE.md:124, ARCHITECTURE.md:347, ARCHITECTURE.md:583, ARCHITECTURE.md:711** · *REGRESSION of I130* + +ARCHITECTURE.md:347: "`compact()` reclaims dead slots and returns an old→new index mapping; the transaction layer is responsible for rewriting any handle-table entries that reference a compacted page." ARCHITECTURE.md:124 repeats it ("Slot indices are stable until compact; compact returns an old→new mapping for callers to rewrite"), as do :583 ("freed slots become tombstones until `compact()` reclaims the space. Compact is invoked by `defrag()`") and :711. `grep -rn "fn compact" src/` returns nothing. data_page.rs:33-35 documents the opposite as a deliberate invariant: "Dead slots are NOT reused by insert(); the transaction layer frees whole pages rather than compacting individual pages." What `defrag` actually does is relocate values through `txm.update` so a sparse page's live-slot count drains to zero and the WHOLE page is freed (defrag.rs:231-268, packing.rs:181-194). :347 also names a `SLOT_FLAG_DEAD = 0x0000` constant that does not exist (data_page.rs defines only `SLOT_FLAG_LIVE`). + +**Why:** The invented contract is the opposite of a real, load-bearing invariant. data_page.rs:31-32 and handle_table.rs:44-46 both rely on "slot indices are stable for the lifetime of the page — the handle table stores (page_id, slot_index) and relies on this". A maintainer implementing the documented `compact()` would renumber slots inside a live page; every handle-table entry pointing into that page that the remap loop missed would then resolve to the wrong value or to a dead slot, which `read_inner` escalates to a fatal `CorruptPage` (read.rs:141-146). The doc actively invites the one change the format forbids. + +**Fix:** Delete the four `compact()` references and the `SLOT_FLAG_DEAD` constant name. Describe defrag's actual mechanism: relocate values via `update` until a sparse page reaches zero live slots, at which point `SlotPacker::release` returns the whole page to `txn_freed_pages`. State plainly that slot indices are immutable for the page's lifetime and that dead slots are reclaimed only by freeing the entire page. + +#### `DESIGN` FREEMAP-4 — defrag()'s public rustdoc describes a 'threshold x max_observed' sparseness rule that the implementation explicitly abandoned +**src/defrag.rs:139, src/transaction/stats.rs:137** · *NEW* + +`defrag`'s public rustdoc, step 2, says: "A page is sparse if its live-slot count is at or below `threshold x max_observed`" (src/defrag.rs:137-140). The implementation is a per-page density ratio with the page's own slot count as the denominator: `let density = live as f64 / stored as f64; if density < threshold_ratio { sparse.insert(page_id); }` (src/transaction/stats.rs:137-140). `sparse_data_pages`' own doc names the discarded design outright: "The metric uses the page's OWN stored-slot count ... not the max-observed count in the database ... The older \"relative to densest\" metric failed for the case of a single remaining sparse page" (src/transaction/stats.rs:99-104). `DefragOptions::sparse_threshold`'s doc (src/defrag.rs:57-59) matches the implementation, so defrag.rs contradicts itself as well. + +**Why:** `defrag` and `DefragOptions` are public API; this rustdoc is what a caller reads to pick a threshold. Under the documented rule, `sparse_threshold(0.25)` on a database whose densest page holds 40 slots would mean 'pages with <= 10 live slots'; under the real rule it means 'pages less than 25% full relative to their own stored slot count'. Those select different page sets, and the documented rule (relative to the densest page) never fires for the one case the implementation was changed to handle — a lone sparse page, which scores 1.0 against itself. Callers tune the knob against a rule that does not exist. + +**Fix:** Replace defrag.rs:137-140 with the actual predicate (live_slots / stored_slots strictly less than `sparse_threshold`, per page), matching the wording already in `sparse_data_pages` and in `DefragOptions::sparse_threshold`. + +#### `SMELL` HANDLES-INDEX-5 — ARCHITECTURE.md says handle-table COW pages always extend via cache.new_page; they have gone through the freemap-aware allocator since the alloc-closure refactor +**ARCHITECTURE.md:597, src/transaction/staging.rs:97, src/transaction/freemap.rs:638, src/handle_table.rs:26** · *REGRESSION of I118 (duplicate)* + +ARCHITECTURE.md:597 states: "Overflow pages and handle-table COW pages do *not* go through `allocate_data_page` (they call `cache.new_page` directly and always extend), but their *frees* still feed the freemap on commit". Every production handle-table mutation instead receives a freemap-aware closure: `staging.rs:97` `let mut alloc = |c: &mut PageCache| self.freemap.cow_alloc_into(c, &mut tree, reuse);` immediately before `self.handle_table.insert(...)`, the same at `freemap.rs:638` for `ht_insert` and `mutate.rs:198` for `handle_table.delete`. handle_table.rs:26-31 says so explicitly: "Page allocation goes through an `alloc` closure the caller injects (the transaction layer's freemap-aware `cow_alloc`), so superseded pages from a prior committed transaction are reused before the file is extended." The only remaining `cache.new_page()` in the module is `create_root` (handle_table.rs:175), which runs once per database. + +**Why:** The doc understates reclamation for the handle table and, read alongside the freemap chapter, tells a maintainer that handle-table growth is extend-only — leading to wrong conclusions about steady-state file size and about which allocations are affected by the savepoint reuse carve-out (`reuse = self.savepoints.is_empty()`, staging.rs:93), a nuance the doc does not mention at all for these pages. + +**Fix:** Rewrite ARCHITECTURE.md:597 to say handle-table and membership-index COW pages allocate through `cow_alloc` (freemap reuse first, extend as fallback, reuse disabled while a savepoint is open), and that only `create_root` and overflow pages extend directly. + +#### `SMELL` CRYPTO-6 — The documented "argon2 params are zero for HKDF" slot invariant is violated by the create path, which writes the OWASP defaults into HKDF slots +**src/transaction/recovery.rs:590, src/superblock/crypto_header.rs:19, src/superblock/crypto_header.rs:244** · *NEW* + +The on-disk record layout is documented twice, identically: "2..14 argon2 params: m_cost(u32) | t_cost(u32) | p_cost(u32) (zero for HKDF)" (crypto_header.rs:19) and ARCHITECTURE.md:663 "2..14 | argon2 params: m_cost(u32) | t_cost(u32) | p_cost(u32) (zero for HKDF)". `wrap_into` honors it: `crate::crypto::Key::Raw(_) => (KdfId::Hkdf, Argon2Params { m_cost: 0, t_cost: 0, p_cost: 0 })` (crypto_header.rs:244-251). `build_create_cipher` does not: `crate::crypto::Key::Raw(_) => (KdfId::Hkdf, Argon2Params::default())` (recovery.rs:590), and `slot.argon2 = params;` (recovery.rs:602) then stamps 19456/2/1 into a slot whose `kdf_id` is HKDF. + +**Why:** A database created with `Key::Raw` has slot 0 carrying m=19456,t=2,p=1 with kdf_id=1, while a second raw credential added later via `add_key` lands in a slot carrying zeros with the same kdf_id — the two paths produce different bytes for identical semantics, and neither matches what the format doc promises. Any external tool, forensic script, or future validator that enforces the documented "zero for HKDF" rule (a reasonable integrity check, since the field is meaningless for HKDF) will reject every raw-key database Chisel has ever created. Because `KeySlot::aad()` covers these bytes, the discrepancy is inert for unwrap today — which is exactly why it will stay unnoticed until something depends on it. + +**Fix:** Make `build_create_cipher` write zeros for the HKDF branch (or make `wrap_into` write the defaults) so the two paths agree, and pick whichever the doc states. Better: delete the duplication by having `build_create_cipher` call `CryptoHeader::wrap_into` — see CRYPTO-7. + +#### `SMELL` PAGE-IO-2 — spillway_max_bytes is documented as a cap on the spillway FILE but only caps the live resident set; the file grows without bound under forget/respill +**src/page_cache.rs:114-115 and src/page_cache.rs:33 (comment text only)** + +The field doc says: "Strict upper bound on the spillway sidecar file in bytes (excluding per-slot headers)." The module header repeats it: "`spillway_max_bytes` caps the spillway file." The actual check in `Spillway::spill` charges only the LIVE resident set: `let post_write_bytes = (self.slots.len() as u64 + 1) * self.payload_size as u64; if post_write_bytes > self.max_bytes { return Err(ChiselError::SpillwayFull {...}) }`. Slot addressing uses the monotonic `next_slot_index`, which `forget` never decrements — spillway.rs's own field doc says so: "forget/respill of the same page reuses an existing slot and does NOT shrink the vec. A long transaction that forget/respills many distinct [pages]...". So the file length is `next_slot_index * (SLOT_HEADER_SIZE + payload_size)`, which is unrelated to `max_bytes`. lib.rs:104 gets it right ("strict upper bound on the spillway's LIVE ..."); page_cache.rs does not. + +**Why:** A spill→`get()` (rehydrate calls `spw.forget`, src/page_cache.rs:977)→re-spill cycle is a normal access pattern under cache pressure, and each cycle consumes a fresh slot. An operator who sizes the volume from `spillway_max_bytes` (default 1024 × cache_max_bytes = 8 GiB) gets ENOSPC from a `.spillway` file that has grown well past the number they configured, with no `SpillwayFull` ever raised — the cap they set was never a file-size cap. + +**Fix:** Correct both comments in page_cache.rs to say the cap bounds the LIVE resident set, not the file, and cross-reference spillway.rs's `next_slot_index` doc for the monotonic-cursor growth. If a real file-size bound is wanted, charge `next_slot_index` (not `slots.len()`) against a second cap, or reclaim forgotten slot indices via a free list. + +#### `SMELL` SWIFT-11 — PARITY.md's line references into ChiselDatabase.swift are stale for 8 rows, in a table whose header claims it was verified by reading the file +**swift/PARITY.md:95, swift/PARITY.md:8** · *NEW* + +PARITY.md:8-9 asserts the Swift column is "verified by reading each file directly, not inferred from the plan." Eight rows disagree with the file. `Chisel.read` → "ChiselDatabase.swift:58" (PARITY.md:95); line 58 is blank. Actual `public func read` is line 73. Likewise `handles` :65→75 (PARITY.md:99), `stats` :67→77 (:107), `counters` :69→79 (:108), `tag` :73→85 (:101), `handles_with_tag` :75→87 (:102), `client_byte` :79→91 (:105), `get_root_name` :83→95 (:110). The neighbouring rows are all correct — `isPoisoned`:54, `close`:57, `fileSizeBytes`:81, `addKey`:99, `transaction`:117, `allocate`:142, `Savepoint.name`:209 — as is every one of the 19 `ChiselStore.swift` references and all six generated-file references at PARITY.md:49-50 (819/854/996/1026/974/1005 all check out). The drift is confined to the block that shifted when the seven-line "Mutating operations ... are NOT mirrored here" comment (ChiselDatabase.swift:64-70) was inserted. + +**Why:** The table is the stated source of truth for Python↔Swift parity and the artifact a reviewer uses to confirm "No known gaps" (PARITY.md:16). Eight of its citations point at blank lines and mid-comment text, so the next person to audit parity either wastes time or — worse — trusts the unverifiable rows because the verified-by-reading claim in the header is still there. + +**Fix:** Regenerate the ChiselDatabase.swift column, or drop bare line numbers in favour of symbol names, which do not rot when a comment block is inserted. + +#### `SMELL` DOCS-COMMENTS-9 — THEORY.md says the Cargo workspace was deferred and that `bench/` is not a workspace member; Cargo.toml declares a workspace with `bench` in `default-members` +**THEORY.md:178** + +THEORY.md:178 lists among rejected alternatives: "Adopting a real Cargo workspace — deferred (I61): members would share edition/rust-version/feature resolution, too restrictive for the PyO3 abi3 binding, and the bench floats its floor faster than the engine wants." THEORY.md:190 adds: "It lives in a `bench/` subcrate that is a *sibling* to `python/`, not a plain workspace member drawn into the engine's own build in a way that would auto-run its 10–25 minutes of tests on every `cargo test`." Cargo.toml:19-31 declares exactly that workspace: `[workspace] members = [".", "python", "bench"]` with `default-members = [".", "bench"]`, headed by the comment "I61 (ISSUES.md, 2026-05-22): Cargo workspace declaration. Members share one Cargo.lock (at workspace root) and one target/ directory". README.md:73 and ARCHITECTURE.md:133 both describe it as landed: "`bench/` is a `default-members` workspace member (I58/I61), so a root `cargo test` runs its tests too." + +**Why:** THEORY is the document a maintainer reads to understand *why* the build is shaped as it is; it asserts a decision ("deferred") that was subsequently reversed under the same issue number, and further asserts a build property (bench tests not run by a root `cargo test`) that the repo's own contributor instructions contradict. Someone acting on THEORY.md:190 would not expect `cargo test` at the root to run the bench crate, and would not understand why a bench-crate compile error breaks the engine's test command. + +**Fix:** Rewrite the MSRV entry's rejected-alternatives line to record that the workspace was adopted (I61) with `python` excluded from `default-members` for the PyO3 linker reason, and correct the "sibling, not a workspace member" sentence in the implementation-history section to match `default-members = [".", "bench"]`. + +#### `SMELL` DOCS-COMMENTS-10 — ARCHITECTURE documents the HKDF info string as "chisel-kek"; the code uses "chisel-kek-v1" +**ARCHITECTURE.md:673** + +ARCHITECTURE.md:673, in the on-disk-encryption section's key-derivation spec: "**Raw key** (`Key::Raw`): KEK = HKDF-SHA256(ikm=key material, salt=slot salt, info=`\"chisel-kek\"`)." The code's constant is `const KEK_INFO: &[u8] = b"chisel-kek-v1";` (src/crypto/mod.rs:140), used at src/crypto/mod.rs:181 (`hk.expand(KEK_INFO, okm.as_mut())`) and pinned by the reference test at src/crypto/mod.rs:455 (`hk.expand(b"chisel-kek-v1", &mut expect)`). + +**Why:** The info string is a hard on-disk format parameter — src/crypto/mod.rs:138-140 says so explicitly ("Changing it is a format break (existing slots would stop unwrapping)"). ARCHITECTURE is the stated byte-level reference for the crypto layout, so anyone reimplementing the unwrap path (a recovery tool, a second-language binding, a security audit) from the documented value derives a different KEK and every existing key slot fails to unwrap — presenting as `InvalidEncryptionKey` on a correct passphrase, with no clue where the divergence came from. + +**Fix:** Correct ARCHITECTURE.md:673 to `info="chisel-kek-v1"`, and note that the `-v1` suffix is the KDF-construction version so a future revision can coexist (the rationale already recorded at src/crypto/mod.rs:138-140). + +#### `SMELL` DOCS-COMMENTS-11 — ARCHITECTURE states superblock ties break by lowest slot index (the code's `max_by_key` picks the highest), and presents the `#[cfg(test)]`-only `Superblock::select` as the production recovery path +**ARCHITECTURE.md:310** + +ARCHITECTURE.md:310: "`Superblock::select` reads up to `MAX_SUPERBLOCKS` (= 16) candidate pages ... and `max_by_key`s on `txn_counter`. Ties break by lowest slot index". The function's own doc says the opposite — src/superblock/mod.rs:559-561: "Tie-break: `max_by_key` returns the LAST maximum in iteration order (highest page index on a tie)" — which is `Iterator::max_by_key`'s documented behaviour. Separately, `select` is gated `#[cfg(test)]` (src/superblock/mod.rs:562) and its doc says "Used in tests only. Production code (`open_existing`) inlines the same `filter_map` + `max_by_key`"; the production selection is src/transaction/recovery.rs:319-325. README.md:378 and the recovery doc comment at src/transaction/recovery.rs:172-175 both name `Superblock::select` as the thing that runs on open. + +**Why:** Two separate wrongnesses in one paragraph. The tie-break direction is inverted, and ties are not hypothetical — ARCHITECTURE itself notes they occur "during the `create_new` seeding window", and `create_new` seeds slot i with counter N-1-i (src/transaction/recovery.rs:86), so a reader reasoning about which slot wins on a fresh encrypted database gets the wrong buffer. That is precisely the failure src/transaction/recovery.rs:311-318 warns about ("decrypt_body builds its AAD from the winner's txn_counter but reads the sealed body from the wrong buffer ... the correct key wrongly fails to open"). The `select`-as-production claim sends a reader auditing crash recovery to a test-only function that is not the code that runs. + +**Fix:** Fix the tie-break sentence to "highest slot index (`max_by_key` returns the last maximum)", and reword ARCHITECTURE/README to say recovery inlines the same filter+max in `TransactionManager::open_existing` (keeping the winning buffer for `decrypt_body`), with `Superblock::select` noted as the test-only mirror. + +#### `SMELL` DOCS-COMMENTS-13 — README's chunk-tags example passes bare `u32` literals where the API takes `Tag`, and the API table says `tag()` returns `0` when untagged instead of `None` +**README.md:191** + +The public tag API is typed: `pub fn allocate_tagged(&mut self, value: &[u8], tag: Tag)` (src/lib.rs:609), `pub fn tag(&self, handle: Handle) -> Result>` (src/lib.rs:618), `handles_with_tag(&self, tag: Tag)` (line 655), `delete_with_tag(&mut self, tag: Tag, max: usize)` (line 742). `Tag` wraps `NonZeroU32` and has no `From` — only `Tag::new(v) -> Option` and `TryFrom` (src/handle.rs:96-133). README.md:191-206 nevertheless writes `db.allocate_tagged(b"row-a", 42)?`, `db.handles_with_tag(42)?`, and `db.delete_with_tag(42, 256)?`, plus `assert_eq!(db.tag(a)?, 42);` at line 195 — the latter comparing an `Option` against an integer. README.md:270's API table says: "`tag(handle)` | Read a handle's tag, `0` if untagged". src/handle.rs:5-7 states the design intent the README contradicts: "'no tag' is the absence of a `Tag` (`Option`), never the in-band sentinel `0`." + +**Why:** None of the four calls in the example compile — `Tag` cannot be built from an integer literal — so the section's entire code block is unusable as written, and it is the only documentation of how to construct a tag. The API-table entry teaches the exact sentinel model (`0` = untagged) that `handle.rs` deliberately removed from the public surface, so a reader writes `if db.tag(h)? == 0` and finds no such comparison exists. + +**Fix:** Rewrite the example to construct tags (`let t = Tag::new(42).expect("non-zero");`) and to match on `Option` for `tag()`; change the API-table row to "Read a handle's tag; `None` if untagged". Mention that `Tag` is non-zero and that untagged values use plain `allocate`. + +#### `SMELL` DOCS-COMMENTS-16 — python/README's `open()` signature omits the `encryption_key` keyword it documents elsewhere, and its `pages_allocated` definition misses freemap-reuse allocations +**python/README.md:281** + +python/README.md:280-288 presents the full `chisel.open(...)` signature — path, cache_max_bytes, spillway_max_bytes, drain_insertion, create_if_missing, read_only, superblock_count — and stops there. The binding's actual signature (python/src/db.rs:167-178) ends with `encryption_key = None`, and the stub file confirms it (python/chisel/chisel.pyi:139: `encryption_key: bytes | str | None = None`). The same README's error table already references the keyword (line 352: "`NoEncryptionKeyError` | Opened an encrypted database without supplying `encryption_key`"), and the root README.md:396 says "Encryption is exposed through the `open()` `encryption_key` keyword". Separately, python/README.md:261 defines "`pages_allocated` — `PageCache.new_page` invocations", while the authoritative description counts both extension and reuse — src/stats.rs:68-72: "page allocations, counting BOTH file extensions (`PageCache::new_page` ...) AND freemap reuses (`PageCache::claim_page` ...). Reuse is the common case once the handle table / membership index allocate COW pages through the freemap-aware path, so a counter that ignored it would read ~0 for a steady-state mutating workload." + +**Why:** The signature block is the section a Python user reads to discover open-time options; omitting `encryption_key` means encryption looks unreachable from Python even though the error table implies it exists — the two halves of the same document disagree. The counter definition understates what `pages_allocated` measures in exactly the steady-state mutating workload stats.rs calls out, so a user benchmarking allocation behaviour from the Python side misreads the number. + +**Fix:** Add `encryption_key=None # bytes (raw 32-byte key) or str (passphrase)` to the signature block with a one-line create-vs-reopen note, and restate `pages_allocated` as "page allocations: both file extensions (`new_page`) and freemap reuses (`claim_page`)", matching src/stats.rs. + +#### `SMELL` FREEMAP-2 — ARCHITECTURE.md says handle-table COW pages bypass the freemap and always extend; the code routes them through cow_alloc +**ARCHITECTURE.md:597 (also :587, :595)** · *REGRESSION of I118 (duplicate of TXN-COMMIT-2)* + +ARCHITECTURE.md:597 states: "Overflow pages and handle-table COW pages do *not* go through `allocate_data_page` (they call `cache.new_page` directly and always extend), but their *frees* still feed the freemap on commit". The code says the opposite for the handle table (and membership index): `cow_alloc`'s doc is "Freemap-aware page allocator shared by data-page allocation and the handle-table / membership-index COW paths" (src/transaction/freemap.rs:20-21), and every handle-table / membership site builds `let mut alloc = |c: &mut PageCache| self.freemap.cow_alloc_into(c, &mut tree, reuse);` (src/transaction/freemap.rs:638, src/transaction/staging.rs:50/97/172, src/transaction/mutate.rs:198). src/freemap.rs:8-19 documents the current behavior correctly ("the handle-table / membership-index COW paths prefer `FreeMap::allocate_first`... so they reach a bounded steady-state page count rather than growing one page per mutation"). Only overflow still extends directly (src/overflow.rs:92). + +**Why:** ARCHITECTURE.md is the stated design reference. A maintainer sizing the file's steady-state growth, or debugging why a reused id came back from `claim_page` on a handle-table COW, is told by the top-level doc that this path can only extend. The doc also still names `allocate_data_page` (ARCHITECTURE.md:587, :595), a function that no longer exists anywhere in src/ — only in three historical comments (src/transaction/freemap.rs:28, src/transaction/packing.rs:78, :271). + +**Fix:** Rewrite ARCHITECTURE.md:587-597 to say that data-page, handle-table and membership-index COW allocation all go through `cow_alloc` (reuse-before-extend), that only overflow still calls `cache.new_page` directly, and replace the `allocate_data_page` references with `cow_alloc` / `insert_into_data_page`. + +#### `SMELL` FREEMAP-10 — ARCHITECTURE.md's defrag section names two API items that do not exist: DefragOptions::max_pages and compact() +**ARCHITECTURE.md:609, ARCHITECTURE.md:583** · *REGRESSION of I122 and I130* + +ARCHITECTURE.md:609 says "The cap parameter (`DefragOptions::max_pages`) bounds the number of *values* relocated in one pass, despite the legacy name (kept for API stability; see C4 in ISSUES.md)." The field is `pub max_values: usize` (src/defrag.rs:80) with builder `max_values` (src/defrag.rs:100) — a crate-wide grep for `max_pages` returns nothing, including the PyO3 binding, which reads `max_values` (python/src/db.rs:519). ARCHITECTURE.md:583 says "freed slots become tombstones until `compact()` reclaims the space"; there is no `compact` function anywhere in src/ or python/src/ — the reclaiming caller is `defrag()` itself via `update()` relocation (src/defrag.rs:250-251). + +**Why:** Both lines send a reader looking for symbols that do not exist, and :609's parenthetical actively argues for keeping a name that was already changed — so a maintainer trusting it would either rename `max_values` back to `max_pages` (a breaking change to the public struct and the Python binding, which extracts the attribute by name at python/src/db.rs:519) or file a compatibility issue for a non-problem. + +**Fix:** Update ARCHITECTURE.md:609 to name `DefragOptions::max_values` and drop the legacy-name caveat; replace the `compact()` reference at :583 with `defrag()` (or with the packer's slot-release / relocation mechanism). + +#### `NIT` PAGE-IO-5 — Half of DataPage's public surface is dead, its docs name production callers that do not exist, and a blanket allow(dead_code) hides it +**src/data_page.rs:221-227 (used_space doc) and src/data_page.rs:251-256 (iter_live doc)** + +`#[allow(dead_code)]` sits on the whole `impl DataPage` block with the note "`free_space`, `used_space`, and `iter_live` are called only from cfg(test) today". Grepping the crate (plus tests/, bench/, python/, swift/) for `DataPage::free_space|used_space|iter_live|slot_count` returns no call site outside data_page.rs itself. Their docs assert callers that do not exist: `used_space` — "Used by defrag/stats to decide whether a page is sparse enough to merit consolidation" (src/defrag.rs never calls it); `iter_live` — "important for defrag, which rewrites handle table entries" (defrag uses `HandleTable::iter_live`, src/handle_table.rs:397, a different function); `free_space` — "so downstream mutators refuse to operate on it" (the only mutator, `insert`, calls `validate_header` directly, src/data_page.rs:167). `slot_count` is not mentioned by the attribute's note at all and is likewise callerless. + +**Why:** The allow-attribute is on the impl block, not the individual items, so it also disarms dead-code detection for `insert`, `read`, and `init_page` — if a future refactor orphans the real API, nothing warns. Meanwhile a maintainer reading `used_space` believes defrag's density heuristic depends on it and will preserve or 'fix' semantics for a consumer that does not exist. + +**Fix:** Move `#[allow(dead_code)]` onto the four genuinely test-only items so the live methods stay warning-covered, and strike the false caller claims from their docs (or delete the four methods — defrag can grow its own helper if it ever needs one). + +#### `NIT` PAGE-IO-11 — ARCHITECTURE.md's headline durability claim says two fsyncs per commit and calls the cache cap 'soft'; both contradict the code and the same document +**ARCHITECTURE.md:29 (primary); ARCHITECTURE.md:120 (weaker half)** + +Line 29 states: "Every commit performs two (three with the I28 pre-drain) `fsync` calls (data, then superblock)." The I28 pre-drain is not conditional — `ctx.cache.borrow_mut().flush()?;` runs unguarded on every commit (src/transaction/commit.rs:78), and `PageCache::flush` always calls `self.io.fsync()?` even with zero dirty pages (src/page_cache.rs:587). Line 577 of the same document gets it right: "The no-spill commit cost is **3 fsyncs**." Line 120 similarly says "Soft eviction at `max_pages`" in the same table cell that then says "`CacheFull` at strict `max_pages`", while page_cache.rs's header states "The cache is a STRICT bound with sidecar overflow" and "The pre-spillway 8× HARD_CEILING_MULTIPLIER design is gone" (src/page_cache.rs:29-38). + +**Why:** Line 29 is the document's three-bullet summary of the engine's durability contract — the part a reader trusts without checking. Someone benchmarking or budgeting IOPS from it under-counts commit cost by 50%, and 'soft eviction' is leftover HARD_CEILING_MULTIPLIER vocabulary that tells a reader the cap is elastic when it is not. + +**Fix:** Line 29: "Every commit performs three `fsync` calls (I28 pre-drain, data, superblock)." Line 120: replace "Soft eviction" with "strict cap at `max_pages`; dirty pages are pinned and spill to the sidecar instead of growing the cache". + +### comment-accuracy (38) + +#### `DESIGN` CRYPTO-2 — `sb_identity_aad`'s doc claims it binds each key-slot's DEK wrap to the superblock identity; it does not — the wraps use `KeySlot::aad()` and are superblock-agnostic +**src/superblock/mod.rs:324, src/superblock/crypto_header.rs:271, src/transaction/recovery.rs:606** + +superblock/mod.rs:324-327 reads: "Build the AAD that binds the sealed body and each key-slot's DEK wrap to this superblock's plaintext identity ... this prevents transplanting a sealed body from a different DB or a different txn_counter." `sb_identity_aad()` (mod.rs:334) is passed to exactly one place — `cipher.seal_body(&aad, ...)` at mod.rs:414 and `cipher.open_body(&aad, ...)` at mod.rs:456. Every DEK wrap uses a different AAD: `crypto::wrap_dek(&kek, dek, &s.wrap_nonce, &s.aad())` (crypto_header.rs:271) and `wrap_dek(&kek, &dek, &wrap_nonce, &aad)` where `let aad = slot.aad();` (recovery.rs:605-606). `KeySlot::aad()` (crypto_header.rs:81-91) contains only state, kdf_id, m/t/p, salt, wrap_nonce — no magic, no format_version, no txn_counter, no superblock_count. + +**Why:** A maintainer reading this comment will believe the key-slot table is cryptographically pinned to the superblock generation it was written in, and will therefore not add such a binding when it is needed. That belief is exactly what makes CRYPTO-1 possible: because the wraps carry no superblock identity, an old key-slot table spliced into (or simply read out of) a different superblock generation still authenticates. The comment asserts the security property whose absence is the actual weakness. + +**Fix:** Correct the doc to say `sb_identity_aad` covers only the sealed body, and state explicitly that key-slot wraps are bound solely by `KeySlot::aad()` (slot-local metadata). If the claimed property is wanted, extend the wrap AAD to include the superblock identity/generation rather than the comment. + +#### `DESIGN` PAGE-IO-1 — page_cache.rs module header states rollback does NOT rewind next_page_id; the production rollback path rewinds it via truncate() +**src/page_cache.rs:24-28 (header), src/page_cache.rs:597-601 (discard doc), src/page_cache.rs:722-724 (truncate rewind), src/transaction/lifecycle.rs:266** · *NEW* + +The module header asserts an invariant: "`next_page_id` is a monotonic allocator. It is seeded from the file's page count at open time, and is bumped on every `new_page()`. Rollback does NOT rewind it (see the note in `discard`); orphaned page IDs are acceptable because they are reclaimed by the freemap after commit or simply re-truncated." `discard`'s doc repeats it: "Note: `next_page_id` is deliberately NOT rewound. If rollback freed IDs back to the allocator, two concurrent savepoint rollbacks could hand the same ID to two different allocations." But `truncate` does exactly that: `if self.next_page_id > n { self.next_page_id = n; }` (src/page_cache.rs:722-724), and its own doc says the opposite of the header — "This is the only path that legitimately rewinds `next_page_id`". The production rollback calls it: `cache.truncate(self.committed_roots.total_pages)?` (src/transaction/lifecycle.rs:266), as does `rollback_to` (src/transaction/savepoints.rs:75). The cited `discard` is itself `#[allow(dead_code)]` with no production caller, so the header points the reader at the one path that is NOT the rollback path. The rationale is also wrong on its own terms: "two concurrent savepoint rollbacks" cannot exist in a deliberately single-threaded, single-client engine. + +**Why:** A maintainer reading the header will believe page ids handed out before a rollback are permanently burned and can never be reissued, and will therefore skip cache/spillway/freemap invalidation for the reissued range when adding a new allocation path — precisely the class of aliasing bug the header claims is impossible. Concretely: allocate page 100 in a txn, rollback (truncate rewinds next_page_id to the committed total, say 90), allocate again — new_page() returns 100 a second time. Anyone who wrote code relying on the header's monotonicity claim has a stale-identity bug. + +**Fix:** Rewrite the header bullet to say next_page_id is monotonic WITHIN a transaction and is rewound only by `truncate` (rollback and rollback_to), and point at `truncate`'s doc rather than the dead `discard`. Drop the "two concurrent savepoint rollbacks" rationale, which contradicts the single-threaded model. + +#### `DESIGN` PAGE-IO-4 — DataPage::insert's doc claims the transaction layer allocates a new page per insert; R1 slot packing has replaced that +**src/data_page.rs:161-165** · *NEW* + +`DataPage::insert` carries: "Note (v1 simplification per ARCHITECTURE.md): the transaction layer calls PageCache::new_page() for every insert rather than scanning existing pages for free slots. Intentional, not a bug — this function itself is correct; it's just underutilized." That is no longer true. `transaction::packing` implements the R1 packing cursor — its header says "Packing cursor: a data page allocated earlier in THIS transaction that still has free space. New values pack into it until it fills, at which point a new page is allocated and becomes the new cursor" — and src/transaction/packing.rs:122 calls `DataPage::insert(buf, value)` on that cursor page, with a fresh allocation + `DataPage::init_page` only on the fill path (src/transaction/packing.rs:144-152). ARCHITECTURE.md:318 also states the current behaviour: "Data pages are slotted: they pack multiple values per page (R1)". + +**Why:** The comment tells a maintainer that the function's multi-slot machinery is dead weight ("underutilized"), which is an invitation to simplify or delete the slot-directory append path that the R1 cursor now depends on for every non-first insert in a transaction. It also mis-sets expectations for anyone debugging page occupancy: they will look for one page per value and not find it. + +**Fix:** Replace the note with the current model — inserts pack into the transaction's cursor page via `SlotPacker`, and a new page is allocated only when the cursor fills or packing is disabled (savepoints active). + +#### `DESIGN` PUBLIC-API-4 — `Stats` field docs contradict how `Chisel::stats` populates them, and `file_size_bytes` under-reports encrypted databases +**src/stats.rs:20, src/stats.rs:23, src/lib.rs:849, src/lib.rs:893** + +`Stats::total_pages` is documented as "Total allocated pages in the file, matching Superblock.total_pages" (src/stats.rs:20-21), but `Chisel::stats` fills it from `self.txm.file_page_count()?` (src/lib.rs:842), which is `PageIo::cached_page_count` — the *physical* file length in stride units (src/page_io.rs:521-523), not the superblock field. `Stats::file_size_bytes` is documented as "Raw size of the database file on disk. May exceed `total_pages * PAGE_SIZE` when a previous crash left orphan pages" (src/stats.rs:23-27), yet it is computed as exactly `page_count.saturating_mul(PAGE_SIZE as u64)` from that same `page_count` (src/lib.rs:859) — it can never exceed itself. `Chisel::file_size_bytes` does the same (src/lib.rs:895). + +**Why:** Both fields mislead in opposite directions, and the PAGE_SIZE multiplication is simply wrong for encrypted databases, whose stride is `ENC_PAGE_SIZE` = 8232 (src/transaction/recovery.rs:67, src/page_io.rs:216-226). Measured on a real encrypted DB from a downstream binary: `file_size_bytes()` reported 876544 (107 × 8192) while `stat` on the file returned 880824 (107 × 8232) — a 4280-byte under-report of the "Physical size of the database file in bytes" that README.md:282 promises. Anyone sizing a backup or a disk-quota alarm off this number is short by ~0.5% on every encrypted database. + +**Fix:** Multiply by `io.stride()` rather than `PAGE_SIZE` in both `stats()` and `file_size_bytes()`, and rewrite the two `Stats` field docs to say what the code actually reports (physical page count and physical byte size), dropping the impossible "may exceed" clause. + +#### `DESIGN` PYTHON-2 — savepoint.rs claims the engine pops the savepoint itself on rollback_to; the engine explicitly keeps it +**python/src/savepoint.rs:99** + +savepoint.rs:99-104 states: "The engine pops the savepoint stack down to AND including this savepoint, so the mark itself is gone after one successful call — a second call would fail at the engine layer with SavepointNotFound regardless." The engine does the opposite: `rollback_to_inner` ends with `self.savepoints.truncate(idx + 1);` (src/transaction/savepoints.rs:103), which RETAINS index `idx`, and the engine's own doc comment says so — "The named savepoint itself remains on the stack and can be rolled back to again or released" (src/transaction/savepoints.rs:47-49). Verified: after `sp.rollback_to()`, `tx.savepoint('s')` raises `DuplicateSavepointError: duplicate savepoint: s`, proving the mark is still there. The same wrong model is repeated in tests/test_exception_contract.py:26 and :34 ("sp1.rollback_to() # pops both sp1 and sp2 from the engine stack" — it pops sp2 only). + +**Why:** The comment is the stated justification for the Python-side guard ('The guard here turns that into a cleaner, more specific AlreadyFinishedError'). A maintainer who trusts it would conclude the guard is a cosmetic re-labelling of an engine error and could delete it — which would then expose repeated rollback_to as a working operation, silently changing the documented Python contract. It also misdescribes why test_exception_contract's SavepointNotFound tests pass. + +**Fix:** Correct the comment to match src/transaction/savepoints.rs:47-49 and :103 (the named savepoint survives; only savepoints layered on top are popped), and fix the two test comments. Then re-justify the guard on its own terms (or drop it — see PYTHON-3). + +#### `DESIGN` DOCS-COMMENTS-3 — Handle-table FLAG byte is documented as "forensic-only, no runtime code reads it" in two places, but `recover_depth` uses it as its loop terminator +**src/handle_table.rs:85** + +src/handle_table.rs:85-88 states: "Currently forensic-only — no live code reads it (the depth walk uses child-pointer presence); kept because a hex-dump reader can tell leaf from interior at a glance." ARCHITECTURE.md:452 repeats it verbatim in substance: "The flag byte at position 1 is forensic-only — no runtime code reads `FLAG_LEAF`/`FLAG_INTERIOR`; the depth walk uses child-pointer presence instead." The depth walk in fact reads it first — `HandleTable::recover_depth`, src/handle_table.rs:433-436: `let buf = cache.get(current)?;` then `if buf[1] != FLAG_INTERIOR { break; }`. Child-pointer presence is only the *secondary* terminator (`if child == 0 { break; }` at line 449). The constant itself was made crate-visible for exactly this purpose — src/handle_table.rs:95-99: "I50 ... `pub(crate)` so transaction.rs's open_existing depth-walk can match against the named constant instead of a raw `0x02` literal." — directly contradicting the "no live code reads it" claim eleven lines above. + +**Why:** `recover_depth` is called on every open (src/transaction/recovery.rs:488) and on every rollback (src/transaction/lifecycle.rs:282-289), and its result is the descent depth for all subsequent lookups. A maintainer who trusts the "forensic-only" comment and stops writing `buf[1] = FLAG_INTERIOR` in `grow` (src/handle_table.rs:534), or repurposes byte 1 for a new header field, makes `recover_depth` return 0 for a depth-N tree; every committed handle then mis-descends and `lookup` returns `Ok(None)` → `InvalidHandle` for live data, with no checksum or type-tag error to signal it. ARCHITECTURE explicitly flags radix-depth re-derivation as load-bearing (line 543), so a comment inviting removal of its terminator is a live trap. + +**Fix:** Replace both comments with the truth: byte 1 carries `FLAG_LEAF`/`FLAG_INTERIOR` and is read at runtime by `HandleTable::recover_depth` as the leaf/interior discriminator on the left-spine walk; the zero-child check is the secondary terminator. Note in ARCHITECTURE's handle-table section that the flag is part of the depth-recovery contract, not decoration. + +#### `SMELL` SUPERBLOCK-RECOVERY-5 — `sb_identity_aad`'s doc claims it binds the cleartext bootstrap fields, but `page_size` stays cleartext and is excluded from the AAD +**src/superblock/mod.rs:325 (doc) — the AAD body at 334-342 and the cleartext write at 406 are both correct as written** · *NEW* + +The doc asserts coverage of the cleartext set: "The four bootstrap fields that stay cleartext in both encrypted and plaintext DBs are included" and "These four MUST stay cleartext even in an encrypted DB precisely because they are the AAD" (mod.rs:326-333). The AAD body covers magic, format_version, txn_counter, superblock_count (mod.rs:335-341). But `serialize_encrypted` leaves FIVE fields cleartext — it writes `buf[48..52].copy_from_slice(&self.page_size.to_le_bytes());` (mod.rs:406), and its own comment at mod.rs:401-402 acknowledges "page_size at 48..52 is a cleartext bootstrap field". `page_size` is therefore cleartext, is consumed at open (`if sb.page_size != PAGE_SIZE as u32`, recovery.rs:406), and is authenticated by nothing but the forgeable XXH3 page checksum. + +**Why:** Two concrete costs. (a) A single flipped byte in 48..52 plus a recomputed XXH3 turns a healthy encrypted database into a permanent `UnsupportedPageSize` failure with no fallback to a sibling slot — the AEAD that would have caught it never sees the field. (b) A maintainer reading this doc concludes that every cleartext bootstrap field is AAD-bound, and will add the next cleartext field (a second stride hint, a flags word) assuming it is authenticated — writing exactly the bug this comment was meant to prevent. + +**Fix:** Either add `page_size` to `sb_identity_aad` (bytes 20..24 are already reserved for exactly this) and bump the format version, or correct the doc to say five fields stay cleartext and state explicitly that `page_size` is deliberately outside the AAD and why. + +#### `SMELL` SUPERBLOCK-RECOVERY-6 — `diagnose`'s documented guarantees are wrong in both directions — it can label data pages as corrupt slots and can omit real slots +**src/superblock/mod.rs:588, src/superblock/mod.rs:598, src/superblock/mod.rs:601** + +The doc promises "this returns one `SlotDefect` per genuine superblock slot", that non-slot pages are excluded because "including them in the defect list would falsely label intact data pages as corrupt superblocks", and that with the MIN fallback "we never under-report" (mod.rs:571-587). The implementation derives its bound from the FIRST buffer whose raw bytes 308..312 happen to land in range — over ALL candidates including non-slot pages: `.find(|&n| (MIN_SUPERBLOCKS..=MAX_SUPERBLOCKS).contains(&n)).unwrap_or(MIN_SUPERBLOCKS)` (mod.rs:589-599), then reports `buffers[..bound.min(buffers.len())]` (mod.rs:601). + +**Why:** Over-report: in a DB with N=2 whose two slots are mangled past recognition, if the data page at index 2 happens to carry a value in 2..=16 at byte offset 308, `bound` becomes that value and pages 2..bound — ordinary intact data pages — are emitted as `BadMagic` superblock defects, the exact outcome the doc says is avoided. Under-report: in a DB with N=4 where all four slots are mangled including their count fields, `bound` falls back to 2 and the operator's `CorruptSuperblock` diagnostic silently omits half the slots. Both mislead whoever is triaging an unopenable file. + +**Fix:** Take the count only from buffers at indices < MAX that also pass the magic check, or drop the heuristic and report all read candidates tagged with whether they are inside the recoverable slot range; either way, correct the doc's two guarantees to match. + +#### `SMELL` HANDLES-INDEX-4 — handle_table.rs claims a lazily created sparse child needs "no further copy"; the recursion immediately copies it and queues it as superseded +**src/handle_table.rs:591, src/membership_index.rs:284** + +handle_table.rs:591-595 says: "Sparse allocation: interior pages don't pre-populate children. A zero pointer means 'no subtree allocated here yet'; we lazily create one only when an insert touches that range. The newly allocated child is already a fresh page, so it IS its own COW clone — no further copy needed." The code does the opposite: the fresh page (`leaf` at :598 or `interior` at :608) is passed as `actual_child` into `self.insert_recursive(cache, actual_child, ...)` at :622, whose first statements are `let new_page = alloc(cache)?;` (:552), a full PAGE_SIZE copy (:557-561), and `freed.push(page_id);` (:567) — so the just-created page is copied to a second fresh page and immediately pushed onto the superseded list. The identical code in membership_index.rs:284-286 documents it correctly: "A fresh child here is immediately re-COWed by the recursive call below (it becomes that frame's superseded `page` and is pushed to `freed` there); benign, first-touch only." + +**Why:** A maintainer reasoning about `freed` from this comment will conclude the list contains only previously-committed pages, when in fact it contains never-committed, this-transaction pages too — precisely the distinction that matters when deciding whether a queued id is safe to hand back out or whether a rollback must un-queue it. It also hides that each sparse-child touch costs two page allocations and an 8 KB copy instead of one. + +**Fix:** Replace the last sentence with the membership_index.rs:284-286 wording, which describes what actually happens; or skip the redundant COW by having the fresh-child branch write the entry directly when `level == 1`. + +#### `SMELL` HANDLES-INDEX-6 — Chisel::tag and Chisel::handles_with_tag doc-comments still describe the pre-newtype u32 API, including a Tag 0 case the type makes unconstructable +**src/lib.rs:614, src/lib.rs:644** + +`src/lib.rs:614` documents `tag` as "Returns 0 for untagged handles", but the signature two lines down (`:618`) is `pub fn tag(&self, handle: Handle) -> Result>` and the body is `self.txm.tag(handle.get()).map(Tag::new) // stored 0 -> None` — untagged yields `None`, never `0`, and `Tag` cannot hold `0` at all (`Tag(NonZeroU32)`, handle.rs:94). `src/lib.rs:644` documents `handles_with_tag` with "Tag 0 always returns an empty Vec (the membership index is not updated for untagged values)", but its parameter is `tag: Tag` (`:655`), so the described call cannot be written — `Tag::new(0)` returns `None` (handle.rs:100-105) and `Tag::try_from(0)` returns `Err(ZeroTagError)` (handle.rs:128-133). + +**Why:** These are the two doc blocks a user reads to learn how "untagged" is represented at the public boundary, and they contradict the exact design decision handle.rs:88-92 was written to make legible ("'No tag' is the ABSENCE of a `Tag` (`Option`) — `Tag(0)` is unconstructable"). A reader following lib.rs:614 will write `if db.tag(h)? == 0`, which does not compile, and will look for a zero-tag branch that cannot exist. + +**Fix:** Change lib.rs:614 to "Returns `None` for untagged handles" and delete the "Tag 0 always returns an empty Vec" sentence at lib.rs:644 (the untagged case is now unrepresentable in the signature). + +#### `SMELL` CRYPTO-9 — Cold-load comment describes a `try_into` that no longer exists, and the equality it claims (stride == ENC_PAGE_SIZE) is asserted in prose but checked nowhere +**src/page_cache.rs:1023** · *NEW* + +page_cache.rs:1022-1027 reads: "// The on-disk unit is exactly ENC_PAGE_SIZE at this stride; // the try_into is infallible (ENC_PAGE_SIZE == 8232 == stride). let unit: [u8; ENC_PAGE_SIZE] = on_disk;". There is no `try_into` — the line is a plain move of a `Copy` array. The invariant it names (cipher present ⟹ `self.io.stride() == ENC_PAGE_SIZE`) is stated only in prose here and in `set_cipher`'s doc, "The caller MUST have already called `self.io_mut().set_stride(ENC_PAGE_SIZE)`" (page_cache.rs:900-902); nothing in `set_cipher` (page_cache.rs:902-904) or on this path checks it. Three lines above, `read_page_unit_into(page_id, &mut on_disk[..stride])` (page_cache.rs:1017-1019) fills only `stride` bytes of the 8232-byte stack buffer. + +**Why:** If the pairing invariant is ever broken by a refactor — cipher installed while stride is still 8192 — the last 40 bytes of `unit` are the stack buffer's zero-initialization rather than the real tag/nonce, `PageCipher::open` fails, and the caller sees `ChiselError::DecryptionFailed { page_id }`, which `is_fatal()` (error.rs:226) and poisons the handle. A configuration mistake would be reported to the user as unrecoverable ciphertext corruption. The stale `try_into` reference removes the one hint a reader has that a length check ever guarded this. + +**Fix:** Delete the `try_into` sentence and replace the prose invariant with a `debug_assert_eq!(stride, ENC_PAGE_SIZE)` on the cipher branch (or fold the stride switch into `set_cipher` so the two cannot be set independently). + +#### `SMELL` BENCH-11 — Equivalence-test comment says "no spillway in bench engines"; every bench Chisel engine enables it +**bench/tests/equivalence.rs:145-149, bench/src/chisel_engine.rs:51-58** · *NEW* + +`make_chisel` is commented "256 pages × 8 KiB = 2 MiB strict cache cap (no spillway in bench engines). Must be large enough for the largest single-tx allocation in any equivalence scenario: scenario_large_overflow allocates 1 MiB, which requires ~130 overflow pages … 256 pages gives comfortable headroom." The constructor it calls, `ChiselEngine::open_in_memory`, does `Options::default().cache_max_bytes(cache_max_bytes).spillway_max_bytes(cache_max_bytes * 1024)` and is itself documented "The spillway is enabled at the same production-default scale as `open_file`". + +**Why:** The comment's whole sizing argument ("must be large enough", "comfortable headroom") rests on a cap that no longer exists. A maintainer adding a larger equivalence scenario will either bloat the cache for no reason or, worse, treat a genuine CacheFull as expected because the comment told them the cap is strict. + +**Fix:** Delete the "(no spillway in bench engines)" parenthetical and the headroom argument, or state the actual spillway budget the engine is opened with. + +#### `SMELL` BENCH-12 — Two runner.rs doc comments contradict their own code: scenario prepop transaction granularity and which modes expose counters +**bench/src/runner.rs:509-510, bench/src/runner.rs:71-77** + +`run_scenario_cell`'s doc says step 2 is "Run `prepopulate_workload` untimed (each Allocate in its own tx / for simplicity; pre-pop time is excluded from the measurement)", but the body batches by `POPULATE_TX_MAX_RECORDS` (500) and `POPULATE_TX_BUDGET_BYTES` (1 MiB) — and the inline comment at 532-542 explicitly says single-op-per-tx prepop was abandoned because it is "~12 min for 100K records". Separately, `supports_internal_counters` is documented "Currently only `ChiselStrict` (the other engines are black-box)" while the body is `matches!(self, Self::ChiselStrict | Self::ChiselMemory)`. + +**Why:** The prepop claim matters for interpretation: a reader reasoning about how much freemap/handle-table COW pressure the pre-populated state carries will model 100K separate commits instead of ~200. The counters claim will send someone hunting for why chisel-mem cells carry counter data the doc says is impossible — and the unit test at runner.rs:647-656 can't catch it because `EngineMode::ALL` deliberately excludes ChiselMemory. + +**Fix:** Rewrite the step-2 line to describe the chunked prepop, and update the `supports_internal_counters` doc to name both Chisel modes. + +#### `SMELL` BENCH-13 — freemap-churn-flat documents its throughput unit as round-trips/sec but sets Criterion's element count to the record count +**bench/benches/freemap_churn.rs:117-118, bench/benches/freemap_churn.rs:138-139** + +The doc says "The throughput unit is \"delete+realloc round-trips per second\" (one round-trip = 2 commits)" and the inline comment repeats "// Throughput: one unit = one (delete + realloc) round-trip." The next line is `group.throughput(criterion::Throughput::Elements(live_count as u64))`, with live_count ∈ {500, 200, 100}, while the timed routine performs exactly one `churn_cycle` per iteration. + +**Why:** Criterion divides the per-iteration time by the element count, so the reported figure is records/sec, 100–500× larger than the round-trips/sec the comment tells the reader they are looking at. Anyone trend-tracking this number across the three cases will also compare values normalized by three different divisors as though they shared a unit. + +**Fix:** Either pass `Throughput::Elements(1)` to match the documented round-trip unit, or restate the doc as "records reclaimed per second" and note that the divisor differs per case. + +#### `SMELL` TESTS-CI-4 — tests/counters.rs documents a 2-fsync commit protocol; the protocol is 3 fsyncs, and the assertion is one fsync weaker than the invariant +**tests/counters.rs:38-45** + +tests/counters.rs:39-40 says "// Commit calls fsync twice (data pages + superblock). Anything below this is a regression in the commit protocol." and asserts `after.fsync_calls >= baseline.fsync_calls + 2` with the message "commit must perform at least 2 fsyncs (data + superblock)". The actual protocol is three: src/transaction/commit.rs:1 titles the module "the 3-fsync commit protocol", commit.rs:78 issues the I28 pre-drain `cache.flush()`, commit.rs:105 the data flush, commit.rs:180 the superblock `fsync()`; `PageCache::flush` fsyncs unconditionally at src/page_cache.rs:586 (`self.io.fsync()?;`, outside any dirty-page conditional), and src/page_cache.rs:578-580 names this "the SECOND of three fsyncs". tests/spillway_integration.rs:166 asserts the delta is exactly 3. + +**Why:** Two defects in one place. (1) The comment contradicts three other in-repo sources and would lead a maintainer editing the commit path to believe removing a flush is legal. (2) The assertion's own stated purpose — "anything below this is a regression in the commit protocol" — is not met: deleting the I28 pre-drain flush at commit.rs:78 (whose only job is keeping CacheFull off the persist_freemap path) drops the count to 2 and this test still passes. Only the spillway test catches it, and only incidentally. + +**Fix:** Correct the comment to three fsyncs and change the assertion to `== baseline + 3`, matching tests/spillway_integration.rs:166 so the two tests pin the same number. + +#### `SMELL` TESTS-CI-10 — Test comments point readers at src/transaction.rs and tests/crash_recovery.rs, neither of which exists +**tests/client_byte.rs:9, tests/api_edge_cases.rs:410, tests/transactions.rs:140, tests/tag_ops.rs:12, tests/freemap_multipage.rs:210, tests/error_and_format.rs:2** · *NEW* + +Six test files cite paths that were removed by the module split. tests/client_byte.rs:8-10: "Poison-path coverage lives in the in-crate unit test `poisoned_manager_rejects_every_public_entry_point` (src/transaction.rs)" — the test is real but lives at src/transaction/tests.rs:210; `src/transaction.rs` does not exist (the module is the directory src/transaction/). tests/api_edge_cases.rs:410: "the existing crash_recovery.rs suite covers poison via real fatal-I/O injection" — there is no tests/crash_recovery.rs; that suite is src/recovery_tests.rs. Likewise tests/transactions.rs:140 ("migrated ... to src/transaction.rs as `reopen_preserves_committed_data`" — actually src/transaction/tests.rs:2290), tests/tag_ops.rs:12 ("`persist_freemap_*` in `src/transaction.rs`"), tests/freemap_multipage.rs:210 ("see the unit-test coverage in transaction.rs"), tests/error_and_format.rs:2 ("the named-root validation path in transaction.rs"). + +**Why:** Every one of these comments exists to route a maintainer to the complementary coverage before they widen or delete a test. Following them lands on a missing file, so the reader concludes the referenced coverage was deleted and either re-adds a duplicate or, worse, removes the integration test believing the in-crate one is gone. This is the same class of rot that produced the duplicated named-root suite in TESTS-CI-9. + +**Fix:** Sweep the six references: `src/transaction.rs` → `src/transaction/tests.rs` (with the function name, which is still accurate in all cases), `crash_recovery.rs` → `src/recovery_tests.rs`. + +#### `SMELL` PYTHON-10 — errors.rs states no test enumerates the Python exception classes; three test modules do, one of them named for the very issue cited +**python/src/errors.rs:17** + +errors.rs:17-19: "No test enumerates the Python classes (ISSUES.md I139); the engine-side is_fatal() exhaustiveness test (I104) guards the classification this fallback depends on." tests/test_exception_contract.py is headed "parametrized typed-exception contract tests (I139)" and pins ~15 concrete classes end-to-end; tests/test_errors.py:8-45 enumerates 13 operational and 10 fatal class names by string and asserts each subclasses the right tier; test_exception_contract.py:288-297 adds the three classes that file was missing. + +**Why:** The comment tells a maintainer that the Rust→Python exception mapping is unverified from the Python side, which invites either redundant new tests or an unchecked reorganization of `to_py_err`'s arms on the belief nothing would catch a swap. The tests that would catch it exist. + +**Fix:** Update the comment to reference tests/test_exception_contract.py and tests/test_errors.py, and state precisely what is still uncovered (per-variant coverage of the fatal arms, which test_exception_contract.py:301-310 itself flags). + +#### `SMELL` SWIFT-8 — Swift `isPoisoned` doc omits that a closed handle also reports true — the Rust doc it wraps says so explicitly +**swift/Sources/Chisel/ChiselDatabase.swift:53, swift/Sources/Chisel/ChiselStore.swift:62, chisel-ffi/src/database.rs:86** · *NEW* + +The Rust side documents both conditions: "True if the handle can no longer do work: the engine reports `is_poisoned()`, OR the handle is already closed" (database.rs:86-87), implemented as `guard.as_ref().map(chisel::Chisel::is_poisoned).unwrap_or(true)` (database.rs:90-93) — `None` (taken by `close`) yields `true`. Both Swift doc comments drop the second half: "True if a fatal error has poisoned the underlying engine handle." (ChiselDatabase.swift:53) and "Whether the engine has poisoned itself (a fatal integrity/I-O failure)." (ChiselStore.swift:62-63). The Rust unit test even asserts the closed case, with a comment acknowledging the conflation: `db.close().unwrap(); ... assert!(db.is_poisoned());` (database.rs:293-296). + +**Why:** A caller that closes cleanly and then checks health — e.g. a diagnostics screen calling `store.isPoisoned()` on a shut-down store, or the README's own drop-and-reopen recovery flow (swift/README.md:144-145) which calls `close()` first — reads `true` and concludes the database file is corrupt, per the documented meaning of "fatal error has poisoned". Correct code, wrong conclusion. + +**Fix:** Copy the Rust phrasing into both Swift doc comments: true if the engine is poisoned OR the handle has been closed. + +#### `SMELL` SWIFT-9 — `transaction`/`savepoint` doc says it rethrows "what `body` threw", but a failing commit/release is also caught, rolled back, and rethrown +**swift/Sources/Chisel/ChiselDatabase.swift:113, swift/Sources/Chisel/ChiselDatabase.swift:181** · *NEW* + +The doc reads "Commits on normal return; on a thrown error, rolls back (best-effort ... swallowed via `try?`) and rethrows what `body` threw." (ChiselDatabase.swift:113-116). The code puts the commit inside the `do`: `let result = try body(txn); try native.commit(); return result } catch { try? native.rollback(); throw error }` (ChiselDatabase.swift:121-127). `savepoint` has the identical shape — "on a thrown error, rolls back to `name` ... then rethrows" (:182-183) with `try native.release(name: name)` inside the `do` (:189). + +**Why:** When `body` returns normally and `commit()` fails, the catch arm still runs: it issues a `rollback()` on an engine that has just poisoned itself on the commit path (src/lib.rs:529-530: "A failure inside the fsync/superblock protocol is fatal and poisons the handle"), and rethrows the COMMIT error — not "what `body` threw", since `body` threw nothing. A maintainer reading the comment would believe the catch arm is reachable only from `body`, and could add cleanup there (e.g. re-running `body`'s compensating action, or logging "user code failed") that misfires on every commit failure. + +**Fix:** State what the code does: the catch arm covers a throw from `body` OR from `commit()`/`release()`, and rethrows whichever one threw. If commit-failure should not trigger a rollback attempt, move `try native.commit()` outside the `do`. + +#### `SMELL` SWIFT-10 — Workspace comment still describes three members and "excludes `python`" after `chisel-ffi` became the fourth excluded member +**Cargo.toml:1, Cargo.toml:21** · *NEW* + +On `design/swift-binding`, `members = [".", "python", "bench", "chisel-ffi"]` (Cargo.toml:20) and `default-members = [".", "bench"]` (Cargo.toml:31), but the surrounding comments were not updated. The header still says the members "exercise the engine, the PyO3 binding, and the bench harness in one shot" (Cargo.toml:3-4) — three of four — and the `default-members` paragraph opens "`default-members` excludes `python` from `cargo build` / `cargo test` from the workspace root" (Cargo.toml:21-22) and then spends nine lines explaining pyo3's `extension-module` linker behaviour, never mentioning `chisel-ffi`. The consequence is spelled out elsewhere, in `.github/workflows/swift.yml:33-37`: "`cargo test` ... runs plain `cargo test --verbose`, which honors `default-members = [\".\", \"bench\"]` and excludes chisel-ffi. So chisel-ffi's Rust test suite needs its own gate". + +**Why:** A maintainer reading Cargo.toml concludes that only `python` is excluded and that a root `cargo test` therefore covers `chisel-ffi`. It does not — the 15 unit tests in `chisel-ffi/src/{database,types,error}.rs` (including the whole error-mapping totality guard at error.rs:347-453) run only via the single macOS-only `cargo test -p chisel-ffi` step in swift.yml. Anyone running the project's documented local pre-push check would ship an untested FFI crate. + +**Fix:** Add `chisel-ffi` to the `default-members` comment with its own one-line reason (Apple-target staticlib/cdylib crate, gated by swift.yml), and update the three-member list in the header. + +#### `SMELL` SWIFT-12 — Test comment claims a 32-bit-truncation boundary check that the chosen value cannot detect +**chisel-ffi/src/types.rs:327** · *NEW* + +The comment reads: "Exercises the u64 -> usize cast in DefragOptions::from at both ends of its range: 0 (no-op limit) and a large value that only differs from its usize form if the cast silently truncated (e.g. on a 32-bit usize). The prior version of this test used max_values: 42, which isn't a boundary of anything." (types.rs:327-331). The values actually used are `max_values: 0` and `max_values: 1_000_000` (types.rs:337, 344), asserted against `0usize` and `1_000_000usize`. The cast under test is `.max_values(o.max_values as usize)` (types.rs:232). + +**Why:** 1,000,000 is roughly 0.02% of `u32::MAX`, so it round-trips identically through a 32-bit `usize` — the test cannot fail for the reason its comment gives, on any platform. Neither is it "both ends of" a `u64` range. A maintainer trusting the comment believes the truncation path is covered and would not add the check when a 32-bit Apple target (or any other 32-bit consumer of `chisel-ffi`) is reintroduced; the first real truncation would be a silently reduced defrag budget, not an error. + +**Fix:** Either assert the real boundary (`u64::MAX` / `u32::MAX as u64 + 1`) with the platform-appropriate expectation, or rewrite the comment to say what the test actually covers (zero and a representative non-zero value) and drop the truncation claim. + +#### `SMELL` DOCS-COMMENTS-7 — `page.rs` and `python/README.md` both say the MINOR-newer write-refusal gate is deferred/a no-op; it is implemented and forces read-only on open +**src/page.rs:104** + +src/page.rs:104-108 documents `FORMAT_MAJOR_VERSION`/`FORMAT_MINOR_VERSION` with: "Write safety across minors is a separate concern — a binary at minor M opening a file at minor M' > M can read but not safely write without clobbering fields it doesn't know about; this check is deferred until the first 1.1 release (at which point the gate grows a \"newer minor ⇒ refuse writes\" arm). See ISSUES.md I29." python/README.md:423 says the same: "starting with the first post-1.0 minor bump, a binary at MINOR = m opening a file at MINOR = m' > m will be restricted to read-only ... Until 1.1 ships this check is a no-op because no minor variants exist." The gate is live: src/transaction/recovery.rs:421-423 runs `if page::format_minor(sb.format_version) > page::FORMAT_MINOR_VERSION { cache.io_mut().force_read_only(); }`, and `PageIo::force_read_only` (src/page_io.rs:202) exists solely for it ("Used by the I29 format-MINOR write-gate"). The root README already says it is done — README.md:363: "The write-refusal arm (refuse writes when file MINOR > binary MINOR) is implemented (I29)". + +**Why:** Three documents disagree about whether a safety gate exists. A maintainer reading the constants block in page.rs — the natural place to look when bumping `FORMAT_MINOR_VERSION` from 1 to 2 — concludes the write-refusal arm still has to be built and may implement a second, conflicting one, or may bump MINOR believing older binaries will happily write the file. The Python doc likewise tells users a real behaviour (a mutation returning `ReadOnlyModeError` after opening a newer-minor file) "cannot happen yet". + +**Fix:** Update src/page.rs:104-108 to state the gate is implemented in `TransactionManager::open_existing` and that a MINOR-newer file opens read-only rather than being rejected, and correct python/README.md:423 to drop "Until 1.1 ships this check is a no-op". + +#### `SMELL` DOCS-COMMENTS-8 — `DataPage::insert`'s comment claims the transaction layer allocates a fresh page per insert and that the function is "underutilized" — R1 slot packing has been implemented since +**src/data_page.rs:162** + +src/data_page.rs:162-165: "Note (v1 simplification per ARCHITECTURE.md): the transaction layer calls PageCache::new_page() for every insert rather than scanning existing pages for free slots. Intentional, not a bug — this function itself is correct; it's just underutilized." The transaction layer no longer does that. `SlotPacker::insert` (src/transaction/packing.rs:119-134) first tries the packing cursor — `if let Some(cursor_page_id) = self.insert_cursor { let buf = cache.get_mut(cursor_page_id)?; let result = DataPage::insert(buf, value); ... }` — and only allocates when the cursor is absent or full (line 141). ARCHITECTURE.md now documents the opposite of what the comment attributes to it (line 579-583, "Slot packing (R1) means a single data page can hold many small values"). + +**Why:** The comment cites ARCHITECTURE as its authority for a behaviour ARCHITECTURE no longer describes, so a reader cross-checking the two gets a self-consistent-looking but false picture: they would expect one value per data page and would not understand why `SlotPacker` exists, why the cursor must be cleared under savepoints (src/transaction/packing.rs:157-162), or why `release()` decrements a count instead of freeing a page. It is the only in-code statement of the data-page allocation policy, and it describes the pre-R1 engine. + +**Fix:** Replace the note with the current model: inserts pack into the transaction's insert cursor when packing is enabled, falling back to a fresh page when the cursor is full or a savepoint is active; `DataPage::insert` is the per-page primitive that model is built on. + +#### `SMELL` DOCS-COMMENTS-12 — `Stats` is documented as three fields in its own module header, its `#[non_exhaustive]` rationale, ARCHITECTURE, and the README API table — it has five +**src/stats.rs:12** + +`Stats` (src/stats.rs:16-48) has five public fields: `handle_count`, `total_pages`, `file_size_bytes`, `spillway_logical_bytes`, `spillway_max_bytes` — the last two added by I74 and populated at src/lib.rs:860-861. Four doc sites still describe three. src/stats.rs:1-4: "A plain snapshot struct returned by Chisel::stats() for observability: handle count, page count, and raw file size ... so that lib.rs and the public API don't have to pull in transaction.rs just to expose these three numbers." src/stats.rs:12-15: "`#[non_exhaustive]` so adding a fifth summary field (e.g. live-handle/total-handle ratio for retirement pressure) is not a breaking change" — there are already five. ARCHITECTURE.md:130: "`Stats` (`handle_count`, `total_pages`, `file_size_bytes`)". README.md:280: "`stats()` | Handle count, page count, file size (takes `&self`)". + +**Why:** The spillway gauges are the only way to observe spillway pressure before `SpillwayFull` fires, and stats.rs's own field docs (lines 42-47) call that out as the intended operator workflow ("Operators predict `SpillwayFull` by watching `spillway_logical_bytes / spillway_max_bytes` climb across commits"). Every summary-level doc omits them, so an operator reading the README API table or ARCHITECTURE's module table never learns the capability exists. The "adding a fifth" rationale is also self-refuting and would confuse the next person deciding whether a sixth field is breaking. + +**Fix:** Add the two spillway fields to the README API-table entry and ARCHITECTURE's module table, and update src/stats.rs's header and the `#[non_exhaustive]` note to say "five" (or drop the count and phrase it as "a future field"). + +#### `SMELL` DOCS-COMMENTS-14 — `overflow.rs`'s module header and ARCHITECTURE both attribute chain freeing to a `delete()` that does not exist; the real function has no side effects +**src/overflow.rs:40** + +src/overflow.rs:40 states "A chain is reachable from exactly one handle; delete() frees all pages", and lines 46-48: "delete() returns the list of page ids to free; the caller (transaction.rs) folds those into txn_freed_pages". ARCHITECTURE.md:373 repeats it: "Cycle detection in `read`/`delete` bounds the walk by `total_length / OVERFLOW_PAYLOAD` (I14)". `Overflow` exposes only `write`, `read`, and `collect_chain_pages` (src/overflow.rs:82, 152, 247); there is no `delete`. `collect_chain_pages`'s own doc is explicit that it is not the freeing step — src/overflow.rs:238-240: "The caller (transaction layer) is responsible for actually releasing the pages — this function deliberately has no side effect on page state so it's safe to call speculatively and discard the result." + +**Why:** Two adjacent comments in the same file disagree about whether the enumeration function frees anything, and the header's version is the one a reader hits first. Someone adding a new overflow-release path could reasonably call `collect_chain_pages` and assume the pages are now freed, silently leaking the whole chain (the ids never reach `txn_freed_pages`, so `persist_freemap` never marks them free). The stale `delete` name also defeats grep from ARCHITECTURE into the source. + +**Fix:** Rename the references in src/overflow.rs's header and ARCHITECTURE.md:373 to `collect_chain_pages`, and state that it only enumerates — the transaction layer pushes the returned ids onto `txn_freed_pages`, and commit's `persist_freemap` is what actually frees them. + +#### `SMELL` DOCS-COMMENTS-15 — `freemap.rs`'s dead-code note is wrong in both directions: `is_free` has a production caller and `allocate_first` has none +**src/freemap.rs:59** + +src/freemap.rs:59-65: "I35 reshape note: capacity and is_free are reached only from src-tests today (capacity from src/freemap.rs's own tests; is_free from src/transaction.rs's I27/I28 regression tests). The production allocator path uses `allocate_first` + `mark_free`." Both halves are false today. `FreeMap::is_free` is on a production path — `FreeMapTree::is_free` (src/freemap_tree.rs:240-244) wraps it, and `reclaim_freemap_orphans` calls that at src/transaction/freemap.rs:465 (`&& !tree.is_free(cache, id)?`). `FreeMap::allocate_first` has no non-test caller at all: `grep -rn "FreeMap::allocate_first" src/` outside freemap.rs returns nothing; the tree claims a bit with `FreeMap::first_free_bit_from` + `FreeMap::clear_bit` (src/freemap_tree.rs:508 and 437). ARCHITECTURE.md:122 carries the same error: "`freemap.rs` | Single-page bitmap primitive: `allocate_first` / `mark_free` on one `[u8; PAGE_SIZE]` buffer." + +**Why:** The whole `impl FreeMap` block sits under `#[allow(dead_code)]` (src/freemap.rs:66), so the compiler will never correct this note — the comment is the only signal about which primitives are live. A maintainer pruning dead code would trust it and delete `first_free_bit_from`/`clear_bit` (the two that actually run) while keeping `allocate_first` (the one that does not), breaking the freemap tree's claim path. ARCHITECTURE's module table points the same reader at the wrong primitive as "the" allocator. + +**Fix:** Rewrite the note to name the current split — `first_free_bit_from` + `clear_bit` are the tree's find-and-claim pair, `mark_free` is the reclaim primitive, `is_free` backs `FreeMapTree::is_free` in the orphan sweep, and `allocate_first`/`capacity` are test-only leftovers (delete `allocate_first` or say plainly that it is retained unused). Update ARCHITECTURE.md:122 to match. + +#### `SMELL` TXN-COMMIT-5 — The transaction module header claims `new_page()` extends the file immediately; it only bumps an in-memory counter, so next_page_id can exceed the physical file length +**src/transaction/mod.rs:31** · *NEW* + +mod.rs:31-33: "NOTE: `new_page()` (file extension) extends the underlying file immediately". `PageCache::new_page` touches no I/O — it reads `self.next_page_id`, increments it, inserts a zeroed dirty `CacheEntry`, pushes to the LRU, bumps a counter and calls `maybe_evict()` (page_cache.rs:361-374). The file grows only when a page is actually written (`write_sealed` → `io.write_page_unit`, page_cache.rs:444/920). `file_page_count()` reports the PHYSICAL size and page_cache.rs:663-667 says so explicitly. + +**Why:** The comment implies the invariant `file_page_count() >= next_page_id`, which is false for any transaction that has allocated but not yet flushed. Code in scope depends on the opposite: `rollback_to_inner` calls `cache.truncate(watermark)` (savepoints.rs:75) with a watermark taken from `cache.next_page_id()` (freemap.rs:582), and `PageCache::truncate` forwards to `io.set_page_count(n)`, documented as "Truncate (or extend) the file to exactly `n` stride-units" and implemented as `file.set_len(n * stride)` (page_io.rs:525-538). So a `rollback_to` whose savepoint watermark exceeds the physical length silently GROWS the file with zero pages — the exact opposite of what a reader of mod.rs:31 would predict, and something commit's `let total_pages = cache.file_page_count()?` (commit.rs:121) then stamps into the superblock. + +**Fix:** Correct the note to say `new_page()` reserves an id and creates a dirty in-memory page; the file is extended when that page is written (flush, or a spill-then-drain). Add the corollary that `next_page_id` may exceed `file_page_count()` mid-transaction and that `truncate` therefore both shrinks and extends. + +#### `SMELL` TXN-COMMIT-6 — Savepoint.freed_pages doc describes it as scaffolding for a future R2 pass that has already landed, and contradicts savepoints.rs on what rollback_to does with it +**src/transaction/mod.rs:120, src/transaction/savepoints.rs:52** + +mod.rs:120-124: "`freed_pages` is still tracked per-savepoint so a future freemap reclamation pass (R2) can restore freed-but-not-yet-reclaimed pages if a savepoint is rolled back to." R2 has landed — `FreemapRecycle::persist` marks `txn_freed_pages` free in the COW tree at commit (freemap.rs:338-351, called from commit.rs:94). And `rollback_to` restores nothing: it drops the layered savepoints outright (`self.savepoints.truncate(idx + 1)`, savepoints.rs:103) and clears the current list (`self.txn_freed_pages.clear()`, savepoints.rs:104). savepoints.rs:51-57 says the correct thing: "Post-R2, `commit()` DOES return freed pages to the freemap; this rollback path simply discards the unfinished accounting." + +**Why:** The field's two real consumers are `release_inner`, which merges released savepoints' lists back into `txn_freed_pages` (savepoints.rs:136-143), and the I27 flattening at the top of `run_commit` (commit.rs:48-50) that exists precisely because dropping those lists is a permanent freemap leak. A maintainer reading mod.rs:120 would conclude the field is dead speculative state and delete it, silently reintroducing the I27 leak for the commit-with-active-savepoint pattern that tests.rs:501 pins. + +**Fix:** Replace the R2 forward-reference with the field's actual roles: it is the per-scope accumulator that `release` merges upward and that `run_commit` must flatten before `persist` runs, and that `rollback_to` intentionally discards. + +#### `SMELL` FREEMAP-3 — commit.rs claims persist marks the prior commit's deferred structural frees free in the tree, contradicting the two-free-streams invariant +**src/transaction/commit.rs:80-82** · *NEW* + +src/transaction/commit.rs:80-83 describes step 0 as: "persist the freemap tree. This marks `txn_freed_pages` (plus the prior commit's deferred structural frees) free in a COW of the committed tree". `FreemapRecycle::persist` does no such thing — it loops only over the data frees: `for id in txn_freed_pages.iter().copied() { self.mark_free_committed_path(cache, roots, id)?; }` (src/transaction/freemap.rs:347-349). The unit's own header states the inverse as a load-bearing rule: "`structural_superseded` / `pending_structural_frees` / `structural_reuse` (FREEMAP-page frees) ... These are NOT marked free in the tree" (src/transaction/freemap.rs:311-315), and "Reusing a DEAD page (vs. a free bit in the tree) preserves the extend-only TERMINATION guarantee" (src/transaction/freemap.rs:127-129). + +**Why:** This is the comment most likely to produce a real bug. A maintainer who trusts commit.rs and 'restores' the supposedly-missing behavior — adding `for id in &self.pending_structural_frees { mark_free_committed_path(...) }` to `persist` — puts pool pages into the bitmap while they are simultaneously poppable by `structural_extend` (src/transaction/freemap.rs:99). The same page would then be handed out once as a data page by `cow_alloc` and once as a freemap COW target, i.e. a silent double-allocation. It also re-opens the self-reference recursion the extend-only rule exists to prevent (src/freemap_tree.rs:21-27). + +**Fix:** Change commit.rs:80-83 to say persist marks only `txn_freed_pages` free, and that the prior commit's deferred structural frees are CONSUMED as COW targets by `structural_extend`, never marked free in the bitmap. Cross-reference the TWO FREE-STREAMS block. + +#### `SMELL` FREEMAP-5 — FreeMap::allocate_first has no production caller, yet two header comments name it as the production allocator path +**src/freemap.rs:9, src/freemap.rs:62, src/freemap.rs:127** + +src/freemap.rs:8-11 says the allocation paths "prefer `FreeMap::allocate_first` (reusing a page freed by a prior committed transaction)", and the I35 note says "The production allocator path uses `allocate_first` + `mark_free`" (src/freemap.rs:62-63). A crate-wide grep for `FreeMap::` shows `FreeMap::allocate_first` appears only inside src/freemap.rs's own `#[cfg(test)]` mod (lines 220, 222, 224) and in one stale comment (src/freemap.rs:178). The real tree allocator never calls it: `FreeMapTree::allocate_first` finds the id with `FreeMap::first_free_bit_from` (src/freemap_tree.rs:508) and claims it with `FreeMap::clear_bit` (src/freemap_tree.rs:437). The same I35 note claims `is_free` is "reached only from src-tests today" and cites "src/transaction.rs's I27/I28 regression tests" — but src/transaction.rs no longer exists (it is the src/transaction/ module), and `FreeMap::is_free` IS on a production path: `FreeMapTree::is_free` (src/freemap_tree.rs:244) is called by `FreemapRecycle::reclaim_orphans` (src/transaction/freemap.rs:465), the defrag orphan sweep. `FreeMapTree::is_free`'s own doc (src/freemap_tree.rs:236-239) states this correctly, so the two files disagree. + +**Why:** `FreeMap::allocate_first` (and `capacity`) are genuinely dead production code hidden behind a blanket `#[allow(dead_code)]` on the whole impl (src/freemap.rs:66), so the compiler cannot flag them. A maintainer optimizing or hardening 'the production allocator' would work on `allocate_first` and change nothing that runs, while `first_free_bit_from` + `clear_bit` — the code that actually decides which page id is handed out — goes untouched. The inverted claim about `is_free` risks the opposite error: deleting or weakening a predicate the orphan sweep depends on to avoid double-reclaiming a page. + +**Fix:** Fix src/freemap.rs:8-11 and :59-65 to name `first_free_bit_from` + `clear_bit` as the production path and `is_free` as production-reachable via `FreeMapTree::is_free` / `reclaim_orphans`; then either delete `FreeMap::allocate_first` and `capacity` or move the `#[allow(dead_code)]` onto just those two so future dead code is still caught. + +#### `SMELL` FREEMAP-7 — FreemapRecycle::rollback claims every freemap page COW'd this transaction sits above the watermark; pool-reused pages sit below it +**src/transaction/freemap.rs:516, src/transaction/freemap.rs:99** · *NEW* + +`FreemapRecycle::rollback`'s doc justifies clearing the session set with: "any freemap pages this aborted transaction COW'd sit above the watermark and were just truncated, so their ids must not be treated as in-place-mutable next transaction" (src/transaction/freemap.rs:515-517). `structural_extend` prefers a pooled id over extension — `if let Some(id) = structural_reuse.pop() { ... cache.claim_page(id)?; Ok(id) }` (src/transaction/freemap.rs:99-103) — and those ids come from `pending_structural_frees`, i.e. freemap pages superseded by a PRIOR commit, which are below the current watermark by construction. What actually drops them is `discard_all_dirty`, not the truncate: `rollback_inner` runs `cache.discard_all_dirty(); cache.truncate(self.committed_roots.total_pages)?;` (src/transaction/lifecycle.rs:265-266), and the (a)/(b) comment above it says step (a) exists precisely to "catch pages REUSED from the freemap whose id is less than the watermark" (src/transaction/lifecycle.rs:250-253). The same false premise is repeated at src/transaction/lifecycle.rs:294-296. + +**Why:** The action (clearing `session_owned`) is correct; the stated reason is not. A maintainer optimizing rollback who believes the truncate alone suffices for freemap pages could drop or reorder `discard_all_dirty` — which would leave the aborted transaction's freemap-leaf contents dirty at a pooled id, to be flushed by a later commit. The same premise would also read as license to let the savepoint path (`rollback_to_inner`, which calls truncate only) COW the freemap; today that is safe only because `cow_alloc` disables reuse under savepoints (src/transaction/freemap.rs:55, src/page_cache.rs:687-692). + +**Fix:** Reword to: pages EXTENDED this transaction are dropped by the watermark truncate; pages drawn from `structural_reuse` sit below the watermark and are dropped by `discard_all_dirty` — both reasons the session set must be cleared. Fix the mirrored comment at src/transaction/lifecycle.rs:294. + +#### `SMELL` FREEMAP-8 — persist's early-return is justified by a claim that the supersede streams are non-empty only when there were frees; allocation alone fills them +**src/transaction/freemap.rs:336, src/transaction/freemap.rs:61** · *NEW* + +`FreemapRecycle::persist` returns early on `if txn_freed_pages.is_empty()` (src/transaction/freemap.rs:344-346), justified as: "A no-op when nothing was freed (the recycle/supersede streams are only ever non-empty when there were frees, so the single emptiness check suffices)" (src/transaction/freemap.rs:335-337). That premise is false. A transaction that frees nothing but allocates still fills `structural_superseded`: `cow_alloc` calls `tree.allocate_first(cache, hint, &mut extend)` (src/transaction/freemap.rs:61), which COWs the containing leaf via `clear_bit` -> `cow_descend` -> `cow_node`, pushing the old leaf id onto `pending_superseded` (src/freemap_tree.rs:402), which `put_tree` drains into `structural_superseded` (src/transaction/freemap.rs:213-214). defrag.rs's own test comment describes exactly this: "step-5 `update()` calls allocate fresh data pages via `cow_alloc` -> `allocate_first` ... The OLD (pre-COW) leaf id lands in `structural_superseded`" (src/defrag.rs:344-347). + +**Why:** No bug today: the streams are promoted by `FreemapRecycle::commit` (src/transaction/commit.rs:200), which runs unconditionally, so the early return skips nothing that matters. But the comment states a property of the recycle that a future change could lean on — e.g. moving stream promotion into `persist`, or asserting `structural_superseded.is_empty()` when `txn_freed_pages` is empty — and that change would silently drop a commit's dead freemap pages out of the recycle, turning steady-state freemap churn back into unbounded file growth. + +**Fix:** Replace the parenthetical with the true reason: persist only marks DATA frees, so with no frees there is nothing to mark; the structural streams are promoted separately and unconditionally by `FreemapRecycle::commit`. + +#### `NIT` SUPERBLOCK-RECOVERY-9 — Property-test comment claims `PartialEq` is not derived on `Superblock`, but it is, and another test in the same file relies on it +**src/superblock/mod.rs:1199, src/superblock/mod.rs:176, src/superblock/mod.rs:808** · *NEW* + +The proptest body carries the justification "PartialEq isn't derived on Superblock, so compare structurally — easier to diagnose if a single field round-trips wrong." (mod.rs:1199-1201). The struct is declared `#[derive(Debug, Clone, PartialEq, Eq)] pub struct Superblock` (mod.rs:176), and `test_superblock_roundtrip` in the same module does `assert_eq!(sb, sb2);` (mod.rs:808), which only compiles because `PartialEq` IS derived. + +**Why:** The stated reason is false, so the field-by-field comparison looks mandatory rather than a diagnostics preference. The real hazard is the reverse of what the comment implies: the hand-rolled comparison enumerates fields explicitly, so a newly added `Superblock` field is silently omitted from the round-trip property while `assert_eq!` would have caught it — and the comment actively discourages switching to the form that would. + +**Fix:** Correct the comment to say the structural comparison is a deliberate diagnostics choice (PartialEq is available), and note that a new field must be added to the list — or just use `prop_assert_eq!(parsed, sb)` and let the derive cover future fields. + +#### `NIT` PAGE-IO-10 — lru.rs claims 16 bytes per Node; Option has no niche, so Node is 32 bytes +**src/lru.rs:50-51** · *NEW* + +The module doc closes its design justification with: "No `unsafe`. Fits in cache-line-sized state per node (16 bytes for the `Node` plus map overhead)." `Node` is `struct Node { prev: Option, next: Option }`. `Option` has no niche, so each field is 16 bytes and `size_of::()` is 32 (verified by compiling the identical struct). The stated figure is half the real one. + +**Why:** The number is the module's stated reason for choosing map-stored neighbour pointers over boxed nodes, so the memory argument a maintainer would re-derive when revisiting that trade-off starts from a 2x-wrong premise — e.g. sizing an LRU for a given cache budget, or deciding whether NonZero/u32 page-id packing is worth it. + +**Fix:** Correct the figure to 32 bytes, and note that packing the two links as sentinel u64s (or NonZeroU64) would halve it if the footprint ever matters. + +#### `NIT` PUBLIC-API-10 — `Chisel::tag`'s doc still describes the pre-newtype `u32` return ("Returns 0 for untagged handles") +**src/lib.rs:613** + +The doc reads "Return the tag stored in the handle-table entry for `handle`. Returns 0 for untagged handles." (src/lib.rs:613-614) while the signature is `pub fn tag(&self, handle: Handle) -> Result>` (src/lib.rs:618) and the body maps stored 0 to `None` via `Tag::new` — `Tag(0)` is unconstructable by design (src/handle.rs:88-105). + +**Why:** A reader writing against this method from the rendered rustdoc looks for a zero sentinel that the type system has deliberately made unrepresentable, and the neighbouring README table (README.md:270) repeats the same stale claim — so the wrong model is reinforced in two places. + +**Fix:** Reword to "Returns `None` for untagged handles" and update the matching README API-table row. + +#### `NIT` TESTS-CI-12 — create_tests.rs module doc lists a cleartext-leak check that no test in the file performs +**src/transaction/create_tests.rs:7, src/transaction/create_tests.rs:140** · *NEW* + +The module header enumerates the file's scope, including "- sensitive fields (named_roots names) are NOT in cleartext" (create_tests.rs:7). The only test that could do so, `create_encrypted_db_sealed_body_is_present`, explicitly disclaims it at create_tests.rs:140-142: "Bytes 52..308 (plaintext named_roots) ARE intentionally zeroed by serialize_encrypted ... We do NOT check those here." It checks only that the 24-byte nonce region is non-zero. + +**Why:** A reader auditing encryption coverage reads the header, ticks off "cleartext leak — covered", and moves on. The property is in fact covered, but by `encrypted_named_root_name_absent_from_cleartext` in src/superblock/mod.rs:1081 — a file the header does not mention. If that superblock test were ever deleted, the header would still claim coverage that no longer exists anywhere. + +**Fix:** Change the bullet to point at src/superblock/mod.rs:1081 ("cleartext-leak coverage lives in superblock::tests::encrypted_named_root_name_absent_from_cleartext") rather than claiming it as this file's scope. + +#### `NIT` DOCS-COMMENTS-17 — `txn_freed_pages`'s field comment names a `current_freemap` field that no longer exists +**src/transaction/mod.rs:181** + +src/transaction/mod.rs:180-182 documents the field as: "Pages whose contents are no longer reachable from the new roots. Merged into `current_freemap` at commit time so subsequent transactions can reuse the space (ISSUES.md I9 / I10 / I11 / R2)." There is no `current_freemap` field on `TransactionManager` (the struct's fields are listed at src/transaction/mod.rs:146-237); the committed freemap is `{freemap_page, freemap_depth}` inside `Roots` (lines 99-103), reconstructed on demand, and the merge is done by `FreemapRecycle::persist` (called from src/transaction/commit.rs:94-95), with commit.rs:185-187 spelling it out: "The committed freemap tree advances automatically: its {root, depth} ride in current_roots ... No separate in-memory freemap copy to advance." + +**Why:** A reader grepping for `current_freemap` to understand where freed pages land finds nothing, and the phrasing implies an in-memory freemap mirror that the design deliberately removed — the same misconception ARCHITECTURE warns against in its freemap section. + +**Fix:** Reword to: merged into the COW freemap tree by `FreemapRecycle::persist` during commit, which updates `current_roots.{freemap_page, freemap_depth}`. + +#### `NIT` TXN-COMMIT-9 — txn_freed_pages field comment names a `current_freemap` field that no longer exists +**src/transaction/mod.rs:181** · *NEW* + +mod.rs:180-182: "Pages whose contents are no longer reachable from the new roots. Merged into `current_freemap` at commit time so subsequent transactions can reuse the space". There is no `current_freemap` anywhere in src/ — the committed freemap is `{freemap_page, freemap_depth}` inside `Roots` (mod.rs:96-103) plus the `FreemapRecycle` in the `freemap` field (mod.rs:194), and the merge happens in `FreemapRecycle::persist` (freemap.rs:338). + +**Why:** A reader grepping for `current_freemap` to understand where frees land finds nothing and has to reverse-engineer the commit path from commit.rs:94 instead. Pure comment rot left over from the pre-tree single-page freemap. + +**Fix:** Point the comment at `FreemapRecycle::persist` and the `Roots.{freemap_page, freemap_depth}` pair. + +### cargo-hygiene (1) + +#### `SMELL` GAP-2 — The published Python distribution declares MIT but ships no license text — the LICENSE file lives at the repo root, outside the sdist/wheel build context +**python/pyproject.toml:11, python/pyproject.toml:30, LICENSE:1, .github/workflows/wheels.yml:113** · *unverified* + +`python/pyproject.toml:11` declares `license = { text = "MIT" }`, and `[tool.maturin] include` at :30-32 lists exactly two extra files — `chisel/py.typed` and `chisel/chisel.pyi`. There is no `license-files` key and no LICENSE file anywhere under `python/` (the only copy is the repo-root `LICENSE`). The wheels workflow builds the sdist with `working-directory: python` (.github/workflows/wheels.yml:112-115) and builds wheels with `package-dir: python`, so the root LICENSE is never in scope for either artifact. + +**Why:** Every wheel and sdist published to PyPI carries the metadata string "MIT" with no accompanying license text, which fails the MIT license's own condition that "the above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software" (LICENSE:10-11). Downstream vendoring and corporate license-scanning tooling that reads the distribution rather than the GitHub repo sees an unlicensed artifact. Nothing in CI checks this; the Rust crate is unaffected because cargo picks up the root LICENSE automatically. + +**Fix:** Add a copy (or symlink resolved at build time) of LICENSE under `python/` and reference it from `[tool.maturin] include` and/or `[project] license-files`, so both the wheel and the sdist carry the text. + +### idiomaticity (1) + +#### `SMELL` CRYPTO-7 — The slot wrap and slot unwrap-trial logic each exist twice, and the doc asserting the two unwrap copies are "byte-identical" is unverified — the wrap copies already diverge +**src/superblock/crypto_header.rs:185, src/superblock/crypto_header.rs:192, src/transaction/recovery.rs:637, src/transaction/recovery.rs:571** + +`CryptoHeader::unlock` (crypto_header.rs:192-224) and the free function `unwrap_first_matching_slot` (recovery.rs:637-665) implement the same trial loop. `unlock`'s doc even says so: "This is byte-identical to the inline trial in `recovery.rs` (`unwrap_first_matching_slot`) — both call `slot.aad()` on the fully-populated slot before passing it to `unwrap_dek`" (crypto_header.rs:185-187). They are not textually identical — `unlock` maps kdf ids via `x if x == KdfId::Hkdf as u8` (crypto_header.rs:205-207) while `unwrap_first_matching_slot` hardcodes `1 =>` / `2 =>` (recovery.rs:645-646). The wrap side is likewise duplicated: `CryptoHeader::wrap_into` (crypto_header.rs:236-276) and `build_create_cipher` (recovery.rs:571-622) both derive a KEK, populate a slot, and call `wrap_dek`. Only the open path uses `unwrap_first_matching_slot` (recovery.rs:342); only the key-management path uses `unlock` (keys.rs:146, 176, 209). + +**Why:** The duplication has already produced a real divergence — the argon2-params discrepancy in CRYPTO-6 exists precisely because `build_create_cipher` and `wrap_into` are separate implementations of the same operation. A comment asserting equivalence is not a mechanism that enforces it: any future change to the AAD layout, the kdf-id mapping, or the slot-skipping policy must now be made in two places on each side, with nothing failing if only one is updated. This is the highest-consequence code in the crate to have four hand-maintained near-copies of. + +**Fix:** Delete `unwrap_first_matching_slot` and have `open_existing` call `header.unlock(k)` (discarding the index); delete the inline wrap in `build_create_cipher` and have it construct an empty `CryptoHeader` and call `wrap_into(0, key, &dek)`, threading the `argon2_override` through a parameter on `wrap_into`. That also closes CRYPTO-6 and gives `add_key`/`rotate_key` the cost-parameter control they currently lack. + +--- + +## Refuted during verification + +- **CRYPTO-3** — `KeySlot::aad` claims it prevents a slot being "transplanted between DBs"; the AAD contains no database-identifying bytes, so transplantation authenticates fine + Refuted: The AAD contents are as described (crypto_header.rs:81-91 carries only slot-local metadata), but the comment is a defensible reading and the claimed consequence does not hold. The comment's stated mechanism — 'Binds the wrap to its salt/params/nonce' — is TRUE and is what the AAD actually achieves: you cannot lift `wrapped_dek`/`wrap_tag` out of one slot and splice them into a record carrying a different salt, nonce, kdf_id, or cost params without breaking the Poly1305 tag. Read that way, 'so a slot can't be transplanted' is about the wrapped-DEK bytes, not a claim of a database identifier. And the whole-record copy the finding describes is already dead on arrival independent of the AAD: unwrapping DB A's transplanted slot inside DB B yields A's DEK, which then fails `sb.decrypt_body` at recovery.rs:348 (AAD = B's `sb_identity_aad`) and is mapped to `InvalidEncryptionKey`. The finding itself concedes there is no practical fallout. No code or comment change is warranted here; CRYPTO-2 already covers the one genuinely false claim in this area. + +--- + +## Repository hygiene (verified by me directly) + +- **`swift/` is 974 MB of untracked, unignored build output.** 24 files outside `.build/`, and **not one of them is a `.swift` source** — they are `.o`, `.d`, `.swiftdeps`, and `Chisel.xcframework` (three static libs). The binding's real sources are on the `design/swift-binding` branch, not here. Nothing is at risk of being lost; the hazard is committing it. + *(An earlier draft of this review described these as 24 source files of finished work. That was wrong — corrected here. The SWIFT-2 finding had it right.)* +- **Nothing gitignores it.** `bench/target`, `python/target` and `python/.venv` are all ignored, but only because Cargo and uv write a `.gitignore` *inside* those directories. SwiftPM writes no such file, so `swift/` is exposed — and `git add swift/`, the natural response to `?? swift/`, commits the lot. +- **`README.md` is not wired into any doctest.** No `include_str!("../README.md")` anywhere in `src/`, so its Rust examples are never compiled. Several of them do not compile today (see DOCS-COMMENTS-13). + +## Open questions + +- **Is GAP-1 exploitable into corruption, or merely untidy?** That needs a trace of how `structural_superseded`/`structural_reuse` are consumed by `persist` at commit after a `rollback_to`. I confirmed the asymmetry but not the consequence. +- **Page-level rollback resistance.** `PageCipher::seal` uses AAD = `page_id` alone (`src/crypto/mod.rs:323`), giving anti-relocation but not anti-rollback: an attacker with file access can substitute an *older* sealed image of the same page id and it authenticates. Whether that is in the threat model is a design call — the spec should state it either way. +- **Is the `swift/` directory intentionally out of tree** (vendored elsewhere, awaiting the `origin/main` orphan-history situation), or an oversight? +- **Was `create_if_missing: false` ever intended to prevent creation over a corrupt file**, or only over a missing one? The docstring at `src/lib.rs:332-337` describes the zero-length case but not the sub-page case. + +## What was not reviewed + +- `ISSUES.md` and `docs/reviews/` — deliberately withheld from the reviewers to keep the pass clean-slate; read afterwards only to compute the delta section. +- `docs/specs/` and `docs/plans/` — design intent was taken from `ARCHITECTURE.md`/`README.md`/`THEORY.md`, which are the documents that ship. +- Third-party dependency source, except `argon2 0.5.3`, which was read to settle the `m_cost` ceiling question. +- `target/`, `bench/target/`, `python/.venv/`, `swift/.build/` — build artifacts. +- No fuzzing, no miri, no sanitizer run, no coverage measurement. The `unsafe` audit was by reading, not by tooling. From 6ee42142142b5e71a2cebcde1371b4444b76259c Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Tue, 4 Aug 2026 08:04:38 -0700 Subject: [PATCH 2/4] docs(adr): bring the decision log onto the published lineage The 17 architecture decision records existed only on design/swift-binding, which descends from the pre-release development lineage. That lineage shares no history with origin/main, so nothing on it can reach the published repository through a pull request, and the decision log has been invisible from every branch that can. Copied verbatim, no edits: these are Accepted records and their bodies are frozen. A path checkout is used rather than a merge precisely because the two lineages have no common ancestor. The Swift binding itself remains stranded on that branch and still needs a route onto this lineage. --- docs/adr/0000-decision-register-overview.md | 32 ++++++++ docs/adr/0001-shadow-paging-not-wal.md | 27 +++++++ ...0002-single-writer-enforced-by-mut-self.md | 27 +++++++ ...r-module-cow-no-centralized-abstraction.md | 26 ++++++ ...-rotating-superblocks-for-atomic-commit.md | 27 +++++++ ...spillway-sidecar-file-over-hard-ceiling.md | 31 ++++++++ docs/adr/0006-poison-model-on-fatal-errors.md | 29 +++++++ docs/adr/0007-two-tier-format-versioning.md | 41 ++++++++++ docs/adr/0008-in-memory-mode.md | 28 +++++++ ...ter-instrumentation-via-chisel-counters.md | 34 ++++++++ ...e-series-cross-engine-dedicated-machine.md | 40 ++++++++++ ...-fsync-fairness-via-pragma-fullfsync-on.md | 29 +++++++ ...012-chunk-tags-reverse-membership-index.md | 33 ++++++++ ...in-session-iteration-stability-contract.md | 31 ++++++++ ...e-spending-the-last-reserved-entry-byte.md | 60 ++++++++++++++ ...yption-xchacha20-poly1305-envelope-keys.md | 41 ++++++++++ docs/adr/0016-swift-binding-via-uniffi.md | 79 +++++++++++++++++++ docs/adr/README.md | 23 ++++++ 18 files changed, 638 insertions(+) create mode 100644 docs/adr/0000-decision-register-overview.md create mode 100644 docs/adr/0001-shadow-paging-not-wal.md create mode 100644 docs/adr/0002-single-writer-enforced-by-mut-self.md create mode 100644 docs/adr/0003-per-module-cow-no-centralized-abstraction.md create mode 100644 docs/adr/0004-n-rotating-superblocks-for-atomic-commit.md create mode 100644 docs/adr/0005-spillway-sidecar-file-over-hard-ceiling.md create mode 100644 docs/adr/0006-poison-model-on-fatal-errors.md create mode 100644 docs/adr/0007-two-tier-format-versioning.md create mode 100644 docs/adr/0008-in-memory-mode.md create mode 100644 docs/adr/0009-counter-instrumentation-via-chisel-counters.md create mode 100644 docs/adr/0010-bench-suite-series-cross-engine-dedicated-machine.md create mode 100644 docs/adr/0011-macos-fsync-fairness-via-pragma-fullfsync-on.md create mode 100644 docs/adr/0012-chunk-tags-reverse-membership-index.md create mode 100644 docs/adr/0013-within-session-iteration-stability-contract.md create mode 100644 docs/adr/0014-client-byte-spending-the-last-reserved-entry-byte.md create mode 100644 docs/adr/0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md create mode 100644 docs/adr/0016-swift-binding-via-uniffi.md create mode 100644 docs/adr/README.md diff --git a/docs/adr/0000-decision-register-overview.md b/docs/adr/0000-decision-register-overview.md new file mode 100644 index 0000000..458c974 --- /dev/null +++ b/docs/adr/0000-decision-register-overview.md @@ -0,0 +1,32 @@ +--- +id: 0000 +title: Decision register (overview) +date: 2026-05-04 +status: Accepted +--- + +# 0000. Decision register (overview) + +Chisel is a single-writer embedded transactional storage engine in Rust. The decisions below are the ones that, if reversed, would require rewriting substantial parts of the engine. Smaller decisions (specific bit layouts, error message wording, individual issue resolutions) live in `ISSUES.md`. + +| # | Decision | Status | Reversibility | +|---|---|---|---| +| 1 | Shadow paging, not WAL | Accepted | Hard — touches commit protocol, recovery, every page-mutation path | +| 2 | Single-writer enforced by `&mut self` | Accepted | Hard — every API signature would change | +| 3 | Per-module COW (no centralized abstraction) | Accepted | Medium — affects 5 modules | +| 4 | N rotating superblocks (configurable 2..=16) | Accepted | Hard — recovery and commit both depend | +| 5 | Spillway sidecar file over hard ceiling | Accepted (2026-05-04) | Medium — supersedes `HARD_CEILING_MULTIPLIER` | +| 6 | Poison model on fatal errors | Accepted | Easy — could be relaxed, but Linux fsyncgate semantics make retry unsafe regardless | +| 7 | Two-tier format versioning (file MAJOR/MINOR + per-page byte) | Accepted | Hard — affects every page header | +| 8 | In-memory mode via `Vec`-backed PageIo | Accepted | Easy — additive; could be removed | +| 9 | Counter instrumentation via `Chisel::counters()` | Accepted (PR 1, bench-suite) | Easy — additive, `#[non_exhaustive]` | +| 10 | Bench-suite series (cross-engine comparison + dedicated machine foundation) | Accepted (PRs 1-8 shipped 2026-04-30 → 2026-05-04) | Easy — bench/ is a sibling crate; no engine impact | +| 11 | macOS-fsync fairness via `PRAGMA fullfsync=ON` on SqliteEngine | Accepted (PR 8, 2026-05-04) | Easy — bench-side only | +| 12 | Chunk tags + reverse membership index | Accepted (2026-06-02) | Medium — new on-disk subsystem; additive format (MINOR) | +| 13 | Within-session iteration-stability contract | Accepted (2026-06-04) | Easy — documents existing behavior; public API contract | +| 14 | Client byte — opaque per-chunk u8 in the last reserved entry byte | Accepted (2026-06-05) | Easy — additive; reuses reserved byte [15], no format change | +| 15 | On-disk encryption (XChaCha20-Poly1305, envelope DEK/KEK, MAJOR=2) | Accepted (2026-06-30) | Hard — first MAJOR format bump; encrypted-stride + sealed superblock + page-I/O seal seam | + +The body of this ADR walks each decision in turn. + +--- diff --git a/docs/adr/0001-shadow-paging-not-wal.md b/docs/adr/0001-shadow-paging-not-wal.md new file mode 100644 index 0000000..9e3527a --- /dev/null +++ b/docs/adr/0001-shadow-paging-not-wal.md @@ -0,0 +1,27 @@ +--- +id: 0001 +title: Shadow paging, not WAL +date: unknown +status: Accepted +--- + +# 0001. Shadow paging, not WAL + +**Context:** Two dominant approaches to ACID durability exist for embedded engines. Write-ahead log (WAL) writes intent records to a sequential journal first, then applies changes to the data file in place; recovery replays unfinished log entries. Shadow paging writes new versions of mutated pages to fresh page slots, leaves old pages intact, and atomically swaps a root pointer to make the new state visible; recovery picks the most recent valid root. + +**Decision:** Shadow paging. Every mutation allocates a fresh page via `PageCache::new_page`; the previously-committed page stays intact at its original position. Commit writes the new pages' bytes, fsyncs, then writes a new superblock to a different slot than the currently-active one and fsyncs again. Recovery on open is `Superblock::select` over the N candidate slots, picking the one with the highest valid `txn_counter`. + +**Alternatives considered:** + +- *WAL with in-place updates.* Standard for production-grade DBs (PostgreSQL, SQLite). Rejected for v1: WAL recovery is a substantial subsystem (replay state machine, checkpoint handling, log truncation) that adds risk surface comparable to the entire rest of Chisel. Shadow paging trades disk space (live + previous version of every mutated page until commit) for code simplicity. +- *Hybrid (WAL for small writes, shadow for large).* Considered briefly, rejected as combining the worst of both — recovery code paths multiply and the boundary between modes becomes another correctness obligation. + +**Consequences:** + +- *Positive:* No log replay; recovery is one read of N superblock slots plus checksum validation. The "is this database open" check is the same code path as crash recovery. Crash safety is provable by inspection: any state where the previous superblock is intact remains recoverable, and `fsync` ordering ensures the new superblock isn't durable until its referenced data pages are. +- *Positive:* COW is a natural fit. Every mutation produces a new page; transactions are simply "the set of new pages plus a candidate new superblock." Rollback is "discard the new pages and the new superblock." +- *Negative:* Disk space cost. Updating a single byte of a page costs an entire new page (8 KB) until the next commit, when the old page becomes freeable. Workloads that write small deltas to many pages have high write amplification. +- *Negative:* Defragmentation becomes necessary over time. `defrag.rs` exists for this. +- *Locked-in:* Reverting to WAL would require rewriting `transaction.rs`, `page_cache.rs` (no more "fresh page per mutation"), and the recovery path in `lib.rs`. + +--- diff --git a/docs/adr/0002-single-writer-enforced-by-mut-self.md b/docs/adr/0002-single-writer-enforced-by-mut-self.md new file mode 100644 index 0000000..1decfa4 --- /dev/null +++ b/docs/adr/0002-single-writer-enforced-by-mut-self.md @@ -0,0 +1,27 @@ +--- +id: 0002 +title: Single-writer enforced by `&mut self` +date: unknown +status: Accepted +--- + +# 0002. Single-writer enforced by `&mut self` + +**Context:** Embedded databases face a choice: single-writer (one mutator at a time, often with multiple concurrent readers) vs. multi-writer (transactions interleave, requiring locking, MVCC, or both). The choice affects the API surface, the storage format (MVCC needs version chains), the recovery model, and the testing burden. + +**Decision:** Single-writer, single-process. Enforced at three levels: (a) the OS via exclusive `flock` in `page_io.rs`, (b) the type system via `&mut self` on every mutating Chisel API, (c) explicit project-memory note that this is *philosophical*, not a v1 simplification. + +**Alternatives considered:** + +- *Multi-writer with internal locking.* Would require RwLock or Mutex around `PageCache`, transaction-conflict detection, deadlock handling. Roughly doubles the engine's complexity. +- *MVCC.* Adds version chains to every page, garbage-collection responsibilities, snapshot-isolation semantics. Out of scope for an embedded single-process engine. +- *Single-writer at v1, multi-writer at v2.* Rejected because the `&mut self` API is load-bearing — relaxing it later would be a breaking change for every consumer, and the type system encodes the invariant in a way internal locking cannot. + +**Consequences:** + +- *Positive:* No internal locking. `RefCell` (not `Mutex`) inside `TransactionManager` lets `read()` / `handles()` / `stats()` take `&self` without external wrapping; the borrow checker handles the rest. +- *Positive:* The type system makes "two concurrent transactions" impossible to express. There is no test for it because there is no API for it. +- *Negative:* Workloads that need concurrent writers must serialize at a higher layer (e.g., Exilis does this via its own `RefCell` inside the storage backend). +- *Locked-in:* See above. Multi-writer would be a v2.0 breaking change, not a minor. + +--- diff --git a/docs/adr/0003-per-module-cow-no-centralized-abstraction.md b/docs/adr/0003-per-module-cow-no-centralized-abstraction.md new file mode 100644 index 0000000..a5ab5e6 --- /dev/null +++ b/docs/adr/0003-per-module-cow-no-centralized-abstraction.md @@ -0,0 +1,26 @@ +--- +id: 0003 +title: Per-module COW, no centralized abstraction +date: unknown +status: Accepted +--- + +# 0003. Per-module COW, no centralized abstraction + +**Context:** Multiple modules need copy-on-write semantics: the handle table (radix tree) must clone the path from root to a modified leaf, the freemap rewrites itself on every commit's `persist_freemap`, data pages reuse the same page across commits via `claim_page`. A centralized COW abstraction (a trait, a generic page-mutation type) would seem to factor out repeated logic. + +**Decision:** Each module implements its own COW. `handle_table.rs` clones the root-to-leaf path. `freemap.rs` allocates a new freemap page in `persist_freemap`. `data_page.rs` mutates in place via `claim_page` (which takes `&mut PageBytes` from `PageCache::write_page`). No `trait Cow` or `enum CowStrategy`. + +**Alternatives considered:** + +- *Centralized `trait Cow` over all page-type modules.* Would require uniform interface (e.g., `fn cow_root(&mut self, cache, root_id) -> Result`). Rejected because the modules' actual COW shapes differ enough that the trait would either be too generic (lose useful information) or too specific (have variants that work for only one module). +- *Generic page-mutation type that wraps a strategy.* Same problem as above plus the ergonomic cost of generics in the public API. + +**Consequences:** + +- *Positive:* Each module's COW logic is co-located with the page-type logic it serves. Reading `handle_table.rs` shows you both the radix tree algorithm and the COW it implements. +- *Positive:* Freedom to evolve. The handle table's COW grew several optimizations (`grow()`, short-circuit at depth boundaries) without affecting other modules. +- *Negative:* Repeated boilerplate across 3-4 modules. Each writes its own "allocate a new page, write the new state, return the new page ID." +- *Negative:* Onboarding cost. A new contributor wonders "where is the COW abstraction?" and the answer is "there isn't one, and that's deliberate." + +--- diff --git a/docs/adr/0004-n-rotating-superblocks-for-atomic-commit.md b/docs/adr/0004-n-rotating-superblocks-for-atomic-commit.md new file mode 100644 index 0000000..02ae340 --- /dev/null +++ b/docs/adr/0004-n-rotating-superblocks-for-atomic-commit.md @@ -0,0 +1,27 @@ +--- +id: 0004 +title: N rotating superblocks for atomic commit +date: unknown +status: Accepted +--- + +# 0004. N rotating superblocks for atomic commit + +**Context:** The commit protocol's atomicity hinges on swapping a root pointer in a single durable write. The simplest implementation is a single superblock at offset 0, overwritten on every commit. But a single superblock is vulnerable: a torn write (kernel buffered the new bytes but crashed before all of them reached disk) leaves the file unrecoverable. + +**Decision:** N superblocks (configurable at create time via `Options::superblock_count`, range 2..=16, default 2) occupy file offsets 0..N. Commit writes to slot `txn_counter % N` — always the slot with the lowest `txn_counter` among the surviving N. Recovery (`Superblock::select`) reads all N slots, validates each (magic + checksum + `superblock_count` in range), and picks the highest valid `txn_counter`. + +**Alternatives considered:** + +- *Single superblock with double-write buffer.* PostgreSQL-style approach (every page is written twice, once to a buffer area and once in place). Rejected because shadow paging already provides the same guarantee for data pages — the only page that needs the double-write is the superblock itself, and N rotating slots is conceptually simpler than maintaining a separate buffer area. +- *Write-side journal for the superblock.* Mini-WAL just for the superblock. Rejected: same complexity argument as ADR-1. + +**Consequences:** + +- *Positive:* Trivial torn-write recovery. A torn slot fails the checksum; `Superblock::select` ignores it and picks the previous slot. Higher N (3..16) survives consecutive torn writes. +- *Positive:* No separate journal. The superblock's own slots ARE the journal. +- *Positive:* Configurable space/durability tradeoff. The user picks N at create time based on their crash tolerance. +- *Negative:* N pages of overhead at the start of every file. With default N=2 and 8 KB pages, that's 16 KB minimum file size before any data. +- *Locked-in:* The on-disk layout reserves the first N pages for superblocks; changing N for an existing file would require migration. + +--- diff --git a/docs/adr/0005-spillway-sidecar-file-over-hard-ceiling.md b/docs/adr/0005-spillway-sidecar-file-over-hard-ceiling.md new file mode 100644 index 0000000..6b7870a --- /dev/null +++ b/docs/adr/0005-spillway-sidecar-file-over-hard-ceiling.md @@ -0,0 +1,31 @@ +--- +id: 0005 +title: Spillway sidecar file over hard ceiling +date: 2026-05-04 +status: Accepted +--- + +# 0005. Spillway sidecar file over hard ceiling + +**Context:** The page cache (`page_cache.rs`) has a strict size cap (`Options::cache_max_bytes`). When every cached entry is dirty (no clean page available for eviction), the cache cannot accept a new dirty page without violating its bound. The pre-2026-05-04 design used a `HARD_CEILING_MULTIPLIER = 8` to allow temporary growth past `cache_max_bytes` up to 8× the configured limit, then errored with `CacheFull`. This worked for moderate transactions but failed silently on workloads that legitimately needed to dirty more than 8× the cache: notably, the bench-suite scenarios with `document-store`'s log-normal value sizes. + +**Decision:** Replaced the `HARD_CEILING_MULTIPLIER` elasticity with a *spillway*: a sidecar file `.spillway` that absorbs LRU-tail dirty pages when the cache is full of dirty pages. The cache becomes a strict bound (no elasticity). The spillway is bounded by `Options::spillway_max_bytes` (default `1024 × cache_max_bytes` = 8 GiB at the 8 MiB cache default). Spillway slots carry their own per-slot XXH3 checksum over `page_id || page_bytes`, distinct from the main-file page checksum. The spillway is never `fsync`ed — its content does not need to survive a crash; it's truncated at open and at every commit/rollback. Setting `Options::spillway_max_bytes = 0` disables the spillway and restores `CacheFull`-at-cap semantics. + +**Alternatives considered:** + +- *Keep the 8× ceiling, raise the multiplier.* Would just delay the same problem. Bench-suite document-store workloads can dirty 100× the cache. +- *No bound at all (unbounded growth).* Rejected — embedded engines must respect their configured memory budget, and "unbounded dirty" is an OOM path under sustained write workloads. +- *Disk-spill into the main file using a reserved area.* Adds a permanent on-disk artifact that has to be checksum-protected and durability-managed. Rejected because the spillway's contents are by definition uncommitted; never needing to fsync them is a key simplification. + +**Consequences:** + +- *Positive:* Bench-suite workloads that exceed cache size now succeed. `document-store` was the motivating case. +- *Positive:* Strict cache bound. The `cache_max_bytes` budget is now actually a budget, not a guideline. +- *Positive:* The spillway's "never fsync" policy is correct because its contents are uncommitted dirty state. A crash with a non-empty spillway just discards its contents on the next open — the previous committed superblock is still active. +- *Negative:* The no-spill commit cost is now 3 fsyncs (I28 pre-drain + main-pages flush + superblock), not 2. The pre-drain handles a subtle interaction in the commit protocol: `persist_freemap`'s `allocate_data_page` could trip `maybe_evict`'s spill-or-error path mid-commit if every cached page is dirty; pre-draining clears every dirty pin so the strict cap is reachable via normal eviction. +- *Negative:* New error variant `SpillwayFull { limit_bytes }` fires when both cache and spillway are exhausted. Operational, recoverable via commit/rollback. +- *Breaking change:* `Options::cache_size: usize` (page count) → `Options::cache_max_bytes: u64` (bytes); default unchanged at 8 MiB. + +Spec: `docs/superpowers/specs/2026-05-03-chisel-spillway-design.md` (frozen at decision time). **Update (2026-06-30):** for encrypted databases the spillway slot widens to carry the 8232-byte sealed on-disk unit and stores *ciphertext* (seal-once on evict, verbatim copy on drain); its per-slot XXH3 still guards the round-trip. See ADR-15. + +--- diff --git a/docs/adr/0006-poison-model-on-fatal-errors.md b/docs/adr/0006-poison-model-on-fatal-errors.md new file mode 100644 index 0000000..2752040 --- /dev/null +++ b/docs/adr/0006-poison-model-on-fatal-errors.md @@ -0,0 +1,29 @@ +--- +id: 0006 +title: Poison model on fatal errors +date: unknown +status: Accepted +--- + +# 0006. Poison model on fatal errors + +**Context:** A failed `fsync()` on Linux post-2018 (the "fsyncgate" period) cannot be safely retried — the kernel may have already discarded the dirty pages, and a subsequent successful `fsync()` does NOT mean the earlier data is durable. Similarly, a checksum mismatch on page load means the on-disk state is corrupt and the in-memory reconstruction may be inconsistent with what's actually written. Continuing to use a `TransactionManager` after either condition risks reading torn or wrong state. + +**Decision:** Any fatal error (commit-path `IoError`, `ChecksumMismatch`, `CorruptSuperblock`, etc. — see `ChiselError::is_fatal()` for the full list) sets a poison flag on `TransactionManager`. Every subsequent call returns `ChiselError::Poisoned`, including reads. The only legal recovery is to drop the `Chisel` handle and call `Chisel::open` again; the shadow-paging recovery path (`Superblock::select`) returns the database to its last-durable state. + +**Alternatives considered:** + +- *Retry the failed fsync.* Rejected per Linux fsyncgate semantics. +- *Restrict poisoning to writes; allow reads to continue.* Rejected because reads share the page cache with writes, and a corrupt page may have already been served to a previous read; we can't know which reads are tainted. +- *Auto-reopen on poison.* Rejected because reopen is a substantial state transition (file descriptors, locks, cache contents) that the caller must orchestrate. Forcing the caller to do it makes the recovery boundary explicit. + +**Consequences:** + +- *Positive:* Mirrors `std::sync::Mutex` poisoning — Rust developers will recognize the pattern. +- *Positive:* The recovery idiom (drop + reopen) exercises the same code path as crash recovery, which has the side benefit of testing the recovery path on every real-world poison event. +- *Positive:* No retry logic in the commit protocol. The protocol assumes happy-path fsync semantics; failure means stop. +- *Negative:* A poisoned `TransactionManager` is a permanent dead state. The caller MUST handle this — long-running services need a "drop and reopen on `Poisoned`" wrapper. + +See `ISSUES.md` I1 for the full design and the project-memory note `project_chisel_i1_poison_decision`. **Update (2026-06-30):** the encryption feature extends the model consistently — `DecryptionFailed { page_id }` (a data-page AEAD auth failure after a valid open = tamper/corruption) is fatal/poisoning; a wrong or missing key at open (`InvalidEncryptionKey` / `NoEncryptionKey` / `EncryptionNotSupported`) and the key-management refusals (`NoFreeKeySlot` / `LastKeySlot`) are operational/retryable. The metadata-only `rewrite_crypto_header` rotation commit poisons on fsync failure exactly like `commit`. See ADR-15. + +--- diff --git a/docs/adr/0007-two-tier-format-versioning.md b/docs/adr/0007-two-tier-format-versioning.md new file mode 100644 index 0000000..f9e8d51 --- /dev/null +++ b/docs/adr/0007-two-tier-format-versioning.md @@ -0,0 +1,41 @@ +--- +id: 0007 +title: Two-tier format versioning +date: unknown +status: Accepted +--- + +# 0007. Two-tier format versioning + +**Context:** On-disk format compatibility is a long-term promise. Binaries from any version should refuse to open files written by an incompatible binary, and ideally accept files written by any compatible binary without forcing migration. But "compatibility" has two granularities: file-level (does the high-level layout match?) and page-level (do individual page layouts match?). Conflating them forces a file-wide format bump for any per-page change. + +**Decision:** Two-tier versioning. (a) The superblock carries a packed `format_version: u32` — upper 16 bits MAJOR, lower 16 bits MINOR. Open-time gate compares MAJOR only; same-major files open regardless of minor; different-major is rejected with `UnsupportedFormatVersion`. (b) Every non-superblock page carries a one-byte `page_format_version` in its header. This lets individual page layouts evolve within a major without a file-wide bump. Today every page reports version 0; future minor changes to a page-type's layout will bump that page-type's version while leaving others alone. + +**Alternatives considered:** + +- *File-level only.* Standard approach; works but every per-page layout tweak forces a file-wide migration. Rejected because Chisel has 4 page types (handle table interior/leaf, data, overflow, freemap) and they're likely to evolve at different rates. +- *Page-level only (no file-level gate).* Would lose the "completely incompatible binary, refuse to open" check. The MAJOR check provides a clear "this binary cannot read this file" failure mode. +- *Schema migration system.* Out of scope for embedded; would require migration scripts, version-jump testing, etc. + +**Consequences:** + +- *Positive:* The README's "sacred within a major version" promise is enforceable. A user upgrading from a 1.x binary to a 1.y binary (y > x) opens their existing files cleanly. +- *Positive:* Per-page-type evolution. A future change to data-page slot layout bumps only that page-type's version; existing handle-table pages, freemap pages, etc. remain untouched and unmigrated. +- *Positive:* Lazy migration is the default. Reads dispatch on the version byte; writes always produce the latest version. An opt-in eager upgrader (deferred to a future minor; see ISSUES.md I31) sweeps remaining old pages. +- *Negative:* Two version checks instead of one. The page-cache load path validates the per-page version on every miss; the cost is one byte of comparison per cache miss. +- *Pre-1.0 caveat:* The on-disk `format_version` constant may receive one final reset to 1 before release. No production databases exist yet (project-memory note `project_chisel_format_version_tentative`), so accumulating in-development bumps can be collapsed. + +**Update (2026-06-02):** Chunk tags (ADR-12) drove the first MINOR bump, `0 → 1` — the two-tier scheme's first real exercise. A minor-0 (pre-tag) database opens cleanly under a minor-1 binary through the MAJOR-only gate and reads as fully untagged, exactly as the design promised. The per-page version byte was untouched: the new `MembershipInterior` (`0x05`) / `MembershipLeaf` (`0x06`) page types are born at `PAGE_FORMAT_VERSION_CURRENT = 0`, confirming an additive page-type change needs no per-page bump. + +**Update (2026-06-21):** The per-page read-dispatch (I31) and the file-MINOR write-gate (I29) landed (`docs/specs/2026-06-21-per-page-format-versioning-design.md`). This corrects two over-statements in the Consequences above: + +- The page-cache load path validates ONLY the XXH3 checksum — it does **not** read or validate the per-page version byte. There is no per-miss version check; that "Negative" consequence never described shipped behavior. Version dispatch is per-module and **decode-only** (a reader branches `if page_format_version(buf) >= K { read field } else { default }`), and is currently **dormant**: with only version 0 in existence, no read path branches on it, so "reads dispatch on the version byte" describes the mechanism, not present behavior. +- Writes always stamp `page::current_version(page_type)` (the single per-type write seam; every `init_page` site calls it), and COW makes upgrade-on-write free. The I29 gate forces a file whose MINOR exceeds the binary's into **read-only** at open (`PageIo::force_read_only`) rather than rejecting it — reads stay safe by the additive invariant; only writes are refused (`ReadOnlyMode`). + +Refined model: a **zero-default additive** field (the ADR-14 client-byte pattern, where zero == absent == default) needs **no** version bump. The per-page version exists *solely* to disambiguate absent-vs-zero for additive fields where zero is a legitimate value. The eager-upgrade sweep remains deferred. + +**Update (2026-06-30):** The MAJOR tier saw its first real bump — encrypted databases stamp **MAJOR=2** (`ENCRYPTED_FORMAT_VERSION = pack(2, 0)`; plaintext stays MAJOR=1). The open-time gate now computes the expected MAJOR from whether a crypto-header is present (`expected = if encrypted { 2 } else { 1 }`), so a new binary accepts both a MAJOR=1 plaintext file and a MAJOR=2 encrypted one, while an encryption-unaware old binary (which gates on MAJOR==1) refuses a MAJOR=2 file — the "completely incompatible binary refuses to open" guarantee working exactly as designed. See ADR-15. + +See `ISSUES.md` I29 (file-level) and I31 (page-level). + +--- diff --git a/docs/adr/0008-in-memory-mode.md b/docs/adr/0008-in-memory-mode.md new file mode 100644 index 0000000..3324d3a --- /dev/null +++ b/docs/adr/0008-in-memory-mode.md @@ -0,0 +1,28 @@ +--- +id: 0008 +title: In-memory mode +date: unknown +status: Accepted +--- + +# 0008. In-memory mode + +**Context:** Tests, benchmarks, and ephemeral workloads (e.g., a script that builds a transient database) shouldn't pay the cost of disk I/O or interact with the filesystem at all. A separate in-memory implementation would duplicate substantial code and risk drifting from the disk-backed engine. + +**Decision:** `Chisel::open_in_memory` runs the full engine against a `Vec`-backed `PageIo` with no filesystem and no `flock`. Same code path, same guarantees except durability. Used for tests, benchmarks, and ephemeral work. Also exposed as `chisel.open(None)` in the Python binding. + +**Alternatives considered:** + +- *Mock filesystem at the OS level (e.g., tmpfs).* Tests would still incur kernel-call overhead and filesystem semantics. Rejected — `Vec` is faster and removes the OS from the test loop entirely. +- *Separate `MemChisel` type.* Would duplicate every method and risk drift. Rejected. +- *Backend trait with `FileBackend` and `MemBackend` impls.* This is essentially what `PageIo` already is internally — the `Backing` enum has `File(...)` and `Memory(Vec)` variants. The decision was to expose this via a constructor (`open_in_memory`) rather than via a public trait surface, keeping the public API simple. + +**Consequences:** + +- *Positive:* Tests are fast and hermetic. +- *Positive:* Same code path means in-memory tests catch bugs that would also affect on-disk operation. +- *Positive:* Python users can experiment with `chisel.open(None)` without managing temp files. +- *Negative:* No `flock` means the in-memory mode cannot detect "two `open_in_memory` calls share a state" — but this is impossible by construction since each call creates its own `Vec`. +- *Negative:* Counters reset on close+reopen because the `Vec` doesn't persist; in-memory mode loses counter history that would have survived for a disk-backed reopen of the same file. + +--- diff --git a/docs/adr/0009-counter-instrumentation-via-chisel-counters.md b/docs/adr/0009-counter-instrumentation-via-chisel-counters.md new file mode 100644 index 0000000..00f8e13 --- /dev/null +++ b/docs/adr/0009-counter-instrumentation-via-chisel-counters.md @@ -0,0 +1,34 @@ +--- +id: 0009 +title: Counter instrumentation via `Chisel::counters()` +date: unknown +status: Accepted +--- + +# 0009. Counter instrumentation via `Chisel::counters()` + +**Context:** Bench harnesses, performance debugging, and long-running operational visibility all want "what did the engine actually do during this operation?" Without counters, the only signal is wall-clock time, which is too noisy for component-level analysis (cache hit rate vs. fsync rate vs. allocation rate). + +**Decision:** Four cumulative-from-open counters exposed via `Chisel::counters() -> ChiselCounters`: `cache_hits`, `cache_misses`, `pages_allocated`, `fsync_calls`. Each is a `Cell` living at the increment site (`PageCache` for the first three, `PageIo` for fsync). `PageCache::counters()` aggregates them into a single struct. `ChiselCounters` is `#[non_exhaustive]` so future counters can be added without a breaking change. + +Three semantic conventions matter: + +- **Counters reset on close + reopen.** No persistent state on disk. +- **Misses, allocations, and hit increments record *attempts*, not successes.** A `CacheFull` allocation still bumps `pages_allocated`. Asymmetric exception: `fsync_calls` counts only *successful* fsyncs, because a failed fsync poisons the engine and the counter on a poisoned engine has no defined further meaning. +- **Reads via `Chisel::counters()` are `&self`** and do not mutate. + +**Alternatives considered:** + +- *Internal logging/tracing.* Would require log-line parsing on the consumer side. Counters give a structured, allocation-free read. +- *Histogram of operation latencies.* Out of scope for v1 — adds dependency on a histogram crate and increases per-operation overhead. The bench harness can compute its own latency distributions externally. +- *Configurable counter set.* Adds complexity for marginal benefit. v1 picks four; `#[non_exhaustive]` keeps the door open. + +**Consequences:** + +- *Positive:* The bench harness reads counters before/after each scenario cell and reports deltas. This is what makes the bench-suite's per-cell analysis possible. +- *Positive:* Operational debugging: "how many cache misses did this query cause?" is one `counters()` call. +- *Negative:* Four counters is a v1 minimum; might want fsync byte counts, spillway hits, etc. later. `#[non_exhaustive]` makes additions cheap. + +Landed as PR 1 of the bench-suite series. + +--- diff --git a/docs/adr/0010-bench-suite-series-cross-engine-dedicated-machine.md b/docs/adr/0010-bench-suite-series-cross-engine-dedicated-machine.md new file mode 100644 index 0000000..a252dc2 --- /dev/null +++ b/docs/adr/0010-bench-suite-series-cross-engine-dedicated-machine.md @@ -0,0 +1,40 @@ +--- +id: 0010 +title: Bench-suite series (cross-engine + dedicated machine) +date: 2026-04-30 +status: Accepted +--- + +# 0010. Bench-suite series (cross-engine + dedicated machine) + +**Context:** "Is Chisel fast?" is meaningless without a baseline. A storage engine's performance is dominated by storage primitives (fsync semantics, page cache behavior, allocation patterns) that vary wildly across engines. Comparing Chisel against itself across time gives regression signal but no absolute calibration. Comparing against established engines (redb, SQLite) gives absolute calibration but requires careful fairness control. + +**Decision:** A `bench/` subcrate (sibling to `python/`, NOT a workspace member of the root chisel crate — separate `cd bench && cargo test` step) provides three measurement layers: + +1. **Cross-engine equivalence tests** — five scenarios × three engines × snapshot/restore checks, asserting all engines produce identical observable state for the same workload. +2. **Criterion micro-grid** — 165 cells of single-tx-shape operations. +3. **YCSB-style scenario tier** — four end-to-end workloads (YCSB-A, YCSB-B, Mutation Log, Document Store) timed with `Instant::now()` rather than Criterion (Criterion's many-samples-per-bench model exceeds the 1-6 minute scenario budget). + +A post-processor (`chisel-bench-summarize`) emits `summary.md`, `results.json`, and `cross-engine.md` (per-metric Chisel-vs-redb-vs-SQLite). A diff binary (`chisel-bench-diff`) consumes two `results.json` files and posts a sticky regression-report comment on each PR via `bench.yml`. + +Eight PRs landed 2026-04-30 → 2026-05-04. PR 8 added the cross-engine artifact + macOS-fsync fairness fix (see ADR-11). + +A future **dedicated bench machine** will host low-noise per-PR runs, canonical release-notes numbers, and (eventually) soak workloads. Foundation spec is `docs/specs/2026-05-04-dedicated-bench-machine-foundation-design.md`; code-first phases (noise-gate Rust binary + ops workflows + operator runbook) shipped 2026-05-04. Operator phases (VM provisioning, runner registration, Phase 5 `bench.yml` migration) await operator setup. + +**Alternatives considered:** + +- *In-tree bench.* Rejected — the bench depends on redb and rusqlite, dependencies the storage engine itself doesn't need. Sibling crate keeps the engine's dependency graph minimal. +- *Workspace member.* Tempting but `cargo test` from the repo root would auto-run bench tests, which take 10-25 minutes. Sibling-crate forces explicit `cd bench && cargo test`, which is also a documented gotcha (see ARCHITECTURE.md "Building from source"). +- *Criterion for everything.* Criterion's many-samples-per-bench model is great for micro-grid but exceeds the time budget for end-to-end scenarios. Hybrid (Criterion for micro, `Instant::now()` for scenarios) is the explicit compromise. + +**Consequences:** + +- *Positive:* Trustworthy cross-engine numbers. `cross-engine.md` is suitable for the README and 1.0 release notes (with the noise-floor caveats from the dedicated-machine spec). +- *Positive:* Per-PR regression signal via the bench workflow's sticky diff comment. +- *Positive:* Counter instrumentation (ADR-9) enables per-cell attribution of throughput differences to fsync count, cache pressure, or page-allocation rate. +- *Negative:* Bench tests don't run from the repo root. The CLAUDE.md / build instructions document this gotcha explicitly. +- *Negative:* GitHub-hosted runner variance (~15% on the scenario tier) limits the diff binary's actionability. Solved by the dedicated-machine foundation when operator setup completes. + +Master spec: `docs/superpowers/specs/2026-04-25-chisel-benchmark-suite-design.md` (covers PRs 1-7). PR 8 has its own spec: `docs/superpowers/specs/2026-05-04-chisel-bench-cross-engine-design.md`. Dedicated-machine: `docs/specs/2026-05-04-dedicated-bench-machine-foundation-design.md`. + +--- diff --git a/docs/adr/0011-macos-fsync-fairness-via-pragma-fullfsync-on.md b/docs/adr/0011-macos-fsync-fairness-via-pragma-fullfsync-on.md new file mode 100644 index 0000000..ca54518 --- /dev/null +++ b/docs/adr/0011-macos-fsync-fairness-via-pragma-fullfsync-on.md @@ -0,0 +1,29 @@ +--- +id: 0011 +title: macOS-fsync fairness via `PRAGMA fullfsync=ON` +date: 2026-05-04 +status: Accepted +--- + +# 0011. macOS-fsync fairness via `PRAGMA fullfsync=ON` + +**Context:** On macOS APFS, Chisel's `sync_all` calls Rust's `File::sync_all` which translates to `fcntl(F_FULLFSYNC)` — a flush through the disk's write cache that is durable against power loss. SQLite's default `fsync()` on macOS only flushes to the disk's write cache (without F_FULLFSYNC). Result: bench measurements of Chisel-strict vs. SQLite-strict on macOS measured Apple-vs-Apple disk-cache semantics, not engine behavior — SQLite was ~3 orders of magnitude faster than Chisel on `Strict` durability for the wrong reason. + +**Decision:** `SqliteEngine::open_file` issues `PRAGMA fullfsync=ON` for `DurabilityMode::Strict`. No `#[cfg(target_os)]` gate — Linux ignores the pragma; macOS uses `fcntl(F_FULLFSYNC)` matching Chisel's `sync_all`. Costs: one extra PRAGMA exec at SQLite open time. + +**Alternatives considered:** + +- *Disable F_FULLFSYNC on Chisel for macOS bench.* Would compromise Chisel's actual durability semantics in the bench, defeating the comparison's purpose. +- *Compare only on Linux.* Loses platform coverage; macOS is the platform where Chisel is most likely to be embedded (single-user developer machines). +- *Document the gap, don't fix it.* Rejected — silent unfairness in published numbers is worse than no numbers. + +**Consequences:** + +- *Positive:* macOS bench numbers now reflect engine behavior, not Apple's default `fsync` semantics. +- *Positive:* Linux numbers unchanged (`PRAGMA fullfsync=ON` is a no-op there). +- *Positive:* No `#[cfg(target_os)]` keeps the bench code platform-uniform. +- *Negative:* SQLite-strict on macOS is now slower than its previous bench numbers showed. README/release-note comparisons must be regenerated post-PR 8. + +Confirmed empirically (PR 8 first-run bench-diff): SQLite-strict cells moved by ≤1.5% on Linux across all four scenarios after the change, validating the no-gate decision. + +--- diff --git a/docs/adr/0012-chunk-tags-reverse-membership-index.md b/docs/adr/0012-chunk-tags-reverse-membership-index.md new file mode 100644 index 0000000..9d9ae85 --- /dev/null +++ b/docs/adr/0012-chunk-tags-reverse-membership-index.md @@ -0,0 +1,33 @@ +--- +id: 0012 +title: Chunk tags + reverse membership index +date: 2026-06-02 +status: Accepted +--- + +# 0012. Chunk tags + reverse membership index + +**Context:** The relational layer that builds on Chisel (the primary client) needs three operations the bare handle API cannot do efficiently: sequentially scan all chunks of one relation, drop a whole relation, and delete a single chunk while removing it from its relation's set — all without an `O(all chunks)` pass. Chisel had no notion of grouping chunks; the client could store a group id inside each value and scan everything, but that is linear in the whole database per operation. + +**Decision:** Attach an optional, immutable `u32` **tag** to each chunk at allocation (`allocate_tagged(value, tag)`); `tag == 0` is the "untagged" sentinel. The mapping is split into a *forward* map and a *reverse* map. The forward map (handle → tag) lives in 4 of the `HandleEntry`'s 5 previously-reserved bytes, so `tag(handle)` is `O(1)` off the entry the engine already loads, and a bare `delete(handle)` self-maintains the index by reading the tag from the entry. The reverse map (tag → {handles}) is a two-level copy-on-write radix (`MembershipIndex` over the generic `RadixU64`): an outer tree keyed by tag whose leaf value bit-packs `(inner_depth | inner_root)`, and a per-tag inner tree keyed by handle. `delete_with_tag(tag, max)` is the bounded relation-drop primitive (returns `TagDropProgress { deleted, complete }`; the caller loops `begin → delete_with_tag → commit` until `complete`). + +**Alternatives considered:** + +- *Store the group id in the value, scan all chunks.* `O(all chunks)` per scan/drop. Rejected — the membership index makes scan and drop `O(members of the relation)`. +- *Single flat radix over a packed `(tag:handle)` key.* Rejected in favor of the two-level form, which reuses the existing `u64`-radix shape directly and makes "enumerate the distinct tags" a first-class `O(T)` operation. +- *`u64` tag.* Rejected — a `u32` fits in the `HandleEntry`'s reserved bytes, which is what makes forward storage *free* (no side table, no extra page reads). `2^32 − 1` tags is effectively unbounded for relation ids. +- *Mutable tags.* Cheap to support (the forward tag is readable), but deliberately deferred; immutability removes the retag code path entirely. To re-tag, the client allocates a new chunk and deletes the old. +- *Bitmap inner sets.* Deferred — per-tag handle sets are expected sparse, so radix is the right default. + +**Consequences:** + +- *Positive:* Untagged chunks cost nothing — tag `0`, no index entry, no index COW. The feature is invisible to any workload that does not use it. +- *Positive:* Reuses the existing radix machinery, COW discipline, superblock-anchored root, freemap-backed page reclaim, and poison model — no new commit-protocol surface. "Radix all the way down," not a foreign B-tree. +- *Positive:* Backward compatible by construction. Old databases open with all chunks untagged and an empty index (zeroed bytes read as tag `0` / `PAGE_ID_NONE`); no migration. The on-disk additions ride ADR-7's two-tier versioning as a MINOR bump. +- *Negative:* 4 of the 5 per-entry reserved bytes are permanently committed to the tag; the 5th (byte `[15]`) became the client byte (ADR-14). +- *Negative:* `delete_with_tag` is bounded by `max`; dropping a large relation requires the caller to loop until `complete`, each batch a separately-durable transaction (this bound also caps a drop's working set within the cache/spillway budget). +- *Reversibility:* Medium — it is a new on-disk subsystem (the `HandleEntry` tag field, the superblock `root_membership_index_page`, and `membership_index.rs`). Because the format additions are additive, reversal reverts them without affecting existing readers, but it removes a module and a public API cluster. + +Spec: `docs/specs/2026-06-02-chunk-tags-design.md`. See ARCHITECTURE.md "Chunk tags (the membership index in use)" and "Membership index pages." Closes the `drop_table` / `drop_index_table` handle-and-page leak (ISSUES.md F1 / I12). + +--- diff --git a/docs/adr/0013-within-session-iteration-stability-contract.md b/docs/adr/0013-within-session-iteration-stability-contract.md new file mode 100644 index 0000000..9b49c88 --- /dev/null +++ b/docs/adr/0013-within-session-iteration-stability-contract.md @@ -0,0 +1,31 @@ +--- +id: 0013 +title: Within-session iteration-stability contract +date: 2026-06-04 +status: Accepted +--- + +# 0013. Within-session iteration-stability contract + +**Context:** `handles()` and `handles_with_tag()` materialize a `Vec` by walking radix trees, and their doc comments explicitly disclaimed any ordering ("order is unspecified"). The relational client wants to scan a relation, do work, and scan it again expecting identical results when it has not mutated the store — to re-drive a query, resume an interrupted pass, or cross-check — without defensively sorting or snapshotting. The implementation already produced a deterministic order (the radix walk is a pure function of tree structure), but the *contract* promised nothing, so a consumer could not rely on it. + +**Decision:** Promote the already-true behavior to a documented, tested guarantee, scoped deliberately narrow: within a single open instance, repeated `handles()` / `handles_with_tag(tag)` calls return an identical `Vec` (same elements, same order) as long as the relevant live set is unchanged between calls and no `defrag` has run. The *order itself stays unspecified* — this is a *repeatability* guarantee, not an ordering one. The guarantee is single-session only (it does not survive close+reopen or `defrag`). No production code changed: the radix walks already satisfy it. The work is the contract (doc comments on `Chisel::handles`, `Chisel::handles_with_tag`, and `RadixU64::iter`) plus adversarial differential tests (`tests/iteration_stability.rs`). + +**Alternatives considered:** + +- *Guarantee a specific order (ascending handle).* Strongest, and already what the implementation produces. Rejected: it would commit the public contract to a particular order, constraining any future change to the index internals. "Repeatable but opaque" preserves that freedom. +- *Snapshot / MVCC isolation* so a scan stays stable even across concurrent mutation. Rejected as out of scope — Chisel is single-writer (ADR-2), and the requirement explicitly excluded mutation between scans. +- *Wider scope* (the guarantee survives reopen and/or `defrag`). Rejected in favor of single-session, which constrains future internals the least while still serving the client's scan-twice use case. +- *Internal `debug_assert!` sortedness canary* as a second enforcement mechanism. Rejected — it would couple internal code to the ascending behavior we explicitly declined to promise; a future intentional reorder would have to delete it. + +**Consequences:** + +- *Positive:* The scan layer can rely on within-session repeatability without defensive sorting or snapshotting. +- *Positive:* Minimal commitment. The order stays opaque and the scope is single-session, so the radix order, reopen layout, and `defrag` reordering all remain free to change. +- *Positive:* The differential tests double as a regression guard for the radix-depth re-derivation invariant (ISSUES.md I99 / C1): a rolled-back grow that failed to restore tree depth would mis-enumerate a later scan, which the rollback/savepoint tests catch from the iteration angle. +- *Negative (subtlety):* "Repeatable" cannot be checked from a single call — only differentially (scan, churn the state the contract permits to vary, scan again, compare). Enforcement is therefore test-only; there is no compile-time or single-call guard. +- *Reversibility:* Easy in code (the guarantee is documentation over existing behavior), but it is now a public contract — removing it would be a breaking change for consumers that rely on within-session scan repeatability. + +Spec: `docs/specs/2026-06-04-stable-chunk-iteration-design.md`. Plan: `docs/plans/2026-06-04-stable-chunk-iteration.md`. Tests: `tests/iteration_stability.rs`. Hardens the tagged-iteration API from ADR-12. + +--- diff --git a/docs/adr/0014-client-byte-spending-the-last-reserved-entry-byte.md b/docs/adr/0014-client-byte-spending-the-last-reserved-entry-byte.md new file mode 100644 index 0000000..e2812e0 --- /dev/null +++ b/docs/adr/0014-client-byte-spending-the-last-reserved-entry-byte.md @@ -0,0 +1,60 @@ +--- +id: 0014 +title: Client byte — spending the last reserved entry byte +date: 2026-06-05 +status: Accepted +--- + +# 0014. Client byte — spending the last reserved entry byte + +**Context:** Chunk tags (ADR-12) committed 4 of the 5 reserved `HandleEntry` +bytes to the `u32` tag, leaving one byte (`[15]`). The relational client wanted a +small per-chunk scratch value it could set and read without rewriting the chunk's +value or spending a tag — opaque metadata Chisel stores but never interprets. + +**Decision:** Expose a per-chunk `u8` "client byte" stored in entry byte `[15]`. +It is mutable (`set_client_byte(handle, u8)`, a transactional handle-table +mutation that COWs only the leaf) and readable (`client_byte(handle) -> u8`, +mirroring `tag()`'s read path). Default `0`. Opaque: no search, no filter, no +index — contrast the tag's membership index (ADR-12). The byte rides every value +`update()` via the same entry carry-forward that preserves the tag, and reverts +with the transaction on rollback. Deleted handles return `InvalidHandle` +(following `read()`). + +Crucially, **no on-disk format change**: byte `[15]` has always been part of the +16-byte entry and always written (as `0`). Activating a *reserved* byte is not a +versioned change — there is nothing for a reader to gate on — so +`FORMAT_MINOR_VERSION` stays `1`. This refines ADR-7: reserved bytes are part of +the format from creation; only new structures or semantics a reader must gate on +warrant a version bump. + +**Alternatives considered:** + +- *Store the byte with the value (data page).* Rejected: forces a full value + rewrite per change and re-couples metadata to value bytes. Entry-resident + storage makes a flip cost one handle-table leaf COW, independent of value size. +- *Immutable / set-at-allocation only (like the tag).* Rejected: the client needs + to change it in place; immutability would force delete + reallocate. +- *Richer `Handle { id, tag, client_byte }` return type.* Rejected: a breaking + change to every handle-returning signature; an accessor keeps `handle: u64`. +- *MINOR bump `1 -> 2` for record-keeping.* Rejected: the layout is byte-identical, + so there is nothing to gate on (see Decision). + +**Consequences:** + +- *Positive:* Cheap, value-size-independent per-chunk metadata with zero format + cost — the payoff of pre-allocating reserved bytes in the original layout. +- *Positive:* No migration; pre-feature databases read byte `[15]` as `0`. +- *Negative (caveat, recorded not gated):* a pre-feature binary hardcodes + `[15] = 0` on every entry rewrite, so opening a client-byte database with an + older binary and rewriting an entry (`update`, defrag) silently clears that + chunk's client byte. Acceptable pre-1.0 (no production databases, single-writer + single-process, opaque metadata); it is exactly the case the deferred I29 + minor-write gate would catch. +- *Note:* `client_byte` / `set_client_byte` reject deleted handles with + `InvalidHandle` (following `read()`), stricter than `tag()`'s current unguarded + read of a tombstone — a pre-existing `tag()` quirk tracked separately. + +Spec: `docs/specs/2026-06-05-client-byte-design.md`. + +--- diff --git a/docs/adr/0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md b/docs/adr/0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md new file mode 100644 index 0000000..a719cb7 --- /dev/null +++ b/docs/adr/0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md @@ -0,0 +1,41 @@ +--- +id: 0015 +title: On-disk encryption (XChaCha20-Poly1305, envelope keys) +date: 2026-06-30 +status: Accepted +--- + +# 0015. On-disk encryption (XChaCha20-Poly1305, envelope keys) + +**Context:** Chisel originally listed "no encryption at rest" as an explicit non-goal (see Out of scope, now superseded), deferring confidentiality to filesystem-level encryption. That is adequate against whole-device theft but not for a client that needs per-database confidentiality *it* controls — a supplied key, not the OS's — with cryptographic tamper-detection. The requirement: the client program supplies a key when opening a database; every byte Chisel then writes is encrypted and integrity-protected, and rotating that credential must be cheap. + +**Decision:** Opt-in **authenticated** at-rest encryption. A client supplies a `Key` (raw bytes or a passphrase) via `Options::encryption_key`; without one, the database is plaintext exactly as before. Core choices: + +- **Cipher: XChaCha20-Poly1305**, a fresh **random 192-bit nonce per page write**, `AAD = page_id` (anti-relocation). AEAD (authenticated) over a length-preserving mode so tampering is *cryptographically* detected, not merely obscured. +- **Envelope keys.** A random per-database **DEK** seals every page and the sensitive superblock fields; the DEK is stored **wrapped** under a **KEK** derived from the client key (HKDF-SHA256 for raw keys, Argon2id for passphrases) in an **8-slot key-slot table** in the superblock's plaintext reserved region (offset 324). Credential rotation re-wraps the DEK — **O(1), no data re-encryption**. +- **Page format: an 8232-byte on-disk stride** (`ENC_PAGE_SIZE` = 8192 ciphertext + 16 tag + 24 nonce); the *logical* page stays 8192, so the freemap/data-page/handle-table geometry is untouched — encryption is a transform at the **page-I/O seam** (`PageCache` owns a `PageCipher`; `page_io` is stride-aware but crypto-agnostic). Encrypted DBs use this uniform stride from birth, including superblock slots (the 8192 image zero-padded into an 8232 unit); open bootstraps by reading page 0 at offset 0, learning the stride from its plaintext crypto-header, then reading the remaining slots at that stride. +- **Format gate: MAJOR bump 1 → 2** (the first real exercise of ADR-7's MAJOR tier). An encryption-unaware binary refuses a MAJOR=2 file; plaintext databases stay MAJOR=1 and byte-identical. +- **Spillway and superblock are in scope.** The spillway holds *sealed* blobs (seal-once on evict, verbatim copy on drain — never plaintext); the sensitive superblock body (roots, counters, and the user-chosen `named_roots`) is DEK-sealed, leaving only the bootstrap header + key-slot table in cleartext. + +**Alternatives considered:** + +- *AES-256-XTS (length-preserving).* Zero per-page overhead, no format change, the FDE industry standard — but **confidentiality only**, no cryptographic tamper-detection (would lean on the non-cryptographic XXH3). Rejected: the client wanted authenticated encryption, and since no production databases exist the one-time format change is free. +- *Deterministic nonce from `(page_id, counter)`.* Tempting (no stored nonce) but **unsafe under shadow paging**: a crashed transaction discards its writes and returns page_ids to the freemap while the durable counter doesn't advance, so the same page_id can get different plaintext at the same counter → keystream reuse. XChaCha's 192-bit **random** nonce sidesteps the whole class (no persisted counter, crash-safe) — this drove the cipher choice. +- *AES-256-GCM.* Fine with AES-NI, but its 96-bit nonce forces the deterministic-nonce hazard above or a stored counter; ChaCha is also constant-time in portable software (no hardware dependence) — the better default for an embedded library that runs wherever the client runs. +- *Encrypt only data pages, leave the superblock plaintext.* Rejected: `named_roots` holds user-chosen names (real user data); a plaintext superblock body leaks them. +- *Full DEK rotation (re-encrypt every page under a new DEK).* Deferred (ISSUES.md I142) — a heavy whole-file operation for the "the DEK itself is compromised" case, distinct from the implemented credential (KEK) rotation. +- *Keeping "encryption is out of scope."* Superseded — the requirement is per-database, client-controlled confidentiality that filesystem encryption cannot provide. + +**Consequences:** + +- *Positive:* Plaintext databases are provably byte-and-behavior identical (every divergence is a `Some/None` cipher branch); zero regression for existing/unencrypted files. +- *Positive:* Credential rotation is O(1) — `add_key` / `rotate_key` / `remove_key` (Rust + Python) re-wrap the stable DEK without touching data; `rotate_key` stages the new key before revoking the old (no zero-key window), and `remove_key` refuses the last active slot (brick prevention). +- *Positive:* Reuses the existing A/B superblock + poison model — a metadata-only `rewrite_crypto_header` commit persists a rotated slot table atomically (write inactive slot → fsync → promote), so a crash mid-rotation leaves the old slot table intact. +- *Threat-model boundary (documented non-guarantees, spec §9):* confidentiality + per-page/superblock tamper-detection + anti-relocation; **no** rollback/replay protection (an attacker substituting a wholly older, validly-signed image is undetectable without an external trust anchor); the DEK is plaintext in process memory during an open session (mitigated by zeroize-on-drop, not by encryption). +- *Negative:* Encrypted files are ~0.5% larger (the 40-byte per-page trailer) and MAJOR=2, so encryption-unaware binaries can't read them (by design). +- *Reversibility:* Hard — a new on-disk format (encrypted stride, sealed superblock, key-slot table) plus a page-I/O seal seam and a key-management API cluster. Additive to plaintext DBs (it does not affect existing readers), but removal would delete `src/crypto/`, `src/superblock/crypto_header.rs`, `src/transaction/keys.rs`, and the public key API. + +Spec: `docs/specs/2026-06-29-on-disk-encryption-design.md`. Plan: `docs/plans/2026-06-29-on-disk-encryption.md`. Implemented 2026-06-30 across 6 phases (crypto core → superblock/key-flow → page-I/O + cache + spillway → public API + Python → key rotation → docs/version). See ARCHITECTURE.md "On-disk encryption" and ISSUES.md I142 (deferred bulk DEK rotation). Public API is deliberately narrow: only `Key`, `Argon2Params`, and the encryption error variants are public; the crypto/superblock internals are `pub(crate)`. + +--- + diff --git a/docs/adr/0016-swift-binding-via-uniffi.md b/docs/adr/0016-swift-binding-via-uniffi.md new file mode 100644 index 0000000..356bab0 --- /dev/null +++ b/docs/adr/0016-swift-binding-via-uniffi.md @@ -0,0 +1,79 @@ +--- +id: 0016 +title: Swift binding via UniFFI +date: 2026-07-19 +status: Accepted +summary: The iOS/macOS Swift binding is a UniFFI-generated FFI over an Arc> wrapper crate, with the engine crate left unchanged. +--- + +# 0016. Swift binding via UniFFI + +## Context + +Chisel needs to be embeddable in iOS and macOS applications as a reusable, +published Swift package. Swift can only call C, not Rust directly — so unlike +the existing Python binding (`python/`, PyO3 linked against the engine in one +address space), a Swift binding needs a C ABI in between. The engine is a +single-writer design (`chisel::Chisel` is `Send + !Sync`, every mutator takes +`&mut self`; see ADR-2), which collides head-on with Swift 6 strict concurrency +(`Sendable`) and ARC. The binding's hardest question is therefore not the CRUD +surface but how to cross into Swift's async world without violating the +one-client-at-a-time invariant — and to do it without modifying the engine. + +## Decision + +Ship a reusable SPM package. Bridge Rust → Swift with **UniFFI** (proc-macros, +library mode). A new `chisel-ffi` crate — the fourth workspace member, kept out +of `default-members` like `python/` — wraps the engine in +`Mutex>` and exports a flat, synchronous API via +`#[uniffi::export]`; UniFFI generates both the C-ABI scaffolding and idiomatic +Swift (a real error enum, `Data` buffers, records). Two hand-written Swift +surfaces sit on top: a synchronous, thread-confined `ChiselDatabase` and an +`async` `actor ChiselStore` backed by a dedicated serial `DispatchQueue`. +Mutations are closure-only (`transaction { txn in … }`); reads, config, and key +management are top-level. The full surface is exposed, including on-disk +encryption (ADR-15) with a Keychain-aware `ChiselKey`. The engine crate is not +touched: the single-client invariant relocates from the borrow checker to the +`Mutex` at the FFI boundary. + +## Alternatives considered + +- **Hand-rolled C ABI + cbindgen** — maximal control and no codegen dependency, + but a rich API becomes pages of `unsafe` boilerplate (length out-params for + every `Vec`, error out-params, manual opaque-pointer lifecycles), each a + place for UB. Rejected: UniFFI generates the error tree, buffer handling, and + the SPM/xcframework packaging for free, and is the proven path for shipping a + Rust engine to iOS (matrix-rust-sdk). +- **swift-bridge** — Swift-specific codegen, arguably more idiomatic in places, + but a smaller, less battle-tested ecosystem, no Kotlin reuse, and thinner + publish-an-xcframework documentation. Rejected on ecosystem maturity. +- **Python-style surface (raw `begin`/`commit` + top-level mutators)** — the + whole-branch review found top-level mutators would be dead-on-arrival: the + engine requires an active transaction for mutations and has no auto-begin, so + `db.allocate()` at the top level always throws `NoActiveTransaction`. Rejected + in favor of closure-only mutations, which make "one transaction, always + resolved" structural. + +## Consequences + +- `Mutex` is `Send + Sync` because `Chisel` is `Send + !Sync` (it uses + `RefCell`, no `Rc`); a compile-time `assert_send::()` in + `chisel-ffi` guards the assumption, with a documented dedicated-thread-actor + fallback if the engine ever becomes `!Send`. The `Option` lets `close(self)` + (by-value in the engine) `.take()` and consume behind UniFFI's `&self`/`Arc`. +- The async `transaction` body is deliberately **synchronous** (`@Sendable`, + non-`async`) so no `await` can suspend mid-transaction and interleave a second + logical transaction on the `!Sync` engine — a type-level guarantee, backed by + a dedicated serial queue so blocking fsync I/O never touches Swift's + cooperative pool. +- `ChiselError.category` (operational vs fatal) follows the **Python binding's** + caller-facing tiering — notably `Poisoned` is fatal — which deliberately + diverges from the engine's manager-internal `is_fatal()` (ADR-6). Every error + variant carries a `message: String` (the engine's `Display`) so Swift callers + get diagnostics, not bare case names. +- Distribution is SPM with a prebuilt `.xcframework` across five Apple targets; + a macOS CI job builds and tests it. The Thread-Sanitizer concurrency check is + best-effort in CI because some hosts' SIP policy blocks sanitizer injection + into the xctest helper. +- Relates to ADR-2 (single-writer), ADR-6 (poison model), ADR-15 (encryption). + Supersedes nothing. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..be9a6e0 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,23 @@ +# Architecture Decision Records + + + +| ID | Title | Status | Date | Summary | +|---:|-------|--------|------|---------| +| [0000](0000-decision-register-overview.md) | Decision register (overview) | Accepted | 2026-05-04 | | +| [0001](0001-shadow-paging-not-wal.md) | Shadow paging, not WAL | Accepted | — | | +| [0002](0002-single-writer-enforced-by-mut-self.md) | Single-writer enforced by `&mut self` | Accepted | — | | +| [0003](0003-per-module-cow-no-centralized-abstraction.md) | Per-module COW, no centralized abstraction | Accepted | — | | +| [0004](0004-n-rotating-superblocks-for-atomic-commit.md) | N rotating superblocks for atomic commit | Accepted | — | | +| [0005](0005-spillway-sidecar-file-over-hard-ceiling.md) | Spillway sidecar file over hard ceiling | Accepted | 2026-05-04 | | +| [0006](0006-poison-model-on-fatal-errors.md) | Poison model on fatal errors | Accepted | — | | +| [0007](0007-two-tier-format-versioning.md) | Two-tier format versioning | Accepted | — | | +| [0008](0008-in-memory-mode.md) | In-memory mode | Accepted | — | | +| [0009](0009-counter-instrumentation-via-chisel-counters.md) | Counter instrumentation via `Chisel::counters()` | Accepted | — | | +| [0010](0010-bench-suite-series-cross-engine-dedicated-machine.md) | Bench-suite series (cross-engine + dedicated machine) | Accepted | 2026-04-30 | | +| [0011](0011-macos-fsync-fairness-via-pragma-fullfsync-on.md) | macOS-fsync fairness via `PRAGMA fullfsync=ON` | Accepted | 2026-05-04 | | +| [0012](0012-chunk-tags-reverse-membership-index.md) | Chunk tags + reverse membership index | Accepted | 2026-06-02 | | +| [0013](0013-within-session-iteration-stability-contract.md) | Within-session iteration-stability contract | Accepted | 2026-06-04 | | +| [0014](0014-client-byte-spending-the-last-reserved-entry-byte.md) | Client byte — spending the last reserved entry byte | Accepted | 2026-06-05 | | +| [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) | On-disk encryption (XChaCha20-Poly1305, envelope keys) | Accepted | 2026-06-30 | | +| [0016](0016-swift-binding-via-uniffi.md) | Swift binding via UniFFI | Accepted | 2026-07-19 | The iOS/macOS Swift binding is a UniFFI-generated FFI over an Arc> wrapper crate, with the engine crate left unchanged. | From aad007f35936d9dff2aa831de0b7bb9f1a388036 Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Tue, 4 Aug 2026 08:04:53 -0700 Subject: [PATCH 3/4] docs: retire ISSUES.md in favour of GitHub issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ISSUES.md had grown to 1868 lines across 119 numbered entries and could no longer be read whole by a session that also needed to hold the code — the 2026-07-29 review deliberately withheld it from its reviewers for that reason, which is an admission that the log had stopped being usable as an input. It also duplicated a tracker the project already used, under a second set of identifiers, and its statuses drifted: I149 and I150 were both still marked OPEN although I150 shipped in PR #128 and I149 shipped two commits ago. Of the 14 open and 3 deferred entries, 6 were already represented in existing issues (I147 in #116, I149 and I150 in #102, I151 in #107, I152 and I154 in #106). The remaining 11 were filed as #138-#148, each carrying its entry verbatim plus a header naming its original id and source review. I78 needed its header line preserved as well, because it recorded a correction that contradicts the caveat in its own body. The 83 entries marked fixed were not migrated. They describe completed work and remain in git history at 0ffe3bc. This is the real cost of the change and ADR 0017 records it as such: those entries are no longer discoverable by grep in a checkout, and some of them hold genuine decision rationale. The ~167 I markers in source comments are deliberately left alone. Most refer to entries closed long ago with no GitHub equivalent, so rewriting them would delete provenance across 167 sites for no navigational gain. ARCHITECTURE.md gains a note explaining what they mean and how to retrieve the retired file; the live links in README.md, ARCHITECTURE.md and THEORY.md now point at docs/adr/ and the issue tracker instead. --- ARCHITECTURE.md | 4 +- ISSUES.md | 1868 ----------------- README.md | 5 +- THEORY.md | 2 +- ...github-issues-replace-tracked-issues-md.md | 96 + docs/adr/README.md | 1 + 6 files changed, 104 insertions(+), 1872 deletions(-) delete mode 100644 ISSUES.md create mode 100644 docs/adr/0017-github-issues-replace-tracked-issues-md.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 2cd70e3..a37f5ea 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,6 +1,8 @@ # Chisel — Architecture and On-Disk Format -This document is the cold-start reference for someone (human or AI) picking up the Chisel codebase in a context-free session. It is a compressed map of the layers, the exact on-disk byte format, the load-bearing invariants, and the landmines. Read it whole at the start of a session. For *what Chisel does* and how to use it, see [`README.md`](README.md). For the running decision log — open issues, closed issues, every design tradeoff with date-stamped rationale — see [`ISSUES.md`](ISSUES.md). +This document is the cold-start reference for someone (human or AI) picking up the Chisel codebase in a context-free session. It is a compressed map of the layers, the exact on-disk byte format, the load-bearing invariants, and the landmines. Read it whole at the start of a session. For *what Chisel does* and how to use it, see [`README.md`](README.md). For the design decisions and their rejected alternatives, see [`docs/adr/`](docs/adr/); for the open work, see [GitHub issues](https://github.com/pgexperts/chisel/issues). + +> **On `I` markers.** Comments throughout the source cite issue ids like `I61 (ISSUES.md, 2026-05-22)`. Until 2026-08-03 the project tracked its issues in a git-tracked `ISSUES.md`; that file was retired in favour of GitHub issues, and its open entries were migrated there. The markers were deliberately left in place rather than rewritten across ~167 sites — they are dated provenance, and the retired file is still readable in git history (`git show 0ffe3bc:ISSUES.md`). Treat an `I` as "see the decision log as it stood on that date", not as a live link. For the theory of operation and the rationale behind these decisions — why shadow-paging over WAL, the rejected alternatives, the implementation history — see [`THEORY.md`](THEORY.md). diff --git a/ISSUES.md b/ISSUES.md deleted file mode 100644 index b300407..0000000 --- a/ISSUES.md +++ /dev/null @@ -1,1868 +0,0 @@ -# Chisel Issues and Backlog - -Tracked work for the Chisel storage engine. Items are grouped by source and -rough category. Each entry carries a priority tag; see legend below. - -Sources: -- **[comment-pass]** — found during the 2026-04-10 commenting pass (read-only review, no tests run) -- **[comment-pass 2026-04-17]** — found during the 2026-04-17 re-commenting pass (also read-only; covered changed `src/` files and the new `python/src/` subcrate) -- **[comment-pass 2026-04-22]** — found during the 2026-04-22 third review pass (read-only; five-agent parallel audit over `src/` and `python/src/`) -- **[perf-review 2026-04-26]** — found during the `chisel-performance` skill fresh-eyes pass after PR-1 + PR-2 of the bench-suite series landed (read-only; output at `docs/reviews/perf-review-2026-04-26.md`) -- **[deepdive 2026-05-22]** — found during the `deepdive-rust` skill fresh-eyes pass after the spillway feature and the runtime-config setters landed (read-only; output at `docs/reviews/review-20260522-073901.md`) -- **[perf-review 2026-06-01]** — found during the `chisel-performance` + `rust-performance` full hot-path sweep (read-only; six-agent parallel audit over `src/` and `bench/`; output at `docs/reviews/perf-review-2026-06-01.md`) -- **[roadmap]** — from the README roadmap -- **[client]** — requested by the primary Chisel client - -Priority legend: -- **P0** — correctness / data loss / unsafe behavior. Fix before relying on Chisel for anything that matters. -- **P1** — real bugs or API pain that block clients or make future work harder. Plan for the next milestone. -- **P2** — known-correct v1 simplifications, latent issues, stat accuracy. Batch with related work. -- **P3** — nice-to-have, forward-compat, speculative, or trivial add-ons to other PRs. - ---- - -## Suggested fix order - -> **Status note (2026-04-17):** every item below has landed. The order is preserved here as historical context — a reader looking at this file for "what's open?" should conclude: nothing from the 2026-04-10 or 2026-04-17 review passes is still actionable. The individual entries in later sections carry the definitive status. -> -> **Status note (2026-04-22):** a fresh third review pass re-opened the file with new items I26 (P1, handle-table bounds), I27 (P2, savepoint freed-pages leak on commit), I28 (P2, `CacheFull` poisons during commit), and a doc-sweep bundle (C4), all resolved the same day. Two pre-1.0 infrastructure items also landed that day: I29 (split `format_version` into packed MAJOR / MINOR so the README's "sacred within a major version" promise is enforceable at the bytes level) and I31 (per-page format-version byte + 64-bit reserved common-header region, the foundation for lazy per-page upgrade). These are NOT in the suggested fix order above — see each entry in its own section below. The 2026-04-22 pass specifically looked for invariant mismatches across module boundaries, which is where the remaining bugs now live. -> -> **Status note (2026-05-22):** the deepdive-rust fresh-eyes review (output at `docs/reviews/review-20260522-073901.md`) adds I35–I71 in a new "Deepdive review findings (2026-05-22)" section below. The cluster is dominated by 1.0-readiness work: public-API surface (`pub mod` exposure of engine internals, missing `#[non_exhaustive]` on `Options` / `ChiselError` / `DrainInsertion` / `SpillwayLocation`), Cargo.toml publication metadata gaps, `License: TBD` in the README, and a small batch of code-quality, performance, and doc-drift items. None are correctness bugs. Highest leverage before the 1.0 freeze: I35 (`pub` → `pub(crate)` reshape), which forces the urgency of I36 (`#[non_exhaustive]` on the types that remain public); then I54–I57 (CI supply-chain check + MSRV pin + publication metadata + license) to unblock crates.io publication. The 2026-04-26 perf-review's deltas (F1/F4/F5 resolved, F2/F3/F6 unchanged) are recorded inline at the top of the new section. -> -> **Status note (2026-06-01):** the `chisel-performance` + `rust-performance` full hot-path sweep (output at `docs/reviews/perf-review-2026-06-01.md`) adds I77–I96 in a new "Perf-review findings (2026-06-01)" section below. No correctness bugs and no Don't-Break-List violations were found — the engine's data-handling is confirmed tight (zero-copy reads, in-place slot mutation, dead-on-prod `compact()`, verified I18 / I51 / 3-fsync discipline). The cluster splits into engine hot-path optimizations (I77–I89) and benchmark-harness integrity (I90–I96). The central conclusion is a **sequencing constraint**: the bench harness cannot currently validate an engine change (no `black_box`, no `[profile.release]`, no in-memory-backend row), so the I90/I91/I92 measurement-integrity fixes GATE the two engine P1s (I77 SipHash hasher, I78 per-insert XXH3 re-stamp) — and `black_box` (I90) must precede the LTO profile (I91) because LTO widens the dead-code-elimination window. Prior deferred perf items are unchanged: F2 (`read` → `to_vec`) and F3 (`Cell` counters) still UNCHANGED; I33 (`delete_many` per-leaf) and I34 (mmap cache) still DEFERRED. -> -> **Status note (2026-06-21):** the `deepdive-rust` fresh-eyes pass (output at `docs/reviews/review-20260621-185541.md`, run after PRs #34–#50 and the per-page format-versioning feature landed) adds **I104–I140** in a new "Deepdive review findings (2026-06-21)" section below. The *entire* prior executive summary (2026-06-16) is resolved with no regressions, so the new cluster is fresh — concentrated in error-classification defaults, one CI lint hole, the radix test gap, and doc drift introduced by the #46 dead-code sweep. No new P0 / data-loss bugs were found. Highest leverage first: **I108** (the 1,300-line PyO3 binding escapes the gating `cargo clippy` because `default-members` excludes it — a real CI hole that ships clippy regressions silently) and **I111** (the radix key-math — the engine's most off-by-one-prone code — has no property tests, and its one "two-level" test only reaches depth 1). Then the cheap zero-risk cluster: **I104** (`is_fatal` exhaustiveness test, NOT a default flip — see the review's countercase), **I127** (the `live_slots` SipHash maps the I77 FxHashMap pass missed), and the doc-drift fixes **I128 / I130 / I131 / I132 / I133**. Carried-not-regressed: F2 (`read`→`to_vec`) and the `delete_with_tag` partial-progress drop (re-filed as **I107**) remain open by design. The ADR-to-repo question (**I129**) is a maintainer process call. Three 2026-05-22 fixes have narrow follow-ups here (I46→I117, I63→I131, I71→I111) — the original fix was correct but didn't cover the site this re-review found. -> -> **Status note (2026-06-21 — lower-priority backlog progress + handoff):** after the higher-leverage cluster shipped (PRs #51–#55), the remaining lower-priority items were worked in themed PRs. **DONE (PRs #56–#59):** I109, I110 (*resolved-not-viable* — `--tests` pulls `getrandom`/edition2024 above MSRV; msrv stays lib-only), I116, I117, I119, I121, I123, I134, I136, I137. Several review premises were **corrected** while fixing: I116 "four unguarded sites" → three already-guarded (the real fix is *saturation*; `set_cache_max_bytes` doesn't decrement); I134 "three copies" → five. **DONE 2026-06-21 (PR #60):** **I120 + I126 + I122** — the full Handle/Tag newtype reshape across the Rust + Python surface (see each entry below). **DONE 2026-06-21 (the API-contract pass, pulled forward by maintainer ahead of priority order):** **I125** (`lookup_live` unification — the deepdive premise was *stale*: behavior was already uniform, so this was a zero-behavior-change DRY refactor, not the tombstone-rejection change the note anticipated) and **I124** (`# Errors` rustdoc on all 33 public methods + a self-enforcing `missing_errors_doc` gate). **STILL OPEN (back to priority order):** **I112** (the `FaultyFile` fault-injection layer, which *subsumes* I114 + I115 — drive a real I/O fault into poison + the IoError/CorruptSuperblock paths) is the next big item — design-heavy, warrants a brainstorm/spec/plan. Then the moderate/cosmetic P3s: **I106** (`CorruptSuperblock` cause field — design ready: `deserialize -> Result<_, SuperblockDefect>` + a `diagnose` helper + `CorruptSuperblock { defects }`, ~6 files incl. the I104 exhaustiveness test), **I107** (`delete_with_tag` runs in the caller's txn, so document "roll back on error" — no partial-progress to attach), **I113** (tighten weak assertions), **I139** (Python typed-exception contract tests), **I118** (plumb the freemap-aware `alloc` into membership `create_root`). - -Dependencies and batching drove this more than raw priority. Earlier items unblocked later ones. - -1. **I2** — first commit wipes the only valid superblock. One-day fix, unblocks every other durability guarantee. -2. **I15** — superblock `format_version` validation. One-hour fix, do while I2 is in review. -3. **I6** — `find_leaf` sentinel returns the root as the leaf. Latent corruption; needs a test that forces a sparse handle range. -4. **I1** — commit error handling. Design decided (poison model — see I1 below); implement after I2/I6 so the recovery path is clean. -5. **I18** — `persist_freemap` can reuse pages still referenced by the last-durable superblock. ✅ FIXED 2026-04-17. -6. **F3** — `read()` → `&self`. Do before F2 and I12 pile more API on top; also unblocks R5 (Python bindings). -7. **F2 + I7** — named roots and handle-table rollback tracking. Both touch the handle table / superblock boundary; one coherent PR. -8. **I3 + I4** — rollback file-extension cleanup and `next_page_id` seeding audit. -9. **Freemap bundle: R2 + I9 + I10 + I11 + I12 (F1)** — wire the freemap, plug the leaks, expose bulk delete. One coherent effort; reclamation has to be consistent. -10. **R1** — pack multiple values per data page. Biggest space/perf win; best done on top of a working freemap. -11. **R3 + I17** — selective defrag (and fix the stat accuracy while rewriting the loop). -12. **I13 + I14** — overflow hardening pass. -13. **Page-cache hardening: I19 + I20** — add bounds/asserts on `maybe_evict` and `claim_page`. ✅ FIXED 2026-04-17. -14. **Python binding cleanup: I21–I25** — ergonomics and dead-code audit. ✅ RESOLVED 2026-04-17 (I23 was a false alarm; the other four landed as one PR). -15. **P3 cleanup sweep** — I5, I8, I16, C1–C3, and the "invariants to verify" section. - -R4 (configurable superblock count) and R5 (Python bindings) sat outside this order — R4 was gated on I2, R5 on F3. Both have shipped. - ---- - -## Durability and crash safety - -### I1. Commit error handling — poison model [comment-pass] — **P0** ✅ IMPLEMENTED 2026-04-10 -**Where:** `transaction.rs` `commit()` - -**Problem:** `txn_counter` is incremented **before** the linearization write (`write_page(inactive)` + second `fsync`). If either fails: -- The in-memory counter is already bumped. -- `PageCache::flush()` has already cleared dirty flags on the first-phase pages. -- A naive retry produces a `txn_counter` gap and will **not** re-flush the now-clean pages. -- Rollback after partial commit failure is fragile because `rollback()` only discards pages currently dirty in cache. - -**Resolution:** Adopt a **poison model** (matches `std::sync::Mutex` semantics). On any commit error, the `TransactionManager` becomes poisoned; the only legal recovery is to `close()` and reopen. Reopen uses the existing shadow-paging recovery path (pick the winning superblock), which returns the database to the last durable state. - -**Rationale — fsyncgate:** On Linux (post-2018), a failed `fsync()` cannot be safely retried. The kernel records the error, reports it on the next `fsync()`, then **clears the error state**. A subsequent successful `fsync()` does not mean earlier data is durable — it may have been dropped from the page cache entirely. The only safe response is to treat the file as corrupt-in-memory and start over from a known-good on-disk state. PostgreSQL `PANIC`s on fsync failure for exactly this reason. macOS `F_FULLFSYNC` has similar semantics. Shadow paging + embedded single-writer means reopen is cheap, and it exercises the same recovery code path as a real crash — which is a testing win, not just a correctness one. - -**Implementation steps:** -1. Add `poisoned: Option` (or `bool` for v1) to `TransactionManager`. -2. Every public entry point checks it first and returns `ChiselError::Poisoned` if set. -3. `commit()` sets the flag on **any** error in its steps. No in-place recovery attempt. -4. `close()` / `Drop` is the only method that may run on a poisoned manager — drops the flock and file handle cleanly. -5. Any fatal `ChiselError` variant encountered outside commit (`ChecksumMismatch`, `CorruptSuperblock`, `IoError`) should also poison — fatal is fatal. -6. Document the recovery procedure: on `Err(Poisoned)`, drop the `Chisel` and call `Chisel::open` again. -7. Add a comment in `PageCache::flush()` noting the window between step 1 and step 5 where cached pages lie about durability, and why it's OK under poison (the manager is about to be discarded). - -**Downstream effects:** -- **C3** (cache dirty flags) — no fix needed under poison, but annotate. -- **I3, I4** — rollback still has to revert file extension, independently of poison. -- **F3** (`read()` → `&self`) — poison flag needs to be accessible from `&self` contexts. A `Cell` or `AtomicBool` solves this; do it as part of F3's refactor. - -### I2. First commit can wipe the only valid superblock [comment-pass] — **P0** ✅ FIXED 2026-04-10 -**Where:** `transaction.rs` `create_new` + commit slot selection - -`create_new` writes slot 0 with `txn_counter = 1` and leaves slot 1 all-zero (invalid). The first user commit increments to 2 (even) and overwrites slot 0 — the only previously valid superblock. A torn write on that commit leaves **no** valid superblock and `open_existing` errors with `CorruptSuperblock`. - -Fix: initialize both slots with staggered valid superblocks (e.g., counters 0 and 1) so there is always a fallback. - -### I3. Rollback does not revert file extension [comment-pass] — **P1** ✅ FIXED 2026-04-10 -**Where:** `transaction.rs` `rollback()` - -`rollback()` calls `cache.discard(id)` but never truncates the file or rewinds `next_page_id`. Every rolled-back transaction permanently grows the file with zero-checksum garbage pages. Unreachable, so crash-safe, but leaked until defrag. - -### I4. `PageCache::new_page()` may return IDs pointing at post-crash garbage [comment-pass] — **P3** (audit) ✅ RESOLVED 2026-04-10 -**Where:** `page_cache.rs` `new_page()` - -`next_page_id` is seeded from physical file length in `PageCache::new()`. If a previous crash (see I3) left the file extended past the authoritative superblock's `total_pages`, and the open path forgets to call `set_next_page_id`, `new_page()` returns IDs pointing at stale content. - -Audit as part of I3 cleanup: confirm `TransactionManager::open` always resets `next_page_id` from the winning superblock. - -### I5. `PageCache::truncate()` silently drops dirty pages [comment-pass] — **P3** ✅ RESOLVED 2026-04-10 (docs only — the behavior is intentional under the watermark rollback design) -**Where:** `page_cache.rs` `truncate()` - -No error or debug_assert if discarded entries are dirty. Safe as long as all callers are post-commit, but there is no runtime guard. Add a `debug_assert!(!entry.dirty)` on any future handle-table PR. - -### I18. `persist_freemap` can reuse pages the last-durable superblock still references [comment-pass 2026-04-17] — **P0** ✅ FIXED 2026-04-17 -**Where:** `transaction.rs` `persist_freemap` - -During commit, `persist_freemap` merged `txn_freed_pages` and `old_freemap_page` into `current_freemap` **before** calling `allocate_data_page` to pick a page for the new freemap snapshot. `FreeMap::allocate_first` returns the lowest free id, which was very likely `old_freemap_page` itself or one of the ids just merged from `txn_freed_pages`. The subsequent `claim_page` + `cache.flush()` then overwrote the bytes of a page that the **currently-committed** on-disk superblock still referenced. A crash in the window between that flush and the superblock fsync would leave the last-durable superblock pointing at overwritten bytes. - -This directly violated the shadow-paging invariant spelled out in `allocate_data_page`'s own doc comment ("pages reused by the freemap must not be referenced by the currently-committed superblock"). - -**Fix (landed 2026-04-17):** restructured `persist_freemap` to allocate the new freemap page BEFORE merging `txn_freed_pages` or `old_freemap_page` into `current_freemap`. At the moment of allocation, `current_freemap` still reflects only committed-state frees minus this transaction's allocations, so `FreeMap::allocate_first` can only return a page that was already free in the committed state or a freshly-extended page — both safe. The merges happen AFTER the allocation; the resulting freemap still serializes to disk with those ids marked free, so future transactions can reclaim them. - -**Regression test:** `persist_freemap_does_not_reuse_committed_live_pages` in `src/transaction.rs`. It seeds a committed freemap page via overflow-delete (R1 slot-packing otherwise keeps multi-slot data pages live), then runs a second commit whose deletes populate `txn_freed_pages`, and asserts the new `committed_roots.freemap_page` is not in the at-risk set (old freemap page ∪ frozen `txn_freed_pages`). The test is framed as a direct invariant check rather than a crash-injection harness because the at-risk set is observable purely in post-commit internal state. - -### I27. `commit()` silently drops `savepoints[*].freed_pages` when savepoints are still active [comment-pass 2026-04-22] — **P2** ✅ FIXED 2026-04-22 -**Where:** `transaction.rs` `commit_inner` (the `self.savepoints.clear()` at the end of commit) vs `release_inner` (which DOES merge freed pages back via `merged_freed.extend_from_slice(&sp.freed_pages)`) - -`release()` and `commit()` are asymmetric about what happens to a savepoint's `freed_pages`. When a savepoint is released, its `freed_pages` are merged back into the enclosing transaction's `txn_freed_pages`, so commit can return those ids to the freemap. When `commit()` runs with savepoints still on the stack, `commit_inner` simply calls `self.savepoints.clear()` — the per-savepoint `freed_pages` lists are dropped on the floor. `persist_freemap` only iterates `self.txn_freed_pages`, so any page freed in a scope enclosed by an unreleased savepoint is permanently orphaned from the freemap. - -**Leak workflow:** `begin → delete(h1) → delete(h2) → savepoint("s") → → commit`. The pages backing h1 and h2 were moved from `txn_freed_pages` into the savepoint's `freed_pages` by `savepoint_inner`, the savepoint was never released, commit clears the stack, those ids never reach the freemap. Nothing corrupts — the superblock is consistent, the pages are unreachable — but the freemap no longer knows they are reusable. Defrag is the only thing that can reclaim them. - -**Why prior passes missed it:** the 2026-04-10 pass predated R2 (freemap wiring) — leaks were known and batched into the freemap bundle. The 2026-04-17 pass focused on the I18 `persist_freemap` restructure and the page-cache hardening; savepoint semantics weren't in scope. The bug has been latent since R2 landed and will trip any workload that commits with a savepoint still active. - -**Fix (landed 2026-04-22):** chose option (1) — at the top of `commit_inner`, before `persist_freemap`, iterate every active savepoint and `append` its `freed_pages` onto `self.txn_freed_pages`. This matches `release_inner`'s merge pattern but applied across the full stack. The existing `savepoints.clear()` at step 5 still runs afterwards; we drain rather than iterate-by-reference so the savepoints don't hold stale `freed_pages` if step 5 ever changes. No new error variant, no caller-visible behaviour change beyond the leak going away. - -**Regression test:** `commit_with_active_savepoint_returns_freed_pages_to_freemap` in `src/transaction.rs`. Seeds two overflow-sized handles, opens a transaction, deletes both (populating `txn_freed_pages`), takes a savepoint (which empties `txn_freed_pages` into `savepoint.freed_pages`), commits WITHOUT release, and asserts `FreeMap::is_free(&committed_freemap, id)` holds for every previously-captured id. Pre-fix, none of the ids were marked free — they were permanently leaked. - -### I29. Split `format_version` into packed MAJOR / MINOR for public stability promise [infrastructure 2026-04-22] — **P1** (pre-1.0 foundation) ✅ PHASE 1 LANDED 2026-04-22 -**Where:** `page.rs` `FORMAT_VERSION`, `transaction.rs::open_existing` gate at the I15 site - -**Motivation:** the README's "sacred within a major version" promise requires distinguishing additive minor changes from structural major changes in the on-disk marker. The pre-I29 scheme was a flat `u32 FORMAT_VERSION = 2` checked with exact equality, which conflated the two: any change bumped the number, any mismatch rejected the file. Layering the public stability guarantee on top of that would have required interpretation conventions that lived outside the field itself. - -**Scheme (byte-packed u32):** -- Upper 16 bits = MAJOR. Lower 16 bits = MINOR. -- `FORMAT_MAJOR_VERSION` and `FORMAT_MINOR_VERSION` are `u16` constants; `FORMAT_VERSION` is derived by `pack_format_version(major, minor)` at compile time. -- First 1.0 release: MAJOR = 1, MINOR = 0, `FORMAT_VERSION = 0x00010000`. -- Helpers: `pack_format_version(major, minor)`, `format_major(v)`, `format_minor(v)`. All `const fn` so they compose in constants. - -**Why packed over decimal-coded** (e.g. `major * 100 + minor`): semantics compile into the data-type (`>> 16`, `& 0xFFFF`) rather than relying on a "why 100?" arithmetic convention. Same `u32` on-disk width, same superblock bytes 4..8. - -**Phase 1 (landed 2026-04-22):** open-time gate now compares MAJOR only (`format_major(sb.format_version) != FORMAT_MAJOR_VERSION`). A file written by any 1.x binary opens in any other 1.x binary regardless of minor drift — which is what makes the README promise true. Minor-newer files are accepted as read+write for now because there are no minor variants yet to protect. - -**Phase 2 (deferred until 1.1 run-up):** add a "refuse writes if file MINOR > binary MINOR" arm to protect against a binary silently clobbering superblock fields added in a later minor. Likely shape: introduce a new operational error (`NewerFormatMinor`) or reuse `ReadOnlyMode`; set a flag on the `TransactionManager` at open time; check it in `begin_inner`. No-op today (no newer-minor files exist), so deferring costs nothing but documenting the intent now. - -**Pre-1.0 compatibility note:** any file written by a prior development build carries `format_version = 1` or `format_version = 2` in the flat scheme. Under the packed interpretation those decode as MAJOR = 0, MINOR = 1 or 2. MAJOR = 0 ≠ current MAJOR = 1 → rejected with `UnsupportedFormatVersion`. This is the documented pre-1.0 break; there are no production DBs to migrate, and release notes call it out. MAJOR = 0 is implicitly reserved forever as "pre-1.0 development" and will never be written by a released binary. - -**Regression test:** `format_version_gate_is_major_only` in `src/transaction.rs`. Creates a fresh database, closes, patches both superblock slots to (a) the current major with a bumped minor — asserts open succeeds (pre-fix this rejected with `UnsupportedFormatVersion`); then patches to a bumped major — asserts open fails with `UnsupportedFormatVersion`. Exercises both halves of the MAJOR-only check. - -**Phase 2 (landed 2026-06-21):** the file-MINOR write-gate is live. If the file's MINOR exceeds the binary's MINOR, the open path forces the store read-only (`TransactionManager::open_existing` calls `PageIo::force_read_only`) — no writes are permitted. See `docs/specs/2026-06-21-per-page-format-versioning-design.md`. - -### I31. Per-page format version byte + reserved common-header space [infrastructure 2026-04-22] — **P1** (pre-1.0 foundation) ✅ PHASE 1 LANDED 2026-04-22 -**Where:** `page.rs` `page_format_version` / `PAGE_FORMAT_VERSION_CURRENT` / `COMMON_RESERVED_*`; every non-superblock page-type module's `init_page` (data_page, overflow, freemap, handle_table) - -**Motivation:** the upgrade plan for post-1.0 format evolution calls for **lazy per-page migration** — reads dispatch on each page's declared format version; writes always produce the current format; pages get migrated as the application happens to touch them. A later task (the "eager upgrader", see below) sweeps remaining cold pages. Both depend on having a per-page format version to dispatch on. Pre-I31 there was no such byte. - -**Scheme:** -- Each non-superblock page carries a one-byte `page_format_version`. `PAGE_FORMAT_VERSION_CURRENT = 0` is "the layout as of the I31 commit." -- Storage offset is per-type, dispatched via `page::page_format_version(buf)`: - - `Data`, `Overflow`, `FreeMap`: byte 1 (was "reserved / padding" today, already zero on every existing page). - - `HandleTable`: byte 2 (byte 1 holds `FLAG_LEAF` / `FLAG_INTERIOR`; the flag is forensic-only, no runtime code reads it, moving it would have cost a gratuitous format break). -- Bytes 8..16 of every non-superblock page are RESERVED for future common-header fields (8 bytes / 64 bits, `COMMON_RESERVED_OFFSET` / `COMMON_RESERVED_LEN`). Universally zero today; a future common field added there will bump the affected page type's per-page version, not the superblock's MAJOR. This generalizes and extends the existing data_page "reserved for future per-page txn_counter" slot. - -**Why per-type dispatch rather than uniform byte 1:** moving `FLAG_LEAF`/`FLAG_INTERIOR` would not have been a behavior change (no code reads them) but WOULD have made every existing handle-table page on disk differ from a freshly-initialized one at byte 1 vs byte 2, which would have required a MAJOR bump to avoid silent reinterpretation. The per-type dispatch costs one `if` in the reader and avoids any break — pre-I31 files Just Work (byte 1 or 2 was already zero = "version 0" = current). - -**Phase 1 (landed 2026-04-22):** byte allocation only. `page_format_version` exists and is testable; every page-type `init_page` writes `PAGE_FORMAT_VERSION_CURRENT` explicitly (even though `buf.fill(0)` already zeroed it) so future CURRENT bumps flow through a single authoritative site per type. No dispatch code yet — there are no non-zero versions in use. - -**Phase 2 (deferred — the "eager upgrader"):** when a realistic format change requires it, the read path in the affected page-type module grows a version switch (`match page_format_version(buf) { 0 => read_v0(buf), 1 => read_v1(buf), _ => Err(Unsupported) }`), writes always produce the latest version, and an opt-in `db.upgrade(on_progress)` method rewrites every cold page. `on_progress: FnMut(UpgradeProgress)` lets the caller surface progress to logs / TUI / IPC. A later phase 3 would wrap this in an async worker thread for fully-unattended upgrade — but per the design discussion, that's polish on top of the synchronous scanner, not a separate architecture. - -**Regression tests:** `page_format_version_dispatches_by_page_type` (pure unit test pinning the per-type offset) and `fresh_pages_report_current_version` (asserts Data/FreeMap `init_page` output reports `PAGE_FORMAT_VERSION_CURRENT` through the `page_format_version` reader; Overflow and HandleTable init through their cache-aware paths and are covered end-to-end by existing integration tests). Both in `src/page.rs`'s test module. - -**Phase 1a (landed 2026-06-21):** the read-side dispatch helpers `page::page_format_version()` and `page::current_version()` are now active — every `init_page` site stamps `current_version(type)` explicitly. The per-module version switch (Phase 2 / "eager upgrader") remains deferred pending a real format change that needs it. See `docs/specs/2026-06-21-per-page-format-versioning-design.md`. - ---- - -## Handle table - -### I6. `find_leaf` sentinel returns the root page, not the leaf [comment-pass] — **P0** ✅ FIXED 2026-04-10 -**Where:** `handle_table.rs:252` - -On a zero child pointer mid-descent, `find_leaf` returns `(root_page_id, 0)` — the original root, **not** the leaf currently being walked. The caller (`lookup`) then reads slot 0 of whatever that page is (possibly an interior page) as a `HandleEntry`. - -Today this "works" because: -- For small child-0 page IDs, byte 2 of the little-endian u64 is usually zero and decodes as `HandleFlags::Deleted`. -- For child-0 pointing at page ID ≥ 2^16, byte 2 can be nonzero (0x01 → `Live`, 0x02 → `Overflow`) and **a bogus HandleEntry will be returned for a handle that does not exist.** - -Fix: return a proper `Option<(page_id, slot)>` or a distinct sentinel, or have `lookup` check for zero child pointers directly during descent. Resolve C1 at the same time. - -### I7. Interior COW pages not recorded in `txn_dirty_pages` [comment-pass] — **P1** ✅ FIXED 2026-04-10 -**Where:** `handle_table.rs` `insert_recursive` - -Only the final new root is pushed to `txn_dirty_pages`. Intermediate cloned interior pages are dirty in the cache but not tracked for rollback, so `rollback()` will not discard them. A subsequent `cache.flush()` from a future commit will write them to disk as orphans. - -Batch with F2 — same area of code. - -### I8. `find_leaf` sentinel accidentally relies on page 0 being the superblock [comment-pass] — **P3** ✅ FIXED 2026-04-10 -**Where:** `handle_table.rs` - -The "zero child pointer is unambiguous" invariant only holds because page 0 is superblock A and never a handle-table node. True today but not enforced. Add an `assert!` when next touching `handle_table.rs`. - -### I26. `find_leaf` does not bounds-check `child_idx` against `PTRS_PER_INTERIOR` [comment-pass 2026-04-22] — **P1** ✅ FIXED 2026-04-22 -**Where:** `handle_table.rs` `find_leaf` (the descent loop around line 419) - -For any `handle >= HandleTable::capacity()`, the descent loop computes `child_idx = remaining / child_span` without bounding the result to `< PTRS_PER_INTERIOR (= 1021)`. The resulting byte offset `DATA_PAGE_HEADER_SIZE + child_idx * CHILD_PTR_SIZE` walks off the valid child-pointer region. At the first-overflow boundary (`child_idx == 1021`, reachable with `handle == 520_710` at depth 1) it reads `buf[CHECKSUM_OFFSET..CHECKSUM_OFFSET + 8]` — the XXH3 checksum bytes of the interior page — and treats that nonzero u64 as a child page id. The descent then calls `cache.get(checksum_as_id)`, which will almost always fail with `InvalidPageId` or `ChecksumMismatch`, both of which the TransactionManager classifies fatal and poisons on. At `child_idx >= 1022` the slice op panics outright with out-of-bounds access. - -`lookup` is the only external call site (reached from `read()`, `update()`, `delete()`, `delete_many()`, and `handles()`), and it does not pre-validate the handle. A caller who supplies a u64 larger than the current tree capacity triggers the bug externally — an operational mistake whose expected response is `InvalidHandle` gets escalated to an engine-poisoning fatal error or a process crash. - -**Same failure shape as the historical I6**: `find_leaf` reporting wrong information for a handle that does not exist in the tree. `insert` is unaffected because it pre-grows via `while handle >= capacity { grow() }`. - -**Fix (landed 2026-04-22):** added a capacity guard at the top of `find_leaf`, scoped to `self.depth > 0`. At depth 0 the existing `handle % ENTRIES_PER_LEAF` is already total (wraps cleanly for any u64) and the descent loop never runs, so the guard only matters on the descent path where the out-of-bounds `child_idx` actually arises. Scoping it this way avoids silently changing depth-0 semantics for callers that happen to pass large handles. - -**Regression test:** `lookup_handle_beyond_capacity_returns_none` in `src/handle_table.rs`. Grows the tree to depth 1 and asserts both the first-overflow boundary (`handle == ENTRIES_PER_LEAF * PTRS_PER_INTERIOR`, the historical reads-checksum-as-child case — which without the fix returned `Err(InvalidPageId { page_id: })`) and `handle == u64::MAX` (the would-panic case) both return `Ok(None)`. - -### I32. `delete_inner` walks the handle table twice per handle [perf-review 2026-04-26] — **P2** ✅ IMPLEMENTED 2026-04-26 -**Where:** `transaction.rs` `delete_inner`; `handle_table.rs` `delete` - -**Problem:** `delete_inner` calls `handle_table::lookup` to read the -existing entry, then calls `handle_table::delete` which is -`insert(deleted_entry)` underneath — that walks the tree a second time, -COWing as it goes. Per handle, the radix tree is descended twice. For -1000 sequential deletes inside one transaction, that's 2000 tree walks -instead of 1000. - -**Fix:** Fuse the two operations. `handle_table::delete` becomes a -single recursive descent that reads the existing entry from the leaf, -writes the tombstone in the same COW pass, and returns -`(new_root, Option)`. The Option lets `delete_inner` -distinguish "was Live/Overflow → escalate to release" from "was absent -or already tombstoned → escalate to InvalidHandle." `delete_inner` -becomes one tree walk per handle. - -**Bonus optimizations falling out of the fusion:** - - For absent handles or already-tombstoned entries, the new - implementation returns `(root, None)` immediately without COWing - the path or growing the tree. Today's code path (unreachable from - `delete_inner` since `lookup` short-circuits first) would have - COWed and possibly grown — wasted writes that no caller benefits - from. - - `delete` no longer calls `grow()`. Tree growth stays in `insert`. - -**Regression tests:** Five new unit tests in `handle_table.rs` -covering the four return-value cases (Live, Overflow, already-Deleted, -absent) plus a no-tree-growth assertion for the beyond-capacity case. - -### I33. `delete_many` is not actually batched per-leaf [perf-review 2026-04-26] — **P3** (deferred) -**Where:** `transaction.rs` `delete_many_inner` - -**Problem:** `delete_many` is a thin loop over `delete_inner`. After -I32, each delete walks the handle table once per handle. For dense -delete patterns (e.g., 1000 handles concentrated in 5 leaves), a true -batched implementation would walk once per unique leaf — 5 walks -instead of 1000, with all tombstones for each leaf written in a -single COW pass. - -**Why deferred:** Sparse delete patterns get no benefit; the win is -shape-specific. No concrete client currently demands bulk delete -latency below the fsync floor. PR 4 of the bench-suite series -(scenario tier S3 "mutation log") will surface whether real workloads -hit the dense pattern; if they do, this becomes actionable. Until -then, YAGNI. - -**Fix when actionable:** - - Sort handles by their target leaf (computed via `handle / span` - decomposition without actually descending). - - Group handles by leaf. - - For each unique leaf: descend the tree once, COW the path, write - all tombstones for that leaf in one pass. - - Parallel optimization for `release_data_slot`: handles whose Live - entries point to the same data page can have their slot-count - decrements batched. - - Estimated 5–10× speedup for dense delete patterns; no change for - sparse. - ---- - -## Space leaks (freemap not wired) - -These are all facets of the v1 simplification that the freemap bitmap is -built but unused. They become real bugs the moment the freemap is wired up -(R2) — reclamation logic has to know what to reclaim. **Fix all of these -together as the freemap bundle.** - -### I9. `update()` inline→inline path leaks the old data-page slot [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `transaction.rs` `update()` - -The old `(page_id, slot_index)` is never freed nor added to `txn_freed_pages`. Combined with "fresh page per insert" (R1), every update leaks a page and consumes a new one. - -### I10. `delete()` leaks the Live data-page slot [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `transaction.rs` `delete()` - -Handle-table entry is removed; the data page lingers forever. - -### I11. `commit()` drops `txn_freed_pages` on the floor [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `transaction.rs` `commit()` - -The field name "freed" implies reclamation, but the vector is cleared on commit without returning pages to any freemap. - -### I12. `delete_subtree(handle)` bulk-delete primitive [client] — **P2** (see also F1) ✅ IMPLEMENTED 2026-04-10 as `delete_many` - -> `drop_table` and `drop_index_table` currently leak row/node handles. Chisel's defrag can reclaim them eventually, but a `delete_subtree(handle)` or similar bulk-delete primitive on Chisel would be cleaner. - -Needs design: what defines the subtree? Options: -- Client provides an iterator of handles to delete in one transaction. -- Chisel grows a "handle group" concept that can be bulk-freed. -- Handles get optional parent pointers (probably too invasive). - -The simplest shape is probably `delete_many(&[Handle])` with atomic semantics. Fold into the freemap bundle. - ---- - -## Other bugs - -### I13. `Overflow::write` panics on zero-length values [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `overflow.rs:50` - -For `value.len() == 0`, `num_pages == 0`, no pages are allocated, and the function unconditionally returns `Ok(page_ids[0])` — index out of bounds. Zero-length values stay inline today, so this is unreachable, but it is a latent panic with no defensive check. - -### I14. Overflow chain read/delete lack cycle detection [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `overflow.rs` `read`, `delete` - -A corrupt chain with a cycle loops forever. `read` also does not bound `result.len()` against `total_length`. Should bail with `ChiselError::CorruptPage` after exceeding expected length or a cap on chain depth. - -### I15. `Superblock::deserialize()` does not validate `format_version` [comment-pass] — **P1** ✅ FIXED 2026-04-10 -**Where:** `superblock.rs` - -A future v2 file opened by a v1 binary will be silently accepted and fields could be misinterpreted. Add a check against `page::FORMAT_VERSION` in `deserialize` or `select`. Cheap; do it early. - -### I16. `PageIo::read_page` returns `UnexpectedEof` instead of `InvalidPageId` [comment-pass] — **P3** ✅ FIXED 2026-04-10 -**Where:** `page_io.rs:47` - -Error-quality only. Add a bounds check on any PR touching `page_io.rs`. - -### I17. Defrag stats count values, not pages [comment-pass] — **P2** ✅ FIXED 2026-04-10 -**Where:** `defrag.rs:59-60` - -`stats.pages_examined` and `stats.pages_freed` are populated from a per-value counter, not actual page counts. Fix while doing R3. - ---- - -## Page-cache hardening - -### I19. `PageCache::maybe_evict` grows unboundedly when every page is dirty [comment-pass 2026-04-17] — **P2** ✅ FIXED 2026-04-17 -**Where:** `page_cache.rs` `maybe_evict` - -When every cached entry was dirty, the eviction loop broke out and the cache silently grew past `max_pages` without bound. A long-running transaction allocating many `new_page()`s without intervening flushes could exhaust memory. - -**Fix:** added a `HARD_CEILING_MULTIPLIER` constant (currently 8×) and a check at the end of `maybe_evict` that returns the new operational `ChiselError::CacheFull { limit }` once `entries.len()` exceeds `max_pages * HARD_CEILING_MULTIPLIER`. The soft-limit semantics for `max_pages` are unchanged — write-heavy transactions can still grow past it — but runaway growth now trips a recoverable error. Caller recovery is to commit (which flushes, freeing the dirty pin) or roll back. The Python binding gets a parallel `CacheFullError` in the OperationalError tier. Tests `cache_full_fires_when_all_pages_dirty_past_hard_ceiling` and `cache_full_is_recoverable_via_flush` in `page_cache.rs` cover trigger and recovery; `fresh_manager`'s test cache size in `transaction.rs` was bumped from 64 to 1024 so existing high-allocation tests stay well under the new ceiling. - -**SUPERSEDED 2026-05-04** by the spillway design (see -`ARCHITECTURE.md` Cross-cutting concepts → Spillway). The -`HARD_CEILING_MULTIPLIER` constant is removed; the cache is now a -strict bound (`Options::cache_max_bytes`); overflow dirty pages -spill to a sidecar `Spillway` file capped at -`Options::spillway_max_bytes`. The pre-existing `CacheFull` variant -remains operational and now fires only when the spillway is -disabled (`spillway_max_bytes = 0`) at the strict cache cap. New -operational error `SpillwayFull { limit_bytes }` fires when both -cache and spillway are exhausted. - -### I20. `PageCache::claim_page` silently discards dirty writes on re-insertion [comment-pass 2026-04-17] — **P3** ✅ FIXED 2026-04-17 -**Where:** `page_cache.rs` `claim_page` - -If the freemap ever handed back an id the current transaction had already dirtied, `claim_page` would silently replace the cached entry and lose the pending writes. The only legitimate caller is `allocate_data_page` via the freemap, which post-I18 is well-behaved — the invariant just wasn't enforced. - -**Fix:** added `debug_assert!(!self.is_dirty(page_id), …)` at the top of `claim_page` so a violation surfaces immediately in debug builds rather than as silent data loss hours later. Release builds are unchanged. Test `claim_page_asserts_on_dirty_page` covers the assertion (gated on `cfg(debug_assertions)`). - -### I28. `CacheFull` raised during commit's `persist_freemap` poisons the manager [comment-pass 2026-04-22] — **P2** ✅ FIXED 2026-04-22 -**Where:** `page_cache.rs` `maybe_evict` (the hard-ceiling check added in I19) × `transaction.rs` `commit()` / `poison_on_fatal` - -I19 introduced `ChiselError::CacheFull` as an **operational** error: documented as "caller recovers by committing (flushes → pages become evictable) or rolling back (discards all dirty pages)", and correctly classified `is_fatal() == false`. But `commit_inner` runs `persist_freemap` → `allocate_data_page` → `claim_page` / `new_page` → `maybe_evict` **before** `cache.flush()` drains dirty pages. If `maybe_evict` fires `CacheFull` at that point, the error propagates out of `commit_inner` and `commit()`'s `poison_on_fatal` wrapper poisons the TransactionManager regardless of `is_fatal()`. - -The resulting behaviour violates the operational contract in a particularly painful way: the recovery advice is "commit to flush," and commit is precisely what failed. A caller encountering `CacheFull` during commit has no legal action other than `close()` + reopen, which is the poison-model recovery — `CacheFull` was effectively reclassified fatal by the commit wrapper without anyone noticing. - -In practice the window is narrow: the transaction has to be at the hard ceiling with every page dirty at the moment `persist_freemap` allocates. But the semantic mismatch is real, and any user who hits it gets an inexplicable downgrade from "operational" to "must-reopen." - -**Fix (landed 2026-04-22):** chose option (1) — added `self.cache.borrow_mut().flush()?;` at the top of `commit_inner`, before `persist_freemap`. The drain clears every dirty pin so `persist_freemap`'s own `allocate_data_page` can evict clean pages rather than trip the hard ceiling. `CacheFull` can no longer surface on the commit path. Cost: one extra fsync per commit (2 → 3 total). Consistent with the project's "durability over performance" posture and cheaper than the alternative ("reclassify `CacheFull` as fatal during commit"), which would require caveats throughout the docs and the Python error hierarchy. - -**Ordering safety:** the pre-drain does not weaken the shadow-paging invariant. Shadow paging requires "data-page writes durable BEFORE superblock write durable" — step 1's existing flush (now operating on just the one freemap page persist_freemap materializes) still runs between persist_freemap and the superblock write. The pre-drain only shifts user-dirty page writes earlier within the same pre-superblock window; both are part of the same durable write set the superblock linearizes. - -**Regression test:** `commit_does_not_poison_when_cache_is_past_hard_ceiling` in `src/transaction.rs`. Constructs a `TransactionManager` with `max_pages=4` (hard ceiling 32), saturates the cache via an allocate-until-`CacheFull` loop in a transaction that also has a non-empty `txn_freed_pages` (so `persist_freemap` does not take its early-exit), then calls `commit()` and asserts both `Ok(())` and `!is_poisoned()`. Pre-fix commit returned `Err(CacheFull { limit: 32 })` and poisoned the manager; post-fix both assertions hold. - -### I34. mmap-backed shadow page cache region [client 2026-04-30] — **P3** (deferred design) -**Where:** `page_cache.rs` (cache storage backing) - -**Problem:** Today's `PageCache` stores pages as `Box<[u8; PAGE_SIZE]>` heap allocations indexed by `HashMap`. Memory is process RSS, capped by `Options::cache_max_bytes` (default 8 MiB). Workloads with working sets larger than the cache cap either need the user to raise `cache_max_bytes` (consuming proportional RSS) or accept high cache-miss rates against the database file. (Note: as of 2026-05-04 the cache is no longer elastic — `cache_max_bytes` is a strict cap; the spillway sidecar handles overflow dirty pages via `Options::spillway_max_bytes`. The deferred mmap design replaces the cache storage backing itself, not the spillway, so the design here remains valid; only the per-`PageCache` memory ceiling math needs adjustment to reference `cache_max_bytes` instead of the removed `max_pages × HARD_CEILING_MULTIPLIER`.) - -**Proposed design:** Keep the cache logic unchanged — `HashMap`, `LruIndex`, `dirty_count`, hit/miss counters. Replace the `Box<[u8; PAGE_SIZE]>` storage with offsets into an mmap'd region backed by a separate ephemeral file: - -- On `PageCache::new`, allocate a sized region in a temp file (preferably `O_TMPFILE` on Linux, or `open(O_CREAT, O_EXCL) + unlink` on macOS — the file has no path on the filesystem after the unlink, and is auto-released on process exit or crash). Cleanup is automatic; no leftover state. -- Each `CacheEntry` stores an offset into the region rather than a heap pointer. Reads / writes go through the mmap pointer. -- The OS pages cold cache entries to the cache file under memory pressure; pages them back in on access. Effective cache capacity becomes the file size (configurable in GBs), not the RSS budget (capped at MBs). - -**Architectural compatibility:** - -- *Checksum-on-load invariant unchanged.* Cache loads still go through `load_page`, which validates the XXH3 checksum from the database file before the bytes enter the cache. The mmap region is a transient, process-private store; what's in it has already been validated. -- *COW lifecycle unchanged.* New pages are allocated via `cache.new_page()` the same way; dirty pages still pinned against eviction; the mmap is just where the bytes live. -- *Counter semantics unchanged.* `cache_hits` / `cache_misses` continue to mean "was the entry in our HashMap?" — orthogonal to whether the kernel currently has the mmap'd page resident in RAM. The Chisel-internal cache abstraction is one layer above the OS's page-resident state. -- *Commit protocol unchanged.* Database-file fsync semantics are unaffected — the cache file is never part of the durability path. The two-fsync ordering, the pre-drain flush, and the poison model all stay exactly as today. - -**Implementation questions for the eventual design pass:** - -- Slot allocator within the mmap region: linear append vs. free-list of fixed-size slots indexed by offset. -- Cache file size policy: fixed at open vs. grow-as-needed via `ftruncate`. -- Behavior at cache-file `ENOSPC`: surface as a new error, evict more aggressively, or fall back to heap allocation. -- `O_TMPFILE` availability is Linux-specific; macOS needs the open-then-unlink dance, which has a tiny window where the path exists. -- Default-on or feature-flagged: a feature flag preserves the current `Box<[u8]>` cache for users who prefer it (tests can stay unchanged), at the cost of two code paths to maintain. - -**Why deferred:** The actual win shows up at working sets larger than ~64 MB (the current hard ceiling). PR 4's micro grid will tell us whether real workloads hit that limit. If they do, this becomes actionable; if they don't, the existing in-process cache is fine and the implementation complexity isn't justified. - -**Source:** Proposed by the Chisel client on 2026-04-30 during PR 3 brainstorming. - ---- - -## Python binding - -These are all from the 2026-04-17 pass over the `python/src/` subcrate. Python-side API surface items; none block the Rust core, but they should be settled before R5 ships broadly. - -### I21. `PyChisel` latent `RefCell` re-entry hazard [comment-pass 2026-04-17] — **P3** ✅ DOCUMENTED 2026-04-17 -**Where:** `python/src/db.rs` - -The existing comment said "Python's GIL prevents concurrent re-entry" — true cross-thread, but not for a hypothetical future same-thread Rust→Python→PyChisel callback. No such callback path exists today, so this was a documentation fix only: the comments at the top of `db.rs` and above `with_inner_io` / `with_inner_mut_io` now distinguish cross-thread from same-thread re-entry and spell out what a future callback API would need to do (use `try_borrow_mut` with an explicit reentrancy error, or reshape the engine call so the mutable borrow is released before the callback fires). - -### I22. `PySavepoint::rollback_to()` is silently idempotent [comment-pass 2026-04-17] — **P3** ✅ FIXED 2026-04-17 -**Where:** `python/src/savepoint.rs` - -An explicit second `release()` or `rollback_to()` on a finished savepoint now raises the new `AlreadyFinishedError` (operational tier) rather than silently succeeding. The `__exit__` path intentionally stays idempotent — the `finished` guard short-circuits without raising so normal `with sp:` usage is unaffected whether the user also called an explicit method inside the block. Regression tests: `test_savepoint_second_release_raises`, `test_savepoint_second_rollback_to_raises`, `test_savepoint_explicit_then_with_exit_is_silent`. - -### I23. `DuplicateSavepointError` may be dead code [comment-pass 2026-04-17] — **P3** ✅ RESOLVED 2026-04-17 (not actually dead; issue entry was incorrect) -**Where:** `python/src/errors.rs` - -The comment-pass entry claimed `ChiselError::DuplicateSavepoint` did not exist in `src/error.rs` and that only `SavepointNotFound(_)` was matched in `to_py_err`. Both assertions were wrong. `ChiselError::DuplicateSavepoint(String)` is declared in `src/error.rs`, is raised by `TransactionManager::savepoint()` when a name is reused (exercised by the existing `operational_error_does_not_poison` unit test at `src/transaction.rs`), and is routed in `python/src/errors.rs::to_py_err` to the Python-side `DuplicateSavepointError` class. No code change needed; this entry is preserved for audit-trail value. - -### I24. `PyTransaction` has no explicit `.commit()` / `.rollback()` methods [comment-pass 2026-04-17] — **P3** ✅ FIXED 2026-04-17 -**Where:** `python/src/transaction.rs`, `python/chisel/chisel.pyi` - -Explicit `.commit()` and `.rollback()` methods are now exposed on `PyTransaction`, mirroring the shape of `PySavepoint.release()` / `.rollback_to()`: both drive the engine and set the `finished` guard so a subsequent `__exit__` short-circuits silently; a second explicit drive raises `AlreadyFinishedError`. `.pyi` stubs updated accordingly. Regression tests: `test_tx_explicit_commit`, `test_tx_explicit_rollback`, `test_tx_second_commit_raises`, `test_tx_commit_then_with_exit_is_silent`. - -### I25. `db.close()` silently cancels live `PyTransaction` / `PySavepoint` objects [comment-pass 2026-04-17] — **P3** ✅ FIXED 2026-04-17 -**Where:** `python/src/db.rs close()` + `with_inner_mut_io` contract - -After `db.close()` clears `inner`, any subsequent call through a still-live `PyTransaction` or `PySavepoint` (including an automatic `__exit__` commit on the enclosing `with` block) now raises the new `ClosedError` (operational tier) instead of `PoisonedError`. The `is_poisoned` getter still reports `True` for a closed handle — it answers the "can this handle still do work?" question — but the distinct exception class lets callers tell "I closed this" apart from "Rust-side corruption". Regression tests: `test_close_then_call_raises_closed`, `test_closed_error_is_not_poisoned_error`, `test_close_inside_transaction_surfaces_as_closed`. - ---- - -## Misleading existing comments - -### C1. `handle_table.rs:252` — "Will read as Deleted." — **P3** (byproduct of I6) ✅ RESOLVED 2026-04-10 -Resolved alongside I6 — the comment was removed when `find_leaf` was changed to return `Option`. - -### C2. `freemap.rs` `allocate_near` — "then falls back to allocate_first" — **P3** ✅ FIXED 2026-04-10 -The doc comment claims a fallback to `allocate_first`, but the implementation just exhausts its own outward radius scan and never calls `allocate_first`. Behaviorally equivalent when only one free bit exists, but the doc is wrong. Fix on any freemap-adjacent PR. - -### C3. `page_cache.rs` original header — "dirty pages are never evicted" — **P3** (annotate under I1) ✅ RESOLVED 2026-04-10 -Annotated in `PageCache::flush()` as part of I1: documented the "durability window" between dirty-flag clearing and the trailing fsync, explained that the I1 poison model is what makes the window benign, and flagged what would need to change if the poison model is ever weakened. - -### C4. Documentation sweep [comment-pass 2026-04-22] — **P3** (batch) ✅ RESOLVED 2026-04-22 - -The 2026-04-22 pass surfaced a cluster of small doc / sharp-edge items that didn't warrant individual entries. Landed as a single cleanup commit. - -- **`superblock.rs:152–153`** — deleted the misleading "a deserialized value of 0 is treated as 'legacy'" sentence; replaced with an explicit note that any out-of-range value (including 0) is rejected by `deserialize` because a zero modulus would be catastrophic for the `txn_counter % superblock_count` slot-selection math. -- **`superblock.rs::select()`** — added a "tie-break policy" paragraph documenting that `max_by_key` returns the first maximum in iteration order (lowest slot index wins) and noting the narrow scenarios where ties can legitimately appear. -- **`error.rs::CorruptSuperblock`** — expanded the variant comment to state that a slot rejected for out-of-range `superblock_count` also surfaces as `CorruptSuperblock`; the generic Display message is documented so operators know to inspect raw slot bytes if a specific cause is needed. -- **`stats.rs::file_size_bytes`** — replaced the "during a commit in progress" phrasing (which implied a concurrent observer that Chisel's single-writer model cannot have) with the real cause: post-crash orphan pages in the file tail, overwritten on next allocation (I4 territory). -- **`page_cache.rs` header** — rewrote the soft-limit blurb to spell out the hard-ceiling design (`max_pages * HARD_CEILING_MULTIPLIER`), reference I19, and note that I28's pre-drain prevents `CacheFull` from ever arising on the commit path. -- **`page_cache.rs::new`** — added `let max_pages = max_pages.max(1);` with an explanatory comment. A caller passing 0 would otherwise have set the hard ceiling to 0 and tripped `CacheFull` on the first allocation regardless of workload. No callers pass 0 in practice, but the clamp turns a confusing constructor-time mistake into correct (if inefficient) behaviour. -- **`freemap.rs` header** — replaced the stale "BUILT BUT NOT WIRED IN" note (accurate pre-R2) with a description of how the module is actually wired: `allocate_data_page` prefers the freemap, reclamation happens in `persist_freemap`, I18 ordering is called out, and the overflow / handle-table carve-outs are noted. -- **`transaction.rs::read`** — merged the two consecutive `/// Read a value by handle.` doc paragraphs into one; the second heading line was a leftover from the F3 doc update. -- **`python/chisel/__init__.py::DefragOptions.max_pages`** — clarified that the cap counts values relocated, not pages examined; flagged the name as a legacy carry-over to explain the surface mismatch. -- **`python/src/transaction.rs` header** — added the `.commit()` / `.rollback()` explicit-drive methods to the initial Semantics block so a first-time reader learns about them before getting to the Design note. - -**Not landed (discussion item):** `LockFailed` classification in the Python error hierarchy. It currently sits under `FatalError` but can only fire at `open()` — before any TransactionManager exists — so it cannot poison. The database file is intact, which argues for `OperationalError`. Left for a future design call; either re-parent it or add a doc comment explaining why it stays put. Not a bug, so out of scope for this sweep. - ---- - -## Invariants to verify — **P3** (one-pass audit) ✅ RESOLVED 2026-04-10 - -Audit pass on the assumptions added during the 2026-04-10 commenting pass. Results inline: - -- `page.rs`: **corrected** — checksum is validated on every disk LOAD (cache miss), not on every cache hit. The old annotation was imprecise. Updated to "validates this checksum on every disk LOAD; cache hits skip revalidation". -- `superblock.rs`: "two slots at fixed page ids 0 and 1, alternating by commit" — **verified** against `create_new` (writes slots 0 and 1) and `commit_inner` (alternates by counter parity). -- `superblock.rs`: "orphaned pages from a crashed commit are cleaned up on next mount" — **corrected**. They are NOT actively cleaned; they remain as dead weight. `open_existing` reseeds `next_page_id` from the authoritative superblock's `total_pages` (I4), so subsequent allocations overwrite the garbage tail. Rewrote the comment to describe this accurately. -- `error.rs`: "reopen after fatal error may recover via alternate superblock" — **corrected** and softened. Only `CorruptSuperblock` on the active slot is recoverable that way; other fatal variants (`ChecksumMismatch`, `IoError`, etc.) indicate damage to the last-committed snapshot itself. Comment now notes the I1 poison-model requires close-and-reopen regardless. -- `data_page.rs`: bytes 8..16 `txn_counter` — **corrected**. The field is allocated in the on-disk layout but NOT written by any live module (init_page zeroes it, compact() faithfully preserves zeros). Re-labeled as "reserved for a future per-page txn_counter". -- `data_page.rs`: byte 1 "reserved / padding" — **verified**; no module reads or writes it. -- `overflow.rs`: bytes 1..16 labeled as "alignment padding to keep the 16-byte common-header shape" — **corrected**. The 16-byte shape is `DATA_PAGE_HEADER_SIZE`, not `COMMON_HEADER_SIZE` (which is 12). Comment now distinguishes the two. -- `handle_table.rs`: depth recovery via leftmost-spine walk is correct **only** because `grow()` installs the old root at child index 0 — **verified** by inspection. -- `page_cache.rs`: read-only opens intentionally take `LOCK_EX` — **verified**. The existing comment in `page_io::open` says so explicitly ("even a reader needs to block concurrent writers"). Not an oversight; intentional for the single-writer shadow-paging model. -- `page_cache.rs`: superblocks bypass the cache entirely — **verified** for BOTH the write path (`commit_inner` → `io_mut().write_page`) AND the read path (`open_existing` → `io_mut().read_page`). `io_mut` doc expanded to document both call sites. - ---- - -## Roadmap items - -From README.md, restated here for visibility. - -### R1. Pack multiple values per data page [roadmap] — **P2** ✅ IMPLEMENTED 2026-04-10 -> Currently each value gets its own page; packing small values together will significantly reduce file size and improve cache efficiency. - -Biggest space/perf win. Best done on top of a working freemap (R2) so page-free-space tracking has a home. - -### R2. Wire the free page map into the allocator [roadmap] — **P2** ✅ IMPLEMENTED 2026-04-10 -> The bitmap is built but allocations currently extend the file; reusing free pages will eliminate file growth after delete-heavy workloads. - -Anchor of the freemap bundle. Depends on resolving I9–I11: reclamation needs to know what to reclaim. Interacts with I3 (rollback file extension) and I4 (`next_page_id` seeding). - -### R3. Selective defragmentation [roadmap] — **P2** ✅ IMPLEMENTED 2026-04-10 -> Consolidate only sparse pages instead of re-inserting every value. - -Fix I17 (defrag stats) while you're rewriting the loop. - -### R4. Configurable superblock count [roadmap] — **P3** (gated on I2) ✅ IMPLEMENTED 2026-04-11 -> Trade commit performance for additional crash durability (3+ superblock copies). - -I2 must be fixed first — the "first commit wipes the only valid superblock" bug affects any N ≥ 2. - -### R5. Python bindings [roadmap] — **P3** (gated on F3) ✅ SHIPPED 2026-04-17 (across the python-binding commit series; see `python/` subcrate and the I21–I25 follow-up batch) -> PyO3-based wrapper exposing the full Chisel API to Python, including context managers for transactions and savepoints. - -Formally gated on F3 (so `&self` reads flow through without wrapping); the binding then landed incrementally as a separate PyO3 subcrate under `python/` with its own `Cargo.toml` / `pyproject.toml` / `maturin develop` workflow, and was further polished by the I21–I25 binding-cleanup batch (explicit `PyTransaction.commit()` / `.rollback()`, `AlreadyFinishedError` on double-drive, `ClosedError` distinct from `PoisonedError`, and RefCell reentrancy docs). - ---- - -## Client feature requests - -### F1. `delete_subtree(handle)` bulk-delete primitive [client] — **P2** ✅ IMPLEMENTED 2026-04-10 as `delete_many` -See I12 — filed in the leaks section because it is the cleanest fix for the client's current orphan-handle problem in `drop_table` / `drop_index_table`. Part of the freemap bundle. - -### F2. Named roots [client] — **P1** ✅ IMPLEMENTED 2026-04-10 -**Motivation (from the client):** -> `rollback` and `rollback_to` reset `meta_root` to 0 on the assumption that handle 0 is always the meta B-tree root. This holds today because `init_meta_root` allocates handle 0 on a fresh database and Chisel preserves handles across updates. But if the meta B-tree ever gets deleted and re-allocated, handle 0 would be orphaned and some other handle would be the root. - -This is a latent correctness bug disguised as a feature request — the client is currently relying on an unwritten invariant. - -Proposed API: -```rust -db.set_root_name("meta", handle)?; -let handle = db.get_root_name("meta")?; -``` - -Named roots would be stored in the superblock (small fixed-size table, or a small dedicated root-names page pointed to from the superblock). The key property is that they survive commit/rollback the same way the handle-table root does, and are not themselves handles that need tracking. - -Design questions (open): -- How many named roots? A fixed small count (e.g., 8) keeps the superblock layout simple. -- Max name length? 16 or 32 bytes is probably plenty. -- Do they take effect at commit time, or immediately? Commit time matches the transactional semantics the client needs. - -Batch with I7 — both touch handle table / superblock. - -### F3. `read()` should take `&self`, not `&mut self` [client] — **P1** ✅ IMPLEMENTED 2026-04-10 -**Motivation (from the client):** -> Chisel's `read()` takes `&mut self` because it goes through the mutable page cache. That forced us to use `RefCell` in `ChiselStorage` for the `&self` read methods on `StorageEngine`. If Chisel internally did its own interior mutability (a `RefCell` or `UnsafeCell` around the page cache), `read()` could take `&self` and we'd eliminate our wrapper layer entirely. - -Pervasive change — reaches from `Chisel::read` down through `TransactionManager`, `PageCache`, and `PageIo`. Cleanest approach is probably a `RefCell` (or `Mutex` if we ever want `Sync`) inside `TransactionManager`, since everything flows through `PageCache`. - -**Design question (open):** `RefCell` (single-threaded, no `Sync`, cheapest) or `Mutex` (leaves the `Sync` door open)? Client is single-threaded today, but committing to single-threaded in the type system is hard to undo. - -Interacts with I5 (truncate dropping dirty pages), I7 (rollback not tracking all dirty pages), and I1's poison flag — all become harder to reason about under interior mutability if reads and writes can now interleave even within a single thread. Do F3 *before* F2/I12 pile more API on top; also unblocks R5. - ---- - -## Deepdive review findings (2026-05-22) - -Source: `docs/reviews/review-20260522-073901.md` (read-only first-contact review of the root `chisel` crate, the `python/` PyO3 binding, and the `bench/` subcrate). - -### Delta from prior perf-review (2026-04-26) - -The 2026-04-26 perf-review used its own internal F1–F6 numbering distinct from ISSUES.md's F-series (client feature requests). The deepdive pass found: - -- **F1 (delete_many is a thin loop) — RESOLVED.** The doc at `src/transaction.rs:1386-1414` now accurately describes the shape and references the deferred I33 batching work. The recommended option (a) of the prior review landed. -- **F4 (`ChiselEngine::internal_counters` masks poison) — RESOLVED.** Fixed at `bench/src/chisel_engine.rs:109-114` with `Ok(Some(self.db.counters()?))`. Poison now propagates as the prior review recommended. -- **F5 (`Identifier` lacks `#[repr(transparent)]`) — RESOLVED.** Attribute applied at `bench/src/engine.rs:29-31`; documented `unsafe` slice transmute at `bench/src/chisel_engine.rs:93-103` eliminates the per-call `Vec` allocation. -- **F2 (`read()` allocates a `Vec`) — UNCHANGED.** `src/transaction.rs:1217` still calls `.to_vec()`. Author classified as deferable; restated below as part of I52 to keep visibility. -- **F3 (per-call `Cell` counter overhead in `PageCache::get`/`get_mut`/`new_page`) — UNCHANGED.** Deliberate trade-off per the prior review's resolution. Not re-flagged. -- **F6 (CI has no supply-chain check) — UNCHANGED.** Promoted to I54 below for proper tracking. - -### Public API and 1.0 readiness - -#### I35. `pub mod` declarations expose engine internals as 1.0 API surface [deepdive 2026-05-22] — **P1** ✅ FIXED 2026-05-22 (PR #11; pub → pub(crate) reshape with selective re-exports of the surface kept public) -**Where:** `src/lib.rs:22-35` - -**Problem:** twelve `pub mod` declarations expose `data_page`, `defrag`, `error`, `freemap`, `handle_table`, `overflow`, `page`, `page_cache`, `page_io`, `stats`, `superblock`, and `transaction`. Every type and method in those modules — `TransactionManager`, `HandleEntry`, `PageCache::set_next_page_id`, `Superblock::serialize`, `OverflowChain`, `DEFAULT_SUPERBLOCK_COUNT`, `MAX_INLINE_VALUE`, and dozens more — becomes part of Chisel's 1.0 stability contract once that release ships. The actual documented public API in `README.md` is 18 methods on `Chisel`; the on-the-wire surface is hundreds of types. - -This is the single highest-leverage decision blocking 1.0. Until it's settled, every other API-stability finding (I36, I37, I39) is provisional — they're only worth fixing on the items that stay public. - -**Direction of fix:** switch internal modules to `pub(crate)` and re-export only the genuinely public types from `lib.rs`: - -```rust -pub use error::{ChiselError, Result}; -pub use stats::{Stats, ChiselCounters}; -pub use defrag::{DefragOptions, DefragStats}; -// Options, DrainInsertion, SpillwayLocation, Chisel stay defined in lib.rs. -``` - -Tests that need access to internals can use `#[cfg(test)] pub use …` re-exports or live inside the modules. The `defrag` module is the trickiest because its `DefragOptions` / `DefragStats` are public; either lift those types into `lib.rs` or keep `pub mod defrag` and rely on `#[non_exhaustive]` to bound the public footprint. - -#### I36. Public types not marked `#[non_exhaustive]` [deepdive 2026-05-22] — **P1** ✅ FIXED 2026-05-22 (PR #12; ChiselError, Options, DrainInsertion, Stats, DefragOptions, DefragStats all gained `#[non_exhaustive]`; fluent builders added on Options + DefragOptions so external callers can still construct them) -**Where:** `src/lib.rs:80-88` (`Options`), `src/lib.rs:99-102` (`DrainInsertion`), `src/lib.rs:107-111` (`SpillwayLocation`), `src/error.rs:16` (`ChiselError`), `src/stats.rs::Stats` - -**Problem:** of all the public types Chisel ships, only `ChiselCounters` carries `#[non_exhaustive]`. Adding a field to `Options`, a variant to `ChiselError` or `DrainInsertion`, or a backing to `SpillwayLocation` is a breaking change today. This bites both struct-literal callers and exhaustive `match` callers — exactly the patterns Rust idiom encourages. - -**Direction of fix:** add `#[non_exhaustive]` to all five types before 1.0. For `Options`, follow up with a `Options::builder()` so callers don't have to construct via `Options { …, …: Default::default() }`. The existing fields stay; only the breakage shape changes — struct-literal construction now requires `..Default::default()`. - -Trade-off: `#[non_exhaustive]` enums force callers to write `_ => …` arms, which is a real ergonomic cost for `match` on `DrainInsertion` (only two variants today). The alternative is to commit to "no new variants, ever" — fine for `DrainInsertion`, defensible but constraining for `ChiselError`. - -#### I37. `SpillwayLocation` is `pub` but used only internally [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (PR #11; folded into the I35 reshape — `pub(crate)`-scoped along with the other engine-internal types) -**Where:** `src/lib.rs:107-111` - -**Problem:** `SpillwayLocation` is constructed only inside `Chisel::open` and `Chisel::open_in_memory_with_options`; it's part of the `PageCache::new` constructor signature, which is only public because `pub mod page_cache`. Users have no reason to construct one; it leaks because `page_cache` does. - -**Direction of fix:** `pub(crate)` once `page_cache` is gated (i.e., as part of I35's reshape). If `page_cache` stays `pub`, leave `SpillwayLocation` `pub` and add `#[non_exhaustive]` (covered by I36). - -#### I38. `Chisel::close() -> Result<()>` is always `Ok(())` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/lib.rs:264-267` - -**Problem:** `close(self)` consumes `self` and always returns `Ok(())`. The `Result` is documented as future-proofing for fsync-on-close failures, but today it's theatre — callers who `?` the result get no observable behaviour. Without `#[must_use]`, callers who *do* care can silently drop the result. - -**Direction of fix:** add `#[must_use = "Chisel::close may surface fsync errors in a future release; ignore explicitly with let _ = if intentional"]` on the method. If you're confident close will stay infallible, change the return type to `()` instead. - -#### I39. `TransactionManager::current_roots() -> (u64, u64, u64)` returns a positional tuple [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/transaction.rs:1693-1699` - -**Problem:** `pub fn current_roots(&self) -> (u64, u64, u64)` returns `(handle_table_page, freemap_page, next_handle)`. A positional 3-tuple of `u64`s is a stringly-typed API in tuple clothing — the caller has to remember which slot is which. The method is exposed only because `transaction` is `pub mod`. - -**Direction of fix:** if the method needs to stay public, introduce `pub struct CurrentRoots { pub handle_table_page: u64, pub freemap_page: u64, pub next_handle: u64 }` with `#[non_exhaustive]`. If it doesn't (probable once I35 lands), drop the `pub` and use it as an internal `pub(crate)` helper. - -#### I40. Runtime setters return `Result<()>` but are infallible [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/page_cache.rs:691, 718, 728` - -**Problem:** `PageCache::set_cache_max_bytes`, `set_spillway_max_bytes`, and `set_drain_insertion` all return `Result<()>` but none can fail in the current implementation. `set_drain_insertion` is literally `self.drain_insertion = policy; Ok(())`. The `Result` shape hedges for future fallibility, but right now the type lies about the API. - -**Direction of fix:** drop the `Result` from `set_drain_insertion` (truly state-free). For the other two, leave the `Result` with a one-line doc comment explaining the hedge — both are plausibly fallible in a future world where shrinking the cache could observe pinned dirty pages. - -#### I41. `ChiselError` has no `source()` impl [deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #14; `source()` returns the inner `io::Error` for the `IoError` arm and `None` for all other variants — preserves the error-chain walker contract for anyhow / eyre / tracing) -**Where:** `src/error.rs:215` (`impl std::error::Error for ChiselError {}`) - -**Problem:** the trait impl is empty. `IoError(io::Error)` wraps an inner cause but exposes it nowhere — `e.source()` returns `None` for every variant. This breaks error-chain walkers (`anyhow::Error::root_cause`, structured-logging adapters, `eyre` reports). The Display message is the only signal an upstream caller can see. - -**Direction of fix:** implement `fn source(&self) -> Option<&(dyn Error + 'static)>` that returns the inner `io::Error` for `ChiselError::IoError(e)` and `None` for the rest. If future variants gain inner causes (e.g., wrapping a deserialization error), extend the match. - -#### I42. Python `to_py_err` discards inner `io::Error` from `IoError(_)` [deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #15; IoError instances now carry `.errno` (raw OS error code) and `.kind` (string form of `std::io::ErrorKind`) as Python attributes — callers branch on those instead of parsing the Display message) -**Where:** `python/src/errors.rs:209-249` - -**Problem:** `to_py_err` formats `ChiselError` via `Display` and then drops the variant. A Python caller cannot programmatically distinguish ENOSPC from EACCES from EIO — they all become `chisel.IoError("I/O error: ")`. The comment at lines 211-213 documents the choice as "the string is the only cross-boundary contract"; defensible but worth re-litigating if any caller wants disk-full-vs-permission-denied handling on the Python side. - -**Direction of fix:** for `IoError`, attach the inner errno (where available) as a Python exception attribute (`errno` or `winerror`-style). PyO3 exception classes can hold arbitrary data; a `PyIoError::new_err((msg, errno))` would surface it. Trade-off: cross-boundary error fidelity vs. holding the Rust error chain in memory across the FFI boundary. - -#### I43. bench `EngineResult` uses `Box` and erases engine class [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `bench/src/engine.rs:43` - -**Problem:** `pub type EngineResult = Result>;` makes engine-specific errors invisible to downstream `match`. The bench crate is `publish = false` and internal-use-only, so this isn't a true public-API leak, but the runner / diff binary / scenarios already do `?`-propagation through `EngineResult` and can't tell `ChiselError::Poisoned` from `redb::Error::Corrupted` without `downcast`. - -**Direction of fix:** introduce a thin enum: - -```rust -pub enum EngineError { - Chisel(ChiselError), - Redb(redb::Error), - Sqlite(rusqlite::Error), - Other(Box), -} -``` - -with `#[from]` impls so `?` keeps working. Keeps the existing call-site ergonomics; adds introspection where needed. - -### Code quality - -#### I44. `libc::flock` `unsafe { … }` block missing `// SAFETY:` comment [deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #14; multi-line SAFETY block documents the fd validity, flag composition, and signal-safety reasoning) -**Where:** `src/page_io.rs:133-141` - -**Problem:** the only `unsafe` block in the core engine — the syscall the entire single-writer contract rests on — has no `// SAFETY:` comment. The Commenting standards section of `ARCHITECTURE.md` calls this out as a convention violation; this is the one place in the engine that violates it. - -The invariants the call upholds: (1) `fd` is valid for the duration of the call because we hold `&File`'s borrow; (2) the syscall returns an `errno`-style int that we check; (3) no resources are leaked because flock release is tied to fd close (which `Drop` handles). - -**Direction of fix:** - -```rust -fn try_lock(file: &File) -> Result<()> { - use std::os::unix::io::AsRawFd; - let fd = file.as_raw_fd(); - // SAFETY: - // * `fd` is valid for the duration of this call: we hold a borrow of - // `&File`, so the descriptor cannot be closed concurrently. - // * `LOCK_EX | LOCK_NB` is a fixed bitflag combination that flock(2) - // accepts on every supported platform (Linux, macOS). - // * The call returns 0 on success, -1 on failure with errno set; we - // do not read errno (`LockFailed` is sufficient diagnostic for the - // "someone else holds the lock" case, the only failure mode in - // practice for a path we can open). - // * No resources are leaked: the lock is released when the underlying - // fd is closed, which happens when `PageIo`'s `Drop` runs. - let rc = unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) }; - if rc != 0 { - return Err(ChiselError::LockFailed); - } - Ok(()) -} -``` - -#### I45. `unreachable!` in `delete_inner` should be `CorruptPage` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/transaction.rs:1375-1379` - -**Problem:** `delete_inner` ends with `unreachable!("handle_table::delete returns None for Deleted entries; None was already escalated to InvalidHandle by ok_or above")`. The "unreachable" depends on `HandleTable::delete`'s cross-module behaviour. A future refactor that changes that contract turns this into a library-reachable panic instead of a typed error. - -**Direction of fix:** convert to `Err(ChiselError::CorruptPage { page_id: entry.page_id })` with a comment noting that reaching this arm would mean the handle table returned a Deleted entry that ok_or didn't catch — i.e., the in-memory state contradicts itself, which is genuinely a corruption signal worth surfacing typed. - -#### I46. `DataPage::insert(...).expect("value fits in empty page")` needs an `// INVARIANT:` comment [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/transaction.rs:1846` - -**Problem:** the `expect` is reachable if `DataPage::insert` ever returns `None` for any reason besides "no room" (e.g., a future size-overflow check on a misuse). The invariant is currently sound — the data page was just allocated and initialized, so insert can't fail for size reasons against a value that was already length-checked against `MAX_INLINE_VALUE` — but it's not asserted at a type level. - -**Direction of fix:** add a comment naming the data-page contract: - -```rust -// INVARIANT: insert can only return None for "no room"; the page was just -// init'd via DataPage::init_page (empty), and the value was length-checked -// against MAX_INLINE_VALUE upstream. If DataPage::insert ever grows other -// failure modes, this expect needs to translate them to typed errors. -let slot = DataPage::insert(buf, value).expect("value fits in empty page"); -``` - -#### I47. `file_size_bytes` multiplication lacks overflow check [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/lib.rs:411` - -**Problem:** `page_count * page::PAGE_SIZE as u64` could overflow at `u64::MAX / 8192 ≈ 2.25 × 10^15` pages (18 EiB). Unreachable for any real database, but unannotated. - -**Direction of fix:** `page_count.saturating_mul(page::PAGE_SIZE as u64)` is one character of armor. - -#### I48. Five invariant-backed `.unwrap()` sites in `page_cache.rs` need `// INVARIANT:` annotations [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/page_cache.rs:188, 211, 353, 384, 914` - -**Problem:** five `.unwrap()` sites on hashmap `get` / `Option` access immediately after the cache was populated or the spillway was just constructed. Each is invariant-backed but unannotated; in aggregate they're a "trust the local code" pattern that a maintenance read can't verify quickly. - -**Direction of fix:** at minimum, annotate each with a one-line `// INVARIANT:` comment naming what guarantees the `Some`. Example for `:188`: - -```rust -// INVARIANT: entry was just inserted by load_page on the miss branch, -// or contains_key returned true on the hit branch. -Ok(&self.entries.get(&page_id).unwrap().buf) -``` - -A stronger fix is to refactor `get` / `get_mut` to return the borrow from inside the `load_page` branch, but the existing shape predates the spillway and the change has knock-on borrow-checker implications. - -#### I49. `expect("LRU referenced page id not in entries")` should be `CorruptPage` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/page_cache.rs:865` - -**Problem:** reachable if the LRU index and entries map ever desync. Currently kept in sync by `discard`/`truncate`/`flush`, but a future refactor that touches one without the other turns this into a library-reachable panic. - -**Direction of fix:** translate to `Err(ChiselError::CorruptPage { page_id: victim_id })` and document that reaching this branch indicates the cache's two-data-structure invariant broke, which is genuinely a corruption signal worth surfacing typed. - -#### I50. Hex literal `0x02` used instead of `FLAG_INTERIOR` constant in `open_existing` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/transaction.rs:417, 423` - -**Problem:** `if root_buf[1] == 0x02 { ... }` reaches for a raw hex literal rather than `handle_table::FLAG_INTERIOR`. The constant exists; using the literal defeats the single-source-of-truth promise for the on-disk format and makes a grep for "interior" miss this site. - -**Direction of fix:** import `handle_table::FLAG_INTERIOR` (exposing it if currently private — it's already implicitly public via the on-disk format) and compare against it. Same fix at both line 417 and the implicit comparison logic at 423. - -### Performance - -#### I51. `read_page` calls `page_count()` (one extra `lseek`) on every call [deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #16; `PageIo` carries a `cached_page_count: Cell` HWM seeded at open and updated on every extending write — eliminates one `lseek` per cache miss; 5 regression tests in src/page_io.rs verify the cache stays coherent across truncate / extend / set_page_count) -**Where:** `src/page_io.rs:166-167` - -**Problem:** `read_page` calls `self.page_count()?` every call, which on the file backing does `file.seek(SeekFrom::End(0))` — one extra syscall per read. The doc comment at lines 156-163 documents the cost and the rationale (no cache invalidation complexity), but underweights it ("absorbed by `PageCache` on cache hits") — the cache miss IS the cost site by definition, and high-miss-rate workloads pay this on every page load. - -**Direction of fix:** cache a high-water-mark on `PageIo`, invalidated by: -- `write_page` past EOF (extend the HWM) -- `set_page_count` (set the HWM exactly) - -`page_count()` returns the cached HWM. Initial seed at open via the existing `seek(End(0))`. Two write paths to update; zero seeks on the read path. Saves one syscall per cache miss. - -#### I52. `flush()` allocates a transient `Vec` per commit [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/page_cache.rs:340-358` - -**Problem:** `flush()` collects dirty IDs into `Vec` for every flush. Sized to `dirty_count`, so for a 10K-page transaction that's 80 KB transient allocation per commit, repeated on every commit. The collect-first idiom is a borrow-checker dodge (the loop needs `&mut self.entries` while iterating). - -**Direction of fix:** keep a scratch `Vec` on `self` and reuse it across flushes (clear before populating). Adds 24 bytes to `PageCache` (the Vec metadata) and eliminates the per-commit allocation. - -Related: `read()` similarly allocates a `Vec` on every call (`src/transaction.rs:1217`, perf-review F2 unchanged). The fixes are different shapes — a `read_borrow(&self, handle) -> Result>` sibling API would close that one for Rust callers (PyO3 callers cannot benefit because `PyBytes` wants owned bytes). - -#### I53. bench `file_size_bytes` triggers an O(live handles) walk via `stats()` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (added `Chisel::file_size_bytes() -> Result` — O(1) single-field accessor that skips the handle-table walk; bench `chisel_engine.rs::file_size_bytes` now calls it instead of materializing the full Stats struct) -**Where:** `bench/src/chisel_engine.rs:106` - -**Problem:** `self.db.stats()?.file_size_bytes` calls `Chisel::stats()`, which walks the entire handle table (O(live handles)) just to populate `handle_count`. Used per measurement cell in the bench harness — for 100K-handle scenarios this is ~milliseconds per call, dragged into every reporting step. - -**Direction of fix:** add a dedicated `Chisel::file_size_bytes() -> Result` that reads `page_count * PAGE_SIZE` directly (the existing math in `Chisel::stats`) without the handle walk. The bench `file_size_bytes` impl calls the new method; the existing `stats()` keeps its current shape because callers want all three fields together. - -### CI, packaging, and publication - -#### I54. CI runs no supply-chain check [perf-review 2026-04-26 / deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #13; `audit` job via `rustsec/audit-check@v1.4.1` runs on every push/PR; the hotfix history at PRs #17 + #26 cleared two transient ignores — paste 1.0.15 unmaintained and pyo3 0.22.x PyString overflow — both via real fixes, not permanent ignores) -**Where:** `.github/workflows/ci.yml` - -**Problem:** three jobs — `test`, `clippy`, `fmt` — all running their respective cargo subcommands. No `cargo audit`, no `cargo deny`, no MSRV pinning. A vulnerable transitive dep would land silently. - -(Promoted from perf-review F6 which was deferred at the time.) - -**Direction of fix:** add an `audit` job to `.github/workflows/ci.yml`: - -```yaml -audit: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: rustsec/audit-check@v1.4.1 - with: - token: ${{ secrets.GITHUB_TOKEN }} -``` - -Costs one CI minute per build. `cargo deny` is the next level up (license + advisory + source policy) and warrants a `deny.toml` policy file. - -#### I55. No MSRV pinned in `Cargo.toml` or CI [deepdive 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (PR #13; `rust-version = "1.82"` in all three subcrates' Cargo.toml; CI `msrv` job uses `dtolnay/rust-toolchain@1.82` and builds `-p chisel` only — scoping per I61 since bench's deps adopt edition2024 faster than we want to float the floor) -**Where:** `Cargo.toml`, `bench/Cargo.toml`, `python/Cargo.toml`, `.github/workflows/ci.yml` - -**Problem:** `rust-version` is absent from every `Cargo.toml`. CI uses `dtolnay/rust-toolchain@stable`, so an unannounced 1.x MSRV bump can land silently. README says "Rust stable, edition 2021"; that's not a pinned MSRV. The codebase uses `let-else` (1.65+), `is_none_or` (1.82+), `is_some_and` (1.70+); actual floor is currently ≥ 1.82. - -**Direction of fix:** pin `rust-version = "1.82"` (or whatever the current actual floor is — verify via `cargo msrv` if available) in `Cargo.toml`. Add a `msrv` job to CI that uses `dtolnay/rust-toolchain@1.82` and runs `cargo build`. If the project doesn't commit to MSRV stability pre-1.0, document that decision in the README. - -#### I56. `Cargo.toml` lacks crates.io publication metadata [deepdive 2026-05-22] — **P1** ✅ FIXED 2026-05-22 (PR #10; root Cargo.toml has license, repository, readme, keywords, categories, description, authors; python and bench subcrates also have publish=false and matching metadata) -**Where:** root `Cargo.toml` - -**Problem:** missing `license`, `repository`, `readme`, `keywords`, `categories`. All required or strongly recommended for crates.io publication. `cargo publish` will refuse without `license` (or `license-file`). - -**Direction of fix:** - -```toml -[package] -name = "chisel" -version = "0.1.0" -edition = "2021" -rust-version = "1.82" # see I55 -description = "Transactional slot-based storage engine with shadow paging" -license = "MIT OR Apache-2.0" # see I57 -repository = "https://github.com/pgexperts/chisel" -readme = "README.md" -keywords = ["database", "storage", "embedded", "transactional", "shadow-paging"] -categories = ["database", "data-structures"] -``` - -#### I57. `License: TBD` blocks any third-party use [deepdive 2026-05-22] — **P1** ✅ FIXED 2026-05-22 (PR #10; MIT license; LICENSE file at repo root, README badge updated, license = "MIT" in Cargo.toml. User explicitly chose MIT-only over the conventional MIT-OR-Apache-2.0 dual) -**Where:** `README.md:326-327` - -**Problem:** with no license, the code is "all rights reserved" by default — no one can legally use or distribute it. For a pre-1.0 project that's worth flagging visibly on the README. - -**Direction of fix:** pick a license now. The prevailing Rust convention is `MIT OR Apache-2.0` dual. Drop `LICENSE-MIT` and `LICENSE-APACHE` files at the repo root, update README from "TBD" to the chosen license, and add `license = "MIT OR Apache-2.0"` to `Cargo.toml` (covered by I56). - -#### I58. `bench/` is not in `ci.yml` [deepdive 2026-05-22, formalizing spillway-rollout lesson #1] — **P2** ✅ FIXED 2026-05-22 (PR #13 added a dedicated `bench-tests` job; superseded by I61's workspace migration in PR #23 — `cargo test` from root now covers bench via `default-members = [".", "bench"]`, so the standalone bench-tests job was removed in the same PR) -**Where:** `.github/workflows/ci.yml`, `bench/Cargo.toml` - -**Problem:** the bench subcrate is a sibling — not a workspace member — so `cargo test` from the repo root doesn't run `bench/`'s tests, and `ci.yml` doesn't either. The spillway-rollout retrospective in `ARCHITECTURE.md` (Implementation history → Lessons learned, lesson #1) flagged this as a pattern that bit a real PR. The mid-PR review caught the missed bench test failures, but there's no CI-side safety net. - -**Direction of fix:** add a `bench-tests` job to `.github/workflows/ci.yml`: - -```yaml -bench-tests: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: dtolnay/rust-toolchain@stable - - uses: Swatinem/rust-cache@v2 - with: - workspaces: bench - - name: Run bench subcrate tests - working-directory: bench - run: cargo test --verbose -``` - -~2 minutes added; closes the real coverage hole. Distinct from `bench.yml` (which runs the scenario tier on PRs for regression-comment purposes and does not run `cargo test`). - -#### I59. `wheels.yml` has no early `cargo test` gate [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `.github/workflows/wheels.yml` - -**Problem:** the wheels workflow builds and tests wheels at tag time. The CI matrix runs Rust tests on push/PR, and wheels.yml runs `pytest` after building wheels, so underlying Rust correctness is covered transitively. But a wheels build on a tag whose underlying commit is broken still runs pytest against broken bindings and fails there — slower feedback than a pre-wheel `cargo test`. - -**Direction of fix:** either add an early `cargo test` step in wheels.yml, or make the wheels job `needs:` the test job (cross-workflow `needs` is supported via a reusable workflow or by re-running tests inline). Less urgent than I54/I55/I56 because wheels.yml is tag-triggered and the underlying problem only manifests if someone tags a broken commit. - -#### I60. Orphaned `bench-disk-cleanup.yml` and `bench-os-update.yml` workflows [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (option b — schedule disabled, dispatch retained) -**Where:** `.github/workflows/bench-disk-cleanup.yml`, `.github/workflows/bench-os-update.yml` - -**Problem:** both workflows are queued waiting for a self-hosted runner that hasn't been provisioned. While they're queueing/expiring, GitHub may surface them as "stuck workflows" in the Actions UI. - -**Direction of fix:** either: -- (a) commit to provisioning the dedicated runner per the "Dedicated bench machine foundation" spec; or -- (b) flip both workflows to `workflow_dispatch:` only with a `# DISABLED until self-hosted runner provisioned` header, so they don't accumulate failed runs. - -Choice (a) is the planned path per the spec; choice (b) is the cleanup if the plan moves out by months. - -#### I61. No workspace manifest [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (default-members = [".", "bench"]; python excluded due to maturin linker requirements) -**Where:** repo root (no `Cargo.toml` `[workspace]`) - -**Problem:** `python/` and `bench/` are sibling subcrates with `chisel = { path = ".." }` path-deps, not workspace members. Each rebuilds `chisel` separately because their `Cargo.lock` files are independent. The README's opening "Rust workspace with three crates" sentence is wrong (see I62). A real workspace would share `target/` and `Cargo.lock`, giving `cargo test --workspace` and `cargo clippy --workspace` coverage of all three at once. - -**Direction of fix:** add a root `[workspace]` declaration: - -```toml -[workspace] -members = [".", "python", "bench"] -resolver = "2" -``` - -Trade-off: workspace members share an `edition` / `rust-version` / unified feature resolution, which can be restrictive for the PyO3 binding (it has different abi3 considerations than the root). Resolver = "2" addresses most of the friction. Probably worth doing; the current setup costs the project a clean way to test the whole tree (and forces I58 as a separate job rather than `cargo test --workspace` covering it for free). - -### Doc fixes - -#### I62. `README.md:71` claims "Rust workspace with three crates" but the repo is not a workspace [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `README.md:71` ("Building from source" intro) - -**Problem:** the README opens "Rust workspace with three crates: the root `chisel` engine, the `python/` PyO3 binding, and the `bench/` benchmark suite." There is no workspace `Cargo.toml`. The same paragraph later acknowledges the truth ("running `cargo test` from the repo root does **not** run the bench subcrate's tests, since `bench/` is a sibling crate, not a workspace member") — those two sentences contradict each other. - -**Direction of fix:** either change the opening sentence to "three sibling crates" (the truth today, and aligns with the existing CLAUDE.md→ARCHITECTURE.md migration), or do I61 first and update the README to reflect the new workspace structure. - -#### I63. `Chisel::commit` docstring says "two fsyncs"; protocol does three [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/lib.rs:278-280` - -**Problem:** `Chisel::commit`'s docstring reads "Performs two fsyncs (dirty data pages, then the alternate superblock)". `ARCHITECTURE.md`'s commit-protocol section (and the I28 fix) document three: pre-drain flush + main-pages flush + superblock. The `no_spill_workload_preserves_two_fsync_commit` test (despite its name) pins to `== 3` per spillway-rollout lesson #3. - -**Direction of fix:** update the docstring to "Performs three fsyncs (pre-drain flush, main pages flush, then the alternate superblock)". Also consider renaming the test to drop the "two_fsync" misnomer; the test's body already documents the three. - -#### I64. `python/src/db.rs:155` uses plain `*` where Rust side uses `saturating_mul` [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `python/src/db.rs:155` - -**Problem:** `let resolved_spillway_max_bytes = spillway_max_bytes.unwrap_or(1024 * cache_max_bytes);` uses plain `*`. `src/lib.rs:118` (`Options::default`) uses `cache_max_bytes.saturating_mul(1024)`. For the default `cache_max_bytes = 8_388_608`, the result fits comfortably in `u64`. A user passing `cache_max_bytes = 1 << 54` (16 PiB) from Python would silently overflow to a small spillway cap rather than saturate to `u64::MAX`. - -**Direction of fix:** mirror the Rust side: - -```rust -let resolved_spillway_max_bytes = spillway_max_bytes.unwrap_or_else(|| cache_max_bytes.saturating_mul(1024)); -``` - -#### I65. `src/spillway.rs` carries stale `#[allow(dead_code)]` on every exported item [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/spillway.rs:39-74` (and similar repeated attributes throughout the file) - -**Problem:** every `pub` item — `SLOT_HEADER_SIZE`, `SLOT_SIZE`, `Backing`, `Spillway`, every `impl` method, `slot_checksum`, `write_slot`, `read_slot` — carries `#[allow(dead_code)]` with the comment "Suppressed until spillway is wired into PageCache (Tasks 7-8)". Tasks 7-8 landed (see `page_cache.rs:824` and surrounding); the spillway IS wired in. The attributes now suppress nothing real, and if any of these items genuinely becomes dead in a future refactor, the attribute will hide that. - -**Direction of fix:** remove every `#[allow(dead_code)]` in `src/spillway.rs` along with the explanatory comments that go with them. A `cargo build` after the removal will fail on anything that was legitimately dead; that's the signal worth surfacing. - -#### I66. Tests use `std::mem::forget(file)` to bypass `NamedTempFile` cleanup [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/transaction.rs:2211`, `src/page_cache.rs:930, 1000`, possibly others - -**Problem:** `NamedTempFile` cleans up its path via `Drop`; tests that need the file to outlive the `NamedTempFile` value (because they re-open the path) leak the temp file deliberately with `std::mem::forget(file)`. This is fragile: the leaked path stays in `/tmp` after the test exits. Over many CI runs this can fill disk on a long-lived runner. - -**Direction of fix:** use `tempfile::TempDir` and construct paths inside it. `TempDir`'s `Drop` cleans up the directory and everything in it, including a re-opened sibling. `src/spillway.rs:338-365` (`open_file_truncates_existing_content`) already does this correctly — use it as the pattern. - -### Idiomaticity - -#### I67. Three sites use awkward `!self.entries.get(&id).is_none_or(|e| e.dirty)` pattern [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/page_cache.rs:419, 702, 834` - -**Problem:** the double negative ("not none-or-dirty" = "some and clean") is acknowledged as awkward in the comment at line 820. Used at three sites for the same eviction-victim search. - -**Direction of fix:** replace with `self.entries.get(&id).is_some_and(|e| !e.dirty)` — reads as "some and clean" and matches the intent directly. Fixes all three sites the same way; no semantic change. - -#### I68. `Chisel::Drop` doesn't fsync (correct, but worth a one-line annotation) [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** `src/lib.rs` (no explicit `Drop` impl on `Chisel`) - -**Problem:** if a user forgets to `commit()`, shadow paging guarantees the on-disk state is the last committed state — so dropping without committing is correct, not a data-loss bug. The type-level doc at `src/lib.rs:139-141` documents this. But a reader coming from other ecosystems (Postgres, RocksDB) expects an explicit "close discards uncommitted work" callout at the `Drop` site too. - -**Direction of fix:** add a `// Drop intentionally omitted — shadow paging guarantees the on-disk state is the last committed state regardless of how the value goes out of scope. See type-level doc for the full semantics.` block at the top of `impl Chisel` or just below the struct declaration. No behaviour change; documentation for the next reader. - -#### I69. `flock` is advisory, not mandatory — worth annotating in an ops/recovery doc [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** README + `ARCHITECTURE.md` (cross-cutting) - -**Problem:** the README and ARCHITECTURE.md both mention that Chisel uses `flock` for single-process exclusion, but neither explicitly states that `flock` is advisory — an external tool that doesn't respect advisory locks (some text editors with "lock files", filesystem dump tools, naive sync utilities) can scribble on the file. This is a Linux/macOS POSIX limitation, not a Chisel bug, but it deserves a sentence so users don't trip over it. - -**Direction of fix:** add a sentence to README's "Platform support" section or to ARCHITECTURE.md's "Cross-cutting concepts" — "Chisel's `flock` is *advisory*: cooperating processes (any other Chisel instance) honour it, but a tool that bypasses advisory locking (e.g., `cp` while a transaction is in flight, some sync utilities) can still corrupt the file. The shadow-paging invariants assume an exclusive owner; respect the lock." - -### Test coverage gaps - -#### I70. No `#[should_panic]` tests for `unreachable!` / `expect` invariant sites [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (reframed post-Group F: typed CorruptPage variant contract covered by `corrupt_page_variant_contract` in src/error.rs; integration-test coverage of the I14 path already existed in src/recovery_tests.rs) -**Where:** test coverage for `src/transaction.rs:1375` (`unreachable!`), `src/page_cache.rs:865` (`expect`), `src/transaction.rs:1846` (`expect`) - -**Problem:** the codebase has good regression tests for documented invariants (I1, I3, I7, I18, I27, I28, I29 all have dedicated tests). The `unreachable!` and `expect` sites name invariants but don't have tests that exercise the invariant-violating path. - -**Direction of fix:** lower priority — and if I45 + I49 convert these to typed `CorruptPage` errors, this finding becomes a test for the `CorruptPage` arm instead. Wait for those decisions; revisit afterward. - -#### I71. No property tests (`proptest` / `quickcheck`) for byte-roundtrip code [deepdive 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** test coverage for `Superblock::serialize` / `deserialize`, `DataPage` slot packing, freemap bitmap operations - -**Problem:** the existing targeted tests cover known cases well; property tests would cover the unknown ones. `Superblock::serialize` round-trips, slot-packing fill / compact invariants, and freemap bit operations are all natural fits. - -**Direction of fix:** add `proptest = "1"` to `[dev-dependencies]` and write three property tests: -- `serialize(deserialize(buf)).map(|sb| sb.serialize()) == Some(buf)` for any well-formed superblock -- `DataPage::insert` then `DataPage::read` round-trips for any value ≤ `MAX_INLINE_VALUE` -- `FreeMap::mark_free(id)` then `FreeMap::is_free(id)` round-trips for any `id < CAPACITY` - -Low priority pre-1.0; high value when format evolution starts in earnest. - -#### I72. Replace `paste` dev-dependency with the maintained `pastey` fork [deepdive follow-up 2026-05-22] — **P3** ✅ FIXED 2026-05-22 -**Where:** root `Cargo.toml` (`paste = "1"` in `[dev-dependencies]`); single use site `tests/common/mod.rs:51` (`paste::paste!` inside `dual_backing_test!` macro). - -**Problem:** RUSTSEC-2024-0436 — `paste 1.0.15` is **unmaintained**. The author (dtolnay) archived the GitHub repo on 2024-10-07 and the README says the project is no longer maintained. RustSec classifies this as informational (no vulnerability, no broken semantics), but `rustsec/audit-check` flags it by default and blocks the I54 supply-chain CI job. - -Surfaced when the I54 audit job landed on main and immediately tripped on this dep. Worked around in the same fix-up commit (`ignore: RUSTSEC-2024-0436` in `.github/workflows/ci.yml`); this entry documents the proper fix. - -**Direction of fix:** swap `paste` for `pastey`, a drop-in fork explicitly created to address this advisory. Two edits: - -```toml -# Cargo.toml -[dev-dependencies] -- paste = "1" -+ pastey = "0.1" -``` - -```rust -// tests/common/mod.rs -- paste::paste! { -+ pastey::paste! { -``` - -Verify `cargo test` still produces both `_file` and `_memory` test variants via the `dual_backing_test!` expansion, then drop the `ignore: RUSTSEC-2024-0436` line from `.github/workflows/ci.yml`. - -Low priority because the warning is informational and only affects a dev-dep; high enough to fix in a small PR before the `ignore` list accumulates more entries. - -#### I73. GitHub Actions Node.js 20 deprecation (every job uses `actions/checkout@v4`) [post-P2 CI run 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (checkout v4 → v5 across all four workflows; other actions already Node 22+) -**Where:** every job in `.github/workflows/ci.yml` (test, clippy, fmt, audit, msrv, bench-tests, python matrix) — currently 7 distinct jobs all pinned to `actions/checkout@v4`. The `dtolnay/rust-toolchain`, `Swatinem/rust-cache@v2`, `actions/setup-python@v5`, and `rustsec/audit-check@v1.4.1` actions should also be re-checked for Node 24 readiness. - -**Problem:** GitHub is forcing Node.js 24 as the default on hosted runners on **June 2nd, 2026** (~10 days from today). Node.js 20 will be removed entirely on **September 16th, 2026**. Surfaced as warning annotations on the first green post-P2 audit run on main: - -> `! Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/checkout@v4. … For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/` - -CI keeps passing today, but the warning is on every run and there is a hard cutoff coming. - -**Direction of fix:** bump each `uses:` line to a version whose action manifest declares `node24` once those versions are GA. As of 2026-05-22, the v5 line of `actions/checkout` is the natural target; the other actions cited above need a one-shot audit of their action.yml `runs.using` field. Until then, the temporary opt-in is the `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true` env var on the runner — but that's worse than just bumping the pin once. Single-PR scope: edit `.github/workflows/ci.yml`, push, verify one CI run goes green. - -Low priority because runners auto-upgrade on the cutoff date anyway; medium-low because if `actions/checkout@v5` is GA before then, this is a five-minute PR that closes the warning noise immediately. - -#### I74. Expose `Spillway::logical_bytes` and `max_bytes` via `Chisel::stats` / `ChiselCounters` [I65 follow-up 2026-05-22] — **P3** ✅ FIXED 2026-05-22 (added `spillway_logical_bytes: Option` and `spillway_max_bytes: Option` to Stats; wired through PageCache → TransactionManager → Chisel::stats; Python `chisel.Stats` dataclass updated; integration tests in tests/spillway_integration.rs + python/tests/test_stats_defrag.py exercise the None → Some(>0) → Some(0) lifecycle) -**Where:** `src/spillway.rs` (`logical_bytes`, `max_bytes` — currently `#[cfg(test)]`); `src/stats.rs` (`Stats` / `ChiselCounters`); `src/lib.rs` (`Chisel::stats`). - -**Problem:** when I65 stripped every `#[allow(dead_code)]` from `src/spillway.rs`, `Spillway::logical_bytes()` and `Spillway::max_bytes()` surfaced as legitimately unused — they had no production caller. Gating them as `#[cfg(test)]` keeps the lib build clean and preserves the methods for the test module, but throws away the chance for operators to read spillway capacity utilisation through the public stats API. - -Spillway capacity is exactly the kind of metric operators want for capacity planning: "how full is the spillway right now, and what's the cap?" Knowing `logical_bytes / max_bytes` answers "are we one transaction away from `SpillwayFull`?" - -**Direction of fix:** add two fields to the `Stats` (or `ChiselCounters`) struct: - -```rust -#[non_exhaustive] -pub struct Stats { - // ...existing fields... - /// Spillway logical bytes in flight (None if spillway never opened). - pub spillway_logical_bytes: Option, - /// Spillway max-bytes cap (None if spillway never opened). - pub spillway_max_bytes: Option, -} -``` - -Wire them into `Chisel::stats` by inspecting `PageCache::spillway` (which is already `pub(crate)`). Then remove the `#[cfg(test)]` from the two `Spillway` methods. The `Option` shape is because the spillway is lazily opened on first spill — `None` distinguishes "no spillway yet" from "spillway has zero bytes in flight." - -Low priority because spillway exhaustion currently surfaces as `SpillwayFull` (a typed error) rather than a silent slowdown; operators can hook on that. But adding observability is the difference between "see the wall coming" and "hit the wall." - -#### I75. Bump `pyo3` from 0.22 to 0.24+ to clear RUSTSEC-2025-0020 [I61 follow-up 2026-05-22] — **P2** ✅ FIXED 2026-05-22 (bumped to 0.24; RefCell→Mutex, Cell→AtomicBool for PyO3 0.24's Sync requirement; cargo audit shows 0 vulns / 0 warns; 85/85 Python tests pass) -**Where:** `python/Cargo.toml` (`pyo3 = { version = "0.22", features = ["extension-module", "abi3-py311"] }`); call sites throughout `python/src/`. - -**Problem:** `pyo3 0.22.x` (we're pinned at 0.22.6 in Cargo.lock) ships `PyString::from_object` with a "Risk of buffer overflow" — `from_object` takes `&str` arguments and forwards them directly to the Python C API without checking for terminating NUL bytes, so the Python interpreter can read beyond the end of the `&str` data and potentially leak the contents of the OOB read by raising a Python exception containing a copy of the data including the overflow. - -RustSec advisory: RUSTSEC-2025-0020. Patched in pyo3 0.24.1+. Our binding may or may not call `PyString::from_object` directly; either way, the vulnerable function ships in our `.so` and gets exposed to any code that links against it (including future internal code). - -Surfaced when I61 unified all three subcrates' lockfiles under a workspace `Cargo.lock` — pre-I61, only the root crate's deps (no pyo3) were audited. Worked around in the I61 PR by adding `ignore: RUSTSEC-2025-0020` to the audit job; this entry documents the proper fix. - -**Direction of fix:** bump `pyo3` from 0.22 → 0.24 (or whatever's current as of the fix). The migration is non-trivial because PyO3 has breaking API changes between majors — notable changes in 0.23/0.24 include: -- `Bound` is the standard reference type now (most APIs that returned `&PyAny` now return `Bound`). -- `value_bound` / `extract_bound` style was made the default. -- Some attribute-setting paths changed. - -Plan: -1. Read PyO3 0.23 and 0.24 migration guides. -2. Bump `pyo3` to the target version in `python/Cargo.toml`. -3. Iterate on `python/src/db.rs`, `transaction.rs`, `savepoint.rs`, `errors.rs` to fix breakage. The errno/kind setattr code (added in I42) and the `_chisel.create_exception!`-style code in `errors.rs` are the most likely friction points. -4. Run `cd python && maturin develop --release && pytest` until green. -5. Remove `ignore: RUSTSEC-2025-0020` from `.github/workflows/ci.yml`. -6. Mark I75 as ✅ FIXED here. - -Higher priority than most other P3s because (a) it's a real (not informational) security advisory and (b) the `ignore:` workaround is a temporary measure that should be cleared promptly. - -#### I76. Periodic clean-checkout `cargo clippy` job to catch lint regressions hidden by build cache [I53/I74 follow-up 2026-05-22] — **P3** -**Where:** `.github/workflows/ci.yml` (would add a new scheduled job). - -**Problem:** CI uses `Swatinem/rust-cache@v2` so per-PR clippy runs are fast — but the cache can mask a clippy regression. Concretely: PR #27 introduced `1024u64 * cache_max_bytes as u64` in `tests/spillway_integration.rs`. The `as u64` cast is redundant (`cache_max_bytes` was inferred to u64), but PR #27's CI ran clippy against a warm cache from the previous PR's build and the lint never fired. PR #28 (cleanup, against a fresh branch) caught it because the cache key changed (different branch, different file set) and clippy did a full re-check. - -The general shape: any time `cargo clippy --all-targets` skips a file because the cached build artifact is still valid, lints that depend on context (`-D warnings`, new lints in a clippy upgrade, lints that fire only against fresh expansion) can be silently missed. The cost is a wrong-feedback signal at PR time — the regression lands and is found by a future unrelated PR. - -**Direction of fix:** add a scheduled-trigger CI job that runs clippy without the cache: - -```yaml -clippy-no-cache: - # Periodic clean-checkout clippy. Catches lint regressions hidden by - # Swatinem/rust-cache@v2's incremental cache on PR-triggered runs. - # See I76 in ISSUES.md. - on: - schedule: - - cron: '0 6 * * 1' # Mondays, 06:00 UTC - workflow_dispatch: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v5 - - uses: dtolnay/rust-toolchain@stable - with: - components: clippy - # No Swatinem/rust-cache step — intentional. - - run: cargo clippy --all-targets --workspace --exclude chisel-py -- -D warnings -``` - -Runs once a week; ~5 minutes per run; would have caught the PR #27 leak immediately on the following Monday. Cheap insurance for a real (if rare) failure mode. - -Low priority because (a) the leak is caught eventually by the next unrelated PR's CI, (b) the leak is always a clippy warning (not a runtime bug), (c) the cleanup PR #28 already absorbed the actual fix. - ---- - -## Perf-review findings (2026-06-01) - -Source: `docs/reviews/perf-review-2026-06-01.md` (read-only full hot-path sweep, `chisel-performance` + `rust-performance` skills; six-agent parallel audit over `src/` and `bench/`). No correctness bugs, no Don't-Break-List violations. Each item carries its review-doc ref (PR-x) and classification — **[STATIC FACT]** (provable by reading) or **[HYPOTHESIS]** (mechanism certain, magnitude needs a bench). Severities are as-assessed by the sweep and open to re-triage. Line numbers are as-of the sweep and prefixed `~`; the symbol names are the durable anchor. - -**Sequencing:** the benchmark-harness items (I90–I92) are prerequisites — the grid cannot trustworthily validate an engine change until they land, and I90 (`black_box`) must precede I91 (LTO profile) because LTO widens the dead-code-elimination window. Measure the two engine P1s (I77, I78) only afterward. Do not gate CI pass/fail on the resulting numbers (shared-runner noise; perf is report-only per the project CI policy). - -### Engine — hot path & commit - -#### I77. Default SipHash on `PageCache.entries` and `LruIndex.nodes` [perf-review 2026-06-01] — **P1** (PR-A, [HYPOTHESIS]) ✅ FIXED 2026-06-01 (rustc_hash::FxHashMap on both maps; zero new transitive deps — only the 3rd runtime dependency. Behavior-preserving: 174 engine tests green. Magnitude still unmeasured — run the read-warm bench, with I90 black_box + I91 profile, on dedicated hardware) -**Where:** `src/page_cache.rs` `entries` (init ~:167) + `get`/`get_mut`; `src/lru.rs` `nodes` (~:67) + `push_front`/`unlink` - -**Problem:** both hot maps use `std::collections::HashMap` with the default SipHash-1-3 `RandomState`. A warm `cache.get(id)` costs ≈ six SipHash probes of a `u64` key (`contains_key` + `get` + the `touch_lru` bookkeeping); a depth-2 read calls `get` four times → ≈ 24 SipHash hashings per cache-hit read. SipHash's DoS-resistance is worthless for a single-writer embedded engine keyed on trusted local page IDs. Related (PR-R): `get`/`get_mut` and the `LruIndex` ops also double-probe (`contains_key` then `get`) — folding those into a single probe is the same area. - -**Direction of fix:** swap both maps to a fast `u64` hasher (`foldhash`, `rustc_hash::FxHashMap`, or `ahash`). Validate on the existing `read-warm` micro-grid row (all cache hits, zero I/O — isolates hashing + LRU cost). In-memory only; no on-disk format, fsync, poison, or `&mut self` interaction; LRU order rides the linked-list pointers, not map iteration order. - -#### I78. Per-insert full-page XXH3 re-stamp on the slot-packing path [perf-review 2026-06-01] — **P1** (PR-B, [HYPOTHESIS]) — ⏸️ OPTIMIZATION DEFERRED 2026-06-01 (pending a bulk-insert bench on Phase 0's tooling / dedicated hardware). Investigation CORRECTED the caveat below: the eager stamp is NOT needed for the evict-mid-transaction case — the spillway round-trip verifies its own `slot_checksum`, never the internal page checksum (only main-file cold-load does, `page_cache.rs:872`). The eager stamp's real job is "valid internal checksum before the main-file write — commit flush OR spill-then-drain"; that drain path is precisely what makes the deferral non-trivial. The misleading `transaction.rs` comment is fixed; see memory `project_chisel_i78_restamp_deferred.md`. -**Where:** `src/transaction.rs` `insert_into_data_page` (cursor path ~:1856-1869, fresh-page path ~:1892); cost in `src/page.rs` `compute_checksum` - -**Problem:** `page::stamp_checksum(buf)` runs after **every** `DataPage::insert`, hashing all 8184 page-body bytes regardless of how few changed. A 1000-small-value transaction packing ≈ 39 values/page re-hashes its data pages once per value instead of once per page — `O(values × 8 KB)` where `O(pages × 8 KB)` would do. - -**Direction of fix:** defer stamping — stamp a data page lazily at flush time and on the eviction path (a "needs-stamp" sub-flag, or stamp on cursor retirement + an eviction hook). **Caveat (Don't-Break #8):** a stamped checksum must still be guaranteed before any eviction-to-spillway/disk and before flush; eager stamping exists for exactly the evict-mid-transaction case (documented at `transaction.rs:1839-1842`). Validate on `allocate-1000pertx` / `update-1000pertx` at small value sizes. - -#### I79. `commit_inner` promotes `current_freemap` / `current_live_slots` by clone, not swap [perf-review 2026-06-01] — **P2** (PR-F, [STATIC FACT]) -**Where:** `src/transaction.rs` `commit_inner` step 5 (~:917-920); mirrored per `begin()` (~:717-721) - -**Problem:** every commit deep-copies the full 8 KB `Box<[u8; PAGE_SIZE]>` freemap and rebuilds the `O(live-pages)` `current_live_slots` HashMap, even for a single-row commit (one bit / one page changed). The bitmap is moved ≈ 3× per small commit (this clone + the `persist_freemap` copy at ~:684). - -**Direction of fix:** `std::mem::swap(&mut committed_*, &mut current_*)` at step 5, then reseed `current_*` from `committed_*` at the next `begin()` (which already clones). Same shape as the accepted I52 fix; strictly fewer allocations, so it does not need a bench to justify (one would size it). **Don't-Break:** the swap must occur after the step-4 fsync linearization point (it does). - -**Re-validated 2026-06-22 — RE-SCOPED; no longer the free win it was when filed (this entry predates the multi-page freemap rewrite + the `transaction.rs` extraction, so its line refs are stale).** (1) The 8 KB freemap-bitmap clone is **gone**: the freemap is now a COW tree (`FreemapRecycle`); `commit` promotes the structural-recycle streams with no full-bitmap copy. That half is obsolete. (2) The `current_live_slots` clone survives, now in `SlotPacker::commit` (`src/transaction/packing.rs:207`). But a `mem::swap` there is **not free**: after a swap, `current_live_slots` holds the PRE-transaction state until the next `begin()` reseeds it, and it is read post-commit by `sparse_data_pages` / `data_page_ids_snapshot` (`src/transaction/stats.rs:124,155`) — so the swap needs a proof those readers never run between `commit` and `begin`, plus a regression test. It is also a microsecond `HashMap` clone sitting **below the fsync floor**. Verdict: do NOT treat as a no-bench win; if revisited, measure first and prove the stats-read timing. - -#### I80. `maybe_evict` re-scans the dirty LRU prefix per eviction in the mixed clean/dirty regime [perf-review 2026-06-01] — **P2** (PR-G, [HYPOTHESIS]) -**Where:** `src/page_cache.rs` `maybe_evict` Phase A (~:912-930) - -**Problem:** the `dirty_count == entries.len()` early-out only fires when **every** entry is dirty. In a mixed regime (clean read-through pages interleaved with dirty pinned ones, clean victims toward the MRU end) the `iter_lru_to_mru().find(|id| !dirty)` walks past the whole dirty prefix from the tail on each call — `O(n²)` across a transaction that evicts repeatedly. - -**Direction of fix:** a separate intrusive "clean LRU" sub-list or a free-victim cursor so eviction is `O(1)` amortized. Confirm the regime occurs first (pure-write hits the all-dirty early-out; pure-read finds the first tail item — the quadratic only bites mixed read+write transactions large enough to evict). No invariant exposure. - -#### I81. No cache buffer pool — malloc/free per page churn + 8 KB memset on `new_page` [perf-review 2026-06-01] — **P2** (PR-H, [HYPOTHESIS]) -**Where:** `src/page_cache.rs` page allocation sites (~:326, :446, :736, :853, :873); eviction drops the `CacheEntry` (frees its Box) - -**Problem:** under sustained pressure (working set > cache) the cache does malloc-on-load / free-on-evict in lockstep — a fresh 8 KB allocation per miss and per `new_page`, the just-freed buffer not reused; `new_page` additionally zeroes 8 KB. (Alloc-heavy → worst case for shared-CI bench noise; measure on dedicated hardware, report-only.) - -**Direction of fix:** a small free-list capped at a few× the cache size — push freed `Box`es on eviction, pop on load / `new_page` (zero only when handing out a `new_page`). The **small-pool** idea, explicitly NOT the deferred I34 mmap-backed-storage redesign. - -#### I82. `page_io` / `spillway` use seek+write (two syscalls/page) instead of positioned pread/pwrite [perf-review 2026-06-01] — **P2** (PR-I, [HYPOTHESIS]) -**Where:** `src/page_io.rs` `read_page` (~:209-213) / `write_page` (~:236-240); `src/spillway.rs` `write_slot` (~:280-284) / `read_slot` (~:307-311) - -**Problem:** each page I/O issues `seek(Start(off))` then `read_exact`/`write_all` — two syscalls where `FileExt::{read_exact_at, write_all_at}` (pread/pwrite) does one. `flush()` calls `write_page` once per dirty page per commit; the spillway drain adds more (and splits header/page into two writes). - -**Direction of fix:** switch the `File` arm to positioned I/O; for the spillway, assemble header+page into one `[u8; SLOT_SIZE]` and issue a single positioned write. Stays inside the I/O modules (invariant 6); the single-writer flock makes shared-offset removal safe. For fsync-dominated commits the win may be in the noise — bench with a large-dirty-set commit + syscall count. - -#### I83. No `read_many` — batched reads re-descend the handle table per key [perf-review 2026-06-01] — **P2** (PR-J, [HYPOTHESIS]) -**Where:** absent from `src/lib.rs` / `src/transaction.rs` (only single `read`) - -**Problem:** the read-side analogue of the deferred I33. K reads pay K independent `find_leaf` descents from the root; keys sharing interior/leaf pages (monotonic handles read in ranges) re-fetch and re-hash the same interior pages K times. - -**Direction of fix:** land I77 first (cuts per-descent cost for free). Whether a leaf-grouping `read_many` beats the simple loop is the hypothesis — settle with a clustered-read bench row before building the batched descent (same YAGNI posture that deferred I33). Read-only `&self`; preserve per-handle error semantics. - -#### I84. Freemap `allocate_first` linear-scans the bitmap from byte 0 with no cursor [perf-review 2026-06-01] — **P3** (PR-O, [HYPOTHESIS]) ✅ ADDRESSED 2026-06-22 (multi-page freemap hint cursor) -**Where:** `src/freemap.rs` `allocate_first` (~:135-146); found independently by the commit and write-path agents - -**Problem:** every freemap-backed allocation (and the per-commit freemap-page placement) scans from byte 0. With a dense low region and frees concentrated high, each allocation re-walks the same leading zero bytes — `O(n²)` across a reuse-heavy transaction. Minor (single 8 KB page, whole-byte skip, L1/L2-resident), but pure repeated work. - -**Direction of fix:** an in-memory `next_free_hint` byte cursor (reset on `begin()` clone / lower-id `mark_free`), optionally `u64`-word scanning. In-memory allocator state only — no on-disk format impact, and the I18 lowest-id ordering is about which ids are visible, not scan order. ARCHITECTURE.md:434 documents the from-0 scan as behavior; this is the perf entry for it. - -**Re-validated 2026-06-22 — ✅ ADDRESSED by the multi-page freemap.** The single-page `src/freemap.rs::allocate_first` this targeted no longer exists; the COW tree's `FreeMapTree::allocate_first` (`src/freemap_tree.rs:470`) now takes a `hint: &mut u64`, does `scan_from(cache, *hint)`, and advances `*hint = found` on each allocation — exactly the cursor this asked for. `freemap_hint` is threaded through `cow_alloc` and persisted across the transaction (lowered on `mark_free`). No work needed; closing. - -#### I85. `update_inner` always relocates; in-code cost-model text inaccurate under R1 [perf-review 2026-06-01] — **P3** (PR-K, [STATIC FACT]; doc + narrow opt) -**Where:** `src/transaction.rs` `update_inner` (~:1251-1321); unused in-place `src/data_page.rs` `DataPage::update` - -**Problem:** a same-size update unconditionally retires the old slot and packs the new value into the cursor page (delete+insert leaving a tombstone), so the cost-model line "update(same size) = 1 data page COW" does not describe reality — there is no in-place data COW. Correct under R1 (a committed packed page can't be rewritten in place without rewriting every co-resident handle); the finding is a doc inaccuracy plus a narrow optimization. - -**Direction of fix:** correct the cost-model text. Optional narrow win: when the old value lives on a page dirtied **in the current transaction** and the new value fits the old slot, overwrite in place via `DataPage::update`. **Don't-Break:** any in-place path MUST be restricted to current-transaction dirty pages — overwriting a committed data page violates shadow paging. - -### Spillway - -#### I86. Spillway re-validates XXH3 on every unspill — redundant with the main-file checksum on the drain path [perf-review 2026-06-01] — **P3** (PR-P, [STATIC FACT]; hazard-flagged) -**Where:** `src/spillway.rs` `rehydrate` / `slot_checksum` (~:235-255); callers `src/page_cache.rs` drain (~:438) and resident read-back (~:848) - -**Problem:** every `rehydrate` recomputes an XXH3 over the slot. On the **drain** path the bytes are immediately written to the main file and re-validated structurally on their next cold load, so that verify is arguably redundant (spilled → checksummed → drain-verified → cold-load-verified). **But** the resident read-back path re-inserts the rehydrated page as dirty without a fresh main-file write, so dropping its checksum would let an undetected sidecar bit-flip enter the cache and later flush with a freshly-stamped valid-looking checksum — silent corruption. - -**Direction of fix:** at most skip the verify on the drain-and-immediately-rewrite path; **keep** it on the resident read-back path (Don't-Break #8). The spill-side checksum (torn-sidecar-write detection) stays. Marginal — bench a spill-heavy workload before touching. - -#### I87. Spillway micro-allocations: rehydrate `Box::new(buf)` copy + per-batch `drain_batch` Vec [perf-review 2026-06-01] — **P3** (PR-Q, [STATIC FACT]) -**Where:** `src/page_cache.rs` rehydrate sites (~:436-448, :848-856, :866-873); `src/spillway.rs` `drain_batch` (~:204-210) - -**Problem:** (a) `read_slot` / `rehydrate` return `[u8; PAGE_SIZE]` by value, then the cache does `Box::new(buf)` — a stack→heap 8 KB copy on top of the file read (the spill *write* side is clean). (b) `drain_batch` allocates a fresh `Vec` per batch — the exact shape I52 eliminated elsewhere with a reused scratch vec, not applied here. - -**Direction of fix:** (a) have `rehydrate` / `read_slot` take a `&mut Box<[u8; PAGE_SIZE]>` out-param and `read_exact` directly into the heap buffer (pairs with I81's pool). (b) reuse a `drain_scratch: Vec` mirroring `dirty_scratch`. Sidecar layout is ephemeral — free to change. - -#### I88. `Spillway::logical_bytes` over-reports after `forget` (high-water, not live) [perf-review 2026-06-01] — **P3** (PR-U, [STATIC FACT]; stats-accuracy, non-perf) -**Where:** `src/spillway.rs` `logical_bytes` (~:128-130) - -**Problem:** `logical_bytes` returns `next_slot_index * PAGE_SIZE`, but `forget` / `forget_above` remove from `slots` without decrementing `next_slot_index` (slots aren't reused until `truncate`). During a drain (which calls `forget` per page) the I74 `spillway_logical_bytes` stat reports the high-water size, not the live resident size. Stats-accuracy quirk surfaced via the I74 surface; not a perf or durability issue. - -**Direction of fix:** track a live-byte counter decremented in `forget` / `forget_above`, or document `logical_bytes` as a high-water mark. Decide alongside any I74 stats follow-up. - -### Error path - -#### I89. Savepoint error variants allocate a `String` at construction [perf-review 2026-06-01] — **P3** (PR-S, [STATIC FACT]) -**Where:** `src/error.rs` `SavepointNotFound(String)` / `DuplicateSavepoint(String)` (~:30-31); constructed at `src/transaction.rs:1005,1054,1093` via `name.to_string()` - -**Problem:** these eagerly heap-allocate at error construction (error branch only). Cold savepoint-control path, so no perf impact — recorded for completeness and as the contrast to `InvalidHandle(u64)`, which correctly carries a bare `u64` with lazy `Display` formatting on the hot lookup-miss path. - -**Direction of fix:** none required for perf. If the error enum is ever reworked, a borrowed / `Box` payload would remove the allocation, but it is not worth a standalone change. - -### Bench harness & build (measurement-integrity cluster — prerequisites) - -#### I90. Bench harness has no `black_box` — read/alloc results are dead-code-eligible [perf-review 2026-06-01] — **P1** (PR-C, [STATIC FACT]) ✅ FIXED 2026-06-01 (std::hint::black_box on both ends of apply_op's Read arm; one funnel covers the scenario tier + warm/cold micro-grid rows + drive_workload_with_tx_granularity) -**Where:** `bench/src/runner.rs` `apply_op` (~:216-250); `bench/benches/micro_grid.rs` read loops (~:111-115, :139-144); `bench/benches/scenarios.rs` timed loop. `grep black_box bench/` → none. - -**Problem:** read results (`Vec`) are dropped, never observed; the criterion closures return `()`, so criterion's built-in `black_box` on the closure return value doesn't protect per-op results. The optimizer may elide the unused-result work — and the risk **grows when the I91 LTO profile lands**. This is the single most important bench-hygiene gap and must be fixed first. - -**Direction of fix:** feed the resolved id through `black_box` on the way in and `black_box` the returned bytes in `apply_op` and the read loops. Bench-only; consumer-neutral. - -#### I91. No `[profile.release]` — Chisel/redb benched under weaker codegen than bundled SQLite [perf-review 2026-06-01] — **P1** (PR-D, [STATIC FACT] asymmetry / [HYPOTHESIS] magnitude) ✅ FIXED 2026-06-01 (workspace-root [profile.release] lto="fat" + codegen-units=1, plus [profile.release-with-debug] for profiling; consumer-neutral — root profile affects only in-repo builds. Tracked baselines need a one-time re-record) -**Where:** workspace `Cargo.toml` (no `[profile.*]`); confirmed absent in `bench/` and `python/`; no `.cargo/config.toml`. - -**Problem:** with no profile override, `cargo bench` builds `chisel` + `redb` at `opt-level=3` but `lto=false`, `codegen-units=16`, `panic=unwind`, while `rusqlite` `bundled` compiles SQLite C at its own `-O2`. The headline cross-engine table handicaps the two Rust engines (no cross-crate inlining across the `chisel`→bench boundary) — a "SQLite is faster here" conclusion is partly a build-config artifact. (The harness already equalizes the macOS `F_FULLFSYNC` path in `sqlite_engine.rs:50-62`; profile parity is the missing build-side half.) - -**Direction of fix:** add a tuned profile at the **workspace root** (`lto = "thin"`/`"fat"`, `codegen-units = 1`) and re-record baselines once (uniform shift, not a regression). Land **after** I90. **Consumer-neutral:** a workspace-root profile affects only in-repo builds (benches/tests/examples); Cargo ignores a dependency's profile, so downstream `chisel = "0.1"` consumers are unaffected. Do NOT tune via the published lib crate. - -#### I92. In-memory Chisel backend exists but is never benched [perf-review 2026-06-01] — **P1** (PR-E, [STATIC FACT]) ✅ FIXED 2026-06-01 (ChiselMemory EngineMode → chisel-mem column in the scenario tier; chisel-mem − chisel-strict wall-clock delta = the fsync tax. Scenario-tier only — micro-grid's fs::copy fixtures can't seed a Vec-backed engine. Regression test pins wiring + the fsync_calls protocol-counter subtlety) -**Where:** `bench/src/runner.rs` `EngineMode` (~:30-98, only `ChiselStrict` → `open_file`); `micro_grid.rs:168`, `scenarios.rs:27-31` all file-backed. `ChiselEngine::open_in_memory` only used in `bench/tests/`. - -**Problem:** without an in-memory row the harness can't separate Chisel's CPU cost (slot packing, XXH3, handle-table walk, COW) from its full durability cost (fsync, `F_FULLFSYNC`, pwrite) — the decomposition that makes a "commit is slow" finding actionable. A pure-CPU regression can be masked by fsync-dominated wall time. (chisel-performance Lever 5.) - -**Direction of fix:** add a `ChiselMemory` `EngineMode` via `open_in_memory(cache_size)`, include it in `EngineMode::ALL` and the scenarios mode list (cleanest through `run_scenario_cell`, which pre-populates in-process). Track both backends so the ratio is visible. Bench-only. - -#### I93. Bench binaries could pin `mimalloc` for realistic best-case + cross-engine allocator parity [perf-review 2026-06-01] — **P2** (PR-N, [STATIC FACT] absent / [HYPOTHESIS] win) ✅ FIXED 2026-06-01 (#[global_allocator] = MiMalloc in the micro_grid + scenarios bench binaries; mimalloc 0.1.52 dev-dep, publish=false — never the library. Builds on MSRV 1.82, fat-LTO release links + launches clean. Re-record baselines) -**Where:** no `#[global_allocator]` anywhere; natural home `bench/benches/{scenarios,micro_grid}.rs` - -**Problem:** Chisel (`Box<[u8;8192]>` per page, `Vec` per read) and redb (`value().to_vec()`) are alloc-heavy; SQLite's C core does far less Rust-side heap traffic. All three run on the platform default allocator, so the tracked numbers reflect "Chisel on system malloc," not its best, and the allocator tax falls unevenly. - -**Direction of fix:** set `#[global_allocator] = MiMalloc` in the two bench binaries (the `publish=false` crate) and re-record baselines. **Consumer-neutral and load-bearing:** a `#[global_allocator]` is process-global and a library must never force one — the bench crate is the only correct home; do NOT add it to `chisel` or `chisel-py`. - -#### I94. Scenario timed region includes begin/commit framing, per-op `Instant::now()`, and payload `Vec` alloc [perf-review 2026-06-01] — **P2** (PR-M, [STATIC FACT] region / [HYPOTHESIS] distortion) -**Where:** `bench/src/runner.rs` scenario timed loop (~:548-563) + `apply_op` payload alloc (~:230-240) - -**Problem:** the CI-tracked scenario tier brackets the whole loop, takes a per-op `Instant::now()` pair inside it, and builds `vec![0u8; size]` inside the timed op. Per-op clock reads (200K on a 100K-op run) and payload alloc/zeroing fold into throughput, taxing fast (read) ops more than fsync-bound (write) ops and compressing cross-engine deltas. (The micro-grid correctly hoists setup into `iter_batched`; the hand-rolled scenario timer does not.) - -**Direction of fix:** pre-build payloads once per size outside the loop; compute throughput from a single outer timer over a loop with no inner `Instant::now()` (run the latency-sampled pass separately); document that scenario throughput is per-transaction. Bench-only. - -#### I95. Noise gate N=1 false-green; gate/diff conflate "no signal" with PASS/FAIL [perf-review 2026-06-01] — **P2** (PR-L, [STATIC FACT]) -**Where:** `bench/src/noise_gate/cov.rs` (~:24-30, :36), `bench/src/bin/noise_gate.rs` (~:147-148), `bench/src/diff/compare.rs` (~:200-208) - -**Problem:** `compute_cov` returns `0.0` for a single sample and the gate treats `cov <= threshold` as pass, so a `--runs 1` invocation reports PASS with zero observed variance — qualifying a noisy machine (defeating the gate's purpose). Zero-mean cells produce `NaN`/`inf` cov and `delta_pct`, marked failing rather than recognized as broken. (The gate is correctly **report-only** — `diff.rs` returns success regardless of regression count and `bench.yml` has no `needs:` into the gating jobs — so it cannot turn CI red on shared-runner noise, consistent with policy; these gaps are about report trustworthiness, not gating.) - -**Direction of fix:** require `runs >= 2`; surface a distinct `INDETERMINATE` / `Undefined` state for N<2 and zero-mean cells so "no signal" renders separately from PASS/FAIL. Guard `baseline == 0.0` in the diff. Bench-only. - -#### I96. Scenario tier has no warmup-discard — cold-start ops inflate tracked p95/p99 [perf-review 2026-06-01] — **P3** (PR-T, [STATIC FACT]) -**Where:** `bench/src/runner.rs` `run_scenario_cell` percentile computation (~:570-574) - -**Problem:** the single-run-no-warmup scenario design folds cold-start ops (cold cache, unwarmed allocator arenas / branch predictors) into the same distribution as steady-state, inflating the tracked `THRESHOLD_PCT_P95/P99` tails. - -**Direction of fix:** discard a warmup prefix from `per_op_ns` before percentile computation, or run a short untimed warmup pass first. Bench-only; document the choice. - ---- - -## Feature requests (2026-06-02) - -Source: primary Chisel client, 2026-06-02. - -### I97. Streaming handle enumeration (`for_each_handle` callback) [client 2026-06-02] — **P3** -**Where:** `src/lib.rs` `Chisel::handles` (~:512); `src/transaction.rs::handles_inner` (~:1435); `src/handle_table.rs::iter_live` / `iter_recursive` (~:353 / ~:604) - -**Motivation:** `Chisel::handles()` already enumerates all live handles, but it MATERIALIZES the full `Vec` (plus a transient `Vec<(u64, HandleEntry)>` inside `iter_live`). For a database with millions of live handles that is a multi-megabyte allocation spike on top of the `O(live handles)` tree walk — the same walk that makes `stats()` O(live handles), see I53. A caller that wants to process every handle without holding them all in memory has no lazy option today. - -**Direction of fix:** add a callback enumerator — e.g. `Chisel::for_each_handle(&self, f: impl FnMut(u64)) -> Result<()>` — that drives `iter_recursive` directly and invokes the closure per live handle instead of pushing into a Vec. A callback, NOT a lazy `Iterator`, is the right shape: the walk holds `&mut PageCache` throughout (pages load/evict as it descends), so a lazy iterator would have to carry that mutable borrow across every `next()` — borrow-checker-hostile. `iter_recursive` already has the structure; it would call `f(handle)` where it currently does `result.push((handle, entry))`. Keep `handles()` as the eager convenience built on top. The Python binding can expose a generator backed by the callback — which would also make the existing `Iterable[int]` `.pyi` annotation honest (it returns an eager `list` today). - -**Thin-impl note (client):** even a callback that internally collects then iterates is a strict ergonomic win over returning the Vec; the zero-materialization version is the actual goal. - -### I98. Make the I28 commit pre-drain conditional — common-case 3→2 fsyncs [client 2026-06-02] — **P2** (investigate; durability-sensitive) -**Where:** `src/transaction.rs::commit_inner` — the pre-drain `self.cache.borrow_mut().flush()?` at ~:855 (I28), ahead of `persist_freemap` (~:864) and the step-1 flush (~:873). The count is pinned by `tests/spillway_integration.rs::no_spill_workload_preserves_two_fsync_commit` (~:111, currently asserts `== 3`). - -**Motivation (client 2026-06-02):** clients want Chisel to minimize fsyncs internally and transparently — no client API needed (the transaction model already batches all ops of a commit into one durable write set; `delete_many` + I33 cover bulk delete). The concrete internal lever is the commit fsync count. Every commit does THREE fsyncs today: the I28 pre-drain (~:855), the step-1 data flush (~:873), and the superblock fsync (step 4). The pre-drain exists ONLY to stop `persist_freemap`'s `allocate_data_page` from tripping `maybe_evict`'s spill-or-`CacheFull`/`SpillwayFull` decision — and per the I28 comment that requires a narrow state: *"every existing entry dirty, nothing evictable, and either spillway disabled or full."* In the COMMON case (cache below the cap, or with clean evictable pages, or spillway headroom) the pre-drain is dead weight: a single step-1 flush could carry every user-dirty page PLUS the one freemap page in ONE fsync, returning the commit to its 2-fsync shadow-paging floor. - -**Direction of fix:** make the pre-drain conditional. Before ~:855, ask the cache whether `persist_freemap`'s single-page allocation could trip the ceiling — a cheap `PageCache` predicate over `entries.len()` vs the cap, the presence of any clean (evictable) entry, and spillway headroom. If allocation is provably safe, SKIP the pre-drain and let the step-1 flush (~:873) fsync everything once (2 fsyncs). Keep the pre-drain whenever the predicate is unsure or the saturated-all-dirty-no-spillway state holds. A conservative predicate (keep the pre-drain on ANY doubt) preserves the I28 guarantee with zero new risk in the rare case while removing a full fsync from every ordinary commit. - -**Don't-Break #2 compliance:** this is NOT "optimizing the pre-drain away" (which the Don't-Break list forbids) — it ADDRESSES the underlying cache-ceiling interaction directly, keeping the flush precisely when the ceiling can bite. The poison-on-`CacheFull` hazard (I28) is untouched in the only state where it can occur. - -**Measurement:** deterministic, not timing-dependent — `ChiselCounters::fsync_calls` counts fsyncs directly. A test asserts a no-spill commit drops 3 → 2, which would restore `no_spill_workload_preserves_two_fsync_commit` to its namesake (it pins `== 3` today only because I28 forced the third fsync — see I63). Wall-time win is one fewer fsync per commit on the file backend — significant on slow storage and on many-small-commit workloads (fsync is the dominant cost per the cost model). The in-memory backend shows the counter drop but no wall-time (its fsyncs are no-ops, cf. I92). - -**Risk / why investigate-not-do:** durability-critical path. The predicate must be provably conservative against the post-spillway `CacheFull`/`SpillwayFull` interaction — the I28 reasoning predates the spillway and now composes with it. Same posture as I78: measure + an explicit architectural argument before implementation, not a blind edit. - -### I99. Handle-table in-memory `depth` not restored on rollback — silent corruption after a rolled-back grow [internal 2026-06-02] — **P1 (critical correctness — silent data loss)** ✅ FIXED 2026-06-02 -**Fix:** extracted the open-time spine walk into `HandleTable::recover_depth` and call it in `rollback_inner` + `rollback_to_inner` (mirroring the `MembershipIndex.outer_depth` fix); regression tests `handle_table_depth_restored_after_rolled_back_grow` / `..._after_rollback_to_savepoint` in `src/transaction.rs`. Direction A (minimal mirror-fix) below was taken; the structural Direction B (depth in `Roots`) remains a possible future cleanup, not needed now. - -**Where:** `src/handle_table.rs` (`HandleTable.depth`, mutated by `grow` at ~:425; descent uses `self.depth` in `find_leaf`/`lookup`/`iter`); `src/transaction.rs::rollback_inner` (~:1005) and `rollback_to_inner` (~:1100), which restore `current_roots` from a snapshot but never touch `handle_table.depth`; the open-time left-spine recovery walk in `open_existing` (~:426–449) is the only place depth is ever re-derived. - -**The bug:** `HandleTable` caches its tree depth in an in-memory `u32` for fast descent. `insert` mutates it (`self.depth += 1`) when the radix tree grows a level. But only the handle-table ROOT page id lives in `Roots`/the superblock — `depth` is NOT part of the snapshotted transactional state. So when a transaction that GREW the handle table is rolled back, `current_roots.handle_table_page` snaps back to the shallow committed root while `handle_table.depth` stays at the grown (too-deep) value. Every subsequent descent then walks one phantom interior level: `find_leaf` reads `buf[DATA_PAGE_HEADER_SIZE..+8]` of a *leaf* page (actually a `HandleEntry`'s `page_id` field) as if it were a child pointer, descends into a data page, and returns `None` → the caller sees `InvalidHandle` for a **previously-committed handle**, or (depending on the bytes) a wrong value / `CorruptPage`. This is silent data loss from valid input. - -**Confirmed (probe, 2026-06-02):** commit exactly `ENTRIES_PER_LEAF` (510) handles (fills depth 0, no grow); `begin`; `allocate` one more (forces grow to depth 1); `rollback`; then `read(5)` for a committed handle → `Err(InvalidHandle(5))`, where the same read returned its 2-byte value moments earlier. Minimal trigger: N=510 committed handles + 1 grown-then-rolled-back allocate + any read of a committed handle. - -**Why it stayed latent:** handle ids are dense and monotonic, so the depth-0→1 grow boundary (510 live handles) is a rare, sequential event that few tests combine with rollback-then-read. The identical bug class in the membership index's outer tree (`MembershipIndex.outer_depth`) was hit immediately during chunk-tags development because tag values are client-supplied and sparse — a single `allocate_tagged(v, tag>=1021)` grows the outer tree — and was FIXED on branch `feature/chunk-tags` (commit `7228c85`) by re-deriving `outer_depth` from the restored root in both rollback paths, mirroring open-time recovery. This issue is the **handle-table half of the same root cause**, left unfixed deliberately to keep that fix scoped and give the core handle table its own focused change + tests + review. - -**Direction of fix (two options):** -- **A — minimal mirror-fix (low risk, proven):** extract the open-time left-spine walk (`open_existing` ~:426–449) into a reusable `HandleTable::recover_depth(cache, root) -> Result` (the analogue of `RadixU64::recover_depth`), call it at open AND in `rollback_inner`/`rollback_to_inner` right after `current_roots` is restored, then `handle_table.set_depth(d)`. Same shape as the membership fix already on `feature/chunk-tags`. Cost: one extra `cache.get` (possibly a disk read) per rollback. Eliminates the bug with the smallest, most local change. -- **B — structural (cleaner, eliminates the class):** make the in-memory depth part of the snapshotted state instead of a side-channel field — add `handle_table_depth` (and fold the membership `outer_depth` in too) to the in-memory `Roots` struct so `begin`/`commit`/`rollback`/`savepoint`'s existing `Roots` clones carry it automatically. No superblock format change needed (depth is derivable from the root; it only needs to ride the in-memory snapshot). Removes the entire "stranded in-memory radix depth across rollback" failure mode for both trees at once, at the cost of a moderate refactor threading depth through `Roots` rather than the `HandleTable`/`MembershipIndex` fields. - -**Test to add (either fix):** the probe above as a regression test — commit `ENTRIES_PER_LEAF` handles, grow+rollback one more, assert every committed handle still reads back; plus the `rollback_to(savepoint)` variant. (The membership-index equivalents — `tagged_membership_survives_rolled_back_outer_grow` / `..._rollback_to_savepoint` in `src/transaction.rs` — are the template.) - -**Severity note:** P1 because it is silent data loss on the engine's most fundamental structure, reachable from ordinary `allocate`+`rollback` usage with no unsafe code or corruption precondition. Mitigating factor: no production databases exist yet (cf. format-version tentativeness), and a process restart accidentally "heals" the in-memory depth via open-time recovery — so the corruption is confined to the lifetime of a single open handle after a rolled-back grow. - -### I100. Callback enumeration `for_each_handle_with_tag` [chunk-tags out-of-scope 2026-06-02] — **P3** -**Where:** `src/transaction.rs` (`handles_with_tag`); `src/membership_index.rs` (`handles_for_tag`); `src/lib.rs` / `python/src/db.rs` public surface. - -**Motivation:** `handles_with_tag(tag)` materializes the full `Vec` of a tag's members, the same multi-megabyte spike I97 calls out for `handles()`. A relation with millions of chunks has no lazy option. Mirror I97's planned `for_each_handle(f)` shape with `for_each_handle_with_tag(tag, f: impl FnMut(u64))` driving the inner radix iteration without collecting. Blocked on I97 landing first so the two share one callback idiom. Additive; the eager `handles_with_tag` stays as the convenience wrapper. The Python binding can expose a generator over the callback. Filed from the chunk-tags spec's "Out of scope for v1". - -### I101. Bitmap inner sets for dense per-tag handle ranges [chunk-tags out-of-scope 2026-06-02] — **P3** (profile-gated) -**Where:** `src/membership_index.rs` — the per-tag inner `RadixU64` (handle → 1 membership bit). - -**Motivation:** the membership index stores each tag's member set as a radix tree, optimal for the EXPECTED sparse handle distribution. If profiling on a real relational workload ever shows DENSE per-tag handle ranges (e.g. a relation whose chunks are allocated contiguously), a bitmap inner set would be both smaller and faster to scan than the radix. Do NOT do this speculatively — the radix is the right default; this is a measured-need optimization only. Filed from the chunk-tags spec's "Out of scope for v1". - -### I102. Batched per-page frees / streaming drop in `delete_with_tag` [chunk-tags out-of-scope 2026-06-02] — **P3** -**Where:** `src/transaction.rs::delete_with_tag_inner` — the `for &h in &take { self.delete_inner(h)?; }` loop. - -**Motivation:** `delete_with_tag` deletes one handle per `delete_inner` call, each a full handle-table walk + per-handle freeing, exactly the per-handle shape I33 wants to batch for `delete_many` (per-leaf batched delete walks the tree once per leaf, not once per handle). When I33's batched-delete primitive lands, route `delete_with_tag`'s bounded batch through it for dense-relation drops. Also relevant: a relation far larger than one `max` batch already drops incrementally (the caller loops `begin → delete_with_tag → commit`), so this is a per-batch efficiency item, not a correctness or unbounded-time gap. Filed from the chunk-tags spec's "Out of scope for v1". - -### I103. Bump `pyo3` from 0.24 to 0.29 to clear RUSTSEC-2026-0176 / RUSTSEC-2026-0177 [audit 2026-06-16] — **P2** ✅ FIXED 2026-06-16 -**Where:** `python/Cargo.toml` (`pyo3 = "0.29"`); migration touches `python/src/{db,transaction,savepoint,errors}.rs`. - -**What:** two RustSec advisories landed against `pyo3` < 0.29 — RUSTSEC-2026-0176 (out-of-bounds read in `PyList` / `PyTuple` `nth` / `nth_back` iterators) and RUSTSEC-2026-0177 (missing `Sync` bound on `PyCFunction::new_closure`). The chisel binding uses NEITHER vulnerable API, but `cargo audit` flags the crate transitively, turning the gating `audit` CI job red repo-wide. Both are patched only in 0.29.0, so the fix is a 0.24 → 0.29 bump (an alternative `cargo audit --ignore` was rejected in favor of the real upgrade). PyO3 0.29 migration applied: `PyObject` → `Py` (dropped from the prelude); `Python::with_gil` → `Python::attach` and `py.allow_threads` → `py.detach` (the attach/detach GIL model); `Bound::downcast` → `Bound::cast`; and `#[pyclass(from_py_object)]` opt-in on `PyDrainInsertion` (the auto-`FromPyObject` derive for `Clone` pyclasses is now opt-in, and this enum IS extracted from Python args). Public Python API unchanged. `cargo audit` → 0 vulnerabilities; the engine/bench suites and `cargo clippy -p chisel-py --all-targets -- -D warnings` pass; the CI Python matrix (pytest, CPython 3.11/3.13 × Linux/macOS) validates runtime. Independent of the handle-table COW page-reclamation work (PR #34). - ---- - -## Deepdive review findings (2026-06-21) - -The `deepdive-rust` fresh-eyes pass at commit `deb0303` (full output at `docs/reviews/review-20260621-185541.md`). `cargo build`/`test`/`clippy --all-targets -- -D warnings`/`fmt --check` were green on that commit. The *entire* 2026-06-16 executive summary is resolved with no regressions (handle-table COW spine leak, tag-map `CacheFull` divergence, corrupt-radix OOB panic, the `Deleted=0x03` doc, the crash-between-fsyncs test — all closed by PRs #34/#40/#44). No new P0 / data-loss bugs. The cluster below (severity tags map to the file's P-legend: BUG/blocking → P1, DESIGN/latent → P2, SMELL/NIT → P3) concentrates in error-classification defaults, a CI lint hole, the radix test gap, and doc drift from the #46 dead-code sweep. - -Suggested order for this cluster: **I108** (CI lint hole) and **I111** (radix proptest gap) first — both are real gaps with cheap fixes; then the zero-risk cheap batch **I104 / I127 / I128 / I130 / I131 / I132 / I133** (a test, a hasher swap, and doc edits — no runtime behaviour change); then the P2 DESIGN items as they get touched. The P3 idiom/NIT items batch with any PR in their file. - -### Error handling and classification - -#### I104. `ChiselError::is_fatal()` is fail-open — unlisted future variants classified non-fatal [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/error.rs:168-181` - -**Problem:** the `matches!`-based classifier returns `false` for any variant not in its Fatal list, so a future fatal variant a maintainer forgets to add is silently classified *operational* and will NOT poison the manager. For a durability-first engine this is the one function whose default leans the wrong way; the in-code INVARIANT comment already admits it's "a durability hole, not a compile error." All 24 current variants are classified correctly — the gap is latent, not a present bug. - -**Direction of fix (NOT a default flip):** the review argues a `_ => true` flip just moves the latent bug (a future *operational* variant would then over-poison and tank availability). Instead add an exhaustiveness test that constructs every `ChiselError` variant and asserts `is_fatal()` matches the value the enum's documented Fatal/Operational block prescribes. Converts the prose invariant into a compile-time-ish signal the `#[non_exhaustive]` enum can't give, catches *both* misclassification directions, changes zero runtime behaviour (~30 lines). See the review's countercase section for the full argument. - -**Fixed (2026-06-21):** added `is_fatal_matches_documented_classification_for_every_variant` to `src/error.rs`'s test module. A test-local `documented_is_fatal` helper classifies every variant via a match with **no `_` arm** — and because the test lives in the defining crate, `#[non_exhaustive]` does NOT suppress the exhaustiveness check, so adding a variant fails to compile until it is placed in the Fatal or Operational block (the compile-time guard `matches!` can't give). The test then asserts `is_fatal()` agrees with the independent documented classification for a constructed instance of all 24 variants, plus a `== 9 fatal` tripwire. No runtime behaviour change; the default was NOT flipped. - -#### I105. Blanket `From` makes every `?` on an I/O call produce a *fatal* `IoError` invisibly [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/error.rs:282-286` (conversion); `:280-281` (the caveat comment) - -**Problem:** the blanket conversion means any `?` on an io call silently produces a fatal `IoError` and poisons, even on a benign `NotFound`. The comment concedes callers "must catch and remap" to classify e.g. `NotFound` as operational, but nothing enforces it — the `FileNotFound` operational variant exists *because* the conversion is too coarse. A future `?` on an io call that should be recoverable will poison. - -**Direction of fix:** drop the blanket `From` (force explicit classification at each I/O site), or have the conversion inspect `ErrorKind` and route the known-operational kinds (`NotFound`, `AlreadyExists`, …) to operational variants. - -**Fixed (2026-06-21, ErrorKind-inspection option, per maintainer decision):** the `From` impl now matches on `e.kind()` and routes `io::ErrorKind::NotFound` → operational `FileNotFound`; every other kind stays fatal `IoError`. Scope was deliberately limited to `NotFound` (not `AlreadyExists`): NotFound can only arise from resolving a missing *path* (open of a non-existent file), never from a read/write on the already-open, flock'd fd, so demoting it is safe — whereas demoting a kind that might signal real corruption would be the same fail-open hazard as I104. `Chisel::open` already pre-checks the missing-file case at `lib.rs:295` and returns `FileNotFound` directly, so the tested open path is unchanged; this makes any *stray* NotFound that reaches a `?` classify consistently instead of poisoning. Full suite green (no test regressed). The stale "every io::Error becomes a fatal IoError" comment was rewritten. - -#### I106. `CorruptSuperblock` folds three distinct causes into one nullary fatal variant [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/error.rs:106` - -**Problem:** bad checksum, bad magic, and out-of-range `superblock_count` all surface as the same nullary, fatal, unrecoverable `CorruptSuperblock`; the comment admits operators "should look at the raw slot bytes." Discarding which-slot-and-why on the engine's worst failure is an operability loss. (Related: `InvalidMagic` at `:111` is likely unreachable because a bad magic already surfaces here via `select` — see I115.) - -**Direction of fix:** add a cause field (`enum SuperblockDefect { Checksum, Magic, BadCount(u32) }`) and/or the offending slot index. - -#### I107. `delete_with_tag` discards its partial-deletion list on a mid-pass error [deepdive 2026-06-21, carried from 2026-05-22] — **P3** ✅ RESOLVED 2026-06-21 (already documented) -**Where:** `src/lib.rs:553`; `src/transaction.rs:2041` (`delete_with_tag_inner`) - -**Problem:** the `?` inside the per-handle loop drops the already-built `deleted` set; the caller gets `Err` with no `TagDropProgress`. On-disk state stays consistent (the failed transaction rolls back), but a resumable relation-drop cannot reconcile what it had dropped. Unchanged since the 2026-05-22 pass — not a regression, restated to keep it tracked under a number. - -**Direction of fix:** attach the partial progress to the error variant, or document that rollback-on-error is mandatory and the caller must restart the drop. (Interacts with I102's batched-drop work.) - -**Resolved (2026-06-21, p3-cleanup batch — premise stale):** the chosen "document" option is ALREADY in place and is more accurate than the deepdive's framing. The `delete_with_tag` rustdoc (`src/lib.rs`) documents that a mid-pass error leaves a *consistent* in-transaction state where BOTH `rollback()` (discard the pass) and `commit()` (keep the partial drop) are safe, that the dropped-this-pass set is unrecoverable from the return value, and that the caller re-enumerates via `handles_with_tag` / re-runs the bounded loop to finish (committing single-element passes to learn exactly which handles dropped). No code change needed — "rollback is mandatory" would be *wrong* since commit-the-partial is also safe. No structured progress is attached to the error (the maintainer's recorded decision). - -### CI and automation - -#### I108. The PyO3 binding (1,300+ lines of Rust) is never clippy-linted in CI [deepdive 2026-06-21] — **P1** ✅ FIXED 2026-06-21 -**Where:** `.github/workflows/ci.yml:40` (root clippy step); python job at `:118-160` - -**Problem:** the gating `cargo clippy -- -D warnings` (no `--workspace`/`-p`) honors `default-members = [".", "bench"]`, which excludes `chisel-py`; the python job runs only `maturin develop` + `pytest`. Confirmed empirically — clippy only checks `chisel` and `chisel-bench`. So clippy regressions in `python/src/*` ship silently. (`cargo fmt -- --check` *does* cover it — rustfmt walks all members — so only clippy has the hole.) - -**Direction of fix:** add `cargo clippy -p chisel-py --all-targets -- -D warnings` (with the maturin linker env) to the python job, or run root clippy with `--workspace`. - -**Fixed (2026-06-21):** took the `--workspace` option — the clippy job now runs `cargo clippy --workspace -- -D warnings`, which lints all three members (chisel, chisel-bench, chisel-py). Verified locally that the run checks `chisel-py v0.1.0` and passes clean. pyo3's `extension-module` feature means no libpython link during a check, so it runs on the bare clippy runner with no Python/maturin setup. Chose `--workspace` over a dedicated python-job step to avoid running clippy 4× across the OS×version matrix and to future-proof any new workspace member. - -#### I109. The release/tag wheel path has no `cargo audit` gate [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `.github/workflows/wheels.yml:11-17` - -**Problem:** the `audit` job lives in `ci.yml` (push/PR to main) but is not a dependency of the tag-triggered wheel build, which runs only `cargo test --release`. A vulnerable dependency introduced and tagged without merging through a PR would publish wheels unaudited. - -**Direction of fix:** add a `cargo audit` step (or reuse the `rustsec/audit-check` action from I54) as a gate on the wheel job. - -**Fixed (2026-06-21):** added `cargo install cargo-audit` + `cargo audit` to `wheels.yml`'s `cargo-test-gate` job (which both `wheels` and `sdist` already `needs:`), so a tag build now fails fast on a vulnerable dependency instead of publishing wheels unaudited. - -#### I110. The MSRV job builds but never `--tests` and skips bench/python [deepdive 2026-06-21] — **P3** ✅ RESOLVED 2026-06-21 (not viable as specified — msrv stays lib-only; reason below) -**Where:** `.github/workflows/ci.yml:98-107` - -**Problem:** `cargo build -p chisel` verifies the library compiles at 1.82 but not its test code, so an MSRV-breaking construct in a test module passes. Low risk. - -**Direction of fix:** `cargo build --tests -p chisel` in the msrv job (a cheap strengthening; keep bench/python scoped out per I55/I61's deliberate floor-floating decision). - -**Resolved (2026-06-21) — NOT VIABLE as specified, reverted:** `--tests` was tried and the msrv CI job went **red**: `cargo build --tests -p chisel` at 1.82 fails to resolve proptest's transitive `getrandom v0.4.2`, which now requires the `edition2024` Cargo feature (Rust 1.85+). So chisel's *test* dependency tree floats above the 1.82 *library* floor — the exact MSRV-floating problem I61 documents for bench/python. Reverted to lib-only `cargo build --verbose -p chisel`; added an inline `ci.yml` comment recording why `--tests` is intentionally NOT used, so it isn't re-attempted. The MSRV promise is about the published library (which has no `getrandom`/proptest in its tree), and that remains verified. Net: the review's "cheap strengthening" was a good idea that the dependency ecosystem makes impractical — a deliberate non-fix, not an oversight. - -### Tests - -#### I111. Radix key-math has no property tests, and the one "two-level" test reaches depth 1 [deepdive 2026-06-21, follow-up to I71] — **P1** ✅ FIXED 2026-06-21 -**Where:** `src/handle_table.rs:1313` (`test_handle_table_grows_to_two_levels`); the `find_leaf`/`span_at_level`/`insert`/`grow`/`recover_depth` paths and the identical `src/membership_index.rs` paths - -**Problem:** `test_handle_table_grows_to_two_levels` is misnamed — it inserts `ENTRIES_PER_LEAF + 10` (520) handles (forces depth 1) and never asserts `ht.depth()`. No test exercises a depth-≥2 descent, so the multi-digit `span_at_level` / `handle % child_span` decomposition — the engine's most off-by-one-prone code, addressing both the stable-handle table and the tag index — is exercised only at depth 0–1. I71 added proptests for slot packing / freemap bit math / checksum round-trip but not for the radix math. - -**Direction of fix:** insert `> ENTRIES_PER_LEAF²` and assert depth ≥2; add a `HashMap`-oracle property (`insert(h,e); lookup(h)==Some(e)` over arbitrary `u64` on a depth-≥2 tree, plus a `grow → recover_depth` round-trip) — ~15 lines that would catch any `span_at_level` off-by-one instantly. `transaction.rs` also lacks a stateful commit/rollback/savepoint proptest despite a ready oracle (`assert_no_reachable_page_is_free`, `:2642`). - -**Fixed (2026-06-21):** **correction to the direction above** — the depth-2 boundary is `capacity(1) = ENTRIES_PER_LEAF * PTRS_PER_INTERIOR = 510 * 1021 = 520_710`, NOT `ENTRIES_PER_LEAF²` (= 260_100, still depth 1). Changes: (1) renamed `test_handle_table_grows_to_two_levels` → `test_handle_table_grows_to_depth_one` and added `assert_eq!(ht.depth(), 1)` so it honestly pins what it exercises; (2) added `test_handle_table_grows_to_depth_two` — inserts a sparse handle set including one ≥ 520_710, asserts `depth() == 2`, reads every entry back, and asserts an uninserted in-capacity handle reads absent (the historical I6/I26 phantom-descent guard); (3) added two `HandleTable` proptests — a `HashMap`-oracle insert/lookup over handles `0..1_100_000` (cases routinely reach depth 2, last-write-wins) and a `grow → recover_depth` round-trip over `0..600M` (depths 0–3); (4) mirrored both proptests onto the membership index's generic `RadixU64` (the second radix implementation — `prop_radix_insert_lookup_matches_oracle` over keys `0..2.5M` crossing its `capacity(1) = 1021² = 1_042_441` boundary, and `prop_radix_recover_depth_matches` over `0..1.1e9`). All six new tests pass; full suite green. The `transaction.rs` stateful-commit proptest noted above is NOT done here — deferred as its own item. - -#### I112. Fault-injection test layer absent — the poison/flush coupling and the fatal-`IoError` path are reasoned, not tested [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/page_cache.rs` flush (clears dirty flags before the trailing fsync); `src/transaction.rs:2833` (`fatal_error_outside_commit_also_poisons`); `error.rs:94` (`IoError`) - -**Problem:** flush clears per-page dirty flags *before* the trailing fsync — safe *only* under the poison model, and the comment warns it breaks if poisoning is weakened. No test asserts a flush/fsync error actually poisons: `test_poison_recovery_by_reopen` can't inject a fault, and `fatal_error_outside_commit_also_poisons` is tautological (`force_poison_for_test()` then asserts `Poisoned`). The most-traveled fatal path (`IoError`) is constructed-only — no test induces a real I/O fault and observes the engine returning it. - -**Direction of fix:** a `Backing::FaultyFile` test variant that fails a chosen syscall closes the poison/flush gap and exercises `IoError` / `CorruptSuperblock` / `InvalidMagic` end-to-end (also addresses open question 4 and several I115 gaps at once). - -**Fixed (2026-06-21, PR #62):** the mechanism landed NOT as a `Backing` variant but as a `#[cfg(test)] fault: Cell` field on `PageIo` — it mirrors the existing `fail_next_membership_op` injection pattern, composes with BOTH backings (so the poison tests run on the fast in-memory `PageIo`), and avoids duplicating the Memory page-store. `Fault` is a `Copy` enum — `None | FailFsync(u32 countdown) | FailWritePage(u64) | FailReadPage(u64)` — checked at the top of `read_page`/`write_page`/`fsync`, one-shot (the io::Error is synthesized at the site so the enum stays `Copy`); `arm_fault` is the test-only setter. Tests: (a) `page_cache.rs::failed_flush_fsync_leaves_pages_clean_but_nondurable` confirms `dirty_count == 0` after a faulted flush — the durability window made observable; (b) `transaction.rs::commit_fsync_failure_poisons_at_each_of_the_three_fsyncs` drives the `FailFsync` countdown at each of commit's three fsyncs (a one-shot would only have hit the pre-drain — the countdown was a required refinement found in design verification); (c) `commit_write_failure_poisons`; (d) `fatal_error_outside_commit_also_poisons` REWRITTEN to drive a real `FailReadPage` fault through a cold reopen instead of the `force_poison_for_test` tautology (`force_poison_for_test` kept — `poisoned_manager_rejects_every_public_entry_point` still uses it as a precondition). Deferred non-goals: faulting `set_page_count` and open-time `IoError`. Subsumes I114 + I115. - -#### I113. Weak assertions that survive a subtle break [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `tests/defrag.rs:63`; `src/page_cache.rs:1294`; `tests/spillway_runtime_mutability.rs:33`; assorted `assert!(x.is_some()/is_err())` sites - -**Problem:** `assert!(pages_freed > 0)` passes if defrag freed 1 of ~45 (the sibling `:154` bounds it correctly); `assert_eq!(dh+dm,1)` passes if a hit is recorded as a miss; the spillway-mutability test sets the cap then asserts nothing about its effect; several `is_some`/`is_err` checks never inspect the value/variant. - -**Direction of fix:** tighten each to assert the actual expected value/variant/bound. - -**Fixed (2026-06-21, p3-cleanup batch):** the three named sites + ~14 `is_*` sites tightened. `tests/defrag.rs:63` `pages_freed > 0` → exact `pages_examined==1`/`values_moved==5`/`pages_freed==1`; `page_cache.rs` `dh+dm==1` → separate `cache_hit_count`/`cache_miss_count` assertions (catches a hit recorded as a miss); `spillway_runtime_mutability.rs` → after `set_spillway_max_bytes(0)`, saturate the cache and assert `CacheFull` actually fires (proves the cap took effect). The `is_some`/`is_err` sites were sharpened to specific variants (`InvalidHandle`, `ReadOnlyMode`, `InvalidSuperblockCount{value}`, `FileNotFound`, `ZeroTagError`, full-entry equality) where a specific value is expected; genuinely-loose ones (`is_ok()` on `()`, a proptest `is_some` whose next line checks the field) left with rationale. One site (`recovery_tests.rs` both-superblocks-corrupt) was tightened to assert `is_fatal()` rather than the `CorruptSuperblock` variant SHAPE, deliberately — I106 (PR #63) concurrently reshapes that variant, and a nullary-variant match would silently fail to compile once both land. - -#### I114. `claim_page_asserts_on_dirty_page` is `#[cfg(debug_assertions)]` and vanishes under `--release` [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/page_cache.rs:1236`; the release-profile gate at `wheels.yml:17` - -**Problem:** the test guarding the I20 dirty-page invariant is debug-only, so it silently disappears under `cargo test --release` — which the wheel gate uses. The invariant is checked only in dev-profile runs. - -**Direction of fix:** either keep the `debug_assert!` but assert the *observable* consequence in a release-safe test, or document why debug-only coverage is acceptable here. - -**Fixed (2026-06-21, PR #62):** kept the `debug_assert!` (cheap dev guard) and the debug-only `claim_page_asserts_on_dirty_page`, and ADDED a release-compiled `claim_page_keeps_dirty_count_consistent` that asserts the *observable* accounting consequence — claiming a clean page (the legitimate freemap-reuse path) adds exactly one dirty entry, so `dirty_count` stays consistent with the live entry set in EVERY profile. Verified passing under both `cargo test` and `cargo test --release`. - -#### I115. Error-variant coverage gaps; `InvalidMagic` likely unreachable; `CorruptSuperblock` never pinned [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/error.rs:111` (`InvalidMagic`); `tests/recovery_tests.rs:533` (the OR-arm); the `Poisoned` end-to-end loop - -**Problem:** `InvalidMagic` is probably unreachable (a bad magic surfaces as `CorruptSuperblock` via `select` — see I106); `CorruptSuperblock` is never *pinned* as the expected variant (only matched in an OR-arm); the `Poisoned` end-to-end path is synthetic (`force_poison_for_test`, never a real fatal error driven into a live manager — see I112). **Memory correction:** the project note "ReadOnlyMode never raised" is stale — `tests/options_validation.rs:240` genuinely drives and matches it. - -**Direction of fix:** pin `CorruptSuperblock` as the sole expected variant in the recovery tests; decide whether `InvalidMagic` is dead (remove) or reachable (add a test); drive a real fatal error for the `Poisoned` loop (depends on I112). - -**Fixed (2026-06-21, PR #62):** `recovery_tests.rs::corrupt_magic_surfaces_as_corrupt_superblock_not_invalid_magic` corrupts the 4-byte magic in EVERY superblock slot (re-stamping the checksum so the magic check, not the checksum check, is the failing gate) and asserts `CorruptSuperblock` as the SOLE expected variant — pinning it AND proving `InvalidMagic` unreachable in one test. `InvalidMagic` was then REMOVED as provably-dead public API (mechanism: a bad magic makes `Superblock::deserialize` return `None` for every slot → `select` returns `None` → the open path surfaces `CorruptSuperblock`, never `InvalidMagic`). Removal spanned the enum decl, `is_fatal`/`Display`/exhaustiveness-test arms, the fatal-variant tripwire count (9 → 8), the PyO3 binding (`InvalidMagicError` exception + registration + match arm), the Python package (`__init__`, `.pyi`, `test_errors.py`), and both READMEs — maintainer-approved (single Chisel client, can adapt). The `Poisoned` end-to-end path is now driven by a real fault (see I112's rewritten `fatal_error_outside_commit_also_poisons`). - -### Correctness - -#### I116. `PageCache::truncate` decrements `dirty_count` unguarded — underflow disables the cache cap [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/page_cache.rs:625-654` (`truncate`); the guarded siblings `discard`, `maybe_evict` Phase B, `set_cache_max_bytes` (justification comment at `:986-994`) - -**Problem:** `self.dirty_count -= 1` per removed dirty entry is the one decrement site not guarded against an `entries`/`dirty_count` desync. On any desync it underflows (debug panic / release wrap to `usize::MAX`), after which `maybe_evict`'s `dirty_count == entries.len()` short-circuit never fires and the cache silently stops enforcing its cap. This `dirty_count`-underflow shape has now appeared in two consecutive reviews at different call sites (2026-06-16 flagged the `maybe_evict` site, since guarded). - -**Direction of fix:** recompute `dirty_count` from surviving entries after the loop, or saturate — and consolidate all four decrement sites behind one helper to end the recurrence (ties into I136's eviction-loop extraction). - -**Fixed (2026-06-21):** added `PageCache::untrack_dirty(was_dirty)` doing `dirty_count = dirty_count.saturating_sub(1)` and routed the decrement sites through it. **Two corrections to the framing:** (1) `truncate` was NOT the lone unguarded outlier — like `discard` and `maybe_evict` Phase B it already had the `if entry.dirty` guard; the genuine fix is making all of them **saturating** (they used plain `-= 1`, which underflows only under a real `entries`/`dirty_count` desync). (2) There are **three** decrement sites, not four: `set_cache_max_bytes` evicts CLEAN entries only and never touches `dirty_count`. Saturation turns a desync into "cap not strictly enforced until the next flush/rollback resets the count" instead of a debug panic / release wrap. Behavior-preserving under the invariant; full suite green. - -#### I117. `insert_into_data_page`'s `.expect("value fits in empty page")` is a library-reachable panic if the inline-size constants drift [deepdive 2026-06-21, follow-up to I46] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/transaction.rs:2576`; the hand-maintained literal `MAX_INLINE_VALUE` at `:66` - -**Problem:** correct today (`MAX_INLINE_VALUE = 8162` exactly equals `CHECKSUM_OFFSET - DATA_PAGE_HEADER_SIZE - SLOT_ENTRY_SIZE`), but `MAX_INLINE_VALUE` is a hand-maintained literal with only a prose "keep in sync" note. If any of the three constants is edited independently, `DataPage::insert` returns `None` on the "fits" path and a plain `allocate()` of the wrong-sized value panics the process. I46 added an `// INVARIANT:` comment here; this asks for a compile-time guard. - -**Direction of fix:** `const _: () = assert!(MAX_INLINE_VALUE == CHECKSUM_OFFSET - DATA_PAGE_HEADER_SIZE - SLOT_ENTRY_SIZE);`, or map the `None` to a typed error. - -**Fixed (2026-06-21):** went one better than a const-assert — made `MAX_INLINE_VALUE` a **derived** const: `page::CHECKSUM_OFFSET - page::DATA_PAGE_HEADER_SIZE - data_page::SLOT_ENTRY_SIZE` (was the `8162` literal). It can no longer drift (it IS the page constants), so the upstream `value.len() > MAX_INLINE_VALUE` check can't diverge from the page's real capacity and the `.expect` is unreachable by construction. Made `data_page::SLOT_ENTRY_SIZE` `pub(crate)` so transaction.rs can reference it. Value unchanged (still 8162); overflow-boundary tests pass. - -#### I118. Fresh inner-tree roots bypass the freemap-aware allocator — churn workloads extend the file [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 (common case; deep-tree residual tracked below) -**Where:** `src/membership_index.rs:603` (`create_root` → `init_page` → `cache.new_page()`); the "bounded steady-state page count" header claim at `:41-49` - -**Problem:** `MembershipIndex::insert` threads the freemap-aware `alloc` closure into `inner.insert`, but a tag's first-ever `create_root` always extends (never reuses a freed page). Not a correctness bug (the id is past the watermark; rollback's `truncate` and commit's freemap merge both handle it), but a workload that repeatedly empties and re-creates a tag's inner tree extends the file once per re-create, undercutting the module's own bounded-page-count claim. - -**Direction of fix:** plumb the `alloc` closure into `create_root`/`init_page`, or narrow the header comment to exclude first-create. - -**Fixed (2026-06-21, p3-cleanup batch):** chose the code fix. Threaded the freemap-aware `alloc` closure through `init_page`/`create_root` and its two production call sites, so re-creating a dropped tag's inner tree reuses a freed page instead of extending. This required also freeing the emptied inner tree's ROOT page in `MembershipIndex::remove` (previously left un-freed) — otherwise there is nothing in the freemap to reuse. A 3-agent adversarial freemap-safety review CONFIRMED the root-free is safe: freed exactly once, fully orphaned (the outer entry is removed alongside), correctly handled by `persist_freemap`'s allocate-first/merge-last discipline (I18), and reusable across reopen — no double-free, no use-after-free, no corruption. Pinned by `tests/tag_ops.rs::recreating_a_dropped_tag_reuses_freed_pages` (drop a tag fully, re-create, `total_pages` stable). **RESIDUAL (follow-up):** `RadixU64::delete` does NOT shrink depth, so a tag whose inner tree had grown to depth ≥ 1 (carried > `SLOTS_PER_PAGE` handles at once) empties to a *chain* of interior pages — only the root is reclaimed here; the empty interiors leak (bounded, non-corrupting, strictly better than the prior free-nothing behavior). The common single-leaf case I118 targets is fully reclaimed. Completing it needs a depth-shrinking `delete` or a walk that frees the whole emptied subtree. - -#### I119. `commit_inner`'s `txn_counter += 1` is an unchecked `u64` increment [deepdive 2026-06-21, carried] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/transaction.rs:889` - -**Problem:** practically unreachable, but a wrapped counter corrupts `Superblock::select`'s "highest counter wins", and a debug overflow panics mid-protocol after dirty flags were cleared. Inconsistent with the `saturating_add` discipline used elsewhere. - -**Direction of fix:** `checked_add` → fatal error on overflow. - -**Fixed (2026-06-21):** `self.txn_counter = self.txn_counter.checked_add(1).expect("…unreachable")`. **Chose a controlled panic over the review's "fatal error" suggestion deliberately:** overflow needs 2^64 commits (structurally unreachable), so a dedicated fatal `ChiselError` variant for it would be speculative public surface (a new enum arm + Display + `is_fatal` + the I104 exhaustiveness test + a Python exception class — all for an impossible event). The real defect was the **silent release-side wrap to 0** corrupting `Superblock::select`; `checked_add().expect()` replaces that with a loud, controlled failure on the invariant. If a typed fatal variant is preferred, it's a one-line swap at the `expect`. - -### API design - -#### I120. Handles are raw `u64` and tags raw `u32` across the whole public surface [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/lib.rs` (`allocate -> u64`; `read`/`update`/`delete`/`tag`/`handle_live_page_id` take `u64`; `delete_tagged(handle, tag)` takes two integers) - -**Problem:** a crate that is `#[non_exhaustive]`-everything and careful about API stability leaves its most-used values primitive-obsessed; `delete_tagged`'s two integer args can be transposed with no compiler error. (Related: `handles()` / `handles_with_tag()` return eager `Vec` with a three-paragraph prose contract — the streaming variant is already tracked as I97/I100; the newtype is the orthogonal half.) - -**Direction of fix:** opaque `Handle(u64)` / `Tag(u32)` newtypes with `From`/`Into` — makes `delete_tagged` un-transposable and distinguishes `handles()` from `handles_with_tag()` return types. - -**Fixed (2026-06-21):** `Handle(u64)` (`#[repr(transparent)]`) and `Tag(NonZeroU32)` newtypes in a new `src/handle.rs`, re-exported from the crate root. The full reshape (maintainer's choice) flips every `Chisel` method in `lib.rs` to take/return `Handle`/`Tag`; the engine (`transaction.rs`, `handle_table.rs`, `membership_index.rs`) stays raw `u64`/`u32` (radix math + on-disk layout are structural). `delete_tagged(Handle, Tag)` is now un-transposable. Per the I120 decision, the newtypes carry `PartialEq` against their raw primitive + `Display` for ergonomics (most call sites compile untouched). PyO3 binding converts at the edge; Python handles/tags stay plain `int`. Design: `docs/specs/2026-06-21-handle-tag-newtype-reshape-design.md`; plan: `docs/plans/2026-06-21-handle-tag-newtype-reshape.md`. - -#### I121. `DefragStats` and `TagDropProgress` are outcome reports but not `#[must_use]` [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/defrag.rs:111` (`DefragStats`); `src/membership_index.rs:538` (`TagDropProgress`) - -**Problem:** `db.defrag(opts)?;` silently discards `pages_freed`; `delete_with_tag(...)?;` discards `complete` — the field the caller must loop on (see I107). The crate already added `#[must_use]` to `close()` (I38), so it values the pattern. - -**Direction of fix:** `#[must_use]` on both type definitions. - -**Fixed (2026-06-21):** `#[must_use = "…"]` on both struct defs. It immediately earned its keep — clippy `--all-targets` flagged one genuine discard (`tests/defrag.rs:81` `db.defrag(..).unwrap();`), now `let _ = …` since that test only checks post-defrag readback. - -#### I122. `DefragOptions::max_pages` counts values relocated, not pages [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/defrag.rs:64-70` (the "DESPITE THE NAME…" comment) - -**Problem:** preserving a misleading public field/setter name "for stability" on a pre-1.0, `#[non_exhaustive]`, no-production-users crate is the wrong trade — the C4 doc-sweep and the Python `__init__.py` note both already paper over the mismatch instead of fixing it. - -**Direction of fix:** rename to `max_values` now (and the Python `DefragOptions.max_pages` mirror), while there are no users to break. - -**Fixed (2026-06-21):** renamed the field, builder method, `Default`, the single consumer in `defrag.rs`, and the Python mirror (`__init__.py` dataclass + docstring, `.pyi`, the duck-typed `getattr` in `python/src/db.rs`). The "DESPITE THE NAME" comment is gone — the name is now honest. Clean break (pre-1.0, `#[non_exhaustive]`, no production users). Landed first as its own commit, independent of the newtype reshape. - -#### I123. `page_count` / `file_page_count` keep `&mut self` "for API stability" on a pure `Cell` read [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/page_io.rs:330,594` - -**Problem:** these now-pure cache reads keep `&mut self`, forcing a `borrow_mut()` on semantically-read `&self` paths — a latent double-borrow risk if a future `&self` path nests a cache read. The comment already concedes a future patch can drop the `&mut`. - -**Direction of fix:** drop the `&mut self` now (the ripple is small). - -**Fixed (2026-06-21):** `PageIo::page_count` and `PageCache::file_page_count` now take `&self` (both are pure `Cell`/field reads). Callers holding `&mut` still compile (coercion); the ripple the review predicted was indeed small — three now-unnecessary `mut` bindings (`PageCache::new`'s `io` param + two page_io tests), de-`mut`'d. - -#### I124. Public fallible methods broadly lack `# Errors` rustdoc; several omit `# Panics` [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `savepoint`/`rollback_to`/`release`/`set_root_name`/`get_root_name` and most `TransactionManager` mutators; assert sites at `transaction.rs:307`, `membership_index.rs:167` - -**Problem:** for a published crate this is the `missing_errors_doc` / `missing_panics_doc` surface clippy would flag under `--all-targets` pedantic. - -**Direction of fix:** a shared "Errors & poisoning" doc convention plus targeted `# Errors`/`# Panics` sections. - -**Fixed (2026-06-21):** the public API is entirely the `Chisel` impl in `src/lib.rs` (the storage internals are `pub(crate)` per I35), so the pedantic lint flags **only** that surface — `cargo clippy -p chisel --lib -W clippy::missing_errors_doc` enumerated exactly **33** `# Errors` gaps there and **0** `# Panics` (the public methods delegate to `pub(crate)` internals, so the assert sites the deepdive cites are not on the published surface). Added the shared convention as a `# Errors and poisoning` section on the `Chisel` type doc — every method's `# Errors` lists only its *operational* errors; `Poisoned` and fatal I/O/corruption are universal and documented once — then a concise `# Errors` to all 33 methods (the three savepoint methods, previously undocumented, also got summaries). Made it self-enforcing: `#![warn(clippy::missing_errors_doc)]` at the crate root, which CI's `-D warnings` promotes to a hard error, so a future public `-> Result` method without `# Errors` fails the build. Verified with `cargo doc -p chisel -D warnings` (anchors/intra-doc links resolve) + the full green gate. `# Panics` left untouched (none needed on the public surface; the brittle `missing_panics_doc` lint was deliberately NOT enabled). Done AFTER the I120 reshape so the documented signatures are final, and AFTER I125 so the `lookup_live`/`InvalidHandle` contract the docs reference is in place. - -### Type system - -#### I125. Three different deleted-handle semantics for the same "look up a live handle" [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/transaction.rs:1563-1602` (`read`, `client_byte`, `tag`); `:2014` (`delete_tagged`) - -**Problem:** `read()` and `client_byte()` reject a tombstone with `InvalidHandle`; `tag()` reads through it; `delete_tagged()` reads `.tag` off the looked-up entry with no Deleted guard. The "a handle is live" invariant lives in scattered per-method guards, so a fourth caller can easily get it wrong. - -**Direction of fix:** one `lookup_live(handle) -> Result` applying the Deleted ⇒ `InvalidHandle` rule once, used by all four sites. - -**Fixed (2026-06-21):** the deepdive premise was already **stale** — every site (`read`, `tag`, `client_byte`, `set_client_byte`, `update`, `delete_tagged`) funnels through `handle_table::lookup`, which collapses `Deleted → None`, then does `.ok_or(InvalidHandle)`, so the *behavior* was already uniform (no semantic change). What remained was the duplication the issue warns about: the `lookup(...)?.ok_or(InvalidHandle(handle))?` incantation was copy-pasted six times, plus two now-**dead** post-lookup guards (`client_byte_inner`'s `match Deleted` arm and `set_client_byte_inner`'s `if matches!(Deleted)` could never fire). Introduced `TransactionManager::lookup_live(&self, handle) -> Result` (read-view root selection + the single `ok_or(InvalidHandle)`), routed all six sites through it, and deleted both dead guards. `read_inner` keeps a Live/Overflow dispatch `match` whose `Deleted` arm is now a documented non-panicking backstop (exhaustiveness only). `handle_live_page_id` deliberately stays on its own Option-returning lookup (absent ≠ error). Pinned by a new `tests/tag_ops.rs::deleted_handle_is_invalid_across_all_entry_points` characterization test (green before and after — confirms the refactor preserved behavior); full Rust + Python suites + clippy `--workspace --all-targets` green. - -#### I126. Tag/client-byte `0` is overloaded as the "untagged"/"unset" sentinel [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 (tag half) -**Where:** `src/lib.rs` tag surface; threaded through the membership layer - -**Problem:** `0` doubles as a valid value and the "no tag" sentinel for both tag (`u32`) and client byte (`u8`); `allocate_tagged(value, tag: u32)` can be called with the sentinel by mistake and "membership index not updated for tag 0" is an implicit rule. - -**Direction of fix:** `Option` at the tag API boundary — "no tag" becomes expressible in the type (composes with the I120 newtype). - -**Fixed (2026-06-21):** `Tag(NonZeroU32)` — `Tag(0)` is unconstructable, so "untagged" can only be expressed by calling `allocate` (not `allocate_tagged`). `tag()` returns `Option` (stored `0 → None`). Python: `tag()` returns `int | None`; passing `tag=0` to a tagged op raises `ValueError` at the binding edge (`require_tag` helper). The on-disk/engine representation is unchanged (`u32` with `0`=untagged; the two `if tag != 0` membership guards stay). **Client-byte half NOT changed** — `0` there is a genuine default, not a functional sentinel (no membership coupling), so it stays a plain `u8`; this entry's resolution covers only the tag half, which was the stated direction of fix. - -### Performance - -#### I127. `current_live_slots` / `committed_live_slots` / `Savepoint.live_slots` use std `HashMap` (SipHash) on the per-op path [deepdive 2026-06-21, follow-up to I77] — **P2** ✅ FIXED 2026-06-21 -**Where:** `src/transaction.rs:144` (field decls ~144/216); probed/updated at `insert_into_data_page:2555/2591`, `release_data_slot:2463` - -**Problem:** these slot-accounting maps are probed and updated on every insert/update/delete. Keys are trusted local `u64` page ids — the identical threat model to the page-cache/LRU maps the I77 pass already moved to `FxHashMap` with an explicit "no DoS surface, SipHash is pure cost" rationale. I77 simply missed these. - -**Direction of fix:** `rustc_hash::FxHashMap` for all three; zero-risk, same free win. (Verified-clean by the same review: LRU is O(1), eviction short-circuits on `dirty_count`, radix descent is allocation-free, XXH3 re-stamps once per touched page — no other perf gaps on the hot path.) - -**Fixed (2026-06-21):** swapped all three maps to `rustc_hash::FxHashMap` in `src/transaction.rs` (field types + `HashMap::new()` → `FxHashMap::default()`; import `std::collections::HashMap` → `rustc_hash::FxHashMap`). `rustc_hash` was already a root dep (I77). Drop-in, no callers changed; full suite green. - -### Docs vs reality - -#### I128. ADR-7 + `ARCHITECTURE.md:580` describe per-page version read-dispatch as live, but no code reads the byte [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** ADR-7 (`.codebase-memory/adr.md`); `ARCHITECTURE.md:580`; `PageCache::load_page` - -**Problem:** both claim "reads dispatch on the per-page version byte" / "the page-cache load path validates the per-page version on every miss." Every page type *writes* `current_version(type)` at init, but `load_page` validates only the XXH3 checksum — there is no version read or dispatch anywhere. The mechanism is reserved-and-stamped, not implemented (consistent with the ADR's own "I31 eager upgrader (deferred)" note; the consequence bullets contradict it). ADR-7 was partially corrected 2026-06-21 — verify the corrected text and `ARCHITECTURE.md:580` both read reserved/future tense. - -**Direction of fix:** reword the live-dispatch claims to reserved/future tense in both ADR-7 and ARCHITECTURE.md. See also I31's "Phase 1a" note (the read helpers exist but are dormant — no production caller). - -**Fixed (2026-06-21):** reworded the ARCHITECTURE.md Page-level versioning bullet — "reads dispatch on the version byte" → "reads *will* dispatch … the decode helpers and `page::page_format_version()` exist but are **dormant today** — `load_page` validates only the XXH3 checksum and nothing reads the version byte yet." The page-header diagram footnote was already accurate (decode-only, cites the spec). ADR-7's text was already corrected during the 2026-06-21 versioning work and stays MCP-only per I129. - -#### I129. The ADR is not git-tracked — a fresh clone has no ADR and `// see ADR-N` comments resolve to nothing [deepdive 2026-06-21] — **P2** ✅ RESOLVED 2026-06-21 (ADR kept MCP-only; dangling comments reworded) -**Where:** `.codebase-memory/adr.md` (the codebase-memory MCP's local store, untracked); the many `// see ARCHITECTURE.md ADR-N` code comments - -**Problem:** the ADR-0..14 record lives only in the untracked MCP store; ARCHITECTURE.md has no ADR sections, so every code comment citing an ADR number is a dangling reference for anyone who clones the repo. This is a process/maintainer decision, not a code fix. - -**Direction of fix:** commit the ADR into the repo as `docs/ADR.md` (and either re-point the `// see …` comments at it or keep them once the file exists). Edit the MCP store via `manage_adr mode=update` with the FULL content — the `sections` param wipes the whole store (recorded footgun). **Open question for the maintainer:** is the ADR meant to be a repo artifact? If yes, this is the migration; if it stays MCP-only, the code comments should stop citing ADR numbers. - -**Resolved (2026-06-21, maintainer decision = keep ADR MCP-only):** the ADR stays in the codebase-memory store (NOT committed as `docs/ADR.md`). The dangling `// … ADR-N` code comments were reworded to stop citing numbers that don't resolve in a clone: `src/page.rs` (ADR-7 → "per-type format evolution; see the per-page-versioning spec") and `python/src/db.rs` ×2 (dropped "ADR-5"; point at ARCHITECTURE.md). `grep -rE 'ADR-[0-9]' src/ python/src/` is now clean of dangling repo refs. - -#### I130. `ARCHITECTURE.md` cites functions deleted in the #46 dead-code sweep [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `ARCHITECTURE.md:246` (`page::page_format_version()`); `:108` and `:471` (`allocate_near`, the latter claiming it's "currently used by data-page allocation" — already false before deletion) - -**Problem:** the #46 sweep updated code comments but not ARCHITECTURE.md, leaving three references to deleted functions. (Note: `page::page_format_version()` was *re-introduced* as a dormant read helper by the 2026-06-21 versioning work per I31's Phase 1a — confirm whether the `:246` reference is now accurate or still describes the deleted signature.) - -**Direction of fix:** reconcile the three references against current code — delete the `allocate_near` ones; correct or keep `page_format_version` depending on its current dormant-but-present status. - -**Fixed (2026-06-21):** removed the `allocate_near` references (deleted in PR #46): the freemap responsibility-table row (now `allocate_first` / `mark_free`) and the detail paragraph (now notes the radius-scan variant was removed in #46). The `page::page_format_version()` ref is NOT stale — the function was re-introduced as a dormant read helper by the I31 work, and the surrounding text already describes it as decode-only (the dispatch-overclaim is handled by I128). - -#### I131. `README.md:14` says commit is "two-phase" but the protocol does three fsyncs [deepdive 2026-06-21, follow-up to I63] — **P3** ✅ FIXED 2026-06-21 -**Where:** `README.md:14`; the `commit()` rustdoc (numbers only two fsyncs) - -**Problem:** I63 fixed the *docstring's* "two fsyncs" claim, but the README's "two-phase durability (fsync data pages, then fsync superblock)" and the rustdoc step-numbering were missed. The protocol does THREE fsyncs (pre-drain I28 flush `transaction.rs:971`, step-1 data flush `:989`, superblock `:1026`); ARCHITECTURE.md gets it right ("3 fsyncs", with a `== 3` regression test). - -**Direction of fix:** README → three-fsync; renumber the `commit()` rustdoc to give the pre-drain a step number. - -**Fixed (2026-06-21):** README.md:14 → "shadow-paging durability (all data pages fsync'd before the superblock fsync; three fsyncs per commit)"; the `commit()` rustdoc gained a "Step 0 — pre-drain (I28)" entry documenting the third fsync, without renumbering the load-bearing steps 1–5. - -#### I132. `ARCHITECTURE.md` says `bench/` is "not a workspace member" — it is [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `ARCHITECTURE.md:588`, `:619`, `:668` (the last draws a now-false build-topology inference) - -**Problem:** `Cargo.toml:20` is `members = [".", "python", "bench"]`; bench is merely excluded from `default-members`. README.md:72 is correct; ARCHITECTURE contradicts it three times. (Post-I61 state.) - -**Direction of fix:** "a workspace member excluded from `default-members`." - -**Fixed (2026-06-21):** corrected all three ARCHITECTURE.md claims (lines had drifted to :591/:622/:671) — the bench-infrastructure intro now says bench IS a workspace member (in `members` + `default-members`, so a root `cargo test` runs it); the PR-2 history note is "a sibling subcrate at this point (I61 later made it a member)"; the lessons-learned item is past-tensed and marked RESOLVED via I58/I61. (Note: bench is in `default-members`; `python` is the member excluded from it.) - -#### I133. `ARCHITECTURE.md` "common 16-byte header" vs `COMMON_HEADER_SIZE = 12` [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 (briefly deferred; a layout workflow proved there's no collision — the constant was just stale) -**Where:** `ARCHITECTURE.md:232`, `:235`; `src/page.rs:41` - -**Problem:** the 16 figure is actually `DATA_PAGE_HEADER_SIZE` (the data-only extended header); the named common-header constant is 12. - -**Direction of fix:** reconcile the number against the constant name. - -**Deferred (2026-06-21) — NOT a doc edit:** investigating this surfaced a genuine *code-constant* inconsistency, so "change ARCHITECTURE's 16 → 12" would create a new error rather than fix one. `COMMON_HEADER_SIZE = 12` (page.rs:41) is the outlier: the I31 reserved region is bytes 8..16 (`COMMON_RESERVED_OFFSET = 8`, `COMMON_RESERVED_LEN = 8`), the page.rs:60 comment calls bytes 8..16 "common-header fields", and ARCHITECTURE.md:232 ("16-byte common header = 0..8 type-specific + 8..16 reserved") is internally consistent with **16** — yet the named constant says 12, and it carries `#[allow(dead_code)]` (no caller, so the contradiction never had to resolve). On a data page (`DATA_PAGE_HEADER_SIZE = 16`) the reserved 8..16 also overlaps the 12..16 slot-metadata region. **Maintainer decision needed:** the canonical common-header size (bump `COMMON_HEADER_SIZE` → 16? shrink `COMMON_RESERVED_LEN` to fit 12?) and whether the reserved region collides with data-page slot metadata. No doc edit is correct until that's decided. - -**Fixed (2026-06-21, after a 10-agent layout workflow — high confidence):** the deferral's collision worry was **wrong** — correcting it: a Data page's slot directory starts at byte **16** (`free_start` inits to `DATA_PAGE_HEADER_SIZE = 16`, data_page.rs:103-104), so bytes 12..16 are part of the *reserved* region, **not** slot metadata. The workflow mapped bytes 0..16 of all six non-superblock page types and ran three adversarial verifiers against "bytes 8..16 are reserved on every non-superblock page" — **0/3 could refute it**; none store load-bearing data in 8..16. (The superblock *does* use 8..16 for `txn_counter`, but it is explicitly exempt from the common header.) So `COMMON_HEADER_SIZE = 12` was simply a stale value predating the I31 8..16 reservation; being dead code (`#[allow(dead_code)]`, no caller), the contradiction never surfaced. **Fix:** bumped `COMMON_HEADER_SIZE` → **16** (= `COMMON_RESERVED_OFFSET + COMMON_RESERVED_LEN` = `DATA_PAGE_HEADER_SIZE`); rejected the "shrink `COMMON_RESERVED_LEN` to 4" alternative (it would silently un-reserve bytes 12..16 that the comments + `compact()` save/restore treat as reserved). Added compile-time `const _: () = assert!(…)` checks in `page.rs` locking the three constants together, plus a `data_page.rs` test that the reserved 8..16 stays zero across init+insert. Reworded the stale `overflow.rs:21` "(= 12)" comment; `ARCHITECTURE.md:232` was already correct ("16-byte common header"). - -### Idiomaticity - -#### I134. The COW page copy bounces through a named 8 KB stack array for the borrow checker [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/handle_table.rs:330-333`; `src/membership_index.rs:205-208`, `:290-292` - -**Problem:** three sites copy a page through a named stack buffer purely to satisfy the borrow checker; the pattern is unnamed and duplicated. - -**Direction of fix:** a shared `cow_copy_page(cache, src, dst)` helper using `copy_from_slice`. - -**Fixed (2026-06-21):** added `PageCache::copy_page(src, dst)` and routed all **five** sites through it (the review counted three; there were two more in membership_index — `:316-319` and the leaf-COW at `:290-293`, plus the second handle_table interior site). The stack bounce now lives in one place. The borrow-checker reason (can't `get(src)` and `get_mut(dst)` at once) is documented on the helper. - -#### I135. `u64::from_le_bytes(buf[off..off+8].try_into().unwrap())` repeated ~15× [deepdive 2026-06-21] — **P3** -**Where:** across `src/handle_table.rs` and `src/superblock.rs::deserialize` - -**Problem:** the load-bearing infallibility of each `unwrap()` is undocumented at every site. - -**Direction of fix:** a tiny `read_u64_le(buf, off)` helper that names the invariant once. - -#### I136. Three hand-copied clean-victim eviction loops with divergent short-circuits [deepdive 2026-06-21, carried] — **P3** ✅ FIXED 2026-06-21 -**Where:** `src/page_cache.rs` — `maybe_evict` Phase A, `set_cache_max_bytes`, `flush` Phase 1b (the last omits the `dirty_count` early-out) - -**Problem:** the three loops are subtly divergent; carried from the prior review. Relates to the `dirty_count` underflow trend (I116). - -**Direction of fix:** extract `evict_clean_to_cap` and consolidate (do alongside I116's `dirty_count` helper). - -**Fixed (2026-06-21):** extracted `PageCache::evict_clean_to_cap()` and routed all three sites through it (the review's "three loops" was accurate this time — confirmed by reading each). `maybe_evict` Phase A and `set_cache_max_bytes` were character-identical; `flush` Phase 1b was the same loop minus the `dirty_count == entries.len()` early-out — now unified, which adds that early-out to Phase 1b harmlessly (Phase 1a has already cleared every dirty flag there, so it never fires). Behavior-preserving; full suite green. - -### Python binding (PyO3) - -#### I137. `db.rs` / `transaction.rs` file-header docs still describe the engine as `RefCell>` — it's a `Mutex` since I75 [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `python/src/db.rs:2-29` (header) vs the actual field at `:76`; `python/src/transaction.rs:44` (same stale claim) - -**Problem:** the file headers argue "why RefCell (and not Mutex)", but I75 migrated the field to `Mutex>` (the struct comment 30 lines below correctly documents the Mutex and contradicts the header above it). A first-time reader gets the wrong concurrency model and the wrong I21 failure mode (RefCell *panics* on re-entry; the Mutex *deadlocks*). - -**Direction of fix:** rewrite both file headers to describe the Mutex model and the deadlock-not-panic re-entry semantics. - -**Fixed (2026-06-21):** rewrote the `python/src/db.rs` file header — `Mutex>` (not RefCell), "Why Mutex (and not RefCell) — I75" (PyO3 0.24+ Sync requirement), and the re-entry hazard now correctly says **deadlock** (non-reentrant `std::sync::Mutex`), not panic. Now consistent with the already-correct `inner`-field comment 30 lines below. Also fixed `python/src/transaction.rs:44` (`RefCell` → `Mutex`, and its stale "PoisonedError" → `ClosedError` per I25). - -#### I138. `errors.rs` catchall routes any future variant to the abstract base, bypassing the Operational/Fatal split [deepdive 2026-06-21] — **P2** ✅ FIXED 2026-06-21 -**Where:** `python/src/errors.rs:326` (the `_` arm); the "exhaustive, no silent fallback" header claim at `:11-13` - -**Problem:** the same fail-open class as engine I104, on the binding side. A new *fatal* engine variant would surface in Python as a bare `ChiselError`, so `except chisel.FatalError:` poison-recovery handlers silently miss it — breaking the documented `FatalError` contract (`errors.rs:5-7`). The header still claims the match is exhaustive with "no silent fallback", which `#[non_exhaustive]` (I36) made false. - -**Direction of fix:** branch the catchall through the Operational/Fatal base class instead of the abstract `ChiselError`; correct the header claim. - -**Fixed (2026-06-21):** `to_py_err` now captures `let fatal = err.is_fatal();` before the match consumes `err`, and the `_` catchall routes `if fatal { FatalError } else { OperationalError }` instead of the abstract `ChiselError` base — so a future fatal variant stays catchable via `except chisel.FatalError:`. The stale header claim that the match is compile-time exhaustive (false since I36 added the `_` arm) was rewritten to describe the tiered fallback and to point at the engine-side I104 test as the guard for the classification this fallback relies on. Dormant today (all 24 current variants hit concrete arms); `cargo clippy --workspace` compiles the binding clean and the Python matrix stays green. - -#### I139. The typed-exception contract is largely unverified [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `python/tests/` (`test_operational_hierarchy`, `test_threading.py`); `python/src/errors.rs::to_py_err` - -**Problem:** routing is correct today, but `SavepointNotFoundError`, `DuplicateSavepointError`, `CacheFullError`, `SpillwayFullError`, `RootNameTableFullError`, `InvalidRootNameError` are never raised in any test, and `test_operational_hierarchy` omits `TagMismatch`/`SpillwayFull`/`TransactionInProgress` — a swapped `to_py_err` arm ships green. `test_threading.py` joins before re-touching, so the Mutex deadlock/poison path (the whole reason for I75's Mutex-over-RefCell decision) is unexercised. - -**Direction of fix:** a parametrized "trigger each variant, assert the concrete class" test, plus a two-thread contention test. - -**Fixed (2026-06-21, p3-cleanup batch):** new `python/tests/test_exception_contract.py` (13 tests) triggers all 8 previously-unverified-or-omitted variants and asserts the EXACT typed class plus its `Operational`/`Fatal` base: `SavepointNotFoundError`, `DuplicateSavepointError`, `InvalidRootNameError` (empty/>24-byte/NUL), `RootNameTableFullError` (fill all 8 slots), `TagMismatchError`, `TransactionInProgressError` (config mutator mid-txn), `CacheFullError` (16-page cache + spillway disabled), `SpillwayFullError` (small cache + small spillway). A two-thread contention test runs concurrent begin/allocate/commit cycles on one shared `Chisel`, asserting no deadlock, all handles readable, and not poisoned. (The Mutex never enters Rust's poisoned state because fatal conditions return `Err(Poisoned)` rather than panicking under the lock — so the "poison propagation across threads" path is N/A by construction, noted in the test.) 113/113 Python tests pass. - -### Cargo hygiene - -#### I140. `python/Cargo.toml:6` carries a stale "PyO3 0.22" comment — the dep is 0.29 [deepdive 2026-06-21] — **P3** ✅ FIXED 2026-06-21 -**Where:** `python/Cargo.toml:6` - -**Problem:** cosmetic — the dependency is `pyo3 = "0.29"` (I103) but the comment still says 0.22. The only cargo-hygiene finding; every declared dependency is otherwise used and MSRV 1.82 holds against every floor. - -**Direction of fix:** update or delete the comment. - -**Fixed (2026-06-21):** dropped the stale "PyO3 0.22 … only needs 1.63" specifics for "PyO3 has a lower MSRV of its own, but the path-dep on the root crate makes its 1.82 floor ours too" — version-agnostic, so it can't drift again. - -#### I141. `transaction.rs` god-module — decomposed through the unit extractions; the final StagingTxn extraction deliberately deferred [deepdive 2026-06-22] — **P3** ⏸ DEFERRED 2026-06-22 -**Where:** `src/transaction/` (the directory module split out of the former `transaction.rs`) - -**Problem:** The 2026-06-22 review tagged `transaction.rs` (~2.6k prod lines) a god-module (filed DESIGN, verifier-adjusted to **SMELL**): every durability invariant (3-fsync ordering, I18 freemap window, BUG#2 atomic staging, R1 cursor accounting, watermark rollback) encoded as prose comments + cross-references rather than types/module boundaries, with test-only `Cell` flags on the production struct. - -**Disposition (decomposed, then deliberately stopped):** Worked incrementally — a directory-module split by concern plus four extracted units: `SlotPacker` (R1 packing, #77), `FreemapRecycle` (structural recycle + persist/reclaim, #78), `CommitProtocol` (the 3-fsync sequence, #79), and a `#[cfg(test)] FaultInjector` (#76). All behavior-preserving (the existing suite is the oracle; FreemapRecycle additionally passed a 4-lens adversarial review). The planned final unit — `StagingTxn` (the BUG#2 atomic `allocate_inner` prepare/install) — was **deliberately NOT extracted**: the candidate-prepare/install vocabulary (`handle_table_insert_candidate`, `membership_insert_candidate`, `membership_remove_candidate`, `abort_allocate_prepare`, `inject_membership_failure`, `insert_into_data_page`) is **shared across `allocate_inner` (staging.rs) AND `update_inner`/`delete_inner` (mutate.rs)**, so a context-based extraction cannot be contained to staging.rs — it would force the most delicate mutation paths + packing.rs through a mechanical wrapper change for little cohesion gain. `staging.rs` is already a focused, holdable-in-context concern file. - -**Future work (incremental only):** if/when the staging paths are touched for a real reason, fold the shared vocabulary into a `StagingTxn` unit at that point. Do NOT re-trigger a standalone extraction from the SMELL alone — the engine passes every test; on a green, pre-production engine a below-bug structural finding is the lowest-value, highest-risk change (this whole decomposition was a large undertaking driven by one SMELL). The remaining `transaction.rs` SMELL items below the extraction (the test-only flags) are already addressed by the `FaultInjector` split. - ---- - -## On-disk encryption - -Source: **[encryption 2026-06-29]** — deferred work captured while implementing the on-disk encryption feature (design at `docs/specs/2026-06-29-on-disk-encryption-design.md`). - -#### I142. Bulk DEK rotation (full re-encryption under a new DEK) is not implemented — **P3** ⏸ DEFERRED 2026-06-29 - -**Where:** `src/lib.rs`, `src/transaction/keys.rs`, `src/crypto/mod.rs` - -**Problem:** There are two distinct "key rotation" operations with very different costs: - -1. **Credential rotation** (KEK re-wrap): derives a new KEK from a new passphrase or raw key and re-wraps the existing DEK into a key slot. This is O(1) — it touches only the superblock — and is fully implemented via `add_key` / `rotate_key` / `remove_key`. - -2. **Bulk DEK rotation** (full re-encryption): generates a fresh DEK, re-encrypts every data page under the new DEK, and replaces the wrapped DEK in all active key slots. This is O(total\_pages) — it must rewrite the entire database file — and is **not yet implemented**. - -Bulk DEK rotation is only needed when the DEK itself is believed compromised (e.g., a process memory dump exposed the in-session DEK). Credential rotation — the far more common operational need (password change, key rollover, adding a second credential) — is already available and O(1). Because no production databases exist today and the DEK is not separately distributed, the risk of DEK compromise is low; the heavy whole-file cost makes this a poor default rotation path. - -**Direction of fix (when needed):** Implement a `rekey(old_key, new_key)` or `rekey_dek(key)` API that (1) generates a fresh DEK, (2) reads, decrypts, re-encrypts, and writes back every non-superblock page in a single pass using the existing stride-aware `PageIo`, (3) re-wraps the new DEK into all currently-active key slots, and (4) commits an updated superblock. The operation must be crash-safe: either complete or leave the original file intact. A copy-then-atomic-rename strategy (write the new file alongside, then rename) is the simplest crash-safe approach for an embedded store; an in-place two-pass strategy is also possible but more complex. Reuse `PageCipher::seal` / `PageCipher::open` and `CryptoHeader` from the existing crypto layer. - ---- - -## Deep review 2026-07-02 - -Source: **[deepdive 2026-07-02]** — fresh-eyes Rust review at `581be36`; full report at `docs/reviews/review-20260702-001902.md`. Every BUG/DESIGN finding here was confirmed by an independent adversarial verification pass (0 refuted). The four BUGs (I143–I146) are fixed in PR #89; the DESIGN/SMELL items are recorded for triage. Delta vs the prior two reviews: **0 regressions, 71 findings verified resolved.** - -#### I143. TOCTOU create-vs-open race: create decision made before the flock [deepdive 2026-07-02] — **P1** ✅ FIXED 2026-07-02 (PR #89) -**Where:** `src/lib.rs:366` - -**Problem:** Chisel::open decides create-vs-open BEFORE the flock is taken: `file_exists` is computed at lines 366-369, the exclusive lock is only acquired inside `PageIo::open` at line 375, and the stale boolean then selects `TransactionManager::create_new` at line 389. The doc comment (lines 339-341) claims the lock is acquired 'before any parsing... rather than racing on the superblock', but the create/open decision itself races. Cross-process TOCTOU with silent data loss: process B stats a nonexistent/zero-length file; process A concurrently creates the DB, commits data, and closes (releasing the flock); B then acquires the lock and runs create_new on the now-populated file, writing a fresh superblock over A's committed database. Multi-process exclusion is an explicit part of the public contract (flock, LockFailed), so this is exactly the race the lock exists to prevent. - -**Direction of fix:** Re-check the file length after the flock is held (e.g. have PageIo::open return whether it created the file, or re-stat via the locked fd) and choose create_new vs open_existing from that post-lock observation. - -**Fixed (2026-07-02, PR #89):** see the commit; a regression test is included where feasible (I144 forges the on-disk stride and asserts a typed error, not a panic). - -#### I144. Unvalidated crypto-header `stride` → div-by-zero panic / huge-alloc DoS on open [deepdive 2026-07-02] — **P1** ✅ FIXED 2026-07-02 (PR #89) -**Where:** `src/transaction/recovery.rs:258` - -**Problem:** open_existing takes the crypto-header's stride field verbatim (`sb.encryption.map(|h| h.stride as usize)`) and calls `cache.io_mut().set_stride(stride)` with no validation. `PageIo::set_stride` (src/page_io.rs:226) computes `len / stride as u64`. The header comment in src/superblock/crypto_header.rs:11 claims stride is "validated by the engine"; no such check exists anywhere (only the two hardcoded ENC_PAGE_SIZE fallback sites are safe). stride is a plaintext field protected only by the forgeable XXH3 page checksum — the codebase itself established this trust boundary when it bounds-checked ct_len in decrypt_body for exactly this reason. A forged stride of 0 is a guaranteed division-by-zero panic at open (DoS, violates the poison-not-panic error model); a huge stride drives multi-GiB read-buffer allocations. Also a first-class comment-vs-code mismatch. - -**Direction of fix:** Before set_stride, require header.stride == ENC_PAGE_SIZE as u32 (the only value ever written) and treat a mismatch like an unsupported format / corrupt slot; that also makes the crypto_header.rs comment true. - -**Fixed (2026-07-02, PR #89):** see the commit; a regression test is included where feasible (I144 forges the on-disk stride and asserts a typed error, not a panic). - -#### I145. `reclaim_freemap_orphans` not poison-wrapped — fatal mid-sweep leaves a usable, indeterminate manager [deepdive 2026-07-02] — **P2** ✅ FIXED 2026-07-02 (PR #89) -**Where:** `src/transaction/freemap.rs:663` - -**Problem:** Minor imprecision only: the failure requires the fatal error to strike mid-sweep (reachable_pages, a live-page cache.get, or mark_free_growing); the CorruptPage/ChecksumMismatch skip on dead pages at freemap.rs:457-458 is intentionally non-poisoning and not part of the bug. The BUG severity and substance are correct as stated. A fatal `IoError`/`CorruptPage` from `reachable_pages`, `cache.get`, or `mark_free_committed_path` during the sweep returns to the caller with the manager NOT poisoned. Worse, `mark_free_committed_path` writes the partially-advanced tree root back into `current_roots` even on error (put_tree on the error path, freemap.rs:291-293), so the un-poisoned manager holds a freemap tree in an indeterminate mid-mutation state that a subsequent `commit()` will happily make durable — exactly the class of state the I1 poison model exists to fence off. - -**Direction of fix:** Wrap the sweep at the TransactionManager boundary the same way as everything else: `let result = self.freemap.reclaim_orphans(...); self.poison_on_fatal(result)` inside `reclaim_freemap_orphans` (freemap.rs:663-673). - -**Fixed (2026-07-02, PR #89):** see the commit; a regression test is included where feasible (I144 forges the on-disk stride and asserts a typed error, not a panic). - -#### I146. README says the newer-minor write-refusal gate is unwired; it shipped as I29 [deepdive 2026-07-02] — **P2** ✅ FIXED 2026-07-02 (PR #89) -**Where:** `README.md:363` - -**Problem:** README.md:363 falsely states the MINOR write-refusal gate is "not yet wired up"; it shipped as the I29 write-gate in src/transaction/recovery.rs:409-411 (force_read_only on file-MINOR > binary-MINOR, mutations return ReadOnlyMode), consistent with THEORY.md:132 (repo root). Only the README sentence needs updating. This is the user-facing compatibility contract, in a README refreshed three commits ago (#86). A user reading README concludes that opening a newer-minor file risks clobbering unknown fields on write; in reality mutations return ReadOnlyMode. The two design docs disagree with each other on a safety guarantee. - -**Direction of fix:** Update README's format-compatibility section to say the read-only write-gate is implemented (matching THEORY.md:132 and recovery.rs:409); drop the "lands with the first post-1.0 minor bump" sentence. - -**Fixed (2026-07-02, PR #89):** see the commit; a regression test is included where feasible (I144 forges the on-disk stride and asserts a typed error, not a panic). - -#### I147. HandleTable::insert grow loop unbounded for handle==u64::MAX (latent; unreachable via monotonic handles) [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `src/handle_table.rs:253` - -**Problem:** HandleTable::insert's grow loop (src/handle_table.rs:253) is unbounded: for handle == u64::MAX, capacity() saturates to u64::MAX at depth 6 and the loop allocates one page per iteration forever, and delete (line 289) reports u64::MAX absent at saturated capacity while find_leaf (lines 694-697) explicitly supports it. Real code-level defect and internally inconsistent with the module's own MAX_DEPTH comment and read path, but unreachable through the engine (handles are monotonic from 1), so latent — fix by bounding the loop with `&& self.depth < MAX_DEPTH` as FreeMapTree::mark_free_growing does. Runaway loop with unbounded page allocation on a key the module's own comments and read path explicitly claim to support. Practically unreachable through the engine today (handles are monotonic from `next_handle` starting at 1, so u64::MAX needs ~1.8e19 allocations), but it is a live in-crate API and the code around it (find_leaf's u64::MAX carve-out, the MAX_DEPTH comment) asserts the case is handled when it is not. - -**Direction of fix:** Mirror freemap_tree: bound the loop (`while handle >= self.capacity() && self.depth < MAX_DEPTH`) and/or make `grow` a no-op at MAX_DEPTH; give `delete` the same `cap != u64::MAX` exemption `find_leaf` has. - -#### I148. `free_subtree` / membership descent has no positional type check — corrupt-but-checksummed page amplifies corruption [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/membership_index.rs:84` - -**Problem:** As stated, with one softening: the remove()-path scenario where leaf values (1) are freed as page ids usually aborts earlier with a misleading ChecksumMismatch (inner.delete/any_present descend the same bogus depth before free_subtree runs), so the sharpest unmitigated instance is a corrupt interior child pointer at the depth-1 boundary of free_subtree, which is pushed into `freed` without any read or validation. Inconsistent hardening across the crate's three radixes against the exact threat model the repo tests everywhere (checksum-valid, structurally wrong pages): freemap_tree, overflow, and data_page fail closed with typed CorruptPage; handle_table and membership_index fail open. For `free_subtree` the failure mode is corruption amplification — bogus ids (potentially live data pages or even superblock slot 1) enter `txn_freed_pages` and get marked reusable at commit. - -**Direction of fix:** Add the freemap_tree-style positional type check (`buf[0]` vs expected MembershipLeaf/MembershipInterior/HandleTable+FLAG) on every descent read in find_leaf/insert_recursive/delete_recursive/iter/free_subtree, returning CorruptPage on mismatch. - -#### I149. `total_length` trusted unbounded before `Vec::with_capacity` — corrupt chain can OOM (contradicts I14) [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/overflow.rs:183` - -**Problem:** As stated, with one precision: the failing page cannot arise from any legitimate Chisel write or from the stale-handle scenario (a genuinely Overflow-typed page always carries a truthfully-written total_length, replicated per page). Reaching the panic requires on-disk corruption/tampering that also passes the XXH3 checksum. Within the module's own I14 standard — corrupt-but-checksummed chains surface as typed CorruptPage, never panic/abort — the gap is real: total_length is the one header field read() trusts unbounded before allocating, and a cheap bound (next_page_id * OVERFLOW_PAYLOAD, the same universe bound reclaim_orphans uses) would close it. The module's own hardening standard (I14: corrupt-but-checksummed chains must surface as typed CorruptPage, never a panic/abort) is contradicted for the one field it trusts most. A well-formed chain can never carry more bytes than the file holds, so a cheap plausibility bound exists and is already used elsewhere (reclaim_orphans uses cache.next_page_id()). - -**Direction of fix:** Bound total_length against a file-derived maximum (cache.next_page_id() * OVERFLOW_PAYLOAD → CorruptPage if exceeded), or clamp the initial with_capacity and let Vec grow as pages are validated. - -#### I150. Attacker-controlled Argon2 params consumed before the AEAD tag rejects — KDF-param OOM/DoS [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/superblock/crypto_header.rs:209` - -**Problem:** As stated, except the twin lives at src/transaction/recovery.rs:625-653 (derive_kek call at line 637), not src/superblock/recovery.rs. Everything else — params-before-auth ordering, u32::MAX m_cost accepted by argon2 0.5.3, re-stampable XXH3 checksum — checks out. Fix is a small read-time ceiling on the slot's Argon2 params (or clamp in derive_kek) before deriving. An attacker who edits a slot's m_cost/t_cost/p_cost and re-stamps the non-cryptographic XXH3 checksum turns every subsequent open into an OOM/allocation-abort (or minutes of grinding). Unauthenticated-KDF-param DoS is inherent to the construction, but an unbounded cost parameter converts "corrupt file fails to open" into "process dies". - -**Direction of fix:** Clamp attacker-controlled params with sanity ceilings before deriving (e.g. m_cost <= a few GiB, t_cost/p_cost small maxima); reject out-of-range slots the same way as an unknown kdf_id (skip/continue), so a tampered slot degrades to InvalidEncryptionKey. - -#### I151. Pack cursor not gated on packing_enabled; rollback restores a live cursor under an active savepoint → in-place write below the watermark [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/transaction/packing.rs:119` - -**Problem:** `SlotPacker::insert` consults `self.insert_cursor` unconditionally — the cursor branch is not gated on `packing_enabled`. `rollback_to_inner` (savepoints.rs:98-102) restores the packer snapshot taken at savepoint creation, which was captured BEFORE `clear_cursor()` ran (savepoints.rs:32-33), so rolling back to the first savepoint of a transaction that had packed inserts restores a live `Some(P)` cursor while that savepoint is still on the stack. This contradicts the module's stated invariant (packing.rs:27-28: "Packing is disabled entirely when savepoints are active") and the savepoint design premise that no in-place mutation happens below the savepoint watermark. After the restore, the next insert writes a slot into pre-watermark page P in place; a second `rollback_to` to the same savepoint truncates only >= watermark, so P keeps the physically-written slot forever — each rollback_to+insert cycle leaks one dead slot into a page that then gets committed (rolled-back value bytes become durable). No API-visible wrong reads, but the rollback mechanism's "nothing below the watermark changed" premise is silently violated. - -**Direction of fix:** Gate the cursor branch on `packing_enabled` (`if packing_enabled { if let Some(cursor) = self.insert_cursor { ... } }`) — one guard at the single decision point, which also makes the restore harmless — or have `restore()` force the cursor to None when a savepoint remains active. - -#### I152. I29 write-gate compares file MINOR against the plaintext constant unconditionally — a future (2,1) file opens writable [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/transaction/recovery.rs:409` - -**Problem:** The I29 write-gate compares the file's MINOR against the plaintext constant `page::FORMAT_MINOR_VERSION` (= 1) unconditionally, but src/page.rs:120-123 declares that the encrypted MAJOR=2 series "carries its own minor series, independent of plaintext" starting at FORMAT_MINOR_VERSION_ENCRYPTED = 0. A future encrypted file at (2,1) opened by today's binary (whose encrypted-series minor is 0) evaluates `1 > 1 == false` and stays WRITABLE — exactly the clobber-newer-fields hazard the gate exists to prevent. Latent today (no (2,≥1) files exist), but the gate silently fails for the second major series the moment its minor is first bumped. - -**Direction of fix:** Select the comparison constant by major series: gate against FORMAT_MINOR_VERSION_ENCRYPTED when the file's MAJOR is 2, FORMAT_MINOR_VERSION when 1. - -#### I153. Per-page temporal replay: page AAD is page_id only, so stale-but-valid pages can be spliced into a current DB [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `src/crypto/mod.rs:321` - -**Problem:** Accurate as written; one refinement: 're-authenticates forever' holds until a (currently deferred) full DEK rotation/re-encryption — KEK rotation does not invalidate old sealed images. An attacker with file access can splice stale versions of individual pages into a current database, producing a mixed state that never existed at any commit (stale freemap or tree pages that pass AEAD and then corrupt structure silently). The design spec's documented non-goal covers only substitution of a "wholly older, validly-signed database image"; per-page temporal splicing is strictly stronger and is not covered by the spec's "cryptographic tamper-detection" / anti-relocation claims. - -**Direction of fix:** At minimum, extend the spec §9 boundary to name per-page replay explicitly. If it should be defended, bind pages to a commit epoch in the AAD (costs rewriting reachable pages on epoch bump) or hash-chain page tags into the sealed superblock body. - -#### I154. Blanket CryptoError→InvalidEncryptionKey conflates bad Argon2Params on create/add_key with a wrong key [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `src/error.rs:397` - -**Problem:** Accurate as stated. Minor precision note: the add_key path (keys.rs:151) reaches InvalidEncryptionKey via an explicit `.map_err`, not the blanket From impl, but the effect (BadKeyLength/Kdf on the new credential reported as "does not match any key slot") is identical. Creating a brand-new encrypted DB with bad Argon2Params yields 'key does not match any key slot' — nonsensical on create where no slots exist yet; the same conflation hits add_key's wrap of the NEW credential (transaction/keys.rs:151 map_err → InvalidEncryptionKey). The comment above the From impl claims a CryptoError here 'is always a key-or-KDF problem on intact on-disk data', which is true, but collapsing 'your params are invalid' into 'your key is wrong' sends users debugging the wrong thing. - -**Direction of fix:** Validate `argon2_params` (and raw-key length) in open()/the setter alongside the existing superblock_count check, or map Kdf/BadKeyLength to a distinct operational variant (e.g. InvalidKeyParameters) instead of the blanket InvalidEncryptionKey. - -#### I155. Finished PyTransaction data ops don't check the `finished` guard — a stray op aliases into the next transaction [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `python/src/transaction.rs:135` - -**Problem:** PyTransaction's data operations (allocate, read, update, delete, delete_many, allocate_tagged, set_root_name, savepoint, etc., lines 135–227) do not check the `finished` guard; only commit()/rollback()/__exit__ do. A finished Transaction object still forwards every operation to the db. After `t1 = db.transaction(); t1.commit(); t2 = db.transaction()`, a stray `t1.allocate(b"x")` silently writes into t2's transaction — the exact 'called the wrong object' bug class that the I22/I24 AlreadyFinishedError guard was added to surface, applied to commit/rollback but not to the data ops. If no new transaction is active the engine at least raises NoActiveTransactionError, but the aliasing case is silent data misattribution. - -**Direction of fix:** Check `finished` at the top of each forwarding method and raise AlreadyFinishedError, consistent with the commit/rollback policy. - -#### I156. GIL held across commit fsyncs and Argon2id passphrase rotate_key — freezes all Python threads for the KDF/fsync duration [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `python/src/db.rs:652` - -**Problem:** Per-op engine calls (including commit's 3 fsyncs, defrag, and add_key/rotate_key/remove_key) hold the GIL for their full duration; only open() detaches (db.rs:276). With passphrase keys, rotate_key runs Argon2id (19 MiB, t=2) once or twice under the GIL — roughly 100-300 ms, not seconds — freezing all other Python threads. The GIL-hold is an explicitly documented design tradeoff (db.rs:634-651 names the consequence and both fix paths), but that comment predates the encryption work (2026-06-20 vs PR #85), so the memory-hard KDF case was never weighed; the comment should at least be updated to name key ops alongside commit/defrag. A passphrase rotate_key or a large commit freezes every other Python thread in the process for the full KDF/fsync duration — seconds-scale for Argon2id. The encryption work postdates the 'per-op calls hold the GIL' comment, which was reasoned about quick engine ops, not a deliberately slow memory-hard KDF. - -**Direction of fix:** Move the mutex acquisition inside py.detach (detach → lock → engine call → unlock → reattach); the closure only needs Chisel: Send, which open() already proves. Do NOT detach while holding the guard — that inverts the GIL/Mutex order and can deadlock. At minimum, detach around the three key-management methods. - -#### I157. redb-strict skips F_FULLFSYNC on macOS — the PR-8 fairness fix was applied to SQLite but not redb [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `bench/src/redb_engine.rs:93` - -**Problem:** DurabilityMode::Strict maps to redb Durability::Immediate, which on macOS commits via File::sync_data() (redb-2.6.3 unix.rs backend); Rust std's sync_data on Darwin issues fcntl(F_BARRIERFSYNC) (and even a plain fdatasync would not flush the disk write cache). Meanwhile chisel-strict commits via sync_all/F_FULLFSYNC (src/page_io.rs:445-448,480) and sqlite-strict is deliberately handicapped to parity with PRAGMA fullfsync=ON (bench/src/sqlite_engine.rs:50-62), whose comment explicitly names the ~3-orders-of-magnitude macOS artifact this causes. The PR-8 fairness fix was applied to SQLite but not redb, so on macOS (the dev machine this repo runs benches on) redb-strict skips the F_FULLFSYNC cost that both other strict engines pay. Cross-engine strict comparisons on macOS systematically flatter redb — exactly the measurement artifact the sqlite comment says the harness exists to prevent. - -**Direction of fix:** redb exposes no full-fsync knob, so either document the asymmetry in the DurabilityMode::Strict doc and the summary renderer (footnote redb-strict on macOS), or treat macOS redb-strict numbers as non-comparable and rely on Linux CI for cross-engine strict columns. - -#### I158. freemap_churn claims aux-metric persistence that never happens — reclamation regressions are invisible [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `bench/benches/freemap_churn.rs:193` - -**Problem:** The comments at freemap_churn.rs lines 14, 126, and 193 claim the churn metrics are persisted to bench/results/aux_metrics.jsonl via AuxMetricsWriter, but this bench never uses that writer (it exists in bench/src/runner.rs and is used only by micro_grid.rs); the pages_allocated and file-size deltas are computed in the timed closures and discarded via black_box, so only wall-clock timing is reported. The bench's two stated purposes — trend-tracking the flat-high-water property and reclamation pages_allocated — are unfulfilled: only wall-clock timing reaches Criterion's output. A freemap reclamation regression (file growing per commit) would be invisible unless it also changed timing. The comments tell a reader the safety net exists when it does not. - -**Direction of fix:** Either wire an AuxMetricsWriter (as micro_grid does) to emit the deltas per case, or delete the three aux-file claims and state that only timing is tracked. - -#### I159. No test combines encryption with the on-disk (Path) spillway sidecar — the 'no plaintext hits disk' promise is only unit-tested in-memory [deepdive 2026-07-02] — **P2** 🔶 OPEN -**Where:** `tests/spillway_integration.rs:22` - -**Problem:** No test anywhere combines encryption with the production on-disk spillway sidecar. Chisel::open always uses SpillwayLocation::Path (src/lib.rs:381), but every encrypted-spillway test (src/page_cache.rs:1985 fresh_encrypted_cache_with_spillway) hardcodes SpillwayLocation::InMemory, and tests/spillway_integration.rs never sets encryption_key (grep confirms zero 'encryption' hits in the spillway integration tests). The spillway sidecar is a second file that receives page payloads mid-transaction. The at-rest-encryption promise ('no plaintext hits disk') is exactly about this path, and the only coverage is unit-level against an in-memory buffer. A regression that routed plaintext bytes into the Path-backed sidecar (e.g. a drain refactor picking the wrong payload_size branch) would pass the entire suite. - -**Direction of fix:** Add one integration test: open an encrypted DB with a ~4-page cache, allocate sentinel-patterned values until pages spill, scan the sidecar file for the sentinel bytes (must be absent), then commit + reopen + read back. - -#### I160. RadixU64::insert(u64::MAX) infinite grow loop + delete lacks find_leaf's cap guard (latent; twin of I147) [deepdive 2026-07-02] — **P3** 🔶 OPEN -**Where:** `src/membership_index.rs:243` - -**Problem:** Latent, unreachable-from-public-API inconsistency in the pub(crate) RadixU64: insert(key = u64::MAX) infinite-loops in the grow() while-loop (capacity saturates at u64::MAX for depth >= 6, and grow has no MAX_DEPTH cap), and delete lacks find_leaf's cap != u64::MAX exemption (benign today since u64::MAX can never be inserted, but contradicts find_leaf's documented u64::MAX support). Production keys are u32 tags and validated engine-minted handles, so no external input reaches these paths; fix is a one-line MAX_DEPTH cap / error in the grow loop plus mirroring find_leaf's guard in delete. Same runaway grow-and-allocate loop; same internal contradiction with find_leaf's explicit u64::MAX support and the MAX_DEPTH comment (lines 22-27, "forces one final grow to depth 6"). Unreachable in practice via engine-assigned handles / u32 tags, but latent for any future in-crate caller of the generic radix. - -**Direction of fix:** Add `&& self.depth < MAX_DEPTH` to the grow loop (or a MAX_DEPTH no-op in `grow`, matching FreeMapTree::grow), and align `delete`'s capacity guard with `find_leaf`'s saturated-capacity exemption. - diff --git a/README.md b/README.md index a1b0546..d643d84 100644 --- a/README.md +++ b/README.md @@ -293,7 +293,7 @@ On create, Chisel generates a random data-encryption key (DEK), encrypts every p The wrapped DEK lives in an **8-slot key table**. Because the DEK itself never changes, credential rotation only re-wraps the DEK in a slot — it is O(1), independent of database size. `add_key` stages a second credential (both open the DB), `rotate_key` replaces one credential in place, and `remove_key` retires one (refusing the last remaining slot with `LastKeySlot`). A full table returns `NoFreeKeySlot`. -See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for the on-disk layout (crypto header, key slots, per-page nonce stride) and [THEORY.md](THEORY.md) for the rationale behind the envelope scheme and the shadow-paging nonce discipline (with [ISSUES.md](ISSUES.md) as the dated decision log). +See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for the on-disk layout (crypto header, key slots, per-page nonce stride) and [THEORY.md](THEORY.md) for the rationale behind the envelope scheme and the shadow-paging nonce discipline (with [`docs/adr/`](docs/adr/) as the dated decision log). ## API reference @@ -473,7 +473,8 @@ db.remove_key("correct horse battery staple") # retire - [`ARCHITECTURE.md`](ARCHITECTURE.md) — living architecture overview: layer model, commit protocol, recovery, full on-disk format byte-by-byte, and cross-cutting concepts. Start here if you're reading the codebase to *act* on it. - [`THEORY.md`](THEORY.md) — theory of operation: *why* the design is what it is — the load-bearing decisions, the rejected alternatives, and the implementation history. Read this to build a durable model before changing the engine. -- [`ISSUES.md`](ISSUES.md) — running decision log: open issues, closed issues, and every design tradeoff with date-stamped rationale. +- [`docs/adr/`](docs/adr/) — architecture decision records: one file per decision, with the context and the alternatives that were rejected. +- [GitHub issues](https://github.com/pgexperts/chisel/issues) — the running issue log. This replaced a tracked `ISSUES.md` on 2026-08-03; `I` markers in code comments refer to entries in that retired file, which remains readable in git history. ## License diff --git a/THEORY.md b/THEORY.md index aa1c454..08676db 100644 --- a/THEORY.md +++ b/THEORY.md @@ -161,7 +161,7 @@ There is one **critical, revised** sub-decision that is easy to get wrong and wo **Why.** Envelope encryption makes credential rotation O(1) — you re-wrap the DEK — instead of O(database size). The per-slot KDF choice matches input entropy: HKDF is fast and correct for high-entropy keys, while Argon2id is memory-hard to resist brute-forcing low-entropy passphrases (its params are recorded per slot). And every rotation op is an ordinary superblock A/B + fsync commit, so it reuses the existing crash-safe protocol wholesale: a metadata-only `rewrite_crypto_header` commit persists a rotated slot table atomically (write the inactive slot, fsync, promote), so a crash mid-rotation leaves the old table intact. -Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9 and ISSUES.md I142. +Two threat-model boundaries are documented rather than solved, and you should know them before you rely on this: there is **no rollback/replay resistance** (an attacker who substitutes a wholly older, validly-signed image is undetectable without an external trust anchor like a TPM), and the DEK sits in plaintext in process memory during a session (mitigated by zeroize-on-drop, not by encryption). See spec `2026-06-29` §3/§5/§9 and [issue #140](https://github.com/pgexperts/chisel/issues/140) (the deferred bulk DEK rotation, formerly I142). ### Encryption page format: 8232-byte stride, logical page stays 8192, MAJOR 1→2 (ADR-15) diff --git a/docs/adr/0017-github-issues-replace-tracked-issues-md.md b/docs/adr/0017-github-issues-replace-tracked-issues-md.md new file mode 100644 index 0000000..99e0842 --- /dev/null +++ b/docs/adr/0017-github-issues-replace-tracked-issues-md.md @@ -0,0 +1,96 @@ +--- +id: 0017 +title: Track issues in GitHub, not a tracked ISSUES.md +date: 2026-08-03 +status: Accepted +summary: The 1868-line ISSUES.md decision log was retired; open entries were migrated to GitHub issues and the file deleted. +--- + +# 0017. Track issues in GitHub, not a tracked ISSUES.md + +## Context + +From the project's start, issues lived in a single git-tracked `ISSUES.md`: +119 numbered entries (`I1`–`I160`, with gaps) across 1868 lines, each carrying +a location, a triaged problem statement, and a direction of fix. It doubled as +the decision log — closed entries recorded not just that something was fixed +but why a particular fix was chosen over the alternatives. + +Three forces made this untenable: + +- **It outgrew a context window.** At 1868 lines the file could not be read + whole by an agent session that also needed to hold the code. Every review + pass had to grep it, and the 2026-07-29 review deliberately withheld it from + its reviewers to keep the pass clean-slate — an admission that the log had + become too large to be an input. +- **It duplicated a tracker the project already used.** The repository is + public and its work already flows through GitHub issues and pull requests. + The 2026-07-29 review's findings were filed as issues #102–#126 while their + predecessors sat in `ISSUES.md`, so the same finding could exist in two + places under two identifiers (the review's own text had to annotate findings + "KNOWN as I149" to reconcile them). +- **Status drifted silently.** Nothing linked an entry to the commit that + fixed it. At the time of this decision `I149` and `I150` were both still + marked `🔶 OPEN` although `I150` had been fixed and merged in PR #128 and + `I149` was fixed in the branch performing this migration. + +Doing nothing meant maintaining a second issue tracker by hand, in a format +that could not be queried, linked from a commit, or closed by a merge. + +## Decision + +We will track issues in GitHub and delete `ISSUES.md`. + +The 14 open and 3 deferred entries were triaged before deletion: 6 were +already represented in existing GitHub issues (`I147`→#116, `I149`/`I150`→#102, +`I151`→#107, `I152`/`I154`→#106) and the remaining 11 were filed as issues +#138–#148, each carrying its original entry verbatim plus a provenance header +naming its `I` and source review. The 83 entries marked fixed were not +migrated; they describe completed work and remain readable in git history. + +The ~167 `I` markers in source comments were deliberately **not** +rewritten. `ARCHITECTURE.md` gained a note explaining what they refer to and +how to retrieve the retired file. + +## Alternatives considered + +- **Keep `ISSUES.md` and sync it with GitHub** — rejected. Two systems of + record with a manual sync step is exactly the drift that produced the stale + `I149`/`I150` statuses. The sync would have no enforcement. + +- **Split `ISSUES.md` into per-issue files under `docs/issues/`**, mirroring + what was done for the ADR log — rejected. It solves the context-window + problem but not the duplication: the project would still be running a + second tracker alongside GitHub, with no way to close an entry from a PR. + The ADR split is different in kind, because decisions genuinely belong in + the repository at the commit that made them; open work does not. + +- **Migrate the 83 fixed entries into ADRs** — rejected for this change, but + worth revisiting. Some closed entries do contain real decision rationale + (the `I119` argument for a controlled panic over a typed fatal error, for + instance). Converting them wholesale would be a large paraphrase-risk + exercise; doing it selectively, when a future change touches the decision, + is cheaper and more accurate. + +- **Rewrite the ~167 `I` comment markers** to point at the new issue + numbers — rejected. Most refer to entries that were closed years of commits + ago and have no GitHub equivalent, so the rewrite would be mostly deletion + of provenance, across 167 sites, for no navigational gain. + +## Consequences + +- Issue status is now derived from GitHub rather than asserted in a file: a + merged PR can close its issue, and an issue cannot be stale-but-marked-open + without someone noticing. +- Labels (`severity:*`, `type:*`) replace the `P1`/`P2`/`P3` priority prefix. + The migrated issues were labelled by nature, not by mechanical translation + of their old priority, so old and new priorities are not comparable. +- The decision log is now split by kind: `docs/adr/` holds decisions, GitHub + holds open work. A reader looking for "why is it like this" goes to the ADRs; + a reader looking for "what is broken" goes to the issue tracker. +- The 83 fixed entries are no longer discoverable by grep in a checkout. They + remain in git history (`git show 0ffe3bc:ISSUES.md`), but retrieving them now + requires knowing they existed — which is what the `ARCHITECTURE.md` note is + for. This is the real cost of this decision. +- Offline work loses the issue list. Anyone working without network access can + no longer read the open issues from the checkout. diff --git a/docs/adr/README.md b/docs/adr/README.md index be9a6e0..058ebca 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -21,3 +21,4 @@ | [0014](0014-client-byte-spending-the-last-reserved-entry-byte.md) | Client byte — spending the last reserved entry byte | Accepted | 2026-06-05 | | | [0015](0015-on-disk-encryption-xchacha20-poly1305-envelope-keys.md) | On-disk encryption (XChaCha20-Poly1305, envelope keys) | Accepted | 2026-06-30 | | | [0016](0016-swift-binding-via-uniffi.md) | Swift binding via UniFFI | Accepted | 2026-07-19 | The iOS/macOS Swift binding is a UniFFI-generated FFI over an Arc> wrapper crate, with the engine crate left unchanged. | +| [0017](0017-github-issues-replace-tracked-issues-md.md) | Track issues in GitHub, not a tracked ISSUES.md | Accepted | 2026-08-03 | The 1868-line ISSUES.md decision log was retired; open entries were migrated to GitHub issues and the file deleted. | From 7da56cc9e1577ab2067780f9e4bf7f1970882bf8 Mon Sep 17 00:00:00 2001 From: Christophe Pettus Date: Tue, 4 Aug 2026 15:48:26 -0700 Subject: [PATCH 4/4] docs: close the review's findings on the ported decision log MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things the docs review caught. The ADR log was ported from the retired lineage with its `ISSUES.md` citations intact, which is correct — an Accepted record is superseded, not edited — but README.md and ARCHITECTURE.md now route readers to `docs/adr/` as *the* decision log, so a reader following that pointer lands on citations with no way to resolve them. Record 0000 is the register overview rather than a decision, so it carries the breadcrumb for the whole log; 0017 records the same gap as an explicit consequence of the retirement it decides. Record 0000's claim that smaller decisions "live in ISSUES.md" was simply false as of the commit that deletes the file. The review record put the I143 regression 'five commits' before its own baseline; `git rev-list --count 04534c0..d87e670` is 2. --- docs/adr/0000-decision-register-overview.md | 4 +++- docs/adr/0017-github-issues-replace-tracked-issues-md.md | 9 +++++++++ docs/reviews/review-20260729-183138.md | 2 +- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/adr/0000-decision-register-overview.md b/docs/adr/0000-decision-register-overview.md index 458c974..2544210 100644 --- a/docs/adr/0000-decision-register-overview.md +++ b/docs/adr/0000-decision-register-overview.md @@ -7,7 +7,9 @@ status: Accepted # 0000. Decision register (overview) -Chisel is a single-writer embedded transactional storage engine in Rust. The decisions below are the ones that, if reversed, would require rewriting substantial parts of the engine. Smaller decisions (specific bit layouts, error message wording, individual issue resolutions) live in `ISSUES.md`. +Chisel is a single-writer embedded transactional storage engine in Rust. The decisions below are the ones that, if reversed, would require rewriting substantial parts of the engine. Smaller decisions (specific bit layouts, error message wording, individual issue resolutions) are tracked as [GitHub issues](https://github.com/pgexperts/chisel/issues); before 2026-08-03 they lived in a tracked `ISSUES.md`, which ADR [0017](0017-github-issues-replace-tracked-issues-md.md) retired. + +**Reading `ISSUES.md` citations.** Records 0006, 0007, 0012, 0013 and 0015 predate that retirement and cite `ISSUES.md` entries by `I`. Those bodies are left as written — an Accepted record is superseded, not edited, and the citations are accurate as dated provenance. Resolve one with `git show 0ffe3bc:ISSUES.md`, the last commit carrying the file. | # | Decision | Status | Reversibility | |---|---|---|---| diff --git a/docs/adr/0017-github-issues-replace-tracked-issues-md.md b/docs/adr/0017-github-issues-replace-tracked-issues-md.md index 99e0842..e341232 100644 --- a/docs/adr/0017-github-issues-replace-tracked-issues-md.md +++ b/docs/adr/0017-github-issues-replace-tracked-issues-md.md @@ -94,3 +94,12 @@ how to retrieve the retired file. for. This is the real cost of this decision. - Offline work loses the issue list. Anyone working without network access can no longer read the open issues from the checkout. +- The decision log itself now contains dead citations. Records 0006, 0007, + 0012, 0013 and 0015 were written while `ISSUES.md` was live and cite it by + `I`; those bodies are Accepted and are superseded rather than edited, + so the citations stay. This is the same trade made for the ~167 code comments + above, and it lands in a worse place — README.md and ARCHITECTURE.md now route + readers to `docs/adr/` as *the* decision log, so a reader following that + pointer can hit a citation with no way to resolve it. Record 0000 carries the + breadcrumb (`git show 0ffe3bc:ISSUES.md`) for the whole log rather than + repeating it in each affected record. diff --git a/docs/reviews/review-20260729-183138.md b/docs/reviews/review-20260729-183138.md index da80eb9..39a7c84 100644 --- a/docs/reviews/review-20260729-183138.md +++ b/docs/reviews/review-20260729-183138.md @@ -32,7 +32,7 @@ One process note: a verifier agent violated the read-only instruction and wrote ## Executive summary -1. **`Chisel::open` silently destroys any existing file smaller than one page, and ignores `create_if_missing: false` while doing it** (`src/lib.rs:366` vs `:384`). Two different "does this file exist" predicates disagree for lengths 1..8191. **This is a regression introduced five commits ago by the I143 fix itself** (`04534c0`): that commit moved the create-vs-open decision to `io.page_count()? > 0` to close a lock race, but left the `create_if_missing` gate on `metadata.len() > 0`. Reproduced end-to-end. Fix by deriving both decisions from one post-lock predicate. +1. **`Chisel::open` silently destroys any existing file smaller than one page, and ignores `create_if_missing: false` while doing it** (`src/lib.rs:366` vs `:384`). Two different "does this file exist" predicates disagree for lengths 1..8191. **This is a regression introduced two commits ago by the I143 fix itself** (`04534c0`): that commit moved the create-vs-open decision to `io.page_count()? > 0` to close a lock race, but left the `create_if_missing` gate on `metadata.len() > 0`. Reproduced end-to-end. Fix by deriving both decisions from one post-lock predicate. 2. **A hostile or bit-rotted database file can hang or abort the process before any version check** (`src/superblock/crypto_header.rs:110`). Argon2 `m_cost`/`t_cost` are parsed straight out of the superblock and handed to the KDF with no bound, and argon2 0.5.3 enforces no ceiling — up to 4 TiB of infallible allocation, retried for up to 8 key slots. This is the same trust boundary the project already hardened twice (I144 stride, ct_len); the key slots were missed. 3. **`swift/` is 974 MB of untracked *and* unignored build output on `main`** — `.o`, `.d`, `.swiftdeps`, `.build/`, `.xcbuild/`, and a three-architecture `Chisel.xcframework`, with **zero `.swift` sources** in it. The actual binding sources live on the `design/swift-binding` branch. So the risk is not data loss, it is that `git add swift/` (the obvious response to `?? swift/`) commits nearly a gigabyte of artifacts. It needs a `.gitignore` rule, not a commit. 4. **The benchmark harness reports nothing and compares unfairly, and its tests cannot catch either.** `discover_cells` walks depth 4 while Criterion writes at depth 5, so every micro-grid timing cell renders as an em-dash — and the test fixtures were built at depth 4, matching the bug, so the suite stays green. Separately, SQLite's cold-read cells time a DELETE→WAL journal conversion the harness itself created, inside the timed region.