Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,9 +195,10 @@ sequenceDiagram

U->>L: open(path, options)
L->>L: validate options.superblock_count ∈ 2..=16
L->>L: file exists & non-empty?
alt file exists
L->>IO: PageIo::open (acquires flock)
L->>L: file_exists? (pre-lock stat;<br/>gates create_if_missing only)
L->>IO: PageIo::open (acquires flock)
L->>IO: page_count() — post-lock length
alt existed (post-lock page_count > 0)
L->>TM: open_existing(cache)
TM->>IO: read pages 0..MAX_SUPERBLOCKS
TM->>SB: deserialize each candidate
Expand All @@ -208,7 +209,6 @@ sequenceDiagram
TM->>TM: re-derive handle-table + membership<br/>outer depth from their roots (I99 / C1)
TM-->>L: TransactionManager
else fresh
L->>IO: PageIo::open
L->>TM: create_new(cache, superblock_count)
Note over TM: I2: write all N slots with<br/>staggered counters so a torn<br/>first commit has a fallback
end
Expand All @@ -217,6 +217,8 @@ sequenceDiagram

The format-version gate after `select` is what makes the README's "sacred within a major version" promise concrete: a file written by a future, incompatible MAJOR is rejected with `UnsupportedFormatVersion`. Same-major files (any minor) open cleanly. (See I29 for the packed-MAJOR/MINOR scheme; I31 for the per-page version byte that supports lazy upgrade within a major.)

**Landmine:** the create-vs-open decision must come from `page_count()` observed *after* `PageIo::open` holds the flock, never from the pre-lock `file_exists` stat — a pre-lock decision races a concurrent creator (process B stats an empty file, process A creates + commits + closes, B's stale decision then runs `create_new` over A's committed data). `file_exists` still gates `create_if_missing`, since a refused open must never materialize an empty file — but it must not also decide create-vs-open (I143).

---

## On-disk format
Expand Down Expand Up @@ -610,6 +612,8 @@ The cap parameter (`DefragOptions::max_pages`) bounds the number of *values* rel

On any fatal error — an `IoError` from `fsync`, a `ChecksumMismatch` on a page load, a `CorruptSuperblock` on open, a `DecryptionFailed`, any error raised after the commit protocol has begun — the `TransactionManager` becomes **poisoned**. Every subsequent call returns `ChiselError::Poisoned`, **including reads**. The only legal recovery is to drop the `Chisel` handle and call `Chisel::open` again; the shadow-paging recovery path then returns the database to its last-durable state. (See I1; rationale — fsyncgate, the Mutex analogy — in THEORY.md.)

**Landmine:** every `TransactionManager` method with a fatal-error path must route its `Result` through `self.poison_on_fatal(result)` before returning. A new entry point that instead propagates the error directly (a bare `?`) bypasses this — the manager stays unpoisoned while a subsequent partial mutation may already have reached `current_roots`, so a later `commit()` can durably persist an indeterminate state (I145: `reclaim_freemap_orphans` missed this wrap).

### Engine-activity counters

`Chisel::counters()` returns a `ChiselCounters` snapshot of four cumulative-from-open counters: `cache_hits`, `cache_misses`, `pages_allocated`, and `fsync_calls`. Each counter is a `Cell<u64>` living at the site that increments it (`PageCache` for the first three, `PageIo` for fsync), and `PageCache::counters()` aggregates them into a single struct read via `PageIo::fsync_count()`.
Expand Down Expand Up @@ -639,6 +643,8 @@ Chisel supports optional authenticated encryption of database files (shipped; MA

**On-disk stride.** Encrypted databases use a uniform 8232-byte stride for every page including the superblock slots. Plaintext databases use the 8192-byte stride; the two are mutually exclusive and the stride is recorded in the superblock's plaintext crypto-header (bytes 325..329) so the engine reads the correct number of bytes before any operation. The `page_io` layer is stride-agnostic: callers set the stride once (via `PageIo::set_stride`) and all subsequent raw reads/writes use it.

**Landmine:** `stride` is guarded only by the page's forgeable XXH3 checksum, not the AEAD tag — it must be readable before any DEK is available, so it cannot live inside the sealed body. `open_existing` validates `stride == ENC_PAGE_SIZE` before calling `set_stride`; skip that check and a forged stride of 0 divides-by-zero panics `set_stride`, while a huge forged stride drives multi-GiB read allocations — both on `open` alone, no key required (I144).

**Envelope (key hierarchy).** A random 256-bit per-database DEK encrypts all page content. The DEK is never stored in plaintext: it is wrapped under a key-encryption key (KEK) and the wrapped form is held in a plaintext key-slot table inside the superblock's reserved region. The crypto-header + key-slot table occupy bytes **324..1356**:

```text
Expand Down
6 changes: 6 additions & 0 deletions python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,11 @@ Catch and continue.
| `InvalidSuperblockCountError` | `superblock_count` outside `2..=16` |
| `CacheFullError` | Page cache hit its strict `cache_max_bytes` cap with every cached page dirty (no clean page available for eviction) AND the spillway is disabled (`spillway_max_bytes=0`); commit or roll back to drain. When the spillway is enabled (default), the cache overflows into it instead and you'll see `SpillwayFullError` only if the spillway also fills. |
| `SpillwayFullError` | Spillway sidecar's `spillway_max_bytes` cap was reached during a transaction; commit or roll back to drain the spillway. Database is intact. |
| `NoEncryptionKeyError` | Opened an encrypted database without supplying `encryption_key` |
| `InvalidEncryptionKeyError` | `encryption_key` was supplied but unwraps no key slot (wrong passphrase or wrong raw bytes) |
| `EncryptionNotSupportedError` | `encryption_key` was supplied but the database is plaintext |
| `NoFreeKeySlotError` | `add_key` / `rotate_key` attempted but all 8 key-slot table entries are in use |
| `LastKeySlotError` | `remove_key` would clear the last active key slot, leaving the database permanently unopenable |
| `TagMismatchError` | `delete_tagged(handle, tag)` was passed a `tag` that doesn't match the handle's stored tag; the chunk and membership index are left untouched |
| `ClosedError` | Any call on a `Chisel`, `Transaction`, or `Savepoint` after `db.close()` |
| `AlreadyFinishedError` | Second explicit drive on a transaction or savepoint |
Expand All @@ -360,6 +365,7 @@ Drop the handle and reopen.
| Class | When it fires |
|---|---|
| `IoError` | Underlying filesystem I/O error. Also subclasses the builtin `OSError`, so it is catchable as `except OSError` and `.errno` is `OSError`'s native attribute (with `.strerror` set when an errno exists). Carries `.errno` (the raw OS error code, or `None` if unavailable) and `.kind` (the Rust `io::ErrorKind` name, e.g. `"PermissionDenied"`), so callers can branch on the cause without parsing the message. |
| `DecryptionFailedError` | A page or the superblock body failed AEAD authentication (wrong key, or the ciphertext was tampered with) |
| `ChecksumMismatchError` | A page's XXH3 checksum did not validate on load |
| `CorruptSuperblockError` | No readable superblock slot found |
| `FileSizeMismatchError` | File size inconsistent with the superblock's claim |
Expand Down
Loading