Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 73 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ Chisel is designed for single-writer embedded use: one process holds the file vi
- **Crash durability** — N configurable superblocks (2–16) with round-robin writes ensure committed data survives crashes. Every page carries an XXH3 checksum for torn-write and bit-rot detection.
- **Transactions** — begin / commit / rollback with shadow-paging durability (all data pages fsync'd before the superblock fsync; three fsyncs per commit).
- **Savepoints** — PostgreSQL-style named savepoints with `rollback_to` (savepoint preserved for retry) and `release` (merges into the enclosing scope).
- **Handles** — store a value, get back a `u64` handle. Read, update, or delete by handle. Handles are stable across updates, defrag, and reopens.
- **Chunk tags** — attach an immutable `u32` tag to a chunk at allocation time; a reverse membership index maps each tag back to its handles. Enumerate a tag's members, delete with a tag-match assertion, or bulk-drop a whole tagged relation in bounded passes. Tag `0` means untagged.
- **Handles** — store a value, get back a `Handle` (a `u64` newtype). Read, update, or delete by handle. Handles are stable across updates, defrag, and reopens.
- **Chunk tags** — attach an immutable `Tag` (a non-zero `u32` newtype) to a chunk at allocation time; a reverse membership index maps each tag back to its handles. Enumerate a tag's members, delete with a tag-match assertion, or bulk-drop a whole tagged relation in bounded passes. Untagged is the absence of a tag, not a zero value: `tag()` returns `None`.
- **Value packing** — slotted data pages pack multiple small values per 8 KB page; values over ~8 KB transparently overflow into chained pages.
- **Named roots** — a small fixed table in the superblock mapping string names to handles. Survives commit / rollback transactionally.
- **Defragmentation** — explicit `defrag()` consolidates sparse pages and returns a count-based stats record.
Expand All @@ -44,7 +44,7 @@ The published crate is named `chisel-storage` (plain `chisel` was already taken)

## Quick Start

```rust
```rust,no_run
use chisel::{Chisel, Options};
use std::path::Path;

Expand Down Expand Up @@ -133,10 +133,15 @@ A handle is a stable `u64` returned by `allocate()`. It maps through a radix-tre
All mutations require an active transaction. `begin()` opens one, `commit()` makes it durable, `rollback()` discards it. Only one transaction is active at a time — Chisel has savepoints, not nested transactions.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
# let mut db = Chisel::open_in_memory()?;
db.begin()?;
let h1 = db.allocate(b"a")?;
let h2 = db.allocate(b"b")?;
db.commit()?; // both h1 and h2 become durable atomically
# Ok(())
# }
```

Rollback is effectively free: pages written during the transaction were never linked from a superblock, so they are simply abandoned. There is no undo log to replay.
Expand All @@ -150,27 +155,37 @@ Savepoints are named marks within a transaction.
- `rollback()` (full rollback) and `commit()` both clear the entire savepoint stack.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
# let mut db = Chisel::open_in_memory()?;
db.begin()?;
let keep = db.allocate(b"keep")?;
db.savepoint("experiment")?;
let _ = db.allocate(b"maybe discard")?;
db.rollback_to("experiment")?; // discards the _ handle; keep remains; sp still open
db.release("experiment")?;
db.commit()?;
# Ok(())
# }
```

### Named roots

A small fixed-size table in the superblock mapping short string names to handles, intended for long-lived entry points such as a meta-B-tree root. Changes are transactional: `set_root_name` takes effect on commit and reverts on rollback.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
# let mut db = Chisel::open_in_memory()?;
db.begin()?;
let meta = db.allocate(b"meta-root-payload")?;
db.set_root_name("meta", meta)?;
db.commit()?;

// Later, possibly after reopen:
let meta = db.get_root_name("meta")?.expect("meta root should be set");
# Ok(())
# }
```

Names are bounded in length and must be valid UTF-8 without embedded NUL; the table has a small fixed capacity. See `TransactionManager::set_root_name` for exact limits.
Expand All @@ -180,6 +195,9 @@ Names are bounded in length and must be valid UTF-8 without embedded NUL; the ta
`defrag()` consolidates sparse data pages: it re-inserts values from pages whose live-slot count falls below a threshold so those pages become fully free and can be reclaimed. It runs inside an active transaction so it composes with other work and commits atomically.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
# let mut db = Chisel::open_in_memory()?;
use chisel::DefragOptions;

db.begin()?;
Expand All @@ -191,40 +209,57 @@ let stats = db.defrag(
.max_values(0), // 0 = no cap on values relocated
)?;
db.commit()?;
# Ok(())
# }
```

### Chunk tags

`allocate_tagged(value, tag)` attaches an immutable `u32` tag to a chunk and registers it in a reverse membership index. The tag is fixed at allocation: `update` preserves it, and retagging means delete + re-allocate. Tag `0` is the untagged sentinel and is never indexed — use plain `allocate` for untagged values. Plain `delete` is self-maintaining (it drops the handle from the index automatically); `delete_tagged` is the verified variant that errors with `TagMismatch` if the stored tag differs.
`allocate_tagged(value, tag)` attaches an immutable tag to a chunk and registers it in a reverse membership index. The tag is fixed at allocation: `update` preserves it, and retagging means delete + re-allocate. Plain `delete` is self-maintaining (it drops the handle from the index automatically); `delete_tagged` is the verified variant that errors with `TagMismatch` if the stored tag differs.

A tag is a `Tag`, not a bare `u32` — `Tag` wraps a `NonZeroU32`, so zero is unrepresentable rather than being an "untagged" sentinel you could pass by accident. Untagged means *no tag at all*: allocate with plain `allocate()`, and `tag()` returns `None` for such a handle.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
# let mut db = Chisel::open_in_memory()?;
use chisel::Tag;

let tag = Tag::new(42).expect("42 is non-zero"); // or Tag::try_from(42u32)?

db.begin()?;
let a = db.allocate_tagged(b"row-a", 42)?;
let _b = db.allocate_tagged(b"row-b", 42)?;
let a = db.allocate_tagged(b"row-a", tag)?;
let _b = db.allocate_tagged(b"row-b", tag)?;
db.commit()?;

assert_eq!(db.tag(a)?, 42);
let members = db.handles_with_tag(42)?; // both handles; order unspecified, but repeatable within a session
assert_eq!(db.tag(a)?, Some(tag)); // None for a handle allocated untagged
let members = db.handles_with_tag(tag)?; // both handles; order unspecified, but repeatable within a session
assert_eq!(members.len(), 2);

// Bounded relation-drop: loop until the tag is fully drained.
loop {
db.begin()?;
let progress = db.delete_with_tag(42, 256)?; // up to 256 chunks per pass
let progress = db.delete_with_tag(tag, 256)?; // up to 256 chunks per pass
db.commit()?;
if progress.complete {
break;
}
}
# Ok(())
# }
```

### In-memory mode

`Chisel::open_in_memory()` creates a memory-backed database using a `Vec<u8>`-backed `PageIo`. Same code path, same guarantees except durability — no filesystem, no `flock`, and all data is lost on drop.

```rust
# fn main() -> chisel::Result<()> {
# use chisel::Chisel;
let mut db = Chisel::open_in_memory()?;
// ... same API as a file-backed Chisel ...
# Ok(())
# }
```

For tuned options (cache size, superblock count), use `Chisel::open_in_memory_with_options(options)`.
Expand All @@ -238,7 +273,8 @@ Encryption is opt-in and driven entirely through `Options::encryption_key`. A ke

Both variants zeroize their material on drop.

```rust
```rust,no_run
# fn main() -> chisel::Result<()> {
use chisel::{Chisel, Key, Options};
use std::path::Path;
use zeroize::Zeroizing;
Expand All @@ -249,6 +285,8 @@ let mut db = Chisel::open(
Path::new("secret.db"),
Options::default().encryption_key(key),
)?;
# Ok(())
# }
```

On create, Chisel generates a random data-encryption key (DEK), encrypts every page under it with XChaCha20-Poly1305, and wraps the DEK under a key-encryption key derived from your supplied key. On reopen, the supplied key must unwrap one of the on-disk key slots or `open` fails with `InvalidEncryptionKey`. Supplying a key to a plaintext DB returns `EncryptionNotSupported`; omitting it on an encrypted DB returns `NoEncryptionKey`.
Expand All @@ -272,13 +310,13 @@ See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for
| `savepoint(name)` | Create a named savepoint |
| `rollback_to(name)` | Undo to savepoint (savepoint preserved) |
| `release(name)` | Merge savepoint into enclosing scope |
| `allocate(value)` | Store a value; returns a `u64` handle |
| `allocate_tagged(value, tag)` | Store a value with an immutable `u32` tag; returns a `u64` handle |
| `allocate(value)` | Store a value; returns a `Handle` |
| `allocate_tagged(value, tag)` | Store a value with an immutable `Tag` (non-zero); returns a `Handle` |
| `read(handle)` | Retrieve a value (takes `&self`) |
| `update(handle, value)` | Replace a value (handle preserved) |
| `delete(handle)` | Remove a handle |
| `delete_many(handles)` | Batch-delete in the current transaction |
| `tag(handle)` | Read a handle's tag, `0` if untagged (takes `&self`) |
| `tag(handle)` | Read a handle's tag as `Option<Tag>`; `None` if untagged (takes `&self`) |
| `handles_with_tag(tag)` | Enumerate live handles carrying `tag`; repeatable within a session, order unspecified (takes `&self`) |
| `client_byte(handle)` | Read the handle's opaque client byte, `0` if unset (takes `&self`) |
| `set_client_byte(handle, byte)` | Set the opaque client byte; mutable, transactional; `update()` preserves it |
Expand All @@ -298,23 +336,24 @@ See [ARCHITECTURE.md#on-disk-encryption](ARCHITECTURE.md#on-disk-encryption) for

## Options

`Options` is `#[non_exhaustive]`, so a downstream crate cannot build one with a struct literal. Start from `Options::default()` and use the chained setters — every field has one:

```rust
use chisel::Options;

let options = Options {
cache_max_bytes: 8 * 1024 * 1024, // in-memory LRU cap, default 8 MiB
spillway_max_bytes: 1024 * 8 * 1024 * 1024, // sidecar overflow file cap, default 8 GiB
drain_insertion: chisel::DrainInsertion::LruTail, // default; use Mru to drain at insertion
create_if_missing: true,
read_only: false,
superblock_count: 2, // 2..=16; only consulted on create
encryption_key: None, // Some(key) to create/open an encrypted DB
argon2_params: None, // None = OWASP defaults; only used on create
};
# fn main() {
use chisel::{DrainInsertion, Options};

let options = Options::default()
.cache_max_bytes(8 * 1024 * 1024) // in-memory LRU cap, default 8 MiB
.spillway_max_bytes(1024 * 8 * 1024 * 1024) // sidecar overflow file cap, default 8 GiB
.drain_insertion(DrainInsertion::LruTail) // default; use Mru to drain at insertion
.create_if_missing(true)
.read_only(false)
.superblock_count(2); // 2..=16; only consulted on create
// .encryption_key(key) to create/open an encrypted DB;
// .argon2_params(params) to override the OWASP defaults (create only).
# }
```

`Options` is `#[non_exhaustive]`, so build a customized value with the chained setters rather than a struct literal from another crate: `Options::default().cache_max_bytes(N).encryption_key(key)`. Every field has a matching setter, including `Options::encryption_key(key)` and `Options::argon2_params(params)`.

`cache_max_bytes` is a strict cap on the in-memory LRU cache. When the cache is full and a dirty page cannot be evicted, overflow dirty pages spill to a sidecar `Spillway` file rather than returning an error. The spillway file is bounded by `spillway_max_bytes` (default 8 GiB). Setting `spillway_max_bytes = 0` disables the spillway entirely, restoring the pre-spillway `CacheFull` semantics at the strict cache cap: the operational error `CacheFull` fires when the cache is full and no eviction is possible. With the spillway enabled, exhausting both the cache and the spillway returns `SpillwayFull { limit_bytes }` (also operational; caller recovers by committing or rolling back).

`read_only = true` still acquires an exclusive `flock` — it only suppresses writes at the application layer. Two read-only opens cannot coexist on the same file. This is a deliberate choice: even a reader must block concurrent writers to keep the shadow-paging invariants intact.
Expand Down Expand Up @@ -347,7 +386,11 @@ On any fatal error — including a failed commit-protocol fsync — the `Chisel`

`commit` is the one method where the operational/fatal split above does not apply. Its only non-poisoning error is `NoActiveTransaction`, checked before the protocol starts. Once `cache.flush()` has run the manager is in a partial-commit state, so *every* error it can then return poisons the handle — including `CacheFull` and `SpillwayFull`, which are operational anywhere else. Do not catch those from `commit` and call `rollback` to recover: the handle is already poisoned and `rollback` will tell you so.

```rust
```rust,no_run
# fn main() -> chisel::Result<()> {
# use chisel::{Chisel, ChiselError, Options};
# let path = std::path::Path::new("db.chisel");
# let mut db = Chisel::open(path, Options::default())?;
match db.commit() {
Ok(()) => (),
// `Poisoned` is matched explicitly: is_fatal() is false for it, so it
Expand All @@ -359,6 +402,8 @@ match db.commit() {
}
Err(e) => return Err(e), // operational — handle per your caller's policy
}
# Ok(())
# }
```

The poison model is mandatory because Linux `fsync` semantics (post-2018 "fsyncgate") do not permit safely retrying a failed fsync: the kernel may have discarded the dirty pages before reporting the error. macOS `F_FULLFSYNC` has similar semantics. PostgreSQL `PANIC`s on fsync failure for exactly this reason.
Expand Down
22 changes: 20 additions & 2 deletions src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,16 @@
// `-D warnings`, which promotes this to a hard error — a new public `-> Result`
// method without an `# Errors` section fails the build.
#![warn(clippy::missing_errors_doc)]
// The README is the entry point for every downstream user, and every Rust
// snippet in it was a compile error against the current API — the `defrag`
// module went `pub(crate)`, `DefragOptions`/`Options` became
// `#[non_exhaustive]`, and tags/handles became `Tag`/`Handle` newtypes, none
// of which the examples followed. Nothing caught it because the README was
// not wired into the build at all. Including it here makes every fence a
// doctest, so `cargo test` fails the next time an API change outruns the
// docs. Fences that would touch the filesystem or need a live failure to
// demonstrate are marked `no_run`: still compiled, just not executed.
#![doc = include_str!("../README.md")]

// I35 (ISSUES.md, 2026-05-22): every storage-internals module is
// pub(crate). The supported public surface is the curated re-export
Expand Down Expand Up @@ -859,7 +869,12 @@ impl Chisel {
/// inside an active transaction). Takes `&self` (F3).
///
/// # Errors
/// Only on poisoning — an unbound `name` returns `Ok(None)`.
/// `InvalidRootName` if `name` could never have been stored in the first
/// place — empty, over `NAMED_ROOT_NAME_LEN` bytes, or containing a NUL.
/// The lookup path runs the same validation `set_root_name` does, so an
/// unrepresentable name is rejected rather than reported as unbound. A
/// merely *unbound* (but valid) name returns `Ok(None)`. Otherwise only
/// on poisoning.
pub fn get_root_name(&self, name: &str) -> Result<Option<Handle>> {
Ok(self.txm.get_root_name(name)?.map(Handle::from))
}
Expand All @@ -868,7 +883,10 @@ impl Chisel {
/// active transaction; becomes durable on commit.
///
/// # Errors
/// `NoActiveTransaction` if no transaction is open.
/// `NoActiveTransaction` if no transaction is open; `InvalidRootName` if
/// `name` is empty, over `NAMED_ROOT_NAME_LEN` bytes, or contains a NUL —
/// the clear path runs the same validation `set_root_name` does, so an
/// unrepresentable name is rejected rather than treated as a no-op.
pub fn clear_root_name(&mut self, name: &str) -> Result<()> {
self.txm.clear_root_name(name)
}
Expand Down
27 changes: 16 additions & 11 deletions src/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,23 @@
pub struct Stats {
/// Number of live handles (u64 ids currently mapped in the handle table).
pub handle_count: u64,
/// Total allocated pages in the file, matching Superblock.total_pages.
/// PHYSICAL page count: how many whole stride-units the file currently
/// holds, from `PageIo::page_count`. This is NOT `Superblock.total_pages`,
/// which is the last-durable *logical* count. The two diverge when a
/// previous crash left orphan pages in the file tail: those pages are
/// counted here but are dead weight the next allocation will overwrite,
/// and the superblock's figure remains the authoritative one for what the
/// database actually contains.
pub total_pages: u64,
/// Raw size of the database file on disk: `page_count × stride`, where
/// `stride` is `PAGE_SIZE` for a plaintext database and `ENC_PAGE_SIZE`
/// (8232) for an encrypted one — an encrypted page carries a 24-byte
/// nonce and a 16-byte tag on top of its 8192 plaintext bytes. May exceed
/// `total_pages * stride` when a previous crash left orphan
/// pages in the file tail — the last-durable superblock's
/// `total_pages` is authoritative, anything beyond it is dead
/// weight that the next allocation will overwrite (see I4).
/// Chisel is single-writer, so there is no concurrent commit
/// that could cause a transient divergence.
/// Raw size of the database file on disk, as `total_pages × stride` —
/// `stride` being `PAGE_SIZE` for a plaintext database and
/// `ENC_PAGE_SIZE` (8232) for an encrypted one, whose pages each carry a
/// 24-byte nonce and a 16-byte tag on top of their 8192 plaintext bytes.
///
/// Since both fields come from the same physical page count, this is
/// exactly `total_pages × stride` and never diverges from it. It is a
/// page-aligned figure rather than a `stat(2)` call, so it will not
/// reflect a trailing partial page mid-extend.
pub file_size_bytes: u64,
/// I74 (ISSUES.md, 2026-05-22): current spillway logical-bytes in
/// flight (`PAGE_SIZE` × LIVE resident spilled pages — a page read back
Expand Down
Loading