Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ As of May 2026, SQLRite has:
- Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md).
- SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i)
- Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md).
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 multi-connection foundation: shipped.** `Connection` is now `Send + Sync` and `Connection::connect()` mints a sibling handle that shares the same backing `Database`. **11.2 logical clock + active-tx registry: shipped on this branch.** New `sqlrite::mvcc` module exposes `MvccClock` (AtomicU64-backed) and `ActiveTxRegistry` (`min_active_begin_ts` for GC). The WAL header bumps v1 → v2 to persist the clock high-water mark; v1 WALs upgrade transparently. Snapshot-isolation reads + `BEGIN CONCURRENT` writes follow in 11.3 / 11.4. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 + 11.2: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock` and `ActiveTxRegistry`. WAL header v1 → v2 persists the clock high-water mark. **11.3 `MvStore` + `PRAGMA journal_mode`: shipped on this branch.** New `MvStore` (the in-memory version index keyed by `RowID`, with the snapshot-isolation visibility rule `begin <= T < end`) and the `JournalMode { Wal, Mvcc }` per-database toggle reachable via `PRAGMA journal_mode = mvcc;`. The executor doesn't consult `MvStore` yet — that wiring lands in 11.4 alongside `BEGIN CONCURRENT` writes (read-side and write-side are coupled). Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md).
- A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete)

See the [Roadmap](roadmap.md) for the full phase plan.
Expand Down
18 changes: 18 additions & 0 deletions docs/design-decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,24 @@ Decisions are grouped by the engine layer they concern: parser, storage, concurr

---

### 12c. `MvStore` data structure + `JournalMode` toggle land before the read path uses them (Phase 11.3)

**Decision.** [`MvStore`](../src/mvcc/store.rs) (the in-memory version index) and the [`JournalMode`](../src/mvcc/mod.rs) enum (with the `PRAGMA journal_mode = wal | mvcc` SQL surface) ship together in Phase 11.3, but the executor's read path **does not consult `MvStore`** until 11.4. `Database` grows two new fields (`mvcc_clock: Arc<MvccClock>`, `mv_store: MvStore`); both are allocated on every `Database::new`, even when the journal mode is `Wal`.

**Why ship the data structures before the read-side wiring.** The snapshot-isolation contract requires that the read path see versions the write path produced. In v0 our writes happen via the legacy `Database.tables` mutation followed by a per-page WAL commit; those don't push into `MvStore`. So if 11.3 wired reads through `MvStore`, every read would see an empty store and return wrong rows. Routing reads through `MvStore` only makes sense once the *commit path* is mirroring writes into it — and that's a non-trivial change (the commit timestamp must come from `MvccClock`, the cap rule has to fire on the previous version, schema changes must invalidate the store). 11.4 ships both halves together because they're coupled. 11.3 ships the parts that *aren't* coupled (the data structure + the toggle) so the diffs stay reviewable.

**Why allocate `mvcc_clock` + `mv_store` even in `Wal` mode.** Two reasons:
- `PRAGMA journal_mode = mvcc;` shouldn't have to lazy-construct anything mid-statement. Constructing `MvccClock` is cheap (one `AtomicU64`); `MvStore` is a `Mutex<HashMap>` (zero-allocation when empty).
- Sibling `Connection::connect` handles can outlive the moment when MVCC was enabled. If the clock were lazy, a sibling connecting before MVCC was first enabled wouldn't observe the same clock as one connecting after — a confusing footgun. Allocating eagerly on `Database::new` means every sibling shares the same `Arc<MvccClock>` from day one.

**Why `Mvcc → Wal` is rejected when the store has committed versions.** The `MvStore` is the only durable record of those versions until 11.5's checkpoint integration drains them into the pager. Switching back to `Wal` mode would either silently strand them (correctness bug) or quietly discard them (data loss). v0 fails the PRAGMA with a typed error and lets the caller decide what to do. When 11.5 lands, "drain to pager then switch" becomes legal.

**Why per-database, not per-connection.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §8 flags this as an open question. Per-connection is more flexible (a maintenance connection can stay in WAL mode while app connections use MVCC); per-database is closer to user expectation and matches SQLite's `PRAGMA journal_mode` semantic. For 11.3 we picked per-database for simplicity — the journal-mode field lives on `Database`, every `Connection::connect` sibling sees the same value. If the per-connection trade-off becomes load-bearing later, the dispatch lives behind `Connection::journal_mode()` already, so callers don't need to change.

**Plan-doc reference.** [`concurrent-writes-plan.md`](concurrent-writes-plan.md) §4.2 (version index), §6 (SQL surface), §8 (open questions on per-connection vs per-database journal mode).

---

## Query execution

### 13. `NULL`-as-false in `WHERE` clauses
Expand Down
16 changes: 11 additions & 5 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -589,19 +589,25 @@ Lift SQLRite past SQLite's single-writer ceiling with multi-version concurrency

`Connection` is a thin handle backed by `Arc<Mutex<Database>>`. Call [`Connection::connect`] to mint a sibling that shares the same engine state — typically one per worker thread. The headline contract: `Connection` is `Send + Sync`, and the engine no longer requires the caller to wrap the public API in their own `Mutex`. Today every operation still serializes through the per-database mutex (and the pager's existing process-level flock), so the behaviour change is *capability*, not throughput; concurrent throughput arrives with `BEGIN CONCURRENT` in 11.4.

### 🚧 Phase 11.2 — Logical clock + active-tx registry *(in progress, plan-doc "Phase 10.2")*
### Phase 11.2 — Logical clock + active-tx registry *(plan-doc "Phase 10.2")*

New [`sqlrite::mvcc`](../src/mvcc/) module:
[`sqlrite::mvcc`](../src/mvcc/) module:

- `MvccClock` — process-wide monotonic `u64` over `AtomicU64`. `tick()` hands out begin- / commit-timestamps; `now()` reads the high-water without advancing it; `observe(value)` advances the clock to `value` if greater (used at WAL replay).
- `ActiveTxRegistry` — `Mutex<BTreeMap>` over in-flight transactions. `register(&clock)` allocates a `TxId`, snapshots `begin_ts`, and returns a RAII `TxHandle`; `min_active_begin_ts()` answers Phase 11.6 GC's "what's still possibly visible" question.
- `TxId` newtype + `TxTimestampOrId` tagged union — defined now so 11.4 can plug in without re-litigating the type shape.

WAL format bumps **v1 → v2**: bytes 24..32 of the WAL header (previously reserved-zero) now carry the persisted `clock_high_water` `u64`. v1 WALs open cleanly — those zero bytes read as "clock never advanced" — and the next checkpoint rewrites the header at v2. No offline upgrade step. `Wal::set_clock_high_water` / `Wal::clock_high_water` accessors expose the field; the setter rejects regressions with a typed error. The clock isn't wired into the executor yet (that's 11.3); the persistence + restore plumbing is in place so 11.3 just reads the high-water at open and seeds the in-memory clock.
WAL format bumps **v1 → v2**: bytes 24..32 of the WAL header (previously reserved-zero) now carry the persisted `clock_high_water` `u64`. v1 WALs open cleanly — those zero bytes read as "clock never advanced" — and the next checkpoint rewrites the header at v2. No offline upgrade step. `Wal::set_clock_high_water` / `Wal::clock_high_water` accessors expose the field; the setter rejects regressions with a typed error.

### Phase 11.3 — `MvStore` skeleton + snapshot-isolation reads *(planned)*
### 🚧 Phase 11.3 — `MvStore` skeleton + `PRAGMA journal_mode` opt-in *(in progress, plan-doc "Phase 10.3")*

In-memory version index + `PRAGMA journal_mode = mvcc` opt-in. Lazy-loads versions from the pager on first touch. Writes still go through the legacy path — only reads change.
Standalone version-index data structure + the per-database journal-mode toggle.

- New [`MvStore`](../src/mvcc/store.rs): `Mutex<HashMap<RowID, Arc<RwLock<Vec<RowVersion>>>>>`. `RowID = (table, rowid)`; each `RowVersion` carries `begin: TxTimestampOrId`, `end: Option<TxTimestampOrId>`, `payload: VersionPayload` (`Present(cols)` or `Tombstone`). `MvStore::read(row, begin_ts)` implements the textbook snapshot-isolation visibility rule (`begin <= T < end`). `push_committed` validates monotonicity + caps the previous latest version's `end`; `push_in_flight` adds a placeholder version that's invisible to other readers until commit rewrites its `begin`.
- New [`JournalMode`](../src/mvcc/mod.rs) enum (`Wal` default, `Mvcc`); per-database setting on `Database`. `PRAGMA journal_mode = wal | mvcc;` toggles; `PRAGMA journal_mode;` returns the current value as a single-row, single-column result. `Connection::journal_mode()` reads the value through the public API. Switching `Mvcc → Wal` is rejected if the store carries committed versions (would silently strand them); v0 is intentionally strict.
- `Database` grows `mvcc_clock: Arc<MvccClock>` and `mv_store: MvStore` fields, allocated on every `Database::new` so the toggle to MVCC mode doesn't require a re-init step. Both are shared across every `Connection::connect` sibling.

The executor doesn't consult `MvStore` yet — that wiring lives in 11.4 alongside `BEGIN CONCURRENT` writes (the read-side and write-side are coupled: snapshot reads make sense only once the commit path is mirroring versions into the store). 11.3's contract is *the data structure + the toggle exist and round-trip*; 11.4 will turn the dial.

### Phase 11.4 — `BEGIN CONCURRENT` writes + commit-time validation *(planned, the meat)*

Expand Down
19 changes: 18 additions & 1 deletion docs/supported-sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,23 @@ PRAGMA auto_vacuum = OFF; -- disable; equivalent: NONE, 'OFF', 'NONE'

Out-of-range values (anything outside `0.0..=1.0`, `NaN`, `±∞`) and unknown identifiers like `WAL` / `FULL` are rejected with a typed error — the trigger never silently saturates or falls back to a default. The setting is per-`Connection` runtime state — it's not persisted in the file header, so every reopen starts at the default `Some(0.25)`.

### `PRAGMA journal_mode` (Phase 11.3, SQLR-22)

Selects the per-database concurrency model. `wal` (default) is the legacy WAL-backed pager every pre-Phase-11 build used; `mvcc` opts the database into multi-version concurrency control (Phase 11 — concurrent writes via `BEGIN CONCURRENT`).

```sql
PRAGMA journal_mode; -- read; renders a single-row "wal" or "mvcc"
PRAGMA journal_mode = mvcc; -- opt into MVCC for this database
PRAGMA journal_mode = wal; -- switch back (rejected if the MvStore
-- already carries committed versions)
```

Case-insensitive on both the pragma name and the value. Quoted values (`'mvcc'`) work; numeric values are rejected (the field is enum-shaped). Unknown modes return a typed error and don't disturb the existing setting.

The setting is **per-database** — every `Connection::connect` sibling sees the same value (the [open-question](concurrent-writes-plan.md) on per-connection vs per-database journal mode resolved to per-database for v0; revisit if a workload requires the per-connection variant). Reachable through the public API as `Connection::journal_mode() -> JournalMode`.

**What 11.3 changes:** the toggle is observable. The data structures backing MVCC (`MvccClock`, `MvStore`, the active-transaction registry) are allocated and round-trip through `PRAGMA`. **What 11.3 does *not* change yet:** the executor's read path. SELECTs still go through the legacy `tables → pager` path regardless of journal mode. End-to-end snapshot-isolation reads + `BEGIN CONCURRENT` writes land together in 11.4 — the read-side and write-side are coupled, and shipping one without the other would surface as wrong rows.

---

## Read-only databases
Expand Down Expand Up @@ -638,7 +655,7 @@ For context when you hit `NotImplemented`. See [Roadmap](roadmap.md) for when th

### Session / schema
- Multiple attached databases (`ATTACH DATABASE`, `DETACH DATABASE`)
- `PRAGMA` statements other than `auto_vacuum` (SQLR-13). The dispatcher is in place — adding a pragma is a single arm in `execute_pragma`. `journal_mode`, `synchronous`, `cache_size`, etc. are not yet wired up
- `PRAGMA` statements other than `auto_vacuum` (SQLR-13) and `journal_mode` (SQLR-22 / Phase 11.3). The dispatcher is in place — adding a pragma is a single arm in `execute_pragma`. `synchronous`, `cache_size`, etc. are not yet wired up
- `REPLACE INTO`, `INSERT OR IGNORE`, `INSERT OR REPLACE` (conflict-resolution clauses)

---
Expand Down
95 changes: 95 additions & 0 deletions src/connection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,17 @@ impl Connection {
self.lock().is_read_only()
}

/// Phase 11.3 — current journal mode. `Wal` (default) keeps every
/// pre-Phase-11 caller's behaviour. `Mvcc` is opt-in via
/// `PRAGMA journal_mode = mvcc;`. Per-database — every
/// [`Connection::connect`] sibling sees the same value.
///
/// Today this is observable but doesn't change query behaviour;
/// 11.4 wires `Mvcc` mode into the read/write paths.
pub fn journal_mode(&self) -> crate::mvcc::JournalMode {
self.lock().journal_mode()
}

/// Escape hatch for advanced callers — locks the shared `Database`
/// and hands back the guard. Not part of the stable API; will move
/// or change as Phase 10's MVCC sub-phases land.
Expand Down Expand Up @@ -1512,6 +1523,90 @@ mod tests {
assert_sync::<Connection>();
}

// -----------------------------------------------------------------
// Phase 11.3 — `PRAGMA journal_mode` round-trip
// -----------------------------------------------------------------

/// Fresh connections default to `wal` mode. The PRAGMA read form
/// renders the current value as a single-row, single-column table
/// the REPL can print.
#[test]
fn journal_mode_defaults_to_wal_and_renders_through_pragma() {
let mut conn = Connection::open_in_memory().unwrap();
assert_eq!(conn.journal_mode(), crate::mvcc::JournalMode::Wal);

// Read form returns "1 row returned." status (matching
// `auto_vacuum`'s shape).
let status = conn.execute("PRAGMA journal_mode;").unwrap();
assert!(
status.contains("1 row returned"),
"unexpected status: {status}"
);
}

/// `PRAGMA journal_mode = mvcc;` flips the per-database mode and
/// is observable through every sibling handle. The headline
/// per-database contract for Phase 11.3.
#[test]
fn journal_mode_set_to_mvcc_propagates_to_siblings() {
let mut primary = Connection::open_in_memory().unwrap();
let sibling = primary.connect();
assert_eq!(sibling.journal_mode(), crate::mvcc::JournalMode::Wal);

primary.execute("PRAGMA journal_mode = mvcc;").unwrap();
assert_eq!(primary.journal_mode(), crate::mvcc::JournalMode::Mvcc);
// Sibling sees the same value — proves the setting lives on
// the shared `Database`, not on the per-handle Connection.
assert_eq!(sibling.journal_mode(), crate::mvcc::JournalMode::Mvcc);

// Switch back is allowed because no MVCC versions exist yet
// (11.4 will populate the store).
primary.execute("PRAGMA journal_mode = wal;").unwrap();
assert_eq!(primary.journal_mode(), crate::mvcc::JournalMode::Wal);
assert_eq!(sibling.journal_mode(), crate::mvcc::JournalMode::Wal);
}

/// The set form is case-insensitive on both the pragma name and
/// the value (matching SQLite). Quoted values work too.
#[test]
fn journal_mode_pragma_is_case_insensitive() {
let mut conn = Connection::open_in_memory().unwrap();
conn.execute("PRAGMA JOURNAL_MODE = MVCC;").unwrap();
assert_eq!(conn.journal_mode(), crate::mvcc::JournalMode::Mvcc);
conn.execute("pragma journal_mode = 'wal';").unwrap();
assert_eq!(conn.journal_mode(), crate::mvcc::JournalMode::Wal);
}

/// Unknown modes return a typed error and don't disturb the
/// existing setting.
#[test]
fn journal_mode_rejects_unknown_value() {
let mut conn = Connection::open_in_memory().unwrap();
let err = conn
.execute("PRAGMA journal_mode = delete;")
.expect_err("unknown mode must error");
let msg = format!("{err}");
assert!(
msg.contains("unknown mode 'delete'"),
"unexpected error: {msg}"
);
// Setting wasn't disturbed.
assert_eq!(conn.journal_mode(), crate::mvcc::JournalMode::Wal);
}

/// Numeric values are rejected — `journal_mode` is enum-shaped.
/// SQLite accepts e.g. `journal_mode = 0` for OFF historically;
/// SQLRite stays explicit.
#[test]
fn journal_mode_rejects_numeric_value() {
let mut conn = Connection::open_in_memory().unwrap();
let err = conn
.execute("PRAGMA journal_mode = 0;")
.expect_err("numeric mode must error");
let msg = format!("{err}");
assert!(msg.contains("numeric"), "unexpected error: {msg}");
}

#[test]
fn prepare_cached_executes_the_same_as_prepare() {
let mut conn = Connection::open_in_memory().unwrap();
Expand Down
Loading
Loading