[samgbafa]

TC-1274 PR B: a maintained epoch_head table, so a write finds its space's head epochs in O(heads) instead of running the epoch/epoch_order anti-join over the space's whole history. Design note (approved): ~/scratch/tc1274/epoch-head-table.md.

Based on main 1cbeb7d (#341 merged).

What changes

• transact: under the existing per-space sequence lock, it reads epoch_head (a PK range of about one row) instead of the anti-join.
  • It then records the new epoch as the space's only head, in the same transaction.
  • With one cached row (the normal case) that's one UPDATE; with none, an INSERT; with two or more (a backfilled fork), DELETE + INSERT.
• Self-validating cache:
  • It's used only when its newest seq + 1 equals the space's next seq, which transact already computes. Every epoch, including one appended by v1.20.x, gets seq = MAX(event_order.seq) + 1 under the same lock, so any epoch the cache doesn't know about makes the check fail.
  • When the check fails, that space falls back to the old anti-join once and the write repairs the cache.
  • Correctness doesn't depend on the backfill, or on every writer being new.
• Parent order: heads are taken in id byte order. The audit (in the design note, §0) found that nothing re-derives an epoch hash from its parents, and parents never leave the node.
• Migration m20261018_000000_epoch_head:
  • CREATE TABLE IF NOT EXISTS epoch_head (space, id, seq, PK(space, id)), with no FKs, so it takes no lock on any existing table.
  • It does no backfill.
  • down drops the table.
• Gentle online backfill ([epoch_head_backfill]):
  • On by default, at 2 spaces/s, with a pause after each space at least as long as that space took.
  • One short transaction per space, under that space's lock; it skips spaces whose cache is current.
  • It stops at the first error with one typed log line (SQLSTATE only, no text or space IDs), and logs progress every 500 spaces.
  • It can be turned off with enabled = false / TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED=false. Startup validates the pacing fields only when the backfill is enabled.
• feat(TC-1274): add epoch_order (space, parent, child) index migration #341 review nits (folded in here):
  • the in-progress check's pg_locks is filtered to the current database;
  • new negative test: a failed index plus a non-building SHARE UPDATE EXCLUSIVE holder still gives the DROP advice.

Parent-order audit: nothing recomputes or verifies an epoch hash from its parents

Searched: tinycloud-node at this PR's head (360751f), js-sdk origin/master 0431d61b (all packages/*/src, including packages/replica (TC-858), packages/cli, sdk-core, node-sdk, web-sdk), tcos 689ff21, tinycloud-app-kit 5059c5b, tinycloud-sdk-rs/wasm, tinycloud-auth.

• epoch_hash is defined at tinycloud-core/src/events/mod.rs:179. It's the only encoder of Epoch { parents, events } and has exactly one caller, transact, at tinycloud-core/src/db.rs:3397. It runs once per new epoch, and the result is stored as epoch.id. No code anywhere re-derives or verifies it.
• Parents never leave the node.
  • They're returned only as Commit.consumed_epochs (db.rs:208, set at :3593). The field has no reader: git grep consumed_epochs finds only those lines.
  • No route serves epoch_order.
  • The node has no peer or replica epoch sync: no libp2p or peer code touches epochs.
• Where the epoch hash leaves the node, it's an opaque stored value:
  • the kv/sync cursor anchor, hex-encoded and decoded as a value (tinycloud-core/src/kv_sync.rs:298, :308);
  • write-hook events, commit.rev.to_cid(0x55) (tinycloud-node-server/src/routes/mod.rs:2370, :2384);
  • the current_kv (seq, epoch, epoch_seq) tie-break on stored bytes.
• js-sdk only carries the hook event's epoch as an opaque string (packages/sdk-services/src/hooks/types.ts:48, HooksService.ts:1178). epoch_hash appears only in comments describing the node's TC-332 PK race (packages/sdk-core/src/space.ts:107, :114, and space.test.ts). The TC-858 replicas sync KV through kv/sync and never see parents. tcos, app-kit, sdk-rs/wasm and auth have no epoch-hash use.
• Is ORDER BY id consistent? There's no verifier to match.
  • The old anti-join had no ORDER BY, so on a multi-head space the parent order depended on the plan.
  • Linear spaces have one parent.
  • Existing epoch IDs are never recomputed.
  • Bytewise id order changes only the (new, opaque) hash of a future merge after a pre-TC-732 fork, and makes it deterministic.

Rollback

Rolling back to v1.20.x has the same requirement as #341. The older node refuses to start while seaql_migrations lists migrations it doesn't ship (sea-orm-migration get_migration_with_status). So first run DELETE FROM seaql_migrations WHERE version IN ('m20261017_000000_epoch_order_space_index', 'm20261018_000000_epoch_head');.

• Keep the index and the epoch_head table: v1.20.x ignores the table.
• On re-upgrade, both migrations are no-ops, and the seq check repairs every space the older node wrote to.
• This is in the CHANGELOG, on both entries.

Follow-up (not in this PR): backfill completion marker

Each boot re-runs the pass. Spaces that are already current cost one short transaction each (advisory lock, MAX(seq) probe, PK lookup) at no more than spaces_per_second. A marker would be safe across nodes, because lazy repair covers any space a rolled-back node later dirties. But it needs a new schema object, and its "done" state outlives rollbacks, so I've kept it out of this PR.

Round-trip budget (TC-951)

postgres_round_trip_budget: the budgets go up by exactly the measured amount: sign-in 75 → 77, kv put 37 → 38, kv get unchanged at 22. Reason, also in the test: the epoch_head cache write replaces the anti-join (TC-1274). - Measured on this machine, main 1cbeb7d gives 75 / 37 / 22 and this PR gives 77 / 38 / 22. Statements go 51 → 53 and 31 → 32.

• The +1 per epoch-writing commit is the cache UPDATE or INSERT; sign-in has two such commits, the host and session /delegates.
• The test now disables the backfill, which is background traffic like the webhook poll.

Measurements (native PG 18.6, synthetic: 1 space × 1,000,000 epochs + 20 × 50,000)

EXPLAIN (ANALYZE, BUFFERS) through PREPARE/EXECUTE, one warm-up then 5 runs; median of execution + planning time. Parallel workers off. Script: ~/scratch/tc1274/bench/{seed.sql,explain.py}.

Head lookup for the 1M-epoch space	Plan	custom median	generic median	buffers
(a) today's anti-join, PK only (prod before Sam's index)	Hash Right Join: seq scan of all of epoch_order (2M rows) and of epoch, 8-batch hash spilling to temp	1,823.6 ms	1,944.5 ms	78,917 (74,359 read, 31,301 temp)
(b) same query + idx_epoch_order_space_parent_child (#341)	Merge Left Join of two index-only range scans (idx_epoch_space_id, the new index), 0 heap fetches	559.2 ms	552.5 ms	38,699 (all hit)
(c) SELECT space, id, seq FROM epoch_head WHERE space IN ($1) (this PR), 200,021 cached heads	Index Scan on epoch_head_pkey, 1 row	0.648 ms	0.645 ms	4
MAX(event_order.seq) probe (unchanged, both before and after)	backward index probe	0.79 ms	0.74 ms	5

• (a) → (b): the index cuts the lookup by about 3.3× but stays O(history). 1M rows are still walked on every write, under the space lock.
• (b) → (c): about 860× faster, and independent of history.
• The generic plan (the sqlx prepared-statement case) matches the custom plan for (b) and (c).

Tests (RED first)

New in tinycloud-core/src/epoch_head.rs (SQLite and PG):

• heads_follow_every_write;
• old_writer_history_with_a_fork_is_detected: a v1.20.x-style writer appends 2 epochs and then a 2-way fork. The new writer takes exactly the anti-join's heads, in id order, and the cache ends at one row;
• repair_fills_upgraded_history_once: an empty cache on existing linear and forked history. The second repair is a no-op, and a write on the 2-row fork collapses it;
• PG only:
  • postgres_concurrent_writers_keep_exactly_one_head: 24 puts from two node instances on one space;
  • postgres_repair_concurrent_with_writers_ends_exact;
  • postgres_v1_20_6_upgrade_with_operator_index: a v1.20.6 schema with history and a fork, plus the operator's CREATE INDEX CONCURRENTLY, then db::migrate, backfill and next write;
• backfill_pages_every_space_in_id_order, cache_is_current_only_when_it_holds_the_newest_seq.

Also:

• The TC-951 assert_linear_history helper now also asserts that the cache holds exactly the chain's last epoch. That covers every existing concurrent-write test, including multi-space ones.
• tinycloud-node-server/src/epoch_head_backfill.rs, with paused time: every space is checked once across pages at the configured rate; the pause is at least the time each space took; it stops at the first error without retrying; it stops on shutdown.

RED evidence (native PG 18.6 and SQLite):

• RED 1, with transact not writing the cache: 8 of the new tests fail, and the failures are the cache assertions.
• RED 2, with a naive cache trusted without the seq check: {sqlite,postgres}_old_writer_history_with_a_fork_is_detected fail, because the parents are the stale cached head rather than the fork's two heads.

GREEN: cargo test -p tinycloud-core passes on SQLite and PG: 470 lib tests plus every integration target. - tinycloud-node: postgres_round_trip_budget was run locally with this change (77 / 38 / 22). The rest of the node-server suite, including the epoch_head_backfill unit tests and clippy, was cut off by the TC-1355 host pause; CI on this PR runs it.

• Unrelated load flakes: two tinycloud-core tests that don't touch epochs (meeting_legacy_guard SQLite "database is locked", and the sql::service 10 s timeout) failed once at load average 25. Both pass in isolation.

No merge or deploy is implied.

[samgbafa]

REQUEST_CHANGES: see TC-1274

Head 360751f04623d2c6a782342974b7de5a79771f77 (base main 1cbeb7d). This is a correctness review of concurrency, the migration and data integrity. I didn't run anything locally (the host is paused), so it relies on CI run 39038867763.

Blocker (one, trivial)

• CI clippy (workspace) fails: clippy::doc_lazy_continuation at tinycloud-core/src/epoch_head.rs:16-18. The cause is line 11, which starts with //! + 1` (TC-732). Rustdoc reads that as a list item, which also splits the `seq = MAX(event_order.seq) + 1` code span. Reflow the paragraph so that + 1 doesn't start a line. The logic needs no change.

Apart from that, everything I checked holds.

1. Seq-check proof: sound. No counterexample for any supported writer

• Old writers: every release since v1.20.1 takes the same tinycloud/kv-seq/v1 advisory lock and sets seq = MAX(event_order.seq) + 1 (v1.20.6 db.rs:3067, :3253). That includes v1.20.6 and v1.20.7.
  • Every epoch has at least one event_order row, because spaces with no eligible events are filtered out before epochs are built. So every locked append strictly raises MAX.
  • Rows written under the lock are exact when written. After any later append, max(head.seq) + 1 < next.
  • An old writer that appends once gives next = s + 2, against s + 1 from the cache. The two can't coincide.
• The proof would break if MAX could shrink. It can't:
  • nothing in production deletes or updates epoch, epoch_order or event_order (the only delete is in a kv_sync.rs test);
  • spaces are never deleted, so a history can't shrink and grow back.
• Forks and multi-head spaces: repair stores every anti-join head, and the check uses the largest seq among them. If the newest epoch in a legacy history isn't a head, the cache never validates and the anti-join runs until the next write rewrites the cache. That costs performance only, and it fixes itself.
• Seq gaps: these don't matter, because the check is equality against next.
• Two nodes: the database lock covers them. postgres_concurrent_writers_keep_exactly_one_head passes in CI.
• Rollback and roll-forward: stale rows survive CREATE TABLE IF NOT EXISTS, and old writers' appends raise MAX, so the stale rows are detected.
• Boundary (non-blocking): a pre-TC-732 writer (v1.17.x or older) takes no lock. It can append a same-seq sibling that the check can't see, and that sibling would stay a head that is never merged.
  • Parents feed only epoch_hash, so the only effect is on the DAG's shape.
  • A fleet that mixes those releases isn't supported.
  • Suggest stating "all writers ≥ v1.20.1" in the module doc and the CHANGELOG.

2. Concurrency: OK

• transact order: in-process guard → SQLite writer lock → BEGIN READ COMMITTED → pg_advisory_xact_lock → MAX → cache read → epoch insert → advance → commit.
  • It's all one transaction, with no window between computing next and writing the cache.
  • Under READ COMMITTED, each statement after the lock gets a new snapshot that includes the previous holder's commit.
• repair_epoch_heads: it takes the locks in the same order and asks for a single key, so it can't deadlock.
• SQLite: the write lock is reserved at BEGIN.
• MySQL: a FOR UPDATE on the space row, unsupported as before.

3. Single-UPDATE optimisation: correct

The row count is read under the lock, and advance always leaves exactly one head:

• One row: UPDATE, including a stale single row whose true heads are a fork. The old-writer fork test covers this.
• No rows: INSERT.
• Two or more rows: DELETE + INSERT. repair_fills_upgraded_history_once covers this.
• One to several happens only through repair's replace.

Optional: assert rows_affected == 1 on the UPDATE.

4. Backfill: OK

• It's idempotent: it skips any space whose cache is current.
• It runs one transaction per space under that space's lock, at most spaces_per_second.
• The pause is at least as long as the work, so the duty cycle is at most 50%.
• It stops at the first error and logs the SQLSTATE only.
• It honours shutdown, and the switch works.

Non-blocking:

• (a) The pass reruns on every boot of every node, indefinitely. That's cheap (the lock plus two reads per space), but it could stop once the cache is filled.
• (b) If statement_timeout hits on one huge stale space, the pass stops for every later space until the next boot. This is by design.
• (c) validate() rejects spaces_per_second = 0 even when enabled = false.

5. Migration: OK

• CREATE TABLE IF NOT EXISTS with a PK only and no FK takes no lock on existing tables, so it's safe under the boot's 30 s lock_timeout. down drops the table.
• SQLite: a BLOB id.
• MySQL: a VARBINARY(255) + VARCHAR(255) PK, within InnoDB's key-length limits.
• Rollback (non-blocking): like feat(TC-1274): add epoch_order (space, parent, child) index migration #341's m20261017, this migration makes a v1.20.x binary refuse to boot (Migration file … is missing) until its seaql_migrations row is deleted.
  • The CHANGELOG calls a rollback safe but leaves out that step. The repo's precedent is the "Roll-forward only" entry in CHANGELOG.md.
  • Data stays safe either way: keeping or dropping the table both work, because of IF NOT EXISTS plus the seq check.

6. Tests and budget: OK

• The RED cases are real:
  • Without the seq check, old_writer_history_with_a_fork_is_detected gets [first] instead of the two fork heads.
  • Without advance, the cache assertions fail, including the strengthened assert_linear_history.
• CI evidence: in test (workspace, PostgreSQL-backed) (job 117414107571, with TINYCLOUD_TEST_POSTGRES_URL set, so nothing was skipped), these all ran and passed:
  • the 11 epoch_head tests, PostgreSQL and SQLite;
  • the 4 backfill tests;
  • feat(TC-1274): add epoch_order (space, parent, child) index migration #341's new negative test;
  • postgres_round_trip_budget.
• Budget: each epoch commit swaps the anti-join for the cache read and adds one UPDATE, so +1 per commit. Sign-in has two such commits (+2), a kv put has one (+1), and a kv get has none. The 75→77 and 37→38 raises are justified.
• Parent-order audit confirmed: epoch_hash has one caller (db.rs:3397), and consumed_epochs has no reader.

[samgbafa]

APPROVE: see TC-1274

Head 447ad70584b3e32a3458ea8af10b7b1eec714329 (base main 1cbeb7d). This follows my REQUEST_CHANGES on 360751f. The new head is a fast-forward from it.

The new commit (447ad70)

• tinycloud-core/src/epoch_head.rs: only the module doc changed. It now puts `seq = MAX(event_order.seq) + 1` on one line, so no line starts with +. The code and tests in the file are unchanged, and CI clippy (workspace) now passes.
• EpochHeadBackfillConfig::validate(): returns Ok when enabled = false. The new test epoch_head_backfill_pacing_is_checked_only_when_enabled covers enabled and disabled, 0/s and progress_every = 0, through resolve_runtime_config, and it passes in CI.
• CHANGELOG rollback notes are accurate:
  • A v1.20.x node refuses to boot until the seaql_migrations rows for m20261017 and m20261018 are deleted.
  • The index and epoch_head can be kept, and v1.20.x never reads the table.
  • On the next upgrade both migrations re-apply as no-ops. m20261017 finds the valid index (PG) or uses IF NOT EXISTS, and m20261018 uses CREATE TABLE IF NOT EXISTS.
  • Any cache rows left from before the rollback fail the seq check once the old node has written, so the next write or the backfill repairs them.
• Completion marker: deferred to a follow-up. That's fine, because it would only make the backfill cheaper and correctness doesn't depend on it.

Local mutation check (host resumed)

I ran this on native PG 18 and SQLite at the default stack, with cargo test -p tinycloud-core --lib epoch_head.

Build	Result
Unmodified (360751f; its tinycloud-core code is the same as 447ad70's)	11 of 11 pass
RED 2: is_current trusts any non-empty cache (no seq check)	{sqlite,postgres}_old_writer_history_with_a_fork_is_detected fail at the "id order" assertion: the parents are the stale cached head, not the fork's two heads. The unit test also fails.
RED 1: advance never writes the cache	8 of 11 fail, all on cache assertions

Both RED claims in the PR body are reproduced exactly. postgres_repair_concurrent_with_writers_ends_exact still passes under RED 1, because the last repair can run after the last write. That test exercises repair running alongside writes, not cache maintenance, so it doesn't weaken the RED evidence. The patches were throwaway, in a scratch worktree. It's been removed, with its target dir and PG instance, and the host-slot claim is released.

CI

Every check on 447ad70 passes: 13 pass and 9 are skipped (deploy and release jobs). That covers clippy, fmt, build, test (workspace, PostgreSQL-backed) (all epoch_head tests, the new config test and postgres_round_trip_budget), the at-rest harness, m1-realdata and the Docker build.

My earlier findings on 360751f stand: the seq-check proof, concurrency, the single UPDATE, the backfill and the migration.

[samgbafa]

APPROVE: see TC-1274 (reviewed exact head 447ad70 merged onto main f613f11)