feat(TC-1274): maintain epoch_head so writes stop scanning epoch history

CHANGELOG.md

@@ -2,7 +2,8 @@
 
 ## [Unreleased]
 
-- New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314).
+- New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314). Rolling back to v1.20.x: the older node refuses to start while `seaql_migrations` lists this migration (`Migration file of version 'm20261017_000000_epoch_order_space_index' is missing`). Delete that row first, and keep the index: v1.20.x runs the same anti-join, and a later upgrade finds the valid index and does nothing.
+- Writes no longer scan a space's whole epoch history to find its head epochs (TC-1274). A new `epoch_head` table (migration `m20261018_000000_epoch_head`) caches each space's heads. Every write maintains it in the same transaction, under the space's existing sequence lock. A write now reads O(heads) rows, normally one, instead of running the `epoch`/`epoch_order` anti-join, and usually costs one more statement, an `UPDATE` of the cached head. The cache is used only when its newest `seq` is the space's newest. Otherwise that space falls back to the old anti-join once and the write repairs the cache, so epochs appended by older releases (a rolling deploy, or a rollback and upgrade) never produce wrong parents. Heads are taken in `id` byte order. Previously the anti-join returned them in no defined order. This affects only a merge after a pre-TC-732 fork, and nothing re-derives an epoch hash from its parents. The migration only creates the empty table, without foreign keys, so it takes no locks on existing tables. After the node starts serving, a gentle online backfill checks each space once. It runs in one short transaction per space, under that space's lock, at no more than `epoch_head_backfill.spaces_per_second` (default 2) and pauses after each space for at least as long as the space took. It stops at its first error with one log line, and logs progress every `progress_every` spaces (default 500). Turn it off with `[epoch_head_backfill] enabled = false` or `TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED=false`. Writes stay correct without it. Keep the `idx_epoch_order_space_parent_child` index: the fallback and the backfill use it. Rolling back to v1.20.x: as for `m20261017_000000_epoch_order_space_index`, the older node refuses to start until its rows are gone. Run `DELETE FROM seaql_migrations WHERE version IN ('m20261017_000000_epoch_order_space_index', 'm20261018_000000_epoch_head');` first. Keep the index and the `epoch_head` table: v1.20.x ignores the table. A later upgrade re-applies both migrations as no-ops (the index check and `CREATE TABLE IF NOT EXISTS`), and the `seq` check repairs every space the older node wrote to.
 - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2).
 - CACAO/SIWE delegation ingestion normalizes ECDSA recovery IDs 0/1 to 27/28 before hashing and persisting the canonical DAG-CBOR bytes (TC-1071). Equivalent low-s signatures now share one delegation CID, including the CID used for revocation lookup. High-s signatures are rejected during decoding, matching the existing EIP-191 verifier and EIP-2 low-s convention; UCAN/JWS bytes remain unchanged. Existing stored delegation IDs are not rewritten.
 - Synchronize the standalone M1, W5 and load-signer Cargo locks with CACAO's `k256` dependency so `--locked` dependency resolution succeeds (TC-1071). Keep the load signer compatible with the SDK's default invocation lifetime.

tinycloud-core/src/db.rs

@@ -541,6 +541,44 @@ where
     pub async fn begin_graph_transaction(&self) -> Result<DatabaseTransaction, DbErr> {
         begin_write_transaction(&self.conn, chain_isolation_level(&self.conn)).await
     }
+
+    /// TC-1274 backfill: up to `limit` space IDs after `after`, in ID order.
+    pub async fn epoch_head_backfill_page(
+        &self,
+        after: Option<&SpaceId>,
+        limit: u64,
+    ) -> Result<Vec<SpaceId>, DbErr> {
+        let mut query = space::Entity::find().order_by_asc(space::Column::Id);
+        if let Some(after) = after {
+            query = query.filter(space::Column::Id.gt(SpaceIdWrap(after.clone())));
+        }
+        Ok(query
+            .limit(limit)
+            .all(&self.conn)
+            .await?
+            .into_iter()
+            .map(|space| space.id.0)
+            .collect())
+    }
+
+    /// TC-1274 backfill of one space: rewrite its cached head epochs from
+    /// history unless they are provably current. Takes the space's sequence
+    /// locks as a write does, so it serializes with that space's writers and
+    /// holds no other space. Returns whether it rewrote the cache.
+    pub async fn repair_epoch_heads(&self, space: &SpaceId) -> Result<bool, DbErr> {
+        let _sequence = self
+            .acquire_space_sequence_guards(std::slice::from_ref(space))
+            .await;
+        let _writer = match &self.writer_lock {
+            Some(lock) => Some(lock.lock().await),
+            None => None,
+        };
+        let tx = self.begin_graph_transaction().await?;
+        crate::kv_sync::lock_space_sequences(&tx, [space]).await?;
+        let repaired = crate::epoch_head::repair(&tx, &SpaceIdWrap(space.clone())).await?;
+        tx.commit().await?;
+        Ok(repaired)
+    }
 }
 
 impl<C, B, K> SpaceDatabase<C, B, K>
@@ -3344,34 +3382,18 @@ pub(crate) async fn transact<C: ConnectionTrait, S: StorageSetup, K: Secrets>(
             }
         }
 
-        // get 'most recent' epochs for each of the spaces
-        let mut most_recent = epoch::Entity::find()
-            .select_only()
-            .left_join(epoch_order::Entity)
-            .filter(
-                Condition::all()
-                    .add(epoch::Column::Space.is_in(event_spaces.keys().cloned().map(SpaceIdWrap)))
-                    .add(epoch_order::Column::Child.is_null()),
-            )
-            .column(epoch::Column::Space)
-            .column(epoch::Column::Id)
-            .into_tuple::<(SpaceIdWrap, Hash)>()
-            .all(db)
-            .await?
-            .into_iter()
-            .fold(
-                HashMap::new(),
-                |mut m: HashMap<SpaceIdWrap, Vec<Hash>>, (space, epoch)| {
-                    m.entry(space).or_default().push(epoch);
-                    m
-                },
-            );
+        // The current heads of each space (TC-1274): O(#heads) from the
+        // `epoch_head` cache, falling back to history only for a space whose
+        // cache is not provably current. The sequence locks taken above are
+        // held to commit.
+        let head_spaces: Vec<SpaceIdWrap> = event_spaces.keys().cloned().map(SpaceIdWrap).collect();
+        let mut heads = crate::epoch_head::current(db, &head_spaces, &max_seqs).await?;
 
         // get all the orderings and associated data
         let (epoch_order, space_order, event_order, epochs) = event_spaces
             .into_iter()
             .map(|(space, events)| {
-                let parents = most_recent.remove(&space).unwrap_or_default();
+                let parents = heads.parents.remove(&space).unwrap_or_default();
                 let epoch = epoch_hash(&space, &events, &parents)?;
                 let seq = max_seqs.remove(&space).unwrap_or(0);
                 Ok((space, (epoch, events, seq, parents)))
@@ -3450,6 +3472,18 @@ pub(crate) async fn transact<C: ConnectionTrait, S: StorageSetup, K: Secrets>(
                 TxError::EpochInsert(e)
             })?;
 
+        // Each space's only head is now the epoch just appended: its parents
+        // were all of the previous heads.
+        crate::epoch_head::advance(
+            db,
+            &heads,
+            space_order
+                .iter()
+                .map(|(space, (seq, epoch, _, _))| (SpaceIdWrap(space.clone()), *epoch, *seq))
+                .collect(),
+        )
+        .await?;
+
         // save epoch orderings
         if !epoch_order.is_empty() {
             epoch_order::Entity::insert_many(epoch_order)