ops: disable epoch_head backfill in the production compose

.github/workflows/docker.yml

@@ -658,7 +658,7 @@ jobs:
               raise SystemExit('::error::Deployment destination is not the fixed reviewed path')
           native = 'ghcr.io/tinycloudlabs/tinycloud-node'
           reviewed = native + '@sha256:4bdae4fc232d886bb56bfbce4ec62ed4b8fb4e1369a01f6afb73d06dd68c188f'
-          n0_hash = '6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e'
+          n0_hash = 'c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7'
           pre_n0_hash = 'df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91'
           pre_n0_template_hash = '28302d6761d841b7cc5c11314aed2d377735e6bbdcaf85972a752e03bc9fdc93'
           rollback = os.environ.get('ROLLBACK_PRE_N0') == 'true'

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 
+- Production compose disables the epoch_head backfill at boot; enable it later once DB CPU has headroom.
 - New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314). Rolling back to v1.20.x: the older node refuses to start while `seaql_migrations` lists this migration (`Migration file of version 'm20261017_000000_epoch_order_space_index' is missing`). Delete that row first, and keep the index: v1.20.x runs the same anti-join, and a later upgrade finds the valid index and does nothing.
 - Writes no longer scan a space's whole epoch history to find its head epochs (TC-1274). A new `epoch_head` table (migration `m20261018_000000_epoch_head`) caches each space's heads. Every write maintains it in the same transaction, under the space's existing sequence lock. A write now reads O(heads) rows, normally one, instead of running the `epoch`/`epoch_order` anti-join, and usually costs one more statement, an `UPDATE` of the cached head. The cache is used only when its newest `seq` is the space's newest. Otherwise that space falls back to the old anti-join once and the write repairs the cache, so epochs appended by older releases (a rolling deploy, or a rollback and upgrade) never produce wrong parents. Heads are taken in `id` byte order. Previously the anti-join returned them in no defined order. This affects only a merge after a pre-TC-732 fork, and nothing re-derives an epoch hash from its parents. The migration only creates the empty table, without foreign keys, so it takes no locks on existing tables. After the node starts serving, a gentle online backfill checks each space once. It runs in one short transaction per space, under that space's lock, at no more than `epoch_head_backfill.spaces_per_second` (default 2) and pauses after each space for at least as long as the space took. It stops at its first error with one log line, and logs progress every `progress_every` spaces (default 500). Turn it off with `[epoch_head_backfill] enabled = false` or `TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED=false`. Writes stay correct without it. Keep the `idx_epoch_order_space_parent_child` index: the fallback and the backfill use it. Rolling back to v1.20.x: as for `m20261017_000000_epoch_order_space_index`, the older node refuses to start until its rows are gone. Run `DELETE FROM seaql_migrations WHERE version IN ('m20261017_000000_epoch_order_space_index', 'm20261018_000000_epoch_head');` first. Keep the index and the `epoch_head` table: v1.20.x ignores the table. A later upgrade re-applies both migrations as no-ops (the index check and `CREATE TABLE IF NOT EXISTS`), and the `seq` check repairs every space the older node wrote to.
 - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2).

docker-compose.dstack-postgres.yaml

@@ -53,6 +53,12 @@ services:
       # headroom for admin/psql access, and raise it only alongside the
       # instance's own max_connections.
       TINYCLOUD_DATABASE__MAX_CONNECTIONS: "15"
+      # TC-1274: production boots with the epoch_head backfill off while the
+      # database is CPU-bound. Writes stay correct without it. Re-enable later
+      # by setting "true" (or removing this line); the pace keys are
+      # TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND / __PROGRESS_EVERY.
+      # Keep the double underscore: a single underscore is silently ignored.
+      TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false"
       # Required for every manual TC-780 deploy. Keep true through offline
       # verification; only the final, explicitly reviewed redeploy sets false.
       TINYCLOUD_DATABASE__WRITE_FENCE: "${SQL_IDENTITY_FENCE:?set true during cutover, false after verification}"

docs/releasing.md

@@ -294,8 +294,8 @@ gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \
 ```
 
 The served/OCI identity remains **1.20.6-dstack / e18578ff9162986ddd813259c60f73866e3e60d1**.
-The N0 raw lock **changed in #346 (TC-959)**, which passes the validated N2 correlation fields through the log sidecar:
-`6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`,
+The N0 raw lock **changed again in TC-1274**, which sets `TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false"` in the production compose (whichever of this and #347 lands second re-pins). It previously changed in #346 (TC-959), which passes the validated N2 correlation fields through the log sidecar; the hash below is the TC-1274 lock:
+`c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7`,
 from #346 final reviewed head.
 It supersedes the #309 lock `ca110b0bae6d5a06a1002fb80df3f71cffec11659c9f9dbfa5d72624735546f6`,
 from #309 reviewed head `44eeb20a1461ef66b5af492e3be4c42be2728b67`,

docs/runbooks/tc-959-n0-log-shipping.md

@@ -146,7 +146,7 @@ Keep these identities distinct:
 | --- | --- |
 | Before: verifier `docker_compose_file` / hash | Raw preserved pre-N0 compose: `df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91` |
 | Before: application `compose_hash` | `bfe9028a58b23238188aa7d70b82282a7cd6414cd83dcc47a3b295c6f64fe2f8`; a distinct reported identity, not the raw-file hash |
-| After: current profile-enabled raw compose | `6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`; bind this artifact to the final reviewed PR head before any promotion |
+| After: current profile-enabled raw compose | `c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7`; bind this artifact to the final reviewed PR head before any promotion |
 
 The raw after hash identifies the profile-enabled compose, not deployment approval. Record its final reviewed PR head and required CI evidence in the officer handoff. Do not infer a raw-file hash from the distinct application-reported compose hash.
 

scripts/ci/fixtures/tc959-n0-compose.yaml

@@ -53,6 +53,12 @@ services:
       # headroom for admin/psql access, and raise it only alongside the
       # instance's own max_connections.
       TINYCLOUD_DATABASE__MAX_CONNECTIONS: "15"
+      # TC-1274: production boots with the epoch_head backfill off while the
+      # database is CPU-bound. Writes stay correct without it. Re-enable later
+      # by setting "true" (or removing this line); the pace keys are
+      # TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND / __PROGRESS_EVERY.
+      # Keep the double underscore: a single underscore is silently ignored.
+      TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false"
       # Required for every manual TC-780 deploy. Keep true through offline
       # verification; only the final, explicitly reviewed redeploy sets false.
       TINYCLOUD_DATABASE__WRITE_FENCE: "${SQL_IDENTITY_FENCE:?set true during cutover, false after verification}"

tinycloud-node-server/src/main.rs

@@ -177,6 +177,42 @@ mod tests {
         assert_eq!(cfg.hooks.max_ticket_ttl_seconds, 777);
     }
 
+    #[test]
+    fn epoch_head_backfill_enabled_reads_only_the_double_underscore_key() {
+        let _lock = lock_env();
+        let single = "TINYCLOUD_EPOCH_HEAD_BACKFILL_ENABLED";
+        let double = "TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED";
+        let rate = "TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND";
+        let extract = || {
+            build_config_figment()
+                .extract::<config::Config>()
+                .expect("config should parse")
+                .epoch_head_backfill
+        };
+
+        let _s = EnvVarGuard::unset(single);
+        let _d = EnvVarGuard::unset(double);
+        let _r = EnvVarGuard::unset(rate);
+        assert!(extract().enabled, "backfill is on by default");
+
+        // The production compose sets this key (TC-1274).
+        let _d = EnvVarGuard::set(double, "false");
+        assert!(!extract().enabled);
+        drop(_d);
+
+        // The trap: a single underscore is a silent no-op.
+        let _s = EnvVarGuard::set(single, "false");
+        assert!(
+            extract().enabled,
+            "single-underscore form must not disable it"
+        );
+        drop(_s);
+
+        // The key the later ramp tunes.
+        let _r = EnvVarGuard::set(rate, "7");
+        assert_eq!(extract().spaces_per_second, 7);
+    }
+
     #[test]
     fn canonical_double_underscore_loads_storage_database() {
         let _lock = lock_env();