From 3c8d65e78f315f3bc724aa38302168e9b8edba4c Mon Sep 17 00:00:00 2001 From: Sam Gbafa Date: Sun, 18 Oct 2026 09:54:08 +0000 Subject: [PATCH 1/2] ops: disable epoch_head backfill in the production compose (TC-1274) --- .github/workflows/docker.yml | 2 +- CHANGELOG.md | 1 + docker-compose.dstack-postgres.yaml | 6 +++++ docs/releasing.md | 4 +-- docs/runbooks/tc-959-n0-log-shipping.md | 2 +- scripts/ci/fixtures/tc959-n0-compose.yaml | 6 +++++ tinycloud-node-server/src/main.rs | 33 +++++++++++++++++++++++ 7 files changed, 50 insertions(+), 4 deletions(-) diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 79fa65af..241af2e0 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -658,7 +658,7 @@ jobs: raise SystemExit('::error::Deployment destination is not the fixed reviewed path') native = 'ghcr.io/tinycloudlabs/tinycloud-node' reviewed = native + '@sha256:4bdae4fc232d886bb56bfbce4ec62ed4b8fb4e1369a01f6afb73d06dd68c188f' - n0_hash = '6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e' + n0_hash = 'c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7' pre_n0_hash = 'df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91' pre_n0_template_hash = '28302d6761d841b7cc5c11314aed2d377735e6bbdcaf85972a752e03bc9fdc93' rollback = os.environ.get('ROLLBACK_PRE_N0') == 'true' diff --git a/CHANGELOG.md b/CHANGELOG.md index a88ce58a..c6e6c513 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [Unreleased] +- Production compose disables the epoch_head backfill at boot; enable it later once DB CPU has headroom. - New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314). Rolling back to v1.20.x: the older node refuses to start while `seaql_migrations` lists this migration (`Migration file of version 'm20261017_000000_epoch_order_space_index' is missing`). Delete that row first, and keep the index: v1.20.x runs the same anti-join, and a later upgrade finds the valid index and does nothing. - Writes no longer scan a space's whole epoch history to find its head epochs (TC-1274). A new `epoch_head` table (migration `m20261018_000000_epoch_head`) caches each space's heads. Every write maintains it in the same transaction, under the space's existing sequence lock. A write now reads O(heads) rows, normally one, instead of running the `epoch`/`epoch_order` anti-join, and usually costs one more statement, an `UPDATE` of the cached head. The cache is used only when its newest `seq` is the space's newest. Otherwise that space falls back to the old anti-join once and the write repairs the cache, so epochs appended by older releases (a rolling deploy, or a rollback and upgrade) never produce wrong parents. Heads are taken in `id` byte order. Previously the anti-join returned them in no defined order. This affects only a merge after a pre-TC-732 fork, and nothing re-derives an epoch hash from its parents. The migration only creates the empty table, without foreign keys, so it takes no locks on existing tables. After the node starts serving, a gentle online backfill checks each space once. It runs in one short transaction per space, under that space's lock, at no more than `epoch_head_backfill.spaces_per_second` (default 2) and pauses after each space for at least as long as the space took. It stops at its first error with one log line, and logs progress every `progress_every` spaces (default 500). Turn it off with `[epoch_head_backfill] enabled = false` or `TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED=false`. Writes stay correct without it. Keep the `idx_epoch_order_space_parent_child` index: the fallback and the backfill use it. Rolling back to v1.20.x: as for `m20261017_000000_epoch_order_space_index`, the older node refuses to start until its rows are gone. Run `DELETE FROM seaql_migrations WHERE version IN ('m20261017_000000_epoch_order_space_index', 'm20261018_000000_epoch_head');` first. Keep the index and the `epoch_head` table: v1.20.x ignores the table. A later upgrade re-applies both migrations as no-ops (the index check and `CREATE TABLE IF NOT EXISTS`), and the `seq` check repairs every space the older node wrote to. - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2). diff --git a/docker-compose.dstack-postgres.yaml b/docker-compose.dstack-postgres.yaml index 025f32b3..24da7bc4 100644 --- a/docker-compose.dstack-postgres.yaml +++ b/docker-compose.dstack-postgres.yaml @@ -53,6 +53,12 @@ services: # headroom for admin/psql access, and raise it only alongside the # instance's own max_connections. TINYCLOUD_DATABASE__MAX_CONNECTIONS: "15" + # TC-1274: production boots with the epoch_head backfill off while the + # database is CPU-bound. Writes stay correct without it. Re-enable later + # by setting "true" (or removing this line); the pace keys are + # TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND / __PROGRESS_EVERY. + # Keep the double underscore: a single underscore is silently ignored. + TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false" # Required for every manual TC-780 deploy. Keep true through offline # verification; only the final, explicitly reviewed redeploy sets false. TINYCLOUD_DATABASE__WRITE_FENCE: "${SQL_IDENTITY_FENCE:?set true during cutover, false after verification}" diff --git a/docs/releasing.md b/docs/releasing.md index bb596bca..396e90b5 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -294,8 +294,8 @@ gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \ ``` The served/OCI identity remains **1.20.6-dstack / e18578ff9162986ddd813259c60f73866e3e60d1**. -The N0 raw lock **changed in #346 (TC-959)**, which passes the validated N2 correlation fields through the log sidecar: -`6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`, +The N0 raw lock **changed again in TC-1274**, which sets `TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false"` in the production compose (whichever of this and #347 lands second re-pins). It previously changed in #346 (TC-959), which passes the validated N2 correlation fields through the log sidecar; the hash below is the TC-1274 lock: +`c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7`, from #346 final reviewed head. It supersedes the #309 lock `ca110b0bae6d5a06a1002fb80df3f71cffec11659c9f9dbfa5d72624735546f6`, from #309 reviewed head `44eeb20a1461ef66b5af492e3be4c42be2728b67`, diff --git a/docs/runbooks/tc-959-n0-log-shipping.md b/docs/runbooks/tc-959-n0-log-shipping.md index e8028fc1..39a944c4 100644 --- a/docs/runbooks/tc-959-n0-log-shipping.md +++ b/docs/runbooks/tc-959-n0-log-shipping.md @@ -146,7 +146,7 @@ Keep these identities distinct: | --- | --- | | Before: verifier `docker_compose_file` / hash | Raw preserved pre-N0 compose: `df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91` | | Before: application `compose_hash` | `bfe9028a58b23238188aa7d70b82282a7cd6414cd83dcc47a3b295c6f64fe2f8`; a distinct reported identity, not the raw-file hash | -| After: current profile-enabled raw compose | `6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`; bind this artifact to the final reviewed PR head before any promotion | +| After: current profile-enabled raw compose | `c33c23021589163b62a71ef245b7194717711922027629f155bdd5cff806b8c7`; bind this artifact to the final reviewed PR head before any promotion | The raw after hash identifies the profile-enabled compose, not deployment approval. Record its final reviewed PR head and required CI evidence in the officer handoff. Do not infer a raw-file hash from the distinct application-reported compose hash. diff --git a/scripts/ci/fixtures/tc959-n0-compose.yaml b/scripts/ci/fixtures/tc959-n0-compose.yaml index 025f32b3..24da7bc4 100644 --- a/scripts/ci/fixtures/tc959-n0-compose.yaml +++ b/scripts/ci/fixtures/tc959-n0-compose.yaml @@ -53,6 +53,12 @@ services: # headroom for admin/psql access, and raise it only alongside the # instance's own max_connections. TINYCLOUD_DATABASE__MAX_CONNECTIONS: "15" + # TC-1274: production boots with the epoch_head backfill off while the + # database is CPU-bound. Writes stay correct without it. Re-enable later + # by setting "true" (or removing this line); the pace keys are + # TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND / __PROGRESS_EVERY. + # Keep the double underscore: a single underscore is silently ignored. + TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED: "false" # Required for every manual TC-780 deploy. Keep true through offline # verification; only the final, explicitly reviewed redeploy sets false. TINYCLOUD_DATABASE__WRITE_FENCE: "${SQL_IDENTITY_FENCE:?set true during cutover, false after verification}" diff --git a/tinycloud-node-server/src/main.rs b/tinycloud-node-server/src/main.rs index 19471198..b2c45056 100644 --- a/tinycloud-node-server/src/main.rs +++ b/tinycloud-node-server/src/main.rs @@ -177,6 +177,39 @@ mod tests { assert_eq!(cfg.hooks.max_ticket_ttl_seconds, 777); } + #[test] + fn epoch_head_backfill_enabled_reads_only_the_double_underscore_key() { + let _lock = lock_env(); + let single = "TINYCLOUD_EPOCH_HEAD_BACKFILL_ENABLED"; + let double = "TINYCLOUD_EPOCH_HEAD_BACKFILL__ENABLED"; + let rate = "TINYCLOUD_EPOCH_HEAD_BACKFILL__SPACES_PER_SECOND"; + let extract = || { + build_config_figment() + .extract::() + .expect("config should parse") + .epoch_head_backfill + }; + + let _s = EnvVarGuard::unset(single); + let _d = EnvVarGuard::unset(double); + let _r = EnvVarGuard::unset(rate); + assert!(extract().enabled, "backfill is on by default"); + + // The production compose sets this key (TC-1274). + let _d = EnvVarGuard::set(double, "false"); + assert!(!extract().enabled); + drop(_d); + + // The trap: a single underscore is a silent no-op. + let _s = EnvVarGuard::set(single, "false"); + assert!(extract().enabled, "single-underscore form must not disable it"); + drop(_s); + + // The key the later ramp tunes. + let _r = EnvVarGuard::set(rate, "7"); + assert_eq!(extract().spaces_per_second, 7); + } + #[test] fn canonical_double_underscore_loads_storage_database() { let _lock = lock_env(); From 1286793d808f429f640e19b079e509cd991657a7 Mon Sep 17 00:00:00 2001 From: Sam Gbafa Date: Sun, 18 Oct 2026 09:56:20 +0000 Subject: [PATCH 2/2] fmt --- tinycloud-node-server/src/main.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tinycloud-node-server/src/main.rs b/tinycloud-node-server/src/main.rs index b2c45056..c2132cf3 100644 --- a/tinycloud-node-server/src/main.rs +++ b/tinycloud-node-server/src/main.rs @@ -202,7 +202,10 @@ mod tests { // The trap: a single underscore is a silent no-op. let _s = EnvVarGuard::set(single, "false"); - assert!(extract().enabled, "single-underscore form must not disable it"); + assert!( + extract().enabled, + "single-underscore form must not disable it" + ); drop(_s); // The key the later ramp tunes.