feat(TC-1274): seek KV list ranges on a COLLATE "C" index; hold the legacy-artifact fence scan for 60 s

CHANGELOG.md

@@ -9,6 +9,8 @@
   - **`current_kv` C-index warning codes.** The log sidecar maps the three constant warnings of the TC-1274 `current_kv` C-index migration to `current_kv_c_index_missing`, `current_kv_c_index_invalid` and `current_kv_c_index_wrong_shape` (TC-1274); until then they would ship as `unclassified`. This is part of the same regenerated Vector configuration.
   - **Ingress.** The `/admin/stats` label takes effect once the ingress image is rebuilt from `deploy/ingress` and its digest is pinned in the compose; until then the deployed ingress logs it as `<other>`.
 - Add `GET /admin/stats` (TC-1482), a read-only list of the lowercase wallet addresses that hold data on the node, for the scoreboard's protocol-user count. It is disabled (`404`, like an unknown route) unless the new `NODE_STATS_READ_TOKEN` is set to a value of at least 32 bytes (`openssl rand -hex 32`); a shorter value, or one equal to `TINYCLOUD_ADMIN_SECRET`, also disables it. The token is separate from `TINYCLOUD_ADMIN_SECRET` (neither opens the other's routes). An owner counts when they hold live KV in any space except `account`, or database artifacts in any space except `account` and `secrets`, except KV in the `public` space whose key starts with `.well-known/vault-` (vault metadata). Automatic sign-in records are excluded: the `account` space, the `secrets` table OpenKey's bootstrap creates, and the vault metadata `vault.unlock()` writes. App first-use databases (for example listen's `conversations`) count. So an owner who only signed in, or only signed in and unlocked the vault, is not listed. One SQL statement however many spaces exist. A result past 200,000 owners (or 400,000 distinct counted spaces) is a bare `507`, never a partial `200`. Neither the token nor any address is logged. The deployment workflow and compose files do not pass the variable yet.
+- New migration `m20261020_000000_current_kv_key_c_index` adds `idx_current_kv_space_deleted_key_c ON current_kv (space, deleted, (key COLLATE "C"))` on PostgreSQL (TC-1274). The KV list range select compares `key COLLATE "C"`, which the existing index, built in the column's own collation, cannot seek on under a non-`C` database collation such as production's `en_US.UTF-8`; it read and filtered every live key of the space. No query changed: the planner matches the new index. **The index is performance-only: the node boots and serves without it, so the migration never locks `current_kv` for a build and never fails the boot over the index.** It does nothing when a valid index of that exact shape exists; builds it when `current_kv` is empty (a fresh database); and, when it is missing on a non-empty `current_kv`, builds nothing, logs one `current_kv_c_index_missing` warning with constant text and keeps serving (KV lists stay slower), and is recorded as applied, so an index built later is used without a restart. An index of that name that is invalid, not ready or not live logs `current_kv_c_index_invalid`; one of another shape logs `current_kv_c_index_wrong_shape` at error level (drop it with `DROP INDEX CONCURRENTLY`); neither stops the boot. **Operator step before deploy** (PostgreSQL), to avoid the slow lists: build the index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` outside any transaction and check `pg_index.indisvalid`, `indisready` and `indislive`; the exact statements and recovery are in `docs/releasing.md`. SQLite and MySQL: no change. Rollback: `DROP INDEX CONCURRENTLY IF EXISTS idx_current_kv_space_deleted_key_c`, and, for a binary rollback, `DELETE FROM seaql_migrations WHERE version = 'm20261020_000000_current_kv_key_c_index'` first (an older node refuses to start with a migration it does not ship).
+- The SQL/DuckDB legacy-artifact fence scan is reused for 60 s instead of 1 s while the durable fence is off (TC-1274). It was an anti-join over every artifact row, re-run by the first SQL/DuckDB request each second. The durable and configured fences are still read on every request, so `fence on` is immediate; a stale answer can only delay a 503 (or lift one) by at most 60 s, and only around an artifact written by an older release or a restore. The argument is on `LEGACY_SCAN_TTL` in `database_migration.rs`.
 
 ## [1.21.2] - 2026-10-19
 

docs/releasing.md

@@ -57,6 +57,97 @@ a version into the `docker.yml` `image_version` input that `Cargo.toml` does not
 already claim. Both have happened (`0b48da2`, and the image tagged `1.10.0`
 built from a `1.9.0` tree); the `version-guard` job now blocks the second one.
 
+## Operator steps before a node deploy
+
+A migration that indexes a large PostgreSQL table cannot be built by the boot:
+the boot runs every pending migration in one transaction, so a plain
+`CREATE INDEX` holds a SHARE lock on the table, blocking its writes, for the
+whole build. For these, the operator builds the index online first. **Order:
+operator step, then the node deploy.**
+
+An index that only speeds up reads is **performance-only**: the node boots and
+serves without it, so its migration never builds it on a non-empty table and
+never fails the boot over it. It logs a typed warning and keeps serving; an
+index built later is picked up by the planner with no restart.
+
+### `idx_current_kv_space_deleted_key_c` (TC-1274, `m20261020_000000_current_kv_key_c_index`)
+
+Ships in the release after 1.21.2. It lets the KV list range select seek on
+its byte-order key bounds instead of reading and filtering every live key of
+the space. **Performance-only:** the node boots and serves without the index,
+and KV lists are slower until it exists. When it is missing on a non-empty
+`current_kv`, the node logs `current_kv_c_index_missing` (WARN) and keeps
+serving.
+
+1. On the PlanetScale primary, **outside any transaction** (the console, or
+   `psql` without `BEGIN`; never through the node or a migration runner):
+
+   ```sql
+   CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_current_kv_space_deleted_key_c ON current_kv (space, deleted, (key COLLATE "C"));
+   ```
+
+   Writes to `current_kv` continue during the build. It waits for transactions
+   that were open when it started, so it can sit in "waiting for old
+   transactions" while one is open; it does not block them. Build time grows
+   with the row count of `current_kv`. Measured locally on an unthrottled
+   PostgreSQL 16: 3.9 s for 1,000,000 rows (112 MB heap, 109 MB index). A
+   CPU-limited primary is slower; the index is about the size of the existing
+   `idx_current_kv_space_deleted_key`.
+
+2. Check the result. All three columns must be `t`:
+
+   ```sql
+   SELECT i.indisvalid, i.indisready, i.indislive FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid WHERE c.relname = 'idx_current_kv_space_deleted_key_c';
+   ```
+
+3. If the build was cancelled or failed, the index is left INVALID, and
+   `IF NOT EXISTS` would skip it. Drop it and redo step 1:
+
+   ```sql
+   DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c;
+   ```
+
+4. Deploy the node. At boot the migration reads `pg_index` for that exact name
+   and **never takes a lock on `current_kv` for a build and never fails the
+   boot over the index**:
+   - a valid, ready, live btree of that name on `current_kv` with the shape
+     above (columns `space, deleted, key`, the third in collation
+     `pg_catalog.C`, the first two in their own columns' collations, default
+     operator classes, ascending, no predicate,
+     expression or INCLUDE): **no DDL, no lock, no warning**, and the index
+     keeps its OID;
+   - no such index and `current_kv` **empty** (a fresh or test database): the
+     migration builds it, which is instant;
+   - no such index and `current_kv` **not empty**: nothing is built. The node
+     logs one `current_kv_c_index_missing` warning, boots and serves with slower
+     KV lists, and the migration is recorded as applied. Run step 1 whenever
+     convenient; the planner uses the index as soon as it is valid. A restart
+     is not needed;
+   - the right shape but INVALID, not ready or not live (a failed or still
+     running `CREATE INDEX CONCURRENTLY`): one `current_kv_c_index_invalid`
+     warning. The index is left alone. If no build is running, follow step 3
+     and redo step 1;
+   - the name with another shape (for example the existing
+     `(space, deleted, key)` shape, `COLLATE "POSIX"`, DESC, another operator
+     class, INCLUDE, a predicate, another table, or a non-index relation): one
+     `current_kv_c_index_wrong_shape` error-level log. The object is left
+     alone, the node serves, and lists are slower. Run
+     `DROP INDEX CONCURRENTLY idx_current_kv_space_deleted_key_c;` and then
+     step 1.
+
+   The log text is constant (no key, space or value; production logs are
+   public, TC-961). The log shipper maps the `warn`/`error` record to
+   `node_warning` / `node_error` with an unclassified code, so look for the
+   event name in the node's own container log.
+
+Rollback: `DROP INDEX CONCURRENTLY IF EXISTS idx_current_kv_space_deleted_key_c;`
+is safe at any time; the list range select goes back to filtering the space,
+and nothing else uses the index. A node image from before this migration
+refuses to start while `seaql_migrations` lists a migration it does not ship, so
+a binary rollback also needs
+`DELETE FROM seaql_migrations WHERE version = 'm20261020_000000_current_kv_key_c_index';`
+first. The migration's `down` keeps the index.
+
 ## Promoting to production (main-only)
 
 Production is built and promoted **only from `main`** (TC-940, 2026-10-13).

tinycloud-core/Cargo.toml

@@ -85,5 +85,6 @@ unicode-normalization = "0.1"
 sea-orm = { version = "1.1", features = ["runtime-tokio-rustls", "sqlx-sqlite"] }
 async-std = { version = "1", features = ["attributes"] }
 tokio = { workspace = true, features = ["test-util"] }
+tracing-subscriber = "0.3"
 iri-string.workspace = true
 proptest = "1"

tinycloud-core/src/database_migration.rs

@@ -438,6 +438,47 @@ async fn has_unmigrated_artifacts_exact<C: ConnectionTrait>(
         .any(|row| !is_digest_name(&row.2) && !registered.contains(&row)))
 }
 
+/// How long `EffectiveFenceCache` reuses a legacy-artifact scan (TC-1274).
+///
+/// This was 1 s, so while the durable fence was off every SQL/DuckDB request
+/// more than 1 s after the last one re-ran `has_unmigrated_artifacts`, an
+/// anti-join over every artifact row (~12% of the database's query CPU).
+///
+/// Why 60 s of staleness is safe:
+/// - **What the boolean gates.** `true` (a SQL/DuckDB artifact with a
+///   non-digest name and no `database_legacy_artifact` row) makes every
+///   SQL/DuckDB `/invoke`, and the share-email SQL store, answer 503 before
+///   any database is touched. `false` lets the request on to `resolve`, which
+///   reads `database_legacy_artifact` and `database_alias` live on every
+///   request, quarantines a registered legacy name or routes it through its
+///   alias. The cached value is only a tripwire for artifacts nobody has
+///   inventoried yet.
+/// - **The operator's fence is never cached.** The configured fence and the
+///   durable `database_identity_fence` row are read on every call before the
+///   cache. `fence on` takes effect at once, and `apply`, `set_alias` and
+///   `clear_alias` all require that row, so every migration step runs behind a
+///   live fence. `set_fence(false)` runs its own uncached scan.
+/// - **Stale `false` (the unsafe direction) needs a writer this binary does
+///   not have.** Artifact names come from `logical_name` (an N2 digest) or
+///   from an alias, which `resolve` returns only when its legacy row is
+///   registered; the legacy selector exists for N3's inventory only. So the
+///   answer turns from `false` to `true` only if an older release or a restore
+///   writes an unregistered legacy artifact. For up to 60 s requests for that
+///   path then resolve to its digest name (an empty database) instead of
+///   503. That is the 1 s window made longer, and it overwrites nothing. A
+///   digest artifact created in the window makes the later `apply` fail with
+///   `Collision` for the operator to resolve.
+/// - **Stale `true` is fail-closed.** After the last legacy artifact is
+///   registered, requests keep getting a retryable 503 for at most 60 s, and
+///   only while the durable fence is off.
+/// - **Restarts and instances.** The cache lives in the process
+///   (`SpaceDatabase`) and starts empty, so a restarted node scans on its
+///   first request. A migration run or the `tinycloud-sql-identity` CLI is
+///   another process and does not invalidate it; nothing above needs that.
+///   Nodes sharing one database each hold their own entry, each at most 60 s
+///   stale. Entries are aged from the start of their scan.
+pub const LEGACY_SCAN_TTL: Duration = Duration::from_secs(60);
+
 /// Caches only the expensive legacy-artifact inventory. The durable metadata
 /// fence is read on every check, so turning it on takes effect even while an
 /// earlier "unfenced" inventory result is cached.
@@ -451,7 +492,7 @@ impl Default for EffectiveFenceCache {
     fn default() -> Self {
         Self {
             legacy: tokio::sync::Mutex::new(None),
-            ttl: Duration::from_secs(1),
+            ttl: LEGACY_SCAN_TTL,
         }
     }
 }
@@ -1761,6 +1802,46 @@ mod tests {
         conn.close().await.unwrap();
     }
 
+    /// The scan is reused for `LEGACY_SCAN_TTL`, measured on the monotonic
+    /// clock from the start of the scan, and re-run after it. The entry's
+    /// timestamp is moved instead of sleeping the TTL.
+    #[tokio::test]
+    async fn effective_fence_reuses_a_legacy_scan_for_the_ttl_and_rescans_after_it() {
+        use sea_orm::Database;
+        use sea_orm_migration::MigratorTrait;
+
+        let conn = Database::connect("sqlite::memory:").await.unwrap();
+        crate::migrations::Migrator::up(&conn, None).await.unwrap();
+        let cache = EffectiveFenceCache::default();
+        let aged = |age: Duration| Instant::now().checked_sub(age).expect("monotonic clock");
+
+        assert!(!cache.check(&conn, false).await.unwrap());
+        artifact("sql", "test-space", "threads")
+            .insert(&conn)
+            .await
+            .unwrap();
+
+        cache.legacy.lock().await.as_mut().unwrap().0 =
+            aged(LEGACY_SCAN_TTL - Duration::from_secs(5));
+        assert!(
+            !cache.check(&conn, false).await.unwrap(),
+            "a scan younger than the TTL is reused"
+        );
+
+        cache.legacy.lock().await.as_mut().unwrap().0 =
+            aged(LEGACY_SCAN_TTL + Duration::from_secs(1));
+        assert!(
+            cache.check(&conn, false).await.unwrap(),
+            "a scan older than the TTL is re-run and sees the new artifact"
+        );
+        let (checked_at, fenced) = cache.legacy.lock().await.unwrap();
+        assert!(fenced);
+        assert!(
+            checked_at.elapsed() < Duration::from_secs(5),
+            "the re-run is timestamped now"
+        );
+    }
+
     #[tokio::test]
     async fn effective_fence_caches_legacy_scan_but_observes_metadata_fence_live() {
         use sea_orm::Database;