feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 
+- Encryption at rest for KV blocks, FileSystem store (TC-887 S3, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks are sealed (TCE1) under anonymous names and served back as plaintext; legacy blocks stay readable in `encrypt`, are refused in `strict`, and a node with locked keys answers 503 for sealed data. **Behaviour change on nodes with at-rest keys only:** a writing invocation is now authorized (and its spaces checked as hosted) before any key work, so a put to a space the node does not host, or without authority there, answers **401 instead of 404**; nodes without keys are unchanged. See `docs/at-rest/INTERFACES.md` §7.
 - New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314).
 - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2).
 - CACAO/SIWE delegation ingestion normalizes ECDSA recovery IDs 0/1 to 27/28 before hashing and persisting the canonical DAG-CBOR bytes (TC-1071). Equivalent low-s signatures now share one delegation CID, including the CID used for revocation lookup. High-s signatures are rejected during decoding, matching the existing EIP-191 verifier and EIP-2 low-s convention; UCAN/JWS bytes remain unchanged. Existing stored delegation IDs are not rewritten.

docs/at-rest-escrow.md

@@ -146,7 +146,7 @@ The same settings as compose env: `TINYCLOUD_AT_REST__MODE`, `TINYCLOUD_AT_REST_
   - it keeps a copy of the sealed key, the bundle, the ceremony record and each space's wrapped data key in the block store (`_tc/at-rest/` in S3, `{datadir}/at-rest/_tc/at-rest/` on disk). Data-key objects are named by a keyed hash, not the space;
   - in this mode a copy that can't be written is logged and the boot goes on (nothing depends on it yet). With `encrypt` or `strict`, every boot-time copy must be written: a block store that refuses the write stops the boot;
   - nothing is encrypted.
-- **`encrypt` and `strict` are refused at boot in this build.** Block storage has no seal path yet, so the node won't claim to encrypt while it writes plaintext; `/info` never shows `encrypting: true`. Storage turns the capability on when the seal path lands (TC-887 S3). From then on, `encrypt` or `strict` refuses to start unless the recipients equal the bundle's, `verified_fp` equals its EFP, and the stored ceremony record verifies under the key inside the TEE. A Static-root node with no recipients may use `escrow_none = true` instead, which acknowledges that no escrow exists.
+- **`encrypt` and `strict` need a block store with a seal path.** The FileSystem store seals KV blocks (TC-887 S3); with S3 block storage, `encrypt` and `strict` are refused at boot in this build, so the node won't claim to encrypt while it writes plaintext. With a seal path, `encrypt` or `strict` refuses to start unless the recipients equal the bundle's, `verified_fp` equals its EFP, and the stored ceremony record verifies under the key inside the TEE. A Static-root node with no recipients may use `escrow_none = true` instead, which acknowledges that no escrow exists.
 - **In a TEE, the escrow recipients and `verified_fp` must be literal values in the measured compose.** With any at-rest config, the node reads its `app-compose.json` from the guest agent (`/Info`, `tcb_info.app_compose`), checks it against `compose_hash`, and refuses to boot unless `TINYCLOUD_AT_REST__ESCROW__RECIPIENTS` and `TINYCLOUD_AT_REST__ESCROW__VERIFIED_FP` are set there as literals (no `${…}`, no pass-through) equal to the effective config. Values from the encrypted environment, a file or anywhere else are refused. If the app-compose can't be read or parsed, the node refuses to boot. The `/Info` shape must be confirmed on a live CVM before the escrow ceremony.
 - **The escrow bundle is made or changed only at boot, never by a request.** Every endpoint below only reads it.
   - **An ARK never gets a second bundle for other holders.** The node refuses to boot (`escrow_holders_changed`) if the configured recipients differ from any existing bundle for its ARK: the active bundle, any stored escrow row, a ceremony record in the block store whose MAC verifies under the ARK, or a bundle object in the block store naming the ARK. This holds after a restore too, when the database lost its escrow rows. The block-store bundle object is unauthenticated, so it can only refuse, never be used. Changing a holder after the ceremony needs an ARK rotation (TC-1345).

docs/at-rest/INTERFACES.md

@@ -65,10 +65,22 @@ pub trait DataKeyProvider: Send + Sync {
     /// by deliberate crypto-shred).
     async fn key(&self, space: &str, dek_id: DekId) -> Result<DataKey, DataKeyError>;
 
+    /// Before work that may read `space`'s sealed blobs under the invoke write
+    /// lock: put back any of the space's keys the database lost (from an
+    /// authenticated mirror). Never creates a key. Locked keys are not an error
+    /// here (the read itself answers 503).
+    async fn prepare_read(&self, space: &str) -> Result<(), DataKeyError>;
+
     /// NK for sealed S3 object names. Same value for the life of the deployment.
     fn object_name_key(&self) -> Result<ObjectNameKey, DataKeyError>;
 }
 
+/// Work under the invoke write lock and transaction runs inside
+/// `write_locked(..)`; there `current_key`/`key` make no key-store writes and
+/// answer `NotPrepared` for a key that would need creating or restoring.
+pub async fn write_locked<F: Future>(f: F) -> F::Output;
+pub fn in_write_lock() -> bool;
+
 #[derive(Debug, thiserror::Error)]
 pub enum DataKeyError {
     /// Node booted locked (W cannot unseal ARK). Callers map to HTTP 503.
@@ -81,6 +93,9 @@ pub enum DataKeyError {
     /// PG/S3/unwrap failure (incl. a wrapped DEK failing AEAD). Opaque, no key bytes.
     #[error("data key backend: {0}")]
     Backend(#[source] Box<dyn std::error::Error + Send + Sync>),
+    /// A key needed creating or restoring inside `write_locked`. HTTP 503.
+    #[error("the space's data key was not prepared before the write lock")]
+    NotPrepared,
 }
 ```
 
@@ -290,8 +305,29 @@ redacted.
   counts legacy + sealed. While locked, a space reports legacy bytes only (writes are 503 then).
   A sealed object with an invalid TCE1 length counts 0 bytes; the warning is rate-limited and
   carries only the length.
-- **Errors:** at-rest causes map explicitly to 503 (keys unavailable / key store) or 500
-  (integrity, unsupported, legacy refused in strict) on `/invoke` and the KV GET routes. The
-  general "store error → 401" catch-all is tracked separately (TC-1347).
+- **Errors:** at-rest causes map explicitly to 503 (keys unavailable, key store unavailable, key
+  not prepared) or 500 (integrity, unsupported, legacy refused in strict, inconsistent key store)
+  on `/invoke` and the KV GET routes. Bodies carry reason codes only; backend detail goes to the
+  server log. Integrity and unsupported failures are logged (typed, rate-limited, no names),
+  including ones that abort a response mid-stream. The general "store error → 401" catch-all is
+  tracked separately (TC-1347).
+- **Prepare before the write lock:** while keys are set up, a writing invocation is authorized and
+  its spaces' hosting checked *before* any key work, so a refused request creates no data key or
+  mirror object. Then every space it touches is prepared: a data key the database lost is restored
+  from its authenticated mirror (never created), and written spaces get their current key. Inside
+  the invoke write lock and transaction the provider only looks keys up (`at_rest::write_locked`);
+  a key that would need creating or restoring there is `NotPrepared` (503), never a key-store
+  write, which on SQLite would wait on the lock the request holds.
+- **Accepted residual (linkability during the transition):** in `encrypt`, a read of a
+  legacy-only block asks for the sealed name first, then the legacy name (`strict` probes the legacy
+  name to answer "legacy refused"). A storage provider's access logs can therefore link a
+  `space_tag` to its space id (and an object tag to its content hash) for every space read before
+  the backfill, for the life of the object-name key.
+- **Downgrade window until `strict`:** legacy objects are served without a hash check. In `off` or
+  `encrypt`, someone who can write the bucket and delete sealed objects can serve any bytes under the
+  legacy names they know. `strict` (after the backfill) closes this.
+- **`remove`:** deletes the sealed copy when keys can name it, else the legacy copy. When both
+  exist, the legacy copy survives, and while keys are locked only the legacy copy is removed. KV
+  delete is logical, so nothing calls it in production today; S5/GC must delete both names.
 - **Reserved names:** The key-continuity key mirror lives under `_tc/` in the block bucket (`datadir/at-rest` on
   FS). The block boot scan skips it.