feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333)

.github/workflows/rust.yml

@@ -355,6 +355,11 @@ jobs:
         (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
          cargo test --locked -p tinycloud-node --test s3_block_store -- --ignored)
 
+    - name: S3 block store with encryption at rest (against the harness MinIO)
+      run: |
+        (set -a; . "$TC887_RUN_DIR/harness.env"; set +a
+         cargo test --locked -p tinycloud-node --test at_rest_s3_store -- --ignored)
+
     - name: Harness logs
       if: failure()
       run: |

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## [Unreleased]
 
-- Encryption at rest for KV blocks, FileSystem store (TC-887 S3, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks are sealed (TCE1) under anonymous names and served back as plaintext; legacy blocks stay readable in `encrypt`, are refused in `strict`, and a node with locked keys answers 503 for sealed data. **Behaviour change on nodes with at-rest keys only:** a writing invocation is now authorized (and its spaces checked as hosted) before any key work, so a put to a space the node does not host, or without authority there, answers **401 instead of 404**; nodes without keys are unchanged. See `docs/at-rest/INTERFACES.md` §7.
+- Encryption at rest for KV blocks, FileSystem and S3 stores (TC-887 S3, off by default). With `at_rest.mode = "encrypt"`/`"strict"` and keys set up, new KV blocks are sealed (TCE1) under anonymous names and served back as plaintext; S3 range reads fetch the header and then only the covering segments, pinned to one object version. Legacy blocks stay readable in `encrypt`, are refused in `strict`, and a node with locked keys answers 503 for sealed data. **Behaviour change on nodes with at-rest keys only:** a writing invocation is now authorized (and its spaces checked as hosted) before any key work, so a put to a space the node does not host, or without authority there, answers **401 instead of 404**; nodes without keys are unchanged. See `docs/at-rest/INTERFACES.md` §7.
 - New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314).
 - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2).
 - CACAO/SIWE delegation ingestion normalizes ECDSA recovery IDs 0/1 to 27/28 before hashing and persisting the canonical DAG-CBOR bytes (TC-1071). Equivalent low-s signatures now share one delegation CID, including the CID used for revocation lookup. High-s signatures are rejected during decoding, matching the existing EIP-191 verifier and EIP-2 low-s convention; UCAN/JWS bytes remain unchanged. Existing stored delegation IDs are not rewritten.

docs/at-rest-escrow.md

@@ -146,7 +146,7 @@ The same settings as compose env: `TINYCLOUD_AT_REST__MODE`, `TINYCLOUD_AT_REST_
   - it keeps a copy of the sealed key, the bundle, the ceremony record and each space's wrapped data key in the block store (`_tc/at-rest/` in S3, `{datadir}/at-rest/_tc/at-rest/` on disk). Data-key objects are named by a keyed hash, not the space;
   - in this mode a copy that can't be written is logged and the boot goes on (nothing depends on it yet). With `encrypt` or `strict`, every boot-time copy must be written: a block store that refuses the write stops the boot;
   - nothing is encrypted.
-- **`encrypt` and `strict` need a block store with a seal path.** The FileSystem store seals KV blocks (TC-887 S3); with S3 block storage, `encrypt` and `strict` are refused at boot in this build, so the node won't claim to encrypt while it writes plaintext. With a seal path, `encrypt` or `strict` refuses to start unless the recipients equal the bundle's, `verified_fp` equals its EFP, and the stored ceremony record verifies under the key inside the TEE. A Static-root node with no recipients may use `escrow_none = true` instead, which acknowledges that no escrow exists.
+- **`encrypt` and `strict` need a block store with a seal path.** Both the FileSystem and the S3 store seal KV blocks (TC-887 S3), so both declare one; without it `encrypt` and `strict` are refused at boot, so the node never claims to encrypt while it writes plaintext. With a seal path, `encrypt` or `strict` refuses to start unless the recipients equal the bundle's, `verified_fp` equals its EFP, and the stored ceremony record verifies under the key inside the TEE. A Static-root node with no recipients may use `escrow_none = true` instead, which acknowledges that no escrow exists.
 - **In a TEE, the escrow recipients and `verified_fp` must be literal values in the measured compose.** With any at-rest config, the node reads its `app-compose.json` from the guest agent (`/Info`, `tcb_info.app_compose`), checks it against `compose_hash`, and refuses to boot unless `TINYCLOUD_AT_REST__ESCROW__RECIPIENTS` and `TINYCLOUD_AT_REST__ESCROW__VERIFIED_FP` are set there as literals (no `${…}`, no pass-through) equal to the effective config. Values from the encrypted environment, a file or anywhere else are refused. If the app-compose can't be read or parsed, the node refuses to boot. The `/Info` shape must be confirmed on a live CVM before the escrow ceremony.
 - **The escrow bundle is made or changed only at boot, never by a request.** Every endpoint below only reads it.
   - **An ARK never gets a second bundle for other holders.** The node refuses to boot (`escrow_holders_changed`) if the configured recipients differ from any existing bundle for its ARK: the active bundle, any stored escrow row, a ceremony record in the block store whose MAC verifies under the ARK, or a bundle object in the block store naming the ARK. This holds after a restore too, when the database lost its escrow rows. The block-store bundle object is unauthenticated, so it can only refuse, never be used. Changing a holder after the ceremony needs an ARK rotation (TC-1345).

docs/at-rest/INTERFACES.md

@@ -299,7 +299,9 @@ redacted.
 - **Accepted residual (header trust):** `Content-Length`/`Content-Range`/416 totals come from the
   unverified header. A forged header can only produce a wrong total or a body that aborts with
   no unauthenticated bytes (test `storage::sealed::tests::forged_header_length_aborts_without_extra_bytes`),
-  never wrong bytes.
+  never wrong bytes. On S3 a range read's segment GET is pinned to the header GET's ETag
+  (`If-Match`): if the object is replaced or removed in between, the read starts over once, then
+  answers 503. It never falls back to the legacy name once a sealed header was read.
 - **Quota:** plaintext bytes. Sealed objects are scanned by `space_tag` and joined to a space when
   the object-name key is unlocked (lazily, per lookup), so the first quota check after an unlock
   counts legacy + sealed. While locked, a space reports legacy bytes only (writes are 503 then).

tinycloud-node-server/src/storage/mod.rs

@@ -9,18 +9,14 @@ use std::sync::Arc;
 use crate::{at_rest::AtRestRuntime, BlockStores};
 use tinycloud_core::storage::either::Either;
 
-/// Links the opened block store to the node's at-rest state (TC-887).
-///
-/// The FileSystem store seals KV blocks when the mode and keys allow, so it
-/// declares a seal path. The S3 store does not seal yet and declares none,
-/// so `at_rest::setup` refuses a non-`off` mode with S3 storage rather than
-/// write plaintext.
+/// Links the opened block store to the node's at-rest state (TC-887). Both
+/// stores seal KV blocks when the mode and keys allow, so storage declares a
+/// seal path.
 pub fn attach_at_rest(stores: &mut BlockStores, runtime: Arc<AtRestRuntime>) {
+    runtime.declare_seal_path();
+    let handle = sealed::AtRestHandle::new(runtime);
     match stores {
-        Either::B(fs) => {
-            runtime.declare_seal_path();
-            fs.attach_at_rest(sealed::AtRestHandle::new(runtime));
-        }
-        Either::A(_) => {}
+        Either::A(s3) => s3.attach_at_rest(handle),
+        Either::B(fs) => fs.attach_at_rest(handle),
     }
 }