test(TC-887): keep the at-rest key mirror out of KV pairing; check it for raw keys

docs/at-rest/INTERFACES.md

@@ -1,5 +1,9 @@
 # Encryption at rest: interfaces (TC-887)
 
+> **Status (TC-887):** on `main` today: the envelope, key types, `DataKeyProvider` and the
+> harness. Not on `main` yet: key continuity (`docs/at-rest-escrow.md`, `at_rest::keyring`,
+> node wiring) and sealed KV storage (§7). They land with PRs #325–#327 and #329/#333.
+
 Normative interface and format for `tinycloud_core::at_rest` and the node's sealed KV block
 storage. Plan: TC-887 (encryption at rest inside the TEE). Key continuity and escrow:
 `docs/at-rest-escrow.md`.

scripts/at-rest-harness/README.md

@@ -107,7 +107,7 @@ then insert (a WAL delta).
 |---|---|---|
 | `green_kv_put_reaches_minio` | every put round-trips through the node and adds a non-empty object to the bucket | pass |
 | `green_sql_write_reaches_postgres` | the writes round-trip, and `database_artifact` has a `payload` and a non-NULL `delta_payload` | pass |
-| `red_kv_blocks_are_sealed_in_minio` | object keys contain no space id, wallet address or blake3 fingerprint of the value (b64url padded/unpadded, base64 padded/unpadded, hex; raw digest and multihash); raw bytes hold no marker; each put's objects together are at least as large as the value; the 64 KiB object doesn't compress (zlib ratio > 0.95); every object of a put in one space shares no 32-byte run with every object of the same put in the other (equal object counts required) | **fail** |
+| `red_kv_blocks_are_sealed_in_minio` | object keys contain no identity of any space the harness created (space id or wallet address: raw, lowercase, hex, base64 and base64url at any alignment, of the text and of the bytes) and no blake3 fingerprint of the value (b64url padded/unpadded, base64 padded/unpadded, hex; raw digest and multihash); raw bytes hold no marker; each put's objects together are at least as large as the value; the 64 KiB object doesn't compress (zlib ratio > 0.95); every object of a put in one space shares no 32-byte run with every object of the same put in the other (equal object counts required); every `_tc/` key-mirror object passes the checks below | **fail** |
 | `red_sql_artifacts_are_sealed_in_postgres` | `payload` doesn't start with `SQLite format 3\0` and `delta_payload` doesn't start with the WAL magic; neither holds a marker, compresses (ratio > 0.95) or is smaller than the 64 KiB value it holds; the two spaces' payloads and deltas share no 32-byte run | **fail** |
 
 The RED checks re-run the green preconditions first, so a RED result is never
@@ -123,6 +123,46 @@ across writes once a per-write salt (SQLite's WAL salts) is mixed in.
 flagged, a salted deflate-9 blob is flagged by the size floor, and sealed-like
 random blobs pass.
 
+### At-rest key mirror (`_tc/…` objects)
+
+With at-rest keys on, the node writes control objects next to its blocks. The
+KV RED check reads them twice: from the boot snapshot `up` saves
+(`boot-snapshot/`, before any test write) and from the bucket after its puts.
+Every one must pass all of these, or the check is RED:
+
+- **Name.** Exactly one of `_tc/at-rest/ark-<generation, 10 digits>-<ark id,
+  32 lowercase hex>.json`, `escrow-<EFP, 64 hex>.json`, `record-<EFP, 64
+  hex>.json` or `dek-<tag, 32 hex>-<data key id, 10 digits>.json`, numbers ≥ 1.
+  Any other `_tc/` object is RED.
+- **Typed schema** per kind (S1's `MirroredArk`, `MirroredDek`,
+  `EscrowBundle`, `VerificationRecord`): exactly the fields, integers as
+  integers, `created_at`/`verified_at` strict RFC 3339 (≤ 35 chars), ids as
+  exact lowercase hex, `sealed`/`chain`/`wrapped` as canonical base64 of 60
+  bytes, escrow shares as armored age files and bech32 recipients, and no TEE
+  identity in a record (the harness node has none). Duplicate keys are refused.
+- **Raw-key scan** over every JSON key and every string outside those exact
+  fields: nothing that decodes (hex, base64, base64url, padded or not,
+  whitespace and prefixes like `0x`/`base64:`/data URLs removed) to 32 bytes,
+  no array of 32 byte values, no two fields of 16 bytes each.
+- **Content.** The same identity, fingerprint and marker scans as blocks,
+  over the name and the text.
+- **Authentic key material.** The check reads the node's Static root
+  (`secrets/node-static-secret`, never printed) and re-derives the hierarchy
+  from S1's spec: every sealed ARK opens under W, each older generation's
+  chain opens under the next, every wrapped DEK opens under KEK_g with its
+  AAD, its tag mapped to a space the harness signed in to (`spaces.log`). An
+  escrow bundle's canary and EFP and a record's MAC are recomputed. A raw key
+  padded to 60 bytes fails here.
+
+Not checkable without the escrow holders: the escrow share commitments and
+the age-encrypted share files themselves.
+
+`selftest_key_hierarchy_reference_vectors` (RFC 5869 vectors, seal/open
+round trips, every one-bit flip and wrong binding rejected),
+`selftest_key_mirror_accepts_s1_shaped_mirrors` and
+`selftest_key_mirror_flags_every_bypass` (one case per rule and per known
+bypass) run in plain `cargo test`.
+
 ## Not covered
 
 - The dstack simulator (`DSTACK_SIMULATOR_ENDPOINT`). No native simulator is

scripts/at-rest-harness/harness.sh

@@ -20,7 +20,7 @@ set -Eeuo pipefail
 ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
 SCRATCH_ROOT=${TC887_SCRATCH_ROOT:-$HOME/scratch/tc887}
 BIN_DIR=${TC887_BIN_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/tc887-harness/bin}
-STATE_SCHEMA=tc887-at-rest-harness/v3
+STATE_SCHEMA=tc887-at-rest-harness/v4
 
 # Pinned MinIO community releases (linux-amd64). dl.min.io no longer serves
 # community binaries (HTTP 410), so these come from the archived GitHub
@@ -322,6 +322,9 @@ TC887_S3_BUCKET=$BUCKET
 TC887_S3_REGION=$REGION
 TC887_S3_ACCESS_KEY_ID=$minio_user
 TC887_S3_SECRET_ACCESS_KEY=$minio_password
+TC887_NODE_ROOT_FILE=$run/secrets/node-static-secret
+TC887_SPACES_LOG=$run/spaces.log
+TC887_BOOT_SNAPSHOT=$run/boot-snapshot
 EOF
   # The node's whole environment; sourced inside the child, never on argv.
   # memory_threshold=0 keeps SQL databases file-backed, so writes after the
@@ -408,12 +411,54 @@ EOF
   write_state "$run" starting
   wait_http "$node_url/healthz" "$UP_NODE_PID" tinycloud-node "${TC887_NODE_READY_SECS:-180}"
 
+  # ---- boot snapshot: every object as the node left it at boot, before any
+  # test write (the at-rest key mirror writes ARK/escrow objects here).
+  MC_HOST_tc887="http://$minio_user:$minio_password@127.0.0.1:$minio_port" \
+    snapshot_bucket "$run"
+
   write_state "$run" up
   trap - EXIT INT TERM
   log "up: state file $run/state.json"
   printf '%s\n' "$run/state.json"
 }
 
+# Copy every object of the harness bucket into $1/boot-snapshot as N.bin, with
+# index.json mapping each object key to its file. Keys never become paths.
+# Needs MC_HOST_tc887 (credentials) in the environment.
+snapshot_bucket() {
+  local run=$1 dir=$1/boot-snapshot i=0 key count
+  mkdir -p "$dir"
+  MC_CONFIG_DIR=$run/mc "$BIN_DIR/mc" --quiet ls --recursive --json "tc887/$BUCKET" \
+    >"$dir/listing.jsonl" 2>>"$run/logs/mc.log"
+  while IFS= read -r -d '' key; do
+    MC_CONFIG_DIR=$run/mc "$BIN_DIR/mc" --quiet cat "tc887/$BUCKET/$key" \
+      >"$dir/$i.bin" 2>>"$run/logs/mc.log"
+    i=$((i + 1))
+  done < <(python3 - "$dir" <<'PY'
+import json, os, sys
+out = sys.argv[1]
+keys = []
+with open(os.path.join(out, "listing.jsonl")) as f:
+    for line in f:
+        if not line.strip():
+            continue
+        entry = json.loads(line)
+        if entry.get("status") != "success":
+            sys.exit("mc ls reported an error")
+        if entry.get("type") == "file":
+            keys.append(entry["key"])
+index = [{"key": k, "file": f"{i}.bin"} for i, k in enumerate(keys)]
+with open(os.path.join(out, "index.json"), "w") as f:
+    json.dump(index, f, indent=1)
+for k in keys:
+    sys.stdout.write(k + "\0")
+PY
+)
+  count=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' "$dir/index.json")
+  [[ "$count" == "$i" ]] || die "boot snapshot: listed $count objects, copied $i"
+  log "boot snapshot: $i object(s) in $dir"
+}
+
 # write_state RUN STATUS; PIDs come from UP_*, the rest from STATE_* env.
 write_state() {
   RUN=$1 STATUS=$2 SCHEMA=$STATE_SCHEMA \