test(TC-887): keep the at-rest key mirror out of KV pairing; check it for raw keys

[samgbafa]

Part of TC-887 (encryption at rest): harness and doc follow-ups from the #331 review (F2, F1), reworked per review 4bcaddcf so that mirrors are checked against an allowlist. No product code changes.

What changed

• tests/at_rest_harness.rs: KV blocks and _tc/ objects are checked separately.

  • With at-rest keys on, the node writes wrapped-key mirrors to the block bucket under _tc/at-rest/.
  • Pairing, the size floor and compressibility apply to blocks only.
  • The marker, fingerprint and identity scans apply to every object, _tc/ names and contents included.

• An exact allowlist for mirror names. A _tc/ object is a mirror only if its name matches one of these exactly (numbers ≥ 1):

  • _tc/at-rest/ark-<10 digits>-<32 hex>.json
  • _tc/at-rest/escrow-<64 hex>.json
  • _tc/at-rest/record-<64 hex>.json
  • _tc/at-rest/dek-<32 hex>-<10 digits>.json

  Any other _tc/ object is RED.

• Every mirror kind is inspected, at boot as well as after the puts.

  • harness.sh up saves a boot snapshot of the bucket before it reports STATE.
  • The KV RED check checks every _tc/ object in both snapshots against the exact schema for its kind: ARK, DEK, escrow bundle or verification record. An unknown kind is RED.

• Typed schemas.

  • Each kind has an exact field set; strict serde structs refuse duplicate keys.
  • JSON integers only.
  • created_at is strict RFC 3339, at most 35 characters.
  • Ids are exact lowercase hex.
  • sealed, chain and wrapped are canonical base64 of exactly 60 bytes.
  • Escrow shares are armored age files. Records carry no TEE identity.

• A raw-key scan over every string and every JSON key.

  • Whitespace and common prefixes are stripped first.
  • RED: anything that decodes as hex, base64 or base64url (padded or not) to 32 bytes.
  • Also RED: an array of 32 byte values, and two 16-byte fields that together make 32.
  • Fields already checked by their exact schema encoding are exempt.
  • Findings never print the values.

• Name check. Every space id and wallet address the harness created is checked, in raw, lowercase, hex (text and bytes), base64 and base64url forms at all byte alignments.

• Re-derived key authentication.

  • The harness owns the node's Static test root. The test re-derives the key hierarchy independently of node code: W, the sealed ARK and its AAD, the chain, KEK_g, the DEK wrap AAD and the mirror tag.
  • Every sealed ARK and every wrapped DEK must authenticate, and generations must run 1..n with valid chains.
  • DEK mirror tags are mapped to the harness's spaces.
  • The escrow canary, EFP and record MAC are recomputed.
  • A raw key padded to 60 bytes doesn't authenticate.

• Self-tests (offline, in plain cargo test):

  • selftest_key_hierarchy_reference_vectors: HKDF matches the RFC 5869 vectors. The test's own sealed ARK and wrapped DEK open, and every single-bit flip of each is rejected; wrong root, id, generation, label, space, dek_id or KEK are each rejected.
  • selftest_key_mirror_accepts_s1_shaped_mirrors: a full ARK, chain, DEK, escrow and record set produces no findings.
  • selftest_key_mirror_flags_every_bypass: 74 negative cases, each required to produce its specific finding, plus loops over every identity form and over a marker or fingerprint inside a mirror.

• scripts/at-rest-harness/{harness.sh,README.md}: the boot snapshot, a spaces.log for mapping tags to spaces, and the residual: escrow share contents can't be checked without the holders' keys, so only their format is checked.

• docs/at-rest/INTERFACES.md: a status note. Key continuity and sealed KV storage aren't on main yet; they land with feat(TC-887): at-rest key escrow crate and ceremony CLI #325–feat(TC-887): node wiring for at-rest keys and escrow recovery #327 and feat(TC-887): sealed KV block helpers and plaintext sizes by space tag #329/feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333/feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336.

How it's tested

• cargo test -p tinycloud-node --test at_rest_harness: the 4 self-tests pass; the 4 live harness checks stay ignored, as before. cargo clippy … --test at_rest_harness -- -D warnings, cargo fmt --check and bash -n harness.sh are clean.
• Live, main's node: the boot snapshot is empty. test green passes, and test red is RED as expected, with 28 KV and 16 SQL findings.
• Live, a node built from the S3 sealing stack (feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336) in encrypt mode with escrow none:
  • the boot snapshot holds the real ARK mirror, and after the puts there are 5 _tc/ objects (the ARK and 4 DEKs);
  • with --expect kv=green,sql=red, KV is GREEN and SQL is RED;
  • the real mirrors passed the checks unchanged on the first run.
• Live tamper on that node: one real DEK's wrapped replaced with a raw key plus padding, and a stray _tc/other/blob added. KV is RED with exactly those 2 findings.
• Same binary with mode off: RED (20 KV, 16 SQL findings). No false GREEN.

[samgbafa]

REQUEST_CHANGES: see TC-887 (head d03f5c1)

[samgbafa]

APPROVE: see TC-887 (head fe18014)