feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333)

[samgbafa]

TC-887, slice S3 part c: the S3 block store seals KV blocks. Stacked: S1 #327 → #329 (S3a) → #333 (S3b, FileSystem) → this PR. Review only the top four commits (c36b098 S3 sealing, f87eb1c double-remove test and escrow doc, 3268564 review fixes, f5cb0b0 review deltas). Off by default: with TINYCLOUD_AT_REST__MODE unset or off, the store behaves as before.

What changed

• tinycloud-node-server/src/storage/s3.rs
  • In encrypt and strict modes, persist writes TCE1 objects under the sealed name v1/<space tag>/<object id> (from feat(TC-887): sealed KV block helpers and plaintext sizes by space tag #329's sealed module). prepare_write (feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333's hook) loads the space's data key before the invoke lock.
  • read streams the whole object through the authenticated decrypting reader.
  • read_range makes two GETs: the header, then only the segments that cover the range. 416 reports the plaintext length.
  • Legacy objects stay readable in encrypt mode and are refused in strict mode, with the same read_plan as the FileSystem store.
  • While locked, writes and sealed reads return the at-rest 503 (feat(TC-887): sealed KV block helpers and plaintext sizes by space tag #329's mapping); legacy reads are still served.
  • remove deletes the sealed copy, or else the legacy object. total_size counts plaintext bytes; sealed bytes are counted by space tag, including by the boot scan.
  • S3Readable = Either<Body, Box<DecryptReader<Body>>>. The read, range and persist futures are boxed behind the async-trait methods, the same fix as feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333's stack-overflow change.
  • The object size check now uses HeadObject. remove used GetObjectAttributes without an attribute list, which S3 and MinIO reject (InvalidArgument: Invalid attribute name specified).
• storage/mod.rs: attach_at_rest attaches the runtime to the S3 store as well as the FileSystem store, and declares the seal path for both. docs/at-rest-escrow.md says both stores seal.
• storage/size.rs test: a double remove floors plaintext sizes at zero under both the tag and the legacy entry.
• tests/at_rest_s3_store.rs: a new test against a real S3 API. It is ignored by default, and the harness CI job runs it against its MinIO in a new step after the RED checks.

How it was tested

• tests/at_rest_s3_store.rs against the harness MinIO (2/2 pass):
  • Round trip: plaintext lengths 0, 17, one segment, one segment + 1, and 3 segments + 9. Each is stored only under its sealed name, as TCE1 at the exact ciphertext length, and reads back with Content::len equal to the plaintext length.
  • Ranges: every range across the segment edges matches the plaintext slice. A range past the end returns 416 with the plaintext total.
  • Quota: plaintext quota, and the boot scan on a freshly opened store.
  • Modes: a legacy object written in mode off stays readable in encrypt mode. While locked, writes and sealed reads return 503 and legacy reads are still served. Strict mode refuses legacy objects and serves sealed ones.
  • Tamper and remove: flipping one byte of a sealed object fails closed with InvalidData, before any plaintext is returned. Removing the block then makes it unreadable.
• cargo test -p tinycloud-node --lib storage:: 27/27 pass; did_resolution_single_entry 10/10 pass. cargo clippy -p tinycloud-node --tests and cargo fmt --check are clean.
• The full suite runs on CI (on a CI-only branch, since stacked PRs don't get CI); the run is linked in a comment below.

Review fixes (security 3c9d19cf, correctness d6a8c93b), commit 3268564

• Range reads pinned by ETag (correctness 1, security N3): the segment GET sends If-Match with the header GET's ETag. If the object was replaced, removed or shrunk in between, the read starts over once, then answers 503. It never falls back to the legacy name once a sealed header was read, and never gives a false integrity 500.
• contains checks only the name a write would use (INTERFACES §7). The S3 store implements prepares/prepare_read and uses LoggedDecrypt.
• New MinIO tests in tests/at_rest_s3_store.rs (6/6 pass locally against the harness MinIO):
  • temp-file staging with From/Suffix ranges at segment edges;
  • off after encrypt, and off with locked keys (503);
  • strict with locked keys (503);
  • contains per §7;
  • a concurrent-replace test: 200 alternating re-PUTs of two valid encryptions during range reads, with no wrong bytes and no integrity error.

Delta round (6d578564, 3f0b1aec), commit f5cb0b0

• R2 on S3: the read and range paths log integrity failures with the sealed-name locator.
• Concurrent-replace test: it now fails if any served body aborts mid-stream.
• Changelog: extended to the S3 store.
• Rebased onto S1 final, which now carries main's fix(TC-1354): size S3 block deletes with HeadObject #337. The duplicate object_size/delete from fix(TC-1354): size S3 block deletes with HeadObject #337 is dropped; feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336's copies are kept. rust.yml keeps both harness steps (s3_block_store and at_rest_s3_store), and remove keeps feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336's sealed-first version.
  • Against MinIO locally: at_rest_s3_store 6/6 and main's s3_block_store 1/1.

[samgbafa]

CI (stacked PR, so it ran on a CI-only branch): run https://github.com/TinyCloudLabs/tinycloud-node/actions/runs/39027726967 on ci/tc-887-s3c (f71b24f = d8dddd3 + the workflow_dispatch trigger). Every job passed. The harness job's new step ran at_rest_s3_store against MinIO: 2/2 passed.

[samgbafa]

Rebased onto S1 round 3 (#327 daef3cf). Exact-head CI on 79878f5 (dispatched with workflow_dispatch on the PR branch): run https://github.com/TinyCloudLabs/tinycloud-node/actions/runs/39032820048, all jobs green.

[samgbafa]

REQUEST_CHANGES: see TC-887 (head 79878f55a7a25672757a49e62c6151979033b275)

[samgbafa]

REQUEST_CHANGES: see TC-887 (head 79878f5)

[samgbafa]

APPROVE: see TC-887 (head 6b01237ed6d22b41f0b04d5f6b5b91ef53d4152c)

[samgbafa]

Review round (6b01237): exact-head CI run https://github.com/TinyCloudLabs/tinycloud-node/actions/runs/39045005162 — all jobs green, including the 6 MinIO tests in the harness job.

[samgbafa]

APPROVE: see TC-887 (head 6b01237)

[samgbafa]

APPROVE: see TC-887 (head dc5e68f)

[samgbafa]

Rebased onto S1 #327 029c703. Exact-head CI on dc5e68f: run https://github.com/TinyCloudLabs/tinycloud-node/actions/runs/39054266209, all jobs green.