feat(TC-887): node wiring for at-rest keys and escrow recovery by samgbafa · Pull Request #327 · TinyCloudLabs/tinycloud-node

samgbafa · 2026-10-17T08:05:44Z

TC-887 S1, part 3 of 3. Stacked on #326 (keyring), which is stacked on #325 (escrow crate and CLI). Off by default: with no [at_rest] config the node does nothing new.

This head encrypts nothing. No storage path seals data yet. mode = "encrypt" and mode = "strict" are refused at boot, and /info never reports encrypting: true. The storage seal path (TC-887 S3) turns the capability on when it lands.

What it adds

[at_rest] config: mode (off | encrypt | strict), escrow_none, and [at_rest.escrow] with recipients, verified_fp, allow_software. Unknown fields are rejected.
AtRestRuntime, created before block storage opens. The keyring is installed after migrations.
- It has a seal-path capability, declare_seal_path, that storage will set.
- Until storage sets it, any mode other than off is refused at boot, and encrypting() stays false.
In a TEE, the escrow recipients and verified_fp must be literal values in the measured app-compose (checked against compose_hash). Anything else refuses the boot.
The escrow bundle is made or changed only at boot. A request never creates, activates or mirrors one. If the configured recipients differ from the active bundle's, the node refuses to boot.
A node whose root can't unseal its keys boots locked. Two holders recover it.
Key mirror on the block store, for S3 and the filesystem. Reads are capped at 64 KiB. FS writes are fsynced and use unique temporary names.
Endpoints:
- GET /at-rest/status, GET /at-rest/escrow, GET /at-rest/escrow/record, GET /at-rest/recovery?nonce=, POST /at-rest/recovery/shares;
- /info gains an at_rest summary;
- keyring errors answer 503 (a retry may help) or 500 (operator action), with a stable, non-secret reason code.
Share intake:
- rejected shares are limited per client;
- only shares that authenticate spend the per-boot budget.
Docs: docs/at-rest-escrow.md, node section, including the limits of multi-node recovery.

Tests

tests/at_rest_ceremony.rs covers, over HTTP:
- the ceremony test-unlock;
- locked-node recovery;
- encrypt and strict refused without a seal path;
- junk from one client not blocking genuine holders;
- boot refusals.
at_rest::tests: measured-compose literals.
routes::at_rest::tests: error statuses and reasons.
dstack::tests: reading app_compose from the guest agent's Info.
tests/at_rest_env.rs: compose env parsing.
Smoke-tested by hand with the real node binary, the CLI and stock age.

Not for merge or deploy yet.

- [at_rest] config (mode off|encrypt|strict, escrow recipients, verified_fp, allow_software, escrow_none), deny_unknown_fields; default off sets nothing up. - AtRestRuntime in two phases: created before block storage opens (for the storage seal path), keyring installed after migrations. It exposes mode(), encrypting() and provider() for storage. - Boot gate: encrypt/strict refuse to start when the keys are unlocked and the escrow gate fails; a node whose root can't unseal boots locked. - Key mirror on the block store: S3 (_tc/at-rest/) and filesystem ({datadir}/at-rest/). - Endpoints: GET /at-rest/status, /at-rest/escrow, /at-rest/escrow/record, /at-rest/recovery?nonce=; POST /at-rest/recovery/shares (body cap, rate limit, no share logging). /info carries an at_rest summary. - Tests: ceremony and recovery through the HTTP API; compose env parsing.

samgbafa · 2026-10-17T10:15:26Z

REQUEST_CHANGES: see TC-887 (security review, head 4872828660801882dec0c4a84980930370a24ce7).

samgbafa · 2026-10-17T11:07:32Z

REQUEST_CHANGES: see TC-887 (head 4872828)

…free junk shares (#327) - AtRestRuntime gains a seal-path capability (declare_seal_path) that storage sets once it seals writes. Until then setup refuses any mode other than off, and encrypting() stays false. - In a TEE, the escrow recipients and verified_fp must be literal values in the measured app-compose (tcb_info.app_compose from the guest agent, checked against compose_hash) equal to the effective config; anything else refuses the boot. - Rejected share submissions are limited per client (5, then 1/min) and never spend the per-boot budget. - Key mirror reads are capped at MAX_MIRROR_OBJECT; FS writes fsync the file and directory. - Docs: B1, measured escrow, pin semantics, share budget, mirror trust. - Tests: encrypt/strict refused without a seal path; junk from one client doesn't block genuine holders; measured-compose literals.

…s; docs (#327) - Keyring errors answer 503 when a retry may help and 500 when an operator must act, each with a stable, non-secret reason code. - With a mode other than off, an unlocked node whose gate is closed (for example a transient store error after recovery) re-runs it every 30 s. - FS mirror writes use a unique temporary name per write. - Boot logs say what passed (escrow verified vs escrow_none). - Docs: the escrow is made or changed only at boot and never by a request; changed recipients refuse the boot; escrow and record are restored after a database restore; HTTP error codes; recovery is per node; the FS mirror path; module docs describe this head (no seal path).

… TEE (#327) - In a TEE, every at-rest configuration is checked against the measured app-compose (check_tee_escrow). If the guest agent's Info has no readable tcb_info.app_compose, it doesn't parse, or the escrow config doesn't match its literals, the node refuses to start. - TC-1357 note at the share route's ClientIp. - Docs: the fail-closed check, the /Info shape to confirm on a live CVM, per-client limits best effort until TC-1357.

- With any at-rest setting, a dstack socket that is present but whose Info can't be read refuses the boot instead of booting as non-TEE, which would skip the measured-compose check and the escrow_none / allow_software refusals (refuse_unknown_tee; test). - Merges #326's non-secret gate reasons, so ShareAnswer.gate and the boot errors never carry raw store or mirror text.

samgbafa · 2026-10-17T13:02:21Z

REQUEST_CHANGES: see TC-887 (security review round 3, head c793c6df0b1975332e128cb767836b3717fc455c).

…ure map (#327) - S3 and FS mirror listings return only names that get accepts; the FS mirror lists regular files only. FS get never follows symlinks (O_NOFOLLOW), treats non-files as absent and bounds the read itself. - require_mirror = mode != off: with mode off, mirror contents never stop the node (with #326's skip-on-unreadable). - The per-client share-failure map is an LRU of 4096 clients, O(1) per call. - Tests: planted names, a directory and a /dev/zero symlink in the real app's datadir with mode off; FS mirror listing, symlink and size cap; 100k distinct clients stay bounded and fast.

samgbafa · 2026-10-17T13:59:58Z

REQUEST_CHANGES: see TC-887 (correctness round 3, head b3b896b22f92b0646b517c99f228960b2fb3898b)

…ig (#327) - FS mirror get opens with O_NOFOLLOW | O_NONBLOCK and requires a regular file (fstat on the handle) before reading, so a FIFO under an escrow or record name can't hang the boot (test). - at_rest.escrow.allow_resplit_before_verify passes through to the keyring's guarded pre-ceremony re-split. - Merges #326 (D1 fail-closed data keys, main with #341). - Docs: the re-split flag, dek_unavailable, the never-overwrite DEK rule.

samgbafa · 2026-10-17T14:42:02Z

REQUEST_CHANGES: see TC-887 (security review round 3 delta, head 7c0c4ce7d7d8cfd026c4509b1ef6c5b2d4c05ed4).

…wiring

…K holder fix (#327) - KeyMirror::delete for the S3 (DeleteObject) and filesystem (remove_file, absent is fine) mirrors, used only to retire an ARK that never protected anything. - Merges #326 (fresh-ARK re-split, retirement markers, S3's write-lock guards and prepare_read). - Docs: correcting a holder before the ceremony replaces the root key and makes old shares worthless; the dek_unavailable runbook; mirror write failures stop the boot with encrypt or strict.

…wiring

…anted-bundle runbook (#327)

samgbafa · 2026-10-17T17:42:39Z

APPROVE: see TC-887 (security review round 4, head 029c703dd3e16c472d47da6723d65d7c1bc0c923).

samgbafa · 2026-10-17T18:36:47Z

APPROVE: see TC-887 (correctness round 4, head 029c703dd3e16c472d47da6723d65d7c1bc0c923)

…wiring

samgbafa force-pushed the skgbafa/tc-887-s1-node-wiring branch from 88528c4 to e92bd43 Compare October 17, 2026 08:41

samgbafa force-pushed the skgbafa/tc-887-s1-keyring branch from 0ae6fe7 to 1a3e685 Compare October 17, 2026 08:47

samgbafa force-pushed the skgbafa/tc-887-s1-node-wiring branch from e92bd43 to 58ff6b9 Compare October 17, 2026 08:47

samgbafa force-pushed the skgbafa/tc-887-s1-keyring branch from 1a3e685 to 5bd7656 Compare October 17, 2026 09:11

samgbafa force-pushed the skgbafa/tc-887-s1-node-wiring branch from 58ff6b9 to 4872828 Compare October 17, 2026 09:11

This was referenced Oct 17, 2026

feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333

Draft

test(TC-887): keep the at-rest key mirror out of KV pairing; check it for raw keys #335

Merged

Merge commit '8f5e8ab' into skgbafa/tc-887-s1-node-wiring

1d44ae6

samgbafa mentioned this pull request Oct 17, 2026

feat(TC-887): seal KV blocks in the S3 store (S3c, stacked on #333) #336

Draft

samgbafa added 7 commits October 17, 2026 11:16

Merge commit '4eb6ebc' into skgbafa/tc-887-s1-node-wiring

ef1b629

Merge commit 'c4e4120' into skgbafa/tc-887-s1-node-wiring

44415ed

Merge commit '27f9430' into skgbafa/tc-887-s1-node-wiring

daef3cf

Merge commit '9b54bb7' into skgbafa/tc-887-s1-node-wiring

31c7df2

samgbafa added 3 commits October 17, 2026 13:10

Merge commit 'c296f66' into skgbafa/tc-887-s1-node-wiring

35ac72c

Merge commit '17e1091' into skgbafa/tc-887-s1-node-wiring

b3b896b

Merge commit '9f17af3' into skgbafa/tc-887-s1-node-wiring

c69a450

samgbafa added 5 commits October 17, 2026 15:58

Merge branch 'skgbafa/tc-887-s1-keyring' into skgbafa/tc-887-s1-node-…

484c502

…wiring

Merge branch 'skgbafa/tc-887-s1-keyring' into skgbafa/tc-887-s1-node-…

e9c2ec6

…wiring

Merge branch 'skgbafa/tc-887-s1-keyring' into skgbafa/tc-887-s1-node-…

f47fe70

…wiring

docs(TC-887): an ARK never gets a second bundle for other holders; pl…

029c703

…anted-bundle runbook (#327)

samgbafa added 2 commits October 17, 2026 18:41

Merge branch 'skgbafa/tc-887-s1-keyring' into skgbafa/tc-887-s1-node-…

448758b

…wiring

docs(TC-887): retirement markers never stop a mode-off boot (#327)

c1a040e

samgbafa mentioned this pull request Oct 18, 2026

fix: serve resolves provider and Auto node identities again (TC-1442) #359

Open

Navigation Menu

Repository navigation

feat(TC-887): node wiring for at-rest keys and escrow recovery - #327

feat(TC-887): node wiring for at-rest keys and escrow recovery#327
samgbafa wants to merge 22 commits into
skgbafa/tc-887-s1-keyringfrom
skgbafa/tc-887-s1-node-wiring

samgbafa commented Oct 17, 2026 •

edited

Loading

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

samgbafa commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it adds

Tests

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

samgbafa commented Oct 17, 2026 •

edited

Loading