feat(TC-887): at-rest keyring, key store and migration by samgbafa · Pull Request #326 · TinyCloudLabs/tinycloud-node

samgbafa · 2026-10-17T07:17:46Z

TC-887 (S1, PR 2 of 3). Adds the at-rest key hierarchy and the DataKeyProvider implementation to tinycloud_core::at_rest, plus its migration. Nothing in the node constructs the keyring yet (that is #327), so the only runtime effect is three new empty tables. Merge only after the release cut, and after #325.

Stack: based on #325 (skgbafa/tc-887-s1-escrow), which includes current main (#320, #322 and later). Updates are merged in, not rebased. The one-line explicit key type in envelope.rs ((&*k_obj).into()) is needed here because the escrow crate brings generic-array 1.x into tinycloud-core's graph. Rust CI is dispatched on this branch's head.

What's added

at_rest/hierarchy.rs:
- W = HKDF(root);
- the ARK sealed under W, with AAD over (ark_id, generation, W label);
- generation chaining, so the newest ARK opens older generations;
- KEK, NK and DEK-mirror tag derivation;
- DEK wrap with length-prefixed AAD over (space, dek_id, generation, ark_id), as agreed in INTERFACES.md;
- key bytes kept only in zeroizing buffers (DataKey/ObjectNameKey::from_zeroizing).
at_rest/store.rs:
- row access with insert-or-ignore followed by a re-read;
- rotation in one transaction;
- escrow activation serialized per generation;
- the KeyMirror trait, with versioned mirror objects (sealed ARK rows, escrow bundles, ceremony records, wrapped DEKs) and a 64 KiB read cap;
- MemoryMirror for tests.
at_rest/keyring.rs: AtRestKeyring.
- Boot:
  - unlock from the newest generation that opens under W with a chain to generation 1;
  - store mirrored rows only once they authenticate that way, or (locked) once a recovered ARK opens them;
  - skip forged, junk, gapped or oversized mirror objects with a warning;
  - rebuild a missing chain under the same root, and clear a stale chain on the newest generation.
  - It refuses to provision over any existing key, escrow or DEK row, over any mirror object, or after a store or mirror error.
  - A different root gives a locked keyring.
- Escrow (made or changed only at boot, after a rotation or after a recovery, never by a request):
  - recipients that differ from the active bundle's refuse the boot (EscrowHoldersChanged);
  - with a pinned verified_fp, only the pinned bundle is used (re-activated from the database or mirror), otherwise EscrowPinned;
  - unpinned, a mirrored bundle whose mirrored record verifies under the ARK is restored, otherwise a new one is made;
  - a missing ceremony record is restored from the mirror when its MAC verifies.
- DataKeyProvider:
  - current_key creates a DEK, converges with concurrent callers, and mirrors the DEK before serving it;
  - key never creates one, and restores a missing DEK from the mirror only if its wrap authenticates for that space;
  - object_name_key comes from ARK generation 1;
  - a locked keyring answers Locked.
- Encrypt gate (gate, read-only):
  - checks the recipients and recipient policy, the pinned EFP, and an ARK-MACed verification record;
  - store and parse failures are reported as Unavailable.
- Share intake (submit_share):
  - commitment-checked, three slots;
  - only authenticated shares spend the per-boot budget;
  - unlocked node: the ceremony test-unlock;
  - locked node: recovery with two shares, then a reseal under the new root. Every older generation must open by its chain.
- rotate_ark: a library function.
  - A node that loses a concurrent rotation gets Conflict.
  - A stale chain gives an explicit, retryable Inconsistent.
  - The operator tooling is TC-1345.
- Errors: AtRestError::is_transient() and reason() give stable, non-secret reason codes.
Models and migration: m20261017_010000_at_rest_keys (ordered after feat(TC-1274): add epoch_order (space, parent, child) index migration #341's m20261017_000000_epoch_order_space_index) creates at_rest_key, at_rest_escrow (one active bundle per generation) and space_data_key. down refuses while the tables have rows.
Dependencies (tinycloud-core): hkdf, lru and parking_lot (already in the lock), and tinycloud-at-rest-escrow. Dev-only: age. The separate test workspaces' lockfiles are refreshed.

Tests (cargo test -p tinycloud-core --lib at_rest: 68 pass with S2's envelope tests and S3's write-lock test, 1 of them on PostgreSQL; clippy -Dwarnings clean):

Provisioning and reopening serve the same keys. Nothing is created without provisioning.
DEK handling:
- DEKs converge across concurrent callers and across two keyrings on one DB;
- a copied wrapped DEK fails;
- an unknown id is UnknownKey;
- DEKs survive a restore to before they were created;
- a different genuine DEK under a mirrored id is refused;
- another space's mirrored DEK is never stored or overwritten (that space fails closed);
- a mirrored DEK under a generation that didn't load is never replaced, and comes back once the generation loads (the review's D1).
Locked and unlocked behaviour:
- a new root boots locked;
- two shares unlock it without a restart;
- the old root is then locked out.
Mirror trust:
- a forged generation, a gap and junk never lock or stop a healthy node;
- an empty database takes back only authenticated rows, under the same root and under a new root;
- only junk fails closed;
- objects the mirror can't read (odd names, directories, symlinks) never stop a node with a healthy database, but stop one whose database is empty;
- mirror seal bytes are never stored;
- with require_mirror off (mode off), a mirror that can't be written never stops the node.
Recovery and chains:
- a locked node trusts only the pinned bundle;
- a missing or foreign chain is rebuilt under the same root;
- recovery fails with a reason on a broken chain or a missing generation;
- a stale node can't rotate over another node's rotation;
- a stale newest chain is cleared.
Escrow:
- two nodes with different holders never flip the bundle, and reads write nothing;
- pinned and unpinned restores bring back the bundle and record, with no new ceremony;
- a forged mirrored record proves nothing;
- a pinned escrow is never re-split;
- a wrong holder can be fixed (allow_resplit_before_verify) only before the ceremony and before any data key (database or mirror). The fix makes a fresh ARK and retires the old one, so old shares rebuild only the retired key, and a database restored to before the fix never loads it;
- a forged retirement marker never drops a live key;
- under S3's write lock, keys are only looked up (NotPrepared), and prepare_read restores a lost DEK without creating one.
Gate refusals: not pinned, pin mismatch, not verified, a forged record, an unreadable record (Unavailable), changed recipients, software holders without the opt-in, self-host without escrow_none.
Store conflicts keep the first row (SQLite and PostgreSQL). Concurrent activations on PostgreSQL leave one active bundle. Junk never spends the share budget. Migration down refuses with rows.

samgbafa · 2026-10-17T07:35:38Z

REQUEST_CHANGES: see TC-887 (security review of head 2a938c7; also covers c338a42)

samgbafa · 2026-10-17T07:35:56Z

REQUEST_CHANGES: see TC-887 (correctness review of head c338a42, reviewed at 2a938c7 plus the c338a42 delta; Linear comment defc9998)

… round 2 (#326) - Rotation writes generation n+1 and generation n's chain in one transaction, with the chain only on a NULL chain; a node that lost the race writes nothing, reloads, and gets AtRestError::Conflict. - Every older generation must open by its chain. At boot under the same root a missing or foreign chain is rebuilt from W and logged; under a new root (recovery) it is an error. Mirrored generations or chains the database lacks are restored at boot. unlock_with checks 1..n (no panic). - One active escrow bundle per generation: activate_escrow locks the generation's key row (FOR UPDATE), plus a partial unique index on PG and SQLite; active_escrow orders deterministically. - DEK rows are mirrored (named by a keyed tag, not the space) before a key is served, restored on a database miss, and a different mirrored DEK with the same id is refused rather than overwritten. - DataKey/ObjectNameKey::from_zeroizing (pub(crate)); the hierarchy no longer copies DEK/NK bytes out of zeroizing buffers. - GateRefusal::Unavailable for store/mirror/parse errors; stored bundles and records go through the versioned parsers; mirror objects carry a version. - Provisioning is refused over escrow rows or any _tc/at-rest/ object; the provisioned log names only the winning ark_id. - Migration down refuses while key rows exist. - Deterministic store-conflict tests (SQLite and PG), multi-thread PG test with concurrent re-splits; excluded test lockfiles refreshed.

Adds the S1 key hierarchy to tinycloud_core::at_rest. Nothing in the node constructs it yet. - hierarchy: W = HKDF(root), ARK sealed under W with AAD (ark_id, generation, W label), generation chaining (newest ARK opens older ones), KEK/NK derivation, per-space DEK wrap with length-prefixed AAD (space, dek_id, generation, ark_id). Key bytes only live in zeroizing buffers. - store: at_rest_key, at_rest_escrow and space_data_key rows (insert-or-ignore + re-read), and the KeyMirror trait for sealed-ARK and escrow copies next to the blocks (in-memory mirror for tests). - keyring: AtRestKeyring implements DataKeyProvider (current_key creates and converges, key never creates, object_name_key from ARK gen 1). Boot opens or provisions the ARK, restores rows from the mirror, never provisions over a store/mirror error or orphaned data keys, and boots locked when the root can't unseal. Escrow is kept to the configured holders (re-split on change, old bundles kept), the encrypt gate checks recipients, policy, pinned EFP and an ARK-MACed verification record, and the per-boot share intake runs the ceremony test-unlock (all three pairs) or locked-node recovery (two shares, reseal under the new root). Intake is commitment-checked, slot-capped and rate limited per boot. - migration m20261017_000000_at_rest_keys.

… round 2 (#326) - Rotation writes generation n+1 and generation n's chain in one transaction, with the chain only on a NULL chain; a node that lost the race writes nothing, reloads, and gets AtRestError::Conflict. - Every older generation must open by its chain. At boot under the same root a missing or foreign chain is rebuilt from W and logged; under a new root (recovery) it is an error. Mirrored generations or chains the database lacks are restored at boot. unlock_with checks 1..n (no panic). - One active escrow bundle per generation: activate_escrow locks the generation's key row (FOR UPDATE), plus a partial unique index on PG and SQLite; active_escrow orders deterministically. - DEK rows are mirrored (named by a keyed tag, not the space) before a key is served, restored on a database miss, and a different mirrored DEK with the same id is refused rather than overwritten. - DataKey/ObjectNameKey::from_zeroizing (pub(crate)); the hierarchy no longer copies DEK/NK bytes out of zeroizing buffers. - GateRefusal::Unavailable for store/mirror/parse errors; stored bundles and records go through the versioned parsers; mirror objects carry a version. - Provisioning is refused over escrow rows or any _tc/at-rest/ object; the provisioned log names only the winning ark_id. - Migration down refuses while key rows exist. - Deterministic store-conflict tests (SQLite and PG), multi-thread PG test with concurrent re-splits; excluded test lockfiles refreshed.

…on by name The test rolled back one migration assuming it was the newest; with m20261017_000000_at_rest_keys after it, locate it by name and roll back it and every later migration.

samgbafa · 2026-10-17T10:15:25Z

REQUEST_CHANGES: see TC-887 (security review round 2, head 5bd76560438099fbaf132fb8da68918579c368f9).

…t; free junk shares (#326) - Boot collects ARK candidates per generation (database row first, then mirror rows) and unlocks from the newest that opens under W with a chain to generation 1. Only rows on that path are stored, resealed under W; forged, junk, gapped or oversized mirror objects are skipped with a warning. A locked node waits for the pinned bundle's generation and stores mirror rows only once the recovered ARK opens them. - restore_dek stores a mirrored DEK only if its wrap authenticates for the space and id; mirror_dek refuses only a genuine different DEK and replaces objects that don't authenticate. - Mirror reads are capped (MAX_MIRROR_OBJECT, MirrorObjectTooLarge); an oversized object counts as absent. - With verified_fp pinned the keyring never re-splits: it uses the pinned bundle (restored from the mirror if the database lost it) or fails with EscrowPinned. - A share spends the per-boot budget only after it decrypts and matches a commitment. - Tests: forged generation, gap and junk on a healthy database; empty database under the same and a new root; a copied DEK object; pinned re-split; junk never spends the budget.

samgbafa · 2026-10-17T11:07:31Z

REQUEST_CHANGES: see TC-887 (head 5bd7656)

…only (#326) - establish_escrow runs at boot, after a rotation and after a recovery. An active bundle for other holders refuses the boot (EscrowHoldersChanged): the node never re-splits on its own. With no active bundle, a pinned bundle is re-activated from the database or mirror (the pin authenticates it); unpinned, a mirrored bundle whose mirrored record verifies under the ARK is restored; otherwise a new one is made. A missing record is restored from the mirror when its MAC verifies. A database restore no longer forces a new ceremony. - escrow_view, verification_record, gate and share intake only read the active bundle; no request writes escrow rows or mirror objects. - The newest generation never keeps a chain: boot clears a stale one, and rotate_ark answers a stale chain with an explicit Inconsistent ("retry"), not a permanent Conflict. - AtRestError::is_transient and reason(): stable, non-secret reason codes for API answers. - Tests: two nodes with different holders (no flip, reads write nothing), restore before the record and before the escrow (pinned and unpinned), a forged mirrored record, a stale newest chain.

# Conflicts: # test/w5-policy-runtime-node-e2e/Cargo.lock

… entry guard) (#326)

…326) - A generation picked from the mirror for its chain keeps the database's own seal (same ARK id), or gets a fresh seal under W if the database lacks it; mirror seal bytes are never stored. Test fails before. - GateRefusal::Unavailable carries AtRestError::reason() (a stable, key-free code), never raw store or mirror error text; the detail is logged server-side.

- With any at-rest setting, a dstack socket that is present but whose Info can't be read refuses the boot instead of booting as non-TEE, which would skip the measured-compose check and the escrow_none / allow_software refusals (refuse_unknown_tee; test). - Merges #326's non-secret gate reasons, so ShareAnswer.gate and the boot errors never carry raw store or mirror text.

samgbafa · 2026-10-17T13:02:19Z

APPROVE: see TC-887 (security review round 3, head 9b54bb77a985ff662bb1f4df29883c680ce20d8d).

#326) - Mirrored ARK objects that can't be read are skipped with a warning whenever the database has rows (still fatal with an empty database, where the mirror is all there is). Record and PITR-bundle restores skip unreadable objects too. - KeyringConfig::require_mirror: boot and recovery mirror writes are fatal only when keys may seal data; otherwise they are logged. - MemoryMirror: plant_unreadable and set_read_only for tests. - Tests: odd names, a directory, a symlink and an unreadable record on a healthy database (pinned, unpinned, locked recovery); fatal with an empty database; a read-only mirror is fatal only with require_mirror.

…st_keys (#326) Orders after #341's m20261017_000000_epoch_order_space_index. No schema change.

…ure map (#327) - S3 and FS mirror listings return only names that get accepts; the FS mirror lists regular files only. FS get never follows symlinks (O_NOFOLLOW), treats non-files as absent and bounds the read itself. - require_mirror = mode != off: with mode off, mirror contents never stop the node (with #326's skip-on-unreadable). - The per-client share-failure map is an LRU of 4096 clients, O(1) per call. - Tests: planted names, a directory and a /dev/zero symlink in the real app's datadir with mode off; FS mirror listing, symlink and size cap; 100k distinct clients stay bounded and fast.

samgbafa · 2026-10-17T13:59:57Z

REQUEST_CHANGES: see TC-887 (correctness round 3, head 17e1091a69b3714c355f75c92a13a120278fd4f6)

# Conflicts: # tinycloud-core/src/migrations/m20261015_000000_delegation_cursor_index.rs # tinycloud-core/src/migrations/mod.rs

…lit (#326) - D1: a mirrored data key that doesn't authenticate is never ignored and never overwritten. It may be the genuine key under a generation that didn't load (a restore plus a failed boot read). restore_dek fails closed for that space (AtRestError::DekUnavailable, transient, reason dek_unavailable) instead of letting create_dek and mirror_dek replace it; mirror_dek never overwrites an existing object with other bytes; an oversized DEK object fails closed too. Test reproduces the review's D1 chain and checks the original key returns once the object reads. - KeyringConfig::allow_resplit_before_verify: a holder change may re-split at boot only with the flag, no pin, an unverified active escrow and no data key ever made; otherwise EscrowHoldersChanged. - Merges main (#341's m20261017_000000_epoch_order_space_index and its migrated_through m20261015 test).

…ig (#327) - FS mirror get opens with O_NOFOLLOW | O_NONBLOCK and requires a regular file (fstat on the handle) before reading, so a FIFO under an escrow or record name can't hang the boot (test). - at_rest.escrow.allow_resplit_before_verify passes through to the keyring's guarded pre-ceremony re-split. - Merges #326 (D1 fail-closed data keys, main with #341). - Docs: the re-split flag, dek_unavailable, the never-overwrite DEK rule.

samgbafa · 2026-10-17T14:42:01Z

REQUEST_CHANGES: see TC-887 (security review round 3 delta, head 9f17af351f2f5907ba9a08c1efeed1eb576d67fc).

…e-lock guards (#326) - allow_resplit_before_verify no longer re-splits the same ARK: the node makes a fresh ARK (generation 1), splits it for the configured holders and retires the old one, so old shares rebuild only a key that never protected anything. - A W-authenticated tombstone per old ARK goes to the mirror first; load() then drops that ARK's key and escrow rows (also from a database restored to before the retirement) and ignores and deletes its ARK, escrow and record objects. Forged or junk tombstones are skipped; a database mixing retired and live rows is Inconsistent. - The database swaps keys in one transaction (replace_all_keys). - may_resplit also reads the mirror: any record- or dek- object, or any stored record, refuses the shortcut. - KeyMirror::delete (MemoryMirror implements it). - Takes S3's #333 keyring changes (efa5bbb): write_locked / in_write_lock, DataKeyError::NotPrepared, the guards on restore_dek, mirror_dek and create_dek, DataKeyProvider::prepare_read and its test. D1 still holds: prepare_read's restore fails closed on a mirrored DEK that doesn't authenticate.

… closed (#326)

…K holder fix (#327) - KeyMirror::delete for the S3 (DeleteObject) and filesystem (remove_file, absent is fine) mirrors, used only to retire an ARK that never protected anything. - Merges #326 (fresh-ARK re-split, retirement markers, S3's write-lock guards and prepare_read). - Docs: correcting a holder before the ceremony replaces the root key and makes old shares worthless; the dek_unavailable runbook; mirror write failures stop the boot with encrypt or strict.

…ders (#326) When no active bundle exists and none can be restored, establish_escrow used to split the ARK again for whatever holders were configured. Now any evidence of an earlier bundle for this ARK with other holders refuses the boot (EscrowHoldersChanged), strongest first: - a stored escrow row for this ARK; - a mirrored ceremony record whose MAC verifies under the ARK against its mirrored bundle (a record that doesn't verify proves nothing); - a mirrored bundle object naming this ARK: unauthenticated, so it can only refuse, never be used. A planted one fails closed (like D1); the log names the object, the runbook is to verify it and remove it. The operator route is the same as for an active bundle: allow_resplit_before_verify (a fresh ARK) before the ceremony with nothing sealed, an ARK rotation (TC-1345) after it. may_resplit no longer takes the active bundle (any stored or mirrored record refuses). Tests: rows lost + holders changed refuses without the flag and gives a fresh ARK with it; after the ceremony the verified record refuses even with the flag; a planted bundle only refuses and removing it boots.

samgbafa · 2026-10-17T17:42:37Z

APPROVE: see TC-887 (security review round 4, head 01055c7a7181d06c0723dc390651fb8b10e02fd4).

samgbafa force-pushed the skgbafa/tc-887-s1-escrow branch from 05c5200 to 65d3f4c Compare October 17, 2026 07:58

samgbafa force-pushed the skgbafa/tc-887-s1-keyring branch from c338a42 to 0ae6fe7 Compare October 17, 2026 07:58

samgbafa force-pushed the skgbafa/tc-887-s1-escrow branch from 65d3f4c to 91c5411 Compare October 17, 2026 08:47

samgbafa force-pushed the skgbafa/tc-887-s1-keyring branch from 0ae6fe7 to 1a3e685 Compare October 17, 2026 08:47

samgbafa added 4 commits October 17, 2026 09:08

feat(TC-887): expose the verified escrow record

5fab9c0

test(TC-887): roll the cursor-index lock test back to its own migrati…

5bd7656

…on by name The test rolled back one migration assuming it was the newest; with m20261017_000000_at_rest_keys after it, locate it by name and roll back it and every later migration.

samgbafa force-pushed the skgbafa/tc-887-s1-escrow branch from 91c5411 to 4222883 Compare October 17, 2026 09:11

samgbafa force-pushed the skgbafa/tc-887-s1-keyring branch from 1a3e685 to 5bd7656 Compare October 17, 2026 09:11

samgbafa mentioned this pull request Oct 17, 2026

feat(TC-887): seal KV blocks in the FileSystem store; prepare writes before the invoke lock #333

Draft

samgbafa added 2 commits October 17, 2026 11:03

Merge commit 'f423ba9' into skgbafa/tc-887-s1-keyring

89967a2

samgbafa added 4 commits October 17, 2026 11:16

Merge commit '8083ca2' into skgbafa/tc-887-s1-keyring

c4e4120

# Conflicts: # test/w5-policy-runtime-node-e2e/Cargo.lock

fix(TC-887): name the chain walk resolve_chain (single DID-resolution…

27f9430

… entry guard) (#326)

Merge commit 'a0d57fa' into skgbafa/tc-887-s1-keyring

c4ab37f

samgbafa added 2 commits October 17, 2026 13:10

chore(TC-887): rename the at-rest migration to m20261017_010000_at_re…

17e1091

…st_keys (#326) Orders after #341's m20261017_000000_epoch_order_space_index. No schema change.

samgbafa added 2 commits October 17, 2026 14:01

Merge branch 'skgbafa/tc-887-s1-escrow' into skgbafa/tc-887-s1-keyring

424dbde

# Conflicts: # tinycloud-core/src/migrations/m20261015_000000_delegation_cursor_index.rs # tinycloud-core/src/migrations/mod.rs

samgbafa added 2 commits October 17, 2026 15:58

chore(TC-887): log the data-key object name when a mirrored DEK fails…

3befa10

… closed (#326)

Navigation Menu

Repository navigation

feat(TC-887): at-rest keyring, key store and migration - #326

feat(TC-887): at-rest keyring, key store and migration#326
samgbafa wants to merge 18 commits into
skgbafa/tc-887-s1-escrowfrom
skgbafa/tc-887-s1-keyring

samgbafa commented Oct 17, 2026 •

edited

Loading

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

samgbafa commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

samgbafa commented Oct 17, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

samgbafa commented Oct 17, 2026 •

edited

Loading