feat: portable workspace checkpoint format and library

[jiashuoz]

Summary

Rainier Serverless cold-suspends a session by terminating its microVM and keeping
only the workspace. The portable workspace checkpoint is the durable,
provider-neutral representation of that workspace — the thing the control plane
has to be able to say a true sentence about, because if it cannot be read back
the session's work is gone.

This PR adds the format and a standalone library for it: a design note, a new
public checkpoint/ package, and its tests. No driver integration —
internal/driver/ is untouched on purpose, because it is being changed in
parallel; wiring Suspend(warm=false) to Writer.Write behind the durability
barrier is a later PR.

Requirements it implements, from tokencanopy/rainier-cloud:

• PRD §10 — a versioned, encrypted, integrity-checked checkpoint containing the
  session filesystem and native agent resume state, restorable on another
  qualified provider, created and restore-tested after every clean cold
  suspension, with a key reference model that supports a maximum checkpoint age.
• Tenancy §11 — envelope encryption with versioned keys and an authenticated
  context of workspace, session and checkpoint generation; ciphertext copied into
  another context must fail (§18 item 39). §14.2 — deletion, per storage class,
  by key-wrapper deletion. §15.1 — nothing in the prohibited-logging list ever
  reaches an error.
• ADR-0003 §2.3 — the warm-dormant and deep-dormant tiers both rest on it.
  §4.4 — suspend is not successful until the upload completes and the manifest
  commits atomically, and deep-dormant needs a verified checkpoint newer than the
  disk's last write.

---

The design decisions

The note is docs/design/2026-09-20-portable-workspace-checkpoint.md and argues
each of these; the short version:

One framed stream, not content-addressed chunks. Chunking buys
deduplication, free resumable upload and parallelism — real wins for a workspace
checkpointed every ten idle minutes. It costs a chunk store with reference
counting and a garbage collector, which turns deletion from an operation into a
distributed refcount problem; tenancy §14.2 requires deletion to be provable
per storage class. It also shares ciphertext across contexts by construction,
which is what §11 forbids. Chosen: one content object, framed internally.
Deduplication can come back below the format (a blob store that chunks
internally) or beside it (a differential second version). A garbage collector
standing between a customer's workspace and its only durable copy cannot be added
later and then removed.

Note that "one stream" cannot mean one AEAD call: crypto/cipher's GCM is
one-shot over a []byte. Framing is what makes a single logical stream
implementable, and it is also what gives per-frame integrity and detectable
truncation.

Two objects, one commit. The content object's key carries a random attempt
suffix, so two writers at the same generation never collide and never have to
reason about whose half-written object they found. The manifest's key does not,
and it is written with put-if-absent: that single conditional write is the atomic
manifest commit ADR §4.4 rests on. Orphaned content objects are unreadable
without their manifest's wrapped key, so leaving one for the retention sweeper
costs storage and discloses nothing.

Three independent bindings of one authenticated context.
FormatVersion \0 workspace \0 session \0 generation, purpose-separated, bound
by the key wrap's AAD, by the manifest's tag, and by every frame's AAD. Any one
would satisfy §18 item 39 on paper; three means a mistake in one degrades the
property instead of deleting it.

The caller supplies the context; the manifest never gets to. Every read path
takes the Context the caller expects and uses it to build every AAD; the
manifest's own identity fields are compared against it and otherwise not
consulted. A library that read the context out of the manifest and then used it to
open the manifest would make the cross-context property vacuous.

Authorization is a required argument. NewReader refuses a nil Authorize
hook, and the hook runs after the manifest is parsed and before the data key is
unwrapped, before one content byte is read, and before the target directory is
touched. The policy is the cell's; that the step exists and runs first is the
library's. Preflight proves authorization and key readiness from the manifest
alone — one small request, no content — which is the call a placement decision
makes when PRD §4.3 filters capacity by "storage and key readiness".

"Verified" means decrypt, integrity, and a structural walk — not a second
copy. Verify reads the committed object back through the blob store
(checking the in-memory bytes would test the encoder against itself), opens every
frame with its own AAD, hashes the ciphertext, walks every entry under the same
rules a restore applies, and writes nothing. It ends by comparing seven
aggregates against the authenticated manifest, including a tree digest that makes
ADR §19's "checksum-equal restore" checkable.

The manifest holds no path, file name, symlink target or file byte. The tree
listing is inside the encrypted stream. That is what lets the manifest be
plaintext, and it is why Summary has nowhere to put a secret.

No compression in v1. The field exists and must be none, so adding zstd
is a declared, strictly-parsed format change. The reasons are a decompression
bomb bound Verify would have to enforce, a second failure layer, a workspace's
bulk being mostly already-compressed, and compressed length being an entropy
estimate of the tree. It is listed as open, to be decided by measurement.

Deletion is one operation. The data key exists only in the manifest, wrapped.
Deleting the manifest makes the content object unreadable by anyone, including
Rainier; destroying the key version makes every checkpoint under it unreadable
without touching object storage at all. Delete orders manifest-first so an
interrupted deletion leaves the unreadable state.

Bounded memory as a rule, not an aspiration. No structure is O(entries) or
O(tree bytes); there is deliberately no API that returns a list of entries. The
test proves it: a 2 GiB sparse tree of 20,202 entries peaks 4.4 MiB above
baseline.

Package placement. Top-level checkpoint/, beside control, controlapp,
v0wire, attachplane, runnerplane and protocol/* — the set
scripts/check-public-control.sh guards and rainier-cloud may import. It cannot
be in internal/, and it must not be duplicated in rainier-cloud, because a
format implemented twice diverges the day the second copy is patched and this
one's divergence mode is unreadable customer work. This PR adds checkpoint to
that script's import-hygiene loop with no relaxations at all.

---

What rainier-cloud must implement

The library is deliberately three ports and no policy. Behind each one:

1. A GCS blob store with put-if-absent.

   PutIfAbsent(ctx context.Context, key string, write func(io.Writer) error) error
   Open(ctx context.Context, key string) (io.ReadCloser, error)
   Delete(ctx context.Context, key string) error

   PutIfAbsent is x-goog-if-generation-match: 0, must return an error
   satisfying errors.Is(err, checkpoint.ErrExists) when the object is there,
   and must either create the whole object or leave nothing — a failed write
   aborts the resumable upload session rather than leaving a partial object. Open
   returns checkpoint.ErrNotFound for a missing object. The callback shape is
   what lets a GCS Writer be handed straight to the format, which is also the
   only resumability the format wants (the protocol-level kind, invisible to the
   format).

2. Key service wiring behind Wrapper. Cloud KMS, with the versioned key
   reference model the interface already expresses: Wrap takes the alias the
   workspace's policy names and returns the concrete key version it used, which
   is what the manifest records and what Unwrap is later asked for. Two
   requirements: the authenticated context MUST be passed through as the KMS
   AdditionalAuthenticatedData (a wrapper that drops it silently removes one of
   three bindings), and it must fail closed — ErrKeyUnavailable or a transport
   error, never a fallback to a key it happens to have. The library passes a
   wrapper's error through rather than flattening it to ErrAuth, so a throttled
   KMS is never reportable as a tampered checkpoint.

3. Scheduling and policy. Everything the library declines: when to checkpoint,
   upload retry and backoff, the maximum-checkpoint-age enforcement and the
   operational view that flags workspaces outside it (PRD §10), the deep-dormant
   precondition (comparing the manifest's authenticated created_at against the
   disk's last write — the library supplies the timestamp and the verification
   report, not the comparison), the retention sweep that collects orphaned content
   objects from lost attempts, the deletion ledger of tenancy §14.2, and the real
   Authorize hook: authoritative workspace/creator lookup, placement generation,
   product-region policy, and key readiness, per tenancy §4 and §8.2.

4. The driver call site, when internal/driver/ settles: Suspend(warm=false)
   flushes and unmounts over vsock, then runs the four-step barrier — upload,
   commit, verify, and only then release the workspace slot.

---

Out of scope, explicitly

• Driver integration. internal/driver/ is not touched in this PR.
• Scheduling, retention, checkpoint-age policy, freshness views, upload retry,
  backoff, concurrency limits.
• Any storage backend but the in-memory one. Any key service but the static-key
  wrapper.
• Authorization policy (a required hook, never a decision).
• Incremental or differential checkpoints, deduplication, compression, provider
  snapshots, disk formats, live migration.
• Agent-home materialization. The library cannot reach the agent home and does not
  try to; whether non-credential agent state should travel is ADR-0003 §9's open
  question and the note's §14 keeps it open rather than quietly answering it.
• Anything in rainier-cloud.

---

The independent review, and what it changed

After the gates passed, a separate reviewer went at the diff and the note
adversarially: cryptographic misuse, plaintext leaks, unbounded memory, vacuous
tests. It read every file, ran the suite, and proved its findings with scratch
programs and overlay tests rather than asserting them. It found no critical
issue — it could not construct a nonce reuse, a cross-context or cross-purpose
confusion, an unauthenticated manifest field, or an undetected
truncation/extension/reorder/splice. Everything below is fixed in 8ac297b.

The two that mattered

A restore that Verify structurally could not vouch for. A directory's
recorded mode was applied when the directory was created, so a tree containing a
0555 directory — which go mod download produces for every module it
extracts; the reviewer counted 246 of them in this repository's own module cache
— packed cleanly, verified cleanly, and failed at restore on its first child
with EACCES. That combination is the one the durability barrier is least able
to survive: ADR-0003 §4.4 releases the workspace and eventually deletes the disk
on Verify's word, so any rule Restore applies that Verify does not turns a
verified checkpoint into a failed cold resume.

A symlink escape the review's own containment argument missed. The reviewer
crafted three chained-symlink streams and concluded containment held, with a
sound-sounding reason: a symlink exists at exactly one real location, so the
directory checkLink resolves against is the one the kernel resolves from. That
is true only while the link is created where its name says — and it is not, if
an ancestor component of the name is itself a link pointing shallower. A probe
confirmed the gap: a/d -> .. is contained (resolves to the root), a/d/up -> .. is contained (resolves to a), but a/d/up is physically created inside
the root, so its .. leaves the target and a/d/up/pwned lands outside it. Two
individually contained links compose into an escape. Only reachable with the data
key, since the stream is authenticated — but "the tree lands inside the target"
is a property §7 of the note claims, and it did not hold.

One change closes both. consumeStream now keeps a stack of the directories the
walk currently has open: every entry's parent must be a directory this walk
created (a symlink never is), and a directory's mode is applied when the walk
leaves it, with directories created 0o700 until then. Entries arrive
depth-first, so the state is a stack of depth O(tree depth) — a few hundred —
not a list of entries, so the bounded-memory budget survives, and the format's
ordering rule becomes enforced rather than assumed. Both checks run in verify
mode too, which is the whole point.

The rest

• Authorize receives an unauthenticated Manifest, and could not do
  otherwise — the tag needs the key whose unwrap it is authorizing. Now
  documented outright: only workspace, session, generation and the content key's
  prefix have been checked; decide with it, do not record from it. Delete
  says the same and bounds what a forged manifest could reach (one sibling
  attempt object in the generation being deleted anyway).
• Cancellation was ignored once streaming started — a cancelled 40 GiB
  restore kept filling the destination. Both directions now check inside the copy
  loop and return context.Canceled.
• ErrTooLarge was unreachable as documented, reduced to "could not be read"
  by a missing %w on the realistic path.
• KeyRef was the one unvalidated free-form field, fully attacker-controlled
  for anyone with bucket write access, reaching an operator through Summary
  with newlines, NULs and ANSI escapes intact. Now printable ASCII only — which
  also makes the hand-rolled canonical rendering unambiguous for the right
  reason rather than by accident of its neighbouring fields' charsets.
• No per-entry byte bound: archive/tar's reader implements PAX sparse
  entries, where the header size is logical and the plaintext far smaller. Now
  bounded against the manifest before a byte is written, and sparse records
  refused.
• Restore(".") refused every entry and blamed the checkpoint for the caller's
  spelling. MemoryStore.Overwrite was a put-if-absent bypass on the public
  surface of a package whose atomicity story is put-if-absent — unexported.
  Manifest now says not to log one and which rendering to log instead.

Test quality

The reviewer's sharpest finding was that the manifest-tamper table proves less
than it looks. Eleven of the twenty-one authenticated fields are refused by a
cheaper, earlier check than the tag — the identity comparison, the
enumerations, the length arithmetic, the wrapper's key-reference check — so
those cases never reach the tag and say nothing about whether it binds them. And
the reflection test proved each field's name appeared in the authenticated
input, not its value: add("frames", "0") would have passed it. There is now
a test that changes every field in turn through reflection and requires the
authenticated input to change with it.

assertNoContentInError's strongest token was the literal "/tmp/", which can
never fire because t.TempDir resolves under a TMPDIR that is not /tmp
here — it now takes the fixture root, and is applied to the error families that
actually have a path available to leak.

The bounded-memory tests got the largest rethink. The ceiling was 64 MiB against
a ~3 MiB budget, which would have watched a per-entry map go straight by. The
first fix was to tighten it and grow the fixture until a map would have to cross
it — which worked, and then starved internal/sandboxexec into a 600s package
timeout on this 4-core box, because a minute of parallel cryptography is not a
free thing to add to a suite. So the claim is now tested as what it actually is:
the peak heap is a function of the frame size, not of the entry count. One
test measures the bytes (2 GiB sparse tree, 12 MiB ceiling, measured 4.5 MiB);
a second measures the slope, writing 2,000 entries and then 40,000 and
requiring the difference to be under 2 MiB — measured 0.67 MiB for twenty times
the entries. That rules out an O(entries) structure of any constant factor,
which no ceiling can, and it costs seconds instead of a minute. The read side
moved off MemoryStore onto a store that streams from disk, so the measurement
is of the library rather than of the test double. Package time under -race
went from 126s to 46s, and sandboxexec is comfortable again.

Also added, for behaviours the note claimed and nothing checked: setuid dropping,
frame-writer boundaries at and around an exact frame multiple, a single write
spanning many frames, and quiesce-violation detection in both directions.

Gates

make verify, go test ./... -race -count=1, go vet ./..., git diff --check
— all clean.

Two things worth recording rather than quietly re-running away. sandboxexec's
600s timeout described above was real, was mine, and is fixed by the rework of
the memory tests. And on one race run TestAttachToFailedSession in
internal/controld failed on a 5s timeout, then passed on a targeted re-run and
on three further full runs of that package; internal/controld does not import
checkpoint (verified with go list -deps) and nothing on this branch touches
it, so that one looks like pre-existing flake rather than anything here.

Final state after the rework, all four gates in one pass: go test ./... -race -count=1 exits 0 with no failures — checkpoint 60.0s,
internal/sandboxexec 22.7s, internal/controld 18.1s — and make verify,
go vet ./... and git diff --check are clean.

🤖 Generated with Claude Code