feat: session bootstrap token and vsock transport for microVM sessions

[jiashuoz]

Summary

PR 2 of the microVM driver sequence: the session bootstrap token and
virtio-vsock as the single host-to-guest control channel, implementing
docs/design/2026-09-20-microvm-bootstrap-token-and-vsock.md (merged on
main), which implements rainier-cloud ADR-0003 §2.7 items 1 and 2.

Two things change, and they are the same thing seen from two ends. An
environment's decrypted secret_refs stop being dispatched to a microVM
runner at all: the create carries their names and a single-use,
placement-fenced, 120-second token, and the guest exchanges it for the
values over the session RPC after it boots. And a microVM guest is configured
over vsock rather than by a file on the host or by MMDS: it opens one
connection, reads its whole configuration off the first control frame, and
runs its terminal, its session RPC and its lifecycle handshake over the same
conn, because internal/relay already multiplexes exactly those.

Nothing about the Docker path changes. That is the compatibility floor, it is
asserted as strictly as the new path, and its own tests are untouched.

The commits

Four, landed in the note's §8 rollout order — ordered by who must be able to refuse
before anybody asks — each green on its own and each separately revertable.

1. 5daa85e — protocol/runner and internal/relay, additive only.
Spec.BootstrapToken and Spec.SecretNames; MethodFetchSessionSecrets and
MethodMintSessionBootstrap; SessionBootstrapProtocolVersion;
CapabilityMicrovmV1; runner.BootConfig (the note's open question 4: beside
Spec, because every field of it is a field of the create it came from);
ControlEvent.Cold and relay.KindBootConfig. Nothing sets any of it, so
every existing message is byte-identical — pinned by repeating the golden
create literally, by a Docker create still carrying its env values verbatim,
by a warm suspend notice still having no cold key, and by an older decoder
reading a cold notice as the plain suspending it has always answered.

2. 6208bef — the control plane. createSpec keys the one new decision
on the placement's runner's announced capabilities (which now travel with the
placement rather than being re-read at dispatch): for microvm.v1 it leaves
material.Environment's values out of Spec.Env, sets SecretNames, and
mints a 32-byte crypto/rand base64url token whose sha256 is stored against
the session with the create's placement generation and a 120-second expiry.
control.SessionBootstrapStore is the seam — two methods, with the spend as
ONE atomic step, because single use enforced by a read followed by a write is
not single use — implemented by the in-memory store under its mutex and by
Postgres as a predicated UPDATE with a second read only to say which of the
four refusals applies. internal/controld grows the two
handleSessionRequest arms; fetch_session_secrets spends first and
re-resolves the secrets through the same launchMaterial the scheduler
dispatches with, so there is one decryption path and not two.

3. 1e0e240 — runnerd and the driver. relay.NetConn (a relay.Conn
over a byte stream, newline-delimited JSON, writes serialized because a Hub
writes from every attached client's goroutine). PUT /vsock with guest_cid
3 and a per-boot uds_path inside the session's state directory; a listener
on <uds_path>_1024 opened before InstanceStart; boot_config as the
first FrameControl on whatever connects. session.json and
buildGuestEnv's copy of spec.Env are deleted entirely. The driver refuses
a create carrying env values with no token, naming the control-plane version.
runnerd announces microvm.v1, shares the /register body with the vsock
door (serveSessionConn), leaves Spec.DialURL empty on that path, and
originates mint_session_bootstrap before a cold resume boots.

4. cb1b7de — sessiond. A func(context.Context) (relay.Conn, error)
behind dialLoop; the Docker path keeps websocket.Dial(dial+"?session="+id)
byte-for-byte (pinned by the request URI a runner receives); the microVM path
dials AF_VSOCK to (2, 1024). The guest reads its configuration off the first
frame, exchanges its token, and applies both to its own environment in the
shape bootEnvFromOS already reads — so prepareBoot, the agent sync and the
exec runner are identical on both paths. A refusal fails the boot chain.
suspending with cold: true flushes, ends the execs, unmounts
/rainier/agents, forgets the delivered secrets, and then answers ready.

5. 94e34de — the review's findings. A fifth commit, because the review
below is part of the work rather than a note about it. It is not a fifth step
of the rollout: every fix belongs to one of the four steps above and each of
those is still green on its own.

Compatibility (the note's §3 table)

controld rolls before runners, the order the fleet already uses.

Pairing	Behaviour	Where it is asserted
new controld + old runner	No microvm.v1 announced, so createSpec mints nothing, withholds nothing, dispatches today's Spec exactly.	TestCreateSpecWithholdsOnlyForMicrovm (four non-microvm rows)
old controld + new runner	The token is absent and Spec.Env carries the secrets. The microVM driver refuses to boot, naming the control-plane version — a refusal, not a quiet write to session.json.	RunContract's twelfth subtest; TestMicrovmRefusesSecretValuesWithNoToken
new sandbox + old controld	handleSessionRequest's default arm answers unknown method …, and the guest then applies the row below.	TestSandboxRequestIsAnswered (pre-existing, unchanged)
new sandbox, no token	Docker: the guest reads its environment block as today and never asks. microVM: an empty SecretNames is a clean boot; a non-empty one with no token, or a refused exchange, fails the boot chain as stage_failed naming the count of undelivered names and none of their values.	TestBootOverVsockWithNoSecretsDeclaredIsACleanBoot, TestAMissingTokenWithDeclaredSecretsFailsTheBootChain, TestARefusedExchangeFailsTheBootChain

What is tested here

• Minting, withholding and single use. The five outcomes the note's test
  plan names — fresh, replayed, expired, superseded placement generation,
  another session's — live in controlapp/repotest (cases B1–B3), so they are
  run against the in-memory store and Postgres rather than against a fake;
  and again end-to-end over the live runner plane in
  TestFetchSessionSecretsTable. createSpec's "the values or the token,
  never both" is a matrix over announced capabilities.
• The log discipline (§15.1), as a test rather than a claim.
  TestTheBootstrapExchangeLeavesNothingInTheLog captures internal/controld's
  whole log output across a mint, a successful fetch and a replay, greps it for
  the fixture secret and for every token that crossed the wire, and then
  proves the grep ran over real output by requiring the session id to be in it.
• Nothing on host disk. TestMicrovmWritesNoDecryptedEnvironment walks the
  whole state directory for the fixture secret and for the bootstrap token,
  and checks the mode of every file and directory it finds.
• The channel. A guest dials the socket Firecracker forwards port 1024 to,
  and the assertions are on what it RECEIVES: the whole configuration, the
  bootstrap pair, the proxy with the session's identity as userinfo, a
  no_proxy with no dial host in it. The socket's mode is pinned at 0600.
• The resume. A cold resume mints a fresh token, opens a fresh socket, and
  leaves the parked session's socket removed; a mint that cannot be made fails
  the resume and leaves the session parked for a later one.
• The guest, over a net.Pipe-backed fake transport. Boot config applied
  (in the shape bootEnvFromOS reads), the exchange sent exactly once, a
  refusal failing the chain with a count and no name or value, a redial NOT
  re-exchanging a spent token and a resume re-exchanging a fresh one, the cold
  suspend forgetting the delivered secrets and the warm one keeping them.

What needs hardware

There is no KVM in CI and none on the machine this was written on, so four
things are asserted by construction and by unit test and are not yet proven
against Firecracker. They are Phase 1's host harness, not go test:

• that Firecracker forwards <uds_path>_1024 to the unix socket the
  driver is listening on, and accepts PUT /vsock in the version and jailer
  configuration we ship (the runbook pins v1.17.0);
• that the guest kernel has CONFIG_VIRTIO_VSOCKETS and /dev/vsock, and
  that the session image sets RAINIER_TRANSPORT=vsock;
• that the TAP firewall drops 169.254.169.254 while the session still
  boots — which is the point of removing MMDS;
• the cold-suspend unmount of /rainier/agents against a real block
  device.

Everything above the socket is exercised here: the host end over a real unix
socket in internal/driver, the guest end over a net.Pipe in
cmd/sessiond.

The twelfth RunContract subtest

"a create carrying secret values and no token is refused", and it is written
so that both drivers have something to fail:

• a driver that does not announce microvm.v1 must accept the create.
  Docker puts the values in a docker run argv and nothing reaches the host
  filesystem; a refusal there would mean a control-plane rollback took every
  self-hosted session down.
• a driver that does announce it must refuse, because that capability
  is precisely the claim the plane withholds on — a driver that announces it
  and then accepts the values has told the plane it will do something it does
  not do. The refusal must name the bootstrap token and microvm.v1, must not
  quote the value, must leave no session behind, and the same create WITH a
  token must be accepted (so the refusal is about the token and not about the
  field existing).

The note's other half is also done: subtest 5 ("snapshot strips the named
environment keys") now carries a token, so it stays a shared subtest rather
than becoming a refusal on one driver and a snapshot on the other, and the
microVM snapshot manifest records the keys of the boot configuration as well
as of the driver's own map — so assertStrippedFromImage's microVM arm reads
the published manifest for what the session was actually configured with,
rather than for a map that is now always empty. Tenancy item 38 is a test.

Gates

make verify (module-path, protocols, control, session-image-security-policy,
tests, build, vet), go test ./... -race -count=1, go vet ./... and
git diff --check all pass.

Skipped for lack of a database: every test in
internal/controld/pgstore skips here with "docker not available and
RAINIER_TEST_PG_DSN unset" — including TestPGStoreRepositories, which is
what runs controlapp/repotest's three new bootstrap cases (B1–B3) against
the Postgres store and the 0015_session_bootstrap migration. Those three
cases do run here against the in-memory store, and the two implementations are
written against the same contract for exactly that reason; the Postgres half
needs a run with a database.

Review

An independent adversarial review was run over the four-commit diff, the
design note and internal/driver/driver.go, asking for correctness bugs,
secret leaks to disk or logs, contract violations and vacuous tests. It
found nine substantive issues. All nine are fixed in 94e34de, each with
the test that would have caught it. In severity order:

1. A sandbox could mint itself unlimited bootstrap tokens. routeControl
   forwards any req:<method> a sandbox sends, and mint_session_bootstrap
   was in that set — so a compromised agent could ask for a fresh token
   whenever it liked and re-read its environment's current secrets forever.
   Single use and the 120-second TTL were enforced against nobody, on Docker
   sessions as well as microVM ones. controld cannot make this check (a
   session_req proves only that some runner sent it), so the runner now
   refuses the method from a sandbox, and refuses a request carrying an id
   from its own reserved space with it — an id space the untrusted end can
   write into is not a space.
2. A refused exchange took the session off the air. The plane spends a
   token on receipt, so a refused or timed-out exchange has spent it;
   sessiond keyed "already done" on success, re-presented the dead token on
   every redial, and dialLoop backed off forever. A session that was merely
   missing its secrets — and had already said so as a failed boot stage —
   became unreachable at the first conn blip. It now keys on attempted, and
   a refused re-exchange on a live session is logged and the conn served.
3. relay.NetConn poisoned its deadline instead of failing. A cancelled
   context set a deadline and never cleared it, so connWriter.writeWithin's
   bounded exec write left the conn readable and unwritable — a session
   silently dead upward while both ends believed it live. It now closes the
   conn, which is what *websocket.Conn does and what writeWithin's own
   doc relies on; the boot preamble consequently closes and re-dials rather
   than handing on a conn it may have broken.
4. The boot exchange used id 1, the RPC dispatcher's first, so a late
   answer would be decoded as an empty agent credential set. The comment
   claiming the counter was seeded past it was false.
5. A failed cold resume made a session unresumable. Only
   <uds_path>_1024 was removed, never <uds_path> — which Firecracker
   binds — and the boot counter does not advance past a failed attempt, so
   the next resume hit EADDRINUSE on a socket nothing served, forever.
6. An unreadable placement minted a token that could never be redeemed:
   generation 0 against a row at 3 is refused as superseded by the very
   placement that minted it, and a guest cannot re-mint. A withholding create
   now refuses; every other create is dispatched exactly as before.
7. The microVM arm of assertStrippedFromImage could pass over an empty
   key list. Subtest 5 now carries a key it does not strip, and the arm
   requires it to survive — so the strip checks cannot go quiet.
8. A runnerd test asserted a hub it never created did not exist, with a
   failure message naming an assertion that was not there. It now builds a
   sandbox and asserts nothing reaches it.
9. A one-byte off-by-one made a frame of exactly maxFrameBytes writable
   and unreadable.

Plus, from the same review: the first dial now retries like every other one
in this process rather than being the one whose transient failure is fatal;
the store's refusal order is written into the port contract so two
implementations cannot describe one failure two ways; the refusal for an
unwithheld create no longer implies every value in the block is secret; and
four comments that claimed more than the code did now say what is true.

The review came back clean on secrets to disk or logs (it checked every
new log line, error string, file write and the snapshot manifest, plus the
file modes), on Docker byte-identity, on the additive wire fields, and
on store atomicity.

Not in the design note

Everything below is a decision this change had to make that the note did not:

• relay.NetConn. The note names vsock but not a framing, and a byte
  stream has none: one JSON value per line, chosen because every message here
  is already produced by encoding/json, which never emits a bare newline
  inside one — so the framing needs no second encoder that could disagree with
  the first. Bounded at 16 MiB, matching both WebSocket ends' SetReadLimit.
  Writes are serialized, which a *websocket.Conn does internally and a
  net.Conn does not.
• runner.BootConfig.Env and .NoProxy. The note's field list says "agent
  manifest"; the map is that plus the per-provider home variables, which are
  paths and names and cannot be enumerated statically (the provider table is
  data). no_proxy is the value noProxyFor derives on the Docker path, kept
  on the wire so there is one source of truth rather than two.
• runner.SessionBootstrapProtocolVersion. The note's payloads carry
  "protocol": 1 and name no constant for it.
• control.SessionBootstrapStore and its four sentinels. The note says
  "a small store seam"; this is its shape. FleetOptions.Bootstraps is
  REQUIRED rather than optional, so a control plane composed without one
  refuses to start instead of choosing between refusing every microVM create
  and quietly putting the secrets back in Spec.Env.
• Where the five-outcome table lives. controlapp/repotest, not a
  controlapp unit test, so it runs against both real stores.
• driver.CapabilityDriver and driver.HostedDriver. Two optional
  interfaces, so the microVM-specific behaviour in runnerd (no dial URL, a
  cold notice) keys on the announced CAPABILITY rather than on a concrete
  type — the claim the control plane acts on and the behaviour the runner
  performs then cannot disagree. SetHost is a setter because the runner is
  composed over the driver and so cannot be a constructor argument.
• sessiond --transport / RAINIER_TRANSPORT. A microVM guest receives
  no per-session environment at all until its boot configuration arrives, so
  which transport it has cannot be inferred from a create. It is a property of
  the rootfs, and the session image sets it — which is image work in a later
  PR of this sequence.
• The undelivered-secrets failure is a boot STAGE. stageSecrets, first
  in the chain, whose script only fails — so the existing watcher reports
  stage_failed with a tail and the agent is never exec'd, through exactly
  the machinery a failed setup already goes through rather than a second path
  that would have to be kept in step with it.
• "Exchange when the token is one I have not spent." The note says "at
  boot and again after every resume" and does not say how a guest tells a
  resume from a redial. A resume is exactly the connection that brings a new
  token, because runnerd mints one before it boots the new VM; a redial brings
  the one already spent. Keying on the token needs no second signal and gets
  both cases right — a sessiond that re-exchanged per connection would be
  refused on its first redial and stay refused forever.
• mint_session_bootstrap is not gated on microvm.v1. A runner that did
  not announce it was dispatched the values in Spec.Env already, so the
  method grants it nothing new; a capability check there would be a second,
  weaker fence in front of the one that holds (the placement guard, the hash,
  the generation, single use), and two fences that can disagree are worse than
  one that cannot.
• runnerd's own RPC id space. The runner now ORIGINATES a session RPC, and
  both directions ride one connection for one session, so its ids come from a
  disjoint space (the high bit) — otherwise a response to the runner's id 1
  would be delivered into a sandbox waiting on its own id 1.
• The cold resume after a runnerd restart still refuses. The token is no
  longer what is missing — the runner mints a fresh one — but the rest of the
  boot configuration is, and persisting it is exactly what §2.7 item 1 rules
  out. Rebuilding it from the control plane on a resume is the
  portable-checkpoint work (ADR-0003 §2.3), not this change.
• A known gap this change does not close, stated rather than left to be
  found. A runnerd restart leaves a RUNNING microVM session's control socket
  unserved: the listener died with the process, and the driver cannot re-open
  one because it has no boot configuration to hand whoever connects (that is
  the same in-memory-only rule as above). The guest's sessiond backs off and
  redials forever, so the session keeps running and is unreachable until it is
  destroyed. It is the same shape as the cold-resume refusal PR 1 introduced
  and has the same fix — a create-shaped resume from the control plane — and
  it is a strictly smaller gap than the Docker path has, since that one has a
  listener a restarted runnerd re-opens.
• unixPathMax and short test directories. A unix socket path is bounded
  at 108 bytes including its NUL, and the path is composed from an operator's
  --microvm-state-dir; the driver checks it and names the flag, because
  "invalid argument" from bind(2) is not something anybody can act on.

🤖 Generated with Claude Code