feat(golang)!: the guest steps instead of blocking on the network, and the client drives it outside its lock

[coderdan]

Summary

The guest is Stack Encrypt compiled to WebAssembly. Today, whenever the guest needs ZeroKMS (our key service), it calls a host function, transport_send, and waits until the HTTP response comes back. A JavaScript edge runtime (Cloudflare Workers, Deno, a browser) can't do that: its fetch only returns a promise, so the guest can't run there. In Go it costs something too: the client holds its lock for the whole network round trip, so every caller queues behind the slowest request.

This PR is the step loop from #1064, as amended by the decisions comment there. An export that needs ZeroKMS no longer waits. Instead it pauses: it hands the host a handle and the HTTP requests it needs. The host performs them however it likes and resumes the operation with the responses, until it is done. The Go client now drives that loop and holds its lock only while the guest is actually running. There is no change for Go users' code.

Changes

Shared guest ABI (packages/stack-guest-abi), first commit

• step module: an operation's work runs as a future, polled once.
  • A request it makes is recorded, and the future stays pending in a table under a handle.
  • resume delivers the responses in order. Each request stores its waker, and resume wakes it first, so the 500-key chunking (buffered) never stalls.
  • abort drops the operation, which wipes everything it held.
• The table's rules:
  • Handles only increase and are never reused, so a stale resume can't reach another operation.
  • At most 64 operations may be paused at once; one more is STATUS_STATE.
  • Work that pauses with nothing to wait on is STATUS_INTERNAL.
  • "Pending with no request" drops the handle.
• STATUS_NEEDS_IO (29) is how an export says it paused. It is not a failure: call::run records no error for it.
• The module holds no encoding, so the credential guest, which doesn't have the vitaminc serde branch, still builds.

Crypto guest and Go client, second commit (breaking for anything that embeds the guest directly)

• The guest no longer imports transport_send; its only import is token_get.
• Every export that can reach ZeroKMS steps: se_cipher_init, se_keyset, se_term, se_encrypt_record, se_decrypt_record and se_query_term.
• New exports:
  • se_io: takes the paused step, handed over once.
  • se_resume(handle, responses): hands back the responses and runs the next step.
  • se_abort(handle): drops the operation.
• The step and the responses are transport frames read with serde (guest/src/io.rs): {handle, requests: [{method, url, headers, body}]} out, and [{status, headers, body}] back. That's field for field what transport_send carried, including the negative-status rule.
• A paused operation:
  • owns its inputs, plaintext included, and wipes them when it ends or is aborted;
  • shares the cipher through an Rc, so se_shutdown drops every paused operation before the cipher, and the cipher's wipe is the last owner's.
• A second se_cipher_init is refused while one is paused.
• Go client (encrypt/step.go, client.go):
  • Client.call and the new Client.callKMS drive the loop.
  • The lock is taken for each step, and released while the requests run.
  • Each step runs under context.WithoutCancel, so the runtime never closes the module under a caller who gave up. The context is checked between steps, and an operation whose caller has gone is aborted in the guest.
  • The token is fetched once per operation, before its first step and outside the lock, and token_get only hands it over. Fetching before every step would ask an OIDC provider once per round trip.
  • A step is read from the guest before the lock is released.
  • Every closure passed to call now receives the step's context.
• Build: mise.toml no longer requires transport_send on the crypto guest's import surface. The credential guest still requires it.
• Docs: encrypt/README.md, the Client doc comment, and a Go CHANGELOG entry.

Verification

• stack-guest-abi:
  • 13 step tests:
    • one request per step;
    • wakers through buffered;
    • several requests in one step;
    • interleaved handles that are never reused;
    • abort drops the work;
    • the wrong number of responses;
    • the 64-handle cap;
    • pending with nothing to wait on;
    • a request made outside a step;
    • the pending slot handed over once, replaced wiped, and emptied by a registry wipe.
  • Also a call::run test that a pause records no error.
  • mise run miri:stack-guest-abi passes, 41 tests.
  • Mutants on its diff: 20 caught, 0 missed (one survivor fixed with a test). This crate isn't in the CI mutants gate.
• Crypto guest:
  • Native cargo test passes, including the step encoding's round trip and refusals that never echo a value.
  • Clippy with -D warnings is clean for wasm32 with default features, eql and deterministic-kms, and natively.
  • mise run wasm:guest:test and wasm:no-http-test pass.
  • All four guests build, and the import-surface check passes with token_get alone.
  • The credential guest still checks with --locked.
• Go:
  • go test ./... passes.
  • go test -race passes on encrypt (524 s) and internal/guest.
  • New tests in encrypt/step_test.go, against a stub ZeroKMS that holds the init's request:
    • another call runs while a request is in flight, so the lock is free;
    • a caller that gives up mid-request gets context.Canceled, the operation's handle is gone, the client stays open, and a new init runs;
    • a context that has already ended never starts a call;
    • the token is fetched outside the lock (probed with TryLock);
    • an init paused and then aborted leaves neither the client key nor the bearer token in guest memory. Without the abort, the bearer stays, so this test can fail.
  • Rewritten tests:
    • TestImportSurfaceIsWASIPlusTokenGet (was …PlusTransport);
    • TestEachRequestIsOneStep (was the sends == 1 counter test), which also checks the handle is spent;
    • TestClientKeyDoesNotRemainInGuestMemory now drives the loop.

Related

• Refs WebAssembly guest blocks on a synchronous ZeroKMS call and cannot run on the edge — export the two halves of a batch so the host does the I/O #1064.
• Based on next, which now carries refactor: read control messages from frames with serde, not FfiValue (part 1: options and stats) #1262, fix(golang): a call whose context ends fails on its own, and leaves the client open #1263 and refactor(stack-encrypt)!: remove FfiValue #1264.

Review notes

• Where to start:
  • packages/stack-guest-abi/src/step.rs;
  • then guest/src/abi.rs (deliver, run, se_io, se_resume, se_abort);
  • then encrypt/client.go (drive, step, abort).
• Two deviations from the design comment, both accepted (2026-10-19):
  • The comment's shape: a Step::Done | NeedsIo frame as every stepping export's output.
  • What this does instead: a done export returns exactly the output it returns today, and a pause is a status (STATUS_NEEDS_IO) plus se_io to take the step.
  • Why: every existing output format is unchanged, and no plaintext output is copied into a wrapper frame.
  • The token: fetched once per operation, before its first step and outside the lock, not once per step. Fetching before every step would ask an OIDC provider once per round trip.
• Rebased over fix(golang): a call whose context ends fails on its own, and leaves the client open #1263 (the Go cancellation fix). Its rules hold for each step:
  • A step waits for the client with c.mu.Acquire(ctx); an abort takes Lock(), because a paused operation must always be dropped.
  • A step runs under guest.Detach(ctx, c.stuck) in place of context.WithoutCancel, so the 30-second backstop for a stuck guest still applies.
  • fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's guest.Caller in token_get and transport_send is gone: the token is fetched before the first step, and requests run between steps under the caller's context.
  • A call that ends because its context ended fails with the context's error. One abandoned while waiting on ZeroKMS matches ErrTransport too, as fix(golang): a call whose context ends fails on its own, and leaves the client open #1263 documents. A step that succeeds after its context ended has its output wiped and fails the same way.
  • New: a step that pauses after its caller gave up sends no request, and a resume whose wait for the client ends with its context still aborts the paused operation.
  • fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's tests now cover these between-step cases. A request that ignores its context no longer closes the client (the guest is paused while it runs); the backstop test now uses a step that ignores cancellation.
• Not done here:
  1. A Go test of two operations interleaved on one instance. Pausing any operation other than the init needs a cipher, and that needs a ZeroKMS that answers load-keyset, which the hermetic stubs don't fake. The Rust step tests cover interleaving, never-reused handles and the cap, and the Go lock test shows a second call running while one is paused.
  2. A memory-scan test with a plaintext needle for an encrypt aborted mid-flight. It needs the same working cipher. The guest drops a paused operation's owned Zeroizing plaintext on abort; that drop is tested natively (the abort test), but there's no linear-memory scan for it.
  3. The edge (JavaScript) host. This PR changes the ABI and the Go host only.
  4. Out of scope, as the decisions say: the credential guest's loop, and fetching keys for several keysets at once under {"any"}.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 8346d2c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5fd8b840-b410-4e94-b0cd-d8e11414121e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

No mutants were generated for the changed lines.

[coderdan]

CI fix, force-pushed: the token is now fetched once per operation, before its first step and outside the lock, and every step of that operation hands it over. Fetching before each step asked an OIDC provider twice for one init (TestOIDCFederationThroughTheClientTransport: provider calls 2). One fetch per operation is what the guest asked for before, so token sources see the same traffic as today. The cargo publish dry-run failure in the "fmt, clippy, nextest, doctests, rustdoc" job is inherited from #1264 (its job fails the same way, from the stack's do-not-merge git [patch]), not this PR's.

[coderdan]

Review of 76285d6 (this PR's diff only, against feat/remove-ffivalue). I read #1064 and its decisions comment, #1263, and the Rust and Go on both sides of the boundary. No earlier review comments exist on this PR.

Findings, most severe first

1. High. The handle space runs out in a long-lived process. step::NEXT is a u32 that only increases and is never reused, and issue() fails with STATUS_STATE once it would overflow. Every Seal and Open pauses at least once (generate_keys / retrieve_keys always go to ZeroKMS), so every one of those calls spends a handle. After 2^32 of them, every ZeroKMS-reaching call on that Client returns ErrState ("this instance has issued every handle") until the process restarts; the client is not closed, so nothing on the Go side notices or recovers. At 1,000 seals a second that is about 50 days of uptime, at 5,000 about ten. Suggest a u64 handle (wasm passes i64 scalars natively, and the Go side already hands the handle over as a uint64 through scalar), or wrap-around with a "not currently open" check, which keeps the stale-resume guarantee because a stale handle is never in the table. Better to fix while the ABI has one host. Inline on step.rs.

2. High. The 64-open cap is a hard failure for the 65th concurrent Go caller. MAX_OPEN = 64 is the right guard against a host that forgets se_abort, but the Go client does nothing to stay under it. Before this PR, 65 goroutines calling Seal queued on c.mu; now the first 64 pause across the network and the 65th se_encrypt_record returns STATUS_STATE, which drive hands back as ErrState ("too many operations are paused: resume or abort one first"). An HTTP server with more than 64 in-flight handlers under ordinary ZeroKMS latency hits this under load, and ErrState is documented as "the client is closed", so a caller will treat the shared client as dead. Suggest a bound in drive (a semaphore of MAX_OPEN acquired before the first step and released when the operation ends or is aborted), so excess callers wait as they did before, plus a test for the 65th caller. Inline on client.go.

3. Low (behaviour). A token is fetched for every callKMS operation, including ones that never ask for it. A query term, a derived term or a keyset lookup on a cached keyset never reaches ZeroKMS, so the guest never calls token_get; before this PR Strategy.Token was not called for them. Now every such call takes Strategy.mu, runs a credential-guest export, and for a device session whose token has lapsed performs a refresh (file lock and network) on behalf of an operation that would never have used it. The failure semantics are preserved (a source that fails only surfaces if the guest asks), so this is cost and a doc accuracy point: the comment on drive says the source is asked "once per operation, as it was when the guest asked for it", which is not true for those operations. Either fetch only when the operation's first step needs it (not possible under decision 4 as written) or say so in the comment and CHANGELOG. Inline on client.go.

4. Low. Marshalled responses are not wiped when the next step short-circuits. The resume closure owns defer wipe(responses), but c.step returns ErrState before calling it when the client was closed while the request was in flight (Close from another goroutine, or a trap on another operation's step). The responses, which can carry wrapped keys, then wait for the collector. Wipe on that path too. Inline on client.go.

5. Low. takeIO calls se_io and se_dealloc outside the allocator's Enter/Exit bracket. Every other guest entry goes through guest.Call, which keeps the memory mapped for the whole call so a Free that arrives mid-call is deferred (the contract on Allocator.Free). takeIO calls inst.io.Call and inst.exports.Free directly. Nothing can close the module during it today (the context is WithoutCancel, the lock is held, the cleanup only fires on an unreachable Client), so it is safe, but it is the one entry that does not keep the discipline. Suggest bracketing it. Inline on step.go.

6. Note, until #1263 and this are merged together. A step runs under context.WithoutCancel with no backstop. Before this PR a guest export that never returned at least died with its caller's deadline (WithCloseOnContextDone); now it holds c.mu forever, and Close takes c.mu too, so the process can only exit. It needs a guest bug to reach, since no step does I/O any more, and the PR's merge plan (take #1263's Detach(ctx, c.stuck) in step) covers it; just make sure whichever lands second keeps the backstop. Related: the comment on TestInterruptedCallClosesTheClient ("An interrupted call closes the module (WithCloseOnContextDone)") is stale, its first subtest now exercises the abort-between-steps path and the module stays open.

What I checked and found sound

• Locks across I/O. c.mu is held per step only; fetchToken runs outside it, so the old c.mu then Strategy.mu nesting is gone and Close (which takes c.mu then releases the credentials) cannot invert against it. transport.held, fetchErr and tokenErr are written and read only under c.mu on the stepping goroutine; sends is atomic. go vet and go test -race on encrypt and internal/guest report no race.
• Interleaving inside the guest. With two operations paused, a std Mutex held across an .await would panic on wasm32 (cannot recursively acquire mutex) and take the instance down. I checked the two: StackCipher::keyset takes keysets() before and after load_keyset(..).await, never across it (cipher.rs:587-617), and WasiHostConnection::send drops the base() guard at the end of the let url statement, before step::request(..).await. StackKms holds no lock of its own on wasm32; HostTokenStrategy is synchronous.
• Handle lifecycle. Issued only on the first pause; resume removes the entry first, so a wrong response count, a pause without a request, or a cap refusal all drop the work; abort and a second abort/resume on a spent handle are STATUS_STATE; se_shutdown runs drop_all before taking the cipher, so the Rc count makes the cipher's wipe the last owner's; clear_pending wipes a step the host never took; wipe_all forgets the pending slot before freeing. In Go, every error after a pause (performStep failure, context ended) aborts the handle, and abort checks closed before calling in.
• Inputs across a pause. Every stepping export copies its inputs before building the future (take_plaintext or to_vec, options parsed up front), so the host freeing its buffers when the export returns is safe; the aborted-init memory scan passes locally. The Initialising guard is moved into the future and so clears on done, failure, abort, cap refusal and shutdown.
• ABI boundary. se_resume validates the responses buffer through take_plaintext (range check, wiped in place), decode_responses is deny_unknown_fields and refuses a non-bytes headers without echoing it, the handle is checked against the table before any response is placed, and se_io is handed over once. Miri on stack-guest-abi is green in CI and the step tests run under it.
• Decisions in #1064 honoured: init and keyset loading on the loop with an in-progress state; a request list with one entry per step; several handles open at once with the cap and no-reuse rule; the last-error slot read under the same lock as the failing step; token refresh on the host; the credential guest out of scope; wakers stored per request and woken before the poll; pending-with-no-request is STATUS_INTERNAL. The one disclosed divergence (a status plus se_io instead of a Step::Done | NeedsIo frame) keeps every output format unchanged and copies no plaintext into a wrapper; it reads as the better shape. Not done, as the PR says: a Go test of two interleaved handles and a plaintext-needle scan of an aborted encrypt.
• Behavioural equivalence. The stored-rows fixture and the whole hermetic suite pass; max_concurrent_reqs(1) and the 500-key split are unchanged; stack-kms has no retry, so none is lost; the request runs under the caller's context as before. The accepted change is one token per operation rather than per request, which the decisions comment calls for.
• CHANGELOG and the !. The Go API is unchanged, so the entry belongs under Changed, as it is; the ! is for the guest ABI, which has no consumer outside this module. Worth keeping in mind when the Go release is cut by hand so it is not read as a major.

Checks run

• cargo test -p stack-guest-abi --all-features (root workspace): 42 passed.
• cargo test in languages/golang/encrypt/guest (native): all passed.
• Built the default, deterministic and EQL guests for wasm32-wasip1 with the import-surface check (token_get is the only transport import).
• go vet and go test -race -count=1 ./encrypt/ ./internal/guest/ (no credentials; the live tests skip): all passed once the EQL guest was built, no race reports.
• mise is untrusted in the review worktree, so the tasks were run by hand with the pinned 1.94.1 toolchain and Go 1.26; trust settings were not changed.
• The failing "fmt, clippy, nextest, doctests, rustdoc" job is the inherited cargo publish --dry-run failure from #1264 (vitaminc-aead-frame has no serde feature on crates.io), not this PR's.

[coderdan]

Rebased onto next now that #1262, #1263 and #1264 have merged, and retargeted to next. Old head 31a0607, new head 7fcb8f0. The first commit applied cleanly; the conflicts were all in the second, with #1263's cancellation work in the Go client.

How they were resolved (the rewrite changes more than hashes):

• Each step waits for the client with fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's Acquire(ctx) and runs under guest.Detach(ctx, stuck), so the 30-second backstop for a stuck guest still applies. An abort takes Lock().
• fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's guest.Caller in token_get and transport_send is gone: the token is fetched before the first step, and requests run between steps under the caller's context.
• fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's rule that a call which succeeds after its context ended fails with the context's error (output wiped) now applies where an operation finishes. A call abandoned while waiting on ZeroKMS still matches ErrTransport too, as fix(golang): a call whose context ends fails on its own, and leaves the client open #1263 documents.
• New: a step that pauses after its caller gave up sends no request, and a resume whose wait for the client ends with its context still aborts the paused operation. Both have tests, and each guard was checked by removing it and watching its test fail.
• fix(golang): a call whose context ends fails on its own, and leaves the client open #1263's tests are adapted to the step loop. A request that ignores its context no longer closes the client, because the guest is paused while it runs. The backstop test now uses a step that ignores cancellation, and the timing-based tests now cancel from inside the request, so they don't depend on the machine's speed.
• guest.Lock gains TryLock (with a test) for the step tests' lock probe.
• doc.go no longer says the guest imports an HTTP send.

Verified locally:

• Every Cargo.lock passes --locked at both commits.
• fmt, nextest workspace --all-features (1227 passed), clippy -D warnings.
• All five guests build, and the guest tests pass with and without eql.
• scripts/go-binding-test.sh languages/golang passes.
• The cancellation and step tests pass under -race -count=3 for encrypt, auth and internal/guest.
• go generate leaves no drift.
• The packages/ diff is byte-identical to the old head's, so mutants weren't rerun.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T10:46:27.912109Z	7fcb8f0	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-19T10:49:51.564767Z	7fcb8f0	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7fcb8f08bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".