Problem
The server is single-node and nothing in the code says so out loud.
Session state lives in two process-local maps:
// internal/session/manager.go
sessions map[string]*Session
resumable map[string]*Session
sessions holds the live PeerConnection, the running pipeline, the LLM conversation and the rolling summary. resumable indexes those same sessions by their current single-use resume token. Both die with the process, and neither is visible to any other instance.
Put two instances behind an ordinary round-robin load balancer and the recovery ladder documented in docs/protocol.md breaks in exactly the situations it exists for:
PATCH /whip/{sessionId} (ICE restart) lands on instance B, which has never heard of that session ID, and returns 404. The client burns its three backed-off restarts against the wrong box.- The resume redial carries a valid token that only instance A can validate. Instance B rejects it, and the client falls back to a fresh session: greeting replays, history gone.
recovered-without-history becomes the normal outcome rather than the rare one.
server.max_sessions caps per process, so a 3-instance deployment silently has 3× the intended cap and 3× the provider spend.
None of this fails loudly. It fails as "reconnection doesn't work in production, works fine locally", which is the worst shape a bug can have.
Proposed change
Two options. They are not mutually exclusive and the first is a prerequisite for taking the second seriously.
A. Document sticky routing, and make the failure loud. The cheap, correct-today answer. Sessions are inherently stateful — the DTLS association and the SRTP keys are per-instance and no external store fixes that. What a second instance can do is route correctly:
- A deployment guide covering session affinity in the common front ends (ALB target-group stickiness, nginx
ip_hash/cookie, Cloud Run session affinity, and the caveat that WHIP's Location header is the natural affinity carrier since the client already echoes it back on PATCH and DELETE).
- A knob to make the server advertise an absolute
Location URL pointing at its own routable address, so a client's follow-up requests bypass the load balancer's choice entirely. This is the smallest change that actually fixes the problem.
max_sessions documented as per-instance, with the multiplication called out.- A distinct error on a
PATCH for an unknown session, so "wrong instance" is distinguishable from "session expired" in a log.
B. External session/token store. For the case where affinity is not available. The resume token is the tractable half: it is a short-lived opaque string mapped to a session ID, and a shared Redis or Postgres index would let instance B answer "this token belongs to instance A" and issue a redirect. The conversation state (history, summary, transcript) would need to travel with it. The live peer never can.
Worth deciding explicitly whether B is in scope at all, or whether the answer is permanently "affinity, documented properly" plus a horizontally-scalable control plane in front. Either answer is fine; the current state — no answer written down anywhere — is not.
Acceptance criteria
Pointers
internal/session/manager.go — sessions and resumable maps ~L29internal/session/session.go — resumeToken, single-use, reissued per resume ~L48internal/signaling/ — WHIP handlers, Location header, PATCH routedocs/protocol.md — the recovery ladder this breaksinfrastructure/aws/ec2/ — the compose + Caddy deployment this would extend
Problem
The server is single-node and nothing in the code says so out loud.
Session state lives in two process-local maps:
sessionsholds the livePeerConnection, the running pipeline, the LLM conversation and the rolling summary.resumableindexes those same sessions by their current single-use resume token. Both die with the process, and neither is visible to any other instance.Put two instances behind an ordinary round-robin load balancer and the recovery ladder documented in
docs/protocol.mdbreaks in exactly the situations it exists for:PATCH /whip/{sessionId}(ICE restart) lands on instance B, which has never heard of that session ID, and returns 404. The client burns its three backed-off restarts against the wrong box.recovered-without-historybecomes the normal outcome rather than the rare one.server.max_sessionscaps per process, so a 3-instance deployment silently has 3× the intended cap and 3× the provider spend.None of this fails loudly. It fails as "reconnection doesn't work in production, works fine locally", which is the worst shape a bug can have.
Proposed change
Two options. They are not mutually exclusive and the first is a prerequisite for taking the second seriously.
A. Document sticky routing, and make the failure loud. The cheap, correct-today answer. Sessions are inherently stateful — the DTLS association and the SRTP keys are per-instance and no external store fixes that. What a second instance can do is route correctly:
ip_hash/cookie, Cloud Run session affinity, and the caveat that WHIP'sLocationheader is the natural affinity carrier since the client already echoes it back onPATCHandDELETE).LocationURL pointing at its own routable address, so a client's follow-up requests bypass the load balancer's choice entirely. This is the smallest change that actually fixes the problem.max_sessionsdocumented as per-instance, with the multiplication called out.PATCHfor an unknown session, so "wrong instance" is distinguishable from "session expired" in a log.B. External session/token store. For the case where affinity is not available. The resume token is the tractable half: it is a short-lived opaque string mapped to a session ID, and a shared Redis or Postgres index would let instance B answer "this token belongs to instance A" and issue a redirect. The conversation state (history, summary, transcript) would need to travel with it. The live peer never can.
Worth deciding explicitly whether B is in scope at all, or whether the answer is permanently "affinity, documented properly" plus a horizontally-scalable control plane in front. Either answer is fine; the current state — no answer written down anywhere — is not.
Acceptance criteria
docs/deployment.md(or a section indocs/configuration.md) states plainly that sessions are process-local and what breaks without affinity.Locationadvertisement, configurable, off by default.max_sessionsdocumented as per-instance.PATCH /whip/{id}for an unknown session returns a distinguishable error and logs it as such.Pointers
internal/session/manager.go—sessionsandresumablemaps ~L29internal/session/session.go—resumeToken, single-use, reissued per resume ~L48internal/signaling/— WHIP handlers,Locationheader,PATCHroutedocs/protocol.md— the recovery ladder this breaksinfrastructure/aws/ec2/— the compose + Caddy deployment this would extend