Problem
The built-in agent runtime forgets every caller the moment the call ends.
Within a call there is real state: an LLM conversation, a TranscriptLog, and a rolling summary that compresses earlier turns so long calls stay inside the context window. internal/session deliberately shares all three across a reconnect, which is what makes ICE restart and resume preserve the conversation. But it is all held on the Session, and the session is reaped after server.session_grace_ms. Call back tomorrow and you are a stranger.
For the product this server is aimed at — a support line, a personal assistant, a device on a desk — that is the difference between a demo and something worth deploying. "As we discussed last week" is table stakes for a voice agent, and today the only way to get it is to abandon the built-in runtime for a BYO agent, which can persist whatever it likes.
Proposed change
Persist a per-caller memory record across sessions, keyed by a stable caller identity, and inject it the same way the rolling summary is injected today.
Identity is the hard part and should be decided first. internal/session already has an identity concept (identity_test.go) and the JWT path in /token carries claims. The key must come from something the caller cannot trivially forge, because memory keyed on a client-supplied string is a cross-user data leak waiting to happen: a JWT subject claim when auth is on, and a caller number on the SIP path. When neither is available, no persistent memory — silently degrading to "no memory" is correct, silently sharing one memory blob between anonymous callers is not.
Storage. Reuse what is already configured rather than adding a dependency. internal/rag already speaks pgvector and Supabase and the server already has a pgx driver, so a memories table alongside the existing vector store is the smallest addition. Keep it behind an interface with an in-memory implementation for tests and single-node use.
Shape. Start with the narrow, useful version rather than general episodic memory: a bounded text digest per caller, written at session end from the rolling summary the pipeline already produces, and prepended to the system context on the next session. Extraction of durable facts ("prefers to be called Sam", "on the enterprise plan") is a natural second step and should not block the first.
Config.
[memory]
enabled = false
provider = "pgvector" # reuses the [pgvector] connection
max_chars = 2000 # ceiling on what gets injected
retention_days = 90
Privacy is a feature requirement here, not a footnote. Persisting what people said on a phone call has obligations attached: retention that actually expires, a documented deletion path for one caller's record, off by default, and an explicit note in the docs about what is stored and for how long. A voice agent that quietly retains transcripts forever is a liability for whoever deploys it.
Open questions
- Does the memory digest belong in the system prompt, or should it be retrieved per turn through the existing RAG path? The second scales better and costs a lookup per turn.
- Should BYO agents be able to read and write it through the agent HTTP contract, or is it strictly a built-in-runtime feature? They can persist their own today, but a shared store would let the two modes interoperate.
Acceptance criteria
Pointers
internal/pipeline/rolling_summary.go — the digest this persistsinternal/pipeline/transcript.go — TranscriptLog, the other candidate sourceinternal/session/session.go — where conversation state is shared across a resume ~L44internal/session/identity_test.go — existing identity handlinginternal/rag/pgvector.go, internal/rag/supabase.go — storage to reusedocs/bring-your-own-agent.md — the escape hatch this closes
Problem
The built-in agent runtime forgets every caller the moment the call ends.
Within a call there is real state: an LLM conversation, a
TranscriptLog, and a rolling summary that compresses earlier turns so long calls stay inside the context window.internal/sessiondeliberately shares all three across a reconnect, which is what makes ICE restart and resume preserve the conversation. But it is all held on theSession, and the session is reaped afterserver.session_grace_ms. Call back tomorrow and you are a stranger.For the product this server is aimed at — a support line, a personal assistant, a device on a desk — that is the difference between a demo and something worth deploying. "As we discussed last week" is table stakes for a voice agent, and today the only way to get it is to abandon the built-in runtime for a BYO agent, which can persist whatever it likes.
Proposed change
Persist a per-caller memory record across sessions, keyed by a stable caller identity, and inject it the same way the rolling summary is injected today.
Identity is the hard part and should be decided first.
internal/sessionalready has an identity concept (identity_test.go) and the JWT path in/tokencarries claims. The key must come from something the caller cannot trivially forge, because memory keyed on a client-supplied string is a cross-user data leak waiting to happen: a JWT subject claim when auth is on, and a caller number on the SIP path. When neither is available, no persistent memory — silently degrading to "no memory" is correct, silently sharing one memory blob between anonymous callers is not.Storage. Reuse what is already configured rather than adding a dependency.
internal/ragalready speaks pgvector and Supabase and the server already has apgxdriver, so amemoriestable alongside the existing vector store is the smallest addition. Keep it behind an interface with an in-memory implementation for tests and single-node use.Shape. Start with the narrow, useful version rather than general episodic memory: a bounded text digest per caller, written at session end from the rolling summary the pipeline already produces, and prepended to the system context on the next session. Extraction of durable facts ("prefers to be called Sam", "on the enterprise plan") is a natural second step and should not block the first.
Config.
Privacy is a feature requirement here, not a footnote. Persisting what people said on a phone call has obligations attached: retention that actually expires, a documented deletion path for one caller's record, off by default, and an explicit note in the docs about what is stored and for how long. A voice agent that quietly retains transcripts forever is a liability for whoever deploys it.
Open questions
Acceptance criteria
[memory]section, off by default, documented indocs/configuration.mdanddocs/agent-runtime.md.max_chars.Pointers
internal/pipeline/rolling_summary.go— the digest this persistsinternal/pipeline/transcript.go—TranscriptLog, the other candidate sourceinternal/session/session.go— where conversation state is shared across a resume ~L44internal/session/identity_test.go— existing identity handlinginternal/rag/pgvector.go,internal/rag/supabase.go— storage to reusedocs/bring-your-own-agent.md— the escape hatch this closes