Retry OpenAI realtime responses that fail before output#6454
Retry OpenAI realtime responses that fail before output#6454RitwijParmar wants to merge 2 commits into
Conversation
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| message_ch=utils.aio.Chan(), | ||
| function_ch=utils.aio.Chan(), | ||
| messages={}, | ||
| response_create_params=self._response_create_params.pop(client_event_id, None) | ||
| if client_event_id | ||
| else None, | ||
| _created_timestamp=time.time(), | ||
| _done_fut=asyncio.Future(), | ||
| ) |
There was a problem hiding this comment.
🔴 Agent turn can hang forever when a new response starts during a failed-response retry
A newly started reply replaces the in-progress reply (self._current_generation = _ResponseGeneration(...) at livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py:1837) without ever closing the earlier reply that is still waiting to be retried, so that earlier reply's output streams stay open forever.
Impact: The conversation turn tied to the abandoned reply never completes, hanging the agent until the session is restarted.
Orphaned generation when a superseding response.created arrives during the retry window
Before this PR, _current_generation was always closed and set to None by _handle_response_done before any new response.created arrived, so the normal branch in _handle_response_created (realtime_model.py:1837-1846) never overwrote a live generation.
Now _maybe_retry_failed_response (realtime_model.py:2205-2255) deliberately leaves self._current_generation open (channels not closed, _done_fut unresolved) while it schedules a delayed retry. During that retry window (0.1s–2s, _interval_for_retry), any new response.created — e.g. a server-VAD auto-response triggered by the user speaking — takes the normal branch at realtime_model.py:1837 and reassigns self._current_generation to a brand new _ResponseGeneration. The previous (retrying) generation object is no longer referenced by self._current_generation, so it is never passed to _close/_close_current_generation, and its message_ch/function_ch are never closed.
The consumer in livekit-agents/livekit/agents/voice/agent_activity.py:3667 (async for msg in generation_ev.message_stream) and the surrounding await speech_handle.wait_if_not_interrupted([...]) at agent_activity.py:3718 block until those channels close, so the abandoned turn hangs indefinitely. The pending retry task later aborts cleanly (self._current_generation is not retry_generation), but it does not close the orphaned generation. The new test test_response_done_failed_does_not_retry_against_new_generation exercises exactly this superseding scenario but does not assert that the old generation was closed.
(Refers to lines 1837-1846)
Prompt for agents
The retry feature (_maybe_retry_failed_response) intentionally keeps self._current_generation open while a delayed retry is pending, instead of closing it like the pre-PR _handle_response_done did. This breaks the previous invariant that _current_generation is always None (or already closed) when _handle_response_created creates a new generation.
Problem: If a new response.created arrives while a retry is pending (for example a server-VAD auto-response triggered by the user speaking during the retry_interval), the normal branch in _handle_response_created (around realtime_model.py:1837) reassigns self._current_generation to a new _ResponseGeneration without closing the previous, still-open retrying generation. That orphaned generation's message_ch/function_ch are never closed and its _done_fut never resolves, so the AgentActivity consumer (agent_activity.py:3667 async-for over message_stream, and the wait at agent_activity.py:3718) hangs forever.
Possible fix: before overwriting self._current_generation in the normal branch of _handle_response_created, detect when the existing self._current_generation is a live _ResponseGeneration that is being superseded (i.e. not the expected reuse path) and close it (close its message channels, resolve modalities/_done_fut) similar to _close_current_generation, and clean up any associated _response_retry_event_ids/_response_retry_generations entries. Also consider handling this when the pending retry task aborts because self._current_generation changed.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
One thing I would want to pin down before this grows is the generation identity across a retry. Reusing streams avoids a duplicate generation in the caller, but the trace and metrics still need to say whether the first attempt failed before output and the second attempt completed it. I would carry one generation id with an attempt number and emit the retry reason once. Then a failed first attempt cannot look like two user generations in dashboards. Is that useful for the direction here? |
2 participants
Fixes #6205.
OpenAI realtime can return response.done with status failed after response.created but before any output item starts. Before this change, that closes the generation and the agent turn can disappear without another response attempt.
This retries that narrow case with the existing connection retry policy:
I did not retry after output starts. That needs a separate policy for partial audio/text cleanup and transcript history.
Checked locally: