Skip to content

SMOODEV-4158: Re-ask an LLM call whose stream breaks mid-response - #209

Merged
brentrager merged 1 commit into
SMOODEV-4057-stream-usagefrom
SMOODEV-4158-stream-retry
Oct 24, 2026
Merged

brentrager merged 1 commit into
SMOODEV-4057-stream-usagefrom
SMOODEV-4158-stream-retry

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Problem

On the SMOODEV-4125 bench, six gpt-6-luna turns died on stream read error: error decoding response body, after 17 to 428s, and were counted INCONCLUSIVE. chat_stream retries only the initial send. A body that breaks in transit failed the whole turn, and the error was formatted with {e}, so the source chain (reset, truncation or timeout) was lost.

Solution

  • Body read errors (OpenAI-compatible and Anthropic paths) are now a typed StreamInterrupted (re-exported) that carries the full source() chain.
  • run_with_channel re-asks the same request up to MAX_STREAM_RETRIES (2) times with 500ms/1s backoff, and emits AgentEvent::LlmStreamRetry { iteration, attempt, max_retries, error }.
  • It retries ONLY when all of these hold:
    • the response has not completed (Done not seen);
    • no answer text reached the consumer (so a server streaming the reply never shows it twice);
    • the error is a StreamInterrupted, not a refusal or a malformed event.
  • Tool calls dispatch and the conversation/cost are recorded only after a complete response, so a retry never repeats an action or bills twice. The retry sits ahead of gateway_list_price_for / record_cost_and_check_budget.
  • MockLlmClient::push_stream_interrupted scripts a mid-stream break.

Verification

  • New tests:
    • re-asked once and the turn completes;
    • bounded at 2;
    • an error after a complete tool-call response is NOT retried and the tool is not dispatched;
    • not retried after answer text;
    • only StreamInterrupted qualifies (including through .context());
    • backoff;
    • the source chain is kept.
  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace: all green (640 core lib tests).

For #208 (1.15.0). @stream-usage owns the changeset.

Jira: SMOODEV-4158

🤖 Generated with Claude Code

On the bench, six gpt-6-luna turns died on "stream read error: error decoding
response body". chat_stream retried only the initial send, so a body that
broke in transit failed the whole turn. The error was also formatted with {e},
which dropped the source chain, so the log could not say whether the cause was
a reset, a truncation or a timeout.

Body read errors are now a typed StreamInterrupted that carries the full
chain. run_with_channel re-asks the same request up to MAX_STREAM_RETRIES (2)
times with 500ms/1s backoff, and emits AgentEvent::LlmStreamRetry. It re-asks
only while that cannot be observed or repeat anything: the response has not
completed, no answer text has reached the consumer, and no tool call has been
dispatched. Nothing is pushed to the conversation and nothing is costed until
a response completes. An error after a complete response is still surfaced
and not retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 24, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 935edb5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant