[Fix] Billed API requests return no response when streams fail silently or end at max_tokens - #1580
Conversation
…ies, unhandled max_tokens stops, and dropped thinking signatures - Bound mid-stream API failure retries (3 automatic attempts), announce every retry through the visible backoff countdown, and ask the user once the budget is exhausted instead of looping silently. - Propagate the response stop_reason through the usage stream and stop retrying when an empty response ended with max_tokens, surfacing remediation guidance instead of re-billing the full context. - Capture Anthropic thinking-block signatures (signature_delta) and replay each signed thinking block unchanged on tool-use continuations.
|
Warning Review limit reachedNext included review available in 40 minutes. View limit detailsLimit details: You’ve used all 4 included reviews currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (7)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Review statusThis PR was opened by an automated account. A human maintainer must verify the change intent, provenance, and validation before merging. Current step: Mark the PR ready. Required CI must pass before CodeRabbit starts. Review-state labels are managed by this workflow; do not edit them manually. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@CodeRabbit review |
|
|
@CodeRabbit review |
|
2 participants
Related GitHub Issue
Reported in Discord (no tracking issue): users are billed for API requests that produce no response — recurring
streaming_failedrows with nothing shown, observed primarily on Sonnet.Description
Three related causes of "billed but no response" are fixed together:
1. Silent, unbounded mid-stream retries (
src/core/task/Task.ts)When a streaming request failed mid-flight (e.g. Anthropic
overloaded_error), the retry loop pushed the same request back onto the stack unconditionally; the backoff countdown was gated onautoApprovalEnabled, so most users saw nothing while every retry re-billed the full input context. Now automatic mid-stream retries are capped atMAX_AUTOMATIC_API_RETRIES = 3, every retry runs through the visiblebackoffAndAnnounce→api_req_retry_delayedcountdown regardless of auto-approval, and once the cap is exhausted the user is asked viaapi_req_failed. Approving resets the retry budget (the turn's user message is popped and re-added exactly once, so history never duplicates); declining surfaces the error and records a synthetic failure assistant message before stopping.2.
stop_reasonwas never inspected (src/api/transform/stream.ts,src/api/providers/anthropic.ts,src/core/task/Task.ts)Anthropic's
message_deltacarriesstop_reason, which was dropped. It now flows through the usage stream chunk, and the empty/no-tool-use response path branches on it: a response that ends withmax_tokensand no usable content (e.g. the model burned its whole output budget on thinking) is not retried — retrying the identical request would fail the same way while re-billing the full context. The user gets a clear error explaining the token limit and suggesting a higher max output tokens setting. Genuinely empty responses keep automatic retries, now bounded by the same cap before asking the user.3. Anthropic thinking signatures were dropped (
src/api/providers/anthropic.ts,src/core/task/apiConversationHistory.ts)signature_deltastream events were ignored, so thinking blocks could never be replayed on tool-use continuations (Anthropic rejects unsigned replays). The handler now accumulates each thinking block's text and signature, emits the existingthinking_completestream chunk per completed block, and exposesgetThinkingBlocks()/getThoughtSignature(). The existing history plumbing replays each signed thinking block individually (signatures only validate against their exact block text, so blocks are never combined under one signature).filterNonAnthropicBlocksalready preservesthinking/redacted_thinkingvia its allowlist; a regression test now pins that.Test Procedure
New coverage:
Task.spec.ts— mid-stream suite: cap-then-ask with exactly 3 visible retry announcements and exact attempt counts; visibility without auto-approval; user-approved budget reset re-adds the user message exactly once; declined retry restores history and records the failure.Task.spec.ts— empty-response suite:stop_reason: max_tokensstops immediately with the remediation error (no retry, no ask, history consistent); automatic empty-response retries are bounded and then ask.anthropic.spec.ts—stop_reasonpropagation on usage chunks;signature_deltacapture withthinking_completeemission andgetThoughtSignature(); per-block pairing across multiple thinking blocks; stale signature reset between requests; signedthinking/redacted_thinkingblocks round-trip unchanged.apiConversationHistory.spec.ts— each thinking block is replayed with its own signature, in order.Pre-Submission Checklist
Visual Snapshots
N/A — extension-host retry/streaming logic with no rendered-UI change.
Videos (interaction / animation only)
N/A
Documentation Updates
Additional Notes
Consolidates what would otherwise be three separate PRs (mid-stream retry bounding, stop_reason handling, thinking-signature preservation) into one reviewable change. The same
signature_delta/stop_reason gaps also exist inanthropic-vertex.ts, which duplicates this streaming loop; left out of scope here and a good follow-up.Get in Touch
@taltas on Discord can reach the requester (on55db) for reproduction details.