Repository navigation
SMOODEV-4057: price streamed turns instead of recording $0 - #208
Open
brentrager wants to merge 3 commits into
Open
brentrager wants to merge 3 commits into
brentrager wants to merge 3 commits into
Conversation
Every streamed turn on llm.smoo.ai recorded cost_usd = 0. LiteLLM flushes its cost headers before the completion is priced, so on a stream they read 0, and the agent fell back to ModelPricing, which prices gpt-6-luna and every smooth-* alias at the free tier. - Read usage.cost from the final (empty-choices) stream chunk and from a non-streaming body. On a stream it supersedes the header, since it is sent after pricing. - When a call carries no gateway cost, price its real tokens at the gateway's published list price from /model/info before falling back to ModelPricing. The price book is cached per gateway+key for an hour, and a failed fetch for five minutes. 0/0 rows count as unpriced, not free. List-priced costs still set cost_estimated. Pearl th-c420bf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: ffcc369 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
#209) On the bench, six gpt-6-luna turns died on "stream read error: error decoding response body". chat_stream retried only the initial send, so a body that broke in transit failed the whole turn. The error was also formatted with {e}, which dropped the source chain, so the log could not say whether the cause was a reset, a truncation or a timeout. Body read errors are now a typed StreamInterrupted that carries the full chain. run_with_channel re-asks the same request up to MAX_STREAM_RETRIES (2) times with 500ms/1s backoff, and emits AgentEvent::LlmStreamRetry. It re-asks only while that cannot be observed or repeat anything: the response has not completed, no answer text has reached the consumer, and no tool call has been dispatched. Nothing is pushed to the conversation and nothing is costed until a response completes. An error after a complete response is still surfaced and not retried. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://smooai.atlassian.net/browse/SMOODEV-4057 · pearl th-c420bf
Problem
~/.smooth/usage.jsonlshowedcost_usd = 0on every Big Smooth turn. The tokens part of the bug was already fixed in core 1.11.0, which sendsstream_options.include_usage(#179). Cost was still broken:x-litellm-response-cost*headers before the completion is priced, so on a stream they read 0, andparse_gateway_costmaps 0 toNone.ModelPricing::for_model, which pricesgpt-6-lunaand most gateway models at the free tier.Solution
usage.cost: the cost on the final stream chunk (the one with emptychoices) is now read and emitted asStreamEvent::Cost. LiteLLM sends it withinclude_cost_in_streaming_usage, and OpenRouter always does. On a stream it supersedes the header. A non-streaming call uses it only when no cost header came.input_cost_per_token/output_cost_per_tokenfrom/model/infobefore falling back toModelPricing.cost_estimated.LlmProvider.cost::gateway_list_price,resolution::{fetch_model_pricing, parse_model_pricing}. Changeset: minor.Verification
/model/infois never fetched.cargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings(0 warnings),cargo test --workspace(633 core tests, all passing).🤖 Generated with Claude Code