Approx usage for interrupted streaming requests#55
Conversation
| def approx_text_tokens(s): return (len(s or '') + 2)//3 | ||
|
|
||
| def approx_obj_tokens(o): | ||
| try: s = json.dumps(obj2dict(o), ensure_ascii=False, default=str) | ||
| except Exception: s = str(o) | ||
| return approx_text_tokens(s) |
There was a problem hiding this comment.
@jph00 Should we instead use the tiktoken based estimator from solveit?
| api_name=api_name, | ||
| vendor_name=vendor_name, | ||
| usage=usage) | ||
| chat._track(self.value) |
There was a problem hiding this comment.
A cancelled request exits AsyncChat._call while yielding chunks (async for chunk in res: yield chunk # exits here) and never reaches the rest of the code:
# AsyncChat._call()
...
if stream:
if self.prefill: yield _mk_prefill(self.prefill)
res = astream_with_complete(self, res, postproc=postproc)
async for chunk in res: yield chunk # exits here
res = res.valueSo we manually call chat._track(self.value) here to set c.use for the interrupted request.
| out.append(_chunk_text(chunk)) | ||
| yield postproc(chunk) | ||
| self.value = chunk | ||
| except (GeneratorExit, asyncio.CancelledError): |
There was a problem hiding this comment.
In solveit PR we use asyncio task.cancel() so it raisesasyncio.CancelledError but we also have aclose() in solveit as a safety nest and so we keep GeneratorExit here too.
| def mk_client(model=None, vendor_name=None, api_name=None, api_key=None, base_url=None, xtra_hdrs=None, | ||
| timeout=httpx.Timeout(connect=30, read=300, write=30, pool=10)): | ||
| # %% ../nbs/06_acomplete.ipynb #c714601e | ||
| def resolve_api_vendor(model=None, vendor_name=None, api_name=None, api_key=None, base_url=None): |
There was a problem hiding this comment.
Factored this out to be able to use it during interrupted Completion construction.
| yield postproc(chunk) | ||
| self.value = chunk | ||
| except (GeneratorExit, asyncio.CancelledError): | ||
| api_name,vendor_name,*_ = resolve_api_vendor(chat.model, chat.vendor_name, chat.api_name, chat.api_key, chat.base_url) |
There was a problem hiding this comment.
api_name and vendor_name are inferred in acomplete inside mk_client and not stored in AsyncChat, so we resolve them here using the new helper.
1 participant
Streaming wrappers now build an interrupted
Completionwhen a stream is cancelled or closed before provider usage is returned. The wrapper estimates prompt/output tokens, assumes 80% of input tokens were cached, normalizes the synthetic usage through the provider’s existingnorm_usage, and tracks it with the normalAsyncChatusage accounting.Providers can now register
approx_raw_usagehooks, so approximate usage keeps the same provider-shaped raw usage and cost path as real responses. This adds hooks for OpenAI Responses, OpenAI Chat, Anthropic, and Gemini.This lets callers such as Solveit show and log approximate token usage/cost for interrupted prompts, including cancellations before the first streamed token.
interrupted_usage_half.mov