Repository navigation
UN-4223 [FIX] Cap a streamed LLM completion at a total time limit - #2314
Open
johnyrahul wants to merge 2 commits into
Open
johnyrahul wants to merge 2 commits into
johnyrahul wants to merge 2 commits into
Conversation
A read timeout resets on every chunk, so it never stops a model that keeps generating. In production single Gemini calls streamed for 2+ hours with no timeout, error or retry logged, and memory grew to ~10 GB until pod liveness took whole executor pods offline. collect_with_retry takes an optional max_seconds that caps the whole call, retries included, and raises DeadlineExceededError when it passes; the stream is closed and not replayed. LLM.complete() applies it to every streamed completion: LLM_STREAM_MAX_SECONDS, default 1800 s, 0 disables it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| f"stream exceeded its {max_seconds:g}s total time " | ||
| f"limit after {len(items)} chunk(s)" | ||
| ) | ||
| return items |
Contributor
There was a problem hiding this comment.
If the last chunk arrives within the limit but the stream finishes after it, this return path never checks the deadline. LLM.complete() can then return a successful response despite exceeding the stated total limit.
Knowledge Base Used: SDK LLM runtime
Prompt To Fix With AI
This is a comment left during a code review.
Path: unstract/sdk1/src/unstract/sdk1/utils/retry_utils.py
Line: 328
Comment:
**Clean finish skips deadline**
If the last chunk arrives within the limit but the stream finishes after it, this return path never checks the deadline. `LLM.complete()` can then return a successful response despite exceeding the stated total limit.
**Knowledge Base Used:** [SDK LLM runtime](https://app.greptile.com/zipstack/-/custom-context/knowledge-base/zipstack/unstract/-/docs/sdk1-llm-runtime.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This comment has been minimized.
This comment has been minimized.
… limits A retryable failure just before the limit could sleep through the backoff and then send a new request after the deadline. Raise instead when the backoff would end past it. LLM_STREAM_MAX_SECONDS values "inf" and "nan" parse as floats and would quietly switch the limit off; treat them like any other invalid value and fall back to the default with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Contributor
Unstract test resultsPer-group results
Critical paths
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
LLM.complete()now stops a streamed completion once it has run forLLM_STREAM_MAX_SECONDSin total (default 1800 s, retries included) and raisesLLMErrornaming the adapter.collect_with_retrygains an optionalmax_secondsthat raisesDeadlineExceededErroronce it passes. The stream is closed and not replayed.Why
UN-4223: on 2026-10-01, 10-02 and 10-03, single
vertex_ai/gemini-3.1-flash-litecalls inunstract-worker-pg-executorran for 2+ hours. Pod liveness eventually took whole executor pods offline, which paged a P0 on the US Checkly check.Prod logs for the two hung processes on 10-02 (
5hgn2/p24,rm772/p12) show what was happening:Retry N/M, no timeout, no error. Those lines are logged at WARNING/ERROR and do appear for other failures in the same pods during that window.collect_with_retryholding an ever-growing list of chunks.A read timeout resets on every chunk, so no timeout setting can stop a model that keeps generating. A total wall-clock limit can, and it works for every provider, not just Gemini.
How
collect_with_retry(..., max_seconds=...):DeadlineExceededErroris raised.LLM._complete_via_streampassesmax_seconds=_stream_max_seconds(), which readsLLM_STREAM_MAX_SECONDS:llm.pycites Anthropic replies of around 16 minutes.EXECUTOR_RESULT_TIMEOUT. The caller gets a real error instead of a timeout on a job that keeps running.Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
LLM_STREAM_MAX_SECONDS, or set it to0to turn the limit off.enable_streaming=false),acomplete()andstream_complete()are unchanged.stream_complete()yields chunks to its caller as they arrive, so the caller owns its pacing; it has no production callers today.collect_with_retrycallers that don't passmax_secondsbehave exactly as before. The existing tests are unchanged and pass.Database Migrations
Env Config
LLM_STREAM_MAX_SECONDS(default 1800,0disables). Documented inworkers/sample.envnext toEXECUTOR_RESULT_TIMEOUT. Nothing needs to be set to get the default.Relevant Docs
workers/sample.envRelated Issues or PRs
timeout, which bounds the silent stream case this PR doesn't cover.terminationGracePeriodSeconds.Dependencies Versions
Notes on Testing
collect_with_retry(3 new cases):LLM.complete()(newtests/test_llm_stream_deadline.py):LLMErrorat the limit, after exactly one call.0, negative, invalid with warning.max_secondsnot passed fromLLM, the endless-stream test hangs (killed bytimeoutafter 20 s), which is the production behaviour. With it, the test fails fast at the limit.ruff0.3.4 (the pre-commit pin) is clean.Screenshots
Checklist
I have read and understood the Contribution Guidelines.
🤖 Generated with Claude Code