You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(ai): cap output tokens and document text per model call #123
Found while working on #109 (AI cost limits). The cost of one model call is bounded only by the model's own limits: GenerateContentConfig in services/ai/src/requestflow_ai/extraction/model_client.py sets no max_output_tokens and no thinking budget, and the prompt carries the whole document text (up to the 20 MB file cap). The upload caps (UPLOAD_MAX_PER_HOUR × UPLOAD_MAX_FILES) bound the number of calls, not their size, so one large document can cost many times the sample figures (≈ 0.01–0.03 USD per document).
Acceptance criteria
max_output_tokens (and, if the model supports it, a thinking budget) set from configuration with a default derived from the eval set (largest recorded output × safety factor); a truncated answer becomes a visible, non-retryable processing error – never a partial found
A cap on the document text sent per call (characters or estimated tokens) with a clear outcome for larger documents (warning + processing error, or a documented truncation that keeps grounding honest) – decision in the PR
Eval gate unchanged (replay), unit tests for both caps, README + .env.example + docs/technical/operations.md → AI cost updated
Related cost multiplier found in the #124 review: when a processing job is retried (model_output_invalid / model_error are retryable, retryLimit: 5 in src/features/jobs/queues.ts), every document of the request is sent to the model again – including documents whose answer already succeeded – and the tokens of failed attempts are never recorded. Worth deciding here (or in a separate issue) whether successful per-document results should survive a job retry.
Activity
Fluory commentedon Oct 16, 2026
Related cost multiplier found in the #124 review: when a processing job is retried (
model_output_invalid/model_errorare retryable,retryLimit: 5insrc/features/jobs/queues.ts), every document of the request is sent to the model again – including documents whose answer already succeeded – and the tokens of failed attempts are never recorded. Worth deciding here (or in a separate issue) whether successful per-document results should survive a job retry.Generated by Claude Code