Skip to content

feat(ai): cap output tokens and document text per model call #123

Open

Description

@Fluory

Goal

Found while working on #109 (AI cost limits). The cost of one model call is bounded only by the model's own limits: GenerateContentConfig in services/ai/src/requestflow_ai/extraction/model_client.py sets no max_output_tokens and no thinking budget, and the prompt carries the whole document text (up to the 20 MB file cap). The upload caps (UPLOAD_MAX_PER_HOUR × UPLOAD_MAX_FILES) bound the number of calls, not their size, so one large document can cost many times the sample figures (≈ 0.01–0.03 USD per document).

Acceptance criteria

  • max_output_tokens (and, if the model supports it, a thinking budget) set from configuration with a default derived from the eval set (largest recorded output × safety factor); a truncated answer becomes a visible, non-retryable processing error – never a partial found
  • A cap on the document text sent per call (characters or estimated tokens) with a clear outcome for larger documents (warning + processing error, or a documented truncation that keeps grounding honest) – decision in the PR
  • Eval gate unchanged (replay), unit tests for both caps, README + .env.example + docs/technical/operations.md → AI cost updated

Not part of this task

Security/Privacy affected?

No – cost and availability only.

Activity

  1. Fluory commented on Oct 16, 2026

    @Fluory
    OwnerAuthor

    Related cost multiplier found in the #124 review: when a processing job is retried (model_output_invalid / model_error are retryable, retryLimit: 5 in src/features/jobs/queues.ts), every document of the request is sent to the model again – including documents whose answer already succeeded – and the tokens of failed attempts are never recorded. Worth deciding here (or in a separate issue) whether successful per-document results should survive a job retry.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    aiLLM, prompts, evalsfeatureNew capability

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions