Skip to content

AI max_tokens budgets sit below a thinking turn on every env-overridable model #254

Description

@NathanTarbert

The four max_tokens budgets in packages/outpost/ai/src/config.ts are sized for models that answer without thinking:

Setting Value Call
maxResponseTokens 2048 generator.generate / generateStream
maxClassifierTokens 512 classifier.classify
maxSentimentTokens 512 analyzeSentiment
maxConfidenceTokens 256 confidence.score

Every one of those models is env-overridable (AI_RESPONSE_MODEL, AI_CLASSIFIER_MODEL, AI_SENTIMENT_MODEL, AI_CONFIDENCE_MODEL), and on Claude Opus 5 and Fable 5 thinking is on by default. A thinking turn consumes the budget before the answer starts, so 256 and 512 are below what one costs.

Why this is the gating item rather than a tuning nit

Two changes have now removed the loud failures on that swap and left this one:

  • #240 stopped temperature being sent to models that reject it, so the swap no longer 400s on every call and publishes the apology fallback.
  • The same PR made an empty extraction a failure at all three secondary call sites, so a thinking-only response degrades visibly instead of returning a fabricated value with degraded: false.

So after #240 the swap is possible and observable. These budgets are what make it safe: with them unchanged, a thinking-default model spends the whole allowance thinking, returns no text, and every classify / sentiment / confidence call degrades to its heuristic on every request. Not broken, not silent — just uniformly useless, which is a worse outcome to debug than a hard failure.

The generator's 2048 is the least alarming of the four and still wants checking, since a truncated answer at stop_reason: max_tokens publishes mid-sentence today — see #231.

What settling it involves

Not just raising the numbers. It needs a decision per call site:

  1. Should the cheap calls think at all? For classify, sentiment and confidence the answer is probably no — they emit small JSON. thinking: { type: 'disabled' } with output_config.effort: 'low' is valid on Opus 5 and cheaper than a raised ceiling, but it is a 400 on Fable 5, where thinking is always on. So "disable thinking" is not portable and the budget has to cover a thinking turn wherever it is not.
  2. What does the generator's ceiling become, given a thinking turn plus a full support answer, and what does that do to cost per ticket.
  3. Where the numbers live. They are constants today; if they have to vary by model, that is a per-model table rather than four integers.

Raising them blindly costs money on every ticket for headroom most requests do not use, which is why this wants an owner and a decision rather than a quick bump.

Origin

Flagged from two directions during review of #240: from the empty-response side (a thinking-only reply is what the new guard catches) and from the budget side (why that reply happens). Filed rather than mentioned a third time in a PR body.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: aiAI response pipeline, groundedness, confidenceroadmapTracked on the Outpost roadmaproadmap: nextRoadmap horizon: after launch path clears

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions