Problem
The classifier judge call sits in front of the routed call. A judge that stalls stalls the turn behind it, and that consultation has no bound of its own — the route waits as long as the upstream is willing to hold the connection.
This is the one LLM call in the request path that the user is not waiting on for content. Its entire value is deciding which tier answers, and that value decays to zero the moment it costs more latency than the routing saves. There is no configuration that expresses "consult the judge, but not for longer than this".
Proposal
judge_deadline_ms on the llm_classifier route:
[routes.assistant]
type = "llm_classifier"
judge_deadline_ms = 1500
On expiry the judge counts as unavailable and routing falls back through exactly the same fail-open path as any other judge failure — no new routing behaviour, no new failure mode to reason about. None (the default) leaves it unbounded, so existing deployments are unaffected.
Two details that matter:
- The whole consultation is bounded, not just the HTTP call. A judge that returns its headers promptly and then stalls mid-stream would otherwise hold the turn open just as long. The timeout therefore wraps both the call and the response aggregation.
- Expiry is reported through the existing fail-open taxonomy (
reason = "deadline_exceeded"), so it lands on /metrics via record_classifier_fail_open alongside the other reasons rather than as an untracked silent path. The message is libsy-authored text plus the configured duration, so it carries no upstream content.
A zero deadline is rejected at load time rather than treated as "no deadline" — that is a config mistake with a silent and expensive failure mode.
Relationship to #277
Different layer, and they compose. #277 bounds an individual upstream request at the client. This bounds the routing decision regardless of how many requests or how much streaming it involves.
Rebased onto current main and green.
Problem
The classifier judge call sits in front of the routed call. A judge that stalls stalls the turn behind it, and that consultation has no bound of its own — the route waits as long as the upstream is willing to hold the connection.
This is the one LLM call in the request path that the user is not waiting on for content. Its entire value is deciding which tier answers, and that value decays to zero the moment it costs more latency than the routing saves. There is no configuration that expresses "consult the judge, but not for longer than this".
Proposal
judge_deadline_mson thellm_classifierroute:On expiry the judge counts as unavailable and routing falls back through exactly the same fail-open path as any other judge failure — no new routing behaviour, no new failure mode to reason about.
None(the default) leaves it unbounded, so existing deployments are unaffected.Two details that matter:
reason = "deadline_exceeded"), so it lands on/metricsviarecord_classifier_fail_openalongside the other reasons rather than as an untracked silent path. The message is libsy-authored text plus the configured duration, so it carries no upstream content.A zero deadline is rejected at load time rather than treated as "no deadline" — that is a config mistake with a silent and expensive failure mode.
Relationship to #277
Different layer, and they compose. #277 bounds an individual upstream request at the client. This bounds the routing decision regardless of how many requests or how much streaming it involves.
Rebased onto current
mainand green.