Skip to content

feat(libsy): bound judge consultations with a configurable deadline #346

Description

@gburachas

Problem

The classifier judge call sits in front of the routed call. A judge that stalls stalls the turn behind it, and that consultation has no bound of its own — the route waits as long as the upstream is willing to hold the connection.

This is the one LLM call in the request path that the user is not waiting on for content. Its entire value is deciding which tier answers, and that value decays to zero the moment it costs more latency than the routing saves. There is no configuration that expresses "consult the judge, but not for longer than this".

Proposal

judge_deadline_ms on the llm_classifier route:

[routes.assistant]
type = "llm_classifier"
judge_deadline_ms = 1500

On expiry the judge counts as unavailable and routing falls back through exactly the same fail-open path as any other judge failure — no new routing behaviour, no new failure mode to reason about. None (the default) leaves it unbounded, so existing deployments are unaffected.

Two details that matter:

  • The whole consultation is bounded, not just the HTTP call. A judge that returns its headers promptly and then stalls mid-stream would otherwise hold the turn open just as long. The timeout therefore wraps both the call and the response aggregation.
  • Expiry is reported through the existing fail-open taxonomy (reason = "deadline_exceeded"), so it lands on /metrics via record_classifier_fail_open alongside the other reasons rather than as an untracked silent path. The message is libsy-authored text plus the configured duration, so it carries no upstream content.

A zero deadline is rejected at load time rather than treated as "no deadline" — that is a config mistake with a silent and expensive failure mode.

Relationship to #277

Different layer, and they compose. #277 bounds an individual upstream request at the client. This bounds the routing decision regardless of how many requests or how much streaming it involves.

Rebased onto current main and green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions