Skip to content

feat(libsy): map graded classifier verdicts onto routing targets #348

Description

@gburachas

Problem

#275 gave custom mode a target_selector policy: a JSON Pointer reads one field out of the verdict and its value is used as a target label. That works when the rubric asks "which target?" and the judge answers with a target name.

Rubrics more often ask for a judgement on a scale that means something to the model — difficulty, complexity, risk — and let the deployment decide what each grade routes to:

{"recommended_tier": "complex"}

Today the only way to route that is to make the rubric answer "strong" instead. That is worse prompting: it asks the model to apply the deployment's routing policy rather than to assess the task, and it means every operating point (conservative / balanced / aggressive) needs its own rewritten rubric even though the assessment is identical and only the mapping differs.

targets cannot express the mapping either — it is the set of models the route may dispatch to, so two entries resolving to one model are rejected.

Proposal

An optional labels map on the target_selector policy:

targets = ["weak", "strong"]
default_target = "strong"

[routes.assistant.policy]
type = "target_selector"
selector = "/recommended_tier"
labels = { simple = "weak", medium = "weak", complex = "strong", reasoning = "strong" }

Several verdict values may share a target. Omit labels and verdict values are target labels exactly as they are today — this is purely additive.

Validation at load time, because both failures are silent at runtime otherwise:

  • every value in labels must name a configured target;
  • the table must not be empty (every verdict would be unroutable, which is a config mistake rather than a policy).

A verdict value absent from the map is unroutable in the same way an unknown target label already is, so default_target decides.

Why this is worth a key rather than a prompt

The prompt workaround is real and we have been running it — an appended line telling the judge to answer with the routing target rather than the grade. It costs a rubric edit per operating point, and it silently couples prompt text to routing config: change targets and the prompt is wrong with nothing to catch it. Moving the mapping into config is what makes one rubric serve several operating points.

Builds directly on #275; no change to the selector mechanism itself. Rebased onto current main and green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions