Problem
#275 gave custom mode a target_selector policy: a JSON Pointer reads one field out of the verdict and its value is used as a target label. That works when the rubric asks "which target?" and the judge answers with a target name.
Rubrics more often ask for a judgement on a scale that means something to the model — difficulty, complexity, risk — and let the deployment decide what each grade routes to:
{"recommended_tier": "complex"}
Today the only way to route that is to make the rubric answer "strong" instead. That is worse prompting: it asks the model to apply the deployment's routing policy rather than to assess the task, and it means every operating point (conservative / balanced / aggressive) needs its own rewritten rubric even though the assessment is identical and only the mapping differs.
targets cannot express the mapping either — it is the set of models the route may dispatch to, so two entries resolving to one model are rejected.
Proposal
An optional labels map on the target_selector policy:
targets = ["weak", "strong"]
default_target = "strong"
[routes.assistant.policy]
type = "target_selector"
selector = "/recommended_tier"
labels = { simple = "weak", medium = "weak", complex = "strong", reasoning = "strong" }
Several verdict values may share a target. Omit labels and verdict values are target labels exactly as they are today — this is purely additive.
Validation at load time, because both failures are silent at runtime otherwise:
- every value in
labels must name a configured target;
- the table must not be empty (every verdict would be unroutable, which is a config mistake rather than a policy).
A verdict value absent from the map is unroutable in the same way an unknown target label already is, so default_target decides.
Why this is worth a key rather than a prompt
The prompt workaround is real and we have been running it — an appended line telling the judge to answer with the routing target rather than the grade. It costs a rubric edit per operating point, and it silently couples prompt text to routing config: change targets and the prompt is wrong with nothing to catch it. Moving the mapping into config is what makes one rubric serve several operating points.
Builds directly on #275; no change to the selector mechanism itself. Rebased onto current main and green.
Problem
#275 gave custom mode a
target_selectorpolicy: a JSON Pointer reads one field out of the verdict and its value is used as a target label. That works when the rubric asks "which target?" and the judge answers with a target name.Rubrics more often ask for a judgement on a scale that means something to the model — difficulty, complexity, risk — and let the deployment decide what each grade routes to:
{"recommended_tier": "complex"}Today the only way to route that is to make the rubric answer
"strong"instead. That is worse prompting: it asks the model to apply the deployment's routing policy rather than to assess the task, and it means every operating point (conservative / balanced / aggressive) needs its own rewritten rubric even though the assessment is identical and only the mapping differs.targetscannot express the mapping either — it is the set of models the route may dispatch to, so two entries resolving to one model are rejected.Proposal
An optional
labelsmap on thetarget_selectorpolicy:Several verdict values may share a target. Omit
labelsand verdict values are target labels exactly as they are today — this is purely additive.Validation at load time, because both failures are silent at runtime otherwise:
labelsmust name a configured target;A verdict value absent from the map is unroutable in the same way an unknown target label already is, so
default_targetdecides.Why this is worth a key rather than a prompt
The prompt workaround is real and we have been running it — an appended line telling the judge to answer with the routing target rather than the grade. It costs a rubric edit per operating point, and it silently couples prompt text to routing config: change
targetsand the prompt is wrong with nothing to catch it. Moving the mapping into config is what makes one rubric serve several operating points.Builds directly on #275; no change to the selector mechanism itself. Rebased onto current
mainand green.