A model is picked before its provider's credentials are known to work, so a deployment that can reach a model by one route fails when the route it happens to try is unconfigured. OpenRouter serves models from many vendors, so it is usually a viable second route for a model named for its origin vendor.
Companion to #106, which covers falling back after a configured provider fails at request time (outage, retries exhausted). This issue is the step before: choosing a route that is credentialed at all.
What happens today
EvaluationRunnerService#judge_provider takes the first provider with any credential, in a fixed order:
%i[anthropic openai openrouter].find do |name|
owner_provider_options(name).any? || global_provider_token?(name)
end
and judge_class then builds an agent with generate_with provider, model: @evaluation.judge_model.
The provider and the model are chosen independently, so a judge model of claude-opus-5 can be paired with whichever provider happened to win the scan. Presence of a key is treated as proof the route works — an ANTHROPIC_API_KEY that is set but out of credit still wins the scan and takes the run down.
ModelSpec.parse has the routing knowledge this needs and is not consulted here: given openrouter among the offered providers it already resolves a vendor-prefixed name to it.
What it should do
Resolve model and provider together, preferring a route that is actually usable:
- An explicitly named provider wins — never silently re-route what the caller asked for.
- A vendor-native route when that vendor is credentialed (
claude-opus-5 → anthropic).
- Otherwise a gateway that serves the model —
anthropic/claude-opus-5 via openrouter — rather than failing.
- Report the chosen route on the run, so a judge that answered from a different vendor than the label suggests is visible rather than surprising.
Worth deciding as part of this: whether "credentialed" should mean more than "a key is present". A cheap liveness check would have caught the empty-balance case above, at the cost of a request per run.
Where this bites
The judge is the clearest case, but the same pairing exists wherever a provider is inferred separately from a model — ScenarioEvaluationRunner#model_specs falls back to @evaluation.agent.provider for a bare name, and AgentExecutionService#requested_provider takes the agent's provider regardless of what requested_model names.
Related: #416 (OpenRouter's provider needs the openai gem at runtime), which is another way a nominally configured route fails only when exercised.
A model is picked before its provider's credentials are known to work, so a deployment that can reach a model by one route fails when the route it happens to try is unconfigured. OpenRouter serves models from many vendors, so it is usually a viable second route for a model named for its origin vendor.
Companion to #106, which covers falling back after a configured provider fails at request time (outage, retries exhausted). This issue is the step before: choosing a route that is credentialed at all.
What happens today
EvaluationRunnerService#judge_providertakes the first provider with any credential, in a fixed order:and
judge_classthen builds an agent withgenerate_with provider, model: @evaluation.judge_model.The provider and the model are chosen independently, so a judge model of
claude-opus-5can be paired with whichever provider happened to win the scan. Presence of a key is treated as proof the route works — anANTHROPIC_API_KEYthat is set but out of credit still wins the scan and takes the run down.ModelSpec.parsehas the routing knowledge this needs and is not consulted here: givenopenrouteramong the offered providers it already resolves a vendor-prefixed name to it.What it should do
Resolve model and provider together, preferring a route that is actually usable:
claude-opus-5→anthropic).anthropic/claude-opus-5viaopenrouter— rather than failing.Worth deciding as part of this: whether "credentialed" should mean more than "a key is present". A cheap liveness check would have caught the empty-balance case above, at the cost of a request per run.
Where this bites
The judge is the clearest case, but the same pairing exists wherever a provider is inferred separately from a model —
ScenarioEvaluationRunner#model_specsfalls back to@evaluation.agent.providerfor a bare name, andAgentExecutionService#requested_providertakes the agent's provider regardless of whatrequested_modelnames.Related: #416 (OpenRouter's provider needs the
openaigem at runtime), which is another way a nominally configured route fails only when exercised.