Skip to content

Resolve a model and its provider together, falling back to a gateway that serves it #431

Description

@TonsOfFun

A model is picked before its provider's credentials are known to work, so a deployment that can reach a model by one route fails when the route it happens to try is unconfigured. OpenRouter serves models from many vendors, so it is usually a viable second route for a model named for its origin vendor.

Companion to #106, which covers falling back after a configured provider fails at request time (outage, retries exhausted). This issue is the step before: choosing a route that is credentialed at all.

What happens today

EvaluationRunnerService#judge_provider takes the first provider with any credential, in a fixed order:

%i[anthropic openai openrouter].find do |name|
  owner_provider_options(name).any? || global_provider_token?(name)
end

and judge_class then builds an agent with generate_with provider, model: @evaluation.judge_model.

The provider and the model are chosen independently, so a judge model of claude-opus-5 can be paired with whichever provider happened to win the scan. Presence of a key is treated as proof the route works — an ANTHROPIC_API_KEY that is set but out of credit still wins the scan and takes the run down.

ModelSpec.parse has the routing knowledge this needs and is not consulted here: given openrouter among the offered providers it already resolves a vendor-prefixed name to it.

What it should do

Resolve model and provider together, preferring a route that is actually usable:

  1. An explicitly named provider wins — never silently re-route what the caller asked for.
  2. A vendor-native route when that vendor is credentialed (claude-opus-5anthropic).
  3. Otherwise a gateway that serves the model — anthropic/claude-opus-5 via openrouter — rather than failing.
  4. Report the chosen route on the run, so a judge that answered from a different vendor than the label suggests is visible rather than surprising.

Worth deciding as part of this: whether "credentialed" should mean more than "a key is present". A cheap liveness check would have caught the empty-balance case above, at the cost of a request per run.

Where this bites

The judge is the clearest case, but the same pairing exists wherever a provider is inferred separately from a model — ScenarioEvaluationRunner#model_specs falls back to @evaluation.agent.provider for a bare name, and AgentExecutionService#requested_provider takes the agent's provider regardless of what requested_model names.

Related: #416 (OpenRouter's provider needs the openai gem at runtime), which is another way a nominally configured route fails only when exercised.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions