feat: canary models, adapters, and policies separately with automatic rollback

[Yash-Chindam]

Seventh PR closing gaps between the design spec and the implementation.

Gap this closes

§13 "Canary models, adapters and router policies separately" and the design target "Automatic rollback after failed readiness or canary criteria". Previously there was one combined plan whose rollback triggers were four strings, and nothing evaluated them.

Behaviour change

A staging adapter used to compete with the production one on measured quality delta, so a staged adapter with the larger gain took 100% of traffic. It now takes canary_traffic_percent (default 10). A Router built without a monitor gives a staged adapter no traffic at all.

What changed

• canary.py: one CanaryPlan per canary on its own track (model, adapter, policy), with criteria derived from the catalog: the strictest tier latency objective among the models served, and their benchmarked quality less a tolerance.
• evaluate(): failed readiness rolls back with no sample; after 50 requests, error rate, p95, or quality over the line rolls back. Error rate and latency are compared with the stable arm, so a shared outage does not roll back an innocent change.
• Adapter track in the gateway: deterministic traffic split by tenant and prompt, automatic suspension on failure, and no caching of canary responses.
• GET /v1/registry/canaries, plus router_canary_requests_total and router_canary_rollbacks_total.
• python -m llm_router.canary prints the plans; with --plan and --observation it exits 0/2/3 for promote/hold/rollback. CD now renders all plans instead of one.
• RoutePolicy gains previous_version, canary settings, and per-tier latency objectives. serving.canary_config is removed in favour of the per-track plans.

Not done

• Model and policy rollback is decided but not actuated. The command returns the verdict and the rollback target; no rollout controller calls it yet. That wiring belongs with the deployment topology work.
• Canary state is per gateway replica and is lost on restart, so a restarted replica re-admits a suspended adapter until it fails its sample again.
• The live quality signal is structured-output validity only, as elsewhere.

Test plan

• 35 new tests, including a failing staged adapter being suspended after exactly its 50-request sample with production serving thereafter, and an outage shared by both arms not triggering a rollback
• ruff format --check ., ruff check ., mypy clean
• pytest tests/unit tests/integration: 314 passed, coverage 98%
• Playwright end-to-end suite: 9 passed locally

🤖 Generated with Claude Code