feat: roll back a failed model or policy canary without a person

[Yash-Chindam]

What

Closes the spec's design target "automatic rollback after failed readiness or canary criteria" (§16) for the model and policy tracks. Until now their rollback was decided and printed, and nothing acted on it.

How

llm_router.rollout:

• deploy runs helm upgrade --install --atomic, so a release whose workloads never become ready is restored by Helm itself.
• evaluate judges one canary plan with the existing rule and, on a rollback verdict for the model or policy track, runs helm rollback to the previous release. Exit codes: 0 promote, 2 hold, 3 rolled back, 4 the rollback command failed.
• A plan with no recorded rollback target runs nothing and stops.
• A failed adapter canary runs nothing: the gateway already withdraws it without a redeploy.
• Nothing runs without --execute.

CD renders the atomic deploy command as an artifact.

Tests

tests/unit/test_rollout.py (16 tests): verdict-to-command for each track; dry run changes nothing; --execute runs the rollback and reports a failed one; a shared outage with a baseline is not blamed on the canary; and a check against the real helm binary that every flag used exists.

Limits

• The Helm commands have not been run against a cluster.
• The observation is a file the caller supplies. Nothing collects a model or policy canary's metrics from Prometheus, and nothing schedules evaluate; that loop belongs to whatever runs the rollout.
• helm rollback restores the previous Helm release, which is the recorded target only if releases were made in catalog order.

🤖 Generated with Claude Code