feat: serve every catalog model from Ray worker pools split by accelerator

[Yash-Chindam]

What

Closes most of the deployment topology gap in the spec (§7.3, §17): Ray head and worker topology, GPU pools separated by accelerator type, NVIDIA GPU Operator support, the GPU exporter, and Grafana. Helm follows in its own PR.

Why an overlay

The base runs one vLLM engine, which can serve one model, while the catalog holds three local models on three accelerator types. deploy/overlays/ray replaces that engine with a KubeRay RayService that serves all of them behind one OpenAI-compatible endpoint. The base is left as it was, so the single-engine path and its gateway-side engine telemetry keep working.

How

• llm_router.topology generates the RayService from the catalog: a head that runs no model, and one worker group per accelerator type sized from the autoscaling bounds of its models. A worker holds as many GPUs as the largest replica on its pool, so a tensor-parallel model fits on one node.
• The embedded Serve application keeps only fields Ray's LLMConfig accepts, and loads weights from the location governance records in MLflow.
• The overlay deletes the vllm-serve Deployment, Service and NetworkPolicy, repoints the gateway, and adds a Ray network policy and a PodMonitor.
• deploy/gpu-operator/values.yaml: device plugin, GPU feature discovery (the gpu.product label the pools select on), and the DCGM exporter with a ServiceMonitor.
• Two Grafana dashboards shipped as a labelled ConfigMap, and a PrometheusRule with eight alerts.
• CD now fails on topology drift and validates both rendered topologies with kubeconform.

Tests

tests/unit/test_topology.py (20 tests): pool sizing, head and worker placement, LLMConfig field allow-list, artifact sources matching governance, pod security contract, generated-file drift, overlay structure, a real kubectl kustomize render of the overlay, and a check that every dashboard and alert expression queries a metric the gateway actually publishes.

Local: ruff and mypy clean, 389 tests pass, 98% coverage.

Limits

• Nothing here has been applied to a cluster. The Ray image digest, GPU product labels and node sizes are placeholders.
• Under the overlay the gateway's own router_engine_* and router_gpu_* gauges stay empty, so live load does not influence routing; engine metrics go to Prometheus from the Ray pods instead.
• Ray resolves adapters at <artifact root>/adapters/<adapter id>, without the revision segment governance records; the deploy step has to publish the promoted revision there.
• Dashboards chart gateway and DCGM series only. Ray's own vLLM series are scraped but not charted, because their exported names were not verified.

🤖 Generated with Claude Code