vllm-switch-controller

An experimental inference switching system with an external control plane and a process-local backup runtime. It routes OpenAI-compatible requests across long-lived model backends and serializes their sleep/wake lifecycle.

Warning

The controller has no authentication, authorization, or TLS. Its data and management APIs share one listener. Keep it on loopback or a trusted management network and place an authenticated reverse proxy in front of it when remote access is required.

Unqualified vLLM refers to the upstream project and its baseline behavior. The fork-specific mechanisms in this repository are named vllm-switch.

Capabilities

For upstream vLLM backends, the controller:

• Routes /v1/chat/completions and /v1/completions by the request's model alias.
• Lists configured aliases at /v1/models.
• Drains in-flight requests before sleeping their backend.
• Serializes transitions and verifies /is_sleeping post-conditions.
• Holds exactly one reservation for complete JSON and streaming request lifetimes.
• Launches a single-GPU backend pool sequentially and stops only verified owned groups.

For vllm-switch backends, additionally:

• Coordinates aggregate CPU-backup accounting without owning tensors or copies.
• Requests cooperative CPU-backup reclaim from vllm-switch under host-memory pressure.

vllm-switch is useful for:

• Running multiple models on a single GPU with minimal HBM usage.
• Managing sleep/wake cycles across multiple vllm-switch processes.
• Using idle CPU memory for backup storage and reclaiming it when needed.
• Using direct I/O for exact disk snapshots and recovery.

This repository ships switch_runtime: pinned CPU pooling, eager snapshots, D2H/H2D, backup publication, concrete reclaim, and exact disk snapshots. The library runs inside each GPU worker through the companion vllm-switch provider interfaces. The controller service only receives aggregate metadata. The vllm-switch-bench repository owns cross-system experiments, results and plots.

Requirements

• Linux and Python 3.11 or newer.
• uv for the source workflow.
• The companion vllm-switch for the full backup feature set. Upstream vLLM can supply basic sleep endpoints but not this coordinator contract.
• Enough GPU memory to initialize each configured model individually.

See the compatibility contract and vllm-switch integration.

Install

From a checkout:

uv sync --frozen --dev
uv run vllm-switch-controller --help

Or install a built wheel:

uv build
uv tool install dist/*.whl

For the optimized worker path, also install this package into the engine environment:

uv pip install --python /path/to/engine/.venv/bin/python /path/to/vllm-switch-controller
export VLLM_SLEEP_BACKEND=switch

The wheel does not install PyTorch or vLLM. It uses the engine's CUDA/PyTorch environment. Keep switch_runtime enabled in VLLM_PLUGINS if a plugin allowlist is configured. The native sleep path remains the default. See Runtime for fixed-weight inference restrictions, engine interfaces, L2 reconstruction, and validation.

Quick start

Create a local configuration. Machine paths belong only in ignored *.local.yaml files:

cp configs/models.launcher.example.yaml configs/models.local.yaml
$EDITOR configs/models.local.yaml

Start the controller:

uv run vllm-switch-controller --config configs/models.local.yaml

In another shell, launch or prepare each backend sequentially:

uv run vllm-switch-launch \
  --config configs/models.local.yaml \
  --pid-file pids.json

Do not send traffic until pool preparation succeeds. Then list aliases and make a request:

curl -fsS http://127.0.0.1:9000/v1/models

curl -fsS http://127.0.0.1:9000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "model-b",
    "messages": [{"role": "user", "content": "Reply with one word."}],
    "max_tokens": 8
  }'

Changing only model drives any necessary drain, sleep, wake, and routing operation. Stop launcher-owned backends with:

uv run vllm-switch-stop --pid-file pids.json

The stop command validates PID, process-group ID, and Linux process start time before signalling the process group. It refuses legacy or stale PID files instead of risking an unrelated process.

Documentation

• Documentation index
• Getting started
• Configuration
• Operations and troubleshooting
• API reference
• Architecture and safety invariants
• CPU backup coordinator protocol
• Process-local runtime and design guides
• vllm-switch integration
• vLLM v0.22.1 delta and runtime call chains
• Compatibility matrix

Development

uv sync --frozen --dev
uv run python -m pytest tests -q
uv run ruff check vllm_switch_controller scripts tests
uv run ruff format --check vllm_switch_controller scripts tests
uv run mypy --ignore-missing-imports vllm_switch_controller
uv build

See CONTRIBUTING.md, SECURITY.md, and the Apache-2.0 license.