Skip to content

feat: record models, adapters, benchmarks and promotions in MLflow - #36

Merged
github-actions[bot] merged 1 commit into
mainfrom
feat/23-mlflow-governance
Oct 4, 2026
Merged

github-actions[bot] merged 1 commit into
mainfrom
feat/23-mlflow-governance

Conversation

@Yash-Chindam

Copy link
Copy Markdown
Owner

What

Closes the MLflow gap in the spec (§6, §7.5, §17): "MLflow stores models, adapters, benchmark evidence and lifecycle state" and "tracks artifacts, benchmarks, stages and promotion history". Also documents the external provider path from #35, which shipped without a README section.

How

llm_router.governance syncs the reviewed catalog into an MLflow model registry and tracking store.

Catalog record In MLflow
Model or adapter revision One model version tagged with its card, artifact location and checksum
Lifecycle stage Version tag plus a staging / production alias
Benchmark run One run: measurements as metrics, provenance as parameters
Stage change One appended run in a promotions experiment (from, to, commit, policy version)
  • plan is a pure diff between catalog and store; sync applies it; verify exits 3 on drift; history prints promotions for one subject.
  • Governance flows one way. The gateway never reads MLflow, so a hand-edited stage changes nothing in production and is reported as drift.
  • A recorded revision that turns up with a different checksum is refused outright.
  • Replaced or removed revisions are retired to deprecated, never deleted, so rollback targets stay on record.
  • Models registered in MLflow by anyone else are not touched.
  • deploy/kubernetes/mlflow.yaml: MLflow server with proxied artifacts, its own ExternalSecret (the gateway is never given the database or object-store credentials), and a network policy admitting only the delivery pipeline namespace.
  • CD renders governance-plan.json alongside the canary plan.

Tests

  • tests/unit/test_governance.py: plan, idempotence, promotion history, supersession, removal, immutability, metadata updates, drift repair, CLI.
  • tests/integration/test_governance_mlflow.py: the same behaviour against a real MLflow client on SQLite, including alias movement on promotion and demotion.
  • Manifest contract tests for the MLflow network policy, credential separation and artifact proxying.

Local: ruff and mypy clean, 369 tests pass, 98% coverage, Playwright 9/9.

Limits

  • The sync records where an artifact belongs; it does not upload weights.
  • Engine variants (quantized / speculative) are not recorded as MLflow versions yet; they carry no artifact revision of their own in the catalog.
  • The MLflow server manifest has not been run on a cluster, and sync against a live server stays disabled in CD with the rest of deploy.
  • dev now installs mlflow-skinny, sqlalchemy and alembic so CI exercises the real client.

🤖 Generated with Claude Code

Sync the governed catalog into an MLflow model registry and tracking store,
verify it for drift, and keep an append-only promotion history. Governance
flows one way: the gateway never reads MLflow. Adds the MLflow server
manifest with its own credentials and network policy, renders the governance
plan in CD, and documents the external provider path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions
github-actions Bot merged commit d639824 into main Oct 4, 2026
6 checks passed
@github-actions
github-actions Bot deleted the feat/23-mlflow-governance branch October 4, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant