feat(instance): attach to an externally-managed LiteLLM deployment

[davidgibbons]

Closes #29.

Adds spec.workload.managed: false so a LiteLLMInstance can describe a LiteLLM proxy the operator did not deploy, making the entity CRDs usable against an existing installation without handing the operator the workload.

This implements the issue's proposal, plus the endpoint half of "worth considering alongside". readinessRef is deliberately not implemented — see below.

What changes

spec.workload.managed (*bool, defaults true). When false, reconcileResources and reconcileAutoRollback are skipped entirely. Nothing is created, nothing existing is adopted or mutated, and no Forbidden is logged on every loop — the instance reports its actual state instead of sitting Degraded about resources it doesn't manage.

spec.workload.endpoint (optional, unmanaged only, CEL-enforced). Without it the endpoint stays today's formula, which requires the CR to be named after a Service it doesn't own. With it you can attach to a Service under another name, in another namespace, or to a proxy outside the cluster.

Readiness for an unmanaged instance comes from the admin API answering at status.endpoint (CheckLiveness, which probeInstanceHealth already performs) rather than a name-matched Deployment. This is what makes endpoint coherent: with an explicit endpoint there may be no Deployment of that name to look at, and the proxy may be a StatefulSet or off-cluster. It also closes a hole in the RBAC workaround — a Deployment that merely shares the CR's name can no longer make an unreachable proxy report ready: true.

updateInstanceStatus computes the endpoint before deriving readiness, since readiness now depends on it.

apiVersion: litellm.palena.ai/v1alpha1
kind: LiteLLMInstance
metadata:
  name: my-gateway
spec:
  workload:
    managed: false
    endpoint: http://litellm.platform.svc:4000   # optional
  masterKey:
    secretRef:
      name: litellm-master-key
      key: LITELLM_MASTER_KEY
  database: {}

Status differences when unmanaged

	Managed (default)	Unmanaged
Workload + optional resources	Created and reconciled	Never touched
Auto-rollback	Active	Skipped
Database migration Job	Created when enabled	Never — the proxy owns its schema
status.ready	A Deployment replica is ready	Admin API answers at status.endpoint
status.replicas / readyReplicas	Deployment counts	0
status.version	spec.image.tag	Empty, unless the proxy discloses litellm_version
PodsHealthy condition	Set	Absent — the operator owns no pods
Ready reason	AllResourcesReady / DeploymentNotReady	ProxyReachable / ProxyNotReachable
Health probing, config sync, entity CRDs, finalizer cleanup	Active	Active

status.version is worth calling out. spec.image.tag describes nothing the operator deployed, so leaving it in place would print a fabricated latest in the VERSION print column. probeInstanceHealth already fetches /health/readiness and discarded the payload, so it now reads litellm_version off it — but that field is only present when the proxy's own general_settings sets allow_public_health_readiness_details: true. The endpoint is unauthenticated, so the master key does not unlock it, and the default payload is {"status", "db"}. In the common case status.version is therefore empty for an unmanaged instance. That is deliberate: empty means "the operator does not know", which is true, where latest would be a guess. Both payload shapes are covered by tests. Say the word if you would rather it fell back to the image tag.

Decisions worth a look

Managed is *bool, not bool. An envtest spec caught this: with a plain bool and no omitempty, a Go-typed client constructing WorkloadSpec{Endpoint: ...} marshals "managed": false explicitly, so the API-server default never fires and the caller silently gets an unmanaged instance. Nil now means managed, matching enableServiceLinks and friends upstream. YAML users see no difference — workload: {} still defaults to true.

No readinessRef. Probing the endpoint covers Deployment, StatefulSet and off-cluster uniformly, so kind-switching would add surface without adding capability. Happy to add it if you'd rather have the Kubernetes-object signal.

spec.database.migration is gated too, and ignored entirely when unmanaged. An externally-managed proxy owns its own schema — LiteLLM migrates on startup, and whatever deployed it ships its own migration hook, so the operator would be a second migrator racing the real one. Mechanically it is worse than that: BuildMigrationJob takes its image from spec.image.tag (falling back to latest), which for an unmanaged instance describes nothing the operator deployed. Leaving it open meant running prisma migrate deploy at an arbitrary schema version against a database the operator does not own. DatabaseReady reports WorkloadUnmanaged, and the message says the migration was ignored rather than skipping it silently when one was configured anyway.

Tests

internal/controller/litellminstance_unmanaged_test.go — no resources created; a name-colliding Helm-owned Deployment left byte-identical, un-mutated and un-adopted; readiness tracking the probe in both directions; a name-matched Deployment not faking readiness; version read from the proxy; workloadManaged and instanceEndpoint tables.

internal/controller/litellminstance_controller_test.go — four envtest specs for the CEL rule and the managed default, since only envtest runs API-server validation.

What was verified against upstream

/health/liveliness and /health/readiness are the paths the operator's shipped client already uses for every managed instance, so the readiness signal reuses a proven code path rather than a new one. /health/liveliness returns the bare string I'm alive!; CheckLiveness passes a nil result and so never unmarshals it, which is why it is safe as a reachability probe. The litellm_version gating described above was checked against litellm/proxy/health_endpoints/_health_endpoints.py on main, not assumed — the field had been declared in this repo's ReadinessResponse since the initial commit but never read by anything until now.

spec.workload.endpoint is trimmed of trailing slashes by the client (strings.TrimRight(endpoint, "/")), so http://host:4000/ and http://host:4000 behave identically. The CRD pattern is ^https?://[^\s/?#]+, which requires a host — the value becomes an outbound request URL, so a scheme alone should not pass.

Housekeeping

make manifests generate sync-helm-crds and make bundle are committed. make test, make lint, helm lint and helm template pass locally. Commit is signed off, README and docs/reference/litellminstance.md document the field, and there's a sample at config/samples/litellm_v1alpha1_litellminstance_unmanaged.yaml.

A small drive-by: the master-key-with-autogenerate-fallback block was copy-pasted in resolveInstance and probeInstanceHealth; it's now one masterKeyRef helper that the new readiness probe also uses.

[schneidermr]

Thanks for this — it's a genuinely excellent PR, and the write-up made reviewing it a pleasure. A few things I want to call out before the nitpicks, because they're the parts I'd have most likely gotten wrong myself:

• Declining readinessRef was the right call. I asked for it, and your reasoning for leaving it out is better than my reasoning for wanting it: probing the endpoint covers Deployment, StatefulSet and off-cluster uniformly, so a kind-switch would have added surface without adding capability. It also closes a hole I'd missed — a Deployment that merely shares the CR's name can no longer make an unreachable proxy report ready: true.
• Managed *bool rather than bool. That a Go-typed client marshals an explicit "managed": false and quietly skips the API-server default is exactly the kind of thing that gets found in production six months later. Nice catch, and good that an envtest spec pinned it.
• Leaving status.version empty instead of echoing spec.image.tag. Empty is honest; latest would have been a guess.

I also test-merged it against current main (which has moved a few commits) and ran the suite. The Go code merges cleanly — common.go, litellminstance_controller.go and litellminstance_resources_test.go all auto-merge — and make test is green afterwards, including your four CEL specs. Only two conflicts, both in generated files. So there's nothing structural in the way here.

Two things from the checklist in #29 are still open, though, and they're the two I'd want to settle before merging.

---

1. Config inertness still needs a status condition

This is the one I care most about. Right now WorkloadUnmanaged appears exactly once in the tree — on DatabaseReady — and everything else is documented only.

The reason docs aren't enough: nobody attaching to an existing proxy starts from a blank CR. They copy a managed one and flip the flag. Then sso, scim, jwtAuth, oauth2Auth, rbac, security, logging, adminUI, caching, routerSettings, callbacks, passThroughEndpoints, secretManager and license injection all silently stop applying, and nothing anywhere says so. The CR looks healthy and half of it is decorative.

What I had in mind is a condition that names only the sections the user actually set — not the whole catalogue, which would be noise for someone who did start clean. Something like:

Type:    WorkloadUnmanaged
Status:  True
Reason:  ConfigSettingsIgnored
Message: spec.sso, spec.caching and spec.routerSettings are ignored while
         workload.managed is false; the proxy owns its own configuration

Happy to be talked out of the exact shape, but I don't think "documented only" survives contact with a copied CR.

2. masterKey.autoGenerate needs enforcing, not just documenting

I traced what actually happens today, and it's worse than the 401 I predicted in the issue.

masterKeyRef() falls back to <name>-master-key, but that Secret is created by reconcileSecrets, which is only ever called from reconcileResources (litellminstance_controller.go:228) — precisely the path that's skipped when unmanaged. So the reference dangles: every entity CRD fails on a missing Secret, and nothing in the error mentions autoGenerate. Someone would burn a genuinely unpleasant afternoon on that.

A spec-level CEL rule closes it, in the same style as the endpoint one you already added:

// +kubebuilder:validation:XValidation:rule="!has(self.workload) || !has(self.workload.managed) || self.workload.managed || has(self.masterKey.secretRef)",message="masterKey.secretRef is required when workload.managed is false"

(untested, and the !has(self.workload.managed) guard is belt-and-braces given the true default — worth confirming in envtest, since that's the only place API-server validation actually runs)

---

One more, found while reviewing

LiteLLMGuardrail reports Ready=True against an unmanaged instance and does nothing. The guardrail controller has no awareness of workload.managed — it validates the CR (instance exists, Secret exists, key present), sets Ready=True, reason=Validated, and stops. But guardrails only ever materialise through the instance ConfigMap, which is never built when unmanaged. So you get a green CR doing nothing at all.

That's a bit worse than the inert config above, because it actively asserts success rather than staying quiet. And @adamdolman mentioned in #29 that guardrails are a likely future want, so it will get hit. Reporting Ready=False with something like InstanceUnmanaged would be honest.

Tiny one

docs/changelog.md is generated — the release workflow syncs it from the root CHANGELOG.md, so it shouldn't be hand-edited. It's also one of the two merge conflicts, so dropping that hunk fixes both at once. The root CHANGELOG.md entry is great as-is.

---

None of this is a rework — it's small, well-scoped stuff on top of a design I'm happy with. If you'd rather not do another round, I'm glad to push the condition and the CEL rule onto your branch myself; just say which you'd prefer.

[davidgibbons]

Thanks for the thorough review — pushed 266e28f addressing all three substantive points plus the changelog nit.

1. Config inertness → WorkloadUnmanaged condition. Added, matching the shape you sketched: it names only the spec sections the CR actually has set among sso, scim, jwtAuth, oauth2Auth, rbac, security, logging, adminUI, caching, routerSettings, callbacks, passThroughEndpoints, secretManager, and license injection — so a clean unmanaged CR gets no condition at all, and a copied one only names what's actually inert.

2. masterKey.autoGenerate dangling reference. Your trace was exactly right — added the CEL rule you proposed (masterKey.secretRef is required when workload.managed is false), verified in envtest for both the autoGenerate-only and neither-set cases, and confirmed it doesn't affect the managed path.

3. LiteLLMGuardrail against an unmanaged instance. Good catch — it now reports Ready=False/InstanceUnmanaged instead of Validated when instanceRef points at an unmanaged instance, since guardrail config only ever renders through the ConfigMap that instance never builds.

4. docs/changelog.md. Reverted to match main; the new entries live in the root CHANGELOG.md only.

Also updated README.md and both docs/reference/*.md reference docs to describe the new condition and reasons, and added unit/envtest coverage for all three behaviors. make test and make lint are clean.