feat(deploy): Argo CD observe-only setup and Qwen judge release by kmr-rohit · Pull Request #266 · kubeflow/docs-agent

kmr-rohit · 2026-10-13T18:50:21Z

What

deploy/argocd/: Argo CD install values (no Dex, no notifications, no ApplicationSet, anonymous and exec off) and Applications for docs-agent, gateway-guardrails, llm-runtime and llm-judge, tracking main.
deploy/envs/prod/: production values for the demo cluster (domain, CORS origins, image overrides) and the public HTTPS ingress for Langfuse and Argo CD. Argo reads them as a second source.
deploy/argocd/README.md: the Kubernetes Secrets the stack needs, by name only. No secret values in Git.
deploy/envs/prod/langfuse/values.yaml + deploy/argocd/apps/langfuse.yaml: Langfuse release values captured from the cluster, including settings that had been applied by hand (public NextAuth URL, sign-up disabled, pull policy). Generated secrets are pinned to the existing langfuse-app Secret, so an upgrade cannot rotate them. LLM connections may call llm-judge.
Argo CD viewer account bound to role:readonly; admin and viewer password hashes are passed at install time (README).
gateway-guardrails 0.2.1: new gateway.tls.acme.createIssuer (default true, no behaviour change). With false the Certificate uses an existing ClusterIssuer, so the ACME contact email stays in the cluster instead of in public values.
llm-judge: a second llm-runtime release with the existing qwen profile, served as flo-judge, on the second GPU node, with an AuthorizationPolicy declaring its intended callers (not enforced yet, see Known gaps).
MCP tracing upstreamed (docs-agent-mcp/mcp-server/flo_tracing.py, co-authored with @SanthoshToorpu): the live MCP server runs a fork image (santhoshtoorpu/...:otel-flo) whose Langfuse tracing was never on main. Ported onto current main (not copied: the image predates main's argument validation, release-date formatting and code-search repo filter). A traced() decorator adds TOOL / RETRIEVER / EMBEDDING observations to the three tools, both retrievers and embed_query; no-op without Langfuse keys. Also fixes top_scores, which was always empty in production. langfuse==4.15.2, as in the live image. Verified in-cluster with the real SDK.
CI no longer deploys (.github/workflows/oke-cicd.yaml): after fix(ci): make the PR checks pass again on main #269 made the tests pass, the deploy job ran again and helm upgraded the demo cluster's docs-agent release with chart defaults (only the MCP image changed; production stayed healthy). The workflow now tests every PR and, on main, only builds and pushes the MCP image; all kubectl/helm/OCI steps and cluster credentials are removed. Since a push runs the workflow file from the merged commit, merging this PR cannot trigger another cluster deploy. deploy/envs/prod pins ghcr.io/kubeflow/mcp-kubeflow-docs:0e49ebd, the image now running, so Git matches the cluster.

Why

Visibility first. No Application has a syncPolicy. Argo reports drift between main and the cluster; a maintainer clicks Sync to deploy. Hand edits now show as OutOfSync.
Config in Git, secrets in the cluster. Environment values are public and reviewed by PR; secrets are Kubernetes Secrets referenced by name.
Gemma answers, Qwen judges. Gemma 4 E4B stays the answer model. Evaluation and feedback triage use Qwen2.5-7B on its own GPU: a different model family, and judging never competes with user traffic.

Tested

Rendered each chart from this branch with deploy/envs/prod/ values and ran kubectl diff against the live cluster: llm-runtime and gateway-guardrails show no difference (the ClusterIssuer is no longer rendered; it must be annotated helm.sh/resource-policy=keep before any helm upgrade, see the chart README); docs-agent differs only in checksum/config.
Default chart values still render the ClusterIssuer (backward compatible).
kubectl apply --dry-run=server passes for all Applications and the rendered llm-judge resources.

Known gaps

In-cluster Istio AuthorizationPolicies in ml-infra (LLM, judge, Milvus, embeddings) are not enforced: docs-agent and ml-infra have no sidecar injection, so a pod in any namespace can reach these services (verified from default). The public gateway policies are enforced and unaffected. Fix planned as a follow-up: sidecar injection per namespace, ml-infra first. Recorded in deploy/argocd/README.md.

Not in this PR

Staging namespace, eval-on-sync hook, chat-widget feedback, and the CI change from helm upgrade to build-only. These follow as separate PRs.

Add Argo CD install values and Applications for docs-agent, gateway-guardrails and llm-runtime, tracking main with no syncPolicy so Argo reports drift without changing the cluster. Deployment-specific values (domain, ACME email, CORS origins, image overrides) come from a private config repo as a second source, so no personal or environment data lives in this repo. Secrets stay as Kubernetes Secrets referenced by name. Add llm-judge: a second llm-runtime release using the existing qwen profile on its own GPU, served as flo-judge, for LLM-as-a-judge evals. Gemma keeps answering; a different model family judges. An Istio AuthorizationPolicy limits callers to eval, triage and Langfuse. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

Per-environment values move to deploy/envs/prod/ in this repo and are read by Argo as a second source; no private config repo is needed. gateway-guardrails gains gateway.tls.acme.createIssuer (default true). With false, the Certificate uses an existing ClusterIssuer, so the ACME contact email lives only in the cluster and not in public values. deploy/secrets/ lists the Secrets the stack needs and adds a script that backs them up to a private local directory outside the repo. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

…ecrets folder Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

Capture the live Langfuse release values in deploy/envs/prod/langfuse, including the settings that were applied by hand (public NextAuth URL, sign-up disabled, image pull policy). Pin salt, encryption key and NextAuth secret to the existing langfuse-app Secret so an upgrade cannot rotate them. Allow LLM connections to llm-judge and llm-stable. Add an observe-only Argo Application for Langfuse, and a viewer account bound to role:readonly. Password hashes are passed at install time. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

…without sidecars Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

The live MCP deployment carried LANGFUSE_PUBLIC_KEY/SECRET_KEY added with kubectl set env. Neither kubectl diff nor Argo flags fields added outside Git, so a clean redeploy would have silently dropped tracing. Add a generic mcp.extraEnv list (default empty) and set the two Secret references in deploy/envs/prod/docs-agent.yaml. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

The live MCP server runs a fork image (santhoshtoorpu/...:otel-flo) whose tracing code was never upstreamed, so any image built from main loses the TOOL / RETRIEVER / EMBEDDING spans. Port it onto current main: - flo_tracing.py (from the live image): no-op unless LANGFUSE_PUBLIC_KEY is set; tracing errors never break a search. Adds a traced() decorator so each function gets one line instead of a re-indented body; functools.wraps keeps the signatures FastMCP builds tool schemas from. - Traced: the three MCP tools, search_docs_auto, search_collection, embed_query. Outputs record hit count, retrieval mode, intent and top scores. - Fix: top_scores read the score from entity, but Milvus puts distance on the hit, so it was always empty in production. - langfuse==4.15.2 (the version in the live image). Only the tracing is taken from the image; it was built from an older snapshot, and main's later fixes (argument validation, release dates, repo filter on code search) are kept. Verified in-cluster with the real SDK and the evaluation project: one search_kubeflow_docs call produced TOOL, RETRIEVER and EMBEDDING observations with environment=staging. tests/test_flo_tracing.py covers the decorator, the output summary and the tool schemas. Co-authored-by: SanthoshToorpu <90833739+SanthoshToorpu@users.noreply.github.com> Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

Merging kubeflow#269 made the tests pass again, and the oke-cicd deploy job ran for the first time since September: it helm-upgraded the docs-agent release on the demo cluster with chart defaults (only the MCP image changed; production stayed healthy). With Argo CD managing the cluster, two writers would keep overwriting each other. The workflow now tests every PR and, on main, only builds and pushes the MCP image. The kubectl/helm/OCI steps and the cluster credentials are gone; the job is renamed "Build and push MCP image" and still gated on ENABLE_OKE_DEPLOY. Rolling out an image is a PR to deploy/envs/<env>/docs-agent.yaml plus a sync in Argo CD. deploy/envs/prod pins ghcr.io/kubeflow/mcp-kubeflow-docs:0e49ebd, the image that CI deployed, so Git matches the cluster instead of pointing back at the old fork image. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

tarekabouzeid · 2026-10-14T21:52:23Z

+        - key: kubernetes.io/hostname
+          operator: NotIn
+          values:
+          - 10.0.10.102


Can you please look into not using IPs?

Removed in 6757c39. The GPU nodes are tainted nvidia.com/gpu:NoSchedule, so Langfuse pods never schedule there; the hostname anti-affinity was redundant.

tarekabouzeid · 2026-10-14T21:52:52Z

+            - key: kubernetes.io/hostname
+              operator: NotIn
+              values:
+              - 10.0.10.102


Removed in 6757c39. The GPU nodes are tainted nvidia.com/gpu:NoSchedule, so Langfuse pods never schedule there; the hostname anti-affinity was redundant.

tarekabouzeid · 2026-10-14T21:53:21Z

+            - key: kubernetes.io/hostname
+              operator: NotIn
+              values:
+              - 10.0.10.102


Removed in 6757c39. The GPU nodes are tainted nvidia.com/gpu:NoSchedule, so Langfuse pods never schedule there; the hostname anti-affinity was redundant.

tarekabouzeid · 2026-10-14T21:53:41Z

+          - key: kubernetes.io/hostname
+            operator: NotIn
+            values:
+            - 10.0.10.102


Here as well

Removed in 6757c39. The GPU nodes are tainted nvidia.com/gpu:NoSchedule, so Langfuse pods never schedule there; the hostname anti-affinity was redundant.

tarekabouzeid · 2026-10-14T21:54:15Z

+      name: langfuse-app
+      key: encryption-key
+  nextauth:
+    url: https://langfuse.kmrrohit.space


Can you look into the url?

These are the demo cluster's hostnames. Kubeflow doesn't have a subdomain for this yet, so they live only in deploy/envs/prod/ (env config); the charts take the domain as a value. Once a project-owned domain exists, it's a change to deploy/envs/prod/ only. I'd rather keep the real hosts in Git so Argo CD reflects what's live, but happy to switch them to placeholders if you prefer.

tarekabouzeid · 2026-10-14T21:54:56Z

+          - key: kubernetes.io/hostname
+            operator: NotIn
+            values:
+            - 10.0.10.102


Check this one rol

Removed in 6757c39. The GPU nodes are tainted nvidia.com/gpu:NoSchedule, so Langfuse pods never schedule there; the hostname anti-affinity was redundant.

GPU nodes carry the nvidia.com/gpu NoSchedule taint, so Langfuse pods never schedule there; the per-hostname rule was redundant. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

The gateway-guardrails chart defaults to ghcr.io/kubeflow/kubeflow-session-issuer, but nothing built it. Push it on main tagged by short SHA; prod moves to it in a follow-up once the first image exists. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

tarekabouzeid · 2026-10-15T18:29:26Z

Thanks @kmr-rohit
/lgtm
/approve

google-oss-prow · 2026-10-15T18:29:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarekabouzeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tarekabouzeid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

server.py imports flo_tracing since kubeflow#266, but the Dockerfile copies an explicit file list that was not updated, so every image built from main since then exits on start with ModuleNotFoundError. Production and staging still run 0e49ebd, built before kubeflow#266, so nothing live broke; promoting any newer commit would have. Found by the first /eval run, whose preview used such an image. A test now checks that every local module the copied files import is copied too. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

… and image/credential fixes (#274) * feat(deploy): staging deploys and evaluates every commit on main The staging image tag is the synced commit's short SHA (the tag CI pushes), so each merge changes staging, Argo CD syncs it and the eval runs. Sync retries cover the minutes before CI has pushed the image. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * feat(deploy): promote production by commit, synced automatically Production Applications pin a commit (# prod-revision) and take both images from it, so the session issuer now comes from ghcr.io/kubeflow and needs no pull secret. deploy/promote.sh opens the promotion PR with the changelog and an eval table; merging it deploys, reverting it rolls back. The root Application flo-apps applies deploy/argocd/apps/ from main; otel-collector also syncs automatically. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * feat(deploy): weekly Langfuse data retention Self-hosted Langfuse retention is an Enterprise feature, so a CronJob deletes traces, observations, scores and raw S3 events older than 4 weeks for Flo-live and 2 weeks for flo-eval. Datasets are kept. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * docs(deploy): document sync policy, promotion and retention Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * fix(deploy): keep Langfuse app credentials out of the chart's own Secret The chart renders a Secret named langfuse-app and fills it only with values it finds through Helm lookup or generates. Our values referenced the salt, encryption key and NextAuth secret from that same Secret, so under Argo CD, where lookup returns nothing, the chart treated them as provided elsewhere and rendered langfuse-app empty. The sync on 2026-10-16 applied it and wiped the values; the running pods kept them in memory, and new pods could not start. Read the credentials from langfuse-app-credentials, which an admin creates and Argo CD does not manage, and ignore /data on langfuse-app so a sync cannot blank it again. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * feat(eval): /eval runs the end-to-end eval on a pull request before merge The staging eval only runs after a merge, so a change that hurts answers is found on main. A maintainer can now comment /eval on a PR: - .github/workflows/pr-eval.yaml checks the commenter is an OWNERS approver, builds the PR's MCP image (pr-<n>-<sha8>) and labels the PR eval and eval-sha:<sha>, which pins the reviewed commit. One PR at a time; an eval that holds the preview for 3 hours is cleared. - The docs-agent-preview ApplicationSet deploys that commit into docs-agent-preview under a restricted AppProject (that namespace, the chart's kinds, nothing cluster-scoped); the namespace enforces Pod Security baseline. - The docs-agent-pr-eval ApplicationSet runs the pr-eval chart from main. Its Job (tests/eval/pr_eval.py) waits for the rollout, evaluates main (staging) and then the PR with the same model, index and judge, comments the comparison with public Langfuse traces per case, and removes the labels, which tears the preview down. Grading always comes from main. staging_eval.py gains evaluate(), a run-name prefix, an environment setting and public traces; the staging PostSync run is unchanged. Preview agent traces go to the eval project as environment preview. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> * fix(mcp): copy flo_tracing.py into the MCP image server.py imports flo_tracing since #266, but the Dockerfile copies an explicit file list that was not updated, so every image built from main since then exits on start with ModuleNotFoundError. Production and staging still run 0e49ebd, built before #266, so nothing live broke; promoting any newer commit would have. Found by the first /eval run, whose preview used such an image. A test now checks that every local module the copied files import is copied too. Signed-off-by: Rohit Kumar <rr7433446@gmail.com> --------- Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

google-oss-prow Bot added the do-not-merge/work-in-progress label Oct 13, 2026

google-oss-prow Bot requested review from franciscojavierarceo and tarekabouzeid October 13, 2026 18:50

google-oss-prow Bot added size/L size/XL and removed size/L labels Oct 13, 2026

This was referenced Oct 13, 2026

feat(deploy): staging environment with eval on every sync #267

Merged

feat(feedback): thumbs up/down in the chat widget and nightly triage #268

Merged

kmr-rohit and others added 8 commits October 14, 2026 03:00

docs(deploy): list required Secrets in the Argo README instead of a s…

b384b59

…ecrets folder Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

docs(deploy): record that in-cluster Istio policies are not enforced …

31ac217

…without sidecars Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

kmr-rohit force-pushed the chore/argocd-observability-docs branch from dc4ec70 to e361a7a Compare October 13, 2026 21:31

google-oss-prow Bot added size/XXL and removed size/XL labels Oct 13, 2026

kmr-rohit marked this pull request as ready for review October 13, 2026 21:33

google-oss-prow Bot removed the do-not-merge/work-in-progress label Oct 13, 2026

google-oss-prow Bot requested a review from chasecadet October 13, 2026 21:33

tarekabouzeid reviewed Oct 14, 2026

View reviewed changes

kmr-rohit added 2 commits October 15, 2026 12:19

fix(deploy): drop node-IP anti-affinity from Langfuse values

6757c39

GPU nodes carry the nvidia.com/gpu NoSchedule taint, so Langfuse pods never schedule there; the per-hostname rule was redundant. Signed-off-by: Rohit Kumar <rr7433446@gmail.com>

google-oss-prow Bot assigned tarekabouzeid Oct 15, 2026

google-oss-prow Bot added the lgtm label Oct 15, 2026

google-oss-prow Bot added the approved label Oct 15, 2026

google-oss-prow Bot merged commit eb53144 into kubeflow:main Oct 15, 2026
8 checks passed

kmr-rohit mentioned this pull request Oct 15, 2026

chore(deploy): promote main eb53144 images to prod #273

Closed

kmr-rohit mentioned this pull request Oct 17, 2026

feat(deploy): promote by commit, pre-merge /eval, Langfuse retention, and image/credential fixes #274

Merged

Conversation

kmr-rohit commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Tested

Known gaps

Not in this PR

Uh oh!

tarekabouzeid Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarekabouzeid commented Oct 15, 2026

Uh oh!

google-oss-prow Bot commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kmr-rohit commented Oct 13, 2026 •

edited

Loading

tarekabouzeid Oct 14, 2026 •

edited

Loading