Skip to content

feat(deploy): staging environment with eval on every sync - #267

Merged
google-oss-prow[bot] merged 9 commits into
kubeflow:mainfrom
kmr-rohit:feat/staging-env
Oct 16, 2026
Merged

feat(deploy): staging environment with eval on every sync#267
google-oss-prow[bot] merged 9 commits into
kubeflow:mainfrom
kmr-rohit:feat/staging-env

Conversation

@kmr-rohit

@kmr-rohit kmr-rohit commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stacked on #266 (review that first; this PR's own changes are the last commits).

What

  • Staging (deploy/argocd/apps/docs-agent-staging.yaml, deploy/envs/staging/docs-agent.yaml): the docs-agent chart in namespace docs-agent-staging, synced automatically from main (self-heal, prune, namespace created by Argo). Shares the model, embeddings and Milvus with production; not public.
  • Eval on every sync (tests/eval/staging_eval.py, chart templates/eval-job.yaml): an Argo CD PostSync Job sends the 58 single-turn cases of flo_eval_dataset.json to the staging agents through the kagent A2A API, checks tool routing and query fidelity, and has the judge model (flo-judge, Qwen) rate each answer's groundedness in the tool results. Gates: routing >= 0.85, grounded >= 0.65, no errors. A failed gate fails the Job and marks the sync failed in Argo.
  • Results in Langfuse (evaluation project): a run of dataset flo-e2e named <sha> pr-<n> @<author> <time>, with routing, query_fidelity, grounded and answers scores on each trace. Contributors find their change by PR number or GitHub username and compare runs side by side.
  • OTel collector in Git (deploy/platform/otel-collector/, observe-only Application): the hand-applied collector, plus namespace-based routing. The k8sattributes processor (read-only RBAC) looks up the sending pod's namespace; spans from docs-agent-staging go to the evaluation project, everything else to the live project.
  • docs-agent chart 0.3.3: optional eval block (off by default).

Tested on the demo cluster (apps pointed at this branch)

  • Staging auto-synced; agents and MCP Ready; answers through the A2A API.
  • PostSync eval at 7944615: 58 cases, 0 errors, routing 1.00, query fidelity 1.00, grounded 0.73, answers 0.78, p50 2.6 s. All 58 traces linked; scores and dataset run written with no errors.
  • An earlier attempt tagged agents with OTEL_RESOURCE_ATTRIBUTES; kagent's runtime ignores it and staging spans reached the live project. Fixed by routing on namespace instead (verified: 58/58 traces in the evaluation project).
  • Collector config validated with otelcol-contrib validate (0.160.0).
  • Production render unchanged (eval.enabled defaults to false).

Before syncing staging (admin, once)

  1. Langfuse: an evaluation project with API keys.
  2. docs-agent-staging Secrets: mcp-server-secret (Milvus password) and mcp-langfuse-keys (evaluation keys).
  3. Authorization-eval (Basic auth for the evaluation project) in docs-agent/langfuse-otlp.

@kmr-rohit kmr-rohit changed the title feat(deploy): staging environment and OTel collector in Git feat(deploy): staging environment with eval on every sync Oct 13, 2026
@kmr-rohit kmr-rohit mentioned this pull request Oct 13, 2026
Merged
@kmr-rohit
kmr-rohit force-pushed the feat/staging-env branch 2 times, most recently from 43e3ffb to 1c4a5b2 Compare October 13, 2026 21:31
@kmr-rohit
kmr-rohit marked this pull request as ready for review October 14, 2026 16:43
@google-oss-prow
google-oss-prow Bot requested a review from chasecadet October 14, 2026 16:44
tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/ingress/langfuse-ingress.yaml
spec:
commonName: langfuse.kmrrohit.space
dnsNames:
- langfuse.kmrrohit.space

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check namespace, also in #266

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the demo cluster's hostnames. Kubeflow doesn't have a subdomain for this yet, so they live only in deploy/envs/prod/ (env config); the charts take the domain as a value. Once a project-owned domain exists, it's a change to deploy/envs/prod/ only. I'd rather keep the real hosts in Git so Argo CD reflects what's live, but happy to switch them to placeholders if you prefer.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/langfuse/values.yaml Outdated
- key: kubernetes.io/hostname
operator: NotIn
values:
- 10.0.10.102

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in #266

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #266 (6757c39); this branch will pick it up on rebase.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/langfuse/values.yaml Outdated
- key: kubernetes.io/hostname
operator: NotIn
values:
- 10.0.10.102

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #266 (6757c39); this branch will pick it up on rebase.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/langfuse/values.yaml Outdated
- key: kubernetes.io/hostname
operator: NotIn
values:
- 10.0.10.102

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #266 (6757c39); this branch will pick it up on rebase.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/langfuse/values.yaml Outdated
- key: kubernetes.io/hostname
operator: NotIn
values:
- 10.0.10.102

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#266

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #266 (6757c39); this branch will pick it up on rebase.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/langfuse/values.yaml Outdated
- key: kubernetes.io/hostname
operator: NotIn
values:
- 10.0.10.102

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#266

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #266 (6757c39); this branch will pick it up on rebase.

tarekabouzeid
tarekabouzeid reviewed Oct 15, 2026
View reviewed changes
Comment thread deploy/envs/prod/gateway-guardrails.yaml
sessionAuth:
enabled: true
enforce: true
image: ghcr.io/kmr-rohit/kubeflow-session-issuer:v0.1.0

@tarekabouzeid tarekabouzeid Oct 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please look into using Kubeflow tags

@kmr-rohit kmr-rohit Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5d9ee8d adds the session-issuer image to the main build, pushing ghcr.io/kubeflow/kubeflow-session-issuer:<sha> (the chart default already points there; nothing built it before). Prod moves to it in a one-line follow-up once the first image exists, and the pull secret goes away then. The package may need to be set public by an org admin, like mcp-kubeflow-docs.

@kmr-rohit
feat(deploy): staging environment and OTel collector in Git
4292635 
Add docs-agent-staging: the docs-agent chart in its own namespace,
synced automatically from main by Argo CD (self-heal, prune). It shares
the model, embeddings and Milvus with production and is not public; the
eval job reaches it through the kagent controller A2A API.

Move the hand-applied otel-collector (Deployment, Service, ConfigMap with
the noise filter) into deploy/platform/otel-collector with an
observe-only Application. A routing connector sends spans tagged
deployment.environment=staging to the Langfuse evaluation project and
everything else to the live project.

docs-agent chart 0.3.2: kagent.agentEnv adds env to every agent pod;
staging uses it to set OTEL_RESOURCE_ATTRIBUTES.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(eval): end-to-end eval of staging after every Argo CD sync
ec70f8a 
tests/eval/staging_eval.py sends the 58 single-turn cases of
flo_eval_dataset.json to the deployed agents through the kagent A2A API,
checks tool routing and query fidelity, and asks the judge model
(flo-judge, Qwen) whether each answer is grounded in the tool results.
Results are recorded in Langfuse as a run of the flo-e2e dataset, named
by commit, PR and author, with per-trace scores. Gates: routing >= 0.85,
grounded >= 0.65, no errors. Standard library only.

docs-agent chart 0.3.3: optional eval Job as an Argo CD PostSync hook
(off by default, on in staging). It fetches the script at the synced
revision; a failed gate fails the sync.

First run against staging: 58 cases, routing 1.00, query fidelity 1.00,
grounded 0.72, answers 0.85, p50 3.5 s, no errors.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
fix(deploy): route staging spans by namespace, not by agent tag
62d4895 
kagent's agent runtime ignores OTEL_RESOURCE_ATTRIBUTES, so staging
agent spans carried no environment tag and landed in the live Langfuse
project. The collector now looks up the sending pod's namespace with the
k8sattributes processor (read-only RBAC on pods, namespaces and
replicasets) and routes docs-agent-staging spans to the evaluation
project. Drop the kagent.agentEnv chart option, which had no effect.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(deploy): tag Langfuse environment production/staging on both tra…
c12e140 
…ce routes

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
fix(eval): record runs as Langfuse v4 Experiments via OTLP experiment…
44ad1e3 
… spans

Langfuse v4 (events_only) ignores dataset-run-items; Experiments are built
from spans carrying langfuse.experiment.* attributes. The eval now sends
one root span per case (input, agent answer, expected tools, link to the
agent trace) over OTLP/HTTP JSON and puts the scores on it, as well as
on the agent trace.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(chart): eval.limit for quick smoke runs of the staging eval
970a74c 
Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(deploy): staging runs the main-built MCP image (ghcr.io/kubeflow…
58d354a 
…/mcp-kubeflow-docs:0e49ebd)

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
style(eval): ruff format staging_eval.py
1bfaec8 
Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
fix(deploy): keep Argo CD from rotating Langfuse credentials on sync
279c73f 
The Langfuse chart reuses generated passwords through Helm lookup, which Argo CD
cannot run, so each render produces new ones. Ignore /data on those Secrets and
respect that on sync, so the live credentials are never overwritten.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@tarekabouzeid

tarekabouzeid commented Oct 16, 2026

Copy link
Copy Markdown
Member

Thanks @kmr-rohit
/lgtm
/approve

@google-oss-prow google-oss-prow Bot added the lgtm label Oct 16, 2026
@google-oss-prow

google-oss-prow Bot commented Oct 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarekabouzeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow
google-oss-prow Bot merged commit 9145c31 into kubeflow:main Oct 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo Awaiting requested review from franciscojavierarceo
@chasecadet chasecadet Awaiting requested review from chasecadet
@tarekabouzeid tarekabouzeid Awaiting requested review from tarekabouzeid

Assignees

@tarekabouzeid tarekabouzeid

Labels

approved lgtm size/XL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kmr-rohit @tarekabouzeid