Skip to content

feat(feedback): thumbs up/down in the chat widget and nightly triage - #268

Merged
google-oss-prow[bot] merged 5 commits into
kubeflow:mainfrom
kmr-rohit:feat/chat-feedback
Oct 16, 2026
Merged

feat(feedback): thumbs up/down in the chat widget and nightly triage#268
google-oss-prow[bot] merged 5 commits into
kubeflow:mainfrom
kmr-rohit:feat/chat-feedback

Conversation

@kmr-rohit

@kmr-rohit kmr-rohit commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Stacked on #267 (and #266). This PR's own changes are the last two commits.

What

  • Widget (frontend/docs_scripts/chatbot.js, chatbot.css): a "Helpful?" bar with thumbs up/down under every finished answer, also on answers restored from history. Thumbs-down offers an optional comment. The vote is stored with the chat; the bar removes itself when the gateway has no feedback route. Light and dark theme.
  • Feedback service (gateway-guardrails 0.3.0, feedback.enabled): POST /api/feedback with the chat's contextId (the Langfuse session id), the 0-based turn and up/down. It finds that turn's trace and writes a BOOLEAN score user_feedback with a deterministic id (a changed vote overwrites the first). If the trace is not ingested yet, the vote goes on the session. The route requires a session JWT and shares the chat rate limits. Standard library only, run from a ConfigMap on a stock python image: no image to build.
  • Nightly triage (docs-agent chart 0.4.0, triage.enabled, 03:00): tests/feedback/triage.py feedback reads the last day's thumbs-down, re-searches the index over MCP, and has the judge (flo-judge, Qwen) classify the failing layer: generation, retrieval, tool_routing, agent_skipped_retrieval (owner: docs-agent), or content_gap_handled / content_gap_hallucination (owner: the component's working group). Results are stored as triage_layer / triage_component scores; the Job log has a Markdown summary grouped by owner. Nothing is filed automatically.

Both are enabled in deploy/envs/prod/.

Tested on the demo cluster

  • Feedback service deployed on its own (no gateway change): invalid contextId → 400; a thumbs-down on a fresh production answer linked to its trace (linked: true) and was read back from /v3/scores.
  • Triage in-cluster, read-only: found that vote, probed MCP, and classified "What is the latest Kubeflow release?" as content_gap_hallucination → kubeflow/manifests, with the user's comment.
  • Unit tests: tests/test_feedback.py (validation, turn → trace, score ids), tests/test_triage.py (component guess, owner routing, report). 22 passed.
  • helm lint and server-side dry runs for the new templates.

Not done yet

  • The gateway route (/api/feedback) is not live until gateway-guardrails is synced from main.
  • The judge's agreement with human labels has not been measured; triage layers are suggestions until it is.

@kmr-rohit
kmr-rohit force-pushed the feat/chat-feedback branch 5 times, most recently from dbcafbf to f662014 Compare October 15, 2026 18:45
@kmr-rohit
feat(feedback): thumbs up/down in the chat widget, recorded in Langfuse
180a544 
Widget: a "Helpful?" bar under each finished answer (and answers restored
from history). A thumbs-down offers an optional comment. The vote is
saved with the chat so it survives reloads; the bar removes itself if the
gateway has no feedback route.

gateway-guardrails 0.3.0: optional feedback service (feedback.enabled).
POST /api/feedback takes the chat's contextId (the Langfuse session id),
the 0-based turn and up/down. It finds that turn's trace and writes a
BOOLEAN score user_feedback with a deterministic id, so a changed vote
overwrites the first. If the trace is not ingested yet the vote goes on
the session. The path requires a session JWT and shares the chat rate
limits. Standard library only, run from a ConfigMap on a stock python
image (no image to build). Enabled in deploy/envs/prod.

Tested on the demo cluster: invalid contextId -> 400; a vote on a fresh
production answer linked to its trace and was read back via /v3/scores.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(feedback): nightly triage of thumbs-down answers
057b53e 
tests/feedback/triage.py, standard library only:
- new "feedback" mode reads user_feedback=false scores (/v3/scores) from
  the last N hours, including votes stored on a session before its trace
  arrived;
- the MCP probe talks streamable HTTP directly (no fastmcp);
- --report prints a Markdown summary grouped by owner: pipeline layers
  to kubeflow/docs-agent, content gaps to the component's repo, with the
  user's comment. Nothing is filed automatically.

docs-agent chart 0.4.0: optional CronJob (triage.enabled, 03:00 daily)
that fetches the script at triage.ref and runs it with --write, judged
by flo-judge. Enabled in deploy/envs/prod.

Tested in-cluster: the smoke-test thumbs-down was found, probed and
classified content_gap_hallucination -> kubeflow/manifests.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
style(feedback): ruff format triage and feedback tests
23c88f3 
Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(frontend): tell users that conversations and feedback are recorded
a306424 
One sentence in the existing disclaimer under the input box, asking users not to
share personal information.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
feat(frontend): Material 3 redesign of the chat widget
93bec3c 
Token-based stylesheet (light and dark differ only in colour roles), a branded header, answers as open text beside the avatar with tonal user bubbles, starter questions under the welcome message, a pill composer, restyled sources and feedback, and a full-screen sheet on phones. Fixes answers overflowing the right edge. The disclaimer is shorter and keeps the data-collection notice.

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
@kmr-rohit
kmr-rohit marked this pull request as ready for review October 16, 2026 19:33
@tarekabouzeid

tarekabouzeid commented Oct 16, 2026

Copy link
Copy Markdown
Member

/lgtm
/approve

@google-oss-prow google-oss-prow Bot added the lgtm label Oct 16, 2026
@google-oss-prow

google-oss-prow Bot commented Oct 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tarekabouzeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow
google-oss-prow Bot merged commit 602f182 into kubeflow:main Oct 16, 2026
8 checks passed
kmr-rohit added a commit to kmr-rohit/docs-agent that referenced this pull request Oct 17, 2026
@kmr-rohit
chore(deploy): pin production to 602f182
df054d7 
Promote the commit that added chat feedback and nightly triage (kubeflow#268). CI on
main built both images for it (mcp-kubeflow-docs and kubeflow-session-issuer).

Signed-off-by: Rohit Kumar <rr7433446@gmail.com>
This was referenced Oct 17, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@franciscojavierarceo franciscojavierarceo Awaiting requested review from franciscojavierarceo
@tarekabouzeid tarekabouzeid Awaiting requested review from tarekabouzeid

Assignees

@tarekabouzeid tarekabouzeid

Labels

approved lgtm size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kmr-rohit @tarekabouzeid