Skip to content

feat(ci): add automated llm for docs-agent issue triaging - #263

Open
modichika wants to merge 3 commits into
kubeflow:mainfrom
modichika:feat/AI-issue-triage-agent
Open

modichika wants to merge 3 commits into
kubeflow:mainfrom
modichika:feat/AI-issue-triage-agent

Conversation

@modichika

Copy link
Copy Markdown
Member

This PR adds a automated CI workflow for triaging issues through GitHub's Agentic Workflows (gh-aw) for kubeflow/docs-agent repository . It reads the issue title and body and analyses the issue based on:

  • 📂 Source context
  • 📊 Scope
  • 📝 Context
  • 🎯 Verdict

The pain-points it solves:

  • Reduces maintainer fatigue to analyse every issue as soon as it gets opened.
  • Decreases the number of issues from new contributors and offloads the project maintainer's burden to analyse each and every one of them.
  • Helps determine the meaningful contributors who truly wants to contribute in the repository.
  • Maintains the repositories health throughout time.
  • Helps triage issues that match the repository needs in the community call.

What this PR does not address and will be addressed in the upcoming PRs?
It does not include PR-triaging and will be included through gh-aw in future.

This PR resolves issue Implement CI Gatekeeper Workflows for Triaged Issues

And it's also part of this KEP-13293 from kubeflow/pipelines repo.
Co-authors: @alyssacgoins @zazulam

Signed-off-by: modichika <shristimodi95@gmail.com>
Signed-off-by: modichika <shristimodi95@gmail.com>
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign franciscojavierarceo for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@modichika

modichika commented Oct 6, 2026 •

Copy link
Copy Markdown
Member Author

The Build, Test, and Deploy to OKE/Compile and Test (pull_request) is failing because:

image

I think we need to add the missing dependency frontmatter.
cc: @kmr-rohit @SanthoshToorpu @chasecadet

@kmr-rohit

kmr-rohit commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Hi @modichika , Fixed test failures , can you do a dummy commit ?. Lets review changes in saturday's biweekly meetup.

@abhishekKokadwar abhishekKokadwar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. The title validation is carefully done: the title reaches bash through env rather than ${{ }} interpolation, the agent only sees type and area parsed from the title, the action is pinned by SHA, and the re-triage check only counts bot or member comments, so an outside commenter can't suppress a retry. A few things I found while comparing it with docs/agents/issue-triage.md from #257:

1. The comment template's placeholders are scrambled. In issue_triage.md the bullets read <Files confirm contradict or report symbols that the>, <Clear ambiguous or>, <Repro / actual expected> and <Ready for maintainer-only needs-info, or pickup,>. The version on main has <Files or symbols that confirm or contradict the report>, <Clear or ambiguous>, <Repro / expected / actual> and <Ready for pickup, needs-info, or maintainer-only>. The ```markdown fence is also never closed (the file ends at - <Single next step>), and the line "Each section is two or three short bullets. No time estimates." was dropped. The agent copies this template, so the wording matters. The lock file will need a gh aw compile after the fix, since it records a body_hash.

2. Most allowlisted labels don't exist on the repo yet. Of the 26, only duplicate, good first issue and help wanted exist today; kind/*, area/*, priority/*, needs-info, needs-triage, maintainer-only and gsoc-2026 don't. It probably makes sense to run scripts/sync-github-labels.sh from #257 before this merges. Otherwise the first runs will either fail to add labels or create them without colours or descriptions.

3. test(...) and ci(...) titles get rejected. .github/ISSUE_TEMPLATE/chore.yml tells reporters to use chore(<area>):, docs(<area>):, test(<area>): or ci(<area>):, and the prompt maps chore/test/ci → kind/chore. But ISSUE_TITLE_PATTERN only allows bug|chore|docs|feat|security, so a test(tests): … issue gets the "Validation Failed" comment. Adding test|ci to the pattern would line the three up.

4. (Question) Should the agent apply the maintainer-judgement labels? The allowlist includes priority/p0, good first issue, gsoc-2026, help wanted and maintainer-only. The prompt treats the issue body as untrusted, but an issue written to steer the model could still self-apply one of those. Restricting automatic labels to kind/*, area/*, needs-info and needs-triage, and leaving the rest to maintainers, would make that a non-issue. Happy to defer if this was already discussed.

The red checks look like the pre-#269 main failures, so a rebase onto current main should clear them.

Signed-off-by: modichika <shristimodi95@gmail.com>
@modichika

Copy link
Copy Markdown
Member Author

Hey @kmr-rohit done the dummy commit. PTAL!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants