fix(triage): verify a comment or body marker landed before calling a run successful - #56
Merged
Merged
Conversation
…run successful Claude Code exiting cleanly only proves the process ran; nothing in this composite action ever checked whether triage's own promised output -- a new issue comment, or a body edit carrying the claude-triage:start marker -- actually landed on GitHub. Both are ordinary tool calls the model makes from its own prompt, so a run that legitimately completes without calling either still reported steps.claude.outcome == 'success', which made the progress-comment step delete the one visible trace of the run and made the composite action's own overall conclusion green. Confirmed happening for real on three separate triage runs (novus-power/hive#1490, #1657, #1661): each reported success and left nothing behind. Adds a step that re-fetches the issue's own comments and body after a triage run reports success, and folds a genuine mismatch into the run's own outcome the same way turn-limit wrap-up already does, rather than trusting the model's self-report.
Mearman
marked this pull request as ready for review
September 10, 2026 20:05
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…own comment --paginate -q filters the jq expression per page rather than across the combined result, so an issue with more prior comments than fit on one page left the comment count as a multi-line value. Separately, "Post Headroom savings comment" posts its own comment unconditionally whenever Headroom proxied anything, under the shipped default configuration, and that comment's id is always greater than the progress comment's -- counting it made the check report success even when triage itself left nothing. Slurps every page into one array before filtering, and excludes any comment carrying the Headroom marker. Both gh api calls in this step are now retried three times like every other one in this file that isn't already guarded by an outer continue-on-error.
claude
Bot
dismissed
their stale review
September 10, 2026 20:21
Superseded by re-review on the latest commit.
…omment's id The verification step's if: required steps.progress-comment.outputs.comment_id, which is empty whenever post_progress_comment is false -- a supported, documented input combination this repeats through a different door. In that configuration verification was skipped entirely and steps.claude.outcome flowed straight through as this composite action's own overall conclusion, unverified. Adds a "Record run start time" step, unconditional and first in the job, and switches the comment check to created_at >= that timestamp instead of id > the progress comment's id. The progress comment (when it exists) and Headroom's own savings comment are still excluded explicitly, by id and by marker respectively, so neither is miscounted as real triage output.
claude
Bot
dismissed
their stale review
September 10, 2026 20:33
Superseded by re-review on the latest commit.
Any comment landing in the run's time window -- a human reply, a different bot, another automation entirely -- satisfied the created_at/id/marker filters and counted as real triage output, even when Claude's own comment never landed. Filters by the identity GH_TOKEN actually authored as: read straight from the progress comment's own API response when one exists (post_progress_comment enabled, the default), falling back to the documented default GITHUB_TOKEN identity when it doesn't and no custom github_token is set. The one combination neither can resolve -- no progress comment and a custom github_token -- omits the author filter rather than guessing or failing a valid input combination outright. "Post progress comment" now captures the real poster login from its own API response alongside the comment id, rather than discarding it.
claude
Bot
dismissed
their stale review
September 10, 2026 20:44
Superseded by re-review on the latest commit.
…en is empty "Run Claude Code" passes inputs.github_token straight through to upstream with no fallback, unlike GH_TOKEN elsewhere in this file (github_token || github.token). With the documented default -- github_token left empty, as every claude-triage.yml example in this repository does -- that asymmetry means Claude's own comment authenticates via an upstream-minted GitHub App token (claude[bot]), while the progress comment authenticates as the ambient github-actions[bot]: two different identities on the same run. Confirmed directly against real production comments on novus-power/hive, which uses exactly this configuration: the progress comment is authored by github-actions[bot], genuine triage comments by claude[bot]. The previous revision would have made every real triage run there report a false failure, having filtered out Claude's own comment by author. Checks for an empty github_token first and uses claude[bot] in that case, before falling back to the progress comment's own observed identity (valid only when both paths share the same real token) or no author filter at all.
claude
Bot
dismissed
their stale review
September 10, 2026 20:54
Superseded by re-review on the latest commit.
examples/direct/claude-triage.yml left github_token unset (the App-identity path) but never declared id-token: write, so the OIDC exchange that mints the Claude Code App token had nothing to authenticate with -- every triage run using this exact shipped example, unmodified, would fail outright at setupGitHubToken() with "Could not fetch an OIDC token", matching examples/direct/claude-review.yml's own already-correct configuration and the same permission novus-power/hive's real workflow had already added independently. Corrected the "Verify triage output landed" comment that cited this example as already granting id-token: write, which overstated what was actually shipped before this fix.
claude
Bot
dismissed
their stale review
September 10, 2026 21:03
Superseded by re-review on the latest commit.
There was a problem hiding this comment.
Re-reviewed at 4ef350b. This commit ("grant id-token: write in the direct-form triage example") directly and correctly addresses the Should-fix from my last round: examples/direct/claude-triage.yml now declares id-token: write, and the code comment in action.yml is updated to accurately reflect that the claude[bot] branch is only reachable on a genuine, successful OIDC-based App-token mint — matching what I verified independently against upstream's setupGitHubToken() source. Thread resolved.
Nothing else changed in this revision. All findings from prior rounds are now fixed and none remain open. Approving.
|
🗜️ Headroom context compression
|
|
🎉 This PR is included in version 1.10.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude Code exiting cleanly only proves the process ran; nothing in this composite action ever checked whether triage's own promised output -- a new issue comment, or a body edit carrying the
claude-triage:startmarker -- actually landed on GitHub. Both are ordinary tool calls the model makes from its own prompt, so a run that legitimately completes without calling either still reportedsteps.claude.outcome == 'success', which made the progress-comment step delete the one visible trace of the run and made the composite action's own overall conclusion green.Confirmed happening for real on three separate triage runs in novus-power/hive: novus-power/hive#1490, novus-power/hive#1657, novus-power/hive#1661 -- each reported success and left nothing behind, no comment, no label, no body edit.
Adds a "Verify triage output landed" step that re-fetches the issue's own comments and body after a triage run reports success (comparing against the progress comment's own id as the "before" marker), and folds a genuine mismatch into the run's own outcome the same way "Wrap up on turn limit" already does, rather than trusting the model's self-report. Scoped to triage mode only, since that's where this was actually observed -- review mode already has an equivalent real check ("Structured review summary" re-fetches the submitted review).