Repository navigation
feat(claude-review): converge reviews in fewer rounds - #99
Merged
crowecawcaw merged 3 commits intoOct 6, 2026
Merged
Conversation
A month of review data (1,506 comments on 174 PRs) showed the reviewer never settles: every revision gets ~2.5 new comments regardless of round, ~40% of later-round comments hit code that existed at round one, and comments average ~1,600 chars. The prompt asked for one partial pass per run, and every push re-reviewed the whole diff. - The first review is a full, thorough pass. Later revisions are reviewed incrementally against the last fully reviewed commit, recorded in a sticky summary comment: open threads are re-checked and marked addressed when fixed (or when a reply's reason holds up), and only newly changed code is reviewed. - Findings carry a severity (blocking / should-fix / nit). Nits are posted only on the full review; incremental should-fix findings must land on changed lines. The prompt excludes pre-existing issues and speculative scenarios and caps comments at ~120 words. - A summary comment and a "Claude review" commit status report what is still open, so authors and their agents know when to stop. The status needs `statuses: write` from the caller; the job now inherits the caller's permissions so that grant is opt-in rather than a startup failure. - The agent no longer holds a GitHub token or any write tool. It returns findings as structured output (--json-schema); a deterministic script validates anchors, dedupes by fingerprint, and posts one batched review. - Debounce 30s and skip heads a newer push superseded; skip bot-triggered runs (Dependabot), which claude-code-action rejects anyway. - Default to Claude Opus 5.5, also for release notes. Opus 5.5 rejects a forced tool_choice, so release notes use auto and keep the existing tool-call check. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
crowecawcaw
force-pushed
the
claude-review-converge
branch
from
October 5, 2026 23:45
24e984c to
bc5d9ff
Compare
Resolving a review thread needs contents: write, which the job that runs the model must never hold. Split the workflow: the review job runs with a read-only token and hands its results over as an artifact; the post job inherits the caller's grants, posts, and resolves addressed threads when the caller grants contents: write (otherwise the marked reply alone closes the thread, as before). Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
… token could resolve them Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
crowecawcaw
marked this pull request as ready for review
October 6, 2026 16:25
andychoquette
approved these changes
Oct 6, 2026
crowecawcaw
added a commit
that referenced
this pull request
Oct 7, 2026
…us 5.5 (#100) * fix(claude-review): harden posting against forged markers, partial posts, and re-runs Follow-ups to #99 from review feedback: - Only trust a review thread's fp marker when the bot wrote its first comment. Anyone can open a thread, and the fp format is predictable, so a forged marker could suppress the real finding, skew the open counts, or get a human's thread replied to and resolved. - Neutralize HTML comments in model text before posting, so a marker the PR induces the model to write cannot outrank the one we append. - A partial post no longer counts as a finished review: the baseline stays put, the status is an error, and the summary lists the findings that did not land. - Findings can anchor to removed code (side LEFT), so a finding on a deleted file is no longer dropped for want of a new-side line. - Name the review artifact per run rather than per attempt, so re-running only a failed post job finds it, and re-read the PR's threads before posting so that re-run does not duplicate the comments that landed. - Release notes: auto tool choice can return plain text; retry up to three times with a reminder instead of failing on the first text answer. An empty entries list is a valid result. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * fix(claude-review): bump claude-code-action to v1.0.244 for Opus 5.5 v1.0.136 bundles Claude Code CLI 2.1.163 (claude-agent-sdk 0.3.163), which Bedrock rejects for us.anthropic.claude-opus-5-5 with a 400: version 2.1.280 or newer is required. v1.0.244 bundles CLI 2.1.292. The action's inputs are unchanged between the two versions. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * fix(claude-review): allow non-write workflow_run actors claude-code-action v1.0.244 checks write access for workflow_run events, for both github.actor and workflow_run.actor. On fork PRs that's the fork author, who never has write access, so every external review failed before Claude ran. The review job's token is read-only, Claude only gets read tools, and posting happens in the separate post job, so bypassing the check is safe here. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> --------- Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was the problem/requirement? (What/Why)
The Claude reviewer takes many rounds to settle. Last month: 1,506 bot comments on 174 PRs.
What was the solution? (How)
blocking/should-fix/nit. Nits only on the first review. Comments are capped at ~120 words.Claude reviewcommit status.reviewruns the model with a read-only token and returns structured output.postruns no model; it validates the findings and posts them.tool_choice: auto, since 5.5 rejects a forced tool choice).What is the impact of this change?
AWS_CLAUDE_PR_REVIEW_ROLErole must allowus.anthropic.claude-opus-5-5.post:statuses: writefor the commit status.contents: write(instead ofcontents: read) to resolve threads. Without it, threads get an "Addressed" reply and count as closed.How was this change tested?
zizmorandactionlintreport nothing new.Was this change documented?
Yes, in workflow comments and the script docstring.
Is this a breaking change?
No. Callers need no changes.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.