AI Code Review Bot

A GitHub App that reviews every pull request with the Claude API. It fetches the PR diff, analyses each changed file for bugs and vulnerabilities, posts inline comments with one-click suggested fixes, and approves or requests changes based on a configurable severity threshold.

The problem

Most LLM review bots fail the same way: the model reports a real issue but attaches it to a line that isn't in the diff, and GitHub rejects the comment — or worse, it lands on unrelated code. This one treats the model's output as untrusted. Claude is called with forced tool-use so findings arrive as structured data, then every finding is validated against the parsed diff before anything is posted: findings in disabled categories are dropped, lines that are off by up to 3 are snapped to the nearest commentable line, and anything outside the diff is discarded rather than posted blindly.

Quick start

1. Create the GitHub App

Settings → Developer settings → GitHub Apps → New GitHub App.

• Webhook URL: https://<your-host>/webhooks/github
• Webhook secret: a long random string
• Permissions: Pull requests Read & write, Contents Read-only, Checks Read & write (for merge gating), Metadata Read-only
• Subscribe to: Pull request, Push, Installation
• Generate and download a private key (.pem), note the App ID, then install the app on your repos

2. Configure

cp .env.example .env
# set GITHUB_APP_ID, GITHUB_WEBHOOK_SECRET, ANTHROPIC_API_KEY
# point GITHUB_APP_PRIVATE_KEY_PATH at your .pem

3. Run

# Docker — Postgres + Redis + API + Celery worker
# place your key at ./github-app.private-key.pem first
docker compose up --build     # API on http://localhost:8000, Swagger at /docs

Or locally:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

uvicorn backend.main:app --reload                              # terminal 1
celery -A backend.tasks.celery_app worker --loglevel=info      # terminal 2 (needs Redis)
cd dashboard && npm install && npm run dev                     # terminal 3 → :5173

For local webhook delivery, tunnel port 8000 (ngrok http 8000) and point the App's webhook URL at the tunnel. Open a PR and the bot posts its review.

How a review works

                 ┌──────────────┐   webhook (HMAC-SHA256 verified)
   GitHub  ─────▶│  FastAPI     │──────────────┐
   PR event      │  /webhooks   │              │
                 └──────┬───────┘              ▼
                        │  enqueue      ┌─────────────────┐
                        ├──────────────▶│  Celery worker  │
                        │   (Redis)     └────────┬────────┘
                 ┌──────▼───────┐                │ 1. fetch diff (GitHub API)
                 │  PostgreSQL  │◀───────────────┤ 2. parse unified diff
                 │  repos       │    persist     │ 3. Claude review (tool-use)
                 │  reviews     │                │ 4. validate findings vs diff
                 │  comments    │                │ 5. post review + check run
                 │  configs     │                ▼
                 │  webhook_logs│         ┌──────────────┐
                 └──────┬───────┘         │  GitHub PR   │
                        │  REST /api/*    │  review API  │
                 ┌──────▼───────┐         └──────────────┘
                 │ React        │
                 │ dashboard    │
                 └──────────────┘

1. Webhook arrives, HMAC-SHA256 signature is verified against GITHUB_WEBHOOK_SECRET, delivery is logged.
2. pull_request.opened / synchronize upserts the repo, creates a Review row, queues a Celery task. Commit SHAs are deduped so the same commit is never reviewed twice.
3. The worker authenticates as the App (JWT → installation access token) and fetches the diff.
4. A hand-written unified-diff parser maps every line to its new-file line number and diff position, tracking which lines are actually commentable.
5. Claude is called with a forced tool schema and language-specific hints, returning structured findings plus a summary.
6. Findings are validated against the diff — category gating, line snapping, out-of-diff removal.
7. One atomic review is posted: inline comments with GitHub suggestion blocks, plus a summary. The event is REQUEST_CHANGES / COMMENT / APPROVE depending on whether any finding meets the repo's block threshold.
8. A check run is posted for optional merge gating, and everything is persisted for the dashboard.

Rule categories (backend/ai/rules.py): bug, null_safety, security, secrets, performance, error_handling, concurrency, maintainability, and style — the last off by default as too noisy. Each is toggleable per repo. Language detection covers 17 extensions, with sharpened prompt guidance for Python, JavaScript/TypeScript, Go, and Java. Severity is critical / warning / suggestion.

Dashboard (React + Vite + TypeScript): installed repos, review history with issue counts and durations, per-review findings detail, a per-repo config page for rule categories and block threshold, and webhook delivery logs.

Layout

backend/
  config.py         pydantic-settings config (env / .env)
  database.py       SQLAlchemy engine + session
  models.py         repositories, repo_configs, reviews, review_comments, webhook_logs
  schemas.py        Pydantic I/O + the AI review contract
  github/           App auth, diff/PR fetching, review + check-run posting, diff parser
  ai/               Claude engine, finding validation, prompts, rule catalogue
  webhooks/         signature verification, event dispatch, handlers
  tasks/            Celery app + end-to-end review pipeline
  api/              repos · reviews · analytics · webhook_logs
  tests/            pytest suite
dashboard/          React + Vite + TypeScript

Tech stack

FastAPI, Pydantic v2 + pydantic-settings, SQLAlchemy 2, Alembic, PostgreSQL (SQLite by default for local runs), Celery + Redis, httpx, anthropic, PyJWT with cryptography for App auth. Dashboard: React, Vite, TypeScript. Tooling: pytest (+ pytest-cov, respx), ruff, mypy.

Configuration

Variable	Default	Description
GITHUB_APP_ID	0	GitHub App ID
GITHUB_APP_PRIVATE_KEY_PATH	./github-app.private-key.pem	Path to the App private key
GITHUB_WEBHOOK_SECRET	empty	Shared secret for HMAC verification
ANTHROPIC_API_KEY	empty	Claude API key
CLAUDE_MODEL	claude-opus-4-8	Model used for reviews
CLAUDE_MAX_TOKENS	4096	Per-review token cap
DATABASE_URL	sqlite:///./review.sqlite3	Postgres in Docker
MAX_FILES_PER_REVIEW	40	Cap on files reviewed per PR
MAX_FILE_ADDITIONS	1500	Skip files with more added lines
IGNORE_GLOBS	*.lock,*.min.js,*.svg,package-lock.json,yarn.lock,*.snap	Always skipped
POST_CLEAN_REVIEWS	true	Comment even when nothing is found

Per-repo overrides (enabled categories, block threshold) live in the repo_configs table and are editable from the dashboard.

Tests

pytest                 # 23 tests
ruff check backend
mypy backend

The suite covers the diff parser (line-number mapping, additions/deletions, new-file detection, commentable-line tracking), the review engine's validation logic (category gating, line snapping, drop behaviour), decide_event severity mapping, and webhook signature verification plus event dispatch. .github/workflows/ci.yml runs ruff, mypy, and pytest across Python 3.10/3.11/3.12 with a Redis service container, plus a dashboard build job.

Status and limitations

• No screenshots or usage metrics are published here. docs/screenshots/ is an empty placeholder — the review quality claims above describe what the code does, not measured results on a benchmark of real PRs.
• "Estimated time saved" in the analytics view is a constant multiplier per issue, not a measurement. Treat it as a display convenience.
• Review quality is bounded by what fits in one Claude call: files beyond MAX_FILES_PER_REVIEW and large files beyond MAX_FILE_ADDITIONS are skipped, and each file is analysed with PR context but without whole-repository context.
• Findings that cannot be mapped to a commentable diff line are dropped, so a genuine issue on an unchanged line will not be reported.
• The mypy step in CI is advisory (|| true); type errors do not fail the build.
• Requires a GitHub App and an Anthropic API key to do anything — there is no offline demo mode.

License

MIT — see LICENSE.