Codebase Investigator

Codebase Investigator is an AI-assisted repository analysis tool for public GitHub repositories.

Paste a GitHub URL, ask a plain-English question, and get an answer grounded in cited files and line ranges. Each answer includes a retrieval trace and a separate trust check so the output is easier to verify.

Live demo: codebase-investigator-81wr.onrender.com

A higher-level project writeup is available in the public case study: codebase-investigator-case-study.

What It Does

• Clones a public GitHub repo.
• Indexes source files.
• Retrieves relevant code with hybrid search:
  • TF-IDF exact matching
  • OpenAI embeddings
  • broad review scan for vague questions
  • duplicate merge
  • MMR reranking
• Sends selected evidence to OpenAI for the answer.
• Audits the answer separately with:
  • programmatic citation validation
  • independent model critique
• Stores session history for follow-up questions.

Why It Matters

Generic LLM answers about unfamiliar codebases are easy to over-trust. This project is built around evidence first: retrieve the relevant files, answer from that context, cite the evidence, and run a separate trust check on the result.

Run Locally

$env:OPENAI_API_KEY = "your-key"
$env:OPENAI_MODEL = "gpt-4o-mini"
$env:OPENAI_AUDIT_MODEL = "gpt-4o-mini"
$env:OPENAI_EMBEDDING_MODEL = "text-embedding-3-small"

python -m investigator serve --host 127.0.0.1 --port 8765

Open:

http://127.0.0.1:8765

CLI

python -m investigator ask https://github.com/owner/repo "How does auth work here?"
python -m investigator chat https://github.com/owner/repo --session review-1

API

POST /api/ask

{
  "repo": "https://github.com/owner/repo",
  "session": "review-1",
  "question": "What problems do you see with this?"
}

Returns:

• answer
• cited evidence
• retrieval trace
• trust check
• session claims

Project Structure

investigator/
  repo.py         clone + index GitHub repos
  retrieval.py    hybrid retrieval + MMR reranking
  embeddings.py   OpenAI embeddings + cache
  llm.py          OpenAI answer generation
  auditor.py      citation checks + separate model audit
  memory.py       session history
  service.py      app orchestration
  web.py          browser UI + API
  cli.py          command interface
  models.py       shared dataclasses
tests/
docs/
.env.example

Tests

python -m unittest discover

Current coverage checks:

• hybrid retrieval
• embedding candidate merge
• broad review questions
• hallucinated citation detection
• separate model audit path
• multi-turn context passing
• API response shape