Does synthesizing organizational context (git history + issues + docs) actually produce better engineering answers than retrieval-augmented generation alone — or is that just a plausible-sounding claim?
Atlas is a benchmark built to answer that question honestly, including about itself. The first run said no. That verdict turned out to be invalid — the benchmark's own methodology had silently broken. Atlas exists to show the fix, the re-run, and the adversarial re-check of that re-run, not just a favorable number.
Atlas synthesizes cross-source organizational knowledge — commit history, issues/PRs, and documentation — into structured, cited context artifacts (ownership, dependencies, constraints, known risks, recent evolution) for the entities in a codebase. The core question this repository exists to test: does giving an LLM that synthesized context, instead of just raw retrieved chunks, produce measurably better answers to real engineering questions?
Engineers answering questions like "who owns this module," "what would break if I refactored this," or "why did this regress" today get either tribal knowledge (doesn't scale, walks out the door) or retrieval-augmented search over raw commits/issues (finds text, doesn't synthesize relationships, ownership, or constraints across sources). Atlas tests whether a synthesis step in between — turning scattered evidence into structured, queryable claims — closes that gap.
Most "we benchmarked our RAG system" write-ups show one clean run. This repository shows three phases of the same investigation, kept in full:
- The first benchmark run reported the hypothesis rejected. An audit found that verdict was invalid — a token-budget misconfiguration against a reasoning model had silently produced empty synthesis output on every entity. The benchmark measured a bug, not the thesis.
- The bug (and two further, previously unknown ones) was fixed, and the benchmark was re-run, producing a genuine positive result.
- That result was then adversarially reviewed — a fresh, hostile pass instructed to try to refute it — which found real problems (a circular bias check, an overclaimed statistical finding) and forced the conclusions to be narrowed before anything was called final.
The current, honest bottom line — Level 2, Scientifically Validated, not the highest available level, with exact reasons why — is in docs/.
| Run A | Run B (repeat) | |
|---|---|---|
| Task Correctness Delta | 18.6% | 24.7% |
| Cross-Source Advantage | 20.3% | 27.7% |
| Verdict | POSITIVE VALIDATION | POSITIVE VALIDATION |
Both independent runs exceeded the pre-registered thresholds. Read this correctly, not optimistically: synthesis measurably helps on questions whose answer shape matches synthesis's own schema (dependencies, ownership, historical change) — confirmed, not just claimed, by direct inspection of the synthesized data. Two significant open risks remain unresolved: the same model generates and judges answers (self-judging, currently unmeasured), and only one reasoning provider has been tested. Full statistical detail, including what does not hold up under scrutiny: docs/ATLAS-SCIENTIFIC-CERTIFICATE.md.
flowchart LR
A[Collect\ngit + issues + docs] --> B[Normalize]
B --> C[Synthesize\nLLM: entity context]
C --> D{Validate\ngate}
D -- fails --> X[Abort +\ndiagnostic report]
D -- passes --> E[Evaluate\nbaseline vs. synthesis]
E --> F[Report\nstatistical verdict]
Every stage is idempotent (skips work already cached in data/) except the validation gate, evaluation, and report, which always run fresh. The validation gate is the mechanism that catches what broke the first run: before any evaluation cost is spent, it checks synthesis isn't degenerate, evidence coverage is adequate, token budgets are safe, and both the reasoning and embedding providers are reachable — and aborts with a written diagnostic rather than silently producing a report from bad data. See docs/ATLAS-CERTIFICATION.md for what it caught, live, during this project's own development.
The reasoning/embedding provider is swappable via configuration only — OpenAI, Anthropic, DeepSeek, OpenRouter, and Ollama are all supported through one adapter interface, with zero business-logic code aware of which is active (docs/provider-audit.md).
npm install
cp .env.example .env
# set REASONING_API_KEY (or provider-specific key) and GITHUB_TOKEN in .env
npm run validateRequires Node ≥20. See .env.example for every configuration option, or the Environment Variables section below for the ones that matter most.
The exact commands used to produce the headline result above, against a cached offline evidence snapshot (no GitHub API calls needed):
rm -f data/synthesized/*.json # force real re-synthesis
npx tsx src/synthesize/index.ts # ~5 min, 11 live LLM calls
npx tsx src/main.ts # ~35 min, validation gate + 84 live LLM calls + reportBoth commands were run to completion, live, twice, during development of this repository — not a theoretical claim. Full reproduction certificate, including two real bugs found and fixed in the process (documented, not silently patched): docs/reproducibility-certificate.md.
Universal provider configuration (see .env.example for the complete reference):
| Variable | Description |
|---|---|
GITHUB_TOKEN |
GitHub personal access token for API access |
REASONING_PROVIDER |
openai | anthropic | deepseek | openrouter | ollama |
REASONING_API_KEY |
API key for the reasoning provider |
REASONING_MODEL |
Reasoning model name |
SYNTHESIS_MAX_TOKENS / ANSWER_MAX_TOKENS / JUDGE_MAX_TOKENS |
Per-call-type output-token ceilings — see below for why these are separate |
REASONING_EFFORT |
low | medium | high — honored by providers that support it (DeepSeek, OpenRouter) |
EMBEDDING_PROVIDER / EMBEDDING_API_KEY / EMBEDDING_MODEL |
Embedding provider config (default: OpenAI text-embedding-3-small) |
TARGET_REPO / EVIDENCE_SINCE / MAX_COMMITS / MAX_ISSUES |
What to collect and how much |
RUNS_PER_TASK |
Evaluation runs per benchmark task (default: 3) |
Why separate per-call-type token budgets? Synthesis, answer generation, and judging have different completion-length needs, and thinking-capable models (DeepSeek, o-series, etc.) draw hidden reasoning tokens from the same budget as the visible output. A ceiling too low for the task is exactly what invalidated the first benchmark run — see docs/history/v1-methodology-audit/synthesis-analysis.md.
npm run collect # Gather evidence from GitHub
npm run normalize # Unify into NormalizedEvidence[]
npm run synthesize # LLM synthesis of entity context
npm run evaluate # Run the benchmark (baseline vs. synthesis)
npm run report # Statistical verdict
npm run validate # Full pipeline: all of the above + the validation gate (recommended entry point)Running the full benchmark takes ~40 minutes. To see baseline vs. synthesis on one question instead:
npx tsx scripts/run-single-task.ts --list # see available questions
npx tsx scripts/run-single-task.ts task-5-implicit-dependencies # run one, live, with timingTiming note: the 4 LLM calls themselves take under a minute combined (~1 min), but building the retrieval index first — embedding all 1,018 evidence records locally via Ollama — took ~10-11 minutes in testing. That one-time cost is unrelated to which question you ask; it's the dominant factor in this script's total runtime, not the LLM calls. Budget accordingly.
npm test # 211 tests
npm run typecheck # tsc --noEmit, strict mode, covers src/ and scripts/A real synthesized entity, routing, spot-checked line-by-line in docs/history/v2-fixes-and-revalidation/synthesis-report.md: named owner and contributors with commit-evidence citations, dependency graph with confidence-scored edges, severity-graded constraints, PR-numbered change history. The full file: data/synthesized/routing.json. Full benchmark output from both certified runs: data/results/ and data/_archive-v2/.
Start at docs/README.md — it indexes the current certification (engineering, scientific, stability, reproducibility, risk assessment, presentation-readiness) and the full audit-and-fix history behind it. If you read one document, make it docs/ATLAS-CERTIFICATION.md.
Stated here because a README that hides them isn't trustworthy, and the scientific certificate states them at length:
- Self-judging: the same model generates and scores both benchmark conditions. No valid check against this bias currently exists in this codebase.
- Single-provider validated: DeepSeek only. A Claude comparison is architecturally ready but blocked on a missing API key — see
docs/provider-comparison.mdfor exact resume steps. - n=7 benchmark tasks: statistically underpowered; the significance test's p-value is not stable across repeat runs (disclosed and measured in
docs/stability-report.md, not glossed over).
Apache 2.0 — see LICENSE.