RepoContextBench

RepoContextBench evaluates whether a coding agent can answer practical questions about a repository by finding the right context, covering required facts, citing evidence, and avoiding unsupported concrete claims.

Version 2.0 has three benchmark tracks:

Track	Subject	Tasks	Purpose
Lite	microsoft/agent-framework at 47fa59f8e9d7b91e382834b42ecff45e22e2d890	20	Broad, practical repository research
Hard	facebook/sapling at 1e764c94ae163869a303fbead19f279a8466ab44	20	Deep failure, concurrency, durability, compatibility, and change-impact analysis
Full	Lite + Hard	40	Primary combined result

Full is a deterministic composition of one complete Lite run and one complete Hard run from the same answerer configuration. It is not a third execution. Its profile is rebuilt from all 40 task-score rows, so Score is a per-task micro average and resource totals are additive.

The published dashboard is available at codealive-ai.github.io/repo-context-bench.

Repository layout

dataset/agent-framework/v1/   Immutable 1.0 dataset
dataset/agent-framework/v2/   2.0 Lite track
dataset/sapling/v2/           2.0 Hard track
dataset/full/v2/              2.0 Full composition manifest and combined tasks
results/v1/                   Immutable 1.x official results
results/v2/                   2.x results when published
src/RepoContextBench/         C# runner, judge, composer, exporter, report server
tests/RepoContextBench.Tests/ Runner, dataset, composition, and publication tests
docs/                         Benchmark, dataset, results, and methodology cards

Runner dependency

The runner depends on an authorized private CodeAlive backend checkout because it measures the actual CodeAlive ContextResearchAgent. Codex CLI, Claude Code, OpenCode, and Kimi Code harnesses currently build through the same dependency graph.

Place repo-context-bench and codealive-app next to each other, or pass:

dotnet build RepoContextBench.slnx \
  -p:CodeAliveBackendRoot=/path/to/codealive-app/src

Validate v2

dotnet run --project src/RepoContextBench -- validate-dataset \
  --dataset dataset/agent-framework/v2/tasks.jsonl \
  --manifest dataset/agent-framework/v2/manifest.json \
  --judge-regression-file dataset/agent-framework/v2/judge-regression.jsonl \
  --source-root /path/to/agent-framework

dotnet run --project src/RepoContextBench -- validate-dataset \
  --dataset dataset/sapling/v2/tasks.jsonl \
  --manifest dataset/sapling/v2/manifest.json \
  --judge-regression-file dataset/sapling/v2/judge-regression.jsonl \
  --source-root /path/to/sapling

dotnet run --project src/RepoContextBench -- validate-dataset \
  --dataset dataset/full/v2/tasks.jsonl \
  --manifest dataset/full/v2/manifest.json \
  --judge-regression-file dataset/full/v2/judge-regression.jsonl

Validate the component tracks against their own source roots. Full validation is structural because its evidence spans belong to two different checkouts.

Run and compose

Run Lite and Hard separately with the same system configuration and the correct repository data source or local checkout. The frozen v2 publication judge is Codex CLI gpt-5.6-sol with medium reasoning effort.

dotnet run --project src/RepoContextBench -- run \
  --dataset dataset/agent-framework/v2/tasks.jsonl \
  --manifest dataset/agent-framework/v2/manifest.json \
  --repository-id "$AGENT_FRAMEWORK_REPOSITORY_ID" \
  --organisation-id "$CODEALIVE_ORGANISATION_ID" \
  --out artifacts/runs/v2/lite/my-system \
  --judge enabled --judge-provider codex_cli \
  --judge-model gpt-5.6-sol --judge-reasoning-effort medium

dotnet run --project src/RepoContextBench -- run \
  --dataset dataset/sapling/v2/tasks.jsonl \
  --manifest dataset/sapling/v2/manifest.json \
  --repository-id "$SAPLING_REPOSITORY_ID" \
  --organisation-id "$CODEALIVE_ORGANISATION_ID" \
  --out artifacts/runs/v2/hard/my-system \
  --judge enabled --judge-provider codex_cli \
  --judge-model gpt-5.6-sol --judge-reasoning-effort medium

dotnet run --project src/RepoContextBench -- compose-full \
  --lite-run artifacts/runs/v2/lite/my-system \
  --hard-run artifacts/runs/v2/hard/my-system \
  --dataset dataset/full/v2/tasks.jsonl \
  --manifest dataset/full/v2/manifest.json \
  --out artifacts/runs/v2/full/my-system

compose-full rejects partial, unhealthy, hash-mismatched, differently judged, or differently configured component runs.

Dashboard

Place complete Lite, Hard, and composed Full run directories under one runs root, then build or serve against the Full task union:

dotnet run --project src/RepoContextBench -- serve \
  --runs artifacts/runs/v2/all \
  --dataset dataset/full/v2/tasks.jsonl \
  --manifest dataset/full/v2/manifest.json \
  --port 8770

The dashboard treats each track's declared task count as complete and defaults the main leaderboard to Full when Full results exist.

Publication contract

A publishable Lite or Hard result requires exactly 20 unique task results. A publishable Full result requires exactly 40 and valid component provenance. Every official result also requires:

• partial_run = false;
• zero fatal, network, and judge failures;
• positive answerer timing;
• Codex CLI gpt-5.6-sol medium as judge;
• dataset, manifest, track, and task count matching the requested export;
• source dataset/manifest snapshots matching the export, complete answerer model-call telemetry, exact per-task token ledgers, and successful per-task judgment artifacts;
• schema 2, a fresh non-resumed run, the frozen v2 dataset/manifest hashes and subject commits, and a score profile recomputed from task scores that themselves reproduce the complete verdicts;
• successful path, internal-ID, and secret sanitization.

Admission never depends on Score. Export each track with its own dataset and manifest; never copy raw private run directories into results/. Runs snapshot both inputs on creation; --resume true refuses to reuse scores when the subject, execution profile, skill hash, or either source hash differs. Resume is for operational recovery and marks the run non-publishable; official runs are clean single-shot executions. annotate-runs remains available only for legacy v1 runs.

Interpretation

Score is the partial-credit, judge-based repository-answer metric. Certification is a stricter gate. Speed, tokens, and cost use answerer measurements only. Full is the primary cross-system result; Lite and Hard remain diagnostic tracks and should not be averaged from rounded dashboard values.

All gold is public, so the release is auditable but vulnerable to contamination and task-specific tuning. Sapling Hard was model-audited and pilot-calibrated on visible gold; human claim calibration and negative-control baselines remain pending for any hidden-test claim.

Token, cost, cache, retry, and startup-time semantics differ across CLI harnesses. The normalized contract and its primary-source rationale are documented in docs/telemetry.md. The displayed cost is an API-equivalent estimate from the pinned standard API price snapshot, not observed subscription spend or an invoice charge.

Kimi Code/K3 setup, read-only policy, and exact session-usage extraction are documented in src/RepoContextBench/run-guides/kimi-code.md.

Licenses and citation

• runner/dashboard code: MIT (LICENSE)
• dataset annotations and results: CC BY 4.0 (DATA_LICENSE)
• Agent Framework evidence: upstream MIT attribution
• Sapling evidence: upstream GPL-2.0 attribution

Citation metadata is in CITATION.cff.