release: v1.9.0 — evidence-driven ranking, half the query latency

.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "codebase-index",
   "displayName": "Codebase Index",
   "description": "Give Claude a precise local map of your codebase: find implementations, trace behavior, and predict change impact with file-line evidence.",
-  "version": "1.8.0",
+  "version": "1.9.0",
   "author": {
     "name": "codebase-index contributors"
   },

.claude/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.8.0
+1.9.0

.codex/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.8.0
+1.9.0

.opencode/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.8.0
+1.9.0

CHANGELOG.md

@@ -6,6 +6,89 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [1.9.0] - 2026-09-02
+
+### Added
+
+- **Objective ground truth from git history.** `tests/eval/gen_queries.py` mints a
+  retrieval benchmark from any git repository by pairing a human-written commit
+  subject with the files that commit actually changed. Unlike docstring-derived
+  benchmarks the query text is not copied into the document being retrieved, so the
+  set is leak-free by construction; merges, reverts, releases, version bumps,
+  sweeping refactors, changelog-style answers and benchmark scaffolding are all
+  filtered out.
+- **Multi-corpus, multi-language evaluation.** `run_eval.py --corpus REPO:QUERIES`
+  pools several repositories into one benchmark so ranking changes are validated
+  outside this repository and outside Python. The 1.9.0 defaults were measured on
+  305 queries across Python, Java and TypeScript corpora.
+- **Significance testing.** Every non-baseline row now reports a paired bootstrap
+  95% confidence interval and a paired permutation p-value (both seeded, therefore
+  reproducible). Query sets of this size have a noise floor of several MRR points;
+  signals now ship on the strength of that test rather than the sign of a delta.
+- **Context-noise metrics.** The report adds mean emitted snippet tokens, duplicate
+  rate of returned results, and p99 latency alongside the existing IR metrics.
+
+### Changed
+
+- **Fusion now scores cross-retriever agreement.** RRF fuses on
+  `(path, line-bucket)`, so a symbol definition at line 40 and a lexical hit at line
+  120 in the same file fused as two unrelated candidates — two retrievers agreeing
+  on a file produced two weak results instead of one strong one, and cross-source
+  agreement never reached the score. Each candidate now also receives, at
+  `file_agreement_weight` (0.4), the RRF mass of every retriever that found its file
+  at another locator, excluding retrievers already counted at that locator.
+  Ablatable via `RetrievalTuning(file_agreement=False)`.
+- **Documentation demotion deepened** from -0.05 to -0.20. Prose describing a
+  feature matches a natural-language question more literally than the code
+  implementing it, so design notes and plans were displacing the modules they
+  describe. Because this only reorders prose relative to code, documentation-seeking
+  queries improved as well (category MRR 0.579 → 0.612). Generated/vendor paths move
+  to -0.25 so they remain the least-preferred role, and `MAX_ABS_PRIOR` now pins the
+  invariant that priors stay tiebreakers.
+- **Fuzzy identifier matching is now a recall fallback.** It runs only when the
+  precise symbol lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` (3) rows. It moved no ranking metric across 305 queries while
+  accounting for ~20% of query latency; typo and acronym recall is unchanged because
+  those are exactly the queries where the precise lookup comes up empty.
+- **Candidate over-fetch is explicit.** `candidate_pool_multiplier` (default 2)
+  replaces the implicit widening that happened whenever dedup or MMR was enabled.
+  Making it explicit revealed that the quality previously credited to SimHash dedup
+  was really the wider pool; dedup is retained for what it does measurably do, which
+  is cutting the duplicate rate of returned snippets from ~1.6% to ~0%.
+
+### Fixed
+
+- **Synonym matches were reported as exact symbol matches.** `is_exact` came from
+  SQL and was relative to whichever needle retrieved the row, so a synonym
+  expansion ("config" for "configuration") marked an unrelated symbol as an exact
+  match — worth a +0.20 rerank bonus and an unconditional `high` confidence.
+  Exactness is now judged against the terms the user actually typed.
+- **Duplicate suppression was order-dependent on ties.** Equal-scoring duplicates
+  handed the slot to whichever copy arrived last, contradicting the documented
+  "ties favor input order" and making the retained snippet depend on retriever
+  emission order.
+
+### Performance
+
+- Query latency roughly halves: p50 78.6 ms → 51.2 ms, p95 193.2 ms → 94.9 ms,
+  p99 277.1 ms → 152.2 ms on the pooled three-repository benchmark, from the fuzzy
+  fallback plus a SimHash fingerprint that folds repeated tokens by multiplicity and
+  caches token digests. Fingerprint output is bit-for-bit unchanged.
+- Mean emitted snippet tokens fall 1183 → 1091 per query.
+
+### Retrieval quality
+
+Pooled over 305 queries (Python, Java, TypeScript), v1.8.0 → 1.9.0:
+
+| Metric | v1.8.0 | 1.9.0 | Δ | p |
+|---|---|---|---|---|
+| MRR | 0.564 | 0.591 | +0.027 | <0.001 |
+| MAP | 0.433 | 0.461 | +0.028 | <0.001 |
+| nDCG@10 | 0.503 | 0.527 | +0.024 | <0.001 |
+| recall@5 | 0.529 | 0.560 | +0.031 | <0.001 |
+| P@5 | 0.182 | 0.192 | +0.009 | 0.001 |
+| hit@3 | 0.649 | 0.666 | +0.016 | 0.124 |
+
 ## [1.8.0] - 2026-09-02
 
 ### Added

README.md

@@ -271,7 +271,7 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.8.0**. It includes:
+The latest released line is **1.9.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
@@ -280,15 +280,21 @@ The latest released line is **1.8.0**. It includes:
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
 - benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- rank fusion that scores cross-retriever agreement at file level, not just at a locator;
 - bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
-  impact analysis.
-
-Planned work is deliberately separated from shipped capability. The next
-product priorities are stronger real-repository evaluations, typed framework
-edges, and an even more direct task-context workflow. See the
-[roadmap](docs/ROADMAP.md).
+  impact analysis;
+- a multi-repository retrieval evaluation with leak-free git-derived ground truth,
+  one-signal ablations, and paired significance tests
+  ([tests/eval](tests/eval/README.md)).
+
+Every shipped ranking signal has to survive that evaluation: 1.9.0 removed the
+cost of two signals that could not demonstrate a benefit and rejected several
+plausible ones outright (IDF-weighted coverage, stemming, graph propagation, MMR,
+a file-length prior). Planned work is deliberately separated from shipped
+capability. The next product priorities are typed framework edges and an even more
+direct task-context workflow. See the [roadmap](docs/ROADMAP.md).
 
 ## Documentation
 

docs/ARCHITECTURE.md

@@ -2,7 +2,7 @@
 
 ## 1. Overview
 
-`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.8.0`
+`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.9.0`
 it has two shipped faces:
 
 1. **A Claude Code Skill** (`.claude/skills/codebase-index/SKILL.md`) that Claude auto-invokes for
@@ -97,8 +97,9 @@ codebase-index/
     ├── graph/                   # builder.py (edge resolution), expand.py (impact),
     │                            #   export.py (HTML graph)
     ├── storage/                 # db.py (pragmas, schema, version guard), schema.sql, repo.py
-    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py,
-    │                            #   budget.py, pipeline.py, types.py
+    ├── retrieval/               # intent.py, searchers.py, fusion.py, rerank.py, priors.py,
+    │                            #   lexical.py, fuzzy.py, diversity.py, skeleton.py,
+    │                            #   tuning.py (ablation contract), budget.py, pipeline.py, types.py
     ├── embeddings/              # backend.py, noop.py (default), local.py, external.py — opt-in
     ├── output/                  # markdown.py, json.py, redact.py
     ├── watch/                   # watcher.py (optional, watchdog-based)

docs/BENCHMARKS.md

@@ -1,10 +1,11 @@
 # Benchmarks
 
-`codebase-index` has three benchmark surfaces. Read them with their status in
+`codebase-index` has four benchmark surfaces. Read them with their status in
 mind — the whole point of this page is to keep evidence and aspiration separate.
 
 | Surface | What it is | Status | Use it as |
 |---|---|---|---|
+| Retrieval eval (`tests/eval/`) | Ranking quality vs a fixed baseline across multiple real repositories, with significance tests | **Proven (relative)** | The gate for ranking changes; measures *deltas*, not absolute superiority |
 | Public suite (`tests/benchmark_public.py`) | Deterministic synthetic multi-language fixture with the full metric framework | **Toy/synthetic** | CI regression gate + metric shape, **not** product-quality evidence |
 | Smoke/perf (`test_perf_smoke.py`, `test_benchmark_comparison.py`) | Latency + output-size guards on a tiny fixture | **Toy/smoke** | Regression checks only |
 | Honest real-repo (`tests/benchmark_honest.py`) | 55k LOC Java repo, recall@3 vs disciplined `rg` baseline, symmetric token accounting | **Proven (one repo)** | The only headline product-quality number we stand behind today |
@@ -15,15 +16,33 @@ Do not write, imply, or ship any of these until a run with published logs exists
 
 - Any 10k / 100k / 1M LOC scale or speed claim (no real run at that size).
 - "Beats Cursor / Sourcegraph / Codebase-Memory MCP" — no head-to-head exists.
-- Per-language quality claims beyond Java (the honest run is Java-only).
+- Per-language *absolute* quality claims beyond Java. The retrieval eval covers
+  Python, Java and TypeScript, but it measures this system against its own earlier
+  versions — it says a change helped, not that the product beats an alternative.
 - Generic "Nx faster" / "Nx fewer tokens" without naming the baseline and repo.
-- Latency claims — the honest run explicitly does not headline latency
-  (Python process start dominates; real `rg` is tens of ms).
+- Latency claims against external tools — the honest run explicitly does not
+  headline latency (Python process start dominates; real `rg` is tens of ms).
+  Version-over-version latency from the retrieval eval is in-process and is only
+  comparable to other runs of that harness.
 
 The defensible headline today is exactly: **on one 55k LOC Java repo, recall@3 was
 70% (index) vs 40% (`rg`+window), using ~13× fewer answer tokens.** Everything
 else is roadmap.
 
+## Retrieval evaluation (ranking gate)
+
+See [tests/eval/README.md](../tests/eval/README.md) for the full protocol. In short:
+ground truth comes from hand-written queries verified against the source tree and
+from commit-subject → changed-files pairs mined from git history (leak-free: the
+query text is not in the indexed corpus). Corpora are pooled across languages, one
+index per corpus is shared by every variant, and every non-baseline row carries a
+paired bootstrap CI and permutation p-value.
+
+`1.9.0` was measured over 305 queries across Python, Java and TypeScript corpora
+against `1.8.0`: MRR +0.027, MAP +0.028, nDCG@10 +0.024, recall@5 +0.031 (all
+p < 0.001), with p50 latency 78.6 ms → 51.2 ms. These are version-over-version
+ranking deltas on those corpora, not a universal quality claim.
+
 ## Public benchmark suite
 
 Run:

docs/RETRIEVAL.md

@@ -65,6 +65,11 @@ source)` list so fusion is source-agnostic.
   first when the query clearly names a path.
 - **Symbol** — query against `symbols` (exact, identifier parts, bounded fuzzy matching). Carries
   `kind` (function/class/method/...) and signature. Primary for `locate_impl` / `find_refs`.
+  Fuzzy identifier matching (acronym / concatenation / edit distance) runs only as a **recall
+  fallback**, when the precise lookup named no symbol and returned fewer than
+  `fuzzy_fallback_min` rows. Measured over 305 queries on three repositories it moved no ranking
+  metric while costing ~20% of query latency, so it is kept for typos and abbreviations but no
+  longer runs when the query already spelled its identifier correctly.
 - **FTS** — FTS5 `bm25()` over the `fts_chunks` virtual table (chunk text + symbol names +
   summaries indexed). Query-time camelCase/snake_case splitting, small down-weighted synonym
   expansion, and soft coverage scoring make natural-language questions robust without weakening
@@ -79,28 +84,66 @@ source)` list so fusion is source-agnostic.
 scores:
 
 ```
-RRF(d) = Σ_r  w_r / (k + rank_r(d))        # k ≈ 60, w_r = per-intent retriever weight
+RRF(d) = Σ_r  w_r · k / (k + rank_r(d))    # k ≈ 60, w_r = per-intent retriever weight
 ```
 
 - Robust to scale differences between BM25 and cosine.
 - Per-intent weights `w_r` let `locate_impl` favor the symbol list and `how_it_works` favor FTS.
+- Scaled by `k` so fused scores and the reranker's bounded bonuses share an O(1) scale. This is a
+  monotonic rescale; fusion order is unchanged.
 - Ties broken by rerank features (next step).
 
-## 4. Reranking (`retrieval/rerank.py`)
+### Cross-locator file agreement
+
+Fusion keys on `(path, line-bucket)`, not `(path, start, end)`, because different retrievers report
+different line ranges for the same place. Bucketing alone was not enough: a symbol defined at line
+40 and a lexical hit at line 120 are genuinely different locators, so a file that **two retrievers
+agreed on** still fused as two separate candidates, each carrying one retriever's evidence — and
+cross-source agreement, the entire point of RRF, never fired.
+
+Each candidate therefore also receives, at weight `file_agreement_weight`, the RRF mass of every
+retriever that found its *file* at some other locator:
+
+```
+score(d) = RRF(d) + α · Σ_{r ∉ sources(d)}  w_r · k / (k + best_rank_r(path(d)))
+```
+
+Retrievers already counted at the candidate's own locator are excluded, so nothing double-counts,
+and the term is bounded by the same weights as fusion itself. `α = 0.4`; the 0.3–0.6 plateau peaks
+there. Set `RetrievalTuning(file_agreement=False)` to recover plain locator-only RRF.
 
-A lightweight, explainable feature score (no external model required) layered on the fused order:
+## 4. Reranking (`retrieval/rerank.py`)
 
-| Feature | Intuition |
-|---|---|
-| symbol-kind match | a `def`/`class` outranks an incidental mention |
-| path proximity | files near a query-named path score higher |
-| graph centrality | high in/out-degree nodes matter more for `architecture` |
-| recency | recently changed files (git mtime) slightly boosted |
-| exact-name bonus | exact symbol-name match dominates fuzzy |
-| test/generated penalty | test files and generated code demoted unless asked |
+A lightweight, explainable feature score (no external model required) layered on the fused order.
+Every term is bounded, so reranking reorders near-neighbours rather than overruling retrieval:
+
+| Feature | Effect | Intuition |
+|---|---:|---|
+| Exact symbol match | +0.20 | the user named a specific symbol |
+| Symbol definition kind | +0.05 | a `def`/`class` outranks an incidental mention |
+| Symbol name among query terms | +0.05 | the name was asked for, not just matched |
+| Path term match | +0.05 | the user supplied a location clue |
+| Graph centrality (`in_degree`) | ≤ +0.08 | `log1p`-damped, so a god class cannot dominate |
+| Reference-count fallback | ≤ +0.04 | for names too common to resolve a precise `in_degree` |
+| Source role prior | −0.25…+0.08 | see below |
+| Generated, or test on a non-test query | −0.15 | supporting evidence, not the answer |
+
+### Source role priors (`retrieval/priors.py`)
+
+| Role | Prior | Rationale |
+|---|---:|---|
+| Implementation | +0.08 | the answer to a code question is usually code |
+| Test | −0.06 | flips to +0.05 when the query or intent is test-oriented |
+| Documentation | −0.20 | prose *about* a feature matches a natural-language question more literally than the code implementing it, so design notes and plans crowded out the modules they describe |
+| Generated / vendor / build | −0.25 | never the answer; kept strictly below documentation |
+
+The documentation prior deepened from −0.05 in 1.8.0. Because it only reorders prose relative to
+code — never below other prose — documentation-seeking queries improved too (MRR 0.579 → 0.612).
+At −0.35 that reverses and the `docs` category collapses, so the optimum is interior, not a
+"more is better" knob. `MAX_ABS_PRIOR` caps every prior so this stays a tiebreaker.
 
 The reranker also produces the human-readable **`reason`** string per result
-(e.g. *"exact symbol match · called by 4 sites · in src/auth/"*).
+(e.g. *"exact symbol match · 4 callers · in src/auth/"*).
 
 ## 5. Graph expansion (`graph/retrieval.py`; `graph/expand.py` for impact APIs)
 
@@ -119,11 +162,17 @@ Expanded nodes retain edge confidence and receive distance-decayed scores so see
 
 ## 6. Diversity and duplicate control
 
-`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate
-suppression. MMR is disabled in the shipped default because the reproducible
-benchmark favored relevance-only ranking; callers that need broader snippet
-coverage can enable `RetrievalTuning(mmr=True)`. Duplicate suppression remains
-available independently.
+`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate suppression.
+
+MMR is disabled in the shipped default: it moved no ranking metric on the benchmark and roughly
+doubled p50 latency. Callers that need broader snippet coverage can enable
+`RetrievalTuning(mmr=True)`.
+
+SimHash duplicate suppression stays on, but on noise grounds rather than ranking grounds: it does
+not move MRR, and it takes the duplicate rate of returned snippets from ~1.6% to ~0%. The
+over-fetch that feeds selection is now an explicit `candidate_pool_multiplier` rather than an
+implicit side effect of enabling dedup — an earlier ablation credited dedup with a quality win that
+was really the wider pool doing the work.
 
 ## 7. Token budgeting (`retrieval/budget.py`)