release: v1.8.0

.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "codebase-index",
   "displayName": "Codebase Index",
   "description": "Give Claude a precise local map of your codebase: find implementations, trace behavior, and predict change impact with file-line evidence.",
-  "version": "1.7.0",
+  "version": "1.8.0",
   "author": {
     "name": "codebase-index contributors"
   },

.claude/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.7.0
+1.8.0

.codex/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.7.0
+1.8.0

.opencode/skills/codebase-index/.skill_version

@@ -1 +1 @@
-1.7.0
+1.8.0

CHANGELOG.md

@@ -6,6 +6,26 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [1.8.0] - 2026-09-02
+
+### Added
+
+- **Reproducible retrieval evaluation.** Added a self-repository ground-truth
+  query suite with Recall@K, MRR, nDCG, hit rate, precision, MAP, useful-context,
+  latency percentiles, and one-signal ablations.
+
+### Changed
+
+- **Hybrid retrieval quality.** Natural-language lexical queries now use safe,
+  down-weighted identifier/synonym expansion and soft term coverage; exact symbol
+  lookup preserves framing-word tolerance and bounded fuzzy matching.
+- **Packaging compatibility.** Cap the build backend below the Metadata 2.5
+  default until the release validation toolchain supports that metadata version.
+- **Ranking defaults are evidence-driven.** Implementation/test/documentation
+  source priors are calibrated from the benchmark. Graph propagation is bounded
+  and intent-directed; graph and MMR signals remain opt-in because ablations
+  reduced direct retrieval quality on the reproducible corpus.
+
 ## [1.7.0] - 2026-07-29
 
 ### Added
@@ -401,7 +421,8 @@ All notable changes to this project are documented here. The format is based on
 - Hooks example + `watch` mode for keeping the index fresh without blocking the edit loop (M8).
 - `doctor`, `stats`, `clean` diagnostics/maintenance commands.
 
-[Unreleased]: https://github.com/denfry/codebase-index/compare/v1.7.0...HEAD
+[Unreleased]: https://github.com/denfry/codebase-index/compare/v1.8.0...HEAD
+[1.8.0]: https://github.com/denfry/codebase-index/compare/v1.7.0...v1.8.0
 [1.7.0]: https://github.com/denfry/codebase-index/compare/v1.6.0...v1.7.0
 [1.6.0]: https://github.com/denfry/codebase-index/compare/v1.5.0...v1.6.0
 [1.5.0]: https://github.com/denfry/codebase-index/compare/v1.4.0...v1.5.0

README.md

@@ -271,14 +271,16 @@ See [MCP.md](docs/MCP.md) for client configuration.
 
 ## Project status
 
-The latest released line is **1.7.0**. It includes:
+The latest released line is **1.8.0**. It includes:
 
 - hybrid and optional vector retrieval;
 - Tree-sitter symbol extraction across the documented language tiers;
 - import, call, reference, and inheritance graphs;
 - architecture communities, central nodes, and surprising cross-module links;
 - shortest dependency paths and node descriptions;
 - token-budgeted and skeletonized retrieval packets;
+- benchmark-calibrated lexical expansion, fuzzy identifier matching, and source-aware ranking;
+- bounded, intent-directed graph discovery with optional diversity and duplicate suppression;
 - CLI, Skill, plugin, and MCP delivery;
 - incremental updates, watch hooks, diagnostics, skill rollback, and diff-aware
   impact analysis.

docs/ARCHITECTURE.md

@@ -2,7 +2,7 @@
 
 ## 1. Overview
 
-`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.7.0`
+`codebase-index` is a **local-first** code intelligence layer for AI coding agents. In `1.8.0`
 it has two shipped faces:
 
 1. **A Claude Code Skill** (`.claude/skills/codebase-index/SKILL.md`) that Claude auto-invokes for

docs/FAQ.md

@@ -14,7 +14,7 @@ pip install codebase-index        # or: pipx install codebase-index
 ```
 
 To pin an exact version or grab an unreleased commit, install from a GitHub tag
-instead: `pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.7.0"`.
+instead: `pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.8.0"`.
 
 Then run `codebase-index init` inside your project and `codebase-index index` to build
 the first index. In Claude Code you can instead install the plugin
@@ -162,11 +162,12 @@ Yes. Use any of these methods:
 
 ## Is it production-ready?
 
-Yes — `codebase-index` is released as **v1.7.0**. The core indexing and search
-functionality is implemented and tested. The current `1.7.0` package includes:
+Yes — `codebase-index` is released as **v1.8.0**. The core indexing and search
+functionality is implemented and tested. The current `1.8.0` package includes:
 
-- Hybrid FTS/path/symbol/vector retrieval
-- Import/call/reference graph expansion and `impact`
+- Hybrid FTS/path/symbol/vector retrieval with benchmark-calibrated lexical expansion
+  and bounded fuzzy identifier matching
+- Import/call/reference graph expansion, intent-directed graph discovery, and `impact`
 - Diff-aware blast-radius analysis for tracked working-tree changes
 - Optional local embeddings, with external embeddings gated behind explicit opt-in
 - Hooks and watch mode for freshness

docs/INSTALLATION.md

@@ -67,7 +67,7 @@ pipx install codebase-index            # isolated environment
 uv tool install codebase-index         # uv-managed tool
 
 # Pin to a GitHub tag for an exact or unreleased version
-pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.7.0"
+pip install "codebase-index @ git+https://github.com/denfry/codebase-index.git@v1.8.0"
 
 # From source (editable mode)
 git clone https://github.com/denfry/codebase-index.git
@@ -90,7 +90,7 @@ pip install -e ".[embeddings-local,watch,dev]"
 
 ### uvx / Homebrew status
 
-As of `1.7.0`, **PyPI is shipped** — `pip install codebase-index` and
+As of `1.8.0`, **PyPI is shipped** — `pip install codebase-index` and
 `pipx install codebase-index` are the verified paths. `uvx codebase-index init`,
 Homebrew tap installation, signed checksums, and SBOMs remain distribution
 targets for a more complete release story.
@@ -130,7 +130,7 @@ Expected output:
 === codebase-index Doctor ===
 
 [OK] Python 3.12 (requires 3.11+)
-[OK] codebase-index package installed (v1.7.0)
+[OK] codebase-index package installed (v1.8.0)
 [OK] tree-sitter is available
 [INFO] Cache directory not yet created: ...
 [INFO] Skill not installed in .claude/skills/

docs/RETRIEVAL.md

@@ -2,7 +2,8 @@
 
 The retrieval engine turns a natural-language or symbolic query into a **compact, ranked,
 token-budgeted** set of file/line ranges for Claude to read. It is hybrid: multiple independent
-retrievers run, their results are fused, reranked, expanded via the graph, then trimmed.
+retrievers run, their results are fused and reranked, then trimmed. Graph expansion and MMR are
+available as bounded opt-in signals; the shipped default prioritizes direct evidence.
 
 ```
 query
@@ -21,13 +22,16 @@ query
 [3] rank fusion  ── Reciprocal Rank Fusion (RRF) across retriever result lists
   │
   ▼
-[4] rerank  ── feature-based score (symbol-kind, path proximity, recency, centrality)
+[4] rerank  ── feature-based score (symbol-kind, path, source role, centrality)
   │
   ▼
-[5] graph expansion  ── pull in imports/callers/callees per intent (bounded)
+[5] optional graph expansion  ── pull in imports/callers/callees per intent (bounded)
   │
   ▼
-[6] token budgeting  ── greedy fill under --token-budget; snippets trimmed + secret-redacted
+[6] diversity / duplicate filtering (MMR opt-in, SimHash duplicate guard)
+  │
+  ▼
+[7] token budgeting  ── greedy fill under --token-budget; snippets trimmed + secret-redacted
   │
   ▼
 ranked results + recommended_reads + fallback_suggestions
@@ -59,10 +63,12 @@ source)` list so fusion is source-agnostic.
 
 - **Path** — exact and glob path matches (`src/auth/*.py`, `auth.py`). Highest precision; surfaced
   first when the query clearly names a path.
-- **Symbol** — query against `symbols` (name exact, prefix, then fuzzy/trigram). Carries `kind`
-  (function/class/method/...) and signature. Primary for `locate_impl` / `find_refs`.
+- **Symbol** — query against `symbols` (exact, identifier parts, bounded fuzzy matching). Carries
+  `kind` (function/class/method/...) and signature. Primary for `locate_impl` / `find_refs`.
 - **FTS** — FTS5 `bm25()` over the `fts_chunks` virtual table (chunk text + symbol names +
-  summaries indexed). Tokenizer is code-aware (splits camelCase/snake_case). Primary lexical signal.
+  summaries indexed). Query-time camelCase/snake_case splitting, small down-weighted synonym
+  expansion, and soft coverage scoring make natural-language questions robust without weakening
+  exact terms.
 - **Vector** *(opt-in)* — cosine similarity over chunk embeddings via `sqlite-vec`. Only runs if
   `embeddings.enabled = true`. Adds semantic recall for paraphrased queries. Absent → pipeline
   degrades gracefully to FTS+symbol.
@@ -96,19 +102,30 @@ A lightweight, explainable feature score (no external model required) layered on
 The reranker also produces the human-readable **`reason`** string per result
 (e.g. *"exact symbol match · called by 4 sites · in src/auth/"*).
 
-## 5. Graph expansion (`graph/expand.py`)
+## 5. Graph expansion (`graph/retrieval.py`; `graph/expand.py` for impact APIs)
+
+Graph expansion runs only when the tuning enables `graph_source` and the intent plan requests a
+graph strategy. It is disabled by default because the reproducible self-repository ablation
+reduced direct-hit MRR when related nodes displaced lexical hits.
 
-After reranking, pull in *related* nodes per the intent's graph strategy, bounded by `--depth`
-(default 1–2) and a node cap:
+When enabled, it is bounded by depth and node cap:
 
 - `impact` → walk **up** edges (callers, importers) = blast radius.
 - `how_it_works` → walk **down** edges (callees, imported defs) = mechanism.
-- `find_refs` → direct reverse edges only.
-- `data_flow` → both directions along call/assignment edges.
+- `find_refs` → walk **up** edges to callers/importers.
+- `data_flow` → walk **both** directions along call/assignment edges.
+
+Expanded nodes retain edge confidence and receive distance-decayed scores so seeds stay on top.
+
+## 6. Diversity and duplicate control
 
-Expanded nodes are merged into results with a discounted score so seeds stay on top.
+`retrieval.diversity` provides bounded MMR selection and SimHash near-duplicate
+suppression. MMR is disabled in the shipped default because the reproducible
+benchmark favored relevance-only ranking; callers that need broader snippet
+coverage can enable `RetrievalTuning(mmr=True)`. Duplicate suppression remains
+available independently.
 
-## 6. Token budgeting (`retrieval/budget.py`)
+## 7. Token budgeting (`retrieval/budget.py`)
 
 Results are trimmed to fit `--token-budget` (default per intent, e.g. 1500 tokens):
 
@@ -121,7 +138,7 @@ Results are trimmed to fit `--token-budget` (default per intent, e.g. 1500 token
 
 The point: Claude gets enough to decide, and a precise list of what to read next — never a dump.
 
-## 7. Confidence & fallback
+## 8. Confidence & fallback
 
 A `confidence` score (high/medium/low) is derived from: top RRF score, score gap between #1 and #2,
 number of agreeing retrievers, and whether a symbol matched exactly.
@@ -131,13 +148,12 @@ number of agreeing retrievers, and whether a symbol matched exactly.
 - **low** → skill instructs Claude to **fall back** to `ripgrep`/Grep/Glob with suggested patterns
   emitted in `fallback_suggestions` (derived from query terms + detected symbols).
 
-## 8. Output payload (shared by Markdown + JSON)
+## 9. Output payload (shared by Markdown + JSON)
 
 ```jsonc
 {
   "query": "where is auth token refresh implemented",
   "intent": "locate_impl",
-  "index": { "exists": true, "stale": false, "built_at": "...", "head_commit": "abc1234" },
   "confidence": "high",
   "results": [
     {

docs/RETRIEVAL_PIPELINE.md

@@ -18,7 +18,7 @@ Intent detection (keyword / symbol / impact / general)
 │ 2. Path-based search                    │
 │ 3. SQLite FTS5 lexical search           │
 │ 4. Vector search (optional embeddings)  │
-│ 5. Graph expansion (from seed results)  │
+│ 5. Graph expansion (explicit opt-in)   │
 └─────────────────────────────────────────┘
     ↓
 Reciprocal Rank Fusion (RRF)
@@ -53,15 +53,16 @@ Ranked retrieval packet with confidence score
 
 ## 3. SQLite FTS5 Lexical Search
 
-**Trigger:** General keyword queries.
+**Trigger:** General keyword and natural-language queries.
 
 **Process:**
-- Build an FTS5 query from the user's text
-- Tokenize: split `snake_case`, expand `camelCase` at query time
-- Search the `fts_chunks` virtual table
-- Return matching chunks with BM25-style scores
+- Parse identifiers into camelCase/PascalCase/snake_case subtokens.
+- Add a small, explicit synonym/inflection vocabulary at lower weight.
+- Use OR groups for soft matching, then require bounded term coverage and rank
+  by original-term coverage plus BM25.
+- Quote every FTS term so query punctuation cannot inject MATCH operators.
 
-**Score:** Based on FTS5 rank — higher for more term matches and rarer terms.
+**Score:** Coverage is the primary signal; BM25 is a bounded tie-break.
 
 ## 4. Vector Search (Optional)
 
@@ -80,52 +81,74 @@ Ranked retrieval packet with confidence score
 
 ## 5. Graph Expansion
 
-**Trigger:** After initial results are found.
+**Trigger:** `RetrievalTuning(graph_source=True)` and an intent plan with a
+graph strategy. It is disabled in the shipped default because the reproducible
+self-repository ablation reduced direct-hit MRR.
 
 **Process:**
-- For each seed result, traverse the dependency/call graph
-- Find related files: callers, callees, imports, inheritors
-- Add related files with a decay factor (distance from seed)
+- Seed from lexical/symbol candidates already found in SQLite.
+- Traverse only indexed, resolved edges with bounded depth and node count.
+- Follow `up` (callers/importers), `down` (callees/imports), or `both`
+  according to the intent plan.
+- Apply distance decay and preserve edge confidence in the candidate reason.
 
-**Score:** Decreases with graph distance — direct connections score higher.
+Graph expansion is an opt-in context-discovery signal, not a replacement for
+direct lexical or symbol evidence.
 
-## Reciprocal Rank Fusion (RRF)
+## 6. Diversity and duplicate control
 
-Combines results from multiple retrievers:
+SimHash suppresses near-duplicate snippets independently of MMR. Bounded
+Maximal Marginal Relevance is available through `RetrievalTuning(mmr=True)`;
+the shipped default keeps relevance-only ordering because the benchmark favored
+direct hits.
+
+## 7. Reciprocal Rank Fusion (RRF)
+
+Combines ranked lists from the enabled retrievers:
 
 ```
-RRF_score(d) = Σ (1 / (k + rank_r(d)))
+RRF_score(d) = Σ w_r · k / (k + rank_r(d))
 ```
 
+The implementation multiplies textbook RRF by `k` so fusion and bounded rerank
+bonuses share a comparable scale; ordering is unchanged.
+
 Where:
 - `k` is a constant (default 60)
 - `rank_r(d)` is the rank of document `d` in retriever `r`
-- Sum is over all retrievers that returned `d`
+- `w_r` is the intent/tuning weight for retriever `r`
 
-This ensures documents that appear in multiple retrievers rank higher.
+The implementation merges co-located chunks into one per-file bucket before
+fusion, preventing a large file from dominating the result list.
 
-## Reranking
+## 8. Reranking
 
-After fusion, apply additional boosts:
+After fusion, apply bounded explainable boosts and penalties:
 
-| Factor | Boost | Rationale |
-|---|---|---|
-| Exact symbol match | +0.3 | User named a specific symbol |
-| File type relevance | +0.1 | `.ts` for TypeScript queries, etc. |
-| Recency | +0.05 | Recently modified files may be more relevant |
-| File size | -0.05 per 10KB | Prefer focused files over large ones |
+| Factor | Effect | Rationale |
+|---|---:|---|
+| Exact symbol match | +0.20 | User named a specific symbol |
+| Symbol definition kind | +0.05 | Prefer actionable definitions |
+| Path term match | +0.05 | User supplied a location clue |
+| Degree / reference evidence | up to +0.08 | Stable structural tiebreaker |
+| Implementation source prior | +0.08 | Prefer source over prose/tests |
+| Documentation source prior | -0.05 | Avoid docs displacing implementation |
+| Generated/vendor/build | -0.12 | Suppress low-value derived code |
+| Test path on non-test query | -0.06 | Keep tests as supporting evidence |
 
-## Confidence Score
+## 9. Confidence
 
-The final confidence score (0.0 to 1.0) determines how Claude should proceed:
+Confidence is categorical (`high`, `medium`, `low`) and is derived from
+exact-symbol evidence, multi-retriever agreement, score separation, and result
+count. Exact symbol matches are `high`, including a single-result response.
 
-| Confidence | Meaning | Action |
-|---|---|---|
-| 0.8 - 1.0 | High | Read recommended ranges and answer directly |
-| 0.5 - 0.8 | Medium | Read ranges; optionally confirm with one Grep |
-| 0.0 - 0.5 | Low | Use fallback suggestions (ripgrep, Glob) |
+| Confidence | Action |
+|---|---|
+| `high` | Read recommended ranges and answer directly |
+| `medium` | Read ranges; optionally confirm with one Grep |
+| `low` | Use fallback suggestions (ripgrep, Glob) |
 
-## Token Budget Enforcement
+## 10. Token Budget Enforcement
 
 The output is capped at a configurable token budget:
 
@@ -134,9 +157,9 @@ The output is capped at a configurable token budget:
 3. Remaining results are listed without snippets
 4. The `recommended_reads` field contains only the most critical line ranges
 
-Default budget: 2000 tokens (configurable in `.codeindex.json`).
+Default budget: 1500 tokens (configurable in `.codeindex.json`).
 
-## Fallback Suggestions
+## 11. Fallback Suggestions
 
 When confidence is low, the pipeline generates fallback strategies: