perf(chunksfts): rank the whole workspace in one FTS5 query

[dvcdsys]

Stage 2 of the search-perf plan, and the change stage 1's timings pointed at.
Follows #263 (int8 scan) and #264 (per-phase timings).

Why BM25 was the whole problem

FTS5 drives the query. It evaluates MATCH over the whole chunks_fts
table — every project on the server — joins each hit to chunks_meta, and only
then discards the rows belonging to other projects. A per-project BM25 query
therefore costs about the same whichever project it names, and a 43-project
workspace paid for the same global match 43 times.

On top of that the 43 queries contended over one index: a repo's BM25 measured
326–542 ms standalone but up to 7,000 ms inside the fan-out.

What changed

chunksfts.SearchProjects replaces the N queries with one statement that ranks
within each project and keeps each project's top rows:

ROW_NUMBER() OVER (PARTITION BY project_path ORDER BY bm ASC, rowid ASC)

The IN list is batched at 500 paths to stay under SQLite's 999-variable
ceiling — the same batch size the vector store already uses. In the handler the
query runs inside the fan-out's errgroup alongside the dense scans rather than
before them, so nothing serialises. Fusion moves out of the per-project
goroutines because it now needs both sides; it was under a millisecond across
the whole fan-out.

bm25_sum_ms and bm25_max_ms collapse into bm25_ms. The split existed to
separate "work done" from "waited for" across N queries; with one query they are
the same number, and keeping both would imply a fan-out that no longer happens.

Measured

45-repo fixture (1.9M chunks, voyage-code-3 @2048), 43-project workspace,
10 queries, medians. Both builds run back to back against the same already-warm
page cache — a process restart does not evict it — with one warm-up pass
discarded each time. This matters: a first comparison across a cold restart
credited the change with twice the improvement it earned.

phase	develop	stage 2
wall	10,235 ms	3,048 ms	3.4x
BM25	111,210 ms summed over 43	2,782 ms in one query	40x
dense sum	15,166 ms	11,649 ms	1.3x
dense max	2,318 ms	1,138 ms	2.0x

The dense rows are worth pausing on: nothing in the dense path changed.
Removing 43 concurrent FTS queries gave the vector scans back the CPU and I/O
they were contending for — 1.4x on the work, 2.1x on the project anyone actually
waits for.

BM25 is no longer the dominant term — it is 19% of the fan-out's work at the
median, down from 78–80%. Dense is now the thing to look at.

Correctness on the fixture

50 queries, full workspace response captured per query (panel with both signals,
chunk list in order), develop vs this branch:

• BM25 signal identical in all 500 panel rows. Zero differences.
• Project panel order identical for all 50 queries.
• Dense scores wobble by ≤0.0015 in a few percent of rows, and the chunk order
  shifts where that changes an RRF rank.

That last one is not this change. Running the same binary twice and
diffing it against itself produces at least as much wobble (30 differing rows
vs 38 across the build change). Its cause is not established and I did not chase
it: single-project search repeats bit-identically across three runs including
scores, and three consecutive identical workspace queries repeat
bit-identically, so it correlates with machine load rather than with the query
or the build. Worth its own look; it predates this PR.

Risks the plan flagged, and where they landed

• BM25 scores are corpus-global either way — verified rather than assumed:
  identical in all 500 panel rows.
• Match-set size. The window materialises the global match before trimming.
  Not a problem at this corpus size; no outer bound added, so this is the thing
  to watch on a much larger server.
• Bound-parameter count. Batched at 500, with a test that spans the boundary.
• A workspace of one must not get slower. No special case: at one project
  the window degenerates to a sort over that project's hits. If a single-project
  regression shows up, SearchProject is still there to fall back to.

The blast radius grew, and the tests say so

BM25 used to fail per project. Now one failing query costs every project its
sparse signal at once. The fallback is the one a pre-FTS install already lives
with — dense-only results, no failed_repos, no 500 — and
TestWorkspaceSearch_SurvivesBM25Failure drops chunks_fts outright to prove
it.

Tests

Each mutation-checked against the bug it describes:

• SearchProjects matches SearchProject per project — same hits, same order,
  same scores — over 4 queries x 2 limits x 4 projects on a deliberately
  tie-heavy corpus;
• the IN list does not prefix-match (project paths routinely share prefixes:
  local:host:/x vs local:host:/x/y);
• the map survives the batch boundary (searchProjectsBatch + 7 projects);
• BM25 hits stay in their own project end-to-end through the handler;
• a total BM25 failure still returns dense results.

One honest gap: the (bm, rowid) tiebreak is defensive, not test-provable.
Ties are the norm in a trigram index — 14 of 16 hits in the package's own corpus
share a score — but SQLite currently returns tied rows in rowid order for both
the LIMIT and the window form, so dropping the explicit tiebreak still passes.
Naming it makes the agreement a property of the queries rather than a
coincidence a future planner is free to break.

Verification

go test ./... green (46 packages), go test -race green on both changed
packages, make openapi-check in sync, go vet and gofmt clean.

---

🤖 Generated with Claude Code

---

Second commit: a regression the tests could not see

Found by clicking the fixture dashboard by hand. google authentication login form took 18.6 s against a 4.6 s median, and stage 1's slow-query log
attributed all of it to the single BM25 statement.

The partitioning was not the problem — what the CTE carried was. Selecting
file_path, content and the rest inside hits makes SQLite materialise all of
it for every matched row before ROW_NUMBER trims. The match set is the
whole server's index, because that is how FTS5 evaluates MATCH: that query
matched 263,515 rows in order to return 2,300.

Timed with loadtests/bench/ftsab through the server's own driver, serially, so
the old shape's cost is not hidden by the fan-out's concurrency:

query	match set	46 queries	payload in CTE	rank first
google authentication login form	263,515	26,085 ms	15,070 ms	2,798 ms
parse JWT token and validate signature	623,913	27,587 ms	29,566 ms	5,492 ms
rate limiter middleware	166,347	16,904 ms	7,692 ms	993 ms
websocket upgrade handshake	21,049	3,505 ms	1,011 ms	156 ms
graceful shutdown on SIGTERM	14,987	3,244 ms	854 ms	424 ms

The JWT row is the one that matters: at a 624k-row match set the first commit
was slower than the 46 queries it replaced. The gain scaled inversely with the
match set — exactly backwards — and the ten-query bench set hid it because the
old shape runs concurrently in the fan-out while these numbers are serial.

The fix ranks on (project_path, rowid, bm) alone and joins chunks_meta and
chunks_fts back for the survivors: one rowid round-trip per returned row,
nothing per discarded row. The query that started this went 18,623 ms → 2,711 ms.

No test guards this. It is a property of the query plan, not of the result —
the equivalence tests pass against both forms, which is precisely the point:
correctness tests cannot see this class of bug. What guards it is
loadtests/bench/ftsab, which times all three shapes against the real corpus,
and a comment on the query explaining why the obvious form is wrong.

It also revises the risk I listed above as "not a problem at this corpus size;
no outer bound added". Match-set size was the problem, and it is now paid for
per returned row rather than per matched row.

Third commit: review findings

N1 — the guard I named was not in the repo. The previous commit closed with
"what guards it is loadtests/bench/ftsab", and /loadtests/ is gitignored, so
that guard existed on one machine. It is now a test that asserts the query
plan: FTS5 reports a rowid lookup as 0:= and a MATCH scan as 0:M…; the
rank-first form does both, the payload-in-CTE form only scans.
workspaceRankQuery is extracted so the test builds the string production
builds rather than a copy free to drift.

Its own mutation check lives in the tree rather than in my shell history:
TestExplainDistinguishesTheTwoQueryShapes builds the slow form and asserts the
plan assertion rejects it — otherwise a change in how SQLite reports plans
could quietly turn the guard into a tautology. Confirmed by mutation: moving the
payload back into the CTE fails the plan test while the equivalence tests
still pass, which is the review's point made executable.

F2 — bm25_ms had no invariant after sum/max collapsed. Added
bm25_ms <= fanout_ms, and the comment states what it does not catch: a
dropped assignment, since 0 <= fanout holds. > 0 isn't available because an
in-memory corpus rounds to 0 ms, and a flaky guard is worse than an honest
partial one.

F3 — both paths now share scanRankedHit. This mattered more than it looks:
the equivalence test compares the two paths against each other, so a mistake
made symmetrically in both would have passed.

F4 — the mutex around bm25ByProject guarded nothing (single writer, readers
after g.Wait()); dropped, with a comment, because a lock that protects nothing
reads like protection to whoever next needs those hits inside the fan-out.

F5 — denseHits[i] released as the fusion loop consumes it.

F7 — placeholders(0) returned "NULL", which matches nothing and is
indistinguishable from "nothing matched". Removed; IN () is a syntax error,
which is loud and points at the wrong caller.

F9 — stale searchPhases header.

F6/F8 — not done, deliberately

Collapsing SearchProject into SearchProjects([]string{p}) would make
TestSearchProjects_MatchesPerProjectQueries compare a function to itself, and
that test is the only independent check that the partitioned ranking matches the
known-good per-project one. The per-project BM25 signal feeds project candidacy,
so a divergence re-ranks the projects panel with no error and no failed_repos.
Structural agreement is worth less here than an oracle. SearchProject's doc
comment now says it has no production caller, why it is kept, and not to delete
it as unused — which is the real fix for F8.