perf(httpapi): measure workspace search per phase, inside the handler

[dvcdsys]

Stage 1 of the three-stage search-perf plan that follows #263. It makes nothing
faster. It exists so stages 2 and 3 are not guesses.

Why a diagnostic gets its own PR

The last round budgeted the 10.5 s workspace query by measuring the dense scan,
the BM25 query and the fan-out's parallel speedup separately and multiplying
them together. The budget closed — which is not the same as being right. Two
optimisations were about to be built on it, and one of the multiplicands was off
by 18x.

What lands

• internal/httpapi/searchtimings.go — a searchPhases accumulator. The serial
  phases (embed, stale-FTS probe, fan-out wall, fuse) are plain durations; the
  two per-project phases keep a sum and a max behind atomics. The sum is the
  work the query did, the max is what the user waited for, and under sublinear
  parallelism those are different questions — eight concurrent project searches
  measured 3.4x faster than the same eight in sequence, so the wall time sits
  between the two numbers and reporting one alone hides which.
• The fan-out records dense and BM25 per project.
• One INFO line per workspace query, plus a timings object on the response,
  documented as WorkspaceSearchTimings in doc/openapi.yaml.
• projects_scanned / projects_returned, because their ratio is the premise
  of stage 3: the fan-out does full work on every project and then thresholds
  the answer down.

Always measured, conditionally reported

Collection is unconditional — it costs a handful of time.Now() calls and two
atomics against a query that reads gigabytes. Where the numbers go is gated
twice, and the gates answer different questions:

• The log line fires only above 2 s. The server already writes one
  http_request line per request carrying the wall time, so a second line on
  every workspace query would be noise added in order to catch the rare slow
  one. A threshold keeps the one property a ?debug flag cannot have: nobody
  needs to have switched anything on before the slow query happened. Two
  seconds is not "wrong" for a fan-out over every project in a workspace — it is
  the point past which the breakdown is worth storing, and low enough that a
  regression on a small workspace still trips it.
• The response object needs ?timings=true. In a response this is a
  debugging aid, not API surface. Every existing caller — CLI, MCP tools,
  dashboard — gets byte-identical responses to before.

One omission versus the plan's spec

No hydrate_ms. Chunk payloads are hydrated inside VectorStore.Search and
chunksfts.SearchProject, so hydration is not separable from the handler; it is
inside dense_sum_ms / bm25_sum_ms and addDense says so. Everything after
fuse is in-memory slicing. On the fixture, wall minus the four serial phases is
~19 ms of 9,911 ms, so nothing material is unaccounted for.

What it measured

45-repo fixture (1.9M chunks, voyage-code-3 @2048), 14-core Mac, int8 scan on,
10 queries against the 43-project workspace, via loadtests/bench/phases.py.

The measurement harness is not in the repository — /loadtests/ is
gitignored (.gitignore:98), fixture and scripts alike, because the fixture is
tens of GB and cost real embedding-provider tokens to build. The numbers below
are therefore not reproducible from a clone. They are reported here for the
decision they drive, not as a check anyone can re-run; what is reproducible
from this PR is the instrument that produced them.

phase	median	range
wall	9,911 ms	3,909 – 16,876
fan-out	9,663 ms	3,689 – 16,657
BM25, summed over projects	93,345 ms	15,026 – 183,177
BM25, slowest single project	3,210 ms	665 – 5,767
dense, summed over projects	26,476 ms	24,307 – 27,834
dense, slowest single project	2,428 ms	1,621 – 2,549
embed	218 ms	203 – 293
stale-FTS probe	11 ms	8 – 12
fuse	0 ms	0

Dense is a constant; BM25 is the variable and wall tracks it — 78% of the
fan-out's work at the median. Its cost follows the number and length of query
terms rather than repo size, because MATCH is evaluated over the whole
server's chunks_fts and filtered by project afterwards, once per repo.

Two estimates it disproved:

• a repo's BM25 measures 326–542 ms standalone (loadtests/bench/ftstest,
  through the server's own driver) but up to 5,767 ms inside the fan-out.
  43 concurrent FTS queries against one index degrade each other by roughly an
  order of magnitude. Stage 2 (one workspace-wide FTS query) removes 42/43 of
  that work and the contention — worth more than it looked, not less.
• the stale-FTS probe costs 11 ms through the server, not the 0.2 ms a
  Python-side measurement suggested.

Tests

Shape and gating, never wall-clock values — a bound on dense_sum_ms would fail
on a loaded runner and teach everyone to ignore the test. Asserted: every field
present when asked for; no timings block when not asked for, or on a
response that ran no search (zeroes would read as "instant"); the counters
matching the fan-out actually performed; max <= sum per phase, which catches a
sum and a max wired to the wrong accumulator; and both sides of the log
threshold from one captured logger, because a test that only proves silence
would still pass if the line were deleted outright. slowWorkspaceQuery is a
var rather than a const purely so that test can cross the threshold without
sleeping; nothing at runtime writes it.

Each gate was mutation-checked — removing the opt-in, removing the threshold,
and deleting the log line each fail the suite.

Verification

go test ./... green (46 packages), make openapi-check reports
openapi.gen.go in sync, go vet clean, gofmt clean on the touched files.
Rebased onto develop @ a82ef2d.

---

🤖 Generated with Claude Code

---

Review round (commit d511513)

F1 — projects_returned measured the panel cap, not the threshold. It came
from projectPayloads, already truncated to top_projects (default 10), so the
scanned:returned ratio read 43:10 on the fixture whether ten repos were relevant
or forty — and moved when a client passed a different top_projects. Since that
ratio is the stated premise of stage 3, the metric two optimisations were going
to rest on was measuring a request parameter. Now len(surviving); the panel
count keeps its own field, projects_in_panel.

F2 — early returns reported nothing at all. Both no-queryable-project paths
passed a literal nil and never entered the reporter, so the two gates
documented as independent were off together there. embed_ms is already paid by
that point, and a hung embedding provider is precisely what the slow-query line
is for. They now report with requested=false: nothing in the response, but a
slow one still writes its line.

F3 — wall_ms did not cover the whole handler, and the unnamed remainder
absorbed access.AccessibleProjectHostPaths — the one pre-fan-out step that
grows with the caller's project count rather than the workspace, and the one the
fixture cannot exercise, since admin and AuthDisabled skip the ACL branch.
started moved to the handler's first line and the resolution step got
resolve_ms. The spec now states what the remainder is instead of implying
there is none.

Documentation-only, from the same review: dense_sum_ms / bm25_sum_ms
include projects whose query failed — keeping them is correct, since the time
was spent and excluding it would put the sums permanently below the wall time
they explain, but a slow failure can own the max, so the spec says so; and the
comment on slowWorkspaceQuery now names the condition its test-only mutability
depends on (nothing here calls t.Parallel(); the first parallel test in this
package has to move the threshold onto Deps first).

Both new tests were mutation-checked against the bugs they describe. Restoring
the panel-capped counter fails the F1 test; restoring the nil early return
fails the F2 test. Honestly not covered: an unrecorded resolve_ms still
passes, because a shape test cannot tell an unset duration from a fast one.

---

What the corrected counter says (measured after d511513)

F1 was worth fixing for a concrete reason, now that the fixed counter has been
run against the fixture (branch build, two runs, counts identical in both since
they depend on scores rather than timing):

	scanned	survived threshold	panel	over-scan
median	43	33	10	1.3x
best case	43	23	10	1.9x
worst case	43	35	10	1.2x

The plan's stated premise for stage 3 — "the fan-out does full work on all 45
repos and then thresholds down to 10, so three quarters of the work is thrown
away" — was reading the top_projects cap. 77% of projects clear the
relevance threshold, so the real over-scan is 1.3x, and the candidacy
distribution across a workspace is flat rather than peaked: the condition under
which bound-based pruning prunes nothing.

That does not change this PR, and it does not change stage 2 (which removes
42/43 of the BM25 work by asking once instead of 43 times, whatever the
threshold later decides). It does mean stage 3 has to be re-scoped or dropped
before anyone builds it — which is exactly the outcome a diagnostic is supposed
to produce, and the reason this PR is separate from the optimisations it is
meant to steer.