perf(vectorstore): scan an int8 copy and rescore the shortlist exactly - #263
Merged
Conversation
Workspace search over the 45-repo load-test fixture took 23.9 s at the median
on a 14-core Mac with NVMe and 36 GB of RAM. Production is an e2-standard-2
(2 vCPU, 8 GB, network PD) behind a Cloudflare tunnel whose 100 s edge timeout
would fire before the answer.
The cause was not the fan-out logic, it was the volume. A 2048-dim float32
embedding is 8192 bytes, past SQLite's 8157-byte local-payload limit on our
8 KiB pages, so every `vectors` row spilled into its own overflow page.
Measured with dbstat on the fixture: 238,727 leaf + 1,909,447 overflow pages
for 1,909,447 vectors = 9,216 bytes read per vector, 17.6 GB per workspace
query. The schema comment still described the 768-dim case ("the scan reads two
rows per page and never follows an overflow chain"), which was true of every
model the store originally shipped with.
Fix: `vectors_q8`, the same vectors at one byte per component plus a per-vector
scale, is what a search now scans. It takes a shortlist, and the shortlist is
rescored against the float32 originals in `vectors`, which stay authoritative
and untouched. Measured on the same fixture: 636,483 leaf pages, zero overflow,
2,731 bytes per vector, 5.2 GB per workspace query — 3.4x less. The file grew
20 GB -> 26.35 GB (+31%).
Why rescoring rather than trusting int8. On 60k vectors of the fixture's
largest collection (ziglang/zig, voyage-code-3 @2048) against 50 REAL
query-side embeddings, recall of the exact float32 top-K:
shortlist k=10 k=20
20 0.998 0.994
40 0.998 0.999
60 1.000 1.000
200 1.000 1.000
int8 alone gives 0.994 at both k. The quantisation misorders near-ties, it does
not lose the documents, so re-reading a few dozen exact vectors recovers all of
them — hence a floor of 64 and 4x the limit above it. An earlier version of
this experiment drew its queries FROM the corpus; a corpus vector is an exact
member of the set being searched, its neighbours are far away, and it made
rescoring look worthless (0.990 either way). Query-side embeddings are the
regime that decides.
Latency, A/B on the same machine and the same warm page cache, back to back,
by flipping CIX_VECTOR_SCAN_QUANT (10 queries, one repeat each):
float32 scan int8 + rescore
single project 15,899 ms 1,422 ms p50
23,642 ms 2,093 ms p95
workspace (45) 23,879 ms 10,544 ms p50
33,366 ms 25,186 ms p95
The single-project scan improves more than the 3.4x byte reduction because
5.2 GB fits this machine's page cache and 17.6 GB does not. The workspace
number improves less, so something other than the dense scan now dominates the
fan-out — but what, exactly, is not established. BM25 is the obvious suspect
(`chunksfts.SearchProject` matches `chunks_fts` across the WHOLE server and
filters by project afterwards, once per repo), and measured through the
server's own driver it costs 326-542 ms per repo on this fixture, which does
not account for 10 s. The next step is per-phase timing inside the handler
rather than another guess.
A note for whoever measures next: the same BM25 query timed through Python's
system sqlite3 on this Mac takes 18 s, repeatably, against 380 ms through
modernc.org/sqlite. Any conclusion about FTS5 cost drawn with a different
SQLite build than the server's is worthless.
End-to-end check that the approximation is invisible: 20 queries, top-20, on
the full 346k-vector zig collection, captured with the compact scan off and
then on. 20/20 byte-identical, including the scores.
Mechanics:
- Scores returned to callers are always the exact cosine, never the int8
estimate. This is load-bearing beyond cosmetics: min_score thresholds on
it, the workspace fan-out min-max normalises across projects with it, and
hybrid search blends it with BM25 — an approximate score would move results
BETWEEN projects in a way no single-project test would catch.
- Candidates are keyed by doc_id, not rowid. `vectors` has a composite
PRIMARY KEY so its rowid is implicit, and SQLite only promises to preserve
implicit rowids across a VACUUM for tables with an INTEGER PRIMARY KEY.
Survivable while the rowid never leaves one query; fatal once a second
table keys off it.
- Readiness is a q8_state row, not a COUNT. Collections created by this code
are marked at creation (empty, therefore complete) and every upsert
maintains both tables in one transaction. Collections written by an older
binary have no flag, keep using the float32 scan, and are converted by a
background pass at open — largest first, 2000-row transactions, 50% duty
cycle, free-space checked up front. The flag is written in the same
transaction as the batch that proves it, so a kill leaves a collection
unmarked and still correct, never marked and incomplete. On the fixture the
backfill converted 1,909,447 vectors in 245 s.
- vectors_q8 carries `language` because that is the only filter any caller
produces (fetchVectorResults, from the `languages` parameter). Any other
`where` key falls back to the float32 scan, which has every column.
`{"language": ""}` is a filter, not the absence of one — chromem compared
metadata["language"] to "" — and the test for that fails against the
obvious `if language != ""` version.
- CIX_VECTOR_SCAN_QUANT=false opts out: the copy is a quarter of the float32
bytes on top of an already large store. Turning it off also withdraws the
completion flag from anything written while off, so turning it back on
rebuilds rather than trusting a stale copy.
- Deletes reach the copy in both directions, and delete-by-file runs BEFORE
the vectors delete because its subquery reads file_path from `vectors`. An
orphaned q8 row is a document the scan keeps shortlisting and the rescore
can no longer score: it vanishes from results with nothing logged.
- The legacy chromem import still writes float32 only; it creates its
collection with raw SQL so nothing marks it complete, and the backfill that
runs right after picks it up.
Also here, because it is the same query path and it was free: the stale-FTS
probe in workspace search used `SELECT COUNT(*) ... LIMIT 1` per repo, which
walks every matching index entry to answer a yes/no question (the LIMIT bounds
the result rows of an aggregate that always returns one). Measured on the
fixture's 1.95M-row chunks_meta across 46 projects: 53.2 ms as COUNT, 0.2 ms as
EXISTS. It runs serially, before the fan-out, on every workspace query.
Tests. The layout ones assert PAGES, not milliseconds, so they mean the same
thing in CI, on a laptop and on the production box — multiply by that machine's
read throughput and you have its latency. TestScanPackingEfficiency also fails
on the 1024-dim case as float32 (8192 bytes read to obtain 4096: one row per
leaf page, half of it air), which was reachable by an operator halving
output_dimension to save time and getting half the vector quality for 89% of
the I/O. The behaviour tests use a corpus with deliberate near-duplicate
clusters, because random unit vectors in 2048 dimensions are nearly orthogonal
and have no near-ties for a quantiser to confuse — real code corpora are the
opposite. TestScanQuantOffThenOn covers the toggle that would otherwise leave a
collection marked complete and missing every row written while it was off.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s, toggle staleness, exactness claims Addresses the review of 838c923. Findings 1-10 plus the minors; nothing was waved through, and two of them were only provable by writing a test that fails against the old code first. ## The backfill was racy, and the comment claiming otherwise was false (1) backfillBatch reads a batch in one implicit transaction, quantises it in Go, and writes it in another. It holds no lock across the gap — deliberately, since the thing most likely to want the write lock is the file watcher reindexing a file somebody just saved — and on the fixture that gap is open across 245 s of live server. Two things went wrong in it, neither of which surfaces as an error: - a doc deleted in the gap had its compact row REINSERTED, and nothing could ever remove it again: DeleteByFile finds doc_ids through `vectors`, where the row no longer is. Every later scan shortlists the orphan and every rescore drops it, so the query silently returns fewer results. - a doc re-embedded in the gap had its fresh compact row OVERWRITTEN by this batch's quantisation of the embedding it had just replaced. The document then ranks by a vector it no longer has. Both are now closed at the statement level rather than by locking: WHERE EXISTS (do not resurrect) and ON CONFLICT DO NOTHING (never be the later writer — the backfill only fills gaps). The empty-batch completion no longer claims to be "in the same transaction as the query that proved it", which was literally untrue; completeness rests on those two clauses, and the comment now says so. TestBackfillSurvivesConcurrentWrites runs the backfill against a churning collection and asserts invariants rather than an interleaving, so it can only fail for a real reason. TestBackfillNeverResurrectsOrOverwrites pins the two clauses deterministically. Both fail against a plain upsert — verified by reverting the SQL and re-running. ## One failed collection stopped the other forty-two (2) backfillQ8 returned on the first per-collection error. The realistic cause is a collection deleted mid-walk (admin project delete, orphan sweep) failing the next insert's foreign key — after which every remaining collection stayed on the float32 scan until somebody restarted the server, with one warn line as the only trace. Now logged per collection and skipped; the completion line reports how many failed. ## A legacy import could hide documents behind a live flag (3) The import writes `vectors` directly and creates its collection with INSERT OR IGNORE, which was justified as "nothing marks it complete". True only when the collection is NEW. An operator who indexed a project live (ensureCollection flags it at creation) and then pointed CIX_CHROMA_PERSIST_DIR at a legacy tree reaches the other case — migration_state is keyed on the legacy collection name and has never seen it — and the imported docs get no compact rows inside a collection whose flag says it is complete. The backfill skips flagged collections, so those documents would never be searchable on the fast path. The import now withdraws the flag unconditionally. ## Indexing must not fail over a performance hint (4) ensureCollection returned the error from markCollectionQ8Ready, which opens its own transaction and can lose a race for the write lock. That aborted a whole UpsertChunks over a row whose absence costs nothing but a slower scan — and which self-heals, because the backfill sets it at the next open. Logged now. ## The two scan paths disagreed on a zero query (5) quantizeInt8 returns scale 0 for an all-zero vector and scanQ8 short-circuited to empty, while the float32 path scores every row 0 and fills the heap. Same broken query, different answers depending on whether the collection had been converted — and in a workspace fan-out, both at once. The short-circuit is gone: a zero query now scores everything 0 on both paths, which is what quantizeInt8's own comment always claimed. ## "Exact" was overclaiming (6) The docs said results and scores "stay exact". Scores do, by construction — the rescore computes them from the float32 vectors. The SET does not, in the worst case: the shortlist is a fixed width and topK rejects boundary ties strictly, so a collection holding more than `shortlist` documents within one quantisation step of each other truncates the tie in scan order, and the rescore cannot recover a document it never received. Measured error is zero on every corpus tried; that is now what the documentation says, with the boundary case named in q8Shortlist next to the fix that would close it. ## Performance and duplication (7-10) - doc_id was scanned into a fresh Go string on every row of both scan loops — ~1.9M allocations per workspace query on the fixture, and the CIX_VECTOR_SCAN_QUANT=false path paid it too, so the opt-out did not actually restore pre-change behaviour. Both loops now read it as RawBytes and materialise the string only for a row that enters the heap. - The backfill paginated `vectors` in doc_id order. `vectors` is a rowid table whose composite primary key is a separate index, so that scattered ~9 kB row lookups across the collection's whole rowid span — the same 1.8x that made scanSQL pick idx_vec_coll in the first place. Now keyset-paginated on rowid through idx_vec_coll. - rescore duplicated scan's streaming loop line for line, putting the float32 decode protocol in three places; both now call streamExact. The IN-list batching duplicated hydrate's; both now call docIDInList. The scan-slot select was pasted twice; acquireScanSlot. - Options.ScanQuant's comment described a design that was not built ("writes maintain it either way"). Rewritten to match: the flag governs the whole lifecycle. ## Minors q8_state's comment claimed a dimension column that does not exist. The row-size expression was pasted three times; sizeExprQ8. q8Ready cached negatives it never used; positives only, presence is the answer. clearQ8Ready ran per 500-chunk batch; hoisted to once per UpsertChunks, and moved BEFORE the first write so a crash mid-run cannot leave the flag set over a half-written collection. Writes with the copy switched off now DELETE the compact rows they touch — without that, a doc re-embedded while off kept its stale compact row, and the backfill's new DO NOTHING would have sealed it in on re-enable. Filter fallback to the exact scan logs at debug, and TestQ8FilterableCoversEveryFilter fails when a new filter column appears without a decision about it. layout_test claimed scanTable was derived from the SQL; it now is, via an assertion. Not done, with reasons: quantizeInt8's per-component division stays a division — precomputing the reciprocal changes stored values for a loop that runs once per chunk at index time, and the end-to-end identity check below is worth more than the microseconds. The maintenance q8 aggregate still walks leaf pages; it sits behind the maintenance service's TTL cache, and the cheap fix if that changes is recording the total at completion, noted at the constant. Verified on the 45-repo fixture, not only in unit tests: one collection's compact copy was wiped, the server rebuilt it (54,063 vectors, 8 s, zero orphans in the whole database), and 20 queries x top-20 came back identical both to the pre-review compact scan and to the exact float32 scan — 20/20, scores included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Dense search scans an int8 copy of every vector and rescores the shortlist
against the float32 originals. Every score returned stays the exact cosine, the
result sets measured identical, and the scan reads 3.4x fewer bytes.
Why
A 2048-dim float32 embedding is 8192 bytes, past SQLite's 8157-byte
local-payload limit on our 8 KiB pages, so every
vectorsrow spilled intoits own overflow page. Measured with
dbstaton the load-testfixture (45 cloned repos, 43 of them in the workspace, 1,909,447 vectors):
238,727 leaf + 1,909,447 overflow pages for 1,909,447 vectors =
9,216 bytes read per vector, 17.6 GB per workspace query.
The schema comment still described the 768-dim case ("the scan reads two rows
per page and never follows an overflow chain") — true of every model the store
originally shipped with, false for every model an operator would pick today.
Numbers
All measured on the load-test fixture — 1,909,447 chunks over 39 collections,
43 of the projects linked into one workspace, voyage-code-3 @2048 — on a
14-core Mac with NVMe. Latency is an A/B on the same machine and the same warm
page cache, back to back, by flipping
CIX_VECTOR_SCAN_QUANT— 10 queries,one repeat each.
Layout (
dbstat, real fixture)vectors(float32)vectors_q8(int8)The database file grew 20 GB -> 26.35 GB (+31%).
Latency
The single-project scan improves more than the 3.4x byte reduction because
5.2 GB fits this machine's page cache and 17.6 GB does not.
Quality
Recall of the exact float32 top-K, measured on 60k vectors of the fixture's
largest collection (
ziglang/zig) against 50 real query-side embeddings:int8 alone gives 0.994 at both k — quantisation misorders near-ties, it does
not lose the documents, so re-reading a few dozen exact vectors recovered every
one. Hence a shortlist floor of 64 and 4x the limit above it.
Two guarantees, deliberately kept apart in the docs and the code comments.
Scores are exact by construction — every number a caller sees comes from
the float32 vector. The result set is an approximation measured at zero
error and not proved at zero: the shortlist is a fixed width and
topKrejectsboundary ties strictly, so a collection holding more than
shortlistdocumentswithin one quantisation step of each other could truncate a tie in scan order.
Named in
q8Shortlistnext to the change that would close it.An earlier version of this experiment drew its queries from the corpus. A
corpus vector is an exact member of the set being searched and its neighbours
are far away, which made rescoring look worthless (0.990 either way). Query-side
embeddings are the regime that decides.
End-to-end: 20 queries x top-20 against the full 346k-vector zig collection,
captured with the compact scan off and then on — 20/20 byte-identical,
scores included. Re-run after the review round with the same result.
Design notes for the reviewer
Scores are always the exact cosine, never the int8 estimate. Load-bearing
beyond cosmetics:
min_scorethresholds on it, the workspace fan-out min-maxnormalises across projects with it, and hybrid search blends it with BM25.
An approximate score would move results between projects in a way no
single-project test would catch.
TestSearchScoresAreExactpins it.Candidates are keyed by
doc_id, not rowid.vectorshas a compositePRIMARY KEY so its rowid is implicit, and SQLite only promises to preserve
implicit rowids across a
VACUUMfor tables with an INTEGER PRIMARY KEY.Survivable while a rowid never leaves one query; fatal once a second table
keys off it.
Readiness is a
q8_staterow, not aCOUNT. Collections created by thiscode are marked at creation (empty, therefore complete — best-effort: a
failure there is logged, not returned, and the backfill covers it) and every
upsert maintains both tables in one transaction. Collections written by an
older binary have no flag, keep using the float32 scan, and are converted by
a background pass at open — largest first, 2000-row transactions, 50% duty
cycle, free space checked up front, keyset-paginated on rowid through
idx_vec_coll.What makes a half-finished conversion safe is not that the flag shares a
transaction with the data — it does not, and the first version of this PR
claimed otherwise. It is that the backfill's insert cannot resurrect a
deleted row (
WHERE EXISTS) and cannot overwrite a newer one(
ON CONFLICT DO NOTHING), so any interleaving of deletes and upserts leavesthe copy consistent, and the flag only ever follows a cursor that ran off the
end. See the review-round section below.
Timings: the first full build converted 1,909,447 vectors in 245 s (before
the rowid-pagination change); after it, one 54,063-vector collection rebuilt
in 8 s.
vectors_q8carrieslanguagebecause that is the only filter any callerproduces (
fetchVectorResults, from thelanguagesparameter). Any otherwherekey falls back to the float32 scan, which has every column.{"language": ""}is a filter, not the absence of one — chromem comparedmetadata["language"]to""— and the test for that fails against theobvious
if language != ""version.CIX_VECTOR_SCAN_QUANT=falseopts out, across the whole lifecycle ratherthan just reads: the backfill does not run, and writes withdraw the
collection's completion flag and DELETE the compact rows of the docs they
touch. Both halves are needed — leaving a stale row behind would be sealed in
by the backfill's
DO NOTHINGon re-enable, and the document would then bescored with a vector it no longer has (
TestScanQuantOffThenOn).Deletes reach the copy in both directions, and delete-by-file runs
before the vectors delete because its subquery reads
file_pathfromvectors. An orphaned q8 row is a document the scan keeps shortlisting andthe rescore can no longer score: it vanishes from results with nothing logged.
Also here
The stale-FTS probe in workspace search used
SELECT COUNT(*) ... LIMIT 1perrepo — which walks every matching index entry to answer a yes/no question (the
LIMITbounds the result rows of an aggregate that always returns exactly one).It runs serially, before the fan-out, on every workspace query.
Numbers, with their provenance, because the two available measurements do not
agree and only one of them is the server's. Through Python's sqlite3 on the
fixture's 1.95M-row
chunks_metaacross 46 projects: 53.2 ms as COUNT, 0.2 msas EXISTS — a 266x ratio, which is a query-plan property and is why the change
was made. Through the server itself, with the EXISTS form in place, the probe
costs ~11 ms per workspace query. The pre-fix cost was never measured
through the server, so this PR does not claim a saving in milliseconds; what it
claims is that the plan stops being O(rows) per repo. See the note at the end
about which SQLite build a measurement came from.
Tests
The layout tests assert pages, not milliseconds, so they mean the same thing
in CI, on a laptop and on the production box — multiply by that machine's read
throughput and you have its latency.
TestScanPackingEfficiencyalso documents the case that was worst and leastobvious: at 1024 dimensions nothing overflows and the float32 scan still
read 8192 bytes to obtain 4096, because two 4.1 kB rows cannot share an 8 KiB
page. An operator halving
output_dimensionto save time got half the vectorquality for 89% of the I/O.
The behaviour tests build a corpus with deliberate near-duplicate clusters:
random unit vectors in 2048 dimensions are nearly orthogonal and have no
near-ties for a quantiser to confuse, while real code corpora are full of them.
Not in this PR
Workspace search is still around 10 s, and it is no longer the dense scan.
That has since been measured from inside the handler on a follow-up branch
(not part of this PR), across the ten benchmark queries:
Dense is a constant: the same work for every query, parallelised down to
~2.4 s for the slowest single project. BM25 is the variable, and wall tracks
it —
chunksfts.SearchProjectevaluatesMATCHover the whole server'schunks_ftsand filters by project afterwards, once per repo, so its costfollows the number and length of query terms rather than repo size. At the
median it is 78% of the fan-out's work. Separately, the fan-out searched
43 projects and returned 10.
Both of those are the next two changes, and neither belongs here.
A note for whoever measures next: the same BM25 query timed through Python's
system sqlite3 on this Mac takes 18 s, repeatably, against 380 ms
through
modernc.org/sqlite. Any conclusion about FTS5 cost drawn with adifferent SQLite build than the server's is worthless.
Review round (commit
5bafefe)Findings 1-10 and the minors from the review of
838c923. The substantiveones:
reads a batch, quantises in Go, writes in a separate transaction, and holds
no lock across the gap. A delete committing in that gap had its compact rows
resurrected as orphans nothing could ever remove (DeleteByFile finds doc_ids
through
vectors); an upsert committing in it had its fresh compact rowoverwritten with the stale quantisation. Closed at the statement level —
WHERE EXISTSandON CONFLICT DO NOTHING— rather than by taking a lockthe file watcher would then queue behind. Two new tests, both verified to
fail against a plain upsert.
imported document from the fast path forever. The import now withdraws the
flag.
limitscore-0 rows from the exact one — the same broken query answered two ways
inside one workspace fan-out. The short-circuit is gone.
doc_idwas allocated per row in both scan loops (~1.9M strings perworkspace query), including on the
CIX_VECTOR_SCAN_QUANT=falsepath, sothe opt-out did not actually restore prior behaviour. RawBytes now, string
only for rows that enter the heap.
doc_idorder, scattering ~9 kB row lookupsacross the collection's rowid span. Same cost model that made
scanSQLpickidx_vec_collin the first place (measured 1.8x there; the backfill's ownbefore/after was not measured separately). Keyset-paginated on rowid now.
rescoreduplicatedscan's streaming loop andhydrate's IN-listbatching; both extracted.
Options.ScanQuant's comment described a designthat was not built.
Verified on the fixture, not only in unit tests: one collection's compact copy
was wiped, the server rebuilt it (54,063 vectors, 8 s, zero orphans in the
whole database), and 20 queries x top-20 came back identical both to the
pre-review compact scan and to the exact float32 scan.
Two findings were declined with reasons in the commit message: precomputing
quantizeInt8's reciprocal (changes stored values for a loop that runs onceper chunk at index time) and making the maintenance q8 aggregate cheaper (it
sits behind the maintenance service's TTL cache).
🤖 Generated with Claude Code