perf(vectorstore): scan an int8 copy and rescore the shortlist exactly

doc/CONFIG_REFERENCE.md

@@ -37,6 +37,7 @@ the DB.
 | `CIX_CHROMA_PERSIST_DIR` | `/data/chroma` | Legacy chromem-go store. Read on startup for the one-time import into the SQLite vector store, then left untouched as the rollback path. See [VECTORSTORE.md](VECTORSTORE.md). |
 | `CIX_VECTORS_DIR` | sibling of `CIX_CHROMA_PERSIST_DIR` (`/data/vectors`) | Vector store directory: one SQLite database per embedding namespace. |
 | `CIX_VECTOR_MMAP_SIZE` | `0` (off) | `PRAGMA mmap_size` for the vector store, in bytes. Roughly 40% lower search latency in exchange for resident memory — mapped database pages count in RSS. |
+| `CIX_VECTOR_SCAN_QUANT` | `true` | Scan a compact int8 copy of each vector instead of the float32 original, rescoring the shortlist against the originals. Every score returned is the exact cosine; which documents reach the shortlist is an approximation, measured at recall 1.000 against exact search (see `doc/VECTORSTORE.md`). 3.4x fewer bytes read per query at 2048 dimensions, in exchange for roughly a quarter more disk. Set `false` if the volume cannot take it; existing copies are then ignored, and writes remove the copies of rows they touch so re-enabling rebuilds instead of trusting stale data. |
 | `CIX_GGUF_CACHE_DIR` | `/data/models` | Where downloaded GGUF files live. |
 | `CIX_PUBLIC_URL` | — | Externally-reachable URL used to build GitHub webhook delivery URLs. Empty disables webhook URL display. |
 

doc/VECTORSTORE.md

@@ -150,21 +150,96 @@ on disk, deliberately: it keeps the package self-contained and
 live in `vectors`. A multi-kilobyte `TEXT` column pushes a row past SQLite's
 local-payload limit, and SQLite then keeps only ~1 kB of the row in the table
 page and spills the rest — *including the embedding* — into an overflow chain,
-roughly doubling the pages a scan touches. Kept apart, a `vectors` row is
-~3.2 kB and two of them share an 8 KiB page. Content is read only for the K
+roughly doubling the pages a scan touches. Kept apart, a 768-dim `vectors` row
+is ~3.2 kB and two of them share an 8 KiB page. Content is read only for the K
 winners of a search: one extra lookup per result.
 
+**Why the scan reads a second copy of every vector.** The paragraph above stops
+being true once the model is bigger than 1024 dimensions. A 2048-dim float32
+embedding is 8192 bytes on its own, past the 8157-byte local-payload limit, so
+every `vectors` row spills into an overflow page and the scan is back to the
+layout splitting out the content was meant to avoid. Measured with `dbstat`
+over 400 rows, bytes a full scan must read per vector:
+
+| dimensions | representation | leaf | overflow | bytes/vector |
+|---|---|---|---|---|
+| 768 | float32 | 200 | 0 | 4096 |
+| 1024 | float32 | 400 | 0 | 8192 |
+| 2048 | float32 | 50 | 400 | 9216 |
+| 768 | int8 | 40 | 0 | 819 |
+| 1024 | int8 | 58 | 0 | 1188 |
+| 2048 | int8 | 134 | 0 | 2744 |
+
+The pathological line is 1024, not 2048: nothing overflows there and the scan
+still reads 8192 bytes to obtain 4096, because two 4.1 kB rows cannot share an
+8 KiB page. Halving `output_dimension` to save time bought half the vector
+quality for 89% of the I/O.
+
+`vectors_q8` removes the whole step function by scanning one byte per component
+instead of four. `TestScanPackingEfficiency` and `TestScanBytesPerVectorBudget`
+assert those numbers — as pages, not milliseconds, so they mean the same thing
+in CI, on a laptop, and on the production box.
+
 ## Search
 
 ```
-SELECT rowid, embedding FROM vectors INDEXED BY idx_vec_coll
- WHERE collection_id = ?  [AND <metadata filters>]
+SELECT doc_id, scale, embedding FROM vectors_q8 INDEXED BY idx_q8_coll
+ WHERE collection_id = ?  [AND language = ?]
 ```
 
-Rows stream past a dot product (embeddings are stored L2-normalised, so cosine
-similarity *is* the dot product) into a top-K min-heap that rejects a losing
-row with one comparison. Metadata and chunk text are fetched afterwards, for
-the winners only.
+Rows stream past an integer dot product into a top-K min-heap that rejects a
+losing row with one comparison. The heap is wider than the caller's limit — the
+int8 ranking chooses a shortlist, it does not produce the answer. The shortlist
+is then rescored against the exact float32 vectors in `vectors`, and metadata
+and chunk text are fetched for the winners only.
+
+**What the approximation costs.** Measured on 60k vectors of the load-test
+fixture's largest collection (`ziglang/zig`, voyage-code-3 @2048) against 50
+real query-side embeddings, recall of the exact float32 top-K:
+
+| shortlist | k=10 | k=20 |
+|---|---|---|
+| 20 | 0.998 | 0.994 |
+| 40 | 0.998 | 0.999 |
+| **60** | **1.000** | **1.000** |
+| 200 | 1.000 | 1.000 |
+
+Without rescoring at all the int8 ranking alone gives 0.994 at both k — the
+quantisation misorders near-ties, it does not lose the documents, which is why
+re-reading a few dozen exact vectors recovered every one of them here.
+`q8Shortlist` therefore uses a floor of 64 and 4x the limit above it. Scan CPU
+in the same run: 127 ms per query float32, 42 ms int8 (3.0x).
+
+Two different guarantees, worth keeping apart. **Scores are exact by
+construction**: every number a caller sees is the cosine against the float32
+vector, computed by the rescore. **The result set is an approximation** whose
+error was measured at zero on this corpus and is not proved at zero in general
+— the shortlist is a fixed size and `topK` rejects boundary ties strictly, so a
+collection holding more than `shortlist` documents inside one quantisation step
+of each other (a file vendored a hundred times, say) can truncate a tie in scan
+order, and the rescore cannot recover a document that was never shortlisted.
+Widening the shortlist to swallow boundary ties would close that; it has not
+been needed on any corpus measured so far.
+
+Scores returned to callers are always the exact cosine, never the int8
+estimate. That is load-bearing beyond cosmetics: `min_score` thresholds on it,
+the workspace fan-out normalises across projects with it, and hybrid search
+blends it with BM25 — an approximate score would move results between projects
+in a way no single-project test would catch. `TestSearchScoresAreExact` pins it.
+
+**Building and rebuilding the copy.** Writes maintain `vectors_q8` in the same
+transaction as `vectors`, so a collection created by this code is complete by
+construction, and `q8_state` records that at creation — the readiness check is
+a primary-key lookup, never a `COUNT`. A store written before the table existed
+is converted by a background pass at open, largest collection first, in 2000-row
+transactions at a 50% duty cycle; until a collection is covered its searches
+take the float32 scan, which is correct and simply slower. Nothing is ever
+marked complete before it is: the flag is written in the same transaction as
+the batch that proves it. Set `CIX_VECTOR_SCAN_QUANT=false` to opt out — the
+copy is roughly a quarter of the float32 bytes on top of an already large
+store, so an operator short of disk needs a way to say no. Turning it off also
+withdraws the completion flag from anything written while it is off, so turning
+it back on rebuilds rather than trusting a stale copy.
 
 `INDEXED BY` is not an optimisation hint, it is a guarantee, and *which* index
 matters. Measured on the real index, scanning its largest (74k-row) collection:
@@ -188,6 +263,10 @@ The metadata filter (`where`) mirrors chromem's semantics exactly, including
 the two odd cases: an unknown key with a non-empty value matches nothing, and
 an unknown key with an empty value matches everything.
 
+`TestQ8ScanUsesCollectionIndex` pins the same guarantee for the compact table:
+`idx_q8_coll`'s keys are `(collection_id, rowid)` for the same reason, and the
+language filter must not change the driving index.
+
 **Concurrency.** One scan per query, and a process-wide semaphore caps
 concurrent scans at `NumCPU`. Splitting a single query across workers was
 measured to buy nothing in the low-memory configuration (109 ms at 1 worker vs

server/cmd/cix-server/main.go

@@ -411,6 +411,7 @@ func run() (restart bool, err error) {
 			Dir:             cfg.VectorDirFor(comps),
 			LegacyChromaDir: cfg.ChromaDirFor(comps),
 			MMapBytes:       cfg.VectorMMapSize,
+			ScanQuant:       cfg.VectorScanQuantEnabled,
 			Logger:          logger,
 		})
 	}

server/internal/config/config.go

@@ -55,7 +55,19 @@ type Config struct {
 	// 0 (the default) leaves it off. Env: CIX_VECTOR_MMAP_SIZE. It buys
 	// roughly 40% lower search latency and costs resident memory: every
 	// connection maps the database file and mapped pages count in RSS.
-	VectorMMapSize          int64
+	VectorMMapSize int64
+	// VectorScanQuantEnabled controls the compact int8 copy the vector store
+	// scans instead of the float32 originals. Env: CIX_VECTOR_SCAN_QUANT,
+	// default true.
+	//
+	// It exists because turning it on costs disk before it saves time: the
+	// copy is about a quarter of the float32 bytes, added to a store that may
+	// already be the largest thing on the volume, and it is built by a
+	// background pass over every existing vector. An operator who is short of
+	// disk, or who wants to isolate a search-quality question from the
+	// approximation, needs a way to say no. Off means every collection keeps
+	// using the exact float32 scan — correct, and as slow as it was before.
+	VectorScanQuantEnabled  bool
 	SQLitePath              string
 	MaxFileSize             int
 	ExcludedDirs            []string
@@ -310,6 +322,12 @@ func Load() (*Config, error) {
 	}
 	c.VectorMMapSize = int64(vecMMap)
 
+	scanQuant, err := getenvBool("CIX_VECTOR_SCAN_QUANT", true)
+	if err != nil {
+		return nil, err
+	}
+	c.VectorScanQuantEnabled = scanQuant
+
 	authOff, err := getenvBool("CIX_AUTH_DISABLED", false)
 	if err != nil {
 		return nil, err

server/internal/httpapi/workspacesearch.go

@@ -509,26 +509,34 @@ func workspaceSearchResponse(
 // reindex before BM25 can contribute. A best-effort detector: if any
 // SQL probe errors out we log + return nil rather than fail the
 // request, since the warning is informational, not load-bearing.
+//
+// EXISTS, not COUNT(*), and the difference is not cosmetic: the question is
+// "are there any rows", and COUNT walks every matching index entry to answer
+// it. Measured on the 45-repo load-test index (1.95M rows in chunks_meta),
+// this loop cost 53 ms per workspace search as COUNT and 0.2 ms as EXISTS —
+// and it runs BEFORE the fan-out, so every query pays it serially. The LIMIT 1
+// that used to be on these statements did nothing: it bounds the result rows
+// of an aggregate that always returns exactly one.
 func (s *Server) detectStaleFTSRepos(ctx context.Context, projectPaths []string) []workspaceSearchStaleFTSRepoPayload {
 	out := make([]workspaceSearchStaleFTSRepoPayload, 0)
 	for _, pp := range projectPaths {
-		var nMeta, nFiles int
+		var hasMeta, hasFiles bool
 		if err := s.Deps.DB.QueryRowContext(ctx,
-			`SELECT COUNT(*) FROM chunks_meta WHERE project_path = ? LIMIT 1`, pp).Scan(&nMeta); err != nil {
+			`SELECT EXISTS(SELECT 1 FROM chunks_meta WHERE project_path = ?)`, pp).Scan(&hasMeta); err != nil {
 			s.Deps.Logger.Warn("workspaces search: stale-fts probe (chunks_meta)",
 				"project_path", pp, "err", err)
 			return nil
 		}
-		if nMeta > 0 {
+		if hasMeta {
 			continue
 		}
 		if err := s.Deps.DB.QueryRowContext(ctx,
-			`SELECT COUNT(*) FROM file_hashes WHERE project_path = ? LIMIT 1`, pp).Scan(&nFiles); err != nil {
+			`SELECT EXISTS(SELECT 1 FROM file_hashes WHERE project_path = ?)`, pp).Scan(&hasFiles); err != nil {
 			s.Deps.Logger.Warn("workspaces search: stale-fts probe (file_hashes)",
 				"project_path", pp, "err", err)
 			return nil
 		}
-		if nFiles > 0 {
+		if hasFiles {
 			out = append(out, workspaceSearchStaleFTSRepoPayload{ProjectPath: pp})
 		}
 	}

server/internal/vectorstore/chromemimport.go

@@ -313,6 +313,25 @@ func (s *Store) importCollection(ctx context.Context, dir, name string) (int, er
 	if err := tx.QueryRowContext(ctx, `SELECT id FROM collections WHERE name = ?`, name).Scan(&collID); err != nil {
 		return 0, err
 	}
+	// No compact scan copy is written here, deliberately: the import is already
+	// the slowest thing a boot can do, and the background backfill that runs
+	// right after it (see startQ8Backfill) converts the result at a duty cycle
+	// that leaves the server usable. Until then those collections search the
+	// float32 way, which is what they did before that table existed.
+	//
+	// But "the import creates the collection, so nothing marked it complete"
+	// is only true when the collection is NEW. INSERT OR IGNORE also succeeds
+	// against a collection this binary created and flagged earlier — an
+	// operator pointing CIX_CHROMA_PERSIST_DIR at a legacy tree after indexing
+	// the same project live reaches exactly that, because migration_state is
+	// keyed on the legacy collection name and has never seen it. Imported docs
+	// would then have no compact rows inside a collection whose flag says it
+	// is complete, and the backfill skips flagged collections: permanently
+	// invisible to search. So the flag comes off here, unconditionally.
+	if err := clearQ8Ready(ctx, tx, collID); err != nil {
+		return 0, err
+	}
+	s.forgetQ8(collID)
 	vecStmt, err := tx.PrepareContext(ctx, upsertVectorSQL)
 	if err != nil {
 		return 0, err