Promote develop → main for server/v0.14.0

.gitignore

@@ -93,3 +93,6 @@ server/internal/httpapi/dashboard/dist/*
 portainer_mcp/
 portainer-mcp*
 tools.yaml
+
+# Local load-test corpus + throwaway server instance (never committed)
+/loadtests/

doc/CONFIG_REFERENCE.md

@@ -37,6 +37,7 @@ the DB.
 | `CIX_CHROMA_PERSIST_DIR` | `/data/chroma` | Legacy chromem-go store. Read on startup for the one-time import into the SQLite vector store, then left untouched as the rollback path. See [VECTORSTORE.md](VECTORSTORE.md). |
 | `CIX_VECTORS_DIR` | sibling of `CIX_CHROMA_PERSIST_DIR` (`/data/vectors`) | Vector store directory: one SQLite database per embedding namespace. |
 | `CIX_VECTOR_MMAP_SIZE` | `0` (off) | `PRAGMA mmap_size` for the vector store, in bytes. Roughly 40% lower search latency in exchange for resident memory — mapped database pages count in RSS. |
+| `CIX_VECTOR_SCAN_QUANT` | `true` | Scan a compact int8 copy of each vector instead of the float32 original, rescoring the shortlist against the originals. Every score returned is the exact cosine; which documents reach the shortlist is an approximation, measured at recall 1.000 against exact search (see `doc/VECTORSTORE.md`). 3.4x fewer bytes read per query at 2048 dimensions, in exchange for roughly a quarter more disk. Set `false` if the volume cannot take it; existing copies are then ignored, and writes remove the copies of rows they touch so re-enabling rebuilds instead of trusting stale data. |
 | `CIX_GGUF_CACHE_DIR` | `/data/models` | Where downloaded GGUF files live. |
 | `CIX_PUBLIC_URL` | — | Externally-reachable URL used to build GitHub webhook delivery URLs. Empty disables webhook URL display. |
 

doc/VECTORSTORE.md

@@ -150,21 +150,96 @@ on disk, deliberately: it keeps the package self-contained and
 live in `vectors`. A multi-kilobyte `TEXT` column pushes a row past SQLite's
 local-payload limit, and SQLite then keeps only ~1 kB of the row in the table
 page and spills the rest — *including the embedding* — into an overflow chain,
-roughly doubling the pages a scan touches. Kept apart, a `vectors` row is
-~3.2 kB and two of them share an 8 KiB page. Content is read only for the K
+roughly doubling the pages a scan touches. Kept apart, a 768-dim `vectors` row
+is ~3.2 kB and two of them share an 8 KiB page. Content is read only for the K
 winners of a search: one extra lookup per result.
 
+**Why the scan reads a second copy of every vector.** The paragraph above stops
+being true once the model is bigger than 1024 dimensions. A 2048-dim float32
+embedding is 8192 bytes on its own, past the 8157-byte local-payload limit, so
+every `vectors` row spills into an overflow page and the scan is back to the
+layout splitting out the content was meant to avoid. Measured with `dbstat`
+over 400 rows, bytes a full scan must read per vector:
+
+| dimensions | representation | leaf | overflow | bytes/vector |
+|---|---|---|---|---|
+| 768 | float32 | 200 | 0 | 4096 |
+| 1024 | float32 | 400 | 0 | 8192 |
+| 2048 | float32 | 50 | 400 | 9216 |
+| 768 | int8 | 40 | 0 | 819 |
+| 1024 | int8 | 58 | 0 | 1188 |
+| 2048 | int8 | 134 | 0 | 2744 |
+
+The pathological line is 1024, not 2048: nothing overflows there and the scan
+still reads 8192 bytes to obtain 4096, because two 4.1 kB rows cannot share an
+8 KiB page. Halving `output_dimension` to save time bought half the vector
+quality for 89% of the I/O.
+
+`vectors_q8` removes the whole step function by scanning one byte per component
+instead of four. `TestScanPackingEfficiency` and `TestScanBytesPerVectorBudget`
+assert those numbers — as pages, not milliseconds, so they mean the same thing
+in CI, on a laptop, and on the production box.
+
 ## Search
 
 ```
-SELECT rowid, embedding FROM vectors INDEXED BY idx_vec_coll
- WHERE collection_id = ?  [AND <metadata filters>]
+SELECT doc_id, scale, embedding FROM vectors_q8 INDEXED BY idx_q8_coll
+ WHERE collection_id = ?  [AND language = ?]
 ```
 
-Rows stream past a dot product (embeddings are stored L2-normalised, so cosine
-similarity *is* the dot product) into a top-K min-heap that rejects a losing
-row with one comparison. Metadata and chunk text are fetched afterwards, for
-the winners only.
+Rows stream past an integer dot product into a top-K min-heap that rejects a
+losing row with one comparison. The heap is wider than the caller's limit — the
+int8 ranking chooses a shortlist, it does not produce the answer. The shortlist
+is then rescored against the exact float32 vectors in `vectors`, and metadata
+and chunk text are fetched for the winners only.
+
+**What the approximation costs.** Measured on 60k vectors of the load-test
+fixture's largest collection (`ziglang/zig`, voyage-code-3 @2048) against 50
+real query-side embeddings, recall of the exact float32 top-K:
+
+| shortlist | k=10 | k=20 |
+|---|---|---|
+| 20 | 0.998 | 0.994 |
+| 40 | 0.998 | 0.999 |
+| **60** | **1.000** | **1.000** |
+| 200 | 1.000 | 1.000 |
+
+Without rescoring at all the int8 ranking alone gives 0.994 at both k — the
+quantisation misorders near-ties, it does not lose the documents, which is why
+re-reading a few dozen exact vectors recovered every one of them here.
+`q8Shortlist` therefore uses a floor of 64 and 4x the limit above it. Scan CPU
+in the same run: 127 ms per query float32, 42 ms int8 (3.0x).
+
+Two different guarantees, worth keeping apart. **Scores are exact by
+construction**: every number a caller sees is the cosine against the float32
+vector, computed by the rescore. **The result set is an approximation** whose
+error was measured at zero on this corpus and is not proved at zero in general
+— the shortlist is a fixed size and `topK` rejects boundary ties strictly, so a
+collection holding more than `shortlist` documents inside one quantisation step
+of each other (a file vendored a hundred times, say) can truncate a tie in scan
+order, and the rescore cannot recover a document that was never shortlisted.
+Widening the shortlist to swallow boundary ties would close that; it has not
+been needed on any corpus measured so far.
+
+Scores returned to callers are always the exact cosine, never the int8
+estimate. That is load-bearing beyond cosmetics: `min_score` thresholds on it,
+the workspace fan-out normalises across projects with it, and hybrid search
+blends it with BM25 — an approximate score would move results between projects
+in a way no single-project test would catch. `TestSearchScoresAreExact` pins it.
+
+**Building and rebuilding the copy.** Writes maintain `vectors_q8` in the same
+transaction as `vectors`, so a collection created by this code is complete by
+construction, and `q8_state` records that at creation — the readiness check is
+a primary-key lookup, never a `COUNT`. A store written before the table existed
+is converted by a background pass at open, largest collection first, in 2000-row
+transactions at a 50% duty cycle; until a collection is covered its searches
+take the float32 scan, which is correct and simply slower. Nothing is ever
+marked complete before it is: the flag is written in the same transaction as
+the batch that proves it. Set `CIX_VECTOR_SCAN_QUANT=false` to opt out — the
+copy is roughly a quarter of the float32 bytes on top of an already large
+store, so an operator short of disk needs a way to say no. Turning it off also
+withdraws the completion flag from anything written while it is off, so turning
+it back on rebuilds rather than trusting a stale copy.
 
 `INDEXED BY` is not an optimisation hint, it is a guarantee, and *which* index
 matters. Measured on the real index, scanning its largest (74k-row) collection:
@@ -188,6 +263,10 @@ The metadata filter (`where`) mirrors chromem's semantics exactly, including
 the two odd cases: an unknown key with a non-empty value matches nothing, and
 an unknown key with an empty value matches everything.
 
+`TestQ8ScanUsesCollectionIndex` pins the same guarantee for the compact table:
+`idx_q8_coll`'s keys are `(collection_id, rowid)` for the same reason, and the
+language filter must not change the driving index.
+
 **Concurrency.** One scan per query, and a process-wide semaphore caps
 concurrent scans at `NumCPU`. Splitting a single query across workers was
 measured to buy nothing in the low-memory configuration (109 ms at 1 worker vs

doc/openapi.yaml

@@ -2630,6 +2630,20 @@ paths:
             minimum: 0
             maximum: 1
             default: 0.4
+        - name: timings
+          in: query
+          required: false
+          description: |
+            Attach a per-phase breakdown of where the query spent its
+            time (see WorkspaceSearchTimings). Diagnostic, not API
+            surface: it exists so a slow workspace query can be taken
+            apart, and it is off unless asked for. The server logs the
+            same breakdown by itself whenever a query is slow, so
+            catching a regression does not depend on someone having
+            passed this flag at the right moment.
+          schema:
+            type: boolean
+            default: false
       responses:
         "200":
           description: Search results
@@ -4137,6 +4151,12 @@ components:
             Size of the tree. Absent when it could not be walked, which keeps
             "unreadable" distinguishable from "empty". The SQLite entry
             includes the -wal and -shm sidecars.
+        partial:
+          type: boolean
+          description: |
+            True when used_bytes undercounts because some entries inside the
+            tree were unreadable and skipped. Absent means the sum is
+            complete.
         fs_total_bytes: { type: integer, format: int64 }
         fs_free_bytes: { type: integer, format: int64 }
 
@@ -6525,6 +6545,95 @@ components:
             under the new schema.
           items:
             $ref: "#/components/schemas/WorkspaceSearchStaleFTSRepo"
+        timings:
+          $ref: "#/components/schemas/WorkspaceSearchTimings"
+
+    WorkspaceSearchTimings:
+      type: object
+      description: |
+        Where this query spent its time, in milliseconds. Returned only
+        when the request passes `timings=true` AND the query actually ran
+        a search — a workspace with no queryable project reports nothing
+        rather than a block of zeroes that would read as "instant".
+
+        The fan-out phases report a sum AND a max, and both are needed: the
+        sum is how much work the query did across every project, the max is
+        how long it waited for the slowest one. With perfect parallelism the
+        wall time is the max; with none it is the sum; in practice it is
+        between them, and one number alone cannot say which.
+
+        `projects_scanned` versus `projects_returned` is the ratio that says
+        how much of the work was discarded: the fan-out runs dense and BM25
+        over every project in the workspace and then thresholds the answer
+        down to the relevant ones. `projects_in_panel` is a separate,
+        smaller question — how many of those the caller was actually shown,
+        after the `top_projects` cap.
+
+        The named phases do not sum to `wall_ms`. The remainder is the
+        workspace visibility check, assembling the projects panel, the
+        round-robin interleave and writing the response — all in memory and,
+        on the load-test fixture, ~19 ms of ~9,900 ms.
+      properties:
+        wall_ms:
+          type: integer
+          description: The whole handler, from its first line to its last.
+        embed_ms:
+          type: integer
+          description: Round-trip to the embedding provider for the query text.
+        resolve_ms:
+          type: integer
+          description: |
+            Loading the workspace's project memberships and applying the
+            per-user access filter. Separate from the rest because it is the
+            one pre-fan-out step that grows with how many projects the
+            caller can see, rather than with the workspace.
+        stale_fts_ms:
+          type: integer
+          description: The pre-fan-out probe for repos with no BM25 mirror.
+        fanout_ms:
+          type: integer
+          description: Wall time of the parallel per-project phase.
+        dense_sum_ms:
+          type: integer
+          description: |
+            Vector-store search summed across projects, including hydration
+            of each project's winning rows, and including projects whose
+            query failed — the time was spent either way, and omitting it
+            would put the sums permanently below the wall time they explain.
+        dense_max_ms:
+          type: integer
+          description: |
+            The slowest single project's dense search. May belong to a
+            project whose query failed; the fan-out logs a warning of its
+            own for those.
+        bm25_ms:
+          type: integer
+          description: |
+            The workspace's BM25 search. One FTS5 statement covering every
+            project, partitioned per project by a window function — not a
+            sum over projects, which is why it has no matching `_max`
+            field. `MATCH` is evaluated over the whole server's index
+            whatever the scope, so asking once per project repeated the
+            same global work N times and the N queries contended over one
+            index on top of that.
+        fuse_ms:
+          type: integer
+          description: Normalisation, candidacy blending and thresholding.
+        projects_scanned:
+          type: integer
+          description: Projects the fan-out searched.
+        projects_returned:
+          type: integer
+          description: |
+            Projects that survived the relevance threshold — NOT the number
+            the caller was shown. Capping this at `top_projects` would peg
+            the scanned:returned ratio to a request parameter instead of
+            measuring how much of the fan-out's work was discarded.
+        projects_in_panel:
+          type: integer
+          description: |
+            Projects present in the response's `projects` array, i.e.
+            `min(projects_returned, top_projects)`.
 
     WorkspaceSearchPendingRepo:
       type: object

server/cmd/cix-server/main.go

@@ -411,6 +411,7 @@ func run() (restart bool, err error) {
 			Dir:             cfg.VectorDirFor(comps),
 			LegacyChromaDir: cfg.ChromaDirFor(comps),
 			MMapBytes:       cfg.VectorMMapSize,
+			ScanQuant:       cfg.VectorScanQuantEnabled,
 			Logger:          logger,
 		})
 	}
@@ -437,6 +438,7 @@ func run() (restart bool, err error) {
 
 	idx := indexer.New(database, vsHolder, embedSvc, logger)
 	idx.SetEmbedIncludePath(cfg.EmbedIncludePath)
+	idx.SetMaxChunkTokens(cfg.MaxChunkTokens)
 	// Record the active embedding model on every indexed project so the
 	// dashboard can highlight stale vectors when the runtime provider /
 	// model changes. Wire it as a live lookup so a runtime provider

server/dashboard/src/modules/search/SearchPage.tsx

@@ -44,18 +44,15 @@ export default function SearchPage() {
   const queryParam = params.get('q') ?? '';
   const [draft, setDraft] = useState(queryParam);
 
-  // Debounce input → URL after 250ms idle; Enter commits immediately.
-  useEffect(() => {
-    const id = setTimeout(() => {
-      if (draft === queryParam) return;
-      const next = new URLSearchParams(params);
-      if (draft.trim()) next.set('q', draft);
-      else next.delete('q');
-      setParams(next, { replace: true });
-    }, 250);
-    return () => clearTimeout(id);
-    // eslint-disable-next-line react-hooks/exhaustive-deps
-  }, [draft]);
+  // Typing changes `draft` and nothing else. The query in the URL — which is
+  // what actually runs a search — moves only on submit.
+  //
+  // This used to debounce draft into the URL after 250ms idle. That is the
+  // usual pattern and it is wrong here: a semantic search embeds the query
+  // through the configured provider, so every pause while typing spent a real
+  // API call and a full fan-out to answer a half-written question. "retry with
+  // exponential backoff" typed at a normal pace fires on "retry", "retry with",
+  // "retry with expo" — three searches nobody asked for and one they did.
 
   // Follow the URL when it changes from outside (a pasted link, back button).
   useEffect(() => {
@@ -172,7 +169,11 @@ function Results({
     );
   }
   if (query.trim().length < 2) {
-    return <Empty title="Type a query">At least two characters, then results appear here.</Empty>;
+    return (
+      <Empty title="Type a query">
+        At least two characters, then press Enter to search.
+      </Empty>
+    );
   }
   switch (mode) {
     case 'semantic':

server/dashboard/src/modules/search/components/SearchBar.tsx

@@ -13,7 +13,7 @@ export function SearchBar({
 }: {
   value: string;
   onChange: (v: string) => void;
-  /** Fired on Enter — bypasses the debounce and commits immediately. */
+  /** Fired on Enter. This is the ONLY thing that runs a search — typing does not. */
   onSubmit?: (v: string) => void;
   placeholder?: string;
   className?: string;