diff --git a/README.md b/README.md index 9e51ea8d..77b4b3d0 100644 --- a/README.md +++ b/README.md @@ -29,16 +29,6 @@ Or open `http://localhost:21847/dashboard` in your browser: The cix dashboard — server status at a glance and a guided “Connect Claude Code to cix” onboarding with copy-paste commands -> [!IMPORTANT] -> **Reindex after upgrading the server.** Until the parsing/chunking/embedding -> pipeline stabilizes, an upgrade can change how code is embedded. A reindex -> brings every project onto the new pipeline; within a version, search is -> consistent once reindexed. -> -> The 0.13.0 vector-store change is the exception: it moves the vectors you -> already have into SQLite by itself, on first boot, without re-embedding -> anything ([`doc/VECTORSTORE.md`](doc/VECTORSTORE.md)). - --- ## Why @@ -83,7 +73,7 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow │ └── embedded React dashboard + Swagger UI │ │ │ │ Indexing pipeline │ -│ ├── tree-sitter/wasm (AST chunking, 30+ langs) (wazero) │ +│ ├── tree-sitter/wasm (AST chunking, 31 langs) (wazero) │ │ ├── embedding provider (local llama.cpp / Voyage / OpenAI) │ │ ├── SQLite vector store (float32 BLOBs, streamed cosine scan) │ │ └── SQLite FTS5 mirror (BM25) + metadata (modernc/sqlite) │ @@ -97,6 +87,8 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow Pure-Go static binary; CUDA-image variants add a CUDA runtime layer for GPU embeddings. Workspace clones live in `/repos/`. +**Why vectors live in SQLite.** Through v0.12.x cix used [chromem-go](https://github.com/philippgille/chromem-go), an in-memory vector database. It decodes every document of every collection into the heap at startup and never evicts, so memory was proportional to the index rather than to the work: a real 312k-document index cost **2.2 GB resident while idle** and 47 seconds before it could answer anything. v0.13.0 replaced it with a SQLite store that keeps embeddings as float32 BLOBs and scans them per query — the same index now idles at **tens of megabytes** and answers about a millisecond after boot. The trade is search latency, roughly 4× higher and far less sensitive to how many results you ask for. Existing indexes are imported automatically on first boot; nothing is re-embedded. Numbers, layout and the migration: [`doc/VECTORSTORE.md`](doc/VECTORSTORE.md). + --- ## Quick Start @@ -346,15 +338,8 @@ projects and teams that make it possible: **Indexing & storage** - [tree-sitter](https://tree-sitter.github.io/tree-sitter/) — AST-aware - chunking across 30+ languages, run via + chunking across 31 languages, run via [wazero](https://github.com/tetratelabs/wazero) (pure-Go WASM runtime). -- [gotreesitter](https://github.com/odvcencio/gotreesitter) — the Go - tree-sitter binding cix's AST chunking first grew from; thank you for the - head start. -- [chromem-go](https://github.com/philippgille/chromem-go) — the - embedded vector store cix shipped through v0.12.x, and the model its - collection semantics still follow. Now kept only to read a pre-0.13 - index during the one-time import. - [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) — cgo-free SQLite for project metadata, symbols, the FTS5/BM25 mirror, and (since v0.13.0) the vectors themselves. @@ -367,6 +352,8 @@ projects and teams that make it possible: [oapi-codegen](https://github.com/oapi-codegen/oapi-codegen) — OpenAPI-as-source-of-truth codegen for the Go interface and TypeScript dashboard types. +- [gronx](https://github.com/adhocore/gronx) — the crontab expressions + behind scheduled database maintenance. - [brotli](https://github.com/andybalholm/brotli) and the [Go](https://go.dev/) standard library and `golang.org/x` ecosystem. @@ -382,19 +369,34 @@ projects and teams that make it possible: - [notify](https://github.com/rjeczalik/notify) — cross-platform filesystem watching for the index-on-change watcher. - [koanf](https://github.com/knadh/koanf) — layered configuration - (flags → env → `~/.cix/config.yaml`). + (flags → env → `~/.cix/config.yaml`), with + [validator](https://github.com/go-playground/validator) checking what + lands there. +- [go-gitignore](https://github.com/sabhiram/go-gitignore) — `.cixignore` + and `.gitignore` matching, so the watcher and the indexer agree with git + about what is source. **Dashboard (web UI)** - [React](https://react.dev/) + [Vite](https://vitejs.dev/) — the embedded - dashboard served at `/dashboard`. + dashboard served at `/dashboard`, routed by + [React Router](https://reactrouter.com/). - [Radix UI](https://www.radix-ui.com/) + [Tailwind CSS](https://tailwindcss.com/) — accessible component primitives and styling (the shadcn/ui pattern). - [TanStack Query](https://tanstack.com/query) — server-state and data fetching. - [openapi-typescript](https://github.com/openapi-ts/openapi-typescript) — generates the dashboard's API types from the OpenAPI spec. -- [lucide](https://lucide.dev/) and [sonner](https://github.com/emilkowalski/sonner) - — icons and toast notifications. +- [sonner](https://github.com/emilkowalski/sonner) — toast notifications. + +**No longer in the stack, still owed thanks** +- [chromem-go](https://github.com/philippgille/chromem-go) — the embedded + vector store cix shipped through v0.12.x, and the model its collection + semantics still follow. It is why the early versions worked at all; it + is kept as a dependency only to read a pre-0.13 index during the + one-time import. +- [gotreesitter](https://github.com/odvcencio/gotreesitter) — the Go + tree-sitter binding cix's AST chunking first grew from, before the move + to WASM grammars on wazero. Thank you for the head start. Full dependency lists with versions live in [`server/go.mod`](server/go.mod), [`cli/go.mod`](cli/go.mod), and diff --git a/doc/LANGUAGES.md b/doc/LANGUAGES.md index 69164af1..dead8861 100644 --- a/doc/LANGUAGES.md +++ b/doc/LANGUAGES.md @@ -1,42 +1,42 @@ # Supported languages -cix uses tree-sitter (via `github.com/odvcencio/gotreesitter`) to extract semantic chunks (functions, classes, methods, types) from source code. Files in unsupported languages still get indexed via a sliding-window fallback — they're searchable, just without per-symbol granularity. +cix uses tree-sitter to extract semantic chunks (functions, classes, methods, types) from source code. The grammars are compiled to WebAssembly and run in [wazero](https://github.com/tetratelabs/wazero), a pure-Go WASM runtime — so the server stays a cgo-free static binary and a runaway grammar hits a memory limit inside the sandbox instead of the process heap (`server/internal/chunker/tswasm`). Files in unsupported languages still get indexed via a sliding-window fallback — they're searchable, just without per-symbol granularity. ## Default language set (31) -| ID | gotreesitter factory | Function | Class | Method | Type | -|---|---|:-:|:-:|:-:|:-:| -| `python` | `PythonLanguage` | ✓ | ✓ | | | -| `typescript` | `TypescriptLanguage` | ✓ | ✓ | ✓ | ✓ | -| `tsx` | `TsxLanguage` | ✓ | ✓ | ✓ | ✓ | -| `javascript` | `JavascriptLanguage` | ✓ | ✓ | ✓ | | -| `go` | `GoLanguage` | ✓ | | ✓ | ✓ | -| `rust` | `RustLanguage` | ✓ | ✓ | | ✓ | -| `java` | `JavaLanguage` | ✓ | ✓ | | ✓ | -| `c` | `CLanguage` | ✓ | ✓ | | ✓ | -| `cpp` | `CppLanguage` | ✓ | ✓ | | ✓ | -| `c_sharp` | `CSharpLanguage` | ✓ | ✓ | ✓ | ✓ | -| `ruby` | `RubyLanguage` | ✓ | ✓ | | | -| `php` | `PhpLanguage` | ✓ | ✓ | ✓ | ✓ | -| `swift` | `SwiftLanguage` | ✓ | ✓ | | ✓ | -| `kotlin` | `KotlinLanguage` | ✓ | ✓ | | | -| `scala` | `ScalaLanguage` | ✓ | ✓ | | ✓ | -| `bash` | `BashLanguage` | ✓ | | | | -| `lua` | `LuaLanguage` | ✓ | | | | -| `dart` | `DartLanguage` | ✓ | ✓ | ✓ | ✓ | -| `r` | `RLanguage` | ✓ | | | | -| `objc` | `ObjcLanguage` | ✓ | ✓ | ✓ | ✓ | -| `html` | `HtmlLanguage` | | | | ✓ | -| `css` | `CssLanguage` | | ✓ | | | -| `scss` | `ScssLanguage` | ✓ | ✓ | | | -| `sql` | `SqlLanguage` | ✓ | | | ✓ | -| `markdown` | `MarkdownLanguage` | | | | ✓ | -| `zig` | `ZigLanguage` | ✓ | ✓ | | | -| `julia` | `JuliaLanguage` | ✓ | | | | -| `fortran` | `FortranLanguage` | ✓ | ✓ | | | -| `haskell` | `HaskellLanguage` | ✓ | | | ✓ | -| `ocaml` | `OcamlLanguage` | ✓ | ✓ | | ✓ | -| `solidity` | `SolidityLanguage` | ✓ | ✓ | | ✓ | +| ID | Function | Class | Method | Type | +|---|:-:|:-:|:-:|:-:| +| `python` | ✓ | ✓ | | | +| `typescript` | ✓ | ✓ | ✓ | ✓ | +| `tsx` | ✓ | ✓ | ✓ | ✓ | +| `javascript` | ✓ | ✓ | ✓ | | +| `go` | ✓ | | ✓ | ✓ | +| `rust` | ✓ | ✓ | | ✓ | +| `java` | ✓ | ✓ | | ✓ | +| `c` | ✓ | ✓ | | ✓ | +| `cpp` | ✓ | ✓ | | ✓ | +| `c_sharp` | ✓ | ✓ | ✓ | ✓ | +| `ruby` | ✓ | ✓ | | | +| `php` | ✓ | ✓ | ✓ | ✓ | +| `swift` | ✓ | ✓ | | ✓ | +| `kotlin` | ✓ | ✓ | | | +| `scala` | ✓ | ✓ | | ✓ | +| `bash` | ✓ | | | | +| `lua` | ✓ | | | | +| `dart` | ✓ | ✓ | ✓ | ✓ | +| `r` | ✓ | | | | +| `objc` | ✓ | ✓ | ✓ | ✓ | +| `html` | | | | ✓ | +| `css` | | ✓ | | | +| `scss` | ✓ | ✓ | | | +| `sql` | ✓ | | | ✓ | +| `markdown` | | | | ✓ | +| `zig` | ✓ | ✓ | | | +| `julia` | ✓ | | | | +| `fortran` | ✓ | ✓ | | | +| `haskell` | ✓ | | | ✓ | +| `ocaml` | ✓ | ✓ | | ✓ | +| `solidity` | ✓ | ✓ | | ✓ | The exact AST node types per language live in `server/internal/chunker/chunker.go` (`defaultRegistry`). File-extension mapping lives in `server/internal/langdetect/langdetect.go`. @@ -66,7 +66,7 @@ These produce sliding-window chunks. Adding semantic chunking is a one-map-entry `erlang, elixir, commonlisp, svelte, graphql, hcl (terraform), cmake, dockerfile, regex, xml, make` -PRs welcome — verify node names with `gotreesitter`'s `cmd/tsquery` against a representative fixture before adding. +PRs welcome — the node names in `defaultRegistry` must match what the grammar actually emits, and `chunker_test.go` walks the registry against the compiled grammars to catch a typo without needing a fixture file per language. ## How the chunker decides diff --git a/doc/TEAM_DEPLOYMENT.md b/doc/TEAM_DEPLOYMENT.md index f5438bbf..4513fb2e 100644 --- a/doc/TEAM_DEPLOYMENT.md +++ b/doc/TEAM_DEPLOYMENT.md @@ -251,10 +251,12 @@ docker compose pull docker compose up -d ``` -> **Reindex after a server upgrade.** Until the parsing/chunking/embedding -> pipeline stabilizes, an upgrade can change how code is embedded. Trigger a -> reindex (dashboard or `cix reindex`) so every project lands on the new -> pipeline. Within a version, search is consistent once reindexed. +> **An upgrade does not normally need a reindex.** Existing indexes keep +> working, and 0.13's move of the vectors into SQLite imports them for you on +> first boot without re-embedding anything. The cases that do call for one — +> the embedding model changing, or a release that backfills new chunk-level +> data — are listed in [`UPDATES.md`](UPDATES.md#3-reindex-after-an-upgrade), +> and the dashboard flags affected projects itself. Server and CLI are released on **independent tag streams** (`server/vX.Y.Z` vs CLI tags) and are wire-compatible across a minor skew — you don't have to diff --git a/doc/UPDATES.md b/doc/UPDATES.md index 78e93169..4a8353ce 100644 --- a/doc/UPDATES.md +++ b/doc/UPDATES.md @@ -127,7 +127,7 @@ Upgrading the **server** can require a reindex in two cases: model (or you change it yourself via Dashboard → Server → Embedding model), every project becomes stale. The dashboard's drift indicator paints affected projects red with a "Stale model" - badge until you reindex. See README's *Drift indicator* section. + badge until you reindex. See [`DASHBOARD.md`](DASHBOARD.md#drift-indicator). 2. **Schema migration adds chunk-level data.** Releases that backfill new chunk metadata (e.g. the FTS5 mirror introduced by `f00e3d3`) may prompt the dashboard to recommend a reindex on existing diff --git a/server/dashboard/src/modules/projects/components/ProjectInfoCard.tsx b/server/dashboard/src/modules/projects/components/ProjectInfoCard.tsx index 9399386c..3607b7a7 100644 --- a/server/dashboard/src/modules/projects/components/ProjectInfoCard.tsx +++ b/server/dashboard/src/modules/projects/components/ProjectInfoCard.tsx @@ -4,7 +4,7 @@ import { formatBytes } from '@/lib/formatBytes'; import { formatDateTime, formatRelative } from '@/lib/formatDate'; // Metadata that would otherwise need an SSH session: which embedding model -// produced the vectors, where this project's SQLite and chromem-go state +// produced the vectors, where this project's SQLite and vector-store state // lives, and how big both are. Storage fields are nullable — an // embeddings-disabled server has no resolvable paths, and "—" beats "0 B". // diff --git a/server/internal/db/schema.go b/server/internal/db/schema.go index 7d686be6..63566db6 100644 --- a/server/internal/db/schema.go +++ b/server/internal/db/schema.go @@ -417,7 +417,7 @@ CREATE INDEX IF NOT EXISTS idx_call_edges_caller ON call_edges(caller_symbol); CREATE INDEX IF NOT EXISTS idx_call_edges_callee ON call_edges(callee_symbol); -- PR14 dropped the workspaces communities/community_members tables. --- Workspace search is now a weighted fan-out across per-project chromem +-- Workspace search is now a weighted fan-out across per-project vector -- collections (no Louvain, no centroid index). migrateDropCommunities -- DROPs the tables on upgrade for installs that ran any of PR5..PR12. @@ -441,8 +441,8 @@ CREATE INDEX IF NOT EXISTS idx_chunks_meta_project_file CREATE INDEX IF NOT EXISTS idx_chunks_meta_project ON chunks_meta(project_path); --- chunks_fts is the BM25-searchable side, parallel to chromem-go's dense --- vector store. Workspace search runs both in parallel per project then +-- chunks_fts is the BM25-searchable side, parallel to the dense vector +-- store. Workspace search runs both in parallel per project then -- fuses by RRF; project-relevance gating uses BM25 signal to drop repos -- that share no token with the query (the dense-only fan-out leaks -- semantically-distant repos as false positives). diff --git a/server/internal/embeddings/provider/ollama/provider.go b/server/internal/embeddings/provider/ollama/provider.go index c3f40a45..3f758abc 100644 --- a/server/internal/embeddings/provider/ollama/provider.go +++ b/server/internal/embeddings/provider/ollama/provider.go @@ -110,9 +110,9 @@ func (p *Provider) ID() string { return "ollama:" + p.cfg.Model } -// Dimension returns 0 — the vector store infers dimension from the -// first upsert (chromem-go behaviour) and CodeRankEmbed-Q8 reports -// it on first call. +// Dimension returns 0 — the vector store takes whatever width the first +// upsert carries (each namespace holds one model's vectors, so they cannot +// mix) and CodeRankEmbed-Q8 reports it on first call. func (p *Provider) Dimension() int { return 0 } // SupportsTokenize is true: llama-server exposes /tokenize. diff --git a/server/internal/embeddings/provider/voyage/voyage.go b/server/internal/embeddings/provider/voyage/voyage.go index 6fb7c209..04318f85 100644 --- a/server/internal/embeddings/provider/voyage/voyage.go +++ b/server/internal/embeddings/provider/voyage/voyage.go @@ -9,7 +9,7 @@ // (256/512/1024/2048). Part of Provider.ID() because changing it // invalidates the existing index. // - output_dtype: float|int8 (binary/ubinary are out of scope — -// chromem-go has no hamming search). For int8 the server returns +// the vector store has no hamming search). For int8 the server returns // a list of integers per dimension; we dequantize to float32 in // this package before returning vectors to the vector store. // @@ -734,7 +734,7 @@ func (p *Provider) embed(ctx context.Context, texts []string, inputType string) // the approximate unit-norm float representation. // // This is the only place in the codebase that handles int8 quantized -// embeddings; chromem-go and the search path both work exclusively +// embeddings; the vector store and the search path both work exclusively // in float32. func dequantize(raw json.RawMessage, dtype string) ([]float32, error) { switch dtype { diff --git a/server/internal/httpapi/router.go b/server/internal/httpapi/router.go index 34dbf858..8ae39917 100644 --- a/server/internal/httpapi/router.go +++ b/server/internal/httpapi/router.go @@ -76,7 +76,7 @@ type Deps struct { // server is started with CIX_EMBEDDINGS_ENABLED=false (e.g. in router // tests). Phase 5 uses it for semantic search. EmbeddingSvc EmbeddingsQuerier - // VectorStore is the chromem-go backed vector store (Phase 4). Nil-safe: + // VectorStore is the SQLite-backed vector store. Nil-safe: // semantic search returns empty results when absent. Typed as the // vectorstore.Interface so production can supply a *vectorstore.Holder // (swappable on provider switch) while tests pass a raw *Store. diff --git a/server/internal/httpapi/workspacesearch.go b/server/internal/httpapi/workspacesearch.go index af835891..d065d83a 100644 --- a/server/internal/httpapi/workspacesearch.go +++ b/server/internal/httpapi/workspacesearch.go @@ -119,14 +119,14 @@ type projectHits struct { // WorkspaceSearch — GET /api/v1/workspaces/{id}/search. // // Hybrid BM25+dense fan-out. Each project runs two queries in -// parallel: dense (chromem cosine) and sparse (SQLite FTS5 BM25 over +// parallel: dense (vector-store cosine) and sparse (SQLite FTS5 BM25 over // chunks_fts). Per project, the two ranked lists are fused via // Reciprocal Rank Fusion. Across projects, an α-blended candidacy // score (with per-query min-max normalization on both signals) plus // a relative threshold (`candidacy ≥ best × 0.4`) keeps the result // set focused on repos that actually share vocabulary or semantics // with the query — pure-dense fan-out leaked every workspace repo at -// noise-level cosine similarity, since chromem returns the N nearest +// noise-level cosine similarity, since a top-K scan returns the N nearest // vectors regardless of how far away "nearest" actually is. // // Observed pre-hybrid: in a workspace of N repos, the repos that @@ -756,9 +756,9 @@ func projectLabel(projectPath string) string { return projectPath } -// round4 rounds f to 4 decimal places — matches the chunk-side -// rounding chromem already applies, so scores in the response look -// consistent across nested fields. +// round4 rounds f to 4 decimal places — matches the chunk-side rounding the +// vector store already applies, so scores in the response look consistent +// across nested fields. func round4(f float32) float32 { if math.IsNaN(float64(f)) { return 0 diff --git a/server/internal/indexer/indexer.go b/server/internal/indexer/indexer.go index 7733721b..03a17ad5 100644 --- a/server/internal/indexer/indexer.go +++ b/server/internal/indexer/indexer.go @@ -818,8 +818,9 @@ func (s *Service) ProcessFilesStreaming( } // ---- Stage 3: WRITE (serial, ordered) -------------------------------- - // Vector-store + per-file DB writes run on this single goroutine: chromem - // is thread-safe but serialising keeps SQLite's WAL writer uncontended and + // Vector-store + per-file DB writes run on this single goroutine: the + // store is thread-safe, but serialising keeps SQLite's WAL writer + // uncontended and // preserves deterministic progress-event ordering. Each write is local and // sub-ms, so serialising costs nothing next to the (now parallel) embeds. for _, p := range prep { @@ -868,7 +869,7 @@ func (s *Service) ProcessFilesStreaming( } } - // Build chunksfts payload from the same chunks pushed to chromem. + // Build chunksfts payload from the same chunks pushed to the vector store. ftsChunks := make([]chunksfts.Chunk, len(p.vsChunks)) for i, c := range p.vsChunks { ftsChunks[i] = chunksfts.Chunk{ diff --git a/server/internal/maintenance/analyze.go b/server/internal/maintenance/analyze.go index df1d2595..d14342fd 100644 --- a/server/internal/maintenance/analyze.go +++ b/server/internal/maintenance/analyze.go @@ -96,7 +96,7 @@ func (s *Service) Analyze(ctx context.Context) (*Analysis, error) { // scanState is the live picture every scanner reconciles against, read once so // five scanners do not issue the same queries five times. type scanState struct { - // liveCollections maps chromem collection name -> host_path, for every + // liveCollections maps vector-store collection name -> host_path, for every // row currently in `projects`. liveCollections map[string]string // liveHashes is the set of projects.HashPath values currently in use. diff --git a/server/internal/vectorstore/store.go b/server/internal/vectorstore/store.go index 02ea3102..30212027 100644 --- a/server/internal/vectorstore/store.go +++ b/server/internal/vectorstore/store.go @@ -11,7 +11,8 @@ // embedding namespace and a query streams the collection past a dot product // with a top-K heap. Opening is one file open (sub-millisecond), resident // memory is a couple of page-cache buffers per active query, and the cost is -// search latency: roughly 3x chromem's on the same data. +// search latency: roughly 4x chromem's on the same data — see the measured +// table in doc/VECTORSTORE.md. // // # Frozen contracts //