Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 25 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,6 @@ Or open `http://localhost:21847/dashboard` in your browser:

<img src="site/public/img/dashboard-home.png" alt="The cix dashboard — server status at a glance and a guided “Connect Claude Code to cix” onboarding with copy-paste commands" width="900">

> [!IMPORTANT]
> **Reindex after upgrading the server.** Until the parsing/chunking/embedding
> pipeline stabilizes, an upgrade can change how code is embedded. A reindex
> brings every project onto the new pipeline; within a version, search is
> consistent once reindexed.
>
> The 0.13.0 vector-store change is the exception: it moves the vectors you
> already have into SQLite by itself, on first boot, without re-embedding
> anything ([`doc/VECTORSTORE.md`](doc/VECTORSTORE.md)).

---

## Why
Expand Down Expand Up @@ -83,7 +73,7 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow
│ └── embedded React dashboard + Swagger UI │
│ │
│ Indexing pipeline │
│ ├── tree-sitter/wasm (AST chunking, 30+ langs) (wazero) │
│ ├── tree-sitter/wasm (AST chunking, 31 langs) (wazero) │
│ ├── embedding provider (local llama.cpp / Voyage / OpenAI) │
│ ├── SQLite vector store (float32 BLOBs, streamed cosine scan) │
│ └── SQLite FTS5 mirror (BM25) + metadata (modernc/sqlite) │
Expand All @@ -97,6 +87,8 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow

Pure-Go static binary; CUDA-image variants add a CUDA runtime layer for GPU embeddings. Workspace clones live in `<data-dir>/repos/`.

**Why vectors live in SQLite.** Through v0.12.x cix used [chromem-go](https://github.com/philippgille/chromem-go), an in-memory vector database. It decodes every document of every collection into the heap at startup and never evicts, so memory was proportional to the index rather than to the work: a real 312k-document index cost **2.2 GB resident while idle** and 47 seconds before it could answer anything. v0.13.0 replaced it with a SQLite store that keeps embeddings as float32 BLOBs and scans them per query — the same index now idles at **tens of megabytes** and answers about a millisecond after boot. The trade is search latency, roughly 4× higher and far less sensitive to how many results you ask for. Existing indexes are imported automatically on first boot; nothing is re-embedded. Numbers, layout and the migration: [`doc/VECTORSTORE.md`](doc/VECTORSTORE.md).

---

## Quick Start
Expand Down Expand Up @@ -346,15 +338,8 @@ projects and teams that make it possible:

**Indexing & storage**
- [tree-sitter](https://tree-sitter.github.io/tree-sitter/) — AST-aware
chunking across 30+ languages, run via
chunking across 31 languages, run via
[wazero](https://github.com/tetratelabs/wazero) (pure-Go WASM runtime).
- [gotreesitter](https://github.com/odvcencio/gotreesitter) — the Go
tree-sitter binding cix's AST chunking first grew from; thank you for the
head start.
- [chromem-go](https://github.com/philippgille/chromem-go) — the
embedded vector store cix shipped through v0.12.x, and the model its
collection semantics still follow. Now kept only to read a pre-0.13
index during the one-time import.
- [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) — cgo-free
SQLite for project metadata, symbols, the FTS5/BM25 mirror, and (since
v0.13.0) the vectors themselves.
Expand All @@ -367,6 +352,8 @@ projects and teams that make it possible:
[oapi-codegen](https://github.com/oapi-codegen/oapi-codegen) —
OpenAPI-as-source-of-truth codegen for the Go interface and TypeScript
dashboard types.
- [gronx](https://github.com/adhocore/gronx) — the crontab expressions
behind scheduled database maintenance.
- [brotli](https://github.com/andybalholm/brotli) and the
[Go](https://go.dev/) standard library and `golang.org/x` ecosystem.

Expand All @@ -382,19 +369,34 @@ projects and teams that make it possible:
- [notify](https://github.com/rjeczalik/notify) — cross-platform filesystem
watching for the index-on-change watcher.
- [koanf](https://github.com/knadh/koanf) — layered configuration
(flags → env → `~/.cix/config.yaml`).
(flags → env → `~/.cix/config.yaml`), with
[validator](https://github.com/go-playground/validator) checking what
lands there.
- [go-gitignore](https://github.com/sabhiram/go-gitignore) — `.cixignore`
and `.gitignore` matching, so the watcher and the indexer agree with git
about what is source.

**Dashboard (web UI)**
- [React](https://react.dev/) + [Vite](https://vitejs.dev/) — the embedded
dashboard served at `/dashboard`.
dashboard served at `/dashboard`, routed by
[React Router](https://reactrouter.com/).
- [Radix UI](https://www.radix-ui.com/) + [Tailwind CSS](https://tailwindcss.com/)
— accessible component primitives and styling (the shadcn/ui pattern).
- [TanStack Query](https://tanstack.com/query) — server-state and data
fetching.
- [openapi-typescript](https://github.com/openapi-ts/openapi-typescript) —
generates the dashboard's API types from the OpenAPI spec.
- [lucide](https://lucide.dev/) and [sonner](https://github.com/emilkowalski/sonner)
— icons and toast notifications.
- [sonner](https://github.com/emilkowalski/sonner) — toast notifications.

**No longer in the stack, still owed thanks**
- [chromem-go](https://github.com/philippgille/chromem-go) — the embedded
vector store cix shipped through v0.12.x, and the model its collection
semantics still follow. It is why the early versions worked at all; it
is kept as a dependency only to read a pre-0.13 index during the
one-time import.
- [gotreesitter](https://github.com/odvcencio/gotreesitter) — the Go
tree-sitter binding cix's AST chunking first grew from, before the move
to WASM grammars on wazero. Thank you for the head start.

Full dependency lists with versions live in
[`server/go.mod`](server/go.mod), [`cli/go.mod`](cli/go.mod), and
Expand Down
70 changes: 35 additions & 35 deletions doc/LANGUAGES.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,42 @@
# Supported languages

cix uses tree-sitter (via `github.com/odvcencio/gotreesitter`) to extract semantic chunks (functions, classes, methods, types) from source code. Files in unsupported languages still get indexed via a sliding-window fallback — they're searchable, just without per-symbol granularity.
cix uses tree-sitter to extract semantic chunks (functions, classes, methods, types) from source code. The grammars are compiled to WebAssembly and run in [wazero](https://github.com/tetratelabs/wazero), a pure-Go WASM runtime — so the server stays a cgo-free static binary and a runaway grammar hits a memory limit inside the sandbox instead of the process heap (`server/internal/chunker/tswasm`). Files in unsupported languages still get indexed via a sliding-window fallback — they're searchable, just without per-symbol granularity.

## Default language set (31)

| ID | gotreesitter factory | Function | Class | Method | Type |
|---|---|:-:|:-:|:-:|:-:|
| `python` | `PythonLanguage` | ✓ | ✓ | | |
| `typescript` | `TypescriptLanguage` | ✓ | ✓ | ✓ | ✓ |
| `tsx` | `TsxLanguage` | ✓ | ✓ | ✓ | ✓ |
| `javascript` | `JavascriptLanguage` | ✓ | ✓ | ✓ | |
| `go` | `GoLanguage` | ✓ | | ✓ | ✓ |
| `rust` | `RustLanguage` | ✓ | ✓ | | ✓ |
| `java` | `JavaLanguage` | ✓ | ✓ | | ✓ |
| `c` | `CLanguage` | ✓ | ✓ | | ✓ |
| `cpp` | `CppLanguage` | ✓ | ✓ | | ✓ |
| `c_sharp` | `CSharpLanguage` | ✓ | ✓ | ✓ | ✓ |
| `ruby` | `RubyLanguage` | ✓ | ✓ | | |
| `php` | `PhpLanguage` | ✓ | ✓ | ✓ | ✓ |
| `swift` | `SwiftLanguage` | ✓ | ✓ | | ✓ |
| `kotlin` | `KotlinLanguage` | ✓ | ✓ | | |
| `scala` | `ScalaLanguage` | ✓ | ✓ | | ✓ |
| `bash` | `BashLanguage` | ✓ | | | |
| `lua` | `LuaLanguage` | ✓ | | | |
| `dart` | `DartLanguage` | ✓ | ✓ | ✓ | ✓ |
| `r` | `RLanguage` | ✓ | | | |
| `objc` | `ObjcLanguage` | ✓ | ✓ | ✓ | ✓ |
| `html` | `HtmlLanguage` | | | | ✓ |
| `css` | `CssLanguage` | | ✓ | | |
| `scss` | `ScssLanguage` | ✓ | ✓ | | |
| `sql` | `SqlLanguage` | ✓ | | | ✓ |
| `markdown` | `MarkdownLanguage` | | | | ✓ |
| `zig` | `ZigLanguage` | ✓ | ✓ | | |
| `julia` | `JuliaLanguage` | ✓ | | | |
| `fortran` | `FortranLanguage` | ✓ | ✓ | | |
| `haskell` | `HaskellLanguage` | ✓ | | | ✓ |
| `ocaml` | `OcamlLanguage` | ✓ | ✓ | | ✓ |
| `solidity` | `SolidityLanguage` | ✓ | ✓ | | ✓ |
| ID | Function | Class | Method | Type |
|---|:-:|:-:|:-:|:-:|
| `python` | ✓ | ✓ | | |
| `typescript` | ✓ | ✓ | ✓ | ✓ |
| `tsx` | ✓ | ✓ | ✓ | ✓ |
| `javascript` | ✓ | ✓ | ✓ | |
| `go` | ✓ | | ✓ | ✓ |
| `rust` | ✓ | ✓ | | ✓ |
| `java` | ✓ | ✓ | | ✓ |
| `c` | ✓ | ✓ | | ✓ |
| `cpp` | ✓ | ✓ | | ✓ |
| `c_sharp` | ✓ | ✓ | ✓ | ✓ |
| `ruby` | ✓ | ✓ | | |
| `php` | ✓ | ✓ | ✓ | ✓ |
| `swift` | ✓ | ✓ | | ✓ |
| `kotlin` | ✓ | ✓ | | |
| `scala` | ✓ | ✓ | | ✓ |
| `bash` | ✓ | | | |
| `lua` | ✓ | | | |
| `dart` | ✓ | ✓ | ✓ | ✓ |
| `r` | ✓ | | | |
| `objc` | ✓ | ✓ | ✓ | ✓ |
| `html` | | | | ✓ |
| `css` | | ✓ | | |
| `scss` | ✓ | ✓ | | |
| `sql` | ✓ | | | ✓ |
| `markdown` | | | | ✓ |
| `zig` | ✓ | ✓ | | |
| `julia` | ✓ | | | |
| `fortran` | ✓ | ✓ | | |
| `haskell` | ✓ | | | ✓ |
| `ocaml` | ✓ | ✓ | | ✓ |
| `solidity` | ✓ | ✓ | | ✓ |

The exact AST node types per language live in `server/internal/chunker/chunker.go` (`defaultRegistry`). File-extension mapping lives in `server/internal/langdetect/langdetect.go`.

Expand Down Expand Up @@ -66,7 +66,7 @@ These produce sliding-window chunks. Adding semantic chunking is a one-map-entry

`erlang, elixir, commonlisp, svelte, graphql, hcl (terraform), cmake, dockerfile, regex, xml, make`

PRs welcome — verify node names with `gotreesitter`'s `cmd/tsquery` against a representative fixture before adding.
PRs welcome — the node names in `defaultRegistry` must match what the grammar actually emits, and `chunker_test.go` walks the registry against the compiled grammars to catch a typo without needing a fixture file per language.

## How the chunker decides

Expand Down
10 changes: 6 additions & 4 deletions doc/TEAM_DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,10 +251,12 @@ docker compose pull
docker compose up -d
```

> **Reindex after a server upgrade.** Until the parsing/chunking/embedding
> pipeline stabilizes, an upgrade can change how code is embedded. Trigger a
> reindex (dashboard or `cix reindex`) so every project lands on the new
> pipeline. Within a version, search is consistent once reindexed.
> **An upgrade does not normally need a reindex.** Existing indexes keep
> working, and 0.13's move of the vectors into SQLite imports them for you on
> first boot without re-embedding anything. The cases that do call for one —
> the embedding model changing, or a release that backfills new chunk-level
> data — are listed in [`UPDATES.md`](UPDATES.md#3-reindex-after-an-upgrade),
> and the dashboard flags affected projects itself.

Server and CLI are released on **independent tag streams** (`server/vX.Y.Z`
vs CLI tags) and are wire-compatible across a minor skew — you don't have to
Expand Down
2 changes: 1 addition & 1 deletion doc/UPDATES.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Upgrading the **server** can require a reindex in two cases:
model (or you change it yourself via Dashboard → Server →
Embedding model), every project becomes stale. The dashboard's
drift indicator paints affected projects red with a "Stale model"
badge until you reindex. See README's *Drift indicator* section.
badge until you reindex. See [`DASHBOARD.md`](DASHBOARD.md#drift-indicator).
2. **Schema migration adds chunk-level data.** Releases that backfill
new chunk metadata (e.g. the FTS5 mirror introduced by `f00e3d3`)
may prompt the dashboard to recommend a reindex on existing
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { formatBytes } from '@/lib/formatBytes';
import { formatDateTime, formatRelative } from '@/lib/formatDate';

// Metadata that would otherwise need an SSH session: which embedding model
// produced the vectors, where this project's SQLite and chromem-go state
// produced the vectors, where this project's SQLite and vector-store state
// lives, and how big both are. Storage fields are nullable — an
// embeddings-disabled server has no resolvable paths, and "—" beats "0 B".
//
Expand Down
6 changes: 3 additions & 3 deletions server/internal/db/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -417,7 +417,7 @@ CREATE INDEX IF NOT EXISTS idx_call_edges_caller ON call_edges(caller_symbol);
CREATE INDEX IF NOT EXISTS idx_call_edges_callee ON call_edges(callee_symbol);

-- PR14 dropped the workspaces communities/community_members tables.
-- Workspace search is now a weighted fan-out across per-project chromem
-- Workspace search is now a weighted fan-out across per-project vector
-- collections (no Louvain, no centroid index). migrateDropCommunities
-- DROPs the tables on upgrade for installs that ran any of PR5..PR12.

Expand All @@ -441,8 +441,8 @@ CREATE INDEX IF NOT EXISTS idx_chunks_meta_project_file
CREATE INDEX IF NOT EXISTS idx_chunks_meta_project
ON chunks_meta(project_path);

-- chunks_fts is the BM25-searchable side, parallel to chromem-go's dense
-- vector store. Workspace search runs both in parallel per project then
-- chunks_fts is the BM25-searchable side, parallel to the dense vector
-- store. Workspace search runs both in parallel per project then
-- fuses by RRF; project-relevance gating uses BM25 signal to drop repos
-- that share no token with the query (the dense-only fan-out leaks
-- semantically-distant repos as false positives).
Expand Down
6 changes: 3 additions & 3 deletions server/internal/embeddings/provider/ollama/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,9 @@ func (p *Provider) ID() string {
return "ollama:" + p.cfg.Model
}

// Dimension returns 0 — the vector store infers dimension from the
// first upsert (chromem-go behaviour) and CodeRankEmbed-Q8 reports
// it on first call.
// Dimension returns 0 — the vector store takes whatever width the first
// upsert carries (each namespace holds one model's vectors, so they cannot
// mix) and CodeRankEmbed-Q8 reports it on first call.
func (p *Provider) Dimension() int { return 0 }

// SupportsTokenize is true: llama-server exposes /tokenize.
Expand Down
4 changes: 2 additions & 2 deletions server/internal/embeddings/provider/voyage/voyage.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
// (256/512/1024/2048). Part of Provider.ID() because changing it
// invalidates the existing index.
// - output_dtype: float|int8 (binary/ubinary are out of scope —
// chromem-go has no hamming search). For int8 the server returns
// the vector store has no hamming search). For int8 the server returns
// a list of integers per dimension; we dequantize to float32 in
// this package before returning vectors to the vector store.
//
Expand Down Expand Up @@ -734,7 +734,7 @@ func (p *Provider) embed(ctx context.Context, texts []string, inputType string)
// the approximate unit-norm float representation.
//
// This is the only place in the codebase that handles int8 quantized
// embeddings; chromem-go and the search path both work exclusively
// embeddings; the vector store and the search path both work exclusively
// in float32.
func dequantize(raw json.RawMessage, dtype string) ([]float32, error) {
switch dtype {
Expand Down
2 changes: 1 addition & 1 deletion server/internal/httpapi/router.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ type Deps struct {
// server is started with CIX_EMBEDDINGS_ENABLED=false (e.g. in router
// tests). Phase 5 uses it for semantic search.
EmbeddingSvc EmbeddingsQuerier
// VectorStore is the chromem-go backed vector store (Phase 4). Nil-safe:
// VectorStore is the SQLite-backed vector store. Nil-safe:
// semantic search returns empty results when absent. Typed as the
// vectorstore.Interface so production can supply a *vectorstore.Holder
// (swappable on provider switch) while tests pass a raw *Store.
Expand Down
10 changes: 5 additions & 5 deletions server/internal/httpapi/workspacesearch.go
Original file line number Diff line number Diff line change
Expand Up @@ -119,14 +119,14 @@ type projectHits struct {
// WorkspaceSearch — GET /api/v1/workspaces/{id}/search.
//
// Hybrid BM25+dense fan-out. Each project runs two queries in
// parallel: dense (chromem cosine) and sparse (SQLite FTS5 BM25 over
// parallel: dense (vector-store cosine) and sparse (SQLite FTS5 BM25 over
// chunks_fts). Per project, the two ranked lists are fused via
// Reciprocal Rank Fusion. Across projects, an α-blended candidacy
// score (with per-query min-max normalization on both signals) plus
// a relative threshold (`candidacy ≥ best × 0.4`) keeps the result
// set focused on repos that actually share vocabulary or semantics
// with the query — pure-dense fan-out leaked every workspace repo at
// noise-level cosine similarity, since chromem returns the N nearest
// noise-level cosine similarity, since a top-K scan returns the N nearest
// vectors regardless of how far away "nearest" actually is.
//
// Observed pre-hybrid: in a workspace of N repos, the repos that
Expand Down Expand Up @@ -756,9 +756,9 @@ func projectLabel(projectPath string) string {
return projectPath
}

// round4 rounds f to 4 decimal places — matches the chunk-side
// rounding chromem already applies, so scores in the response look
// consistent across nested fields.
// round4 rounds f to 4 decimal places — matches the chunk-side rounding the
// vector store already applies, so scores in the response look consistent
// across nested fields.
func round4(f float32) float32 {
if math.IsNaN(float64(f)) {
return 0
Expand Down
7 changes: 4 additions & 3 deletions server/internal/indexer/indexer.go
Original file line number Diff line number Diff line change
Expand Up @@ -818,8 +818,9 @@ func (s *Service) ProcessFilesStreaming(
}

// ---- Stage 3: WRITE (serial, ordered) --------------------------------
// Vector-store + per-file DB writes run on this single goroutine: chromem
// is thread-safe but serialising keeps SQLite's WAL writer uncontended and
// Vector-store + per-file DB writes run on this single goroutine: the
// store is thread-safe, but serialising keeps SQLite's WAL writer
// uncontended and
// preserves deterministic progress-event ordering. Each write is local and
// sub-ms, so serialising costs nothing next to the (now parallel) embeds.
for _, p := range prep {
Expand Down Expand Up @@ -868,7 +869,7 @@ func (s *Service) ProcessFilesStreaming(
}
}

// Build chunksfts payload from the same chunks pushed to chromem.
// Build chunksfts payload from the same chunks pushed to the vector store.
ftsChunks := make([]chunksfts.Chunk, len(p.vsChunks))
for i, c := range p.vsChunks {
ftsChunks[i] = chunksfts.Chunk{
Expand Down
2 changes: 1 addition & 1 deletion server/internal/maintenance/analyze.go
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ func (s *Service) Analyze(ctx context.Context) (*Analysis, error) {
// scanState is the live picture every scanner reconciles against, read once so
// five scanners do not issue the same queries five times.
type scanState struct {
// liveCollections maps chromem collection name -> host_path, for every
// liveCollections maps vector-store collection name -> host_path, for every
// row currently in `projects`.
liveCollections map[string]string
// liveHashes is the set of projects.HashPath values currently in use.
Expand Down
3 changes: 2 additions & 1 deletion server/internal/vectorstore/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
// embedding namespace and a query streams the collection past a dot product
// with a top-K heap. Opening is one file open (sub-millisecond), resident
// memory is a couple of page-cache buffers per active query, and the cost is
// search latency: roughly 3x chromem's on the same data.
// search latency: roughly 4x chromem's on the same data — see the measured
// table in doc/VECTORSTORE.md.
//
// # Frozen contracts
//
Expand Down
Loading