Conversation
Full replacement of the dashboard UI. Every screen is rebuilt on a small token layer instead of ad-hoc utility classes, so the look is enforced by the build rather than by review. The system, in five rules: cream surfaces and ink outlines (no greys, no blur shadows); cards round at 12px and controls stay square, with borderRadius overridden globally to 0 so a stray `rounded-md` cannot quietly drift; depth is a hard 4px offset shadow, one per screen region; mono for every machine value, right-aligned; status is a 9x9 square plus a word, never colour alone. Surfaces are a measured tonal ladder rather than a set of moods — field > surface > canvas > head, each a fixed CIE L* step apart, the device Material 3 uses for surface-container roles and Carbon for layers. A control the user types into gets its own `field` tone: sharing the card's fill made a form read as a grid of identical outlines with nothing for the eye to land on. Text is three tones that all clear WCAG AA on every surface they are allowed on, plus `faint`, which does not and is therefore limited to placeholders and disabled text. Also in here: - Palette moved to RGB channel vars so Tailwind opacity modifiers keep working while `.dark` swaps the whole thing. - JetBrains Mono self-hosted as subset woff2 with unicode-range; the icon library (lucide-react) and @radix-ui/react-scroll-area are dropped. - The status bar spans the full window as a sibling of the sidebar+main row, and pages publish their middle-slot fact through useStatusFact(). - Fixed invalid nesting that shipped before: buttons inside buttons in the search result header, and buttons inside an anchor in the project card. - Fixed card corners reading as cut off — a square filled header strip painting over a 12px arc needs overflow-hidden on the card. - Workspace projects are a table like every other list in the dashboard, not a stack of cards with the facts crammed into one wrapped line. - devmock/ (gitignored) boots the real App against a mock fetch, so every authenticated screen can be reviewed without a server or a login. ?at= picks the route, ?theme= pins light/dark. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`make run` depends on bundle depends on fetch-llama, so every single server restart re-downloaded the pinned llama.cpp release from GitHub, re-extracted it, copied 52 MB into the bundle and deep-re-signed it, then ran tsc and vite to emit the same dashboard assets. On a fast link that is 5.4s of nothing; when GitHub is slow it was measured at 114s. Restart the dev server a few times a minute and that is the whole session. Each step now skips when its output would be identical: - fetch-llama.sh stamps DEST_DIR with the version it built from and exits early when the stamp matches and llama-server is still there. The stamp is removed before extraction and rewritten only after every sanity check passes, so a half-finished fetch is never mistaken for a good one. Verified archives are also cached outside the repo, in ~/.cache/cix/llama, which is what makes a forced re-fetch or a fresh clone cost no network. - bundle compares that stamp against the one in the bundle and skips the copy and the codesign together — they are tied, because `cp -R` creates files Sequoia's amfid treats as untrusted. - dashboard-build hashes every input the build reads and skips when the digest matches the last successful build. Escape hatches: LLAMA_FORCE=1 (which implies a bundle rebuild, since a forced re-fetch means the operator distrusts what is staged), BUNDLE_FORCE=1, DASHBOARD_FORCE=1, and `make dashboard-clean`, which now drops the stamp along with the output. Warm `make bundle` goes from 5.4s to 0.5s. LLAMA_STRICT=1 still fails on an unpinned version, cache or no cache — the strict gate is checked before the skip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `cp .env .env.bak-$(date …)` before editing leaves a file with the same API key and bootstrap password in it, one `git add -A` away from being committed. `.env` was ignored; its backups were not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Saving on /server made the embedding-model card, the runtime-params card and the sidecar rail disappear until a manual reload. `/status` reports the LIVE provider — Service.CurrentKind(), which returns "" whenever the provider is nil, i.e. exactly the window in which the sidecar is being torn down and rebuilt. The page read that as "we are on a remote backend" and unmounted every ollama-only card, because `?? 'ollama'` guards null and undefined but not "". useRestartSidecar then invalidates this very query on settle, so the refetch landed at the emptiest possible moment and the result sat in cache for the full 30s poll interval — hence the reload. "" is unknown, not remote. Hold the last kind actually reported and only switch layout on a positively-reported one, so a transient blank changes nothing. A blank seen before any real kind still falls back to ollama, which is what the old default did and what a fresh dashboard should show. Verified against a devmock that reports the blank kind: before, /server dropped from five cards to two; after, it stays at five. A genuine openai still collapses to the remote-backend layout with a plain "Save", and a blank arriving *after* openai no longer flips the page to ollama. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat(dashboard): rewrite the frontend on the cream & ink design system
cix-server idles at 4-5 GB RSS on a large index because chromem-go is an
in-memory vector database: NewPersistentDB decodes every document of every
collection at startup and never evicts. Idle memory therefore tracks the size
of the whole index, not the working set.
Most of that was unreachable. projects.Delete removed the row and let FK
CASCADE take the chunks and symbols, but nothing ever dropped the vector
collection or the cloned checkout. On the box this was found, 40 of 87
collections (447k of 751k documents, 2.4 GB) belonged to projects that no
longer existed, and 86% of the clone directory was likewise orphaned — all of
it resident for the life of the process.
Two halves:
* projects.Delete gains an optional Artifacts hook so deleting a project
also drops its collection and checkout, and DeleteProject clears queued
clone/index jobs first so a job cannot re-create what was just removed.
Artifact failures are wrapped in ErrArtifactCleanup — the row is already
gone, so the caller logs rather than reporting a failed delete.
* A new internal/maintenance package plus three admin endpoints report what
the server is using and reclaim the backlog, in five separately selectable
categories. Analyze is synchronous with a single-flight cache; the
analysis id it returns is an identity for "what the admin was shown", not
a safety mechanism — every item is re-validated against live state
immediately before deletion, so a project that reappears in between is
skipped rather than wiped.
Safety, since this deletes things: the active namespace, live collections, the
active model and in-flight downloads are never candidates; an inactive
namespace larger than the active one is listed but not pre-selected, because
that pattern means the server booted with the wrong embedding model rather
than that the real index is garbage; and orphaned collections are held back
entirely while a clone/index job runs, since a queued job identifies its
target by a SHA-1 path hash and a collection by an MD5 of the same path, with
no surviving row to join them through.
Also fixes enrichProjectStorage, which built the per-project vector directory
by joining the collection NAME onto the namespace path. chromem stores a
collection under a hash of that name, so the path never existed and
chroma_size_bytes was silently omitted on every project. The store now
answers where its own collections live, pinned by a test against the real
on-disk layout.
Verified against a copy of a real 751k-document index: heap 4.12 GB -> 1.28 GB
after reclaiming 40 collections, with the surviving document count matching
chunks_meta exactly. Note that debug.FreeOSMemory returns pages to the OS on
Linux but only marks them reclaimable on darwin, so a local macOS run shows
the heap fall while RSS holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Surfaces what the server is holding — Go heap, resident memory, resident vector documents and the four storage locations — and drives the two-phase cleanup: Clean analyses first, the result is rendered in place with one checkbox per category, and only then does a confirmation dialog spell out exactly what the selected boxes will delete. Analyze is a mutation rather than a query on purpose: it is admin-triggered, takes seconds, and has a server-side side effect. Its data IS the rendered analysis, so reset() is how the section forgets a spent one — which it does after every clean, and on the 409 that means the analysis expired. Category labels, descriptions and which boxes start ticked all come from the server, so the rules about what is safe to reclaim are decided in one place instead of being restated in React. Unused models arrive unticked and flagged destructive, and the dialog then warns about the re-download cost. Also hoists formatBytes into src/lib. It existed as two private copies that had already drifted on their unit ceiling — a difference a screen reporting multi-gigabyte vector stores would have made visible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine findings, all real: DeleteProject dequeued clone/index work but never cancelled a session already running. Deleting a `running` job row does not stop the goroutine behind it, so an index in flight would keep writing and re-create the collection and checkout moments after the artifact cleanup removed them — the very race this feature closes, one floor up. Now mirrors ForceStopIndex and calls CancelIndexing too. RemoveCloneDir took no lock while the file handlers take a read lock and the clone worker takes a write one, precisely so a tree is not read or rewritten mid-change. It now takes the same write lock before os.RemoveAll. A single-flight follower returned whatever sat in the cache once the leader finished, without checking that it was the leader's result or still inside its TTL. A failed leader could hand back a twenty-minute-old analysis: 200 with a stale picture whose id then 409s on clean. Followers now require both. Clean read the analysis and invalidated it separately, so two concurrent calls with the same id both ran. The deletions are idempotent, but the second would report bytes the first had already reclaimed. Consuming the analysis under the same lock as the read makes an id good for exactly one clean — and the selection is validated first, so a typo no longer costs a scan. Usage had neither single-flight nor a cancellable walk: DirSizeBytes ignored the context entirely, so an abandoned request still walked the whole tree, and N concurrent callers meant N full walks. It now shares one walk behind a short TTL and checks the context as it goes. The GGUF cache location came only from the embedding service, so with embeddings disabled both the model-cache disk row and the unused-models category silently vanished while the files sat there. Falls back to config for reporting; deletion is still gated on knowing the active model. Empty namespace directories were skipped by the leaf walk, so a clean could report "nothing left" while leaving empty directories behind. Now listed — but only when completely empty, since loose files in a directory we do not understand are not ours to delete. Also: the primary button said "Clean" while calling analyze, which is the last place a destructive feature should let a label disagree with its action; and formatBytes rendered 0 as an em dash, which is right for a measurement that could not be taken but turns "Reclaimed 0 B" into "Reclaimed —". Totals now opt into a real zero. Adds tests for each: single-use-under-concurrency, a rejected selection leaving the analysis usable, follower staleness, and DeleteProject taking its collection and checkout with it. The anonymous-access test now uses each route's real method — GET on a POST-only path returned 405 before auth ran and proved nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The follower path in Analyze re-entered Analyze when the leader produced nothing usable, while Usage had already been written to fall through instead, with a comment explaining why. Two neighbouring functions solving the same problem two different ways is a maintenance trap on its own, and the recursive one is the wrong half of the pair: `done` is nil by then, so each failed round adds a stack frame per goroutine, and a scan that keeps failing under a steady stream of requests grows those stacks without bound. Both now fall through and compute for themselves. Beyond the stack, that is also better behaviour under failure: each caller gets its own error instead of queueing behind a leader that cannot deliver. Two tests, because the obvious one is not enough. The liveness test — every caller returns promptly against a closed database — passes with the recursion still in place, so it proves nothing about the shape of the retry. The second one samples the call stack from inside the scan and asserts no goroutine is ever two Analyze frames deep. Restoring the recursion to check: it reached 7 frames and failed, while the liveness test stayed green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…urces The page had grown two unrelated jobs: editing embedding configuration, and reading what the process is using. They share nothing — different data, no common action, and one of them is read-only until you deliberately reach for a destructive button. A tab each. "Save & restart" now renders only on the tab it applies to. A primary action hovering over a storage screen is noise, and worse, it reads as though it might act on what is shown there. Unsaved edits survive a tab switch while the button does not, so the Runtime tab carries a dot when the draft is dirty — the status bar already says "unsaved changes", but the tab that owns them should say which tab they are in. Resources is now independent of the runtime-config query, which matters more than it sounds: memory and disk are exactly what an operator wants to read when the rest of this page is failing to load. Previously a failed config fetch replaced the entire page with an error. That path was broken anyway. The loading branch tested `cfg.isLoading || !draft`, and the draft is only ever built from a successful fetch — so on failure `!draft` was true, the skeletons rendered forever and the error callout underneath was unreachable. Error is checked first now. Also renames the card head from "Resources" to "Storage & memory": with the tab already named Resources, a card head repeating its container tells the reader nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported as "the button does not react to changes, saving is impossible".
It was in the page header, above both tabs, which made it look like it
belonged to whatever was on screen — and it read as broken twice over.
First, placement. A header action only makes sense while it applies to the
whole page. Once the page grew tabs it either had to follow the active tab,
appearing and vanishing under the title, or hover over a storage screen it
has nothing to do with. It now sits in a sticky bar at the top of the tab
that owns it, directly above the form it acts on, and still cannot be
scrolled away — which was the original reason for putting it in the header.
Second, and the actual cause of "impossible to save": the button is disabled
whenever the sidecar reports state=disabled, no matter how many fields have
been edited. That is correct — with CIX_EMBEDDINGS_ENABLED=false there is no
service to apply anything to, and the server refuses a provider switch for
the same reason — but the control said none of it. The bar now carries the
reason next to the button, and names the env var, because a greyed-out
control with no explanation is indistinguishable from a broken one.
Worth stating since it comes up with it: switching provider is a SEPARATE
control inside the provider card and is not gated on this state. When
embeddings are off it fails server-side instead (embeddings.ErrDisabled), so
both roads lead back to the env var — which is what the message now says.
The bar doubles as the change counter ("3 unsaved changes" / "no changes"),
so the state that drives the button is visible next to it rather than only in
the status bar.
Also updates the design-system note in ui/page.tsx, which cited this very
button as the argument for header actions and would otherwise now contradict
the code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bar was pinned to the top of the Runtime settings tab so it could not be scrolled away. In practice cards slid underneath a floating strip and the result read as a rendering fault rather than as a deliberate toolbar. It now sits in normal flow at the end of the tab, above a rule, closing the form it belongs to. The property it was pinned for does not buy much here: the form is short enough that its end is where you arrive anyway, and the change counter it carries is duplicated in the status bar for anyone who has scrolled past it. Behaviour is unchanged — same enable/disable rules, same reason text next to the button when it is blocked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat: Resources tab — report memory/disk usage and reclaim orphaned storage
SQLite never shrinks a file when rows are deleted; the pages go on the
freelist and are reused. The dev instance's database is 8.86 GB of which
48% is freelist. This is the first half of a feature that reports that
waste and reclaims it.
Compaction has to be built on VACUUM INTO, and VACUUM INTO produces a
*snapshot*: measured on a clone of the real database, 18 200 rows were
written during a 76 s copy and 850 of them reached the copy. Adopting
such a copy would silently discard the rest. Everything here is shaped
by that: writes are frozen for the duration, and the copy is adopted at
boot rather than under a running server, because thirteen services hold
the pool and none of them can be repointed at a new file.
internal/dbmaint carries the state that cannot live in the database it
is replacing:
- state.go the journal, written tmp+fsync+rename+fsync-dir, plus an
append-only event trail that spans the restart
- reconcile.go the file set decides, the journal only explains: every
combination of (files present, journal phase) maps to
exactly one action, and the original is deleted last
- verify.go a copy is fingerprinted against the frozen source before
anything irreversible happens to the original
- stats.go size, waste, advice; header reads only, safe to poll
- reclaim.go incremental reclaim and WAL checkpoint
Reconcile runs from main() and from the offline password reset, in the
same slot as the legacy adoption and before anything opens the file.
Two things measurement corrected. incremental_vacuum does not shrink the
file in WAL mode until a checkpoint follows it, so the test that proves
the file shrinks uses a reclaim small enough to stay under SQLite's
auto-checkpoint threshold — without that bound it passed with the
checkpoint removed. And wal_checkpoint(TRUNCATE) reports the state after
truncation, so a fully successful checkpoint reports zero pages; the byte
figures are the honest ones.
The build is red until the compaction and schedule handlers land: the
spec is written and regenerated, so the generated interface demands three
methods that do not exist yet.
…he copy Adds the working half of database compaction: the endpoints, the write freeze, the compactor itself, and the restart that adopts the result. The freeze is three layers, and the ordering is the whole design. A Go route gate refuses writes in microseconds; background writers are stopped and drained; and only then does the compactor hold a write transaction as a backstop. Leading with the SQL lock alone would be a disaster — measured, a refused write sits in SQLite's busy handler for 5.06 s holding one of eight pool connections, and eight of those stall reads too, which is exactly what the feature promises will keep working. The gate classifies per route, never by method. Five search endpoints plus file and tree are POSTs that only read, and a method-based gate would refuse them. It is installed outside requireAuth because the GitHub webhook is a write that skips authentication. /health now answers 200 without touching the database while frozen. It pinged the database with a one-second budget, so a freeze would have failed the container healthcheck and let a restart policy kill the compaction it was running. Compaction copies under the freeze, verifies the copy against a fingerprint of the frozen source, journals the intent and asks the process to re-execute itself. run() now returns a restart flag so the exec happens from main() after every deferred cleanup has unwound: the listener has to be closed or the new image cannot bind, and the embeddings sidecar has to be reaped or the new image collides with it. The copy runs on a dedicated pool with no auto_vacuum in its DSN — modernc applies DSN pragmas to every connection and VACUUM INTO carries the pending mode into the copy, so a shared pool would silently produce an incremental copy on every run regardless of what was asked for. Everything that can refuse happens before the first Quiesce: past that point the job queue's one-shot lifecycle means there is no way back except a restart, so a failed compaction also restarts, cleanly, on the untouched original. Also: migration 19 and the schedule it configures, with outcomes read from the journal rather than the table because a compaction replaces the database; CIX_DB_MAINTENANCE_* for compose-driven deployments; and Stop on the poll scheduler, which enqueues jobs and so had to become drainable rather than merely cancellable.
The block lives in the Resources tab and reports what the server now knows about its own database: file size, how much of it is empty space, the write-ahead log, and the reclaim mode read live from the file rather than from any stored setting. Three actions, deliberately distinct. Compact rebuilds and restarts. Reclaim returns free pages without a window and only appears when the database can do it. Checkpoint folds the log back in and only appears when there is a log worth folding. The incremental-reclaim toggle opens the same confirmation as Compact, because on a populated database that is what it costs. A switch that silently triggers a rebuild and a restart would be the most surprising thing in this feature, so the dialog says which of the two reasons brought the admin there and describes the same interruption either way. The dialog is where the honesty lives: read-only for an estimated duration rather than down, search and reads still served, indexing and new logins refused, then a restart. It also states that nothing is lost if the machine dies mid-operation, because "rebuild the database" reads as dangerous and here it genuinely is not. The banner is mounted next to UpdateBanner and owns its own polling rather than going through react-query. It has a requirement nothing else in the dashboard has: it must keep rendering while its own backend goes away, since the operation it reports restarts the server. A failed poll is the expected middle of the operation, so it renders "reconnecting" instead of unmounting — react-query's two retries would give up and leave stale cached data on screen at exactly the wrong moment. It reads the public status endpoint directly, which is outside /api/v1 and outside auth for the same reason. Schedules render "never run" rather than an epoch date, and say when the values they show are defaults nobody has configured — an upgraded server comes up that way by design.
…ent the feature The default was held behind a measurement, because incremental mode maintains pointer-map pages and this server's hot path is bulk-inserting chunk and symbol rows. On an indexing-shaped workload — 120k wide rows in batched transactions, then a bulk delete — it costs +1.5% on insert, with no measurable difference to deletes or file size. That is a cheap price for a database that can return its own space, so fresh files get it. Existing databases are untouched: SQLite silently ignores the pragma on a file that already has tables, so an upgrade changes nothing until an admin asks for a compaction. That pragma now being on every connection makes a real hazard live, so it is tested rather than assumed. A connection carries a pending auto_vacuum change into any VACUUM INTO it runs, and measured directly, a mode-none database opened with the pragma only in its DSN produces an incremental copy. Copying through the shared pool would therefore convert a legacy database on every run and make enable_incremental decorative. The compactor is guarded twice — its own pool omits the pragma and the mode is set explicitly — and the test fails only when both guards are removed, which is the honest statement of what protects what. Two earlier single-guard mutations passed, which is how the second guard was found to be redundant-but-cheap rather than load-bearing. Also fixes two things the end-to-end run surfaced: an idle journal serialised started_at as year 1, which reads as a bug rather than as "nothing has happened", so it is now omitted; and durations were logged as raw nanoseconds. doc/DATABASE_MAINTENANCE.md documents what an admin is agreeing to.
… run costs Running the whole pipeline against a copy of a production database — 8.25 GB, 47% waste — found thirty seconds of pure downtime that bought nothing. PRAGMA quick_check reads every page, and the boot-time verification runs before the listener binds. On the 4.5 GB copy it did not finish inside its 30-second budget, so the deadline fired and the result was discarded. The server was unavailable for that whole time for a check whose answer was never used. It is now skipped above 512 MB. That is safe because quick_check was never what did the work: a bad copy is caught by the fingerprint — row counts taken from the source under the write freeze, which neither a copy of another database nor an earlier state of this one can match — and by the header's own claim about the file's length, which catches truncation. A test lowers the threshold so the skip path is exercised and proves the fingerprint still rejects the wrong database. The run itself behaved as designed. Throughout the 95-second copy, reads answered 200, writes answered 503, and /health stayed 200 — 24 probe writes refused, none admitted. The process kept its pid across the re-exec. Afterwards: 8.25 GB → 4.18 GB, 4.07 GB returned, and 48 projects / 297 563 chunks / 2 users unchanged.
The toggle only turned incremental reclaim on. Turning it off was left out because it costs the same rebuild — which was the wrong reason to remove a capability. The whole design is built on stating the price and letting the admin decide, and this was the one place that decided for them. The request field becomes tri-state as a result: a boolean enable_incremental cannot express "switch it off", so compaction now takes auto_vacuum of keep, none or incremental, defaulting to keep. That also makes the common case explicit rather than implied — a compaction that is only about reclaiming space says so. The dialog names the direction and what follows from it: off means space comes back only by compacting again; on means free pages return without a rebuild, at the measured 1.5% on indexing writes.
The reclaim mode was a parameter on the compaction request, which forced an invented third value — "keep" — to express "I am asking for space back and have no opinion about settings". That third value was a symptom: two unrelated decisions had been fused into one endpoint, and the fusion needed papering over. They are now separate. POST /database/compact reclaims space and leaves the mode exactly as it found it — no request body at all. PUT /database/auto-vacuum sets the mode, and asking for the mode the database is already in does nothing and answers 200 rather than rebuilding anything. Underneath they share startRebuild, because changing the mode of a populated database requires a rebuild. That is a reason for the setting to call into compaction; it was never a reason for compaction to carry a setting.
The CIX_DB_MAINTENANCE_* layer never went through ValidateSchedule. Only the API path did, and config.Load checks nothing beyond the mode enum — so a compose file could configure a schedule the server would then honour in part. Half a window is the case that matters. DueNow consults the window only when both bounds are set, so a WINDOW_START_HOUR with no WINDOW_END_HOUR does not narrow the schedule, it removes it: a full compaction meant for 03:00 freezes the server and restarts it in the middle of the afternoon. An interval of zero fires on every tick. A percentage of 150 never fires and never says why. The env layer is now validated at startup, where an operator can see it, and the scheduler declines to act on a resolved schedule that does not validate. Refusing is the right failure: the alternative is a maintenance window the admin did not ask for. Also: the journal has one owner at a time. A scheduled reclaim finishing during the quiesce that precedes a rebuild used to Save "reclaim done" straight over the rebuild's "preparing" — the dashboard would report a finished reclaim for a server that was frozen and about to restart, and a crash in that window left the reconciler believing the compaction never started. The reclaim now checks for an owner while holding the lock it would need to become one, and a reclaim interrupted by shutdown is not journalled as a failure at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The swap deletes the displaced original's -wal, and it has to: a log left under the old name shadows whatever file takes that name next. But an uncheckpointed log holds committed transactions the database file itself does not, and the displaced original is exactly what the rollback path restores if the copy is then lost. So a failed checkpoint was warned about and stepped over, and the two facts only met in the rare chain — checkpoint fails, swap proceeds, copy is lost, next boot rolls back — where the restored database is quietly short of its last writes. Nothing else has the file open at that point in the boot, so a failure there is real. It now aborts the adoption: the database is kept exactly as it is, the copy is discarded, and a re-run costs a few minutes. Also bound the public read of the event trail. Status() parsed the whole of maintenance.log on every call, /maintenance/status is unauthenticated, and nothing ever truncated the file — so the cost of an anonymous request grew with the lifetime of the server, at whatever rate the caller liked. Only the last 128 KB is read now, and the trail rolls over at 4 MB keeping one generation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mode before the write lock Four faults in one path. **The claim came after the preflight.** startRebuild checked s.running, released the lock, then spent tens of milliseconds on pragmas, a job count and an fsynced journal write before setting the flag. A double-clicked button, or a scheduled full run landing on a manual one, put two rebuilds past the same gate — and the loser's cleanup deletes the winner's half-written copy and thaws the write gate in the middle of its snapshot. Writes accepted after that thaw are absent from a copy that is then adopted, which is the one outcome this whole feature is built to prevent. The flag is now claimed in the same critical section as the check, and given back by every path that refuses. **Setting the reclaim mode needs a write transaction.** It was being set on the copy connection *after* the freezer had taken BEGIN IMMEDIATE, so it blocked for the full busy timeout and failed the run with SQLITE_BUSY. It only showed on an incremental source — asking a mode-none database for mode none is short-circuited before any lock is needed, and a legacy database is what the end-to-end run happened to use. So every rebuild of a database created by a recent build stalled 30 seconds and gave up. The copy connection is now opened and its mode pinned before the lock exists; measured on the same fixture, 30.3 s and failed becomes 42 ms and ready. **A full auto-vacuum database was demoted to none.** copyInto mapped everything but incremental to NONE, so a compaction of a database in full mode silently changed a setting nobody had asked about — while the journal recorded the mode it had not produced. **blocked_reason was declared, documented, consumed and never assigned.** The dashboard disables the Compact control on it, so it was always enabled and the refusal arrived as a toast after the click. It is now computed from the same checks the request refuses on. And the guard those checks run was blind to half the work it was guarding against: ActiveJobs counted the jobs table, but a CLI push creates no row — the three-phase protocol lives in the indexer's session map. Compaction could start mid-index, 503 the next /index/files, and in full mode restart the process mid-protocol. Both sources are counted now, from one constructor, because the two hand-built copies had already drifted. Also drops the progress ticker: it fsynced the journal every two seconds during the read-only window to record a figure recovery never reads. The copy's size on disk is the progress, and Status() measures it on demand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pragma was in the shared DSN, applied to every connection, on the strength of a claim that turns out to be half true: SQLite ignores auto_vacuum on a populated database *only* when honouring it would mean moving pages — that is, going to or from `none`. Between `full` and `incremental` it applies immediately, at any time. So a database somebody had deliberately put in full auto-vacuum was converted to incremental by nothing more than an upgrade, silently, and the compaction that followed then recorded the converted mode as if it had always been there. The whole point of putting the mode on a switch of its own was that nothing else gets to move it. It is now set once, on a file this build is creating. That has to happen before journal_mode=WAL writes the header — measured: a pragma issued after the main pool is open is ignored whatever order the statements appear in — so a new file is seeded on its own connection first, and an existing file is not opened for the purpose at all. Also widens the spec's auto_vacuum enum: `full` cannot be requested, but a compaction of a database already in that mode preserves it and says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On the restart path a Shutdown timeout returned an error, and main exits on it — before reaching the exec. One slow in-flight request outliving the 10 s budget therefore took the server down permanently with a verified .compact file sitting beside the database and nothing to pick it up: re-executing *is* the restart mechanism here, there is no supervisor behind it. The timeout is now logged, the listener forced closed so the new image can bind, and the exec happens anyway. The copy is verified and journalled by that point; the reconciler does the rest at boot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two ways it reported a compaction that was not happening. A non-2xx from /maintenance/status resolved straight to "unreachable", with none of the guard the network-error path has. A proxy answering 404 for a route it does not know, on the very first poll, rendered "The server is restarting to adopt the compacted database — reconnecting…" and locked the page into a two-second poll loop on a server where nothing had ever run. A failed request now only means "restarting" if an operation had actually been seen. And the terminal strip replayed forever. It was keyed on component mount with no recency check, and the journal keeps its last entry by design — it is the only record of an operation whose result could not be written to the database it replaced. So every page load for the rest of the server's life showed "Database compacted", and a nightly scheduled reclaim brought it back nightly — worded as a compaction, because the strip ignored the operation's kind. It now appears only while the outcome is still news, and says which operation it is describing. Also collapses three hand-written definitions of "an active phase" — already diverging across the banner, the polling interval and the disabled state of the controls — into one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reclaim schedule was an interval plus an optional hour window, and
"every day at 00:00" was expressible only by accident: interval 24 with a
window of 0→1, fired by an hourly tick somewhere inside that hour. Having
to be told that is the diagnosis.
An interval is also the wrong shape. It is measured from the last run, so
one manual compaction at 18:00 moves every subsequent nightly run to
18:00, and it drifts from there. cron is anchored to the clock, which is
what an operator means by "every night".
So: a crontab expression, and a scheduler that is general rather than a
database feature. internal/schedule is a table of named tasks, one
goroutine watching the clock, and a handler called in-process when a task
is due. Polling, cleanup and update checks can hang off the same
machinery; the database's reclaim and compaction are simply its first two
entries, with a switch and an expression each.
It is deliberately not a job queue. This server already has one — the
jobs table, with retries, dedupe and a worker — and a second persistence
model beside it would mean two places to look when something did not run.
A task that wants durable retryable work enqueues into `jobs`; that is
the seam. Compaction is why the trigger could not live in that queue in
the first place: it drains the queue as part of taking the server
read-only, so a trigger inside it would be draining itself.
**The library is not trusted on its own.** Both maintained zero-dependency
cron parsers evaluated get next-fire-time wrong on expressions whose day
does not exist in every month:
NextTickAfter("0 0 31 * *", 2026-08-31) -> 2026-10-02
NextTickAfter("0 0 29 2 *", 2026-08-13) -> 2027-03-04
Neither answer satisfies its own expression, and go-quartz has the same
class of defect on 29 February — so this is a property of the ecosystem,
not of one dependency. gronx's due-check is correct for the same instants,
so the candidate is verified against it and stepped over when the two
disagree. The three cases are pinned as regression tests, and an
expression that can never match at all is refused rather than accepted as
a schedule that silently never fires.
Crontab semantics, decided rather than inherited by accident:
- The next run is computed from the clock, never from when the last one
finished, so a slow run cannot make the schedule drift.
- A run that overruns its own slot loses the slots it ran through instead
of firing a burst afterwards.
- A slot missed while the process was down is skipped for compaction —
noticing at 09:00 would freeze the server in the middle of the working
day — and caught up for reclaim, which costs milliseconds and would
otherwise never run on a laptop that is asleep every night.
- Daylight saving is wall-clock: an hour that does not exist on the spring
forward is skipped, an hour that happens twice fires once. Documented
and pinned rather than fought.
- The slot is claimed on disk *before* the handler runs. That is
correctness, not bookkeeping: compaction re-executes the process as its
final step, and a slot still marked due when the new process starts
would fire it again, and again.
next_runs is computed server-side by the parser that fires them, so the
preview beside the field cannot disagree with what the server does.
Also drops maintenance_settings' schedule columns and, with them, the
configurable thresholds table: what an admin adjusts is *when* a task
runs, and how much waste is worth acting on is a deployment property —
now a built-in default with an environment override, one fewer table and
one fewer form. CIX_DB_MAINTENANCE_CRON replaces the three interval and
window variables, which shipped in the same unreleased change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Resources tab dropped both cards straight into TabsContent with no wrapper, so they butted against each other and read as one card with two headers. Every other multi-card view on the page stacks in a flex column with a gap; this one just skipped it — the stray indentation on the second line was the tell. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The checkpoint control is gone — from the UI, the API, the spec and the docs. It offered to fold the write-ahead log back into the database by hand, and measured on the real instance that log sits at 4.11 MB against a 4.4 GB database: SQLite auto-checkpoints at 1000 pages and keeps it there. Reclaim already checkpoints as part of its work, and so does the swap. The one case where the log does grow is a long-held read transaction — during the compaction copy it grew 79 MB — and in that window the server is frozen and the button unusable, after which the next automatic checkpoint clears it anyway. So it duplicated the automatic behaviour in the ordinary case and was unavailable in the only case worth reaching for it. The log's size stays on the stats strip: those megabytes are real disk that the "File size" figure does not include. Then the review round, blockers first. **Stop did not stop anything.** Handlers ran in detached goroutines and Stop returned as soon as the loop exited — despite promising the compactor that background writers had finished. A reclaim mid-`incremental_vacuum` would sail through the quiesce and go on writing into the database being copied, where neither the HTTP route gate (it holds no request) nor the write lock (taken later) could see it. Handlers are now tracked and joined, bounded by the caller's context. **The "not while a rebuild runs" gate was lost in the rewrite.** The old tick checked it first; the new reclaim task checked mode, thresholds and jobs but not that. Restored on both tasks — `incremental_vacuum` and its checkpoint write straight through the shared pool. **One env variable drove both schedules.** Both tasks were registered with CIX_DB_MAINTENANCE_CRON, so an operator's nightly reclaim time became the compaction's time the moment anybody switched compaction on — a freeze and a restart every night at 03:00 — and Save then persisted it, so clearing the variable no longer helped. The variable now defaults the reclaim task only, and a request that does not mention `cron` no longer freezes the resolved value into the row. **The spring forward swallowed a run.** In Kyiv 03:00 becomes 04:00, so `0 3 * * *` matches no instant that day and wall-clock arithmetic moved to tomorrow: a nightly task silently missing a night, once a year, with nothing in the log — and CatchUp powerless, because no slot was ever armed to be late for. It now fires at the first valid instant after the jump, which is what vixie cron does. The arithmetic is on calendar fields rather than instants: an hour before the transition *as an instant* lands at 02:00, while the hour that vanished is 03:00–04:00. **A run interrupted by a power cut stayed "running" forever.** The status is written before the handler and cleared after it, so nothing would ever correct it on a host that powers off nightly. Stale rows are reconciled at startup, and a run cancelled by shutdown records itself as interrupted. Also: `DefaultEnabled` is asked at resolve time rather than captured at registration, so switching the reclaim mode no longer leaves the default stale until a restart, in either direction; thresholds are pointers end-to-end, since an explicit 0 means "run whenever the schedule says" and a plain int could not tell that from unset; the dashboard's verdict is computed from the resolved thresholds rather than from constants it only claimed to share; and `maintenance_settings` is dropped outright — it was migrated with care and read by nothing. The schedule endpoints now have success-path tests. The route table proved a non-admin cannot reach them; with no registry in the fixture, an admin could not either, so the error mapping and the response envelope were untested. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Migration 20 was amended to drop the table alongside its other work, which is correct for a database that has not run it yet and a no-op for one that has — and this branch already has installs in the second category, where the table survives as a thing an admin can write to that nothing reads. Both migrations are unreleased, so a step of its own costs nothing and is deterministic whichever version of 20 a database happened to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The login form kept its submit button disabled until both state values were non-empty. A browser skips implicit submission when the form's default button is disabled, so any fill React never saw as an input event — a password manager, browser autofill — left the form visibly complete with a dead Enter key, and a dead first click with it. It is now disabled only while the request is in flight; `required` on both inputs is what rejects an actually-empty submit, and it does so without taking the keyboard away. All three auth forms also read their values from the submitted form rather than from state, for the same reason: a field filled without an input event would otherwise post an empty password against a form that looks filled in. State stays the source of truth for rendering; the FormData read is the fallback, so nothing changes when the events do arrive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(dashboard): let Enter submit the auth forms again
… list The loop woke every thirty seconds to ask two daily tasks whether it was midnight yet — 2 880 wakeups a day, each doing one SELECT per task. It now sleeps until the earliest armed run, and a saved schedule nudges it awake rather than waiting out a sleep armed for the old expression. The wait is capped at five minutes anyway, and that cap is the interesting part: a suspended laptop does not advance Go's monotonic clock, so a timer armed for eight hours comes back whenever the lid opens. The "did this slot pass while we were away" grace is therefore larger than the cap, so ordinary lateness can never look like a missed slot to a task that refuses to run late. The rest of the review's cleanup list: - `claim` re-checks `enabled` in its WHERE and reports whether it took the slot. The window between the loop's read and its write is sub-millisecond, but a run of `db.compact` is a frozen server and a restart, and "we had already decided" is not an answer to "I turned it off". - `loadAll` replaces the N+1: the loop and the API both want the whole table, and it will always fit in a page. `Save` renders the one task it changed instead of recomputing every task's next three runs. - `arm` and `claim` refresh `updated_at`, which they had been leaving stale. - A `NextRuns` failure is logged and surfaced instead of silently rendering an empty preview that reads exactly like "not due for a while". - `running`, `last_millis` and `updated_by` are rendered. They were plumbed through the schema, the wire and the types and then dropped on the floor — a column nothing reads is a column nobody can trust. `interrupted` joins the status enum, which the server had been able to write since the power-cut fix but the spec did not admit. - The dashboard backs off on a failing poll instead of knocking every three seconds until the lid closes. Not done, deliberately: `versioncheck` keeps its own ticker. Its period is CIX_VERSION_CHECK_INTERVAL, a released duration-valued variable, and a duration does not survive the trip through crontab — 6h maps cleanly, 7h does not exist. Moving it means either breaking that variable or carrying both forms, which is a decision of its own rather than a tidy-up. The package comment and the docs say so rather than claiming a generality the code has not earned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
untilNext took the earliest next_run_at in the table without asking whether anything would ever act on it. A task switched off keeps the time it was last armed for; considerOne steps over it without re-arming; the moment passes and that row is now both the earliest and permanently in the past, so the loop wakes on its one-second floor for good — roughly 260k queries a day against a task an admin explicitly disabled, and worse than the polling this branch came to remove. A row left behind by a task deleted from the code does the same thing on every database that ever ran the older build. Both are now skipped, and Save arms a disabled task for nothing rather than leaving the time behind. The daylight-saving rescue searched the vanished hour a minute at a time, so `30 30 3 * * *` — gronx takes a leading seconds field — matched nothing and lost the day the rescue exists for. It now steps at the expression's own resolution. It also looked at one transition. ZoneBounds only answers about the interval containing the instant handed to it, so `0 3 28 3 *` asked in June met October's fallback first, stopped there, and skipped from 2027 to 2028. It now walks every transition up to the candidate. A threshold of zero is legal and means that dimension does not gate, but advise compared against it: `>= 0` holds for every database ever measured, so an empty freelist came out Urgent, offering to return 0 B to the filesystem, and VerdictOK was unreachable. Zero now means unset, and the multiplication behind the urgent threshold saturates instead of wrapping a large environment value into a negative one. Last, a schedule with no upcoming run no longer reports " (this schedule has no upcoming run)" with a space where the error would have been. Each fix has a test that fails without it, at the measured values above. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat: reclaim wasted space in the SQLite database
CIX_VECTORS_DIR names the container for the SQLite vector stores — one
database per embedding namespace — and VectorDirFor mirrors ChromaDirFor's
component nesting so a namespace's database and the chromem directory it will
be imported from are always derivable from the same identity components.
The default is a SIBLING of CIX_CHROMA_PERSIST_DIR ("<...>/vectors" next to
"<...>/chroma") rather than a fresh path under the data dir: every container
image overrides only CIX_CHROMA_PERSIST_DIR, and anything else would put the
vectors somewhere that is not the persistent volume.
CIX_VECTOR_MMAP_SIZE is the opt-in mmap knob, off by default — it trades
resident memory for roughly 40% lower search latency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chromem-go loads every document of every collection into the Go heap at open
and never evicts, so memory was proportional to the index rather than to the
work: 2209 MB resident and a 47-second cold boot on a real 312k-document
index, before answering a single query. Embeddings now live as BLOBs in one
SQLite database per embedding namespace and a search streams the collection
past a dot product into a top-K heap. Measured on the same data: 19 MB
resident idle, 44 MB after a hundred searches and a 47-collection fan-out,
sub-millisecond open. The cost is latency — roughly 3x chromem's, and flat in
the result limit (k=500 is 1.6% dearer than k=10).
Storage. <data>/vectors/<kind>/<model-slug>/vectors.db, a sibling tree to the
legacy <data>/chroma so the gob files stay untouched as the rollback path.
Chunk text is stored (duplicating chunks_fts on disk, deliberately: the
package stays self-contained and SearchResult.Content is unchanged) but in its
own table — a multi-kilobyte TEXT column in `vectors` would push every row past
SQLite's local-payload limit and spill the EMBEDDING into an overflow chain,
roughly doubling the pages a scan touches.
Traps honoured, all measured in the prototype:
- modernc.org/sqlite sorts _pragma DSN options lexicographically instead of
applying them in order, which silently drops page_size on a fresh database.
Pragmas are applied by hand on each connection through a driver.Connector.
- page_size=8192. 4 KiB fits one row per page; 16 KiB is one byte-class over
modernc.org/memory's slab limit, so every page buffer costs mmap+munmap.
- cache_size stays at the default: measured to buy no latency, and it is per
connection, so raising it multiplies RSS.
- The scan runs INDEXED BY idx_vec_coll_file. Delete-by-file plus reinsert —
every save the watcher sees — scatters a collection's rows across the
table, and a plain table scan then degrades 3.3x. Pinned by an EXPLAIN
QUERY PLAN test.
- Idle connections are closed after 30s: SQLite's page cache lives in mmap'd
arenas outside the Go heap, so closing the connection is the only thing
that returns it.
Migration. Opening a namespace imports any chromem collection not yet in
migration_state, one transaction per collection (which also writes its
migration_state row), so a crash redoes exactly the collection it was in the
middle of and never duplicates a finished one. Free space is checked first.
Nothing under the chroma tree is ever written. The server does not link
chromem-go at all — the importer decodes the gobs through mirror structs, and
chromem is now a test-only dependency, kept because a fixture written by the
real thing is the only evidence the mirrors still match.
Maintenance. ListCollections reads from SQL and is no longer free; its doc
comment says so. Orphaned collections no longer claim to free RAM (they do not
— the store holds nothing per collection), EstimatedRAMBytes is left unset and
Clean's stop-the-world FreeOSMemory is gone. CollectionDir is replaced by
CollectionSizeBytes + DBPath, since there is no per-collection directory to
stat. The abandoned-namespace scan now covers both trees, protecting the
active namespace in each — which is what keeps the pre-migration gob files off
the deletion list.
Frozen contracts are untouched: collection names, document IDs, the Interface
method set and the HTTP API are all unchanged, which is what lets the existing
gob files be imported verbatim and every Interface consumer stay as it was.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
doc/VECTORSTORE.md covers the layout on disk, the schema and why chunk text lives in its own table, the search path and the pragma choices behind it, the two new environment variables, and how the one-time import from chromem-go behaves on first boot (including that the legacy files are kept as the rollback path and are not reclaimed automatically yet). The rest is de-chromem-ing prose that has gone stale: the config reference, the search-algorithm and deployment docs, the OpenAPI descriptions for chroma_path and the now-unset estimated_ram_bytes, and the architecture diagram on the site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scan was driven by idx_vec_coll_file, which has the right prefix but the wrong order: its keys sort by file_path, so the row lookups jump around the collection's whole rowid span. A dedicated idx_vec_coll sorts by rowid — SQLite appends it to every index key — so the same scan walks the table sequentially. Measured against the real 312k-document index, scanning its largest 74k-row collection: 137 ms via idx_vec_coll, 244 ms via idx_vec_coll_file, 267 ms with no index at all (that one walks all 312k rows and discards 76% of them). idx_vec_coll_file stays: delete-by-file needs it. Existing databases pick the index up on the next open (CREATE INDEX IF NOT EXISTS runs in the schema). The plan test now pins idx_vec_coll specifically and fails if SQLite picks the file-path index back up. Also replaces the estimates in doc/VECTORSTORE.md with figures measured through this implementation on that index: 19 MB resident at open, 26 MB after a fan-out over all 47 collections, 510 ms for that warm fan-out, 17 s to import the whole thing, 1.86 GB on disk with chunk text included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SQLite vector store keeps its page cache in modernc's mmap arenas, which runtime.MemStats cannot see — the dashboard showed 3.9 MB "heap in use" on a process holding tens of MB resident. The Resources headline tile now shows current RSS (with the Go heap in its tooltip), and darwin learns to report RSS via ps(1), since mach task_info has no cgo-free path and this only runs on an admin-endpoint hit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SQLite vector store imports the chromem gob files once and then leaves them alone forever: they are the rollback path, and the abandoned-namespace category protects the ACTIVE namespace in both trees for exactly that reason. On the reference install that is 2.5 GB no process will ever open again, with no way to reclaim it short of rm -rf. Add an opt-in "Legacy chromem data" category that releases that protection deliberately. It lists the active namespace's chromem directory only when the tree is provably redundant — every collection directory in it has a migration_state row in that namespace's vectors.db. Anything less (migration in flight, a directory the importer could not read, a missing or unreadable vectors.db) and the namespace is not offered at all rather than offered and disabled: a disabled row still advertises gigabytes that are not garbage yet. The warnings say which case it was. Never pre-selected, disk only (nothing about the tree is in RAM, so estimated_ram_bytes stays 0), and the description carries the irreversibility warning verbatim. The full-migration check is repeated immediately before the delete, which is what a runtime embedding-model switch can invalidate; a running index/clone job deliberately does NOT hold the category back, because this binary never writes the gob tree — only an in-flight import can make it matter again, and the re-check asks about that directly. New vector-store surface: Maintainer.MigratedCollections (migration_state, with an explicit "unknown" that must not be read as "nothing was migrated") and vectorstore.LegacyMigrationStatus. Deleting the tree changes nothing else: migration_state rows are kept, so the next boot finds no legacy directory and starts normally. Dashboard needs no changes — categories render generically from the analysis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
decodeDocs decoded an ENTIRE collection into a slice before the per-collection transaction opened and held it until commit: measured 268 MB peak HeapAlloc for a 30k-document collection (~9 kB of live heap per document), so ~650 MB for 74k and ~1.8 GB for a 200k-document monorepo. The first boot after the upgrade therefore demanded exactly the memory this store exists to give back, and OOM-killed containers sized for the new ~28 MB idle footprint. Decoding and writing are now pipelined: NumCPU decode workers feed a channel and the writer drains it into the SAME per-collection transaction 2000 documents at a time, dropping each batch's references before pulling the next. Peak heap scales with the batch, not the collection. Resumability is unchanged (one transaction per collection, migration_state still written last inside it), as are cancellation, the unreadable-collection accounting and the progress-log cadence. Also: - PRAGMA journal_size_limit=64MB in the shared connection setup. A checkpoint rewinds the WAL but by default leaves the FILE at its high-water mark forever — measured a permanent 159 MB -wal beside a 158 MB database — and that sidecar is counted in the dashboard's "Vector store" disk row. It benefits steady-state operation too. - importSpaceFactor 0.5 -> 0.9. The "1121 MB from 2.5 GB" comment was a stale prototype number measured WITHOUT chunk content; the shipping schema stores it and the same tree produces 1.86 GB (0.74x). The rest of the guard is the WAL: every page a transaction touches stays in it until commit, so a per-collection transaction still peaks at roughly one collection whatever the write batch is. - Migration start / progress / completion logs raised info -> warn. Production runs at warn level and the HTTP listener only comes up after the store opens, so at info the operator watches a server that answers nothing and says nothing for the whole import — silence that has repeatedly been read as "the server is down" and answered with a restart. Tests: a collection spanning several batches is byte-exact and resumable mid-collection; exact-batch-multiple boundaries (3 full batches, exactly one batch, one partial batch) neither drop nor duplicate the last flush; cancellation unwinds instead of deadlocking on the channel send and commits nothing; journal_size_limit is pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…emental vacuum
Two problems that get much more expensive to fix once the store has real
deployments, so they are fixed in one migration.
(a) vectors.collection_id had no foreign key, PRAGMA foreign_keys was
never enabled, and collections.id was a plain INTEGER PRIMARY KEY (the
rowid) with no AUTOINCREMENT. A collection id is CACHED by callers
(Store.collIDs, and through it the indexer) across the window in which an
admin can delete the collection, and both halves of that were unguarded:
- A late upsert holding a stale id committed rows whose collection_id
has no collections row. ListCollections joins FROM collections, so
those rows were invisible to every count, every size figure and the
orphan sweep — they held disk forever with nothing able to find them.
- Deleting the highest-numbered collection freed its rowid for REUSE,
so the next collection created inherited both the id and any rows
that outlived the delete, and search answered one project's query
with another project's chunks.
(b) auto_vacuum was off. The Resources screen advertises reclaimed bytes
when collections are deleted, but the pages only reached the freelist:
the file never shrank and df never confirmed what the UI claimed.
Schema v2: AUTOINCREMENT collection ids, ON DELETE CASCADE foreign keys
on both child tables, PRAGMA foreign_keys=ON per connection, and
auto_vacuum=INCREMENTAL set before the first write on fresh databases.
PRAGMA incremental_vacuum runs after DeleteCollectionByName; deliberately
NOT after DeleteByFile, where the watcher's delete-and-reinsert churn is
already covered by freelist recycling (measured: 100 cycles over 72k rows
grew the file 3.8 MB and ended with an empty freelist) — documented in a
comment at the call site.
None of the three can be reached with ALTER TABLE, so existing files are
upgraded by rebuilding: user_version (0 on current files) selects the
path, a sibling temp file gets the v2 schema and pragmas, the data is
copied with ATTACH + INSERT SELECT, and the result is fsynced and renamed
over the original with the stale -wal/-shm removed. One pass delivers all
three, because a rebuild IS the vacuum. Free space is checked first
(needs one extra copy of the file) and progress is logged at warn, for
the same reason the legacy import is. Orphan rows that a v1 file already
holds cannot enter a database that enforces the constraint, so the copy
filters them and reports how many it dropped — they were unreachable
already. Measured: a 152 MB database rebuilds in 0.32 s.
UpsertChunks now returns an error wrapping ErrCollectionDeleted when its
collection was deleted mid-flight, and drops the stale id from the cache
so the store is usable again. It does not retry — re-creating a
collection an admin just deleted is the caller's decision, not a silent
side effect of a failed batch.
Tests: the v1 -> v2 upgrade path end to end (v1 fixture built from a
frozen copy of the old schema; v2 pragmas, AUTOINCREMENT DDL and
sqlite_sequence, data intact, orphans dropped, stale -wal cleared, no
leftover temp file); a file already at v2 is not rebuilt (same inode);
create/delete/create yields a fresh id; the stale-id race fails cleanly
through both upsertBatch and UpsertChunks and leaves zero orphan rows;
DeleteCollectionByName shrinks page_count and drains the freelist.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- maintenance/analyze.go scanStaleNamespaces: the ctx check sat in the inner loop over one namespace tree's leaves, so a bare break only ended that tree and the scan carried on into the next one — walking and sizing a whole second directory after the caller had given up. Labelled break instead. Also refreshed the neighbouring comment on orphan-collection sizing: DeleteCollectionByName now runs an incremental vacuum, so "the file does not shrink until a VACUUM" no longer describes what happens. - CIX_VECTORS_DIR and CIX_VECTOR_MMAP_SIZE were documented only in doc/CONFIG_REFERENCE.md and doc/VECTORSTORE.md. Added them, commented out with a one-line explanation in the local style, to .env.example and to the env blocks of docker-compose.yml, docker-compose.cuda.yml, portainer-stack.yml and portainer-stack-cuda.yml. The mmap note is per-file: it is a bad idea under the CPU compose's 2G limit and affordable under the CUDA stack's 10G. CIX_CHROMA_PERSIST_DIR, which was bare in all five, gained the one line saying it is now only the legacy import source and rollback path. - maintenance.Category.EstimatedRAMBytes: marked Deprecated in the godoc-recognised form — nothing has set it since the vector store left the Go heap, it is kept for wire compatibility with older dashboard builds, and it is a candidate for removal in the next breaking API revision. doc/openapi.yaml already described it as legacy; it now also carries deprecated: true and a matching x-deprecated-reason, so the generated client field is marked deprecated too (openapi.gen.go regenerated, openapi-check green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… stamps Two defects in the v2 upgrade path, both crash-window shaped. A kill between the driver creating the file and the schema being written leaves a zero-byte vectors.db; "file exists" sent it to the upgrader, which ATTACHed the empty database and died on its first SELECT — a fatal boot error where the old code self-healed. And initDatabase stamped user_version unconditionally, so one old-binary run against a newer data dir remarked a v3 file as v2, defeating the downgrade guard upgradeSchema had just honoured. Zero-byte files now count as fresh, and the stamp is written only on files this process created. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er import Review sweep, round two. The incremental vacuum moves out of DeleteCollectionByName into Maintainer.ReclaimFreePages, called once after the clean loop — per-delete it re-shuffled the file tail under the write lock (~170 ms per 20k-doc collection) for every orphan in the batch. The import ends with wal_checkpoint(TRUNCATE) so a freshly migrated idle server does not carry ~100 MB of dead WAL in its usage row. The free-space guard now prices the WAL peak against the largest single collection instead of a flat share of the tree, and the rebuild removes stale -wal/-shm sidecars before the rename rather than after, removing the need to reason about salt mismatches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The zero-byte guard covered one case of a class. A kill between PRAGMA journal_mode (which materialises page 1) and schemaSQL leaves a header-only file: not zero bytes, still no schema, same fatal no-such-table boot error — and it cannot be adopted as fresh either, because page 1 has already frozen the default page_size. openDB now asks sqlite_master instead of stat: an empty master proves the file holds nothing, so it is removed and recreated. One check closes the whole undersized-file class. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Batching the vacuum into maintenance.Clean left the other delete path
dry: DELETE /projects/{path} → DropCollection freed pages only into the
freelist, so on a server where projects come and go the file keeps its
high-water size forever and the Resources row overstates by exactly that
amount — the complaint the vacuum was added to fix. A project delete is
a single admin action, not a batch, so it reclaims inline; the indexer's
DeleteCollection before a full reindex stays vacuum-free on purpose,
since the reindex reuses those pages immediately. Also updates the
analyze.go size comment that still described the per-delete vacuum, and
drops a leftover chromem-era "resident in RAM" phrase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round-four review sweep. An operator who points CIX_VECTORS_DIR at
somebody else's SQLite file used to die on "no such table:
old.collections" — the emptiness probe now also classifies the file
(collections is the anchor table of every schema version) and refuses a
foreign database by name, leaving it byte-intact. And the one behaviour
this PR shipped without a regression test — DELETE /projects/{path}
actually reclaiming pages, wiring that already went missing once in the
batched-vacuum refactor — is now pinned behaviourally in an httpapi test:
logical size must shrink, freelist must end empty, the surviving
project's data must stay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A views-only database has zero tables but is still somebody's data — the emptiness probe's WHERE type='table' filter routed it into the recreate branch, so the commit that added the foreign-file refusal shipped with a silent-destruction hole exactly for table-less foreign files. Emptiness is now literal: no rows in sqlite_master at all. The crash-on-first-boot class always has a literally empty master, so the stricter definition costs nothing, and views-only files now get the same loud, byte-intact refusal as any other foreign database. Also restores the readUserVersion doc comment that the probe's insertion had absorbed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(server): replace chromem-go with a SQLite-BLOB vector store
…roken /health only answers at the very end of startup — the llama model load runs first (minutes from cold on a host that is powered off nightly) and, on the first start after this release, so does the one-time chromem -> SQLite vector-store import. start_period was 120s, so a normal cold boot spent roughly a minute flagged `unhealthy` before the server ever got a chance to answer. Nothing acts on that flag: restart policies react to exits, not health, and there is no autoheal or depends_on: service_healthy anywhere in these files. The flag's only consumer is a human reading Portainer — which is exactly how this has failed three times (19-21 Jul), each ending in a restart of a server that was merely still starting. After the vector-store change a restart there also discards the collection the import was in the middle of. So the asymmetry is total: an over-long window delays a label nobody reads automatically, a short one manufactures false alarms. 600s in all four compose/stack files and in the image's own HEALTHCHECK, which is the default whenever the image runs without a compose override — leaving that at 120s would have made the image contradict the stack. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ncy gate Four checks failed on the develop -> main promotion (#247). None of them was caused by the promotion itself; the gate simply runs against main, and develop-based PRs are not scanned, so this is the first time these were asked. govulncheck (server) — five reachable Go standard-library advisories on go1.26.5, among them GO-2026-6089 (ReadHeaderTimeout on the unencrypted HTTP/2 check) reached straight from http.ListenAndServe. go.mod moves to go 1.26.6. govulncheck (cli) — the same shape one release line down: four advisories on go1.25.12, fixed in go1.25.13. Both workflows resolve their toolchain through go-version-file, so the go directive is the only lever, and both modules scan clean on their declared toolchain (the CLI has to be checked with GOTOOLCHAIN=go1.25.13 locally — a newer host toolchain is used as-is and hides the fix). Trivy code-scanning — go-git v5.19.1 carried CVE-2026-71556 (high) and CVE-2026-71557, fixed in v5.19.2; golang.org/x/mod v0.37.0 carried the two sumdb-forgery advisories, fixed in v0.40.0; react-router 7.18.1 carried GHSA-qwww-vcr4-c8h2, which this repo had recorded as unfixable and which now has a fix in 7.18.2. go mod tidy carried x/crypto, x/net, x/text and x/tools forward with them. GO-2026-5932 (x/crypto/openpgp unmaintained) has no fixed version and no call path into it — `go list -deps` finds the package nowhere in the build — so it stays accepted, as before. The dashboard lockfile also loses its entries for lucide-react and @radix-ui/react-scroll-area. That is not a removal: both were dropped from package.json in e7c2dd5 and only their lockfile rows survived. Nothing under src/ imports either, npm ci and the production build both pass without them. TestSearchLatencyGate — flaky, not broken. It failed with "P95=238.0ms" on a commit that changed one YAML field, while a second CI run of the identical tree passed, and the same test measures a 1.7ms median here under GOMAXPROCS=2 with the machine loaded. P95 over 50 samples is the 48th slowest; on a shared runner three descheduling stalls produce it, so that statistic reports the runner, not the store. The gate moves to the median, which no handful of stalls can shift and which still catches the failure the test exists for — a search that got catastrophically slower moves every sample. The tail is not dropped silently: median, p95 and max are all logged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion of
developtomainfor the server/v0.13.0 release. 47 non-merge commits, five feature PRs.Why 0.13.0 and not 0.12.10
The vector store is a different piece of software than it was in 0.12.9, the on-disk layout gains a directory, and the first boot after upgrading does one-time work. That is a minor bump, not a patch.
What is in it
SQLite vector store (#246) — chromem-go is no longer linked by the server. Embeddings live as little-endian float32 BLOBs in one SQLite file per embedding namespace, searched by a streamed exact scan with a top-K heap. On the reference index (312k documents / 47 collections): idle RSS 2209 MB → 28 MB, time to ready 47 s → ~1 s, at the cost of ~3x search latency (34 ms → 110 ms p50 on a 74k-document collection; 47 ms end-to-end for
cix searchagainst 31 ms).Database maintenance (#243) — offline compaction with a boot reconciler and a write gate, incremental auto-vacuum for new databases, and recurring maintenance on a crontab expression instead of a bare interval.
Admin Resources (#242) and dashboard redesign (#241) — the Server page splits into Runtime settings and Resources; reclaimable disk is analysed and cleaned by category, including the pre-migration chromem tree once it is provably imported.
Fixes — Enter submits the auth forms again (#245), the maintenance banner stops crying wolf, the scheduler stops spinning.
First boot after this upgrade
The store imports the legacy
chroma/tree into<data>/vectors/once, before the HTTP listener comes up: 17.4 s for 312k documents in live testing, logged atwarnso it is visible at production log level. It is resumable per collection and needs roughly0.74 xthe gob tree in free space plus the largest collection's worth of WAL. Nothing underchroma/is written or deleted — it stays as the rollback path.start_periodon the healthcheck goes from 120 s to 600 s in all four compose/stack files and in the image's ownHEALTHCHECK./healthonly answers once the llama model has loaded and the import has run, and the old window meant a normal cold boot spent about a minute flaggedunhealthy. Nothing acts on that flag automatically — its only consumer is a human reading Portainer, which is how this produced three false "the server is down" alarms in July.Rolling back
Revert the image tag; the chromem tree is exactly where the old binary expects it. Anything indexed after the upgrade lives only in
vectors/and needs a reindex on rollback.New configuration
CIX_VECTORS_DIRCIX_CHROMA_PERSIST_DIR, i.e./data/vectorsCIX_VECTOR_MMAP_SIZE0(off)PRAGMA mmap_sizein bytes. ~40% lower search latency, paid for in resident memory. Not for a tight memory ceiling.CIX_CHROMA_PERSIST_DIRis still read — it is the import source and the rollback path.Verification
go test ./...green (43 packages),go vetclean,-raceclean on vectorstore / httpapi / maintenance,make openapi-checkin sync. The vector-store change went through six review rounds; every finding closed with a fix plus a regression test for the specific scenario.Outstanding before the tag
The trivy security gate against the current production tag has not run yet — it builds on the production host, which was unreachable at the time of writing. It must pass before
server/v0.13.0is pushed.