Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ WorldScript-Studio/
│ ├── unit/ # Vitest tests (co-located naming convention)
│ ├── e2e/ # Playwright specs (CI-only)
│ └── setup.ts # Global Vitest setup
├── workers/ # Web Workers: inference, duckdb, plugin, v2 inference/duckdb
├── workers/ # Web Workers: plugin.worker.ts; v2/ is the sole worker generation since ADR-0015 (inference, duckdb, webllm)
├── scripts/ # Build/deploy helpers (i18n bundle, SW version sync, bundle budget, edge build)
├── infra/low-end-ci/ # Local CI stack for constrained hardware
├── src-tauri/ # Tauri 2 desktop app (Rust)
Expand Down Expand Up @@ -425,7 +425,7 @@ Edge builds run `scripts/build-edge.mjs` which sets `DEPLOY_TARGET=edge` and pat
### Local Inference

- 4-layer stack: WebLLM → ONNX Runtime Web → Transformers.js → heuristics fallback.
- `services/localAiFacade.ts` wraps WebLLM via `packages/ai-core` + `workers/inference.worker.ts`.
- `services/localAiFacade.ts` wraps WebLLM via `packages/ai-core` + `workers/v2/inference.worker.ts`.
- Tab-leader election via BroadcastChannel prevents multi-tab GPU contention.
- **Adaptive AI Engine** (`services/ai/adaptiveAiEngine.ts`) — runtime hardware-aware backend/model selection. Gated by `enableAdaptiveAiEngine` flag.
- **Device Profiler** (`services/ai/localAiDeviceProfiler.ts`) — WebGPU/WebNN/NPU/battery detection, 30s TTL cache.
Expand All @@ -448,19 +448,20 @@ Edge builds run `scripts/build-edge.mjs` which sets `DEPLOY_TARGET=edge` and pat

### WorkerBus v2

Central orchestration layer for all background worker tasks. Messages use short kind literals (`TASK`, `CANCEL`, `PING`, `PONG`, `PROGRESS`, `RESULT`) validated by Zod.
Central orchestration layer for all background worker tasks — since ADR-0015, the **sole** worker generation (v1's `workers/duckdbWorker.ts`/`workers/inference.worker.ts` deleted). Messages use short kind literals (`TASK`, `CANCEL`, `PING`, `PONG`, `PROGRESS`, `RESULT`) validated by Zod.

- `packages/worker-bus/src/` — WorkerBus, WorkerPool, PriorityTaskQueue, CircuitBreaker, DeadLetterQueue, ProtocolHandler, workerBootstrap, constants, schemas.
- `services/workerBusManager.ts` — singleton lifecycle; registers `inference` and `duckdb` pools.
- `packages/worker-bus/src/` — WorkerBus (`hasPool()`/`terminatePool()` for scoped pool lifecycle), WorkerPool, PriorityTaskQueue (`critical > high > normal > low`, no `background` tier), CircuitBreaker, DeadLetterQueue, ProtocolHandler (unused by the live `runTask()` path), workerBootstrap (`WorkerHandlerContext.emitProgress(stage, progress, message?)` — a flat function, not `context.progress.emit(...)`), constants, schemas.
- `services/workerBusManager.ts` — singleton lifecycle; registers `inference`, `duckdb`, `webllm`, `plugin` pools. `ensureDuckDbPool()`/`ensureInferencePool()`/`ensureWebLlmPool()` force-init and re-register a pool removed via `terminatePool()`, decoupled from `enableWorkerBusV2` (these are core features, not experimental infra).
- `services/hybridRouter.ts` — routes to Web Worker pool or Rust TaskSupervisor (Tauri only) when `enableRustCompute` is on.
- `services/legacyWorkerBusAdapter.ts` — shims old `@domain/ai-core` WorkerBus API onto v2.
- `services/tauriTaskBridge.ts` — `invokeRustTask()`, `isRustComputeAvailable()` (60s TTL ping cache).
- Feature flags: `enableWorkerBusV2` (on by default), `enableRustCompute` (on by default; effective on Tauri desktop only).
- v2 workers: `workers/v2/inference.worker.ts` (text + embed via Hugging Face transformers), `workers/v2/duckdb.worker.ts` (init/query/exec/shutdown).
- Workers: `workers/v2/inference.worker.ts` (text + embed via Hugging Face transformers, prepared-statement-free), `workers/v2/duckdb.worker.ts` (init/query/exec/shutdown, binds params via DuckDB-WASM prepared statements), `workers/v2/webllm.worker.ts` (WebGPU, ADR-0005), `workers/plugin.worker.ts` (sandboxed plugin execution, outside `v2/` but on the same protocol).
- **Known gap:** `timeoutMs` is not enforced anywhere in the live `runTask()` path (no timer rejects a hung task) — pre-existing, tracked in ADR-0015/TODO.md.

### DuckDB Analytics

- `workers/duckdbWorker.ts` runs DuckDB-WASM off main thread (OPFS persistence → in-memory fallback).
- `workers/v2/duckdb.worker.ts` runs DuckDB-WASM off main thread via the shared `duckdb` pool (OPFS persistence → in-memory fallback). `duckdbClient.ts` is an adapter over `ensureDuckDbPool()`/`bus.enqueue()` that auto-reinits on a respawned worker's "not initialized" error.
- `services/duckdb/duckdbClient.ts` is a singleton proxy with AbortSignal and init retry.
- Schema (`duckdbSchema.ts`): 10 tables + 5 views including `rag_chunks` (FLOAT[384]), `cross_project_index`, `codex_*`.
- Gated behind `featureFlagsSlice.enableDuckDbAnalytics` (on by default).
Expand Down
4 changes: 2 additions & 2 deletions AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ change landed — all 14 findings CONFIRMED, no baseline drift, base SHA `f5f9c1
| F-08 | 🟠 P1 | Tauri `connect-src` missing LanguageTool's port and the Hugging Face hosts WebLLM/Transformers.js resolve models from | Fixed; scope widened during verification — the LanguageTool port was missing on **all 5** surfaces, not just Tauri; the real weight-file CDN (`us.aws.cdn.hf.co`) traced empirically via `curl`, not guessed |
| F-09 | 🟠 P1 | DuckDB-WASM loaded from an unversioned, floating-`latest` third-party CDN — already unreachable under the (correctly-scoped) `worker-src` CSP, so this was dead code, not just a supply-chain risk | Self-hosted from the pinned npm dependency via `scripts/copy-duckdb-assets.mjs` (gitignored, ~72 MB, never committed) |
| F-10 | 🟡 P2 | Two conflicting production URLs; the in-app link and the Italian locale pointed at a dead domain | Empirically resolved (live 200 vs. 404 check) — no maintainer input needed. Unified into a `PRODUCTION_URL` constant + new drift gate |
| F-11 | 🟡 P2 | Release/tag drift — `v1.24.1` tag 5 commits behind `HEAD` | Version bumped to `1.24.2` in this sprint (WS-7); tag/release publish remains a maintainer action |
| F-11 | 🟡 P2 | Release/tag drift — `v1.24.1` tag 5 commits behind `HEAD` | Version bumped to `1.24.2` in this sprint (WS-7); `v1.24.2` tagged and published 2026-07-29 |
| F-12 | 🟡 P2 | Storybook built 3× per CI run; `ci.yml` header still said "StoryCraft Studio" | Deduped to 1×, header fixed. Bonus: the Storybook test-runner's flags were invalid for the installed CLI version, so it had never actually executed a single story on any prior CI run — fixed, and its `\|\| true` (which was itself hiding the failure) replaced with `continue-on-error: true` |
| F-13 | 🟡 P2 | Coverage ratchet stale since 2026-06-06; 4 non-blocking CI gates with no stated exit criterion | Ratchet raised to CI-measured values (see quality-gate line above); `docs/CI.md` gained an exit-criteria table for all 4 |
| F-14 | 🟡 P2 | Two live worker generations (v1 + WorkerBus v2) for DuckDB/inference, confirmed via real call-site tracing | Documented, not consolidated — [ADR-0014](docs/adr/0014-worker-generation-duplication.md); a dedicated migration sprint is the correct scope for consolidation |
| F-14 | 🟡 P2 | Two live worker generations (v1 + WorkerBus v2) for DuckDB/inference, confirmed via real call-site tracing | **Resolved.** Consolidated as the dedicated migration sprint [ADR-0014](docs/adr/0014-worker-generation-duplication.md) called for — [ADR-0015](docs/adr/0015-worker-generation-consolidation.md), PRs #286–288/#290. Both v1 files deleted; correction loops on the migration PRs also caught and fixed 4 real bugs the untested v2 files had drifted on (dropped SQL params, pool-termination hangs, respawn connection loss, a missing export that broke production builds) |

### Post-mortem: why no gate caught F-01 for two months

Expand Down
45 changes: 45 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,51 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

> Worker-generation consolidation sprint: the v1/WorkerBus-v2 duplication [ADR-0014](docs/adr/0014-worker-generation-duplication.md)
> deferred is now closed out. Executed as 4 stacked PRs (#286–288, #290); see
> [ADR-0015](docs/adr/0015-worker-generation-consolidation.md) for the full decision record.

### Changed

- **DuckDB analytics and local inference (embeddings + NLP) now route through WorkerBus v2**,
replacing dedicated `new Worker(...)` instances per service. `services/duckdb/duckdbClient.ts`,
`services/ai/localEmbeddingService.ts`, and `services/ai/localNlpService.ts` keep their exact
public APIs — an adapter pattern, not a consumer rewrite, so none of their real callers needed
changes. `ensureDuckDbPool()`/`ensureInferencePool()` added to `services/workerBusManager.ts`,
decoupled from `enableWorkerBusV2` (mirrors the existing `ensureWebLlmPool()` pattern — these are
core features, not experimental infra).
- The shared `inference` WorkerBus pool's `maxWorkers` capped at 2 (was 4) — each replica
independently loads its own transformers.js pipeline with no cross-replica cache sharing, so 4
concurrent workers under a burst could mean 4x the model memory footprint.

### Fixed

- **DuckDB query/exec parameters were silently dropped**, executing an unbound query — already live
and breaking `services/ai/telemetryService.ts`'s parameterized `INSERT` writes. Fixed via
DuckDB-WASM prepared statements.
- **`WorkerBus.terminatePool()` could leave tasks hanging forever** — removing a pool didn't settle
tasks already routed to it. Fixed by tracking task→pool assignment and cancelling matching tasks
before removal; a pool removed this way is now also transparently re-registered on next use
instead of failing every subsequent call.
- **A respawned DuckDB worker lost its connection**, failing every query after an idle timeout with
"DuckDB not initialized." Fixed by transparently re-initializing once and retrying.
- **DuckDB's OPFS-unavailable fallback silently swallowed the failure**, giving users no warning
their analytics wouldn't persist across reloads (private browsing, old Safari) — restored via the
existing progress channel. A follow-up fix made the fallback's own cleanup step best-effort, so a
secondary failure there can no longer block the fallback path itself.
- **A missing `ensureInferencePool` export broke production builds** on Vercel (rolldown's
`[MISSING_EXPORT]`) — not caught by Vitest (the consumer test mocks the whole module) or by local
`tsgo` typecheck. Root-caused by reproducing Vercel's *exact* build command
(`pnpm run build:edge`, which differs from the plain `pnpm run build` used for an earlier,
misleading local repro attempt) and fixed by adding the missing function.

### Removed

- `workers/duckdbWorker.ts` and `workers/inference.worker.ts` (the v1 worker generation) — both
workloads now exclusively use `workers/v2/duckdb.worker.ts` and `workers/v2/inference.worker.ts`.

## [1.24.2] — 2026-07-29

> CSP functional-truth, desktop crypto, and documentation-truth hardening sprint
Expand Down
12 changes: 7 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,9 @@ locales/ → i18n source JSON — 19 locales (de/en/es/fr/it/ar/he/el/j
tests/ → unit/ (Vitest) + e2e/ (Playwright); shared E2E helpers in tests/e2e/helpers.ts
types/ → Supplemental TypeScript definitions (duckdb-wasm-worker.d.ts, tauri-plugins.d.ts)
types.ts → Core shared interfaces and types (root level)
workers/ → inference.worker.ts (@huggingface/transformers v3), duckdbWorker.ts (DuckDB-WASM)
v2/ → WorkerBus v2 workers: inference.worker.ts, duckdb.worker.ts, webllm.worker.ts (P1-1, @mlc-ai/web-llm)
workers/ → plugin.worker.ts (sandboxed plugin execution, P0-2)
v2/ → sole worker generation since ADR-0015: inference.worker.ts (@huggingface/transformers v3),
duckdb.worker.ts (DuckDB-WASM), webllm.worker.ts (P1-1, @mlc-ai/web-llm)
infra/low-end-ci/ → Local CI stack: Forgejo + act + systemd units + bash scripts
scripts/ → Build/deploy helpers (sync-deploy-base, cf-pages-deploy, graphify-update, etc.)
```
Expand Down Expand Up @@ -188,7 +189,7 @@ Wrap each major view root with `components/ui/ViewErrorBoundary.tsx` — provide

### DuckDB Analytics

`workers/duckdbWorker.ts` off main thread (OPFS → in-memory fallback). `duckdbClient.ts`: singleton, init retry 3× backoff. Schema: 10 tables + 5 views incl. `rag_chunks` (FLOAT[384]). Gate all paths behind `enableDuckDbAnalytics`. Dual-write via `duckdbListenerLoader.ts` (dynamically imported). `ragVectorMigration.ts`: FLOAT[64]→FLOAT[384] upgrade. `useDuckDb.ts` 30s timeout; `useAnalytics.ts` parallelizes 4 queries.
`workers/v2/duckdb.worker.ts` off main thread via the shared WorkerBus v2 `duckdb` pool (OPFS → in-memory fallback, ADR-0015). `duckdbClient.ts`: adapter over `ensureDuckDbPool()`/`bus.enqueue()`, auto-reinits on a respawned worker's "not initialized" error. Schema: 10 tables + 5 views incl. `rag_chunks` (FLOAT[384]). Gate all paths behind `enableDuckDbAnalytics`. Dual-write via `duckdbListenerLoader.ts` (dynamically imported). `ragVectorMigration.ts`: FLOAT[64]→FLOAT[384] upgrade. `useDuckDb.ts` 30s timeout; `useAnalytics.ts` parallelizes 4 queries.

### Logging

Expand Down Expand Up @@ -218,7 +219,7 @@ Real-time P2P via Yjs + `packages/collab-transport`. Signaling AES-256-GCM/PBKDF

### WorkerBus v2 (`packages/worker-bus`)

Central orchestration layer for all background tasks. Key files: `workerBus.ts` (orchestrator, priority queue + circuit breakers), `workerPool.ts` (auto-scaling `MIN`→`MAX_WORKERS_INFERENCE`, idle timeout), `taskQueue.ts` (heap: `critical > high > normal > low > background`), `circuitBreaker.ts` (per-worker health gate), `deadLetterQueue.ts`, `protocolHandler.ts` (typed postMessage + version negotiation), `workerBootstrap.ts` (`registerTaskHandler` inside worker scripts, `WorkerHandlerContext.progress.emit`).
Central orchestration layer for all background tasks — since ADR-0015, the **sole** worker generation (no more v1/v2 duplication). Key files: `workerBus.ts` (orchestrator, priority queue + circuit breakers, `hasPool()`/`terminatePool()` for scoped pool lifecycle), `workerPool.ts` (auto-scaling `MIN`→`MAX_WORKERS_INFERENCE`, idle timeout), `taskQueue.ts` (priority queue: `critical > high > normal > low` — no `background` tier), `circuitBreaker.ts` (per-worker health gate), `deadLetterQueue.ts`, `protocolHandler.ts` (typed postMessage + version negotiation, currently unused by the live `runTask()` path — not wired up), `workerBootstrap.ts` (`registerTaskHandler` inside worker scripts; handlers receive `WorkerHandlerContext.emitProgress(stage, progress, message?)`, a flat function, not `context.progress.emit(...)`).

**All constants** re-exported from `constants.ts`. **Schemas** (Zod) in `schemas.ts` gate cross-thread messages. After changes: `pnpm exec vitest run tests/unit/workerBus`.

Expand Down Expand Up @@ -405,7 +406,8 @@ Feature-specific implementation patterns (Plot Board, ProForge Pipeline, scene-l

See `AUDIT.md` and `TODO.md`. Key items:
- `app/listenerMiddleware.ts` — redux-undo `StateWithHistory` typing at boundaries.
- `workers/inference.worker.ts` — `@huggingface/transformers` v3 path alias in `tsconfig.json`; if alias breaks, restore `@ts-expect-error`.
- `workers/v2/inference.worker.ts` (v1 deleted, ADR-0015) — `@huggingface/transformers` v3 path alias in `tsconfig.json`; if the alias breaks, fix the path alias or the package's type declaration directly — do not suppress with `@ts-expect-error` (conflicts with the suppression-ratchet policy above).
- **`WorkerBus.runTask()` doesn't enforce `timeoutMs`** — confirmed via inspection, no timer on either the bus or worker-side rejects a hung task. Pre-existing, tracked in ADR-0015/TODO.md, not yet fixed.
- **DS-5:** Delete legacy bridge block from `index.css` — deferred until DS-1 verified in production.
- **B-1 (IDB encryption):** Passphrase UX complete (`IdbUnlockModal`, `PassphraseModal`). Actual IDB read/write integration for stores is Phase 4 (service-layer only currently).
- **B-2 (Voice WASM):** Engine + download UI shipped. Remaining: E2E integration test coverage.
Expand Down
24 changes: 15 additions & 9 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,19 @@ Status: 🔄 in progress | ⬜ open | ✅ done
- ✅ **DuckDB-WASM self-hosted** — replaced an unversioned, already-CSP-dead third-party CDN.
- ✅ **Doc-truth fixes** — fabricated `tauri-plugin-stronghold` claim removed; canonical Vercel URL unified with a new drift gate; CI Storybook 3×→1× + a genuinely broken test-runner invocation fixed.
- ✅ **Coverage ratchet** raised to CI-measured values (L79/F72/B65/S77).
- ⬜ **Worker-generation consolidation (F-14, new this sprint)** — both a v1 and a WorkerBus-v2
generation of the DuckDB and local-inference workers are live simultaneously
(`docs/adr/0014-worker-generation-duplication.md`). Needs a dedicated migration sprint: decide
the target generation (v2, per `CLAUDE.md`'s own architecture framing), plan the 3 v1 call-site
migrations (`services/duckdb/duckdbClient.ts`, `services/ai/localNlpService.ts`,
`services/ai/localEmbeddingService.ts`), then remove the v1 worker files. Until then, any worker-
level fix (CSP, protocol, error handling) must be checked against **both** generations.
- ✅ **Worker-generation consolidation (F-14)** — the dedicated migration sprint
`docs/adr/0014-worker-generation-duplication.md` deferred, executed as 4 stacked PRs:
#286 (parity gate + real handler tests for the previously-untested v2 workers), #287 (DuckDB —
`duckdbClient.ts` migrated, `workers/duckdbWorker.ts` deleted), #288 (embeddings —
`localEmbeddingService.ts` migrated), #290 (NLP — `localNlpService.ts` migrated,
`workers/inference.worker.ts` deleted, closing out v1 entirely). Decision record:
`docs/adr/0015-worker-generation-consolidation.md` (supersedes 0014). Correction loops on #287/#288
surfaced and fixed real bugs the original v2 files had never been exercised against: DuckDB
`params` were silently dropped (already live in `telemetryService.ts`'s parameterized writes),
`terminatePool()` could leave tasks hanging forever, a respawned DuckDB worker lost its
connection, and `ensureInferencePool()` was imported but never defined (broke Vercel's prod
build — see the entry below). **Merge status:** #286 merged; #287/#288/#290 open, stacked, CI
green — merge in order before the next release.
- ✅ **Pre-existing, unrelated E2E a11y finding surfaced during this sprint's CI runs** —
`tests/e2e/a11y.spec.ts` "writer version control panel has no serious axe violations" failed
deterministically (`color-contrast` 4.47 vs. 4.5 required, `--sc-accent` `#92400e` over its own
Expand All @@ -38,8 +44,8 @@ Status: 🔄 in progress | ⬜ open | ✅ done
active-toggle classes reused `--sc-accent` for both background tint and text instead of the
already-vetted `--nav-background-active`/`--nav-text-active` token pair; switched all 3
occurrences (Writer VC desktop/mobile + mobile ProForge, same shared pattern).
- **Tag `v1.24.2` + publish the GitHub Release** — maintainer action (STOP-AND-ASK per this
sprint's own hard rules); the version-bump commit is in, the tag/release is not.
- **Tag `v1.24.2` + publish the GitHub Release** — stale entry, already done (verified via
`git tag`/`gh release list`: `v1.24.2` tagged and published 2026-07-29T11:25:42Z).


## v1.24 — Post-release feature-flag refinement (2026-06-21)
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0014-worker-generation-duplication.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ADR 0014 — Two live worker generations (v1 and WorkerBus v2)

- **Status:** Accepted (documented as known debt — consolidation explicitly out of scope this sprint)
- **Status:** Superseded by [[0015-worker-generation-consolidation]] — migrated 2026-07-29
- **Date:** 2026-07-29
- **Deciders:** Maintainer + Claude Code
- **Context tags:** workers, architecture, tech-debt
Expand Down
Loading
Loading