Darwin improvements: indexing scaling, retention, analytics dashboard, tests

.claude/commands/runtests.md

@@ -0,0 +1,48 @@
+---
+description: Run the project's automated checks (TypeScript, mypy, pytest if scoped) and summarise failures.
+allowed-tools: Bash, Read
+---
+
+Run the standard validation suite for whatever scope the user gave (default: both backend + frontend). Output a tight summary with pass/fail counts and the first error per failing tool — don't paste full logs back unless the user asks.
+
+# Steps
+
+1. **Decide scope from the user's message.**
+   - If they wrote `/runtests backend` → only backend checks.
+   - If `/runtests web` or `/runtests frontend` → only frontend checks.
+   - If `/runtests <path>` → only checks scoped to that path (mypy on Python files, tsc on TS files, pytest on tests dir).
+   - Otherwise → run both backend and frontend.
+
+2. **Backend checks (run sequentially, stop reporting on each):**
+   - `cd backend && python -m mypy . 2>&1 | tail -50` — mypy across the whole backend.
+   - `cd backend && pytest -x --tb=short 2>&1 | tail -50` — fast-fail pytest. Most modules don't have tests, so a "no tests collected" result is normal; that's not a failure.
+   - `cd backend && alembic heads 2>&1` — confirm single head (multiple heads means a migration was added on a parallel branch and needs merging).
+
+3. **Frontend checks:**
+   - `cd web && npx tsc --noEmit --incremental 2>&1 | tail -40` — TypeScript across `web/src/`.
+   - `cd web && npm run lint 2>&1 | tail -40` — ESLint.
+   - `cd web && npx prettier --check "src/**/*.{ts,tsx,js,jsx}" 2>&1 | tail -20` — prettier formatting check.
+
+4. **Skip what's not relevant.** Don't run frontend checks for backend-only changes and vice versa. If you can infer from `git diff --stat` that only `backend/` was touched, skip frontend.
+
+5. **Summarise**, format like:
+
+   ```
+   Backend
+     mypy        ✓ clean   (or ✗ N errors — first: <file:line — message>)
+     pytest      ✓ N passed (or ✗ N failed)
+     alembic     ✓ single head
+   Frontend
+     tsc         ✓ clean   (or ✗ N errors — first: <file:line — message>)
+     lint        ✓ clean
+     prettier    ✓ clean
+   ```
+
+6. **If anything failed**, follow up with the user: "X failures above. Want me to fix them, or are you debugging a specific area?" Don't auto-fix unless explicitly told to.
+
+# Notes
+
+- Everything in this file's command list is in `.claude/settings.json`'s allow-list, so none of it should prompt for permission.
+- `pytest` runs against the live local stack (Postgres / Vespa / model server). If they're not running, integration-style tests will fail — flag that as "stack not running" rather than as test failures.
+- Don't run `npx playwright test` from this command. E2E is too slow for an inner-loop check; use it on demand only.
+- Don't run `pre-commit run --all-files` from here either — that's slow and overlaps with the per-tool checks above. The user can `pre-commit run` themselves before commit.

.claude/hooks/check_changed.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+#
+# Stop hook: validates whatever Claude touched in the current working tree.
+# Runs once per Claude response (not per Edit), so a multi-file task
+# triggers a single type-check pass rather than N redundant ones.
+#
+# Scope:
+#   - If any web/**/*.ts(x) is dirty -> run `tsc --noEmit --incremental`
+#     across the whole web project (file-scoped tsc would need its own
+#     tsconfig; full project + incremental cache is the right balance).
+#   - If any backend/**/*.py is dirty -> run mypy *only on those files*
+#     (file-scoped mypy is fast).
+#
+# Output is piped through `tail` so it can't dominate Claude's context.
+# Exits 0 even on failure — we want Claude to see the error in its next
+# turn, not block the response from completing.
+set -u
+
+PROJECT_DIR="${CLAUDE_PROJECT_DIR:-$(git rev-parse --show-toplevel 2>/dev/null)}"
+[ -z "$PROJECT_DIR" ] && exit 0
+cd "$PROJECT_DIR" || exit 0
+
+# Modified files in the working tree (staged + unstaged + untracked).
+# Filename is field 2 in `git status --porcelain` output. We strip the
+# "XY " prefix manually so paths with spaces survive.
+DIRTY=$(git status --porcelain 2>/dev/null | sed 's/^...//')
+[ -z "$DIRTY" ] && exit 0
+
+ts_changed=false
+py_changed=()
+
+while IFS= read -r f; do
+  case "$f" in
+    web/*.ts|web/*.tsx|web/**/*.ts|web/**/*.tsx) ts_changed=true ;;
+    backend/*.py|backend/**/*.py) py_changed+=("${f#backend/}") ;;
+  esac
+done <<< "$DIRTY"
+
+# --- TypeScript ----------------------------------------------------------
+if [ "$ts_changed" = true ] && [ -d "web" ]; then
+  echo "── tsc (web) ──"
+  ( cd web && npx --no-install tsc --noEmit --incremental 2>&1 | tail -40 ) || true
+fi
+
+# --- Python (mypy on changed files only) ---------------------------------
+# Prefer the project venv's Python (where mypy is installed). Fall back to
+# python3 / python on PATH if the venv doesn't exist.
+PY=""
+if [ -x "$PROJECT_DIR/.venv/bin/python" ]; then
+  PY="$PROJECT_DIR/.venv/bin/python"
+elif command -v python3 >/dev/null 2>&1; then
+  PY="$(command -v python3)"
+elif command -v python >/dev/null 2>&1; then
+  PY="$(command -v python)"
+fi
+
+if [ "${#py_changed[@]}" -gt 0 ] && [ -d "backend" ] && [ -n "$PY" ]; then
+  echo "── mypy (changed files) ──"
+  ( cd backend && "$PY" -m mypy --no-error-summary "${py_changed[@]}" 2>&1 | tail -25 ) || true
+fi
+
+exit 0

.claude/settings.json

@@ -0,0 +1,100 @@
+{
+  "$schema": "https://json.schemastore.org/claude-code-settings.json",
+  "_comment": "Project-shared Claude Code settings. Commit this file. Personal overrides go in .claude/settings.local.json (gitignored). Permissions allow Claude to run validation commands without prompting; the deny list always wins. Hooks auto-run type-checkers after Edit/Write so type regressions surface immediately.",
+
+  "permissions": {
+    "allow": [
+      "Bash(git status:*)",
+      "Bash(git diff:*)",
+      "Bash(git log:*)",
+      "Bash(git show:*)",
+      "Bash(git branch:*)",
+      "Bash(git stash list)",
+      "Bash(git remote -v)",
+
+      "Bash(ls:*)",
+      "Bash(grep:*)",
+      "Bash(find:*)",
+      "Bash(cat:*)",
+      "Bash(head:*)",
+      "Bash(tail:*)",
+      "Bash(wc:*)",
+      "Bash(tree:*)",
+      "Bash(file:*)",
+
+      "Bash(ps aux)",
+      "Bash(ps -ef)",
+      "Bash(docker ps:*)",
+      "Bash(docker logs:*)",
+      "Bash(docker exec * psql:*)",
+      "Bash(docker network inspect:*)",
+      "Bash(redis-cli ping)",
+      "Bash(redis-cli llen:*)",
+      "Bash(redis-cli info:*)",
+
+      "Bash(python -m mypy:*)",
+      "Bash(pytest:*)",
+      "Bash(pre-commit run:*)",
+      "Bash(black --check:*)",
+      "Bash(alembic heads)",
+      "Bash(alembic history:*)",
+      "Bash(alembic current)",
+      "Bash(alembic check)",
+      "Bash(PYTHONPATH=* python -c:*)",
+      "Bash(PYTHONPATH=* python -m:*)",
+
+      "Bash(npx tsc --noEmit:*)",
+      "Bash(npx tsc --noEmit --incremental:*)",
+      "Bash(npm run lint:*)",
+      "Bash(npm run typecheck:*)",
+      "Bash(npx prettier --check:*)",
+      "Bash(npx playwright test --list:*)",
+
+      "Bash(gh api repos/:*)",
+      "Bash(gh pr view:*)",
+      "Bash(gh issue view:*)",
+      "Bash(gh pr list:*)",
+      "Bash(gh issue list:*)",
+      "Bash(gh pr checks:*)",
+      "Bash(gh run view:*)",
+      "Bash(gh run list:*)"
+    ],
+
+    "deny": [
+      "Bash(git push:*)",
+      "Bash(git push --force:*)",
+      "Bash(git push -f:*)",
+      "Bash(git reset --hard:*)",
+      "Bash(git checkout --:*)",
+      "Bash(git clean -fd:*)",
+      "Bash(git branch -D:*)",
+      "Bash(rm -rf /:*)",
+      "Bash(rm -rf ~:*)",
+      "Bash(sudo:*)",
+      "Bash(docker compose down -v:*)",
+      "Bash(docker volume rm:*)",
+      "Bash(docker rm -f:*)",
+      "Bash(alembic downgrade:*)",
+      "Bash(npm publish:*)",
+      "Bash(gh pr merge:*)",
+      "Bash(gh pr close:*)"
+    ]
+  },
+
+  "env": {
+    "PYTHONDONTWRITEBYTECODE": "1"
+  },
+
+  "hooks": {
+    "Stop": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/check_changed.sh\""
+          }
+        ]
+      }
+    ]
+  }
+}

.gitignore

@@ -8,3 +8,19 @@
 *.sw?
 docker-compose.dev.yml
 venv/
+
+# Claude Code: keep shared project settings (.claude/settings.json) but not
+# personal overrides or session state.
+.claude/settings.local.json
+.claude/projects/
+.claude/cache/
+
+# Claude Code /export outputs (timestamped conversation transcripts).
+# Pattern: YYYY-MM-DD-HHMMSS-<slug>.txt — typically dropped into the cwd
+# from which Claude Code was launched.
+[0-9][0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]-*.txt
+
+# Stray LLM API request payloads sometimes dumped during local debugging.
+# These contain prompt content (chat history, internal references, names).
+backend/requestdata.json
+requestdata.json