diff --git a/docs/README.md b/docs/README.md index 98666d3134..5158e9ea5d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -100,6 +100,7 @@ npm run docs:check-links - [search-rag-master-plan.md](search-rag-master-plan.md) / [search-rag-master-context.md](search-rag-master-context.md) — search/RAG roadmap and shared context - [rag-improvement/README.md](rag-improvement/README.md) — reviewed/updated RAG improvement programme: answer-quality track (intent-aware related information, length) + corrected eval/safety infra track - [rag-improvement/HANDOVER.md](rag-improvement/HANDOVER.md) — multi-session handover: per-session work packets, status table, checklists, and paste-ready prompts for executing the programme +- [rag-improvement/COORDINATION.md](rag-improvement/COORDINATION.md) — coordinator handover: programme history, wave/session decisions, babysit playbook, approvals map, and the coordination-chat bootstrap prompt - [rag-hybrid-findings-and-todo.md](rag-hybrid-findings-and-todo.md) — hybrid retrieval findings backlog - [reindex-shadow-harness-design.md](reindex-shadow-harness-design.md) — designed-only shadow reindex harness (driver not built) - [ingestion-concurrency-fix-workorder.md](ingestion-concurrency-fix-workorder.md) — ingestion concurrency workorder diff --git a/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md b/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md new file mode 100644 index 0000000000..51109058f0 --- /dev/null +++ b/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-plan-review-guide-vhrls9 | 0171669b37f4c9c7fb72a7a931b8b326d4eb72a1 | docs/rag-improvement coordination handover + HANDOVER status sync + hazard H5a note + 6 inbox requests (PR #2024, post-#2022/#2023 merge) | docs-only; conflict with merged S1 row resolved; state re-verified against origin/main 2bd146eed and canary runs 31964560921 -> 32025082010 | prettier on touched docs; verify:pr-local docs scope (see PR body) | diff --git a/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md b/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md new file mode 100644 index 0000000000..b55ceeb4e1 --- /dev/null +++ b/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-plan-review-guide-vhrls9 | 704f8aebe0ca9df115b3b2b87fea735c847c8e88 | docs: coordination-chat handover (COORDINATION.md, HANDOVER status corrections, catalogue) | clean | verify:pr-local docs-focused scope green (format, docs gates, ledger checks) | diff --git a/docs/clinical-hazard-analysis.md b/docs/clinical-hazard-analysis.md index 98c8615411..e3cb7567e8 100644 --- a/docs/clinical-hazard-analysis.md +++ b/docs/clinical-hazard-analysis.md @@ -117,6 +117,22 @@ _Pathway: coverage/confidence/trust gating. Note: the **coverage gate** (`evalua | **H5d** | Hardcoded clozapine / patient-property branches in the coverage gate use looser accept criteria than the generic path ([`rag.ts:3726`](../src/lib/rag/rag.ts)) — inconsistent strictness per drug. | Low–Medium | Medium | downstream gates apply uniformly | none asserts cross-drug parity | Per-drug literals in a safety-adjacent gate. | | **H5e** | Out-of-order response painting the wrong answer under a new question — **controlled**: a monotonic `searchRequestSeqRef` guards every state write incl. streamed progress ([`ClinicalDashboard.tsx`](../src/components/ClinicalDashboard.tsx), audit M10). | High (if it regressed) | Low | request-id guard | none (client React logic) | Minor: superseded stream has no `AbortController` (resource hygiene only). | +**H5a update (2026-08-17, coordinator review):** the fast-path half of H5a is now partially +mitigated — `deriveConfidence` (`src/lib/rag/rag-answer-support.ts`) computes +`strongestNonSynthetic` by excluding rows tagged `similarity_origin: "synthetic_text"` and +gates `"high"` on that value, so title-match synthetic scores no longer reach "high" on their +own. The **live residual** is a different, untagged path: `buildDocumentSummaryResults` +(`src/lib/rag/rag-row-contracts.ts`) stamps `similarity: 1` on document-summary rows with no +origin tag; its only caller is the document-summary route in `rag.ts`, never the general +answer path. **Owner decision 2026-08-17 — Option B:** add a distinct provenance value +(`"document_context"`) to the `similarity_origin` union and stamp it on those rows, keep the +confidence derivation unchanged (on the summary route the "query" is the document itself and +citation support is still verified by the same pipeline), pin the behaviour with tests, and +keep telemetry's `synthetic_similarity_count` from counting it. Option A (tag as +`synthetic_text` so summaries cap at "medium") was considered and rejected as a label +downgrade without a measured safety gain. Implementation packet: `docs/rag-improvement/HANDOVER.md` +§G1 (no retrieval behaviour change; no canary). + ### H6 — Adversarial / corrupted document content changes the answer _Pathway: uploaded document text → model context → answer. Detailed in the companion threat model; summarized here as a hazard._ diff --git a/docs/outstanding-issues-inbox/0649db55-c4ea-4683-b51c-4c6679754f81.json b/docs/outstanding-issues-inbox/0649db55-c4ea-4683-b51c-4c6679754f81.json new file mode 100644 index 0000000000..dc86b416e5 --- /dev/null +++ b/docs/outstanding-issues-inbox/0649db55-c4ea-4683-b51c-4c6679754f81.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "0649db55-c4ea-4683-b51c-4c6679754f81", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "task", + "summary": "R2 + R3: claim-support strictness rejects verbatim-faithful guideline restatements (directive normativity; topic-overlap dilution) — packet S1c", + "detail": "R2: normativeDirectiveActions in src/lib/rag/rag-claim-support.ts has no pattern for 'usual / recommended ... dose is ...' guideline phrasing, so imperative claims ('start lithium at 500 mg nocte') fail against descriptive norms; reproduced offline on the EMHS lithium chunk. R3: a claim synthesising two adjacent source bullets fails the >=50% single-segment topic-overlap requirement even when every atom matches; reproduced offline. Fix R2 with a small pattern addition plus adversarial negatives; MEASURE R3 before loosening. One PR after S1b merges and its canary is green; RAG impact behaviour change; canary pair. Packet: docs/rag-improvement/HANDOVER.md S1c. Stop: no grounding-gate weakening beyond the two named artefacts.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS71R6BG9X2VAGXMM1A1G" + } +} diff --git a/docs/outstanding-issues-inbox/64dd6966-0697-4b70-a7ee-c8d526729404.json b/docs/outstanding-issues-inbox/64dd6966-0697-4b70-a7ee-c8d526729404.json new file mode 100644 index 0000000000..d4f269ec87 --- /dev/null +++ b/docs/outstanding-issues-inbox/64dd6966-0697-4b70-a7ee-c8d526729404.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "64dd6966-0697-4b70-a7ee-c8d526729404", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P3", + "type": "task", + "summary": "Governance Option B decided: tag document-summary rows with similarity_origin 'document_context', keep the confidence label — implement as packet G1", + "detail": "Owner decision 2026-08-17 on the question queued by #212 tranche 3 (buildDocumentSummaryResults stamps similarity: 1 with no similarity_origin; deriveConfidence excludes only synthetic_text). Option B: add a new similarity_origin value document_context to the union in src/lib/types.ts and to src/lib/answer-stream-contract.ts, stamp it in buildDocumentSummaryResults (src/lib/rag/rag-row-contracts.ts), keep deriveConfidence unchanged and pin it with discriminating tests, keep rag.ts synthetic_similarity_count from counting it, and update docs/clinical-hazard-analysis.md H5a. Rationale: the only caller is the document-summary route where the query is the document itself and citation support is still verified; Option A (tag as synthetic_text so summaries cap at medium) rejected as a label downgrade without a measured safety gain. RAG impact: no retrieval behaviour change; no canary. Packet: docs/rag-improvement/HANDOVER.md G1. Closes the P1 governance question row once landed.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS7GT0MSNT8C71GX4KF5V" + } +} diff --git a/docs/outstanding-issues-inbox/6c0d5004-45a4-42c8-b001-92428f76a0b0.json b/docs/outstanding-issues-inbox/6c0d5004-45a4-42c8-b001-92428f76a0b0.json new file mode 100644 index 0000000000..f6a7c9acab --- /dev/null +++ b/docs/outstanding-issues-inbox/6c0d5004-45a4-42c8-b001-92428f76a0b0.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "6c0d5004-45a4-42c8-b001-92428f76a0b0", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P1", + "type": "task", + "summary": "R1: unbudgeted strong escalation makes provider_timeout the dominant lithium fallback — route the dosing class to strong before the deadline (packet S1b)", + "detail": "S1 (PR #2022) post-fix live probes: 'Lithium dosing?' 4/4 source-only, 3/4 as provider_timeout. fast_unsupported_retry_strong launches a strong generation into the fast route's leftover ~10-13 s; only the truncation self-heal is deadlineAllowsGenerationRetry-gated. Ladder rung 3 (README A1): route medication_dose_risk / dosing to the strong route in chooseAnswerRoute (src/lib/rag/rag-routing.ts) BEFORE the route deadline is created — not in shouldRetryWithStrongAfterFast, and NOT a budget change (#231 stop condition stands). Own PR, RAG impact behaviour change, canary pair, Clinical Governance Preflight, check:production-readiness. Owner decided 2026-08-17 this lands before S2 (A2/A3 add length; length under the unbudgeted retry pushes more dosing queries into timeout). Packet: docs/rag-improvement/HANDOVER.md S1b.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS6Y4DP6M3GKV5EFQXYPF" + } +} diff --git a/docs/outstanding-issues-inbox/a3ba2812-2984-4fad-b959-6dfe69fb6aa8.json b/docs/outstanding-issues-inbox/a3ba2812-2984-4fad-b959-6dfe69fb6aa8.json new file mode 100644 index 0000000000..f3d7e7ad97 --- /dev/null +++ b/docs/outstanding-issues-inbox/a3ba2812-2984-4fad-b959-6dfe69fb6aa8.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "a3ba2812-2984-4fad-b959-6dfe69fb6aa8", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Recurring 'Unhandled server request error' on /api/search and /api/search/universal in Sentry — unowned, pre-dates #1946", + "detail": "Sentry (clinibase-xz): three issue groups in 24h, 17 events, 0 users impacted, on /api/search and /api/search/universal, all with culprit chunk 1261.js:2:4801. First seen 2026-08-14T08:44:37Z on release c9b089c9, about six hours before PR #1946 merged, so not caused by the row contracts. Recorded in the #212 tranche-1 handover; never captured durably until now. Next: triage the Sentry groups (read-only Sentry MCP or dashboard), map chunk 1261.js to source via the release's source maps, reproduce locally with the request shapes Sentry recorded. Stop: do not silence the error path; search routes are clinical output.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS758BTVMVKSAMK68GMS5" + } +} diff --git a/docs/outstanding-issues-inbox/c156f1c0-8d83-460a-b1b5-ba970d664405.json b/docs/outstanding-issues-inbox/c156f1c0-8d83-460a-b1b5-ba970d664405.json new file mode 100644 index 0000000000..fd280753a9 --- /dev/null +++ b/docs/outstanding-issues-inbox/c156f1c0-8d83-460a-b1b5-ba970d664405.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "c156f1c0-8d83-460a-b1b5-ba970d664405", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P3", + "type": "rec", + "summary": "eval-canary.yml is post-merge only (repository_dispatch + Sunday cron, no ref input) — record this in docs/rag-behaviour so sessions stop expecting a branch canary", + "detail": ".github/workflows/eval-canary.yml triggers on repository_dispatch type eval-canary and schedule cron 0 18 * * 0; it always loads the default branch and has no workflow_dispatch or ref input, so a canary can only ever measure main. Consequence for the RAG programme: 'canary pair' means latest green run on main before the merge -> a dispatch after the merge (gh api repos/BigSimmo/Database/dispatches -f event_type=eval-canary), compared with npm run eval:retrieval:compare -- --fail-on-regression on the eval-canary-output artifacts. Recorded correctly for S1 (baseline run 31964560921 -> post run 32025082010, zero regressions). Next: add one paragraph to docs/rag-behaviour/safeguards.md canary protocol; no workflow change.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS7D3TYJ0XPNKTBCWS5KV" + } +} diff --git a/docs/outstanding-issues-inbox/ff73ce15-6890-44a1-8ddb-2919306a0e86.json b/docs/outstanding-issues-inbox/ff73ce15-6890-44a1-8ddb-2919306a0e86.json new file mode 100644 index 0000000000..00a8551ffe --- /dev/null +++ b/docs/outstanding-issues-inbox/ff73ce15-6890-44a1-8ddb-2919306a0e86.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "ff73ce15-6890-44a1-8ddb-2919306a0e86", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P3", + "type": "rec", + "summary": "source_metadata pin in rag-row-contracts.ts is data-backed only — add check (jsonb_typeof(metadata) = 'object') on documents or loosen the pin", + "detail": "src/lib/rag/rag-row-contracts.ts requires source_metadata to be a JSON object (z.record) while documents.metadata jsonb permits arrays and scalars. Live query 2026-08-14 (project sjrfecxgysukkwxsowpy): select jsonb_typeof(metadata), count(*) from public.documents group by 1 returned a single row, object = 2851, so nothing breaks today — but the guarantee is data, not schema, and a future ingest could make retrieval throw RetrievalRowShapeError for that document. Options: a check constraint via a new migration (role postgres; run check:migration-role) or loosen the pin. The module's doc comment also claims every required field is 'not null in supabase/schema.sql', which is true for nine fields and false for source_metadata — one-clause docs fix in the same PR.", + "source": "RAG programme coordinator, S1 PR #2022 residuals and #212 handover follow-ups, 2026-08-17", + "issueUlid": "01M07SS78VND10QT3784W1137Y" + } +} diff --git a/docs/rag-improvement/COORDINATION.md b/docs/rag-improvement/COORDINATION.md new file mode 100644 index 0000000000..f9d98a2928 --- /dev/null +++ b/docs/rag-improvement/COORDINATION.md @@ -0,0 +1,209 @@ +# RAG improvement programme — coordination handover + +**Status:** maintained coordinator context (2026-08-13). This file hands the **coordinator +role** to a future chat: the session that dispatches worker sessions, drives their PRs to +green, merges them, approves canaries, and keeps the programme moving. Worker sessions read +[HANDOVER.md](HANDOVER.md); the design authority is [README.md](README.md); this file is +everything the coordinator additionally needs — the programme's history, the operating +decisions already made with the owner, and the playbook learned running the first cycle. + +The three files are deliberately layered: README = _what to build and why_, HANDOVER = +_what each worker session does_, COORDINATION = _how the owner's chat runs the programme_. +Do not duplicate content across them; link instead. + +--- + +## 1. Programme history (what has already happened) + +Chronological, with the durable artefacts: + +1. **PDF review + programme guide.** The owner's uploaded "Clinical RAG Improvement + Programme" PDF was reviewed against the repo; agreed items kept, seven corrections made + (stale pin, `check:rag:fixtures` name collision, intent detection already existing in + triplicate, missing engagement with `docs/rag-behaviour/refuted-approaches.md`, `#231`'s + real stop condition, single-owner roles, wrong follow-up assumptions). Result: + `docs/rag-improvement/README.md` — **PR #1895, merged 2026-08-13**. The owner's + priorities baked in: answer-quality track first, moderate (~1.5×) length increase. +2. **A1 phase 1 landed via babysit.** **PR #1899** ("record the structured + generation-quality verdict on fallback", branch + `claude/lithium-generation-quality-debug-ji1vce`) — structured `GenerationQualityError` + diagnostics, `generation_quality_gate:*` retry reasons, fallback metadata, + `scripts/probe-generation-quality.ts`. Driving it to merge required three fixes (see the + playbook in §5): a real merge conflict with main resolved by keeping **both** sides' + additive fallback diagnostics; a pre-existing test failure fixed by updating the expected + verdict `template_like_answer` → `empty_after_sanitize` (current `sanitizeAnswerText` + strips that fixture to empty, so the first rung of the failure-reason ladder wins); and a + maintainability-budget breach fixed by extracting the prompt constant verbatim to + `src/lib/rag/rag-answer-instructions.ts`. **Merged 2026-08-13** (merge commit `c924b65`). +3. **Worker handover pack.** `docs/rag-improvement/HANDOVER.md` — packets S1–S7+, status + table, checklists, per-packet paste-ready prompts. **PR #1908, merged 2026-08-13.** + +Nothing else from the programme has started. S1 and S4 are the next dispatches. + +## 2. The coordinator role + +The coordinator chat does what worker sessions must not: + +- **Dispatch** exactly one worker session per packet, using the HANDOVER §7 prompt for that + packet — or, preferably, the _tailored_ version the previous worker session emitted as its + closing "handoff for the next chat" section (see §4). +- **Babysit** each worker PR to green and **merge** it (workers stop at the open PR by + contract). Use the playbook in §5. +- **Approve and dispatch canaries.** Live eval-canary pairs (~$1–2/run, `eval-canary` + repository dispatch, no `workflow_dispatch`) are per-run owner decisions. The coordinator + relays the owner's approval; it never self-approves. Same for provider-backed evals + (`eval:rag`, `eval:quality`, `eval:retrieval:quality`, `verify:release`, + `check:supabase-project`, `test:live`) and live Supabase reads. +- **Keep the status table honest** — verify HANDOVER §2 rows against live GitHub/git state + at the start of each coordination turn; correct stale rows in the next docs-carrying PR. +- **Capture follow-ups** into `docs/outstanding-issues.md` via `/issues` before context is + lost. + +Worker sessions implement one packet, run gates, open one PR, append the ledger, update +their status row, and stop. They never merge, never watch CI, never dispatch canaries. + +## 3. Scheduling and session-configuration decisions (already made with the owner) + +**Wave plan** (Track A is strictly consecutive — shared files and evidence dependencies; +Track B is the parallel lane): + +| Wave | Dispatch (one fresh chat per task) | Gate to the next wave | +| ---- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| 0 | **D1** (this docs PR) + **S4** + **S1b** + **#212 T4** in parallel; **C1** post-S1 canary (done) | D1 merged (unblocks R0); S1b merged + canary green (unblocks S1c) | +| 1 | **R0** `issues:reconcile` (serialized) + **G1** (any time, disjoint) + **S1c** (after S1b canary) | S1c merged + canary green | +| 2 | **S2** (+S2b if split) + **S5** + **S6** (S5/S6 after S4 merges) | S2 merged + canary pair + `eval:answer-quality` + Gate E | +| 3 | **S3** + #212 closure | S3 merged | +| 4 | S7+ only after explicit owner decisions (Gate B verdict, experiment appetite) | — | + +Waves 1–2 of the original plan (S1 + S4, then S2 + S5 + S6) were re-cut on 2026-08-17 after S1 +landed: the owner chose **R1 before S2** (S1's residual R1 explains most remaining lithium +timeouts and A2/A3 add length), and **governance Option B** for the document-summary +`similarity: 1` question. Track A stays strictly consecutive; Track B and the `#212` sibling +stream are the parallel lanes. + +One session per packet, never two — the duplicate-PR trap is real and recorded (`#292`: +PRs #1766/#1767 shipped the same conversion twice, four hours apart). + +**Per-packet session settings** (agreed with the owner): + +| Task | Model | Reasoning effort | Plan mode | Fast mode | +| ----------------------- | ----------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------ | --------- | +| Coordinator (this chat) | Fable | Medium — routing, not building | Only while re-planning | No | +| C1 canary compare | coordinator | Low | No | — | +| D1 docs / R0 reconcile | Sonnet | Medium / Low | No | OK | +| S1 (done) | Fable | High (not xhigh — the plan exists; the session is build/diagnosis) | Yes — approve the mitigation-rung choice before edits | No | +| S1b (R1 routing) | Fable | High | Yes — approve the class list before edits | No | +| S1c (R2 + R3) | Fable | High | Yes | No | +| G1 (Option B tag) | Opus | High | Optional | No | +| S2 (and S2b if split) | Fable | High | Yes — eyeball the composition menu + length targets (clinical judgement) | No | +| S3 | Opus | Medium–high | Optional | OK | +| S4 | Opus | Medium | No | OK | +| S5/S6 | Fable | High for sandbox/isolation design, medium for plumbing | For the design decisions | No | +| #212 T4 (worker) | Fable | High — concurrent job claimer, retry/poison-pill risk | Yes — approve per-site fail/skip decisions | No | + +Rationale anchors: `AGENTS.md` "Reasoning effort calibration" (RAG surfaces = top row; +xhigh planning over-produces; debugging inverts plan/build effort), and plan mode only in +sessions the owner will check back on — a fire-and-forget session parked in plan mode +stalls indefinitely. + +## 4. The worker-prompt pattern (chained handoffs) + +Every packet prompt from HANDOVER §7 gets one closing paragraph added when dispatched +(established with the owner for S1; reuse the pattern for all packets): + +> **Handoff for the next chat:** as the final section of your last message, produce the +> ready-to-paste prompt for the next session. Start from the next packet's prompt in +> `docs/rag-improvement/HANDOVER.md` §7 and tailor it with this session's outcomes: your PR +> number and branch, the key findings/decisions the next packet depends on, any approval +> still pending, and anything you discovered that changes the next packet (flag it as a +> proposed HANDOVER edit rather than silently deviating). Label it clearly "Paste into the +> next session after you merge my PR". + +The S1 dispatch prompt already exists in tailored form (HANDOVER §7 S1 + the reading order +including merged PR #1899's diff, the `rag.ts` 4,362-line budget warning, and the handoff +paragraph). If the coordinator needs to regenerate it, compose: HANDOVER §7 S1 prompt + +"read merged #1899's diff first; do not re-implement it" + budget warning + handoff +paragraph. + +## 5. Babysit playbook (learned driving #1899 to merge) + +- **Subscribe, don't poll:** `subscribe_pr_activity` for the PR; webhook events wake the + session. `send_later` self check-ins may require an MCP approval the session cannot grant + non-interactively — if blocked, rely on events and a manual owner nudge. +- **`dirty` is a claim, not a fact:** confirm with + `git merge-tree --write-tree origin/main ` before treating GitHub's + `mergeable_state: dirty` as a real conflict. Behind-but-clean → sync once, late. +- **Resolve conflicts in a scratch worktree** (`git worktree add … `), symlink + `node_modules` from the main checkout for gates. When two sessions added _different + additive_ fields at the same site (the common case here), the right resolution keeps both + sides. Regenerate derived docs (`npm run docs:update`) rather than hand-resolving counts. +- **A failing test on the merge head may predate you:** check it against the PR's own head + before blaming the merge. The #1899 case was a semantic drift — main's sanitizer changed + which quality-verdict token a fixture trips; the fix was updating the expectation with a + comment, not weakening the mechanism. +- **`src/lib/rag/rag.ts` has a 4,362-line no-growth budget** + (`scripts/check-maintainability-budgets.mjs`, currently ~4,343 after the + `rag-answer-instructions.ts` extraction). Additive diagnostics PRs can breach it on + merge. Fix by extracting a cohesive module (pure move), never by raising the budget. +- **Expected noise:** CodeRabbit rate-limit comment edits (no action); the `ci-triage` bot + comment restating failures you are already fixing (no action); `PR required` failing as + the aggregate of a failure you've fixed on a superseded head (no action once pushed). +- **The PR-handoff stop hook re-arms** after every PR the coordinator session itself + creates, blocking GitHub PR reads and loop tools. The owner's explicit babysit/merge + instruction is the unlock; the deny message names the exact marker-removal command. +- **After merge:** verify the landing by content (the merged tree contains the expected + files/lines — squash merges orphan branch SHAs), clean up worktrees, and skip + babysit-only ledger pushes (the repo forbids tips whose sole delta is a babysit record). + +## 6. Standing approvals map + +Granted by the programme's standing decisions (no fresh ask needed): + +- Dispatching worker sessions with the HANDOVER prompts; merging worker PRs that are green + with all required checks and no unresolved actionable threads, when the owner has asked + for babysit-to-merge on that PR or wave. +- Offline gates, `verify:pr-local`, focused tests, `eval:rag:offline`, + `check:rag:fixtures` — all local/offline verification. +- Docs-only PRs maintaining HANDOVER/COORDINATION state. + +Always a fresh owner ask, every single time: + +- Live eval-canary dispatches (each run), any provider-backed eval or live test, live + Supabase queries, reindexing, migrations, deployment actions, enabling any flag in + production, and anything in `docs/rag-behaviour/refuted-approaches.md` territory. + +## 7. Current state and next actions (update on change) + +- **Done:** PR #1895 (guide), PR #1899 (A1 phase 1), PR #1908 (worker handover) — merged + 2026-08-13. **PR #2022 (S1)** merged 2026-08-17 as squash `2bd146eed`, landing verified by + content; canary pair baseline run 31964560921 (`8f8d111ab`) → post run 32025082010 + (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45. Sibling + `#212` T3 (PR #2023) merged as `440a34f71`. +- **Owner decisions 2026-08-17:** R1 before S2; governance Option B. +- **Next (Wave 0):** dispatch S4, S1b and #212 T4 as three fresh chats; merge this docs PR; + then R0 reconcile. G1 may start any time. Prompts: HANDOVER §7 (S1b, S1c, G1, S4) and the + #212 handover artifact (T4). +- **Waiting on owner:** merges as PRs open; S1b's canary approval when it merges; later S2's + canary + `eval:answer-quality` + Gate E. +- **Live board (artifact, owner-private):** RAG Master Plan v2 — + `https://claude.ai/code/artifact/d5dba709-0df3-40e3-8a45-15997231533d`. + +## 8. Coordination-chat bootstrap prompt + +Paste this into a fresh chat to stand up the coordinator: + +> You are the coordination chat for the Clinical KB RAG improvement programme in +> BigSimmo/Database. Read, in order: `docs/rag-improvement/COORDINATION.md` (your role, +> history, decisions, playbook), `docs/rag-improvement/HANDOVER.md` (worker packets, +> status table, worker prompts), and skim `docs/rag-improvement/README.md` §5 for the +> track structure. Then verify live state before trusting any of it: check the HANDOVER +> status table against the actual open/merged PRs and branches on GitHub, and note any +> stale rows for correction in the next docs PR. +> +> Report back: (1) current programme state, (2) which wave we are in and what is ready to +> dispatch, (3) the exact tailored prompt(s) for the next worker session(s) per +> COORDINATION §4, and (4) anything waiting on my decision (merges, canary approvals). +> Then wait for my instruction — dispatching sessions, babysitting a PR to merge, or +> approving a canary are my calls, per the standing approvals map in COORDINATION §6. +> Never start implementing a packet yourself in this chat; the coordinator dispatches and +> merges, workers build. diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index 6bb3e97976..3564aeecf8 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -1,11 +1,13 @@ # RAG improvement programme — multi-session handover -**Status:** maintained handover ledger (2026-08-13). This is the context file every cloud +**Status:** maintained handover ledger (2026-08-17). This is the context file every cloud session implementing the programme reads first. The design authority is [README.md](README.md) in this directory; the protected-surface rules live in `docs/rag-behaviour/`. This file adds what the guide deliberately does not carry: current programme state, the per-session work packets, the paste-ready prompts, and the status table -each session must update. +each session must update. The coordinator-side layer — who dispatches sessions, merges their +PRs, and approves canaries, plus the babysit playbook — lives in +[COORDINATION.md](COORDINATION.md). **How to use this file (for the agent reading it at session start):** @@ -20,7 +22,7 @@ each session must update. --- -## 1. Programme state snapshot (as of 2026-08-13) +## 1. Programme state snapshot (as of 2026-08-17) - **Landed:** the reviewed/updated programme guide (`docs/rag-improvement/README.md`, PR #1895, merged 2026-08-13). @@ -29,29 +31,59 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured `GenerationQualityError` diagnostics, `generation_quality_gate:*` retry reasons, fallback metadata, and the provider-safe `scripts/probe-generation-quality.ts` probe. Sessions S1+ must treat that content as existing code. Do not re-implement it. +- **Landed:** **A1 phase 2 (S1) was PR #2022**, merged 2026-08-17 as squash `2bd146eed` — + ladder rung 1. On healthy retrieval latency, generation-quality gates still dominated + source-only fallbacks (eight owner-approved live probes; the decisive sertraline case + generated in 3.9 s and still fell back), so route budgets stayed untouched. Two + text-normalisation artefacts were fixed: markdown-emphasis atom splitting + (`foldMarkdownEmphasis` in `src/lib/answer-verification.ts`) and PDF visual-wrap + claim-segment fragmentation (`reflowBoundedSourceLines(..., { requireContinuationStart })` + in `src/lib/rag/rag-source-segmentation.ts`, used by `src/lib/rag/rag-claim-support.ts`). + Post-merge canary pair: baseline run 31964560921 (`8f8d111ab`) → post run 32025082010 + (`2bd146eed`), document/content recall 1.0/1.0, zero per-case rr regressions, answer gate + 45/45. **Residuals recorded, not fixed:** R1 unbudgeted strong escalation + (`fast_unsupported_retry_strong` launches strong generation into the fast route's leftover + ~10–13 s; now the dominant lithium fallback mode as `provider_timeout`) → packet S1b; R2 + directive-normativity strictness (`normativeDirectiveActions` lacks "usual / recommended … + dose is …" phrasing) and R3 topic-overlap dilution → packet S1c. +- **Owner decisions 2026-08-17:** (1) **R1 before S2** — A2/A3 add answer length, and length + under the still-unbudgeted strong retry pushes more dosing queries into `provider_timeout`, + not fewer; (2) **governance Option B** for the document-summary `similarity: 1` question + (see G1 below): add a provenance tag, keep the confidence label, no canary. +- **Sibling stream sharing `src/lib/rag/**`:** ledger `#212` runtime row contracts. T1 (PR + #1946, `rag.ts`), T2 (PR #1981, `rag-candidate-sources.ts`) and T3 (PR #2023, + `src/app/api/**`, squash `440a34f71`) are merged; the RAG surface is complete for that + defect class. T4 (`worker/main.ts`) is its own PR. Any RAG-surface packet here must not + re-touch the row-contract helpers in `src/lib/rag/rag-row-contracts.ts` except G1. - **Key issue refs:** `#231` (source-only degradation with healthy retrieval — A1), `#001` (semantic rerank stays off — constrains B6), `#100` (perceived latency / no token - streaming — constrains A1), `#292` (check open PRs before acting on a queued item). + streaming — constrains A1), `#292` (check open PRs before acting on a queued item), `#212` + (row contracts), `#324` / `#330` (verify landing by content after squash merges). - **Decisive constraint from #231:** the 35–40 s route-budget probes were tested and rejected — generation completed inside the deadline and still failed the quality gate. Never "fix" A1 by raising `answerRouteBudgetMs` without new evidence that directly rebuts - that recorded result. + that recorded result. S1's evidence (2026-08-17) re-confirmed it. ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: evidence-chosen mitigation | `claude/s1-rag-mitigation-231-86c182` | #2022 | PR open — rung 1 verification-faithfulness fixes | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | -| S2 | A2: composition menu | `claude/rag-a2-composition-` | — | Blocked on S1 evidence | — | -| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | -| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | -| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-` | — | Ready (parallel-safe) | — | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Blocked on S4 | — | -| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Blocked on S4 | — | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| ---------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/rag-a1-r1-routing-` | — | Ready — dispatch now (owner decided R1 before S2, 2026-08-17) | needs canary pair | +| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/rag-a1-r2-r3-claim-support-` | — | Blocked on S1b + its canary | needs canary pair | +| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | +| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1b + S1c | canary pair + `eval:answer-quality` + Gate E | +| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | +| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | +| S4 | B0: adversarial fixtures + baseline + register | `claude/rag-b0-adversarial-fixtures-` | — | Ready — dispatch now (parallel-safe) | — | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Blocked on S4 | — | +| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Blocked on S4 | — | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | +| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-` | — | Ready — dispatch now | Clinical Governance Preflight; closes #212 if the audit supports it | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows @@ -94,6 +126,66 @@ owner merges) and never at watching CI. - **Done:** PR open with the mitigation, its evidence trail (which gate reasons dominated, why this rung of the ladder), and a fallback-rate non-inferiority argument. +### S1b — A1 rung 3 (R1): route the dosing class to the strong route before the deadline + +- **Precondition:** S1 merged (PR #2022, `2bd146eed`) and its canary pair green. Owner decision + 2026-08-17: R1 lands before S2. Check the open PR list for a routing PR first (`#292`). +- **Evidence:** S1's post-fix probes left "Lithium dosing?" 4/4 source-only, 3/4 as + `provider_timeout`: `fast_unsupported_retry_strong` launches a strong generation into the + fast route's leftover ~10–13 s and only the truncation self-heal is + `deadlineAllowsGenerationRetry`-gated. The fix is routing, not budget. +- **Work:** in `chooseAnswerRoute` (`src/lib/rag/rag-routing.ts`) route the dosing / + `medication_dose_risk` class (and `drug_dosing` only if evidence supports) to the strong + route **before** the route deadline is created. Do not change + `shouldRetryWithStrongAfterFast` as the fix; do not touch `answerRouteBudgetMs`, + `OPENAI_ANSWER_TIMEOUT_MS`, any quality gate, or fallback caching. Non-dosing classes keep + their routing unchanged and the PR must argue that explicitly. +- **Files:** `src/lib/rag/rag-routing.ts`, targeted tests. `rag.ts` has a 4,362-line + no-growth budget — extract, never raise. +- **Gates:** focused vitest on routing, `eval:rag:offline`, `check:rag:fixtures`, + `check:production-readiness` (answer routing changed), `verify:pr-local`; `RAG impact: +behaviour change — canary pair -> ` (owner + approves the dispatch); Clinical Governance Preflight. Live probes only with owner approval + and always `node --env-file=.env.local scripts/run-tsx.mjs scripts/probe-generation-quality.ts ""`. +- **Done:** PR open with the routing diff, offline proof, and the fallback-rate + non-inferiority argument for non-dosing classes. + +### S1c — A1 residuals R2 + R3: claim-support strictness + +- **Precondition:** S1b merged and its canary green (keeps canary attribution clean). +- **Work:** R2 — add a `normativeDirectiveActions` pattern in `src/lib/rag/rag-claim-support.ts` + for guideline phrasing "usual / recommended … dose is …" so imperative claims ("start + lithium at 500 mg nocte") verify against descriptive norms, with adversarial negatives + proving unrelated imperatives still fail. R3 — a claim synthesising two adjacent source + bullets fails the ≥50 % single-segment topic-overlap requirement even when every atom + matches; **measure** how often before loosening anything, and loosen only with a + discriminating negative test. Both reproduce offline on the EMHS lithium chunk. +- **Hard boundaries:** no grounding-gate weakening beyond the two named artefacts; no + retrieval/ranking edit; no budget change. +- **Gates:** as S1b; `RAG impact: behaviour change — canary pair …`; Clinical Governance + Preflight; `check:production-readiness`. + +### G1 — governance decision: provenance tag for document-summary rows (Option B) + +- **Decision (owner, 2026-08-17):** `buildDocumentSummaryResults` + (`src/lib/rag/rag-row-contracts.ts`) stamps `similarity: 1` on document-summary rows with + no `similarity_origin`; its only caller is the document-summary route in `rag.ts`, never + the general answer path; `deriveConfidence` (`src/lib/rag/rag-answer-support.ts`) excludes + only `"synthetic_text"` from `strongestNonSynthetic`. On the summary route the "query" is + the document itself and citation support is still verified by the same pipeline, so the + "high" label stays. Option A (tag as `synthetic_text` → summaries cap at "medium") is + recorded as rejected. +- **Work:** add a new origin value (proposed `"document_context"`) to the + `similarity_origin` union in `src/lib/types.ts` and to `src/lib/answer-stream-contract.ts`; + stamp it in `buildDocumentSummaryResults`; leave `deriveConfidence` unchanged and pin it + with tests (tagged doc-summary rows still reach "high"; `synthetic_text` rows still do + not); `rag.ts` `synthetic_similarity_count` must not count the new value; update + `docs/clinical-hazard-analysis.md` H5a. +- **Gates:** focused vitest, `eval:rag:offline`, `check:rag:fixtures`, `verify:pr-local`; + `RAG impact: no retrieval behaviour change — provenance tag only; confidence derivation +unchanged, pinned by test`; Clinical Governance Preflight. No canary. +- **Precondition:** none — disjoint from S1b/S1c/T4; may run in parallel. + ### S2 — A2 composition, with a separately reviewable A3 length fallback - **Precondition:** S1 merged (its evidence determines how much length headroom exists). @@ -251,6 +343,41 @@ inside the session. > condition. No provider-backed commands or live canary dispatches without asking me first. > Finish at an open PR with ledger append and an updated HANDOVER status row, then stop. +**S1b (A1 rung 3 — R1 routing):** + +> Implement packet S1b from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: route the +> dosing / `medication_dose_risk` class to the strong route in `chooseAnswerRoute` +> (`src/lib/rag/rag-routing.ts`) before the route deadline is created, per README §A1 ladder +> rung 3 and S1's recorded residual R1. Read the S1b packet, README §A1, and +> `docs/rag-behaviour/` first — protected surface; flag RAG impact in your first message. Do +> not change route budgets, quality gates, or `shouldRetryWithStrongAfterFast`. Reproduce +> routing offline with focused tests; ask me before any live probe (probe needs +> `node --env-file=.env.local`). Gates: focused vitest, `eval:rag:offline`, +> `check:rag:fixtures`, `check:production-readiness`, `verify:pr-local` — paste decisive +> lines; `npm run format` and commit before push. PR: full template, `RAG impact: behaviour +change — canary pair …`, Clinical Governance Preflight, non-inferiority argument for +> non-dosing classes. Ledger append, HANDOVER S1b row, stop at the open PR. Handoff for the +> next chat: produce the tailored S1c prompt. + +**S1c (R2 + R3):** + +> Implement packet S1c from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: the +> `normativeDirectiveActions` phrasing pattern (R2) and the measured topic-overlap decision +> (R3) in `src/lib/rag/rag-claim-support.ts`, with adversarial negatives. Protected surface — +> flag RAG impact first. No grounding-gate weakening beyond the two named artefacts, no +> retrieval edit, no budget change. Gates and PR body as S1b. Stop at the open PR. + +**G1 (governance Option B):** + +> Implement packet G1 from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: add the +> `similarity_origin` value `"document_context"` to the union in `src/lib/types.ts` and to +> `src/lib/answer-stream-contract.ts`, stamp it in `buildDocumentSummaryResults` +> (`src/lib/rag/rag-row-contracts.ts`), keep `deriveConfidence` unchanged and pin it with +> discriminating tests, keep `synthetic_similarity_count` from counting it, and update +> `docs/clinical-hazard-analysis.md` H5a with the decision. Protected surface — flag RAG +> impact first; `RAG impact: no retrieval behaviour change — provenance tag only`. Clinical +> Governance Preflight. No canary. Stop at the open PR. + **S2 (A2 + A3):** > Implement packet S2 from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: