From 704f8aebe0ca9df115b3b2b87fea735c847c8e88 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 09:31:59 +0000 Subject: [PATCH 1/3] docs: add coordination-chat handover for the RAG improvement programme MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/rag-improvement/COORDINATION.md — the coordinator-side layer over the worker HANDOVER: programme history (PRs #1895/#1899/#1908), the coordinator role definition, the wave dispatch plan and per-packet model/effort/plan-mode settings, the chained worker-prompt handoff pattern, the babysit playbook learned driving #1899 to merge, the standing-approvals map, and a paste-ready coordination-chat bootstrap prompt. Links it from the docs catalogue and HANDOVER, and corrects HANDOVER's now-verifiable status rows (#1908 merged). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UfJeqoq59y5xKmgRGHjqyA --- docs/README.md | 1 + docs/rag-improvement/COORDINATION.md | 188 +++++++++++++++++++++++++++ docs/rag-improvement/HANDOVER.md | 6 +- 3 files changed, 193 insertions(+), 2 deletions(-) create mode 100644 docs/rag-improvement/COORDINATION.md diff --git a/docs/README.md b/docs/README.md index 98666d3134..5158e9ea5d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -100,6 +100,7 @@ npm run docs:check-links - [search-rag-master-plan.md](search-rag-master-plan.md) / [search-rag-master-context.md](search-rag-master-context.md) — search/RAG roadmap and shared context - [rag-improvement/README.md](rag-improvement/README.md) — reviewed/updated RAG improvement programme: answer-quality track (intent-aware related information, length) + corrected eval/safety infra track - [rag-improvement/HANDOVER.md](rag-improvement/HANDOVER.md) — multi-session handover: per-session work packets, status table, checklists, and paste-ready prompts for executing the programme +- [rag-improvement/COORDINATION.md](rag-improvement/COORDINATION.md) — coordinator handover: programme history, wave/session decisions, babysit playbook, approvals map, and the coordination-chat bootstrap prompt - [rag-hybrid-findings-and-todo.md](rag-hybrid-findings-and-todo.md) — hybrid retrieval findings backlog - [reindex-shadow-harness-design.md](reindex-shadow-harness-design.md) — designed-only shadow reindex harness (driver not built) - [ingestion-concurrency-fix-workorder.md](ingestion-concurrency-fix-workorder.md) — ingestion concurrency workorder diff --git a/docs/rag-improvement/COORDINATION.md b/docs/rag-improvement/COORDINATION.md new file mode 100644 index 0000000000..8259fc28f0 --- /dev/null +++ b/docs/rag-improvement/COORDINATION.md @@ -0,0 +1,188 @@ +# RAG improvement programme — coordination handover + +**Status:** maintained coordinator context (2026-08-13). This file hands the **coordinator +role** to a future chat: the session that dispatches worker sessions, drives their PRs to +green, merges them, approves canaries, and keeps the programme moving. Worker sessions read +[HANDOVER.md](HANDOVER.md); the design authority is [README.md](README.md); this file is +everything the coordinator additionally needs — the programme's history, the operating +decisions already made with the owner, and the playbook learned running the first cycle. + +The three files are deliberately layered: README = _what to build and why_, HANDOVER = +_what each worker session does_, COORDINATION = _how the owner's chat runs the programme_. +Do not duplicate content across them; link instead. + +--- + +## 1. Programme history (what has already happened) + +Chronological, with the durable artefacts: + +1. **PDF review + programme guide.** The owner's uploaded "Clinical RAG Improvement + Programme" PDF was reviewed against the repo; agreed items kept, seven corrections made + (stale pin, `check:rag:fixtures` name collision, intent detection already existing in + triplicate, missing engagement with `docs/rag-behaviour/refuted-approaches.md`, `#231`'s + real stop condition, single-owner roles, wrong follow-up assumptions). Result: + `docs/rag-improvement/README.md` — **PR #1895, merged 2026-08-13**. The owner's + priorities baked in: answer-quality track first, moderate (~1.5×) length increase. +2. **A1 phase 1 landed via babysit.** **PR #1899** ("record the structured + generation-quality verdict on fallback", branch + `claude/lithium-generation-quality-debug-ji1vce`) — structured `GenerationQualityError` + diagnostics, `generation_quality_gate:*` retry reasons, fallback metadata, + `scripts/probe-generation-quality.ts`. Driving it to merge required three fixes (see the + playbook in §5): a real merge conflict with main resolved by keeping **both** sides' + additive fallback diagnostics; a pre-existing test failure fixed by updating the expected + verdict `template_like_answer` → `empty_after_sanitize` (current `sanitizeAnswerText` + strips that fixture to empty, so the first rung of the failure-reason ladder wins); and a + maintainability-budget breach fixed by extracting the prompt constant verbatim to + `src/lib/rag/rag-answer-instructions.ts`. **Merged 2026-08-13** (merge commit `c924b65`). +3. **Worker handover pack.** `docs/rag-improvement/HANDOVER.md` — packets S1–S7+, status + table, checklists, per-packet paste-ready prompts. **PR #1908, merged 2026-08-13.** + +Nothing else from the programme has started. S1 and S4 are the next dispatches. + +## 2. The coordinator role + +The coordinator chat does what worker sessions must not: + +- **Dispatch** exactly one worker session per packet, using the HANDOVER §7 prompt for that + packet — or, preferably, the _tailored_ version the previous worker session emitted as its + closing "handoff for the next chat" section (see §4). +- **Babysit** each worker PR to green and **merge** it (workers stop at the open PR by + contract). Use the playbook in §5. +- **Approve and dispatch canaries.** Live eval-canary pairs (~$1–2/run, `eval-canary` + repository dispatch, no `workflow_dispatch`) are per-run owner decisions. The coordinator + relays the owner's approval; it never self-approves. Same for provider-backed evals + (`eval:rag`, `eval:quality`, `eval:retrieval:quality`, `verify:release`, + `check:supabase-project`, `test:live`) and live Supabase reads. +- **Keep the status table honest** — verify HANDOVER §2 rows against live GitHub/git state + at the start of each coordination turn; correct stale rows in the next docs-carrying PR. +- **Capture follow-ups** into `docs/outstanding-issues.md` via `/issues` before context is + lost. + +Worker sessions implement one packet, run gates, open one PR, append the ledger, update +their status row, and stop. They never merge, never watch CI, never dispatch canaries. + +## 3. Scheduling and session-configuration decisions (already made with the owner) + +**Wave plan** (Track A is strictly consecutive — shared files and evidence dependencies; +Track B is the parallel lane): + +| Wave | Dispatch in parallel | Gate to the next wave | +| ---- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | **S1 + S4** | both PRs merged (S2 needs S1's code on main; S5 needs S4's fixtures) | +| 2 | **S2 + S5 + S6** | S2 merged + its canary pair approved and green (S2b, the split-out A3 length step, follows S2 in the same lane if HANDOVER keeps them separate) | +| 3 | **S3** (after S2b when used) | S3 merged | +| 4 | S7+ only after explicit owner decisions (Gate B verdict, experiment appetite) | — | + +One session per packet, never two — the duplicate-PR trap is real and recorded (`#292`: +PRs #1766/#1767 shipped the same conversion twice, four hours apart). + +**Per-packet session settings** (agreed with the owner): + +| Packet | Model | Reasoning effort | Plan mode | Fast mode | +| --------------------- | ------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------ | --------- | +| S1 | Fable | High (not xhigh — the plan exists; the session is build/diagnosis) | Yes — approve the mitigation-rung choice before edits | No | +| S2 (and S2b if split) | Fable | High | Yes — eyeball the composition menu + length targets (clinical judgement) | No | +| S3 | Fable or Opus | Medium–high | Optional | OK | +| S4 | Cheaper model fine (Opus) | Medium | No | OK | +| S5/S6 | Fable preferred | High for sandbox/isolation design, medium for plumbing | For the design decisions | No | + +Rationale anchors: `AGENTS.md` "Reasoning effort calibration" (RAG surfaces = top row; +xhigh planning over-produces; debugging inverts plan/build effort), and plan mode only in +sessions the owner will check back on — a fire-and-forget session parked in plan mode +stalls indefinitely. + +## 4. The worker-prompt pattern (chained handoffs) + +Every packet prompt from HANDOVER §7 gets one closing paragraph added when dispatched +(established with the owner for S1; reuse the pattern for all packets): + +> **Handoff for the next chat:** as the final section of your last message, produce the +> ready-to-paste prompt for the next session. Start from the next packet's prompt in +> `docs/rag-improvement/HANDOVER.md` §7 and tailor it with this session's outcomes: your PR +> number and branch, the key findings/decisions the next packet depends on, any approval +> still pending, and anything you discovered that changes the next packet (flag it as a +> proposed HANDOVER edit rather than silently deviating). Label it clearly "Paste into the +> next session after you merge my PR". + +The S1 dispatch prompt already exists in tailored form (HANDOVER §7 S1 + the reading order +including merged PR #1899's diff, the `rag.ts` 4,362-line budget warning, and the handoff +paragraph). If the coordinator needs to regenerate it, compose: HANDOVER §7 S1 prompt + +"read merged #1899's diff first; do not re-implement it" + budget warning + handoff +paragraph. + +## 5. Babysit playbook (learned driving #1899 to merge) + +- **Subscribe, don't poll:** `subscribe_pr_activity` for the PR; webhook events wake the + session. `send_later` self check-ins may require an MCP approval the session cannot grant + non-interactively — if blocked, rely on events and a manual owner nudge. +- **`dirty` is a claim, not a fact:** confirm with + `git merge-tree --write-tree origin/main ` before treating GitHub's + `mergeable_state: dirty` as a real conflict. Behind-but-clean → sync once, late. +- **Resolve conflicts in a scratch worktree** (`git worktree add … `), symlink + `node_modules` from the main checkout for gates. When two sessions added _different + additive_ fields at the same site (the common case here), the right resolution keeps both + sides. Regenerate derived docs (`npm run docs:update`) rather than hand-resolving counts. +- **A failing test on the merge head may predate you:** check it against the PR's own head + before blaming the merge. The #1899 case was a semantic drift — main's sanitizer changed + which quality-verdict token a fixture trips; the fix was updating the expectation with a + comment, not weakening the mechanism. +- **`src/lib/rag/rag.ts` has a 4,362-line no-growth budget** + (`scripts/check-maintainability-budgets.mjs`, currently ~4,343 after the + `rag-answer-instructions.ts` extraction). Additive diagnostics PRs can breach it on + merge. Fix by extracting a cohesive module (pure move), never by raising the budget. +- **Expected noise:** CodeRabbit rate-limit comment edits (no action); the `ci-triage` bot + comment restating failures you are already fixing (no action); `PR required` failing as + the aggregate of a failure you've fixed on a superseded head (no action once pushed). +- **The PR-handoff stop hook re-arms** after every PR the coordinator session itself + creates, blocking GitHub PR reads and loop tools. The owner's explicit babysit/merge + instruction is the unlock; the deny message names the exact marker-removal command. +- **After merge:** verify the landing by content (the merged tree contains the expected + files/lines — squash merges orphan branch SHAs), clean up worktrees, and skip + babysit-only ledger pushes (the repo forbids tips whose sole delta is a babysit record). + +## 6. Standing approvals map + +Granted by the programme's standing decisions (no fresh ask needed): + +- Dispatching worker sessions with the HANDOVER prompts; merging worker PRs that are green + with all required checks and no unresolved actionable threads, when the owner has asked + for babysit-to-merge on that PR or wave. +- Offline gates, `verify:pr-local`, focused tests, `eval:rag:offline`, + `check:rag:fixtures` — all local/offline verification. +- Docs-only PRs maintaining HANDOVER/COORDINATION state. + +Always a fresh owner ask, every single time: + +- Live eval-canary dispatches (each run), any provider-backed eval or live test, live + Supabase queries, reindexing, migrations, deployment actions, enabling any flag in + production, and anything in `docs/rag-behaviour/refuted-approaches.md` territory. + +## 7. Current state and next actions (update on change) + +- **Done:** PR #1895 (guide), PR #1899 (A1 phase 1), PR #1908 (worker handover) — all + merged 2026-08-13. +- **Next:** dispatch Wave 1 — S1 (Fable, high, plan mode) and S4 (cheaper model, medium, + no plan mode) in parallel, one session each. +- **Waiting on owner:** nothing until Wave 1 PRs open; then merges, and later S2's canary + approval. + +## 8. Coordination-chat bootstrap prompt + +Paste this into a fresh chat to stand up the coordinator: + +> You are the coordination chat for the Clinical KB RAG improvement programme in +> BigSimmo/Database. Read, in order: `docs/rag-improvement/COORDINATION.md` (your role, +> history, decisions, playbook), `docs/rag-improvement/HANDOVER.md` (worker packets, +> status table, worker prompts), and skim `docs/rag-improvement/README.md` §5 for the +> track structure. Then verify live state before trusting any of it: check the HANDOVER +> status table against the actual open/merged PRs and branches on GitHub, and note any +> stale rows for correction in the next docs PR. +> +> Report back: (1) current programme state, (2) which wave we are in and what is ready to +> dispatch, (3) the exact tailored prompt(s) for the next worker session(s) per +> COORDINATION §4, and (4) anything waiting on my decision (merges, canary approvals). +> Then wait for my instruction — dispatching sessions, babysitting a PR to merge, or +> approving a canary are my calls, per the standing approvals map in COORDINATION §6. +> Never start implementing a packet yourself in this chat; the coordinator dispatches and +> merges, workers build. diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index 954600281f..68659bd597 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -5,7 +5,9 @@ session implementing the programme reads first. The design authority is [README.md](README.md) in this directory; the protected-surface rules live in `docs/rag-behaviour/`. This file adds what the guide deliberately does not carry: current programme state, the per-session work packets, the paste-ready prompts, and the status table -each session must update. +each session must update. The coordinator-side layer — who dispatches sessions, merges their +PRs, and approves canaries, plus the babysit playbook — lives in +[COORDINATION.md](COORDINATION.md). **How to use this file (for the agent reading it at session start):** @@ -42,7 +44,7 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured | Packet | Scope | Branch | PR | State | Canary / evidence refs | | -------- | ----------------------------------------------- | ------------------------------------------------ | ----- | ---------------------- | ---------------------- | | Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover | `claude/rag-plan-review-guide-vhrls9` | #1908 | Open — this PR | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 | Merged 2026-08-13 | docs-only | | S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | | S1 | A1 phase 2: evidence-chosen mitigation | `claude/rag-a1-mitigation-` | — | Not started | — | | S2 | A2: composition menu | `claude/rag-a2-composition-` | — | Blocked on S1 evidence | — | From 9f4d4d40b6b6d581435e7f714b8d4be0cefd3150 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 09:32:25 +0000 Subject: [PATCH 2/3] docs: record review ledger entry for coordination handover PR Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01UfJeqoq59y5xKmgRGHjqyA --- ...0e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md diff --git a/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md b/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md new file mode 100644 index 0000000000..b55ceeb4e1 --- /dev/null +++ b/docs/branch-review-records/e5b5d137733340e2e8440c1a3f3fb4e4d0fb9654a76f6444a1af1eb0d6467fa3.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-plan-review-guide-vhrls9 | 704f8aebe0ca9df115b3b2b87fea735c847c8e88 | docs: coordination-chat handover (COORDINATION.md, HANDOVER status corrections, catalogue) | clean | verify:pr-local docs-focused scope green (format, docs gates, ledger checks) | From 8e788591379373044e906f3f1009b9d732e55302 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 20:06:06 +0800 Subject: [PATCH 3/3] docs(ledger): record the PR #2024 coordination-handover sync review Co-Authored-By: Claude Fable 5 --- ...81c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md diff --git a/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md b/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md new file mode 100644 index 0000000000..51109058f0 --- /dev/null +++ b/docs/branch-review-records/446cd9ee36ef481c9c26ae8b85d66d4e88b005ab7e601c5123d71834c246ceda.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-plan-review-guide-vhrls9 | 0171669b37f4c9c7fb72a7a931b8b326d4eb72a1 | docs/rag-improvement coordination handover + HANDOVER status sync + hazard H5a note + 6 inbox requests (PR #2024, post-#2022/#2023 merge) | docs-only; conflict with merged S1 row resolved; state re-verified against origin/main 2bd146eed and canary runs 31964560921 -> 32025082010 | prettier on touched docs; verify:pr-local docs scope (see PR body) |