From e0db503cb93ec0bc9843c91cc7b84a0e8627ea6e Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 23:33:29 +0800 Subject: [PATCH 1/2] docs(rag): record the S1b canary pair, root-cause packet S1d, and mark S4/T4 merged MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - HANDOVER: S1b canary evidence (baseline 32025082010 -> post 32039841070, first post run 32038751592 red on one non-golden case, root-caused); S1c/S5/S6 Ready; S4 and #212 T4 Merged; new packet S1d (final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free provider_source_gap) with §3 body and §7 prompt. - COORDINATION §3 wave 1 re-cut (S1c + S1d + G1 + S5 + S6 parallel; RAG behaviour PRs merged one at a time with own canary pairs); §7 state. - One immutable inbox request for S1d (P2 task). Co-Authored-By: Claude Fable 5 --- .../e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1.json | 14 ++++ docs/rag-improvement/COORDINATION.md | 39 ++++----- docs/rag-improvement/HANDOVER.md | 81 ++++++++++++++----- 3 files changed, 98 insertions(+), 36 deletions(-) create mode 100644 docs/outstanding-issues-inbox/e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1.json diff --git a/docs/outstanding-issues-inbox/e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1.json b/docs/outstanding-issues-inbox/e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1.json new file mode 100644 index 0000000000..cefc3915f6 --- /dev/null +++ b/docs/outstanding-issues-inbox/e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "e8a5480e-0b0e-4ab1-b7bd-6090699dd7c1", + "createdOn": "2026-08-17", + "action": "add", + "payload": { + "pri": "P2", + "type": "task", + "summary": "S1d: a hedged, cited, low-confidence fast answer collapses to a citation-free provider_source_gap in the finalizer with no extractive recovery — turned the S1b canary red on summary-discharge-guidance", + "detail": "First post-S1b canary run 32038751592 failed one case (summary-discharge-guidance): strong_routine_retrieval; final_quality_gate:provider_source_gap, grounded false, 0 citations, citation_failure_rate 0.0227. Re-run 32039841070 and 3/3 cache-bypassed live probes took the other branch: generation_fallback:generation_quality_failed; source_backed_extractive_fallback, grounded, 4 citations. Live index unchanged (0 documents/chunks/jobs changed between runs); retrieval 36/36 identical; S1b's diff is dosing-class only. Mechanism: in src/lib/rag/rag.ts (~3580-3625) every fast-route failure shape throws into shouldRecoverFastFailureExtractively (fast + strong_routine_retrieval + results -> source-backed extractive answer), but a hedged CITED confidence:low fast answer whose lead misses providerSourceGapLeadPattern and whose body matches the broader gapLikeAnswer regex in finalizeRagAnswerQualityCore (src/lib/rag/rag-extractive-answer.ts ~3822-3845) survives the loop and is converted to a citation-free evidence_gap with NO extractive recovery. Model phrasing alone chooses the branch. Fix: give the finalizer gap conversion the same extractive recovery for fast + routine + results (or detect the broad gap in-loop); keep the citation-free terminal for genuinely empty retrieval; offline fixtures for both. Own PR, RAG impact behaviour change, canary pair, Clinical Governance Preflight. Lands BEFORE S2 (A2/A3 lengthen answers, more hedged prose). Packet: docs/rag-improvement/HANDOVER.md S1d.", + "source": "RAG programme coordinator, S1b canary root-cause 2026-08-17", + "issueUlid": "01M085NVEB2K48464KM44VGVCE" + } +} diff --git a/docs/rag-improvement/COORDINATION.md b/docs/rag-improvement/COORDINATION.md index f9d98a2928..0a22f53173 100644 --- a/docs/rag-improvement/COORDINATION.md +++ b/docs/rag-improvement/COORDINATION.md @@ -67,13 +67,13 @@ their status row, and stop. They never merge, never watch CI, never dispatch can **Wave plan** (Track A is strictly consecutive — shared files and evidence dependencies; Track B is the parallel lane): -| Wave | Dispatch (one fresh chat per task) | Gate to the next wave | -| ---- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| 0 | **D1** (this docs PR) + **S4** + **S1b** + **#212 T4** in parallel; **C1** post-S1 canary (done) | D1 merged (unblocks R0); S1b merged + canary green (unblocks S1c) | -| 1 | **R0** `issues:reconcile` (serialized) + **G1** (any time, disjoint) + **S1c** (after S1b canary) | S1c merged + canary green | -| 2 | **S2** (+S2b if split) + **S5** + **S6** (S5/S6 after S4 merges) | S2 merged + canary pair + `eval:answer-quality` + Gate E | -| 3 | **S3** + #212 closure | S3 merged | -| 4 | S7+ only after explicit owner decisions (Gate B verdict, experiment appetite) | — | +| Wave | Dispatch (one fresh chat per task) | Gate to the next wave | +| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| 0 | **D1** (this docs PR) + **S4** + **S1b** + **#212 T4** in parallel; **C1** post-S1 canary (done) | D1 merged (unblocks R0); S1b merged + canary green (unblocks S1c) | +| 1 | **S1c** + **S1d** + **G1** + **S5** + **S6** in parallel (disjoint files); S1c and S1d merged one at a time, each with its own canary pair (R0 reconcile landed as #2045) | S1c and S1d merged + canaries green | +| 2 | **S2** (+S2b if split) + **S5** + **S6** (S5/S6 after S4 merges) | S2 merged + canary pair + `eval:answer-quality` + Gate E | +| 3 | **S3** + #212 closure | S3 merged | +| 4 | S7+ only after explicit owner decisions (Gate B verdict, experiment appetite) | — | Waves 1–2 of the original plan (S1 + S4, then S2 + S5 + S6) were re-cut on 2026-08-17 after S1 landed: the owner chose **R1 before S2** (S1's residual R1 explains most remaining lithium @@ -174,17 +174,20 @@ Always a fresh owner ask, every single time: ## 7. Current state and next actions (update on change) -- **Done:** PR #1895 (guide), PR #1899 (A1 phase 1), PR #1908 (worker handover) — merged - 2026-08-13. **PR #2022 (S1)** merged 2026-08-17 as squash `2bd146eed`, landing verified by - content; canary pair baseline run 31964560921 (`8f8d111ab`) → post run 32025082010 - (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45. Sibling - `#212` T3 (PR #2023) merged as `440a34f71`. -- **Owner decisions 2026-08-17:** R1 before S2; governance Option B. -- **Next (Wave 0):** dispatch S4, S1b and #212 T4 as three fresh chats; merge this docs PR; - then R0 reconcile. G1 may start any time. Prompts: HANDOVER §7 (S1b, S1c, G1, S4) and the - #212 handover artifact (T4). -- **Waiting on owner:** merges as PRs open; S1b's canary approval when it merges; later S2's - canary + `eval:answer-quality` + Gate E. +- **Done (2026-08-17):** #1895, #1899, #1908 (2026-08-13); **#2022 S1** (squash `2bd146eed`); + **#2024 D1** (`78fe906b8`); **#2035 S1b** (merge `92f7618c0`); **#2036 S4** (`f5b093291`); + **#2037 #212 T4** (`1726537b7`); **#2045 R0 reconcile** (28 requests; #212 closed). Canary + pairs: S1 baseline 32025082010 → S1b post 32039841070, recall 1.0/1.0, zero per-case rr + regressions, answer gate 45/45. The first S1b post run (32038751592) was red on one non-golden + case; root-caused to the finalizer gap-recovery hole → packet S1d, not S1b. +- **Owner decisions:** R1 before S2; governance Option B; S1d lands before S2. +- **Wave 1 (dispatch now, parallel, disjoint files):** S1c (`rag-claim-support.ts`), S1d + (`rag-extractive-answer.ts` finalizer / `rag.ts` fast-failure block), G1 (`types.ts`, + `answer-stream-contract.ts`, `rag-row-contracts.ts`), S5 (harness/telemetry), S6 (`eval/docling/`). + Prompts: HANDOVER §7. Merge order for the RAG-behaviour ones: S1c and S1d each get their own + canary pair; merge one, run its canary, then the next. +- **Wave 2:** S2 (+S2b) after S1c and S1d canaries are green. **Wave 3:** S3; S7+ owner decisions. +- **Waiting on owner:** merges as PRs open; canary approvals for S1c, S1d, S2. - **Live board (artifact, owner-private):** RAG Master Plan v2 — `https://claude.ai/code/artifact/d5dba709-0df3-40e3-8a45-15997231533d`. diff --git a/docs/rag-improvement/HANDOVER.md b/docs/rag-improvement/HANDOVER.md index e258c297fd..99f374c148 100644 --- a/docs/rag-improvement/HANDOVER.md +++ b/docs/rag-improvement/HANDOVER.md @@ -66,24 +66,25 @@ generation-quality verdict on fallback`), merged 2026-08-13 — structured ## 2. Status table — update in every programme PR -| Packet | Scope | Branch | PR | State | Canary / evidence refs | -| ---------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | -| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | -| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | -| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | -| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | -| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/rag-a1-r2-r3-claim-support-` | — | Blocked on S1b + its canary | needs canary pair | -| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | -| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1b + S1c | canary pair + `eval:answer-quality` + Gate E | -| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | -| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | -| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | PR open 2026-08-17 — awaiting owner merge | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | -| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Blocked on S4 | — | -| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Blocked on S4 | — | -| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | -| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | -| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | PR open 2026-08-17 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | +| Packet | Scope | Branch | PR | State | Canary / evidence refs | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | --------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Guide | Programme guide | `claude/rag-plan-review-guide-vhrls9` | #1895 | Merged 2026-08-13 | docs-only | +| Handover | Multi-session handover + coordination | `claude/rag-plan-review-guide-vhrls9` | #1908 / #2024 | Merged 2026-08-13; coordination layer PR #2024 | docs-only | +| S0 | A1 phase 1: structured fallback diagnostics | `claude/lithium-generation-quality-debug-ji1vce` | #1899 | Merged 2026-08-13 | offline 93/93 focused | +| S1 | A1 phase 2: rung-1 verification-faithfulness fixes | `claude/s1-rag-mitigation-231-86c182` | #2022 | Merged 2026-08-17 (squash `2bd146eed`, landed by content) | 8 pre-fix + 5 post-fix live probes 2026-08-17; offline 583/583; canary pair run 31964560921 (baseline `8f8d111ab`) -> run 32025082010 (`2bd146eed`): recall 1.0/1.0, zero per-case rr regressions, answer gate 45/45; rung-2 measurement in `docs/audit/live-drift-forensics-2026-08.md` §5 | +| S1b | A1 rung 3 (R1): pre-deadline strong routing for dosing class | `claude/s1b-rag-dosing-routing-6u1mik` | #2035 | Merged 2026-08-17 (PR #2035, merge `92f7618`) | canary pair pending: baseline run 32025082010 (`2bd146eed`) -> post-merge dispatch (owner-approved); offline 586/586 + verify:pr-local heavy scope green | +| S1c | A1 residuals R2 + R3: claim-support strictness | `claude/rag-a1-r2-r3-claim-support-` | — | Ready — dispatch now (S1b merged, canary pair green 2026-08-17) | needs canary pair | +| S1d | A1 final-gate gap recovery: hedged cited low-confidence fast answers must recover extractively, not collapse to a citation-free `provider_source_gap` | `claude/rag-a1-final-gate-gap-recovery-` | — | Ready — dispatch now; parallel-safe with S1c (different file); lands before S2 | needs canary pair; evidence: canary runs 32038751592 (red) vs 32039841070 (green), 3/3 live probes on the extractive branch | +| G1 | Governance: provenance tag for document-summary rows (Option B) | `claude/rag-g1-document-context-origin-` | — | Ready — disjoint; owner decided Option B 2026-08-17 | no canary (no behaviour change) | +| S2 | A2 (+A3): composition menu + moderate length | `claude/rag-a2-composition-` | — | Blocked on S1b + S1c | canary pair + `eval:answer-quality` + Gate E | +| S2b | A3: moderate length (if separate review needed) | `claude/rag-a3-length-` | — | Blocked on S2 | — | +| S3 | A4: follow-up suggestion refinement | `claude/rag-a4-follow-ups-` | — | Blocked on S2 + S2b | — | +| S4 | B0: adversarial fixtures + baseline + register | `claude/packet-s4-adversarial-fixtures-5ho5tp` | #2036 | Merged 2026-08-17 (squash `f5b093291`) | Offline only: `check:rag:adversarial-fixtures` 24 cases / 8 categories / 6 canaries; `eval:rag:offline` 24 suites, 597 tests. Baseline `scripts/fixtures/rag-adversarial-baseline.v1.json` marks the three provider-backed gates `pending_owner_run` | +| S5 | B1+B2: telemetry assessment + offline harness | `claude/rag-b1-b2-harness-` | — | Ready — dispatch now (S4 merged) | — | +| S6 | B3: Docling lab benchmark | `claude/rag-b3-docling-lab-` | — | Ready — dispatch now (S4 merged) | — | +| S7+ | B4 shadow / B5 Ragas / B6 reranker / B7 DSPy | — | — | Gated — owner decision | — | +| #212 T1–T3 | Runtime row contracts (rag.ts, rag-candidate-sources.ts, src/app/api) — sibling stream sharing `src/lib/rag/**` | — | #1946 / #1981 / #2023 | Merged (T3 squash `440a34f71` 2026-08-17) | see the #212 ledger row; RAG surface complete for the cast class | +| #212 T4 | Runtime row contracts: `worker/main.ts` (11 casts) — sibling stream | `claude/ledger-212-tranche-4-worker-q3y6i4` | #2037 | Merged 2026-08-17 (squash `1726537b7`); #212 closed by reconcile PR #2045 | Governance Preflight complete; audit: 1 inbound cast (claim rows, per-row fail-soft) + 2 read-back param casts contracted, 9 outbound/interop left; closes #212 (inbox `done` queued in the PR) | Update rule: the session that opens a packet's PR edits its row (branch, PR number, state) in the same PR. A later session updating another packet may also correct stale rows @@ -186,6 +187,36 @@ behaviour change — canary pair -> impact first; `RAG impact: no retrieval behaviour change — provenance tag only`. Clinical > Governance Preflight. No canary. Stop at the open PR. +**S1d (final-gate gap recovery):** + +> Implement packet S1d from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: make a hedged, +> cited, low-confidence fast-route answer whose body matches the finalizer's gap-like regex recover +> to the source-backed extractive fallback (as every in-loop fast failure already does for +> `fast` + `strong_routine_retrieval` + results) instead of collapsing to a citation-free +> `final_quality_gate:provider_source_gap`. Read the S1d packet, README §A1, and +> `docs/rag-behaviour/` first — protected surface; flag RAG impact in your first message. Keep the +> citation-free gap terminal for genuinely empty retrieval. Offline fixtures for both cases. Do not +> touch `rag-claim-support.ts` (S1c) or `rag-routing.ts` (S1b). Gates: focused vitest, +> `eval:rag:offline`, `check:rag:fixtures`, `check:production-readiness`, `verify:pr-local` — paste +> decisive lines; format + commit before push. PR: full template, `RAG impact: behaviour change — +canary pair …`, Clinical Governance Preflight. Ledger append, HANDOVER S1d row, stop at the open PR. + **S2 (A2 + A3):** > Implement packet S2 from `docs/rag-improvement/HANDOVER.md` in BigSimmo/Database: From 10c19a69e418dbbfbf1895e498449d575936ae59 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Mon, 17 Aug 2026 23:33:30 +0800 Subject: [PATCH 2/2] docs(ledger): record the D2 handover sync review Co-Authored-By: Claude Fable 5 --- ...86de82505984d1baef6b335c14020361671fcd7b83b680aef2c.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/19c1cf380274d86de82505984d1baef6b335c14020361671fcd7b83b680aef2c.record.md diff --git a/docs/branch-review-records/19c1cf380274d86de82505984d1baef6b335c14020361671fcd7b83b680aef2c.record.md b/docs/branch-review-records/19c1cf380274d86de82505984d1baef6b335c14020361671fcd7b83b680aef2c.record.md new file mode 100644 index 0000000000..b2aa3d16bf --- /dev/null +++ b/docs/branch-review-records/19c1cf380274d86de82505984d1baef6b335c14020361671fcd7b83b680aef2c.record.md @@ -0,0 +1 @@ +| 2026-08-17 | claude/rag-d2-handover-sync | e0db503cb93ec0bc9843c91cc7b84a0e8627ea6e | docs/rag-improvement HANDOVER/COORDINATION sync after S1b canary pair + S1d packet + S4/T4 rows (docs-only) | docs-only; S1b canary root cause recorded; S1d packet added | verify:pr-local docs scope |