From c3ca68fa7f13e2d7ae6c4c96ccfcafb67a59135a Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Sat, 22 Aug 2026 02:59:55 +0800 Subject: [PATCH 1/5] =?UTF-8?q?docs(db):=20Phase=205=20close-out=20?= =?UTF-8?q?=E2=80=94=20measurement=20baselines,=20staging=20parity,=20D4?= =?UTF-8?q?=20settled=20(#316)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Discharges the last follow-on on `#316` and records the evidence. - Phase 5 close-out in the forensics dossier: like-for-like end-to-end before/after probes (text fast path 31,610 -> 955 ms; hybrid 21,757 -> 6,720 ms), a new EXPLAIN forward baseline for all four RPCs `explain_retrieval_rpc` supports, and the `rag_retrieval_logs` miss-scan baseline that §1.3 left pending. - Records two instrument limits honestly: `explain_retrieval_rpc` exposes only an outer `Function Scan` (so plan flips are not observable through it) and raises `22023` for the `_v2` RPC family. Extending it is production DDL and was deliberately not written in a read-only phase. - Finding, not absorbed: all 22 restored indexes report `idx_scan = 0` against never-reset statistics, including both trigram indexes credited with the incident fix, which makes the co-administered `ANALYZE` the better-supported explanation for the recovery. Queued as its own ledger item. - Staging brought to parity: `20260820120000` applied to `ikoiolksxqxfxgiyqpnu` by real `db push`; 211 rows, function present, `stmt_count 3` matching production. - Alarm observed clear: live-drift run `32514326022` green on `main`, first success since 2026-07-19, pinned issue #1963 auto-closed. - D4 corrected to the 2026-08-21 dashboard fact ("Deploy to production" ENABLED, production branch `main`, automatic branching ON) in both the coordination board and the forensics dossier, with the superseded safe-either-way reasoning retained and marked rather than deleted. Co-Authored-By: Claude Opus 5 --- docs/audit/live-drift-forensics-2026-08.md | 300 +++++++++++++++++- docs/database-remediation-coordination.md | 16 +- .../0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json | 12 + .../2040d1fb-6d26-4977-902f-d4a3c2c404c2.json | 14 + .../3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json | 14 + .../98e6ae7f-4190-49f3-a64d-fbf632ef6643.json | 11 + .../d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json | 14 + 7 files changed, 369 insertions(+), 12 deletions(-) create mode 100644 docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json create mode 100644 docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json create mode 100644 docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json create mode 100644 docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json create mode 100644 docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 76072e0b50..167c20c6a2 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -1684,6 +1684,228 @@ choosing any code mitigation. Residual: hybrid fan-out still costs ~8.5 s worst- by the remaining remediation phases, not a route-budget change (`#231`'s stop condition stands). `check:production-readiness` on the final state is **pending**. +### Phase 5 close-out — 2026-08-22 Perth (2026-08-21 UTC) + +_Owner-authorised **read-only** production window (`sjrfecxgysukkwxsowpy`) plus a staging-only +apply. A dedicated worktree carried the link; the main checkout stayed on its staging link and +every call re-verified `supabase/.temp/project-ref` immediately beforehand. No production DDL, no +data mutation, no paid eval._ + +#### Production is unchanged since 2026-08-21 — 8 of 8 baseline checks match + +| Check | Expected | Read | +| --------------------------------------- | -------- | ---------------- | +| `public` indexes | 210 | **210** | +| invalid / not-ready indexes | 0 | **0** | +| `supabase_migrations` rows | 211 | **211** | +| rows with no executed statements | 20 | **20** | +| `migration_history_versions()` probe | ok | **ok** | +| `migration_history_versions()` versions | 211 | **211** | +| `search_schema_health()` `ok` | true | **true** | +| latest version | — | `20260820120000` | + +Nothing diverged, so no escalation was raised on the production reading. + +#### 5.1(a) End-to-end probes — the like-for-like row §1.3 asked for + +Same endpoint and query style as the incident measurements (`POST /api/search`, semantic clinical +queries, live production data). `supabase_rpc_latency_ms` is the server-measured retrieval +component and is the comparable figure; `total` additionally carries local dev-server render +overhead in this run, so it is directionally useful rather than strictly comparable. + +| Measurement | Before (2026-08-14) | After restore + ANALYZE (2026-08-14) | **Now (2026-08-22)** | +| ------------------------------------------ | ------------------- | ------------------------------------ | -------------------- | +| Semantic, text fast path — total / rpc ms | 37.7 s / 31,610 | 4.8 s / 1,535 | **8.4 s / 955** | +| Semantic, hybrid strategy — total / rpc ms | 29.9 s / 21,757 | 17.2 s / 8,519 | **10.9 s / 6,720** | +| Warm repeat (`search_cache`) — total / rpc | not measured | not measured | **0.41 s / 0** | + +Retrieval latency has held its recovery and improved further on both strategies: the fast path is +**33× better than the incident** and 1.6× better than the 2026-08-14 post-restore reading; hybrid is +3.2× better than the incident and 1.3× better than post-restore. Nothing is materially worse than +the 2026-08-17 healthy baseline, so no escalation was raised. + +#### 5.1(b) EXPLAIN forward baseline — captured now, on the healthy system + +§1.3 never captured an EXPLAIN set; its before-numbers were end-to-end probes. This is that missing +baseline, recorded so a future incident has something to diff against. Captured with +`npm run profile:retrieval -- --analyze` (which calls the service-role-only +`public.explain_retrieval_rpc`), query `clozapine monitoring requirements`, `--match-count 24`, +`--samples 3`, public scope. Sample 1 is `first_unprimed`; managed Supabase buffers are never +flushed, so no sample is truly cold. + +| RPC | unprimed exec ms | warm median exec ms | warm median RTT ms | shared_hit (warm) | shared_read (unprimed) | +| ----------------------------------- | ---------------: | ------------------: | -----------------: | ----------------: | ---------------------: | +| `match_documents_for_query` | 42.642 | 5.949 | 102.783 | 1,175 | 10 | +| `match_document_chunks_text` | 867.942 | 35.821 | 150.652 | 7,243 | 42 | +| `match_document_lookup_chunks_text` | 13.237 | 3.013 | 126.042 | 511 | 7 | +| `match_document_table_facts_text` | 1,722.211 | 99.183 | 179.852 | 2,516 | 1,070 | + +Zero errors across all twelve samples. **The new slowest path is +`match_document_table_facts_text`** (1.72 s unprimed, 1,070 unprimed `shared_read`) — recorded here +as the outlier to watch, replacing the trigram-absence signature that dominated the incident. + +**Two limits of this instrument, stated so the baseline is not over-read:** + +1. **It cannot show plan flips.** `explain_retrieval_rpc` runs `EXPLAIN` over + `select * from public.(…)`, so the plan is a single `Function Scan`: a PL/pgSQL body's inner + plan is not exposed. Every sample reports `plan_node_types: ["Function Scan"]` and + `index_names: []`. Plan §5.1's "record plan flips (seq scan → index scan)" is therefore **not + answerable through this tool**; the index-usage read below answers the same question directly and + more durably. +2. **It cannot reach the two v2 RPCs named in the Phase 5 brief.** `explain_retrieval_rpc` accepts + exactly four names (`match_documents_for_query`, `match_document_chunks_text`, + `match_document_lookup_chunks_text`, `match_document_table_facts_text`) — verified in its only + defining migration `20260626020000_phase7_retrieval_rpc_performance.sql`, in `supabase/schema.sql`, + and on production itself: + + ``` + ERROR: 22023: Unsupported retrieval RPC: match_document_chunks_text_v2 + CONTEXT: PL/pgSQL function explain_retrieval_rpc(text,text,integer,uuid,uuid[],boolean) line 25 at RAISE + ERROR: 22023: Unsupported retrieval RPC: match_document_index_units_hybrid_v2 + ``` + + Extending the function to cover the `_v2` family is production DDL and was **not** authorised for + this phase, so it was not written. Queued as its own ledger item; the v1 siblings profiled above + cover the same two owning tables in the meantime. + +#### 5.1(c) Index usage — the finding this phase actually turned up + +`pg_stat_database.stats_reset` is **NULL** on production: index statistics are cumulative and have +never been reset, so a zero here means genuinely never scanned, not recently zeroed. Across `public`, +163 of 210 indexes show scans, `max(idx_scan)` is 262,335,345 and the total is 344,747,815 — the +counters are clearly live. + +Against that, **all 22 indexes restored or repaired by the incident response report `idx_scan = 0`** — +the 20 Phase 4 rebuilds (2026-08-19) and, more pointedly, both trigram indexes restored on 2026-08-14: + +| Index | Size | `idx_scan` | Present since | OID | +| ---------------------------------- | ------ | ---------: | ------------- | ------- | +| `document_chunks_content_trgm_idx` | 68 MB | **0** | 2026-08-14 | 1491258 | +| `documents_title_trgm_idx` | 648 kB | **0** | 2026-08-14 | 1491257 | + +The OID `1491258` is the same value §Phase 4 Step 4 recorded as _unchanged across the 2026-08-19 +push_, which is what makes the zero meaningful: the index has not been dropped and recreated, so its +counter has been accumulating for the full seven days since restoration. Both are `indisvalid` and +`indisready` with canonical `coalesce(...)` definitions. + +What _is_ serving retrieval on the same tables: + +| Index | Kind | `idx_scan` | +| ----------------------------------------------- | ---------------- | ---------: | +| `document_chunks_search_idx` | GIN `search_tsv` | **37,717** | +| `document_chunks_embedding_hnsw_idx` | HNSW vector | **4,906** | +| `documents_title_search_idx` | GIN tsvector | **37,299** | +| `document_table_facts_title_row_param_trgm_idx` | GIN trigram | **2,372** | + +The last row matters: a _different_ trigram index on a sibling table is used thousands of times, so +"trigram indexes are never chosen here" is not a blanket property of the planner or the extension. + +**Consequence for the incident narrative — stated as a correction, not a certainty.** The 2026-08-14 +Phase 5 table above attributes the 31,610 ms → 1,535 ms recovery to "restore + ANALYZE", treating the +two trigram indexes as the fix. Seven days of cumulative statistics show the restored trigram indexes +have served **zero** scans, so they cannot be carrying that recovery. `ANALYZE` was an unseparated +co-intervention in the same operation — and `20260626020000` ends by running `analyze` over exactly +`documents`, `document_chunks`, `document_table_facts`, `document_embedding_fields` and +`document_index_units` — which makes stale planner statistics the better-supported explanation for +both the 31-second regression and its recovery. This does **not** retract the repair: the two indexes +are repo-defined, the chain commands them, and drift is correctly closed by their presence. It +retracts the _attribution_, and it changes what a future 30-second retrieval incident should check +first — planner statistics before missing indexes. Raised as its own ledger item rather than absorbed +here; ~69 MB of currently unscanned index is a secondary, non-urgent question that follows from it. + +#### 5.1(d) `rag_retrieval_logs` miss-scan baseline — §1.3's second pending item, now closed + +| Table | `seq_scan` | `seq_tup_read` | `idx_scan` | live rows | total size | avg rows / seq scan | +| -------------------- | ---------: | -------------: | ---------: | --------: | ---------: | ------------------: | +| `rag_retrieval_logs` | 47 | 6,620 | 98 | 640 | 1440 kB | 140.9 | +| `rag_query_misses` | 421 | 29,482 | 509 | 177 | 320 kB | 70.0 | +| `rag_queries` | 51 | 33,792 | 419 | 258 | 4760 kB | 662.6 | +| `rag_aliases` | 5,829 | 379,728 | 23,583 | 68 | 224 kB | 65.1 | +| `rag_response_cache` | 2,133 | 13,203 | 5,352 | 0 | 656 kB | 6.2 | + +**Reading: not currently a latency risk, and that is the point of recording it.** Every one of these +relations is under 5 MB with at most a few hundred live rows, so a sequential scan is a handful of +pages and cheaper than an index descent — the planner's choice is correct at this scale. The row to +watch is `rag_aliases`: 5,829 sequential scans over 68 rows is free today and becomes a real cost if +that table ever grows. This table is the baseline; the trigger for action is growth in +`seq_tup_read`, not the presence of `seq_scan`. + +#### 5.2 — confirmed already satisfied, and re-confirmed with fresh data + +Ledger `#231` already records that plan §5.2 "is satisfied by S1's 2026-08-17 healthy-latency +probes", and re-graded itself P1 → P2 on that basis with residual **R4** (chronic ~30 s strong-route +`provider_timeout` on `metformin-renal-dosing` and `valproate-pregnancy`, with a safe source-backed +extractive fallback, never model synthesis). This phase **confirms that reading rather than +reopening it**: measured retrieval now costs 955 ms (fast path) to 6,720 ms (hybrid) against an +`answerRouteBudgetMs.fast` of 25,000 ms, so retrieval consumes 4–27% of the budget and is no longer +capable of binding it. The 2026-08-14 verdict — that pre-generation latency _was_ the binding cause +during the incident window — stands for that window and is now closed out. R4 is a generation-side +residual and keeps `#231`'s existing stop condition; **no separate R4 row was created**, per `#231`'s +own instruction. + +One inconsistency found and queued, not edited here: the _recommended-queue_ row for `#231` still +presents it as A1 / "immediate approved live investigation", contradicting the P2 re-grade in its own +detail row. Queued through `issues:update` for the next reconcile. + +#### 5.3 — no eval canary owed + +Plan §5.3 gates `eval:retrieval:quality` on "if any Phase 3 RPC deploy changed behaviour". Phase 3 +was reframed to repo-side codification and deployed **zero** behaviour change: `#316` records manifest +`def_hash` equal to live for all ten `match_*` functions, and live-drift `32131517648` reported 0 +function mismatches. §5.3 therefore reduces to `check:production-readiness`, run once at the end of +this phase. No paid eval was run and none was requested. + +``` +$ npm run check:production-readiness +[Production Readiness] +Project: Clinical KB Database (sjrfecxgysukkwxsowpy) +PASS (9): + - Node runtime 24.19.0 matches required Node 24.x. + - Boot guard invokes requireQueryHashSecret(); the query-hash HMAC fails closed in production (PIA-2). + - Local override file .env.local is present + - Server env includes required Supabase project values. + - OpenAI API key is configured. + - OpenAI safety identifiers use a deployment-secret HMAC; raw owner IDs are not sent. + - RAG_QUERY_HASH_SECRET is set; logged clinical-query hashes are keyed HMAC pseudonyms (PIA-2). + - HEALTH_DEEP_PROBE_SECRET is set for authorized deep health probes. + - Supabase project config points to Clinical KB Database. +READY: no blocking production-readiness failures. +``` + +Nine checks pass, zero blocking failures, and the gate independently confirms the operator +environment is bound to `Clinical KB Database` (`sjrfecxgysukkwxsowpy`). **§1.3's "`check:production-readiness` +on the final state is pending" is closed.** + +#### Staging catch-up — the owed apply, now closed + +Staging (`ikoiolksxqxfxgiyqpnu`) was one version behind after the 2026-08-20 window, where both write +paths were denied by that session's auto-mode classifier. Applied here, by real `db push` (never +`migration repair`); the pending set was confirmed first as exactly one version with zero remote-only: + +``` +$ supabase migration list --linked --project-ref ikoiolksxqxfxgiyqpnu +… {"local":"20260820120000","remote":""} +$ supabase db push --linked --project-ref ikoiolksxqxfxgiyqpnu --skip-vault --yes +Applying migration 20260820120000_migration_history_versions_rpc.sql... +{"upToDate":false,"dryRun":false,"migrations":["20260820120000_migration_history_versions_rpc.sql"],…} +``` + +`--include-all` was correctly not needed: the pending version sorts after the remote tip +`20260819110500`. + +| Staging check | Before | After | +| -------------------------------------- | ---------------- | ---------------------------------- | +| `supabase_migrations` rows | 210 | **211** | +| latest version | `20260819110500` | **`20260820120000`** | +| `migration_history_versions()` present | absent | **present** | +| history row `stmt_count` | — | **3** (executed, not mark-applied) | +| probe / version_count | — | **ok / 211** | +| `public` indexes | 210 | **210** | + +`stmt_count 3` matches production's row exactly, so both tiers now carry the same executed +migration rather than a repaired history entry. **Staging parity, open by one version since +2026-08-20, is closed.** + ## Phase 6 — Future-proofing (repo-side; one migration authored, NOT deployed) _2026-08-18 (repo-only session; no hosted read or mutation)._ Built per plan §6.1–6.3, worker chat @@ -2053,7 +2275,43 @@ $ supabase migration list --linked --project-ref sjrfecxgysukkwxsowpy empty and the step should now pass. **No guard migration is owed**: the row carries executed statements, so it is not a history repair and cannot surface in the `migration_history` probe. -#### D4: strong evidence of deploy-on-merge — but the toggle itself has not been read +#### D4 — SETTLED 2026-08-21 by a direct dashboard read: deploy-on-merge is ON + +**This question is closed. The Supabase dashboard was read on 2026-08-21 and shows the GitHub +integration with "Deploy to production" ENABLED, production branch `main` — "Apply changes to your +production database when you merge into your configured production GitHub branch". Automatic +branching is also ON (limit 3, "Supabase changes only" enabled).** The inference recorded below was +correct; it is no longer an inference, and the safe-either-way hedge it justified is retired. + +Consequences, which `AGENTS.md` § "Supabase project safety" now carries as the authoritative rule: + +- **Merge approval IS production-deploy approval.** Any migration merged to `main` is applied to the + live clinical database automatically, within seconds (34 s measured, §3.7). There is no separate + deploy step to forget and no window in which to hold it back. Never merge a + `supabase/migrations/**` PR outside an approved window, and never arm auto-merge on one. +- **The post-merge gate is a green `live-drift` run**, requiring BOTH `check:drift` and + `check:migration-history`. It is _not_ `supabase migration list`: that reads recorded history only, + so it cannot distinguish an applied migration from a history row whose statements never executed — + precisely the shape of the twenty no-statements rows. `check:drift` compares the live schema itself. +- **A migration that cannot run inside a transaction cannot ship this way.** The integration applies + each migration in one transaction, so a bare `CREATE INDEX CONCURRENTLY` migration fails outright. + Index work stays operator-prebuild plus a validate-only guard migration (the `20260804110240` + pattern). +- **Branching Compute is not covered by the organisation's Spend Cap** (stated on the same dashboard + screen), so preview databases are an uncapped cost that scales with PRs touching `supabase/**`. + CI's `Migration replay` job (`db-reset-verify`, `supabase migration up --local`) independently + replays the whole chain on every database-touching PR, so preview branches are a second net rather + than the only one — that is the context for any later decision to reduce or disable them. Queued as + its own ledger item. + +The `list_branches` reading below remains accurate and is now corroborated rather than superseded: +production is bound to git `main`, and the toggle that acts on that binding is enabled. + +_The analysis that follows was written before the dashboard was read. It is retained because it +records how the question was narrowed, and why the safe-either-way rule was the correct posture while +the toggle was still unknown._ + +##### Superseded reasoning, retained: how D4 stood before the dashboard read `created_by` and `idempotency_key` are NULL for every row from `20260818090000` to `20260820120000`, including the ones this programme applied by operator `db push`, so the history table carries **no @@ -2087,14 +2345,14 @@ Production is **still bound to git `main`**, and `updated_at` predates 2026-08-1 changed that day never touched this binding, and the §3.7 mechanism is intact. Together with §3.7's 34-second apply and `20260820120000` arriving unpushed, that is strong evidence of deploy-on-merge. -**It remains an inference, and the difference matters.** No field of the branch record reports the +**SUPERSEDED 2026-08-21 — do not act on this paragraph or the rule that follows it; the dashboard read above replaces both.** _It remains an inference, and the difference matters._ No field of the branch record reports the "Deploy to production" setting. The superseded 2026-08-19 account describes that setting being changed without the binding being deleted, so "toggle off, binding intact" cannot be ruled out from here — and the two failure modes are not symmetric. Declaring D4 ON tells operators to skip `db push`; if the toggle is in fact off, every merged migration then sits unapplied and drift returns silently, which is the original incident. Declaring it OFF risks only a redundant no-op push. -**So the operative rule is the one that is correct under both states, and it must not be shortened:** +**SUPERSEDED 2026-08-21.** _The operative rule at the time — correct under both states, and deliberately not shortened:_ 1. Never merge a migration PR outside its approved window — correct if deploys happen on merge. 2. After any migration merges, run `supabase migration list --linked --project-ref ` and @@ -2104,7 +2362,7 @@ One extra command, wrong under neither hypothesis. What would replace this rule dashboard read of the toggle, or a deployment-settings API that reports it. Raised as a P1 by the Codex review of PR #2205, and correctly: the earlier wording here presented the inference as a direct read. -#### Staging is one version behind — blocked, not skipped +#### Staging is one version behind — blocked, not skipped (CLOSED 2026-08-22, see Phase 5 close-out) Staging (`ikoiolksxqxfxgiyqpnu`, verified before every call) reads **210** history rows, latest `20260819110500`, `to_regprocedure('public.migration_history_versions()') is null`. Its pending set is @@ -2116,3 +2374,37 @@ MCP `execute_sql` alike — under the live-Supabase confirmation rule added by P Read-only calls were unaffected, which is why every verification above exists. This is an authorisation gap, not a technical one: **staging parity, closed by Phase 4, is open again by one version until an operator applies it.** + +**CLOSED 2026-08-22.** An operator session applied `20260820120000` to staging by real `db push`; +staging now reads 211 history rows with `migration_history_versions()` present and `stmt_count 3`, +matching production exactly. Evidence in the "Phase 5 close-out" section, "Staging catch-up". + +## Alarm cleared — live-drift green on `main`, 2026-08-21 (run `32514326022`) + +The alignment fix landed on 2026-08-20 but had **never been observed passing**: two sessions existed +because `live-drift` was red on its `Align migration history for Supabase Preview` step (PGRST106). +Dispatched here on `main` with the fix live on production and staging brought to parity. + +`https://github.com/BigSimmo/Database/actions/runs/32514326022` — **job conclusion: `success`.** + +``` +live-drift Compare live schema drift No unexpected schema drift between live and supabase/schema.sql. +live-drift Align migration history for Supabase Preview Remote migration versions: 211 (read via rpc) +live-drift Align migration history for Supabase Preview Remote-only (Preview blockers): 0 +``` + +Every step reported `success`, including `Guard Supabase project identity`, `Compare live schema +drift`, `Align migration history for Supabase Preview`, and `Capture drift and migration-history +findings`. + +**This is the first green `live-drift` run since 2026-07-19T19:37:29Z** — measured, not assumed: of +the last 60 runs, exactly four succeeded (`2026-07-13T09:58:30Z`, `2026-07-13T17:10:55Z`, +`2026-07-19T19:37:29Z`, and this one), and the oldest run in that window is `2026-07-09`. That is a +**33-day continuous red streak**, closed. + +**Pinned issue `#1963` "Live drift check failing" auto-closed at `2026-08-21T18:38:42Z`**, by the +`drift-routing` job, seconds after the alignment step reported zero Preview blockers — the +self-closing behaviour the routing was built for, now demonstrated end to end for the first time. + +Both halves of the original `#316` alarm are therefore closed _and observed closed_: drift findings +are empty, and the job that reports them is green. diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 7e11617ecc..46e5055184 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -90,13 +90,13 @@ trusted as-is._ | 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | | 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | | 4 — Index restoration | ☑ COMPLETE 2026-08-19 — production **and** staging | **#2151** (branch `claude/db-remediation-phase4-indexes-a1661a`, head `5d3dca4dd`); forensics §Phase 4 completion; live-drift run `32171070287` | Owner-authorised off-peak window, D4 **OFF** so nothing auto-deployed. All **20** `missing_live` indexes rebuilt with `CREATE INDEX CONCURRENTLY` from canonical definitions — Batch A 14/14, Batch B 6/6, every one `indisvalid AND indisready` with normalised `pg_get_indexdef` matching canonical; zero invalid builds, zero retries, zero skips, zero lock waits; no transactional build attempted; `#102` held out. Both `unexpected_live` orphans **DROPPED** concurrently (not codified) — each is a strict leading-column subset of a present canonical index and the chain already commands both drops (`20260620000000`, `20260703030000`/`20260708000000`); neither is in `schema.sql`, so the drop moves live _into_ mirror agreement. Live now 210 indexes vs manifest 210, zero invalid anywhere. Codified by real `db push` (never `migration repair`; all four history rows carry executed statements): `20260819100000`/`100100` guard Batch A/B, **`20260819100200` discharges the 4.4 trigram-guard debt** that `20260804110240` never covered, `20260819100300` takes `required_indexes` 22→30 adopting **all 8** Phase 6.3 monitor-candidates (`search-health-unmonitored-indexes.json` 44→36, no `monitor-candidate` left; production `ok: true`). **Staging also closed:** `document_chunks_content_trgm_idx` rebuilt into the canonical `coalesce(content,'')` form (§3.3(d)), the four migrations applied by the Phase 2 method with all md5s matching, `check:drift` vs staging **green, 0 unexpected (was 19)**, corpus untouched. Live-drift **37 → 16**: `missing_live` **20 → 0**, `unexpected_live` **2 → 0**; the lone function mismatch is merge-pending and proven (live `def_hash 85df52de…` == this branch's regenerated manifest). **Two escalations, not absorbed:** PITR is OFF on production (new P2 queued) and the `migration_history` block did **not** drop — no allowlist entry was earned (empty intersection), so the 15 stay `#Q5JHBJ`'s work. | -| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | +| 5 — Measure | ☑ COMPLETE 2026-08-22 | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` against never-reset statistics, including both trigram indexes credited with the incident fix — the recovery is better explained by the co-administered `ANALYZE` than by the indexes. Queued as its own ledger item. **Two instrument gaps recorded:** `explain_retrieval_rpc` cannot show plan flips (outer `Function Scan` only) and cannot reach the `_v2` RPC family (`22023`), which would need production DDL. | | 6 — Future-proofing | ☑ COMPLETE 2026-08-19 (6.1 live; 6.2 guards live on both tiers; 6.3 repo-side) | **#2058** merged `9c660af1f` (6.1 / 6.2 contract / 6.3); follow-up #2090 merged `79e5d2a68`; **Phase 6.2 guards: branch `claude/migration-history-drift-allowlist-37444c`** (forensics §"6.2 completion"); live-drift run `32251326536` | 6.1 probe LIVE on production since 2026-08-18. **6.2 done 2026-08-19 (`#Q5JHBJ`)**: all fifteen no-statements versions classified — every one `validation`, none `superseded`/`no_ddl` (no single later executed migration re-creates every object; `COMMENT ON` is a catalog write) — and guarded by six fail-fast migrations `20260819110000`…`110500` (dropped objects / comments + cron / document FKs / 46 operational index shapes / `index_generation_id` promotion / 15 function `def_hash`es read from `schema_drift_snapshot()`). Allowlist 5 superseded + 15 validation = 20. Proof: whole chain replayed into the scratch image `Applied 210/210`, `CHAIN == MANIFEST` (every hand repair in §2.3/§3.3/Phase 4 reproduced by the chain; no reconcile needed), seven mutants raise; all six dry-ran green on production and a mutant failed there; real `db push` (pending = exactly six; rows `no_statements false`; 210 rows; corpus untouched); staging by the Phase 2 method, six md5-matched, drift green. **Live-drift on the branch: `No unexpected schema drift`, all 20 history rows allowed — `#316`'s finding set is EMPTY for the first time since 2026-07-26.** Job still red on the Phase 0 `Align migration history` step (PGRST106: `supabase_migrations` not exposed to PostgREST; it ran for the first time ever) — queued as its own P2, owner decision (expose schema / use access token / RPC). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) redundant, owner may delete. | | 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | **Owner decisions (2026-08-18):** D1 = codify-as-live (128 MB on the four hybrids, 64 MB on the other six) — DECIDED. D2 = `work_mem` eval-canary exemption — GRANTED. D3 = 6.1 deploy bundled into -the Phase 3 production window — DECIDED. D4 = Supabase GitHub auto-deploy — **REOPENED 2026-08-20, treat as ON** (the 2026-08-19 Phase 4 "disabled" reading never tested deploy-on-merge; see the 2026-08-20 window update below). Until the dashboard toggle is re-verified, assume merging a migration PR deploys it to production. +the Phase 3 production window — DECIDED. D4 = Supabase GitHub auto-deploy — **SETTLED 2026-08-21 by a direct dashboard read: "Deploy to production" is ENABLED, production branch `main` ("Apply changes to your production database when you merge into your configured production GitHub branch"); automatic branching is ON (limit 3, "Supabase changes only").** Merging a migration PR to `main` IS a production deployment, applied within seconds. Per `AGENTS.md` § "Supabase project safety": never merge a `supabase/migrations/**` PR outside an approved window, never arm auto-merge on one, and prove the result with a green `live-drift` run (BOTH `check:drift` and `check:migration-history`) — never `supabase migration list`, which reads recorded history only and cannot tell an applied migration from a row whose statements never executed. **Production window — CLOSED 2026-08-18 without a push (PR #2123 → forensics §3.7).** The authorised window's pre-flight `supabase migration list` showed all five `20260818*` migrations @@ -105,7 +105,7 @@ no mark-applied path was used, and production received zero writes. Verified rea `ok`; ten `work_mem` values = D1; live-drift run `32131517648` = **0 function mismatches**, 20 `missing_live`, 2 `unexpected_live`, 15 `migration_history`. The RPC track of `#316` is closed. -**KEY FINDING — D4: strong evidence of deploy-on-merge; the "disabled" reading does not hold (2026-08-20, below). The toggle itself has not been read — act on the safe-either-way rule, not on the inference.** The Supabase GitHub integration (Branching, production +**KEY FINDING — D4 is SETTLED (2026-08-21, direct dashboard read): "Deploy to production" is ENABLED with production branch `main`, so deploy-on-merge is a fact, not an inference. The safe-either-way hedge below is retired and retained only as superseded reasoning; `AGENTS.md` carries the operative rule.** The Supabase GitHub integration (Branching, production bound to git `main`, branch record from 2026-06-27) auto-applies every migration merged to `main` onto production — live-drift bracketed `110000–112000` to ~34 s after #2106 merged. This undermines the explicit-window model the plan and playbook assume: a merged migration IS a production deploy. @@ -150,13 +150,13 @@ needs the same migration by the Phase 2 method. Until that window runs, the week this one step and #1963 will not self-close. **Update 2026-08-20 (window) — the alignment fix is LIVE on production; staging is one version behind; -D4: strong evidence of deploy-on-merge, toggle not directly read.** PR #2198 merged before the window (squash `a341832af`), and the pre-flight +D4 has SINCE BEEN SETTLED (2026-08-21, dashboard read) — see the D4 correction above.** PR #2198 merged before the window (squash `a341832af`), and the pre-flight found `20260820120000` **already applied** — `stmt_count 3`, executed statements, not mark-applied — so `db push` was never run against production. Verified read-only: `prosecdef true`, `provolatile s`, `search_path=""`, `proacl postgres=X/postgres | service_role=X/postgres`, and the function returns `probe: ok` with `version_count 211` against 211 history rows and 211 local files. No guard migration is -owed. **Two things for the owner:** (1) **D4: the "disabled" reading does not hold, but the toggle has -not been read.** `created_by` is NULL on every recent history row, so the database itself proves +owed. **Two things for the owner:** (1) **D4 — SUPERSEDED 2026-08-21; retained as the reasoning of the time. The toggle has since been read and deploy-on-merge is ON.** _As written then: the "disabled" reading does not hold, but the toggle had +not been read._ `created_by` is NULL on every recent history row, so the database itself proves nothing. `list_branches(sjrfecxgysukkwxsowpy)` still returns a live Git branch record binding **production** to git `main`: `{"name":"main","is_default":true,"git_branch":"main","project_ref":"sjrfecxgysukkwxsowpy","created_at":"2026-06-27","updated_at":"2026-07-04"}`. @@ -176,9 +176,9 @@ fact. (2) Staging but both write paths were denied by the session auto-mode classifier (live-Supabase confirmation rule from PR #2196), so Phase 4's staging parity is open again by one version until an operator applies it. -**Next dispatches:** merge the 6.2 PR (its CI `Migration replay` and Supabase Preview are the last chain proofs), then Phase 5 close-out (after-EXPLAIN set, `#231` re-test on healthy latency, `check:production-readiness`), then one serialized `issues:reconcile`. Every future migration still needs its own approved window and its own `db push`. **D4 is REOPENED (2026-08-20): treat merging a migration PR as a production deployment** until the Supabase "Deploy to production" toggle is re-verified in the dashboard — so never merge a migration PR outside its approved window, and apply and verify the migration inside that window before it lands. +**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22)** — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window and its own `db push`. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. -**Where the programme stands after Phase 4 (2026-08-19).****Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both +**Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both tiers and staging is at full parity, so the remaining live-drift findings are exactly one category: the fifteen `migration_history` no-statements rows, which are `#Q5JHBJ` and need fail-fast guard migrations, never bare allowlisting. Phase 4 earned no allowlist entry for any of them — measured, not diff --git a/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json new file mode 100644 index 0000000000..77f30a23eb --- /dev/null +++ b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "0a0ab127-0cc1-4b14-b4ce-fd839a98386c", + "createdOn": "2026-08-21", + "action": "update", + "payload": { + "id": "#231", + "detail": "PHASE 5.2 CONFIRMED SATISFIED with fresh data 2026-08-22, not reopened. This row already recorded that remediation-plan Phase 5.2 is satisfied by S1's 2026-08-17 healthy-latency probes; the Phase 5 close-out re-measured production end to end and confirms it. Retrieval now costs 955 ms on the text fast path and 6,720 ms on hybrid (from 31,610 ms and 21,757 ms at the incident), against answerRouteBudgetMs.fast of 25,000 ms -- so retrieval consumes 4-27% of the fast budget and is no longer capable of binding it. The 2026-08-14 verdict that pre-generation latency WAS the binding cause stands for that window and is now closed out. Residual R4 (chronic ~30 s strong-route provider_timeout on metformin-renal-dosing and valproate-pregnancy, with a safe source-backed extractive fallback) is generation-side and unchanged; no separate R4 row was created, per this row's own instruction. INCONSISTENCY TO FIX AT RECONCILE: the recommended-execution-queue row for #231 still presents it as A1 / 'immediate approved live investigation' with the old framing ('live answers degrade to source-only when answerRouteBudgetMs.fast binds while retrieval is healthy'), which contradicts the P1 -> P2 re-grade recorded in this detail row. The queue entry should be re-graded to match P2 and re-scoped to the R4 generation-side residual, so the queue stops advertising a retrieval investigation that the measurements have closed.", + "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(a) and 5.2; production probes 2026-08-22", + "baseRowFingerprint": "a2d8deb8060b77d1ed3718818a01b31372c4b30281165dc049102fa616b396e9" + } +} diff --git a/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json b/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json new file mode 100644 index 0000000000..aede97f70c --- /dev/null +++ b/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "2040d1fb-6d26-4977-902f-d4a3c2c404c2", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P3", + "type": "task", + "summary": "explain_retrieval_rpc cannot profile the _v2 retrieval RPC family, so the two Phase 1.2 outliers have no EXPLAIN baseline", + "detail": "Phase 5 close-out was asked to capture an EXPLAIN baseline for match_document_chunks_text_v2 and match_document_index_units_hybrid_v2 (the two section 1.2 outliers) and could not. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else. Verified three ways: its only defining migration 20260626020000_phase7_retrieval_rpc_performance.sql, the supabase/schema.sql live mirror, and a direct call against production returning the 22023 error for both names. Extending the p_rpc branch list is production DDL, and with D4 confirmed ON a migration merged to main deploys automatically, so it needs its own approved window; it was deliberately NOT written during a read-only phase. SECOND, SMALLER GAP recorded at the same time: the function EXPLAINs `select * from public.(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore not answerable through this instrument at all; pg_stat_user_indexes is the durable substitute and is what Phase 5 used. NEXT: author one migration extending the branch list to the _v2 family and ship it in an approved window. STOP: not urgent -- the v1 siblings profiled in Phase 5 cover the same owning tables.", + "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/migrations/20260626020000_phase7_retrieval_rpc_performance.sql", + "issueUlid": "01M0JTGY6SJYJEM6MAR4FGVWA8" + } +} diff --git a/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json new file mode 100644 index 0000000000..f21d89f5a2 --- /dev/null +++ b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "3c71ce2a-b752-4f1b-a8ef-4870f8027bf7", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Restored trigram indexes have never been scanned, so the 2026-08-14 incident fix is probably mis-attributed to them rather than to ANALYZE", + "detail": "Measured on production 2026-08-22, read-only. pg_stat_database.stats_reset is NULL, so index counters are cumulative and have never been reset; 163 of 210 public indexes show scans, max(idx_scan) 262,335,345, total 344,747,815 -- the counters are demonstrably live. Against that, ALL 22 indexes restored or repaired by the incident response report idx_scan = 0: the 20 Phase 4 rebuilds of 2026-08-19, and both trigram indexes restored 2026-08-14 -- document_chunks_content_trgm_idx (68 MB, OID 1491258) and documents_title_trgm_idx (648 kB, OID 1491257). The OID matters: 1491258 is the same value Phase 4 Step 4 recorded as UNCHANGED across the 2026-08-19 push, so the index was never dropped and recreated and its counter has accumulated for the full seven days since restoration. What actually serves retrieval on those tables is document_chunks_search_idx (GIN tsvector, 37,717 scans), document_chunks_embedding_hnsw_idx (HNSW, 4,906) and documents_title_search_idx (37,299). Note that a DIFFERENT trigram index on a sibling table, document_table_facts_title_row_param_trgm_idx, is used 2,372 times, so this is not a blanket property of the planner or of pg_trgm. CONSEQUENCE: forensics Phase 5 attributes the 31,610 -> 1,535 ms recovery to 'restore + ANALYZE', treating the trigram restore as the fix. Zero scans mean those indexes cannot be carrying that recovery, and ANALYZE was an unseparated co-intervention in the same operation -- 20260626020000 itself ends by running analyze over documents, document_chunks, document_table_facts, document_embedding_fields and document_index_units. Stale planner statistics is the better-supported explanation for both the 31-second regression and its recovery. THIS DOES NOT RETRACT THE REPAIR: the two indexes are repo-defined, the migration chain commands them, and drift is correctly closed by their presence. It retracts the ATTRIBUTION, and it changes triage order -- the next 30-second retrieval incident should check planner statistics BEFORE hunting missing indexes. Secondary and non-urgent: roughly 69 MB of currently unscanned index. NEXT: (a) in an approved window, separate the two interventions by measuring retrieval latency before and after a fresh ANALYZE, and (b) only then decide whether the two trigram indexes earn their storage. STOP: do NOT drop them -- they are repo-defined and dropping them reopens live drift.", + "source": "production read-only window 2026-08-22; docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(c)", + "issueUlid": "01M0JTGYEJ47M1XD0V0EGS5BDD" + } +} diff --git a/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json new file mode 100644 index 0000000000..5371e39f7f --- /dev/null +++ b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "98e6ae7f-4190-49f3-a64d-fbf632ef6643", + "createdOn": "2026-08-21", + "action": "done", + "payload": { + "id": "#316", + "outcome": "CLOSED 2026-08-22 -- Phase 5, this row's last remaining follow-on, is complete and the alarm is OBSERVED clear rather than merely believed clear. (1) MEASUREMENT, like-for-like with the section 1.3 probe style: text fast path 31,610 -> 955 ms (33x better than the incident, 1.6x better than the 2026-08-14 post-restore reading); hybrid 21,757 -> 6,720 ms. Nothing was worse than the 2026-08-17 healthy baseline, so nothing was escalated. A new EXPLAIN forward baseline was captured on the healthy system for all four RPCs that explain_retrieval_rpc supports, zero errors across twelve samples; the new slowest path is match_document_table_facts_text (1.72 s unprimed, 1,070 unprimed shared_read). Section 1.3's two pending items are now closed: missing-index sizing is moot (0 missing) and the rag_retrieval_logs miss-scan baseline is recorded (47 seq scans, 6,620 tuples read, 640 live rows, 1.4 MB -- cheap at this scale; the row to watch as it grows is rag_aliases at 5,829 seq scans over 68 rows). (2) 5.2 confirmed ALREADY SATISFIED via #231 and not reopened: retrieval now consumes 4-27% of the 25 s fast-route budget, so it cannot bind it. (3) 5.3 needed NO eval canary -- Phase 3 deployed zero behaviour change -- so it reduced to check:production-readiness: READY, 9 PASS, 0 blocking failures. (4) PRODUCTION RE-VERIFIED UNCHANGED, 8/8 against the 2026-08-21 baseline: 210 public indexes, 0 invalid, 211 history rows, 20 no-statement rows, migration_history_versions() probe ok with 211 versions, search_schema_health() ok true. (5) STAGING PARITY CLOSED: 20260820120000 applied to ikoiolksxqxfxgiyqpnu by real db push (never migration repair); staging now reads 211 rows with the function present and stmt_count 3, matching production exactly. (6) THE ALARM IS OBSERVED CLEAR: live-drift run 32514326022, dispatched on main, is GREEN -- 'No unexpected schema drift between live and supabase/schema.sql', 'Remote migration versions: 211 (read via rpc)', 'Remote-only (Preview blockers): 0', every step success. That is the FIRST green live-drift since 2026-07-19T19:37:29Z (of the last 60 runs only four succeeded), ending a 33-day red streak, and pinned issue #1963 auto-closed at 2026-08-21T18:38:42Z. (7) D4 DOCUMENTATION CORRECTED: this row's 2026-08-19 'auto-deploy is OFF' sentence was already superseded by the 2026-08-21 correction; the coordination board and the forensics dossier are now aligned to the dashboard fact (Deploy to production ENABLED, production branch main, automatic branching ON limit 3), with the superseded safe-either-way reasoning retained and marked rather than deleted. THREE FINDINGS QUEUED SEPARATELY rather than absorbed here: the uncapped Branching Compute cost; the explain_retrieval_rpc gap on the _v2 RPC family (needs production DDL, deliberately not written in a read-only phase); and the discovery that all 22 restored indexes report idx_scan = 0 against never-reset statistics, which makes ANALYZE rather than the trigram restore the better-supported explanation for the incident recovery. RESIDUAL WORK LIVES ON ITS OWN ROWS, not here: PITR is still OFF on production, and the fifteen no-statements history rows remain #Q5JHBJ.", + "baseRowFingerprint": "17b4358a583c8bf5d2b7cc77a6ccf0b8eaf714b92a4e7301f3756a9e831b46ad" + } +} diff --git a/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json b/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json new file mode 100644 index 0000000000..49eaceac97 --- /dev/null +++ b/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "d873ec0d-e25d-41fe-b24c-d2c3aa9375ee", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "rec", + "summary": "Supabase preview-branch compute is an uncapped cost sitting outside the organisation Spend Cap", + "detail": "Dashboard read 2026-08-21 (the same read that settled D4) shows Automatic branching ON with limit 3 and \"Supabase changes only\" enabled, and the same screen warns that Branching Compute is NOT covered by the organisation's Spend Cap. Preview databases are therefore an uncapped cost that scales with the number of open PRs touching supabase/**, on a project that already runs auto-deploy to production. CONTEXT FOR ANY DECISION TO REDUCE OR DISABLE THEM: CI's Migration replay job (db-reset-verify, which runs `supabase migration up --local`) independently replays the entire migration chain on every database-touching PR, so preview branches are a SECOND net rather than the only one -- turning them down would not leave the chain unproven. Preview branches did earn their keep once (the 20260819100200 guard failure on PR #2151 was caught by a preview branch building from the chain alone), so this is a cost/benefit decision, not a cleanup. NEXT: decide whether to keep branching at limit 3, lower the limit, or disable it, and record the decision with its date. STOP: do not change Supabase project settings without explicit owner approval; this row is a decision request, not a licence to reconfigure the project.", + "source": "Supabase dashboard read 2026-08-21; docs/audit/live-drift-forensics-2026-08.md D4 section; AGENTS.md Supabase project safety", + "issueUlid": "01M0JTGY049X40BT8EZTS24YYX" + } +} From 67ab9e23cf94d3973d1eff7b767ba8a6281580a6 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Sat, 22 Aug 2026 03:19:28 +0800 Subject: [PATCH 2/5] docs(ledger): record the Phase 5 close-out review for claude/phase-5-closeout Co-Authored-By: Claude Opus 5 --- ...febc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md diff --git a/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md b/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md new file mode 100644 index 0000000000..dcddda298f --- /dev/null +++ b/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md @@ -0,0 +1 @@ +| 2026-08-21 | claude/phase-5-closeout | c3ca68fa7f13e2d7ae6c4c96ccfcafb67a59135a | Phase 5 close-out: measurement baselines, staging parity, D4 documentation correction (#316) | Self-review; no blocking findings. Docs-only diff plus five ledger inbox requests. Read-only production window, one staging-only db push, one live-drift dispatch (green). Two limits and one attribution finding recorded rather than absorbed; no RAG surface touched. | verify:pr-local (11/11, failed: none); check:production-readiness (READY, 9 PASS, 0 blocking); format (whole tree, committed); live-drift run 32514326022 green | From 9646a7dbd1e14a61147411a83ce10d35f31d50ba Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Sat, 22 Aug 2026 03:36:22 +0800 Subject: [PATCH 3/5] docs(db): correct three Codex review findings on the Phase 5 close-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - P1: drop the stale "every future migration needs its own `db push`" sentence. With D4 confirmed ON the merge IS the deployment, so a pre-merge push puts live ahead of the reviewed repo state and a post-merge push is a redundant production mutation. The gate is the post-merge `live-drift` run, matching AGENTS.md. - P2: stop presenting cumulative `pg_stat_user_indexes.idx_scan` as plan-flip evidence. It aggregates every workload on the table and no before/after delta was captured, so it can show an index is never chosen by anything but cannot show that this query changed plan. Plan §5.1's plan-flip deliverable is now marked explicitly OPEN rather than discharged. - P2: correct a false coverage claim. `match_document_index_units_hybrid_v2` delegates to `match_document_index_units_hybrid_scoped` over `document_index_units`, which none of the four supported RPCs touches, so that path has no EXPLAIN baseline at all — not a partial stand-in. Only `document_chunks` has a usable v1 sibling. - Phase 5 status now reads "COMPLETE, two deliverables explicitly OPEN". - Ledger: cancelled the inbox request carrying the same false claim and queued a corrected one, re-graded P3 -> P2. Co-Authored-By: Claude Opus 5 --- docs/audit/live-drift-forensics-2026-08.md | 23 +++++++++++++++---- docs/database-remediation-coordination.md | 4 ++-- .../05bd88ac-f24f-464a-8853-5d7ff55b375b.json | 14 +++++++++++ .../1c89922d-3d01-4811-a255-ed78e2ed11c3.json | 10 ++++++++ 4 files changed, 45 insertions(+), 6 deletions(-) create mode 100644 docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json create mode 100644 docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 167c20c6a2..48ef0429a7 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -1750,8 +1750,14 @@ as the outlier to watch, replacing the trigram-absence signature that dominated `select * from public.(…)`, so the plan is a single `Function Scan`: a PL/pgSQL body's inner plan is not exposed. Every sample reports `plan_node_types: ["Function Scan"]` and `index_names: []`. Plan §5.1's "record plan flips (seq scan → index scan)" is therefore **not - answerable through this tool**; the index-usage read below answers the same question directly and - more durably. + answerable through this tool**. The index-usage read in 5.1(c) is a _different and weaker_ signal, + not a substitute: `pg_stat_user_indexes.idx_scan` is cumulative across every workload that touches + the table, and no before/after counter delta was captured around these samples, so it can show + that an index is never chosen by **anything**, but it cannot show whether **this** profiled query + moved from a sequential scan to an index scan. **Plan §5.1's plan-flip deliverable is therefore + left explicitly OPEN, not discharged.** Closing it needs query-specific inner-plan evidence — + which requires the RPC extension queued below, or an `auto_explain`-style capture — and that is + production-side work this read-only phase could not do. 2. **It cannot reach the two v2 RPCs named in the Phase 5 brief.** `explain_retrieval_rpc` accepts exactly four names (`match_documents_for_query`, `match_document_chunks_text`, `match_document_lookup_chunks_text`, `match_document_table_facts_text`) — verified in its only @@ -1765,8 +1771,17 @@ as the outlier to watch, replacing the trigram-absence signature that dominated ``` Extending the function to cover the `_v2` family is production DDL and was **not** authorised for - this phase, so it was not written. Queued as its own ledger item; the v1 siblings profiled above - cover the same two owning tables in the meantime. + this phase, so it was not written. Queued as its own ledger item. **Substitute coverage is partial, and + unevenly so — the earlier claim that the v1 siblings cover both owning tables was wrong.** + `match_document_chunks_text`, profiled above, is the v1 sibling of `match_document_chunks_text_v2` + and shares its owning table `document_chunks`, so that target has a usable stand-in. The other one + does not: `match_document_index_units_hybrid_v2` delegates to + `match_document_index_units_hybrid_scoped` over `document_index_units` + (`supabase/schema.sql:8033-8054`), and the four supported names reach only `documents`, + `document_chunks`, lookup chunks and `document_table_facts`. **The `document_index_units` + retrieval path therefore has no EXPLAIN baseline at all**, and that is an open deliverable rather + than a covered one. Its index-level usage is still recorded in 5.1(c), which is the weaker signal + described above, not a plan. #### 5.1(c) Index usage — the finding this phase actually turned up diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 46e5055184..6f9edd89ce 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -90,7 +90,7 @@ trusted as-is._ | 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | | 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | | 4 — Index restoration | ☑ COMPLETE 2026-08-19 — production **and** staging | **#2151** (branch `claude/db-remediation-phase4-indexes-a1661a`, head `5d3dca4dd`); forensics §Phase 4 completion; live-drift run `32171070287` | Owner-authorised off-peak window, D4 **OFF** so nothing auto-deployed. All **20** `missing_live` indexes rebuilt with `CREATE INDEX CONCURRENTLY` from canonical definitions — Batch A 14/14, Batch B 6/6, every one `indisvalid AND indisready` with normalised `pg_get_indexdef` matching canonical; zero invalid builds, zero retries, zero skips, zero lock waits; no transactional build attempted; `#102` held out. Both `unexpected_live` orphans **DROPPED** concurrently (not codified) — each is a strict leading-column subset of a present canonical index and the chain already commands both drops (`20260620000000`, `20260703030000`/`20260708000000`); neither is in `schema.sql`, so the drop moves live _into_ mirror agreement. Live now 210 indexes vs manifest 210, zero invalid anywhere. Codified by real `db push` (never `migration repair`; all four history rows carry executed statements): `20260819100000`/`100100` guard Batch A/B, **`20260819100200` discharges the 4.4 trigram-guard debt** that `20260804110240` never covered, `20260819100300` takes `required_indexes` 22→30 adopting **all 8** Phase 6.3 monitor-candidates (`search-health-unmonitored-indexes.json` 44→36, no `monitor-candidate` left; production `ok: true`). **Staging also closed:** `document_chunks_content_trgm_idx` rebuilt into the canonical `coalesce(content,'')` form (§3.3(d)), the four migrations applied by the Phase 2 method with all md5s matching, `check:drift` vs staging **green, 0 unexpected (was 19)**, corpus untouched. Live-drift **37 → 16**: `missing_live` **20 → 0**, `unexpected_live` **2 → 0**; the lone function mismatch is merge-pending and proven (live `def_hash 85df52de…` == this branch's regenerated manifest). **Two escalations, not absorbed:** PITR is OFF on production (new P2 queued) and the `migration_history` block did **not** drop — no allowlist entry was earned (empty intersection), so the 15 stay `#Q5JHBJ`'s work. | -| 5 — Measure | ☑ COMPLETE 2026-08-22 | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` against never-reset statistics, including both trigram indexes credited with the incident fix — the recovery is better explained by the co-administered `ANALYZE` than by the indexes. Queued as its own ledger item. **Two instrument gaps recorded:** `explain_retrieval_rpc` cannot show plan flips (outer `Function Scan` only) and cannot reach the `_v2` RPC family (`22023`), which would need production DDL. | +| 5 — Measure | ☑ COMPLETE 2026-08-22 — two deliverables explicitly OPEN, queued | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` against never-reset statistics, including both trigram indexes credited with the incident fix — the recovery is better explained by the co-administered `ANALYZE` than by the indexes. Queued as its own ledger item. **Two instrument gaps recorded, and they leave two §5.1 deliverables genuinely OPEN rather than discharged:** `explain_retrieval_rpc` exposes only an outer `Function Scan`, so no query-specific plan-flip evidence exists (cumulative `idx_scan` counters are a weaker, whole-workload signal and are not a substitute); and it raises `22023` for the `_v2` family, so `document_index_units` — which no supported RPC touches — has no EXPLAIN baseline at all. Both need production DDL and are queued. | | 6 — Future-proofing | ☑ COMPLETE 2026-08-19 (6.1 live; 6.2 guards live on both tiers; 6.3 repo-side) | **#2058** merged `9c660af1f` (6.1 / 6.2 contract / 6.3); follow-up #2090 merged `79e5d2a68`; **Phase 6.2 guards: branch `claude/migration-history-drift-allowlist-37444c`** (forensics §"6.2 completion"); live-drift run `32251326536` | 6.1 probe LIVE on production since 2026-08-18. **6.2 done 2026-08-19 (`#Q5JHBJ`)**: all fifteen no-statements versions classified — every one `validation`, none `superseded`/`no_ddl` (no single later executed migration re-creates every object; `COMMENT ON` is a catalog write) — and guarded by six fail-fast migrations `20260819110000`…`110500` (dropped objects / comments + cron / document FKs / 46 operational index shapes / `index_generation_id` promotion / 15 function `def_hash`es read from `schema_drift_snapshot()`). Allowlist 5 superseded + 15 validation = 20. Proof: whole chain replayed into the scratch image `Applied 210/210`, `CHAIN == MANIFEST` (every hand repair in §2.3/§3.3/Phase 4 reproduced by the chain; no reconcile needed), seven mutants raise; all six dry-ran green on production and a mutant failed there; real `db push` (pending = exactly six; rows `no_statements false`; 210 rows; corpus untouched); staging by the Phase 2 method, six md5-matched, drift green. **Live-drift on the branch: `No unexpected schema drift`, all 20 history rows allowed — `#316`'s finding set is EMPTY for the first time since 2026-07-26.** Job still red on the Phase 0 `Align migration history` step (PGRST106: `supabase_migrations` not exposed to PostgREST; it ran for the first time ever) — queued as its own P2, owner decision (expose schema / use access token / RPC). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) redundant, owner may delete. | | 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | @@ -176,7 +176,7 @@ fact. (2) Staging but both write paths were denied by the session auto-mode classifier (live-Supabase confirmation rule from PR #2196), so Phase 4's staging parity is open again by one version until an operator applies it. -**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22)** — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window and its own `db push`. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. +**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22), with two §5.1 deliverables explicitly left OPEN and queued** (query-specific plan-flip evidence, and any EXPLAIN baseline for the `document_index_units` path) — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window, but **not** a separate manual `db push`: with D4 ON the merge itself is the deployment, and the post-merge `live-drift` run — BOTH `check:drift` and `check:migration-history` green — is the schema-application gate. A pre-merge push would put the live schema ahead of the reviewed repository state; a post-merge push would be a redundant production mutation. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. **Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both tiers and staging is at full parity, so the remaining live-drift findings are exactly one category: diff --git a/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json b/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json new file mode 100644 index 0000000000..0efa3eb5d6 --- /dev/null +++ b/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "05bd88ac-f24f-464a-8853-5d7ff55b375b", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "task", + "summary": "The document_index_units retrieval path has no EXPLAIN baseline, and Phase 5 has no query-specific plan-flip evidence", + "detail": "Two Phase 5.1 deliverables are explicitly OPEN, not discharged. Re-graded P3 -> P2 versus the withdrawn request 2040d1fb, because that request understated the gap by claiming substitute coverage that does not exist. (A) NO EXPLAIN BASELINE FOR THE INDEX-UNITS PATH. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else, proven against production for both match_document_chunks_text_v2 and match_document_index_units_hybrid_v2. For the first of those the v1 sibling match_document_chunks_text shares the owning table document_chunks and is a usable stand-in. For the second there is none: match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units (supabase/schema.sql:8033-8054), and no supported RPC touches that table. document_index_units is one of the two section 1.2 outliers, so the outlier that most needed a baseline is the one that has none. (B) NO QUERY-SPECIFIC PLAN-FLIP EVIDENCE. explain_retrieval_rpc EXPLAINs `select * from public.(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore unanswerable through this instrument. The pg_stat_user_indexes read captured in Phase 5.1(c) is a WEAKER and DIFFERENT signal, not a substitute: idx_scan is cumulative across every workload touching the table and no before/after delta was captured around the samples, so it can prove an index is never chosen by anything but cannot prove that a given query changed plan. NEXT: one migration extending the explain_retrieval_rpc p_rpc branch list to the _v2 family (at minimum match_document_index_units_hybrid_v2 and match_document_chunks_text_v2), shipped in an approved window -- with D4 ON, merging it to main deploys it, so it needs the window and a green post-merge live-drift run. Then re-run npm run profile:retrieval --analyze to capture the missing baseline. For (B), consider whether an auto_explain-style capture is a better fit than widening the RPC. STOP: do not record the cumulative index-usage read as plan-flip evidence; that conflation is exactly what this row exists to prevent.", + "source": "Codex review of PR #2250 (P2, comment 3833062803 and 3833062807); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/schema.sql:8033-8054", + "issueUlid": "01M0JWT29F8VAY97CG3PM5X2QW" + } +} diff --git a/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json b/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json new file mode 100644 index 0000000000..63434e9f0e --- /dev/null +++ b/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "1c89922d-3d01-4811-a255-ed78e2ed11c3", + "createdOn": "2026-08-21", + "action": "cancel", + "payload": { + "requestId": "2040d1fb-6d26-4977-902f-d4a3c2c404c2", + "reason": "Withdrawn and replaced: the request asserted that the v1 siblings profiled in Phase 5 cover the same owning tables as the two unreachable _v2 RPCs. That is false for one of the two. match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units, and none of the four RPCs explain_retrieval_rpc supports touches that table, so the index-units retrieval path has no EXPLAIN baseline at all rather than a partial stand-in. Raised by Codex review of PR #2250 and confirmed against supabase/schema.sql:8033-8054. Superseded by a corrected add request queued in the same commit." + } +} From 560e0ae81d152fb2e76c87cffbe23720747e4b88 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 19:42:24 +0000 Subject: [PATCH 4/5] =?UTF-8?q?docs(db):=20address=20PR=20#2250=20review?= =?UTF-8?q?=20=E2=80=94=20D4=20db-push=20contradiction,=20measurement=20li?= =?UTF-8?q?mits,=20timezone?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex + CodeRabbit review findings on the Phase 5 close-out documentation. - Remove the mandatory separate production `db push` from the coordination board's "Next dispatches". D4 is settled: merging a migration PR to `main` IS the production deployment, so a pre-merge push puts the live schema ahead of reviewed repo state and a post-merge push is a redundant production mutation. Approved-window and green post-merge `live-drift` requirements are kept; `db push` is reserved for authorised staging or recovery work. - Stop treating cumulative `idx_scan` counters as query-plan evidence. Plan §5.1's seq-scan -> index-scan deliverable is now recorded as explicitly OPEN, with the reason the index-usage read cannot substitute for an inner plan and what would actually close it. - Correct the `_v2` RPC coverage claim. `match_document_index_units_hybrid_v2` delegates to `match_document_index_units_hybrid_scoped`, which reads `document_index_units`; none of the four profiled RPCs touches that table, so it has no EXPLAIN baseline. Only the `document_chunks` half is covered. - Qualify the "counters have never been reset" conclusion. `pg_stat_database.stats_reset IS NULL` rules out a database-wide reset only; `pg_stat_reset_single_table_counters(oid)` resets a single relation without touching it. The zero-scan observation and the ANALYZE attribution stand as well-supported rather than proven, and the follow-up records how to settle it. - Make the Perth/UTC boundary explicit rather than changing dates. The window ran across local midnight: 2026-08-22 Perth is 2026-08-21 UTC, which is why records dated 2026-08-21 describe 2026-08-22 measurements. - Add `text` language identifiers to four fenced blocks (markdownlint MD040). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01ApB8NBygQn9cosxCQ8omk9 --- docs/audit/live-drift-forensics-2026-08.md | 69 ++++++++++++++----- docs/database-remediation-coordination.md | 4 +- .../0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json | 4 +- .../3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json | 4 +- .../98e6ae7f-4190-49f3-a64d-fbf632ef6643.json | 2 +- 5 files changed, 58 insertions(+), 25 deletions(-) diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 167c20c6a2..efa6b9cc99 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -1686,6 +1686,12 @@ by the remaining remediation phases, not a route-budget change (`#231`'s stop co ### Phase 5 close-out — 2026-08-22 Perth (2026-08-21 UTC) +_**Dates in this section are Perth local time (UTC+8), with the UTC date given alongside where the +two differ.** The window ran across the local-midnight boundary: the review and every commit are +dated 2026-08-21 UTC, which is 2026-08-22 in Perth. Where a record elsewhere in the repository says +"2026-08-22" bare for this phase, it means 2026-08-22 Perth = 2026-08-21 UTC — no measurement was +taken after the review date._ + _Owner-authorised **read-only** production window (`sjrfecxgysukkwxsowpy`) plus a staging-only apply. A dedicated worktree carried the link; the main checkout stayed on its staging link and every call re-verified `supabase/.temp/project-ref` immediately beforehand. No production DDL, no @@ -1713,11 +1719,11 @@ queries, live production data). `supabase_rpc_latency_ms` is the server-measured component and is the comparable figure; `total` additionally carries local dev-server render overhead in this run, so it is directionally useful rather than strictly comparable. -| Measurement | Before (2026-08-14) | After restore + ANALYZE (2026-08-14) | **Now (2026-08-22)** | -| ------------------------------------------ | ------------------- | ------------------------------------ | -------------------- | -| Semantic, text fast path — total / rpc ms | 37.7 s / 31,610 | 4.8 s / 1,535 | **8.4 s / 955** | -| Semantic, hybrid strategy — total / rpc ms | 29.9 s / 21,757 | 17.2 s / 8,519 | **10.9 s / 6,720** | -| Warm repeat (`search_cache`) — total / rpc | not measured | not measured | **0.41 s / 0** | +| Measurement | Before (2026-08-14) | After restore + ANALYZE (2026-08-14) | **Now (2026-08-22 Perth = 2026-08-21 UTC)** | +| ------------------------------------------ | ------------------- | ------------------------------------ | ------------------------------------------- | +| Semantic, text fast path — total / rpc ms | 37.7 s / 31,610 | 4.8 s / 1,535 | **8.4 s / 955** | +| Semantic, hybrid strategy — total / rpc ms | 29.9 s / 21,757 | 17.2 s / 8,519 | **10.9 s / 6,720** | +| Warm repeat (`search_cache`) — total / rpc | not measured | not measured | **0.41 s / 0** | Retrieval latency has held its recovery and improved further on both strategies: the fast path is **33× better than the incident** and 1.6× better than the 2026-08-14 post-restore reading; hybrid is @@ -1750,28 +1756,52 @@ as the outlier to watch, replacing the trigram-absence signature that dominated `select * from public.(…)`, so the plan is a single `Function Scan`: a PL/pgSQL body's inner plan is not exposed. Every sample reports `plan_node_types: ["Function Scan"]` and `index_names: []`. Plan §5.1's "record plan flips (seq scan → index scan)" is therefore **not - answerable through this tool**; the index-usage read below answers the same question directly and - more durably. + answerable through this tool, and that deliverable stays OPEN.** The index-usage read below is a + useful but _different_ measurement and does not substitute for it: `pg_stat_user_indexes.idx_scan` + is a cumulative, database-wide counter that aggregates every workload touching the index, and no + before/after delta was captured around these samples. It can therefore show that an index is + never used by anything, but it cannot show that _this_ profiled query moved from a sequential scan + to an index scan. **Do not read the counters below as plan evidence when comparing a future + incident against this baseline.** Closing §5.1 properly needs a query-specific inner plan — either + `explain_retrieval_rpc` extended to run `EXPLAIN` on the body's own statements, or an + `auto_explain`-style capture with nested statements enabled — both of which are production DDL or + configuration changes that this read-only phase was not authorised to make. 2. **It cannot reach the two v2 RPCs named in the Phase 5 brief.** `explain_retrieval_rpc` accepts exactly four names (`match_documents_for_query`, `match_document_chunks_text`, `match_document_lookup_chunks_text`, `match_document_table_facts_text`) — verified in its only defining migration `20260626020000_phase7_retrieval_rpc_performance.sql`, in `supabase/schema.sql`, and on production itself: - ``` + ```text ERROR: 22023: Unsupported retrieval RPC: match_document_chunks_text_v2 CONTEXT: PL/pgSQL function explain_retrieval_rpc(text,text,integer,uuid,uuid[],boolean) line 25 at RAISE ERROR: 22023: Unsupported retrieval RPC: match_document_index_units_hybrid_v2 ``` Extending the function to cover the `_v2` family is production DDL and was **not** authorised for - this phase, so it was not written. Queued as its own ledger item; the v1 siblings profiled above - cover the same two owning tables in the meantime. + this phase, so it was not written. Queued as its own ledger item. **Coverage of the two unreached + targets is uneven, and only half of it is covered:** + + - `match_document_chunks_text_v2` — its owning table `document_chunks` **is** exercised by the + profiled v1 siblings `match_document_chunks_text` and `match_document_lookup_chunks_text`. + - `match_document_index_units_hybrid_v2` — **no profiled RPC touches its owning table.** The v2 + function delegates to `match_document_index_units_hybrid_scoped`, which reads + `public.document_index_units` (`supabase/schema.sql`, the `match_document_index_units_hybrid_v2` + and `..._scoped` definitions). The four profiled RPCs read `documents`, `document_chunks`, + `document_chunks` again, and `document_table_facts` — none of them is an index-units sibling. + **`document_index_units` therefore has no EXPLAIN baseline at all after this phase**, and that + gap is carried forward with the ledger item above rather than treated as covered. #### 5.1(c) Index usage — the finding this phase actually turned up -`pg_stat_database.stats_reset` is **NULL** on production: index statistics are cumulative and have -never been reset, so a zero here means genuinely never scanned, not recently zeroed. Across `public`, +`pg_stat_database.stats_reset` is **NULL** on production, so **no database-wide reset** +(`pg_stat_reset()`) has been recorded. That is the limit of what this read proves: a per-relation +`pg_stat_reset_single_table_counters(oid)` call resets one table's or one index's counters without +touching `pg_stat_database.stats_reset`, and this window captured no evidence either way for the +individual indexes below. A zero is therefore _very likely_ to mean genuinely never scanned rather +than recently zeroed — strongly so given the OID continuity established below — but it is not proven. +Settling it needs a per-index reset time, which `pg_stat_all_indexes.stats_reset` carries only on +PostgreSQL 18 and later; the running server version was not read in this window. Across `public`, 163 of 210 indexes show scans, `max(idx_scan)` is 262,335,345 and the total is 344,747,815 — the counters are clearly live. @@ -1803,7 +1833,10 @@ The last row matters: a _different_ trigram index on a sibling table is used tho **Consequence for the incident narrative — stated as a correction, not a certainty.** The 2026-08-14 Phase 5 table above attributes the 31,610 ms → 1,535 ms recovery to "restore + ANALYZE", treating the two trigram indexes as the fix. Seven days of cumulative statistics show the restored trigram indexes -have served **zero** scans, so they cannot be carrying that recovery. `ANALYZE` was an unseparated +have served **zero** scans. Subject to the reset caveat recorded above — a per-relation +`pg_stat_reset_single_table_counters()` call would be invisible to the database-wide `stats_reset` +read, so "zero scans since restoration" is strongly supported rather than proven — they cannot be +carrying that recovery. `ANALYZE` was an unseparated co-intervention in the same operation — and `20260626020000` ends by running `analyze` over exactly `documents`, `document_chunks`, `document_table_facts`, `document_embedding_fields` and `document_index_units` — which makes stale planner statistics the better-supported explanation for @@ -1855,7 +1888,7 @@ was reframed to repo-side codification and deployed **zero** behaviour change: ` function mismatches. §5.3 therefore reduces to `check:production-readiness`, run once at the end of this phase. No paid eval was run and none was requested. -``` +```text $ npm run check:production-readiness [Production Readiness] Project: Clinical KB Database (sjrfecxgysukkwxsowpy) @@ -1882,7 +1915,7 @@ Staging (`ikoiolksxqxfxgiyqpnu`) was one version behind after the 2026-08-20 win paths were denied by that session's auto-mode classifier. Applied here, by real `db push` (never `migration repair`); the pending set was confirmed first as exactly one version with zero remote-only: -``` +```text $ supabase migration list --linked --project-ref ikoiolksxqxfxgiyqpnu … {"local":"20260820120000","remote":""} $ supabase db push --linked --project-ref ikoiolksxqxfxgiyqpnu --skip-vault --yes @@ -2362,7 +2395,7 @@ One extra command, wrong under neither hypothesis. What would replace this rule dashboard read of the toggle, or a deployment-settings API that reports it. Raised as a P1 by the Codex review of PR #2205, and correctly: the earlier wording here presented the inference as a direct read. -#### Staging is one version behind — blocked, not skipped (CLOSED 2026-08-22, see Phase 5 close-out) +#### Staging is one version behind — blocked, not skipped (CLOSED 2026-08-22 Perth = 2026-08-21 UTC, see Phase 5 close-out) Staging (`ikoiolksxqxfxgiyqpnu`, verified before every call) reads **210** history rows, latest `20260819110500`, `to_regprocedure('public.migration_history_versions()') is null`. Its pending set is @@ -2375,7 +2408,7 @@ Read-only calls were unaffected, which is why every verification above exists. T authorisation gap, not a technical one: **staging parity, closed by Phase 4, is open again by one version until an operator applies it.** -**CLOSED 2026-08-22.** An operator session applied `20260820120000` to staging by real `db push`; +**CLOSED 2026-08-22 Perth (2026-08-21 UTC).** An operator session applied `20260820120000` to staging by real `db push`; staging now reads 211 history rows with `migration_history_versions()` present and `stmt_count 3`, matching production exactly. Evidence in the "Phase 5 close-out" section, "Staging catch-up". @@ -2387,7 +2420,7 @@ Dispatched here on `main` with the fix live on production and staging brought to `https://github.com/BigSimmo/Database/actions/runs/32514326022` — **job conclusion: `success`.** -``` +```text live-drift Compare live schema drift No unexpected schema drift between live and supabase/schema.sql. live-drift Align migration history for Supabase Preview Remote migration versions: 211 (read via rpc) live-drift Align migration history for Supabase Preview Remote-only (Preview blockers): 0 diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 46e5055184..affd38529e 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -90,7 +90,7 @@ trusted as-is._ | 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | | 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | | 4 — Index restoration | ☑ COMPLETE 2026-08-19 — production **and** staging | **#2151** (branch `claude/db-remediation-phase4-indexes-a1661a`, head `5d3dca4dd`); forensics §Phase 4 completion; live-drift run `32171070287` | Owner-authorised off-peak window, D4 **OFF** so nothing auto-deployed. All **20** `missing_live` indexes rebuilt with `CREATE INDEX CONCURRENTLY` from canonical definitions — Batch A 14/14, Batch B 6/6, every one `indisvalid AND indisready` with normalised `pg_get_indexdef` matching canonical; zero invalid builds, zero retries, zero skips, zero lock waits; no transactional build attempted; `#102` held out. Both `unexpected_live` orphans **DROPPED** concurrently (not codified) — each is a strict leading-column subset of a present canonical index and the chain already commands both drops (`20260620000000`, `20260703030000`/`20260708000000`); neither is in `schema.sql`, so the drop moves live _into_ mirror agreement. Live now 210 indexes vs manifest 210, zero invalid anywhere. Codified by real `db push` (never `migration repair`; all four history rows carry executed statements): `20260819100000`/`100100` guard Batch A/B, **`20260819100200` discharges the 4.4 trigram-guard debt** that `20260804110240` never covered, `20260819100300` takes `required_indexes` 22→30 adopting **all 8** Phase 6.3 monitor-candidates (`search-health-unmonitored-indexes.json` 44→36, no `monitor-candidate` left; production `ok: true`). **Staging also closed:** `document_chunks_content_trgm_idx` rebuilt into the canonical `coalesce(content,'')` form (§3.3(d)), the four migrations applied by the Phase 2 method with all md5s matching, `check:drift` vs staging **green, 0 unexpected (was 19)**, corpus untouched. Live-drift **37 → 16**: `missing_live` **20 → 0**, `unexpected_live` **2 → 0**; the lone function mismatch is merge-pending and proven (live `def_hash 85df52de…` == this branch's regenerated manifest). **Two escalations, not absorbed:** PITR is OFF on production (new P2 queued) and the `migration_history` block did **not** drop — no allowlist entry was earned (empty intersection), so the 15 stay `#Q5JHBJ`'s work. | -| 5 — Measure | ☑ COMPLETE 2026-08-22 | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` against never-reset statistics, including both trigram indexes credited with the incident fix — the recovery is better explained by the co-administered `ANALYZE` than by the indexes. Queued as its own ledger item. **Two instrument gaps recorded:** `explain_retrieval_rpc` cannot show plan flips (outer `Function Scan` only) and cannot reach the `_v2` RPC family (`22023`), which would need production DDL. | +| 5 — Measure | ☑ COMPLETE 2026-08-22 Perth (2026-08-21 UTC) | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` with no database-wide statistics reset recorded (`pg_stat_database.stats_reset` is NULL), including both trigram indexes credited with the incident fix — on that evidence the recovery is better explained by the co-administered `ANALYZE` than by the indexes. A per-relation reset via `pg_stat_reset_single_table_counters()` is not excluded by that read, so the attribution is well-supported rather than proven. Queued as its own ledger item. **Two instrument gaps recorded:** `explain_retrieval_rpc` cannot show plan flips (outer `Function Scan` only) and cannot reach the `_v2` RPC family (`22023`), which would need production DDL. | | 6 — Future-proofing | ☑ COMPLETE 2026-08-19 (6.1 live; 6.2 guards live on both tiers; 6.3 repo-side) | **#2058** merged `9c660af1f` (6.1 / 6.2 contract / 6.3); follow-up #2090 merged `79e5d2a68`; **Phase 6.2 guards: branch `claude/migration-history-drift-allowlist-37444c`** (forensics §"6.2 completion"); live-drift run `32251326536` | 6.1 probe LIVE on production since 2026-08-18. **6.2 done 2026-08-19 (`#Q5JHBJ`)**: all fifteen no-statements versions classified — every one `validation`, none `superseded`/`no_ddl` (no single later executed migration re-creates every object; `COMMENT ON` is a catalog write) — and guarded by six fail-fast migrations `20260819110000`…`110500` (dropped objects / comments + cron / document FKs / 46 operational index shapes / `index_generation_id` promotion / 15 function `def_hash`es read from `schema_drift_snapshot()`). Allowlist 5 superseded + 15 validation = 20. Proof: whole chain replayed into the scratch image `Applied 210/210`, `CHAIN == MANIFEST` (every hand repair in §2.3/§3.3/Phase 4 reproduced by the chain; no reconcile needed), seven mutants raise; all six dry-ran green on production and a mutant failed there; real `db push` (pending = exactly six; rows `no_statements false`; 210 rows; corpus untouched); staging by the Phase 2 method, six md5-matched, drift green. **Live-drift on the branch: `No unexpected schema drift`, all 20 history rows allowed — `#316`'s finding set is EMPTY for the first time since 2026-07-26.** Job still red on the Phase 0 `Align migration history` step (PGRST106: `supabase_migrations` not exposed to PostgREST; it ran for the first time ever) — queued as its own P2, owner decision (expose schema / use access token / RPC). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) redundant, owner may delete. | | 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | @@ -176,7 +176,7 @@ fact. (2) Staging but both write paths were denied by the session auto-mode classifier (live-Supabase confirmation rule from PR #2196), so Phase 4's staging parity is open again by one version until an operator applies it. -**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22)** — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window and its own `db push`. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. +**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22 Perth = 2026-08-21 UTC)** — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window and a green post-merge `live-drift` run as its application gate; it does **not** need a separate production `db push` — merging to `main` is the deployment, so a pre-merge push would put the live schema ahead of the reviewed repo state and a post-merge push is a redundant production mutation. `db push` is reserved for explicitly authorised staging or recovery work. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. **Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both tiers and staging is at full parity, so the remaining live-drift findings are exactly one category: diff --git a/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json index 77f30a23eb..75412e4b86 100644 --- a/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json +++ b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json @@ -5,8 +5,8 @@ "action": "update", "payload": { "id": "#231", - "detail": "PHASE 5.2 CONFIRMED SATISFIED with fresh data 2026-08-22, not reopened. This row already recorded that remediation-plan Phase 5.2 is satisfied by S1's 2026-08-17 healthy-latency probes; the Phase 5 close-out re-measured production end to end and confirms it. Retrieval now costs 955 ms on the text fast path and 6,720 ms on hybrid (from 31,610 ms and 21,757 ms at the incident), against answerRouteBudgetMs.fast of 25,000 ms -- so retrieval consumes 4-27% of the fast budget and is no longer capable of binding it. The 2026-08-14 verdict that pre-generation latency WAS the binding cause stands for that window and is now closed out. Residual R4 (chronic ~30 s strong-route provider_timeout on metformin-renal-dosing and valproate-pregnancy, with a safe source-backed extractive fallback) is generation-side and unchanged; no separate R4 row was created, per this row's own instruction. INCONSISTENCY TO FIX AT RECONCILE: the recommended-execution-queue row for #231 still presents it as A1 / 'immediate approved live investigation' with the old framing ('live answers degrade to source-only when answerRouteBudgetMs.fast binds while retrieval is healthy'), which contradicts the P1 -> P2 re-grade recorded in this detail row. The queue entry should be re-graded to match P2 and re-scoped to the R4 generation-side residual, so the queue stops advertising a retrieval investigation that the measurements have closed.", - "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(a) and 5.2; production probes 2026-08-22", + "detail": "PHASE 5.2 CONFIRMED SATISFIED with fresh data 2026-08-22 Perth (2026-08-21 UTC), not reopened. This row already recorded that remediation-plan Phase 5.2 is satisfied by S1's 2026-08-17 healthy-latency probes; the Phase 5 close-out re-measured production end to end and confirms it. Retrieval now costs 955 ms on the text fast path and 6,720 ms on hybrid (from 31,610 ms and 21,757 ms at the incident), against answerRouteBudgetMs.fast of 25,000 ms -- so retrieval consumes 4-27% of the fast budget and is no longer capable of binding it. The 2026-08-14 verdict that pre-generation latency WAS the binding cause stands for that window and is now closed out. Residual R4 (chronic ~30 s strong-route provider_timeout on metformin-renal-dosing and valproate-pregnancy, with a safe source-backed extractive fallback) is generation-side and unchanged; no separate R4 row was created, per this row's own instruction. INCONSISTENCY TO FIX AT RECONCILE: the recommended-execution-queue row for #231 still presents it as A1 / 'immediate approved live investigation' with the old framing ('live answers degrade to source-only when answerRouteBudgetMs.fast binds while retrieval is healthy'), which contradicts the P1 -> P2 re-grade recorded in this detail row. The queue entry should be re-graded to match P2 and re-scoped to the R4 generation-side residual, so the queue stops advertising a retrieval investigation that the measurements have closed.", + "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(a) and 5.2; production probes 2026-08-22 Perth (2026-08-21 UTC)", "baseRowFingerprint": "a2d8deb8060b77d1ed3718818a01b31372c4b30281165dc049102fa616b396e9" } } diff --git a/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json index f21d89f5a2..98809a6867 100644 --- a/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json +++ b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json @@ -7,8 +7,8 @@ "pri": "P2", "type": "issue", "summary": "Restored trigram indexes have never been scanned, so the 2026-08-14 incident fix is probably mis-attributed to them rather than to ANALYZE", - "detail": "Measured on production 2026-08-22, read-only. pg_stat_database.stats_reset is NULL, so index counters are cumulative and have never been reset; 163 of 210 public indexes show scans, max(idx_scan) 262,335,345, total 344,747,815 -- the counters are demonstrably live. Against that, ALL 22 indexes restored or repaired by the incident response report idx_scan = 0: the 20 Phase 4 rebuilds of 2026-08-19, and both trigram indexes restored 2026-08-14 -- document_chunks_content_trgm_idx (68 MB, OID 1491258) and documents_title_trgm_idx (648 kB, OID 1491257). The OID matters: 1491258 is the same value Phase 4 Step 4 recorded as UNCHANGED across the 2026-08-19 push, so the index was never dropped and recreated and its counter has accumulated for the full seven days since restoration. What actually serves retrieval on those tables is document_chunks_search_idx (GIN tsvector, 37,717 scans), document_chunks_embedding_hnsw_idx (HNSW, 4,906) and documents_title_search_idx (37,299). Note that a DIFFERENT trigram index on a sibling table, document_table_facts_title_row_param_trgm_idx, is used 2,372 times, so this is not a blanket property of the planner or of pg_trgm. CONSEQUENCE: forensics Phase 5 attributes the 31,610 -> 1,535 ms recovery to 'restore + ANALYZE', treating the trigram restore as the fix. Zero scans mean those indexes cannot be carrying that recovery, and ANALYZE was an unseparated co-intervention in the same operation -- 20260626020000 itself ends by running analyze over documents, document_chunks, document_table_facts, document_embedding_fields and document_index_units. Stale planner statistics is the better-supported explanation for both the 31-second regression and its recovery. THIS DOES NOT RETRACT THE REPAIR: the two indexes are repo-defined, the migration chain commands them, and drift is correctly closed by their presence. It retracts the ATTRIBUTION, and it changes triage order -- the next 30-second retrieval incident should check planner statistics BEFORE hunting missing indexes. Secondary and non-urgent: roughly 69 MB of currently unscanned index. NEXT: (a) in an approved window, separate the two interventions by measuring retrieval latency before and after a fresh ANALYZE, and (b) only then decide whether the two trigram indexes earn their storage. STOP: do NOT drop them -- they are repo-defined and dropping them reopens live drift.", - "source": "production read-only window 2026-08-22; docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(c)", + "detail": "Measured on production 2026-08-22 Perth (2026-08-21 UTC), read-only. pg_stat_database.stats_reset is NULL, so no database-wide reset has been recorded -- note this does NOT prove the counters were never reset, because pg_stat_reset_single_table_counters(oid) resets one table or index without touching that database-level timestamp; 163 of 210 public indexes show scans, max(idx_scan) 262,335,345, total 344,747,815 -- the counters are demonstrably live. Against that, ALL 22 indexes restored or repaired by the incident response report idx_scan = 0: the 20 Phase 4 rebuilds of 2026-08-19, and both trigram indexes restored 2026-08-14 -- document_chunks_content_trgm_idx (68 MB, OID 1491258) and documents_title_trgm_idx (648 kB, OID 1491257). The OID matters: 1491258 is the same value Phase 4 Step 4 recorded as UNCHANGED across the 2026-08-19 push, so the index was never dropped and recreated and its counter has accumulated for the full seven days since restoration. What actually serves retrieval on those tables is document_chunks_search_idx (GIN tsvector, 37,717 scans), document_chunks_embedding_hnsw_idx (HNSW, 4,906) and documents_title_search_idx (37,299). Note that a DIFFERENT trigram index on a sibling table, document_table_facts_title_row_param_trgm_idx, is used 2,372 times, so this is not a blanket property of the planner or of pg_trgm. CONSEQUENCE: forensics Phase 5 attributes the 31,610 -> 1,535 ms recovery to 'restore + ANALYZE', treating the trigram restore as the fix. Zero scans mean those indexes are very unlikely to be carrying that recovery -- unlikely rather than impossible, because a per-relation counter reset would be invisible to the read above; the OID continuity across the 2026-08-19 push is what makes the zero credible, and ANALYZE was an unseparated co-intervention in the same operation -- 20260626020000 itself ends by running analyze over documents, document_chunks, document_table_facts, document_embedding_fields and document_index_units. Stale planner statistics is the better-supported explanation for both the 31-second regression and its recovery -- better-supported, not established. THIS DOES NOT RETRACT THE REPAIR: the two indexes are repo-defined, the migration chain commands them, and drift is correctly closed by their presence. It retracts the ATTRIBUTION, and it changes triage order -- the next 30-second retrieval incident should check planner statistics BEFORE hunting missing indexes. Secondary and non-urgent: roughly 69 MB of currently unscanned index. NEXT: (a) in an approved window, first rule out a per-index counter reset -- on PostgreSQL 18+ read pg_stat_all_indexes.stats_reset for these two OIDs, and on earlier versions fall back to whatever reset or audit record exists (the production server version was not read in this window); (b) separate the two interventions by measuring retrieval latency before and after a fresh ANALYZE; and (c) only then decide whether the two trigram indexes earn their storage. STOP: do NOT drop them -- they are repo-defined and dropping them reopens live drift.", + "source": "production read-only window 2026-08-22 Perth (2026-08-21 UTC); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(c)", "issueUlid": "01M0JTGYEJ47M1XD0V0EGS5BDD" } } diff --git a/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json index 5371e39f7f..88b248dab9 100644 --- a/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json +++ b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json @@ -5,7 +5,7 @@ "action": "done", "payload": { "id": "#316", - "outcome": "CLOSED 2026-08-22 -- Phase 5, this row's last remaining follow-on, is complete and the alarm is OBSERVED clear rather than merely believed clear. (1) MEASUREMENT, like-for-like with the section 1.3 probe style: text fast path 31,610 -> 955 ms (33x better than the incident, 1.6x better than the 2026-08-14 post-restore reading); hybrid 21,757 -> 6,720 ms. Nothing was worse than the 2026-08-17 healthy baseline, so nothing was escalated. A new EXPLAIN forward baseline was captured on the healthy system for all four RPCs that explain_retrieval_rpc supports, zero errors across twelve samples; the new slowest path is match_document_table_facts_text (1.72 s unprimed, 1,070 unprimed shared_read). Section 1.3's two pending items are now closed: missing-index sizing is moot (0 missing) and the rag_retrieval_logs miss-scan baseline is recorded (47 seq scans, 6,620 tuples read, 640 live rows, 1.4 MB -- cheap at this scale; the row to watch as it grows is rag_aliases at 5,829 seq scans over 68 rows). (2) 5.2 confirmed ALREADY SATISFIED via #231 and not reopened: retrieval now consumes 4-27% of the 25 s fast-route budget, so it cannot bind it. (3) 5.3 needed NO eval canary -- Phase 3 deployed zero behaviour change -- so it reduced to check:production-readiness: READY, 9 PASS, 0 blocking failures. (4) PRODUCTION RE-VERIFIED UNCHANGED, 8/8 against the 2026-08-21 baseline: 210 public indexes, 0 invalid, 211 history rows, 20 no-statement rows, migration_history_versions() probe ok with 211 versions, search_schema_health() ok true. (5) STAGING PARITY CLOSED: 20260820120000 applied to ikoiolksxqxfxgiyqpnu by real db push (never migration repair); staging now reads 211 rows with the function present and stmt_count 3, matching production exactly. (6) THE ALARM IS OBSERVED CLEAR: live-drift run 32514326022, dispatched on main, is GREEN -- 'No unexpected schema drift between live and supabase/schema.sql', 'Remote migration versions: 211 (read via rpc)', 'Remote-only (Preview blockers): 0', every step success. That is the FIRST green live-drift since 2026-07-19T19:37:29Z (of the last 60 runs only four succeeded), ending a 33-day red streak, and pinned issue #1963 auto-closed at 2026-08-21T18:38:42Z. (7) D4 DOCUMENTATION CORRECTED: this row's 2026-08-19 'auto-deploy is OFF' sentence was already superseded by the 2026-08-21 correction; the coordination board and the forensics dossier are now aligned to the dashboard fact (Deploy to production ENABLED, production branch main, automatic branching ON limit 3), with the superseded safe-either-way reasoning retained and marked rather than deleted. THREE FINDINGS QUEUED SEPARATELY rather than absorbed here: the uncapped Branching Compute cost; the explain_retrieval_rpc gap on the _v2 RPC family (needs production DDL, deliberately not written in a read-only phase); and the discovery that all 22 restored indexes report idx_scan = 0 against never-reset statistics, which makes ANALYZE rather than the trigram restore the better-supported explanation for the incident recovery. RESIDUAL WORK LIVES ON ITS OWN ROWS, not here: PITR is still OFF on production, and the fifteen no-statements history rows remain #Q5JHBJ.", + "outcome": "CLOSED 2026-08-22 Perth (2026-08-21 UTC) -- Phase 5, this row's last remaining follow-on, is complete and the alarm is OBSERVED clear rather than merely believed clear. (1) MEASUREMENT, like-for-like with the section 1.3 probe style: text fast path 31,610 -> 955 ms (33x better than the incident, 1.6x better than the 2026-08-14 post-restore reading); hybrid 21,757 -> 6,720 ms. Nothing was worse than the 2026-08-17 healthy baseline, so nothing was escalated. A new EXPLAIN forward baseline was captured on the healthy system for all four RPCs that explain_retrieval_rpc supports, zero errors across twelve samples; the new slowest path is match_document_table_facts_text (1.72 s unprimed, 1,070 unprimed shared_read). Section 1.3's two pending items are now closed: missing-index sizing is moot (0 missing) and the rag_retrieval_logs miss-scan baseline is recorded (47 seq scans, 6,620 tuples read, 640 live rows, 1.4 MB -- cheap at this scale; the row to watch as it grows is rag_aliases at 5,829 seq scans over 68 rows). (2) 5.2 confirmed ALREADY SATISFIED via #231 and not reopened: retrieval now consumes 4-27% of the 25 s fast-route budget, so it cannot bind it. (3) 5.3 needed NO eval canary -- Phase 3 deployed zero behaviour change -- so it reduced to check:production-readiness: READY, 9 PASS, 0 blocking failures. (4) PRODUCTION RE-VERIFIED UNCHANGED, 8/8 against the 2026-08-21 baseline: 210 public indexes, 0 invalid, 211 history rows, 20 no-statement rows, migration_history_versions() probe ok with 211 versions, search_schema_health() ok true. (5) STAGING PARITY CLOSED: 20260820120000 applied to ikoiolksxqxfxgiyqpnu by real db push (never migration repair); staging now reads 211 rows with the function present and stmt_count 3, matching production exactly. (6) THE ALARM IS OBSERVED CLEAR: live-drift run 32514326022, dispatched on main, is GREEN -- 'No unexpected schema drift between live and supabase/schema.sql', 'Remote migration versions: 211 (read via rpc)', 'Remote-only (Preview blockers): 0', every step success. That is the FIRST green live-drift since 2026-07-19T19:37:29Z (of the last 60 runs only four succeeded), ending a 33-day red streak, and pinned issue #1963 auto-closed at 2026-08-21T18:38:42Z. (7) D4 DOCUMENTATION CORRECTED: this row's 2026-08-19 'auto-deploy is OFF' sentence was already superseded by the 2026-08-21 correction; the coordination board and the forensics dossier are now aligned to the dashboard fact (Deploy to production ENABLED, production branch main, automatic branching ON limit 3), with the superseded safe-either-way reasoning retained and marked rather than deleted. THREE FINDINGS QUEUED SEPARATELY rather than absorbed here: the uncapped Branching Compute cost; the explain_retrieval_rpc gap on the _v2 RPC family (needs production DDL, deliberately not written in a read-only phase); and the discovery that all 22 restored indexes report idx_scan = 0 with no database-wide statistics reset recorded, which makes ANALYZE rather than the trigram restore the better-supported explanation for the incident recovery (a per-relation pg_stat_reset_single_table_counters() call would not show in that read, so this is well-supported, not proven). RESIDUAL WORK LIVES ON ITS OWN ROWS, not here: PITR is still OFF on production, and the fifteen no-statements history rows remain #Q5JHBJ.", "baseRowFingerprint": "17b4358a583c8bf5d2b7cc77a6ccf0b8eaf714b92a4e7301f3756a9e831b46ad" } } From f6281f7dd55e4ad60e362ca3b064c0833501c99f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 21 Aug 2026 19:49:52 +0000 Subject: [PATCH 5/5] docs(ledger): supersede the Phase 5 close-out review record with the decisive gate line MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior record's checks cell read only "verify:pr-local (11/11, failed: none)". Repo policy is that evidence is never compressed, so the record now carries the gate's own summary output verbatim. The earlier record could not be edited to carry it: check:branch-review-ledger pins each record's filename to the SHA-256 of its row and rejects any prose outside that single table row, so an in-place edit fails the gate (reproduced — "filename must be the SHA-256 content address for its record row"). The sanctioned route is ledger:append --supersede, which is what this is. It also records the correct head for the review sweep. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01ApB8NBygQn9cosxCQ8omk9 --- ...d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md diff --git a/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md b/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md new file mode 100644 index 0000000000..5414325b9f --- /dev/null +++ b/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md @@ -0,0 +1 @@ +| 2026-08-21 | claude/phase-5-closeout | 653712cbeda0059979e71131e828241f921f17da | PR #2250 review-thread sweep: Codex P1/P2 + CodeRabbit findings on the Phase 5 close-out docs and ledger inbox | Fixed and resolved. P1 db-push contradiction removed (db push reserved for authorised staging/recovery); plan-flip and index-units deliverables left explicitly OPEN; never-reset claim qualified to database-wide only; Perth/UTC boundary made explicit; four MD040 fences labelled. Supersedes the c3ca68fa record, whose checks cell compressed the gate output. | verify:pr-local decisive line — "PR-local verification summary: - completed: check:runtime, check:installed-lock-parity, format:changed, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline / - failed: (none) / - not reached: (none)"; format (whole tree, committed) |