diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 76072e0b50..5e7c2c1675 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -1684,6 +1684,258 @@ choosing any code mitigation. Residual: hybrid fan-out still costs ~8.5 s worst- by the remaining remediation phases, not a route-budget change (`#231`'s stop condition stands). `check:production-readiness` on the final state is **pending**. +### Phase 5 close-out — 2026-08-22 Perth (2026-08-21 UTC) + +_**Dates in this section are Perth local time (UTC+8), with the UTC date given alongside where the +two differ.** The window ran across the local-midnight boundary: the review and every commit are +dated 2026-08-21 UTC, which is 2026-08-22 in Perth. Where a record elsewhere in the repository says +"2026-08-22" bare for this phase, it means 2026-08-22 Perth = 2026-08-21 UTC — no measurement was +taken after the review date._ + +_Owner-authorised **read-only** production window (`sjrfecxgysukkwxsowpy`) plus a staging-only +apply. A dedicated worktree carried the link; the main checkout stayed on its staging link and +every call re-verified `supabase/.temp/project-ref` immediately beforehand. No production DDL, no +data mutation, no paid eval._ + +#### Production is unchanged since 2026-08-21 — 8 of 8 baseline checks match + +| Check | Expected | Read | +| --------------------------------------- | -------- | ---------------- | +| `public` indexes | 210 | **210** | +| invalid / not-ready indexes | 0 | **0** | +| `supabase_migrations` rows | 211 | **211** | +| rows with no executed statements | 20 | **20** | +| `migration_history_versions()` probe | ok | **ok** | +| `migration_history_versions()` versions | 211 | **211** | +| `search_schema_health()` `ok` | true | **true** | +| latest version | — | `20260820120000` | + +Nothing diverged, so no escalation was raised on the production reading. + +#### 5.1(a) End-to-end probes — the like-for-like row §1.3 asked for + +Same endpoint and query style as the incident measurements (`POST /api/search`, semantic clinical +queries, live production data). `supabase_rpc_latency_ms` is the server-measured retrieval +component and is the comparable figure; `total` additionally carries local dev-server render +overhead in this run, so it is directionally useful rather than strictly comparable. + +| Measurement | Before (2026-08-14) | After restore + ANALYZE (2026-08-14) | **Now (2026-08-22 Perth = 2026-08-21 UTC)** | +| ------------------------------------------ | ------------------- | ------------------------------------ | ------------------------------------------- | +| Semantic, text fast path — total / rpc ms | 37.7 s / 31,610 | 4.8 s / 1,535 | **8.4 s / 955** | +| Semantic, hybrid strategy — total / rpc ms | 29.9 s / 21,757 | 17.2 s / 8,519 | **10.9 s / 6,720** | +| Warm repeat (`search_cache`) — total / rpc | not measured | not measured | **0.41 s / 0** | + +Retrieval latency has held its recovery and improved further on both strategies: the fast path is +**33× better than the incident** and 1.6× better than the 2026-08-14 post-restore reading; hybrid is +3.2× better than the incident and 1.3× better than post-restore. Nothing is materially worse than +the 2026-08-17 healthy baseline, so no escalation was raised. + +#### 5.1(b) EXPLAIN forward baseline — captured now, on the healthy system + +§1.3 never captured an EXPLAIN set; its before-numbers were end-to-end probes. This is that missing +baseline, recorded so a future incident has something to diff against. Captured with +`npm run profile:retrieval -- --analyze` (which calls the service-role-only +`public.explain_retrieval_rpc`), query `clozapine monitoring requirements`, `--match-count 24`, +`--samples 3`, public scope. Sample 1 is `first_unprimed`; managed Supabase buffers are never +flushed, so no sample is truly cold. + +| RPC | unprimed exec ms | warm median exec ms | warm median RTT ms | shared_hit (warm) | shared_read (unprimed) | +| ----------------------------------- | ---------------: | ------------------: | -----------------: | ----------------: | ---------------------: | +| `match_documents_for_query` | 42.642 | 5.949 | 102.783 | 1,175 | 10 | +| `match_document_chunks_text` | 867.942 | 35.821 | 150.652 | 7,243 | 42 | +| `match_document_lookup_chunks_text` | 13.237 | 3.013 | 126.042 | 511 | 7 | +| `match_document_table_facts_text` | 1,722.211 | 99.183 | 179.852 | 2,516 | 1,070 | + +Zero errors across all twelve samples. **The new slowest path is +`match_document_table_facts_text`** (1.72 s unprimed, 1,070 unprimed `shared_read`) — recorded here +as the outlier to watch, replacing the trigram-absence signature that dominated the incident. + +**Two limits of this instrument, stated so the baseline is not over-read:** + +1. **It cannot show plan flips.** `explain_retrieval_rpc` runs `EXPLAIN` over + `select * from public.(…)`, so the plan is a single `Function Scan`: a PL/pgSQL body's inner + plan is not exposed. Every sample reports `plan_node_types: ["Function Scan"]` and + `index_names: []`. Plan §5.1's "record plan flips (seq scan → index scan)" is therefore **not + answerable through this tool**. The index-usage read in 5.1(c) is a _different and weaker_ signal, + not a substitute: `pg_stat_user_indexes.idx_scan` is cumulative across every workload that touches + the table, and no before/after counter delta was captured around these samples, so it can show + that an index is never chosen by **anything**, but it cannot show whether **this** profiled query + moved from a sequential scan to an index scan. **Plan §5.1's plan-flip deliverable is therefore + left explicitly OPEN, not discharged.** Closing it needs query-specific inner-plan evidence — + which requires the RPC extension queued below, or an `auto_explain`-style capture — and that is + production-side work this read-only phase could not do. +2. **It cannot reach the two v2 RPCs named in the Phase 5 brief.** `explain_retrieval_rpc` accepts + exactly four names (`match_documents_for_query`, `match_document_chunks_text`, + `match_document_lookup_chunks_text`, `match_document_table_facts_text`) — verified in its only + defining migration `20260626020000_phase7_retrieval_rpc_performance.sql`, in `supabase/schema.sql`, + and on production itself: + + ```text + ERROR: 22023: Unsupported retrieval RPC: match_document_chunks_text_v2 + CONTEXT: PL/pgSQL function explain_retrieval_rpc(text,text,integer,uuid,uuid[],boolean) line 25 at RAISE + ERROR: 22023: Unsupported retrieval RPC: match_document_index_units_hybrid_v2 + ``` + + Extending the function to cover the `_v2` family is production DDL and was **not** authorised for + this phase, so it was not written. Queued as its own ledger item. **Substitute coverage is partial, and + unevenly so — the earlier claim that the v1 siblings cover both owning tables was wrong.** + `match_document_chunks_text`, profiled above, is the v1 sibling of `match_document_chunks_text_v2` + and shares its owning table `document_chunks`, so that target has a usable stand-in. The other one + does not: `match_document_index_units_hybrid_v2` delegates to + `match_document_index_units_hybrid_scoped` over `document_index_units` + (`supabase/schema.sql:8033-8054`), and the four supported names reach only `documents`, + `document_chunks`, lookup chunks and `document_table_facts`. **The `document_index_units` + retrieval path therefore has no EXPLAIN baseline at all**, and that is an open deliverable rather + than a covered one. Its index-level usage is still recorded in 5.1(c), which is the weaker signal + described above, not a plan. + +#### 5.1(c) Index usage — the finding this phase actually turned up + +`pg_stat_database.stats_reset` is **NULL** on production, so **no database-wide reset** +(`pg_stat_reset()`) has been recorded. That is the limit of what this read proves: a per-relation +`pg_stat_reset_single_table_counters(oid)` call resets one table's or one index's counters without +touching `pg_stat_database.stats_reset`, and this window captured no evidence either way for the +individual indexes below. A zero is therefore _very likely_ to mean genuinely never scanned rather +than recently zeroed — strongly so given the OID continuity established below — but it is not proven. +Settling it needs a per-index reset time, which `pg_stat_all_indexes.stats_reset` carries only on +PostgreSQL 18 and later; the running server version was not read in this window. Across `public`, +163 of 210 indexes show scans, `max(idx_scan)` is 262,335,345 and the total is 344,747,815 — the +counters are clearly live. + +Against that, **all 22 indexes restored or repaired by the incident response report `idx_scan = 0`** — +the 20 Phase 4 rebuilds (2026-08-19) and, more pointedly, both trigram indexes restored on 2026-08-14: + +| Index | Size | `idx_scan` | Present since | OID | +| ---------------------------------- | ------ | ---------: | ------------- | ------- | +| `document_chunks_content_trgm_idx` | 68 MB | **0** | 2026-08-14 | 1491258 | +| `documents_title_trgm_idx` | 648 kB | **0** | 2026-08-14 | 1491257 | + +The OID `1491258` is the same value §Phase 4 Step 4 recorded as _unchanged across the 2026-08-19 +push_, which is what makes the zero meaningful: the index has not been dropped and recreated, so its +counter has been accumulating for the full seven days since restoration. Both are `indisvalid` and +`indisready` with canonical `coalesce(...)` definitions. + +What _is_ serving retrieval on the same tables: + +| Index | Kind | `idx_scan` | +| ----------------------------------------------- | ---------------- | ---------: | +| `document_chunks_search_idx` | GIN `search_tsv` | **37,717** | +| `document_chunks_embedding_hnsw_idx` | HNSW vector | **4,906** | +| `documents_title_search_idx` | GIN tsvector | **37,299** | +| `document_table_facts_title_row_param_trgm_idx` | GIN trigram | **2,372** | + +The last row matters: a _different_ trigram index on a sibling table is used thousands of times, so +"trigram indexes are never chosen here" is not a blanket property of the planner or the extension. + +**Consequence for the incident narrative — stated as a correction, not a certainty.** The 2026-08-14 +Phase 5 table above attributes the 31,610 ms → 1,535 ms recovery to "restore + ANALYZE", treating the +two trigram indexes as the fix. Seven days of cumulative statistics show the restored trigram indexes +have served **zero** scans. Subject to the reset caveat recorded above — a per-relation +`pg_stat_reset_single_table_counters()` call would be invisible to the database-wide `stats_reset` +read, so "zero scans since restoration" is strongly supported rather than proven — they cannot be +carrying that recovery. `ANALYZE` was an unseparated +co-intervention in the same operation — and `20260626020000` ends by running `analyze` over exactly +`documents`, `document_chunks`, `document_table_facts`, `document_embedding_fields` and +`document_index_units` — which makes stale planner statistics the better-supported explanation for +both the 31-second regression and its recovery. This does **not** retract the repair: the two indexes +are repo-defined, the chain commands them, and drift is correctly closed by their presence. It +retracts the _attribution_, and it changes what a future 30-second retrieval incident should check +first — planner statistics before missing indexes. Raised as its own ledger item rather than absorbed +here; ~69 MB of currently unscanned index is a secondary, non-urgent question that follows from it. + +#### 5.1(d) `rag_retrieval_logs` miss-scan baseline — §1.3's second pending item, now closed + +| Table | `seq_scan` | `seq_tup_read` | `idx_scan` | live rows | total size | avg rows / seq scan | +| -------------------- | ---------: | -------------: | ---------: | --------: | ---------: | ------------------: | +| `rag_retrieval_logs` | 47 | 6,620 | 98 | 640 | 1440 kB | 140.9 | +| `rag_query_misses` | 421 | 29,482 | 509 | 177 | 320 kB | 70.0 | +| `rag_queries` | 51 | 33,792 | 419 | 258 | 4760 kB | 662.6 | +| `rag_aliases` | 5,829 | 379,728 | 23,583 | 68 | 224 kB | 65.1 | +| `rag_response_cache` | 2,133 | 13,203 | 5,352 | 0 | 656 kB | 6.2 | + +**Reading: not currently a latency risk, and that is the point of recording it.** Every one of these +relations is under 5 MB with at most a few hundred live rows, so a sequential scan is a handful of +pages and cheaper than an index descent — the planner's choice is correct at this scale. The row to +watch is `rag_aliases`: 5,829 sequential scans over 68 rows is free today and becomes a real cost if +that table ever grows. This table is the baseline; the trigger for action is growth in +`seq_tup_read`, not the presence of `seq_scan`. + +#### 5.2 — confirmed already satisfied, and re-confirmed with fresh data + +Ledger `#231` already records that plan §5.2 "is satisfied by S1's 2026-08-17 healthy-latency +probes", and re-graded itself P1 → P2 on that basis with residual **R4** (chronic ~30 s strong-route +`provider_timeout` on `metformin-renal-dosing` and `valproate-pregnancy`, with a safe source-backed +extractive fallback, never model synthesis). This phase **confirms that reading rather than +reopening it**: measured retrieval now costs 955 ms (fast path) to 6,720 ms (hybrid) against an +`answerRouteBudgetMs.fast` of 25,000 ms, so retrieval consumes 4–27% of the budget and is no longer +capable of binding it. The 2026-08-14 verdict — that pre-generation latency _was_ the binding cause +during the incident window — stands for that window and is now closed out. R4 is a generation-side +residual and keeps `#231`'s existing stop condition; **no separate R4 row was created**, per `#231`'s +own instruction. + +One inconsistency found and queued, not edited here: the _recommended-queue_ row for `#231` still +presents it as A1 / "immediate approved live investigation", contradicting the P2 re-grade in its own +detail row. Queued through `issues:update` for the next reconcile. + +#### 5.3 — no eval canary owed + +Plan §5.3 gates `eval:retrieval:quality` on "if any Phase 3 RPC deploy changed behaviour". Phase 3 +was reframed to repo-side codification and deployed **zero** behaviour change: `#316` records manifest +`def_hash` equal to live for all ten `match_*` functions, and live-drift `32131517648` reported 0 +function mismatches. §5.3 therefore reduces to `check:production-readiness`, run once at the end of +this phase. No paid eval was run and none was requested. + +```text +$ npm run check:production-readiness +[Production Readiness] +Project: Clinical KB Database (sjrfecxgysukkwxsowpy) +PASS (9): + - Node runtime 24.19.0 matches required Node 24.x. + - Boot guard invokes requireQueryHashSecret(); the query-hash HMAC fails closed in production (PIA-2). + - Local override file .env.local is present + - Server env includes required Supabase project values. + - OpenAI API key is configured. + - OpenAI safety identifiers use a deployment-secret HMAC; raw owner IDs are not sent. + - RAG_QUERY_HASH_SECRET is set; logged clinical-query hashes are keyed HMAC pseudonyms (PIA-2). + - HEALTH_DEEP_PROBE_SECRET is set for authorized deep health probes. + - Supabase project config points to Clinical KB Database. +READY: no blocking production-readiness failures. +``` + +Nine checks pass, zero blocking failures, and the gate independently confirms the operator +environment is bound to `Clinical KB Database` (`sjrfecxgysukkwxsowpy`). **§1.3's "`check:production-readiness` +on the final state is pending" is closed.** + +#### Staging catch-up — the owed apply, now closed + +Staging (`ikoiolksxqxfxgiyqpnu`) was one version behind after the 2026-08-20 window, where both write +paths were denied by that session's auto-mode classifier. Applied here, by real `db push` (never +`migration repair`); the pending set was confirmed first as exactly one version with zero remote-only: + +```text +$ supabase migration list --linked --project-ref ikoiolksxqxfxgiyqpnu +… {"local":"20260820120000","remote":""} +$ supabase db push --linked --project-ref ikoiolksxqxfxgiyqpnu --skip-vault --yes +Applying migration 20260820120000_migration_history_versions_rpc.sql... +{"upToDate":false,"dryRun":false,"migrations":["20260820120000_migration_history_versions_rpc.sql"],…} +``` + +`--include-all` was correctly not needed: the pending version sorts after the remote tip +`20260819110500`. + +| Staging check | Before | After | +| -------------------------------------- | ---------------- | ---------------------------------- | +| `supabase_migrations` rows | 210 | **211** | +| latest version | `20260819110500` | **`20260820120000`** | +| `migration_history_versions()` present | absent | **present** | +| history row `stmt_count` | — | **3** (executed, not mark-applied) | +| probe / version_count | — | **ok / 211** | +| `public` indexes | 210 | **210** | + +`stmt_count 3` matches production's row exactly, so both tiers now carry the same executed +migration rather than a repaired history entry. **Staging parity, open by one version since +2026-08-20, is closed.** + ## Phase 6 — Future-proofing (repo-side; one migration authored, NOT deployed) _2026-08-18 (repo-only session; no hosted read or mutation)._ Built per plan §6.1–6.3, worker chat @@ -2053,7 +2305,43 @@ $ supabase migration list --linked --project-ref sjrfecxgysukkwxsowpy empty and the step should now pass. **No guard migration is owed**: the row carries executed statements, so it is not a history repair and cannot surface in the `migration_history` probe. -#### D4: strong evidence of deploy-on-merge — but the toggle itself has not been read +#### D4 — SETTLED 2026-08-21 by a direct dashboard read: deploy-on-merge is ON + +**This question is closed. The Supabase dashboard was read on 2026-08-21 and shows the GitHub +integration with "Deploy to production" ENABLED, production branch `main` — "Apply changes to your +production database when you merge into your configured production GitHub branch". Automatic +branching is also ON (limit 3, "Supabase changes only" enabled).** The inference recorded below was +correct; it is no longer an inference, and the safe-either-way hedge it justified is retired. + +Consequences, which `AGENTS.md` § "Supabase project safety" now carries as the authoritative rule: + +- **Merge approval IS production-deploy approval.** Any migration merged to `main` is applied to the + live clinical database automatically, within seconds (34 s measured, §3.7). There is no separate + deploy step to forget and no window in which to hold it back. Never merge a + `supabase/migrations/**` PR outside an approved window, and never arm auto-merge on one. +- **The post-merge gate is a green `live-drift` run**, requiring BOTH `check:drift` and + `check:migration-history`. It is _not_ `supabase migration list`: that reads recorded history only, + so it cannot distinguish an applied migration from a history row whose statements never executed — + precisely the shape of the twenty no-statements rows. `check:drift` compares the live schema itself. +- **A migration that cannot run inside a transaction cannot ship this way.** The integration applies + each migration in one transaction, so a bare `CREATE INDEX CONCURRENTLY` migration fails outright. + Index work stays operator-prebuild plus a validate-only guard migration (the `20260804110240` + pattern). +- **Branching Compute is not covered by the organisation's Spend Cap** (stated on the same dashboard + screen), so preview databases are an uncapped cost that scales with PRs touching `supabase/**`. + CI's `Migration replay` job (`db-reset-verify`, `supabase migration up --local`) independently + replays the whole chain on every database-touching PR, so preview branches are a second net rather + than the only one — that is the context for any later decision to reduce or disable them. Queued as + its own ledger item. + +The `list_branches` reading below remains accurate and is now corroborated rather than superseded: +production is bound to git `main`, and the toggle that acts on that binding is enabled. + +_The analysis that follows was written before the dashboard was read. It is retained because it +records how the question was narrowed, and why the safe-either-way rule was the correct posture while +the toggle was still unknown._ + +##### Superseded reasoning, retained: how D4 stood before the dashboard read `created_by` and `idempotency_key` are NULL for every row from `20260818090000` to `20260820120000`, including the ones this programme applied by operator `db push`, so the history table carries **no @@ -2087,14 +2375,14 @@ Production is **still bound to git `main`**, and `updated_at` predates 2026-08-1 changed that day never touched this binding, and the §3.7 mechanism is intact. Together with §3.7's 34-second apply and `20260820120000` arriving unpushed, that is strong evidence of deploy-on-merge. -**It remains an inference, and the difference matters.** No field of the branch record reports the +**SUPERSEDED 2026-08-21 — do not act on this paragraph or the rule that follows it; the dashboard read above replaces both.** _It remains an inference, and the difference matters._ No field of the branch record reports the "Deploy to production" setting. The superseded 2026-08-19 account describes that setting being changed without the binding being deleted, so "toggle off, binding intact" cannot be ruled out from here — and the two failure modes are not symmetric. Declaring D4 ON tells operators to skip `db push`; if the toggle is in fact off, every merged migration then sits unapplied and drift returns silently, which is the original incident. Declaring it OFF risks only a redundant no-op push. -**So the operative rule is the one that is correct under both states, and it must not be shortened:** +**SUPERSEDED 2026-08-21.** _The operative rule at the time — correct under both states, and deliberately not shortened:_ 1. Never merge a migration PR outside its approved window — correct if deploys happen on merge. 2. After any migration merges, run `supabase migration list --linked --project-ref ` and @@ -2104,7 +2392,7 @@ One extra command, wrong under neither hypothesis. What would replace this rule dashboard read of the toggle, or a deployment-settings API that reports it. Raised as a P1 by the Codex review of PR #2205, and correctly: the earlier wording here presented the inference as a direct read. -#### Staging is one version behind — blocked, not skipped +#### Staging is one version behind — blocked, not skipped (CLOSED 2026-08-22 Perth = 2026-08-21 UTC, see Phase 5 close-out) Staging (`ikoiolksxqxfxgiyqpnu`, verified before every call) reads **210** history rows, latest `20260819110500`, `to_regprocedure('public.migration_history_versions()') is null`. Its pending set is @@ -2116,3 +2404,37 @@ MCP `execute_sql` alike — under the live-Supabase confirmation rule added by P Read-only calls were unaffected, which is why every verification above exists. This is an authorisation gap, not a technical one: **staging parity, closed by Phase 4, is open again by one version until an operator applies it.** + +**CLOSED 2026-08-22 Perth (2026-08-21 UTC).** An operator session applied `20260820120000` to staging by real `db push`; +staging now reads 211 history rows with `migration_history_versions()` present and `stmt_count 3`, +matching production exactly. Evidence in the "Phase 5 close-out" section, "Staging catch-up". + +## Alarm cleared — live-drift green on `main`, 2026-08-21 (run `32514326022`) + +The alignment fix landed on 2026-08-20 but had **never been observed passing**: two sessions existed +because `live-drift` was red on its `Align migration history for Supabase Preview` step (PGRST106). +Dispatched here on `main` with the fix live on production and staging brought to parity. + +`https://github.com/BigSimmo/Database/actions/runs/32514326022` — **job conclusion: `success`.** + +```text +live-drift Compare live schema drift No unexpected schema drift between live and supabase/schema.sql. +live-drift Align migration history for Supabase Preview Remote migration versions: 211 (read via rpc) +live-drift Align migration history for Supabase Preview Remote-only (Preview blockers): 0 +``` + +Every step reported `success`, including `Guard Supabase project identity`, `Compare live schema +drift`, `Align migration history for Supabase Preview`, and `Capture drift and migration-history +findings`. + +**This is the first green `live-drift` run since 2026-07-19T19:37:29Z** — measured, not assumed: of +the last 60 runs, exactly four succeeded (`2026-07-13T09:58:30Z`, `2026-07-13T17:10:55Z`, +`2026-07-19T19:37:29Z`, and this one), and the oldest run in that window is `2026-07-09`. That is a +**33-day continuous red streak**, closed. + +**Pinned issue `#1963` "Live drift check failing" auto-closed at `2026-08-21T18:38:42Z`**, by the +`drift-routing` job, seconds after the alignment step reported zero Preview blockers — the +self-closing behaviour the routing was built for, now demonstrated end to end for the first time. + +Both halves of the original `#316` alarm are therefore closed _and observed closed_: drift findings +are empty, and the job that reports them is green. diff --git a/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md b/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md new file mode 100644 index 0000000000..dcddda298f --- /dev/null +++ b/docs/branch-review-records/da9eadddac4befebc995b1000caa73f1fa0ced0d25d52b5789223410bac06c08.record.md @@ -0,0 +1 @@ +| 2026-08-21 | claude/phase-5-closeout | c3ca68fa7f13e2d7ae6c4c96ccfcafb67a59135a | Phase 5 close-out: measurement baselines, staging parity, D4 documentation correction (#316) | Self-review; no blocking findings. Docs-only diff plus five ledger inbox requests. Read-only production window, one staging-only db push, one live-drift dispatch (green). Two limits and one attribution finding recorded rather than absorbed; no RAG surface touched. | verify:pr-local (11/11, failed: none); check:production-readiness (READY, 9 PASS, 0 blocking); format (whole tree, committed); live-drift run 32514326022 green | diff --git a/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md b/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md new file mode 100644 index 0000000000..5414325b9f --- /dev/null +++ b/docs/branch-review-records/eecaf3f51b0f4d8dad88a76dbb9c162887d5067cd48bdde0bbaa87a410f4701a.record.md @@ -0,0 +1 @@ +| 2026-08-21 | claude/phase-5-closeout | 653712cbeda0059979e71131e828241f921f17da | PR #2250 review-thread sweep: Codex P1/P2 + CodeRabbit findings on the Phase 5 close-out docs and ledger inbox | Fixed and resolved. P1 db-push contradiction removed (db push reserved for authorised staging/recovery); plan-flip and index-units deliverables left explicitly OPEN; never-reset claim qualified to database-wide only; Perth/UTC boundary made explicit; four MD040 fences labelled. Supersedes the c3ca68fa record, whose checks cell compressed the gate output. | verify:pr-local decisive line — "PR-local verification summary: - completed: check:runtime, check:installed-lock-parity, format:changed, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline / - failed: (none) / - not reached: (none)"; format (whole tree, committed) | diff --git a/docs/database-remediation-coordination.md b/docs/database-remediation-coordination.md index 7e11617ecc..12413630c4 100644 --- a/docs/database-remediation-coordination.md +++ b/docs/database-remediation-coordination.md @@ -90,13 +90,13 @@ trusted as-is._ | 2 — Staging parity (`#056`) | ☑ COMPLETE 2026-08-18 (re-measured) | **#2093** merged `e546840a1` (§2.0–2.4); **#2104** merged `f19cf8f60` (§2.5 re-measure); `#056` update `5ed2f873` pending reconcile | Gap was 28. Staging `ikoiolksxqxfxgiyqpnu` replayed to parity, corpus untouched, production never targeted. `check:drift` vs staging RED 19 at base `ed43a64f2`; **re-measured at current `main` after applying `20260818090000`: still exactly 19** (same keys, same hashes). Snapshot v2 `migration_history` block runs end-to-end on staging (probe `ok`, 0 rows). Findings: (a) 7 `work_mem` fn mismatches; (b) 8 objects `schema.sql` declares that no migration creates; (c) 3 column-set mismatches; (d) `document_chunks_content_trgm_idx` def mismatch. All four dispositioned by Phase 3. §2.4 traps: staging KEPT the indexes production lost; four duplicate migration version pairs; `20260725000000` points staging's worker GUC at production — **never seed `cron_ingestion_jwt` / `indexing_v3_agent_secret` / `ingestion_webhook_secret` into staging's vault**; **never run `check:drift --prune-stale` against staging** (the five seeded `migration_history` allowlist entries are production's and read stale there — warning only). | | 3 — Reconciliation (reframed) | ☑ COMPLETE 2026-08-18 — repo-side, staging-proven, **and live on production** (auto-applied; see §3.7) | **#2106** merged `72aa18865` (migrations `20260818110000` work_mem ×10, `111000` eight schema-only objects, `112000` chain-stale columns; forensics §3.1–3.6); **#2111** merged `9b52eb075` (staging proof + `20260818113000` forward-codify of three hybrid RPC bodies the chain never received; `#316` final + `#183` rider + 2 captures + 3 cancels queued) | D1 codify-as-live, D2 exemption granted. Manifest `def_hash` == live production hash 10/10 — the ten RPC "divergences" are closed as a mirror gap. §3.3: `token_estimate` was chain-stale (added by `112000`, no-op on prod); `rag_visual_eval_*` id defaults rebound to `pg_catalog.gen_random_uuid()` (no-op on prod); (d) production's restored `document_chunks_content_trgm_idx` (`coalesce(content,'')` form, `8499c3d3…`) IS canonical — staging carries the 2026-06-06 form; repair = Phase 4.4 guard migration + staging drop-and-recreate. Staging at 199 migrations, single drift residual = that index. **New finding:** `20260712000000`'s owner step never landed for three hybrid RPCs — chain-stale, forward-codified by `113000` (not a production hole). Tooling: `scripts/check-drift.ts:192` clips column diffs to 240 chars (never names the column) — queued P3; PR template lacks `RAG impact:` guidance — queued P3. | | 4 — Index restoration | ☑ COMPLETE 2026-08-19 — production **and** staging | **#2151** (branch `claude/db-remediation-phase4-indexes-a1661a`, head `5d3dca4dd`); forensics §Phase 4 completion; live-drift run `32171070287` | Owner-authorised off-peak window, D4 **OFF** so nothing auto-deployed. All **20** `missing_live` indexes rebuilt with `CREATE INDEX CONCURRENTLY` from canonical definitions — Batch A 14/14, Batch B 6/6, every one `indisvalid AND indisready` with normalised `pg_get_indexdef` matching canonical; zero invalid builds, zero retries, zero skips, zero lock waits; no transactional build attempted; `#102` held out. Both `unexpected_live` orphans **DROPPED** concurrently (not codified) — each is a strict leading-column subset of a present canonical index and the chain already commands both drops (`20260620000000`, `20260703030000`/`20260708000000`); neither is in `schema.sql`, so the drop moves live _into_ mirror agreement. Live now 210 indexes vs manifest 210, zero invalid anywhere. Codified by real `db push` (never `migration repair`; all four history rows carry executed statements): `20260819100000`/`100100` guard Batch A/B, **`20260819100200` discharges the 4.4 trigram-guard debt** that `20260804110240` never covered, `20260819100300` takes `required_indexes` 22→30 adopting **all 8** Phase 6.3 monitor-candidates (`search-health-unmonitored-indexes.json` 44→36, no `monitor-candidate` left; production `ok: true`). **Staging also closed:** `document_chunks_content_trgm_idx` rebuilt into the canonical `coalesce(content,'')` form (§3.3(d)), the four migrations applied by the Phase 2 method with all md5s matching, `check:drift` vs staging **green, 0 unexpected (was 19)**, corpus untouched. Live-drift **37 → 16**: `missing_live` **20 → 0**, `unexpected_live` **2 → 0**; the lone function mismatch is merge-pending and proven (live `def_hash 85df52de…` == this branch's regenerated manifest). **Two escalations, not absorbed:** PITR is OFF on production (new P2 queued) and the `migration_history` block did **not** drop — no allowlist entry was earned (empty intersection), so the 15 stay `#Q5JHBJ`'s work. | -| 5 — Measure | ◐ PARTIAL (incident) | #1960 §Phase 5 | Text fast path 31,610 → 1,535 ms; hybrid 8,519 ms residual. Full EXPLAIN set, `work_mem` before/after (only if D1 changes values), `check:production-readiness` pending. | +| 5 — Measure | ☑ COMPLETE 2026-08-22 Perth (2026-08-21 UTC) — two deliverables explicitly OPEN, queued | forensics §Phase 5 close-out; live-drift run `32514326022` | Text fast path **31,610 → 955 ms** (33× vs incident), hybrid **21,757 → 6,720 ms**; nothing worse than the 2026-08-17 healthy baseline. New EXPLAIN forward baseline captured on the healthy system for all four RPCs `explain_retrieval_rpc` supports (new slowest path: `match_document_table_facts_text`, 1.72 s unprimed). §1.3's two pending items closed: missing-index sizing is moot (0 missing) and the `rag_retrieval_logs` miss-scan baseline is recorded. **5.2 confirmed already satisfied** by `#231`'s 2026-08-17 healthy-latency probes — retrieval now consumes 4–27% of the 25 s fast budget, so it cannot bind it; R4 stays generation-side, no separate row. **5.3 needed no eval canary** (Phase 3 deployed zero behaviour change); `check:production-readiness` **READY, 9 PASS / 0 blocking**. **Finding, not absorbed:** all 22 restored indexes report `idx_scan = 0` with no database-wide statistics reset recorded (`pg_stat_database.stats_reset` is NULL), including both trigram indexes credited with the incident fix — the recovery is better explained by the co-administered `ANALYZE` than by the indexes. A per-relation reset via `pg_stat_reset_single_table_counters()` is not excluded by that read, so the attribution is well-supported rather than proven. Queued as its own ledger item. **Two instrument gaps recorded, and they leave two §5.1 deliverables genuinely OPEN rather than discharged:** `explain_retrieval_rpc` exposes only an outer `Function Scan`, so no query-specific plan-flip evidence exists (cumulative `idx_scan` counters are a weaker, whole-workload signal and are not a substitute); and it raises `22023` for the `_v2` family, so `document_index_units` — which no supported RPC touches — has no EXPLAIN baseline at all. Both need production DDL and are queued. | | 6 — Future-proofing | ☑ COMPLETE 2026-08-19 (6.1 live; 6.2 guards live on both tiers; 6.3 repo-side) | **#2058** merged `9c660af1f` (6.1 / 6.2 contract / 6.3); follow-up #2090 merged `79e5d2a68`; **Phase 6.2 guards: branch `claude/migration-history-drift-allowlist-37444c`** (forensics §"6.2 completion"); live-drift run `32251326536` | 6.1 probe LIVE on production since 2026-08-18. **6.2 done 2026-08-19 (`#Q5JHBJ`)**: all fifteen no-statements versions classified — every one `validation`, none `superseded`/`no_ddl` (no single later executed migration re-creates every object; `COMMENT ON` is a catalog write) — and guarded by six fail-fast migrations `20260819110000`…`110500` (dropped objects / comments + cron / document FKs / 46 operational index shapes / `index_generation_id` promotion / 15 function `def_hash`es read from `schema_drift_snapshot()`). Allowlist 5 superseded + 15 validation = 20. Proof: whole chain replayed into the scratch image `Applied 210/210`, `CHAIN == MANIFEST` (every hand repair in §2.3/§3.3/Phase 4 reproduced by the chain; no reconcile needed), seven mutants raise; all six dry-ran green on production and a mutant failed there; real `db push` (pending = exactly six; rows `no_statements false`; 210 rows; corpus untouched); staging by the Phase 2 method, six md5-matched, drift green. **Live-drift on the branch: `No unexpected schema drift`, all 20 history rows allowed — `#316`'s finding set is EMPTY for the first time since 2026-07-26.** Job still red on the Phase 0 `Align migration history` step (PGRST106: `supabase_migrations` not exposed to PostgREST; it ran for the first time ever) — queued as its own P2, owner decision (expose schema / use access token / RPC). Stale branch `claude/database-drift-allowlist-48839e` (tip `3fe1deafe`) redundant, owner may delete. | | 7 — Deferred | ☐ not started | — | Each its own session, per playbook. | **Owner decisions (2026-08-18):** D1 = codify-as-live (128 MB on the four hybrids, 64 MB on the other six) — DECIDED. D2 = `work_mem` eval-canary exemption — GRANTED. D3 = 6.1 deploy bundled into -the Phase 3 production window — DECIDED. D4 = Supabase GitHub auto-deploy — **REOPENED 2026-08-20, treat as ON** (the 2026-08-19 Phase 4 "disabled" reading never tested deploy-on-merge; see the 2026-08-20 window update below). Until the dashboard toggle is re-verified, assume merging a migration PR deploys it to production. +the Phase 3 production window — DECIDED. D4 = Supabase GitHub auto-deploy — **SETTLED 2026-08-21 by a direct dashboard read: "Deploy to production" is ENABLED, production branch `main` ("Apply changes to your production database when you merge into your configured production GitHub branch"); automatic branching is ON (limit 3, "Supabase changes only").** Merging a migration PR to `main` IS a production deployment, applied within seconds. Per `AGENTS.md` § "Supabase project safety": never merge a `supabase/migrations/**` PR outside an approved window, never arm auto-merge on one, and prove the result with a green `live-drift` run (BOTH `check:drift` and `check:migration-history`) — never `supabase migration list`, which reads recorded history only and cannot tell an applied migration from a row whose statements never executed. **Production window — CLOSED 2026-08-18 without a push (PR #2123 → forensics §3.7).** The authorised window's pre-flight `supabase migration list` showed all five `20260818*` migrations @@ -105,7 +105,7 @@ no mark-applied path was used, and production received zero writes. Verified rea `ok`; ten `work_mem` values = D1; live-drift run `32131517648` = **0 function mismatches**, 20 `missing_live`, 2 `unexpected_live`, 15 `migration_history`. The RPC track of `#316` is closed. -**KEY FINDING — D4: strong evidence of deploy-on-merge; the "disabled" reading does not hold (2026-08-20, below). The toggle itself has not been read — act on the safe-either-way rule, not on the inference.** The Supabase GitHub integration (Branching, production +**KEY FINDING — D4 is SETTLED (2026-08-21, direct dashboard read): "Deploy to production" is ENABLED with production branch `main`, so deploy-on-merge is a fact, not an inference. The safe-either-way hedge below is retired and retained only as superseded reasoning; `AGENTS.md` carries the operative rule.** The Supabase GitHub integration (Branching, production bound to git `main`, branch record from 2026-06-27) auto-applies every migration merged to `main` onto production — live-drift bracketed `110000–112000` to ~34 s after #2106 merged. This undermines the explicit-window model the plan and playbook assume: a merged migration IS a production deploy. @@ -150,13 +150,13 @@ needs the same migration by the Phase 2 method. Until that window runs, the week this one step and #1963 will not self-close. **Update 2026-08-20 (window) — the alignment fix is LIVE on production; staging is one version behind; -D4: strong evidence of deploy-on-merge, toggle not directly read.** PR #2198 merged before the window (squash `a341832af`), and the pre-flight +D4 has SINCE BEEN SETTLED (2026-08-21, dashboard read) — see the D4 correction above.** PR #2198 merged before the window (squash `a341832af`), and the pre-flight found `20260820120000` **already applied** — `stmt_count 3`, executed statements, not mark-applied — so `db push` was never run against production. Verified read-only: `prosecdef true`, `provolatile s`, `search_path=""`, `proacl postgres=X/postgres | service_role=X/postgres`, and the function returns `probe: ok` with `version_count 211` against 211 history rows and 211 local files. No guard migration is -owed. **Two things for the owner:** (1) **D4: the "disabled" reading does not hold, but the toggle has -not been read.** `created_by` is NULL on every recent history row, so the database itself proves +owed. **Two things for the owner:** (1) **D4 — SUPERSEDED 2026-08-21; retained as the reasoning of the time. The toggle has since been read and deploy-on-merge is ON.** _As written then: the "disabled" reading does not hold, but the toggle had +not been read._ `created_by` is NULL on every recent history row, so the database itself proves nothing. `list_branches(sjrfecxgysukkwxsowpy)` still returns a live Git branch record binding **production** to git `main`: `{"name":"main","is_default":true,"git_branch":"main","project_ref":"sjrfecxgysukkwxsowpy","created_at":"2026-06-27","updated_at":"2026-07-04"}`. @@ -176,9 +176,9 @@ fact. (2) Staging but both write paths were denied by the session auto-mode classifier (live-Supabase confirmation rule from PR #2196), so Phase 4's staging parity is open again by one version until an operator applies it. -**Next dispatches:** merge the 6.2 PR (its CI `Migration replay` and Supabase Preview are the last chain proofs), then Phase 5 close-out (after-EXPLAIN set, `#231` re-test on healthy latency, `check:production-readiness`), then one serialized `issues:reconcile`. Every future migration still needs its own approved window and its own `db push`. **D4 is REOPENED (2026-08-20): treat merging a migration PR as a production deployment** until the Supabase "Deploy to production" toggle is re-verified in the dashboard — so never merge a migration PR outside its approved window, and apply and verify the migration inside that window before it lands. +**Next dispatches:** Phase 5 close-out is **COMPLETE (2026-08-22 Perth = 2026-08-21 UTC), with two §5.1 deliverables explicitly left OPEN and queued** (query-specific plan-flip evidence, and any EXPLAIN baseline for the `document_index_units` path) — end-to-end before/after probes, a new EXPLAIN forward baseline, the `rag_retrieval_logs` miss-scan baseline, `#231`/5.2 confirmed already satisfied, staging brought to 211, and `live-drift` observed **green** on `main` (run `32514326022`, first success since 2026-07-19; pinned issue #1963 auto-closed). Remaining: one serialized `issues:reconcile`. Every future migration still needs its own approved window, but **not** a separate manual `db push`: with D4 ON the merge itself is the deployment, and the post-merge `live-drift` run — BOTH `check:drift` and `check:migration-history` green — is the schema-application gate. A pre-merge push would put the live schema ahead of the reviewed repository state; a post-merge push would be a redundant production mutation. `db push` is reserved for explicitly authorised staging or recovery work. **D4 is SETTLED (2026-08-21, dashboard read): merging a migration PR to `main` IS a production deployment** — never merge one outside its approved window, never arm auto-merge on one, and prove the outcome with a green `live-drift` run rather than `supabase migration list`. -**Where the programme stands after Phase 4 (2026-08-19).****Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both +**Where the programme stands after Phase 4 (2026-08-19).** The index track of `#316` is closed on both tiers and staging is at full parity, so the remaining live-drift findings are exactly one category: the fifteen `migration_history` no-statements rows, which are `#Q5JHBJ` and need fail-fast guard migrations, never bare allowlisting. Phase 4 earned no allowlist entry for any of them — measured, not diff --git a/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json b/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json new file mode 100644 index 0000000000..0efa3eb5d6 --- /dev/null +++ b/docs/outstanding-issues-inbox/05bd88ac-f24f-464a-8853-5d7ff55b375b.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "05bd88ac-f24f-464a-8853-5d7ff55b375b", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "task", + "summary": "The document_index_units retrieval path has no EXPLAIN baseline, and Phase 5 has no query-specific plan-flip evidence", + "detail": "Two Phase 5.1 deliverables are explicitly OPEN, not discharged. Re-graded P3 -> P2 versus the withdrawn request 2040d1fb, because that request understated the gap by claiming substitute coverage that does not exist. (A) NO EXPLAIN BASELINE FOR THE INDEX-UNITS PATH. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else, proven against production for both match_document_chunks_text_v2 and match_document_index_units_hybrid_v2. For the first of those the v1 sibling match_document_chunks_text shares the owning table document_chunks and is a usable stand-in. For the second there is none: match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units (supabase/schema.sql:8033-8054), and no supported RPC touches that table. document_index_units is one of the two section 1.2 outliers, so the outlier that most needed a baseline is the one that has none. (B) NO QUERY-SPECIFIC PLAN-FLIP EVIDENCE. explain_retrieval_rpc EXPLAINs `select * from public.(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore unanswerable through this instrument. The pg_stat_user_indexes read captured in Phase 5.1(c) is a WEAKER and DIFFERENT signal, not a substitute: idx_scan is cumulative across every workload touching the table and no before/after delta was captured around the samples, so it can prove an index is never chosen by anything but cannot prove that a given query changed plan. NEXT: one migration extending the explain_retrieval_rpc p_rpc branch list to the _v2 family (at minimum match_document_index_units_hybrid_v2 and match_document_chunks_text_v2), shipped in an approved window -- with D4 ON, merging it to main deploys it, so it needs the window and a green post-merge live-drift run. Then re-run npm run profile:retrieval --analyze to capture the missing baseline. For (B), consider whether an auto_explain-style capture is a better fit than widening the RPC. STOP: do not record the cumulative index-usage read as plan-flip evidence; that conflation is exactly what this row exists to prevent.", + "source": "Codex review of PR #2250 (P2, comment 3833062803 and 3833062807); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/schema.sql:8033-8054", + "issueUlid": "01M0JWT29F8VAY97CG3PM5X2QW" + } +} diff --git a/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json new file mode 100644 index 0000000000..75412e4b86 --- /dev/null +++ b/docs/outstanding-issues-inbox/0a0ab127-0cc1-4b14-b4ce-fd839a98386c.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "0a0ab127-0cc1-4b14-b4ce-fd839a98386c", + "createdOn": "2026-08-21", + "action": "update", + "payload": { + "id": "#231", + "detail": "PHASE 5.2 CONFIRMED SATISFIED with fresh data 2026-08-22 Perth (2026-08-21 UTC), not reopened. This row already recorded that remediation-plan Phase 5.2 is satisfied by S1's 2026-08-17 healthy-latency probes; the Phase 5 close-out re-measured production end to end and confirms it. Retrieval now costs 955 ms on the text fast path and 6,720 ms on hybrid (from 31,610 ms and 21,757 ms at the incident), against answerRouteBudgetMs.fast of 25,000 ms -- so retrieval consumes 4-27% of the fast budget and is no longer capable of binding it. The 2026-08-14 verdict that pre-generation latency WAS the binding cause stands for that window and is now closed out. Residual R4 (chronic ~30 s strong-route provider_timeout on metformin-renal-dosing and valproate-pregnancy, with a safe source-backed extractive fallback) is generation-side and unchanged; no separate R4 row was created, per this row's own instruction. INCONSISTENCY TO FIX AT RECONCILE: the recommended-execution-queue row for #231 still presents it as A1 / 'immediate approved live investigation' with the old framing ('live answers degrade to source-only when answerRouteBudgetMs.fast binds while retrieval is healthy'), which contradicts the P1 -> P2 re-grade recorded in this detail row. The queue entry should be re-graded to match P2 and re-scoped to the R4 generation-side residual, so the queue stops advertising a retrieval investigation that the measurements have closed.", + "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(a) and 5.2; production probes 2026-08-22 Perth (2026-08-21 UTC)", + "baseRowFingerprint": "a2d8deb8060b77d1ed3718818a01b31372c4b30281165dc049102fa616b396e9" + } +} diff --git a/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json b/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json new file mode 100644 index 0000000000..63434e9f0e --- /dev/null +++ b/docs/outstanding-issues-inbox/1c89922d-3d01-4811-a255-ed78e2ed11c3.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "1c89922d-3d01-4811-a255-ed78e2ed11c3", + "createdOn": "2026-08-21", + "action": "cancel", + "payload": { + "requestId": "2040d1fb-6d26-4977-902f-d4a3c2c404c2", + "reason": "Withdrawn and replaced: the request asserted that the v1 siblings profiled in Phase 5 cover the same owning tables as the two unreachable _v2 RPCs. That is false for one of the two. match_document_index_units_hybrid_v2 delegates to match_document_index_units_hybrid_scoped over document_index_units, and none of the four RPCs explain_retrieval_rpc supports touches that table, so the index-units retrieval path has no EXPLAIN baseline at all rather than a partial stand-in. Raised by Codex review of PR #2250 and confirmed against supabase/schema.sql:8033-8054. Superseded by a corrected add request queued in the same commit." + } +} diff --git a/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json b/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json new file mode 100644 index 0000000000..aede97f70c --- /dev/null +++ b/docs/outstanding-issues-inbox/2040d1fb-6d26-4977-902f-d4a3c2c404c2.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "2040d1fb-6d26-4977-902f-d4a3c2c404c2", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P3", + "type": "task", + "summary": "explain_retrieval_rpc cannot profile the _v2 retrieval RPC family, so the two Phase 1.2 outliers have no EXPLAIN baseline", + "detail": "Phase 5 close-out was asked to capture an EXPLAIN baseline for match_document_chunks_text_v2 and match_document_index_units_hybrid_v2 (the two section 1.2 outliers) and could not. public.explain_retrieval_rpc accepts exactly four names -- match_documents_for_query, match_document_chunks_text, match_document_lookup_chunks_text, match_document_table_facts_text -- and raises 22023 Unsupported retrieval RPC for anything else. Verified three ways: its only defining migration 20260626020000_phase7_retrieval_rpc_performance.sql, the supabase/schema.sql live mirror, and a direct call against production returning the 22023 error for both names. Extending the p_rpc branch list is production DDL, and with D4 confirmed ON a migration merged to main deploys automatically, so it needs its own approved window; it was deliberately NOT written during a read-only phase. SECOND, SMALLER GAP recorded at the same time: the function EXPLAINs `select * from public.(...)`, so a PL/pgSQL body's inner plan is never exposed and every sample reports a single Function Scan with no index names. Plan section 5.1's 'record plan flips (seq scan -> index scan)' is therefore not answerable through this instrument at all; pg_stat_user_indexes is the durable substitute and is what Phase 5 used. NEXT: author one migration extending the branch list to the _v2 family and ship it in an approved window. STOP: not urgent -- the v1 siblings profiled in Phase 5 cover the same owning tables.", + "source": "docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(b); supabase/migrations/20260626020000_phase7_retrieval_rpc_performance.sql", + "issueUlid": "01M0JTGY6SJYJEM6MAR4FGVWA8" + } +} diff --git a/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json new file mode 100644 index 0000000000..98809a6867 --- /dev/null +++ b/docs/outstanding-issues-inbox/3c71ce2a-b752-4f1b-a8ef-4870f8027bf7.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "3c71ce2a-b752-4f1b-a8ef-4870f8027bf7", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "issue", + "summary": "Restored trigram indexes have never been scanned, so the 2026-08-14 incident fix is probably mis-attributed to them rather than to ANALYZE", + "detail": "Measured on production 2026-08-22 Perth (2026-08-21 UTC), read-only. pg_stat_database.stats_reset is NULL, so no database-wide reset has been recorded -- note this does NOT prove the counters were never reset, because pg_stat_reset_single_table_counters(oid) resets one table or index without touching that database-level timestamp; 163 of 210 public indexes show scans, max(idx_scan) 262,335,345, total 344,747,815 -- the counters are demonstrably live. Against that, ALL 22 indexes restored or repaired by the incident response report idx_scan = 0: the 20 Phase 4 rebuilds of 2026-08-19, and both trigram indexes restored 2026-08-14 -- document_chunks_content_trgm_idx (68 MB, OID 1491258) and documents_title_trgm_idx (648 kB, OID 1491257). The OID matters: 1491258 is the same value Phase 4 Step 4 recorded as UNCHANGED across the 2026-08-19 push, so the index was never dropped and recreated and its counter has accumulated for the full seven days since restoration. What actually serves retrieval on those tables is document_chunks_search_idx (GIN tsvector, 37,717 scans), document_chunks_embedding_hnsw_idx (HNSW, 4,906) and documents_title_search_idx (37,299). Note that a DIFFERENT trigram index on a sibling table, document_table_facts_title_row_param_trgm_idx, is used 2,372 times, so this is not a blanket property of the planner or of pg_trgm. CONSEQUENCE: forensics Phase 5 attributes the 31,610 -> 1,535 ms recovery to 'restore + ANALYZE', treating the trigram restore as the fix. Zero scans mean those indexes are very unlikely to be carrying that recovery -- unlikely rather than impossible, because a per-relation counter reset would be invisible to the read above; the OID continuity across the 2026-08-19 push is what makes the zero credible, and ANALYZE was an unseparated co-intervention in the same operation -- 20260626020000 itself ends by running analyze over documents, document_chunks, document_table_facts, document_embedding_fields and document_index_units. Stale planner statistics is the better-supported explanation for both the 31-second regression and its recovery -- better-supported, not established. THIS DOES NOT RETRACT THE REPAIR: the two indexes are repo-defined, the migration chain commands them, and drift is correctly closed by their presence. It retracts the ATTRIBUTION, and it changes triage order -- the next 30-second retrieval incident should check planner statistics BEFORE hunting missing indexes. Secondary and non-urgent: roughly 69 MB of currently unscanned index. NEXT: (a) in an approved window, first rule out a per-index counter reset -- on PostgreSQL 18+ read pg_stat_all_indexes.stats_reset for these two OIDs, and on earlier versions fall back to whatever reset or audit record exists (the production server version was not read in this window); (b) separate the two interventions by measuring retrieval latency before and after a fresh ANALYZE; and (c) only then decide whether the two trigram indexes earn their storage. STOP: do NOT drop them -- they are repo-defined and dropping them reopens live drift.", + "source": "production read-only window 2026-08-22 Perth (2026-08-21 UTC); docs/audit/live-drift-forensics-2026-08.md Phase 5 close-out 5.1(c)", + "issueUlid": "01M0JTGYEJ47M1XD0V0EGS5BDD" + } +} diff --git a/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json new file mode 100644 index 0000000000..88b248dab9 --- /dev/null +++ b/docs/outstanding-issues-inbox/98e6ae7f-4190-49f3-a64d-fbf632ef6643.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "98e6ae7f-4190-49f3-a64d-fbf632ef6643", + "createdOn": "2026-08-21", + "action": "done", + "payload": { + "id": "#316", + "outcome": "CLOSED 2026-08-22 Perth (2026-08-21 UTC) -- Phase 5, this row's last remaining follow-on, is complete and the alarm is OBSERVED clear rather than merely believed clear. (1) MEASUREMENT, like-for-like with the section 1.3 probe style: text fast path 31,610 -> 955 ms (33x better than the incident, 1.6x better than the 2026-08-14 post-restore reading); hybrid 21,757 -> 6,720 ms. Nothing was worse than the 2026-08-17 healthy baseline, so nothing was escalated. A new EXPLAIN forward baseline was captured on the healthy system for all four RPCs that explain_retrieval_rpc supports, zero errors across twelve samples; the new slowest path is match_document_table_facts_text (1.72 s unprimed, 1,070 unprimed shared_read). Section 1.3's two pending items are now closed: missing-index sizing is moot (0 missing) and the rag_retrieval_logs miss-scan baseline is recorded (47 seq scans, 6,620 tuples read, 640 live rows, 1.4 MB -- cheap at this scale; the row to watch as it grows is rag_aliases at 5,829 seq scans over 68 rows). (2) 5.2 confirmed ALREADY SATISFIED via #231 and not reopened: retrieval now consumes 4-27% of the 25 s fast-route budget, so it cannot bind it. (3) 5.3 needed NO eval canary -- Phase 3 deployed zero behaviour change -- so it reduced to check:production-readiness: READY, 9 PASS, 0 blocking failures. (4) PRODUCTION RE-VERIFIED UNCHANGED, 8/8 against the 2026-08-21 baseline: 210 public indexes, 0 invalid, 211 history rows, 20 no-statement rows, migration_history_versions() probe ok with 211 versions, search_schema_health() ok true. (5) STAGING PARITY CLOSED: 20260820120000 applied to ikoiolksxqxfxgiyqpnu by real db push (never migration repair); staging now reads 211 rows with the function present and stmt_count 3, matching production exactly. (6) THE ALARM IS OBSERVED CLEAR: live-drift run 32514326022, dispatched on main, is GREEN -- 'No unexpected schema drift between live and supabase/schema.sql', 'Remote migration versions: 211 (read via rpc)', 'Remote-only (Preview blockers): 0', every step success. That is the FIRST green live-drift since 2026-07-19T19:37:29Z (of the last 60 runs only four succeeded), ending a 33-day red streak, and pinned issue #1963 auto-closed at 2026-08-21T18:38:42Z. (7) D4 DOCUMENTATION CORRECTED: this row's 2026-08-19 'auto-deploy is OFF' sentence was already superseded by the 2026-08-21 correction; the coordination board and the forensics dossier are now aligned to the dashboard fact (Deploy to production ENABLED, production branch main, automatic branching ON limit 3), with the superseded safe-either-way reasoning retained and marked rather than deleted. THREE FINDINGS QUEUED SEPARATELY rather than absorbed here: the uncapped Branching Compute cost; the explain_retrieval_rpc gap on the _v2 RPC family (needs production DDL, deliberately not written in a read-only phase); and the discovery that all 22 restored indexes report idx_scan = 0 with no database-wide statistics reset recorded, which makes ANALYZE rather than the trigram restore the better-supported explanation for the incident recovery (a per-relation pg_stat_reset_single_table_counters() call would not show in that read, so this is well-supported, not proven). RESIDUAL WORK LIVES ON ITS OWN ROWS, not here: PITR is still OFF on production, and the fifteen no-statements history rows remain #Q5JHBJ.", + "baseRowFingerprint": "17b4358a583c8bf5d2b7cc77a6ccf0b8eaf714b92a4e7301f3756a9e831b46ad" + } +} diff --git a/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json b/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json new file mode 100644 index 0000000000..49eaceac97 --- /dev/null +++ b/docs/outstanding-issues-inbox/d873ec0d-e25d-41fe-b24c-d2c3aa9375ee.json @@ -0,0 +1,14 @@ +{ + "version": 2, + "id": "d873ec0d-e25d-41fe-b24c-d2c3aa9375ee", + "createdOn": "2026-08-21", + "action": "add", + "payload": { + "pri": "P2", + "type": "rec", + "summary": "Supabase preview-branch compute is an uncapped cost sitting outside the organisation Spend Cap", + "detail": "Dashboard read 2026-08-21 (the same read that settled D4) shows Automatic branching ON with limit 3 and \"Supabase changes only\" enabled, and the same screen warns that Branching Compute is NOT covered by the organisation's Spend Cap. Preview databases are therefore an uncapped cost that scales with the number of open PRs touching supabase/**, on a project that already runs auto-deploy to production. CONTEXT FOR ANY DECISION TO REDUCE OR DISABLE THEM: CI's Migration replay job (db-reset-verify, which runs `supabase migration up --local`) independently replays the entire migration chain on every database-touching PR, so preview branches are a SECOND net rather than the only one -- turning them down would not leave the chain unproven. Preview branches did earn their keep once (the 20260819100200 guard failure on PR #2151 was caught by a preview branch building from the chain alone), so this is a cost/benefit decision, not a cleanup. NEXT: decide whether to keep branching at limit 3, lower the limit, or disable it, and record the decision with its date. STOP: do not change Supabase project settings without explicit owner approval; this row is a decision request, not a licence to reconfigure the project.", + "source": "Supabase dashboard read 2026-08-21; docs/audit/live-drift-forensics-2026-08.md D4 section; AGENTS.md Supabase project safety", + "issueUlid": "01M0JTGY049X40BT8EZTS24YYX" + } +}