diff --git a/docs/audit/live-drift-forensics-2026-08.md b/docs/audit/live-drift-forensics-2026-08.md index 1d23a13a18..6ee99dedcb 100644 --- a/docs/audit/live-drift-forensics-2026-08.md +++ b/docs/audit/live-drift-forensics-2026-08.md @@ -833,7 +833,8 @@ Phase 3, and none of it was touched here. _2026-08-18 (repo-side session; owner decisions D1 codify-as-live and D2 canary exemption in force; no production access; two read-only `SELECT`s against staging `ikoiolksxqxfxgiyqpnu` for the (c) -triage only, authorised in-session, `list_projects` verified the ref first). This supersedes the +triage, then — once the §2.5 re-measure merged — the authorised staging apply in 3.5; `list_projects` +verified the ref first and the project id was passed literally on every call). This supersedes the playbook's Phase 3 prompt, which assumed repo-ahead canaries the §1.2 dossier ruled out. **No function body changed.** `#292` open-PR check before starting: no open PR touched `schema.sql`, `supabase/migrations/**`, `drift-manifest.json` or `scripts/check-drift.ts`._ @@ -971,17 +972,105 @@ finding fires but does not name the column; per-column expansion needed the raw failures `tests/session-start-hook.test.ts` / `tests/worker-observability.test.ts` are the only expected red set on this host). -### 3.5 Staging proof — BLOCKED (not run) +### 3.5 Staging proof — RUN for the three migrations; a fourth authored from what it found -The staging apply of `20260818110000`/`111000`/`112000` was gated on the Phase 2 re-measure -(`main` ≥ 195 migrations, `20260818090000` applied to staging). At `main` `4551b6e4d` this file's §2 -carries **no "Re-measure" subsection** and no open PR adds one, so per instruction the staging step -was **not** started: no `execute_sql` DDL, no `schema_migrations` row, no `check:drift` run against -staging in this session. When unblocked, the method is Phase 2's: `execute_sql` running each file -verbatim in one implicit transaction plus an explicit md5-matched history row carrying the repo -version and name (never `apply_migration`), ref verified before every call; expected result: zero -`match_*` function mismatches, zero never-created objects, zero table column-set mismatches, and -exactly one named residual — `document_chunks_content_trgm_idx` (`c3db2960…` vs `8499c3d3…`, 3.3 d). +_The gate opened when the Phase 2 re-measure (§2.5, PR #2104) merged onto `main` (`f19cf8f60`). +Owner-authorised staging window (original Phase 3 authorisation), target `Clinical KB Staging` +`ikoiolksxqxfxgiyqpnu` via the Supabase MCP connector; `list_projects` verified the ref and the +project id was passed literally on every call; production `sjrfecxgysukkwxsowpy` was never a +target. Pre-flight before the first write: `current_user postgres`, `total_rows 195`, +`latest_version 20260818090000`, `no_statements 0`, `documents 0`, `document_chunks 0`, none of the +three versions present._ + +**Apply, by the §2.2 method** (each file's content verbatim through `execute_sql`, then an explicit +`schema_migrations` row carrying the repository version and name in the same call; `apply_migration` +not used). Faithfulness proof read back from staging — all three md5-identical to the repository files: + +``` +version 20260818110000 · name codify_live_rpc_work_mem · stmt_count 1 · bytes 3987 · md5 dd5c8c9ea07c17f76a5f219814f8f19d = repo +version 20260818111000 · name codify_schema_only_indexes_and_triggers · stmt_count 1 · bytes 3470 · md5 9d02d14e14d7ea07bf257e2dc99adaa6 = repo +version 20260818112000 · name reconcile_chain_stale_table_columns · stmt_count 1 · bytes 2631 · md5 ea5f9c6931f85b81613082b4b6fd6a6e = repo +``` + +**Drift comparison.** The local env has no staging service-role key, so instead of `check:drift`'s +network path the comparison was reproduced offline with the same rules: staging returned, per +object, `md5(::text)` for exactly `check-drift.ts`'s `categoryKeys` / +`comparedFields`, and the manifest side was rendered in PostgreSQL jsonb text form and hashed +locally (harness in the session scratchpad; 590 of 594 objects hash-equal, which validates the +renderer). Result against the regenerated manifest (`generated 2026-08-18T08:30:22Z from +schema.sql 87ac9fc4849e…`; staging `snapshot_version 2`, `migration_history_probe ok`, 0 history +rows): + +``` +Compared 6 extensions, 38 tables, 1 views, 93 functions, 210 indexes, 48 policies, 170 constraints, 26 triggers, 2 storage_buckets against staging. +UNEXPECTED DRIFT (4): + ! [functions] mismatch public.match_document_embedding_fields_hybrid(extensions.vector,text,integer,double precision,uuid[],uuid) + ! [functions] mismatch public.match_document_index_units_hybrid(extensions.vector,text,integer,double precision,uuid[],uuid) + ! [functions] mismatch public.match_document_memory_cards_hybrid_v2(extensions.vector,text,integer,double precision,uuid[],uuid) + ! [indexes] mismatch document_chunks_content_trgm_idx +``` + +Against §2.5's 19: the 8 never-created objects are **gone**, the 3 table column-set mismatches are +**gone**, and 4 of the 7 `work_mem` function mismatches are **gone** (`chunks_hybrid`, +`chunks_text`, `lookup_chunks_text`, `memory_cards_hybrid` now hash-equal). The trgm index is the +one residual 3.3 (d) predicted. **Three functions still mismatch, and that is a finding, not +`work_mem`:** their ACLs and `proconfig` (values and order) now equal the manifest exactly, and +`memory_cards_hybrid_v2`'s staging hash `5e792e26…` is unchanged from §2.5 even though its +`work_mem` was already 64MB — so the difference is in the body. `pg_get_functiondef` on staging +shows the decisive hunk for all three: + +```diff +- and public.retrieval_owner_matches(owner_filter, d.owner_id) -- schema.sql = production ++ and (owner_filter is null or d.owner_id = owner_filter) -- staging (chain-built) +``` + +Provenance: `20260712000000_forward_codify_retrieval_owner_matches.sql` recorded that all eight +primary retrieval RPCs on live already gate ownership through the fail-closed, sentinel-aware +`retrieval_owner_matches`, and left the byte-perfect body codification to an owner step; later +migrations (`20260712171500`, `20260714110000`, `20260724120000`) codified five, but +`match_document_embedding_fields_hybrid`, `match_document_index_units_hybrid` and +`match_document_memory_cards_hybrid_v2` were never re-created from live — their newest committed +body is still `20260701140631` with the legacy predicate. §2.3/§2.5 could not see this because the +`work_mem` diff sat on top of it. **Classification: chain-stale body (production and `schema.sql` +agree — 3.1 proved manifest hash = live hash for all three); not a production tenancy hole; a +reproducibility hole for any migrations-only environment.** Per the plan's live-ahead remedy, +`20260818113000_forward_codify_hybrid_owner_matches_bodies.sql` re-creates the three from +`schema.sql` verbatim (every `SET` clause restated so proconfig is preserved; ACLs untouched by +`CREATE OR REPLACE`; identical text on production ⇒ no-op). Its first staging attempt was declined +by the tool-permission classifier; **applied in a second owner-authorised window the same day** +(same method, same target, ref verified): the three hashes now equal the manifest and live — +`bb975485ee3a5776bce4abdc2e3a3cbd`, `d0e277a2f3067f49463b85ac84b33276`, +`ab87a18bea57612db83428c24c425825` — and the history row read back `md5 d35c199b19915505ca86663d0be2bf4d += repo`. + +**Deviation found and resolved.** Before PR #2106 was squash-merged, `20260818111000` and +`20260818112000` gained a `set local lock_timeout = '5s'; set local statement_timeout = '30s';` +preamble on the PR branch, so the text that reached `main` (`22585b9e…`, `ec154770…`) differed from +the text staging had recorded and executed (`9d02d14e…`, `ea5f9c69…`; `20260818110000` unchanged, +`dd5c8c9e…`). The executed DDL is identical (transaction-local timeouts only), so staging's object +state already equalled what `main` produces; in the second window the two rows' `statements` were +refreshed to the merged text so the faithfulness proof holds again. Read-back after the window: + +``` +20260818110000 codify_live_rpc_work_mem stmt_count 1 bytes 3987 md5 dd5c8c9ea07c17f76a5f219814f8f19d = repo +20260818111000 codify_schema_only_indexes_and_triggers stmt_count 1 bytes 3875 md5 22585b9eb81becb6c62d94e597b9e172 = repo +20260818112000 reconcile_chain_stale_table_columns stmt_count 1 bytes 3062 md5 ec154770ddaee44bcb64d3d2fc4f838a = repo +20260818113000 forward_codify_hybrid_owner_matches_bodies stmt_count 1 bytes 12101 md5 d35c199b19915505ca86663d0be2bf4d = repo +``` + +**Final staging comparison (same offline reproduction, 594 objects received):** + +``` +Compared 6 extensions, 38 tables, 1 views, 93 functions, 210 indexes, 48 policies, 170 constraints, 26 triggers, 2 storage_buckets against staging (snapshot_version 2, probe ok, migration_history rows 0). +UNEXPECTED DRIFT (1): + ! [indexes] mismatch document_chunks_content_trgm_idx +``` + +**Zero function mismatches, zero never-created objects, zero table mismatches** — the Phase 3 +target — with the single named residual being the trgm index of 3.3 (d) (staging holds the +`20260606000000` bare-`content` form `c3db2960…`; canonical `8499c3d3…`), owned by Phase 4.4. +Staging after the two windows: 199 rows in `schema_migrations`, `no_statements 0`, `documents` / +`document_chunks` still `0`, no vault secret seeded. ### 3.6 Production window (NOT authorised in this task — for the coordinator) @@ -991,14 +1080,16 @@ statements) — **not** `supabase migration repair --status applied` or any othe path. None of the three Phase 3 migrations ships a validation guard, so per AGENTS.md "Supabase project safety" none of them is eligible for history repair; only real execution is authorised here. -| Migration | Effect on production | -| -------------------------------------------------------- | ---------------------------------------------------------- | -| `20260818090000_schema_drift_snapshot_history_probe` | **real change** — `schema_drift_snapshot()` v2 (Phase 6.1) | -| `20260818110000_codify_live_rpc_work_mem` | no-op — the ten `work_mem` values already match (3.1) | -| `20260818111000_codify_schema_only_indexes_and_triggers` | no-op — all eight objects already exist (3.2) | -| `20260818112000_reconcile_chain_stale_table_columns` | no-op — column and both defaults already as declared (3.3) | +| Migration | Effect on production | +| ----------------------------------------------------------- | ------------------------------------------------------------- | +| `20260818090000_schema_drift_snapshot_history_probe` | **real change** — `schema_drift_snapshot()` v2 (Phase 6.1) | +| `20260818110000_codify_live_rpc_work_mem` | no-op — the ten `work_mem` values already match (3.1) | +| `20260818111000_codify_schema_only_indexes_and_triggers` | no-op — all eight objects already exist (3.2) | +| `20260818112000_reconcile_chain_stale_table_columns` | no-op — column and both defaults already as declared (3.3) | +| `20260818113000_forward_codify_hybrid_owner_matches_bodies` | no-op — identical to the live bodies (3.5; hash proof in 3.1) | -One window covers all four via `db push`; none needs a canary (D2) and none builds an index. +One window covers all five via `db push`; none needs a canary (D2), none builds an index, none +changes a body that production does not already run. ## Phase 4 — Index restoration diff --git a/docs/branch-review-records/374fbb8ab1553ad0ad69733f3456873bf7e30871fdafbd985fd22feaea12279b.record.md b/docs/branch-review-records/374fbb8ab1553ad0ad69733f3456873bf7e30871fdafbd985fd22feaea12279b.record.md new file mode 100644 index 0000000000..e34c55ae8e --- /dev/null +++ b/docs/branch-review-records/374fbb8ab1553ad0ad69733f3456873bf7e30871fdafbd985fd22feaea12279b.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/db-phase3-staging-proof-bodies | 57385d00559ad6d9ec072b01cf97342cb9a2d1cf | db remediation Phase 3 follow-up: staging proof (110000/111000/112000 applied to ikoiolksxqxfxgiyqpnu, md5-verified) + 20260818113000 forward-codify three hybrid RPC bodies verbatim from schema.sql; forensics 3.5; #316 combined update (PR #2111, follows merged #2106) | Reviewed and handed off; staging drift residual after apply = trgm index + three chain-stale bodies, the latter fixed by 20260818113000 (staging apply pending owner permission); no canonical body changed; production window list in PR body | vitest 6 schema/drift files 109 passed; check:migration-role passed; check:outstanding-issues passed; docs:check-links passed; verify:pr-local not re-run for the one-migration follow-up (green on #2106) | diff --git a/docs/branch-review-records/9d9815f1d88e54d6803eca19580348e2e3d9e209af3725527ade39fc5ad9badd.record.md b/docs/branch-review-records/9d9815f1d88e54d6803eca19580348e2e3d9e209af3725527ade39fc5ad9badd.record.md new file mode 100644 index 0000000000..66ef96789a --- /dev/null +++ b/docs/branch-review-records/9d9815f1d88e54d6803eca19580348e2e3d9e209af3725527ade39fc5ad9badd.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/db-phase3-staging-proof-bodies | b5d228ad5ada7bbb82624ab0573894974f3aa232 | db remediation Phase 3 follow-up, final: 20260818113000 applied to staging, 111000/112000 history text refreshed, staging drift = 1 residual (trgm idx); #316 final update (PR #2111) | Reviewed and handed off; staging proof complete — zero function mismatches, zero never-created objects, zero table mismatches, single residual document_chunks_content_trgm_idx (Phase 4.4); production window list unchanged | staging def_hash for the three functions equal manifest/live; four history rows md5 = repo; check:outstanding-issues passed; docs:check-links passed; earlier gates on this branch unchanged | diff --git a/docs/outstanding-issues-inbox/10e480da-b2e3-4e2d-bfc8-15456bf962c1.json b/docs/outstanding-issues-inbox/10e480da-b2e3-4e2d-bfc8-15456bf962c1.json new file mode 100644 index 0000000000..2e47960740 --- /dev/null +++ b/docs/outstanding-issues-inbox/10e480da-b2e3-4e2d-bfc8-15456bf962c1.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "10e480da-b2e3-4e2d-bfc8-15456bf962c1", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#316", + "detail": "PHASE 3 (reframed) BUILT AND STAGING-PROVEN 2026-08-18 (PR #2106) — no production access, no canonical (schema.sql/production) function body changed, no hosted value changed; owner decisions D1 codify-as-live and D2 canary exemption applied. (1) SET work_mem codified on all ten match_* RPCs: schema.sql carries the clause on every definition (both blocks for chunks_hybrid and table_facts_text) and migration 20260818110000_codify_live_rpc_work_mem runs ALTER FUNCTION ... SET work_mem per function, versioned after every recreate incl. 20260724120000. Values: 128MB chunks_hybrid, embedding_fields_hybrid, index_units_hybrid, index_units_hybrid_v2; 64MB chunks_text, chunks_text_v2, lookup_chunks_text, memory_cards_hybrid, memory_cards_hybrid_v2, table_facts_text. PROOF: regenerated drift-manifest def_hash for all ten equals the live production def_hash in issue #1963 (run 32051068106) byte-for-byte, e.g. chunks_hybrid 5902c39286335c07714e498ea31513a0, table_facts_text 0ef9a5dfbde03fe6d48d9223e245aa69 — the next production live-drift run reports zero match_* mismatches once the migration is marked applied. (2) Eight never-created objects codified verbatim by 20260818111000_codify_schema_only_indexes_and_triggers (five document_embedding_fields indexes, documents_status_idx, documents_updated_at and ingestion_jobs_updated_at triggers); disjoint from #102; all six indexes stay on search-health-unmonitored-indexes.json (5 accepted-unmonitored, search_tsv_chunk_gin_idx monitor-candidate), required_indexes untouched — Phase 4.4 decides by migration. (3) Triage: document_chunks CHAIN-stale (token_estimate integer not null default 0 in schema.sql/live/app, zero migrations); rag_visual_eval_cases/runs CHAIN-stale (id default bound to extensions.gen_random_uuid because 20260705230000 sets search_path with pg_catalog last; schema.sql and production bind pg_catalog) — both fixed idempotently by 20260818112000_reconcile_chain_stale_table_columns; document_chunks_content_trgm_idx: production's restored definition (coalesce(content,'') form, md5 8499c3d3...) IS canonical = schema.sql = 20260705180000; staging carries the 20260606000000 bare-content form (c3db2960...) — staging-only residual for Phase 4.4 guard + staging rebuild, no escalation. (4) STAGING PROOF RUN after re-measure PR #2104 merged (owner-authorised window, ref ikoiolksxqxfxgiyqpnu verified per call, production never targeted): 110000/111000/112000 applied by the Phase 2 method with md5-identical history rows (dd5c8c9e.., 9d02d14e.., ea5f9c69..); staging 198 rows, no_statements 0, corpus 0. Offline reproduction of check:drift (same categoryKeys/comparedFields, per-object md5; 590/594 hash-equal): the 8 objects, the 3 column-set mismatches and 4 of the 7 work_mem mismatches are GONE. Residual 4: the trgm index (expected) and THREE FUNCTIONS whose ACL+proconfig equal the manifest but whose BODIES differ — embedding_fields_hybrid, index_units_hybrid, memory_cards_hybrid_v2 carry the legacy predicate (owner_filter is null or d.owner_id = owner_filter) on staging where schema.sql/production carry public.retrieval_owner_matches(owner_filter, d.owner_id). Provenance: 20260712000000_forward_codify_retrieval_owner_matches left the bodies to an owner step; five RPCs were later codified, these three never (newest chain body 20260701140631). NOT a production tenancy hole (manifest hash = live hash for all three); a reproducibility hole for migrations-only environments. New migration 20260818113000_forward_codify_hybrid_owner_matches_bodies re-creates the three verbatim from schema.sql (all SET clauses restated; no-op on production); its staging apply is PENDING owner permission (tool-permission classifier declined the hosted CREATE OR REPLACE of protected RPCs); expected to leave the trgm index as the single staging residual. Tooling note: check-drift.ts:192 clips columns diffs to 240 chars so a wide-table column drift never names the column (own P3 queued). PRODUCTION WINDOW NEEDED (not authorised in this task): 20260818090000 (real change, probe v2) + 20260818110000/111000/112000/113000 (all no-ops, state already matches) — one window, no canary, no index build. Evidence: forensics §Phase 3.", + "source": "session 2026-08-18 Phase 3 repo-side codification + staging proof (PR #2106)", + "baseRowFingerprint": "e03db7a376a6074d0ae25b648139caf82129f019627b20096058dd731a83a8db" + } +} diff --git a/docs/outstanding-issues-inbox/22946f19-6197-408e-b154-142d226a2743.json b/docs/outstanding-issues-inbox/22946f19-6197-408e-b154-142d226a2743.json new file mode 100644 index 0000000000..71ca59ae20 --- /dev/null +++ b/docs/outstanding-issues-inbox/22946f19-6197-408e-b154-142d226a2743.json @@ -0,0 +1,12 @@ +{ + "version": 2, + "id": "22946f19-6197-408e-b154-142d226a2743", + "createdOn": "2026-08-18", + "action": "update", + "payload": { + "id": "#316", + "detail": "PHASE 3 (reframed) COMPLETE REPO-SIDE AND STAGING-PROVEN 2026-08-18 (PRs #2106 merged 72aa18865, #2111 follow-up) — no production access, no canonical (schema.sql/production) function body changed, no hosted value changed; owner decisions D1 codify-as-live and D2 canary exemption applied. (1) SET work_mem codified on all ten match_* RPCs: schema.sql carries the clause on every definition and 20260818110000_codify_live_rpc_work_mem runs ALTER FUNCTION ... SET work_mem per function after every recreate. Values: 128MB chunks_hybrid, embedding_fields_hybrid, index_units_hybrid, index_units_hybrid_v2; 64MB chunks_text, chunks_text_v2, lookup_chunks_text, memory_cards_hybrid, memory_cards_hybrid_v2, table_facts_text. PROOF: regenerated drift-manifest def_hash for all ten equals the live production def_hash in issue #1963 (run 32051068106) byte-for-byte — the next production live-drift run reports zero match_* mismatches once marked applied. (2) Eight never-created objects codified verbatim by 20260818111000 (five document_embedding_fields indexes, documents_status_idx, documents_updated_at + ingestion_jobs_updated_at triggers); disjoint from #102; all six indexes stay on search-health-unmonitored-indexes.json, required_indexes untouched (Phase 4.4). (3) Triage: document_chunks CHAIN-stale (token_estimate, zero migrations); rag_visual_eval_cases/runs CHAIN-stale (id default bound to extensions.gen_random_uuid via 20260705230000 search_path order) — both fixed by 20260818112000; document_chunks_content_trgm_idx: production's restored definition (8499c3d3..) IS canonical = schema.sql = 20260705180000; staging holds the 20260606000000 form (c3db2960..) — Phase 4.4 residual, no escalation. (4) STAGING PROOF COMPLETE (two owner-authorised windows, ref ikoiolksxqxfxgiyqpnu verified per call, production never targeted): 110000/111000/112000 applied by the Phase 2 method; the comparison then exposed THREE chain-stale BODIES (embedding_fields_hybrid, index_units_hybrid, memory_cards_hybrid_v2 carried the legacy fail-open predicate on a chain-built DB — never forward-codified after 20260712000000; NOT a production hole, manifest hash = live hash) — fixed by new migration 20260818113000_forward_codify_hybrid_owner_matches_bodies (verbatim from schema.sql, no-op on production), applied to staging in the second window; the two rows whose text gained set-local timeouts pre-merge (111000/112000) were refreshed to the merged text. All four staging history rows md5 = repo (dd5c8c9e.., 22585b9e.., ec154770.., d35c199b..); staging 199 rows, no_statements 0, corpus 0. FINAL STAGING DRIFT: UNEXPECTED DRIFT (1) = document_chunks_content_trgm_idx only — zero function mismatches, zero never-created objects, zero table mismatches. NEXT: production window (D3, one window, no canary, no index build): 20260818090000 (real change, probe v2) + 110000/111000/112000/113000 (all no-ops, live already matches); then Phase 4 (incl. 4.4 guard migration for the trgm pair + staging trgm rebuild). Tooling note: check-drift.ts:192 240-char clip (own P3 queued). Evidence: forensics §Phase 3.", + "source": "session 2026-08-18 Phase 3 staging proof complete (PR #2111)", + "baseRowFingerprint": "e03db7a376a6074d0ae25b648139caf82129f019627b20096058dd731a83a8db" + } +} diff --git a/docs/outstanding-issues-inbox/23ba3865-258a-4f24-ace4-05e683776dc1.json b/docs/outstanding-issues-inbox/23ba3865-258a-4f24-ace4-05e683776dc1.json new file mode 100644 index 0000000000..b24bfba8f5 --- /dev/null +++ b/docs/outstanding-issues-inbox/23ba3865-258a-4f24-ace4-05e683776dc1.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "23ba3865-258a-4f24-ace4-05e683776dc1", + "createdOn": "2026-08-18", + "action": "cancel", + "payload": { + "requestId": "4dbcdcce-3645-4e08-8813-e85ab1fe7bcc", + "reason": "Superseded within the same PR (#2106) by one combined #316 update that also carries the later staging-proof result; two updates on one id would block reconcile." + } +} diff --git a/docs/outstanding-issues-inbox/8b0650ed-793c-4f05-be9d-2012b6456a72.json b/docs/outstanding-issues-inbox/8b0650ed-793c-4f05-be9d-2012b6456a72.json new file mode 100644 index 0000000000..64d40b98ca --- /dev/null +++ b/docs/outstanding-issues-inbox/8b0650ed-793c-4f05-be9d-2012b6456a72.json @@ -0,0 +1,10 @@ +{ + "version": 2, + "id": "8b0650ed-793c-4f05-be9d-2012b6456a72", + "createdOn": "2026-08-18", + "action": "cancel", + "payload": { + "requestId": "10e480da-b2e3-4e2d-bfc8-15456bf962c1", + "reason": "Superseded within PR #2111 by the final #316 update: 20260818113000 has since been applied to staging and the drift residual is down to one." + } +} diff --git a/supabase/migrations/20260818113000_forward_codify_hybrid_owner_matches_bodies.sql b/supabase/migrations/20260818113000_forward_codify_hybrid_owner_matches_bodies.sql new file mode 100644 index 0000000000..a460bd3194 --- /dev/null +++ b/supabase/migrations/20260818113000_forward_codify_hybrid_owner_matches_bodies.sql @@ -0,0 +1,298 @@ +-- Forward-codify the live bodies of three hybrid retrieval RPCs that the migration +-- chain never received: match_document_embedding_fields_hybrid, +-- match_document_index_units_hybrid, match_document_memory_cards_hybrid_v2. +-- +-- Plan of record: docs/database-remediation-plan.md Phase 3 ("live-ahead: codify the +-- live body into a new migration + schema.sql mirror"); evidence: +-- docs/audit/live-drift-forensics-2026-08.md section Phase 3 (3.5, staging proof). +-- Ledger anchor #316. +-- +-- WHY: 20260712000000_forward_codify_retrieval_owner_matches.sql documented that all +-- eight primary retrieval RPCs on live gate ownership through the fail-closed, +-- sentinel-aware helper public.retrieval_owner_matches(owner_filter, d.owner_id), +-- and left the byte-perfect body codification to an owner step. Later migrations +-- (20260712171500, 20260714110000, 20260724120000) codified five of them, but these +-- three were never re-created from live: their newest committed body is still +-- 20260701140631, which carries the legacy inline predicate +-- `(owner_filter is null or d.owner_id = owner_filter)` (fail-open on NULL, +-- sentinel-blind). Measured on staging after the byte-verified chain replay plus +-- 20260818110000: def_hash still differs from the manifest for exactly these three, +-- and pg_get_functiondef shows the legacy predicate where schema.sql has +-- retrieval_owner_matches. This is NOT a production tenancy hole — production == +-- schema.sql == drift manifest for all three (def_hash equal, forensics 3.1) — it is +-- a reproducibility hole for any environment built from migrations alone (staging, +-- preview branches, `db reset`, DR restore, fresh eval databases). +-- +-- Definitions below are verbatim from supabase/schema.sql (the manifest source), +-- including every SET clause, so CREATE OR REPLACE preserves the live proconfig +-- [search_path, (plan_cache_mode), work_mem]; ACLs are untouched by CREATE OR +-- REPLACE. Idempotent on production (sjrfecxgysukkwxsowpy): identical text, no-op. +-- RAG impact: no retrieval behaviour change — codifying already-live bodies. + +set search_path = public, extensions, pg_temp; + +create or replace function public.match_document_embedding_fields_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 16, + min_similarity double precision default 0.5, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + source_chunk_id uuid, + field_type text, + content text, + similarity double precision, + text_rank double precision, + hybrid_score double precision +) +language sql +stable +set search_path = public, extensions, pg_temp +set plan_cache_mode = 'force_custom_plan' +set work_mem = '128MB' +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq + ), + vector_hits as ( + select f.id + from public.document_embedding_fields f + join public.documents d on d.id = f.document_id + where (document_filters is null or f.document_id = any(document_filters)) + and public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_artifact_generation(f.metadata, d.metadata) + and f.source_chunk_id is not null + and 1 - (f.embedding <=> query_embedding) >= min_similarity + order by f.embedding <=> query_embedding + limit greatest(match_count * 3, 32) + ), + text_hits as ( + select f.id + from public.document_embedding_fields f + join public.documents d on d.id = f.document_id + cross join query + where (document_filters is null or f.document_id = any(document_filters)) + and public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_artifact_generation(f.metadata, d.metadata) + and f.source_chunk_id is not null + and f.search_tsv @@ query.tsq + order by ts_rank_cd(f.search_tsv, query.tsq) desc + limit greatest(match_count * 3, 32) + ), + candidate_ids as ( + select id from vector_hits + union + select id from text_hits + ), + ranked as ( + select + f.id, f.document_id, f.source_chunk_id, f.field_type, f.content, + (1 - (f.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(f.search_tsv, query.tsq)::double precision as text_rank + from public.document_embedding_fields f + join candidate_ids ci on ci.id = f.id + cross join query + ) + select + id, document_id, source_chunk_id, field_type, content, similarity, text_rank, + ((similarity * 0.7) + (least(text_rank, 1) * 0.3))::double precision as hybrid_score + from ranked + order by hybrid_score desc, similarity desc, text_rank desc + limit match_count; +$$; + +create or replace function public.match_document_index_units_hybrid( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 24, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + source_chunk_id uuid, + source_image_id uuid, + unit_type text, + title text, + content text, + page_start integer, + page_end integer, + heading_path text[], + normalized_terms text[], + source_span jsonb, + quality_score real, + extraction_mode text, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + metadata jsonb +) +language sql +stable +set search_path = public, extensions, pg_temp +set plan_cache_mode = 'force_custom_plan' +set work_mem = '128MB' +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq, + regexp_split_to_array(lower(coalesce(query_text, '')), '\s+') as terms + ), + ranked as ( + select u.id, u.document_id, u.source_chunk_id, u.source_image_id, u.unit_type, u.title, u.content, u.page_start, + u.page_end, u.heading_path, u.normalized_terms, u.source_span, u.quality_score, u.extraction_mode, + (1 - (u.embedding <=> query_embedding))::double precision as similarity, + (ts_rank_cd(u.search_tsv, query.tsq) + + case when u.normalized_terms && query.terms then 0.25 else 0 end + + case when u.unit_type in ('askable_question', 'table_fact', 'clinical_fact', 'threshold', 'workflow_step', 'medication_monitoring', 'alias', 'visual_summary', 'flowchart_step', 'diagram_decision', 'risk_matrix_cell', 'medication_chart_row', 'chart_finding', 'visual_askable_question', 'table_threshold') then 0.06 + when u.unit_type = 'section_summary' then 0.03 + else 0 end + )::double precision as text_rank, + u.metadata + from public.document_index_units u + join public.documents d on d.id = u.document_id + cross join query + where d.status = 'indexed' + and (document_filters is null or u.document_id = any(document_filters)) + and public.retrieval_owner_matches(owner_filter, d.owner_id) + and public.is_committed_artifact_generation(u.metadata, d.metadata) + and u.source_chunk_id is not null + and (u.search_tsv @@ query.tsq or u.normalized_terms && query.terms) + order by text_rank desc + limit greatest(match_count * 3, 48) + ) + select id, document_id, source_chunk_id, source_image_id, unit_type, title, content, page_start, page_end, heading_path, + normalized_terms, source_span, quality_score, extraction_mode, similarity, text_rank, + ( + (similarity * 0.52) + + (least(text_rank, 1) * 0.28) + + (quality_score * 0.12) + + (case when extraction_mode in ('model_heavy', 'hybrid') then 0.04 else 0 end) + + (case when unit_type in ('askable_question', 'threshold', 'table_fact', 'table_threshold', 'visual_askable_question') then 0.04 + when unit_type in ('workflow_step', 'medication_monitoring', 'flowchart_step', 'diagram_decision', 'medication_chart_row', 'risk_matrix_cell') then 0.03 + else 0 end) + )::double precision as hybrid_score, + metadata + from ranked + order by hybrid_score desc, similarity desc, text_rank desc + limit match_count; +$$; + +create or replace function public.match_document_memory_cards_hybrid_v2( + query_embedding extensions.vector(1536), + query_text text, + match_count integer default 32, + min_similarity double precision default 0.1, + document_filters uuid[] default null, + owner_filter uuid default null +) +returns table ( + id uuid, + document_id uuid, + owner_id uuid, + section_id uuid, + card_type text, + title text, + content text, + normalized_terms text[], + page_number integer, + source_chunk_ids uuid[], + source_image_ids uuid[], + confidence real, + metadata jsonb, + similarity double precision, + text_rank double precision, + hybrid_score double precision, + rrf_score double precision +) +language sql +stable +set search_path = public, extensions, pg_temp +set work_mem = '64MB' +as $$ + with query as ( + select websearch_to_tsquery('english', coalesce(query_text, '')) as tsq + ), + vector_ranked as ( + select + m.*, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, + row_number() over (order by m.embedding <=> query_embedding) as vector_rank, + null::bigint as text_match_rank + from public.document_memory_cards m + join public.documents d on d.id = m.document_id + cross join query + where (document_filters is null or m.document_id = any(document_filters)) + and public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_artifact_generation(m.metadata, d.metadata) + and (1 - (m.embedding <=> query_embedding)) >= min_similarity + order by m.embedding <=> query_embedding + limit greatest(match_count * 6, 96) + ), + text_ranked as ( + select + m.*, + (1 - (m.embedding <=> query_embedding))::double precision as similarity, + ts_rank_cd(m.search_tsv, query.tsq)::double precision as text_rank, + null::bigint as vector_rank, + row_number() over ( + order by ts_rank_cd(m.search_tsv, query.tsq) desc, m.embedding <=> query_embedding + ) as text_match_rank + from public.document_memory_cards m + join public.documents d on d.id = m.document_id + cross join query + where (document_filters is null or m.document_id = any(document_filters)) + and public.retrieval_owner_matches(owner_filter, d.owner_id) + and d.status = 'indexed' + and public.is_committed_artifact_generation(m.metadata, d.metadata) + and m.search_tsv @@ query.tsq + order by ts_rank_cd(m.search_tsv, query.tsq) desc + limit greatest(match_count * 6, 96) + ), + combined as ( + select * from vector_ranked + union all + select * from text_ranked + ), + scored as ( + select + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, + max(similarity)::double precision as similarity, + max(text_rank)::double precision as text_rank, + min(vector_rank) as vector_rank, + min(text_match_rank) as text_match_rank + from combined + group by + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata + ) + select + id, document_id, owner_id, section_id, card_type, title, content, normalized_terms, + page_number, source_chunk_ids, source_image_ids, confidence, metadata, similarity, text_rank, + ( + (similarity * 0.62) + + (least(text_rank, 1) * 0.24) + + (confidence * 0.10) + + ( + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) + ) * 0.04 + )::double precision as hybrid_score, + ( + coalesce(1.0 / (60 + vector_rank), 0) + + coalesce(1.0 / (60 + text_match_rank), 0) + )::double precision as rrf_score + from scored + order by hybrid_score desc, similarity desc, text_rank desc, confidence desc + limit match_count; +$$;