fix(pg): retry the boot connect, mount in place, and make readiness assert the mount (TASK-168)

[lilyshen0722]

Incident

lily-shen, 20:45Z: a transient Postgres timeout at boot breaks production chat until someone restarts the pod.

backend/server.ts ran connectPG() once at boot. If that attempt returned null — the observed failure was Connection terminated due to connection timeout — the PG routes were never mounted and nothing retried, ever. /api/pg/messages 404'd for the life of the pod, so chat history could not load and socket writes fell through to Mongo. Nothing surfaced it: five placeholder handlers under /api/pg/status each answered {available:false} for their own failure branch, /api/health/ready returned 200 (degraded: ['postgresql']), and the readiness probe gated only Mongo. A rollout therefore replaced the healthy pod with this one and sent it 100% of the traffic (replicaCount: 1). The only remedy was a human restart.

Acceptance 1 + 3 — retry and mount in place (98f20db9)

backend/services/pgBootService.ts, new: a createPgBoot(deps) factory with injected mountRoutes / connect / initialize / onMounted / sleep / schedule / log / state, and module-level pgBootState (mounted, attempts, fellBack, lastError, mountedAt).

• Five attempts with exponential backoff (500ms → 30s cap; PG_BOOT_RETRY_ATTEMPTS, PG_BOOT_RETRY_BASE_DELAY_MS, PG_BOOT_RETRY_MAX_DELAY_MS), then a 30s background interval that keeps trying.
• Routes mount only after connect AND schema init both succeed — a connected pool over an unapplied schema is not a working chat. Idempotent: a second success does not mount twice.
• Never throws: a failing connector lands in pgBootState.lastError and the process stays up.
• server.ts: pgStatusRoutes is required at the top and mounted unconditionally (the five placeholders are gone), and the retention + installation-cleanup crons moved into onMounted, because they need the same guarantee the routes do.

Acceptance 2 — the pod reports the capability it is missing (aa3c1bb3)

The old gate asked whether a live PG query worked. It structurally could not see this defect: config/db-pg.ts creates the pool lazily, so SELECT 1 started succeeding the moment the transient passed while /api/pg/messages stayed unmounted. Per lily's 20:57Z scope note — "readiness must not reuse that block's outcome as its own evidence; assert the mount":

• /api/health/ready returns 503 {status:'not_ready', reason:'PostgreSQL routes are not mounted on this pod'} when PG_HOST is set and the PG router is absent from the app, and 200 the moment the retry mounts it — no restart, which is the half of the incident that needed a human.
• What it deliberately still does not gate: PG liveness after a successful mount. Failing every pod's readiness on a shared-dependency blip turns partial degradation into a full API outage; health.ready.test.js keeps pinning the degraded-200, and values.yaml now states both halves of the policy.
• The probe is routerIsMounted(app, pgMessageRoutes) — a walk of the Express stack for the router object, not pgBootState.mounted / pgAvailable and not a path string. Identity rather than prefix-parsing: a pod whose boot block believes it is fine while the route table disagrees is the 2026-09-25 shape.

A structural assertion, and why it exists. No reachable state in server.test.js separates the flag from the table — no PG, both false; PG working, both true. A rewiring to pgAvailable would keep every behavioural case green while restoring exactly the blindness that caused the outage, so that test also reads server.ts: setPgMountProbe(() => routerIsMounted(app, pgMessageRoutes)) is required, and setPgMountProbe(() => pgAvailable|pgBootState) is refused.

Evidence

check	result
full backend suite (npx jest --forceExit, Node 22)	445 suites / 4158 tests green, 24 skipped, 0 failed
npm run lint:ts	0 errors (1210 pre-existing warnings)
tsc --noEmit	clean; only the pre-existing root-level test-discord-interactions.ts errors
focus	pgBootService 13, server.test.js 9, health.ready 6

Nine mutations across both commits, each anchored, each restored byte-identical with a green baseline after:

AC1/AC3 — connect-once without retry → red; mount despite a failed connect → red; mount before initialize succeeds → red; mount not idempotent → red; backoff cap removed → red; throwing connector escaping start() → red; eager mount on the failed path → red.
AC2 — mount gate deleted → 2 red; gate back on the pool proxy → 2 red; probe wired to pgAvailable → the structural assertion; probe wired to true → the unmounted-route assertion; routerIsMounted answering true for anything → 4 red; mountRoutes not registering the router → 1 red.

One instrument note, since it cost a cycle: the first form of the AC2 probe parsed the mount prefix back out of Express's Layer.regexp.source, and its own regex literal was malformed — the suite reported "Test suite failed to run", which reads like a held guard and is red for the wrong reason. The identity walk has no parsing step to get wrong.

NOT VERIFIED

• Not exercised against a real PG that is down at boot. Every case injects the failure; what is proven is the logic and the wiring, not the end-to-end timing of a real connection timeout.
• The routeRateLimitGuard baseline and the ci(deploy-dev): fail the run when the new backend cannot serve chat #1899 deploy-dev probe are untouched by this PR.

Deliberately not in this PR

Deleting the Mongo chat fallback (pgAvailable in the socket branch at server.ts:607, pgAvailable() in messageController.ts). lily's note asks for it — the Mongo path's newest document is 2026-08-05, so it hides failures while serving stale rows — and the argument is sound. It is a user-visible behaviour change on the chat write path rather than part of "make the pod report the capability it is missing", so it belongs in a review of its own, not inside an incident fix. Flagged on the row for a scope call.

Deploy note

No migration. values.yaml probe config is unchanged — the comment now states both halves of the policy.

---

The kind smoke red this PR caused, and what it turned out to be (853413ae, 35c048f9)

The first push failed Smoke Tests (kind cluster) on the deploy step: backend pod 0/1 Running for the full 300s, every other pod Ready, helm --wait timed out. That is the new gate doing its job and the gate being unable to explain itself, so both halves are fixed here.

The pod was not slow — it was in a state it could never leave. Root cause, in backend/config/db-pg.ts, which this PR now also fixes:

Using CA certificate from: /app/certs/ca.pem
SSL configuration added with CA certificate
PostgreSQL connection error: The server does not support SSL connections

The chart sets PG_SSL_ENABLED (false locally, true in dev/prod) and local-secrets.yaml carries the comment "Empty cert — not used when PG_SSL_ENABLED=false" — but the code never read that variable. The SSL decision was made by the presence of the CA path, and the local chart mounts a zero-byte ca.pem under a real path, so the smoke cluster's backend went into SSL mode against an in-cluster Postgres with no TLS. connectPG() returned null, the PG routes were therefore never mounted, and readiness — correctly — never passed.

This has been true on main for as long as the local chart has existed, and a green smoke never showed it. The old readiness probe ignored PG entirely: the pod served Mongo-fallback chat, returned 200, and the suite passed. So the smoke suite has never proven the PG routes were mounted — the same blindness, in the same place, that made the 2026-09-25 production incident possible. This gate did not create the bug; it is the first instrument to see it.

The fix reads PG_SSL_ENABLED (unset still means ON, so dev and prod are behaviourally identical), and refuses an empty CA file instead of configuring { ca: '' } — which forces TLS and gives Node nothing to verify against, a shape any instance gets if its CA secret materializes empty, not just kind. The decision is now a pure exported function (resolvePgSsl) with 9 cases, rather than a side effect at import time that had drifted from the two comments documenting the intended rule.

Reproduced and fixed against a real Postgres, app run from source, using the smoke cluster's exact shape (PG_SSL_ENABLED=false + a zero-byte CA at a real path):

	before	after
boot	The server does not support SSL connections	PostgreSQL connected: PostgreSQL 17.10
connectPG()	null	pool
PG routes	never mounted	PostgreSQL routes registered for chat functionality
/api/pg/status	{"available":false}	{"available":true}
/api/health/ready	503 not_ready	200 ready

And the smoke job can now explain itself: on failure it prints the backend deployment's last 120 log lines, the backend pod's events, and an in-cluster curl /api/health/ready — instead of Error: context deadline exceeded with a kubectl describe that, on a five-pod install, is entirely postgres. if: failure() only.

Two full-suite reds under node 22 are load flakes, not this change: agentsRuntime.rateLimitTiers ("socket hang up" in the 3,000-request case) and oauthController (TASK-133) both pass in isolation and neither imports db-pg. Measured: 444 suites / 4166 tests passed with those two red; 23/23 in isolation.

[lilyshen0722]

Arrival check against a moved base — main advanced into BOTH files this PR edits. Recorded because a clean auto-merge of two edits to one file is exactly where a semantic break hides.

67d8e252 → f77c18e5, refactor(authz): the socket write path runs the shared membership rule, not a copy (#1943). Files: backend/server.ts (+17/−12 region) and backend/__tests__/unit/server.test.js (64 changed lines) — the same two files #1901 touches.

Conflict: none. git merge-tree --write-tree --name-only origin/main ca0d1f29 → tree a0c4245480a06a6b2a5b77168c21364da1aec7bc, exit 0, no conflicted paths. The two sides edit disjoint regions: #1943 replaces the local isPodMember copy at server.ts:500-509 with isListedPodMember from utils/isPodMember and adds two socket witnesses; #1901's changes are the PG boot block, the readiness mount gate and the server pg boot routes describe.

Semantic check, by grep on the reconstructed tree:

• server.ts:408 still carries fix(pg): retry the boot connect, mount in place, and make readiness assert the mount (TASK-168) #1901's setPgMountProbe(() => routerIsMounted(app, pgMessageRoutes)); :494/:521 carry refactor(authz): the socket write path runs the shared membership rule, not a copy (TASK-165) #1943's isListedPodMember. The module's own isPodMember export is gone.
• server.test.js:222 is the only remaining isPodMember reference and it is isListedPodMember from ../../utils/isPodMember — no stale require('../../server').isPodMember, which is the way this merge would have broken silently.

Reconstructed merge is green: 4 suites / 45 tests, incl. #1901's four boot cases and #1943's two new socket cases (refuses a departed creator on the socket write path, admits a populated member document).

Instrument note. The first run of that command looked red (8 failures, Cannot read properties of undefined (reading 'prototype') in buffer-equal-constant-time/index.js:37 — SlowBuffer, removed in newer Node). Positive control: the same suite at ca0d1f29 without the merge failed identically, so the failure was my shell's default node v26.0.0, not the merge. Re-run under node v22.23.1: 45/45. Anything reproducing these suites needs Node 22 — this is the second time it has read as a code failure here.

Stale-base distance is 28 against the ceiling 40; head unchanged at ca0d1f29, mergeStateStatus: CLEAN. Nothing in this advance touches docs/development/review-checklist.md or docs/assets/readme, so #1885's arrival check and the frame pin set stay closed.