fix(runtime): carve liveness out of the identity step so a config fault cannot restart a pod (#16561)

`HttpDispatcher.dispatch()` resolved a per-request identity before any route
handler ran, and that step reads the tenancy posture for every request —
credentialed or not. A `tenancy` service that is registered and fails to build
is re-raised as 503 rather than absorbed into "there is no posture", so an
uncredentialed liveness probe was answered 503 for the length of the outage: a
liveness 503 means "restart me", the service fails to build again on the new
pod, and the restart loop hides the fault the 503 exists to make loud.

A route may now declare `liveness: true` on its registry entry. `dispatch()`
runs such a route's handler directly — no identity resolution, no gate, nothing
that reads configuration or credentials. `/health` declares it; its payload was
already process-local. `/ready` is untouched and keeps the full identity step
and its 503 body, so traffic is still withheld until the fault is fixed.

Which routes count as liveness is DERIVED from the dispatcher's own route
table: `DomainHandlerRegistry.resolveLiveness()` is `resolve()` plus one field
read, so it answers through the same matcher that picks the handler and cannot
drift from the routes that exist. There is no second list of paths.


Claude-Session: https://claude.ai/code/session_01YFY46JydE1gMxQG1TqBcMZ

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-sales

.changeset/liveness-carved-out-of-the-identity-step.md

@@ -0,0 +1,13 @@
+---
+"@objectstack/runtime": minor
+---
+
+`GET /api/v1/health` answers 200 whenever the process can serve HTTP, even while a configuration fault is making every other route 503.
+
+The dispatcher resolves a per-request identity before any route handler runs, and that step reads the tenancy posture for every request — credentialed or not. Since a `tenancy` service that is registered and fails to build is (correctly) re-raised as a 503 rather than absorbed into "there is no posture", an uncredentialed liveness probe was answered 503 for the length of the outage. A liveness probe reads 503 as *restart me*; the service then fails to build again on the new pod. A restart cannot fix a service that cannot build, so the result was a restart loop that hid the very fault the 503 exists to make loud.
+
+**Liveness is now carved out of the identity step.** `GET /health` runs its handler directly: no identity resolution, no configuration read, no credential read — the payload it answers (`status`, `timestamp`, `version`, `uptime`) was already process-local. Wire it to `livenessProbe`.
+
+**Readiness is unchanged, deliberately.** `GET /ready` keeps the full identity step and its 503 body, so traffic is withheld until the fault is fixed and existing operator dashboards keep the signal they have. Wire it to `readinessProbe`. Nothing else about the 503 moved: every other route, and an environment-scoped `/environments/:id/health`, answers exactly as before.
+
+Which routes count as liveness is **derived from the dispatcher's own route table** rather than listed anywhere: a route declares `liveness: true` on its registry entry, and `DomainHandlerRegistry.resolveLiveness()` answers through the same matcher that picks the handler — so the set cannot drift from the routes that exist. `DomainRoute.liveness` and `resolveLiveness()` are additive public surface on `@objectstack/runtime`; a route that does not declare the flag is untouched.

content/docs/deployment/cli.mdx

@@ -225,6 +225,15 @@ os serve --preset minimal  # Skip auto-loaded auth/i18n/ui plugins
 - `--preset minimal | default | full` — Override the auto-registration
   tier (see below)
 
+**Probes.** A served process exposes `GET /api/v1/health` (liveness — process
+only, and deliberately blind to configuration and credentials, so a
+configuration fault never restarts the pod) and `GET /api/v1/ready` (readiness —
+the full request pipeline, answering `503` while booting, draining, or faulted).
+Wire the first to Kubernetes' `livenessProbe` and the second to
+`readinessProbe`, never the reverse — the field mapping and the reference
+manifest live in
+[Health checks & orchestration](/docs/deployment/self-hosting#health-checks--orchestration).
+
 **Tier presets**
 
 `os serve` decides which optional plugins to auto-register from a tier

content/docs/deployment/self-hosting.mdx

@@ -265,6 +265,20 @@ questions:
   inconclusive — no data engine at all, or the probe itself errors — the replica
   stays ready rather than black-holing a working deployment.
 
+**A configuration fault never restarts the pod.** The split above is enforced
+one step earlier than the two handlers: `/health` is served *before* the request
+pipeline resolves an identity, so it evaluates nothing that depends on
+configuration or credentials — a misconfigured or unbuildable security service
+leaves it answering `200` for as long as the process can serve HTTP. `/ready`
+keeps the whole pipeline and answers `503` for exactly that class of fault, with
+the body it has always returned. So wire `path: /api/v1/health` to
+`livenessProbe` and `path: /api/v1/ready` to `readinessProbe` — **never the
+other way round, and never the same path to both**. Crossed over, a
+configuration fault makes the orchestrator kill and recreate the pod; the new
+one reads the same configuration and fails the same way, and the operator sees
+`CrashLoopBackOff` instead of the fault. Withholding traffic is the response
+that helps; restarting is not.
+
 ### Kubernetes
 
 The same image works unchanged. A minimal reference Deployment — secrets from

packages/runtime/src/domain-handler-registry.ts

@@ -68,6 +68,27 @@ export interface DomainRoute {
     match?: 'prefix' | 'exact' | 'segment';
     /** Restrict to these UPPERCASE HTTP methods. Omit = all methods. */
     methods?: string[];
+    /**
+     * This route is a LIVENESS probe: `dispatch()` runs its handler WITHOUT the
+     * per-request identity step or the gates that follow it.
+     *
+     * Declared here, on the route itself, and nowhere else — that is the whole
+     * point of the field. "Which routes are liveness" is a question with exactly
+     * one honest source, the table `dispatch()` already routes on, so
+     * {@link DomainHandlerRegistry.resolveLiveness} answers it through the SAME
+     * matcher that picks the handler. A separate array of liveness paths would
+     * be a second list of routes, and this repo has measured what those cost:
+     * they drift from the thing they describe and the drift is silent.
+     *
+     * ⛔ Do not set this on a route whose body reads configuration, credentials
+     * or any service. A liveness handler may report process-local facts only
+     * (the process is executing code, the server is listening) — anything else
+     * puts a configuration fault back on the route whose consumer answers by
+     * restarting the pod, and a restart cannot fix a service that cannot build.
+     * Readiness is where a dependency check belongs; its failure mode (leave the
+     * load-balancer rotation) is the one that helps.
+     */
+    liveness?: boolean;
     handler: DomainHandler;
 }
 
@@ -302,6 +323,23 @@ export class DomainHandlerRegistry {
         return undefined;
     }
 
+    /**
+     * The route claiming `path` (+`method`) when — and only when — it declared
+     * itself a liveness probe ({@link DomainRoute.liveness}); otherwise
+     * `undefined`.
+     *
+     * DERIVED, not listed: it is {@link resolve} plus one field read, so the
+     * liveness set is a projection of the live route table and cannot name a
+     * route that is not registered, miss one that is, or disagree with the
+     * matcher about which route a path reaches. First-match-wins is inherited
+     * too — a non-liveness route registered earlier shadows here exactly as it
+     * shadows in `resolve`, because that is the route the request would get.
+     */
+    resolveLiveness(path: string, method: string): DomainRoute | undefined {
+        const route = this.resolve(path, method);
+        return route?.liveness ? route : undefined;
+    }
+
     private static matches(route: DomainRoute, path: string): boolean {
         switch (route.match) {
             case 'exact':