Skip to content

Commit e564d8b

Browse files
committed
fix(runtime): carve liveness out of the identity step so a config fault cannot restart a pod
`HttpDispatcher.dispatch()` resolved a per-request identity before any route handler ran, and that step reads the tenancy posture for every request — credentialed or not. A `tenancy` service that is registered and fails to build is re-raised as 503 rather than absorbed into "there is no posture", so an uncredentialed liveness probe was answered 503 for the length of the outage: a liveness 503 means "restart me", the service fails to build again on the new pod, and the restart loop hides the fault the 503 exists to make loud. A route may now declare `liveness: true` on its registry entry. `dispatch()` runs such a route's handler directly — no identity resolution, no gate, nothing that reads configuration or credentials. `/health` declares it; its payload was already process-local. `/ready` is untouched and keeps the full identity step and its 503 body, so traffic is still withheld until the fault is fixed. Which routes count as liveness is DERIVED from the dispatcher's own route table: `DomainHandlerRegistry.resolveLiveness()` is `resolve()` plus one field read, so it answers through the same matcher that picks the handler and cannot drift from the routes that exist. There is no second list of paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YFY46JydE1gMxQG1TqBcMZ
1 parent f48f3f1 commit e564d8b

7 files changed

Lines changed: 433 additions & 3 deletions

File tree

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
"@objectstack/runtime": minor
3+
---
4+
5+
`GET /api/v1/health` answers 200 whenever the process can serve HTTP, even while a configuration fault is making every other route 503.
6+
7+
The dispatcher resolves a per-request identity before any route handler runs, and that step reads the tenancy posture for every request — credentialed or not. Since a `tenancy` service that is registered and fails to build is (correctly) re-raised as a 503 rather than absorbed into "there is no posture", an uncredentialed liveness probe was answered 503 for the length of the outage. A liveness probe reads 503 as *restart me*; the service then fails to build again on the new pod. A restart cannot fix a service that cannot build, so the result was a restart loop that hid the very fault the 503 exists to make loud.
8+
9+
**Liveness is now carved out of the identity step.** `GET /health` runs its handler directly: no identity resolution, no configuration read, no credential read — the payload it answers (`status`, `timestamp`, `version`, `uptime`) was already process-local. Wire it to `livenessProbe`.
10+
11+
**Readiness is unchanged, deliberately.** `GET /ready` keeps the full identity step and its 503 body, so traffic is withheld until the fault is fixed and existing operator dashboards keep the signal they have. Wire it to `readinessProbe`. Nothing else about the 503 moved: every other route, and an environment-scoped `/environments/:id/health`, answers exactly as before.
12+
13+
Which routes count as liveness is **derived from the dispatcher's own route table** rather than listed anywhere: a route declares `liveness: true` on its registry entry, and `DomainHandlerRegistry.resolveLiveness()` answers through the same matcher that picks the handler — so the set cannot drift from the routes that exist. `DomainRoute.liveness` and `resolveLiveness()` are additive public surface on `@objectstack/runtime`; a route that does not declare the flag is untouched.

content/docs/deployment/cli.mdx

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,15 @@ os serve --preset minimal # Skip auto-loaded auth/i18n/ui plugins
225225
- `--preset minimal | default | full` — Override the auto-registration
226226
tier (see below)
227227

228+
**Probes.** A served process exposes `GET /api/v1/health` (liveness — process
229+
only, and deliberately blind to configuration and credentials, so a
230+
configuration fault never restarts the pod) and `GET /api/v1/ready` (readiness —
231+
the full request pipeline, answering `503` while booting, draining, or faulted).
232+
Wire the first to Kubernetes' `livenessProbe` and the second to
233+
`readinessProbe`, never the reverse — the field mapping and the reference
234+
manifest live in
235+
[Health checks & orchestration](/docs/deployment/self-hosting#health-checks--orchestration).
236+
228237
**Tier presets**
229238

230239
`os serve` decides which optional plugins to auto-register from a tier

content/docs/deployment/self-hosting.mdx

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,6 +265,20 @@ questions:
265265
inconclusive — no data engine at all, or the probe itself errors — the replica
266266
stays ready rather than black-holing a working deployment.
267267

268+
**A configuration fault never restarts the pod.** The split above is enforced
269+
one step earlier than the two handlers: `/health` is served *before* the request
270+
pipeline resolves an identity, so it evaluates nothing that depends on
271+
configuration or credentials — a misconfigured or unbuildable security service
272+
leaves it answering `200` for as long as the process can serve HTTP. `/ready`
273+
keeps the whole pipeline and answers `503` for exactly that class of fault, with
274+
the body it has always returned. So wire `path: /api/v1/health` to
275+
`livenessProbe` and `path: /api/v1/ready` to `readinessProbe`**never the
276+
other way round, and never the same path to both**. Crossed over, a
277+
configuration fault makes the orchestrator kill and recreate the pod; the new
278+
one reads the same configuration and fails the same way, and the operator sees
279+
`CrashLoopBackOff` instead of the fault. Withholding traffic is the response
280+
that helps; restarting is not.
281+
268282
### Kubernetes
269283

270284
The same image works unchanged. A minimal reference Deployment — secrets from

packages/runtime/src/domain-handler-registry.ts

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,27 @@ export interface DomainRoute {
6868
match?: 'prefix' | 'exact' | 'segment';
6969
/** Restrict to these UPPERCASE HTTP methods. Omit = all methods. */
7070
methods?: string[];
71+
/**
72+
* This route is a LIVENESS probe: `dispatch()` runs its handler WITHOUT the
73+
* per-request identity step or the gates that follow it.
74+
*
75+
* Declared here, on the route itself, and nowhere else — that is the whole
76+
* point of the field. "Which routes are liveness" is a question with exactly
77+
* one honest source, the table `dispatch()` already routes on, so
78+
* {@link DomainHandlerRegistry.resolveLiveness} answers it through the SAME
79+
* matcher that picks the handler. A separate array of liveness paths would
80+
* be a second list of routes, and this repo has measured what those cost:
81+
* they drift from the thing they describe and the drift is silent.
82+
*
83+
* ⛔ Do not set this on a route whose body reads configuration, credentials
84+
* or any service. A liveness handler may report process-local facts only
85+
* (the process is executing code, the server is listening) — anything else
86+
* puts a configuration fault back on the route whose consumer answers by
87+
* restarting the pod, and a restart cannot fix a service that cannot build.
88+
* Readiness is where a dependency check belongs; its failure mode (leave the
89+
* load-balancer rotation) is the one that helps.
90+
*/
91+
liveness?: boolean;
7192
handler: DomainHandler;
7293
}
7394

@@ -302,6 +323,23 @@ export class DomainHandlerRegistry {
302323
return undefined;
303324
}
304325

326+
/**
327+
* The route claiming `path` (+`method`) when — and only when — it declared
328+
* itself a liveness probe ({@link DomainRoute.liveness}); otherwise
329+
* `undefined`.
330+
*
331+
* DERIVED, not listed: it is {@link resolve} plus one field read, so the
332+
* liveness set is a projection of the live route table and cannot name a
333+
* route that is not registered, miss one that is, or disagree with the
334+
* matcher about which route a path reaches. First-match-wins is inherited
335+
* too — a non-liveness route registered earlier shadows here exactly as it
336+
* shadows in `resolve`, because that is the route the request would get.
337+
*/
338+
resolveLiveness(path: string, method: string): DomainRoute | undefined {
339+
const route = this.resolve(path, method);
340+
return route?.liveness ? route : undefined;
341+
}
342+
305343
private static matches(route: DomainRoute, path: string): boolean {
306344
switch (route.match) {
307345
case 'exact':

0 commit comments

Comments
 (0)