Split the REST edge probe from its dependency check

[HMarzban]

Problem

The two backend services answer the edge probe in opposite ways. Only one of the two is deliberate.

The collaboration service points Traefik at a route that always answers 200 (docker-compose.prod.yml:335). The comment above it says why (:333-334): a dependency probe would pull a replica with live rooms off the pool on a transient database blip. Both replicas share one database (replicas: 2, :359), so such a probe fails on both at once and leaves the edge with no servers.

The REST service does the opposite. Traefik probes /health (docker-compose.prod.yml:233), and that route runs the full dependency check. checkOverallHealth answers 503 whenever the result is not ok (apps/hocuspocus.server/src/api/controllers/health.controller.ts:10), and checkAllServices is not ok when Redis is unhealthy (apps/hocuspocus.server/src/api/services/health.service.ts:104-106). REST also runs replicas: 2 (docker-compose.prod.yml:255).

So a Redis problem takes both REST replicas out of the pool together. Every route behind prodback.docs.plus/api then fails, including the many routes that never touch Redis.

Second defect, same route. The check has no deadline that fits inside the probe.

checkAllServices awaits three checks one after another (health.service.ts:97-99). Only the Supabase check bounds itself, at 2000 ms (:73). The Traefik probe timeout is 3s (docker-compose.prod.yml:235). The Redis check has no bound of its own. The request context holds a live client (apps/hocuspocus.server/src/index.ts:55) whose offline queue is enabled (apps/hocuspocus.server/src/lib/redis.ts:80), so ping() waits for the command timeout instead. That timeout is wired from config (apps/hocuspocus.server/src/lib/redis.ts:77) through REDIS_COMMAND_TIMEOUT (apps/hocuspocus.server/src/config/env.ts:49). Its default is 60000 ms (apps/hocuspocus.server/src/config/env.schema.ts:67), twenty times the probe timeout.

A slow dependency therefore fails the probe on the clock, not on health. The admin dashboard hits the same wall. It reads the same route with a 5000 ms abort (apps/admin-dashboard/src/services/api.ts:203, called from apps/admin-dashboard/src/pages/system.tsx:27).

What to do

Give the REST service the same split the collaboration service already has.

Point the Traefik probe at a liveness route that calls no dependency. Keep the full dependency check on the route the admin dashboard reads.

Bound the dependency check so it can finish inside the probe. Run the three checks together rather than in sequence, and give each one its own deadline.

Acceptance

• The traefik.http.services.rest-api.loadbalancer.healthcheck.path label in docker-compose.prod.yml names a route that calls no dependency.
• apps/hocuspocus.server/src/api/services/health.service.ts runs the three checks concurrently, and each one carries its own deadline.
• With Redis stopped in a local or staging run, a GET on an /api route still answers. GET /health answers in under 3 seconds.

Notes

Every route on the REST health router today calls a dependency (apps/hocuspocus.server/src/api/routers/health.router.ts:8-14), so the liveness route is a new one.

The /health prefix already has its own Traefik router for monitoring (docker-compose.prod.yml:226), and the global rate limiter skips /health and /health/* (apps/hocuspocus.server/src/middleware/index.ts:213). The REST router is mounted on that prefix (apps/hocuspocus.server/src/index.ts:70), so a new liveness route there inherits both. No new label or exemption is needed.

The REST container's own Docker healthcheck also uses /health (docker-compose.prod.yml:242). Decide whether it follows the edge probe or keeps the dependency check.