feat(health): add dependency-aware readiness and recovery diagnostics - #140
Open
woahwhattheheck wants to merge 1 commit into
Open
woahwhattheheck wants to merge 1 commit into
woahwhattheheck wants to merge 1 commit into
Conversation
Separate process liveness from traffic readiness. Ready probes now check store, payments (Stellar), and FX under a per-dependency timeout, return redacted reason codes on failure, and re-evaluate every request so recovery does not require a restart. Liveness stays dependency-free so outages do not flap orchestrator restarts. Closes RemitFlow#134
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #134.
Separates process liveness from traffic readiness and adds bounded, redacted dependency probes so a process can no longer report healthy while its store, payment provider, or FX dependency is down.
What changed
GET /api/health/livestays process-only and dependency-free so orchestrators do not restart instances during dependency outages.GET /api/health/readyprobesstore,payments(Stellar), andfxunder a per-check timeout (HEALTH_CHECK_TIMEOUT_MS, default 1000ms).STORE_UNAVAILABLE,PAYMENTS_TIMEOUT, etc.). Raw messages, stacks, and connection material never appear in the payload.ping()hooks onrateServiceandstellarService; shared logic lives independencyHealthService.Design tradeoffs
/api/healthand/api/health/livepayloads are unchanged./api/health/readychecksvalues move from plain"ok"strings to{ status, latencyMs, reason? }objects — callers that only checked top-levelstatuscontinue to work; callers that deep-equalledchecks.store === 'ok'need the new shape (covered in smoke).Acceptance criteria mapping
test/dependencyHealth.test.js— live stays 200 while ready is 503*_TIMEOUTand elapsed ≪ hang delayTest evidence
Out of scope