docs: add API monitoring options writeup BDMS-1023 - #772
Conversation
Reference doc surveying production API monitoring options for OcotilloAPI. Force-added because docs/ is still gitignored on staging (the ignore removal rides a separate PR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jirhiker
changed the title
jirhiker
changed the title
…tus page Adds Option 4 (OpenStatus) — a status-page-first, AGPL-3.0 tool with public status pages, subscriber notifications, and monitoring-as-code (YAML/Terraform) that fits this repo's config-as-code habits. Reframes the doc's goal to include a public status page for users and developers, adds a probe-independence note (keep the monitor external to GCP so the page survives platform outages), extends the comparison table, and re-weights the recommendation toward OpenStatus when the public status page is the priority. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ommendation Restructure to lead with the decision: TL;DR, an explicit five-tool shortlist, and a polished recommendation with a concrete OpenStatus rollout plan. Broaden from the original three self-hosted monitors to the full status-page landscape (OpenStatus, Better Stack, Instatus, Gatus, Uptime Kuma, OneUptime, Checkmate, Cachet, Statping-ng, Vigil, UptimeRobot, Atlassian Statuspage, Checkly, incident.io, Blackbox Exporter), reframed around a public status page for users and developers. Add a probe-independence constraint (keep the monitor external to GCP), note that the live /health returns no DB check and should be extended, add a focused comparison table and a when-to-pick-what decision guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chasetmartin
left a comment
chasetmartin
left a comment
There was a problem hiding this comment.
Great start to monitoring, I think the OpenStatus recommendation is well-reasoned and the probe independence requirement is spot-on and having this status page is great.
One note: the SELECT 1 health check is a solid addition, and if all workers are down it'll absolutely catch that. The trickier case might be what we were seeing yesterday where some workers were running hot on memory while others were fine. The health check could land on a healthy instance and come back clean, even though users are getting errors from the struggling ones. I'm guessing this health check would eventually see an issue even in this case, but might be worth also checking a heavier common endpoint or query, like thing(well) details.
For future improvement on this as the app grows within the Bureau and the API usage increases: We could implement a common open source monitoring stack such as Prometheus and Grafana. There is a prometheus-fastapi-instrumentator library, and storing the metrics provided + visualizing with Grafana would give us further insight into error rates across all endpoints, latency, etc. We could then use K6 load testing to formally test for regressions or improvement as needed.
likithabommasani21
left a comment
likithabommasani21
left a comment
There was a problem hiding this comment.
This looks good overall, nice comparison and clear reasoning for the recommendation. One thought: since Ocotillo depends on external services like the auth provider and any other services, it might be worth adding those as their own components on the status page too. They're a separate failure domain from the DB/API, so a green health check wouldn't necessarily catch an issue there , but it's still something users would notice directly (login not working). Just a thought for completeness, not a blocker.
3 participants
Adds
docs/api-monitoring-options.md— a reference survey of production API monitoring options for OcotilloAPI.🤖 Generated with Claude Code