Problem
The HTTP latency histogram stops at five seconds: buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5] (apps/hocuspocus.server/src/lib/metrics.ts:31).
The REST server runs with idleTimeout: 60 (apps/hocuspocus.server/src/index.ts:145). So every request that takes between 5 and 60 seconds lands in the same overflow bucket.
An operator reading a slow-route alert cannot tell a 6-second request from a 55-second one. Both look identical on the dashboard, so nobody can say whether a fix helped.
What to do
Add buckets above 5, up to at least 60. This is one line at apps/hocuspocus.server/src/lib/metrics.ts:31, for example:
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60],
Acceptance
Notes
Two consumers read this histogram. The p95 alert query is at scripts/observability/grafana/provisioning/alerting/rules-app-slo.yml:68. A dashboard panel is at scripts/observability/grafana/provisioning/dashboards/json/app-performance.json:1139.
Both call histogram_quantile. While 5 is the highest finite bucket, that function cannot report a p95 above 5 seconds. Wider buckets fix the alert and the panel at once.
The route that serves the numbers is app.get('/metrics', ...) at apps/hocuspocus.server/src/index.ts:65.
Problem
The HTTP latency histogram stops at five seconds:
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5](apps/hocuspocus.server/src/lib/metrics.ts:31).The REST server runs with
idleTimeout: 60(apps/hocuspocus.server/src/index.ts:145). So every request that takes between 5 and 60 seconds lands in the same overflow bucket.An operator reading a slow-route alert cannot tell a 6-second request from a 55-second one. Both look identical on the dashboard, so nobody can say whether a fix helped.
What to do
Add buckets above 5, up to at least 60. This is one line at
apps/hocuspocus.server/src/lib/metrics.ts:31, for example:buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60],Acceptance
/metricsshowshttp_request_duration_seconds_bucketrows withle="10",le="30"andle="60".le="10"bucket, not only inle="+Inf".Notes
Two consumers read this histogram. The p95 alert query is at
scripts/observability/grafana/provisioning/alerting/rules-app-slo.yml:68. A dashboard panel is atscripts/observability/grafana/provisioning/dashboards/json/app-performance.json:1139.Both call
histogram_quantile. While 5 is the highest finite bucket, that function cannot report a p95 above 5 seconds. Wider buckets fix the alert and the panel at once.The route that serves the numbers is
app.get('/metrics', ...)atapps/hocuspocus.server/src/index.ts:65.