Skip to content

Widen the HTTP latency buckets past the server idle timeout #219

Description

@HMarzban

Problem

The HTTP latency histogram stops at five seconds: buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5] (apps/hocuspocus.server/src/lib/metrics.ts:31).

The REST server runs with idleTimeout: 60 (apps/hocuspocus.server/src/index.ts:145). So every request that takes between 5 and 60 seconds lands in the same overflow bucket.

An operator reading a slow-route alert cannot tell a 6-second request from a 55-second one. Both look identical on the dashboard, so nobody can say whether a fix helped.

What to do

Add buckets above 5, up to at least 60. This is one line at apps/hocuspocus.server/src/lib/metrics.ts:31, for example:

buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60],

Acceptance

  • /metrics shows http_request_duration_seconds_bucket rows with le="10", le="30" and le="60".
  • A request that takes about 6 seconds is counted in the le="10" bucket, not only in le="+Inf".

Notes

Two consumers read this histogram. The p95 alert query is at scripts/observability/grafana/provisioning/alerting/rules-app-slo.yml:68. A dashboard panel is at scripts/observability/grafana/provisioning/dashboards/json/app-performance.json:1139.

Both call histogram_quantile. While 5 is the highest finite bucket, that function cannot report a p95 above 5 seconds. Wider buckets fix the alert and the panel at once.

The route that serves the numbers is app.get('/metrics', ...) at apps/hocuspocus.server/src/index.ts:65.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions