Quantile Aggregation Lab

Eight p95s went in. One fictional p95 came out.

This interactive R 4.6 lab shows why precomputed quantiles cannot be averaged across service instances. It runs one deterministic, traffic-skewed latency population through four telemetry representations:

1. an unweighted average of instance summaries,
2. a traffic-weighted average of instance summaries,
3. merged classic histogram buckets, and
4. a base-2 exponential histogram with a trace exemplar.

At the default workload, api-01 carries 45% of requests and has a 952.2 ms p95. The real service-wide p95 is 883.8 ms, but the average of eight instance p95s reports 199.6 ms and falsely clears a 300 ms SLO. Traffic weighting still misses by 404.6 ms. Compatible classic buckets get the verdict right with 18.2 ms of estimation error. The exponential histogram reports 884.7 ms, 0.9 ms from the deterministic ground truth, and retains a trace exemplar.

Why this lab exists

A quantile is an order statistic: it identifies a rank within one population. It is not additive. Once an instance exports only its p95, the observations below and above that rank are gone. Neither an average nor a request-weighted average can reconstruct the global ordering.

Histograms preserve additive population counts. Merge the buckets first, then calculate the service quantile from the merged distribution.

This lab measures:

• the true and reported service quantile,
• absolute and relative estimation error,
• correct breach, false-clear, and false-alarm decisions,
• whether the representation is mathematically mergeable,
• telemetry series cost,
• per-instance traffic and p50/p95/p99 latency, and
• the trace exemplar retained from the hot tail.

The deterministic statistical model is in R/simulation.R. The JSON encoder and HTTP server use only base R; there are no runtime package dependencies.

Run it

Requires R 4.6.

make run

Open http://127.0.0.1:8080.

Choose another port:

HOST=0.0.0.0 PORT=9000 make run

Print the default model:

make json | jq

With Docker:

docker build -t quantile-aggregation-lab .
docker run --rm -p 8080:8080 quantile-aggregation-lab

The image is pinned to the official multi-architecture r-base:4.6.1 image and runs as a non-root user.

Verify it

make check

The gate parses and style-checks every R file, verifies the deterministic quantile semantics and JSON output, validates JavaScript syntax, and exercises the live HTTP routes, input clamps, assets, methods, security headers, and environment validation. CI separately builds and health-checks the non-root container.

API

GET /api/simulate accepts:

Query parameter	Default	Range
instances	8	2–32
requests_per_second	2,400	100–10,000
minutes	10	1–60
hot_instance_share_percent	45	10–85
hot_slow_percent	18	1–40
baseline_latency_ms	85	20–500
tail_latency_ms	900	baseline + 50–5,000
jitter_percent	18	0–50
target_quantile	95	90–99.9
slo_ms	300	50–2,000
classic_bucket_gap_ms	450	50–1,500
exponential_scale	4	0–8

Example:

curl 'http://127.0.0.1:8080/api/simulate?target_quantile=99&hot_instance_share_percent=60' | jq

Telemetry recipe

Use a mergeable histogram to detect the service tail, attach an exemplar to reach one representative trace, and preserve structured request events for ad-hoc regrouping:

duration_ms
service.instance.id
service.version
http.route
cloud.region
dependency.name
trace.id
metric.exemplar.trace_id

With telemetry.sh, those fields keep the underlying population queryable by the dimensions you discover during an incident instead of reducing every instance to one irreversible percentile.

Model boundaries

This is an educational deterministic model, not a benchmark of a particular metrics backend. It isolates two production semantics:

• precomputed summary quantiles cannot be meaningfully aggregated, while histogram bucket counts can be; and
• fixed classic buckets bound error by bucket width, while base-2 exponential histograms provide mergeable relative resolution across a wide value range.

See Prometheus's current guide to histograms and summaries and the OpenTelemetry metrics data model for the production semantics behind the experiment.

License

MIT