[jtslear]

Summary

Enable clustering on prometheus.scrape "pods" and prometheus.scrape "cadvisor" in the collector config.

Review of coder/dogfood#702 (v0.8.0 bump), Linear PLE-22 (metrics gaps and Prometheus out-of-order samples): https://github.com/coder/dogfood/pull/702#discussion_r4286742478 . The review noted the cause as likely, from reading the config: https://github.com/coder/dogfood/pull/702#pullrequestreview-5538197905

Why

Alloy runs as a DaemonSet with alloy.clustering.enabled: true, but only pyroscope.scrape has a clustering block. Each Alloy pod therefore scrapes every pod and every node. remote_write drops the instance label, so every pod writes the same series, and Prometheus rejects the copies with out of order sample. This is the same on main and on the previous grafana-agent config.

Test

Using Kind for testing, the cluster has one node, so a DaemonSet gives one peer. I simulated several nodes with --set alloy.controller.type=statefulset --set alloy.controller.replicas=2. Both pods join the cluster (cluster_node_peers = 2).

Totals for both Alloy pods, from prometheus_remote_storage_samples_total and ..._failed_total:

	samples/s sent	failed/s	failed share
main, 2 peers	about 1900	about 600	about 31%
this branch, 2 peers	about 1000	44 to 130	4% to 12%

Target split on this branch (prometheus_target_scrape_pool_targets):

pool	alloy-0	alloy-1
prometheus.scrape.pods	8	7
prometheus.scrape.cadvisor	1	0

Each target now belongs to one peer, and all targets are still scraped (10 pods with up == 1).

Limits

• The failures do not go to zero. I did not find the remaining source. One likely cause is that several ports of one pod give targets with the same labels, because instance is dropped (CRF-18 in the review). I did not test that.
• A single-node kind cluster cannot show a multi-node DaemonSet. The StatefulSet test shows the same clustering mechanism.
• Behaviour change: when an Alloy pod leaves or joins, the targets are redistributed between peers.

Notes

• compiled/resources.yaml regenerated with make build.