You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b1cc27a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/prometheus-high-cardinality-metrics-label-design/index.mdx
+24-14Lines changed: 24 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,26 +12,32 @@ featured: true
12
12
*[OOM]: Out of Memory
13
13
*[TSDB]: Time Series Database
14
14
15
-
> Just add a label for debugging.
15
+
> Just add a label for debugging
16
+
>
17
+
> — A senior engineer during sprint planning
16
18
17
19
Those five words killed our monitoring during the worst possible moment. A team I worked with instrumented their API with response time histograms and added labels for endpoint, status code, and `user_id` — that last one "for debugging." With 50 endpoints, 10 status codes, and 100,000 users, they'd created 50 million potential time series.
18
20
19
21
Initially, only active users generated metrics. Maybe 10,000 series. Prometheus hummed along. Over months, more users became active. Memory usage crept up until a marketing campaign drove a traffic spike. Memory jumped. Prometheus OOM'd. Monitoring went dark during the incident they needed to debug.
20
22
21
-
The fix took five minutes: remove `user_id` from labels, add it to traces instead. The lesson took three days of firefighting to learn.
23
+
<Highlighter>The fix took five minutes: remove <spanclass="font-mono">user_id</span> from labels, add it to traces instead.</Highlighter> The lesson took three days of firefighting to learn.
22
24
23
25
<Callouttype="warning">
24
26
A single unbounded label can destroy your Prometheus deployment. User IDs, request IDs, email addresses, IP addresses — any label that grows with your data will eventually exhaust memory. Design labels for known, bounded sets of values.
25
27
</Callout>
26
28
27
29
## The Math That Kills Your Prometheus
28
30
29
-
Every unique combination of metric name and label values creates a separate time series. Labels don't add — they __multiply__.
31
+
Every unique combination of metric name and label values creates a separate time series. Labels don't add — they _multiply_.
30
32
31
33
Consider a basic HTTP metrics setup: 5 HTTP methods × 20 endpoints × 10 status codes = 1,000 series. Manageable. Add a `user_id` label with 100,000 possible values? Now you're looking at 10 billion potential series. The following table shows how quickly labels multiply:
Copy file name to clipboardExpand all lines: src/content/articles/prometheus-high-cardinality-metrics-label-design/pdf.mdx
+38-25Lines changed: 38 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,18 +20,18 @@ import singleUnboundedLabelDiagram from "./diagrams/a-single-unbounded-label-tra
20
20
*[UUID]: Universally Unique Identifier
21
21
*[WAL]: Write-Ahead Log
22
22
23
-
"Just add a label for debugging" is the most dangerous sentence in observability.
23
+
_"Just add a label for debugging"_ is the most dangerous sentence in observability.
24
24
25
25
Every unique combination of metric name and label values creates a separate time series in Prometheus. A metric with three labels — each having 100 possible values — creates up to one million time series (100 × 100 × 100). Prometheus stores each time series independently, keeping recent data in memory. Cardinality isn't about the number of metrics you have; it's about the combinatorial explosion of label values.
26
26
27
27
I watched a team learn this the hard way. They instrumented their API with response time histograms and added labels for endpoint, status code, and `user_id` "for debugging." With 50 endpoints, 10 status codes, and 100,000 users, they'd created 50 million potential time series. Initially, only active users generated metrics — maybe 10,000 series. Prometheus hummed along. Over months, more users became active. Memory usage crept up. Then a marketing campaign drove a traffic spike. Memory usage jumped. Prometheus OOM'd. Monitoring went dark during the incident.
28
28
29
-
The fix took five minutes: remove `user_id` from the metric labels, add it to traces instead, implement cardinality limits. Prometheus stabilized at 50,000 series. The lesson took three days of firefighting to learn: labels are multiplicative, not additive.
30
-
31
29
<Callouttype="warning">
32
30
A single unbounded label can destroy your Prometheus deployment. User IDs, request IDs, email addresses, IP addresses — any label that grows with your data will eventually exhaust memory. Design labels for known, bounded sets of values.
33
31
</Callout>
34
32
33
+
The fix took five minutes: remove `user_id` from the metric labels, add it to traces instead, implement cardinality limits. Prometheus stabilized at 50,000 series. The lesson took three days of firefighting to learn: labels are multiplicative, not additive.
34
+
35
35
## Understanding Cardinality
36
36
37
37
### Time Series Math
@@ -53,7 +53,8 @@ Three series. Now multiply: 5 HTTP methods × 20 endpoints × 10 status codes =
53
53
The math is straightforward but the implications aren't intuitive. Here's how quickly things escalate:
54
54
55
55
<Table
56
-
variant="vertical-column-delineation-table"
56
+
variant="grid-and-accent-header-table"
57
+
fullWidth={false}
57
58
content={{
58
59
thead: {
59
60
th: ['Design', 'Labels', 'Values per Label', 'Total Series', 'Approx. Memory'],
@@ -84,6 +85,8 @@ Memory estimation follows a rough formula: each active series consumes 1-3 KB fo
84
85
85
86
<Diagram
86
87
src={singleUnboundedLabelDiagram}
88
+
magnifyButtonPosition="top-left"
89
+
size="full"
87
90
title="A single unbounded label transforms manageable metrics into infrastructure-killing cardinality."
88
91
description="Top-to-bottom flowchart showing how metric cardinality multiplies across labels. In the grouped section Label Multiplication, a metric named http_requests is followed by labels method with 4 values, endpoint with 10 values, status with 5 values, and user_id with 100 thousand values. One path stops before user_id and enters the grouped section Without user_id, producing 200 series from 4 times 10 times 5 and an estimated memory cost of about 2 megabytes. A second path continues through the unbounded user_id label into the grouped section With user_id, producing 20 million series from 4 times 10 times 5 times 100 thousand and an estimated memory cost of about 200 gigabytes. The manageable series count and memory estimate are highlighted positively, while the exploded series count and memory estimate are highlighted negatively. The diagram illustrates how adding a single high-cardinality label can turn a safe metric design into one that overwhelms Prometheus memory."
89
92
/>
@@ -101,7 +104,7 @@ Queries compound the problem. A query like `sum(rate(http_requests_total[5m])) b
101
104
The failure modes are predictable once you understand the architecture:
102
105
103
106
<Table
104
-
variant="vertical-column-delineation-table"
107
+
variant="report-table"
105
108
content={{
106
109
thead: {
107
110
th: ['Failure Mode', 'Symptoms', 'Root Cause'],
@@ -146,46 +149,51 @@ The difference between a good label and a bad one comes down to one question: ca
146
149
Good labels share four characteristics: bounded cardinality (you know the finite set of values), meaningful for aggregation (you'll actually `group by` or `sum by` this dimension), stable over time (values don't churn constantly), and shared across many series (the label adds structure, not just uniqueness).
147
150
148
151
<Table
149
-
variant="vertical-column-delineation-table"
152
+
variant="numbered-review-table"
153
+
classes={{
154
+
tbody:
155
+
'[&_td:nth-child(4)]:text-center',
156
+
}}
150
157
content={{
158
+
figure: 'Good label examples with bounded cardinality.',
151
159
thead: {
152
160
th: ['Label', 'Values', 'Cardinality', 'Why It Works'],
153
161
},
154
162
tbody: {
155
163
tr: [
156
164
{
157
-
th: '<span class="font-mono">http_method`',
165
+
th: '<span class="font-mono">http_method</span>',
158
166
td: [
159
-
'<span class="font-mono">GET, POST, PUT, DELETE, PATCH',
@@ -194,15 +202,15 @@ Good labels share four characteristics: bounded cardinality (you know the finite
194
202
},
195
203
],
196
204
},
197
-
figure: 'Good label examples with bounded cardinality.',
198
205
}}
199
206
/>
200
207
201
208
Bad labels grow without bound. User IDs scale with your user base. Request IDs are unique per request — literally infinite cardinality. IP addresses span billions of possibilities. Error messages are free-form strings. Any of these as labels will eventually kill your Prometheus instance.
@@ -234,7 +242,6 @@ Bad labels grow without bound. User IDs scale with your user base. Request IDs a
234
242
},
235
243
],
236
244
},
237
-
figure: 'Bad label examples to avoid.',
238
245
}}
239
246
/>
240
247
@@ -480,7 +487,10 @@ The raw-data approach has trade-offs. You're storing higher cardinality in Prome
480
487
[^relabel-storage]: This statement requires nuance. If you use `metric_relabel_configs` to transform labels (rather than recording rules), Prometheus applies the relabeling __before__ writing to the TSDB — only the normalized version hits disk. However, the high-cardinality data still creates operational issues during ingestion. For the duration of each scrape, Prometheus must hold the raw, un-normalized data in memory to process the relabeling rules. If thousands of targets send hundreds of unique status codes, this transient data can spike memory and CPU usage during the scrape phase. Complex regex patterns across millions of samples per second are CPU-intensive; if normalization logic is too complex, Prometheus may struggle to keep up, causing scrape gaps. The key distinction: __persistent storage__ sees only normalized data, but __ingestion memory__ briefly sees everything.
481
488
482
489
<Table
483
-
variant="vertical-column-delineation-table"
490
+
variant="grid-and-accent-header-table"
491
+
classes={{
492
+
tbody: '!text-content',
493
+
}}
484
494
content={{
485
495
thead: {
486
496
th: ['Approach', 'Pros', 'Cons'],
@@ -722,7 +732,6 @@ Consider HTTP request metrics with handler, method, status_class, and instance l
722
732
723
733
```yaml title="recording-rules-aggregation.yaml"
724
734
# Recording rules to reduce query-time cardinality
This gives you the best of both worlds: full detail locally for troubleshooting recent issues, aggregated data in long-term storage for capacity planning and historical trends. The cost savings on long-term storage can be substantial — you're storing orders of magnitude fewer series.
@@ -988,30 +1001,30 @@ After the crisis, conduct a proper root cause analysis. The questions that matte
988
1001
Prevent recurrence with process changes:
989
1002
990
1003
<List
991
-
variant="check-icons-list"
1004
+
variant="card-grid-list"
992
1005
items={[
993
1006
{
994
-
lead: 'Code review',
1007
+
lead: '1. Code review',
995
1008
text: 'Add cardinality review to your PR checklist. Require justification for every new label: "what bounded set of values will this have?"',
996
1009
},
997
1010
{
998
-
lead: 'CI/CD gates',
1011
+
lead: '2. CI/CD gates',
999
1012
text: 'Lint metric definitions for unbounded patterns. Flag any label that looks like an ID, UUID, email, or IP address.',
1000
1013
},
1001
1014
{
1002
-
lead: 'Metric unit testing',
1015
+
lead: '3. Metric unit testing',
1003
1016
text: 'Write unit tests that exercise metric instrumentation. If a test generates label combinations in a loop, fail the build when unique series exceed a threshold. This catches cardinality bugs before they reach production.',
1004
1017
},
1005
1018
{
1006
-
lead: 'Staging validation',
1019
+
lead: '4. Staging validation',
1007
1020
text: 'Deploy to staging first and check cardinality growth before promoting to production.',
1008
1021
},
1009
1022
{
1010
-
lead: 'Monitoring',
1023
+
lead: '5. Monitoring',
1011
1024
text: 'Alert on series count approaching limits _and_ on growth rate. A 10% hourly increase that goes unnoticed for a day means you\'ve doubled.',
1012
1025
},
1013
1026
{
1014
-
lead: 'Documentation',
1027
+
lead: '6. Documentation',
1015
1028
text: 'Maintain an approved label schema. Document cardinality budgets per service.',
0 commit comments