Skip to content

Commit b1cc27a

Browse files
committed
Update styling in List and Tables for prometheus-high-cardinality-metrics-label-design article
1 parent a4429d4 commit b1cc27a

2 files changed

Lines changed: 62 additions & 39 deletions

File tree

  • src/content/articles/prometheus-high-cardinality-metrics-label-design

‎src/content/articles/prometheus-high-cardinality-metrics-label-design/index.mdx‎

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -12,26 +12,32 @@ featured: true
1212
*[OOM]: Out of Memory
1313
*[TSDB]: Time Series Database
1414

15-
> Just add a label for debugging.
15+
> Just add a label for debugging
16+
>
17+
> — A senior engineer during sprint planning
1618
1719
Those five words killed our monitoring during the worst possible moment. A team I worked with instrumented their API with response time histograms and added labels for endpoint, status code, and `user_id` — that last one "for debugging." With 50 endpoints, 10 status codes, and 100,000 users, they'd created 50 million potential time series.
1820

1921
Initially, only active users generated metrics. Maybe 10,000 series. Prometheus hummed along. Over months, more users became active. Memory usage crept up until a marketing campaign drove a traffic spike. Memory jumped. Prometheus OOM'd. Monitoring went dark during the incident they needed to debug.
2022

21-
The fix took five minutes: remove `user_id` from labels, add it to traces instead. The lesson took three days of firefighting to learn.
23+
<Highlighter>The fix took five minutes: remove <span class="font-mono">user_id</span> from labels, add it to traces instead.</Highlighter> The lesson took three days of firefighting to learn.
2224

2325
<Callout type="warning">
2426
A single unbounded label can destroy your Prometheus deployment. User IDs, request IDs, email addresses, IP addresses — any label that grows with your data will eventually exhaust memory. Design labels for known, bounded sets of values.
2527
</Callout>
2628

2729
## The Math That Kills Your Prometheus
2830

29-
Every unique combination of metric name and label values creates a separate time series. Labels don't add — they __multiply__.
31+
Every unique combination of metric name and label values creates a separate time series. Labels don't add — they _multiply_.
3032

3133
Consider a basic HTTP metrics setup: 5 HTTP methods × 20 endpoints × 10 status codes = 1,000 series. Manageable. Add a `user_id` label with 100,000 possible values? Now you're looking at 10 billion potential series. The following table shows how quickly labels multiply:
3234

3335
<Table
34-
variant="vertical-column-delineation-table"
36+
variant="grid-and-accent-header-table"
37+
classes={{
38+
tbody:
39+
'!text-content [&_td:nth-child(3)]:text-center [&_td:nth-child(4)]:text-center [&_td:nth-child(5)]:text-center',
40+
}}
3541
content={{
3642
thead: {
3743
th: ['Design', 'Labels', 'Values per Label', 'Total Series', 'Approx. Memory'],
@@ -69,46 +75,51 @@ The difference between a good label and a bad one comes down to one question: __
6975
If you can list them exhaustively, it's probably safe. If the value set grows with your data — users, requests, sessions — it's toxic.
7076

7177
<Table
72-
variant="vertical-column-delineation-table"
78+
variant="numbered-review-table"
79+
classes={{
80+
tbody:
81+
'[&_td:nth-child(4)]:text-center',
82+
}}
7383
content={{
84+
figure: 'Good label examples with bounded cardinality.',
7485
thead: {
7586
th: ['Label', 'Values', 'Cardinality', 'Why It Works'],
7687
},
7788
tbody: {
7889
tr: [
7990
{
80-
th: '<span class="font-mono">http_method`',
91+
th: '<span class="font-mono">http_method</span>',
8192
td: [
82-
'<span class="font-mono">GET, POST, PUT, DELETE, PATCH',
93+
'<span class="font-mono">GET</span>, <span class="font-mono">POST</span>, <span class="font-mono">PUT</span>, <span class="font-mono">DELETE</span>, <span class="font-mono">PATCH</span>',
8394
'~7',
8495
'Fixed set defined by HTTP spec',
8596
],
8697
},
8798
{
88-
th: '<span class="font-mono">environment`',
99+
th: '<span class="font-mono">environment</span>',
89100
td: [
90101
'<span class="font-mono">production</span>, <span class="font-mono">staging</span>, <span class="font-mono">development</span>',
91102
'3',
92103
'You control how many environments exist',
93104
],
94105
},
95106
{
96-
th: '<span class="font-mono">status_class`',
107+
th: '<span class="font-mono">status_class</span>',
97108
td: ['<span class="font-mono">2xx</span>, <span class="font-mono">3xx</span>, <span class="font-mono">4xx</span>, <span class="font-mono">5xx</span>',
98109
'4',
99110
'Bucketed from individual codes',
100111
],
101112
},
102113
{
103-
th: '`region`',
114+
th: '<span class="font-mono">region</span>',
104115
td: [
105116
'<span class="font-mono">us-east-1</span>, <span class="font-mono">us-west-2</span>, <span class="font-mono">eu-west-1</span>',
106117
'~10',
107118
'Bounded by your infrastructure footprint',
108119
],
109120
},
110121
{
111-
th: '`service`',
122+
th: '<span class="font-mono">service</span>',
112123
td: [
113124
'<span class="font-mono">api</span>, <span class="font-mono">worker</span>, <span class="font-mono">scheduler</span>',
114125
'~20',
@@ -117,15 +128,15 @@ If you can list them exhaustively, it's probably safe. If the value set grows wi
117128
},
118129
],
119130
},
120-
figure: 'Good label examples with bounded cardinality.',
121131
}}
122132
/>
123133

124134
Bad labels grow without bound:
125135

126136
<Table
127-
variant="vertical-column-delineation-table"
137+
variant="numbered-review-table"
128138
content={{
139+
figure: 'Bad label examples to avoid.',
129140
thead: {
130141
th: ['Label', 'Problem', 'Cardinality', 'Alternative'],
131142
},
@@ -153,7 +164,6 @@ Bad labels grow without bound:
153164
},
154165
],
155166
},
156-
figure: 'Bad label examples to avoid.',
157167
}}
158168
/>
159169

‎src/content/articles/prometheus-high-cardinality-metrics-label-design/pdf.mdx‎

Lines changed: 38 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,18 @@ import singleUnboundedLabelDiagram from "./diagrams/a-single-unbounded-label-tra
2020
*[UUID]: Universally Unique Identifier
2121
*[WAL]: Write-Ahead Log
2222

23-
"Just add a label for debugging" is the most dangerous sentence in observability.
23+
_"Just add a label for debugging"_ is the most dangerous sentence in observability.
2424

2525
Every unique combination of metric name and label values creates a separate time series in Prometheus. A metric with three labels — each having 100 possible values — creates up to one million time series (100 × 100 × 100). Prometheus stores each time series independently, keeping recent data in memory. Cardinality isn't about the number of metrics you have; it's about the combinatorial explosion of label values.
2626

2727
I watched a team learn this the hard way. They instrumented their API with response time histograms and added labels for endpoint, status code, and `user_id` "for debugging." With 50 endpoints, 10 status codes, and 100,000 users, they'd created 50 million potential time series. Initially, only active users generated metrics — maybe 10,000 series. Prometheus hummed along. Over months, more users became active. Memory usage crept up. Then a marketing campaign drove a traffic spike. Memory usage jumped. Prometheus OOM'd. Monitoring went dark during the incident.
2828

29-
The fix took five minutes: remove `user_id` from the metric labels, add it to traces instead, implement cardinality limits. Prometheus stabilized at 50,000 series. The lesson took three days of firefighting to learn: labels are multiplicative, not additive.
30-
3129
<Callout type="warning">
3230
A single unbounded label can destroy your Prometheus deployment. User IDs, request IDs, email addresses, IP addresses — any label that grows with your data will eventually exhaust memory. Design labels for known, bounded sets of values.
3331
</Callout>
3432

33+
The fix took five minutes: remove `user_id` from the metric labels, add it to traces instead, implement cardinality limits. Prometheus stabilized at 50,000 series. The lesson took three days of firefighting to learn: labels are multiplicative, not additive.
34+
3535
## Understanding Cardinality
3636

3737
### Time Series Math
@@ -53,7 +53,8 @@ Three series. Now multiply: 5 HTTP methods × 20 endpoints × 10 status codes =
5353
The math is straightforward but the implications aren't intuitive. Here's how quickly things escalate:
5454

5555
<Table
56-
variant="vertical-column-delineation-table"
56+
variant="grid-and-accent-header-table"
57+
fullWidth={false}
5758
content={{
5859
thead: {
5960
th: ['Design', 'Labels', 'Values per Label', 'Total Series', 'Approx. Memory'],
@@ -84,6 +85,8 @@ Memory estimation follows a rough formula: each active series consumes 1-3 KB fo
8485

8586
<Diagram
8687
src={singleUnboundedLabelDiagram}
88+
magnifyButtonPosition="top-left"
89+
size="full"
8790
title="A single unbounded label transforms manageable metrics into infrastructure-killing cardinality."
8891
description="Top-to-bottom flowchart showing how metric cardinality multiplies across labels. In the grouped section Label Multiplication, a metric named http_requests is followed by labels method with 4 values, endpoint with 10 values, status with 5 values, and user_id with 100 thousand values. One path stops before user_id and enters the grouped section Without user_id, producing 200 series from 4 times 10 times 5 and an estimated memory cost of about 2 megabytes. A second path continues through the unbounded user_id label into the grouped section With user_id, producing 20 million series from 4 times 10 times 5 times 100 thousand and an estimated memory cost of about 200 gigabytes. The manageable series count and memory estimate are highlighted positively, while the exploded series count and memory estimate are highlighted negatively. The diagram illustrates how adding a single high-cardinality label can turn a safe metric design into one that overwhelms Prometheus memory."
8992
/>
@@ -101,7 +104,7 @@ Queries compound the problem. A query like `sum(rate(http_requests_total[5m])) b
101104
The failure modes are predictable once you understand the architecture:
102105

103106
<Table
104-
variant="vertical-column-delineation-table"
107+
variant="report-table"
105108
content={{
106109
thead: {
107110
th: ['Failure Mode', 'Symptoms', 'Root Cause'],
@@ -146,46 +149,51 @@ The difference between a good label and a bad one comes down to one question: ca
146149
Good labels share four characteristics: bounded cardinality (you know the finite set of values), meaningful for aggregation (you'll actually `group by` or `sum by` this dimension), stable over time (values don't churn constantly), and shared across many series (the label adds structure, not just uniqueness).
147150

148151
<Table
149-
variant="vertical-column-delineation-table"
152+
variant="numbered-review-table"
153+
classes={{
154+
tbody:
155+
'[&_td:nth-child(4)]:text-center',
156+
}}
150157
content={{
158+
figure: 'Good label examples with bounded cardinality.',
151159
thead: {
152160
th: ['Label', 'Values', 'Cardinality', 'Why It Works'],
153161
},
154162
tbody: {
155163
tr: [
156164
{
157-
th: '<span class="font-mono">http_method`',
165+
th: '<span class="font-mono">http_method</span>',
158166
td: [
159-
'<span class="font-mono">GET, POST, PUT, DELETE, PATCH',
167+
'<span class="font-mono">GET</span>, <span class="font-mono">POST</span>, <span class="font-mono">PUT</span>, <span class="font-mono">DELETE</span>, <span class="font-mono">PATCH</span>',
160168
'~7',
161169
'Fixed set defined by HTTP spec',
162170
],
163171
},
164172
{
165-
th: '<span class="font-mono">environment`',
173+
th: '<span class="font-mono">environment</span>',
166174
td: [
167175
'<span class="font-mono">production</span>, <span class="font-mono">staging</span>, <span class="font-mono">development</span>',
168176
'3',
169177
'You control how many environments exist',
170178
],
171179
},
172180
{
173-
th: '<span class="font-mono">status_class`',
181+
th: '<span class="font-mono">status_class</span>',
174182
td: ['<span class="font-mono">2xx</span>, <span class="font-mono">3xx</span>, <span class="font-mono">4xx</span>, <span class="font-mono">5xx</span>',
175183
'4',
176184
'Bucketed from individual codes',
177185
],
178186
},
179187
{
180-
th: '`region`',
188+
th: '<span class="font-mono">region</span>',
181189
td: [
182190
'<span class="font-mono">us-east-1</span>, <span class="font-mono">us-west-2</span>, <span class="font-mono">eu-west-1</span>',
183191
'~10',
184192
'Bounded by your infrastructure footprint',
185193
],
186194
},
187195
{
188-
th: '`service`',
196+
th: '<span class="font-mono">service</span>',
189197
td: [
190198
'<span class="font-mono">api</span>, <span class="font-mono">worker</span>, <span class="font-mono">scheduler</span>',
191199
'~20',
@@ -194,15 +202,15 @@ Good labels share four characteristics: bounded cardinality (you know the finite
194202
},
195203
],
196204
},
197-
figure: 'Good label examples with bounded cardinality.',
198205
}}
199206
/>
200207

201208
Bad labels grow without bound. User IDs scale with your user base. Request IDs are unique per request — literally infinite cardinality. IP addresses span billions of possibilities. Error messages are free-form strings. Any of these as labels will eventually kill your Prometheus instance.
202209

203210
<Table
204-
variant="vertical-column-delineation-table"
211+
variant="numbered-review-table"
205212
content={{
213+
figure: 'Bad label examples to avoid.',
206214
thead: {
207215
th: ['Label', 'Problem', 'Cardinality', 'Alternative'],
208216
},
@@ -234,7 +242,6 @@ Bad labels grow without bound. User IDs scale with your user base. Request IDs a
234242
},
235243
],
236244
},
237-
figure: 'Bad label examples to avoid.',
238245
}}
239246
/>
240247

@@ -480,7 +487,10 @@ The raw-data approach has trade-offs. You're storing higher cardinality in Prome
480487
[^relabel-storage]: This statement requires nuance. If you use `metric_relabel_configs` to transform labels (rather than recording rules), Prometheus applies the relabeling __before__ writing to the TSDB — only the normalized version hits disk. However, the high-cardinality data still creates operational issues during ingestion. For the duration of each scrape, Prometheus must hold the raw, un-normalized data in memory to process the relabeling rules. If thousands of targets send hundreds of unique status codes, this transient data can spike memory and CPU usage during the scrape phase. Complex regex patterns across millions of samples per second are CPU-intensive; if normalization logic is too complex, Prometheus may struggle to keep up, causing scrape gaps. The key distinction: __persistent storage__ sees only normalized data, but __ingestion memory__ briefly sees everything.
481488

482489
<Table
483-
variant="vertical-column-delineation-table"
490+
variant="grid-and-accent-header-table"
491+
classes={{
492+
tbody: '!text-content',
493+
}}
484494
content={{
485495
thead: {
486496
th: ['Approach', 'Pros', 'Cons'],
@@ -722,7 +732,6 @@ Consider HTTP request metrics with handler, method, status_class, and instance l
722732

723733
```yaml title="recording-rules-aggregation.yaml"
724734
# Recording rules to reduce query-time cardinality
725-
726735
groups:
727736
- name: cardinality_reduction
728737
interval: 30s
@@ -839,7 +848,11 @@ Code: Remote write config with cardinality filtering.
839848
This gives you the best of both worlds: full detail locally for troubleshooting recent issues, aggregated data in long-term storage for capacity planning and historical trends. The cost savings on long-term storage can be substantial — you're storing orders of magnitude fewer series.
840849

841850
<Table
842-
variant="vertical-column-delineation-table"
851+
variant="grid-and-accent-header-table"
852+
fullWidth={false}
853+
classes={{
854+
tbody: '!text-content',
855+
}}
843856
content={{
844857
thead: {
845858
th: ['Strategy', 'Use Case', 'Cardinality Impact'],
@@ -988,30 +1001,30 @@ After the crisis, conduct a proper root cause analysis. The questions that matte
9881001
Prevent recurrence with process changes:
9891002

9901003
<List
991-
variant="check-icons-list"
1004+
variant="card-grid-list"
9921005
items={[
9931006
{
994-
lead: 'Code review',
1007+
lead: '1. Code review',
9951008
text: 'Add cardinality review to your PR checklist. Require justification for every new label: "what bounded set of values will this have?"',
9961009
},
9971010
{
998-
lead: 'CI/CD gates',
1011+
lead: '2. CI/CD gates',
9991012
text: 'Lint metric definitions for unbounded patterns. Flag any label that looks like an ID, UUID, email, or IP address.',
10001013
},
10011014
{
1002-
lead: 'Metric unit testing',
1015+
lead: '3. Metric unit testing',
10031016
text: 'Write unit tests that exercise metric instrumentation. If a test generates label combinations in a loop, fail the build when unique series exceed a threshold. This catches cardinality bugs before they reach production.',
10041017
},
10051018
{
1006-
lead: 'Staging validation',
1019+
lead: '4. Staging validation',
10071020
text: 'Deploy to staging first and check cardinality growth before promoting to production.',
10081021
},
10091022
{
1010-
lead: 'Monitoring',
1023+
lead: '5. Monitoring',
10111024
text: 'Alert on series count approaching limits _and_ on growth rate. A 10% hourly increase that goes unnoticed for a day means you\'ve doubled.',
10121025
},
10131026
{
1014-
lead: 'Documentation',
1027+
lead: '6. Documentation',
10151028
text: 'Maintain an approved label schema. Document cardinality budgets per service.',
10161029
},
10171030
]}

0 commit comments

Comments
 (0)