You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ac98329
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/alert-fatigue-reduction-triage-actionable-alerts/download.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ pages: 24
14
14
15
15
On-call rotations with high alert volume drain engineers and slow incident response. An audit found one team managing 47 alerts per 24-hour shift—44 required no action, while the real issue went unnoticed in the noise.
16
16
17
-
This complete guide walks you through the systematic process used to reduce a 200-alert-per-week system to 35 actionable alerts, with a 90% action rate—transforming on-call from a burnout machine into a sustainable feedback loop.
17
+
This complete guide walks you through the systematic process used to reduce a 200-alert-per-week system to 35 actionable alerts, with a 90% action rate — transforming on-call from a burnout machine into a sustainable feedback loop.
Copy file name to clipboardExpand all lines: src/content/articles/alert-fatigue-reduction-triage-actionable-alerts/index.mdx
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,21 +17,21 @@ featured: true
17
17
18
18
I inherited a monitoring setup where the on-call engineer averaged 47 alerts per day. Forty-four of those required no action — thresholds set too aggressively, alerts for non-problems, duplicate notifications for the same underlying issue. The forty-fifth was a memory leak that had been slowly building for three hours. By the time anyone noticed it in the noise, the service had already crashed and restarted twice.
19
19
20
-
That's the core problem: comprehensive monitoring and actionable alerting are often at odds. Teams add alerts because something _might_ go wrong, and removing an alert feels like removing a safety net. But the math works against you—an engineer who has acknowledged thirty false positives is not in the right headspace to notice the thirty-first is real.
20
+
That's the core problem: comprehensive monitoring and actionable alerting are often at odds. Teams add alerts because something _might_ go wrong, and removing an alert feels like removing a safety net. But the math works against you — an engineer who has acknowledged thirty false positives is not in the right headspace to notice the thirty-first is real.
21
21
22
-
<Callouttype="warning">Every alert you keep dilutes the ones that matter. An alert that fires but requires no action is worse than no alert at all—it trains engineers to ignore pages.</Callout>
22
+
<Callouttype="warning">Every alert you keep dilutes the ones that matter. An alert that fires but requires no action is worse than no alert at all — it trains engineers to ignore pages.</Callout>
23
23
24
24
This article walks through the audit process that took us from 47 alerts per day to about 5 meaningful pages, with an action rate above 90%.
25
25
26
26
## Why Alert Noise Compounds
27
27
28
-
The most visible symptom of alert fatigue is MTTA drift—mean time to acknowledge. When engineers expect noise, acknowledgment slows. I've watched MTTA drift from under two minutes to over fifteen as alert volume increased, because the on-call engineer stopped keeping their phone nearby.
28
+
The most visible symptom of alert fatigue is MTTA drift — mean time to acknowledge. When engineers expect noise, acknowledgment slows. I've watched MTTA drift from under two minutes to over fifteen as alert volume increased, because the on-call engineer stopped keeping their phone nearby.
29
29
30
-
There's solid research on this from healthcare, where alarm fatigue kills patients. Studies in ICUs found that 72-99% of alarms required no clinical intervention[^1], and staff developed coping mechanisms—turning down volume, disabling non-critical alarms, or simply not rushing to respond. Software operations is no different.
30
+
There's solid research on this from healthcare, where alarm fatigue kills patients. Studies in ICUs found that 72-99% of alarms required no clinical intervention[^1], and staff developed coping mechanisms — turning down volume, disabling non-critical alarms, or simply not rushing to respond. Software operations is no different.
31
31
32
32
[^1]: Sendelbach S, Funk M. "Alarm fatigue: a patient safety concern." AACN Adv Crit Care. 2013 Oct-Dec: 378-86.
33
33
34
-
The feedback loop makes it worse. After an incident caused by a missed alert, the instinct is to add more alerts. But if the alert was missed because of volume, adding more alerts makes the next miss more likely, not less. You can't fix what you haven't measured—and most teams have never actually measured their alert usefulness.
34
+
The feedback loop makes it worse. After an incident caused by a missed alert, the instinct is to add more alerts. But if the alert was missed because of volume, adding more alerts makes the next miss more likely, not less. You can't fix what you haven't measured — and most teams have never actually measured their alert usefulness.
35
35
36
36
## The Three-Step Audit
37
37
@@ -43,7 +43,7 @@ Export your alert definitions and firing history. Prometheus/Alertmanager stores
43
43
44
44
<Newsletter />
45
45
46
-
For each alert, capture: alert name, firing frequency over the last 90 days, action rate (percentage of firings that required human intervention), owning team, and whether a runbook exists. The action rate column is the most important and the hardest to populate—you'll need to cross-reference alert firings with incident tickets or on-call logs.
46
+
For each alert, capture: alert name, firing frequency over the last 90 days, action rate (percentage of firings that required human intervention), owning team, and whether a runbook exists. The action rate column is the most important and the hardest to populate — you'll need to cross-reference alert firings with incident tickets or on-call logs.
47
47
48
48
### Step 2: Classify by Action Rate
49
49
@@ -95,13 +95,13 @@ I use four buckets:
95
95
}}
96
96
/>
97
97
98
-
The "noisy" bucket is where most of the work happens. These alerts fire for real issues, but the threshold is wrong or multiple alerts fire for the same underlying problem. Fixing them requires understanding _why_ action wasn't taken—was the alert a false positive, or did the problem resolve itself before anyone could respond?
98
+
The "noisy" bucket is where most of the work happens. These alerts fire for real issues, but the threshold is wrong or multiple alerts fire for the same underlying problem. Fixing them requires understanding _why_ action wasn't taken — was the alert a false positive, or did the problem resolve itself before anyone could respond?
99
99
100
100
### Step 3: Delete the Noise
101
101
102
102
Deleting alerts is politically difficult. Someone created that alert for a reason. Maybe there was an incident, and the alert was the action item from the postmortem. Deleting it feels like ignoring the lessons learned.
103
103
104
-
But an alert that fires constantly and gets ignored isn't a lesson learned—it's a lesson forgotten. The incident that created it is no longer prevented by the alert; the alert just adds to the noise that makes the _next_ incident harder to catch.
104
+
But an alert that fires constantly and gets ignored isn't a lesson learned — it's a lesson forgotten. The incident that created it is no longer prevented by the alert; the alert just adds to the noise that makes the _next_ incident harder to catch.
105
105
106
106
The talking points that work:
107
107
@@ -127,19 +127,19 @@ The talking points that work:
127
127
128
128
## One Key Technique: Alert on Symptoms
129
129
130
-
Most alert configurations work backwards. They alert on causes—CPU spikes, memory pressure, disk I/O, connection pool exhaustion—hoping to catch problems before users notice. The result is dozens of alerts for a single incident, each describing a different aspect of the same failure.
130
+
Most alert configurations work backwards. They alert on causes — CPU spikes, memory pressure, disk I/O, connection pool exhaustion — hoping to catch problems before users notice. The result is dozens of alerts for a single incident, each describing a different aspect of the same failure.
131
131
132
132
Flip this around. <Highlighter>Alert on symptoms instead: what users actually experience.</Highlighter> A single "checkout latency exceeds SLO" alert replaces the CPU alert, the database alert, the cache alert, and the memory alert. They're all symptoms of the same underlying problem, and the responder doesn't need to be told about each one separately.
133
133
134
-
Cause-based alerts still have a place—as diagnostic information in dashboards and runbooks, not as paging conditions. When the symptom alert fires, the responder can check the cause metrics to understand _why_. But the page itself should describe the user impact, not the infrastructure state.
134
+
Cause-based alerts still have a place — as diagnostic information in dashboards and runbooks, not as paging conditions. When the symptom alert fires, the responder can check the cause metrics to understand _why_. But the page itself should describe the user impact, not the infrastructure state.
135
135
136
-
This principle extends to alert grouping and dependency suppression. If the database is down, you don't need six "connection refused" alerts from downstream services—you need one "database down" alert. Map your dependencies and configure your alerting system to suppress the noise.
136
+
This principle extends to alert grouping and dependency suppression. If the database is down, you don't need six "connection refused" alerts from downstream services — you need one "database down" alert. Map your dependencies and configure your alerting system to suppress the noise.
137
137
138
138
## Maintaining the Gains
139
139
140
140
Alert counts grow naturally. Someone adds an alert during an incident, another gets added during a deployment, a third comes from a vendor integration. Rarely does anyone go back and remove alerts. Without discipline, the count ratchets upward.
141
141
142
-
Establish a cadence: monthly reviews of which alerts fired most and what their action rates were, quarterly reviews of whether alerts still make sense given how your system has evolved. Every alert needs an owner—encode it in a label. Orphaned alerts are the first candidates for deletion.
142
+
Establish a cadence: monthly reviews of which alerts fired most and what their action rates were, quarterly reviews of whether alerts still make sense given how your system has evolved. Every alert needs an owner — encode it in a label. Orphaned alerts are the first candidates for deletion.
@@ -165,7 +165,7 @@ Counter the growth with a deletion budget: every quarter, each team must delete
165
165
166
166
## The Results
167
167
168
-
Remember the system I inherited? Forty-seven alerts per day, forty-four requiring no action—a 6% action rate. MTTA had drifted to 12 minutes. On-call satisfaction was 4/10.
168
+
Remember the system I inherited? Forty-seven alerts per day, forty-four requiring no action — a 6% action rate. MTTA had drifted to 12 minutes. On-call satisfaction was 4/10.
169
169
170
170
After six months of systematic auditing, classification, and deletion: 5 meaningful pages per day, 90% action rate, 3-minute MTTA, on-call satisfaction at 8/10. That's an 80% reduction in volume with fifteen times the signal quality.
0 commit comments