Skip to content

Commit ac98329

Browse files
committed
Second pass wrap mermaid diagrams in Diagram component
1 parent 12f6447 commit ac98329

85 files changed

Lines changed: 6108 additions & 2687 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎src/content/articles/alert-fatigue-reduction-triage-actionable-alerts/download.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ pages: 24
1414

1515
On-call rotations with high alert volume drain engineers and slow incident response. An audit found one team managing 47 alerts per 24-hour shift—44 required no action, while the real issue went unnoticed in the noise.
1616

17-
This complete guide walks you through the systematic process used to reduce a 200-alert-per-week system to 35 actionable alerts, with a 90% action rate—transforming on-call from a burnout machine into a sustainable feedback loop.
17+
This complete guide walks you through the systematic process used to reduce a 200-alert-per-week system to 35 actionable alerts, with a 90% action rate — transforming on-call from a burnout machine into a sustainable feedback loop.
1818

1919
Read this e-book to understand:
2020

‎src/content/articles/alert-fatigue-reduction-triage-actionable-alerts/index.mdx‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,21 +17,21 @@ featured: true
1717

1818
I inherited a monitoring setup where the on-call engineer averaged 47 alerts per day. Forty-four of those required no action — thresholds set too aggressively, alerts for non-problems, duplicate notifications for the same underlying issue. The forty-fifth was a memory leak that had been slowly building for three hours. By the time anyone noticed it in the noise, the service had already crashed and restarted twice.
1919

20-
That's the core problem: comprehensive monitoring and actionable alerting are often at odds. Teams add alerts because something _might_ go wrong, and removing an alert feels like removing a safety net. But the math works against you—an engineer who has acknowledged thirty false positives is not in the right headspace to notice the thirty-first is real.
20+
That's the core problem: comprehensive monitoring and actionable alerting are often at odds. Teams add alerts because something _might_ go wrong, and removing an alert feels like removing a safety net. But the math works against you — an engineer who has acknowledged thirty false positives is not in the right headspace to notice the thirty-first is real.
2121

22-
<Callout type="warning">Every alert you keep dilutes the ones that matter. An alert that fires but requires no action is worse than no alert at all—it trains engineers to ignore pages.</Callout>
22+
<Callout type="warning">Every alert you keep dilutes the ones that matter. An alert that fires but requires no action is worse than no alert at all — it trains engineers to ignore pages.</Callout>
2323

2424
This article walks through the audit process that took us from 47 alerts per day to about 5 meaningful pages, with an action rate above 90%.
2525

2626
## Why Alert Noise Compounds
2727

28-
The most visible symptom of alert fatigue is MTTA drift—mean time to acknowledge. When engineers expect noise, acknowledgment slows. I've watched MTTA drift from under two minutes to over fifteen as alert volume increased, because the on-call engineer stopped keeping their phone nearby.
28+
The most visible symptom of alert fatigue is MTTA drift — mean time to acknowledge. When engineers expect noise, acknowledgment slows. I've watched MTTA drift from under two minutes to over fifteen as alert volume increased, because the on-call engineer stopped keeping their phone nearby.
2929

30-
There's solid research on this from healthcare, where alarm fatigue kills patients. Studies in ICUs found that 72-99% of alarms required no clinical intervention[^1], and staff developed coping mechanisms—turning down volume, disabling non-critical alarms, or simply not rushing to respond. Software operations is no different.
30+
There's solid research on this from healthcare, where alarm fatigue kills patients. Studies in ICUs found that 72-99% of alarms required no clinical intervention[^1], and staff developed coping mechanisms — turning down volume, disabling non-critical alarms, or simply not rushing to respond. Software operations is no different.
3131

3232
[^1]: Sendelbach S, Funk M. "Alarm fatigue: a patient safety concern." AACN Adv Crit Care. 2013 Oct-Dec: 378-86.
3333

34-
The feedback loop makes it worse. After an incident caused by a missed alert, the instinct is to add more alerts. But if the alert was missed because of volume, adding more alerts makes the next miss more likely, not less. You can't fix what you haven't measured—and most teams have never actually measured their alert usefulness.
34+
The feedback loop makes it worse. After an incident caused by a missed alert, the instinct is to add more alerts. But if the alert was missed because of volume, adding more alerts makes the next miss more likely, not less. You can't fix what you haven't measured — and most teams have never actually measured their alert usefulness.
3535

3636
## The Three-Step Audit
3737

@@ -43,7 +43,7 @@ Export your alert definitions and firing history. Prometheus/Alertmanager stores
4343

4444
<Newsletter />
4545

46-
For each alert, capture: alert name, firing frequency over the last 90 days, action rate (percentage of firings that required human intervention), owning team, and whether a runbook exists. The action rate column is the most important and the hardest to populate—you'll need to cross-reference alert firings with incident tickets or on-call logs.
46+
For each alert, capture: alert name, firing frequency over the last 90 days, action rate (percentage of firings that required human intervention), owning team, and whether a runbook exists. The action rate column is the most important and the hardest to populate — you'll need to cross-reference alert firings with incident tickets or on-call logs.
4747

4848
### Step 2: Classify by Action Rate
4949

@@ -95,13 +95,13 @@ I use four buckets:
9595
}}
9696
/>
9797

98-
The "noisy" bucket is where most of the work happens. These alerts fire for real issues, but the threshold is wrong or multiple alerts fire for the same underlying problem. Fixing them requires understanding _why_ action wasn't taken—was the alert a false positive, or did the problem resolve itself before anyone could respond?
98+
The "noisy" bucket is where most of the work happens. These alerts fire for real issues, but the threshold is wrong or multiple alerts fire for the same underlying problem. Fixing them requires understanding _why_ action wasn't taken — was the alert a false positive, or did the problem resolve itself before anyone could respond?
9999

100100
### Step 3: Delete the Noise
101101

102102
Deleting alerts is politically difficult. Someone created that alert for a reason. Maybe there was an incident, and the alert was the action item from the postmortem. Deleting it feels like ignoring the lessons learned.
103103

104-
But an alert that fires constantly and gets ignored isn't a lesson learned—it's a lesson forgotten. The incident that created it is no longer prevented by the alert; the alert just adds to the noise that makes the _next_ incident harder to catch.
104+
But an alert that fires constantly and gets ignored isn't a lesson learned — it's a lesson forgotten. The incident that created it is no longer prevented by the alert; the alert just adds to the noise that makes the _next_ incident harder to catch.
105105

106106
The talking points that work:
107107

@@ -127,19 +127,19 @@ The talking points that work:
127127

128128
## One Key Technique: Alert on Symptoms
129129

130-
Most alert configurations work backwards. They alert on causes—CPU spikes, memory pressure, disk I/O, connection pool exhaustion—hoping to catch problems before users notice. The result is dozens of alerts for a single incident, each describing a different aspect of the same failure.
130+
Most alert configurations work backwards. They alert on causes — CPU spikes, memory pressure, disk I/O, connection pool exhaustion — hoping to catch problems before users notice. The result is dozens of alerts for a single incident, each describing a different aspect of the same failure.
131131

132132
Flip this around. <Highlighter>Alert on symptoms instead: what users actually experience.</Highlighter> A single "checkout latency exceeds SLO" alert replaces the CPU alert, the database alert, the cache alert, and the memory alert. They're all symptoms of the same underlying problem, and the responder doesn't need to be told about each one separately.
133133

134-
Cause-based alerts still have a place—as diagnostic information in dashboards and runbooks, not as paging conditions. When the symptom alert fires, the responder can check the cause metrics to understand _why_. But the page itself should describe the user impact, not the infrastructure state.
134+
Cause-based alerts still have a place — as diagnostic information in dashboards and runbooks, not as paging conditions. When the symptom alert fires, the responder can check the cause metrics to understand _why_. But the page itself should describe the user impact, not the infrastructure state.
135135

136-
This principle extends to alert grouping and dependency suppression. If the database is down, you don't need six "connection refused" alerts from downstream services—you need one "database down" alert. Map your dependencies and configure your alerting system to suppress the noise.
136+
This principle extends to alert grouping and dependency suppression. If the database is down, you don't need six "connection refused" alerts from downstream services — you need one "database down" alert. Map your dependencies and configure your alerting system to suppress the noise.
137137

138138
## Maintaining the Gains
139139

140140
Alert counts grow naturally. Someone adds an alert during an incident, another gets added during a deployment, a third comes from a vendor integration. Rarely does anyone go back and remove alerts. Without discipline, the count ratchets upward.
141141

142-
Establish a cadence: monthly reviews of which alerts fired most and what their action rates were, quarterly reviews of whether alerts still make sense given how your system has evolved. Every alert needs an owner—encode it in a label. Orphaned alerts are the first candidates for deletion.
142+
Establish a cadence: monthly reviews of which alerts fired most and what their action rates were, quarterly reviews of whether alerts still make sense given how your system has evolved. Every alert needs an owner — encode it in a label. Orphaned alerts are the first candidates for deletion.
143143

144144
<Download
145145
resource="alert-fatigue-reduction-triage-actionable-alerts"
@@ -165,7 +165,7 @@ Counter the growth with a deletion budget: every quarter, each team must delete
165165

166166
## The Results
167167

168-
Remember the system I inherited? Forty-seven alerts per day, forty-four requiring no action—a 6% action rate. MTTA had drifted to 12 minutes. On-call satisfaction was 4/10.
168+
Remember the system I inherited? Forty-seven alerts per day, forty-four requiring no action — a 6% action rate. MTTA had drifted to 12 minutes. On-call satisfaction was 4/10.
169169

170170
After six months of systematic auditing, classification, and deletion: 5 meaningful pages per day, 90% action rate, 3-minute MTTA, on-call satisfaction at 8/10. That's an 80% reduction in volume with fifteen times the signal quality.
171171

0 commit comments

Comments
 (0)