You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dafc77c
Browse filesBrowse the repository at this point in the historyBrowse files
A single test with a 5% flake rate will block CI once every 20 runs. That sounds manageable. But a suite of 100 tests where each has just a 1% flake rate? That suite will fail 63% of builds. The probability compounds: $$(1 - 0.01)^{100} \approx 0.37$$, meaning only 37% of builds pass cleanly.
16
+
A single test with a 5% flake rate will block CI once every 20 runs. That sounds manageable. <Highlighter>A suite of 100 tests where each has just a 1% flake rate? That suite will fail 63% of builds.</Highlighter> The probability compounds: $$(1 - 0.01)^{100} \approx 0.37$$, meaning only 37% of builds pass cleanly.
19
17
20
18
I've watched this pattern destroy teams' ability to ship. A few random failures appear. The first response is always the same: "just retry and merge." Within a few months, nobody trusts red builds anymore. Developers stop investigating failures because it's faster to retry than debug. Then a real regression ships because the failure was dismissed as "probably flaky." That's when teams lose the ability to distinguish signal from noise.
Code: Form submission race — wait for the API response before asserting.
46
44
47
-
The flaky version assumes the success message will appear immediately after click. The stable version explicitly waits for the API response, __then__ checks the DOM. The key insight: wait for the __event__ that causes the state change, not the state change itself.
45
+
The flaky version assumes the success message will appear immediately after click. The stable version explicitly waits for the API response, _then_ checks the DOM. The key insight: wait for the _event_ that causes the state change, not the state change itself.
48
46
49
47
### The Debounced Search Race
50
48
@@ -94,10 +92,10 @@ Race conditions are the biggest category, but they're not the only one. The seco
94
92
95
93
Environment issues cause roughly 25% of flakes. The test itself is often correct — it's the environment that's unstable. Tests depend on external state that varies between runs: cookies from previous tests, database records that weren't cleaned up, system time that behaves differently in different timezones.
96
94
97
-
The solution is isolation: each test should run in a pristine environment with no pollution from previous tests or external factors.
98
-
99
95
<Newsletter />
100
96
97
+
The solution is isolation: each test should run in a pristine environment with no pollution from previous tests or external factors.
98
+
101
99
### Browser State and Animations
102
100
103
101
Cookies, localStorage, and session data can leak between tests if you're reusing browser contexts. Modern SPAs also introduce animation timing issues — clicking an element mid-transition causes flakes. The cleanest approach is fresh contexts for each test with animations disabled:
@@ -129,7 +127,12 @@ Starting fresh is more reliable than cleaning up. Cleanup can fail silently (a c
129
127
Database pollution is trickier. Three common approaches, each with tradeoffs:
@@ -170,12 +173,12 @@ Tests that work on your machine but fail in CI at specific hours often have time
170
173
171
174
For unit and integration tests, use your test framework's built-in time mocking. Vitest provides `vi.useFakeTimers()` which lets you control Date.now(), setTimeout, and other time-dependent APIs. For E2E tests running in the browser, you'll need to inject a fixed date via `page.addInitScript()` or use a library that patches the browser's Date object.
172
175
173
-
The simpler alternative is making assertions timezone-agnostic: compare timestamps rather than formatted date strings, and avoid assertions that depend on "today" or "this month."
174
-
175
176
<Callouttype="info">
176
177
If your test fails at midnight UTC, around month boundaries, or in different timezones, you have a time-dependent test. Either freeze time or make assertions timezone-agnostic.
177
178
</Callout>
178
179
180
+
The simpler alternative is making assertions timezone-agnostic: compare timestamps rather than formatted date strings, and avoid assertions that depend on "today" or "this month."
181
+
179
182
## The Path Forward
180
183
181
184
Race conditions and environment issues cause 85% of flakes. Fixing these two categories transforms your test suite from a liability into an asset.
@@ -203,21 +206,24 @@ Race conditions and environment issues cause 85% of flakes. Fixing these two cat
203
206
The approach is systematic:
204
207
205
208
<List
206
-
variant="check-icons-list"
209
+
variant="chevron-list"
210
+
classes={{
211
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
212
+
}}
207
213
items={[
208
214
{
209
-
lead: "Replace them with waits for specific conditions — network responses, DOM elements, URL changes.",
210
-
text: "For race conditions, find every place your test uses `waitForTimeout()` or assumes immediate state changes after actions.",
215
+
lead: 'For race conditions, find every place your test uses <span class="font-mono">waitForTimeout()</span> or assumes immediate state changes after actions.',
216
+
text: "Replace them with waits for specific conditions — network responses, DOM elements, URL changes.",
211
217
},
212
218
{
213
-
lead: "Clear browser state between tests. Isolate database state through transactions, per-test databases, or seed scripts. Freeze time or use timezone-agnostic assertions. Disable animations.",
214
-
text: "For environment issues, audit your test setup.",
219
+
lead: "For environment issues, audit your test setup.",
220
+
text: "Clear browser state between tests. Isolate database state through transactions, per-test databases, or seed scripts. Freeze time or use timezone-agnostic assertions. Disable animations.",
215
221
},
216
222
]}
217
223
/>
218
224
219
-
The deeper lesson is that flaky tests are symptoms. They reveal race conditions in your tests, instability in your application, or inconsistency in your environments. Fixing flakes often uncovers real bugs — an API that's slower under load, a component that renders before its data arrives, a cleanup process that doesn't handle edge cases.
220
-
221
225
<Callouttype="success">
222
226
Start small: pick your three worst flakes (highest impact, not necessarily highest flake rate), fix them this sprint, and measure CI pass rate before and after. A 10% improvement in CI reliability often translates to hours saved per week across the team.
223
227
</Callout>
228
+
229
+
The deeper lesson is that flaky tests are symptoms. They reveal race conditions in your tests, instability in your application, or inconsistency in your environments. Fixing flakes often uncovers real bugs — an API that's slower under load, a component that renders before its data arrives, a cleanup process that doesn't handle edge cases.
0 commit comments