You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0f74dbb
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/database-schema-migrations-continuous-deployment-zero-downtime/index.mdx
+33-28Lines changed: 33 additions & 28 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,20 +20,21 @@ Application code can roll back in seconds. Schema changes can take hours to reve
20
20
21
21
The core tension: CD assumes stateless, independent deployments. Databases are stateful and shared. Your application runs across multiple pods that can be replaced atomically, but every instance talks to the same database. A migration that takes 200ms on your development dataset can lock tables for four minutes against 50 million production rows. During that window, writes queue up, connection pools exhaust, timeouts cascade, and users see errors.
22
22
23
-
I've watched teams debate mid-incident: do we wait out the lock, or roll back and make things worse? Neither option is good once you're in that situation. The answer is to never get there — to treat migrations as a distinct deployment concern with their own safety patterns.
23
+
I've watched teams debate mid-incident: do we wait out the lock, or roll back and make things worse? Neither option is good once you're in that situation. The answer is to never get there — to <Highlighter>treat migrations as a distinct deployment concern with their own safety patterns.</Highlighter>
24
24
25
25
## Why Migrations Break Deployments
26
26
27
27
### The Lock Problem
28
28
29
29
Most DDL operations acquire exclusive locks. While you hold that lock, every other query on that table waits — reads, writes, everything. On a small table, nobody notices. On a table with tens of millions of rows, the lock can persist for minutes while the database rewrites data or rebuilds indexes.
30
30
31
-
Here's what happens when you run `ALTER TABLE users ADD COLUMN status VARCHAR(20) DEFAULT 'active'` on older PostgreSQL (pre-11): the database acquires an ACCESS EXCLUSIVE lock, rewrites the entire table to add the column with the default value, rebuilds indexes, then finally releases the lock. During that rewrite, every application thread trying to touch that table blocks. Connection pools fill up with waiting queries. Those queries eventually timeout. The timeouts cascade through your application as dependent operations fail.
31
+
Here's what happens when you run `ALTER TABLE users ADD COLUMN status VARCHAR(20) DEFAULT 'active'` on older PostgreSQL (pre-v11): the database acquires an ACCESS EXCLUSIVE lock, rewrites the entire table to add the column with the default value, rebuilds indexes, then finally releases the lock. During that rewrite, every application thread trying to touch that table blocks. Connection pools fill up with waiting queries. Those queries eventually timeout. The timeouts cascade through your application as dependent operations fail.
32
32
33
-
Modern PostgreSQL (11+) made some operations instant — adding a nullable column or a column with a default no longer rewrites the table. But plenty of operations still require rewrites: changing column types, adding NOT NULL constraints to existing columns, certain index operations. Here's how that blocking cascade looks:
33
+
Modern PostgreSQL (v11+) made some operations instant — adding a nullable column or a column with a default no longer rewrites the table. But plenty of operations still require rewrites: changing column types, adding NOT NULL constraints to existing columns, certain index operations. Here's how that blocking cascade looks:
34
34
35
35
<Diagram
36
36
src={tableLockDiagram}
37
+
magnifyButtonPosition="top-left"
37
38
title="Table lock during migration blocks all application threads."
38
39
description="Flowchart showing a migration lifecycle and the application impact of a table lock. The main sequence runs from Migration Starts to Acquire Table Lock, then Rewrite Table Data, Rebuild Indexes, and finally Release Lock. A separate grouped region labeled Blocked Application Threads contains four representative threads: Thread 1: Waiting..., Thread 2: Waiting..., Thread 3: Timeout, and Thread 4: Timeout. A dotted arrow labeled blocks points from Acquire Table Lock to that grouped region. The diagram illustrates that once an exclusive table lock is taken, normal application work stalls behind it, some requests remain queued, and others eventually fail with timeouts before the migration completes and the lock is released."
39
40
/>
@@ -51,31 +52,31 @@ The solution is that every schema change must be backward-compatible. The schema
51
52
These failure modes appear repeatedly across teams:
52
53
53
54
<Table
54
-
variant="vertical-column-delineation-table"
55
+
variant="grid-and-accent-header-table"
55
56
content={{
56
57
figure: "Common migration failures and their prevention.",
@@ -90,6 +91,13 @@ The expand-contract pattern (sometimes called parallel change) is the fundamenta
90
91
91
92
The pattern has three phases:
92
93
94
+
<Diagram
95
+
src={expandContractDiagram}
96
+
size="2xl"
97
+
title="Expand-contract pattern phases."
98
+
description="Flowchart showing the three phases of a zero-downtime schema change. In Phase 1: Expand, Add new column leads to Deploy dual-write code, which leads to Backfill existing data. The flow then continues into Phase 2: Migrate, where All data in new column leads to Deploy code reading from new, then to Stop writing to old column. From there the process moves into Phase 3: Contract, where Old column unused leads to Remove old column. Connecting arrows between phases show that each stage depends on the previous one being completed safely. The diagram emphasizes that schema changes are broken into compatible, sequential phases rather than performed as one destructive operation."
99
+
/>
100
+
93
101
<List
94
102
variant="numbered-with-background-list"
95
103
items={[
@@ -108,38 +116,35 @@ The pattern has three phases:
108
116
]}
109
117
/>
110
118
111
-
<Diagram
112
-
src={expandContractDiagram}
113
-
title="Expand-contract pattern phases."
114
-
description="Flowchart showing the three phases of a zero-downtime schema change. In Phase 1: Expand, Add new column leads to Deploy dual-write code, which leads to Backfill existing data. The flow then continues into Phase 2: Migrate, where All data in new column leads to Deploy code reading from new, then to Stop writing to old column. From there the process moves into Phase 3: Contract, where Old column unused leads to Remove old column. Connecting arrows between phases show that each stage depends on the previous one being completed safely. The diagram emphasizes that schema changes are broken into compatible, sequential phases rather than performed as one destructive operation."
115
-
/>
116
-
117
119
### A Concrete Example
118
120
119
121
Let's trace through renaming a column from `email` to `email_address`. A direct `ALTER TABLE RENAME COLUMN` would break every query referencing the old name instantly. Here's the expand-contract approach:
120
122
121
123
<List
122
-
variant="check-icons-list"
124
+
variant="timeline-list"
125
+
classes={{
126
+
wrapper: "sm:ml-6 sm:mr-12",
127
+
}}
123
128
items={[
124
129
{
125
130
lead: "Week 1 (Expand):",
126
-
text: "Run a migration to add the `emailaddress` column. Deploy code that writes to both columns on every insert and update, but still reads from `email`. At this point, new rows have both columns populated; old rows only have `email`.",
131
+
text: 'Run a migration to add the <span class="font-mono">emailaddress</span> column. Deploy code that writes to both columns on every insert and update, but still reads from <span class="font-mono">email</span>. At this point, new rows have both columns populated; old rows only have <span class="font-mono">email</span>.',
127
132
},
128
133
{
129
134
lead: "Week 2 (Migrate):",
130
-
text:"Run a backfill job to copy `email` values to `email_address` for existing rows. Once complete, deploy code that reads from `email_address` instead of `email`. Keep writing to both columns — old pods might still be running.",
135
+
text:'Run a backfill job to copy <span class="font-mono">email</span> values to <span class="font-mono">email_address</span> for existing rows. Once complete, deploy code that reads from <span class="font-mono">email_address</span> instead of <span class="font-mono">email</span>. Keep writing to both columns — old pods might still be running.',
131
136
},
132
137
{
133
138
lead: "Week 3 (Contract):",
134
-
text: "After all old pods are gone and you've verified nothing reads `email`, deploy code that only uses `email_address`. Then run a migration to drop the `email` column.",
139
+
text: 'After all old pods are gone and you\'ve verified nothing reads <span class="font-mono">email</span>, deploy code that only uses <span class="font-mono">email_address</span>. Then run a migration to drop the <span class="font-mono">email</span> column.',
135
140
},
136
141
]}
137
142
/>
138
143
139
-
<Newsletter />
140
-
141
144
Total time: 2-3 weeks for a "simple" rename. This is why you think carefully before renaming columns in production.
142
145
146
+
<Newsletter />
147
+
143
148
Here's the application code during the expand phase — the dual-write pattern:
144
149
145
150
```ruby title="expand-phase-dual-write.rb"
@@ -177,9 +182,9 @@ Now that you understand the fundamental pattern, here's a quick reference for co
177
182
178
183
### Adding Columns
179
184
180
-
Modern PostgreSQL (11+) makes this easy. Adding a nullable column or a column with a default is instant — the default is stored in the catalog, not written to every row. The danger is adding NOT NULL without a default, which requires a table scan.
185
+
Modern PostgreSQL (v11+) makes this easy. Adding a `nullable` column or a column with a default is instant — the default is stored in the catalog, not written to every row. The danger is adding `NOT NULL` without a default, which requires a table scan.
181
186
182
-
Safe approach: add nullable first, backfill values, then add the NOT NULL constraint.
187
+
Safe approach: add `nullable` first, backfill values, then add the `NOT NULL` constraint.
0 commit comments