Skip to content

Commit 1b217d3

Browse files
committed
Final pass kubernetes-cluster-upgrade-playbook-risk-reduction article
1 parent 1f88135 commit 1b217d3

3 files changed

Lines changed: 110 additions & 95 deletions

File tree

‎_TODO.md‎

Lines changed: 4 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -181,13 +181,7 @@ Need a tooltip component for consistency. List to add tooltips to:
181181

182182
## Downloads / Gated Content
183183

184-
- We need a gating system, where the user gets a token to be able to download a PDF and the token is checked before downloading. If a reader has already given their email address - newsletter signup, contact form, download registration, then the download CTA on short form articles should go directly to the HTML version of the deep dive, and it should have a PDF download button.
185-
186-
- Make `/deep-dive/` SSR loaded so we can check if they've given their e-mail address.
187-
188-
- If they haven't and they get to the path from sharing a link or something, we should use the `index.astro` file in that directory to explain that they've accessed gated content, and ask for their e-mail address. Then redirect to the content they want when they give it.
189-
190-
- If they've given their email link, the content switcher button to switch should be enabled.
184+
- We need a gating system, where the user gets a token to be able to download a PDF and the token is checked before downloading. If a reader has already given their email address - newsletter signup, contact form, download registration, then the download CTA on short form articles should show the PDF download button instead of trying to collect email addresses again.
191185

192186
- We had this copyright statement for articles, it should be in the front matter of PDFs:
193187

@@ -197,8 +191,6 @@ The text, diagrams, and images in this work are licensed under CC BY-NC 4.0
197191
All code samples in this article are licensed under the MIT License. Feel free to use, modify, and distribute them in any project.
198192
```
199193

200-
- When someone's already given their email address, the Download CTA should just have a download button - not require them to sign up again
201-
202194
## Search
203195

204196
- How do we handle the keywords in the long form / pdf files from a search perspective? Can we return the result in the search results if the short form content is not returned in the search results, and highlight it somehow in the search results to show that it is gated content? And clicking on its link takes the user to the Download page for that item?
@@ -218,8 +210,6 @@ https://mermaid.js.org/config/directives.html
218210

219211
- Uppy, Tus server, whatever other server needed for file upload on Contact Form component
220212

221-
- Add Inset component and convert `text` code blocks to use it.
222-
223213
- Focus-visible / active handling on ToC nav items
224214

225215
- Style "Share to Mastodon" modal in src/components/Social/Mastodon/client/index.ts
@@ -236,12 +226,6 @@ https://mermaid.js.org/config/directives.html
236226

237227
- Improve `<abbr>` styling: https://codepen.io/ire/pen/NoqWpm
238228

239-
- Service worker isn't caching favicon
240-
241-
- Image component with zoom + and -, and magnify glass. Modal to expand full size.
242-
243-
- Don't lazy load hero image
244-
245229
## List Component
246230

247231
- Task list checked variant Markdown in dark theme is awkward, it has a dark shadow
@@ -256,18 +240,12 @@ https://mermaid.js.org/config/directives.html
256240

257241
## Content Issues
258242

259-
- "### Geographic/Currency Mismatches" in deep-dive/cdn-edge-caching-cache-keys-vary-headers has a table -> callout -> table back to back
260-
261243
- Need an article on OpenStack
262244

263245
- Add a Social Share on some compelling text in each article
264246

265247
- Home page reorganization: move the "What I Deliver" box from the Hero into the Backstage image. Move the Backstage image / video to the hero.
266248

267-
- Determine if any of the python code blocks should have `#!/bin/python` added
268-
269-
- Need a Q & A format to use in `blameless-postmortem-incident-analysis-systemic-causes`. Might be one of the list formats.
270-
271249
- We need to check for short form and deep article articles where the deep-dive index.pdf has a non-featured tag like "argo-cd" only in the pdf.mdx. In those cases, we should make sure the callout for the deep dive includes the name of that non-featured (technology) tag
272250

273251
- deep-dive/kubernetes-pod-resource-requests-limits-qos-classes is showing the Download hero image
@@ -278,12 +256,7 @@ https://mermaid.js.org/config/directives.html
278256

279257
- Need to update Case Studies with lists and tables too
280258

281-
### This file was badly mangled during refactoring, need to compare against original:
282-
283-
src/content/articles/internal-developer-portal-platform-self-service-actions/pdf.mdx
284-
285-
├──
286-
└──
259+
### Snippets
287260

288261
<span class="font-mono">
289262
</span>
@@ -323,15 +296,15 @@ src/content/articles/internal-developer-portal-platform-self-service-actions/pdf
323296
]}
324297
/>
325298

326-
### Stretching the badge out to the same size and centering text:
299+
#### Stretching the badge out to the same size and centering text:
327300

328301
<List
329302
variant="badge-list"
330303
classes={{
331304
titleClass: "block w-full text-center",
332305
}}
333306

334-
### Spacing out the boxes:
307+
#### Spacing out the boxes:
335308

336309
<List
337310
variant="zebra-list"

‎src/content/articles/kubernetes-cluster-upgrade-playbook-risk-reduction/index.mdx‎

Lines changed: 26 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -9,45 +9,45 @@ tags: ["cloud-platforms","kubernetes","aws","azure"]
99
featured: true
1010
---
1111

12-
*[K8s]: Kubernetes
13-
*[API]: Application Programming Interface
14-
*[CRD]: Custom Resource Definition
1512
*[CNI]: Container Network Interface
13+
*[CRD]: Custom Resource Definition
1614
*[CSI]: Container Storage Interface
1715
*[etcd]: Distributed key-value store used by Kubernetes
16+
*[K8s]: Kubernetes
1817

19-
I watched a company avoid Kubernetes upgrades for 18 months. When they finally __had__ to upgrade for security compliance, they faced a five-version jump. Deprecated APIs were everywhere. Custom controllers broke. Workloads failed in ways nobody expected. What should have been four routine 2-hour maintenance windows became a three-week crisis involving weekend war rooms and executive escalations.
18+
I watched a company avoid Kubernetes upgrades for 18 months. When they finally _had_ to upgrade for security compliance, they faced a five-version jump. Deprecated APIs were everywhere. Custom controllers broke. Workloads failed in ways nobody expected. What should have been four routine 2-hour maintenance windows became a three-week crisis involving weekend war rooms and executive escalations.
2019

21-
The lesson is counterintuitive but consistent: frequent, incremental upgrades are __less__ risky than infrequent large jumps. Kubernetes releases new versions roughly every four months, so each version you skip accumulates deprecated APIs, changed behaviors, and incompatible add-ons. A cluster three or more versions behind isn't technical debt - it's an emergency waiting to happen. The goal of this playbook is to make upgrades boring: predictable procedures that happen quarterly without drama.
20+
The lesson is counterintuitive but consistent: frequent, incremental upgrades are _less_ risky than infrequent large jumps. Kubernetes releases new versions roughly every four months, so each version you skip accumulates deprecated APIs, changed behaviors, and incompatible add-ons. <Highlighter>A cluster three or more versions behind isn't technical debt - it's an emergency waiting to happen.</Highlighter> The goal of this playbook is to make upgrades boring: predictable procedures that happen quarterly without drama.
2221

2322
## The Pre-Upgrade Checklist
2423

2524
Most upgrade failures trace back to skipped preparation. Before touching the cluster, work through four categories of readiness checks.
2625

2726
<List
28-
variant="check-icons-list"
27+
variant="zebra-list"
2928
items={[
3029
{
3130
lead: "Timing and planning",
32-
text: "comes first. Review the release notes for your target version - not just the highlights, but the deprecations and breaking changes. Kubernetes only supports upgrading one minor version at a time (1.28 to 1.29 is fine, 1.28 to 1.30 isn't), so verify your upgrade path. Schedule a maintenance window with buffer time.",
31+
text: "Comes first. Review the release notes for your target version - not just the highlights, but the deprecations and breaking changes. Kubernetes only supports upgrading one minor version at a time (1.28 to 1.29 is fine, 1.28 to 1.30 isn't), so verify your upgrade path. Schedule a maintenance window with buffer time.",
3332
},
3433
{
3534
lead: "Cluster health",
36-
text: "establishes your baseline. All nodes should show Ready status. There shouldn't be unexpected pending pods. Control plane components and etcd should be healthy. Resource utilization should be below 70% - upgrades temporarily reduce capacity as nodes drain and restart.",
35+
text: "Establishes your baseline. All nodes should show Ready status. There shouldn't be unexpected pending pods. Control plane components and etcd should be healthy. Resource utilization should be below 70% - upgrades temporarily reduce capacity as nodes drain and restart.",
3736
},
3837
{
3938
lead: "Backup and recovery",
40-
text: "is your safety net. Take an etcd snapshot before starting. Document current cluster state. Most importantly, document and test your rollback procedure in a non-production environment before you need it.",
39+
text: "These are your safety net. Take an etcd snapshot before starting. Document current cluster state. Most importantly, document and test your rollback procedure in a non-production environment before you need it.",
4140
},
4241
{
4342
lead: "Compatibility verification",
44-
text: "catches the issues that break workloads. Check that your add-ons - CNI plugin, CSI drivers, ingress controller, cert-manager, monitoring stack - are compatible with the target version. Validate workload manifests for deprecated APIs.",
43+
text: "Catches the issues that break workloads. Check that your add-ons - CNI plugin, CSI drivers, ingress controller, cert-manager, monitoring stack - are compatible with the target version. Validate workload manifests for deprecated APIs.",
4544
},
4645
]}
4746
/>
4847

4948
<Table
50-
variant="vertical-column-delineation-table"
49+
variant="grid-and-accent-header-table"
50+
fullWidth={false}
5151
content={{
5252
figure: "Pre-upgrade checklist summary.",
5353
thead: {
@@ -110,7 +110,10 @@ Kubernetes components have strict version compatibility requirements. The API se
110110
This is by design. Kubernetes explicitly allows version skew during upgrades. If you're upgrading from 1.28 to 1.29, you'll temporarily have 1.29 API servers serving requests alongside 1.28 kubelets - and that's fine.
111111

112112
<List
113-
variant="check-icons-list"
113+
variant="timeline-list"
114+
classes={{
115+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
116+
}}
114117
items={[
115118
{
116119
lead: "Phase 1: Preparation (30 minutes).",
@@ -136,7 +139,8 @@ This is by design. Kubernetes explicitly allows version skew during upgrades. If
136139
/>
137140

138141
<Table
139-
variant="vertical-column-delineation-table"
142+
variant="criteria-highlight-table"
143+
fullWidth={false}
140144
content={{
141145
figure: "Upgrade phases with risk assessment.",
142146
thead: {
@@ -207,19 +211,22 @@ __Control plane rollback__ is the nuclear option. For self-managed clusters, rol
207211
Managed Kubernetes rollback options are limited:
208212

209213
<List
210-
variant="check-icons-list"
214+
variant="side-by-side-list"
215+
classes={{
216+
wrapper: "sm:mr-18 mb-6",
217+
}}
211218
items={[
212219
{
213220
lead: "EKS",
214-
text: "doesn't support control plane downgrades - your options are restoring from Terraform state or creating a new cluster",
221+
text: "Doesn't support control plane downgrades - your options are restoring from Terraform state or creating a new cluster",
215222
},
216223
{
217224
lead: "GKE",
218-
text: "supports rollback within the maintenance window",
225+
text: "Supports rollback within the maintenance window",
219226
},
220227
{
221228
lead: "AKS",
222-
text: "doesn't support downgrades either",
229+
text: "Doesn't support downgrades either",
223230
},
224231
]}
225232
/>
@@ -229,7 +236,8 @@ If you're on managed Kubernetes, test what "rollback" actually means for your pr
229236
__Worker node rollback__ is much simpler. If a node is misbehaving after upgrade, you can downgrade its kubelet without affecting the rest of the cluster: cordon, drain, downgrade, uncordon. Some teams use a blue-green node pool strategy - creating new nodes at the target version, then draining old nodes. With this approach, rollback is trivial: uncordon the old pool and delete the new one.
230237

231238
<Table
232-
variant="vertical-column-delineation-table"
239+
variant="grid-and-accent-header-table"
240+
fullWidth={false}
233241
content={{
234242
figure: "Rollback scenario risk assessment.",
235243
thead: {

0 commit comments

Comments
 (0)