You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 98d2de6
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/mtls-certificate-rotation-service-mesh-authentication/index.mdx
+29-16Lines changed: 29 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,18 +22,22 @@ A team enables Istio mTLS across their 50-service mesh. Initial rollout goes smo
22
22
23
23
Recovery takes four hours because the on-call engineer has never manually rotated Istio certificates and the runbook doesn't exist yet.
24
24
25
-
If this sounds familiar, you're not alone. Enabling mTLS is a configuration change; operating it reliably is an ongoing commitment to understanding certificate lifecycles, building automation for rotation, and monitoring for expiration failures.
25
+
If this sounds familiar, you're not alone. <Highlighter>Enabling mTLS is a configuration change; operating it reliably is an ongoing commitment to understanding certificate lifecycles, building automation for rotation, and monitoring for expiration failures.</Highlighter>
26
26
27
27
## The Operational Reality of mTLS
28
28
29
29
Standard TLS — what you use when visiting any HTTPS website — is a one-way trust relationship. The server proves its identity to the client by presenting a certificate. The client validates that certificate against its trust store. The server has no idea who the client is at the transport layer.
30
30
31
31
Mutual TLS adds a second handshake step: after the client validates the server's certificate, the server requests and validates a certificate from the client. Both parties cryptographically prove their identity before any application data flows.
32
32
33
-
The operational cost difference is significant. With standard TLS, you manage certificates for servers — maybe dozens or hundreds. With mTLS, __every__ service needs a certificate, and every service needs to validate certificates from every other service it communicates with. In a 100-service mesh, that's potentially thousands of certificate validation paths to maintain.
33
+
The operational cost difference is significant. With standard TLS, you manage certificates for servers — maybe dozens or hundreds. With mTLS, _every_ service needs a certificate, and every service needs to validate certificates from every other service it communicates with. In a 100-service mesh, that's potentially thousands of certificate validation paths to maintain.
34
34
35
35
<Table
36
-
variant="vertical-column-delineation-table"
36
+
variant="grid-and-accent-header-table"
37
+
fullWidth={false}
38
+
classes={{
39
+
tbody: '!text-content',
40
+
}}
37
41
content={{
38
42
thead: {
39
43
th: ['Aspect', 'Standard TLS', 'Mutual TLS'],
@@ -79,22 +83,25 @@ __CA rotation__ is harder. When you rotate an intermediate or root CA, you're ch
79
83
The safe order for CA rotation:
80
84
81
85
<List
82
-
variant="numbered-with-background-list"
86
+
variant="timeline-list"
87
+
classes={{
88
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
89
+
}}
83
90
items={[
84
91
{
85
-
text: 'Add the new CA to all trust bundles (services now trust both old and new)',
92
+
lead: 'Add the new CA to all trust bundles (services now trust both old and new)',
86
93
},
87
94
{
88
-
text: 'Verify all services have received the updated trust bundle',
95
+
lead: 'Verify all services have received the updated trust bundle',
89
96
},
90
97
{
91
-
text: 'Start issuing certificates from the new CA',
98
+
lead: 'Start issuing certificates from the new CA',
92
99
},
93
100
{
94
-
text: 'Wait for all workload certificates to rotate to the new CA',
101
+
lead: 'Wait for all workload certificates to rotate to the new CA',
95
102
},
96
103
{
97
-
text: 'Remove the old CA from trust bundles',
104
+
lead: 'Remove the old CA from trust bundles',
98
105
},
99
106
]}
100
107
/>
@@ -103,6 +110,7 @@ This sequence diagram shows the rotation flow:
103
110
104
111
<Diagram
105
112
src={caRotationDiagram}
113
+
size="full"
106
114
title="CA rotation sequence showing the safe order of operations."
107
115
description="Sequence diagram with three participants: Workload, Certificate Authority, and Trust Store. The first phase is Step 1, Add New CA to Trust. The Certificate Authority sends a new CA certificate to the Trust Store, and the Trust Store updates the Workload with a trust bundle that includes both the old and new CA. The second phase is Step 2, Verify Distribution, where the Workload confirms it has received the new trust bundle. The third phase is Step 3, Issue New Certs, where the Workload sends a CSR for a new certificate to the Certificate Authority, and the Certificate Authority returns a certificate signed by the new CA. The fourth phase is Step 4, Verify Full Rotation, where the Workload asks the Certificate Authority whether any certificates signed by the old CA remain, and the response is that none remain. The fifth phase is Step 5, Remove Old CA, where the Trust Store removes the old CA and the Workload receives an updated trust bundle containing only the new CA. The diagram emphasizes that trust must be updated before new certificates are issued, and the old CA must only be removed after all workloads have rotated."
108
116
/>
@@ -146,7 +154,8 @@ groups:
146
154
Different certificate levels need different alert thresholds:
@@ -184,31 +193,35 @@ mTLS failures produce cryptic errors. The TLS handshake fails, and you get a gen
184
193
### Common Failure Modes
185
194
186
195
<List
187
-
variant="check-icons-list"
196
+
variant="zebra-list"
188
197
items={[
189
198
{
190
199
lead: 'Certificate expired',
191
-
text: 'The most common cause of mTLS outages. Error messages include `x509: certificate has expired`. Check expiration with `openssl x509 -enddate -noout -in cert.pem`. Fix by forcing rotation or restarting the workload to trigger certificate renewal.',
200
+
text: 'The most common cause of mTLS outages. Error messages include <span class="font-mono">x509: certificate has expired</span>. Check expiration with <span class="font-mono">openssl x509 -enddate -noout -in cert.pem</span>. Fix by forcing rotation or restarting the workload to trigger certificate renewal.',
192
201
},
193
202
{
194
203
lead: 'Trust chain broken',
195
-
text: 'The certificate is valid but the CA that signed it isn\'t in the trust store. You\'ll see `x509: certificate signed by unknown authority`. This happens during CA rotation if trust bundles aren\'t updated before new certificates are issued.',
204
+
text: 'The certificate is valid but the CA that signed it isn\'t in the trust store. You\'ll see <span class="font-mono">x509: certificate signed by unknown authority</span>. This happens during CA rotation if trust bundles aren\'t updated before new certificates are issued.',
196
205
},
197
206
{
198
207
lead: 'SAN mismatch',
199
-
text: 'The certificate is valid but doesn\'t include the hostname being used. Error: `x509: certificate is valid for X, not Y`. This commonly happens when DNS names change or when certificates are issued with incomplete SAN lists.',
208
+
text: 'The certificate is valid but doesn\'t include the hostname being used. Error: <span class="font-mono">x509: certificate is valid for X, not Y</span>. This commonly happens when DNS names change or when certificates are issued with incomplete SAN lists.',
200
209
},
201
210
{
202
211
lead: 'Wrong key usage',
203
-
text: 'The certificate exists but wasn\'t issued for mTLS. If it only has `serverAuth` in Extended Key Usage, it can\'t be used as a client certificate. Reissue with both `serverAuth` and `clientAuth`.',
212
+
text: 'The certificate exists but wasn\'t issued for mTLS. If it only has <span class="font-mono">serverAuth</span> in Extended Key Usage, it can\'t be used as a client certificate. Reissue with both <span class="font-mono">serverAuth</span> and <span class="font-mono">clientAuth</span>.',
204
213
},
205
214
]}
206
215
/>
207
216
208
217
I once spent two hours debugging a "connection reset by peer" that turned out to be a certificate with `serverAuth` only — no `clientAuth`. The error message mentioned nothing about key usage. The fix was a one-line change to the certificate spec, but finding it required systematically ruling out every other possibility.
0 commit comments