You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1ef370b
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/kubernetes-secrets-external-secrets-operator-csi-vault/index.mdx
+10-5Lines changed: 10 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,16 +18,17 @@ import patternDecisionDiagram from "./diagrams/pattern-decision-tree.jpg"
18
18
19
19
It's 3 AM and Vault is down. Your on-call engineer gets paged because deployments are failing — pods stuck in ContainerCreating, blocking a critical hotfix. Meanwhile, another team's services keep humming along despite the same outage. The difference isn't luck. It's how secrets get into pods.
20
20
21
-
Both teams use Vault. Both followed the documentation. But one team chose External Secrets Operator, which syncs secrets periodically and caches them as native Kubernetes secrets. The other chose the Secrets Store CSI Driver, which fetches secrets on-demand when pods start. When Vault went down, ESO's cached secrets kept working. CSI's synchronous fetches failed, and pods couldn't start. This isn't about which tool is better — it's about understanding the failure mode you've chosen __before__ it matters.
21
+
Both teams use Vault. Both followed the documentation. But one team chose External Secrets Operator, which syncs secrets periodically and caches them as native Kubernetes secrets. The other chose the Secrets Store CSI Driver, which fetches secrets on-demand when pods start. When Vault went down, ESO's cached secrets kept working. CSI's synchronous fetches failed, and pods couldn't start. This isn't about which tool is better — it's about understanding the failure mode you've chosen _before_ it matters.
22
22
23
23
## The Two Patterns That Matter
24
24
25
-
External secrets management in Kubernetes has consolidated around two dominant patterns. External Secrets Operator runs as a controller in your cluster, periodically syncing secrets from Vault (or AWS Secrets Manager, Azure Key Vault, etc.) into native Kubernetes Secret objects. The Secrets Store CSI Driver takes a different approach: it mounts secrets directly into pods as volumes, fetching them from the external manager when pods start.
25
+
<Highlighter>External secrets management in Kubernetes has consolidated around two dominant patterns.</Highlighter> External Secrets Operator runs as a controller in your cluster, periodically syncing secrets from Vault (or AWS Secrets Manager, Azure Key Vault, etc.) into native Kubernetes Secret objects. The Secrets Store CSI Driver takes a different approach: it mounts secrets directly into pods as volumes, fetching them from the external manager when pods start.
26
26
27
27
Both work fine when your secret manager is healthy. The difference is what happens when it isn't. ESO decouples secret fetching from pod lifecycle — the controller syncs independently, and pods consume cached Kubernetes Secrets. CSI couples them tightly — pods can't start until secrets are fetched. This architectural difference determines everything about how your applications behave during an outage.
@@ -77,7 +78,10 @@ The pod __cannot start__ until this volume mount succeeds. If Vault is unavailab
77
78
This has cascading implications that aren't obvious until you experience them:
78
79
79
80
<List
80
-
variant="check-icons-list"
81
+
variant="timeline-list"
82
+
classes={{
83
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
84
+
}}
81
85
items={[
82
86
{
83
87
text: 'New deployments block entirely — no pods can start',
@@ -108,6 +112,7 @@ The decision between ESO and CSI comes down to two questions: Can your applicati
108
112
109
113
<Diagram
110
114
src={patternDecisionDiagram}
115
+
size="full"
111
116
title="Pattern decision tree."
112
117
description="Flowchart guiding the choice between Kubernetes secret injection patterns based on failure tolerance. The process starts at Choose Secret Pattern and asks whether the application can tolerate minutes of staleness. If yes, the recommendation is ESO. If no, the chart asks whether the team accepts pod failures when the secret manager is down. If yes, the recommendation is CSI Driver. If no, the chart asks whether custom fallback logic is needed. If yes, the recommendation is Init Container. If no, the recommendation is ESO plus a short refresh interval. The diagram highlights that the main decision criteria are tolerance for stale secrets, tolerance for blocked pod startup during secret-manager outages, and whether the team is willing to build custom fallback behavior."
113
118
/>
@@ -130,8 +135,8 @@ For most organizations, ESO is the right default. It's operationally simpler, Gi
130
135
131
136
The mistake isn't choosing either pattern. It's not understanding which failure mode you've chosen. The team that slept through the 3 AM Vault outage didn't get lucky — they understood that ESO's cached secrets would keep their services running. The team that got paged made a valid choice too; for their payment system, blocking on fresh credentials was the right call. Their runbooks reflected it.
132
137
133
-
Whatever you choose, document it. When the next outage happens, your incident responders shouldn't be learning your secret injection architecture for the first time.
134
-
135
138
<Callouttype="success">
136
139
Start with ESO for the majority of workloads, with a 15-minute refresh interval, and monitor sync status with alerts on stale ExternalSecrets. Add CSI for specific high-security applications where staleness is unacceptable. This gives you operational simplicity with escape hatches for edge cases.
137
140
</Callout>
141
+
142
+
Whatever you choose, document it. When the next outage happens, your incident responders shouldn't be learning your secret injection architecture for the first time.
Copy file name to clipboardExpand all lines: src/content/articles/kubernetes-secrets-external-secrets-operator-csi-vault/pdf.mdx
+45-19Lines changed: 45 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,24 +29,27 @@ import threeSecretDiagram from "./diagrams/three-secret-injection-patterns.jpg"
29
29
30
30
It's 3 AM and Vault is down. Your on-call engineer gets paged because deployments are failing — pods stuck in ContainerCreating, blocking a critical hotfix. Meanwhile, another team's services keep humming along despite the same outage. The difference isn't luck. It's how secrets get into pods.
31
31
32
-
Kubernetes secrets have a fundamental problem: they're base64-encoded, not encrypted. They sit in etcd alongside your cluster state, readable by anyone with RBAC access to the namespace. External secret managers like Vault, AWS Secrets Manager, and Azure Key Vault solve the security problem by keeping secrets outside the cluster — but they create a new problem. Your pods now depend on an external service, and that dependency has failure modes you need to understand __before__ the 3 AM page.
32
+
Kubernetes secrets have a fundamental problem: they're base64-encoded, not encrypted. They sit in etcd alongside your cluster state, readable by anyone with RBAC access to the namespace. External secret managers like Vault, AWS Secrets Manager, and Azure Key Vault solve the security problem by keeping secrets outside the cluster — but they create a new problem. Your pods now depend on an external service, and that dependency has failure modes you need to understand _before_ the 3 AM page.
33
33
34
34
Three patterns dominate secret injection:
35
35
36
36
<List
37
-
variant="check-icons-list"
37
+
variant="badge-list"
38
+
classes={{
39
+
titleClass: "block w-full text-center",
40
+
}}
38
41
items={[
39
42
{
40
-
lead: 'External Secrets Operator',
41
-
text: 'syncs secrets on a schedule and caches them as native Kubernetes secrets.',
43
+
title: 'External Secrets Operator',
44
+
lead: 'Syncs secrets on a schedule and caches them as native Kubernetes secrets.',
42
45
},
43
46
{
44
-
lead: 'Secrets Store CSI Driver',
45
-
text: 'fetches secrets on-demand when pods start.',
47
+
title: 'Secrets Store CSI Driver',
48
+
lead: 'Fetches secrets on-demand when pods start.',
46
49
},
47
50
{
48
-
lead: 'Init containers',
49
-
text: 'give you complete control but require you to implement everything yourself.',
51
+
title: 'Init containers',
52
+
lead: 'Gives you complete control but require you to implement everything yourself.',
50
53
},
51
54
]}
52
55
/>
@@ -55,6 +58,7 @@ Each pattern fails differently when the secret manager goes away. Consider a 30-
@@ -83,6 +87,8 @@ ESO's pods keep running with their last-synced secrets, and new pods start succe
83
87
84
88
<Diagram
85
89
src={threeSecretDiagram}
90
+
magnifyButtonPosition="top-left"
91
+
size="full"
86
92
title="Three secret injection patterns."
87
93
description="Flowchart comparing three ways Kubernetes workloads receive secrets from an external secret manager. At the top, an external group labeled External Secret Manager contains Vault, AWS Secrets Manager, or another provider. Inside a grouped Kubernetes Cluster section, the first pattern labeled Pattern 1: ESO shows ESO Controller writing to a Kubernetes Secret, which is then consumed by a pod through envFrom or volumeMount. The second pattern labeled Pattern 2: CSI Driver shows the CSI Driver DaemonSet providing a pod volume mount. The third pattern labeled Pattern 3: Init Container shows an Init Container writing secret data into a shared emptyDir, which the Main Container then reads. The external manager connects to ESO through a path labeled Periodic sync and connects to both the CSI Driver and Init Container through paths labeled On pod start. The diagram emphasizes that ESO caches into native Kubernetes Secrets, CSI fetches during pod startup, and init containers implement a custom startup-time retrieval path."
88
94
/>
@@ -164,7 +170,11 @@ Even with encryption at rest, native secrets have operational limitations. There
164
170
External Secrets Operator is a Kubernetes operator that syncs secrets from external managers — Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager — into native Kubernetes secrets. The operator runs in your cluster, periodically fetches secrets from the external source, and creates or updates Kubernetes Secret objects that your pods consume normally.
@@ -338,6 +348,7 @@ The key insight with ESO is the decoupling between secret fetching and pod lifec
338
348
339
349
<Diagram
340
350
src={esoSyncDiagram}
351
+
size="full"
341
352
title="ESO sync flow."
342
353
description="Sequence diagram showing how External Secrets Operator synchronizes secret values into Kubernetes. The participants are ExternalSecret, ESO Controller, Secret Manager, Kubernetes Secret, and Pod. The flow begins when the ESO Controller watches an ExternalSecret resource. Inside a loop that repeats every refreshInterval, the controller fetches secret values from the external Secret Manager, receives the values in response, and then creates or updates a Kubernetes Secret. Separately, a Pod mounts that Kubernetes Secret and receives the secret data. A note over the ESO Controller and Secret Manager explains that if the Secret Manager is down, the Kubernetes Secret keeps its last value. The diagram highlights that ESO fetches asynchronously on a controller schedule and that pods consume cached Kubernetes Secret data rather than making direct startup-time calls to the external manager."
343
354
/>
@@ -472,14 +483,15 @@ spec:
472
483
```
473
484
Code: Pod with CSI secret volume.
474
485
475
-
The pod __cannot start__ until the volume mounts successfully — which means until secrets are fetched from Vault. This is the key difference from ESO: the dependency is synchronous.
486
+
The pod _cannot start_ until the volume mounts successfully — which means until secrets are fetched from Vault. This is the key difference from ESO: the dependency is synchronous.
476
487
477
488
The CSI driver optionally supports rotation. Enable it by setting `--enable-secret-rotation=true` on the driver DaemonSet and adding `rotationPollInterval` to your SecretProviderClass. When enabled, the driver periodically re-fetches secrets and updates the mounted files. Your application must detect file changes (using inotify or periodic re-reads) to pick up rotated values — the driver updates files, but it can't restart your process.
478
489
479
490
The flow differs fundamentally from ESO: secrets are fetched synchronously during pod startup, not asynchronously by a controller.
480
491
481
492
<Diagram
482
493
src={csiDriverDiagram}
494
+
size="full"
483
495
title="CSI driver secret fetch flow."
484
496
description="Sequence diagram showing how the Secrets Store CSI Driver fetches secrets during pod startup. The participants are Pod, CSI Driver, Vault Provider, and HashiCorp Vault. The flow begins when the Pod sends a mount volume request to the CSI Driver. The CSI Driver asks the Vault Provider to fetch secrets. The provider authenticates to Vault and reads the requested values. Vault returns the secret values to the provider, the provider returns secret data to the CSI Driver, and the CSI Driver mounts the secrets as files into the Pod. A note explains that the pod starts only after the secrets are mounted. An alternate failure path shows the provider attempting to authenticate to Vault when Vault is unavailable. Vault returns a connection failure, the provider returns an error to the CSI Driver, the CSI Driver reports that the mount fails, and a note states that the Pod stays Pending. The diagram emphasizes that CSI secret retrieval is synchronous and directly coupled to pod startup."
485
497
/>
@@ -687,7 +699,8 @@ The __init container__ pattern makes sense when you need behavior that ESO and C
687
699
With three patterns to choose from, the decision comes down to two questions: how fresh do your secrets need to be, and what failure mode can your application tolerate?
@@ -728,38 +741,44 @@ With three patterns to choose from, the decision comes down to two questions: ho
728
741
}}
729
742
/>
730
743
744
+
The table captures raw capabilities, but choosing a pattern requires weighting those capabilities against your operational reality. A team with strong GitOps practices and tolerance for minutes of staleness will land in a different place than a team running payment infrastructure where a stale database credential means failed transactions. The deciding factor is usually the failure mode: ESO fails silently with stale data, CSI fails loudly by blocking pod startup, and init containers fail however you code them to. Each is the right answer for different risk profiles.
745
+
731
746
<List
732
-
variant="check-icons-list"
747
+
variant="card-grid-list"
733
748
items={[
734
749
{
735
750
lead: 'ESO',
736
-
text: 'fits most workloads. It\'s the simplest operationally: install the operator, create ExternalSecrets, and secrets flow into your cluster. GitOps works naturally because ExternalSecrets are declarative resources. You accept potential staleness — your secrets are only as fresh as your `refreshInterval`. For web applications, APIs, and most microservices, minutes of staleness is fine.',
751
+
text: 'Fits most workloads. It\'s the simplest operationally: install the operator, create ExternalSecrets, and secrets flow into your cluster. GitOps works naturally because ExternalSecrets are declarative resources. You accept potential staleness — your secrets are only as fresh as your `refreshInterval`. For web applications, APIs, and most microservices, minutes of staleness is fine.',
737
752
},
738
753
{
739
754
lead: 'CSI Driver',
740
-
text: 'fits workloads that need fresh secrets at startup and can tolerate blocking failures. Payment processing, credential rotation during incident response, or compliance requirements that prohibit caching secrets in Kubernetes — these are CSI use cases. You accept brittleness: when Vault is down, pods don\'t start.',
755
+
text: 'Fits workloads that need fresh secrets at startup and can tolerate blocking failures. Payment processing, credential rotation during incident response, or compliance requirements that prohibit caching secrets in Kubernetes — these are CSI use cases. You accept brittleness: when Vault is down, pods don\'t start.',
741
756
},
742
757
{
743
758
lead: 'Init containers',
744
-
text: 'fit edge cases where ESO and CSI don\'t provide enough control. Legacy applications that need secrets in unusual formats, multi-source fallback logic, or integration with systems that don\'t have ESO/CSI providers. You\'re writing and maintaining the logic yourself.',
759
+
text: 'Fits edge cases where ESO and CSI don\'t provide enough control. Legacy applications that need secrets in unusual formats, multi-source fallback logic, or integration with systems that don\'t have ESO/CSI providers. You\'re writing and maintaining the logic yourself.',
745
760
},
746
761
{
747
762
lead: 'Native secrets',
748
-
text: 'fit non-sensitive configuration and development environments. With encryption at rest configured, they\'re acceptable for data that doesn\'t require rotation or audit trails.',
763
+
text: 'Fits non-sensitive configuration and development environments. With encryption at rest configured, they\'re acceptable for data that doesn\'t require rotation or audit trails.',
749
764
},
750
765
]}
751
766
/>
752
767
753
768
<Diagram
754
769
src={patternDecisionDiagram}
770
+
size="full"
755
771
title="Pattern decision tree."
756
772
description="Flowchart guiding the choice between Kubernetes secret injection patterns based on failure tolerance. The process starts at Choose Secret Pattern and asks whether the application can tolerate minutes of staleness. If yes, the recommendation is ESO. If no, the chart asks whether the team accepts pod failures when the secret manager is down. If yes, the recommendation is CSI Driver. If no, the chart asks whether custom fallback logic is needed. If yes, the recommendation is Init Container. If no, the recommendation is ESO plus a short interval. The diagram highlights that the main decision points are tolerance for stale secrets, tolerance for blocked pod startup, and willingness to build custom fallback behavior."
757
773
/>
758
774
759
775
### Recommendations by Use Case
760
776
761
777
<List
762
-
variant="check-icons-list"
778
+
variant="zebra-list"
779
+
classes={{
780
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
781
+
}}
763
782
items={[
764
783
{
765
784
lead: 'Web applications and APIs',
@@ -797,7 +816,7 @@ Whichever pattern you choose, getting secrets into pods is only half the problem
797
816
Whichever pattern you choose, the security fundamentals remain the same: authenticate with short-lived credentials, scope access narrowly, and restrict network paths.
0 commit comments