feat(azure): sync Key Vault to AKS via External Secrets Operator

docs/CONFIGURATION.md

@@ -407,8 +407,39 @@ clusters:
 
 **Default:** `false` (safety checks are respected during upgrades)
 
+## Secret Management Options (Azure)
+
+### external_secrets_enabled
+
+Deploys the External Secrets Operator (ESO) to an AKS cluster and generates a per-site
+`ExternalSecret`, so Azure Key Vault secrets sync into native Kubernetes Secrets instead
+of being applied by hand. AKS only — AWS uses the Secrets Store CSI driver.
+
+```yaml
+clusters:
+  "20250115":
+    external_secrets_enabled: true
+    components:
+      external_secrets_version: "2.8.0"  # optional; chart version
+```
+
+Key Vault secrets follow the `<compound>-<site>-<field>` convention; the site's
+`ExternalSecret` selects `^<compound>-<site>-` and strips that prefix so the resulting
+Secret keys match what team-operator reads.
+
+**Before enabling on an existing cluster**, the Key Vault entries must already exist under
+the new names and match the live cluster values — otherwise the `ExternalSecret` (which
+owns the target Secret) will reproduce it incompletely. Some Key Vault secrets are created
+by PTD code and some must be created by hand.
+
+**Default:** `false`
+
+See [External Secrets on AKS](guides/external-secrets-aks.md) for the secret-ownership
+tables, the migration procedure, and the post-migration cleanup checklist.
+
 ## See Also
 
 - [Getting Started](GETTING_STARTED.md)
 - [CLI Reference](cli/PTD_CLI_REFERENCE.md)
+- [External Secrets on AKS](guides/external-secrets-aks.md)
 - [Examples](../examples/)

docs/README.md

@@ -23,6 +23,7 @@ Welcome to the Posit Team Dedicated (PTD) documentation.
 ### Infrastructure
 
 - [Kubernetes Guide](infrastructure/kubernetes.md) - Kubernetes-specific documentation
+- [External Secrets on AKS](guides/external-secrets-aks.md) - Key Vault → Kubernetes Secret sync, secret ownership (code vs. hand-created), and the existing-cluster migration procedure
 
 ### Misc
 - [Known Issues](KNOWN_ISSUES.md) - Known issues and rough edges

docs/guides/external-secrets-aks.md

@@ -0,0 +1,286 @@
+# External Secrets on AKS (Azure Key Vault → Kubernetes)
+
+This guide describes how PTD syncs Azure Key Vault secrets into native Kubernetes
+Secrets on AKS using the [External Secrets Operator (ESO)](https://external-secrets.io),
+and the one-time procedure for migrating an **existing** Azure workload cluster onto
+this model.
+
+## Overview
+
+On AKS, ESO replaces the previous approach of hand-applying Kubernetes Secrets (or
+baking Key Vault values into Secrets at deploy time). It is the AKS counterpart to
+the AWS Secrets Store CSI driver. team-operator is **unaware** of ESO — it continues
+to read native Secrets by name (`SecretType: kubernetes`), so no operator changes are
+required.
+
+ESO is installed by the `clusters` step when a cluster sets:
+
+```yaml
+clusters:
+  "<release>":
+    external_secrets_enabled: true
+```
+
+The step deploys the ESO controller into `posit-team-system` (authenticated to Key
+Vault via workload identity) and creates a cluster-scoped `ClusterSecretStore` named
+`azure-keyvault` pointing at the workload vault (`kv-ptd-<name[:17]>`).
+
+## Key Vault naming convention
+
+Product secrets are stored in Key Vault as **1:1 entries** (not JSON blobs) named:
+
+```
+<compound>-<site>-<field>
+```
+
+- `<compound>` — the workload (target) name, e.g. the value returned by `Target.Name()`.
+- `<site>` — the site name (e.g. `main`).
+- `<field>` — the Kubernetes Secret key the workload expects (e.g. `dev-db-password`).
+
+Example (illustrative): `<compound>-<site>-dev-db-password`.
+
+> **This convention is gated on `external_secrets_enabled`.** The `bootstrap` step uses
+> it only when at least one cluster in the workload has external secrets enabled;
+> otherwise it keeps the historical `<site>-<field>` name so vaults on unmigrated
+> workloads are left untouched. **Set the flag before running `bootstrap` on a new
+> workload** — otherwise bootstrap writes legacy names that the site's ExternalSecret
+> selector will not match.
+
+A per-site `ExternalSecret` selects everything under `^<compound>-<site>-` and rewrites
+the key to strip that prefix, so the resulting Secret keys match exactly what
+team-operator reads:
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: azure-keyvault
+  target:
+    name: <compound>-<site>-posit-team   # the Secret team-operator reads
+    creationPolicy: Orphan
+  dataFrom:
+    - find:
+        name:
+          regexp: "^<compound>-<site>-"
+      rewrite:
+        - regexp:
+            source: "^<compound>-<site>-(.*)"
+            target: "$1"
+```
+
+**The `<compound>-<site>-` prefix (not a bare `<compound>-` prefix) is required**: the
+infrastructure secrets (`<compound>-mimir-auth`, `<compound>-postgres-admin-secret`,
+`<compound>-grafana-postgres-admin-secret`, `<compound>-<release>-postgres-grafana-user`)
+also start with the compound name, and must **not** be swept into the product Secret.
+Those remain deploy-time (Pulumi) managed for now.
+
+> AWS stores the same data as a **single JSON blob** per site
+> (`<compound>-<site>.posit.team`); the ESO equivalent there uses `dataFrom.extract`
+> on that one secret rather than `find`.
+
+## Secret lifecycle policies
+
+The generated ExternalSecret uses `creationPolicy: Orphan` and the default
+`deletionPolicy: Retain`. Together these mean the Kubernetes Secret is adopted, kept
+updated, and pruned of keys with no Key Vault source — but it is never destroyed by ESO:
+
+| Event | Result |
+| --- | --- |
+| Key Vault value changes | Secret is updated on the next refresh |
+| Key has no Key Vault source | Key is pruned from the Secret |
+| Key Vault entry deleted | Secret is **retained** (`deletionPolicy: Retain`) |
+| ExternalSecret deleted (e.g. `external_secrets_enabled: false`) | Secret is **retained** (`creationPolicy: Orphan` sets no `ownerReferences`) |
+
+`creationPolicy: Owner` is ESO's default and would otherwise be the natural choice, but
+it sets an `ownerReference` on the Secret — so deleting the ExternalSecret, or removing
+the ESO CRDs, would garbage-collect a live product Secret and break the workload. The
+trade-off is that a Secret can outlive its ExternalSecret and go stale; deleting it is a
+deliberate manual step.
+
+### CRDs are retained
+
+The chart renders its CRDs as ordinary templates, so a rollback of a failed `Atomic`
+upgrade would delete them — and deleting a CRD cascade-deletes every `ExternalSecret` and
+`ClusterSecretStore` in the cluster. The release therefore sets
+`crds.annotations."helm.sh/resource-policy": keep` so Helm leaves the CRDs in place
+(the team-operator release does the same via `crd.keep`).
+
+If a `clusters` apply involving ESO does fail, run
+`ptd ensure <target> --only-steps clusters --refresh` before re-applying: a rollback can
+remove resources Pulumi still believes exist, and without a refresh they are not
+recreated — leaving Secrets present but unmanaged.
+
+## Key Vault secret ownership
+
+Every Key Vault secret is either **created by PTD code** or **created by hand/CLI**.
+Anything in the hand-created column will not exist on a new workload until someone adds
+it, and it is never regenerated — so it must be seeded during a migration and recreated
+during a rebuild.
+
+### Created by PTD code
+
+| Key Vault secret | Step | Notes |
+| --- | --- | --- |
+| `<compound>-<site>-dev-db-password` | `bootstrap` | random, `secrets.NewSiteSecret` |
+| `<compound>-<site>-keycloak-db-user` | `bootstrap` | derived from site name |
+| `<compound>-<site>-keycloak-db-password` | `bootstrap` | random |
+| `<compound>-<site>-pkg-db-password` | `bootstrap` | random |
+| `<compound>-<site>-pkg-secret-key` | `bootstrap` | `rskey` generated |
+| `<compound>-<site>-pub-db-password` | `bootstrap` | random |
+| `<compound>-<site>-pub-secret-key` | `bootstrap` | `rskey` generated |
+| `<compound>-postgres-admin-secret` | `persistent` | JSON `{fqdn,username,password}` |
+| `<compound>-mimir-auth` | `persistent` | random password |
+| `<compound>-<release>-postgres-grafana-user` | `postgres_config` | JSON `{database,password,role}`, per cluster |
+
+### Created by hand / CLI (never written by code)
+
+| Key Vault secret | Consumed by | Notes |
+| --- | --- | --- |
+| `<compound>-<site>-dev-license` | Workbench | license workflow |
+| `<compound>-<site>-pub-license` | Connect | license workflow |
+| `<compound>-<site>-pkg-license` | Package Manager | license workflow |
+| `<compound>-<site>-dev-admin-token` | Workbench (OIDC only) | often empty; see below |
+| `<compound>-<site>-dev-user-token` | Workbench (OIDC only) | often empty; see below |
+| `<compound>-workload-main-database-url` | team-operator (workload secret) | see "Workload-level secrets" |
+| `<compound>-grafana-postgres-admin-secret` | `postgres_config`, `helm` steps | **read-only in code** — the steps fail/warn if absent |
+| `ptd-dockerhub-username` | ACR pull-through cache | shared, not per-workload |
+| `ptd-dockerhub-oat` | ACR pull-through cache | shared, not per-workload |
+
+Notes:
+- `dev-admin-token` / `dev-user-token` are Workbench API tokens that team-operator only
+  **reads** (mounted read-only), and only when Workbench auth is OIDC. Key Vault cannot
+  store an empty value, so when they are unused the site's ExternalSecret emits them as
+  empty literals via `target.template` (`mergePolicy: Merge`) to preserve the Secret's
+  shape.
+- `home-auth-map` is deprecated and is **not** synced.
+
+### Workload-level secrets
+
+The Site CR also references a **workload** Secret (`workloadSecret.vaultName` →
+`<compound>-posit-team`) holding `main-database-url`. On AWS the `persistent` step writes
+this automatically; that code path is **AWS-only**, so on Azure it is hand-created.
+
+Its Key Vault entries use a `<compound>-workload-<field>` prefix — **not**
+`<compound>-<field>`. A field named `main-database-url` under the bare compound prefix
+would produce `<compound>-main-database-url`, which collides with the `^<compound>-main-`
+site selector for a site named `main` (the default) and would be swept into the site
+Secret. **`workload` is therefore a reserved site name.**
+
+> There is currently no ExternalSecret generated for the workload Secret — it remains
+> hand-applied. Seeding `<compound>-workload-main-database-url` prepares for that.
+
+### Kubernetes Secrets NOT sourced from Key Vault
+
+Do not attempt to bring these under ESO:
+
+| Secret | Created by |
+| --- | --- |
+| `<component>-connect-key`, `<component>-packagemanager-key`, `<component>-workbench-key`, `<component>-workbench-config` | **team-operator** generates and owns these (keys, launcher PEM, DSN config). Leave them alone. |
+| `azure-storage-account-<account>-secret` | `clusters` step, from the Azure Storage API |
+| `external-dns/azure-config-file`, `grafana/grafana-db-url`, `alloy/mimir-auth` | `helm` step — Pulumi reads Key Vault and writes a *transformed* value (e.g. a connection string), so these are not 1:1 syncs |
+| `<compound>-postgres-admin-secret` (in-cluster) | hand-applied today; could later use `dataFrom.extract` on the JSON blob |
+
+## Secret value format
+
+Key Vault values must be stored as **raw strings** — no surrounding JSON quotes and no
+trailing newline — because ESO syncs the bytes verbatim into the Kubernetes Secret. A
+quoted or newline-padded value would break consumers (e.g. a DB password carrying
+literal `"` characters). The `bootstrap` step stores strings verbatim; when setting
+values by hand, always use `--file` (see below) rather than `--value`.
+
+## Migrating an existing cluster
+
+For a cluster that is **already running** with live Secrets, the **cluster is the
+source of truth**. Do not let `bootstrap` regenerate the code-generated secrets under
+the new names — it would create fresh random values that don't match the live database
+and app state. Instead, seed the new-named Key Vault entries from the live cluster
+first (`CreateSecretIfNotExists` then no-ops).
+
+For each existing Azure workload cluster:
+
+1. **Enumerate** the site's live Secret keys (the `<compound>-<site>-posit-team` Secret
+   and any separate product Secrets team-operator reads).
+2. **Seed Key Vault from the cluster.** For every key — both code-generated and
+   external — copy the live value into `<compound>-<site>-<field>`. Copy exact bytes via
+   a temp file so there is no quoting, no newline stripping, and no plaintext in the
+   process arguments:
+   ```bash
+   umask 077; tmp=$(mktemp)
+   ptd workon <target> -- kubectl get secret <secret> -n posit-team \
+     -o jsonpath="{.data.<key>}" | base64 -d > "$tmp"
+   az keyvault secret set --vault-name <vault> --name "<compound>-<site>-<field>" \
+     --file "$tmp" --encoding utf-8 >/dev/null
+   rm -P "$tmp"
+   ```
+   Include licenses, and (OIDC sites) the Workbench `dev-admin-token` / `dev-user-token`.
+3. **Verify** each entry matches the cluster by hash (never print plaintext):
+   ```bash
+   # Key Vault side
+   az keyvault secret show --vault-name <vault> --name "<compound>-<site>-<field>" \
+     -o json | jq -j '.value' | shasum -a 256 | cut -c1-16
+   # Cluster side
+   ptd workon <target> -- kubectl get secret <secret> -n posit-team \
+     -o jsonpath="{.data.<key>}" | base64 -d | shasum -a 256 | cut -c1-16
+   ```
+   The two hashes must match before proceeding.
+4. **Enable and apply**: set `external_secrets_enabled: true` on the cluster and run
+   `ptd ensure <target> --only-steps clusters` (preview first).
+5. **Confirm the sync**: the `ExternalSecret` should report `Ready=True` /
+   `SecretSynced`, and the reproduced Secret should match the original key-for-key
+   (hash each key as in step 3).
+6. **Clean up**: see the checklist below.
+
+### Post-migration cleanup checklist
+
+Seed the new names *in parallel* and leave the old ones in place during the migration —
+reverting is then just "delete the new entries". Once the ExternalSecret is confirmed
+healthy and the reproduced Secret matches key-for-key, clean up:
+
+- [ ] **Old-named Key Vault product secrets** — the pre-migration `<site>-<field>` entries
+      (e.g. `main-dev-db-password`) that `<compound>-<site>-<field>` replaces.
+- [ ] **Old-named license entries** — the `-lic` spellings (e.g. `main-dev-lic`) replaced
+      by `<compound>-<site>-dev-license`.
+- [ ] **Any test/scratch Key Vault secrets** created while validating the sync (e.g. a
+      throwaway `…-findtest-*` prefix or a single-value sync probe).
+- [ ] **Any test ExternalSecrets and their target Secrets** in the cluster. Delete the
+      **ExternalSecret first** — otherwise ESO immediately recreates the Secret on its
+      next refresh and it looks like the delete failed.
+- [ ] **The `external-secrets` namespace**, if the cluster was first deployed with ESO
+      there before it moved to `posit-team-system`. Pulumi removes it on the next
+      `clusters` apply; confirm it is gone.
+- [ ] **Verify nothing else matched the selector** — list the Key Vault entries matching
+      `^<compound>-<site>-` and confirm each one is an intended product key:
+      ```bash
+      az keyvault secret list --vault-name <vault> \
+        --query "[?starts_with(name,'<compound>-<site>-')].name" -o tsv
+      ```
+
+Azure Key Vault deletes are **soft deletes**: the entries remain recoverable (and their
+names reserved) for the vault's retention period. Purge only if a name must be reused
+immediately.
+
+### Greenfield clusters
+
+New clusters need no migration, provided `external_secrets_enabled: true` is set
+**before** the first `bootstrap` run — that is what selects the
+`<compound>-<site>-<field>` naming. `bootstrap` then generates the code-generated
+secrets under the correct names. (If bootstrap already ran without the flag, the
+secrets exist under `<site>-<field>`; because `CreateSecretIfNotExists` never
+overwrites, enabling the flag and re-running creates a *second* set under the new
+names with fresh random values — seed those from the cluster as in the migration
+above, then delete the legacy entries.)
+
+Everything in
+[Created by hand / CLI](#created-by-hand--cli-never-written-by-code) must still be added
+to Key Vault manually — notably the three licenses, the workload
+`main-database-url`, and `<compound>-grafana-postgres-admin-secret` (the `postgres_config`
+and `helm` steps only ever *read* that one, so a missing entry surfaces as a step failure
+or a warning rather than being created for you).
+
+## See also
+
+- External Secrets Operator docs: <https://external-secrets.io>
+- Azure Key Vault provider: <https://external-secrets.io/latest/provider/azure-key-vault/>

lib/azure/secretstore.go

@@ -61,7 +61,10 @@ func (s *SecretStore) EnsureWorkloadSecret(ctx context.Context, credentials type
 	// Create a KeyVault secret for each populated field in the map since Azure Secret Provider
 	// doesn't support json blobs, we must create a KV entry for each field.
 	for fieldName, fieldValue := range secretMap {
-		fieldValueStr := fmt.Sprintf("%v", fieldValue)
+		fieldValueStr, err := encodeSecretValue(fieldValue)
+		if err != nil {
+			return fmt.Errorf("failed to encode field %s: %w", fieldName, err)
+		}
 		if fieldValueStr == "" {
 			continue
 		}
@@ -102,14 +105,29 @@ func (s *SecretStore) CreateSecretIfNotExists(ctx context.Context, credentials t
 		return err
 	}
 
-	mSecret, err := json.Marshal(secret)
+	value, err := encodeSecretValue(secret)
 	if err != nil {
 		return err
 	}
 
 	if !s.SecretExists(ctx, azureCreds, secretName) {
-		return createSecret(ctx, azureCreds, s.vaultName, secretName, string(mSecret))
+		return createSecret(ctx, azureCreds, s.vaultName, secretName, value)
 	}
 
 	return
 }
+
+// encodeSecretValue renders a secret payload for storage in Azure Key Vault.
+// Strings are stored verbatim: External Secrets syncs Key Vault values byte-for-byte
+// into Kubernetes Secrets, so json.Marshal's surrounding quotes would corrupt them.
+// Non-strings are still JSON-encoded (e.g. the {fqdn,username,password} DB secret).
+func encodeSecretValue(secret any) (string, error) {
+	if str, ok := secret.(string); ok {
+		return str, nil
+	}
+	b, err := json.Marshal(secret)
+	if err != nil {
+		return "", err
+	}
+	return string(b), nil
+}