fix(zitadel): restart env-at-start OIDC consumers after a client rotation

[Smana]

Why

After an OpenBao rebuild, zitadel-oidc-clients.sh sync rotates the ZITADEL OIDC client (#2078), but a Deployment that reads its client id from env keeps the old id until its pods restart. headlamp-oauth2-proxy (namespace tooling) showed "App not found" until someone ran kubectl rollout restart deployment headlamp-oauth2-proxy -n tooling, and every resource reported healthy the whole time.

What changes

restart_rotated_consumers runs at the end of cmd_sync, right after force_sync_mirrored. That covers every stage-3/4 call site on both clouds (aws/eks/init stage 4, gcp/gke/init stage 3, hosting and consuming), with no workflow change.

flowchart LR
    loop["consumer loop"] -->|"stored client id != ZITADEL's"| rot["rotated: key=new id"]
    rot --> ctx{"kubectl on --cluster?<br/>(*-CLUSTER-vars CM)"}
    ctx -- no --> warn["WARN, list keys"]
    ctx -- yes --> es["force-sync every ExternalSecret<br/>reading the key"]
    es --> wait{"Secret carries new id?<br/>(one 180s deadline)"}
    wait -- no --> warn2["WARN + manual command"]
    wait -- yes --> rs["rollout restart each Deployment<br/>with env/envFrom on that Secret"]

Loading

Property	How
Scoped to real rotations	A key counts as rotated only when it held a client id and this run replaced it with a different one, on the create path or the converge path. A first bootstrap (no previous id) restarts nothing, because those pods start once the Secret appears.
Idempotent	A re-run with nothing rotated passes no keys and makes no kubectl calls.
Right cluster only	Requires kubectl to see flux-system/*-<cluster>-vars. aws/eks/init's consuming sync for gcp-0 runs with kubectl on aws-0, so it only warns there.
New pods read the new id	Force-syncs each ExternalSecret reading the key, then waits until the Secret holds the new id before restarting.
Readers found, not listed	Deployments in the Secret's namespace that reference it via env[].valueFrom.secretKeyRef or envFrom[].secretRef. Volume mounts are excluded.
Non-fatal	Warn-only, like force_sync_mirrored: the clients are already written.

ExternalSecret matching covers both store shapes: OpenBao at the key's bao-map.sh path (headlamp-envvars, grafana-envvars) and clustersecretstore under the bare key (the unmapped headlamp-oauth2-proxy).

Env-at-start readers this reaches today:

Cluster	Deployment	Secret
gcp-0	tooling/headlamp-oauth2-proxy	headlamp-oauth2-proxy (secretKeyRef)
aws-0	tooling/headlamp	headlamp-envvars
both	Grafana, observability	victoria-metrics-k8s-stack-grafana-envvars (envFromSecret)

Not covered: Harbor and the Flux UI take their client through HelmRelease valuesFrom, so a refresh reaches them through a Helm upgrade and not through a Deployment env reference. OpenBao is reconciled through its API by reconcile_openbao_oidc.

Known gap

With AWS as primary, gcp-0's suffixed clients are rotated by aws-0's stage 4, where kubectl points at aws-0. That run only warns. gcp-0's own stage 3 runs the sync later, sees the store already converged, and restarts nothing. Closing this needs the AWS stage to reach gcp-0's kubeconfig, which is out of scope here.

Evidence

• New scripts/ci/tests/test-zitadel-oidc-clients-restart.sh: 14/14 ok. It runs the shipped function bodies against a kubectl stub. Two mutants are killed: dropping the context guard, and dropping the Secret-refreshed check.
• test-zitadel-oidc-clients-convergence.sh gains a rotation case: headlamp-envvars=existing-client-id reaches the restart, while a same-id converge passes nothing.
• bash scripts/ci/tests/run.sh: 40 passed, 1 skipped (vector not installed), 0 failed
• shellcheck -x scripts/provision/zitadel-oidc-clients.sh: clean
• validate-manifests.sh (MemoryMax=6G): exit 0, Valid: 2096, Invalid: 0, "All gates passed"
• validate-doc-claims.sh: 30 claims match. validate-links.sh and verify-doc-paths.sh: clean
• trivy config --exit-code=1 --ignorefile=./.trivyignore.yaml scripts/: exit 0
• tofu validate: not applicable, because no .tf or .tm.hcl changed (the change is in the script the stages call). No terramate or tofu apply was run.

No ADR: this is a fix, not a technology choice.

[github-actions]

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅