fix(gcp): a GCP-hosted ZITADEL keeps its consumers in step

[Smana]

Summary

A GCP-hosted ZITADEL now keeps its consumers in step. On a fresh directory, gcp/gke/init's hosting stage 3 does four things:

1. registers the Google IdP and the groups Action;
2. registers the OIDC clients;
3. mirrors every client secret into the OpenBao path its ExternalSecret reads, then force-syncs those ExternalSecrets;
4. rotates OpenBao's own auth/oidc.

A new stage 5 then halts the deploy if OpenBao's OIDC client has drifted. This closes the known G-1 gap: a ZITADEL rotation left GCP OpenBao OIDC stale.

It fixes three bugs:

• 09-11 bug 9: the sync wrote Secret Manager only, while gcp-0 reads OpenBao.
• Bug 8: zitadel_project_id was committed per directory. A GCP-hosted project id now reaches gke/configure through Secret Manager zitadel-project-id.
• GP-20: the chart's fresh admin PAT wins.

The stage-3 and stage-5 changes are inert while AWS is primary. main keeps primary_cloud = "aws", so the hosting branch does not run and stage 5 prints its skip. GP-20 is live on both clouds. For a seed-restored aws-0 it is a no-op.

Commit	What
1fc5c619 refactor(secrets): one managed-store to OpenBao map, shared	scripts/lib/bao-map.sh, read by secret-store.sh migrate and the new mirror
b91e295c feat(zitadel): mirror consumer secrets into the OpenBao path they are read from	--mirror-openbao (requires --openbao-url, runs only under --apply)
ef915a1b fix(zitadel): the OpenBao mirror repairs itself, writes only what it owns, and fails loudly	Mirrors on the already-converged path too; MIRRORED_FIELDS only onto an existing value; a failed mirror makes the run exit 1
6ba74878 fix(secrets): store_write refuses a short payload write	A failed mktemp, cat or jq no longer ships a truncated payload to the cloud
5917a4b6 fix(gcp): a hosted ZITADEL's project id reaches gke/configure through Secret Manager	Bug 8
d8d39323 fix(zitadel): the chart's fresh admin PAT wins over a stored one	GP-20
21c2788e fix(zitadel): a consumer never overwrites the IdP cloud's PAT, and the project id is written only on change	resolve_zitadel_pat hosting|consuming; publish_project_id compares before it writes
95fdd664 fix(gcp): a hosting stage 3 registers the IdP, then mirrors every client into OpenBao	Stage-3 wiring; the #2078 guard is split into hosting and consumer
ac96645b fix(gcp): mirror an absent OpenBao path in full, print a complete recovery, and halt on OIDC drift	Final-review round, detailed below

Final-review round (ac96645b)

• I-1. A 404 now gets the full store payload. migrate skips a path that exists. So an owned-fields-only write left grafana-envvars without GF_SECURITY_ADMIN_USER/PASSWORD, and grafana-operator could never authenticate. The ownership filter still protects an existing value.
• I-2. The flag lists live in IDP_SYNC_ARGS and CLIENT_SYNC_ARGS. The real calls and the ZITADEL-not-ready recovery both expand them, so the printed recovery is the IdP sync, then the clients sync with every OpenBao flag. sso.md now gives gcp-0's hosting and consuming cases, and states GP-20.
• Stage 5. gcp/gke/init gains stage5-verify-openbao-oidc: openbao-oidc-check.sh --cloud gcp --project … is its last statement, so exit 1 and exit 2 both halt. It keeps aws-0's job name, so feat(openbao): reconcile the OIDC client after ZITADEL rotates it #2078's contract suite guards both clouds (flags known to the parser, the call is the last statement). A consuming gcp-0 skips it and says why. The stale "There is no stage 3" header is rewritten.
• Minors.
  • The project-id failure message now says what really happens.
  • store_write adds the version when secrets create reports ALREADY_EXISTS.
  • The clients script's header warns that kubectl must point at the hosting cluster.
  • A consumer no longer needs kubectl on aws-0.
  • Mirrored paths' ExternalSecrets are force-synced: warn-only, and only under --apply.

Stage 3 and stage 5, hosting branch

flowchart LR
  S2["stage 2<br/>writes gke/configure/.tls/ca.pem"] --> W["wait for ZITADEL"]
  W -- "not ready" --> R["print recovery:<br/>IdP sync + clients sync<br/>from the same arg arrays"]
  W --> IDP["zitadel-idp.sh sync IDP_SYNC_ARGS<br/>Google IdP + groups Action<br/>(PAT resolved as hosting)"]
  IDP --> CL["zitadel-oidc-clients.sh sync CLIENT_SYNC_ARGS<br/>--openbao-* / --mirror-openbao"]
  CL --> SM[("Secret Manager<br/>client secrets, zitadel-project-id")]
  CL --> MIR["mirror: full payload if absent,<br/>else MIRRORED_FIELDS"]
  CL --> OIDC["reconcile_openbao_oidc<br/>auth/oidc config + role"]
  MIR --> BAO[("OpenBao gcp-0<br/>bao.priv.gcp.ogenki.io")]
  OIDC --> BAO
  MIR --> FS["force-sync ExternalSecrets<br/>on mirrored paths"]
  BAO --> S5{"stage 5<br/>openbao-oidc-check.sh"}
  S5 -- "exit 1 / 2" --> HALT["deploy halts"]

Loading

The consumer branch (--idp-cloud) is unchanged, and a guard keeps it that way: it passes no --openbao-* flag and no --mirror-openbao.

When stage 5 halts (e.g. ZITADEL not ready within 45 min, or stage 3 exiting early on a failed get-credentials), the deploy ends red, as on aws-0, and the standalone gke/configure apply after gke/init does not run. After the printed recovery, re-run the deploy (or at least gke/configure) so the vars ConfigMap takes the newly published zitadel_project_id. On a first-ever build, also run secret-store.sh grant for headlamp-oauth2-proxy, which the recovery text does not print.

GP-20: the fresh PAT wins only on the hosting cloud

• Hosting (IDP_CLOUD == CLOUD, and always for zitadel-idp.sh): security/iam-admin-pat is read first. It overwrites the stored zitadel-iam-admin-pat when they differ.
• Consuming: the IdP cloud's store is read, never kubectl, and it is never written. An empty store fails and says to run the hosting sync first.
• Both scripts' headers say that a hosting run must point kubectl at the IdP-hosting cluster. Otherwise a leftover security/iam-admin-pat elsewhere would overwrite that cloud's stored PAT.

Owner pre-flight (P-11)

Before the first GCP deploy, delete the stale Secret Manager zitadel-iam-admin-pat. GP-20 only adds a new version, so the stale one stays enabled and readable by version number.

Live objects --apply mutates

Code path	Object	Mutation
resolve_zitadel_pat hosting (both scripts)	GCP Secret Manager zitadel-iam-admin-pat / AWS zitadel/iam-admin-pat	New version when it differs from the cluster token; created if absent
same	in-cluster security/iam-admin-pat	Read only
zitadel-idp.sh sync	ZITADEL: instance Google IdP, the groups Action, flow 2 (CustomiseToken) triggers	Created if missing; the IdP's client id is corrected in place
zitadel-oidc-clients.sh sync	ZITADEL project, roles and apps; Secret Manager client-secret keys	Created or converged
publish_project_id (GCP hosting)	Secret Manager zitadel-project-id	Created with managed-by=zitadel-oidc-clients; new version only on change
gke/configure	ConfigMap key zitadel_project_id	From the secret, when this cluster hosts ZITADEL and it exists
--mirror-openbao	OpenBao KV platform/*, apps/* (per bao-map.sh)	Full payload onto an absent path; MIRRORED_FIELDS patched onto an existing one
force_sync_mirrored	ExternalSecrets whose openbao-<mount> store and key match a mirrored path	force-sync=<epoch> annotation (--overwrite); a failure only warns
reconcile_openbao_oidc	OpenBao auth/oidc/config, auth/oidc/role/<role>	Client id, secret and bound_audiences rotated to what ZITADEL issued
stage 5	none	Read only; halts the deploy on exit 1 or 2

Merge class

Platform fix: merged when green and reviewed (owner, 2026-09-29; GP-21).

No ADR: fixes, not a technology choice.

Gates

The branch was rebased onto origin/main 5dc3e23e (#2124, a victoria-metrics chart bump) before this push, and every gate was re-run after the rebase.

Command	Result
test-zitadel-oidc-clients-mirror.sh	Red first against 71c56398 (404 wrote only GF_AUTH_GENERIC_OAUTH_CLIENT_ID; force_sync_mirrored missing). Now exit 0, 22 ok
test-cloud-secret-store.sh	Red first: 2 FAIL (ALREADY_EXISTS). Now exit 0, 38 ok
test-gcp-gke-init-workflow.py	Red first: 25 FAIL. Now PASS. Five workflow mutants are all caught
test-zitadel-oidc-clients-openbao.sh	exit 0, 285 ok, including gcp-0's stage-5 flag and last-statement guards
10 zitadel suites, 10 openbao/bao suites, 3 secret-store suites, test-no-secret-argv.sh	24/24 exit 0, including test-openbao-oidc-check.sh with 91 ok
test-terramate-script-refs.sh	exit 0, 102 refs checked, 0 failed
./scripts/ci/validate-manifests.sh (via task check)	Valid: 2096, Invalid: 0, Skipped: 0
validate-links.sh / verify-doc-paths.sh / validate-doc-claims.sh	exit 0 / exit 0 / 30 claims match
validate-idp-topology.sh	aws hosts, all other clouds suspended
shellcheck -x -S warning (7 touched scripts)	exit 0
terramate list	TM-OK; script info deploy lists stage5-verify-openbao-oidc for both clouds
task check	exit 0; ci:test 39 passed, 1 skipped (vector not installed), 0 failed

Live evidence

Placeholder for Phase 8.3–8.4 (gcp-0 hosting deploy from integration/agent-factory):

• Stage 3 log: IdP + Action registered, then clients, with no [warn]; [synced ] lines for the mirrored ExternalSecrets
• Stage 5: openbao-oidc-check.sh exit 0
• grafana-envvars in OpenBao carries GF_SECURITY_ADMIN_USER/PASSWORD, and grafana-operator authenticates
• An ExternalSecret reading a mirrored platform/* path is SecretSynced
• OpenBao UI SSO login through https://auth.gcp.cloud.ogenki.io

[github-actions]

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅