Skip to content

fix(openbao): Stage 2 on GCP's OpenBao - #2123

Merged
Smana merged 9 commits into
mainfrom
fix/openbao-stage2-gcp
Sep 29, 2026
Merged

Smana merged 9 commits into
mainfrom
fix/openbao-stage2-gcp

Conversation

@Smana

@Smana Smana commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes gcp_only_broken_since_stage2: GCP's gcp/openbao/management stack only had lineage + pki, so GCP's OpenBao never got Stage 2's mounts, policies or JWT logins, and anything on gcp-0 reading platform/ or apps/ failed. Salvaged from ac62abf2 (verified live 2026-09-11) and rebuilt on main's current semantics.

Change Where
Stage 2 store of record as a shared module (platform/ + apps/ mounts, five policies, JWT/userpass/OIDC logins, #2078's ignore_changes guards), tested offline with tofu test opentofu/shared/modules/openbao-store-of-record/
GCP's management stack calls the module; OIDC is gated list-before-read, so a missing openbao-oidc secret means OIDC off, never an error or a destroyed mount opentofu/gcp/openbao/management/store-of-record.tf
Policy-parity gate: every policy a JWT role names must be defined on that cloud, and the module's policies must stay byte-identical to AWS's copies. Wired into CI and task check scripts/ci/validate-openbao-policies.sh
secret-store.sh migrate --keys: migrate named keys without a cluster, for a cluster already pointed at OpenBao scripts/provision/secret-store.sh
New-lineage switch OPENBAO_NEW_LINEAGE=true, opt-in, off by default scripts/provision/openbao-config.sh, failover guide

AWS is untouched: no file under opentofu/aws/ changes, and the module's policies are byte-identical to AWS's. The agents/ mount is not here; it arrives with G-5.

flowchart LR
  M["shared module<br/>openbao-store-of-record"] --> G["gcp/openbao/management<br/>store-of-record.tf"]
  A["aws/openbao/management<br/>(inline, unchanged)"] -. "byte-identical policies<br/>(parity gate)" .- M
  S[("Secret Manager<br/>openbao-oidc")] -- "list, then read<br/>(absent → OIDC off)" --> G
  G --> B["GCP OpenBao<br/>platform/ apps/ · JWT · userpass · OIDC"]
  R["openbao-config.sh rehydrate"] -->|default: restore lineage| B
  R -. "OPENBAO_NEW_LINEAGE=true<br/>(refused on any own-seal object)" .-> B
Loading

The new-lineage switch. It lets rehydrate initialise a new lineage on a node whose seal no top-level snapshot carries. It is refused when an object under the node's own seal exists, when combined with a named OPENBAO_SNAPSHOT_KEY, or when any top-level snapshot's seal is unknown, and it never overrides the moved-aside refusal (rc=2). After a new-lineage init with surviving tofu state, the break-glass user must be force-recreated (-replace=module.store_of_record.vault_generic_endpoint.admin_user), because disable_read hides that it is gone; the switch prints this and the guide states it. The owner chose to restore the existing GCP lineage (GP-22), so the switch is not exercised now.

Platform fix: merged when green and reviewed (owner, 2026-09-29; GP-21). No ADR: this is a shared-module code layout, not a technology choice between competing options.

Known gap between this PR and G-3

Until G-3 merges, a ZITADEL client rotation leaves GCP OpenBao's OIDC stale (App.NotFound) until zitadel-oidc-clients.sh sync is re-run with --openbao-url pointed at GCP. Userpass break-glass is unaffected. The comment at scripts/provision/zitadel-oidc-clients.sh:111-113 ("OpenBao OIDC exists only on AWS") is now false; G-3 rewrites it when it wires the GCP sync.

Gates run

Gate Result
module tofu test 3 passed, 0 failed
validate-openbao-policies.sh (roles → policies, AWS ↔ module byte parity) exit 0
test-validate-openbao-policies.sh, test-secret-store-migrate-keys.sh (10/10), test-openbao-new-lineage.sh, test-openbao-oidc-lifecycle.sh exit 0
validate-manifests.sh Invalid 0, Skipped 0
validate-links.sh, verify-doc-paths.sh exit 0
task check (go-task 3.53.1) exit 0; ci:test 30 passed, 1 skipped (test-vector-vrl, no vector binary), 0 failed
tofu validate + trivy config on gcp/openbao/management exit 0

Live evidence

To be filled in by Task 8.3 (restore of the existing GCP lineage, Stage 2 mounts present, migrate --keys, SSO and userpass login).

@github-actions

Copy link
Copy Markdown
Contributor

🔍 Rendered manifest diff — this PR vs main (desired state)

No changes to the rendered desired state. ✅

Comment thread opentofu/shared/modules/openbao-store-of-record/oidc.tf Dismissed
Smana and others added 2 commits September 29, 2026 10:43
… guard shared-policy parity

A new-lineage init leaves vault_generic_endpoint.admin_user's disable_read=true
hiding the fact that no apply will ever recreate the break-glass user again --
the proceed arm and the failover guide now both name the required
tofu apply -replace=module.store_of_record.vault_generic_endpoint.admin_user.

Also: drop two citations to a design doc and an ADR that do not apply here
(salvage provenance is ac62abf2 instead), and add a cmp guard so the shared
module's five policy files can never silently drift from AWS's copies.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011xuNsQcGpQPcEH8m1h3rDD
The printed and documented -replace command omitted -var-file, so
secret_owning_apps fell back to [] and the apply would destroy every app
persona; it also lacked the stack's -parallelism=1. It now runs after the
deploy finishes, not mid-deploy. The policy parity gate also treats a copy
missing on one side as divergence, so deleting one no longer passes.
@Smana
Smana merged commit 8e3f22a into main Sep 29, 2026
12 checks passed
@Smana
Smana deleted the fix/openbao-stage2-gcp branch September 29, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants