fix(gcp): GKE rebuilds that can repeat

[Smana]

Platform fix — merged when green and reviewed, after G-1 (#2123) (owner, 2026-09-29; GP-21). Stacked on fix/openbao-stage2-gcp; GitHub retargets it to main once #2123 merges.

Every change below is a bug a live gcp-0 rebuild hit. Together they make a GKE teardown-and-rebuild repeatable.

What changes

Bug	Symptom on a rebuild	Fix	Commit
09-11 bug 2	A destroyed custom role's ID is reserved for 37 days (undelete only for 7), so a rebuild inside that window cannot recreate xplane_* roles	Role IDs take a generation suffix (_v3); the destroy state rms the roles, the deploy adopts them (scripts/ops/gcp/adopt-custom-roles.sh). external-dns reads the role from the new gcp_dns_editor_role ConfigMap key	414075c, 601bd3a
09-11 bug 3	Stage 2's inline gke/configure apply bypasses the configure script: no .tls/ca.pem, and a restored jwt/gcp-0 mount keeps the destroyed cluster's issuer	Stage 2 writes the CA and adopts jwt/gcp-0 before its apply	03c1887
missing deploy_identity_provider	The same inline apply never passed it, so it published the consumed (AWS) identity_provider_url until the standalone configure run	Passed on the inline apply and its jwt-adopt call	03c1887
09-11 bug 6	gcp-0's private external-dns waited on aws-load-balancer-controller, which gcp-0 never runs: no private record was ever written	gcp-0 overlay removes that dependsOn	29d5a68
09-11 bug 7	Crossplane's grant allowlist lacked objectCreator, so the OpenBao snapshot identity's bucket grant was denied: no snapshots	Role added to the allowlist	f69a557
LB orphans (GP-22)	A deleted GKE cluster leaves a forwarding rule, a target pool and k8s-* firewall rules in no tofu state; the firewall rules block the VPC delete	teardown.sh reports them; on a confirmed teardown it runs the new guarded sweep-lb-orphans.sh --apply, and retries the destroy once only if the network still stands	cac5a07, 8e44f93

81524a7 restores the dated incident (2026-09-16, #2044) behind stage 2's flux-operator state rm comment.

Custom-role hardening (601bd3a), from review of 414075c:

• The destroy's state rm runs only on the roles actually in state, with -lock-timeout=5m and no || true. A state rm lost to a lock, backend or auth error now stops the destroy, instead of letting the cluster destroy delete the roles and burn their IDs.
• adopt-custom-roles.sh treats only NOT_FOUND as absent. Any other describe error (expired auth, invalid_rapt) exits 1 rather than letting the apply create a role that exists.
• The [absent ] line names the burned-ID case: a role deleted more than 7 days ago is NOT_FOUND but its ID stays reserved, so if the create fails, bump custom_role_suffix.
• custom_role_suffix has no default: a run without variables.tfvars fails instead of planning the live roles' replacement.

scripts/ops/tasks.yaml now indexes both new operator scripts: ops:gcp:sweep-lb-orphans and ops:gcp:adopt-custom-roles.

Teardown and sweep flow

flowchart TD
    A[teardown.sh] --> B[terramate --reverse --continue-on-error destroy]
    B --> C{TM_CLOUD wants gcp<br/>and TM_DESTROY_CONFIRMED=true?}
    C -- no --> V
    C -- yes --> D[sweep-lb-orphans.sh --apply]
    D --> E{any GKE cluster in the project,<br/>in any state, or list failed?}
    E -- yes --> R[exit 3: nothing swept,<br/>clusters named] --> N
    E -- no --> F["delete project-wide forwarding rules<br/>whose description carries kubernetes.io/service-name"]
    F --> G[delete target pools<br/>same description marker]
    G --> H["delete k8s-* firewall rules<br/>on vpc-europe-west4-dev only"]
    H --> N{network vpc-europe-west4-dev<br/>still describes?}
    N -- yes --> J[retry destroy once<br/>shared with the AWS lane]
    N -- no --> V
    J --> V[verify against GCP:<br/>clusters, instances, forwarding rules,<br/>target pools, k8s-* firewall rules, disks]
    V --> K{anything left<br/>or unverifiable?}
    K -- yes --> X[exit non-zero]
    K -- no --> Y[exit 0]

Loading

• The sweep is dry-run unless --apply.
• It refuses (exit 3) while any GKE cluster exists in the project, or while clusters cannot be listed.
  • The forwarding-rule and target-pool lists are project-wide: GKE's description marker is the same for every cluster. So a second cluster's live LoadBalancers would be in them.
  • Only the firewall list is scoped to the network.
  • --cluster only names the teardown in the output.
• The retry mirrors the AWS lane's VPC check.
  • It runs only while the network still describes, so a clean teardown does not re-run the whole destroy.
  • Both lanes share one retry, so TM_CLOUD=all destroys at most twice.
• A retry can fail against stacks the first pass already destroyed (typically gcp/openbao/cluster's pre-destroy snapshot of a gone node). The snapshot is deliberately not auto-skipped, since the first pass may never have reached that stack. When the exit is non-zero but the cloud verifies clean, the summary prints a note: re-run with TM_OPENBAO_SKIP_SNAPSHOT=true only if the first pass's snapshot succeeded. The same note is in website/content/docs/reference/commands.md.

What a live apply does

google_project_iam_custom_role.crossplane_{dns,storage,role_reader} keep their addresses; only the ForceNew role_id changes.

Starting point	adopt-custom-roles.sh	tofu apply
Torn-down stack, empty state (expected after the reset)	[absent ]	creates the three _v3 roles
_v3 roles live, out of state (after a teardown by the new destroy)	imports each	no-op on roles
_v3 role soft-deleted ≤ 7 days	[deleted]: the apply's create undeletes it	provider undeletes on create
_v3 role deleted > 7 days ago (describe reads NOT_FOUND, ID still reserved up to 37 days)	[absent ], naming the bump	create fails: bump custom_role_suffix to _v4
State still holds unsuffixed / _v2 roles	[ok] (checks the address, not the ID)	replaces each role and its bindings

• A describe error other than NOT_FOUND (expired auth, invalid_rapt) stops the deploy at the adopt step, before any apply.
• Kept roles are not inert: Crossplane's ProjectIAMMember for ns/kube-system/sa/external-dns outlives the cluster, so that grant stays live until gcp-0 returns.
• preview does not adopt (it must not mutate state), so after a teardown it shows the three roles as creates that deploy will import.
• gke/configure now reads dns_editor_role from init's state; a configure plan against an init state from older code fails until init is re-applied (deploy order already applies init first).

Gates

Command	Result
task check (go-task 3.53.1)	exit 0
↳ validate-manifests.sh	RENDER: overlays=95 charts=70 standalone=108 failed=0; Valid: 2093, Invalid: 0, Skipped: 0
↳ scripts/ci/tests/run.sh	36 passed, 1 skipped, 0 failed (skip: test-vector-vrl, no vector binary)
↳ validate-links.sh / validate-doc-claims.sh / verify-doc-paths.sh	all links resolve / 30 claims match (51 page checks) / every path exists
bash scripts/ci/tests/test-teardown-gcp-sweep.sh	PASS. Against cac5a07's scripts it fails on "another cluster alive" (exit 0, LBs deleted) and "a clean teardown with the network gone" (2 destroy runs, not 1). The failed-list and filter-scoping cases already held there; each fails on a one-line mutant that removes the guard or a filter
bash scripts/ci/tests/test-adopt-custom-roles.sh	all checks passed
python3 scripts/ci/tests/test-gcp-gke-init-workflow.py	PASS
bash scripts/ci/tests/test-terramate-script-refs.sh	99 references checked, 0 failed
shellcheck -x -S warning on sweep-lb-orphans.sh, teardown.sh, adopt-custom-roles.sh and the sweep test	exit 0
tofu fmt -check opentofu/gcp/gke/init	exit 0 (this round's HCL changes are comments)
trivy config --exit-code=1 on opentofu/gcp/gke/init	exit 0
tofu -chdir=opentofu/gcp/gke/configure validate	Success! The configuration is valid.

Tests stub gcloud, terramate and aws; nothing here ran against GCP. The teardown test runs the real teardown.sh with TM_DESTROY_CONFIRMED=true, so its setup fails closed and it aborts before every invocation unless the stub directory is absolute and each CLI resolves to, and executes as, its stub. Proved with logging decoys on PATH: an unguarded version reached a decoy terramate destroy 4 times; with a relative TMPDIR, the version before 1a081247 still leaked 3 (teardown.sh cds, so a relative stub entry stops resolving); the current test: 0.

ADR

None: these are fixes, not a technology choice.

Live evidence

Phase 8, after the 2026-09-29 21:00 reset: a gcp-0 deploy → teardown → rebuild from integration/agent-factory.

• Stage 2 writes opentofu/gcp/gke/configure/.tls/ca.pem and adopts jwt/gcp-0
• _v3 roles created, then adopted on the rebuild
• private external-dns writes records on gcp-0
• an OpenBao snapshot lands in the bucket
• teardown.sh reports Target pools / k8s-* firewall rules, the sweep clears them, and the VPC deletes

[github-actions]

🔍 Rendered manifest diff — this PR vs main (desired state)

2 changed · 0 added · 0 removed

Rendered with kustomize build + helm template (source of truth = git), so Helm-expanded workloads are included. Shows what Flux will apply — not a diff against live cluster state (drift is alerted on separately), and not CRD-defaulted / webhook-mutated output. Secret values are redacted; per-render noise (webhook caBundles, checksum/* annotations, render timestamps) is normalized out.

🟡 changed — cloud.ogenki.io/v1alpha1/GCPWorkloadIdentity/kube-system/xplane-external-dns

--- a/cloud.ogenki.io/v1alpha1/GCPWorkloadIdentity/kube-system/xplane-external-dns
+++ b/cloud.ogenki.io/v1alpha1/GCPWorkloadIdentity/kube-system/xplane-external-dns
@@ -5,6 +5,6 @@
   namespace: kube-system
 spec:
   roles:
-  - projects/ogenki-435905/roles/xplane_dns_editor
+  - projects/ogenki-435905/roles/xplane_dns_editor_v3
   serviceAccount:
     name: external-dns

🟡 changed — helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/external-dns

--- a/helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/external-dns
+++ b/helm.toolkit.fluxcd.io/v2/HelmRelease/kube-system/external-dns
@@ -12,9 +12,6 @@
         name: external-dns
         namespace: kube-system
       version: 1.22.0
-  dependsOn:
-  - name: aws-load-balancer-controller
-    namespace: kube-system
   driftDetection:
     mode: enabled
   install: