diff --git a/docs/design/allnamespaces-install-mode_design.md b/docs/design/allnamespaces-install-mode_design.md new file mode 100644 index 00000000000..d8c34de17f4 --- /dev/null +++ b/docs/design/allnamespaces-install-mode_design.md @@ -0,0 +1,294 @@ +# Enable AllNamespaces Install Mode for OADP Operator + +## Abstract + +- Enable `AllNamespaces` install mode alongside the existing `OwnNamespace` mode in a single CSV. +- The operator's runtime behavior is unchanged — it watches only the namespace it is deployed in. +- Two CSV metadata changes and one minimal Go code change are required. + +## Background + +### OLMv0 — Current model + +OLMv0 uses a five-resource model: + +- **`CatalogSource`** — where to find operators (catalog image) +- **`OperatorGroup`** — which namespaces the operator watches + RBAC generation +- **`Subscription`** — install request +- **`InstallPlan`** — auto-generated execution plan +- **`ClusterServiceVersion` (CSV)** — the operator's metadata, permissions, and deployment spec + +OLMv0 supports four install modes: `AllNamespaces`, `OwnNamespace`, `SingleNamespace`, and `MultiNamespace`. This multi-tenancy model allowed multiple independent operator instances in different namespaces, each watching its own scope. OADP currently uses `OwnNamespace`, meaning the operator only watches CRs in the namespace where it is installed. + +### OLMv1 — The new model + +OLMv1 is not a version bump — it is a ground-up redesign that consolidates the five OLMv0 resources into two: + +- **`ClusterCatalog`** — replaces `CatalogSource` +- **`ClusterExtension`** — replaces `OperatorGroup` + `Subscription` + `InstallPlan` + `CSV` + +It is declarative, GitOps-friendly, and uses a cluster-admin security model instead of namespace-scoped ServiceAccount RBAC. + +**OLMv1 GA only supports AllNamespaces mode.** The rationale, as documented in the [OLMv1 single/own namespace enhancement proposal](https://github.com/openshift/enhancements/pull/1849): + +1. **CRDs are cluster-scoped singletons.** Only one definition of a CRD can exist per cluster. OLMv0's multi-tenancy promise — that multiple operator instances in different namespaces could each own their own CRDs — was architecturally flawed. +2. **Dependency resolution requires a global view.** OLMv1's explicit dependency model cannot work at namespace scope. +3. **PSA and security are simpler at cluster scope.** Managed platforms (ROSA, OSD) operate cluster-admin anyway. + +`OwnNamespace` and `SingleNamespace` were added as Tech Preview in OCP 4.19 ([OCPSTRAT-1711](https://issues.redhat.com/browse/OCPSTRAT-1711)) behind the `TechPreviewNoUpgrade` feature gate — a cluster-wide toggle that, while enabled, blocks upgrading to the next OCP minor version: + +- OCP 4.21 release notes explicitly state these modes "continued as a Technology Preview feature" and are "not recommended for production use." +- A planned GA promotion was subsequently reversed. [OPRUN-4514](https://issues.redhat.com/browse/OPRUN-4514) ("Revert Single/Own Namespace promotion to GA") explicitly states the goal was to return the feature to `TPNU` status; the upstream revert is [operator-controller#2568](https://github.com/operator-framework/operator-controller/pull/2568). + +`OwnNamespace` and `SingleNamespace` are not a viable option for production operators. + +### OADP Operator + +Against this backdrop, OADP today is installed via OLM with only `OwnNamespace` install mode supported. At runtime, the `WATCH_NAMESPACE` environment variable controls which namespace the controller-runtime cache monitors. + +Two distinct pieces of information are at play, set by different actors: + +- **`olm.targetNamespaces`** — a pod annotation set by OLM at install time, based on the OperatorGroup. In OwnNamespace mode this is `openshift-adp`; in AllNamespaces mode this is `""` (empty). Per the [OLM OperatorGroup design](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/operatorgroups.md), the empty string is the intentional signal for "watch all namespaces" — *"the consuming operator must know to treat `""` as an all namespace configuration."* +- **`metadata.namespace`** — a standard Kubernetes pod field set by Kubernetes itself. Always the pod's actual namespace (`openshift-adp`), regardless of install mode or OperatorGroup. + +In OwnNamespace mode these happen to be the same value, which is why the distinction hasn't mattered until now. + +`WATCH_NAMESPACE` is currently sourced from `olm.targetNamespaces`. In AllNamespaces mode it becomes `""`, which breaks PSA labeling, STS credential flow, CLI/VMDP setup, and the cache configuration — all of which need to know *where the operator lives*, not just *what it watches*. + +To understand why this can't be fixed purely at the CSV level, it helps to trace what happens at each phase: + +- **Bundle time**: `operator-sdk generate bundle` generates the CSV from the manager deployment spec. The SDK's `setNamespacedFields` function ([source](https://github.com/operator-framework/operator-sdk/blob/299157a814e674f8f40d1b91ee77d64a90e850de/internal/generate/clusterserviceversion/clusterserviceversion_updaters.go)) unconditionally rewrites any env var named `WATCH_NAMESPACE` to source from `olm.targetNamespaces`. This is hardcoded and cannot be disabled. +- **Install time**: OLM deploys the operator pod and sets `olm.targetNamespaces` based on the OperatorGroup — `openshift-adp` in OwnNamespace mode, `""` in AllNamespaces mode. +- **Runtime**: The kubelet resolves the fieldRef. `WATCH_NAMESPACE` is `""` under AllNamespaces. + +The fix has two parts: a new `OPERATOR_NAMESPACE` env var (sourced from `metadata.namespace`, which the SDK does not rewrite) gives the operator its own namespace. A minimal Go code change reads `WATCH_NAMESPACE` and, if empty, falls back to `OPERATOR_NAMESPACE` for the watch scope — aligning with OLM's stated contract that operators must handle `""` themselves. + +Additionally, OLM requires every ServiceAccount declared in `clusterPermissions` to have a corresponding `permissions` (namespace-scoped Role) entry when running in AllNamespaces mode. Two of the three OADP ServiceAccounts currently lack this entry. + +### OLMv1 Alignment + +AllNamespaces is the strategically correct direction: + +- OLMv1 GA (OCP 4.18) shipped with AllNamespaces-only support. +- OwnNamespace was added as Tech Preview in OCP 4.19 ([OCPSTRAT-1711](https://issues.redhat.com/browse/OCPSTRAT-1711)) and remains TP. +- Enabling AllNamespaces now positions OADP for OLMv1 readiness without requiring disruptive changes to runtime behavior — the operator continues watching only its own namespace. + +> **Note:** AllNamespaces here refers to the install mode (how OLM deploys the operator), not the operator's watch scope. + +## Goals + +- Enable `AllNamespaces` install mode alongside `OwnNamespace` in the same CSV. +- Keep runtime behavior identical: the operator watches only the namespace it is deployed in. +- Maintain full backward compatibility for existing `OwnNamespace` installations. + +## Non Goals + +- Actually watching all namespaces or supporting multi-tenant Velero (one DPA per namespace). +- `SingleNamespace` or `MultiNamespace` install mode support. + +## High-Level Design + +Two CSV metadata changes and one minimal Go code change: + +| # | Change | Why | +|---|---|---| +| 1 | Enable `AllNamespaces: true` in `installModes` | Allow installation with a global OperatorGroup | +| 2 | Add `OPERATOR_NAMESPACE` env var (from `metadata.namespace`) + Go fallback | `WATCH_NAMESPACE` is `""` in AllNamespaces mode by OLM design; operator needs its own namespace separately | +| 3 | Add `permissions` entries for `non-admin-controller` and `velero` SAs | OLM requires these to create the ServiceAccounts in AllNamespaces mode | + +The RBAC `clusterPermissions` and all runtime behavior remain identical. + +**How the operator resolves its namespace in each mode:** + +| OperatorGroup | `WATCH_NAMESPACE` (`olm.targetNamespaces`) | `OPERATOR_NAMESPACE` (`metadata.namespace`) | Operator watches | +|---|---|---|---| +| Namespaced (OwnNamespace) | `openshift-adp` | `openshift-adp` | `openshift-adp` (from `WATCH_NAMESPACE`) | +| Global (AllNamespaces) | `""` (empty — OLM's all-namespaces signal) | `openshift-adp` | `openshift-adp` (fallback to `OPERATOR_NAMESPACE`) | + +## Detailed Design + +### Change 1: Enable AllNamespaces Install Mode + +```yaml +installModes: +- supported: true + type: OwnNamespace +- supported: false + type: SingleNamespace +- supported: false + type: MultiNamespace +- supported: true # ← changed from false + type: AllNamespaces +``` + +Adding installMode support is a safe superset change in OLM — it never blocks upgrades. Existing customers with a namespaced OperatorGroup are unaffected; their install mode stays OwnNamespace. + +### Change 2: OPERATOR_NAMESPACE env var + Go fallback + +Add a new `OPERATOR_NAMESPACE` env var to the manager deployment, sourced from `metadata.namespace`: + +```yaml +- name: OPERATOR_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace +- name: WATCH_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.annotations['olm.targetNamespaces'] # unchanged; SDK manages this +``` + +`WATCH_NAMESPACE` is left sourcing from `olm.targetNamespaces` — the SDK's `setNamespacedFields` function rewrites any env var named `WATCH_NAMESPACE` to this unconditionally ([source](https://github.com/operator-framework/operator-sdk/blob/299157a814e674f8f40d1b91ee77d64a90e850de/internal/generate/clusterserviceversion/clusterserviceversion_updaters.go)), so fighting it with a post-gen patch is fragile. Instead, `OPERATOR_NAMESPACE` uses a different name and is not touched by the SDK. + +The Go code change reads both vars and falls back when `WATCH_NAMESPACE` is empty: + +```go +watchNamespace := os.Getenv("WATCH_NAMESPACE") +if watchNamespace == "" { + watchNamespace = os.Getenv("OPERATOR_NAMESPACE") +} +``` + +This is the approach the [OLM OperatorGroup design](https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/operatorgroups.md) expects: `""` is OLM's intentional all-namespaces signal, and *"the consuming operator must know to treat `""` as an all namespace configuration"* — or in OADP's case, fall back to its own namespace rather than watching everything. + +Under OwnNamespace mode, `WATCH_NAMESPACE` is `openshift-adp` and the fallback is never reached — fully backward-compatible. + +### Change 3: Add Permissions for Missing ServiceAccounts + +The CSV declares `clusterPermissions` for three SAs but only `openshift-adp-controller-manager` has a `permissions` entry. Add `permissions` entries for `non-admin-controller` and `velero` with leader-election rules (configmaps, leases, events). + +- The `velero` SA does not actually perform leader election — these are placeholder rules required solely to satisfy OLM's SA creation requirement. +- In AllNamespaces mode, OLM promotes these namespace-scoped Role/RoleBinding entries to ClusterRoles/ClusterRoleBindings (via its internal `ensureSingletonRBAC` reconciler, which merges all operator permissions into a single cluster-scoped set). +- In OwnNamespace mode, they remain namespace-scoped Roles/RoleBindings — no change to existing behavior. + +### OLM Behavior in AllNamespaces Mode + +These OLM behaviors do not affect operator functionality but should be understood: + +- **CSV copies**: OLM copies the CSV to every namespace. On large clusters, `OLMConfig.spec.features.disableCopiedCSVs: true` disables this. +- **CRD ownership**: The operator globally owns its CRDs. This is a pre-existing constraint of OADP's cluster-scoped CRD ownership — customers running standalone upstream Velero alongside OADP would hit `InterOperatorGroupOwnerConflict` regardless of install mode. +- **Dual installation prevention**: OLM prevents installing the operator in two namespaces with overlapping OperatorGroups. + +### OperatorHub User Experience + +When both install modes are enabled, OperatorHub (OpenShift 4.14+) presents install mode radio buttons and a namespace dropdown: + +- The existing `suggested-namespace: openshift-adp` annotation defaults the namespace to `openshift-adp` in both modes. +- For fresh AllNamespaces installs, the Console automatically creates the namespace, a global OperatorGroup, and a Subscription. + +This follows the pattern used by the Loki Operator (`openshift-operators-redhat`) and OpenShift Serverless (`openshift-serverless`). + +## Implementation + +| Phase | Scope | Risk | Depends on | +|---|---|---|---| +| 1. CSV changes | installModes, WATCH_NAMESPACE source, permissions | Low | None | +| 2. Deploy + CI | Makefile target + Prow job with existing e2e suite | Low | Phase 1 | +| 3. AllNamespaces e2e tests | AllNamespaces-specific test scenarios | Medium | Phase 2 | +| 4. Migration documentation | OperatorGroup swap procedure | Low | Phase 3 | + +### Phase 1: CSV + Go Changes + +- Add `OPERATOR_NAMESPACE` env var to `config/manager/manager.yaml`. +- Add the `WATCH_NAMESPACE` empty-string fallback to the Go startup code. +- Enable `AllNamespaces: true` in `installModes`. +- Add `permissions` entries for `non-admin-controller` and `velero` SAs. +- Run `make bundle` to regenerate — no post-gen patch needed. + +### Phase 2: Deploy + CI for AllNamespaces + +- Add a `deploy-olm-allnamespaces` Makefile target using `operator-sdk run bundle --install-mode AllNamespaces`. +- Add a Prow presubmit job that deploys with this target and runs the existing e2e suite unchanged. +- This gives immediate signal before writing any new test code. + +### Phase 3: AllNamespaces Install E2E Tests + +Test scenarios specific to AllNamespaces behavior: + +- **AllNamespaces fresh install**: verify CSV Succeeded, `WATCH_NAMESPACE` = `""`, `OPERATOR_NAMESPACE` = pod namespace, DPA reconciles, Velero deploys. +- **OwnNamespace to AllNamespaces migration**: install OwnNamespace, swap OperatorGroup to global, verify operator re-deploys and existing DPA continues functioning. +- **AllNamespaces to OwnNamespace rollback**: reverse the migration, verify operator recovers. +- **Upgrade to dual-mode CSV**: upgrade from a prior version (OwnNamespace-only) to the new version, verify the existing namespaced OperatorGroup continues to work. +- **Upgrade test parameterization**: the existing upgrade test hardcodes a namespaced OperatorGroup — parameterize to also test with a global OperatorGroup. + +### Phase 4: Migration Documentation + +Since both modes coexist in the same CSV, migration is a single step: swap the OperatorGroup. + +1. Delete the existing namespaced OperatorGroup. +2. Create a global OperatorGroup in `openshift-adp` (`spec: {}`). +3. OLM re-deploys the operator. Behavior is identical. + +Notes: +- Rollback is the reverse. No CSV downgrade is needed. +- Brief operator unavailability occurs during the swap (seconds to ~1 minute). +- In-flight backups or restores should be completed before migration. + +## Upgrade Path + +- Upgrading from a prior OADP version (OwnNamespace-only CSV) to the version with this change is seamless. +- Adding `AllNamespaces: true` is a safe superset change — OLM does not reject the upgrade. +- The customer's existing namespaced OperatorGroup continues to work. +- No customer action is required unless they want to switch to AllNamespaces mode. + +## Alternatives Considered + +### Two separate channels (one per install mode) + +Rejected: +- A single CSV supports both install modes — OLM uses the OperatorGroup to determine the active mode. +- Two channels would require separate bundles, kustomize overlays, catalog changes, and cross-channel update graphs. +- It would also create an OLM upgrade deadlock if a future release dropped OwnNamespace support. + +### Two separate OLM packages + +Rejected: +- Requires customers to uninstall and reinstall to migrate. +- The single-CSV approach requires only an OperatorGroup swap. + +### Rewrite WATCH_NAMESPACE source in the CSV via post-gen patch + +Rejected: +- `operator-sdk generate bundle` unconditionally rewrites any `WATCH_NAMESPACE` env var to source from `olm.targetNamespaces` ([source](https://github.com/operator-framework/operator-sdk/blob/299157a814e674f8f40d1b91ee77d64a90e850de/internal/generate/clusterserviceversion/clusterserviceversion_updaters.go)). A post-gen `sed` patch to restore `metadata.namespace` would be silently clobbered on every `make bundle` run unless the patch is also part of the target — and would need to survive `bundle-isupdated` CI checks. +- The OLM OperatorGroup design explicitly defines `""` as the all-namespaces signal and places the handling responsibility on the operator. Fighting the SDK to avoid a three-line Go change is the wrong trade-off. + +## Security Considerations + +- The `velero` SA's ClusterRole grants near-cluster-admin permissions (`apiGroups: ['*'], resources: ['*']`). These are bound via `ClusterRoleBinding`, which is cluster-scoped regardless of install mode. The `OwnNamespace` OperatorGroup does not restrict these permissions — the RBAC posture is identical in both modes. +- The new `permissions` entries add minimal RBAC (configmaps, leases, events) and are strictly less permissive than the existing `clusterPermissions` for those SAs. In AllNamespaces mode, OLM promotes these to ClusterRoles. +- A security review of the velero SA permissions is recommended as a general hygiene item, independent of this change. + +## Compatibility + +- Existing OwnNamespace installations are fully backward-compatible. No OperatorGroup change required. +- Upgrade from prior releases does not alter the install mode. +- Migration to AllNamespaces is optional and requires an explicit OperatorGroup swap. +- Under OwnNamespace, `metadata.namespace` and `olm.targetNamespaces` resolve to the same value. +- Minimum OpenShift version for AllNamespaces namespace selection in OperatorHub: 4.14. + +## Open Issues + +1. **Target OADP release**: Which version will include this change? + +## Validation + +Tested on OpenShift 4.22.0-ec.3 (2026-08-24) with the final implementation (`OPERATOR_NAMESPACE` + Go fallback, no post-gen patch). + +| Test | Mode | Result | +|---|---|---| +| CSV install | AllNamespaces | PASS | +| CSV install | OwnNamespace | PASS | +| `WATCH_NAMESPACE` value at runtime | AllNamespaces | `""` (correct — OLM's signal) | +| `OPERATOR_NAMESPACE` value at runtime | AllNamespaces | `openshift-adp` (correct) | +| PSA labeling (`patching operator namespace`) | AllNamespaces | PASS | +| Manager started, leader election acquired | AllNamespaces | PASS | +| `make bundle` idempotent (no post-gen patch) | — | PASS | + +## Future Enhancements + +### Cluster-Wide Watching + +If a future requirement is for the operator to watch all namespaces, `WATCH_NAMESPACE` would remain empty (OLM's signal) and the Go fallback to `OPERATOR_NAMESPACE` would be removed. `OPERATOR_NAMESPACE` is already in place from this design, so the operator already knows its own namespace for PSA labeling, STS, and CLI/VMDP setup. What would still be required: + +- Changes to the cache configuration (remove the `DefaultNamespaces` scope). +- A design decision on DPA singleton scope (one globally or one per namespace).