From 5d73b83d1d529908f50ef0afb58bd43c60688395 Mon Sep 17 00:00:00 2001 From: Vladimir Videlov Date: Thu, 25 Jun 2026 17:08:56 +0200 Subject: [PATCH 1/5] Add webhook-injector propagation alerts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds five new alerts covering the propagation-path metrics that webhook-injector exposes. The two existing alerts (WebhookCertificate NearExpiry, WebhookCertificateAboutToExpire) cover the cert expiry side; these five cover the apply / stamp / GRM-bridge side that is just as important for caBundle freshness reaching the shoot. Alerts: - WebhookTargetStampErrors (critical, 10m for): rate of target_stamps _total{result="error"} > 0. MWC/VWC apply on the target cluster is failing — the freshly rotated caBundle is NOT reaching the admission webhooks and subsequent CR writes may be rejected with TLS errors. - WebhookManagedResourceStampErrors (critical, 10m for): rate of managed_resource_stamps_total{result="error"} > 0. MR Secret stamping on the hosting cluster is failing — embedded CRDs will not refresh in the shoot via GRM, and conversion webhooks for those CRDs will fail TLS once the old cert is pruned. - WebhookManagedResourceUnhealthy (warning, 15m for): rate of managed_resource_unhealthy_total > 0. A labeled MR has ResourcesApplied or ResourcesHealthy != True. GRM is unable to apply or keep resources healthy in the shoot — caBundle propagation is degraded. - WebhookManagedResourceKeepObjectsViolation (warning, 15m for): rate of keep_objects_violations_total > 0. A labeled MR ships CRDs without spec.keepObjects=true — destructive shape: deleting that MR (intentionally or accidentally) would cascade-delete shoot-side CRDs and every custom resource of those kinds. - WebhookManagedResourceForceReconcilePatchErrors (warning, 15m for): rate of force_reconcile_total{result="error"} > 0. Patching gardener.cloud/operation=reconcile on MRs is failing. GRM will still pick up the change on its own watch/sync cadence, but with extra latency. Style follows the existing webhook cert alerts and the rest of this file: Helm dig templating for for/severity/service/support_group, disabled toggle, additionalRuleLabels include, playbook link. Each alert can be individually disabled via .Values.prometheusRules.disabled. Verified: - helm lint charts/controlplane-operations: clean - promtool check rules: SUCCESS, 11 rules found (6 existing + 5 new) --- .../alerts/controlplane-remote.yaml | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/charts/controlplane-operations/alerts/controlplane-remote.yaml b/charts/controlplane-operations/alerts/controlplane-remote.yaml index 841d688..bc128fe 100644 --- a/charts/controlplane-operations/alerts/controlplane-remote.yaml +++ b/charts/controlplane-operations/alerts/controlplane-remote.yaml @@ -99,3 +99,83 @@ groups: summary: "Webhook certificate is about to expire." description: "Certificate expires in less than 3 days." {{- end }} + +{{- if not (.Values.prometheusRules.disabled.WebhookTargetStampErrors | default false) }} + - alert: WebhookTargetStampErrors + expr: | + sum(rate(webhook_injector_target_stamps_total{result="error"}[10m])) > 0 + for: {{ dig "WebhookTargetStampErrors" "for" "10m" .Values.prometheusRules }} + labels: + {{ include "controlplane-operations.additionalRuleLabels" . }} + severity: {{ dig "WebhookTargetStampErrors" "severity" "critical" .Values.prometheusRules }} + playbook: https://operations.global.cloud.sap/docs/support/playbook/webhook-target-stamp-errors/ #TODO: add playbook + service: {{ dig "WebhookTargetStampErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} + support_group: {{ dig "WebhookTargetStampErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} + annotations: + summary: "Webhook caBundle apply to target cluster is failing." + description: "webhook-injector failed to apply MutatingWebhookConfiguration or ValidatingWebhookConfiguration to the target cluster — the freshly rotated caBundle is NOT reaching the admission webhooks and subsequent CR writes may be rejected with TLS errors." +{{- end }} + +{{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceStampErrors | default false) }} + - alert: WebhookManagedResourceStampErrors + expr: | + sum(rate(webhook_injector_managed_resource_stamps_total{result="error"}[10m])) > 0 + for: {{ dig "WebhookManagedResourceStampErrors" "for" "10m" .Values.prometheusRules }} + labels: + {{ include "controlplane-operations.additionalRuleLabels" . }} + severity: {{ dig "WebhookManagedResourceStampErrors" "severity" "critical" .Values.prometheusRules }} + playbook: https://operations.global.cloud.sap/docs/support/playbook/webhook-managed-resource-stamp-errors/ #TODO: add playbook + service: {{ dig "WebhookManagedResourceStampErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} + support_group: {{ dig "WebhookManagedResourceStampErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} + annotations: + summary: "Webhook caBundle stamp on ManagedResource Secret is failing." + description: "webhook-injector failed to stamp the freshly rotated caBundle onto a labeled ManagedResource Secret on the hosting cluster — the embedded CRDs will not refresh in the shoot via gardener-resource-manager, and conversion webhooks for those CRDs will fail TLS once the old cert is pruned." +{{- end }} + +{{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceUnhealthy | default false) }} + - alert: WebhookManagedResourceUnhealthy + expr: | + sum(rate(webhook_injector_managed_resource_unhealthy_total[15m])) > 0 + for: {{ dig "WebhookManagedResourceUnhealthy" "for" "15m" .Values.prometheusRules }} + labels: + {{ include "controlplane-operations.additionalRuleLabels" . }} + severity: {{ dig "WebhookManagedResourceUnhealthy" "severity" "warning" .Values.prometheusRules }} + playbook: https://operations.global.cloud.sap/docs/support/playbook/webhook-managed-resource-unhealthy/ #TODO: add playbook + service: {{ dig "WebhookManagedResourceUnhealthy" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} + support_group: {{ dig "WebhookManagedResourceUnhealthy" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} + annotations: + summary: "ManagedResource reported as not healthy by gardener-resource-manager." + description: "A labeled ManagedResource has ResourcesApplied or ResourcesHealthy condition with Status != True, meaning GRM is unable to apply or keep the resources healthy in the shoot. caBundle propagation to shoot-side CRDs is degraded; check the MR's status conditions for Reason/Message." +{{- end }} + +{{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceKeepObjectsViolation | default false) }} + - alert: WebhookManagedResourceKeepObjectsViolation + expr: | + sum(rate(webhook_injector_managed_resource_keep_objects_violations_total[15m])) > 0 + for: {{ dig "WebhookManagedResourceKeepObjectsViolation" "for" "15m" .Values.prometheusRules }} + labels: + {{ include "controlplane-operations.additionalRuleLabels" . }} + severity: {{ dig "WebhookManagedResourceKeepObjectsViolation" "severity" "warning" .Values.prometheusRules }} + playbook: https://operations.global.cloud.sap/docs/support/playbook/webhook-managed-resource-keep-objects-violation/ #TODO: add playbook + service: {{ dig "WebhookManagedResourceKeepObjectsViolation" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} + support_group: {{ dig "WebhookManagedResourceKeepObjectsViolation" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} + annotations: + summary: "Labeled ManagedResource ships CRDs without spec.keepObjects=true." + description: "A labeled ManagedResource ships at least one CRD but does not set spec.keepObjects=true. Deleting that MR (intentionally or accidentally) would cascade-delete the shoot-side CRDs and every custom resource of those kinds. Fix by setting spec.keepObjects=true on the MR — the stamp still works either way, but the destructive shape needs to be addressed." +{{- end }} + +{{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceForceReconcilePatchErrors | default false) }} + - alert: WebhookManagedResourceForceReconcilePatchErrors + expr: | + sum(rate(webhook_injector_managed_resource_force_reconcile_total{result="error"}[15m])) > 0 + for: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "for" "15m" .Values.prometheusRules }} + labels: + {{ include "controlplane-operations.additionalRuleLabels" . }} + severity: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "severity" "warning" .Values.prometheusRules }} + playbook: https://operations.global.cloud.sap/docs/support/playbook/webhook-managed-resource-force-reconcile-patch-errors/ #TODO: add playbook + service: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} + support_group: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} + annotations: + summary: "Failed to patch gardener.cloud/operation=reconcile on ManagedResource." + description: "webhook-injector stamped a ManagedResource Secret but failed to patch the gardener.cloud/operation=reconcile annotation on the parent MR. GRM may take up to its own watch/sync interval to notice the change rather than reconciling immediately — caBundle propagation to the shoot will still happen, but with extra latency." +{{- end }} From ec3a2b34bb72aae136a895b6fddc4bca04fed4c4 Mon Sep 17 00:00:00 2001 From: Vladimir Videlov Date: Thu, 25 Jun 2026 17:12:50 +0200 Subject: [PATCH 2/5] Bump chart + plugin version to 1.1.11 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch bump for the additive webhook-injector propagation alerts. Follows the convention from #23 (1.1.9 → 1.1.10 for the previous additive alerts PR). --- charts/controlplane-operations/Chart.yaml | 2 +- charts/controlplane-operations/plugindefinition.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/charts/controlplane-operations/Chart.yaml b/charts/controlplane-operations/Chart.yaml index fd9e517..0be59d2 100644 --- a/charts/controlplane-operations/Chart.yaml +++ b/charts/controlplane-operations/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 name: controlplane-operations -version: 1.1.10 +version: 1.1.11 description: A set of Plutono dashboards and Prometheus alerting rules combined with playbooks to ensure effective operations of Controlplane clusters. maintainers: - name: Vladimir Videlov (d051408) diff --git a/charts/controlplane-operations/plugindefinition.yaml b/charts/controlplane-operations/plugindefinition.yaml index 113033a..0ecc745 100644 --- a/charts/controlplane-operations/plugindefinition.yaml +++ b/charts/controlplane-operations/plugindefinition.yaml @@ -3,7 +3,7 @@ kind: PluginDefinition metadata: name: controlplane-operations spec: - version: 1.1.10 + version: 1.1.11 displayName: Controlplane operations bundle description: Operations bundle for Controlane clusters docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/controlplane-operations/main/README.md @@ -11,7 +11,7 @@ spec: helmChart: name: controlplane-operations repository: oci://ghcr.io/cloudoperators/controlplane-operations/charts - version: 1.1.10 + version: 1.1.11 options: - name: prometheusRules.create description: Create Prometheus rules From 123291f89b0e6dc1f01311ce84fdd239eea27d93 Mon Sep 17 00:00:00 2001 From: Vladimir Videlov Date: Thu, 25 Jun 2026 17:27:47 +0200 Subject: [PATCH 3/5] Shorten rate window on propagation alerts; bump to 1.1.12 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses Copilot review feedback on #26: Using rate(metric[N]) > 0 together with for: N is a known Prometheus gotcha — a single error increment can fall out of the trailing N-minute rate window right as the for clause would otherwise satisfy, so a sustained but low-frequency error stream fails to fire the alert. Fix: keep the for: durations (which encode the persistence threshold we actually want — 10m for critical apply failures, 15m for warning-severity MR conditions) but shrink the rate window to a fixed [5m]. That ensures the rate observation continues to evaluate > 0 across the full for period for any error stream that's persistent at all. Same change applied to all five new alerts: - WebhookTargetStampErrors - WebhookManagedResourceStampErrors - WebhookManagedResourceUnhealthy - WebhookManagedResourceKeepObjectsViolation - WebhookManagedResourceForceReconcilePatchErrors Patch-bump chart + plugin to 1.1.12. Verified: - helm lint charts/controlplane-operations: clean - promtool check rules: SUCCESS, 11 rules found. --- charts/controlplane-operations/Chart.yaml | 2 +- .../alerts/controlplane-remote.yaml | 10 +++++----- charts/controlplane-operations/plugindefinition.yaml | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/charts/controlplane-operations/Chart.yaml b/charts/controlplane-operations/Chart.yaml index 0be59d2..60d42d7 100644 --- a/charts/controlplane-operations/Chart.yaml +++ b/charts/controlplane-operations/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 name: controlplane-operations -version: 1.1.11 +version: 1.1.12 description: A set of Plutono dashboards and Prometheus alerting rules combined with playbooks to ensure effective operations of Controlplane clusters. maintainers: - name: Vladimir Videlov (d051408) diff --git a/charts/controlplane-operations/alerts/controlplane-remote.yaml b/charts/controlplane-operations/alerts/controlplane-remote.yaml index bc128fe..9f0f340 100644 --- a/charts/controlplane-operations/alerts/controlplane-remote.yaml +++ b/charts/controlplane-operations/alerts/controlplane-remote.yaml @@ -103,7 +103,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookTargetStampErrors | default false) }} - alert: WebhookTargetStampErrors expr: | - sum(rate(webhook_injector_target_stamps_total{result="error"}[10m])) > 0 + sum(rate(webhook_injector_target_stamps_total{result="error"}[5m])) > 0 for: {{ dig "WebhookTargetStampErrors" "for" "10m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -119,7 +119,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceStampErrors | default false) }} - alert: WebhookManagedResourceStampErrors expr: | - sum(rate(webhook_injector_managed_resource_stamps_total{result="error"}[10m])) > 0 + sum(rate(webhook_injector_managed_resource_stamps_total{result="error"}[5m])) > 0 for: {{ dig "WebhookManagedResourceStampErrors" "for" "10m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -135,7 +135,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceUnhealthy | default false) }} - alert: WebhookManagedResourceUnhealthy expr: | - sum(rate(webhook_injector_managed_resource_unhealthy_total[15m])) > 0 + sum(rate(webhook_injector_managed_resource_unhealthy_total[5m])) > 0 for: {{ dig "WebhookManagedResourceUnhealthy" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -151,7 +151,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceKeepObjectsViolation | default false) }} - alert: WebhookManagedResourceKeepObjectsViolation expr: | - sum(rate(webhook_injector_managed_resource_keep_objects_violations_total[15m])) > 0 + sum(rate(webhook_injector_managed_resource_keep_objects_violations_total[5m])) > 0 for: {{ dig "WebhookManagedResourceKeepObjectsViolation" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -167,7 +167,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceForceReconcilePatchErrors | default false) }} - alert: WebhookManagedResourceForceReconcilePatchErrors expr: | - sum(rate(webhook_injector_managed_resource_force_reconcile_total{result="error"}[15m])) > 0 + sum(rate(webhook_injector_managed_resource_force_reconcile_total{result="error"}[5m])) > 0 for: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} diff --git a/charts/controlplane-operations/plugindefinition.yaml b/charts/controlplane-operations/plugindefinition.yaml index 0ecc745..28ab325 100644 --- a/charts/controlplane-operations/plugindefinition.yaml +++ b/charts/controlplane-operations/plugindefinition.yaml @@ -3,7 +3,7 @@ kind: PluginDefinition metadata: name: controlplane-operations spec: - version: 1.1.11 + version: 1.1.12 displayName: Controlplane operations bundle description: Operations bundle for Controlane clusters docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/controlplane-operations/main/README.md @@ -11,7 +11,7 @@ spec: helmChart: name: controlplane-operations repository: oci://ghcr.io/cloudoperators/controlplane-operations/charts - version: 1.1.11 + version: 1.1.12 options: - name: prometheusRules.create description: Create Prometheus rules From 14f8f53977fa0c2af65990fbd19ad16eee479576 Mon Sep 17 00:00:00 2001 From: Vladimir Videlov Date: Fri, 26 Jun 2026 11:39:53 +0200 Subject: [PATCH 4/5] Enrich propagation alerts with namespace + per-metric labels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Annotations on the five propagation alerts now identify the affected shoot and the per-metric labels in summary/description, following the pattern from controlplane-gardener.yaml ({{` $labels. `}}). Switch sum() → sum by (...) so the labels survive aggregation: - WebhookTargetStampErrors: sum by (namespace, kind) → summary names {{ $labels.kind }} (Mutating/Validating) and {{ $labels.namespace }} (the shoot's control-plane namespace). - WebhookManagedResourceStampErrors: sum by (namespace) → identifies the shoot whose MR Secret stamp failed. - WebhookManagedResourceUnhealthy: sum by (namespace, condition, status) → summary spells out which condition (ResourcesApplied/Healthy) is at which status (False/Progressing/Unknown). - WebhookManagedResourceKeepObjectsViolation: sum by (namespace) → identifies the shoot whose MR ships CRDs without keepObjects=true. - WebhookManagedResourceForceReconcilePatchErrors: sum by (namespace) → identifies the shoot. All descriptions now suggest checking webhook-injector logs in the affected namespace, since the exact MR name is not a metric label (by design — cardinality-bounded) but is in the log line. Cardinality bound is small: target_stamps has 2×3 series max, managed_resource_stamps and force_reconcile have 3, unhealthy has 2×3, keep_objects_violations has 1 — all per-Pod. {result="error"} narrows further. Patch-bump chart + plugin to 1.1.13. Verified: - helm lint: clean - helm template + promtool check rules: SUCCESS, 11 rules found - rendered output contains literal {{ $labels.namespace }} etc. as Prometheus expects, not Helm template artifacts. --- charts/controlplane-operations/Chart.yaml | 2 +- .../alerts/controlplane-remote.yaml | 30 +++++++++---------- .../plugindefinition.yaml | 4 +-- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/charts/controlplane-operations/Chart.yaml b/charts/controlplane-operations/Chart.yaml index 60d42d7..8519207 100644 --- a/charts/controlplane-operations/Chart.yaml +++ b/charts/controlplane-operations/Chart.yaml @@ -1,6 +1,6 @@ apiVersion: v2 name: controlplane-operations -version: 1.1.12 +version: 1.1.13 description: A set of Plutono dashboards and Prometheus alerting rules combined with playbooks to ensure effective operations of Controlplane clusters. maintainers: - name: Vladimir Videlov (d051408) diff --git a/charts/controlplane-operations/alerts/controlplane-remote.yaml b/charts/controlplane-operations/alerts/controlplane-remote.yaml index 9f0f340..95b1cea 100644 --- a/charts/controlplane-operations/alerts/controlplane-remote.yaml +++ b/charts/controlplane-operations/alerts/controlplane-remote.yaml @@ -103,7 +103,7 @@ groups: {{- if not (.Values.prometheusRules.disabled.WebhookTargetStampErrors | default false) }} - alert: WebhookTargetStampErrors expr: | - sum(rate(webhook_injector_target_stamps_total{result="error"}[5m])) > 0 + sum by (namespace, kind) (rate(webhook_injector_target_stamps_total{result="error"}[5m])) > 0 for: {{ dig "WebhookTargetStampErrors" "for" "10m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -112,14 +112,14 @@ groups: service: {{ dig "WebhookTargetStampErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} support_group: {{ dig "WebhookTargetStampErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} annotations: - summary: "Webhook caBundle apply to target cluster is failing." - description: "webhook-injector failed to apply MutatingWebhookConfiguration or ValidatingWebhookConfiguration to the target cluster — the freshly rotated caBundle is NOT reaching the admission webhooks and subsequent CR writes may be rejected with TLS errors." + summary: "{{`{{ $labels.kind }}`}} apply to shoot {{`{{ $labels.namespace }}`}} is failing." + description: "webhook-injector failed to apply {{`{{ $labels.kind }}`}} to the target cluster of shoot {{`{{ $labels.namespace }}`}}. The freshly rotated caBundle is NOT reaching the admission webhooks on that shoot and subsequent CR writes may be rejected with TLS errors." {{- end }} {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceStampErrors | default false) }} - alert: WebhookManagedResourceStampErrors expr: | - sum(rate(webhook_injector_managed_resource_stamps_total{result="error"}[5m])) > 0 + sum by (namespace) (rate(webhook_injector_managed_resource_stamps_total{result="error"}[5m])) > 0 for: {{ dig "WebhookManagedResourceStampErrors" "for" "10m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -128,14 +128,14 @@ groups: service: {{ dig "WebhookManagedResourceStampErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} support_group: {{ dig "WebhookManagedResourceStampErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} annotations: - summary: "Webhook caBundle stamp on ManagedResource Secret is failing." - description: "webhook-injector failed to stamp the freshly rotated caBundle onto a labeled ManagedResource Secret on the hosting cluster — the embedded CRDs will not refresh in the shoot via gardener-resource-manager, and conversion webhooks for those CRDs will fail TLS once the old cert is pruned." + summary: "ManagedResource Secret caBundle stamp is failing on shoot {{`{{ $labels.namespace }}`}}." + description: "webhook-injector failed to stamp the freshly rotated caBundle onto a labeled ManagedResource Secret in namespace {{`{{ $labels.namespace }}`}}. The embedded CRDs will not refresh in the shoot via gardener-resource-manager, and conversion webhooks for those CRDs will fail TLS once the old cert is pruned. Check the webhook-injector logs in namespace {{`{{ $labels.namespace }}`}} for the offending MR name." {{- end }} {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceUnhealthy | default false) }} - alert: WebhookManagedResourceUnhealthy expr: | - sum(rate(webhook_injector_managed_resource_unhealthy_total[5m])) > 0 + sum by (namespace, condition, status) (rate(webhook_injector_managed_resource_unhealthy_total[5m])) > 0 for: {{ dig "WebhookManagedResourceUnhealthy" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -144,14 +144,14 @@ groups: service: {{ dig "WebhookManagedResourceUnhealthy" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} support_group: {{ dig "WebhookManagedResourceUnhealthy" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} annotations: - summary: "ManagedResource reported as not healthy by gardener-resource-manager." - description: "A labeled ManagedResource has ResourcesApplied or ResourcesHealthy condition with Status != True, meaning GRM is unable to apply or keep the resources healthy in the shoot. caBundle propagation to shoot-side CRDs is degraded; check the MR's status conditions for Reason/Message." + summary: "ManagedResource on shoot {{`{{ $labels.namespace }}`}} is unhealthy ({{`{{ $labels.condition }}`}}={{`{{ $labels.status }}`}})." + description: "A labeled ManagedResource in namespace {{`{{ $labels.namespace }}`}} has condition {{`{{ $labels.condition }}`}} with status {{`{{ $labels.status }}`}} (expected True). gardener-resource-manager is unable to apply or keep the resources healthy in the shoot, so caBundle propagation to shoot-side CRDs is degraded. Check the MR's status conditions for Reason/Message and the webhook-injector logs in namespace {{`{{ $labels.namespace }}`}} for the affected MR name." {{- end }} {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceKeepObjectsViolation | default false) }} - alert: WebhookManagedResourceKeepObjectsViolation expr: | - sum(rate(webhook_injector_managed_resource_keep_objects_violations_total[5m])) > 0 + sum by (namespace) (rate(webhook_injector_managed_resource_keep_objects_violations_total[5m])) > 0 for: {{ dig "WebhookManagedResourceKeepObjectsViolation" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -160,14 +160,14 @@ groups: service: {{ dig "WebhookManagedResourceKeepObjectsViolation" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} support_group: {{ dig "WebhookManagedResourceKeepObjectsViolation" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} annotations: - summary: "Labeled ManagedResource ships CRDs without spec.keepObjects=true." - description: "A labeled ManagedResource ships at least one CRD but does not set spec.keepObjects=true. Deleting that MR (intentionally or accidentally) would cascade-delete the shoot-side CRDs and every custom resource of those kinds. Fix by setting spec.keepObjects=true on the MR — the stamp still works either way, but the destructive shape needs to be addressed." + summary: "ManagedResource on shoot {{`{{ $labels.namespace }}`}} ships CRDs without spec.keepObjects=true." + description: "A labeled ManagedResource in namespace {{`{{ $labels.namespace }}`}} ships at least one CRD but does not set spec.keepObjects=true. Deleting that MR (intentionally or accidentally) would cascade-delete the shoot-side CRDs and every custom resource of those kinds. Fix by setting spec.keepObjects=true on the MR — the stamp still works either way, but the destructive shape needs to be addressed. Check the webhook-injector logs in namespace {{`{{ $labels.namespace }}`}} for the offending MR name." {{- end }} {{- if not (.Values.prometheusRules.disabled.WebhookManagedResourceForceReconcilePatchErrors | default false) }} - alert: WebhookManagedResourceForceReconcilePatchErrors expr: | - sum(rate(webhook_injector_managed_resource_force_reconcile_total{result="error"}[5m])) > 0 + sum by (namespace) (rate(webhook_injector_managed_resource_force_reconcile_total{result="error"}[5m])) > 0 for: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "for" "15m" .Values.prometheusRules }} labels: {{ include "controlplane-operations.additionalRuleLabels" . }} @@ -176,6 +176,6 @@ groups: service: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "service" .Values.prometheusRules.defaultService .Values.prometheusRules }} support_group: {{ dig "WebhookManagedResourceForceReconcilePatchErrors" "support_group" .Values.prometheusRules.defaultSupportGroup .Values.prometheusRules }} annotations: - summary: "Failed to patch gardener.cloud/operation=reconcile on ManagedResource." - description: "webhook-injector stamped a ManagedResource Secret but failed to patch the gardener.cloud/operation=reconcile annotation on the parent MR. GRM may take up to its own watch/sync interval to notice the change rather than reconciling immediately — caBundle propagation to the shoot will still happen, but with extra latency." + summary: "gardener.cloud/operation=reconcile patch failing on shoot {{`{{ $labels.namespace }}`}}." + description: "webhook-injector stamped a ManagedResource Secret in namespace {{`{{ $labels.namespace }}`}} but failed to patch the gardener.cloud/operation=reconcile annotation on the parent MR. gardener-resource-manager may take up to its own watch/sync interval to notice the change rather than reconciling immediately — caBundle propagation to the shoot will still happen, but with extra latency. Check the webhook-injector logs in namespace {{`{{ $labels.namespace }}`}} for the affected MR name." {{- end }} diff --git a/charts/controlplane-operations/plugindefinition.yaml b/charts/controlplane-operations/plugindefinition.yaml index 28ab325..5754888 100644 --- a/charts/controlplane-operations/plugindefinition.yaml +++ b/charts/controlplane-operations/plugindefinition.yaml @@ -3,7 +3,7 @@ kind: PluginDefinition metadata: name: controlplane-operations spec: - version: 1.1.12 + version: 1.1.13 displayName: Controlplane operations bundle description: Operations bundle for Controlane clusters docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/controlplane-operations/main/README.md @@ -11,7 +11,7 @@ spec: helmChart: name: controlplane-operations repository: oci://ghcr.io/cloudoperators/controlplane-operations/charts - version: 1.1.12 + version: 1.1.13 options: - name: prometheusRules.create description: Create Prometheus rules From 89e695fa1e8288d93e2dd38016412ee8afb10ba9 Mon Sep 17 00:00:00 2001 From: Vladimir Videlov Date: Fri, 26 Jun 2026 11:42:47 +0200 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Dmitri Fedotov <13087245+defo89@users.noreply.github.com> Signed-off-by: Vladimir Videlov --- charts/controlplane-operations/plugindefinition.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/charts/controlplane-operations/plugindefinition.yaml b/charts/controlplane-operations/plugindefinition.yaml index 5754888..0351070 100644 --- a/charts/controlplane-operations/plugindefinition.yaml +++ b/charts/controlplane-operations/plugindefinition.yaml @@ -5,7 +5,7 @@ metadata: spec: version: 1.1.13 displayName: Controlplane operations bundle - description: Operations bundle for Controlane clusters + description: Operations bundle for Controlplane clusters docMarkDownUrl: https://raw.githubusercontent.com/cloudoperators/controlplane-operations/main/README.md icon: https://raw.githubusercontent.com/cloudoperators/controlplane-operations/main/charts/controlplane-operations/kubernetes-logo.png helmChart: