Skip to content

CNV-87535: k8s: add orphan AlertRelabelConfig GC - #1174

Open
sradco wants to merge 1 commit into
openshift:main-alerts-management-apifrom
sradco:alert-mgmt-restructured-11-orphan-gc
Open

sradco wants to merge 1 commit into
openshift:main-alerts-management-apifrom
sradco:alert-mgmt-restructured-11-orphan-gc

Conversation

@sradco

@sradco sradco commented Aug 24, 2026

Copy link
Copy Markdown

Detect and remove orphan
AlertRelabelConfig resources that no
longer have a matching PrometheusRule,
preventing stale relabel configs from
accumulating.

Cover orphan deletion and keeper cases
in e2e (live rule, GitOps, unannotated).

Signed-off-by: Shirly Radco sradco@redhat.com
Co-authored-by: AI Assistant noreply@cursor.com

Summary by CodeRabbit

  • New Features

    • Automatically removes plugin-managed alert relabel configurations associated with deleted alert rules.
    • Preserves configurations linked to active rules, manually created configurations, and GitOps-managed resources.
    • Runs cleanup during startup and after PrometheusRule changes; secret-only updates no longer trigger cleanup.
    • Reports GitOps-managed orphaned configurations without deleting them.
  • Bug Fixes

    • Cleanup continues processing other configurations when individual resources cannot be read or deleted.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 24, 2026

Copy link
Copy Markdown

@sradco: This pull request references CNV-80608 which is a valid jira issue.

Details

In response to this:

Detect and remove orphan
AlertRelabelConfig resources that no
longer have a matching PrometheusRule,
preventing stale relabel configs from
accumulating.

Depends on #1171,
#1172, and #1173 (read API stack).

Signed-off-by: Shirly Radco sradco@redhat.com
Co-authored-by: AI Assistant noreply@cursor.com

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Advanced

Run ID: fae1803b-66fe-4bc4-8651-729becd80d1f

📥 Commits

Reviewing files that changed from the base of the PR and between 3c01803 and b6230df.

📒 Files selected for processing (1)
  • test/e2e/orphan_arc_gc_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The change adds garbage collection for orphaned plugin-managed AlertRelabelConfigs. Syncs collect live rule IDs and run cleanup for initial and PrometheusRule events. Unit and end-to-end tests cover deletion and retention rules.

Changes

AlertRelabelConfig garbage collection

Layer / File(s) Summary
Orphan detection and deletion
pkg/k8s/alert_relabel_config_gc.go, pkg/k8s/alert_relabel_config_gc_test.go
The manager lists AlertRelabelConfigs, identifies orphaned plugin-managed resources, preserves live and GitOps-managed resources, and deletes eligible orphans. Unit tests cover filtering, errors, nil values, and deletion tracking.
Sync-triggered garbage collection
pkg/k8s/relabeled_rules.go, test/e2e/orphan_arc_gc_test.go
The sync flow collects all rule IDs and runs garbage collection for initial-sync and prometheus-rule-sync keys. Secret-only syncs do not run garbage collection. The end-to-end test verifies deletion and retention behavior.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~30 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant PrometheusRule
  participant relabeledRulesManager
  participant AlertRelabelConfigClient
  PrometheusRule->>relabeledRulesManager: trigger PrometheusRule sync
  relabeledRulesManager->>relabeledRulesManager: collect live rule IDs
  relabeledRulesManager->>AlertRelabelConfigClient: list AlertRelabelConfigs
  relabeledRulesManager->>AlertRelabelConfigClient: delete eligible orphaned ARCs
Loading

Merge Risk: ⚪ Minimal · up to b6230

The added garbage-collection end-to-end coverage has no substantiated merge-blocking issue.

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding Kubernetes garbage collection for orphaned AlertRelabelConfig resources.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No unstable test titles were introduced. The added tests use static Go test function names such as TestGCOrphanedARCs_DeletesOrphan and TestOrphanAlertRelabelConfigGC. The dynamic namespace, times…
Test Structure And Quality ✅ Passed PASS. The changed tests use standard Go testing, not Ginkgo, so BeforeEach/AfterEach and Eventually/Consistently requirements do not apply. The e2e test cleans the namespace and ARC resources with def…
Microshift Test Compatibility ✅ Passed The added e2e test is not a Ginkgo test. It uses Go's standard testing package and defines TestOrphanAlertRelabelConfigGC; it has no It, Describe, Context, or When declarations. Therefore,…
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS. The new e2e test is a standard Go TestOrphanAlertRelabelConfigGC, not a Ginkgo test. It creates Kubernetes resources, PrometheusRule objects, and four AlertRelabelConfigs, then polls API state…
Topology-Aware Scheduling Compatibility ✅ Passed The pull request adds AlertRelabelConfig garbage collection and changes sync/event handling only. The authoritative diff contains no deployment or pod-spec changes and no affinity, topology spread, …
Ote Binary Stdout Contract ✅ Passed No OTE stdout contract violation was introduced. The authoritative diff adds no main(), init(), TestMain(), suite setup, RunSpecs setup, or top-level initializer that writes output. Searches found no …
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The added e2e test uses Go's standard testing.T, not Ginkgo constructs. It contains no hardcoded IPv4 address, IPv4-only parsing, CIDR, network-policy, or external URL. Its API calls use Kuber…
No-Weak-Crypto ✅ Passed No prohibited weak cryptography or secret comparison was introduced. The added and modified lines contain no MD5, SHA-1, DES/3DES, RC4, Blowfish, or ECB usage, and no custom cryptographic implementati…
Container-Privileges ✅ Passed The pull request changes only Go source and test files. The authoritative diff adds no container or Kubernetes manifest fields. Added lines contain no privileged: true, hostPID, hostNetwork, `ho…
No-Sensitive-Data-In-Logs ✅ Passed PASS. The pull request adds logs for ARC namespace/name, orphan status, a rule ID, and operation errors. The rule ID is a SHA-256-derived opaque value (rid_...), and the logs do not include ARC spec…
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Some tools did not complete. Review the errors below.

🔧 golangci-lint (2.13.2)

level=error msg="[linters_context] typechecking error: build constraints exclude all Go files in /test/e2e"


Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from jgbernalp and zhuje August 24, 2026 14:28
@openshift-ci

openshift-ci Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sradco
Once this PR has been reviewed and has the lgtm label, please assign jgbernalp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sradco sradco changed the title CNV-80608: k8s: add orphan AlertRelabelConfig GC CNV-87535: k8s: add orphan AlertRelabelConfig GC Aug 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 24, 2026

Copy link
Copy Markdown

@sradco: This pull request references CNV-87535 which is a valid jira issue.

Details

In response to this:

Detect and remove orphan
AlertRelabelConfig resources that no
longer have a matching PrometheusRule,
preventing stale relabel configs from
accumulating.

Depends on #1171,
#1172, and #1173 (read API stack).

Signed-off-by: Shirly Radco sradco@redhat.com
Co-authored-by: AI Assistant noreply@cursor.com

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sradco
sradco force-pushed the alert-mgmt-restructured-11-orphan-gc branch from 0d89ed9 to d056eaa Compare August 27, 2026 11:10
@sradco
sradco force-pushed the alert-mgmt-restructured-11-orphan-gc branch from d056eaa to 63a08fb Compare September 6, 2026 14:51
@sradco
sradco force-pushed the alert-mgmt-restructured-11-orphan-gc branch from 63a08fb to 3c01803 Compare September 14, 2026 09:32
@simonpasquier

Copy link
Copy Markdown
Contributor

/test ?

@simonpasquier

Copy link
Copy Markdown
Contributor

/test e2e-management-api

Detect and remove orphan
AlertRelabelConfig resources that no
longer have a matching PrometheusRule,
preventing stale relabel configs from
accumulating.

Cover orphan deletion and keeper cases
in e2e (live rule, GitOps, unannotated).

Signed-off-by: Shirly Radco <sradco@redhat.com>
Co-authored-by: AI Assistant <noreply@cursor.com>
@sradco
sradco force-pushed the alert-mgmt-restructured-11-orphan-gc branch from 3c01803 to b6230df Compare September 14, 2026 14:50
@PeterYurkovich

Copy link
Copy Markdown
Contributor

/test e2e-management-api

@openshift-ci

openshift-ci Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

@sradco: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-management-api b6230df link false /test e2e-management-api
ci/prow/security b6230df link false /test security
ci/prow/images b6230df link true /test images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.


arcs, err := rrm.alertRelabelConfigs.List(ctx, "")
if err != nil {
log.Errorf("orphan ARC GC: failed to list ARCs: %v", err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) we shouldn't use acronyms in log messages.

}

if err := rrm.sync(ctx); err != nil {
if err := rrm.sync(ctx, "initial-sync"); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls use constants

}

arcs, err := rrm.alertRelabelConfigs.List(ctx, "")
if err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how will cluster admins know that something's not going correctly? e.g. can we add metrics + alerting rule?

continue
}

if err := rrm.alertRelabelConfigs.Delete(ctx, arc.Namespace, arc.Name); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question here about the ability to know that something failed.

//
// Only ARCs carrying the plugin's alertRuleId annotation are considered.
// GitOps-managed ARCs are never deleted automatically; a warning is logged
// so that operators can clean them up manually.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a log would go unnoticed, there needs to be an alerting rule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants