fix: skip ThanosQuerier reconciliation during deletion to prevent foregroundDeletion deadlock

[jeffdyoung]

Problem

The ThanosQuerier controller enters a race condition with the Kubernetes garbage collector during CR deletion, causing ThanosQuerier resources to get stuck in terminating state for 600+ seconds.

How the deadlock occurs

1. A consumer (e.g. the ODH/RHOAI operator) creates ThanosQuerier CRs with an ownerReference to a parent resource (e.g. a Monitoring CR) with blockOwnerDeletion: true and controller: true.

2. When the parent is deleted or the consumer's GC action removes the ThanosQuerier, Kubernetes initiates foreground cascading deletion: it adds a foregroundDeletion finalizer to the ThanosQuerier and begins deleting its dependents (the Deployment, Service, ServiceAccount, ConfigMap that the ThanosQuerier controller created via Owns()).

3. The ThanosQuerier controller's Reconcile method has no DeletionTimestamp check. It continues to run while the CR is being deleted — calling thanosComponentReconcilers() which updates the same Deployment, Service, etc. that the garbage collector is trying to delete.

4. The controller's updates bump resourceVersion on the owned resources. The GC, which was working with the previous resourceVersion, encounters a conflict and retries. On the next attempt, the controller updates again. This cycle prevents the GC from deleting all dependents, so the foregroundDeletion finalizer is never removed.

5. The ThanosQuerier CR is stuck in terminating state indefinitely.

Observed impact

In the opendatahub-operator E2E suite, this causes Test_ThanosQuerier_not_deployed_without_metrics to time out after 600s waiting for the ThanosQuerier to be deleted. The test disables metrics on the Monitoring CR, the operator's GC action deletes the ThanosQuerier, and the foreground deletion deadlock prevents it from completing. This failure is deterministic and reproduces on every CI run.

Failing build: pull-ci-opendatahub-io-opendatahub-operator-main-opendatahub-operator-rhoai-e2e / 2090374980800876544

Fix

Add a DeletionTimestamp check at the top of Reconcile to skip reconciliation when the ThanosQuerier is being deleted. This allows the garbage collector to delete owned resources without interference and clear the foregroundDeletion finalizer.

This is consistent with the MonitoringStack controller, which already has the same guard at monitoring-stack/controller.go:132:

if !ms.ObjectMeta.DeletionTimestamp.IsZero() {
    logger.V(6).Info("removing cluster scoped resources")
    // ...cleanup...
}

The ThanosQuerier controller doesn't need active cleanup (unlike MonitoringStack, which has cluster-scoped resources) — it relies entirely on Kubernetes GC via Owns() for owned Deployments, Services, ServiceAccounts, and ConfigMaps. Skipping reconciliation is sufficient.

🤖 Generated with Claude Code

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jeffdyoung
Once this PR has been reviewed and has the lgtm label, please assign machine424 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/controllers/monitoring/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

Hi @jeffdyoung. Thanks for your PR.

I'm waiting for a rhobs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: b86c747b-fd82-4c17-9c3f-d8b2398260ba

📥 Commits

Reviewing files that changed from the base of the PR and between 20ac77b and 057dde7.

📒 Files selected for processing (1)

• pkg/controllers/monitoring/thanos-querier/controller.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

Reconcile now checks whether a ThanosQuerier resource has a deletion timestamp. If deletion is in progress, the controller logs the condition and returns without reconciling resources, a result, or an error.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to 057dd

The controller now stops reconciling resources while a ThanosQuerier is being deleted, allowing dependent resources to finish cleanup and preventing the reported termination deadlock. No actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: simonpasquier, slashpai

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes skipping ThanosQuerier reconciliation during deletion to prevent a foreground deletion deadlock.
Description check	✅ Passed	The description directly explains the deletion race, its impact, and the proposed DeletionTimestamp guard.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}