Skip to content

kubelet-config: register MCP informer to trigger reconciliation on pool changes#6219

Open
ankimaha-sys wants to merge 2 commits into
openshift:mainfrom
ankimaha-sys:kubelet-config-mcp-informer
Open

kubelet-config: register MCP informer to trigger reconciliation on pool changes#6219
ankimaha-sys wants to merge 2 commits into
openshift:mainfrom
ankimaha-sys:kubelet-config-mcp-informer

Conversation

@ankimaha-sys

@ankimaha-sys ankimaha-sys commented Jun 23, 2026

Copy link
Copy Markdown

Summary

  • The kubelet config controller did not register event handlers on the MachineConfigPool informer, so adding a new MCP never triggered reconciliation of kubelet-related MachineConfigs.
  • Adds MCP Add/Update/Delete event handlers that re-enqueue all KubeletConfigs, FeatureGates, and NodeConfigs, ensuring per-pool generated MachineConfigs (97-<pool>-generated-kubelet, 98-<pool>-generated-kubelet) are created without requiring a controller restart.

Root Cause

The New() constructor in the kubelet config controller registered informer event handlers for KubeletConfig, FeatureGate, NodeConfig, and APIServer — but not for MachineConfigPool. When a new MCP was created:

  1. syncFeatureHandler (which generates 97-<pool>-generated-kubelet) was never triggered for the new pool
  2. syncKubeletConfig (which generates 98-<pool>-generated-kubelet) was never triggered for matching KubeletConfigs
  3. syncNodeConfigHandler was never triggered for the new pool

The only workaround was restarting the machine-config-controller pod, which caused initial ADD events to fire for FeatureGates/KubeletConfigs, triggering the missing reconciliation.

Fix

Register AddFunc, UpdateFunc, and DeleteFunc handlers on the MCP informer:

  • Add: When a new MCP is created, re-enqueue all KubeletConfigs, the cluster FeatureGate, and the cluster NodeConfig for reconciliation
  • Update: When MCP labels change (which affects pool selector matching), trigger the same re-sync
  • Delete: Log the deletion (generated MC cleanup is handled by other controllers)

This follows the same pattern used by other MCO controllers (render, pinned-image-set, node) that already watch for MCP changes.

How to Reproduce (before fix)

  1. Have an existing cluster with KubeletConfig CRs
  2. Create a new MachineConfigPool:
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfigPool
    metadata:
      name: demo
      labels:
        machineconfiguration.openshift.io/role: demo
    spec:
      machineConfigSelector:
        matchExpressions:
          - key: machineconfiguration.openshift.io/role
            operator: In
            values: [worker, demo]
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/demo: ""
  3. Watch MachineConfigs: oc get machineconfig -w
  4. Expected: 97-demo-generated-kubelet, 98-demo-generated-kubelet, and rendered-demo-* are created
  5. Actual: Only rendered-demo-* is created; 97/98 configs only appear after restarting the controller

Fixes: #5521

Made with Cursor

Summary by CodeRabbit

  • Bug Fixes
    • Kubelet configuration controller now monitors and responds to configuration pool events, automatically triggering controller reconciliation to keep cluster configurations synchronized when configuration pools are added, updated, or deleted.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci

openshift-ci Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ankimaha-sys
Once this PR has been reviewed and has the lgtm label, please assign umohnani8 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot requested review from pablintino and yuqi-zhang June 23, 2026 02:46
@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 23, 2026
@openshift-ci

openshift-ci Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Hi @ankimaha-sys. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@coderabbitai

coderabbitai Bot commented Jun 23, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@ankimaha-sys, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 4 minutes and 39 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3caea9d2-9da5-4b80-95bc-948240a636b0

📥 Commits

Reviewing files that changed from the base of the PR and between 5e58077 and 135ef52.

📒 Files selected for processing (1)
  • pkg/controller/kubelet-config/kubelet_config_controller.go

Walkthrough

The kubelet config controller's New(...) function now registers Add, Update, and Delete event handlers on the mcpInformer. On MCP add or label-change events, the controller lists all KubeletConfig objects and enqueues them, then attempts to enqueue the cluster FeatureGate and NodeConfig objects. MCP delete events are only logged.

Changes

MCP-driven kubelet config reconciliation

Layer / File(s) Summary
MCP handler registration and reconciliation logic
pkg/controller/kubelet-config/kubelet_config_controller.go
New(...) wires mcpInformer with addMachineConfigPool, updateMachineConfigPool, and deleteMachineConfigPool handlers. The add/update handlers call requeueKubeletConfigsForPool, which enqueues all KubeletConfig objects plus the cluster FeatureGate and NodeConfig if they exist; the delete handler only logs the event.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: registering MCP informer handlers to trigger kubelet config reconciliation when pools change.
Linked Issues check ✅ Passed The PR directly addresses issue #5521 by implementing MCP event handlers to auto-generate kubelet configs, eliminating the need for controller restart workarounds.
Out of Scope Changes check ✅ Passed The changes are narrowly scoped to registering MCP event handlers in the kubelet config controller, directly addressing the linked issue without introducing unrelated modifications.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names ✅ Passed The PR modifies only regular Go tests (using testing.T), not Ginkgo tests. This codebase uses standard Go testing; Ginkgo is only in vendor dependencies. The check is not applicable.
Test Structure And Quality ✅ Passed This PR does not include any Ginkgo tests. The check for Ginkgo test structure and quality is not applicable; the kubelet-config codebase uses traditional Go unit tests with testify assertions.
Microshift Test Compatibility ✅ Passed This PR does not add any new Ginkgo e2e tests. It only modifies pkg/controller/kubelet-config/kubelet_config_controller.go to fix MCP event handler registration. The check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests are added in this PR. Changes are controller implementation only (kubelet_config_controller.go), so SNO test compatibility check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed The PR modifies only the kubelet config controller (pkg/controller/kubelet-config/kubelet_config_controller.go) to add MachineConfigPool event handlers for reconciliation logic. No deployment manif...
Ote Binary Stdout Contract ✅ Passed PR modifies kubelet config controller (not OTE binary) by adding event handlers with klog.V(4).Infof() calls that execute at runtime, not during process initialization. No stdout writes in process-...
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds only controller production code (MCP event handlers), not new Ginkgo e2e tests. Custom check requires new e2e tests, so it does not apply here.
No-Weak-Crypto ✅ Passed No weak cryptography detected. The PR adds MachineConfigPool event handlers for controller synchronization with no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or non-constant-time secr...
Container-Privileges ✅ Passed PR adds MCP event handlers to kubelet-config controller code only. No container privilege escalation settings (privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation) or man...
No-Sensitive-Data-In-Logs ✅ Passed All logging in the new MCP handlers logs only resource names and error messages; no sensitive data like passwords, tokens, API keys, PII, or customer data is exposed. Error handling follows existin...

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…ol changes

The kubelet config controller registers event handlers for KubeletConfig,
FeatureGate, NodeConfig, and APIServer informers but not for
MachineConfigPool. When a new MCP is created, the controller is never
notified and fails to generate the per-pool kubelet MachineConfigs
(97-<pool>-generated-kubelet, 98-<pool>-generated-kubelet). Users must
restart the machine-config-controller pod for the configs to appear.

Register Add/Update/Delete event handlers on the MCP informer so that
pool additions and label changes trigger reconciliation of all
KubeletConfigs, FeatureGates, and NodeConfigs, ensuring the generated
MachineConfigs are created without requiring a controller restart.

Fixes: openshift#5521
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Ankit Mahajan <ankimaha@redhat.com>
@ankimaha-sys ankimaha-sys force-pushed the kubelet-config-mcp-informer branch from 756cab2 to 5e58077 Compare June 23, 2026 02:46

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/kubelet-config/kubelet_config_controller.go`:
- Around line 371-379: The current error handling in the lister Get calls for
features and nodeConfig only proceeds when err is nil but ignores all other
errors. Instead of silently dropping non-NotFound errors from
ctrl.featLister.Get and ctrl.nodeConfigLister.Get, add proper error handling
that distinguishes between NotFound errors (which are acceptable and should be
skipped) and other transient errors (which should trigger a requeue). Check if
the error is not nil and not a NotFound error, then requeue the kubelet config
for reconciliation to ensure transient cache failures don't cause missed
reconciliation cycles.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 87991c0c-84ff-4592-bb56-88fea32157c4

📥 Commits

Reviewing files that changed from the base of the PR and between 90c582b and 5e58077.

📒 Files selected for processing (1)
  • pkg/controller/kubelet-config/kubelet_config_controller.go

Comment thread pkg/controller/kubelet-config/kubelet_config_controller.go
Log transient errors from FeatureGate and NodeConfig lister lookups
in requeueKubeletConfigsForPool instead of silently ignoring them.
NotFound errors remain acceptable and are skipped.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Ankit Mahajan <ankimaha@redhat.com>
@ankimaha-sys ankimaha-sys force-pushed the kubelet-config-mcp-informer branch from 9fe40f0 to 135ef52 Compare June 23, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The controller doesn't generate %s-generated-kubeletconfig-%s files for new MCPs

1 participant