Detect nodes that never recover after reboot (reboot timeout → degraded)

[ptalgulk01]

Summary

The operator has no mechanism to detect a node that fails to come back after a
reboot. Once a node enters Rebooting, if it never returns (bad image, hardware
fault, node shutdown), it stays in that state indefinitely — the BootcNode is
never marked Degraded and the reboot slot is held forever.

This blocks two scenarios from #69:

• E2e test planning #69 item 4 — "Shutdown (manually) a node after the reboot to simulate a
  faulty node, after the upgrade. It should be reported as degraded."
  → Today it is not reported as degraded.
• E2e test planning #69 item 8 — "Switch to a completely different image e.g. fedora to centos
  stream." → Node reboots into a kubelet-less image and gets stuck in
  Rebooting forever. (Expected behavior for this case is undefined and should
  be decided as part of this work.)

Reproduction (verified on a bink cluster)

Faulty node after upgrade (#69 item 4): node updates successfully
(updatedCount=1), then the VM is stopped. Observed, stable for 6+ min:

Signal	Value
Node Ready	Unknown
BootcNode Degraded	False
Pool degradedCount	0

Cross-distro bad image, no kubelet (#69 item 8): node reboots into the bad
image and never returns. Observed, stable for 5+ min:

Signal	Value
Node Ready	Unknown
BootcNode Idle	False (Rebooting)
Pool degradedCount	0

Controller log confirms the slot is wedged:
Rollout state ... rebooting: 1, degraded: 0, occupiedSlots: 1, availableSlots: 0, candidates: []

Root cause

• classifyNode() (internal/controller/rollout.go:614) classifies a node as rebooting while Idle=False/Reason=Rebooting (:658). There is no time bound on this state.
• findUnhealthySlots() (rollout.go:460) only scans degraded and upToDate nodes. A node stuck in rebooting is in neither list, so it is never counted as unhealthy.
• The halt logic requires >= unhealthySlotHaltThreshold (== 2, rollout.go:34,96) unhealthy nodes, so a single stuck node never halts the rollout.
• The daemon sets Rebooting then calls systemctl reboot (internal/bootc/executor.go:144); if the node never returns it cannot report anything because it is down. Detection must therefore be controller-side.

Proposed change

1. Reboot timeout (controller-side). Track time since a node entered Rebooting (the Idle=False/Rebooting condition's LastTransitionTime). If it exceeds a timeout, transition the BootcNode to Degraded with a new reason RebootTimeout.
2. API. Add RebootTimeoutSeconds to DisruptionSpec/RolloutSpec (default ~10 min) and a NodeReasonRebootTimeout constant.
3. Feed existing halt logic. Timed-out nodes become degraded, which flows into findUnhealthySlots() / PoolRolloutHalted (interacts with Support configuring rollout halt condition #99).
4. Decide E2e test planning #69 item 8 semantics. Define expected behavior for a node that boots an image without kubelet (reject upfront? same reboot-timeout →degraded?).

Related

• E2e test planning #69 (E2e test planning — items 4 and 8)
• Support configuring rollout halt condition #99 (configurable rollout halt threshold)