Proposal: Restart handover mode (roll the target instead of surging replicas) #12

Problem

The scale handover (raise spec.replicas, then steer the scale-down with pod-deletion-cost) suits workloads that tolerate a second copy. For single-replica targets, rolling the target out would avoid several of its properties:

• The update strategy is reported, not honoured. A Recreate Deployment is often single-replica because two copies must never coexist: exclusive upstream sessions, nonce-ordered transaction senders, ReadWriteOnce volumes. ZeroDowntimeUpdateStrategy=False is set (understudy_controller.go#L130-L131), but the surge still runs both pods.
• A surge lasts as long as the signal, not the eviction. A cordon that is never followed by an eviction (a manual cordon during an incident) keeps both copies running until the node is uncordoned, because nothing ends a surge whose stand-in is ready.
• It writes spec.replicas. Managing that is what baselineReplicas and the Argo ignoreDifferences contract exist for, and a kubectl apply of a manifest with replicas: 1 during a handover still undoes the surge.
• It needs update/patch on every pod in the cluster (rbac.yaml#L27-L29), only to set controller.kubernetes.io/pod-deletion-cost.
• Availability is re-derived by the reconciler and the webhook instead of by the controller that owns it; Count a stand-in only once it is available, honouring minReadySeconds #9 adds minReadySeconds to that logic.

Proposal

Add an opt-in spec.handover field: Scale (today's behaviour, and the default) or Restart.

In Restart mode, when a serving pod's node is doomed, the operator patches a pod-template annotation once per doom episode, for example understudy.sh/handover-at: <RFC3339>. That is the mechanism behind kubectl rollout restart. The Deployment controller then replaces the pod using the workload's own strategy:

• RollingUpdate with maxSurge: 1, maxUnavailable: 0 brings the new pod up first, honours readiness gates, minReadySeconds and progressDeadlineSeconds, and removes the old pod itself. The overlap is exactly a deploy's.
• Recreate stops the old pod before starting the new one. There's no availability gain, but it never runs two copies.

The webhook keeps holding the drainer until the rollout has completed or the old pod is gone, and readinessDeadlineSeconds still bounds the hold.

For these targets the replica machinery goes away: no scale-back, no deletion-cost stamping, no base-replica annotations. A release that starts mid-handover supersedes it as the newer ReplicaSet.

Trade-offs

• Each handover leaves a ReplicaSet revision, so rollout history and rollout undo get noisier.
• A restart replaces every replica, so the mode only fits single-replica targets. It should reject or warn when replicas > 1.
• The template annotation is drift for self-healing GitOps. Reverting it would trigger a second rollout, so it needs an ignoreDifferences entry. Plain kubectl apply leaves it alone, as it does for kubectl rollout restart.
• A canary Rollout would run its full steps and analysis on a template change, so Restart probably fits Deployments only, at least at first.
• Every trigger now costs a restart, so flapping signals are worth excluding in this mode. One example is DeletionCandidateOfClusterAutoscaler, which only marks a node as unneeded.
• deployments: patch is still required, but cluster-wide pod writes are not.

Open questions

• If the target is already mid-release when a node is doomed, should the handover wait for the release, or count it as the handover?
• With Restart, a plain PodDisruptionBudget with minAvailable: 1 could replace the webhook hold where blocking consolidation is acceptable, since it lets the eviction through once the new pod is Ready. It counts Ready pods, not available ones (disruption.go#L929), so it can release up to minReadySeconds early. Is that worth documenting, or out of scope?

🤖 Generated with Claude Code