Prototype prepared Agent surge with capacity fallback

[AliDatadog]

What does this PR do?

Prototypes a default-off prepared Agent rollout using native DaemonSet maxSurge.

The Operator renders a compatible Pod, injects its Pod UID, mounts private rollout state, treats prepared as Ready, and sets surge from the existing maxUnavailable budget while forcing rollout maxUnavailable to zero. Kubernetes performs the normal handoff by deleting the old Pod after the replacement is Ready; the Operator does not orchestrate that handoff.

For profile-managed DaemonSets, the Operator first performs a conventional rollout that narrows the existing broad required anti-affinity, waits for it to become fully Available, and only then enables surge.

If a surged replacement is Pending solely because of node CPU or memory, the Operator can fall back to delete-before-create for that node. It revalidates scheduling, affinity, taints, resource fit, Pod and DaemonSet identities, and consumes the existing maxUnavailable budget before deleting the old Pod with a UID precondition.

flowchart LR
  A[Old Agent active] --> B[Native maxSurge creates replacement]
  B --> C[Replacement pulls, initializes, constructs, reports prepared]
  C --> D[Kubernetes deletes old Pod]
  D --> E[Replacement observes no older sibling and activates]
  B -->|CPU or memory cannot fit both| F[Operator validates budget and fit]
  F --> D

Loading

Motivation

Slow image pulls and process construction should not force collection downtime. Native surge already supplies the create-before-delete lifecycle; this PoC removes the scheduling and process-start conflicts that prevent two Agent Pods from overlapping on one node.

Compatibility boundary

• hostNetwork: true: declared ports are removed; Agent processes still bind after activation. Admission must not add ports back.
• Pod networking without nonzero hostPort: supported for direct Pod-IP and UDS traffic. Node-local Services can select the sleeping Ready Pod and are excluded from the pilot.
• Pod networking with nonzero hostPort: rejected.
• APM and DogStatsD hostPath UDS remain untouched while the replacement sleeps.
• Exactly agent, direct trace-agent, init-volume, and init-config are supported initially.
• ExtendedDaemonSet must be disabled for this native DaemonSet experiment.

Describe how you validated your changes

• Focused controller and renderer packages passed
• Full make lint reported 0 issues
• Generated rollout state uses a private EmptyDir and probes validate PID plus /proc start time
• Skeptical review found no remaining PoC correctness blockers

Coordinated branches:

• Agent: Add experimental prepared startup for Agent rollouts datadog-agent#54032
• Experimental cluster configuration: https://github.com/DataDog/k8s-datadog-agent-ops/pull/9138

Checklist

• Commits are signed
• Enhancement and QA labels applied

[datadog-official]

 

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 65.55% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 65.55%
• Overall Coverage: 47.44% (+0.47%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d6d5717 | Docs | Datadog PR Page | Give us feedback!}

[datadog-official]

Bits couldn’t fix the CI

Warning

This comment is related to an earlier commit.

🟢 Investigated · 🔴 Fix failed · ⚪ Validate · ⚪ Ready

---

View in Datadog | Reviewed commit 0f17b44 · Any feedback? Reach out in #deveng-pr-agent