Skip to content

fix: constrain NVIDIA device plugin to GPU nodes - #8866

Open
atharvaHJoshi wants to merge 1 commit into
eksctl-io:mainfrom
atharvaHJoshi:fix/8858-nvidia-device-plugin
Open

atharvaHJoshi wants to merge 1 commit into
eksctl-io:mainfrom
atharvaHJoshi:fix/8858-nvidia-device-plugin

Conversation

@atharvaHJoshi

Copy link
Copy Markdown

Fixes #8858

Related to #8859.

What this PR does

The NVIDIA device-plugin DaemonSet had no nodeSelector/node affinity, so a DaemonSet
replica was scheduled on every node, including CPU-only nodes where it has no GPU
to manage — crash-looping and potentially taking down unrelated nodes via DiskPressure.

How this differs from #8859

#8859 adds a nodeSelector on nvidia.com/gpu.present: true. That label is not set
by anything in the standard eksctl flow
(eksctl installs the device plugin directly; it
does not use the GPU Operator / node-feature-discovery), so under that PR the DaemonSet
would never schedule at all and GPUs would not be exposed as resources.

This implementation instead:

  • derives NVIDIA-capable instance types from the eksctl cluster configuration
    (unmanaged InstanceType + InstancesDistribution.InstanceTypes, managed
    InstanceType + InstanceTypes),
  • applies a required node affinity using the standard
    node.kubernetes.io/instance-type label (always populated by kubelet), and
  • also excludes Fargate/hybrid/auto compute nodes via
    eks.amazonaws.com/compute-type NotIn.

This works reliably, including on mixed instance-type nodegroups where only the GPU
variants run the plugin.

Changes

  • Add SetNodeAffinity to the DevicePlugin interface; no-op for Neuron/EFA.
  • Implement NVIDIA-specific node affinity in NvidiaDevicePlugin.SetNodeAffinity.
  • Add regression tests (unmanaged, managed, and mixed instance distributions;
    no-op for CPU-only clusters).

Validation

  • go build ./...
  • golangci-lint run --timeout=30m ./pkg/...
  • go test ./pkg/addons/...

Pre-existing, unrelated failures in pkg/info and pkg/iam/oidc are environmental
(kubectl/cfssl missing locally) and remain unchanged.

Signed-off-by: Atharva Joshi <josh.atharva2005@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Hello atharvaHJoshi 👋 Thank you for opening a Pull Request in eksctl project. The team will review the Pull Request and aim to respond within 1-10 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] nvidia-device-plugin-daemonset has no nodeSelector, gets scheduled onto non-GPU nodes too (and can take down an unrelated node with DiskPressure)

1 participant