Skip to content

fix(samples): respect CUDA_VISIBLE_DEVICES in multi-GPU cuda.core samples - #3220

Open
mangeshraut712 wants to merge 3 commits into
NVIDIA:mainfrom
mangeshraut712:fix/multi-gpu-visible-devices
Open

mangeshraut712 wants to merge 3 commits into
NVIDIA:mainfrom
mangeshraut712:fix/multi-gpu-visible-devices

Conversation

@mangeshraut712

@mangeshraut712 mangeshraut712 commented Oct 25, 2026 •

Copy link
Copy Markdown

Closes #3215.

Summary

Updated simpleP2P, simpleMultiGpu, and multiGPUGradientAverage to size device enumeration with len(Device.get_all_devices()), which follows CUDA runtime visibility including CUDA_VISIBLE_DEVICES. Corrected the two affected READMEs and added a focused regression test. Kept system.get_num_devices() unchanged for NVML system inventory use.

Files

  • samples/cuda_core/simpleP2P/simpleP2P.py
  • samples/cuda_core/simpleP2P/README.md
  • samples/cuda_core/simpleMultiGpu/simpleMultiGpu.py
  • samples/cuda_core/simpleMultiGpu/README.md
  • samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py
  • cuda_core/tests/example_tests/test_sample_verification.py

Tip

b462a85c70f6184c928a21dff11a19f50bd49d91

Test plan

  • git diff --check
  • AST/parse check on touched sample + test modules
  • Await NVIDIA runner CI after copy-pr-bot / metadata gate clears

@copy-pr-bot

copy-pr-bot Bot commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-25T21:47:17.603005Z
We couldn’t complete this review. Comment @codex review to try again.
b462a85 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 25, 2026
@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Updates
    • Multi-GPU samples now count devices visible to CUDA, so their reported count reflects CUDA_VISIBLE_DEVICES settings rather than all CUDA-capable GPUs.
    • Samples still require at least two visible devices to run multi-GPU workflows; they skip those workflows when fewer are available.
    • Updated sample documentation to describe device visibility, device counting, and the minimum-device requirement.

The three multi-GPU samples now count CUDA-visible devices with len(Device.get_all_devices()) instead of system.get_num_devices(). Their README text and a parametrized test were updated to reflect and check this API.

Changes: CUDA-visible device enumeration

Layer / File(s) Summary
Update multi-GPU sample device counting
samples/cuda_core/simpleP2P/simpleP2P.py, samples/cuda_core/simpleMultiGpu/simpleMultiGpu.py, samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py, samples/cuda_core/simpleP2P/README.md, samples/cuda_core/simpleMultiGpu/README.md, cuda_core/tests/example_tests/test_sample_verification.py
All three samples count devices with len(Device.get_all_devices()). The README text describes the API, and the test checks that the samples use it instead of get_num_devices.

Assessment against linked issues

Objective Addressed Explanation
Count devices available to the process in the three multi-GPU samples [#3215] ✅
Update the README descriptions of device counting [#3215] ✅

Possibly related PRs

  • NVIDIA/cuda-python#3198 — Both changes use process-visible device enumeration. This PR applies it to sample device counts.
  • NVIDIA/cuda-python#3126 — Both changes concern visible-device mapping. This PR changes sample device counts.

Suggested reviewers: danielfrg

Priority: Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Low · up to 0f119

The samples work with visible-device enumeration, but the simpleP2P README should be updated to match the output before merging.

Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Mangesh Raut <mbr63@drexel.edu>
@mangeshraut712

Copy link
Copy Markdown
Author

I checked the review feedback and local lint. The Codex review did not publish a concrete finding and CodeRabbit reported no actionable comments; I only fixed the Ruff formatting violation in the sample-verification test.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Minor · 📐 Maintainability & Code Quality · README.md:135

samples/cuda_core/simpleP2P/README.md:135
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

suggestion: Update the expected output label.

The sample now prints CUDA-visible device count, but the README still documents CUDA-capable device count. Update the expected output to match the sample.

Suggested fix
-CUDA-capable device count: 2
+CUDA-visible device count: 2

Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 10d7010 and 0f1192a.

Files selected for processing: 3
  • samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py
  • samples/cuda_core/simpleMultiGpu/simpleMultiGpu.py
  • samples/cuda_core/simpleP2P/simpleP2P.py
Files with no reviewable changes: 2
  • samples/cuda_core/multiGPUGradientAverage/multiGPUGradientAverage.py
  • samples/cuda_core/simpleP2P/simpleP2P.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: multi-GPU cuda.core samples take the CUDA device count from NVML and break under CUDA_VISIBLE_DEVICES

1 participant