test(cuda_core): capture machine state on the first CUDA OOM

[juenglin]

Issue #2381 reports intermittent runs where ~190 tests fail with
CUDA_ERROR_OUT_OF_MEMORY.

The error name is misleading: the failures route through Device.memory_resource -> DeviceMemoryResource.__init__ -> _DMR_init, where cuDeviceGetMemPool returns a null handle. Pool setup is failing, not allocation, which is why even a zero-byte allocation fails while thousands of other tests in the same run still pass.

Diagnosing that from a pytest log alone is not possible, because the log cannot show whether the device really was out of memory, which processes held GPU memory, or what the driver returned independently of cuda.core's error reporting. Reproducing it also appears to require a specific driver model, so the evidence has to come from the reporter's machine.

Add helpers/oom_diagnostics.py, which records that evidence at the moment
of failure:

• the driver queried directly (cuMemGetInfo, cuDeviceGetMemPool,
  cuDeviceGetDefaultMemPool), distinguishing genuine exhaustion from an
  unavailable mempool
• nvidia-smi -q for driver model and compute mode
• nvidia-smi --query-compute-apps to identify every process holding
  device memory, which settles whether another tenant is responsible

Capture is latched to the first OOM. A failing run produces ~190 of them,
and invoking nvidia-smi on each would add minutes and bury the log. The
report is written via terminalreporter rather than print() so it survives
the stdout redirection these runs use, and pytest_terminal_summary points
at the artifact from the end of the run, where it is actually noticed.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[juenglin]

Example cuda_core_oom_diagnostics.txt:

==============================================================================
cuda_core diagnostics: first CUDA_ERROR_OUT_OF_MEMORY of this session
==============================================================================
test:      tests/test_zz_demo_delete_me.py::test_first_oom
phase:     call
pid:       18384
platform:  win32
exception: CUDA_ERROR_OUT_OF_MEMORY: The API call failed because it was unable to allocate enough memory or other resources to perform the requested operation.

--- direct driver probe (bypasses cuda.core error reporting) ---
cuCtxGetCurrent() -> (<CUresult.CUDA_SUCCESS: 0>, <CUcontext 0x2cc0c554080>)
cuMemGetInfo() -> (<CUresult.CUDA_SUCCESS: 0>, 24275582976, 25650855936)
cuDeviceGetCount() -> (<CUresult.CUDA_SUCCESS: 0>, 1)
cuDeviceGetMemPool(dev 0) -> (<CUresult.CUDA_SUCCESS: 0>, <CUmemoryPool 0x2cc0cb07cf0>)
cuDeviceGetDefaultMemPool(dev 0) -> (<CUresult.CUDA_SUCCESS: 0>, <CUmemoryPool 0x2cc0cb07cf0>)

$ nvidia-smi -q
(exit 0)
==============NVSMI LOG==============

Timestamp                                              : Thu Jul 30 13:41:04 2026
Driver Version                                         : 595.71
CUDA Version                                           : 13.2

Attached GPUs                                          : 1
GPU 00000000:01:00.0
    Product Name                                       : NVIDIA RTX PRO 5000 Blackwell Generation Laptop GPU
    Product Architecture                               : Blackwell
    Driver Model
        Current                                        : WDDM
        Pending                                        : WDDM
    ... ~80 lines of clocks, ECC, temperature, power ...
    FB Memory Usage
        Total                                          : 24463 MiB
        Reserved                                       : 326 MiB
        Used                                           : 222 MiB
        Free                                           : 23916 MiB
    BAR1 Memory Usage
        Total                                          : 32768 MiB
    ... ~140 more lines ...
    Capabilities
        EGM                                            : disabled

$ nvidia-smi --query-compute-apps=timestamp,pid,process_name,used_memory --format=csv
(exit 0)
timestamp, pid, process_name, used_gpu_memory [MiB]
2026/07/30 13:41:04.256, 18384, C:\Program Files\Python313\python.exe, [N/A]
==============================================================================

[juenglin]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2458/
https://nvidia.github.io/cuda-python/pr-preview/pr-2458/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2458/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2458/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.