Skip to content

[BUG]: Device.set_current() discards real CUDA error and raises misleading ValueError #2460

Description

@juenglin

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Summary

When cuDevicePrimaryCtxRetain fails inside Device.set_current(), the actual CUDA driver error is silently discarded and the user gets:

ValueError: Cannot set NULL context as current

The real cause (e.g. CUDA_ERROR_DEVICE_UNAVAILABLE, CUDA_ERROR_OUT_OF_MEMORY, exclusive compute mode conflict) is lost.

Root cause

In cuda_core/cuda/core/_device.pyx lines 1289–1292, get_primary_context() returns an empty handle when the driver call fails and records the error in the thread-local slot. The caller never reads it:

h_context = get_primary_context(self._device_id)
if h_context.get() == NULL:
    raise ValueError("Cannot set NULL context as current")

This raises the wrong exception type (every other driver failure in cuda.core raises CUDAError) and leaves the thread-local error state polluted, potentially causing a stale error to be misreported by a later call.

How to Reproduce

This is hit whenever cuDevicePrimaryCtxRetain fails for a valid device ordinal. One way to trigger it:

  1. Set the GPU to exclusive compute mode: nvidia-smi -c EXCLUSIVE_PROCESS
  2. Hold a context from another process
  3. Call Device(0).set_current() from a second process

Expected behavior

Device.set_current() should surface the underlying driver error as a CUDAError, matching the pattern used elsewhere (e.g. _context.pyx:59–61).

Operating System

any

nvidia-smi output

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions