fix(cuda.core): fall back to driver when nvJitLink < 12.3 is installed by atiaomar1978-hub · Pull Request #2409 · NVIDIA/cuda-python

atiaomar1978-hub · 2026-07-23T04:37:00Z

Summary

When nvJitLink 12.0–12.2 is installed, Linker.which_backend() and Linker construction raised FunctionNotFoundError: function nvJitLinkVersion is not found instead of warning and falling back to the driver (cuLink) backend.

This is a regression introduced in cuda-core 0.7.0. cuda-core 0.6.0 handled this correctly.

Root cause

nvJitLink 12.0–12.2 only export versioned symbols (__nvJitLink*_12_X); the unversioned nvJitLinkVersion export was added in 12.3.

Since cuda-core >= 0.7.0, _decide_nvjitlink_or_driver() probed availability via:

_optional_cuda_import(
    "cuda.bindings.nvjitlink",
    probe_function=lambda module: module.version(),
)

On nvJitLink 12.0–12.2, module.version() calls the missing unversioned symbol and raises FunctionNotFoundError. _optional_cuda_import() only treats DynamicLibNotFoundError as "unavailable", so the exception escapes instead of triggering driver fallback.

cuda-core 0.6.0 used _inspect_function_pointer("__nvJitLinkVersion") (symbol pointer inspection, no function call), which handled this correctly.

Fix

Stop probing via module.version(). Import cuda.bindings.nvjitlink without a probe function and rely on the existing _nvjitlink_has_version_symbol() helper for the >= 12.3 check.

Reproduction (before fix)

pip install "cuda-core==1.1.0" "cuda-bindings==12.9.7" "cuda-toolkit[nvjitlink]==12.0.1"
python -c "from cuda.core import Linker; print(Linker.which_backend())"

Result: FunctionNotFoundError: function nvJitLinkVersion is not found

Independently confirmed on Windows x86_64 (no NVIDIA GPU) with Python 3.12.10 and nvidia-nvjitlink-cu12==12.0.140.

Expected (and restored) behavior:

>>> Linker.which_backend()
RuntimeWarning: ... is too old (<12.3) ...
'driver'

Tests added

test_decide_nvjitlink_or_driver_falls_back_when_nvjitlink_too_old
test_decide_nvjitlink_or_driver_selects_nvjitlink_when_version_symbol_present
test_which_backend_falls_back_when_nvjitlink_too_old (Linker API regression test for [BUG]: Linker raises FunctionNotFoundError instead of falling back to the driver when nvJitLink < 12.3 is installed #2408)
Updated existing optional-import tests to assert no version() probe on import

Test plan

Added unit/integration coverage for nvJitLink < 12.3 fallback path
Ran cuda_pathfinder/tests/test_optional_cuda_import.py locally (5 passed)
cuda_core tests require NVIDIA driver/GPU locally (nvcuda.dll); CI will run full suite
Manual repro with cuda-toolkit[nvjitlink]==12.0.1 returns "driver" with RuntimeWarning

CI note

The failing PR has assignee, labels, and milestone check is a metadata gate, not a code/test failure. External contributors cannot add the required bug label, assignee, or milestone. Could a maintainer please add:

Label: bug
Milestone: cuda.core 1.2.0 (or appropriate release)
Assignee: triage owner

Signed-off-by: Omar Atie atiaomar1978-hub@users.noreply.github.com

copy-pr-bot · 2026-07-23T04:37:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

atiaomar1978-hub · 2026-07-23T04:59:36Z

CI run 29979997352

The failing job \PR has assignee, labels, and milestone\ is a metadata gate (missing \�ug\ label, assignee, and milestone). I cannot set these as an external contributor.

Updates in latest push

Added regression coverage:
- \ est_which_backend_falls_back_when_nvjitlink_too_old\
- \ est_decide_nvjitlink_or_driver_falls_back_when_nvjitlink_too_old\
- \ est_decide_nvjitlink_or_driver_selects_nvjitlink_when_version_symbol_present\
Local tests run:
- \cuda_pathfinder/tests/test_optional_cuda_import.py: 5 passed
- \cuda_core/tests/test_optional_dependency_imports.py: blocked locally by missing
  vcuda.dll\ (no GPU on this machine); CI should cover these

Could a maintainer please add the \�ug\ label, assignee, and milestone so the metadata check passes?

atiaomar1978-hub · 2026-07-23T16:08:31Z

About me:
I’m Omar Atie, an open‑source contributor with experience building high‑performance integration systems, AI components, and modern data‑processing features in Apache Camel. My work spans distributed systems, API design, and component development, with a focus on reliability, performance, and clean architecture. I’m now expanding into CUDA open‑source development to contribute to GPU‑accelerated computing, improve developer experience, and help strengthen NVIDIA’s ecosystem through practical, high‑impact contributions.

Request:
I’d appreciate feedback from maintainers on how to best align this contribution with NVIDIA’s internal coding standards.
If workflow approvals, GitHub Actions permissions, or metadata updates are required, please guide me and I’ll make the necessary adjustments.
I’m also interested in becoming an active contributor in the CUDA ecosystem, so any guidance on how to integrate effectively with the team and contribution process would be greatly appreciated.

atiaomar1978-hub · 2026-07-24T00:49:04Z

Added release notes entry

Pushed a follow-up commit adding a cuda.core 1.2.0 release note under Fixes and enhancements documenting the #2408 regression fix (cuda_core/docs/source/release/1.2.0-notes.rst).

All commits on this branch are signed and show as verified on GitHub.

Current branch

Commit	Purpose	Verified
`4ee6ae0`	Fix: fall back to driver when nvJitLink < 12.3	✅
`7f6a534`	Tests: regression coverage	✅
`91a1f4f`	Docs: 1.2.0 release note	✅

Still blocked only by maintainer metadata (assignee, \�ug\ label, milestone) and the external-contributor workflow approval (\copy-pr-bot). Happy to adjust anything on request.

atiaomar1978-hub · 2026-07-24T23:34:40Z

Hi @rparolin — could you help unblock this PR for #2408?

Verified status (as of latest push on `fix/linker-nvjitlink-fallback-2408`)

Item	Status
Commits	`4ee6ae0` (fix), `7f6a534` (tests), `91a1f4f` (1.2.0 release note) — DCO `Signed-off-by`, GitHub verified signatures
`pre-commit.ci - pr`	Pass
Module label on PR	`cuda.core` (present)
PR metadata gate	Fail — missing assignee, `bug` type label, and milestone on the PR
Issue #2408 metadata	Already has `bug`, milestone `cuda.core 1.2.0`, and assignee @atiaomar1978-hub (issue assignment does not carry over to the PR)

I cannot set PR assignee / milestone / some labels as an external contributor. Once those are on the PR, the metadata workflow should re-run on labeled / assigned events.

NVIDIA CI / copy-pr-bot

copy-pr-bot reported that workflows on NVIDIA runners need additional validation before they can run. After metadata is fixed, could you (or another CPR vetter) vet/approve so the full cuda.core test pipeline can run?

Happy to adjust anything in review. Thanks for taking a look.

— Omar (@atiaomar1978-hub)

rparolin · 2026-07-30T00:36:52Z

@isVoid Could please review this? Thanks.

isVoid · 2026-07-30T01:14:12Z

+    # Do not probe via module.version(): nvJitLink 12.0-12.2 lacks the unversioned
+    # nvJitLinkVersion symbol, so calling version() raises FunctionNotFoundError.
+    # Use _nvjitlink_has_version_symbol() below instead (symbol pointer inspection).
+    nvjitlink_module = _optional_cuda_import("cuda.bindings.nvjitlink")


The general solution looks good: we validate the nvjitlink module by checking the existence of __nvJitLinkVersion to see if it's "too old". _nvjitlink_has_version_symbol under the hood loads the nvjitlink module and inspect the function pointer, in theory this provide as strong validation as module.version, and is safer.

Nit to change: if we lift the probe function to below, this function is now out of the protection scope of try: ... except DynamicLibNotFoundError inside _optional_cuda_import. Imagine a case where the dylib is missing while the binding module exists, this error could surface and we won't be able to gracefully fallback to culink. We should place the version check inside this try except block somehow.

Thanks @isVoid - addressed in 5478f2d.

The version-symbol check now runs inside _optional_cuda_import's probe (_probe_nvjitlink), so a missing nvJitLink dylib raises DynamicLibNotFoundError and falls back to cuLink. We still avoid module.version() so nvJitLink 12.0-12.2 does not raise FunctionNotFoundError (#2408).

Distinct warnings are preserved:

binding/dylib unavailable -> not available

symbol missing (<12.3) -> too old (<12.3)

Also added coverage for the missing-dylib path and a guard that the probe does not call module.version().

atiaomar1978-hub · 2026-07-30T03:14:48Z

Addressed review feedback (@isVoid)

Pushed 5478f2d (signed / verified):

Keep the >=12.3 version-symbol check inside _optional_cuda_import's probe so DynamicLibNotFoundError (missing dylib) still falls back to cuLink
Continue avoiding module.version() (fixes [BUG]: Linker raises FunctionNotFoundError instead of falling back to the driver when nvJitLink < 12.3 is installed #2408 on nvJitLink 12.0–12.2)
Added tests for missing-dylib fallback + probe-must-not-call-version() guard

Local tests

test_optional_dependency_imports.py: 9 passed
cuda_pathfinder/.../test_optional_cuda_import.py: 5 passed

isVoid

Additionally, please add "test authorship" markers per requested in AGENTS.md.

isVoid · 2026-07-30T17:47:29Z

+    has_version_symbol = [False]
+
+    def _probe_nvjitlink(module):
+        from cuda.bindings._internal import nvjitlink
+
+        has_version_symbol[0] = _nvjitlink_has_version_symbol(nvjitlink)


I'm not really a fan of introducing side-effects inside the function. Here I think using an explicit try-except block is probably more readable:

mod = _optional_cuda_import(...) if mod is None: ... else: try: has_version_symbol = _nvjitlink_has_version_symbol(mod) except DynamicLibNotFoundError: ...

Thanks @isVoid — updated in d84fa43.

Dropped the probe side-effect. We now import with _optional_cuda_import(...) (no probe), then use an explicit try/except DynamicLibNotFoundError around _nvjitlink_has_version_symbol, matching the shape you suggested. Missing dylib still falls back to cuLink; nvJitLink <12.3 still gets the “too old” warning without calling module.version().

atiaomar1978-hub

Also addressed the authorship-marker request: newly added tests now use @pytest.mark.agent_authored(model="grok-4.5") per AGENTS.md.

Stop probing nvJitLink availability via module.version(), which calls the unversioned nvJitLinkVersion symbol missing in nvJitLink 12.0-12.2. Use symbol pointer inspection via _nvjitlink_has_version_symbol() instead, restoring cuda-core 0.6.0 fallback behavior. Fixes NVIDIA#2408 Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Add regression tests for Linker.which_backend() and _decide_nvjitlink_or_driver() when the nvJitLinkVersion symbol is missing (nvJitLink 12.0-12.2). Related to NVIDIA#2408 Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Document the NVIDIA#2408 regression fix in the cuda.core 1.2.0 release notes. Related to NVIDIA#2408 Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

… guard Address review feedback: keep the >=12.3 version-symbol check inside _optional_cuda_import's probe so a missing nvJitLink dylib still falls back to cuLink. Continue avoiding module.version(), which raises FunctionNotFoundError on nvJitLink 12.0-12.2 (NVIDIA#2408). Add coverage for missing-dylib fallback and a guard that the probe does not call module.version(). Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

Address review feedback: drop the probe side-effect and catch DynamicLibNotFoundError around _nvjitlink_has_version_symbol so missing dylibs still fall back to cuLink. Keep avoiding module.version() for nvJitLink <12.3 (NVIDIA#2408). Mark newly added tests with agent_authored authorship markers. Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

isVoid · 2026-07-30T23:13:38Z

/ok to test 210c7a2

isVoid

One more nit on review round trips: let's try not to force push. This rewrites the review history and could make the PR hard to read.

isVoid · 2026-07-30T23:12:51Z

+    # Do not call module.version(): nvJitLink 12.0-12.2 lack the unversioned
+    # nvJitLinkVersion export, so version() raises FunctionNotFoundError (#2408).
+    # Inspect the symbol pointer instead, and catch DynamicLibNotFoundError when
+    # the dylib is missing so we can fall back to cuLink.


We can probably drop these comments. They simply document a fixed bug.

Thanks @isVoid — dropped those comments in e6f3923.

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

atiaomar1978-hub

Thanks for the note on force-pushes — understood. I rebased earlier to sync with main; I will avoid rewriting history on further review rounds and push incremental commits instead.

isVoid · 2026-07-30T23:19:25Z

/ok to test e6f3923

atiaomar1978-hub · 2026-07-31T00:12:36Z

@isVoid I noticed CI is showing three failing jobs. Is there anything I should do on my side, or is this something that needs to be triggered/handled by a maintainer?

isVoid · 2026-07-31T00:17:15Z

No I think these are either one time hiccups or CI breakage that exists on other PRs too.

github-actions · 2026-07-31T00:30:32Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2409/
https://nvidia.github.io/cuda-python/pr-preview/pr-2409/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2409/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2409/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

github-actions Bot added the cuda.core Everything related to the cuda.core module label Jul 23, 2026

atiaomar1978-hub mentioned this pull request Jul 23, 2026

[BUG]: Linker raises FunctionNotFoundError instead of falling back to the driver when nvJitLink < 12.3 is installed #2408

Open

1 task

atiaomar1978-hub force-pushed the fix/linker-nvjitlink-fallback-2408 branch from f0c5775 to 7f6a534 Compare July 23, 2026 23:08

isVoid self-requested a review July 29, 2026 22:21

isVoid added this to the cuda.core 1.2.0 milestone Jul 29, 2026

rparolin assigned atiaomar1978-hub Jul 30, 2026

isVoid requested changes Jul 30, 2026

View reviewed changes

isVoid added the bug Something isn't working label Jul 30, 2026

atiaomar1978-hub requested a review from isVoid July 30, 2026 03:17

isVoid reviewed Jul 30, 2026

View reviewed changes

atiaomar1978-hub commented Jul 30, 2026

View reviewed changes

atiaomar1978-hub and others added 5 commits July 30, 2026 15:59

atiaomar1978-hub force-pushed the fix/linker-nvjitlink-fallback-2408 branch from d84fa43 to 210c7a2 Compare July 30, 2026 23:03

atiaomar1978-hub requested a review from isVoid July 30, 2026 23:04

isVoid reviewed Jul 30, 2026

View reviewed changes

fix(cuda.core): drop obsolete nvJitLink probe comments

e6f3923

Signed-off-by: Omar Atie <atiaomar1978-hub@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

atiaomar1978-hub commented Jul 30, 2026

View reviewed changes

Uh oh!

Conversation

atiaomar1978-hub commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Fix

Reproduction (before fix)

Tests added

Test plan

CI note

Uh oh!

copy-pr-bot Bot commented Jul 23, 2026

Uh oh!

atiaomar1978-hub commented Jul 23, 2026

CI run 29979997352

Updates in latest push

Uh oh!

atiaomar1978-hub commented Jul 23, 2026

Uh oh!

atiaomar1978-hub commented Jul 24, 2026

Added release notes entry

Current branch

Uh oh!

atiaomar1978-hub commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verified status (as of latest push on fix/linker-nvjitlink-fallback-2408)

NVIDIA CI / copy-pr-bot

Uh oh!

rparolin commented Jul 30, 2026

Uh oh!

isVoid Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub commented Jul 30, 2026

Addressed review feedback (@isVoid)

Local tests

Uh oh!

isVoid left a comment

Choose a reason for hiding this comment

Uh oh!

isVoid Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isVoid commented Jul 30, 2026

Uh oh!

isVoid left a comment

Choose a reason for hiding this comment

Uh oh!

isVoid Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

atiaomar1978-hub left a comment

Choose a reason for hiding this comment

Uh oh!

isVoid commented Jul 30, 2026

Uh oh!

atiaomar1978-hub commented Jul 31, 2026

Uh oh!

isVoid commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Reviewers

Assignees

Labels

atiaomar1978-hub commented Jul 23, 2026 •

edited

Loading

atiaomar1978-hub commented Jul 24, 2026 •

edited

Loading

Verified status (as of latest push on `fix/linker-nvjitlink-fallback-2408`)

atiaomar1978-hub left a comment •

edited

Loading