Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux by Copilot · Pull Request #29590 · microsoft/onnxruntime

Copilot · 2026-07-07T05:04:21Z

Description

onnxruntime-gpu 1.27 introduced a hard NEEDED libcudart.so.13 entry in onnxruntime_pybind11_state.so, causing ImportError at import onnxruntime on CPU-only Linux machines — before any provider is selected.

Root cause: cmake/onnxruntime_python.cmake was changed to compile fpA_intB_gemm_adaptor.cu and fpA_intB_gemm_preprocessors_impl.cu directly into onnxruntime_pybind11_state.so and link CUDA::cudart (dynamic). This embeds a load-time CUDA dependency in the Python module itself.

Fix: Move the CUDA weight-preprocessing entry point (pack_weights_for_cuda_mixed_gemm) out of the main pybind module and into a standalone extension module, onnxruntime_cuda_quant_preprocess, that links CUDA::cudart on its own. The main onnxruntime_pybind11_state.so no longer compiles or links any CUDA code, so import onnxruntime has no libcudart dependency. The new module is imported lazily by onnxruntime/python/tools/quantization/cuda_quantizer.py only when weight prepacking is actually requested — never at import onnxruntime time.

These preprocessing APIs are offline-only helpers: they are used by quantization tooling and model builders to produce prepacked weight initializers ahead of time, and are not part of the inference runtime hot path. Because nothing in the runtime imports them, isolating them into a separate, on-demand DLL has no runtime cost and cleanly keeps CUDA out of the base import onnxruntime path.

Why not the provider bridge: An earlier iteration routed the call through the ProviderInfo_CUDA virtual interface (TryGetProviderInfo_CUDA()). That does not work for the CUDA-EP-as-plugin build (onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON): cuda_provider_factory.cc is excluded from the plugin sources and there is no provider bridge, so TryGetProviderInfo_CUDA() returns nullptr and the call throws. The standalone module has no such dependency and works for both the legacy in-tree CUDA EP build and the plugin build.

Key Changes

File	Change
`onnxruntime/python/onnxruntime_pybind_cuda_quant.cc`	New. Self-contained `pack_weights_for_cuda_mixed_gemm` (device malloc + transpose/convert + arch permutation) and a `PYBIND11_MODULE(onnxruntime_cuda_quant_preprocess, …)` entry point.
`cmake/onnxruntime_python.cmake`	Add the `onnxruntime_cuda_quant_preprocess` module target (built when `onnxruntime_USE_CUDA AND NOT WIN32`, compiling the two `fpA_intB` `.cu` files + `CUDA::cudart` + cutlass, hidden visibility) and copy it into `onnxruntime/capi/`. Main pybind module keeps no CUDA sources/links.
`onnxruntime/python/onnxruntime_pybind_quant.cc`	Remove the `USE_CUDA` `PackWeightsForMixedGemm` and its registration. The CPU-only `pack_fp4_weights_for_cuda_moe_gemm` stays in the main module.
`onnxruntime/core/providers/cuda/cuda_provider_factory.{h,cc}`	Revert the `PackWeightsForMixedGemm` `ProviderInfo_CUDA` addition (no longer needed; absent in plugin builds).
`onnxruntime/python/tools/quantization/cuda_quantizer.py`	`_get_pack_weights_for_cuda_mixed_gemm()` now imports `onnxruntime.capi.onnxruntime_cuda_quant_preprocess` lazily; add `has_cuda_weight_prepacking()` capability helper.
`setup.py`	Package `onnxruntime_cuda_quant_preprocess.so` in the Linux/macOS wheels.
`onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py`	Point the prepacked-weight parity test and its skip guard at the new module.
`docs/contrib_ops/cuda/matmul_nbits.md`	Update the offline-packer code snippets to import the new module.

Motivation and Context

import onnxruntime must succeed on CPU-only machines even when the GPU wheel is installed. CUDA dependency errors should surface only when a CUDA provider is explicitly loaded/selected, or when offline CUDA weight prepacking is explicitly requested. This restores the 1.26 behavior where onnxruntime_pybind11_state.so had no NEEDED libcudart.so.* entry, and — unlike the provider-bridge approach — it also works in the CUDA-EP-as-plugin build.

Testing Notes

Built both modules in the CUDA build; readelf -d onnxruntime_pybind11_state.so shows no libcudart NEEDED entry, while onnxruntime_cuda_quant_preprocess.so has NEEDED libcudart.so.13.
import onnxruntime and lazy loading of onnxruntime.capi.onnxruntime_cuda_quant_preprocess both succeed; has_cuda_weight_prepacking() returns True on a CUDA machine.
test_op_matmulnbits_prepacked_cuda.py passes (INT4/INT8 prepacked-vs-runtime parity), confirming the relocated packer produces byte-identical prepacked weights.

Copilot

Pull request overview

This PR removes a load-time CUDA runtime dependency (libcudart.so.*) from onnxruntime_pybind11_state.so by moving the CUDA mixed-GEMM weight prepacker (pack_weights_for_cuda_mixed_gemm) into a standalone, lazily-imported Python extension module (onnxruntime_cuda_quant_preprocess). This restores import onnxruntime behavior on CPU-only Linux machines even when the GPU wheel is installed, while still enabling offline CUDA weight prepacking when explicitly requested.

Changes:

Introduce a new standalone pybind extension module (onnxruntime_cuda_quant_preprocess) that links CUDA::cudart and hosts pack_weights_for_cuda_mixed_gemm.
Update Python quantization tooling/tests/docs to import the new module lazily and use it for prepacked-weight generation.
Update wheel packaging to include the new extension module in Linux/macOS artifacts.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`cmake/onnxruntime_python.cmake`	Stops compiling CUDA sources into the main pybind module; adds a new standalone CUDA prepack extension target.
`onnxruntime/python/onnxruntime_pybind_cuda_quant.cc`	New standalone pybind module implementing `pack_weights_for_cuda_mixed_gemm`.
`onnxruntime/python/onnxruntime_pybind_quant.cc`	Removes the CUDA mixed-GEMM packer from the main pybind module; keeps CPU-only FP4 packer.
`onnxruntime/python/tools/quantization/cuda_quantizer.py`	Lazily imports the new extension and adds a capability helper for CUDA weight prepacking.
`onnxruntime/test/python/quantization/test_op_matmulnbits_prepacked_cuda.py`	Updates parity test to use the new extension module (and skips when unavailable).
`docs/contrib_ops/cuda/matmul_nbits.md`	Updates offline packer snippets and references to the new module.
`setup.py`	Adds the new `.so` to packaged binaries list for Linux/macOS wheels.

…ProviderInfo_CUDA interface - Remove CUDA source files (fpA_intB_gemm_adaptor.cu, fpA_intB_gemm_preprocessors_impl.cu) from onnxruntime_pybind11_state build, and remove CUDA::cudart linkage. This eliminates the NEEDED libcudart.so.13 entry that caused import failures on CPU-only machines. - Add PackWeightsForMixedGemm() virtual method to ProviderInfo_CUDA interface so the pybind module can call it via TryGetProviderInfo_CUDA() (dynamic load) instead of linking CUDA directly. - Implement PackWeightsForMixedGemm() in ProviderInfo_CUDA_Impl (cuda_provider_factory.cc) using existing GPU kernels. - Update onnxruntime_pybind_quant.cc to call through the provider interface; move PackFP4WeightsForMoE outside the USE_CUDA guard since it is pure CPU code. Fixes: import onnxruntime fails with libcudart.so.13 error on CPU-only Linux (issue #29500)

- Use static_cast<int8_t*>(d_transposed.get()) explicitly for consistency with the 8-bit path in cuda_provider_factory.cc - Fix integer division order: compute (8/bits) as size_t division (static_cast<size_t>(8) / static_cast<size_t>(bits)) in both cuda_provider_factory.cc and onnxruntime_pybind_quant.cc

…NO_CUDA_IN_PYBIND The new onnxruntime_pybind_cuda_quant.cc was picked up by the python/*.cc glob and compiled into onnxruntime_pybind11_state, breaking CPU-only builds (fatal error: cuda_runtime.h not found) and re-adding CUDA link deps on every platform. Remove it from the main pybind sources so it only builds inside the standalone onnxruntime_cuda_quant_preprocess module. Restore the ORT_NO_CUDA_IN_PYBIND compile definition on Windows CUDA builds: it is still consumed by onnxruntime_pybind_mlvalue.cc / onnxruntime_pybind_ortvalue.cc to avoid direct CUDA runtime calls (cudaMemcpy/cudaStreamSynchronize) in the pybind module. Dropping it caused LNK2001 unresolved cudart symbols on Windows. Add onnxruntime_cuda_quant_preprocess.so to dl_libs so manylinux wheels package it (the manylinux path builds 'data' from dl_libs only, not libs).

…scalar init Two CI regressions from removing the CUDA runtime link from the main pybind module: - Linux CUDA/TensorRT/Plugin test jobs failed at import with 'undefined symbol: cudaMemcpy'. The pybind module no longer links CUDA::cudart, but ORT_NO_CUDA_IN_PYBIND was only defined on Windows, so onnxruntime_pybind_mlvalue.cc still emitted direct cudaMemcpy calls on Linux. Define ORT_NO_CUDA_IN_PYBIND for all CUDA builds so host/device copies route through the CUDA provider bridge (ProviderInfo_CUDA) on every platform. - macOS arm64 CPU-only build jobs (cpu/coreml/xnnpack/webgpu) failed compiling PackFP4WeightsForMoE with clang -Werror,-Wbraced-scalar-init. This CPU-only function is now compiled unconditionally; replace the braced scalar py::array_t<uint8_t> output({size}) with the array_t(ssize_t count) constructor output(size), which yields the identical 1-D array.

skottmckay · 2026-07-08T07:15:41Z

+    must succeed.
+    """
+
+    @unittest.skipIf(sys.platform.startswith("win"), "cuda quant preprocess module is not built on Windows")


do we need to skip on macos or we don't run tests there? would != 'linux' be simpler in that case?

That is a long time ago. Cuda does not support macos since Apple has its own GPU.

BTW, I removed this check since I add a build option to enable/disable the new DLL.
I add -v to run python unittest, so it's able to see whether tests have skipped or not.

Copilot AI assigned Copilot and skottmckay Jul 7, 2026

Copilot started work on behalf of skottmckay July 7, 2026 05:04 View session

Copilot AI linked an issue Jul 7, 2026 that may be closed by this pull request

onnxruntime-gpu 1.27.0 Linux wheel fails to import without CUDA 13 due to libcudart.so.13 dependency in pybind library #29500

Open

Copilot AI changed the title ~~[WIP] Fix onnxruntime-gpu import error without CUDA 13~~ Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux Jul 7, 2026

Copilot AI requested a review from skottmckay July 7, 2026 05:23

Copilot finished work on behalf of skottmckay July 7, 2026 05:23

tianleiwu reviewed Jul 7, 2026

View reviewed changes

Comment thread onnxruntime/python/onnxruntime_pybind_quant.cc Outdated

tianleiwu marked this pull request as ready for review July 7, 2026 09:22

tianleiwu requested a review from Copilot July 7, 2026 16:23

Copilot started reviewing on behalf of tianleiwu July 7, 2026 16:23 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread setup.py

Comment thread cmake/onnxruntime_python.cmake

tianleiwu marked this pull request as draft July 7, 2026 19:14

Copilot AI and others added 5 commits July 7, 2026 23:30

Initial plan

74f8d83

move cuda quant preprocess to a new dll

5e9fa25

tianleiwu force-pushed the copilot/fix-onnxruntime-gpu-import-error branch from 793e0ea to c948c7d Compare July 7, 2026 23:32

tianleiwu marked this pull request as ready for review July 7, 2026 23:33

tianleiwu added 2 commits July 8, 2026 01:02

Clean up

8b32759

skottmckay approved these changes Jul 8, 2026

View reviewed changes

add a test

9897958

skottmckay reviewed Jul 8, 2026

View reviewed changes

Add build option and torch reference

081ac22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux#29590

Fix libcudart.so.13 hard dependency in pybind module breaking import on CPU-only Linux#29590
skottmckay with Copilot wants to merge 9 commits into
mainfrom
copilot/fix-onnxruntime-gpu-import-error

Copilot AI commented Jul 7, 2026 •

edited by tianleiwu

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

skottmckay Jul 8, 2026

Uh oh!

tianleiwu Jul 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Jul 7, 2026 • edited by tianleiwu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Motivation and Context

Testing Notes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

skottmckay Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

tianleiwu Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 7, 2026 •

edited by tianleiwu

Loading

tianleiwu Jul 8, 2026 •

edited

Loading