沐曦 Metax GPU 服务器上编译失败

[megemini]

参考 https://github.com/PaddlePaddle/community/blob/master/pfcc/paddle-hardware/%E6%B2%90%E6%9B%A6-FastDeploy-%E7%BC%96%E8%AF%91%E9%AA%8C%E8%AF%81-%E6%89%93%E5%8D%A1%E4%BB%BB%E5%8A%A1.md

https://gitee.com/paddlepaddle/FastDeploy/blob/release/2.4/docs/zh/get_started/installation/metax_gpu.md

在沐曦的服务器上编译失败（使用 release/2.4 和 develop 都是同样的错误），提示很多错误，这里只摘取部分：

/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_clear_accept_nums.cu compile failed, command '/usr/local/cuda/bin/nvcc' failed: No such file or directory
In file included from /data/FastDeploy/custom_ops/gpu_ops/remote_cache_kv_ipc.cc:15:
/data/FastDeploy/custom_ops/gpu_ops/remote_cache_kv_ipc.h:28:10: fatal error: driver_types.h: No such file or directory
   28 | #include "driver_types.h"
      |          ^~~~~~~~~~~~~~~~ [2026-03-28 10:02:04,113] [    INFO] spawn.py:38 - gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/maca/include -I/opt/maca/include/mcr -I/opt/maca/include/common -I/opt/maca/include/mcfft -I/opt/maca/include/mcrand -I/opt/maca/include/mcsparse -I/opt/maca/include/mcblas -I/opt/maca/include/mcsolver -I/opt/conda/lib/python3.10/site-packages/paddle/include -I/opt/conda/lib/python3.10/site-packages/paddle/include/third_party -I/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat -I/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/torch/csrc/api/include -I/opt/conda/include/python3.10 -I/opt/conda/include/python3.10 -c /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_update_input_ids_cpu.cc -o /data/FastDeploy/custom_ops/build/fastdeploy_ops/lib.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_update_input_ids_cpu.o -O3 -w -DPADDLE_WITH_CUSTOM_KERNEL -DPADDLE_EXTENSION_NAME=fastdeploy_ops -D_GLIBCXX_USE_CXX11_ABI=1 -DPADDLE_WITH_CUDA -std=c++17
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_set_stop_value_multi_seqs.cu compile failed, command '/usr/local/cuda/bin/nvcc' failed: No such file or directory
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/api.h:5,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/all.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:29,
                 from /data/FastDeploy/custom_ops/gpu_ops/get_img_boundaries.cc:15:
/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/tensor.h:23:10: fatal error: cuda_runtime.h: No such file or directory
   23 | #include <cuda_runtime.h>
      |          ^~~~~~~~~~~~~~~~

[StareAtYou]

看样子是没有进行 maca 运行环境的初始化，麻烦在终端执行下述 shell 命令后（export 检查下是否环境变量都设置上了？并且相应路径是否都存在指定文件/文件夹），再编译试试（推荐使用 FastDeploy develop 分支）：

export MACA_PATH=/opt/maca
if [ ! -d ${HOME}/cu-bridge ]; then
            `${MACA_PATH}/tools/cu-bridge/tools/pre_make`
fi
export CUDA_PATH=${HOME}/cu-bridge/CUDA_DIR
export LD_LIBRARY_PATH=${CUDA_PATH}/lib64:${MACA_PATH}/lib:${MACA_PATH}/mxgpu_llvm/lib:$LD_LIBRARY_PATH
export PADDLE_XCCL_BACKEND=metax_gpu
export FLAGS_weight_only_linear_arch=80
export FD_MOE_BACKEND=cutlass
export FD_METAX_KVCACHE_MEM=8
export FD_ENC_DEC_BLOCK_NUM=0

如果仍然有编译报错，请提供以下信息：
1、服务器系统&内核版本
2、maca sdk/driver 版本（mx-smi 命令截图即可）
3、paddlepaddle & paddle_metax_gpu 包版本
4、FastDeploy 源码分支 & commit ID 信息
5、编译出现的第一个或最开始出现的那一部分报错信息
6、当前使用的 maca docker 镜像版本/链接

[megemini]

非常感谢！之前 cuda 的问题解决了，但是，编译还是有问题，以下是我使用的环境：

https://ai.gitee.com/compute/metax
曦云 C500 - 16 GB

# 内核版本

(base) root@e61bf7addbc8:/data/FastDeploy# uname -a
Linux e61bf7addbc8 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
(base) root@e61bf7addbc8:/data/FastDeploy# cat /proc/version
Linux version 5.15.0-58-generic (buildd@lcy02-amd64-033) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023



# mx-smi
(base) root@e61bf7addbc8:/data/FastDeploy# mx-smi
mx-smi  version: 2.2.9

=================== MetaX System Management Interface Log ===================
Timestamp                                         : Wed Apr  1 05:03:56 2026

Attached GPUs                                     : 1
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.9                       Kernel Mode Driver Version: 3.0.11           |
| MACA Version: 3.2.1.10             BIOS Version: 1.27.5.0                       |
|------------------+-----------------+---------------------+----------------------|
| Board       Name | GPU   Persist-M | Bus-id              | GPU-Util      sGPU-M |
| Pwr:Usage/Cap    | Temp       Perf | Memory-Usage        | GPU-State            |
|==================+=================+=====================+======================|
| 0     MetaX C500 | 0           Off | 0000:0f:00.0        | 0%           Enabled |
| 57W / 350W       | 36C          P0 | 1242/65536 MiB      | Available            |
+------------------+-----------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Sliced GPU                                                                      |
|------------------------------------+---------------------+----------------------|
| Minor    GPU    sGPU-Id    Compute | Vram Quota          | sGPU-Util            |
|====================================+=====================+======================|
| 001      0      0          25%     | 0/16000 MiB         | 0%                   |
+------------------------------------+---------------------+----------------------+

+---------------------------------------------------------------------------------+
| Process:                                                                        |
|  GPU                    PID         Process Name                 GPU Memory     |
|                                                                  Usage(MiB)     |
|=================================================================================|
|  no process found                                                               |
+---------------------------------------------------------------------------------+

End of Log



# paddle
(base) root@e61bf7addbc8:/data/FastDeploy# pip list | grep paddle
paddle-metax-gpu               3.3.0.dev20260327+maca3.3.0.15
paddlepaddle                   3.4.0.dev20260330



# fastdeploy
(base) root@e61bf7addbc8:/data/FastDeploy# git log --oneline
3651113ee (HEAD -> develop, origin/develop, origin/HEAD) [DataProcessor]Remove ENABLE_V1_DATA_PROCESSOR (#7052)




# 编译
(base) root@e61bf7addbc8:/data/FastDeploy# time bash build.sh 
find python version 3.10
[init] removing building directory...
Looking in indexes: http://mirrors.aliyun.com/pypi/simple
Requirement already satisfied: setuptools_scm in /opt/conda/lib/python3.10/site-packages (10.0.5)
Requirement already satisfied: vcs-versioning>=1.0.0.dev0 in /opt/conda/lib/python3.10/site-packages (from setuptools_scm) (1.1.1)
Requirement already satisfied: packaging>=20 in /opt/conda/lib/python3.10/site-packages (from setuptools_scm) (25.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from setuptools_scm) (69.5.1)
Requirement already satisfied: tomli>=1 in /opt/conda/lib/python3.10/site-packages (from setuptools_scm) (2.3.0)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from setuptools_scm) (4.15.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[init] init success

I0401 05:00:05.377974 109236 init.cc:254] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I0401 05:00:05.377996 109236 init.cc:162] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:07.632588 109236 custom_device_load.cc:51] Succeed in loading custom runtime in lib: /opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so
I0401 05:00:07.632627 109236 custom_device_load.cc:58] Skipped lib [/opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so]: no custom engine Plugin symbol in this lib.
I0401 05:00:07.650096 109236 custom_kernel.cc:68] Succeed in loading 973 custom kernel(s) from loaded lib(s), will be used like native ones.
I0401 05:00:07.650621 109236 init.cc:174] Finished in LoadCustomDevice with libs_path: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:07.650660 109236 init.cc:260] CustomDevice: metax_gpu, visible devices count: 1
I0401 05:00:13.810226 109322 init.cc:254] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I0401 05:00:13.810249 109322 init.cc:162] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:16.057623 109322 custom_device_load.cc:51] Succeed in loading custom runtime in lib: /opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so
I0401 05:00:16.057658 109322 custom_device_load.cc:58] Skipped lib [/opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so]: no custom engine Plugin symbol in this lib.
I0401 05:00:16.075213 109322 custom_kernel.cc:68] Succeed in loading 973 custom kernel(s) from loaded lib(s), will be used like native ones.
I0401 05:00:16.075651 109322 init.cc:174] Finished in LoadCustomDevice with libs_path: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:16.075702 109322 init.cc:260] CustomDevice: metax_gpu, visible devices count: 1
[MODE] Building from source (ops)...
[build] build and install fastdeploy_ops...
I0401 05:00:22.378149 109414 init.cc:254] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I0401 05:00:22.378171 109414 init.cc:162] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:24.624779 109414 custom_device_load.cc:51] Succeed in loading custom runtime in lib: /opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so
I0401 05:00:24.624814 109414 custom_device_load.cc:58] Skipped lib [/opt/conda/lib/python3.10/site-packages/paddle_custom_device/libpaddle-metax-gpu.so]: no custom engine Plugin symbol in this lib.
I0401 05:00:24.642194 109414 custom_kernel.cc:68] Succeed in loading 973 custom kernel(s) from loaded lib(s), will be used like native ones.
I0401 05:00:24.642621 109414 init.cc:174] Finished in LoadCustomDevice with libs_path: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
I0401 05:00:24.642661 109414 init.cc:260] CustomDevice: metax_gpu, visible devices count: 1
I0401 05:00:30.983469 109500 init.cc:254] ENV [CUSTOM_DEVICE_ROOT]=/opt/conda/lib/python3.10/site-packages/paddle_custom_device
I0401 05:00:30.983490 109500 init.cc:162] Try loading custom device libs from: [/opt/conda/lib/python3.10/site-packages/paddle_custom_device]
[05:00:31.137][MCC][E]Installed MACA LLVM Version (12.0.0) is lower than ELF LLVM Version(19.1.3), please use LLVM with the same major version (either upgrade MACA or recompile the ELF)
[05:00:31.137][MCC][E]If you want to skip this check and continue running, try: export MACA_DEVICE_IMAGE_CHECK=1.
[05:00:31.138][MCC][E]Installed MACA LLVM Version (12.0.0) is lower than ELF LLVM Version(19.1.3), please use LLVM with the same major version (either upgrade MACA or recompile the ELF)


...


[2026-04-01 05:00:33,881] [    INFO] spawn.py:38 - /root/cu-bridge/CUDA_DIR/bin/nvcc -I/opt/maca/include -I/opt/maca/include/mcr -I/opt/maca/include/common -I/opt/conda/lib/python3.10/site-packages/paddle_custom_device/include -I/opt/conda/lib/python3.10/site-packages/paddle/include -I/opt/conda/lib/python3.10/site-packages/paddle/include/third_party -I/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat -I/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/torch/csrc/api/include -I/root/cu-bridge/CUDA_DIR/include/ -I/root/cu-bridge/CUDA_DIR/include/nvtx3 -I/opt/conda/include/python3.10 -I/opt/conda/include/python3.10 -c /data/FastDeploy/custom_ops/metax_ops/moe_reduce.cu -o /data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/metax_ops/moe_reduce.cu.o -DPADDLE_WITH_CUDA -DEIGEN_USE_GPU -ccbin cc -Xcompiler -fPIC --expt-relaxed-constexpr -DNVCC -O3 -Ithird_party/nlohmann_json/include -Igpu_ops -DPADDLE_DEV -DPADDLE_WITH_CUSTOM_DEVICE_METAX_GPU -w -DPADDLE_WITH_CUSTOM_KERNEL -DPADDLE_EXTENSION_NAME=fastdeploy_ops -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++17
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/get_output_msg_with_topk.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/get_output_msg_with_topk.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/get_output_msg_with_topk.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/get_output.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/get_output.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/get_output.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/save_with_output_msg.h:22,
                 from /data/FastDeploy/custom_ops/gpu_ops/save_with_output_msg.cc:15:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/draft_model/../speculate_msg.h:22,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/draft_model/mtp_save_first_token.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_get_output_with_topk.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/save_with_output.cc:25:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/get_output_ep.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/draft_model/mtp_save_first_token.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/draft_model/mtp_save_first_token.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/save_with_output_msg.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/save_with_output_msg.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_get_output_with_topk.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_get_output_with_topk.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/get_output_ep.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/get_output_ep.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/save_with_output.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/save_with_output.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_save_output.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_save_output_with_topk.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/save_output_msg_with_topk.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/ngram_match.cc:21:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_save_output.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_save_output.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_save_output_with_topk.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_save_output_with_topk.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/save_output_msg_with_topk.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/save_output_msg_with_topk.o is compiled
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/ngram_match.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/ngram_match.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/transfer_output.cc:26:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/transfer_output.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/transfer_output.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_update_input_ids_cpu.cc:15:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_update_input_ids_cpu.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_update_input_ids_cpu.o is compiled
In file included from /root/cu-bridge/CUDA_DIR/include/cublasXt.h:3,
                 from /opt/conda/lib/python3.10/site-packages/paddle_custom_device/include/paddle/phi/backends/dynload/cublas.h:17,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_types.h:27,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/backends/gpu/gpu_info.h:24,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/core/platform/device/gpu/gpu_info.h:25,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/c10/core/Device.h:31,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18,
                 from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46,
                 from /data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_get_output.cc:20:
/root/cu-bridge/CUDA_DIR/include/bridge/blas/cublasXt_wrapper.h:5:10: fatal error: mcblasXt.h: No such file or directory
    5 | #include "mcblasXt.h"
      |          ^~~~~~~~~~~~
compilation terminated.
/data/FastDeploy/custom_ops/gpu_ops/speculate_decoding/speculate_get_output.cc compile failed, command '/usr/bin/gcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/speculate_decoding/speculate_get_output.o is compiled
In file included from /data/FastDeploy/custom_ops/gpu_ops/moe/moe_topk_select.cu:21:
In file included from gpu_ops/moe/fused_moe_helper.h:18:
gpu_ops/cutlass_extensions/wint_type_traits.h:22:10: fatal error: 'cutlass/layout/layout.h' file not found
#include "cutlass/layout/layout.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /data/FastDeploy/custom_ops/metax_ops/moe_reduce.cu:17:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
In file included from gpu_ops/cutlass_kernels/moe_gemm/fused_moe_gemm_kernels.h:23:
gpu_ops/cutlass_extensions/wint_type_traits.h:22:10: fatal error: 'cutlass/layout/layout.h' file not found
#include "cutlass/layout/layout.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /data/FastDeploy/custom_ops/metax_ops/moe_dispatch.cu:20:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
In file included from gpu_ops/cutlass_kernels/moe_gemm/fused_moe_gemm_kernels.h:23:
gpu_ops/cutlass_extensions/wint_type_traits.h:22:10: fatal error: 'cutlass/layout/layout.h' file not found
#include "cutlass/layout/layout.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe.cu:17:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_op.h:22:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
In file included from gpu_ops/cutlass_kernels/moe_gemm/fused_moe_gemm_kernels.h:23:
gpu_ops/cutlass_extensions/wint_type_traits.h:22:10: fatal error: 'cutlass/layout/layout.h' file not found
#include "cutlass/layout/layout.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /data/FastDeploy/custom_ops/gpu_ops/fused_get_rotary_embedding.cu:24:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18:
/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:469:37: error: cannot pass object of non-trivial type
      'std::shared_ptr<phi::Allocation>' through variadic function; call will abort at runtime [-Wnon-pod-varargs]
      MaybeResetHolder(dense.get(), compat_holder, 0);
                                    ^
1 error generated when compiling for xcore1000.
/data/FastDeploy/custom_ops/gpu_ops/moe/moe_topk_select.cu compile failed, command '/root/cu-bridge/CUDA_DIR/bin/nvcc' failed with exit code 1
/data/FastDeploy/custom_ops/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/moe/moe_topk_select.cu.o is compiled
In file included from /data/FastDeploy/custom_ops/gpu_ops/read_data_ipc.cu:16:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18:
/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:469:37: error: cannot pass object of non-trivial type
      'std::shared_ptr<phi::Allocation>' through variadic function; call will abort at runtime [-Wnon-pod-varargs]
      MaybeResetHolder(dense.get(), compat_holder, 0);
                                    ^
In file included from /data/FastDeploy/custom_ops/gpu_ops/set_flags.cu:15:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18:
/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:469:37: error: cannot pass object of non-trivial type
      'std::shared_ptr<phi::Allocation>' through variadic function; call will abort at runtime [-Wnon-pod-varargs]
      MaybeResetHolder(dense.get(), compat_holder, 0);
                                    ^
In file included from /data/FastDeploy/custom_ops/gpu_ops/set_value_by_flags_and_idx.cu:15:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/extension.h:46:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/utils/pybind.h:18:
In file included from /opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBody.h:18:
/opt/conda/lib/python3.10/site-packages/paddle/include/paddle/phi/api/include/compat/ATen/core/TensorBase.h:469:37: error: cannot pass object of non-trivial type
      'std::shared_ptr<phi::Allocation>' through variadic function; call will abort at runtime [-Wnon-pod-varargs]
      MaybeResetHolder(dense.get(), compat_holder, 0);
                                    ^
                                    ^
                                    

[StareAtYou]

你好，基于上述现象，需要以下调整：
1、镜像版可能不适配（编译报错提示缺失的库文件可能未合入低版本maca镜像，先去 “/opt/maca/include” 路径及其子路径下确认，是否存在编译提示缺失的那些文件！！！），推荐使用 MACA开发者平台官方 paddle-metax 镜像（https://developer.metax-tech.com/softnova/docker?chip_name=%E6%9B%A6%E4%BA%91C500%E7%B3%BB%E5%88%97&package_kind=AI&dimension=docker&deliver_type=%E5%88%86%E5%B1%82%E5%8C%85&ai_frame=paddle-metax&system=ubuntu&arch=amd64&python_version=3.10）
2、截至 2026.04.01，paddle_metax_gpu 未完全适配最新版本的 paddlepaddle，若需要使用 FastDeploy 框架（develop分支），请先将 paddlepaddle 版本锁定在 20260319（https://paddle-whl.bj.bcebos.com/nightly/cpu/paddlepaddle/paddlepaddle-3.4.0.dev20260319-cp310-cp310-linux_x86_64.whl）

[Kane2011]

@megemini 关于环境配置和代码分支做了调整更新， 请参考新的指导文档进行， Thanks！

[JiwaniZakir]

The errors point to the build system still invoking /usr/local/cuda/bin/nvcc and including driver_types.h (a CUDA-specific header), even though the Metax environment provides MACA at /opt/maca/ rather than a standard CUDA installation. The custom_ops/gpu_ops/remote_cache_kv_ipc.h:28 unconditionally includes driver_types.h, which only exists in a CUDA toolkit — on MACA-only systems there is no /usr/local/cuda and no NVCC to compile .cu files. The likely root cause is that the CMake/build scripts select the compiler and GPU headers based on a CUDA-present assumption rather than detecting the Metax/MACA toolchain and substituting its equivalents (e.g., the MACA counterpart to driver_types.h). A fix would need to guard CUDA-specific includes and the NVCC invocation behind a WITH_CUDA or similar flag, and add a corresponding WITH_METAX_MACA path that routes .cu compilation and GPU type headers through /opt/maca/.

[megemini]

@Kane2011 感谢回复～

我这里编译通过了，不过，需要注释掉 custom_ops/setup_ops.py 中 metax 的 moe 相关的算子：

        "gpu_ops/swap_cache_batch.cu",
        "gpu_ops/gelu_tanh.cu",
        #"metax_ops/moe_dispatch.cu",
        #"metax_ops/moe_ffn.cu",
        #"metax_ops/moe_reduce.cu",
        #"metax_ops/fused_moe.cu",
        "metax_ops/cache_kv_with_rope.cu",
        "metax_ops/cpp_extensions.cc",
        "metax_ops/split_merge_qkv.cu",

这样可以编译和安装

(base) root@e3ebf1fec84d:/data/FastDeploy# pip show fastdeploy-metax-gpu
Name: fastdeploy-metax-gpu
Version: 2.5.0.dev0
Summary: FastDeploy: Large Language Model Serving.
Home-page: https://github.com/PaddlePaddle/FastDeploy
Author: PaddlePaddle
Author-email: dltp@baidu.com
License: Apache 2.0
Location: /opt/conda/lib/python3.10/site-packages

如果不注释掉这几个算子，仍然会编译报错：

In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:29:26: error: no member named 'MCTLASS_EX_DATATYPE_BF16' in 'mctlassExDataType'; did you mean 'MCTLASS_EX_DATA_TYPE_END'?
      mctlassExDataType::MCTLASS_EX_DATATYPE_BF16;
      ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
                         MCTLASS_EX_DATA_TYPE_END
/opt/maca/include/mctlassEx/mctlassEx.h:68:5: note: 'MCTLASS_EX_DATA_TYPE_END' declared here
    MCTLASS_EX_DATA_TYPE_END
    ^
In file included from /data/FastDeploy/custom_ops/metax_ops/moe_ffn.cu:16:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:35:26: error: no member named 'MCTLASS_EX_DATATYPE_INT8' in 'mctlassExDataType'; did you mean 'MCTLASS_EX_DATA_TYPE_END'?
      mctlassExDataType::MCTLASS_EX_DATATYPE_INT8;
      ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
                         MCTLASS_EX_DATA_TYPE_END
/opt/maca/include/mctlassEx/mctlassEx.h:68:5: note: 'MCTLASS_EX_DATA_TYPE_END' declared here
    MCTLASS_EX_DATA_TYPE_END
    ^
In file included from /data/FastDeploy/custom_ops/metax_ops/moe_ffn.cu:16:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:153:5: error: unknown type name 'mctlassExContiguousGroupedDesc_t'; did you mean 'mctlassExContiguousGroupedGemmAlgo_t'?
    mctlassExContiguousGroupedDesc_t contiguous_group_desc;
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    mctlassExContiguousGroupedGemmAlgo_t
/opt/maca/include/mctlassEx/mctlassEx.h:51:6: note: 'mctlassExContiguousGroupedGemmAlgo_t' declared here
enum mctlassExContiguousGroupedGemmAlgo_t {
     ^
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe.cu:17:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:29:26: error: no member named 'MCTLASS_EX_DATATYPE_BF16' in 'mctlassExDataType'; did you mean 'MCTLASS_EX_DATA_TYPE_END'?
      mctlassExDataType::MCTLASS_EX_DATATYPE_BF16;
      ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
                         MCTLASS_EX_DATA_TYPE_END
/opt/maca/include/mctlassEx/mctlassEx.h:68:5: note: 'MCTLASS_EX_DATA_TYPE_END' declared here
    MCTLASS_EX_DATA_TYPE_END
    ^
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe.cu:17:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:35:26: error: no member named 'MCTLASS_EX_DATATYPE_INT8' in 'mctlassExDataType'; did you mean 'MCTLASS_EX_DATA_TYPE_END'?
      mctlassExDataType::MCTLASS_EX_DATATYPE_INT8;
      ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
                         MCTLASS_EX_DATA_TYPE_END
/opt/maca/include/mctlassEx/mctlassEx.h:68:5: note: 'MCTLASS_EX_DATA_TYPE_END' declared here
    MCTLASS_EX_DATA_TYPE_END
    ^
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe.cu:17:
In file included from /data/FastDeploy/custom_ops/metax_ops/fused_moe_helper.h:17:
/data/FastDeploy/custom_ops/metax_ops/fused_moe_gemm_kernels.h:153:5: error: unknown type name 'mctlassExContiguousGroupedDesc_t'; did you mean 'mctlassExContiguousGroupedGemmAlgo_t'?
    mctlassExContiguousGroupedDesc_t contiguous_group_desc;
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    mctlassExContiguousGroupedGemmAlgo_t
/opt/maca/include/mctlassEx/mctlassEx.h:51:6: note: 'mctlassExContiguousGroupedGemmAlgo_t' declared here
enum mctlassExContiguousGroupedGemmAlgo_t {
     ^
/data/FastDeploy/custom_ops/build/fastdeploy_ops/lib.linux-x86_64-cpython-310/build/fastdeploy_ops/temp.linux-x86_64-cpython-310/gpu_ops/transfer_output.o is compiled
3 errors generated when compiling for xcore1000.

我参考 https://github.com/PaddlePaddle/community/blob/master/pfcc/paddle-hardware/%E6%B2%90%E6%9B%A6-FastDeploy-%E7%BC%96%E8%AF%91%E9%AA%8C%E8%AF%81-%E6%89%93%E5%8D%A1%E4%BB%BB%E5%8A%A1.md 中更新的方式，用 fastdeploy 的 develop 版本进行的编译。 release/2.4 的问题更多，这里就不重复了 ～