From a69978956bbb80016bf2e2e28fdaaeb211c30a86 Mon Sep 17 00:00:00 2001 From: tcherckez-nvidia <127761168+tcherckez-nvidia@users.noreply.github.com> Date: Thu, 9 Jul 2026 07:18:32 -0700 Subject: [PATCH] [https://nvbugs/6367792][fix] Disable piecewise CUDA graph for Nemotron Nano V3 Under enable_chunked_prefill=true, Nemotron Nano V3 (hybrid Mamba2+MoE) produces device-side NaNs during piecewise CUDA-graph replay for small, heavily-padded token buckets, tripping the FlashInfer sampler's NaN assert on the first generation step. Disable piecewise CUDA-graph capture to avoid the faulty replay path while the underlying issue is investigated. Un-waive the TestNemotronNanoV3 accuracy variants (bf16/fp8 verified passing; nvfp4 to be confirmed in Blackwell CI). Signed-off-by: tcherckez-nvidia <127761168+tcherckez-nvidia@users.noreply.github.com> --- examples/auto_deploy/model_registry/configs/nano_v3.yaml | 3 ++- tests/integration/test_lists/waives.txt | 5 ----- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/examples/auto_deploy/model_registry/configs/nano_v3.yaml b/examples/auto_deploy/model_registry/configs/nano_v3.yaml index 3a86c70acb87..488e319e778a 100644 --- a/examples/auto_deploy/model_registry/configs/nano_v3.yaml +++ b/examples/auto_deploy/model_registry/configs/nano_v3.yaml @@ -54,7 +54,8 @@ transforms: insert_cached_ssm_attention: backend: flashinfer_ssm compile_model: - piecewise_enabled: true + # Disabled due to https://nvbugs/6367792 (piecewise + chunked-prefill NaN). + piecewise_enabled: false fuse_nvfp4_moe: backend: trtllm_gen mlir_elementwise_fusion: diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 688f5710254e..884fc386dd6a 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -25,10 +25,6 @@ accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_vide accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[qwen3vl_2b_instruct] SKIP (https://nvbugs/6422294) accuracy/test_llm_api.py::TestLlama3_1_8BInstruct::test_guided_decoding_4gpus[xgrammar] SKIP (https://nvbugs/5346443) accuracy/test_llm_api_autodeploy.py::TestMiniMaxM2::test_finegrained_fp8 SKIP (https://nvbugs/6396422) -accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[bf16-4-attn_dp_off-trtllm] SKIP (https://nvbugs/6367792) -accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[fp8-4-attn_dp_off-trtllm] SKIP (https://nvbugs/6367792) -accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm] SKIP (temporary ToT main waive; B200 AutoDeploy NVFP4 GSM8K accuracy below threshold) -accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-4-attn_dp_off-trtllm] SKIP (https://nvbugs/6367792) accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[bf16_ws4_180gb-flashinfer] SKIP (https://nvbugs/6418090) accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[bf16_ws4_180gb-trtllm] SKIP (https://nvbugs/6422315) accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[fp8_ws4_80gb-trtllm] SKIP (https://nvbugs/6422094) @@ -255,7 +251,6 @@ full:GB200/disaggregated/test_ad_disagg_trtllm_serve.py::test_openai_completion full:GB200/test_e2e.py::test_qwen_e2e_cpprunner_large_new_tokens[DeepSeek-R1-Distill-Qwen-1.5B-DeepSeek-R1-Distill-Qwen-1.5B] SKIP (https://nvbugs/6414760) full:GB300/accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_gen_first[adp-mtp2] SKIP (https://nvbugs/6295740) full:GB300/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_auto_dtype[True-False-False-False] SKIP (https://nvbugs/6316984) -full:GB300/accuracy/test_llm_api_autodeploy.py::TestNemotronNanoV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm] SKIP (https://nvbugs/6329165) full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-enable_chunked_prefill=True-v2_kv_cache=True] SKIP (https://nvbugs/6422343) full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_fp8_block_scales_4gpus[pp4-mtp_nextn=0-fp8kv=True-attention_dp=False-cuda_graph=True-overlap_scheduler=True-torch_compile=True-sampler_async_worker=False] SKIP (https://nvbugs/6388153) full:GB300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-pp4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=False] SKIP (https://nvbugs/6388153)