`1.3.0rc15` on sm_121a / GB10: MTP path fails — CUTLASS sm_120 grouped-GEMM init → SMEM shortfall → Triton fallback emits `.rs` PTX rejected by PTXAS

[zentradev-rabih]

Summary

On nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15 running Nemotron-3-Super-120B-A12B-NVFP4 on NVIDIA GB10 (sm_121a), enabling MTP (speculative_config.decoding_type: MTP, num_nextn_predict_layers: 3) fails at engine startup with three stacked errors:

1. CUTLASS sm_120 grouped-GEMM kernel fails to initialize for the MTP-shaped MoE GEMM
2. CUTLASS GroupedGEMM kernel reports insufficient shared memory at the MTP-shaped tensor sizes
3. Triton fallback emits PTX containing the .rs (stochastic-rounding) feature, which PTXAS rejects for .target 'sm_121a'

Without MTP, the same image + same hardware + same yaml otherwise runs cleanly through a 20-min, 10-concurrent, 15%-cancel soak (validated in issue #14500 after applying a separate two-line patch suggested there).

So this issue isolates the MTP-specific path on sm_121a.

Environment

• GPU: NVIDIA GB10 (Grace-Blackwell, integrated GPU, unified memory, sm_121a)
• OS: DGX OS 7 (Ubuntu 24.04 aarch64)
• Model: Nemotron-3-Super-120B-A12B-NVFP4 (MoE; hybrid Mamba/SSM + attention; NVFP4)
• Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15
• Backend: pytorch (AutoDeploy via trtllm-serve)
• (also applied the fused_moe/quantization.py two-line patch from 1.3.0rc15: trtllm-serve hangs in fused_moe/quantization.py during cold-start on GB10 / sm_121 — 1.3.0rc12 works with identical config #14500 — needed to get past cold-start on this hardware; not relevant to this MTP failure path)

Reproducer

Launch the standard trtllm-serve command (same as #14500) and add this to the mounted extra_llm_api_options yaml:

speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3

Everything else identical to the baseline yaml (CUTLASS MoE backend, enable_block_reuse: false, mamba_ssm_cache_dtype: float16, mamba_ssm_stochastic_rounding: true, enable_chunked_prefill: true, max_batch_size: 8, etc.). MTP-off vs MTP-on is the only knob that flips between "soak passes clean" and the failure chain below.

Failure chain

After MTPSampler initializes successfully, the MoE autotuner begins profiling tactics. Three distinct errors stack:

1) CUTLASS sm_120 grouped-GEMM init

[Autotuner] Failed when profiling runner=<…MoERunner…>, tactic=6,
shapes=[torch.Size([1, 512]), torch.Size([512, 2688, 64]), torch.Size([0]),
        torch.Size([512, 1024, 168]), torch.Size([0])].
Error: [TensorRT-LLM][ERROR] Assertion failed: Failed to initialize cutlass TMA WS grouped gemm.
       Error: Error Internal
       (tensorrt_llm/kernels/cutlass_kernels/cutlass_instantiations/gemm_grouped/120/
        cutlass_kernel_file_gemm_grouped_sm120_M128_BS_group2.generated.cu:39)

[Autotuner] New tuning error occurs:Total failed profiling tactics occurs: 15
for custom_op=trtllm::fused_moe::gemm2.

2) CUTLASS shared-memory shortfall on the MTP-shaped GEMM

[Autotuner] Failed when profiling runner=<…MoERunner…>, tactic=8,
shapes=[torch.Size([1, 1024]), torch.Size([512, 2688, 1024]), torch.Size([0]),
        torch.Size([512, 1024, 2688]), torch.Size([0])].
Error: [TensorRT-LLM][ERROR] Assertion failed: GPU lacks the shared memory resources to run GroupedGEMM kernel
       (../tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_gemm_template_dispatch.h:174)

[Autotuner] New tuning error occurs:Total failed profiling tactics occurs: 15
for custom_op=trtllm::fused_moe::gemm1.

Note: the same model + same hardware runs CUTLASS MoE GEMM fine without MTP. The MTP-shape change is what blows the SMEM budget.

3) Triton fallback — PTXAS rejects .rs feature on sm_121a

Internal Triton PTX codegen error
ptxas /tmp/tmpacy3oa91.ptx, line 1200; error : Feature '.rs' not supported on .target 'sm_121a'
ptxas /tmp/tmpacy3oa91.ptx, line 1205; error : Feature '.rs' not supported on .target 'sm_121a'
ptxas /tmp/tmpacy3oa91.ptx, line 1210; error : Feature '.rs' not supported on .target 'sm_121a'
ptxas /tmp/tmpacy3oa91.ptx, line 1215; error : Feature '.rs' not supported on .target 'sm_121a'
ptxas /tmp/tmpacy3oa91.ptx, line 1220; error : Feature '.rs' not supported on .target 'sm_121a'
…(continues — many lines per failing kernel)

After ~9 minutes of compile failures the container exits and crashloops. Reverting MTP off (no other changes) brings the engine back up cleanly.

Test matrix (rc15 on GB10/sm_121a, Nemotron-3-Super-120B-A12B-NVFP4)

Config	Result
moe_config.backend: CUTLASS, MTP off, with the #14500 patch applied	✅ 20-min / 10-concurrent / 15%-cancel soak clean; 0 engine deaths, 0 cancel hangs
moe_config.backend: CUTLASS, MTP on, with the #14500 patch applied	❌ this issue — three stacked errors above
moe_config.backend: TRTLLM, MTP either	❌ NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above (separate gap)

Why this matters

MTP is the largest documented throughput knob for this model. The combination of "MTP off is the only viable config on GB10" + "the documented MTP+mamba slot-accounting fix [PR #13151] needs MTP on to validate" leaves sm_121a/GB10 deployments unable to take the throughput recovery from #13151 even when running the latest release.

Likely fix surfaces

1. The Triton MTP MoE kernel emitting .rs — if .rs (stochastic rounding) is not actually required for this particular kernel on sm_120+, gate the emission so it's omitted for sm_121a. If it IS required, sm_121a (the integrated GB10 SASS variant) may need a different SASS path or an explicit non-.rs fallback codegen
2. The CUTLASS gemm_grouped_sm120 MTP-shape SMEM blowup — if MTP-shaped MoE GEMM legitimately needs more SMEM than baseline, a kernel variant that fits in the integrated-GPU SMEM budget would prevent the Triton fallback from being needed at all
3. Documentation — clarifying that decoding_type: MTP is currently unsupported on sm_121a (until either of the above lands) so users on GB10 don't keep rediscovering this

Cross-reference

• Issue 1.3.0rc15: trtllm-serve hangs in fused_moe/quantization.py during cold-start on GB10 / sm_121 — 1.3.0rc12 works with identical config #14500 — separate cold-start hang on rc15+GB10 (now resolved by a two-line patch from contributor ssam18, who also flagged that the .rs/PTXAS path was likely separate)
• The yaml-comment chain that originally documented the rc14 PTXAS issue (which now matches what we see in rc15) — this is the same wall, just hit on a different trigger path

Happy to capture full failing PTX dumps, TLLM_LOG_LEVEL=DEBUG logs, py-spy at any of the three failure points, or test a candidate patch on this hardware.

[zentradev-rabih]

Resolved on 1.3.0rc17. The MTP path now initializes and serves on sm_121a / GB10.

Root cause / fix: rc17 includes #13773 (FlashInferNvfp4Sm12xFusedMoE). With moe_config.backend: CUTLASS, NVFP4, SM120/121, and ep_size == 1, the MoE backend now auto-promotes to the FlashInfer b12x path for decode (m < 64) while keeping CUTLASS GroupGEMM for prefill. That routes the decode-time MoE GEMM away from the CUTLASS TMA-WS grouped-GEMM init that was failing → Triton fallback emitting .rs (stochastic-rounding) PTX that ptxas rejected on sm_121a. With that path bypassed, the failure chain in this issue no longer occurs.

Validation (GB10, sm_121, Nemotron-Super-120B-NVFP4, rc17):

• speculative_config: {decoding_type: MTP, num_nextn_predict_layers: 3} — engine reaches Application startup complete, no ptxas/.rs error, generation correct.
• 3-min concurrency soak (12 streams, above the 8-slot batch ceiling, ~15% mid-stream cancellation): 0 engine stalls, 0 leaked slots on cancel, 0 errors.
• Single-user decode throughput (same image, 420-token generation): ~15 tok/s with MTP off → ~27 tok/s with MTP on (~1.8× from speculation).

The CUTLASS autotuner still logs non-fatal Failed to initialize cutlass TMA WS grouped gemm / CheckTactic failed status:15 warnings while profiling; those tactics are discarded and the engine proceeds.

One dependency: rc17 still cold-start-hangs without the fix in #14586 (the .storage().data_ptr() → .data_ptr() change in fused_moe/quantization.py, which on rc17 lands at lines 2715 and 2806). With #14586 applied, cold-start + MTP are both clean.

Suggest closing this as resolved by #13773 once #14586 lands. Thanks to everyone who looked at it.

[tbraun96]

Disclosure: I work on Atlas and we run Nemotron-3-Super-120B on GB10, so this MTP-on-sm_121a failure mode is familiar territory. Your three stacked errors are actually two separate sm_12x walls wearing a trenchcoat.

The CUTLASS sm_120 grouped-GEMM "insufficient shared memory" at MTP-shaped tensor sizes is the one to focus on. On the SM_120 grouped-GEMM builder the pipeline depth is computed from the shared-memory budget and the tile shapes are a small fixed set, with ClusterShape forced to 1x1x1 on GB10. The MTP path reshapes the MoE GEMM (the draft-head shapes are different from the main decode shapes), and at num_nextn_predict_layers: 3 those shapes push the SMEM request past what the sm_120 kernel was built to fit, so init fails. It then falls back to Triton, and Triton on sm_121a emits the .rs (stochastic-rounding) variant which PTXAS rejects for that target, so you see all three at once but the SMEM shortfall is the trigger.

The lever that helped us was shrinking the MTP draft depth. On hybrid models we found K=2 already past the acceptance sweet spot, and K=1 keeps a much smaller MTP-shaped GEMM that tends to fit the sm_120 SMEM budget:

speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 1   # was 3; smaller MTP-shaped grouped-GEMM, fits sm_120 SMEM

@zentradev-rabih does dropping num_nextn_predict_layers from 3 to 1 get you past the CUTLASS SMEM init failure (even if acceptance is lower), or does the grouped-GEMM still fail to initialize at K=1? That tells us whether this is purely a shape/SMEM budget issue or something more structural in the sm_120 grouped-GEMM for MTP shapes.

[mihai-chiorean]

PR #12704 addresses items 1 and 2 of this report — the CUTLASS sm_120 grouped-GEMM init failure and the SMEM shortfall on MTP-shaped tensor sizes — by filtering the SM120 heuristic so it returns only tile configs that fit in 99 KiB SMEM. The fix uses cudaDevAttrMaxSharedMemoryPerBlockOptin at runtime so it applies uniformly to SM120 (RTX PRO 6000) and SM121 (GB10 / DGX Spark).