[https://nvbugs/6434512][fix] Select Marlin for Qwen3.5 MoE on Hopper

[lfr-0531]

Description

Qwen3.5 MoE checkpoints with NVFP4 experts currently inherit the Qwen3-Next defaults without selecting a Hopper-compatible MoE GEMM backend. This affects both globally NVFP4 and mixed-precision checkpoints. On H20, the default CUTLASS W4A4 path requires Blackwell and fails during warmup because no supported MoE tactic remains.

This change preserves the existing Qwen3-Next defaults and selects Marlin for Qwen3.5 text and VL MoE models when the checkpoint quantization algorithm is NVFP4 or MIXED_PRECISION and the GPU is in the Marlin-supported SM range. Other quantization configurations and Blackwell keep their existing backend selection.

The change restores the default H20 execution path without introducing an API, dependency, documentation, ownership, or architecture change. Same-node 8-GPU H20 measurements confirmed that the fixed automatic selection matches an explicit-Marlin reference within normal run-to-run variation.

Related NVBug: https://nvbugs/6434512

Test Coverage

• Diff-scoped pre-commit checks: passed.
• pytest tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl_moe.py -k qwen35_moe_model_defaults: 3 passed, 6 deselected.
• Full source build and editable installation in the H20 Docker environment: passed.
• Runtime verification confirmed that the benchmark loaded commit 050a0d3036 and the modified modeling_qwen3_5.py.
• The seven Qwen3.5 H20 cases from tests/integration/test_lists/qa/llm_perf_core.yml: all passed on 8 H20 GPUs.
• Global-NVFP4 same-node H20 A/B results versus explicit Marlin: +0.787% output throughput for 128/128, +0.017% for max-batch-size-1 8000/1000, and -0.151% for max-batch-size-512 8000/1000.
• Mixed-precision Qwen3.5-397B-A17B-NVFP4-V2 verification on 8 H20 GPUs:
  • Checkpoint metadata: MIXED_PRECISION with 375 FP8 and 60 NVFP4 layer entries.
  • Unfixed automatic path: failed during warmup with No supported MoE GEMM tactic remains after replacing unsupported NO_SMEM epilogues.
  • Explicit Marlin reference: 1442.346951 output tokens/s.
  • Fixed automatic path: 1458.025686 output tokens/s and selected MoeConfig(backend='MARLIN') with allowed_backends=['marlin'].
  • Fixed automatic versus explicit Marlin: +1.087%.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

Walkthrough

Qwen3.5 MoE models now share hardware-aware backend defaults. Supported Hopper configurations select MARLIN for MoE and NVFP4 GEMM. Tests cover Hopper, mixed-precision, and Blackwell configurations.

Changes

Qwen3.5 NVFP4 backend defaults

Layer / File(s)	Summary
Backend default resolution tensorrt_llm/_torch/models/modeling_qwen3_5.py	Adds shared defaults for Qwen3.5 MoE models. The causal-LM and VLM classes use the helper for hardware-aware backend selection.
Backend default validation tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl_moe.py	Tests Hopper NVFP4 selection, mixed-precision behavior, Blackwell NVFP4 behavior, and KV-cache defaults for both model classes.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: yechank-nvidia, nv-guomingz, bowenfu

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title follows the required ticket and type format and clearly summarizes the Marlin backend fix for Qwen3.5 MoE on Hopper.
Description check	✅ Passed	The description explains the issue, solution, test coverage, performance results, and checklist status with sufficient detail.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/models/modeling_qwen3_5.py`:
- Line 58: Annotate every listed function: add parameter and return annotations
to _get_qwen35_moe_model_defaults, Qwen3_5MoeForCausalLM.get_model_defaults, and
Qwen3_5MoeVLModel.get_model_defaults in
tensorrt_llm/_torch/models/modeling_qwen3_5.py; annotate monkeypatch and add ->
None to the three affected test functions in
tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl_moe.py at lines 161,
184, and 201.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 88037976-5a87-4cc1-85d9-fe18ea04a3b1

📥 Commits

Reviewing files that changed from the base of the PR and between 7608520 and ec8827d.

📒 Files selected for processing (2)

• tensorrt_llm/_torch/models/modeling_qwen3_5.py
• tests/unittest/_torch/modeling/test_modeling_qwen3_5_vl_moe.py

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #63983 [ run ] triggered by Bot. Commit: 7ea8d3f Link to invocation

[tensorrt-cicd]

PR_Github #63983 [ run ] completed with state SUCCESS. Commit: 7ea8d3f
/LLM/main/L0_MergeRequest_PR pipeline #51917 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64036 [ run ] triggered by Bot. Commit: 7ea8d3f Link to invocation

[tensorrt-cicd]

PR_Github #64036 [ run ] completed with state SUCCESS. Commit: 7ea8d3f
/LLM/main/L0_MergeRequest_PR pipeline #51966 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64064 [ run ] triggered by Bot. Commit: 7ea8d3f Link to invocation

[tensorrt-cicd]

PR_Github #64064 [ run ] completed with state SUCCESS. Commit: 7ea8d3f
/LLM/main/L0_MergeRequest_PR pipeline #51993 completed with status: 'SUCCESS'

CI Report

Link to invocation

[brnguyen2]

Approving — the comments below are optional touch-ups, not blockers.

Change looks right for the reported H20 failure and the marlin-only GEMM list is fine in practice — the other NVFP4 backends are all SM100+, so nothing is lost on Ada/Hopper.

Two things to confirm before merge (details inline): whether the trigger covers the ModelOpt MIXED_PRECISION NVFP4 exports this file already normalizes, and whether the dense Qwen3.5 classes need the same nvfp4_gemm_config treatment.

Description nit: the auto-generated summary claims three new test functions (test_qwen35_moe_model_defaults_select_marlin_on_hopper, ..._keep_non_nvfp4_backends, ..._keep_blackwell_backends) — the diff actually adds one parametrized test with three params. Worth fixing so a future reader grepping for those names isn't confused.

One more question: MarlinFusedMoE doesn't support the MoE load balancer. Since this now flips the backend by default, is there a configuration (EPLB / wide-EP) where the new default turns a working run into a startup error? If so, a note in the helper's comment would help the next person.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64574 [ run ] triggered by Bot. Commit: 050a0d3 Link to invocation

[tensorrt-cicd]

PR_Github #64574 [ run ] completed with state SUCCESS. Commit: 050a0d3
/LLM/main/L0_MergeRequest_PR pipeline #52438 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64824 [ run ] triggered by Bot. Commit: dad9d67 Link to invocation

[tensorrt-cicd]

PR_Github #64824 [ run ] completed with state SUCCESS. Commit: dad9d67
/LLM/main/L0_MergeRequest_PR pipeline #52667 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[lfr-0531]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #64907 [ run ] triggered by Bot. Commit: dad9d67 Link to invocation

[tensorrt-cicd]

PR_Github #64907 [ run ] completed with state SUCCESS. Commit: dad9d67
/LLM/main/L0_MergeRequest_PR pipeline #52749 completed with status: 'SUCCESS'

CI Report

Link to invocation