Skip to content

[None][fix] Fix Qwen3 w4a8 model execution failure and add unit test #14527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leo0519 wants to merge 4 commits into from
Closed

[None][fix] Fix Qwen3 w4a8 model execution failure and add unit test #14527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8de485a
Fix the runtime error of w4a8 model with ConfiguragleMoE
leo0519 May 18, 2026
54950d7
Add unit test for w4a8 model
leo0519 May 25, 2026
79af0a2
Add QA test for Qwen3-30B-A3B w4a8 model
leo0519 Jun 28, 2026
0a871d3
Merge branch 'main' into qwen3-w4a8
leo0519 Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions tensorrt_llm/_torch/modules/fused_moe/configurable_moe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,6 +647,12 @@ def create_weights(self):
assert hasattr(self.backend, "create_weights"), (
f"Backend {self.backend.__class__.__name__} must implement create_weights()"
)
# Sync quant_config to backend before creating weights.
# apply_layerwise_quant_config() matches modules by name (e.g. "model.layers.X.mlp.experts"),
# but the backend's module name has a ".backend" suffix that prevents matching.
# Propagate the (possibly updated) quant_config from ConfigurableMoE to backend here
# so that backend._get_quant_method() uses the correct per-layer quantization config.
self.backend.quant_config = self.quant_config
return self.backend.create_weights()

def load_weights(self, weights: List[Dict], allow_partial_loading: bool = False):
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/defs/accuracy/references/mmlu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,8 @@ Qwen3/Qwen3-30B-A3B:
- quant_algo: NVFP4
kv_cache_quant_algo: FP8
accuracy: 79.53
- quant_algo: W4A8_AWQ
accuracy: 79.53
- quant_algo: W4A8_MXFP4_FP8
accuracy: 79.78
- quant_algo: W4A8_MXFP4_MXFP8
Expand Down
23 changes: 23 additions & 0 deletions tests/integration/defs/accuracy/test_llm_api_pytorch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4757,6 +4757,29 @@ def test_nvfp4(
task = GSM8K(self.MODEL_NAME)
task.evaluate(llm)

@skip_pre_hopper
@skip_post_blackwell
@pytest.mark.parametrize(
"tp_size,pp_size,ep_size,attention_dp,cuda_graph,overlap_scheduler",
[(2, 1, 1, False, False, True)],
ids=["tp2_ep1"])
def test_w4a8(self, tp_size, pp_size, ep_size, attention_dp, cuda_graph,
overlap_scheduler):
pytorch_config = dict(
disable_overlap_scheduler=not overlap_scheduler,
cuda_graph_config=CudaGraphConfig() if cuda_graph else None)

llm = LLM(
f"{llm_models_root()}/Qwen3/saved_models_Qwen3-30B-A3B_w4a8_hf",
tensor_parallel_size=tp_size,
pipeline_parallel_size=pp_size,
moe_expert_parallel_size=ep_size,
**pytorch_config,
enable_attention_dp=attention_dp)
Comment on lines +4766 to +4778

@coderabbitai coderabbitai Bot May 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Cover the actual regression trigger (w4a8 + fp8 kv cache) in this test.

This test currently runs W4A8 only, but the PR fix is specifically about W4A8 combined with FP8 KV cache. Add kv_cache_config=KvCacheConfig(dtype="fp8", ...) (and keep this test path) so it actually guards the reported failure mode.

As per coding guidelines, QA/test coverage should include relevant failure modes for the feature/fix, and this PR objective explicitly targets W4A8 + FP8 KV cache behavior.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py` around lines 4919 -
4931, The test test_w4a8 currently only exercises W4A8 quantization; update the
LLM instantiation in test_w4a8 to enable an FP8 KV cache by passing
kv_cache_config=KvCacheConfig(dtype="fp8", ...) to LLM so the test covers the
W4A8 + FP8 KV cache regression; ensure you construct KvCacheConfig with the same
required FP8 parameters used elsewhere in tests (or import the shared FP8
config), keep the existing model path and other args, and add any necessary
import for KvCacheConfig so the test actually exercises the faulty combination.

with llm:
task = MMLU(self.MODEL_NAME)
task.evaluate(llm)

@pytest.mark.parametrize("moe_backend", ["CUTLASS", "TRTLLM"])
@pytest.mark.parametrize("tp_size,pp_size,ep_size", [
(1, 1, 1),
Expand Down
1 change: 1 addition & 0 deletions tests/integration/test_lists/qa/llm_function_core.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -725,6 +725,7 @@ accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_nvfp4[tep4_latency_moe
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_nvfp4[tep4_latency_moe_trtllm-torch_compile=True]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a16_mxfp4[latency-TRITON]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a16_mxfp4[latency-TRTLLM]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8[tp2_ep1]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8_mxfp4[fp8-latency-CUTLASS]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8_mxfp4[fp8-latency-TRTLLM]
accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8_mxfp4[mxfp8-latency-CUTLASS]
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/test_lists/test-db/l0_dgx_h100.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ l0_dgx_h100:
# llmapi
- unittest/llmapi/test_mpi_session.py::test_llmapi_launch_multiple_tasks
- accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_gen_only_spec_dec
# ------------- Model specific tests ---------------
- accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_w4a8[tp2_ep1]
- condition:
ranges:
system_gpu_count:
Expand Down