[https://nvbugs/6287721][chore] Unwaive a test for update weights by shikicloud · Pull Request #16102 · NVIDIA/TensorRT-LLM

shikicloud · 2026-07-08T05:17:33Z

Summary by CodeRabbit

Removed the part0 and specific FP8 Qwen3 8B waivers.
Retained the multi-GPU part1 and part4 waivers.
Replaced the 4-GPU Ray part0 marker with explicit FP8 weight-update tests for Qwen3 8B and Qwen3 30B-A3B.

Dev Engineer Review

The changes use valid test-list formats.
The explicit test entries match the intended Qwen3 FP8 weight-update coverage.
No code, API, performance, or error-handling changes are included.
No duplicate or unintended public test entries were identified.

QA Engineer Review

Modified tests/integration/test_lists/waives.txt:
- Removed the part0 waiver.
- Removed the specific FP8 Qwen3 8B waiver.
- Retained the part1 and part4 waivers.
Modified tests/integration/test_lists/test-db/l0_dgx_b200.yml:
- Removed the part0 marker-based entry.
- Added explicit FP8 weight-update tests for Qwen3 8B and Qwen3 30B-A3B.
CI confirmed that the unwaived tests passed.
Verdict: sufficient.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

shikicloud · 2026-07-08T05:18:16Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

coderabbitai · 2026-07-08T05:19:36Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e6bd2046-79ad-4d14-9e66-0af0d2ea7a7d

📥 Commits

Reviewing files that changed from the base of the PR and between bfc4966 and 0f87f28.

📒 Files selected for processing (2)

tests/integration/test_lists/test-db/l0_dgx_b200.yml
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

Walkthrough

The Ray 4-GPU pre-merge list now runs explicit FP8 weight-update tests for Qwen3 8B and Qwen3 30B-A3B. The obsolete part0 and Qwen3 8B waivers were removed.

Changes

Ray FP8 test selection

Layer / File(s)	Summary
Configure explicit FP8 tests and waivers `tests/integration/test_lists/test-db/l0_dgx_b200.yml`, `tests/integration/test_lists/waives.txt`	The test list replaces the `part0` marker with explicit Qwen3 8B and Qwen3 30B-A3B FP8 weight-update tests. The waiver list removes the matching `part0` and Qwen3 8B entries while retaining `part1` and `part4` waivers.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: qijune, brnguyen2, pcastonguay, bowenfu

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description retains the template but leaves the Description and Test Coverage sections empty and provides no implementation or testing details.	Add a short explanation of the issue and solution, and list the relevant test coverage and results.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBugs ticket, change type, and primary change: removing a waiver for an update-weights test.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-08T05:24:02Z

PR_Github #58151 [ run ] triggered by Bot. Commit: 2e0fedd Link to invocation

Superjomn · 2026-07-08T05:29:45Z

 unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_partial_update_weights_nvfp4[auto-Qwen3/Qwen3-8B] SKIP (https://nvbugs/6372690)
 unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_partial_update_weights_nvfp4[fp8-Qwen3/Qwen3-30B-A3B] SKIP (https://nvbugs/6372690)
 unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_partial_update_weights_nvfp4[fp8-Qwen3/Qwen3-8B] SKIP (https://nvbugs/6372690)
+unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_update_weights_fp8[Qwen3/Qwen3-8B-Qwen3/Qwen3-8B-FP8] SKIP (https://nvbugs/6287721)


If skipping both tests is as expected from a functionality perspective, then you may use pytest.skip in the code. If not, and both should be TODOs, then keep it.

Superjomn

LGTM

shikicloud · 2026-07-08T06:15:39Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T06:21:04Z

PR_Github #58176 [ run ] triggered by Bot. Commit: a31e9e0 Link to invocation

tensorrt-cicd · 2026-07-08T06:24:31Z

PR_Github #58151 [ run ] completed with state ABORTED. Commit: 2e0fedd

Link to invocation

tensorrt-cicd · 2026-07-08T07:51:46Z

PR_Github #58176 [ run ] completed with state FAILURE. Commit: a31e9e0
/LLM/main/L0_MergeRequest_PR pipeline #46826 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

shikicloud · 2026-07-08T14:13:44Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

shikicloud · 2026-07-08T15:01:52Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T15:08:17Z

PR_Github #58252 [ run ] triggered by Bot. Commit: ad85689 Link to invocation

tensorrt-cicd · 2026-07-08T16:37:42Z

PR_Github #58252 [ run ] completed with state FAILURE. Commit: ad85689
/LLM/main/L0_MergeRequest_PR pipeline #46893 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

shikicloud · 2026-07-08T16:48:37Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1"

tensorrt-cicd · 2026-07-08T16:55:17Z

PR_Github #58267 [ run ] triggered by Bot. Commit: 3f55b11 Link to invocation

tensorrt-cicd · 2026-07-08T18:39:21Z

PR_Github #58267 [ run ] completed with state SUCCESS. Commit: 3f55b11
/LLM/main/L0_MergeRequest_PR pipeline #46906 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

Superjomn · 2026-07-13T07:40:25Z

The unwaived tests have passed: https://prod.blsm.nvidia.com/sw-tensorrt-llm-github-2/blue/organizations/jenkins/LLM%2Fmain%2FL0_Test-x86_64-Multi-GPU/detail/L0_Test-x86_64-Multi-GPU/2273/tests

Superjomn · 2026-07-13T07:40:43Z

/bot skip --comment "the unwaived tests passed"

tensorrt-cicd · 2026-07-13T07:46:37Z

PR_Github #58929 [ skip ] triggered by Bot. Commit: 3f55b11 Link to invocation

tensorrt-cicd · 2026-07-13T08:00:19Z

PR_Github #58929 [ skip ] completed with state SUCCESS. Commit: 3f55b11
Skipping testing for commit 3f55b11

Link to invocation

shikicloud · 2026-08-05T07:23:58Z

/bot run --stage-list "DGX_B200-4_GPUs-PyTorch-Ray-1" -- unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_update_weights_fp8[Qwen3/Qwen3-8B-Qwen3/Qwen3-8B-FP8] --run-ray

VALLIS-NERIA

This PR does not change KV cache manager code. No objection from the KV cache manager ownership scope.

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

coderabbitai · 2026-08-11T08:13:49Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

shikicloud · 2026-08-11T08:15:29Z

/bot run

tensorrt-cicd · 2026-08-11T08:20:56Z

PR_Github #65243 [ run ] triggered by Bot. Commit: 0f87f28 Link to invocation

tensorrt-cicd · 2026-08-11T09:23:23Z

PR_Github #65243 [ run ] completed with state FAILURE. Commit: 0f87f28
/LLM/main/L0_MergeRequest_PR pipeline #53021 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

shikicloud · 2026-08-11T09:29:12Z

/bot run

tensorrt-cicd · 2026-08-11T09:34:31Z

PR_Github #65261 [ run ] triggered by Bot. Commit: 0f87f28 Link to invocation

tensorrt-cicd · 2026-08-11T10:09:24Z

PR_Github #65261 [ run ] completed with state FAILURE. Commit: 0f87f28
/LLM/main/L0_MergeRequest_PR pipeline #53038 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

shikicloud · 2026-08-11T10:41:43Z

/bot run

tensorrt-cicd · 2026-08-11T10:49:10Z

PR_Github #65273 [ run ] triggered by Bot. Commit: 0f87f28 Link to invocation

brnguyen2

Approving — the comments below are optional touch-ups, not blockers.

Unwaive itself looks right: part0 currently contains exactly test_llm_update_weights_fp8[...8B] and [...30B-A3B], so the two explicit entries reproduce the old marker's coverage, and the part1/part4 waivers correctly stay.

Two things worth adding to the description: what actually fixed https://nvbugs/6287721 (or a link to the CI run that shows the two tests passing), and why the marker entry was replaced with explicit IDs rather than just dropping the waiver line.

brnguyen2 · 2026-08-11T12:13:07Z

  tests:
    - unittest/llmapi/test_llm_multi_gpu_pytorch.py -m "gpu4"
-    - unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py -m "part0"
+    - unittest/_torch/ray_orchestrator/multi_gpu/test_llm_update_weights_multi_gpu.py::test_llm_update_weights_fp8[Qwen3/Qwen3-8B-Qwen3/Qwen3-8B-FP8]


Pinning the two parametrizations explicitly means any test added later with @pytest.mark.part0 will not be scheduled here — the marker entry picked those up automatically. If the goal is only to unwaive, keeping -m "part0" and just deleting the waives.txt lines achieves the same thing without that gap. If you do want the explicit form (e.g. to re-waive a single param later), a short comment saying so would help the next person.

github-actions Bot assigned shikicloud Jul 8, 2026

shikicloud changed the title ~~[https://nvbugs/6287721][chore] Unwaive update weights test for specific models~~ [https://nvbugs/6287721][chore] Waive update weights test for specific models Jul 8, 2026

Superjomn reviewed Jul 8, 2026

View reviewed changes

Superjomn approved these changes Jul 8, 2026

View reviewed changes

shikicloud force-pushed the waive-nvbugs branch from 2e0fedd to a31e9e0 Compare July 8, 2026 06:13

shikicloud changed the title ~~[https://nvbugs/6287721][chore] Waive update weights test for specific models~~ [https://nvbugs/6287721][chore] Unwaive a test for update weights Jul 8, 2026

Superjomn enabled auto-merge (squash) July 13, 2026 07:41

auto-merge was automatically disabled August 5, 2026 08:40
Head branch was pushed to by a user without write access

shikicloud requested review from a team as code owners August 5, 2026 08:40

Barry-Delaney approved these changes Aug 7, 2026

View reviewed changes

yibinl-nvidia approved these changes Aug 7, 2026

View reviewed changes

VALLIS-NERIA approved these changes Aug 10, 2026

View reviewed changes

[https://nvbugs/6287721][test] Isolate part0 FP8 update-weight tests

0f87f28

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

shikicloud force-pushed the waive-nvbugs branch from 3f55b11 to 0f87f28 Compare August 11, 2026 08:13

shikicloud removed request for a team August 11, 2026 08:16

BowenFu approved these changes Aug 11, 2026

View reviewed changes

brnguyen2 approved these changes Aug 11, 2026

View reviewed changes

Uh oh!

Conversation

shikicloud commented Jul 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Dev Engineer Review

QA Engineer Review

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

shikicloud commented Jul 8, 2026

Uh oh!

coderabbitai Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Superjomn Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Superjomn left a comment

Choose a reason for hiding this comment

Uh oh!

shikicloud commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shikicloud commented Jul 8, 2026

Uh oh!

shikicloud commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

shikicloud commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Superjomn commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Superjomn commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

tensorrt-cicd commented Jul 13, 2026

Uh oh!

shikicloud commented Aug 5, 2026

Uh oh!

VALLIS-NERIA left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Aug 11, 2026

Uh oh!

shikicloud commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

shikicloud commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

tensorrt-cicd commented Aug 11, 2026

Uh oh!

shikicloud commented Aug 11, 2026

Uh oh!

shikicloud commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 8, 2026 •

edited

Loading

Superjomn commented Jul 13, 2026 •

edited

Loading