[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na… - #17031
[NVBUG-6327718][test] Unwaive test_disaggregated_videomme[nemotron_na…#17031aswinvisva wants to merge 1 commit into
Conversation
|
/bot run --stage-list "DGX_B200-PyTorch-4" |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
WalkthroughTwo waiver entries are removed from the integration test waiver list, allowing the disaggregated VideoMME test to run on B200 and H20. ChangesTest waiver cleanup
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #62622 [ run ] triggered by Bot. Commit: |
|
PR_Github #62622 [ run ] completed with state |
|
The two lines removed are Two other gaps against the description: nvbugs/6327718 is still P0 Could you attach a passing run of the fp8 variant on B200 and H20 (host + commit + pytest command), the way repair-bot did for nvfp4? For what it's worth the fp8 variant already runs unwaived pre-merge via |
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
This looks like a no-op. [tests/integration/test_lists/waives.txt:16](https://github.com/NVIDIA/TensorRT-LLM/pull/17031/files#diff-621bd2af82a3b97c7a5948368d36c14582ffdf73361deb7f42d2ff395b22167eR16) still waives the same node ID unconditionally (accuracy/test_epd_disagg_multimodal.py::TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8], nvbugs/6478692), so the test stays skipped on every platform and CI can't confirm the fix. Also, the fp8 param only appears in test-db/l0_h100.yml:164 and qa/llm_function_core.txt:86 — l0_b200.yml lists only the qwen3vl/nvfp4 variants — so the B200 entry was already vestigial. Worth confirming the 6327718 signature matches what you believe is fixed, and updating that bug alongside the removal.
…no_v3_omni_fp8] and stress-repeat 20x
Removes two waivers linked to NVBug 6327718 for the Nemotron Nano V3 Omni fp8
E/PD-disagg VideoMME test on B200 and H20. The recorded signature on 6327718
is a stochastic `Fatal Python error: Aborted` (SIGABRT) that appeared in about
2.5%% of pre-merge runs from 2026-06-09 through 2026-07-19 under the 128-way
ThreadPoolExecutor in the videomme driver. This test is currently believed to
be fixed on main; this PR is a stress-test to demonstrate that.
Changes:
* tests/integration/test_lists/waives.txt: drop two `full:B200/` and
`full:H20/` skip entries for 6327718.
* tests/integration/defs/accuracy/test_epd_disagg_multimodal.py: add a
`_repeat` parametrize dimension so each variant runs 20 iterations per
stage. Because the SIGABRT is stochastic, a single passing run does not
beat the ~2.5%% observed rate; 20 back-to-back reps give ~40%% detection
probability if the bug were still latent.
* tests/integration/test_lists/test-db/l0_h100.yml: enumerate 20 rep IDs
for the fp8 variant (the only one wired into H100 pre-merge).
* tests/integration/test_lists/test-db/l0_b200.yml: enumerate 20 rep IDs
for the nvfp4 variant; keep qwen3vl_2b_instruct at a single rep so it
still gets collected on B200.
* tests/integration/test_lists/qa/llm_function_core.txt: update all three
variants to the new `-rep00` naming for the QA collection.
This is a throwaway PR intended for stress verification, not merge.
Trigger with:
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast
Related: NVBug 6478692 (a separate, previously-global waiver on the same
node ID) has since been removed on main; no reference remains in waives.txt.
Signed-off-by: Aswin Visva <31215515+aswinvisva@users.noreply.github.com>
08015b1 to
eefdfc4
Compare
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65073 [ run ] triggered by Bot. Commit: |
|
PR_Github #65073 [ run ] completed with state
|
|
/bot run --stage-list "DGX_H100-PyTorch-1, DGX_B200-PyTorch-4" --disable-fail-fast |
|
PR_Github #65111 [ run ] triggered by Bot. Commit: |
|
PR_Github #65111 [ run ] completed with state |
5 participants
…no_v3_omni_fp8] on B200 and H20
The root cause of the pre-fix failure —
TypeError: isinstance() arg 2 must be a type, a tuple of types, or a unionattensorrt_llm/executor/proxy.pywhen the test infrastructure's session-reuse cache monkey-patchesMpiPoolSessionin the proxy module namespace with a factory function — is already fixed on main. The proxy code was refactored to identify pool-backed sessions by excluding(MpiCommSession, RemoteMpiCommSessionClient)instead ofisinstance(x, MpiPoolSession).Remove the two waivers linked to NVBug 6327718 so CI can verify the fix on both platforms.
Dev Engineer Review
nvbug: 6327718waivers for the targeted test on B200 and H20.tests/integration/test_lists/waives.txtwith no code or API changes.QA Engineer Review
test-db/orqa/files were modified.TestVideoMMEEPD::test_disaggregated_videomme[nemotron_nano_v3_omni_fp8]onfull:B200andfull:H20.Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.