[https://nvbugs/6517846][fix] Raise AGG server-ready timeout to 3600s and unwaive 4 perf-sanity cases

[chenfeiz0326]

Summary

Four aggregated perf-sanity cases in tests/integration/test_lists/waives.txt were re-run on current ToT. All four pass end to end, so this PR removes their waives.

Three were simply stale waives. The fourth — gb300_deepseek-v4-pro ctx_only — needed a one-line fix first: it was failing on a server-readiness budget that no per-case knob can raise, so this PR raises AGG_SERVER_READY_TIMEOUT from 1800s to 3600s and unwaives the case alongside it.

The readiness fix

test_perf_sanity.py gates server startup on:

timeout=min(self.timeout, server_ready_timeout(AGG_SERVER_READY_TIMEOUT, "AGG"))

The min() means a case's test-db TIMEOUT (N) annotation cannot buy more startup time — readiness is global per mode, previously 1800s for aggregated and 3600s for disaggregated. The largest aggregated case needs ~2000s of model load + autotune before /health answers, so it failed readiness while the server was still coming up correctly.

This PR raises the aggregated bound to match the disaggregated one. Nothing else changes: the constant is only ever consumed by that one min(), and it is still overridable by TRTLLM_TEST_AGG_SERVER_READY_TIMEOUT.

Worth noting for reviewers: the cap is newer than the waive it blocks. It was introduced by ebd197f776 (#16403, 2026-07-29), while this case was waived by 1ae9b86f12 (#16882, 2026-07-27) — two days earlier. So the cap was never the original waive cause; it is a second obstacle that appeared later and would have kept the case red even after the original issue was gone.

Cases unwaived

case	job	pytest	request accounting
aggr_upload-deepseek_r1_fp4_v2_2_nodes_grace_blackwell-r1_fp4_v2_dep8_mtp1_8k1k	2929320	1 passed in 1386.44s	10240 / 10240 successful, 0 failed
aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell-k25_thinking_fp4_tep8_32k8k	2929336	1 passed in 689.88s	10 / 10 successful, 0 failed (num_prompts=10, max_concurrency=2)
aggr_upload-deepseek_r1_fp4_v2_grace_blackwell-r1_fp4_v2_dep4_mtp1_1k8k	2929319	1 passed in 4879.88s	10240 / 10240 successful, 0 failed
aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL	2944440	1 passed in 7041.40s	43010 / 43010 successful, 0 failed

Every run was CI-faithful — no server-readiness env override. The v4-pro run (2944440) used test_perf_sanity.py checked out verbatim from this PR's commit, so the 3600s came from the source constant, not from an env var.

Each case was confirmed against independent sources, because a green pytest summary alone is not sufficient here (see below):

• pytest summary 1 passed
• report.xml: tests="1" failures="0" errors="0" skipped="0" — executed, not deselected
• sacct: COMPLETED 0:0
• the launcher's own final line: Aggregated test completed successfully
• every MPI rank exit code 0, zero Bus error occurrences
• request accounting from <run_dir>/aggr-*/trtllm-benchmark.*.log — note 200 OK never appears in slurm stdout, so the case subdirectory is the only place this can be read

v4-pro before / after

check	before (1800s)	after (this PR)
sacct	CANCELLED	COMPLETED 0:0
pytest	1 failed in 1802.20s	1 passed in 7041.40s
launcher verdict	Error: Aggregated test failed	Aggregated test completed successfully
readiness timeouts	4	0
Bus error	7	0
MGMN rank exit codes	135 / 135 / 135	0 / 0 / 0
requests	— (never served)	43010 / 43010, 0 failed

One caveat reviewers should weigh

An earlier v4-pro run that worked around the readiness cap with an env override reached 1 passed in 7116.39s and still failed the job: after rank 0 shut down cleanly, all three peer mgmn_worker_node ranks hit Signal: Bus error (7) / Non-existant physical address → exit 135, so srun --kill-on-bad-exit=1 returned non-zero and the launcher's cleanup_on_failure failed the stage.

That did not recur in the verification run above (0 bus errors, all ranks 0). But this PR's change cannot have fixed it — raising a readiness timeout does not touch MPI or shared-memory teardown. The honest reading is that the SIGBUS is intermittent, not repaired. One clean run shows it is not deterministic; it is not proof it can never recur.

If it reappears post-merge, it should get its own bug rather than a re-waive of this case — the workload itself is functionally healthy in both runs (43010/43010 successful, 0 failed), so a teardown-ordering fault says nothing about serving correctness.

Two adjacent things that look alarming but are not: sacct reporting CANCELLED by <uid> is the launcher scancelling itself, not preemption; and TPOT / ITL / User throughput reading 0.00 with Avg Decoded Tokens per Iter = 1.00 is expected for a ctx_only case, which runs random_output_len=1 and so has no inter-token phase.

Bug status

• nvbugs/6422339 (r1_fp4_v2_dep4_mtp1_1k8k, k25_thinking_fp4_tep8_32k8k) — already Dev - Closed - Verified (fixed 2026-07-20). Cleanly stale.
• nvbugs/6530213 (r1_fp4_v2_dep8_mtp1_8k1k) — still Dev - Open - To fix, but the case passes on ToT, so the waive is stale regardless.
• nvbugs/6517846 (v4-pro ctx_only) — this PR addresses the readiness half. Two sibling gb300_glm-5-fp4 cases under the same bug remain waived and are untouched here; they were not re-tested.

The disagg_upload- v4-pro variants under nvbugs/6581075 are also untouched — this PR unwaives only the aggr_upload-ctx_only variant.

Test Coverage

The unwaived cases are re-enabled in these post-merge stages, which run pre-merge on this PR:

• GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-* — r1_fp4_v2_dep4_mtp1_1k8k
• GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1 / -3
• GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1
• GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-* — the v4-pro ctx_only case, confirmed at l0_gb300_multi_gpus_perf_sanity.yml:33 and mapped by L0_Test.groovy:5428-5430

The AGG_SERVER_READY_TIMEOUT change affects every aggregated perf-sanity case, not just the four here, so the whole aggregated set in those stages is the real coverage for it. The change can only lengthen how long a case is willing to wait for a healthy server, so a case that passed before still passes; the risk it carries is the opposite one — a genuinely dead server now burns up to 3600s instead of 1800s before failing.

Note the verification runs were executed on GB300 (supported_gpus is advisory in perf-sanity and never read by test_perf_sanity.py), so the GB200 stages above are what confirm the GB200 hardware path.

PR Checklist

• Commit message follows the required format
• Commits are signed off (DCO)
• Test coverage described above

Dev Engineer Review

• Removed four stale waiver entries from tests/integration/test_lists/waives.txt.
• Added three GLM5 context-only waiver entries.
• Increased AGG_SERVER_READY_TIMEOUT from 1800 to 3600 seconds.
• Verified all four unwaived cases with 43,010/43,010 successful requests and no readiness timeouts, bus errors, launcher failures, or MPI rank failures.
• The waiver format and test scope require confirmation against the intended GB200 and GB300 post-merge stages.
• CI was unstable because the required ci: full pre-merge approved label was missing.

QA Engineer Review

• No test-db/ or qa/ files were modified.
• Modified test code: AGG_SERVER_READY_TIMEOUT in tests/integration/defs/perf/test_perf_sanity.py.
• No test functions were added, modified, or removed.
• All four affected perf-sanity cases are represented in tests/integration/test_lists/waives.txt.
• CBTS coverage data is unavailable.
• Verdict: needs follow-up.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-*,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1"

[coderabbitai]

Walkthrough

The pull request increases the aggregated-server readiness timeout from 1800 to 3600 seconds. It also removes several performance-test waivers and adds GLM5 context-only waivers.

Changes

Performance test configuration

Layer / File(s)	Summary
Extend aggregated-server readiness timeout tests/integration/defs/perf/test_perf_sanity.py	Increases AGG_SERVER_READY_TIMEOUT to 3600 seconds and updates the large-model initialization comments.
Update performance-test waiver entries tests/integration/test_lists/waives.txt	Removes DeepSeek V4 Pro FP4, DeepSeek R1 FP4 Grace Blackwell, and Kimi K2 Thinking waivers. Adds GLM5 context-only waivers and retains other existing entries.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#16920: Removes related Kimi Thinking performance waivers.
• NVIDIA/TensorRT-LLM#17386: Removes Kimi K2 Thinking performance waivers.
• NVIDIA/TensorRT-LLM#17444: Removes obsolete entries from the same performance-test waiver list.

Suggested labels: ci: full pre-merge approved

Suggested reviewers: brnguyen2, qijune, xinhe-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the timeout increase and removal of four perf-sanity waivers.
Description check	✅ Passed	The description explains the problem, solution, affected cases, risks, test evidence, coverage, and checklist status.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tensorrt-cicd]

PR_Github #64938 [ run ] triggered by Bot. Commit: bc6933f Link to invocation

[tensorrt-cicd]

PR_Github #64938 [ run ] completed with state FAILURE. Commit: bc6933f
/LLM/main/L0_MergeRequest_PR pipeline #52773 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chenfeiz0326]

Correction on the v4-pro case (not unwaived here)

My description above understated how healthy the v4-pro run actually is, so to be accurate:

Under TRTLLM_TEST_AGG_SERVER_READY_TIMEOUT=3600, aggr-*/trtllm-benchmark.0.0.log reports a fully successful benchmark:

Total requests:          43010
Successful requests:     43010
Failed requests:          0
Benchmark duration (s):   5085.05
Total Token throughput:   62530.98 tok/s

(TPOT / ITL / User throughput = 0.00 and Avg Decoded Tokens per Iter = 1.00 are expected here — the case runs random_output_len=1, so a ctx_only run emits one output token and has no inter-token phase.)

So the workload is functionally fine. Both obstacles are harness-level:

1. 1800s AGG readiness cap — deterministic failure under CI-faithful settings; the server needs ~2000s. This alone is why the case cannot be unwaived in this PR.
2. Teardown SIGBUS — strictly after serving completes and rank 0 exits 0, four peer mgmn_worker_node processes fault (Signal code: Non-existant physical address) → exit 135 → --kill-on-bad-exit=1 → cleanup_on_failure. A teardown-ordering / shm fault, not a compute failure.

Whether (2) reproduces in Jenkins is open — a shm bus error is also what a container /dev/shm / enroot difference produces, and I have not compared my container environment against a real CI stage. Flagging it so nobody treats it as confirmed CI behavior.

Neither point affects the three cases this PR unwaives; they were verified independently and CI-faithfully.

[chenfeiz0326]

v4-pro ctx_only verified passing under this PR (aws-cmh GB300, job 2944440)

Re-ran the case with test_perf_sanity.py taken verbatim from this PR — so the 3600s comes from the AGG_SERVER_READY_TIMEOUT source constant, with no TRTLLM_TEST_*_SERVER_READY_TIMEOUT env override in the launcher (verified absent). The previous SIGBUS does not reproduce.

check	result
sacct	COMPLETED 0:0, elapsed 02:03:12
pytest	1 passed, 981 deselected in 7041.40s
launcher verdict	Aggregated test completed successfully (exit 0)
report.xml	tests="1" failures="0" errors="0" skipped="0"
readiness timeouts	0 (was 4 at 1800s)
Bus error count	0 (was 7)
per-rank exit codes	Rank0 task 0, Rank0 MPI comm 0, Rank1/2/3 MGMN 0 (were 135)
request accounting	43010 / 43010 successful, 0 failed, 62639.39 tok/s

Both blockers on nvbugs/6517846 are therefore cleared:

1. Readiness (fixed by this PR). 8x Cache size after warmup and zero readiness timeouts, purely from the raised constant.
2. Teardown SIGBUS (did not recur). All peer mgmn_worker_node ranks exit 0 this time, versus four exit-135 bus errors before. Worth stating plainly that this makes the earlier crash look intermittent rather than fixed by anything here — the 3600s change does not touch MPI/shm teardown. One clean run is evidence it is not deterministic; it is not proof it can never recur. If it reappears in post-merge, it should get its own bug rather than a re-waive of this case.

This PR now unwaives all 4 cases originally in the cohort.

[chenfeiz0326]

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-"

[chenfeiz0326]

Rebased onto 10689401f1 to resolve a waives.txt conflict.

The conflict was purely positional. Three upstream commits churned the same region of waives.txt (#17458 waived 23 cases, #17418 unwaived some, #17467 waived 5 more), and one of them landed a new waive — aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_... (nvbugs/6581075) — on the line immediately following the v4-pro line this PR deletes. test_perf_sanity.py was not touched by any upstream commit in that range, so the AGG_SERVER_READY_TIMEOUT change rebased cleanly.

Resolution kept every upstream line and dropped only the four this PR intends to unwaive. Verified post-rebase: git diff origin/main is exactly 4 deletions, 0 additions in waives.txt, plus the timeout constant in test_perf_sanity.py. Specifically preserved:

• the new glm-5-fp4_..._con1024_... waive from [None][infra] Waive 23 failed cases for main in post-merge 2894 #17458
• the two sibling gb300_glm-5-fp4 waives still under nvbugs/6517846
• both disagg_upload- v4-pro waives under nvbugs/6581075 (this PR unwaives only the aggr_upload-ctx_only variant)

No re-verification run is needed: the rebase changed no test code, and the aws-cmh GB300 verification above already exercised the exact test_perf_sanity.py content now on this branch.

/bot run --disable-fail-fast --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB200-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-"

[chenfeiz0326]

/bot skip --comment "Only unwaive perf tests, no need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #65021 [ skip ] triggered by Bot. Commit: 6c50743 Link to invocation

[tensorrt-cicd]

PR_Github #65021 [ skip ] completed with state SUCCESS. Commit: 6c50743
Skipping testing for commit 6c50743

Link to invocation