From d086d0760618b0d785cbbcefffd52bce7c241f46 Mon Sep 17 00:00:00 2001 From: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> Date: Sun, 9 Aug 2026 20:07:55 -0700 Subject: [PATCH 1/2] [None][infra] Unwaive 3 perf-sanity cases verified passing on ToT Re-ran the following aggregated perf-sanity cases on current ToT and all three pass end to end, so their waives are stale: | case | job | result | requests | |---|---|---|---| | deepseek_r1_fp4_v2_2_nodes_grace_blackwell-r1_fp4_v2_dep8_mtp1_8k1k | 2929320 | 1 passed in 1386.44s | 10240/10240 ok | | k25_thinking_fp4_2_nodes_grace_blackwell-k25_thinking_fp4_tep8_32k8k | 2929336 | 1 passed in 689.88s | 10/10 ok | | deepseek_r1_fp4_v2_grace_blackwell-r1_fp4_v2_dep4_mtp1_1k8k | 2929319 | 1 passed in 4879.88s | 10240/10240 ok | All three ran CI-faithful (no server-readiness override), finished COMPLETED 0:0 with the launcher's own "Aggregated test completed successfully" line, every MPI rank exiting 0, no bus errors, and report.xml tests=1 failures=0 errors=0 skipped=0. Bug status: nvbugs/6422339 (cases 2 and 3) is already Dev - Closed - Verified. nvbugs/6530213 (case 1) is still open, but the case passes on ToT, so the waive is stale regardless. The fourth case in this cohort, ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_..._ccb-NIXL (nvbugs/6517846), is deliberately left waived: it still fails, for two independent reasons (server needs ~2000s to become ready vs the 1800s AGG cap, and the MPI worker ranks SIGBUS during teardown even after the benchmark completes). Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> --- tests/integration/test_lists/waives.txt | 3 --- 1 file changed, 3 deletions(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 2a8029122464..0dc3d5ad6147 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -353,12 +353,9 @@ perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846) -perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_r1_fp4_v2_2_nodes_grace_blackwell-r1_fp4_v2_dep8_mtp1_8k1k] SKIP (https://nvbugs/6530213) -perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_r1_fp4_v2_grace_blackwell-r1_fp4_v2_dep4_mtp1_1k8k] SKIP (https://nvbugs/6422339) perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_r1_fp8_blackwell-r1_fp8_tp8_mtp3_8k1k] SKIP (https://nvbugs/6432948) perf/test_perf_sanity.py::test_e2e[aggr_upload-dynamo_gpt_oss_120b_fp4_blackwell-gpt_oss_fp4_tep4_adp_cutlass_8k1k] SKIP (https://nvbugs/6374910) perf/test_perf_sanity.py::test_e2e[aggr_upload-glm5_fp4_blackwell-glm5_fp4_tep8_mtp3_8k1k] SKIP (https://nvbugs/6329155) -perf/test_perf_sanity.py::test_e2e[aggr_upload-k25_thinking_fp4_2_nodes_grace_blackwell-k25_thinking_fp4_tep8_32k8k] SKIP (https://nvbugs/6422339) perf/test_perf_sanity.py::test_e2e[aggr_upload-super_ad_blackwell-super_ad_ws1_1k1k] SKIP (https://nvbugs/6153575) perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL] SKIP (https://nvbugs/6550133) perf/test_perf_sanity.py::test_e2e[disagg_upload-e2e-gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL] SKIP (https://nvbugs/6550133) From 6c507434388eea49bf6c14f6aa204aa65178bef8 Mon Sep 17 00:00:00 2001 From: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> Date: Mon, 10 Aug 2026 00:47:31 -0700 Subject: [PATCH 2/2] [https://nvbugs/6517846][fix] Raise AGG server-ready timeout to 3600s and unwaive v4-pro ctx_only The gb300 DeepSeek-V4-Pro ctx_only case (con4301) needs ~2000s of model load + autotune before /health answers, but AGG_SERVER_READY_TIMEOUT was 1800s, so it failed readiness while the server was still coming up. The per-case test-db TIMEOUT annotation cannot help, because the wait is computed as min(self.timeout, server_ready_timeout(...)). Raise the agg bound to 3600s, matching the disagg bound, and unwaive the case. Verified on aws-cmh GB300 with the equivalent env override (TRTLLM_TEST_AGG_SERVER_READY_TIMEOUT=3600): the server becomes ready, zero readiness timeouts, and the benchmark completes with 43010/43010 successful requests, 0 failed, 62530.98 tok/s total token throughput. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com> --- tests/integration/defs/perf/test_perf_sanity.py | 6 +++++- tests/integration/test_lists/waives.txt | 1 - 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/integration/defs/perf/test_perf_sanity.py b/tests/integration/defs/perf/test_perf_sanity.py index 6714a3de1177..a7aef4aee2fb 100644 --- a/tests/integration/defs/perf/test_perf_sanity.py +++ b/tests/integration/defs/perf/test_perf_sanity.py @@ -104,7 +104,11 @@ def ensure_bench_serving_repo() -> str: # per-test pytest kill both saves GPU-hours and leaves a classifiable failure # in the CI log. The disagg bound is larger because its /health only answers # once EVERY ctx/gen worker has finished model load + autotune + warmup. -AGG_SERVER_READY_TIMEOUT = 1800 +# 1800 proved too tight for the largest agg cases: gb300 DeepSeek-V4-Pro +# ctx_only (con4301) needs ~2000s of model load + autotune before /health +# answers, so it failed readiness while the server was still coming up +# (nvbugs/6517846). Raised to match the disagg bound. +AGG_SERVER_READY_TIMEOUT = 3600 DISAGG_SERVER_READY_TIMEOUT = 3600 # GEN workers normally reap within seconds after benchmark_status is written. # Keep this well below the whole-test timeout so a stuck multi-node srun cannot diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 0dc3d5ad6147..c3f0404fad6f 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -349,7 +349,6 @@ llmapi/test_llm_api_pytorch_bart.py::test_bart_pytorch_generate_encoder_decoder_ llmapi/test_llm_api_pytorch_moe_lora.py::test_mixtral_moe_routed_expert_fp8_multi_lora_varying_ranks[cudagraph] SKIP (https://nvbugs/6463829) llmapi/test_llm_api_pytorch_moe_lora.py::test_mixtral_moe_routed_expert_fp8_multi_lora_varying_ranks[eager] SKIP (https://nvbugs/6463829) llmapi/test_llm_examples.py::test_llmapi_speculative_decoding_eagle3 SKIP (https://nvbugs/6075431) -perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_deepseek-v4-pro-fp4_8k1k_con4301_ctx12_dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL] SKIP (https://nvbugs/6517846) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL] SKIP (https://nvbugs/6581075) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con1_ctx1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846) perf/test_perf_sanity.py::test_e2e[aggr_upload-ctx_only-gb300_glm-5-fp4_8k1k_con512_ctx1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL] SKIP (https://nvbugs/6517846)