refactor(nvidia): run multi-node fixed-sequence recipes on the InferenceX client - #3415
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline finding, I checked two other things and ruled them out: the dropped use_chat_template/random_range_ratio fields — many old sa-bench blocks (e.g. the b200-fp8 dsr1 disagg-1p1d recipe) never set them in the first place, so nothing is silently lost there; and the 22 telemetry-enabled Qwen3.5 GB200/GB300 recipes, which do retain benchmark.concurrencies on the new custom block (matching the PR description's claim that srt-slurm still needs it to size expected power windows).
Extended reasoning...
This is a mechanical, repo-wide swap of the benchmark: block in 228 recipe YAMLs from sa-bench to a shared custom script invocation; it touches no auth/crypto/permission logic. A confirmed finding (missing perf-changelog.yaml entry per AGENTS.md's non-negotiable invariant) is already queued as an inline comment, so I did not restate it in the body. I additionally spot-checked two other candidate concerns from the ruled-out list against the actual diff (dropped use_chat_template/random_range_ratio fields, and retained concurrencies on telemetry recipes) and confirmed both are non-issues, which is worth surfacing since a human reviewer would otherwise re-check the same things.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
perf-changelog.yaml— This PR changes measured benchmark behavior (10x vs 16x/20x num-prompts multiplier, req-rate inf vs 300/700, 2x-at-inf warmup) across 228 recipes with no perf-changelog.yaml entry, so downstream consumers of the changelog lose the record of this performance-relevant change. AGENTS.md's non-negotiable rule states every recipe modification requires a new append-only perf-changelog.yaml entry (AGENTS.md:124), and none of the 228 recipe edits add one. Fix: append a perf-changelog.yaml entry (at the tail, preserving existing bytes) describing the client migration and its measured-behavior deltas (prompt multiplier, request rate, warmup) before merging. [also at: benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-1p1d-p-tp4-d-tp8-b512-c32-mtp.yaml:145 - This PR changes the benchmark client (sa-bench to custom/srt_fixed_sequence.sh) for 228 recipes, changing measured request rate, warmup, and prompt-count behavior for every one of them, but does not add a perf-changelog.yaml entry, violating AGENTS.md:124 ("Every change that can…]Why this was flagged
AGENTS.md:124 requires: 'Every change that can affect benchmark performance and every recipe addition or modification requires a new perf-changelog.yaml entry... append only at the tail.' This diff (git diff bb089d3 HEAD) modifies benchmark: blocks in 228 recipe YAMLs, switching sa-bench to a custom srt_fixed_sequence.sh client and changing num_prompts multiplier (16x/20x -> 10x in 34 recipes), request rate (300/700 -> inf in 16 recipes), and warmup count/rate per the PR description itself. perf-changelog.yaml's tail (checked via git diff) shows no new entry added in this diff. No safeguard catches this since it's a manual convention, not a CI-enforced one visible in this diff.
Verification: normal (policy invariant violation this PR introduces). AGENTS.md:124 is a "Non-negotiable benchmark invariant": "Every change that can affect benchmark performance and every recipe addition or modification requires a new perf-changelog.yaml entry. The file is append-only and byte-sensitive... append only at the tail." The diff (git diff bb089d3..HEAD) modifies the benchmark: block of 228…
bad6dd0 to
22f8235
Compare
0f03491 to
1898327
Compare
…nceX client Move the 228 recipe files behind the 27 active NVIDIA multi-node fixed-sequence configs from srt-slurm's sa-bench to benchmarks/multi_node/srt_fixed_sequence.sh. Each recipe keeps its ISL/OSL and sends the same served model name and /model tokenizer sa-bench used; concurrencies come from the matrix. Telemetry recipes keep benchmark.concurrencies, which srt-slurm needs for their expected power windows. Deprecated recipes are unchanged.
…ence power lanes The client runs CONC_LIST, but GB200, GB300 and B200 injected it into benchmark.concurrencies only for AgentX, so a narrowed fixed-sequence power run expected windows it never measured. Inject on every DCGM power lane, as H200 already does.
The pinned srt-slurm passes --publish-metrics to Dynamo TRT-LLM workers by default, which Dynamo builds before ai-dynamo/dynamo#12162 reject, so workers exited at startup. Set engine.publish_metrics: false on the 156 active TRT-LLM recipes.
The first container step on an H100 node takes about eight minutes to set up the job image, and etcd is that step, so it missed srt-slurm's 300s readiness default. Allow 1200s in the 12 active H100 recipes.
Drop the hardcoded MODEL from the converted recipes; the client now asks the frontend, which serves the name the workers registered.
Node container setup plus loading DeepSeek-R1 FP8 weights over NFS took longer than srt-slurm's 30-minute health default. Use 360 attempts, as the GB200 recipes do.
Each service ran in its own ~40 GB instance of the job image on the head node, beside prefill rank 0, which was then killed while loading weights. etcd and NATS are static binaries from /configs, so use the small nginx image the profile already provides.
TRTLLM_UCX_INTERFACE named RDMA devices (mlx5_*), not a network
interface, so TRT-LLM could not find a local IP ("Can't get local ip
from NIC Interface") and KV cache never reached decode. No other
cluster sets it; drop it from the 26 active B300 TRT-LLM recipes.
SGLang aborts when a TP rank finishes loading 480 s before the slowest one. On H100 the weights come over NFS, and a node reused from an earlier run loads from page cache, so the head node's ranks finished first and the run failed on every job that reused it. Read the weights into each worker node's page cache in host_setup before workers start.
B300 TRT-LLM disagg is broken on the DSXE cluster regardless; leave these recipes unchanged here.
22f8235 to
2957ea0
Compare
1 participant
Stacked on #3413.
Background
NVIDIA multi-node fixed-sequence recipes benchmark with srt-slurm's built-in
sa-bench, each with its own prompt counts, request rates and concurrency list. #3413 adds one InferenceX client,benchmarks/multi_node/srt_fixed_sequence.sh, that takes its concurrencies from the matrixCONC_LIST, discovers the served model name from the frontend, and writes a power measurement window per concurrency.Changes
Client. The 228 recipe files used by the 27 non-deprecated multi-node fixed-sequence configs in
configs/nvidia-master.yamlswitch to:concurrencies, and the 8 override variants that only set them, are removed. The exception is the 22 telemetry-enabled Qwen3.5 GB200/GB300 recipes: srt-slurm needsbenchmark.concurrenciesto build their expected power windows.inf, and 2x warmup atinf.Power. On every DCGM power lane,
launch_gb200-nv.sh,launch_gb300-nv.shandlaunch_b200-nscale-slurm.shinjectCONC_LISTintobenchmark.concurrencies. They previously did this on AgentX lanes only. Without it, srt-slurm expects windows for the recipe's concurrencies rather than the ones the client runs.TRT-LLM. The 156 TRT-LLM recipes set
engine.publish_metrics: false. The pinned srt-slurm passes--publish-metrics, which Dynamo builds before ai-dynamo/dynamo#12162 reject.H100. All 12 recipes:
nginx-sqshimage with a 1200 s readiness timeout. Each used to be a ~40 GB copy of the job image on the head node, and the first container step on a node takes minutes.health_check.max_attempts: 360(1 h). Loading DeepSeek-R1 FP8 over NFS outlasts the 30 min default.host_setupreads the weights into each worker's page cache before workers start. SGLang aborts if one TP rank finishes loading 480 s before another, which happened whenever one node already had the weights cached.Testing
One e2e run per lane on this branch (8k1k,
--no-evals):require-powerpower_valid: 1require-power, concurrencies 1 and 2customwindow andpower_valid: 1per concurrencyevals-only,eval-limit=50B300 TRT-LLM multi-node is broken on the DSXE cluster independently of this PR; see #3425. Background:
rdmap*), not InfiniBand (mlx5_*).TRTLLM_UCX_INTERFACEtomlx5_*devices.ai-dynamo/tensorrtllm-runtime:0.8.1.post1:backend 'LIBFABRIC' not found).main.