Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @xinli-sw's task in 4m 3s —— View job Review of PR #3396
One 🔴 BLOCKING issue (inline on Other checks passed: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | ||
| # shared script also serves H200, where these SM100 backends are unsupported. | ||
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | ||
| ATTENTION_ARGS=() | ||
| if (( GPU_MEM_MIB >= 150000 )); then | ||
| if (( TP == 2 )); then | ||
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| else | ||
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| fi | ||
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | ||
| fi |
There was a problem hiding this comment.
🔴 BLOCKING: This gate is not GB300-specific. dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh are symlinks to this file, and both B200 (~183 GB) and GB200 pass GPU_MEM_MIB >= 150000, so dsv41flash-fp4-b200-vllm-agentic-dspark and dsv41flash-fp4-gb200-vllm-agentic-dspark also start receiving --attention-config {FLASHINFER_MLA_SPARSE_DSV41 | FLASHMLA_MEGA_ATTN_DSV41, indexer_kv_dtype=mxfp4, indexer_sparse_logits=true} --kv-cache-dtype fp8. The comment above only accounts for H200.
Why it matters: Those two arms stay pinned to vllm/vllm-openai:nightly-cd10ed6f (2026-09-15), while the PR describes these as newly merged Blackwell settings landing in the 2026-09-22 nightly. vLLM's AttentionConfig rejects unknown keys and resolves backend through the AttentionBackendEnum name, so an older image that lacks indexer_sparse_logits, the mxfp4 indexer dtype, or the *_DSV41 backend names fails at vllm serve argument parsing and the whole B200/GB200 grid goes red. Even if the old nightly did accept them, this silently changes the B200/GB200 recipe (fp8 KV cache, MXFP4 indexer KV, different attention backends) while perf-changelog.yaml only lists the GB300 config key, which violates the AGENTS.md rule that every perf-affecting recipe change needs a changelog entry for the affected configs. (I could not reach the upstream vLLM source from this session to confirm which nightly introduced the keys, so please verify against cd10ed6f if you intend to keep the memory gate.)
Fix: Opt in from the GB300 launcher instead of inferring from GPU memory, following the existing VLLM_SAFETENSORS_LOAD_STRATEGY pattern in this script. Add export DSV41_SPARSE_INDEXER_LOGITS=1 next to export VLLM_ENGINE_READY_TIMEOUT_S=7200 in runners/launch_gb300-nv.sh (line 130), and gate here on that variable:
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | |
| # shared script also serves H200, where these SM100 backends are unsupported. | |
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | |
| ATTENTION_ARGS=() | |
| if (( GPU_MEM_MIB >= 150000 )); then | |
| if (( TP == 2 )); then | |
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| else | |
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| fi | |
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | |
| fi | |
| # Sparse indexer logits are opt-in in the upstream Blackwell recipe. This | |
| # shared script also serves H200, B200 and GB200; H200 lacks these SM100 | |
| # backends and the B200/GB200 arms pin an older nightly, so the GB300 | |
| # launcher opts in explicitly via DSV41_SPARSE_INDEXER_LOGITS=1. | |
| # TP2 uses FlashInfer; TP4 uses MegaAttention. fp8 resolves backend KV layout. | |
| ATTENTION_ARGS=() | |
| if [[ "${DSV41_SPARSE_INDEXER_LOGITS:-0}" == "1" ]]; then | |
| if (( TP == 2 )); then | |
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| else | |
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | |
| fi | |
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) | |
| fi |
Alternatively, if the intent is for B200 and GB200 to adopt the same settings, re-pin both to nightly-0961bbae in configs/nvidia-master.yaml and add their config keys and the recipe change to the perf-changelog.yaml entry.
| ATTENTION_ARGS=() | ||
| if (( GPU_MEM_MIB >= 150000 )); then | ||
| if (( TP == 2 )); then | ||
| ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| else | ||
| ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' | ||
| fi | ||
| ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8) |
There was a problem hiding this comment.
🔴 This script is symlinked to by dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh too, so the new ATTENTION_ARGS block also fires for those arms, not just GB300. GPU_MEM_MIB>=150000 is true on B200 (180GB) and GB200 (256GB) as well as GB300, but only the gb300-vllm-agentic-dspark image was bumped; b200 and gb200 still pin vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, the same build documented as lacking these SM100 sparse-indexer backends on H200. Passing --attention-config with FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 and --kv-cache-dtype fp8 to that older image will likely fail vllm serve startup, breaking the b200 and gb200 agentic-coding benchmark runs. …
Why this was flagged
…Fix: gate ATTENTION_ARGS on the image/nightly actually shipping this feature (e.g. per-config-key or image-tag check), not solely on GPU_MEM_MIB, and add matching perf-changelog entries if those arms' behavior does change too.
dsv41flash_fp4_b200_vllm_mtp.sh and dsv41flash_fp4_gb200_vllm_mtp.sh are symlinks to this file (benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh), confirmed via ls -la showing both -> dsv41flash_fp4_vllm_mtp.sh. Their configs (configs/nvidia-master.yaml lines ~8092 and ~8155, dsv41flash-fp4-gb200-vllm-agentic-dspark and dsv41flash-fp4-b200-vllm-agentic-dspark) still use image vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, unchanged by this diff. Both are Blackwell so GPU_MEM_MIB>=150000 at line 84 is true, so the new --attention-config/--kv-cache-dtype fp8 args at line 90 get added to their vllm serve invocation. The PR/changelog only pins and documents this feature for dsv41flash-fp4-gb300-vllm-agentic-dspark on the new nightly-0961bbae image; the older shared nightly is the same one the script's own comment says lacks SM100…
Verification: normal. The new block at benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh:83-91 is gated only on hardware memory (if (( GPU_MEM_MIB >= 150000 ))), and GPU_MEM_MIB comes from nvidia-smi --query-gpu=memory.total (line 59). The script's own comment at line 51 enumerates the Blackwell arms sharing this file — "B200 180 GB, GB200 256 GB, GB300 277 GB" — all far above 150000 MiB; the…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36084687434 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36084687434 |
2585e75 to
3494e65
Compare
更新 GB300 DeepSeek-V4.1-Flash vLLM nightly 配方，并在 TP2 和 TP4 使用 FlashInfer 稀疏注意力。
3494e65 to
b4fa176
Compare
|
InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest |
This branch has not been deployed
2 participants
Pin the GB300 DeepSeek-V4.1-Flash AgentX arm to vLLM nightly
0961bbae. Enable the merged Blackwell sparse indexer settings and select FlashInfer at TP2, MegaAttention at TP4. Preserve the existing topology and memory limits. Also pass the engine readiness timeout from the launcher where needed.The vLLM tracking issue and upstream recipe document the included changes. Local shell, YAML, and exact-key matrix checks pass; GPU sweep and eval are pending.
中文
将 GB300 DeepSeek-V4.1-Flash AgentX 配置固定到 vLLM nightly
0961bbae。启用已合并的 Blackwell 稀疏索引器参数，TP2 选择 FlashInfer，TP4 选择 MegaAttention；保留现有拓扑与显存限制，并由启动脚本传入所需的引擎就绪超时。相关改动见 vLLM 跟踪问题 与上游配方。本地 shell、YAML 和指定配置的矩阵检查通过；GPU sweep 与 eval 待运行。