Skip to content

Qualify B200 DeepSeek-V4.1-Flash on vLLM nightly 0961bbae / 在 B200 上验证 DeepSeek-V4.1-Flash 的 vLLM nightly 0961bbae #3393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xinli-sw wants to merge 1 commit into from
+23 −1
Closed

Qualify B200 DeepSeek-V4.1-Flash on vLLM nightly 0961bbae / 在 B200 上验证 DeepSeek-V4.1-Flash 的 vLLM nightly 0961bbae #3393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,15 @@ if (( TP == 2 || (TP < 8 && GPU_MEM_MIB < 150000) )); then
LOW_TP_ARGS=(--max-num-batched-tokens 4096 --max-num-seqs "$MAX_NUM_SEQS")
fi

# The SKU launcher opts into the Blackwell sparse indexer settings only
# with an image that contains these backends. This script also serves H200.
# TP2 and TP4 use FlashInfer sparse attention; fp8 resolves the backend KV layout.
ATTENTION_ARGS=()
if [[ "${DSV41_BLACKWELL_ATTENTION:-}" == true ]]; then
ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8)
fi
Comment on lines +83 to +87

@claude claude Bot Sep 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 BLOCKING: This gate also fires on GB200 and GB300, which are not part of this PR.

Why it matters: dsv41flash_fp4_gb200_vllm_mtp.sh, dsv41flash_fp4_gb300_vllm_mtp.sh and dsv41flash_fp4_h200_vllm_mtp.sh are symlinks to this file. The GB200 arm (256 GiB per GPU, nightly-cd10ed6f, 2026-09-15) and the GB300 arm (277 GiB per GPU, nightly-af1c0149, 2026-09-16) both clear GPU_MEM_MIB >= 150000, and both run TP2 and TP4, so every one of their points now gets --attention-config {...DSV41...,"indexer_sparse_logits":true} --kv-cache-dtype fp8. The PR description says these settings need the 2026-09-22 nightly; on the older images an unknown backend name or indexer_sparse_logits key would fail at argument parsing, and even if they happen to parse, the GB200/GB300 recipes change without a perf-changelog.yaml entry (only dsv41flash-fp4-b200-vllm-agentic-dspark is listed). I could not confirm the flag availability at cd10ed6f/af1c0149 from this sandbox (no network, source MCP server timed out), so please treat the crash as unverified but the scope leak as certain.

Fix: Gate on a launcher-exported variable, the same way this script already handles VLLM_SAFETENSORS_LOAD_STRATEGY, and export it only from runners/launch_b200-nscale-slurm.sh next to the new VLLM_ENGINE_READY_TIMEOUT_S line:

Suggested change
ATTENTION_ARGS=()
if (( GPU_MEM_MIB >= 150000 )); then
if (( TP == 2 )); then
ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
else
ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
fi
ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8)
fi
ATTENTION_ARGS=()
if [[ "${DSV41_BLACKWELL_SPARSE_INDEXER:-0}" == "1" ]]; then
if (( TP == 2 )); then
ATTENTION_CONFIG='{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
else
ATTENTION_CONFIG='{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
fi
ATTENTION_ARGS=(--attention-config "$ATTENTION_CONFIG" --kv-cache-dtype fp8)
fi

and in runners/launch_b200-nscale-slurm.sh inside the MODEL_PREFIX == "dsv41flash" block:

export DSV41_BLACKWELL_SPARSE_INDEXER=1

(Update the comment on lines 80-82 accordingly; it currently only mentions excluding H200.) If the intent really is to roll these settings out to GB200/GB300 as well, those images need bumping to a nightly that carries them and the changelog needs entries for both config keys.

Fix this →

Comment on lines +83 to +87

@claude claude Bot Sep 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 GB200/GB300 agentic runs pick up the new sparse-indexer attention flags via this shared script, even though those images weren't bumped for it. The GPU_MEM_MIB>=150000 gate at line 84 also fires for gb200-nv (256GB) and gb300-nv (277GB), since dsv41flash_fp4_gb200_vllm_mtp.sh and dsv41flash_fp4_gb300_vllm_mtp.sh are symlinks to this file. But configs/nvidia-master.yaml leaves their images unchanged: gb200 stays on nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3 (the exact build b200 is moving off in this PR) and gb300 on nightly-af1c01499b289be555c475669ba50a88e96d846e. If either predates the new FLASHINFER_MLA_SPARSE_DSV41/FLASHMLA_MEGA_ATTN_DSV41 backends, vllm serve rejects --attention-config on the next gb200/gb300 sweep. …

Why this was flagged

…Fix: gate ATTENTION_ARGS on the vLLM image/version that actually ships the backend, not on GPU memory alone, or bump gb200/gb300 images together with b200.

dsv41flash_fp4_gb200_vllm_mtp.sh and dsv41flash_fp4_gb300_vllm_mtp.sh are symlinks to dsv41flash_fp4_vllm_mtp.sh (verified via ls -la), so the new ATTENTION_ARGS block (lines 83-91) runs for them too. Its only gate is GPU_MEM_MIB>=150000 at line 84, true for GB200 (256GB) and GB300 (277GB) per the script's own comments at lines 50-51. configs/nvidia-master.yaml:8093 pins gb200-vllm-agentic-dspark to nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3, unbumped by this PR and identical to b200's pre-bump image; gb300 (line 8196) stays on nightly-af1c01499b289be555c475669ba50a88e96d846e, also untouched. On base, this script never passed --attention-config, so gb200/gb300 ran unaffected; now, if either nightly lacks the new backend keys, vllm serve fails to start for those SKUs on the next sweep.

Verification: Severity: normal. The gb200/gb300 agentic vLLM arms regress because the shared script now injects new attention flags onto their un-bumped images. Structural chain (all verified): - dsv41flash_fp4_gb200_vllm_mtp.sh and dsv41flash_fp4_gb300_vllm_mtp.sh are symlinks to dsv41flash_fp4_vllm_mtp.sh (ls -la: both "-> dsv41flash_fp4_vllm_mtp.sh"). So the new block added by the diff at lines 83-91…


# Pyxis shares the host network; port 8888 can already belong to a host service.
select_available_server_port
export AIPERF_SERVER_URL="http://localhost:${PORT}"
Expand All @@ -99,6 +108,7 @@ VLLM_CMD=(
--tool-call-parser deepseek_v41 --enable-auto-tool-choice
--reasoning-parser deepseek_v41
--engram-config '{"cpu_offload":true}'
"${ATTENTION_ARGS[@]}"
--speculative-config "$SPEC_CONFIG"
--max-model-len 1048576
--max-cudagraph-capture-size "$CAPTURE_SIZE"
Expand Down
2 changes: 1 addition & 1 deletion configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8433,7 +8433,7 @@ dsv41flash-fp4-b300-vllm-agentic-dspark:
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [2, 4, 8, 16, 32, 64, 128] }

dsv41flash-fp4-b200-vllm-agentic-dspark:
image: vllm/vllm-openai:nightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3
image: vllm/vllm-openai:nightly-0961bbae2894d574be790d219651824eb199318e
model: deepseek-ai/DeepSeek-V4.1-Flash
model-prefix: dsv41flash
runner: cluster:b200-nscale
Expand Down
8 changes: 8 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8887,3 +8887,11 @@
- "Add DeepSeek-V4-Pro-0813 golden AL for draft lengths 4, 5, 7 and 8 (3.36 / 3.61 / 3.73 / 3.47)."
- "Agentic PD router: pin --decode-policy round_robin so decode no longer inherits the prefill --policy (consistent_hashing)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3256

- config-keys:
- dsv41flash-fp4-b200-vllm-agentic-dspark
scenario-type:
- agentic-coding
description:
- "Pin B200 DeepSeek-V4.1-Flash vLLM to the 2026-09-22 nightly 0961bbae and opt in from its launcher to Blackwell sparse indexer logits, MXFP4 indexer KV, and fp8 KV cache; use FlashInfer sparse attention at TP2 and MegaAttention at TP4."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3393
4 changes: 4 additions & 0 deletions runners/launch_b200-nscale-slurm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -804,6 +804,10 @@ run_agentic() {
if [[ "$MODEL_PREFIX" == "dsv41flash" ]]; then
# Cover DSpark5 verification for concurrent AgentX subagents at c1/c2/c4.
export DSV41_MIN_CUDAGRAPH_CAPTURE_SIZE=64
export VLLM_ENGINE_READY_TIMEOUT_S=7200
if [[ "$FRAMEWORK" == "vllm" ]]; then
export DSV41_BLACKWELL_ATTENTION=true
fi
CONTAINER_MOUNT_DIR=/ix
export INFMAX_CONTAINER_WORKSPACE=/ix
export RESULT_DIR=/ix/results
Expand Down
Loading