[Benchmark] Add DFlash vs baseline accuracy/perf sweep scripts - #13
Open
TianHao65 wants to merge 1 commit into
Open
[Benchmark] Add DFlash vs baseline accuracy/perf sweep scripts#13TianHao65 wants to merge 1 commit into
TianHao65 wants to merge 1 commit into
Conversation
Add five standalone benchmark drivers under benchmark/dflash/ that launch
servers across (attention_backend, tp_size) configs and run a workload for
each (concurrency, num_questions) setting, reporting DFlash throughput and
acceptance length against a baseline run.
- bench_dflash_gsm8k_sweep.py GSM8K
- bench_dflash_math500_sweep.py MATH500, scored by \boxed{...} string
equivalence (conservative lower bound)
- bench_dflash_humaneval_sweep.py HumanEval pass@1, sandboxed subprocess
- bench_dflash_mbpp_sweep.py MBPP pass@1, sandboxed subprocess
- bench_dflash_mtbench_sweep.py MT-Bench, performance only (no judge)
For the code benchmarks, test execution runs after the timed generation
region so it never contaminates throughput or accept-length numbers.
Launch environment matches launch_qwen3.5-397B-fp8_tp8_prefix_cache_DFlash.sh
and is applied via setdefault before importing torch/sglang, so explicit
caller exports still win.
These are benchmark scripts, not CI tests -- they are long-running by design.
Owner
|
No CI required since this PR is only for benchmarks. I have manually canceled the run. |
There was a problem hiding this comment.
Pull request overview
This PR adds standalone benchmark drivers under benchmark/dflash/ to sweep DFLASH vs baseline across multiple (attention_backend, tp_size) server configs and (concurrency, num_questions) workloads, reporting throughput and speculative acceptance-length; the code-eval benchmarks also compute pass@1 in a post-timing phase.
Changes:
- Added GSM8K and MATH500 sweep scripts that measure throughput/accept-length and compute basic string-based accuracy.
- Added HumanEval and MBPP sweep scripts that measure throughput/accept-length and compute pass@1 via sandboxed subprocess execution after timing.
- Added an MT-Bench sweep script that measures performance only (no judge), using first-turn prompts as single-turn requests.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| benchmark/dflash/bench_dflash_gsm8k_sweep.py | GSM8K DFLASH vs baseline sweep with throughput + accept-length + numeric-answer accuracy. |
| benchmark/dflash/bench_dflash_math500_sweep.py | MATH500 DFLASH vs baseline sweep with throughput + accept-length + boxed-answer string equivalence accuracy. |
| benchmark/dflash/bench_dflash_humaneval_sweep.py | HumanEval DFLASH vs baseline sweep with throughput + accept-length + pass@1 via subprocess test execution (post-timing). |
| benchmark/dflash/bench_dflash_mbpp_sweep.py | MBPP DFLASH vs baseline sweep with throughput + accept-length + pass@1 via subprocess assertion execution (post-timing). |
| benchmark/dflash/bench_dflash_mtbench_sweep.py | MT-Bench DFLASH vs baseline sweep focused on performance only (no judge). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+499
to
+508
| for conc in concurrencies: | ||
| n = num_questions_by_conc[conc] | ||
| _flush_cache(base_url) | ||
| print( | ||
| f"[warmup] run 1 warmup batch (size={conc}) after /flush_cache; excluded from metrics." | ||
| ) | ||
| metrics = _run_gsm8k_requests( | ||
| base_url, | ||
| prompts=prompts[: n + conc], | ||
| labels=labels[: n + conc], |
Comment on lines
+381
to
+385
| "--disable-custom-all-reduce", | ||
| "--kv-cache-dtype", | ||
| "fp8_e4m3", | ||
| "--page-size", | ||
| str(int(args.page_size) if args.page_size is not None else 64), |
Comment on lines
+557
to
+566
| for conc in concurrencies: | ||
| n = num_questions_by_conc[conc] | ||
| _flush_cache(base_url) | ||
| print( | ||
| f"[warmup] run 1 warmup batch (size={conc}) after /flush_cache; excluded from metrics." | ||
| ) | ||
| metrics = _run_requests( | ||
| base_url, | ||
| prompts=prompts[: n + conc], | ||
| labels=labels[: n + conc], |
Comment on lines
+432
to
+436
| "--disable-custom-all-reduce", | ||
| "--kv-cache-dtype", | ||
| "fp8_e4m3", | ||
| "--page-size", | ||
| str(int(args.page_size) if args.page_size is not None else 64), |
Comment on lines
+573
to
+582
| for conc in concurrencies: | ||
| n = num_questions_by_conc[conc] | ||
| _flush_cache(base_url) | ||
| print( | ||
| f"[warmup] run 1 warmup batch (size={conc}) after /flush_cache; excluded from metrics." | ||
| ) | ||
| metrics = _run_requests( | ||
| base_url, | ||
| prompts=prompts[: n + conc], | ||
| tasks=tasks[: n + conc], |
Comment on lines
+448
to
+452
| "--disable-custom-all-reduce", | ||
| "--kv-cache-dtype", | ||
| "fp8_e4m3", | ||
| "--page-size", | ||
| str(int(args.page_size) if args.page_size is not None else 64), |
Comment on lines
+589
to
+598
| for conc in concurrencies: | ||
| n = num_questions_by_conc[conc] | ||
| _flush_cache(base_url) | ||
| print( | ||
| f"[warmup] run 1 warmup batch (size={conc}) after /flush_cache; excluded from metrics." | ||
| ) | ||
| metrics = _run_requests( | ||
| base_url, | ||
| prompts=prompts[: n + conc], | ||
| tasks=tasks[: n + conc], |
Comment on lines
+464
to
+468
| "--disable-custom-all-reduce", | ||
| "--kv-cache-dtype", | ||
| "fp8_e4m3", | ||
| "--page-size", | ||
| str(int(args.page_size) if args.page_size is not None else 64), |
Comment on lines
+460
to
+468
| for conc in concurrencies: | ||
| n = num_questions_by_conc[conc] | ||
| _flush_cache(base_url) | ||
| print( | ||
| f"[warmup] run 1 warmup batch (size={conc}) after /flush_cache; excluded from metrics." | ||
| ) | ||
| metrics = _run_requests( | ||
| base_url, | ||
| prompts=prompts[: n + conc], |
Comment on lines
+336
to
+340
| "--disable-custom-all-reduce", | ||
| "--kv-cache-dtype", | ||
| "fp8_e4m3", | ||
| "--page-size", | ||
| str(int(args.page_size) if args.page_size is not None else 64), |
sammysun0711
requested changes
Aug 7, 2026
sammysun0711
left a comment
sammysun0711
left a comment
Collaborator
There was a problem hiding this comment.
@TianHao65, please help to check coplot review comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add five standalone benchmark drivers under benchmark/dflash/ that launch servers across (attention_backend, tp_size) configs and run a workload for each (concurrency, num_questions) setting, reporting DFlash throughput and acceptance length against a baseline run.
equivalence (conservative lower bound)
For the code benchmarks, test execution runs after the timed generation region so it never contaminates throughput or accept-length numbers.
Launch environment matches launch_qwen3.5-397B-fp8_tp8_prefix_cache_DFlash.sh and is applied via setdefault before importing torch/sglang, so explicit caller exports still win.
These are benchmark scripts, not CI tests -- they are long-running by design.
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ci