fix(agentx): adopt additive AIPerf main warmup - #2415
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @cquil11's task in 4m 6s —— View job Review of PR #2415
SummaryThe core change is sound: One 🟡 warning posted inline (benchmarks/benchmark_lib.sh:1745): the removed Notes on verification scope: this environment couldn't reach the No blocking issues found. |
cquil11
added
agentx-fast
ChatGPT Codex Connector
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30495403097 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30495544822 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30495544822 |
786f856 to
9543505
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30502957473 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30505223590 |
9543505 to
aac2334
Compare
cquil11
removed
sweep-enabled
agentx-fast
cquil11
changed the title
Pin the shared AIPerf submodule to the synchronized main revision with additive per-lane warmup after mandatory primers while retaining timing-faithful spread phase starts. Align the workflow documentation with the request-count fast preset. 中文：将共享 AIPerf 子模块固定到已同步的 main 版本，在必需预热请求之后执行额外的逐 lane 预热，并保留符合轨迹时序的分散阶段启动。同步更新工作流文档，使其准确描述基于请求数量的快速预设。
aac2334 to
3f788b0
Compare
Pin the shared AIPerf submodule to the synchronized main revision with additive per-lane warmup after mandatory primers while retaining timing-faithful spread phase starts. Align the workflow documentation with the request-count fast preset. 中文：将共享 AIPerf 子模块固定到已同步的 main 版本，在必需预热请求之后执行额外的逐 lane 预热，并保留符合轨迹时序的分散阶段启动。同步更新工作流文档，使其准确描述基于请求数量的快速预设。
…armup Run 30511545046 tried lazy alone and is uninterpretable. It landed just after #2415 swapped the agentic warmup from --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 (benchmark_lib.sh:1793, plus a new utils/aiperf pin), so it moved two variables at once. Its "0 successful / 49 total, OSL=1" is the new warmup's one-token lane primers, not a lazy-offload symptom -- the cell never left warmup. That also invalidates the seven earlier fp8+offload failures as baselines: all of them ran the old time-based warmup. So run the eager control at c8 on the SAME warmup as lazy, in one dispatch, and compare those two only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…armup Run 30511545046 tried lazy alone and is uninterpretable. It landed just after #2415 swapped the agentic warmup from --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 (benchmark_lib.sh:1793, plus a new utils/aiperf pin), so it moved two variables at once. Its "0 successful / 49 total, OSL=1" is the new warmup's one-token lane primers, not a lazy-offload symptom -- the cell never left warmup. That also invalidates the seven earlier fp8+offload failures as baselines: all of them ran the old time-based warmup. So run the eager control at c8 on the SAME warmup as lazy, in one dispatch, and compare those two only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reference run 30163138459 (v0.5.14, 2026-07-25) is no longer comparable to anything, because the aiperf load model changed underneath us. Upstream SemiAnalysisAI#2415 (ddeb02e, 2026-07-30) dropped --agentic-cache-warmup-duration 600 for --warmup-requests-per-lane 10, and SemiAnalysisAI#2457 (ce8a34e, 2026-08-03) added --trace-idle-gap-cap-seconds 300, which makes the replay honour the trace's recorded think-time gaps instead of firing back-to-back. Both landed after that run and reached this fork through the routine main merge 105fa44. This is not a local regression. No config key here overrides an AIPERF_* timing variable; the defaults are upstream hard-codes at benchmark_lib.sh:1810 and :2123; our benchmark_lib.sh delta is remote-bench plumbing only (API-key redaction, REMOTE_RESET_URL, TLS bypass, cancel trap) and touches no timing line; and the aiperf submodule moved 0d2aa057 -> 754356e9 with no local pin. Measured size of the distortion, server-side, reference against run 32585334238 at the same nominal c24: mean #running-req 3.36 -> 2.27, warmup-dropped records 1496 -> 264, total requests 4241 -> 2694. Aggregate throughput therefore reads low and latency reads good on the newer run purely because the server idles more of the hour. Any agg_bmk.json comparison across 2026-08-03 is invalid. glm5.2v0514_fp4_h200_sglang.sh is a byte-identical copy of glm5.2_fp4_h200_sglang.sh, pinned so later edits to that file cannot silently redefine this arm -- in particular the pending addition of --dsa-prefill-backend flashmla_sparse_q8, which the baseline needs on v0.5.18 and which would defeat this arm's purpose. Verified with git diff against the reference run's sha 70a7911: the file matches what that run executed. The arm is safe on v0.5.14 without that flag, and omits it deliberately. Crash #1 (num_splits must have shape (b+1)) needs a prefill CUDA graph captured at runtime, and v0.5.14 defaults prefill to backend='disabled' where v0.5.18 defaults it to 'breakable'. Neither run passes --cuda-graph-backend-prefill, so that is an upstream default flip, and crash #1 is a v0.5.18 regression rather than a latent bug. The reference run confirms the config survives a full 3600s. conc-list is [16, 24] to match the reference; c8 is omitted because the reference has no c8 point to compare against. Read this against glm5.2dplm (run 32585334238), not against the v0.5.18 baseline arm, which has never produced a result. That pair is a five-variable bundle -- HiCache #30393, the prefill-graph default flip, flashmla_sparse_q8, --enable-dp-lm-head, --enable-dp-attention-local-control-broadcast -- and is the honest end-to-end upgrade delta, not an isolation of any one change. The isolated evidence for #30393 is already in hand from accept metrics: collapsed decode batches (accept rate < 0.15) fall from 3.28% at c16 and 2.02% at c24 to 0.00% at every concurrency, mean accept length 2.60 -> 3.04, accept rate 0.53 -> 0.68, and v0.5.14's accept length degrades with batch size (2.81 at 2 running requests down to 2.31 at 5) where v0.5.18 holds flat at 3.03-3.07. Verified: bash -n clean, runners/test_launch_h200_greennode.py passes at 21 benchmarks, inferencex-agentic-dispatch preflight PRECHECK PASSED with 2 matrix rows on h200-greennode_03. 参照运行 30163138459（v0.5.14，2026-07-25）已无法与任何运行比较，因为 aiperf 的负载 模型在我们脚下发生了变化。上游 SemiAnalysisAI#2415（ddeb02eb，2026-07-30）以 --warmup-requests-per-lane 10 取代了 --agentic-cache-warmup-duration 600，SemiAnalysisAI#2457 （ce8a34e4，2026-08-03）新增了 --trace-idle-gap-cap-seconds 300，使回放遵循 trace 中 记录的真实思考间隔，而非连续发送。两者均在该运行之后合入，并通过常规的 main 合并 105fa44 进入本 fork。 这并非本地引入的回归。此处没有任何 config key 覆盖 AIPERF_* 计时变量；默认值是上游在 benchmark_lib.sh:1810 与 :2123 的硬编码；我们对 benchmark_lib.sh 的改动仅为 remote-bench 管道（API key 脱敏、REMOTE_RESET_URL、TLS 跳过、取消陷阱），未触及任何 计时行；aiperf 子模块由 0d2aa057 变为 754356e9，且无本地 pin。 失真幅度的服务端实测，参照运行与运行 32585334238 在同一标称 c24 下对比：平均 #running-req 3.36 -> 2.27，warmup 丢弃记录 1496 -> 264，总请求数 4241 -> 2694。因此 较新运行的聚合吞吐偏低、延迟偏好，纯粹是因为服务器在这一小时中空闲更多。任何跨越 2026-08-03 的 agg_bmk.json 比较均无效。 glm5.2v0514_fp4_h200_sglang.sh 是 glm5.2_fp4_h200_sglang.sh 的逐字节副本，固定下来以 免后续对该文件的修改悄然重定义本臂 —— 尤其是待加入的 --dsa-prefill-backend flashmla_sparse_q8，它在 v0.5.18 上为 baseline 所必需，却会使本臂 失去意义。已用 git diff 对照参照运行的 sha 70a7911 验证：本文件与该运行执行的内容一致。 本臂在 v0.5.14 上无需该开关即可安全运行，且系刻意省略。崩溃 #1 （num_splits must have shape (b+1)）需要在运行期捕获 prefill CUDA graph，而 v0.5.14 的 prefill 默认为 backend='disabled'，v0.5.18 则默认为 'breakable'。两次运行均未传入 --cuda-graph-backend-prefill，故这是上游默认值翻转，崩溃 #1 是 v0.5.18 的回归而非潜伏 缺陷。参照运行已证明该配置可完整跑满 3600 秒。 conc-list 取 [16, 24] 以对齐参照运行；省略 c8，因参照运行没有可比的 c8 数据点。 请将本臂与 glm5.2dplm（运行 32585334238）对读，而非与从未产出结果的 v0.5.18 baseline 臂对读。该配对是五变量的组合 —— HiCache #30393、prefill graph 默认值翻转、 flashmla_sparse_q8、--enable-dp-lm-head、 --enable-dp-attention-local-control-broadcast —— 它是诚实的端到端升级差值，而非对其中 任一项的隔离测量。#30393 的隔离证据已由 accept 指标给出：坍塌的 decode batch （accept rate < 0.15）由 c16 的 3.28%、c24 的 2.02% 降至各并发下的 0.00%，平均 accept length 2.60 -> 3.04，accept rate 0.53 -> 0.68；且 v0.5.14 的 accept length 随 batch 增大而退化（2 个 running request 时 2.81，降至 5 个时 2.31），而 v0.5.18 稳定在 3.03-3.07。 已验证：bash -n 通过，runners/test_launch_h200_greennode.py 在 21 个基准下通过， inferencex-agentic-dispatch preflight 在 h200-greennode_03 上 PRECHECK PASSED， 生成 2 个 matrix 行。
1 participant
Summary
be758d62on synchronizedSemiAnalysisAI/aiperfmain.agentx-fast, use one additional warmup request per lane and a 20-minute profile.--burst-phase-starts.The 10-second cap applies only to whole-system idle periods. Recorded per-trace end-to-start delays remain intact.
Validation
python -m pytest utils/matrix_logic/ utils/test_process_changelog.py -q— 238 passedThe workflow's unrelated full SWE-bench eval was externally canceled; all AgentX throughput jobs completed successfully.