Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Live 2P/1D DPA validation update:\n\n- Slurm 39443 completed all 64 model shards and CUDA-graph capture on both prefills and the decode.\n- Startup then failed deterministically in every logical worker because each ATOM data-parallel EngineCore tried to bind the same KV-event ZMQ endpoint. One DP rank bound successfully; the others raised Address already in use from Infera BlockManager hook initialization, so the worker could not register.\n- This is an Infera multi-DP KV-event publisher bug, not Slurm, MI355X hardware, model loading, Mooncake, or the srt-slurm topology.\n- The upstream fix is open as AMD-AGI/Infera#121. It gives every DP rank a local PUB ingress and relays all rank streams through the one endpoint advertised for the logical worker, retaining complete KV-aware cache visibility.\n- Focused engine/router validation is green: 50 tests passed, including two simultaneous EngineCore publishers through one advertised endpoint; Ruff, pre-commit, and diff checks passed.\n- The current exact-head workflow was not cancelled. Its 1P/1D arm is still running naturally as Slurm 39448. I will pin the reviewed fix and launch a fresh exact-head validation only after this attempt closes. |
8f0a94a to
0eb21ea
Compare
61b4789 to
5e21bed
Compare
8581966 to
1d4522c
Compare
5e21bed to
4c4a61b
Compare
1d4522c to
6da0605
Compare
4c4a61b to
514a519
Compare
|
Claude finished @cquil11's task in 4m 33s —— View job Review complete
SummaryFound 1 blocking issue and 1 minor issue (posted as inline comments):
Everything else checked out: the changelog entry is correctly appended at the end of the file and both master-config and changelog were updated together; Note: the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Use the hardware-proven Mooncake TCP transport for the initial correctness sweep; validate the updated ROCm RDMA path separately before changing production transport." | ||
| - "Use the ATOM image's bundled native AToMesh binary directly, without Infera source overlays or Infera routing, and archive the complete runtime logs from the custom benchmark." | ||
| - "Run the current rocm/atom-dev nightly_202608251555 image at ATOM commit 4e0848dcdf94f3d4be0c1e3ce51db99bc93f8b8c, whose bundled native AToMesh and Mooncake implementation are used without downstream source patches." | ||
| - "Pin SemiAnalysisAI/srt-slurm PR #7 at 10a58d2ebb4d756c423424049c736b909b72e14f so this ATOM/AToMesh port shares the latest validated AMD orchestration base, including native-router data-parallel rank preservation and ATOM recipe flag normalization." |
There was a problem hiding this comment.
🔴 BLOCKING: This changelog entry claims srt-slurm PR #7 is pinned at 10a58d2ebb4d756c423424049c736b909b72e14f, but nothing in this PR (or the repo) actually pins that commit. The shared launcher runners/launch_mi355x-amds-srt.sh:8 still has SRT_SLURM_COMMIT="c87d7b34b009be920896126013ad6dc74c5a99d5", which was set by the parent PR (#2542) for the SGLang-native lanes, and this PR does not modify the launcher.
Why it matters: The two new recipes depend on srt-slurm PR7 features (backend: type: atom, frontend: type: atomesh, connector: mooncake, mooncake_protocol, atom_config). If c87d7b34 predates PR7's native ATOM/AToMesh support, srtctl apply will fail schema validation for both topology jobs and the sweep can never pass. Note the PR body also cites a third SHA (76e7d76961…) as the validated PR7 head, so the intended pin is ambiguous — right now three different commits are referenced and only c87d7b34 is effective.
Fix: Bump SRT_SLURM_COMMIT in runners/launch_mi355x-amds-srt.sh to the exact PR7 head you validated against (and make the changelog text match that SHA), or if c87d7b34 genuinely contains the ATOM/AToMesh support, correct this changelog line to the real pin. Fix this →
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33665959215 |
769b6d8 to
df1481b
Compare
|
Replacement native full-sweep validation: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33717234735 Head df1481b, shared base #2542 at 0f390f6, srt-slurm PR #7 pinned to 527525d43ce2c1d33aa1d429aae0d8f338637b49. Synced with upstream main 6829ed5; the ATOM recipe YAML and AMD matrix are byte-for-byte unchanged from the previous attempt. This corrects an orchestrator defect: native ATOMesh was mistakenly sent through Dynamo NATS/etcd startup and failed before any engine worker started. No engine/image/transport/workload change. The new regression test covers ATOMesh, SGLang, and Dynamo infrastructure selection. Existing validated #2555 and #2630 heads remain unchanged. Expected validation: 1P1D c4,8,16,32,64,128; 2P1D c256,512,768,1024,2048; both native GSM8K evaluations. Results will only be staged after the full native run and its artifacts are checked. |
df1481b to
d3b8c38
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33717234735 |
|
Current native full sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33720270808 Head d3b8c38; shared base #2542 at 01676c3; runtime PR #7 at 527525d43ce2c1d33aa1d429aae0d8f338637b49. The engine recipes and AMD matrix are unchanged from the preceding attempt. The previous run got past NATS/etcd startup but hit the workflow-to-container Python cache-prefix leak. Both 1P1D jobs failed before weight loading; the two still-queued 2P1D jobs were canceled, without touching unrelated jobs. The exact-image CPU-only A/B probe (Slurm 42144, 1 CPU, zero GPUs) reproduced the import failure with the inherited prefix and passed with it absent. The fix is in the shared launcher, not the model settings. This new run was triggered by full-sweep-enabled after the new head passed its standalone changelog gate. It must validate all 11 throughput points and both native accuracy evaluations before results are staged. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33720270808 |
d3b8c38 to
1fb90fe
Compare
|
Resuming ATOM only. Fresh full-sweep-enabled validation: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33782782708 on 1fb90fe. Runtime fa994733 fixes the confirmed evaluation model-name mismatch; shared base 38be99b restores legacy unlimited RDMA memlock. No engine options, images, transport, concurrency lists, benchmark script, or accuracy thresholds changed. Prior run 33720270808 produced five successful 2P1D throughput points, but 1P1D throughput and both evals failed, so nothing from it is staged. The GPU/NIC registration probe 42195 passed all eight devices; memlock A/B probe 42198 is queued. 420 focused local runtime tests pass. Need all 11 throughput points and both evaluations green before staging. MiniMax M3 FP4 vLLM remains next, after ATOM completes. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33782782708 |
|
Validation update for run 33782782708, head 1fb90fe / runtime fa994733:
I restored the exact old commit under immutable-reference tag https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33782782708 No engine/workload config or scheduler bin-packing policy was changed. The later single-entrypoint cleanup still requires its own full native PR sweep after this baseline run. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33782782708 |
|
Fresh native full PR sweep on the unmodified official ATOM nightly and clean single-entrypoint launcher: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33825304569 (head fe8e9f9, runtime 81d46274f508e18ab14d1f123b75132005818dcf). No patches, custom images, overlays, or setup hooks. Includes all 11 throughput points and both complete GSM8K evals. The previous run is failed because its old native ATOM API dropped the requested token limit; its successful throughput and 1P1D accuracy results are preserved, not being reused as a substitute for this fresh full run. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fe8e9f9. Configure here.
| echo "[clear_caches] ${base_url}: L3 clear http=${code:-000} (optional backend unavailable)" | ||
| fi | ||
| done | ||
| } |
There was a problem hiding this comment.
Cache flush hits wrong worker APIs
Medium Severity
The new flush posts SGLang /flush_cache to every AIPERF_SERVER_METRICS_URLS target, a list already used for vLLM/TRT drain scraping. Failed flushes retry for FLUSH_DRAIN_TIMEOUT per worker, so non-SGLang agentic jobs stall, and SGLang recipes that never set that list skip the reset and reuse prefix cache across concurrencies.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit fe8e9f9. Configure here.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33825304569 |
|
/stage-results 33825304569 |
|
@cquil11 staged run 33825304569: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-04~r33825304569 This run remains available across future |
0d7ca6f to
6d439d6
Compare
fe8e9f9 to
1538638
Compare
1 participant
Scope
Port MI355X archived DeepSeek V4 ATOM/AToMesh to the shared AMD-enabled srt-slurm integration.
engineandroles) and preserve their engine, image, topology, and workload settings.benchmark_serving.pycustom-benchmark interface; no engine patches.CONC_LIST, replacing placeholder[1]rows and private recipe sweep lists where present. This preserves the existing measured points while matching current result collection.This workload is deprecated upstream and remains in
configs/deprecated/; it is not restored to active sweeps. Retained for the previously requested runtime-path coverage.Stack
Based directly on shared AMD integration #2542, which is rebased onto InferenceX
mainat96bccfe12b637dad23d32f0bf7ceef0ea86de15a. This child contains only its workload configs, schema-2 recipes, and append-only performance changelog entry. Shared launcher/profile/adapter changes live in #2542.The base selects SemiAnalysisAI/srt-slurm
03863ab23804f8a31811d9aaccc54abc8961a41c: #1 → #11 → #7 → #14, synchronized with NVIDIA srt-slurm85086d3c1fe7b319a1d0d7db8781047dd1940772. NVIDIA runners retain their existing runtime selection.Refresh status
Offline matrix validation: 2 topology/workload rows covering 11 benchmark concurrency points; all 4 prepared throughput/eval configurations pass the pinned runtime's schema/topology checks.
No fresh E2E tests or sweeps were started or rerun. Sweep-trigger labels are removed. Historical runtime evidence is listed in the base PR and does not validate these new heads. Full GPU validation remains pending authorization; this refresh makes no new performance or merge-readiness claim.
AI model disclosure
The exact AI model/version for this refresh is not exposed by the runtime and could not be verified. No subagents contributed to this refresh; earlier model identities are not reliably recorded.