Skip to content

feat(amd): move AMD multi-node fixed-sequence benchmarks to srt-slurm #2542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 6 commits into from
Sep 25, 2026
Merged

feat(amd): move AMD multi-node fixed-sequence benchmarks to srt-slurm #2542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
552462a
refactor: standardize srt-slurm cluster setup hooks
cquil11 Sep 23, 2026
0b57274
feat(amd): add disaggregated SGLang and vLLM to synthetic acceptance
cquil11 Sep 23, 2026
087aae5
feat(amd): run MI355X multi-node recipes through srt-slurm
cquil11 Sep 23, 2026
e836794
feat(amd): run MI355X multi-node fixed-sequence configs on srt-slurm
cquil11 Sep 23, 2026
97a721a
refactor(amd): remove legacy multi-node fixed-sequence orchestration
cquil11 Sep 23, 2026
41e8bfc
fix(amd): serve Qwen3.5 FP8 disagg from the full MI355X checkpoint
cquil11 Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,15 @@ check_env_vars IS_MULTINODE MODEL_NAME PRECISION
- Do not add launcher-name aliases to `runners/runtime_settings.sh` or elsewhere for scripts that no runner resolves to. A launcher without a pool is dead code; a pool without a launcher fails at job start.
- When a pool is retired, delete its launcher in the same PR rather than keeping it as a fallback for another pool.

## SRT Slurm cluster hooks

- Put reusable host-check functions in `runners/srt-slurm/hooks/common.sh`. Sourcing it must only define functions, without running checks, changing environment variables, or initializing benchmarks. Cluster-only helpers stay beside their setup script.
- Keep cluster-specific host prerequisites in `runners/srt-slurm/hooks/<cluster>/setup.sh`, invoked explicitly by the matching cluster profile's `default_host_setup`. These run after allocation, before services and workers start.
- Hooks are only for checks and setup required by that cluster's hosts or fabric. Keep them small, workload-independent, and safe to run repeatedly. Prefer native srt-slurm configuration whenever it can express the requirement.
- Do not put benchmark execution, model selection, engine flags, concurrency tuning, evaluation, result collection, or job orchestration in hooks. Those belong in recipes, benchmark scripts, or the existing orchestration layer.
- Do not use hooks to patch engines or containers, bypass failed checks, or hide runtime bugs behind retries and ad hoc workarounds. Fix problems in the component that owns them.
- Pass settings explicitly from the cluster profile. Scope mutations to the allocated nodes, preserve other jobs' resources, and register teardown for temporary state that needs restoring. See [cluster profiles](docs/configuration-procedures.md#cluster-profiles).

## SRT Slurm synthetic acceptance

- **Do not hard-code synthetic acceptance lengths in SRT recipes, master configs, or launchers.** InferenceX automatically selects the measured value from [`golden_al_distribution/`](golden_al_distribution/) for speculative AgentX throughput runs. Do not add manual `SYNTHETIC_ACCEPTANCE_LENGTH`, vLLM `synthetic_acceptance_length`, SGLang `SGLANG_SIMULATE_ACC_LEN`, or TRT-LLM `TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS` settings.
Expand Down
29 changes: 2 additions & 27 deletions benchmarks/benchmark_lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -558,33 +558,8 @@ _write_amd_smi_sidecar() {
fi
}

# Poll rocm-smi VRAM% every 10s for up to 15 min until the busiest GPU is at or
# below the threshold percent (default 10); return 1 otherwise so the caller
# aborts instead of starting on GPUs still draining the previous job.
# Pass a stricter threshold when the run sizes its KV cache from device-wide free
# memory (torch.cuda.mem_get_info): on 288 GB parts the 10% gate admits ~28.8 GB
# of residual, which the engine folds into non_torch and subtracts from the KV
# pool, so the pool drifts run to run.
wait_for_amd_gpu_clean() {
local threshold="${1:-10}"
local gpu_clean=false vram_max i
for i in $(seq 1 90); do
vram_max=$(rocm-smi --showmemuse 2>/dev/null \
| grep -oE "GPU Memory Allocated \(VRAM%\): [0-9]+" \
| awk '{if ($NF > m) m = $NF} END {print m+0}')
if [ "${vram_max:-0}" -le "$threshold" ]; then
echo "GPUs clean (vram%max=$vram_max <= $threshold after $((i * 10))s)"
gpu_clean=true
break
fi
echo "waiting for prior-job GPU memory reclaim: vram%max=$vram_max (target <= $threshold)"
sleep 10
done
if [ "$gpu_clean" != "true" ]; then
echo "Error: GPUs still draining prior job's memory after 15min" >&2
return 1
fi
}
# shellcheck source=runners/srt-slurm/hooks/common.sh
source "$(dirname "${BASH_SOURCE[0]}")/../runners/srt-slurm/hooks/common.sh" || return 1

# Return success only while a PID exists and is not a zombie waiting to be
# reaped. `kill -0` alone treats zombies as live processes.
Expand Down
105 changes: 0 additions & 105 deletions benchmarks/multi_node/amd_utils/bench.sh
View file
Open in desktop

This file was deleted.

77 changes: 4 additions & 73 deletions benchmarks/multi_node/amd_utils/env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
source "$(dirname "${BASH_SOURCE[0]}")/../../benchmark_lib.sh" --validation-only
check_env_vars ENGINE
# MoRI-IO queue-pair tuning, the UCX RoCE GID index, SGLang router logging and the
# SGLang decode cuda-graph NCCL workaround. Only the SGLang and vLLM MoRI KV paths
# below read these. ENGINE=tilert moves KV over mooncake and starts no SGLang
# SGLang decode cuda-graph NCCL workaround. Only the SGLang MoRI KV path
# below reads these. ENGINE=tilert moves KV over mooncake and starts no SGLang
# router, so it is neither given nor reads them: validating them there would force
# the recipe to invent MoRI tuning for a transport it never uses.
if [[ "$ENGINE" != "tilert" ]]; then
Expand All @@ -15,7 +15,7 @@ if [[ "$ENGINE" != "tilert" ]]; then
fi
# Dual-engine environment setup for multi-node disaggregated serving.
#
# ENGINE=sglang-disagg or vllm-disagg selects the engine-specific block.
# ENGINE=sglang-disagg or tilert selects the engine-specific block.
#
# REQUIRED ENVIRONMENT VARIABLES:
# IBDEVICES - RDMA/InfiniBand device names (e.g., ionic_0,ionic_1,... or mlx5_0,mlx5_1,...)
Expand Down Expand Up @@ -119,76 +119,7 @@ else
fi
fi

if [[ "$ENGINE" == "vllm-disagg" ]]; then
export VLLM_USE_V1=1
export VLLM_SERVER_DEV_MODE=0
export VLLM_DISABLE_REQUEST_ID_RANDOMIZATION=1

set -x

# UCX_NET_DEVICES: Use the first tw-eth interface for UCX TCP transport
if [[ -z "$UCX_NET_DEVICES" ]]; then
UCX_NET_DEV=$(ip -o link show 2>/dev/null | awk -F': ' '/tw-eth/{print $2}' | head -1)
if [[ -n "$UCX_NET_DEV" ]]; then
export UCX_NET_DEVICES="$UCX_NET_DEV"
else
FIRST_IB=$(echo "$IBDEVICES" | cut -d',' -f1)
if [[ -n "$FIRST_IB" ]]; then
export UCX_NET_DEVICES="${FIRST_IB}:1"
fi
fi
echo "[INFO] Auto-set UCX_NET_DEVICES=$UCX_NET_DEVICES"
else
echo "[INFO] Using UCX_NET_DEVICES=$UCX_NET_DEVICES (set by environment)"
fi

# RoCEv2: use IPv4-mapped GID (index 1) for inter-node RDMA routing
export UCX_IB_GID_INDEX

if [[ -n "$UCX_IB_TRAFFIC_CLASS" ]]; then
echo "[INFO] Using UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS (set by environment)"
elif command -v nicctl &> /dev/null; then
ND_PRIO=$(nicctl show qos 2>/dev/null | awk '/PFC no-drop priorities/ {print $NF; exit}')
ND_DSCP=$(nicctl show qos 2>/dev/null | awk -v p="$ND_PRIO" '
$1 == "DSCP" && $2 == ":" && $NF == p {
print $3; exit
}')
# nicctl may emit trailing commas (e.g. "24,"); keep the leading integer so the
# arithmetic can't choke and unparseable output falls back to hostname detection.
ND_PRIO="${ND_PRIO%%,*}"; ND_PRIO="${ND_PRIO//[!0-9]/}"
ND_DSCP="${ND_DSCP%%,*}"; ND_DSCP="${ND_DSCP//[!0-9]/}"
if [[ "$ND_DSCP" =~ ^[0-9]+$ ]] && [[ "$ND_PRIO" =~ ^[0-9]+$ ]]; then
export UCX_IB_TRAFFIC_CLASS=$(( 4 * ND_DSCP ))
export UCX_IB_SL=$ND_PRIO
echo "[INFO] Detected QoS from nicctl: UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS, UCX_IB_SL=$UCX_IB_SL"
else
echo "[WARN] nicctl available but QoS data unavailable; trying hostname detection."
NODENAME=$(hostname -s)
if [[ $NODENAME == GPU* ]] || [[ $NODENAME == smci355-ccs-aus* ]]; then
export UCX_IB_TRAFFIC_CLASS=96
echo "[INFO] Auto-detected UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS from hostname $NODENAME"
elif [[ $NODENAME == mia1* ]]; then
export UCX_IB_TRAFFIC_CLASS=104
echo "[INFO] Auto-detected UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS from hostname $NODENAME"
fi
fi
else
NODENAME=$(hostname -s)
if [[ $NODENAME == GPU* ]] || [[ $NODENAME == smci355-ccs-aus* ]]; then
export UCX_IB_TRAFFIC_CLASS=96
echo "[INFO] Auto-detected UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS from hostname $NODENAME"
elif [[ $NODENAME == mia1* ]]; then
export UCX_IB_TRAFFIC_CLASS=104
echo "[INFO] Auto-detected UCX_IB_TRAFFIC_CLASS=$UCX_IB_TRAFFIC_CLASS from hostname $NODENAME"
else
echo "[INFO] No nicctl and unable to detect from hostname. Skipping QoS configuration."
fi
fi

set +x
echo "[INFO] IBDEVICES=$IBDEVICES UCX_NET_DEVICES=$UCX_NET_DEVICES NCCL_SOCKET_IFNAME=$NCCL_SOCKET_IFNAME UCX_IB_GID_INDEX=$UCX_IB_GID_INDEX UCX_IB_TRAFFIC_CLASS=${UCX_IB_TRAFFIC_CLASS:-unset}"

elif [[ "$ENGINE" == "tilert" ]]; then
if [[ "$ENGINE" == "tilert" ]]; then
echo "[INFO] tilert: IBDEVICES=$IBDEVICES NCCL_SOCKET_IFNAME=$NCCL_SOCKET_IFNAME NCCL_IB_HCA=$NCCL_IB_HCA"

else
Expand Down
40 changes: 0 additions & 40 deletions benchmarks/multi_node/amd_utils/env_atom.sh
View file
Open in desktop

This file was deleted.

Loading
Loading