diff --git a/.claude/commands/add-model-hardware.md b/.claude/commands/add-model-hardware.md index 0100361404..02c1efc043 100644 --- a/.claude/commands/add-model-hardware.md +++ b/.claude/commands/add-model-hardware.md @@ -1,5 +1,5 @@ --- -description: Add a new model+hardware single-node benchmark recipe (script + master-config entry + perf-changelog + launcher routing), open a [Klaud Cold] PR, label full-sweep-fail-fast, and monitor CI +description: Add a new model+hardware single-node benchmark recipe (srt-slurm recipe + master-config entry + perf-changelog), open a [Klaud Cold] PR, label full-sweep-fail-fast, and monitor CI argument-hint: [recipes-link] [draft-model-link] [mtp] --- @@ -40,45 +40,43 @@ Check `MODELS.md` before choosing a model, scenario, or precision. Do not reintr **A. In-codebase research (primary because this repo is the source of truth):** ```bash -# similar benchmark scripts: same model on other SKUs, AND same SKU on other models -ls benchmarks/single_node/fixed_seq_len/_*.sh benchmarks/single_node/fixed_seq_len/*_*.sh +# similar srt-slurm recipes: same model on other SKUs, AND same SKU on other models +ls -d benchmarks/single_node/srt-slurm-recipes//*/* benchmarks/single_node/srt-slurm-recipes/*/*/-* # similar master-config entries (search spaces, image, parallelism), this model + analogues grep -nE "-|.*--" configs/{nvidia,amd}-master.yaml -# the runner launcher for this SKU (script-name routing, env, mounts, MODEL_PATH rewrite) -sed -n '1,80p' runners/launch_*.sh -# shared helpers the scripts rely on -grep -nE "run_benchmark_serving|setup_eval_context|wait_for_server_ready|start_gpu_monitor" benchmarks/benchmark_lib.sh +# how a matrix point selects exactly one recipe variant (TP, GPUs, CONC, KV_OFFLOADING, image) +sed -n '/def select_recipe/,/^def runtime_arguments/p' infx/srt_slurm/single_node.py ``` -- **Read multiple sibling scripts** end-to-end for the exact env vars and serve shape (`VLLM_*`, +- **Read multiple sibling recipes** end-to-end for the exact engine args, env vars and serve shape (`VLLM_*`, `SGLANG_*`, device mapping, download/cache handling, `--enforce-eager` vs graph capture, - KV-cache dtype, attention/MoE backend, parsers). These are the truth for each runner. + KV-cache dtype, attention/MoE backend, parsers, `setup_script`). These are the truth for each runner. - **Compare several master-config search spaces** (e.g. `dsr1`, `qwen3.5`, the same model on a sibling SKU) to choose `{tp, ep, dp-attn} × concurrency` combos that fit *this* hardware's memory. Small-memory SKUs like h100/mi300x go TP8-only, while bigger SKUs add tp4/tp2/DEP. - **Internalize the fixed-seq-len nuances from the existing configs**: `8k1k` runs do **not** need the full `MAX_MODEL_LEN` (the matrix supplies `isl + osl + slack`), and graph-capture batch sizes are scaled to concurrency/scenario (and spec-token count for MTP), not maxed. - Copy how sibling scripts/configs already do it. + Copy how sibling recipes/configs already do it. **B. External research (confirm against upstream guidance):** - **`WebFetch` the model-link card + its `config.json`** → confirm `model:` id, precision, max context, architecture, spec-decode fields (`num_mtp_modules`, etc.). - **`WebFetch` the recipes-link** (if given) → canonical `vllm serve` flags + troubleshooting. - Reconcile with what the sibling scripts do. If they conflict, follow the repo and note why. + Reconcile with what the sibling recipes do. If they conflict, follow the repo and note why. - If a **draft-model-link** is given, note its id for `--speculative-config` and check the card for method (`eagle3` vs native `mtp`) and recommended token count. - Pick the **image tag** from the sibling's master-config entry (or recipes page) and **verify it exists** on the registry before using it. -This research directly feeds Step 2 (script flags/env) and Step 3 (search space). +This research directly feeds Step 2 (recipe args/env) and Step 3 (search space). -## What you're producing (4–5 files) +## What you're producing (3 files) -1. `benchmarks/single_node/fixed_seq_len/__[_][_mtp].sh` +1. `benchmarks/single_node/srt-slurm-recipes///-[-mtp]/8k1k.yaml` + (an `agentic.yaml` beside it for AgentX) 2. an entry in either master config, **`configs/nvidia-master.yaml`** (b*/h*/gb* SKUs) or - **`configs/amd-master.yaml`** (mi* SKUs) + **`configs/amd-master.yaml`** (mi* SKUs), with `srt-recipe:` on every search-space row 3. a `perf-changelog.yaml` entry (this diff vs main is what selects the sweep) -4. (if missing) `SPEC_SUFFIX`/framework-suffix routing in `runners/launch_*.sh` ## Step 1 — branch + find the sibling to copy @@ -86,17 +84,20 @@ This research directly feeds Step 2 (script flags/env) and Step 3 (search space) git checkout main && git pull origin main git checkout -b feat/-[-mtp]-dayzero # nearest sibling: same model other SKU, or same SKU other model -ls benchmarks/single_node/fixed_seq_len/_* # same model, other hardware -ls benchmarks/single_node/fixed_seq_len/*_*.sh # same hardware, other model +ls -d benchmarks/single_node/srt-slurm-recipes//*/* # same model, other hardware +ls -d benchmarks/single_node/srt-slurm-recipes/*/*/-* # same hardware, other model grep -n "--" configs/{nvidia,amd}-master.yaml ``` -Read the closest sibling script **and** its master-config entry. Copy their flag shapes and +Read the closest sibling recipe **and** its master-config entry. Copy their flag shapes and search-space structure rather than inventing. The right model is "same model on a sibling SKU, adjusted for this hardware's quirks." -## Step 2 — write the benchmark script +## Step 2 — write the srt-slurm recipe -Copy the sibling script and adjust. Things that vary and must be checked against the sibling / +Copy the sibling recipe (`base:` plus one `override_*` variant per matrix point) and adjust. +Engine flags are `roles.agg.args` keys without the leading `--`; env vars go in +`roles.agg.env`; each variant names its `CONC` (and `KV_OFFLOADING` for AgentX) in +`benchmark.env`, and `model.container` must equal the master-config `image`. Things that vary and must be checked against the sibling / the model's `recipes.vllm.ai` page: - **Mandatory model flags** (carry from the sibling): block size, parser flags (`--tool-call-parser` / `--reasoning-parser`), `--language-model-only` for text-only sweeps, @@ -113,7 +114,7 @@ the model's `recipes.vllm.ai` page: - **Memory headroom.** Bigger checkpoints constrain TP/EP. If the sibling on a smaller-memory SKU is TP8-only (e.g. h100), match that. -Validate as you go: `bash -n