GKE generated AICR GB300 recipies - #1985
Conversation
Introduces authoritative Google Kubernetes Engine (GKE) cloud provider recipe overlays, component defaults, and simulated KWOK node profiles for Google Cloud's upcoming A4X-Max (NVIDIA GB300 NVL72) bare metal platform. All manifests are generated programmatically from internal GKE Version Vector compatibility registries and GCE hardware compute specifications. Key Architectural Decisions & GKE Tuning: - GKE Managed RDMA & GDRCopy Networking: On GKE A4/A4X/A5X clusters, high-speed inter-node multi-GPU communication utilizes GPU Operator with GDRCopy enabled (`gdrcopy: enabled: true`) across GCP native multi-NIC optical rails. When node pools operate under GKE Managed RDMA (`--gpudirect-strategy=rdma`), GKE natively automates the installation and lifecycle of Google InfiniBand (gIB) plugins and NCCL tuners (`/usr/local/gib/`). Standalone networking installer DaemonSets (`gke-nccl-roce` or `gke-nccl-tcpxo`) are intentionally bypassed in overlays to prevent overriding GKE managed networking. - GKE COS Host Driver Alignment: On Container-Optimized OS (COS), NVIDIA drivers are natively mounted at `/home/kubernetes/bin/nvidia`. GPU Operator is configured with `driver.enabled: false` to instruct the operator not to deploy containerized driver daemonsets over managed host installations. - Workload Tuning Overlays: Configures Kubeflow distributed training leaves with GDRCopy acceleration (`NCCL_NET_GDR_LEVEL: 5`, `NCCL_DEBUG: INFO`), and Dynamo inference platforms parameterically matched to ARM64 host topologies and DRA release gates (`K8s.server.version: >= 1.34`). - Validation Alignment: Adopts deployment phase checks with GPU Operator v25.10+ version floors and suppresses static NCCL performance hard gates pending runtime RoCE testbed characterization. Signed-off-by: Adel Abouchaev <adelab@google.com>
|
Welcome to AICR, @rawsocket! Thanks for your first pull request. Before review, please ensure:
A maintainer will review this soon. |
📝 WalkthroughWalkthroughAdds GB300 support for GKE COS inference and training. The change adds KWOK node profiles, tuning metadata, NCCL RoCE values, base recipes, Dynamo and Kubeflow overlays, and generated recipe coverage expectations. Estimated code review effort: 4 (Complex) | ~45 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
recipes/components/gke-nccl-roce/values.yaml (1)
1-41: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winRegister and wire
gke-nccl-roce, or remove it.
recipes/components/gke-nccl-roce/values.yamlhas no registry entry orcomponentRefsconsumer. GB300 overlays explicitly bypass standalone networking installers in favor of GKE Managed RDMA.If this component is retained, add its registry and recipe wiring. Then run
make bom-docsand commitdocs/user/container-images.md.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@recipes/components/gke-nccl-roce/values.yaml` around lines 1 - 41, Either remove the unused gke-nccl-roce component, or register it in the component registry and wire it through a recipe consumer using componentRefs. If retained, update the generated container image documentation by running make bom-docs and include the resulting docs/user/container-images.md changes.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kwok/profiles/gke/a4x-maxgpu-4g-metal.yaml`:
- Around line 25-31: Update the gpu.memory value in the NVIDIA-GB300 profile
from 245760 to 285696 MiB per GPU, keeping the existing product, count, and
driver settings unchanged.
In `@recipes/overlays/gb300-gke-cos-inference-dynamo.yaml`:
- Around line 77-79: Update the Deployment.gpu-operator.version constraint in
the overlay’s constraints section from the outdated v24.6.0 floor to the
required >= v25.10.0 floor.
In `@recipes/overlays/gb300-gke-cos-inference.yaml`:
- Around line 54-64: Change the intent override in the nodewright-customizations
block from multiNodeTraining to inference so it matches this recipe’s
criteria.intent and selects the correct inference tuning profile; leave the
service, accelerator, and dependencyRefs values unchanged.
---
Outside diff comments:
In `@recipes/components/gke-nccl-roce/values.yaml`:
- Around line 1-41: Either remove the unused gke-nccl-roce component, or
register it in the component registry and wire it through a recipe consumer
using componentRefs. If retained, update the generated container image
documentation by running make bom-docs and include the resulting
docs/user/container-images.md changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 1cfd0cc4-f2e2-40ba-a596-fc5819707e01
📒 Files selected for processing (10)
docs/integrator/components/nodewright.mdkwok/profiles/gke/a4x-maxgpu-4g-metal.yamlkwok/profiles/gke/system-n2.yamlpkg/recipe/testdata/coverage_golden.yamlpkg/tuning/compute_test.gorecipes/components/gke-nccl-roce/values.yamlrecipes/overlays/gb300-gke-cos-inference-dynamo.yamlrecipes/overlays/gb300-gke-cos-inference.yamlrecipes/overlays/gb300-gke-cos-training-kubeflow.yamlrecipes/overlays/gb300-gke-cos-training.yaml
| gpu: | ||
| product: NVIDIA-GB300 | ||
| count: 4 | ||
| memory: 245760 | ||
| driver: "580.95.05" | ||
| driverMajor: "580" | ||
| driverMinor: "95" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Compare gpu.memory conventions across existing KWOK GPU profiles.
fd . kwok/profiles -e yaml --exec grep -n -A2 "^ gpu:" {} \;Repository: NVIDIA/aicr
Length of output: 335
🏁 Script executed:
#!/bin/bash
set -eu
files=$(fd . kwok/profiles -e yaml)
for f in $files; do
if grep -q '^ gpu:' "$f"; then
echo "=== $f ==="
sed -n '/^ gpu:/,/^[^[:space:]]/p' "$f" | head -n 12
fi
done
printf '\n=== gpu.memory references ===\n'
rg -n 'gpu:|memory:|gpu\.memory|GPU.*memory|memory.*MiB|memory.*GB' kwok .github docs 2>/dev/null | head -n 250Repository: NVIDIA/aicr
Length of output: 6626
🏁 Script executed:
#!/bin/bash
set -eu
echo '=== profile ==='
cat -n kwok/profiles/gke/a4x-maxgpu-4g-metal.yaml
echo '=== GPU profile processing ==='
sed -n '80,125p;185,215p' kwok/scripts/apply-nodes.sh
echo '=== profile schema/docs references ==='
rg -n -C 4 'profile|gpu\.memory|MiB per GPU|GB300|GB200|H100' kwok docs README.md 2>/dev/null | head -n 300Repository: NVIDIA/aicr
Length of output: 39102
🌐 Web query:
NVIDIA GB300 NVL72 per GPU HBM3e memory 288 GB official specifications
💡 Result:
The NVIDIA Blackwell Ultra GPU, used in the GB300 NVL72 platform, features an official memory capacity of 288 GB of HBM3e memory per GPU [1][2][3][4]. While some technical documentation may reference slightly lower figures in certain contexts (such as 279 GB of usable memory [5][6][7]), 288 GB is the headline capacity specification consistently cited across official NVIDIA technical communications and partner product documentation [1][2][8][3][4]. Key specifications for the Blackwell Ultra GPU memory include: - Capacity: 288 GB HBM3e per GPU [1][2][3][4]. - Bandwidth: Up to 8 TB/s per GPU [1][5][6]. - System-level: The GB300 NVL72 rack-scale system utilizes 72 of these GPUs to provide a total pool of 20 TB of HBM3e memory [9][2][8][7][3].
Citations:
- 1: https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/
- 2: https://www.supermicro.com/datasheet/datasheet_SuperCluster_GB300_NVL72.pdf
- 3: https://www.superx.sg/sites/default/files/2026-01/NVIDIA%2520GB300%2520Datasheet.pdf
- 4: https://lenovopress.lenovo.com/lp2357-lenovo-nvidia-gb300-nvl72-rack-scale-ai
- 5: https://www.freyrtech.ai/images/Datasheet-for-GB300-NVL72.pdf
- 6: https://nor-tech.com/wp-content/uploads/2026/03/blackwell-datasheet-B300.pdf
- 7: https://www.gigabyte.com/Enterprise/GIGAPOD-Rack-Scale/AI-DLC-Rack_NVIDIA-GB300-NVL72
- 8: https://www.nvidia.com/en-gb/data-center/gb300-nvl72/
- 9: https://www.nvidia.com/en-us/data-center/gb300-nvl72/
🌐 Web query:
GKE a4x-maxgpu-4g-metal GB300 gpu memory 245760
💡 Result:
The a4x-maxgpu-4g-metal machine type in Google Kubernetes Engine (GKE) is a bare metal instance equipped with 4 NVIDIA GB300 (Blackwell Ultra) GPUs [1][2]. According to official Google Cloud documentation, the total GPU memory for this machine type is 1,116 GB of HBM3e [1][3][4]. This total corresponds to 279 GB of HBM3e memory per individual GB300 GPU (1,116 GB ÷ 4 GPUs = 279 GB per GPU) [5][3]. While some industry literature references 288 GB for the Blackwell Ultra GPU [6][7], Google Cloud specifies the usable GPU memory as 279 GB per device for this specific machine configuration [5][3]. The value 245,760 mentioned in your query (245,760 MB) equals exactly 240 GB, which does not match the documented 279 GB per GPU or the 1,116 GB total for the instance [1][5].
Citations:
- 1: https://docs.cloud.google.com/ai-hypercomputer/docs/gpu
- 2: https://docs.cloud.google.com/compute/docs/instances/bare-metal-instances
- 3: https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines
- 4: https://docs.cloud.google.com/compute/docs/gpus/gpu-network-bandwidth
- 5: https://docs.cloud.google.com/compute/docs/gpus
- 6: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nd-gb300-v6-series
- 7: https://lenovopress.lenovo.com/lp2357-lenovo-nvidia-gb300-nvl72-rack-scale-ai
Set gpu.memory to 285696 MiB per GPU. GKE specifies 279 GB per GB300 GPU, and sibling profiles encode GB values as MiB. 245760 represents only 240 GiB.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kwok/profiles/gke/a4x-maxgpu-4g-metal.yaml` around lines 25 - 31, Update the
gpu.memory value in the NVIDIA-GB300 profile from 245760 to 285696 MiB per GPU,
keeping the existing product, count, and driver settings unchanged.
| constraints: | ||
| - name: Deployment.gpu-operator.version | ||
| value: ">= v24.6.0" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Compare gpu-operator version floors across GKE inference-dynamo overlays and locate the recipe merge logic.
rg -n "Deployment.gpu-operator.version" recipes/overlays/
rg -n "func.*[Mm]erge" pkg/recipe -g '*.go'Repository: NVIDIA/aicr
Length of output: 6417
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- overlay and base constraints ---'
sed -n '1,105p' recipes/overlays/gb300-gke-cos-inference-dynamo.yaml
sed -n '60,90p' recipes/overlays/gb300-gke-cos-inference.yaml
printf '%s\n' '--- sibling constraint ---'
sed -n '50,75p' recipes/overlays/b200-gke-cos-inference-dynamo.yaml
printf '%s\n' '--- validation merge implementation ---'
sed -n '80,135p' pkg/recipe/validation.go
sed -n '1150,1325p' pkg/recipe/metadata.go
printf '%s\n' '--- focused merge tests ---'
sed -n '690,825p' pkg/recipe/metadata_test.go
sed -n '995,1055p' pkg/recipe/metadata_test.goRepository: NVIDIA/aicr
Length of output: 19340
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- complete validation constraint merge ---'
sed -n '120,205p' pkg/recipe/validation.go
printf '%s\n' '--- merge chain and recipe loading ---'
sed -n '930,1015p' pkg/recipe/metadata_store.go
printf '%s\n' '--- validation constraint tests ---'
rg -n -A35 -B10 "Constraints|constraint.*overlay|overlay.*constraint|same-name" pkg/recipe/*_test.go pkg/recipe/*.go
printf '%s\n' '--- GB300-related GPU Operator floors and rationale ---'
rg -n -i -A4 -B4 "gb300|blackwell|gpu-operator v25|v25\.10\.0|v24\.6\.0" recipes/overlays recipes/components docsRepository: NVIDIA/aicr
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -eu
log=/tmp/coderabbit-shell-logs/shell-output-5Id0PB
printf '%s\n' '--- validation merge body ---'
sed -n '1,180p' "$log" | sed -n '/--- complete validation constraint merge ---/,$p' | head -n 180
printf '%s\n' '--- merge chain source ---'
sed -n '940,1010p' pkg/recipe/metadata_store.go
printf '%s\n' '--- focused validation merge references ---'
rg -n -A18 -B8 "mergeValidationPhase|Constraints: same nil|overlay value wins|Deployment.*Constraints" pkg/recipe/validation.go pkg/recipe/metadata.go pkg/recipe/metadata_test.go
printf '%s\n' '--- focused accelerator floors ---'
rg -n -i -A5 -B5 "v25\.10\.0|v24\.6\.0|gb300|b300|blackwell" \
recipes/overlays/gb300-gke-cos-inference-dynamo.yaml \
recipes/overlays/gb300-gke-cos-inference.yaml \
recipes/overlays/b200-gke-cos-inference-dynamo.yaml \
recipes/overlays/b200-any.yaml \
recipes/overlays/gb300-any.yaml 2>/dev/null || trueRepository: NVIDIA/aicr
Length of output: 31701
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
def deployment_gpu_operator_floor(path):
text = Path(path).read_text()
match = re.search(
r'(?ms)^\s*deployment:\s*\n.*?^\s*constraints:\s*\n'
r'\s*-\s*name:\s*Deployment\.gpu-operator\.version\s*\n'
r'\s*value:\s*"([^"]+)"',
text,
)
if not match:
raise SystemExit(f"constraint not found in {path}")
return match.group(1)
base = deployment_gpu_operator_floor("recipes/overlays/gb300-gke-cos-inference.yaml")
overlay = deployment_gpu_operator_floor("recipes/overlays/gb300-gke-cos-inference-dynamo.yaml")
# Mirrors mergeValidationPhase: same-name overlay constraints replace base values.
effective = overlay if overlay else base
print(f"base={base}")
print(f"overlay={overlay}")
print(f"effective={effective}")
assert base == ">= v25.10.0"
assert overlay == ">= v24.6.0"
assert effective == overlay
PYRepository: NVIDIA/aicr
Length of output: 206
Restore the GB300 GPU Operator floor to >= v25.10.0. The overlay overrides the base constraint, making >= v24.6.0 effective.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@recipes/overlays/gb300-gke-cos-inference-dynamo.yaml` around lines 77 - 79,
Update the Deployment.gpu-operator.version constraint in the overlay’s
constraints section from the outdated v24.6.0 floor to the required >= v25.10.0
floor.
| - name: nodewright-customizations | ||
| type: Helm | ||
| manifestFiles: | ||
| - components/nodewright-customizations/manifests/tuning-gke.yaml | ||
| overrides: | ||
| service: gke | ||
| accelerator: gb300 | ||
| intent: multiNodeTraining | ||
| dependencyRefs: | ||
| - nodewright-operator | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix the copy-pasted intent: multiNodeTraining override.
This file's criteria.intent is inference (line 17), but the nodewright-customizations override at line 61 sets intent: multiNodeTraining. The upstream b200-gke-cos-inference.yaml recipe sets intent: inference in the equivalent block. This block is identical to the one in gb300-gke-cos-training.yaml, which confirms this was carried over from the training recipe by mistake.
This override selects the tuning-gke.yaml manifest's applied settings. Applying multi-node training tuning to an inference recipe applies the wrong tuning profile to inference nodes.
🐛 Proposed fix
overrides:
service: gke
accelerator: gb300
- intent: multiNodeTraining
+ intent: inference
dependencyRefs:
- nodewright-operator📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: nodewright-customizations | |
| type: Helm | |
| manifestFiles: | |
| - components/nodewright-customizations/manifests/tuning-gke.yaml | |
| overrides: | |
| service: gke | |
| accelerator: gb300 | |
| intent: multiNodeTraining | |
| dependencyRefs: | |
| - nodewright-operator | |
| - name: nodewright-customizations | |
| type: Helm | |
| manifestFiles: | |
| - components/nodewright-customizations/manifests/tuning-gke.yaml | |
| overrides: | |
| service: gke | |
| accelerator: gb300 | |
| intent: inference | |
| dependencyRefs: | |
| - nodewright-operator | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@recipes/overlays/gb300-gke-cos-inference.yaml` around lines 54 - 64, Change
the intent override in the nodewright-customizations block from
multiNodeTraining to inference so it matches this recipe’s criteria.intent and
selects the correct inference tuning profile; leave the service, accelerator,
and dependencyRefs values unchanged.
Recipe evidence checkOther affected recipes without evidence yet: 4These recipes are affected by this PR but carry no committed evidence pointer, so there is
This gate is warning-only and never blocks merge. See ADR-007 for the trust model. |
There was a problem hiding this comment.
Welcome to AICR. The mechanical parts of adding an accelerator to this repo are easy to get wrong and you got them right. coverage_golden.yaml is correctly regenerated, pkg/tuning/compute_test.go and docs/integrator/components/nodewright.md are updated in step, and the inheritance chain (gb300-gke-cos-{training,inference} -> -kubeflow/-dynamo) matches how the b200 family is laid out. go test ./pkg/recipe/... ./pkg/tuning/... passes clean at this head.
The three blockers are all in the same category — config that ships to users and looks right but does the wrong thing — rather than anything structural. Each is a small fix.
Scope validated: all 10 changed files read at be8b984; recipe resolution run locally for the gb300 dynamo/inference/training paths; pkg/recipe + pkg/tuning tests run; apply-nodes.sh profile selection traced against this PR's own Tier 2 CI logs.
CI: Tier 1 / oke-ol-training (argocd-git) and the dependent KWOK Test Summary are red, but that is infrastructure, not your change — the log shows curl: (35) Recv failure: Connection reset by peer followed by Error: repo argo not found while adding the Argo CD Helm repo, in an OKE lane this PR does not touch. A re-run should clear it. The PR is also currently BEHIND main and will need a rebase.
Findings inline. Happy to re-review as soon as you have pushed.
| - check-nvidia-smi | ||
| constraints: | ||
| - name: Deployment.gpu-operator.version | ||
| value: ">= v24.6.0" |
There was a problem hiding this comment.
Blocking — this lowers an inherited validation floor instead of adding to it.
This overlay's base is gb300-gke-cos-inference, which sets the same constraint to >= v25.10.0 (gb300-gke-cos-inference.yaml:79). Overlay constraints union by name with the overlay winning on collision (mergeValidationPhase, pkg/recipe/validation.go:141-148), so this line replaces the base floor.
Confirmed by resolving the recipe at this head:
$ aicr recipe --service gke --accelerator gb300 --os cos --intent inference --platform dynamo
deployment:
constraints:
- name: Deployment.gpu-operator.version
value: '>= v24.6.0'
Consequence: a GB300 cluster running gpu-operator v24.6–v25.9 passes the dynamo deployment gate that the non-dynamo GB300 recipe fails it on — a false PASS on the more specialized recipe. The sibling b200-gke-cos-inference-dynamo.yaml:65 deliberately holds >= v25.10.0 and comments why.
Fix: set ">= v25.10.0", or drop the constraint block to inherit the base floor. If v24.6 is genuinely correct for GB300 + Dynamo, please add a comment saying why it is lower than its own base.
| overrides: | ||
| service: gke | ||
| accelerator: gb300 | ||
| intent: multiNodeTraining |
There was a problem hiding this comment.
Blocking — training tuning profile applied to an inference recipe.
criteria.intent here is inference (L16), but the nodewright-customizations override sets intent: multiNodeTraining. This block is byte-identical to the one in gb300-gke-cos-training.yaml:61, which is where it belongs.
The value is not inert — it lands in the rendered NodeWright tuning ConfigMap (recipes/components/nodewright-customizations/manifests/tuning-gke.yaml:85, configMap.intent: {{ $cust.intent }}), so inference nodes receive the multi-node-training tuning profile.
Every other inference recipe passes inference here (b200-gke-cos-inference.yaml:61, gb200-eks-inference.yaml:73); the *-training overlays are the ones using multiNodeTraining.
Fix: intent: inference.
(CodeRabbit flagged this as well — I verified it independently against the head and it is real and still open.)
There was a problem hiding this comment.
Changing the intent to inference is necessary but not sufficient here. I pulled the pinned package (ghcr.io/nvidia/nodewright-packages/nvidia-tuning-gke:0.1.2, the digest referenced by tuning-gke.yaml) and its profiles/ directory contains only gb200/ and h100/ — there is no gb300 profile. The prepare script (skyhook_dir/prepare_nvidia_configs.sh) validates the accelerator against those directories and exits 1 with "ERROR: unknown accelerator 'gb300'" before the intent value is ever read. So with accelerator: gb300, Skyhook tuning fails on every GB300 node for both leaves — training included — whatever the intent says.
Two ways to resolve: ship GB300 with tuning disabled (keep the component wired, pin enabled: false, so enabling later is a one-line change), or point GB300 at the existing gb200 profile — GB300 shares the Grace host architecture, so the gb200 settings may well be applicable, but that combination has not been officially validated on GB300.
@ayuskauskas as the NodeWright maintainer — would you be comfortable with the gb200 profile on GB300 nodes, or should this ship disabled until a validated gb300 profile is published? Same applies to gb300-gke-cos-training.yaml.
There was a problem hiding this comment.
A GB200 profile on a GB300 should be fine. I agree on the intent though that should match.
There was a problem hiding this comment.
Thanks Alex — that settles it. So the resolution for both leaves is:
- keep nodewright-customizations enabled, but set accelerator: gb200 in the overrides so the package selects the existing GB200 profile
- fix the intent per leaf: inference here, multiNodeTraining in gb300-gke-cos-training.yaml
@rawsocket one knock-on for the same push: the tuning table in docs/integrator/components/nodewright.md and the expected rows in pkg/tuning/compute_test.go are generated from the recipes' accelerator override, so the gke/gb300 row added in this PR will change once the override becomes gb200 — please re-run make tuning-docs and update the test expectation in the same commit. A short comment in the overlays noting that gb300 intentionally maps to the gb200 tuning profile (per this thread) would also help the next reader.
|
|
||
| spec: | ||
| instanceType: a4x-maxgpu-4g-metal | ||
| arch: arm64 |
There was a problem hiding this comment.
Blocking — this profile is never loaded; the GB300 KWOK lanes ran on amd64 H100 nodes.
get_profiles() in kwok/scripts/apply-nodes.sh:52 maps service+accelerator to profile paths, and the gke branch is still hardcoded to EKS fallbacks:
gke)
# GKE uses same profiles for now
case "$accelerator" in
gb200) gpu_profile="eks/p6-gb200.yaml" ;;
*) gpu_profile="eks/p5-h100.yaml" ;;
esacNothing in the tree references kwok/profiles/gke/, and kwok/README.md:131 calls out updating get_profiles() as a required step when adding a profile.
The effect is visible in this PR's own CI. From Tier 2 / gb300-gke-cos-training (helm):
[INFO] Recipe: gb300-gke-cos-training (service=gke, accelerator=gb300)
[INFO] Profiles: system=eks/system-m7i.yaml, gpu=eks/p5-h100.yaml
[INFO] Creating 4 GPU nodes (p5.48xlarge, 8x NVIDIA-H100-80GB-HBM3)
So all four green Tier 2 / gb300-* checks validated arm64 GB300 recipes against amd64 H100 nodes — the arch: arm64 on this line, and the arm64 image pins in the overlays, were never exercised.
Fix: add a gb300) gpu_profile="gke/a4x-maxgpu-4g-metal.yaml" case and a gke system-profile entry pointing at gke/system-n2.yaml. If you would rather land the recipes first, deleting these two profile files is an equally clean resolution — the concern is shipping them in a state where they appear wired up but are not.
Separately and non-blocking, same file: memory: 245760 (L28) — both sibling GPU profiles annotate the unit (memory: 81920 # MiB per GPU, memory: 196608 # MiB per GPU (192GB)). 245760 MiB is 240 GiB, which does not line up with GB300's published ~288 GB HBM3e per GPU. Worth re-checking against the Version Vector source and adding the unit comment.
| # GPUDirect RDMA over RoCE configuration for NVIDIA GPUs on GKE. | ||
| # Derived dynamically from authoritative GKE Version Vector compatibility matrices and GCE SoT. | ||
|
|
||
| image: |
There was a problem hiding this comment.
Non-blocking — this values file is unreachable.
gke-nccl-roce is not in recipes/registry.yaml (only gke-nccl-tcpxo, L176) and no overlay declares it as a componentRef. The only mentions anywhere are the comments in the two GB300 overlays (L52) explaining that the RoCE installer is intentionally bypassed under GKE Managed RDMA.
Because it is unregistered it also will not appear in docs/user/container-images.md, so these pins (nccl-roce-installer:v2.28.9-arm64, gib-plugin:v1.1.2-arm64) sit outside the BOM freshness gate and the weekly refresh will not catch drift on them.
Suggest dropping it from this PR and adding it together with its registry entry when the component is actually wired up.
| @@ -0,0 +1,96 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. | |||
There was a problem hiding this comment.
Non-blocking — missing Apache license header.
This file and three others in the PR carry only the copyright line, without the Apache-2.0 block:
recipes/overlays/gb300-gke-cos-inference.yamlrecipes/overlays/gb300-gke-cos-training.yamlkwok/profiles/gke/a4x-maxgpu-4g-metal.yamlkwok/profiles/gke/system-n2.yaml
These are the only four files of their kind in the tree without it — every other overlay, kwok profile, and component values.yaml has it. The other two new files here (-dynamo, -kubeflow) do have it, so this looks like the generator emitting two header variants. No CI gate catches it today.
yuanchen8911
left a comment
yuanchen8911
left a comment
There was a problem hiding this comment.
Reviewed at be8b984 with recipe resolution run locally, the pinned tuning package pulled from ghcr.io, and image manifests verified against registry APIs.
Change requests
-
NodeWright tuning cannot work as configured: nvidia-tuning-gke:0.1.2 contains only gb200/ and h100/ profiles, and its prepare script rejects
accelerator: gb300before intent is read, failing Skyhook tuning on every GB300 node. Either ship withenabled: falseor reuse the gb200 profile — maintainer question in the nodewright thread. -
gb300-gke-cos-inference-dynamo sets
Deployment.gpu-operator.versionto>= v24.6.0, which replaces (not tightens) the parent's>= v25.10.0on merge. GB300 support shipped in v25.10 — drop the constraint or set>= v25.10.0. -
The new GKE KWOK profiles are never selected: apply-nodes.sh still maps gke/gb300 to eks/p5-h100, and this PR's Tier 2 CI logs show the GB300 lanes ran on amd64 H100 nodes. Wire them in or remove them. Also set
gpu.memoryto 285696 MiB (Google lists 279 GB per GPU; 245760 encodes only 240 GB). -
The Kubeflow
runtime.image/runtime.envoverrides are consumed by nothing, the proposed 25.01-py3-arm64 tag does not exist on nvcr.io (404), and the manifest's hardcoded pytorch/pytorch digest is amd64-only — it cannot run on A4X Max's arm64 hosts. Parameterize the manifest image and pin a multi-arch CUDA 13 digest (NVIDIA PyTorch 25.08+).
Cleanup (config that reads as functional but is not)
- Drop
gdrcopy.enabled: true(inert without an operator-managed driver; values-gke-cos.yaml documents gdrcopy as unsupported on COS), the deaddriver.versionpin, anddriver.nvidiaDriverRoot(not a gpu-operator key). - Drop the dynamo leaf's
engine.cudaVersion/engine.arch(chart 1.2.1 does not consume them); document instead that GB300 requires the CUDA 13 runtime variants (vllm-runtime:1.2.1-cuda13 is published multi-arch). - Correct the Managed RDMA comments: A4X Max additionally requires the asapd-lite DaemonSet, mrdma.google.com resource claims, and ComputeDomain/IMEX setup — document these as external provisioning prerequisites.
- Remove unreachable gke-nccl-roce/values.yaml (no registry entry, no consumer); add the Apache header to the four files missing it.
Branch is behind main with a merge commit — please rebase onto origin/main and squash to a single signed/DCO commit.
| - gpu-operator | ||
| overrides: | ||
| runtime: | ||
| image: "nvcr.io/nvidia/pytorch:25.01-py3-arm64" |
There was a problem hiding this comment.
These runtime.image / runtime.env values are not consumed anywhere: the chart has no runtime key, and the ClusterTrainingRuntime manifest (components/kubeflow-trainer/manifests/torch-distributed-cluster-training-runtime.yaml) templates only acceleratedNodeSelector/acceleratedTolerations — its container image is hardcoded to a digest-pinned pytorch/pytorch:2.11.0-cuda12.8 image. I verified the resolved recipe carries these overrides as dead values.
Two registry checks make this blocking for GB300: nvcr.io returns 404 for the 25.01-py3-arm64 tag (it does not exist; 25.01-py3 does), and the pinned pytorch/pytorch digest is a single-platform amd64 manifest, so the generated runtime cannot execute on A4X Max's arm64 hosts as shipped. The fix needs to parameterize the manifest image and pin a multi-arch digest from NVIDIA's validated GB300 stack — CUDA 13 / NVIDIA PyTorch 25.08 or newer.
| - kai-scheduler | ||
| overrides: | ||
| engine: | ||
| cudaVersion: "13.0" |
There was a problem hiding this comment.
Chart ai-dynamo 1.2.1 does not consume engine.cudaVersion or engine.arch — the component values file has no engine key and no other dynamo overlay sets one, so these land in the release values unused. The underlying requirement is real: Dynamo requires CUDA 13 for B300/GB300, and the plain vllm-runtime:1.2.1 is CUDA 12.9, so it is not a supported GB300 runtime. But runtime selection happens in workload DynamoGraphDeployments, not these chart values. Suggest removing the block and adding a comment that GB300 model-serving workloads must use the CUDA 13 runtime variants — vllm-runtime:1.2.1-cuda13 is published multi-arch on NGC.
| cdi: | ||
| enabled: true | ||
| gdrcopy: | ||
| enabled: true |
There was a problem hiding this comment.
Three of these keys need cleanup (same block in the inference leaf):
- gdrcopy.enabled: true — the inherited values-gke-cos.yaml sets this false with the comment "not supported on COS (host-managed driver)", and GPU Operator deploys GDRCopy as a sidecar of its driver pod, so with driver.enabled: false the flag cannot provision anything. The resolved recipe ships gdrcopy: true, which at best is inert and at worst contradicts the documented COS support stance. The block comment's claim that GDRCopy provides the cross-fabric transport should go with it.
- driver.version: 580.95.05 — dead pin on a disabled driver. The value matches Google's documented A4X Max host-driver floor, so it belongs in a comment or prerequisite doc, but nothing reads it here (and nothing can validate it today: the GPU collector emits no driver-version measurement).
- driver.nvidiaDriverRoot — not a gpu-operator chart key; that name belongs to the nvidia-dra-driver-gpu chart, which already inherits the correct /home/kubernetes/bin/nvidia from the gke-cos base. The operator's COS path is already set via hostPaths.driverInstallDir in values-gke-cos.yaml.
driver.enabled: false and cdi.enabled: true merely restate the inherited values, so after cleanup this override block may not be needed at all.
| # (`--gpudirect-strategy=rdma`), GKE natively automates the lifecycle and host | ||
| # installation of Google InfiniBand (gIB) plugins and NCCL tuners (/usr/local/gib/). | ||
| # Consequently, standalone networking installer DaemonSets (gke-nccl-roce or | ||
| # gke-nccl-tcpxo) are intentionally bypassed to prevent overriding GKE managed networking. |
There was a problem hiding this comment.
This comment overstates what GKE provisions on A4X Max. The gIB-plugin/NCCL-tuner automation it describes is the A4/A4X flow; for A4X Max, Google's documentation requires deploying the asapd-lite DaemonSet (without it "child interfaces aren't created, and the dranet driver filters out the RDMA NICs"), Managed DRANET with ResourceClaimTemplates against deviceClassName mrdma.google.com, and ComputeDomain/IMEX setup for MNNVL.
To be fair about what AICR does cover: the recipe installs nvidia-dra-driver-gpu with ComputeDomains enabled, and the dra-support conformance check validates compute-domain ResourceSlices. What remains external and unvalidated is asapd-lite, the mrdma.google.com claims, and the behavioral ComputeDomain → ResourceClaimTemplate → IMEX flow (already tracked as TODO #1649 in the validator). Suggest correcting the comment and documenting asapd-lite/DRANET as external provisioning prerequisites (docs/integrator/eks-dynamo-networking.md is the precedent).
|
|
||
| constraints: | ||
| - name: K8s.server.version | ||
| value: ">= 1.34.3-gke.1318000" |
There was a problem hiding this comment.
FYI, no change requested: 1.34.3-gke.1318000 is the documented A4X Max floor for the 1.34 track, but AICR's version comparison parses the -gke.N suffix into Extras and never compares it (pkg/version/doc.go), so this constraint enforces only >= 1.34.3 — a cluster at 1.34.3-gke.1000000 passes, and the separate 1.35 track floor (1.35.0-gke.2745000) cannot be expressed at all. A comment noting the suffix is informational would help; we'll file a follow-up for provider-aware build-version comparison.
| - components/nodewright-customizations/manifests/tuning-gke.yaml | ||
| overrides: | ||
| service: gke | ||
| accelerator: gb300 |
There was a problem hiding this comment.
Change this to be gb200 instead of gb300 as the nvidia-tuning-gke doesn't support a gb300 accelerator and they would use the same tuning.
apply-nodes.sh::get_profiles silently fell back to eks/p5-h100.yaml for any unknown service+accelerator. NVIDIA#1985 showed the failure mode: GB300 GKE lanes reported green while running amd64 H100 nodes. Replace the hardcoded map with label-driven discovery in kwok/scripts/lib/profile-select.sh (match metadata.labels {provider,nodeType,accelerator}). Adding a profile is now a YAML file, no bash edit. Layered fail-closed: - Direct (apply-nodes.sh, make kwok-e2e): full diagnostic, exit 1. - Batch (run-all-recipes.sh): explicit invocations — every CI matrix cell — fail on unmapped SKIP; implicit get_recipes() still SKIPs for local make kwok-test-all. - CI discovery (kwok-recipes.yaml classify): unmapped recipes never enter the matrix; malformed overlay YAML aborts classify; workflow_dispatch on an unmapped recipe fails at classify. Selector tightened: reject booleans/integers in criteria (yq's // alternative was falsy-swallowing false); malformed profile YAML is fatal; ambiguous match returns rc=1, genuine no-match returns PROFILE_SELECT_RC_NO_MATCH (rc=2) so batch can distinguish "no profile yet" from "broken tree". Tests wired into the discover job's script-tests step (profile-select_test.sh: 22 cases; run-all-recipes_test.sh: 2 cases proving explicit unmapped cannot be passed). kwok/README.md updated. Fixes NVIDIA#1997 Signed-off-by: framsouza <fram.souza14@gmail.com>
4 participants
Summary
Recipes for GB300.
Motivation / Context
Updating the AICR repo with validate versions.
Type of Change
Component(s) Affected
pkg/recipe)Implementation Notes
GB300 recipes generated with up to date component versions.
Testing
# Commands run (prefer `make qualify` for non-trivial changes) make qualifyRisk Assessment
Checklist
make testwith-race)make lint)git commit -S) — GPG signing info