chore(dynamo-platform): coordinate NATS removal with Dynamo 1.4+ bump - #1983
chore(dynamo-platform): coordinate NATS removal with Dynamo 1.4+ bump#1983mohityadav8 wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughThe Dynamo platform configuration now targets Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/integrator/eks-dynamo-networking.md`:
- Line 25: Insert a blank line between the preceding blockquote and the shell
fenced code block in the documentation, leaving the block contents unchanged.
- Around line 43-51: Update the networking probe instructions around the
tcp-probe command to cover cross-nodegroup connectivity: after identifying the
actual request and KV-event listener endpoints, run a probe from the GPU
nodegroup to a system-nodegroup endpoint and another from the system nodegroup
to a GPU-nodegroup endpoint. Preserve the existing node selectors, required
tolerations, and endpoint-specific ports.
- Around line 81-83: Update the networking documentation rules around the
GPU-to-system security-group entries and corresponding AWS commands to use
separate direction-specific, per-port rules: document frontend-to-worker access
independently from worker-to-frontend ZMQ KV events on ports 5557 and dp_rank,
and after the 1.4.x chart is available, verify and document the exact TCP
request-plane listener separately.
In `@recipes/components/dynamo-platform/values.yaml`:
- Around line 28-31: Update the NATS opt-in comment near the global.nats.install
setting to explicitly use the full Helm key global.nats.install: true, matching
the preserved opt-in path and upgrade command.
In `@recipes/registry.yaml`:
- Line 527: Update the dynamo-platform recipe’s defaultVersion from the
placeholder to the published stable version 1.3.0, and update its matching
values comment. Run make qualify and make bom-docs, then commit the regenerated
image documentation so it lists 1.3.0 instead of 1.2.1.
In `@tests/manifests/dynamo-vllm-smoke-test.yaml`:
- Around line 23-27: Update the Dynamo container environment configuration in
the smoke-test manifest to explicitly set DYN_EVENT_PLANE=zmq and
DYN_REQUEST_PLANE=tcp, matching the documented Kubernetes-native ZMQ event plane
and TCP request plane. Ensure these variables are applied to the relevant Dynamo
containers so the configuration does not fall back to NATS.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 69a50282-d433-43bf-bc66-fcd156aa4e69
📒 Files selected for processing (9)
docs/integrator/eks-dynamo-networking.mdrecipes/components/dynamo-platform/values.yamlrecipes/registry.yamltests/chainsaw/ai-conformance/README.mdtests/chainsaw/ai-conformance/cluster/assert-dynamo.yamltests/chainsaw/ai-conformance/cluster/chainsaw-test.yamltests/chainsaw/ai-conformance/kind-inference-dynamo/assert-dynamo.yamltests/chainsaw/ai-conformance/kind-inference-dynamo/chainsaw-test.yamltests/manifests/dynamo-vllm-smoke-test.yaml
| > for dp_rank > 0. The TCP request plane does not have one fixed, | ||
| > documented port the way NATS had `4222` — confirm the actual listening | ||
| > port(s) on a live cluster before finalizing the SG rule below: | ||
| > ```shell |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add a blank line before the fenced block.
markdownlint-cli2 reports MD031 at Line 25. Put a blank line between the preceding blockquote and the shell fence.
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 25-25: Fenced code blocks should be surrounded by blank lines
(MD031, blanks-around-fences)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/integrator/eks-dynamo-networking.md` at line 25, Insert a blank line
between the preceding blockquote and the shell fenced code block in the
documentation, leaving the block contents unchanged.
Source: Linters/SAST tools
| You can confirm reachability directly from a GPU node before re-running. The | ||
| toleration is required because the GPU node groups on these clusters are | ||
| tainted (`NoSchedule`/`NoExecute`); without it the probe pod stays `Pending` | ||
| and never runs: | ||
|
|
||
| ```shell | ||
| kubectl run nats-probe --rm -i --restart=Never --image=busybox:1.36 \ | ||
| kubectl run tcp-probe --rm -i --restart=Never --image=busybox:1.36 \ | ||
| --overrides='{"spec":{"nodeSelector":{"<gpu-node-label-key>":"<value>"},"tolerations":[{"operator":"Exists"}]}}' \ | ||
| -- sh -c 'nc -zv -w 5 dynamo-platform-nats.dynamo-system.svc.cluster.local 4222' | ||
| -- sh -c 'nc -zv -w 5 <worker-pod-ip-or-svc> <PORT>' |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- target file structure ---'
ast-grep outline docs/integrator/eks-dynamo-networking.md --lang markdown || true
printf '%s\n' '--- target lines and surrounding content ---'
sed -n '1,120p' docs/integrator/eks-dynamo-networking.md
printf '%s\n' '--- related networking and probe references ---'
rg -n -i -C 3 'GPU|system|nodegroup|node group|request port|event port|security.?group|tcp-probe|worker endpoint|frontend|worker' docs/integrator docs README.md 2>/dev/null | head -300Repository: NVIDIA/aicr
Length of output: 32806
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- relevant tracked files ---'
git ls-files | rg -i 'dynamo|eks|manifest|validator|prometheus' | head -250
printf '%s\n' '--- Dynamo deployment and scheduling definitions ---'
rg -n -i -C 4 'dynamo-platform|frontend|worker|nodeSelector|toleration|tolerations|kv-events-config|5557|request.*port|service.*port|port:' recipes tests pkg manifests components 2>/dev/null | head -500
printf '%s\n' '--- exact target documentation line numbers ---'
nl -ba docs/integrator/eks-dynamo-networking.md | sed -n '1,115p'Repository: NVIDIA/aicr
Length of output: 48555
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- smoke-test manifest ---'
sed -n '35,100p' tests/manifests/dynamo-vllm-smoke-test.yaml
printf '%s\n' '--- Dynamo component values and EKS overlays ---'
sed -n '1,220p' recipes/components/dynamo-platform/values.yaml
for f in recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml recipes/overlays/rtx-pro-6000-eks-ubuntu-inference-dynamo.yaml recipes/overlays/eks-inference.yaml; do
if [ -f "$f" ]; then
printf '\n--- %s ---\n' "$f"
sed -n '1,220p' "$f"
fi
done
printf '%s\n' '--- focused diff for the documentation ---'
git diff -- docs/integrator/eks-dynamo-networking.md
printf '%s\n' '--- exact target line numbers ---'
awk '{printf "%5d %s\n", NR, $0}' docs/integrator/eks-dynamo-networking.md | sed -n '35,70p'Repository: NVIDIA/aicr
Length of output: 18517
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import re, json
text = Path("docs/integrator/eks-dynamo-networking.md").read_text()
block = re.search(r"```shell\n(.*?)\n```", text, re.S).group(1)
override = re.search(r"--overrides='([^']+)'", block).group(1)
spec = json.loads(override)
target = re.search(r"nc -zv -w 5 (\S+) (\S+)", block).groups()
assert spec["spec"]["nodeSelector"] == {
"<gpu-node-label-key>": "<value>"
}
assert target[0] == "<worker-pod-ip-or-svc>"
print("probe node selector:", spec["spec"]["nodeSelector"])
print("probe target:", target[0])
print("probe port:", target[1])
print("result: source is constrained to the GPU nodegroup and destination is constrained to a worker endpoint; this does not exercise a GPU-to-system path")
PYRepository: NVIDIA/aicr
Length of output: 281
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import json, re
text = Path("docs/integrator/eks-dynamo-networking.md").read_text()
start = text.index("kubectl run tcp-probe")
block = text[start:text.index("```", start)]
override = re.search(r"--overrides='([^']+)'", block).group(1)
spec = json.loads(override)
target = re.search(r"nc -zv -w 5 (\S+) (\S+)", block).groups()
print("probe node selector:", spec["spec"]["nodeSelector"])
print("probe tolerations:", spec["spec"]["tolerations"])
print("probe target:", target[0])
print("probe port:", target[1])
assert "<gpu-node-label-key>" in spec["spec"]["nodeSelector"]
assert target[0] == "<worker-pod-ip-or-svc>"
print("result: the documented source is constrained to the GPU nodegroup and the documented destination is a worker endpoint; this is not a GPU-to-system probe")
PYRepository: NVIDIA/aicr
Length of output: 460
Add cross-nodegroup probes.
The current probe runs on the GPU nodegroup and targets a worker endpoint, so it does not test GPU-to-system or system-to-GPU traffic. After identifying the actual request and KV-event listeners, run probes from each nodegroup to an endpoint in the other nodegroup.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/integrator/eks-dynamo-networking.md` around lines 43 - 51, Update the
networking probe instructions around the tcp-probe command to cover
cross-nodegroup connectivity: after identifying the actual request and KV-event
listener endpoints, run a probe from the GPU nodegroup to a system-nodegroup
endpoint and another from the system nodegroup to a GPU-nodegroup endpoint.
Preserve the existing node selectors, required tolerations, and
endpoint-specific ports.
| Allow ingress from the GPU node security group to the system node security | ||
| group on: | ||
| - TCP `<PORT>` - Dynamo request plane + KV events (dynamo-platform) — confirm exact port(s) on-cluster, see TODO above |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 \
'authorize-security-group-ingress|<PORT>|5557|frontend|worker' \
docs/integrator/eks-dynamo-networking.md tests recipesRepository: NVIDIA/aicr
Length of output: 50370
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- networking document ---'
sed -n '1,130p' docs/integrator/eks-dynamo-networking.md
printf '%s\n' '--- relevant endpoint and port declarations ---'
rg -n -C 4 \
--glob 'dynamo-vllm-smoke-test.yaml' \
--glob 'phases.sh' \
--glob 'eks-dynamo-networking.md' \
'kv-events-config|endpoint|port|PORT|frontend|worker|DYN_ROUTER_MODE' \
docs tests
printf '%s\n' '--- all Dynamo-related networking declarations ---'
rg -n -C 3 \
'dynamo|NATS|ZMQ|5557|4222|fromPort|toPort|authorize-security-group-ingress' \
docs tests recipesRepository: NVIDIA/aicr
Length of output: 50370
🌐 Web query:
ai-dynamo Dynamo 1.4 TCP request plane port frontend worker ZMQ KV events 5557 dp_rank
💡 Result:
In the ai-dynamo Dynamo 1.4 architecture, these terms relate to the system's KV (Key-Value) cache routing and inter-component messaging mechanisms [1][2][3]. Dynamo uses a TCP-based request plane for communication between frontend and worker components [1][4]. For KV-aware routing, worker engines (such as vLLM or SGLang) publish raw KV cache events—which track cache state across data parallel ranks—over a ZMQ (ZeroMQ) socket [3][5]. The specific technical details are as follows: - ZMQ Port 5557: This is a standard default ZMQ endpoint used by Dynamo workers for publishing KV events [3][6]. Engines like vLLM are configured to publish these events to tcp://127.0.0.1:5557 [3]. - Frontend Worker ZMQ KV Events: Dynamo workers (or a relay process) use a KvEventPublisher to subscribe to this ZMQ socket [3][7]. The events are relayed to Dynamo's internal event plane, enabling KV-aware request routing, which allows the router to direct requests to workers that already have the required KV cache blocks [1][3]. - dp_rank: This stands for Data Parallel Rank [3]. It is a critical identifier used in KV events and worker registration (e.g., with the Standalone KV Indexer) to distinguish cache states across different data parallel replicas of the same model instance [3][8][9]. It defaults to 0 if data parallelism is not enabled [3][9]. For high-concurrency TCP request plane traffic, documentation recommends tuning parameters such as DYN_TCP_WORKER_POOL_SIZE and DYN_TCP_WORK_QUEUE_SIZE to prevent throughput degradation [10].
Citations:
- 1: https://pypi.org/project/ai-dynamo/1.4.0.dev20260727/
- 2: https://github.com/ai-dynamo/dynamo
- 3: https://github.com/ai-dynamo/dynamo/blob/main/docs/integrations/kv-events-custom-engines.md
- 4: https://github.com/NVIDIA/aicr/blob/main/docs/user/validation.md
- 5: https://docs.nvidia.com/dynamo/integrations/kv-cache-integrations/kv-events-for-custom-engines
- 6: https://deepwiki.com/ai-dynamo/dynamo/7.2-backend-end-to-end-tests
- 7: https://github.com/ai-dynamo/dynamo/blob/f1dbea4f/lib/bindings/python/src/dynamo/_core.pyi
- 8: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/router/standalone-indexer.md
- 9: https://docs.nvidia.com/dynamo/components/router/standalone-indexer
- 10: https://github.com/ai-dynamo/dynamo/releases
🌐 Web query:
site:docs.nvidia.com/dynamo 1.4 request plane TCP frontend worker port
💡 Result:
In the NVIDIA Dynamo framework, the request plane transport layer manages communication between components such as the frontend, router, and workers [1][2]. When using the default TCP request plane, the worker port is dynamically assigned by the operating system (using port 0 by default) to avoid port conflicts [1][3]. If a specific, static port is required—for example, to accommodate firewall rules—you can explicitly set the port by configuring the DYN_TCP_RPC_PORT environment variable [1][4]. The host address for this communication can be set via DYN_TCP_RPC_HOST (defaulting to auto-detected) [1][3]. Note that this request plane port is distinct from the frontend's HTTP API port, which typically defaults to 8000 (configurable via the --http-port parameter) [5][6].
Citations:
- 1: https://docs.nvidia.com/dynamo/v-0-9-0/design-docs/request-plane
- 2: https://docs.nvidia.com/dynamo/v-0-9-0/design-docs/architecture-flow
- 3: https://docs.nvidia.com/dynamo/design-docs/communication-planes/request-plane
- 4: https://docs.nvidia.com/dynamo/v-0-9-1/design-docs/request-plane
- 5: https://docs.nvidia.com/dynamo/v-0-9-0/components/router/router-guide
- 6: https://docs.nvidia.com/dynamo/dev/components/frontend
🌐 Web query:
site:github.com/ai-dynamo/dynamo "5557" "dp_rank" kv events
💡 Result:
The term 5557 in the context of KV (Key-Value) events refers to the default TCP port used for ZMQ (ZeroMQ) PUB/SUB communication in systems like Dynamo, vLLM, and SGLang [1][2][3]. Key details regarding this setup include: 1. ZMQ PUB Port: LLM engines often utilize a ZMQ PUB socket (e.g., tcp://*:5557) to stream KV cache events, such as block stored, removed, or cleared events, to subscribers [1][2][4]. 2. Payload Structure: The messages published over this channel are typically formatted as a Msgpack payload consisting of a timestamp, a list of events, and a data parallel rank (dp_rank) [5][1]. 3. Data Parallel Rank (dp_rank): This field identifies the data parallel rank associated with the event [1]. In multi-rank configurations, engines may offset the base port (e.g., 5557) by the dp_rank to multiplex events from different ranks or handle them via distinct subscribers [2][6]. 4. Reliability: Because ZMQ PUB/SUB is inherently lossy, these systems often implement a supplementary mechanism, such as a secondary ROUTER socket or a sequence-number-based replay buffer, to allow consumers to detect gaps and request missing event data [2][4].
Citations:
- 1: https://github.com/ai-dynamo/dynamo/blob/main/docs/integrations/kv-events-custom-engines.md
- 2: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/router/kv-event-replay-comparison.md
- 3: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/router/router-operations.md
- 4: https://github.com/ai-dynamo/dynamo/blob/7c7da216/docs/components/router/kv-event-replay-comparison.md
- 5: https://github.com/ai-dynamo/dynamo/blob/main/docs/fern/pages/developer-guide/advanced-customizations/writing-custom-backends/publish-kv-events.md
- 6: https://github.com/ai-dynamo/dynamo/blob/f1dbea4f/components/src/dynamo/sglang/publisher.py
🌐 Web query:
site:github.com/ai-dynamo/dynamo "DYN_TCP_WORKER_POOL_SIZE" request plane
💡 Result:
The DYN_TCP_WORKER_POOL_SIZE environment variable is used to configure the size of the TCP worker pool in the Dynamo system's TCP request plane [1][2]. When operating the TCP request plane at high concurrency (specifically exceeding 10,000 concurrent requests), it is recommended to tune this parameter to ensure optimal throughput [1][2]. The following tuning guidelines apply: - Set DYN_TCP_WORKER_POOL_SIZE to be greater than or equal to your maximum expected concurrency [1][2]. - Set DYN_TCP_WORK_QUEUE_SIZE to 4 times the value of DYN_TCP_WORKER_POOL_SIZE [1][2]. Failure to appropriately size these parameters relative to high concurrency demands may result in a significant drop in throughput [1][2]. In recent updates, the default value for the internal TCP worker pool size was increased to 10,000 [3]. Operators are encouraged to monitor related metrics, such as heap usage and the dynamo_work_handler_permit_wait_seconds semaphore wait metric, to ensure the configuration remains appropriate for their specific workload [3].
Citations:
- 1: https://github.com/ai-dynamo/dynamo/releases/tag/v1.3.0-deepseek-v4-dev.1
- 2: https://github.com/ai-dynamo/dynamo/releases
- 3: perf: bump the worker concurrency limit by raising the default tcp worker pool size ai-dynamo/dynamo#9090
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
doc = Path("docs/integrator/eks-dynamo-networking.md").read_text()
manifest = Path("tests/manifests/dynamo-vllm-smoke-test.yaml").read_text()
uat = Path("tests/uat/lib/phases.sh").read_text()
rules = re.findall(
r'(?m)^- TCP (\S+) - (.+)$',
doc,
)
aws = re.findall(
r'(?ms)aws ec2 authorize-security-group-ingress --group-id (\S+) \\\n'
r'\s+--protocol tcp --port (\S+) --source-group (\S+)',
doc,
)
endpoint = re.findall(r'"endpoint":"([^"]+)"', manifest)
placements = {
"frontend": bool(re.search(r'(?ms)- name: Frontend.*?type: frontend', manifest)),
"worker": bool(re.search(r'(?ms)- name: VllmDecodeWorker.*?type: worker', manifest)),
"worker_gpu_selector": bool(re.search(
r'(?ms)- name: VllmDecodeWorker.*?nodeSelector:', manifest
)),
}
print("document_ingress_rules:", rules)
print("document_aws_ingress_rules:", aws)
print("manifest_kv_endpoints:", endpoint)
print("manifest_worker_endpoint_binds_all_interfaces:",
any(e.startswith("tcp://*:") for e in endpoint))
print("manifest_component_shapes:", placements)
print("manifest_runtime_versions:",
sorted(set(re.findall(r'vllm-runtime:([^\s]+)', manifest))))
print("uat_worker_endpoint_present:",
bool(re.search(r'--kv-events-config.*?5557', uat, re.S)))
print("uat_runtime_versions:",
sorted(set(re.findall(r'vllm-runtime:([^\s]+)', uat))))
PY
printf '%s\n' '--- upstream request-plane port statements ---'
curl -fsSL https://docs.nvidia.com/dynamo/design-docs/communication-planes/request-plane \
| grep -Eio '.{0,100}(DYN_TCP_RPC_PORT|port 0|dynamic|worker port).{0,160}' \
| head -20 || trueRepository: NVIDIA/aicr
Length of output: 879
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for url in \
https://docs.nvidia.com/dynamo/v-0-9-1/design-docs/request-plane \
https://docs.nvidia.com/dynamo/v-0-9-0/design-docs/request-plane \
https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/integrations/kv-events-custom-engines.md \
https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/components/router/kv-event-replay-comparison.md
do
printf '\n--- %s ---\n' "$url"
curl -fsSL "$url" \
| grep -Ein -C 3 \
'DYN_TCP_RPC_PORT|port 0|dynamic|listen|bind|connect|5557|endpoint|subscriber|publisher' \
| head -120 || true
doneRepository: NVIDIA/aicr
Length of output: 50368
Add direction-specific, per-port security-group rules.
The current GPU SG -> system SG rule does not allow frontend-to-worker connections. Do not combine the TCP request plane with ZMQ KV events. After the 1.4.x chart is published, verify the request-plane listener and document it separately from 5557 + dp_rank. Update the rules and AWS commands at lines 81-83 and 112-114.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/integrator/eks-dynamo-networking.md` around lines 81 - 83, Update the
networking documentation rules around the GPU-to-system security-group entries
and corresponding AWS commands to use separate direction-specific, per-port
rules: document frontend-to-worker access independently from worker-to-frontend
ZMQ KV events on ports 5557 and dp_rank, and after the 1.4.x chart is available,
verify and document the exact TCP request-plane listener separately.
| # global.nats.install is intentionally left unset (chart default: false). | ||
| # Re-enable only if a workload explicitly opts into a NATS-based transport | ||
| # (legacy DYN_REQUEST_PLANE=nats, DYN_EVENT_PLANE=nats, or a NATS | ||
| # request-trace sink), via `nats: install: true`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use the full Helm key for the NATS opt-in.
The comment says nats: install: true, but the required key is global.nats.install, as shown by the upgrade command below. Write global.nats.install: true explicitly so users do not apply the override at the wrong path.
The PR objective names global.nats.install as the preserved opt-in key.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@recipes/components/dynamo-platform/values.yaml` around lines 28 - 31, Update
the NATS opt-in comment near the global.nats.install setting to explicitly use
the full Helm key global.nats.install: true, matching the preserved opt-in path
and upgrade command.
| defaultRepository: https://helm.ngc.nvidia.com/nvidia/ai-dynamo | ||
| defaultChart: dynamo-platform | ||
| defaultVersion: "1.2.1" | ||
| defaultVersion: "<NEW_VERSION>" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
! rg -n 'defaultVersion:\s*"<NEW_VERSION>"' recipes/registry.yaml
test -s docs/user/container-images.md
rg -n 'dynamo-platform' docs/user/container-images.mdRepository: NVIDIA/aicr
Length of output: 275
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- registry context ---'
sed -n '500,540p' recipes/registry.yaml
printf '%s\n' '--- generated documentation context ---'
sed -n '30,45p;105,120p' docs/user/container-images.md
printf '%s\n' '--- related version references ---'
rg -n -C 2 'dynamo-platform|<NEW_VERSION>|1\.2\.1|1\.4' recipes docs .github 2>/dev/null || true
printf '%s\n' '--- repository status ---'
git status --short
git diff --statRepository: NVIDIA/aicr
Length of output: 33023
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- Helm repository index ---'
curl -fsSL https://helm.ngc.nvidia.com/nvidia/ai-dynamo/index.yaml |
sed -n '1,160p'
printf '%s\n' '--- component values version marker ---'
sed -n '1,35p' recipes/components/dynamo-platform/values.yaml
printf '%s\n' '--- version references in tracked files ---'
rg -n '<NEW_VERSION>|defaultVersion:|kubernetes-operator:' recipes/components/dynamo-platform recipes/registry.yaml docs/user/container-images.mdRepository: NVIDIA/aicr
Length of output: 9123
🏁 Script executed:
#!/bin/bash
set -euo pipefail
curl -fsSL https://helm.ngc.nvidia.com/nvidia/ai-dynamo/index.yaml |
awk '
/^ dynamo-platform:/ { in_section=1 }
in_section && /^ [A-Za-z0-9-]+:/ && $0 !~ /^ dynamo-platform:/ { exit }
in_section && /^( version:| appVersion:| created:| name:)/ { print }
'
printf '%s\n' '--- all repository references to the target chart versions ---'
rg -n -C 2 '1\.4\.[0-9]+|1\.2\.1|<NEW_VERSION>' recipes docsRepository: NVIDIA/aicr
Length of output: 8032
Pin dynamo-platform to a published chart version.
Use the latest stable published version, 1.3.0, instead of <NEW_VERSION>. Update the matching values comment, run make qualify and make bom-docs, and commit the regenerated image documentation, which currently lists 1.2.1.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@recipes/registry.yaml` at line 527, Update the dynamo-platform recipe’s
defaultVersion from the placeholder to the published stable version 1.3.0, and
update its matching values comment. Run make qualify and make bom-docs, then
commit the regenerated image documentation so it lists 1.3.0 instead of 1.2.1.
Source: Coding guidelines
| # Uses the Dynamo 1.4+ Kubernetes path: Kubernetes-native discovery, with | ||
| # the request plane on TCP (default) and the KV event plane on ZMQ | ||
| # (default) — no bundled NATS relay in between. The frontend runs | ||
| # KV-cache-aware routing; the local vLLM engine publishes KV events over | ||
| # ZMQ and the KV router consumes them directly, end-to-end. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 \
'vllm-runtime:|DYN_(REQUEST|EVENT)_PLANE|kv-events-config|global\.nats\.install|defaultVersion' \
recipes testsRepository: NVIDIA/aicr
Length of output: 35541
🏁 Script executed:
set -euo pipefail
printf '%s\n' '--- smoke manifest ---'
cat -n tests/manifests/dynamo-vllm-smoke-test.yaml | sed -n '1,110p'
printf '%s\n' '--- Dynamo platform values ---'
cat -n recipes/components/dynamo-platform/values.yaml | sed -n '1,90p'
printf '%s\n' '--- related transport references ---'
rg -n -C 4 'DYN_REQUEST_PLANE|DYN_EVENT_PLANE|NATS_SERVER|kv-events-config|vllm-runtime|Dynamo 1\.4|ZMQ|NATS' \
recipes tests demos docs README.md 2>/dev/null || true
printf '%s\n' '--- repository state summary ---'
git status --short
git diff --statRepository: NVIDIA/aicr
Length of output: 33983
🌐 Web query:
NVIDIA Dynamo vllm-runtime 1.2.1 DYN_REQUEST_PLANE DYN_EVENT_PLANE ZMQ NATS
💡 Result:
NVIDIA Dynamo is a distributed inference framework [1][2]. The vllm-runtime 1.2.1 refers to a specific version of the containerized environment used to run vLLM-based inference within the Dynamo ecosystem [3][4]. The parameters mentioned relate to Dynamo's communication planes, which define how different components (frontend, router, worker) interact [5][6]. 1. DYN_REQUEST_PLANE: This environment variable configures the request distribution transport, determining how RPC requests flow between the frontend, router, and workers [5][7]. - Valid values: tcp (default, optimized for speed), nats [5][7]. - Can be set via the --request-plane CLI flag [7][8]. 2. DYN_EVENT_PLANE: This environment variable configures the pub/sub layer used for near real-time event exchange, such as KV cache updates, worker load metrics, and sequence tracking [6]. - Valid values: nats (default), zmq [6][7]. - Can be set via the --event-plane CLI flag [7][8]. 3. NATS and ZMQ: These are the underlying messaging technologies used to implement the communication planes [6][9]. - NATS (used for both request and event planes) is a messaging system that provides durable messaging (JetStream) and is often used for events and distributed synchronization [5][6]. - ZMQ (ZeroMQ) is an alternative transport option specifically for the event plane, which does not require an external server, as workers dynamically discover each other [6]. In summary, these settings allow users to tune Dynamo's distributed architecture to their specific infrastructure requirements (e.g., choosing between ZMQ for easier, serverless event communication or NATS for more robust, durable messaging) [5][6][2].
Citations:
- 1: https://github.com/ai-dynamo/dynamo/blob/main/docs/backends/vllm/README.md
- 2: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/-
- 3: https://docs.nvidia.com/dynamo/dev/resources/release-artifacts
- 4: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/1.2.1-cuda13/tags
- 5: https://docs.dynamo.nvidia.com/dynamo/design-docs/communication-planes/request-plane
- 6: https://docs.dynamo.nvidia.com/dynamo/dev/design-docs/communication-planes/event-plane
- 7: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/frontend/configuration.md
- 8: https://ithub.global.ssl.fastly.net/ai-dynamo/dynamo/blob/main/docs/components/router/router-operations.md
- 9: https://docs.dynamo.nvidia.com/dynamo/dev/design-docs/communication-planes/request-plane
🌐 Web query:
site:github.com/ai-dynamo/dynamo DYN_REQUEST_PLANE DYN_EVENT_PLANE
💡 Result:
In the Dynamo distributed inference framework, DYN_REQUEST_PLANE and DYN_EVENT_PLANE are environment variables used to configure separate communication channels for system operations [1][2]. DYN_REQUEST_PLANE This variable controls the transport mechanism for RPC requests moving between components (e.g., frontend, router, and worker) [1]. - Valid values: tcp (default) and nats [3][4][5]. - The tcp mode is considered the fastest and requires minimal infrastructure [1]. - The nats mode uses a NATS message broker for request distribution [1][6]. DYN_EVENT_PLANE This variable specifies the transport mechanism for distributing KV cache events and router synchronization messages [1]. - Valid values: zmq and nats [3][4][5]. - If left unset, the default behavior depends on the discovery backend: - Local backends (file, mem) default to zmq [7][8]. - Distributed backends (etcd, kubernetes) default to nats [7][8]. - The zmq option allows the router to function without external services like NATS or etcd [2]. - When using nats, the system uses a NATS server (defaulting to localhost:4222) for event publishing [1][2]. These two planes operate independently; for example, a system can be configured with a tcp request plane while using a nats or zmq event plane [1][2].
Citations:
- 1: https://github.com/ai-dynamo/dynamo/blob/main/docs/design-docs/request-plane.md
- 2: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/router/router-operations.md
- 3: https://github.com/ai-dynamo/dynamo/blob/main/docs/components/frontend/configuration.md
- 4: https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/frontend/frontend_args.py
- 5: https://github.com/ai-dynamo/dynamo/blob/74b79d04/components/src/dynamo/mocker/args.py
- 6: https://github.com/ai-dynamo/dynamo/blob/main/docs/design-docs/distributed-runtime.md
- 7: https://github.com/ai-dynamo/dynamo/blob/ce0cb901/lib/runtime/src/distributed.rs
- 8: https://github.com/ai-dynamo/dynamo/blob/74b79d04/lib/runtime/src/config/environment_names.rs
🏁 Script executed:
set -euo pipefail
python3 - <<'PY'
import json
import urllib.request
urls = [
"https://api.github.com/repos/ai-dynamo/dynamo/tags?per_page=100",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/lib/runtime/src/distributed.rs",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/lib/runtime/src/config/environment_names.rs",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/components/src/dynamo/frontend/frontend_args.py",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/design-docs/event-plane.md",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/components/router/router-operations.md",
]
for url in urls:
print(f"\n--- {url} ---")
try:
with urllib.request.urlopen(url, timeout=20) as response:
data = response.read().decode()
if url.endswith("/tags?per_page=100"):
tags = json.loads(data)
print("\n".join(t["name"] for t in tags if "1.4" in t["name"] or "1.3" in t["name"] or "1.2" in t["name"] or "1.5" in t["name"]))
else:
for i, line in enumerate(data.splitlines(), 1):
if any(x in line for x in ("DYN_EVENT_PLANE", "DYN_REQUEST_PLANE", "event_plane", "request_plane", "kubernetes", "zmq", "nats", "default")):
print(f"{i}:{line}")
except Exception as exc:
print(type(exc).__name__, exc)
PY
printf '%s\n' '--- version and release references in repository ---'
rg -n -C 3 '1\.2\.1|1\.4\+|DYN_EVENT_PLANE|DYN_REQUEST_PLANE|event plane defaults|Kubernetes.*NATS|Kubernetes.*ZMQ' \
. --glob '!vendor/**' --glob '!node_modules/**' --glob '!build/**' --glob '!dist/**' 2>/dev/null || trueRepository: NVIDIA/aicr
Length of output: 27985
🌐 Web query:
site:docs.nvidia.com/dynamo OR site:docs.dynamo.nvidia.com Dynamo 1.4 event plane Kubernetes ZMQ default
💡 Result:
In NVIDIA Dynamo, the event plane transport used for near real-time event exchange (such as KV cache updates and worker load metrics) is determined by the configured discovery backend [1][2]. For deployments using Kubernetes as the discovery backend (which the Dynamo operator automatically configures for Kubernetes environments), the default event plane transport is NATS [2][3]. Key details regarding Dynamo event plane configuration: Transport Selection Logic: When the event plane transport (DYN_EVENT_PLANE) is not explicitly set, Dynamo selects the default based on the discovery backend [2][3]: - Kubernetes or etcd (distributed backends): Defaults to NATS [2][3]. - File or memory (local backends): Defaults to ZMQ (peer-to-peer, no external infrastructure required) [2][3]. Overriding Defaults: You can override the automatic selection by setting the DYN_EVENT_PLANE environment variable to either nats or zmq [1][2]. Deployment Considerations: - NATS: Recommended for large-scale deployments, but requires a NATS server [1][2]. - ZMQ: Provides a peer-to-peer transport with low operational overhead, requiring no external server [1][2]. When using ZMQ, workers bind ZMQ PUB sockets and register their addresses via the discovery system, allowing subscribers to automatically discover and connect to active publishers [1][2]. In Kubernetes, the Dynamo operator can inject the DYN_EVENT_PLANE environment variable into pods, allowing you to switch between these transports as needed for your cluster architecture [1][3].
Citations:
- 1: https://docs.dynamo.nvidia.com/dynamo/dev/design-docs/communication-planes/event-plane
- 2: https://docs.nvidia.com/dynamo/dev/design-docs/communication-planes/event-plane
- 3: https://docs.dynamo.nvidia.com/dynamo/design-docs/communication-planes/event-plane.md
🌐 Web query:
site:catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime 1.2.1 Dynamo version
💡 Result:
Dynamo version 1.2.1 refers to a release of the NVIDIA Dynamo vLLM Runtime, a containerized framework designed for distributed large language model (LLM) inference [1][2]. As of August 2, 2026, version 1.2.1 is identified as the latest stable release tag in the NVIDIA NGC catalog for the vLLM runtime environment [1][3]. There are also experimental development builds under the 1.3.0 series (e.g., 1.3.0-cosmos3-dev.1) available [2][4].
Citations:
- 1: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/-/tags
- 2: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/1.3.0-kimi-k2.6-dev.1
- 3: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/1.2.1-efa-amd64
- 4: https://catalog.ngc.nvidia.com/orgs/nvidia/ai-dynamo/containers/vllm-runtime/1.3.0-cosmos3-dev.1/tags
🏁 Script executed:
set -euo pipefail
python3 - <<'PY'
import json
import urllib.request
urls = [
"https://api.github.com/repos/ai-dynamo/dynamo/tags?per_page=100",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/lib/runtime/src/distributed.rs",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/lib/runtime/src/config/environment_names.rs",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/components/src/dynamo/frontend/frontend_args.py",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/design-docs/event-plane.md",
"https://raw.githubusercontent.com/ai-dynamo/dynamo/main/docs/components/router/router-operations.md",
]
for url in urls:
print(f"\n--- {url} ---")
try:
with urllib.request.urlopen(url, timeout=20) as response:
data = response.read().decode()
if url.endswith("/tags?per_page=100"):
tags = json.loads(data)
print("\n".join(t["name"] for t in tags if any(v in t["name"] for v in ("1.2", "1.3", "1.4", "1.5"))))
else:
for i, line in enumerate(data.splitlines(), 1):
if any(x in line for x in ("DYN_EVENT_PLANE", "DYN_REQUEST_PLANE", "event_plane", "request_plane", "kubernetes", "zmq", "nats", "default")):
print(f"{i}:{line}")
except Exception as exc:
print(type(exc).__name__, exc)
PY
printf '%s\n' '--- version and release references in repository ---'
rg -n -C 3 '1\.2\.1|1\.4\+|DYN_EVENT_PLANE|DYN_REQUEST_PLANE|event plane defaults|Kubernetes.*NATS|Kubernetes.*ZMQ' \
. --glob '!vendor/**' --glob '!node_modules/**' --glob '!build/**' --glob '!dist/**' 2>/dev/null || trueRepository: NVIDIA/aicr
Length of output: 27985
🏁 Script executed:
set -euo pipefail
fetch() {
url="$1"
printf '\n--- %s ---\n' "$url"
curl -ksSfL --max-time 30 "$url"
}
fetch 'https://api.github.com/repos/ai-dynamo/dynamo/tags?per_page=100' |
python3 -c '
import json, sys
for tag in json.load(sys.stdin):
name = tag["name"]
if any(x in name.lower() for x in ("1.2", "1.3", "1.4", "1.5")):
print(name, tag["commit"]["sha"])
'
for path in \
lib/runtime/src/distributed.rs \
lib/runtime/src/config/environment_names.rs \
components/src/dynamo/frontend/frontend_args.py \
docs/design-docs/communication-planes/event-plane.md \
docs/design-docs/event-plane.md \
docs/components/router/router-operations.md
do
fetch "https://raw.githubusercontent.com/ai-dynamo/dynamo/main/$path" 2>/dev/null |
rg -n -C 3 'DYN_EVENT_PLANE|DYN_REQUEST_PLANE|event_plane|request_plane|kubernetes|zmq|nats|default' || true
done
printf '\n--- upstream release references ---\n'
curl -ksSfL --max-time 30 \
'https://api.github.com/repos/ai-dynamo/dynamo/releases?per_page=100' |
python3 -c '
import json, sys
for release in json.load(sys.stdin):
name = release.get("name") or ""
tag = release.get("tag_name") or ""
if any(x in (name + " " + tag).lower() for x in ("1.2", "1.3", "1.4", "1.5")):
print(tag, name)
'Repository: NVIDIA/aicr
Length of output: 755
Set the Dynamo event plane explicitly.
With Kubernetes discovery, an unset DYN_EVENT_PLANE defaults to NATS. --kv-events-config only configures vLLM’s local ZMQ publisher; it does not select Dynamo’s event plane. Set DYN_EVENT_PLANE=zmq and DYN_REQUEST_PLANE=tcp on the Dynamo containers, or keep bundled NATS enabled.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/manifests/dynamo-vllm-smoke-test.yaml` around lines 23 - 27, Update the
Dynamo container environment configuration in the smoke-test manifest to
explicitly set DYN_EVENT_PLANE=zmq and DYN_REQUEST_PLANE=tcp, matching the
documented Kubernetes-native ZMQ event plane and TCP request plane. Ensure these
variables are applied to the relevant Dynamo containers so the configuration
does not fall back to NATS.
Source: Learnings
1 participant
Summary
Stage the coordinated NATS removal for the Dynamo 1.4+ bump: drop
global.nats.install, the NATS storage-class/node-scheduling paths, and NATS-specific test/doc/conformance references, since Dynamo 1.4+ defaults the request plane to TCP and the KV event plane to ZMQ.Motivation / Context
Upstream Dynamo 1.4+ disables bundled NATS by default (ai-dynamo/dynamo#11951) because the request plane now defaults to TCP and the KV event plane to ZMQ. AICR currently pins
dynamo-platform1.2.1 withglobal.nats.install: trueset explicitly, so nothing breaks on the current pin — but the next version bump needs to land together with this NATS removal, not as a baredefaultVersionchange.This is a draft.
dynamo-platform1.4.x hasn't been published tohelm.ngc.nvidia.com/nvidia/ai-dynamoyet — everything here is staged with<NEW_VERSION>/<PORT>placeholders so it's ready to finish and merge the moment it ships.Fixes: #1836
Related: N/A
Type of Change
Component(s) Affected
pkg/recipe)docs/,examples/)tests/chainsaw/ai-conformance/*,tests/manifests/dynamo-vllm-smoke-test.yamlImplementation Notes
recipes/components/dynamo-platform/values.yaml: removedglobal.nats.install: trueand thenats.config.jetstreamPVC block.global.nats.installis left unset so the chart's new default (false) applies; opt back in per-workload vianats: install: trueif a legacy NATS transport is ever needed.recipes/registry.yaml: removed thestorageClassPathsentry (was NATS-only) and the two NATSnodeSelectorPaths/tolerationPathsentries undernodeScheduling.system.tests/manifests/dynamo-vllm-smoke-test.yaml: updated the event-plane comment — KV events now flow ZMQ end-to-end, no NATS relay.docs/integrator/eks-dynamo-networking.md: rewritten. TheTCP 4222SG rule is gone, but the TCP request plane / ZMQ KV events are still direct frontend↔worker pod-to-pod connections crossing the same GPU↔system nodegroup SG boundary, so the doc still needs a port — left as<PORT>pending verification on a real 1.4+ EKS cluster (ss -tlnpcommands included inline).tests/chainsaw/ai-conformance/{cluster,kind-inference-dynamo}/assert-dynamo.yaml+ theirchainsaw-test.yamldescriptions +tests/chainsaw/ai-conformance/README.md: removed thedynamo-platform-natsStatefulSet assertion and NATS wording. Not called out in Dynamo 1.4+ bump: bundled NATS disabled by default (TCP request plane, ZMQ KV events) #1836's listed scope, but the assertion hard-fails once NATS stops being bundled by default, so folded it into this PR.defaultVersion/version comments (<NEW_VERSION>, 5 files), the 2×vllm-runtime:1.2.1image tags in the smoke test,make bom-docsregeneration.Upgrade impact for standing clusters: an in-place
helm upgradeaccepting the new default removes bundled NATS and dropsNATS_SERVERfrom operator-managed workload specs, which triggers rolling restarts. Fresh bundle installs are unaffected. Pass--set global.nats.install=trueduring the upgrade to opt out and avoid the restart.Testing
Risk Assessment
dynamo-platform), but scoped and has a documented opt-out.Rollout notes: Standing clusters upgrading in-place should either accept the NATS removal + rolling restart, or pass
--set global.nats.install=trueto defer it. Fresh installs unaffected. No AICR-side feature flag; this follows the upstream chart default.Checklist
make testwith-race) — pending real chart versionmake lint)git commit -S)