feat: add dynamo single-node example (frontend + workers, Maru KV sharing) - #77
Draft
seohui-XCENA wants to merge 1 commit into
Draft
feat: add dynamo single-node example (frontend + workers, Maru KV sharing)#77seohui-XCENA wants to merge 1 commit into
seohui-XCENA wants to merge 1 commit into
Conversation
…ring) examples/dynamo/single_node/ runs the full serving-stack topology on one node: an OpenAI-compatible dynamo.frontend (--router-mode direct) in front of two dynamo.vllm workers sharing KV through one Maru CXL pool. The test pins a long prompt to worker A (cold prefill, KV stored), then to worker B, which retrieves A's chunks instead of recomputing — verified by worker B's 'External prefix cache hit rate' going to ~70% while A stays at 0% (prefix caching disabled, Maru is the only cache). Operational lessons encoded in the scripts and README: - frontend starts before any worker (file-discovery generate leases expire in ~10s without a watching frontend) - workers start sequentially (concurrent starts race vLLM's memory profiling when sharing a GPU: the second computes a negative KV budget) - readiness is probed with a direct-routed 1-token completion (/health flaps 503 after registration; untargeted requests are 400 in direct router mode) - DISCOVERY_BACKEND=file (default, zero dependencies) or etcd; the file backend's mtime-lease + inotify implementation can intermittently miss or drop a worker's generate registration, so workers get one automatic retry and README documents the etcd alternative for reliability E2E validated over etcd discovery (0.5B model, both workers on one GPU): identical answers from both workers, cross-instance hit counter 69.6%, clean teardown. Note: requires the vLLM-0.23 paged-KV axis-order fix in the connector (xcena-dev#72) — on current main the store path crashes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
examples/dynamo/single_node/: one node, one OpenAI-compatibledynamo.frontend(--router-mode direct) in front of twodynamo.vllmworkers withMaruKVConnector, sharing KV through one Maru CXL pool.single_node_example.shbrings the whole stack up in the safe order, runs a cross-instance sharing test (a long prompt direct-routed to worker A, then to worker B, which retrieves A's chunks from the pool instead of recomputing them), and tears everything down.Operational lessons (encoded in the scripts and README)
generateleases expire in ~10s without a watching frontend, after which every request is503 Model not ready, forever./healthflaps 503 after registration, and untargeted requests are 400 in direct router mode.DISCOVERY_BACKEND=etcdis documented as the reliable alternative (real leases, lossless watches).Note
The example requires the vLLM-0.23 paged-KV axis-order fix in the connector (#72) — on current
mainthe store path crashes with a gather-index assert. It runs cleanly once #72 lands.Test plan
External prefix cache hit rate: 69.6%(worker A: 0.0%; prefix caching disabled, so Maru is the only cache source); clean teardown.