Skip to content

feat: add dynamo single-node example (frontend + workers, Maru KV sharing) - #77

Draft
seohui-XCENA wants to merge 1 commit into
xcena-dev:mainfrom
seohui-XCENA:feat/dynamo-single-node-example
Draft

feat: add dynamo single-node example (frontend + workers, Maru KV sharing)#77
seohui-XCENA wants to merge 1 commit into
xcena-dev:mainfrom
seohui-XCENA:feat/dynamo-single-node-example

Conversation

@seohui-XCENA

Copy link
Copy Markdown
Collaborator

Summary

Adds examples/dynamo/single_node/: one node, one OpenAI-compatible dynamo.frontend (--router-mode direct) in front of two dynamo.vllm workers with MaruKVConnector, sharing KV through one Maru CXL pool. single_node_example.sh brings the whole stack up in the safe order, runs a cross-instance sharing test (a long prompt direct-routed to worker A, then to worker B, which retrieves A's chunks from the pool instead of recomputing them), and tears everything down.

Operational lessons (encoded in the scripts and README)

  • Frontend first. The frontend must watch discovery before any worker registers — file-discovery generate leases expire in ~10s without a watching frontend, after which every request is 503 Model not ready, forever.
  • Workers start sequentially when sharing a GPU: concurrent vLLM memory profiling races to a negative KV budget.
  • Readiness is probed with a direct-routed 1-token completion/health flaps 503 after registration, and untargeted requests are 400 in direct router mode.
  • The file discovery backend can intermittently miss or drop a worker's registration, so each worker gets one automatic retry, and DISCOVERY_BACKEND=etcd is documented as the reliable alternative (real leases, lossless watches).

Note

The example requires the vLLM-0.23 paged-KV axis-order fix in the connector (#72) — on current main the store path crashes with a gather-index assert. It runs cleanly once #72 lands.

Test plan

  • E2E (0.5B model, both workers on one GPU, etcd discovery, fix(maru_vllm): adapt LMCache kernel integration to relocated KV-type enums #72 connector applied locally): both workers registered and served through the frontend, identical answers from both, and the cross-instance retrieve confirmed by worker B's External prefix cache hit rate: 69.6% (worker A: 0.0%; prefix caching disabled, so Maru is the only cache source); clean teardown.

…ring)

examples/dynamo/single_node/ runs the full serving-stack topology on one
node: an OpenAI-compatible dynamo.frontend (--router-mode direct) in
front of two dynamo.vllm workers sharing KV through one Maru CXL pool.
The test pins a long prompt to worker A (cold prefill, KV stored), then
to worker B, which retrieves A's chunks instead of recomputing —
verified by worker B's 'External prefix cache hit rate' going to ~70%
while A stays at 0% (prefix caching disabled, Maru is the only cache).

Operational lessons encoded in the scripts and README:
- frontend starts before any worker (file-discovery generate leases
  expire in ~10s without a watching frontend)
- workers start sequentially (concurrent starts race vLLM's memory
  profiling when sharing a GPU: the second computes a negative KV budget)
- readiness is probed with a direct-routed 1-token completion (/health
  flaps 503 after registration; untargeted requests are 400 in direct
  router mode)
- DISCOVERY_BACKEND=file (default, zero dependencies) or etcd; the file
  backend's mtime-lease + inotify implementation can intermittently miss
  or drop a worker's generate registration, so workers get one automatic
  retry and README documents the etcd alternative for reliability

E2E validated over etcd discovery (0.5B model, both workers on one GPU):
identical answers from both workers, cross-instance hit counter 69.6%,
clean teardown. Note: requires the vLLM-0.23 paged-KV axis-order fix in
the connector (xcena-dev#72) — on current main the store path crashes.
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant