Skip to content

docs(vllm-omni): refresh for v1.5 (0.26.0) — changelog, bidi streaming, s3tokenizer - #6510

Open
Yadan-Wei wants to merge 1 commit into
mainfrom
auto/vllm-omni-doc-update/20260807-v1.5-bidi
Open

docs(vllm-omni): refresh for v1.5 (0.26.0) — changelog, bidi streaming, s3tokenizer#6510
Yadan-Wei wants to merge 1 commit into
mainfrom
auto/vllm-omni-doc-update/20260807-v1.5-bidi

Conversation

@Yadan-Wei

Copy link
Copy Markdown
Contributor

Summary

Refreshes the vLLM-Omni AL2023 DLC docs for the v1.5 / vLLM-Omni 0.26.0 release and documents the new SageMaker bidirectional WebSocket streaming feature. Generated by the vllm-omni-doc-update audit skill (drift + fact-check + coverage-gap), baseline v1.4.0 / omni v0.21.0rc1.

Changes

  • changelog/index.md — new v1.5.0 entry (vLLM-Omni 0.26.0 / vLLM v0.26.0): Rust frontend (vllm-rs), SageMaker bidirectional WebSocket streaming, s3tokenizer/CosyVoice3 fix, FlashInfer 0.6.14.
  • deployment/sagemaker.md — new Bidirectional Streaming (WebSocket) section: the capability LABEL, the 6 supported WS routes (matched against WS_ALLOWED_PATHS), slashless model_invocation_path, and the inference_ami_version requirement.
  • index.md — added s3tokenizer to "What's Included".
  • examples/vllm-omni/bidi/deploy_bidi_stream.py — new runnable end-to-end example (deploy → stream TTS over the bidi WebSocket → teardown).

Verification

The bidi example was verified end-to-end against the released prod image (vllm:omni-sagemaker-cuda-v1): deployed a real SageMaker endpoint, streamed 253 KB of audio back over the bidirectional WebSocket, and tore down cleanly. Re-verified under profile-only credentials (no env-var creds) to confirm the credential bootstrap path works for profile/SSO/role users, not just env-var users.

Doc claims were fact-checked against the code:

  • 6 WS routes match WS_ALLOWED_PATHS in scripts/docker/vllm/omni_sagemaker_serve.py
  • Capability LABEL matches docker/vllm_omni/Dockerfile.amzn2023
  • Example request/response shape matches test/vllm-omni/sagemaker/test_sm_omni_bidi_endpoint.py
  • Changelog version facts match .github/config/image/vllm-omni/sagemaker-amzn2023.yml

Punch list for a human (NOT addressed here)

  • [HIGH — likely a code bug, not a doc fix] EC2 port mismatch. Every example in docs/vllm-omni/deployment/ec2.md uses docker run -p 8080:8080 + curl localhost:8080, but the EC2 entrypoint scripts/docker/vllm/omni_dockerd_entrypoint.sh is exec vllm serve --omni "$@" with no --port, so the container binds vLLM's default 8000. SageMaker sets --port 8080 explicitly; EC2 does not. Either add --port 8080 to the EC2 entrypoint (recommended — makes the docs correct as written) or change every EC2 doc example to -p 8080:8000. Left for an owner to decide since it touches the entrypoint.
  • [low] Floating image tag. A couple of doc spots reference vllm:omni-cuda (unversioned) vs the config's vllm:omni-cuda-v1; likely an intentional rolling tag — flagging only.

Test plan

  • Verify rendered docs after merge
  • Confirm pre-commit run --all-files passes (passed locally)
  • Decide on the EC2 port punch-list item

…g, s3tokenizer

- Add v1.5.0 changelog entry (vLLM-Omni 0.26.0 / vLLM v0.26.0): Rust
  frontend, SageMaker bidirectional WebSocket streaming, s3tokenizer/CosyVoice3,
  FlashInfer 0.6.14.
- Document SageMaker Bidirectional Streaming (WebSocket) in deployment/sagemaker.md:
  capability LABEL, the 6 WS routes, slashless model_invocation_path,
  inference_ami_version requirement.
- Add a runnable, end-to-end-verified bidi example
  (examples/vllm-omni/bidi/deploy_bidi_stream.py) — deploy → stream TTS over the
  bidi WebSocket → teardown. Verified against the released prod image
  (219 KB audio streamed back).
- Add s3tokenizer to "What's Included".

Auto-generated by skill: vllm-omni-doc-update (coverage-gap example verified
end-to-end via verify_examples).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant