TeleFuser: Streaming Inference Framework for World Models and Multimodal Generation

中文 | English

TeleFuser is an open-source streaming inference and serving framework for real-time world models and multimodal generation. It supports continuous video generation, stateful sessions, bidirectional control, distributed GPU inference, FastAPI serving, and LiveKit WebRTC streaming for workloads such as interactive world models, speech-driven animation, and streaming visual systems.

See the world model streaming inference guide for the execution model, runtime path, supported workloads, and reproducible real-time gate.

News 📰

• ✨ 2026-08-05: Added MiniMax H3 T2VA, FL2VA, and Ref2VA joint audio-video generation with standard telefuser serve support. On the matched 768p, five-second, 50-step T2VA benchmark after warmup, the resident 4 x H100 80 GB profile matched pinned local SGLang SP2+TP2 performance while using less GPU memory; see the reproducible validation and performance notes.

• ✨ 2026-08-03: Validated LingBot-World v2 target-side real-time generation on 4 x H100 80 GB at 832x480 and 16 FPS. The current 77-frame gate reached 17.14 steady compute FPS; see the reproducible benchmark.

• ✨ 2026-07-27: Unified streaming on LiveKit with room sessions, retained multi-session admission, LingBot chunk-boundary time slicing, reconnect-friendly browser transport, and server-push/bidirectional contracts.

• ✨ 2026-07-22: Added LingBot-Video support for Dense and MoE T2I/T2V/TI2V generation, native four-GPU CFG/SP execution, and in-memory MoE refinement.

• ✨ 2026-07-15: Added LingBot-World v2 support for offline generation, interactive WebRTC streaming, and multi-GPU inference.

• ✨ 2026-07-06: Added external CacheSeek latent cache integration for service-mode cross-request reuse. Cache hits can skip the first N denoising steps; the Wan2.2 cache-enabled service example snapshots [5, 10, 15, 20, 25] by default. See docs/en/latent_cache.md.

Why TeleFuser

Most open-source inference stacks are optimized for one of three cases:

• one-shot image generation
• offline video generation
• general LLM serving

Real-time world models need a different runtime profile: continuous execution, streaming output, bidirectional interaction, stateful sessions, long-context efficiency, and stable performance under concurrency. TeleFuser focuses on those runtime problems directly.

The project treats a world model as more than a function that returns a single clip. It provides the infrastructure needed to run a model as a continuously updated system that can receive input, keep state, and emit frames progressively.

What TeleFuser Provides

• World-model-oriented runtime: Support for continuous video generation, interactive sessions, and bidirectional control loops.
• ADF (AI Dev First): Repository layers, pipeline contracts, examples, and docs are structured for coding agents to discover capabilities, follow project conventions, and extend pipelines efficiently.
• Streaming pipeline scheduler: Actor-owned stateful stages, bounded artifact edges, per-session ordering, backpressure, lifecycle cleanup, and explicit resource groups.
• Streaming transport: LiveKit-backed WebRTC for server-push media and resilient bidirectional sessions, with room lifecycle, reconnect handling, participant roles, and reliable controls.
• Scalable GPU runtime: Multi-GPU execution with tensor parallelism, sequence parallelism, optional Ray workers, and distributed service replicas.
• Inference optimization stack: Triton kernels, optimized attention backends, quantization, offload, feature caching, and CacheSeek latent cache integration.
• Unified serving: Local Python API, telefuser serve for task APIs, and telefuser stream-serve for LiveKit rooms and media.

Quick Start

Install

pip install -e .

For development:

pip install -e ".[dev]"

TeleFuser does not require tf-kernel to run. The project does not publish prebuilt tf-kernel wheels or a source distribution to a public package index. Build the optional extension with the Makefile under tf-kernel/; a locally built wheel may be distributed only to compatible environments. See the tf-kernel README and installation and usage guide for build, verification, and artifact compatibility details.

The base installation includes the LiveKit Python SDKs used by telefuser stream-serve. A LiveKit Cloud project or self-hosted LiveKit Server is operated separately.

1. Batch Video Inference

from telefuser.pipelines.wan_video.wan21_video import Wan21VideoPipeline
import torch

pipe = Wan21VideoPipeline.from_pretrained(
    model_id_or_path="Wan-AI/Wan2.1-T2V-1.3B",
    device="cuda",
    torch_dtype=torch.bfloat16,
)

video = pipe(
    prompt="A cat playing piano",
    num_frames=81,
    height=480,
    width=832,
)

2. Real-Time World Model WebRTC Demo

TeleFuser streams LingBot-World v2 through LiveKit. LingBot-World v2 uses camera control and its v2 PPL defaults; its streaming example caps a session at two minutes.

The validated four-H100 configuration sustains 17.14 target-side compute FPS for the default 77-frame, 832x480 request, above its 16 FPS playback target. This is a synchronized pipeline-compute metric; model loading, LiveKit encoding, network delivery, and client rendering are measured separately. See the LingBot example guide for the exact command and chunk timings.

LingBot streaming uses the actor-based scheduler for both offline and service execution. Encode, DiT, and decode may overlap even on the same GPU; move stages only when memory placement requires it. See the streaming scheduler guide.

The checked-in browser page forces a TCP TURN relay so the same setup works through VS Code Remote SSH. The complete local development stack therefore has four processes: coturn, LiveKit Server, TeleFuser, and the browser page. Install the LiveKit Server and your platform's coturn package once:

# Debian/Ubuntu; use the equivalent coturn package on other platforms.
sudo apt-get update
sudo apt-get install -y coturn

# Install LiveKit Server once.
curl -sSL https://get.livekit.io | bash

Then run each command below in a separate terminal from the repository root.

Terminal 1 — start the development-only TCP TURN relay:

turnserver -n -m 1 \
  --listening-ip=127.0.0.1 --relay-ip=127.0.0.1 \
  --listening-port=3478 --min-port=49160 --max-port=49200 \
  --user=livekit-demo:livekit-demo-password \
  --realm=livekit.local --fingerprint --lt-cred-mech \
  --no-tls --no-dtls --no-cli --allow-loopback-peers

Terminal 2 — start LiveKit with its development credentials (devkey / secret):

livekit-server --dev

Terminal 3 — load the four-GPU LingBot-World v2 service:

TF_MODEL_ZOO_PATH=/path/to/model_zoo \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
telefuser stream-serve examples/lingbot/lingbot_world_v2_image_to_video_h100.py \
  --livekit-url ws://127.0.0.1:7880 \
  --livekit-api-key devkey --livekit-api-secret secret \
  --num-workers 1 --worker-gpu-map 0,1,2,3 \
  --max-sessions-per-worker 2 --control-idle-timeout 10 \
  --port 8088 --skip-validation

This is one four-GPU model worker and one loaded LingBot service instance, not four replicas. It can retain two independent user sessions; the shared LingBot execution lease runs at most one session chunk at a time and yields at a chunk boundary after the active controller becomes idle while another session waits.

Terminal 4 — serve the browser controller and proxy its session API:

python examples/stream_server/livekit_bidirectional_demo.py \
  --server-url http://127.0.0.1:8088 --port 8092 --no-open

For VS Code Remote SSH, forward remote TCP ports 8092, 7880, and 3478 to the same local ports; 8088 does not need forwarding because the page proxies the TeleFuser API. Open http://127.0.0.1:8092, select an initial image, click Start, and use the on-page controls or W/A/S/D and arrow keys. A successful connection shows a video track plus control_state, generation-stage, and chunk status messages.

Check the server independently with curl http://127.0.0.1:8088/v1/service/health. To stop the stack, stop the browser session or close the page first, then press Ctrl+C in terminals 4, 3, 2, and 1. These loopback addresses, static credentials, disabled TURN TLS, and --allow-loopback-peers are for trusted development only. See the stream server guide for LiveKit Cloud, production networking, session APIs, and troubleshooting.

3. Batch Service Mode

telefuser serve examples/wan_video/wan22_14b_text_to_video_h100.py --task t2v --port 8000

TeleFuser exposes:

• native task APIs under /v1/tasks/*
• OpenAI-compatible image and video APIs under /v1/images and /v1/videos
• service metadata that reflects the pipeline contract

See docs/en/service.md for full API details.

Architecture

TeleFuser uses a layered runtime architecture that maps cleanly to the repository structure:

1. Access layer: FastAPI task APIs and LiveKit-backed stream room/session entrypoints.
2. Service layer: request routing, task management, stream sessions, replica pools, and integration with pipeline execution.
3. Pipeline abstraction layer: model-specific BasePipeline / BaseStage components, with an actor-based streaming orchestrator for bounded dataflow, session ordering, metrics, and cleanup.
4. Model and optimization layer: model loading, attention selection, quantization, offload, LoRA, and cache integration.
5. Execution backend layer: optimized ops, Triton kernels, and device-specific implementations.

Relevant directories:

telefuser/
├── service/         # REST APIs and LiveKit-backed streaming
├── orchestrator/    # Request orchestration and actor-based streaming scheduler
├── pipelines/       # Model-specific pipelines
├── distributed/     # TP / SP / FSDP / Ray utilities
├── feature_cache/   # AdaTaylorCache
├── ops/             # Compile-aware operator dispatch
├── kernel/triton/   # Triton kernels
└── models/          # DiT, VAE, encoders, decoders

Supported Pipelines

World Model and Real-Time Oriented

Pipeline	Task	Notes
LingBot-World v2	Bidirectional world-model streaming	LiveKit control loop via examples/lingbot/lingbot_world_v2_image_to_video_h100.py
LiveAct	S2V	Speech-driven talking head generation via examples/liveact/liveact_s2v_h100.py
FlashVSR	VSR	Streaming video super-resolution via examples/flashvsr/README.md

Video Generation

Pipeline	Task	Notes
WanVideo (Wan2.1 / Wan2.2)	T2V, I2V, FL2V	Main video generation family, including async and service examples in examples/wan_video/README.md
HunyuanVideo	T2V, I2V	Supported via examples/hunyuan_video/README.md
LTX Video	I2V + Audio	Unified audio-video generation via examples/ltx_video/README.md
MiniMax H3	T2VA, FL2VA, Ref2VA + Audio	Local 768p joint audio-video generation via examples/minimax_h3/README.md
LongCat-Video	T2V, I2V, VC	Long-form generation and continuation via examples/longcat_video/README.md
NEW LingBot-Video	T2I, T2V, TI2V, MoE refiner	Dense/MoE generation with native CFG/SP and an in-memory base-to-refiner path; see examples/lingbot_video/README.md

Image Generation and Other Multimodal Pipelines

Pipeline	Task	Notes
Qwen-Image	T2I, Edit	examples/qwen_image/README.md
Z-Image	T2I	examples/z_image/README.md
Flux2 Klein	T2I	examples/flux2_klein/README.md

See examples/README.md for the example runner and baseline comparison workflow.

Documentation

• docs/en/service.md: REST serving, task APIs, OpenAI-compatible APIs
• docs/en/stream_server.md: LiveKit streaming, session APIs, data topics, and deployment
• docs/en/stream_scheduler.md: actor-based stage scheduling, backpressure, lifecycle, metrics, and LingBot placement
• docs/en/parallel.md: distributed inference architecture
• docs/en/communication.md: collectives, CUDA IPC, synchronization, and transport lifecycle
• docs/en/latent_cache.md: CacheSeek latent cache integration
• docs/en/feature_cache.md: AdaTaylorCache
• docs/en/model_loading.md: model loading patterns
• docs/en/attention.md: attention backends and configuration
• docs/en/torch_compile_compatibility.md: compile-related constraints
• docs/en/adding_new_model.md: integrating new models
• docs/en/adding_new_example.md: authoring examples and pipeline contracts

Known Limitations

• AdaTaylorCache is only calibrated for selected model families.
• torch.compile support is still experimental in parts of the stack.
• Some optimized paths require specific GPU architectures and CUDA versions.
• World-model examples such as LingBot-World v2 require external checkpoints and environment setup.
• Multi-machine deployment exists in the architecture but may require project-specific integration and validation.

Development

pip install -e ".[dev]"
pre-commit install
pytest tests/

See CONTRIBUTING.md for contribution workflow and AGENTS.md for project-specific agent guidance.

License

Apache 2.0 License. See LICENSE.

Acknowledgements

TeleFuser builds on and is inspired by a broad set of open-source efforts in multimodal generation and inference systems, including:

• DiffSynth-Studio
• DiffSynth-Engine
• LightX2V
• cache-dit
• diffusers
• Wan2.1 / Wan2.2
• Qwen-Image
• Z-Image
• FlashVSR