[PERF] Add CUDA IPC Ulysses overlap and lossless LingBot streaming

[lzx1413]

Description

This PR adds an optional, source-built CUDA IPC backend for same-host Ulysses scatter operations. It writes directly into cached peer target buffers on a high-priority CUDA communication stream and groups Q/K/V transfers behind one GPU-memory handshake, while retaining PyTorch/NCCL fallbacks for unsupported configurations.

MiniMax H3 now starts its strided V transfer before Q/K normalization and RoPE, then submits tagged Q and K transfers. Other eligible attention implementations also use tagged transfers without changing their public pipeline configuration.

The PR additionally adds lossless LingBot-World streaming delivery and hardens CUDA IPC/session teardown based on the four-H100 validation results.

Motivation

Ulysses attention previously relied on NCCL collectives and intermediate packing or relayout work. This serialized communication with Q/K preprocessing and prevented Copy Engine transfers from overlapping useful GPU computation.

Long-running LingBot sessions also needed an explicit delivery contract: target completion did not guarantee that every published frame had reached the client. Session cancellation could interrupt asynchronous cleanup, while multi-consumer CUDA IPC tensors shared one refcounter and could emit lifecycle warnings during shutdown.

This change improves communication overlap, preserves lossless frame delivery when requested, and makes IPC resource ownership and shutdown deterministic.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change
• Performance improvement
• Code refactoring
• Documentation update
• Other

Changes Made

• Added a source-built tf-kernel CUDA IPC Ulysses backend:
  • Direct peer writes into final-layout target buffers.
  • High-priority CUDA communication stream.
  • Up to 12 cached targets keyed by tag, shape, dtype, mode, and device.
  • One grouped stream-memory handshake for tagged Q/K/V transfers.
  • FP16/BF16, strided-source, target-reuse, stress, and explicit-close coverage.
• Integrated the backend through telefuser.distributed.ulysses_comm:
  • Lazy same-host capability detection.
  • Tagged Q/K/V scatter submissions.
  • NCCL fallback when the extension, peer access, or topology is unsupported.
  • NCCL remains in use for standalone collectives and output gather where faster.
• Overlapped model work with communication:
  • MiniMax H3 submits strided V before Q/K norm and RoPE.
  • Q and K are submitted afterward and share the grouped completion handshake.
  • Eligible LingBot, Wan, Qwen-Image, LiveAct, FlashVSR, and LingBot-Video attention paths use tagged transfers.
• Added lossless LingBot-World delivery:
  • FIFO backpressure for video and terminal outputs.
  • Published-frame completion accounting.
  • Reliable client delivery acknowledgement before LiveKit track teardown.
  • AIPerf support for validating declared versus received frames.
• Hardened IPC and session lifecycle:
  • Each tensor-channel consumer receives an independent CUDA IPC refcounter.
  • WorkerTensorChannel owns producer-pool and consumer-mapping cleanup.
  • Ulysses groups close before distributed process-group teardown.
  • Session stop waits for cooperative generator cleanup before cancellation.
  • Child workers use parent-coordinated shutdown, allowing one Ctrl-C to exit.
• Added correctness, parity, benchmark, profiler, and lifecycle tests.
• Updated English/Chinese parallel docs, MiniMax H3 measurements, LingBot instructions, and AIPerf results.

Testing

• Focused unit tests pass
• Manual four-GPU testing performed
• Benchmarks added and updated

Verification performed:

• ruff check
• 33 focused Ulysses/model unit tests
• 68 focused TeleFuser lossless-streaming tests
• 6 AIPerf adapter tests
• 97 focused IPC/session lifecycle unit tests
• 4-GPU tensor-channel integration tests: 3 passed
• 4-GPU CUDA IPC Ulysses correctness and stress test
• MiniMax H3 Ulysses2, Ulysses4, and TP2+Ulysses2 parity
• 4-H100 MiniMax H3 FlashAttention, SageAttention, and AdaTaylorCache benchmarks
• 1-minute 4-H100 LingBot-World v2 AIPerf replay: 1/1 session succeeded
• Single Ctrl-C service shutdown with no CUDA IPC lifecycle warning

Representative commands:

ruff check telefuser tests tools/validation tf-kernel/tf_kernel

pytest -q \
  tests/unit/distributed/test_ulysses_comm.py \
  tests/unit/models/test_minimax_h3_dit.py \
  tests/unit/models/test_lingbot_world_fast_dit.py \
  tests/unit/service/livekit \
  tests/unit/worker/test_parallel_worker.py \
  tests/unit/worker/test_tensor_channel.py

pytest -q -s tests/integration/test_minimax_h3_distributed.py
pytest -q -s tests/integration/test_worker_tensor_channel.py

cd tf-kernel
torchrun --standalone --nproc-per-node=4 \
  tests/distributed/ulysses_correctness.py

Benchmark and profiler commands are documented in:

• examples/minimax_h3/README.md
• examples/lingbot/README.md
• docs/en/benchmark_aiperf.md

Checklist

• Code follows the project's coding standards (ruff)
• Pre-commit hooks pass (pre-commit run --all-files)
• All tests pass (pytest tests/)
• New tests added for new functionality
• Documentation updated
• Commit messages are clear and descriptive
• PR title follows the convention: [TYPE] Brief description

Related Issues

N/A

Additional Notes

The optimized backend is optional and belongs to the separately built tf-kernel package. TeleFuser does not build or install it automatically. Missing kernels, multi-host process groups, compilation, or unsupported CUDA IPC configurations retain the existing NCCL behavior.

A grouped transfer cannot safely switch to NCCL after some tagged operations have already started; such failures are surfaced instead of mixing collective protocols mid-request.

The complete pytest tests/ and pre-commit run --all-files suites were not run. Focused suites covering the changed paths passed.

GPU Architecture Support

• SM80 (Ampere, Ada Lovelace)
• SM90 (Hopper H100)
• SM100+ (Blackwell)

The CUDA IPC implementation is not intentionally SM90-specific, but this PR was validated only on four H100 80 GB GPUs.

Performance Impact

Environment: 4 x H100 80 GB, PyTorch 2.11.0+cu128, CUDA 12.8, NCCL 2.28.9, source-built SM90 tf-kernel.

MiniMax H3

Configuration	Pipeline time	DiT time	Speedup
FlashAttention 4	77.32 s	74.64 s	1.00x
SageAttention SM90	72.41 s	69.85 s	1.07x
FlashAttention 4 + AdaTaylorCache	42.39 s	39.82 s	1.82x

SageAttention reduced pipeline latency by 6.34% and DiT latency by 6.43%. AdaTaylorCache reduced pipeline latency by 45.2%.

The CUDA trace confirmed 157.7 us of overlap between the strided V Copy Engine transfer and Q/K normalization plus RoPE preprocessing.

LingBot-World v2

Measurement	Result
Direct-path steady compute FPS, previous	17.14
Direct-path steady compute FPS, CUDA IPC Ulysses	19.08
One-minute AIPerf steady compute FPS	17.5824
Generated frames / chunks	957 / 60
Successful AIPerf sessions	1 / 1

The direct-path measurement improved by approximately 11.3% while preserving all requested output frames.