Skip to content

[Plugin] Add livekit-plugins-funasr for self-hosted STT #5897

Description

@LauraGPT

Implementation status

PR #6176 adds livekit-plugins-funasr as an optional local STT plugin.

The current implementation is intentionally narrow:

  • direct local inference with iic/SenseVoiceSmall;
  • batch STT.recognize integration using LiveKit audio frames;
  • 16-bit PCM conversion and resampling to 16 kHz;
  • cleaned transcription text with language detection;
  • an asyncio.Lock around shared model inference;
  • the livekit-agents[funasr] optional dependency extra and complete Torch/Torchaudio runtime dependencies.

SenseVoiceSmall supports Mandarin, Cantonese, English, Japanese, and Korean. It can also emit speech-emotion and audio-event tags; the plugin returns clean transcription text and does not present nospeech as a language. Model weights follow the license linked from the model card.

Validation

The current PR head has been verified with:

  • three focused FunASR plugin tests;
  • Ruff, formatting, and mypy across the repository;
  • wheel and sdist builds;
  • standalone Python 3.10 dependency resolution;
  • real SenseVoiceSmall inference through STT.recognize;
  • GitHub CI with no failures.

Whisper and other LiveKit STT plugins remain unchanged. Streaming, diarization, and a remote server mode are outside the scope of this first plugin.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions