Add livekit-plugins-funasr (FunASR/SenseVoice local STT)

[LauraGPT]

This PR adds livekit-plugins-funasr, a fully local speech-to-text plugin backed by FunASR and SenseVoice.

Why

SenseVoice is a non-autoregressive multilingual ASR model with strong Chinese and Cantonese support. It runs on-device without an API key and gives LiveKit agents a local alternative for Chinese, Cantonese, English, Japanese, and Korean speech.

What

• Adds FunASRSTT as a non-streaming LiveKit STT implementation.
• Combines incoming audio, resamples it to 16 kHz, converts stereo to mono, runs FunASR off the event loop, strips SenseVoice rich tags, and returns the final transcript plus detected language.
• Supports configurable model, device, language, and inverse text normalization.
• Serializes model.generate calls because a shared FunASR model is not guaranteed to be thread-safe.
• Registers the package in the root workspace, livekit-agents[funasr], and uv.lock.
• Declares the complete runtime dependencies, including both PyTorch and torchaudio.
• Includes package-import, resampler, and non-retryable local-error regression tests.

Usage

from livekit.plugins import funasr

stt = funasr.STT(model="iic/SenseVoiceSmall")

CPU is the safe default. For GPU inference, install a PyTorch and torchaudio build matching the host CUDA runtime and pass device="cuda".

Verification

Validated on exact head 666d54029d after merging LiveKit main through ecc97f2f51:

• GitHub recomputed the PR from DIRTY to mergeable; current main later advanced to 845fc28371, and git merge-tree --write-tree HEAD upstream/main still exits 0 with a clean synthetic merge
• the stable patch-id of every non-generated PR file is unchanged from the previously tested head
• uv lock --check and uv sync --all-extras --dev --frozen pass; the regenerated lock retains the upstream mcp 1.28.1 and nltk 3.10.0 security updates plus the complete FunASR dependency graph
• FunASR, AssemblyAI, and FishAudio affected suites pass 121/121
• full repository Ruff check and format check pass; 856 files are formatted
• full type check passes with no issues across 622 source files
• the 1.6.5 wheel and source distribution build successfully; wheel metadata exposes FunASR, LiveKit Agents, NumPy, PyTorch, and torchaudio as runtime requirements
• real SenseVoiceSmall CPU inference through livekit.plugins.funasr.STT.recognize transcribes the official sample as 欢迎大家来体验达摩院推出的语音识别模型。, reports language zh, and completes recognition in 0.234 seconds after model load

[LauraGPT]

Thanks for the review! Addressed the findings:

• Added the py.typed marker (PEP 561), so the package is type-checkable.
• model property now returns the configured model id instead of a hardcoded string.
• Excluded the nospeech classification label from the reported detected language (it is not a language).
• Serialized model.generate with an asyncio.Lock for thread-safety across concurrent recognitions sharing one instance.

CI is green (ruff + type-check). The transcription core (16-bit PCM -> resample to 16k -> SenseVoice -> cleaned text) was verified locally.

[LauraGPT]

Fresh FunASR/SenseVoice-side validation recheck (2026-07-07 UTC):

• PR is open, non-draft, and mergeable at head 348c46f; mergeable_state=blocked is currently the review gate.
• Thread-aware review audit shows 6 total review threads and 0 current unresolved threads. The previous AudioResampler(..., num_channels=...) discussion is resolved.
• GitHub checks visible on the head are green/skipped as expected: ruff, type-check (3.10), type-check (3.13), unit-tests, platform blockguard tests, release gate, Devin Review, and license/cla are successful; matrix/evaluation plugin jobs are skipped.
• Local focused validation on the current PR checkout:
  • git diff --check upstream/main...HEAD passed.
  • python3 -m py_compile livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/stt.py livekit-plugins/livekit-plugins-funasr/livekit/plugins/funasr/__init__.py tests/test_plugin_funasr_stt.py passed.
  • livekit-plugins/livekit-plugins-funasr/pyproject.toml parses; package name is livekit-plugins-funasr with dependencies livekit-agents>=1.2.0, funasr>=1.1.0, and numpy.

Caveat: I did not rerun the full local pytest target in the bare ops checkout because the monorepo runtime package dependencies (livekit.api etc.) are not installed there; GitHub's own unit/type/ruff checks have already run successfully on this head.

[LauraGPT]

Updated this PR to current main and completed the packaging/runtime pass at afab30a:

• the branch is 0 commits behind and the plugin now follows the 1.6.5 release line;
• added livekit-agents[funasr], a regenerated uv.lock, complete PyTorch/torchaudio runtime dependencies, and a real package-import regression;
• local verification passes: 3 FunASR tests, full Ruff/format, mypy across 622 files, wheel/sdist build, and standalone Python 3.10 dependency resolution;
• real SenseVoiceSmall inference through STT.recognize returns the expected Chinese transcript;
• GitHub CI is complete with 9 successes, 3 expected fork skips, and no failures or pending jobs.

@tinalenguyen, would you be able to review the new local STT plugin when convenient?

[LauraGPT]

Synced this PR to current main; exact head cf327cc3c53ac1fad921fd746f745660152e6d99 is 0 commits behind and mergeable. The merge changed none of the FunASR implementation or test files.

Fresh local verification passes the 3 focused plugin regressions, Ruff and format checks, full mypy across 622 source files, uv lock --check, wheel/sdist build, and git diff --check. GitHub has also completed with all visible checks green (plus the expected fork skips) and no unresolved review threads.

@tinalenguyen, could you review the local FunASR/SenseVoice STT plugin when convenient?

[LauraGPT]

Refreshed this PR on top of current livekit/agents@e731eac and pushed head ef94cb81475c350ad6daf5119426d72082b27920.

What changed in the refresh:

• resolved the livekit-agents/pyproject.toml optional-dependency conflict by keeping the upstream >=1.6.6 plugin baseline and preserving the new funasr extra
• aligned livekit-plugins-funasr with the current release baseline: livekit-agents>=1.6.6, funasr>=1.3.16, and numba>=0.61.0 so the lock does not resolve an old Python-incompatible numba stack
• regenerated uv.lock; uv lock --check now passes

Local validation on the refreshed head:

• python -m pytest tests/test_plugin_funasr_stt.py -q -> 3 passed
• python -m pytest tests/test_plugin_funasr_stt.py tests/test_stt_context.py tests/test_inference_stt_context.py tests/test_plugin_assemblyai_stt.py -q -> 157 passed
• ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
• git diff --check

GitHub now reports the PR mergeable again; repository CI is running on the refreshed head.

[LauraGPT]

Current-head verification for e07dd96bae706670f38ac3353127e0c2168e93d7:

• Rechecked the Devin threads with GraphQL: all review threads are resolved. The latest version-mismatch finding is fixed: livekit-agents[funasr] now requires livekit-plugins-funasr>=1.6.6, and livekit/plugins/funasr/version.py reports __version__ = "1.6.6".
• The py.typed marker exists, and local inference failures are raised as APIConnectionError(..., retryable=False).
• GitHub check rollup currently has no bad or pending checks; the PR is mergeable and only waiting for review.

Fresh focused validation on ind-gpu8:

.venv/bin/python -m pytest tests/test_plugin_funasr_stt.py -q
# 3 passed, 4 warnings

.venv/bin/ruff check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# All checks passed!

.venv/bin/ruff format --check livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
# 5 files already formatted

/cpfs_speech/user/zhifu.gzf/venv_uv_bootstrap/bin/uv lock --check
# exit 0, resolved 359 packages

python -m compileall -q livekit-plugins/livekit-plugins-funasr tests/test_plugin_funasr_stt.py
git diff --check upstream/main...HEAD