Skip to content

feat: add optional FunASR/SenseVoice ASR backend#584

Open
LauraGPT wants to merge 2 commits into
Huanshere:mainfrom
LauraGPT:codex/videolingo-funasr-provider-20260714
Open

feat: add optional FunASR/SenseVoice ASR backend#584
LauraGPT wants to merge 2 commits into
Huanshere:mainfrom
LauraGPT:codex/videolingo-funasr-provider-20260714

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • add an opt-in local FunASR backend using iic/SenseVoiceSmall by default
  • preserve VideoLingo's WhisperX-style result contract, including native word timestamps, clip offsets, model caching, and serialized inference per model/device
  • split native and fallback tokens longer than 30 characters with proportional timestamps so the downstream subtitle processor cannot silently discard them
  • wire the backend into the ASR dispatcher, Streamlit settings, launcher preflight, resumable installer, configuration, and documentation
  • keep WhisperX as the default; installing FunASR does not select it automatically

Validation

  • TDD regression: native and fallback 61-character tokens both produced zero downstream rows before the fix; both now preserve the complete text in ordered chunks of at most 30 characters
  • python -m unittest discover -v (15 passed; previously discovered 0 tests)
  • python -m unittest discover -s tests -v (15 passed)
  • real CPU SenseVoice transcription through transcribe_audio() on a Chinese MP3 fixture (1 segment, 10 timestamped words, expected non-empty text)
  • Black check on the changed Python implementation and tests
  • compileall on core, tests, installer.py, and st.py
  • git diff --check

This repository currently has no GitHub Actions workflow, so the reproducible local commands above are the available automated verification for this head.

Scope

The initial UI exposes Chinese, English, and Japanese, matching VideoLingo's current downstream language handling. The model field remains configurable for compatible FunASR models. Streaming, server mode, and speaker diarization are not part of this local provider.

The default SenseVoiceSmall weights are downloaded separately and are not covered by VideoLingo's Apache 2.0 source license. The README now links the model card and FunASR model license and explains that users must select the FunASR runtime after installation.

Closes #564

@LauraGPT

Copy link
Copy Markdown
Author

Fresh independent validation on exact head 97c36dd:

  • FunASR 1.3.14 on CPU, using the same SenseVoice/VAD arguments as this branch, returned key, text, timestamp, and words with 5 native words and 5 timestamp pairs.
  • Feeding that raw result through the branch adapter produced all 5 downstream subtitle rows, preserved the cleaned text, and applied the outer clip offset exactly (20.98s to 25.96s).
  • Both unittest discovery modes pass (15/15 each); Black on the new implementation/tests, Ruff, compileall, and git diff --check also pass.

I found no additional FunASR-side blocker in this pass.

@LauraGPT

Copy link
Copy Markdown
Author

Current-head maintenance check for 97c36dd8632a7103e617476e0bc2beecf60c7420:

  • GitHub compare is still 2 ahead / 0 behind against Huanshere/VideoLingo:main; no sync needed.
  • Fresh focused validation on Python 3.12: python -m pytest tests/test_funasr_local.py -q -> 15 passed.
  • python -m py_compile on the touched implementation/wiring/tests passes.
  • python -m compileall -q core/asr_backend/funasr_local.py tests/test_funasr_local.py passes.
  • git diff --check passes.

No new FunASR-side blocker found in this pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Add FunASR as an optional transcription backend

1 participant