feat(asr): add SenseVoice-Small on-device ASR for Cantonese#579
feat(asr): add SenseVoice-Small on-device ASR for Cantonese#579hoishing wants to merge 7 commits into
Conversation
Introduce SettingsStore.SenseVoiceLanguage, the language choice for the SenseVoiceSmall model. - Cases: cantonese (default), mandarin, auto. - embedIndex maps each case to the model's lid_int_dict language embedding input (auto=0, mandarin/zh=3, cantonese/yue=13). - displayName provides localized labels for the settings picker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit c70bd973dbe7e80d9d552b6715658ddd48bef144)
Add SenseVoiceSmall as a selectable on-device ASR model with strong Cantonese accuracy, running on the Apple Neural Engine via CoreML. SenseVoiceProvider.swift (new): - Vendors the 3-stage CoreML pipeline + greedy-CTC decode from FluidInference/FluidAudio (Apache-2.0), since the pinned FluidAudio fork omits the SenseVoice module. Core inference runs in a SenseVoiceEngine actor off the main thread. - Downloads the int8 encoder variant from FluidInference/sensevoice-small-coreml via the app's HuggingFaceModelDownloader, reusing its HTML/markup corrupt-cache detection. - Reads the selected language (SettingsStore.senseVoiceLanguage) per call. - Chunks audio longer than one encoder pass (~100 s cap): splits at the quietest sample near the cap to avoid mid-word cuts, transcribes each chunk, and joins the pieces so long recordings are fully transcribed. - Intel builds get a not-available stub. SettingsStore.swift: - Add the .senseVoiceSmall SpeechModel case with its metadata (display name, download size, accuracy/speed, FunASR provider, brand color) across the exhaustive switches; mark it Apple-Silicon-only and non-streaming. - isInstalled checks the provider's on-disk artifacts on arm64. - Add the persisted senseVoiceLanguage setting (defaults to Cantonese). ASRService.swift: - Route .senseVoiceSmall to a cached SenseVoiceProvider in both the active and download-only provider factories, and clear it on reset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 18ae6cd)
Show a language menu (Cantonese / Mandarin / Auto Detect) in the voice engine settings when the SenseVoice Small model is selected, mirroring the existing per-model language chip. Selecting a language updates SettingsStore.senseVoiceLanguage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 0d8d67e6080303dede2fea6ae25526f9bc24cd9a)
artifactIsComplete required metadata.json for every .mlmodelc, but valid compiled bundles can omit it — the SenseVoice export ships coremldata.bin + model.mil + weights/weight.bin and no metadata.json. That made ensureModelsPresent throw "artifacts incomplete" after a fully successful download and left SpeechModel.isInstalled permanently false. Require only the load-critical files (coremldata.bin + weights/weight.bin); metadata.json and model.mil are descriptive and now optional. No regression for existing bundles (Parakeet etc.), which already contain both required files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 38e6eea47af64d122235eb4e485e47d09172fdc8)
The CoreML preprocessor constrains its waveform input to 3200..480000 samples (0.2..30 s), rejecting anything longer. The chunk cap was 1,600,000 samples (~100 s), aimed at the encoder's ~108 s frame limit, so any recording over 30 s crashed with an out-of-range input error. - Lower maxChunkSamples to 460,000 (28.75 s), safely below 480,000; the single-pass path now only ever sees sub-limit audio and long recordings chunk correctly. - Zero-pad waveforms below the 3200-sample floor so very short recordings and small trailing chunks don't trip the same input constraint. Verified end-to-end against the int8 CoreML model: 28 s single-pass and a 181 s recording (7 chunks) both transcribe fully with no crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit dcfbb23e347f13af5934a68ebd357c85c5e145d4)
The Cantonese embedIndex was 13, but 13 is the model's <|nospeech|> control token; the correct <|yue|> embedding index is 7. Verified against the model's vocab.json (token <|yue|> maps to embed index 7 in lid_int_dict). Feeding 13 still transcribed clear Cantonese because the acoustic model dominates for clean audio, but it passed "nospeech" as the language hint — wrong semantics that could suppress or degrade output on noisy/borderline speech. Mandarin (3) and auto (0) were already correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit efeb959e70c3537e5a3e9d4ca2c3b212410a071a)
|
The PR Policy check is blocking this PR because required template information is missing. Please update the PR description with:
Visual files detected:
Screenshots or video are required for UI, UX, settings, onboarding, overlay, menu bar, or visual behavior changes. If this PR has no visual changes, check the no-visual-change box in the template. If this remains incomplete for 48 hours after opening, the PR may be closed. |
github-actions
Bot
added
needs PR template
Greptile SummaryThis PR adds SenseVoice-Small (FunASR) as a selectable on-device ASR model, targeting Cantonese transcription where existing Parakeet/Whisper options are weak. The CoreML pipeline (preprocessor → int8 encoder → greedy CTC) is vendored from FluidInference/FluidAudio under Apache-2.0, pinned to a specific commit SHA, and wired into the existing provider/download infrastructure.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
@claude update the PR with this screenshot
|
github-actions
Bot
removed
needs PR template
… pinned model Resolve three review findings on the SenseVoice transcription path: - Chunk join no longer inserts a literal space at every silence cut. A content-aware `joinChunks` adds a separator only at Latin-script boundaries; CJK (Cantonese/Mandarin/Japanese/Korean) chunks are concatenated directly, so long recordings no longer produce output like `喺度 做嘢`. - transcribe() returns the trimmed text (not the raw string), matching the confidence check and the other providers' contract. - Pin the model download to a specific commit (cdea3526163035c19915d4a10268992d018ebd46) instead of `main`, so an upstream re-upload can't silently ship a different/incompatible encoder to fresh installs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the review — all three addressed in 16ce3cd:
Release build passes; the screenshot in the description shows the language picker. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 16ce3cd550
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let end = Self.chunkEnd(in: samples, offset: offset) | ||
| let chunk = Array(samples[offset..<end]) | ||
| let piece = try await engine.transcribe(audio: chunk, language: language) |
There was a problem hiding this comment.
For recordings only slightly longer than maxChunkSamples (for example, about 29s at 16 kHz), chunkEnd can cut near 460000 and the next loop sends the remaining <1s tail through SenseVoice, despite the minChunkSamples guard. Because runPreprocessor pads short inputs with silence before decoding, that tiny trailing sliver can produce spurious text appended to an otherwise good transcript; choose a cut that leaves at least minChunkSamples, or merge/drop the short tail.
Useful? React with 👍 / 👎.
1 participant
Description
Adds SenseVoice-Small (FunASR) as a selectable on-device ASR model, primarily to provide strong Cantonese transcription (the current Parakeet/Whisper options are weak on Cantonese). SenseVoice-Small is non-autoregressive, runs on the Apple Neural Engine via CoreML, uses ~0.3–0.5 GB RAM, and transcribes far faster than real time.
What's included:
SenseVoiceProvider— the 3-stage CoreML pipeline (preprocessor → int8 encoder+CTC → greedy-CTC decode) vendored from FluidInference/FluidAudio (Apache-2.0), since the pinned FluidAudio fork does not ship the SenseVoice module. Core inference runs in an actor off the main thread. Models are downloaded fromFluidInference/sensevoice-small-coremlvia the existingHuggingFaceModelDownloader. Intel builds get a not-available stub.SenseVoiceLanguagesetting (Cantonese / Mandarin / Auto) mapped to the model'slid_int_dictembedding index, with a picker in the voice-engine settings.HuggingFaceModelDownloaderno longer requiresmetadata.jsoninside a.mlmodelcbundle (SenseVoice's export omits it); it now validates oncoremldata.bin+weights/weight.bin. No regression for existing bundles (Parakeet etc.).Type of Change
Related Issue or Discussion
Closes #582
Testing
swiftlint --strict --config .swiftlint.yml Sourcesswiftformat --config .swiftformat SourcesEnd-to-end verified by driving the exact CoreML pipeline (preprocessor → int8 encoder → CTC decode) against real 16 kHz audio:
lidindex was corrected toyue = 7(index 13 is<|nospeech|>).Screenshots / Video
SenseVoice Small as the active Voice Engine model, showing the language selector (Cantonese / Mandarin / Auto Detect):
Notes
🤖 Generated with Claude Code