feat(voice): local speech stack by default — whisper.cpp STT + piper TTS - #10
Merged
Conversation
Speech no longer needs an API key. videoeditor-voice grows a second backend for each direction, local and default: - STT: whisper.cpp (`whisper-cli` over pinned ggml-base.en weights), word-level timestamps normalized to the same transcript shape Scribe returns, so analyze and the recorder coach work unchanged. `VIDEOEDITOR_STT=elevenlabs` opts back into Scribe. - TTS: a piper voice (en_US-lessac-medium) rendered by sherpa-onnx-offline-tts, encoded to the pipeline's mp3 via ffmpeg. `tts: elevenlabs` frontmatter (or `VIDEOEDITOR_TTS`) opts back into ElevenLabs; voice_id is now only required on that path. nixpkgs' piper-tts is broken on aarch64-darwin, hence sherpa-onnx as the engine. The nix closure pins the whole stack: whisper-cpp, sherpa-onnx, and both models, with WHISPER_MODEL / PIPER_VOICE baked into the wrapper and exported by the dev shell. hello-bench now renders keyless end to end. The recorder e2e also unsets WHISPER_MODEL: whisper hallucinates words on the fake mic's tone, which would make coaching assertions flaky. New: `cargo run -p videoeditor-voice --example transcribe -- <audio>` prints the normalized transcript for any file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Speech runs local and keyless by default.
videoeditor-voicenow has two backends per direction:whisper-cli, ggml-base.en)VIDEOEDITOR_STT=elevenlabstts: elevenlabsfrontmatter /VIDEOEDITOR_TTSBoth STT backends return the same transcript shape (
{text, words:[{type,text,start,end}]}), soanalyzeand the recorder coach consume either without change.voice_idfrontmatter is now only required on the ElevenLabs path.Nix closure
whisper-cpp1.8.7,sherpa-onnx1.13.3, and both models (~215 MB) are pinned; the wrapper bakesWHISPER_MODEL/PIPER_VOICEdefaults and the dev shell exports the same pins.examples/hello-benchrenders keyless end to end.Why sherpa-onnx and not nixpkgs
piper-tts: that package fails to evaluate on aarch64-darwin (brokenpysilero-vaddep); sherpa-onnx runs genuine piper voices, is cached for all four flake platforms, and synthesizes at ~0.07 RTF on Apple Silicon.Notes
WHISPER_MODELalongsideELEVENLABS_API_KEY: whisper hallucinates words on the fake mic's tone, which would make coaching assertions flaky.WHISPER_MODELswaps in a multilingual ggml. Whisper emits noaudio_evententries, so background-noise coaching remains Scribe-only.cargo run -p videoeditor-voice --example transcribe -- <audio>.Verified
just check,cargo test --workspace, 9/9 recorder e2e (nix dev shell)nix build .#videoeditor, then the store binary with env stripped: piper voiced all 5 hello-bench clips;analyzetranscribed a video of that narration back word-for-word with a correct timing map — zero configuration, zero keys.🤖 Generated with Claude Code