Skip to content

Add Fish Audio as a third voice engine (cloned voices, streams PCM, no ffmpeg) - #9

Open
BigBeedle wants to merge 2 commits into
jaredrhod:mainfrom
BigBeedle:feat/fish-audio-engine
Open

Add Fish Audio as a third voice engine (cloned voices, streams PCM, no ffmpeg)#9
BigBeedle wants to merge 2 commits into
jaredrhod:mainfrom
BigBeedle:feat/fish-audio-engine

Conversation

@BigBeedle

Copy link
Copy Markdown

What this adds

Fish Audio (fish.audio) as a third voice engine, beside Kokoro and ElevenLabs. The draw: Fish Audio lets anyone clone their own voice (or build a character voice) in their account and drive it by API — so an agent can literally speak in the voice its person made for it. Engine order when enabled: fish → elevenlabs → Kokoro.

Why it fits this codebase

  • Degrade, never mute — the fish path is wrapped in the exact same try/fallback shape as ElevenLabs; any cloud failure logs one line and falls back to the configured Kokoro voice.
  • No new dependencies. Fish Audio streams format: "pcm" (16-bit mono at the configured sample_rate), so chunks go from httpx.stream straight into the existing playback path. Unlike the ElevenLabs route there is no ffmpeg involved. Uses only httpx + numpy, both already in the tree.
  • Key handling mirrors _get_elevenlabs_key — macOS Keychain (item backtalk-fish, renameable via fish.key_slot), Linux secret-tool, FISH_AUDIO_API_KEY env var as the Windows last resort. Never in a file.
  • Backwards compatiblemouth reads the block with .get, so a backtalk.json written before this engine existed keeps working unchanged; the fish block ships disabled in DEFAULTS.

Config

"fish": {
  "enabled": true,
  "reference_id": "<your voice model id on fish.audio>",
  "model": "s2.1-pro-free"
}

The default model is s2.1-pro-free deliberately: it works on an unfunded account. The paid tiers (s1, s2-pro, s2.1-pro) return 402 Payment Required until the account holds credits — the DEFAULTS comment says so explicitly, so nobody debugs a 402 blind. The person's reference_id is listable via GET https://api.fish.audio/model?self=true.

Tested

Windows 11, real account, real cloned voice: config parses, sentences stream and play through the long-lived output stream (audio laws respected — the engine only feeds PCM into the existing _play_stream path), interrupt behavior unchanged, and the Kokoro fallback fires correctly on both a missing key and an unfunded model tier (402). macOS/Linux keychain lookups follow the ElevenLabs code path verbatim but I could only exercise the env-var branch — worth one audition on a Mac before merge.

🤖 Generated with Claude Code

Rigid-IG and others added 2 commits August 26, 2026 08:47
Fish Audio (fish.audio) lets anyone clone their own voice and drive it
by API, which makes it a natural fit for an agent that should sound like
its person wants. This wires it in as a third engine beside Kokoro and
ElevenLabs, following the existing doctrine end to end:

- Engine order: fish -> elevenlabs -> Kokoro. Degrade, never mute — any
  cloud failure falls back exactly like the ElevenLabs path does.
- Streams format "pcm" (16-bit mono at the configured sample_rate), so
  bytes go from the wire straight into the existing playback path: no
  ffmpeg dependency, unlike the ElevenLabs mp3 route.
- Key handling mirrors _get_elevenlabs_key: macOS Keychain (item
  backtalk-fish, renameable via fish.key_slot), Linux secret-tool,
  FISH_AUDIO_API_KEY env var as the Windows last resort. Never in a
  file.
- Config: a documented "fish" block in DEFAULTS. Defaults to the
  s2.1-pro-free model, which works on an unfunded account; the paid
  tiers (s1, s2-pro, s2.1-pro) return 402 until credits are added, and
  the comment says so, so nobody debugs a 402 blind.
- mouth reads the block with .get, so configs written before this
  engine existed keep working unchanged.

Tested on Windows 11 against a real cloned voice (reference_id from
GET /model?self=true): parses, streams, plays through the long-lived
output stream, and falls back to Kokoro cleanly when the model tier is
unfunded (402) or the key is absent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A fresh httpx client per sentence pays a TLS handshake inside the gap
between spoken sentences. One process-lifetime client shaves 0.2-0.4s
off each subsequent chunk (measured: 0.36s cold vs 0.22s reused to
first byte).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants