slng: rewrite plugin around the Unmute Bridge (v2)

[metehan-slng]

Breaking change

This is a major rewrite of livekit-plugins-slng. We would like it released as 2.0.0 (please bump accordingly at release time; version.py is left untouched per contribution guidelines).

What changed

The plugin now connects exclusively through SLNG's Unmute Bridge, the gateway's normalized protocol layer. A model identifier is all that is needed (slng.STT(model="deepgram/nova:3")); the plugin builds the bridge WebSocket endpoint itself.

• Bridge-only routing. The 1.x model_endpoint / model_endpoints parameters were removed and now raise a clear ValueError pointing at the replacement (model= or connections=[...]).
• Failover chains for STT and TTS. connections=[...] accepts model identifiers, bridge endpoint URLs, or typed STTConnectionConfig / TTSConnectionConfig objects. STT fails over at safe stream boundaries and replays buffered audio (including a pending finalize) onto the new connection; TTS fails over only before first audio. HTTP 413 is treated as terminal instead of walking the chain, since every candidate would reject the same oversized payload. Recovery back to the primary is cooldown-based.
• Faster end-of-utterance. The stream sends the bridge a finalize signal when the agent's user state transitions to listening, with a watchdog (final_timeout_s, opt-in) guarding against providers that never deliver the final transcript. Interim results no longer disarm the watchdog.
• BYOK support. provider_api_key forwards the customer's own provider credential as the gateway's X-Slng-Provider-Key header.
• Geographic zone routing. In addition to the existing region_override, a new world_part_override constrains routing to a broad geographic zone (maps to the gateway's X-World-Part-Override header; region_override takes precedence). Both propagate to fallback candidates.
• Typed plugin events. slng_event emits structured events for gateway session IDs and fallback attempt / switch / exhaustion.
• Removed implicit behavior. Client-side language and voice normalization are gone by design: values are passed verbatim so they always match the selected model's contract (e.g. Sarvam requires BCP-47 hi-IN). Voice IDs are provider-native.
• Honest capabilities. Only pcm_s16le STT input is accepted (previously other encodings were silently mislabeled), recognize() raises NotImplementedError (the bridge is WebSocket-only), and offline_recognize is reported as False.
• Hardening. Bounded WebSocket receives with the connection timeout (mirroring the Cartesia plugin), lazy-starting fallback streams that support plain async for and collect(), per-run stream state so base-class retries produce audio, consistent watchdog and speech-event state across options-change reconnects, removal of a dead connection pool, and no leaked per-connection timing state.

Testing

• Unit tests included under livekit-plugins/livekit-plugins-slng/tests/ (no network required; scripted fake WebSockets; all modules marked pytest.mark.unit), covering failover with audio + finalize replay, options-change reconnect state and event brackets, watchdog behavior, retry-safe stream construction, fallback stream contracts, phrase batching, geo override propagation, option forwarding, and encoding validation.
• make check passes (ruff format, ruff lint, strict mypy via scripts/check_types.py).
• Verified live against the production SLNG gateway: STT flush-to-final latency, TTS synthesis across ElevenLabs / Cartesia / Deepgram / Rime bridge models, failover chains, BYOK, and world-part override steering (confirmed via gateway routing logs).

Notes for reviewers

• README.md was rewritten to document the 2.0 API, including a "Migrating from 1.x" section.
• END_OF_SPEECH is emitted per final transcript (rather than waiting for provider utterance-end events) deliberately: it preserves the low end-of-utterance latency the finalize flow provides.
• TTS treats a server close after audio was received as end-of-segment deliberately: several bridge providers (Rime, Cartesia) terminate segments via close.
• Several findings from this PR's automated review were confirmed and fixed in follow-up commits, each with a regression test.

[tinalenguyen]

hi, thanks for the PR! could you remove the test files to prevent noise?

[metehan-slng]

hi, thanks for the PR! could you remove the test files to prevent noise?

Done, removed the test files. Thanks!

[devin-ai-integration]

Devin Review found 2 new potential issues.

View 8 additional findings in Devin Review.

[devin-ai-integration]

🔴 Text-to-speech warm-up crashes with a stack overflow when failover is configured

When a text-to-speech instance has fallback candidates and the active choice is the primary (self._candidate_tts[self._candidate_state.start()].prewarm() at livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py:726), the warm-up call keeps calling itself forever instead of warming a connection, so it never finishes.
Impact: Any agent configured with multiple TTS fallback connections crashes at start-up (RecursionError) as soon as warm-up runs, taking the whole session down.

Self-dispatch recursion in prewarm

self._candidate_tts[0] is self (set at livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py:416). On the parent instance _is_candidate is False, and CandidateState.start() returns 0 in the normal (non-failed-over) state (connection.py:45-53). Therefore self._candidate_tts[self._candidate_state.start()] resolves to self, and self.prewarm() re-enters prewarm with the exact same condition len(self._candidate_tts) > 1 and not self._is_candidate still true, recursing until the stack overflows. Only fallback candidates (index > 0) have _is_candidate=True and would reach the standby branch; the primary never does. This triggers regardless of warm_standby_enabled, since the recursion happens before that branch.

Suggested change

	if len(self._candidate_tts) > 1 and not self._is_candidate:
	self._candidate_tts[self._candidate_state.start()].prewarm()
	return
	if len(self._candidate_tts) > 1 and not self._is_candidate:
	active = self._candidate_tts[self._candidate_state.start()]
	if active is not self:
	active.prewarm()
	return

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Changing voice or language at runtime is ignored by text-to-speech fallback providers

Updating the voice or language at runtime only changes the primary instance's settings (self._opts.voice/self._opts.language at livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py:670-673) and never forwards the change to the configured fallback providers, so after a failover the wrong voice or language is used.
Impact: When speech falls over to a backup provider, it keeps speaking with the stale voice/language that was set at construction time, producing audio that does not match the requested settings.

update_options does not propagate to candidate TTS instances

Each fallback candidate is a full TTS object with its own _opts (built in __init__ at livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py:445-476 and stored in self._candidate_tts). update_options mutates only self._opts and closes only self's warm standby (tts.py:677-679); it iterates neither self._candidate_tts[1:] nor their standby connections. Metrics and plugin events are forwarded from candidates to the parent (tts.py:477-478), showing candidates are intended to be managed by the parent, so the missing option propagation is an oversight. After _FallbackStreamBase._handle_failure advances to a candidate, synthesis uses that candidate's outdated voice/language.

Prompt for agents

TTS.update_options in livekit-plugins/livekit-plugins-slng/livekit/plugins/slng/tts.py updates only the primary instance's self._opts (voice/language) and closes only the primary's warm standby. It does not forward the change to the fallback candidate TTS instances stored in self._candidate_tts[1:], each of which keeps its own _opts and its own warm-standby connection. As a result, after a failover the fallback provider synthesizes with the voice/language captured at construction time rather than the updated value. Update update_options so that, when not a candidate, it also applies the same voice/language changes to each candidate in self._candidate_tts[1:] (and invalidates their warm-standby connections) so all candidates stay consistent with the primary.

---

Was this helpful? React with 👍 or 👎 to provide feedback.