feat: add smallest ai asr and tts extensions#2203
Conversation
harshitajain165
force-pushed
the
feat/smallest-ai-integration
branch
from
63273f3 to
e2f2b42
Compare
|
@halajohn @plutoless |
|
Hi @plutoless 👋 The earlier CI failure was in smallest_tts_python's test_headers_configuration (tests/test_params.py) — the client now sets a default X-Source: ten-framework header, so the header-count assertions were off by one. I've fixed the assertions and pushed the update. The workflow runs are currently sitting in "action_required" (the fork-PR approval gate), so they haven't started. Could you approve/re-run the CI workflows when you get a chance? And a review whenever you have time would be much appreciated. Thanks a lot! |
| message=str(e), | ||
| ), | ||
| ) | ||
| asyncio.create_task(self._handle_reconnect()) |
There was a problem hiding this comment.
When the vendor WebSocket closes, _process_messages() awaits _handle_reconnect() from inside the current _message_task.
The reconnect path calls start_connection(), which first calls stop_connection(). stop_connection() then cancels the
same _message_task that is currently executing the reconnect path, so the reconnect flow is interrupted before a new
message-processing task is created.
Add two new vendor extensions for Smallest AI: - smallest_asr_python: real-time speech-to-text via the Pulse live WebSocket API (binary PCM16 in, interim/final transcripts out, finalize control message, reconnect with exponential backoff). - smallest_tts_python: text-to-speech via the Lightning SSE streaming endpoint (base64 PCM16 chunks decoded on the fly, ~100 ms to first audio, output_format pinned to pcm). Both include mocked extension-local test suites, test configs, and README docs. Configured via params.api_key or SMALLEST_API_KEY.
…sults - guard _send_finalize against a down websocket and complete the finalize handshake locally so asr_finalize_end is always emitted and the turn cannot stall - reset the reconnect retry budget on received transcriptions instead of right after the websocket handshake, so an accept-then-close failure hits the max-attempts fatal error instead of looping forever - reformat _handle_error_message to be black-clean
- tts: stop substring-matching 401/403 in exception text; real auth failures are already classified by the HTTP status-code branch, so transport-level exceptions are always plain errors now - asr: latch the utterance start when its first transcript arrives and reuse it for word-less results, so interim start_ms stays monotonic instead of pointing at the end of the audio stream
The client now sets a default X-Source: ten-framework header, so the header-count assertions in test_headers_configuration were off by one.
…ancel When the vendor WebSocket dropped mid-session, _process_messages (running inside _message_task) awaited _handle_reconnect, whose start_connection -> stop_connection cancels _message_task itself, interrupting the reconnect before a new message task was created. Route every reconnect trigger through a _schedule_reconnect helper that runs the attempt on an independent, tracked task, so the message loop can exit cleanly while the reconnect proceeds. In-flight reconnect tasks are retained against GC and cancelled on deinit. Add a regression test that drops the socket mid-session and asserts the reconnect delivers a transcript.
harshitajain165
force-pushed
the
feat/smallest-ai-integration
branch
from
70675f2 to
d17c5d9
Compare
3 participants
What
Adds two new vendor extensions for Smallest AI:
smallest_asr_python— real-time speech-to-text using thePulse
live WebSocket API (
wss://api.smallest.ai/waves/v1/stt/live). 38 languages,64 ms time-to-first-transcript.
smallest_tts_python— text-to-speech using theLightning
SSE streaming endpoint (
/waves/v1/tts/live). ~100 ms to first audio chunk,12 languages, voice cloning;
lightning_v3.1andlightning_v3.1_promodels.Implementation notes
ASR (extends
AsyncASRBaseExtension):final=false, finals onPulse's
is_final.asr_finalizemaps to Pulse's{"type": "finalize"}control message (sessionstays open).
word_timestamps=trueby default so final results carry accuratestart_ms/duration_ms.TTS (extends
AsyncTTS2HttpExtension):data: {"audio": "<base64>"}/{"done": true}) decodedincrementally; a partial line is never split across chunk boundaries.
output_formatpinned topcm: Lightning's raw PCM is already signed 16-bitLE mono, matching the
pcm_framecontract with no conversion, and skipping thecontainer header keeps time-to-first-audio low.
invalid_api_keyclassified asINVALID_KEY_ERROR(fatal);everything else non-fatal.
Both authenticate via
params.api_key/SMALLEST_API_KEYand sendX-Source: ten-frameworkfor API-side attribution. All otherparamskeys passthrough verbatim (query string for ASR, request body for TTS).
Testing
ASR — result shape, finalize, dump, metrics, reconnect, vendor error, invalid
params; TTS — basic/dump/flush, error classification, metrics, params/URL
resolution, robustness, state machine.
python -m black --check --line-length 80clean; syntax verified.task asr-guarder-test EXTENSION=smallest_asr_python— 10/10 passed against the live APItask tts-guarder-test EXTENSION=smallest_tts_python— 15/15 passed (+1 optional skip) against the live APIvoice_assistant_smallestgraph (Pulse STT + GPT-4o + Lightning TTS over Agora RTC), including barge-in/interruptionSupersedes #2200 (auto-closed by a force-push during a history cleanup).