Skip to content

Add Telnyx streaming TTS provider - #74

Merged
jason-shen merged 2 commits into
streamcoreai:mainfrom
a692570:telnyx-tts-provider
Sep 3, 2026
Merged

Add Telnyx streaming TTS provider#74
jason-shen merged 2 commits into
streamcoreai:mainfrom
a692570:telnyx-tts-provider

Conversation

@a692570

@a692570 a692570 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What this adds

A telnyx TTS provider speaking Telnyx's streaming text-to-speech WebSocket API (wss://api.telnyx.com/v2/text-to-speech/speech?voice=...&audio_format=linear16&sample_rate=16000). Selected with tts.provider = "telnyx", configured through a new [telnyx] section (api_key, voice, voice_speed), overridable via TELNYX_API_KEY, documented in English and Chinese.

Why one WebSocket per utterance

This design is verified against the live protocol, not a guess. The server emits its isFinal frame only after the client sends an empty-text teardown ({"text": ""}). There is no per-utterance completion marker, and plain text frames are buffered without starting synthesis (verified: two sentences sent 12s apart produced zero audio until teardown). Instead of a persistent connection with idle-gap or byte-estimate completion heuristics, the adapter opens a fresh connection per utterance: dial, init ({"text": " ", "voice_settings": {"voice_speed": ...}}), text with flush: true, teardown, stream audio until isFinal, and the server closes 1000. Measured: 3.9s of audio synthesized in 1.6s wall with first audio 854ms after dial. Synthesis outpaces playback ~2.4x, so the per-utterance dial costs nothing on the live path and every completion heuristic is avoided.

Protocol details that shaped the code:

  • Server frames carry no type field; routing is by keys: error, then isFinal, then audio. Frames with audio: null and isFinal false (cache-status notifications) are skipped silently.
  • Audio is base64 linear16 at 16 kHz mono, the pipeline's native rate, so nothing resamples.
  • On barge-in the client sends {"force": true} (best-effort, 2s write deadline) and closes the connection; the server stops audio and never emits a final for the interrupted utterance.
  • An unknown voice, or one the account is not provisioned for, fails the handshake with HTTP 403. The dial error names the voice and points at GET /v2/text-to-speech/voices.
  • voice_speed rides the init frame, so per-utterance speed control works through SynthesizeStreamWithControls, clamped to 0.8-1.2 like Cartesia's conversational band. [telnyx] voice_speed sets the baseline for untagged sentences.
  • Default voice Telnyx.Qwen3TTS.d9348e0d-988a-42cc-a64e-18093fe45c03 ("Delta", female, Qwen3TTS family — Qwen3 voices use UUID catalog ids); availability varies by account.

How I verified it

  • gofmt -l . clean, go build ./..., go vet ./..., go test -race ./... all pass, including the hard config gates TestConfigExampleDocumentsEveryField and TestEveryCredentialHasAnEnvOverride.
  • Hermetic tests in internal/tts/telnyx_test.go (no network, no key) cover frame routing (audio decode, final, cache-status skip, error frames, malformed input), the exact client wire-format bytes, speed clamping, and defaults.
  • Real-audio smoke per CONTRIBUTING: a throwaway cmd/telnyx-smoke (removed before committing) streamed "Hello from the Telnyx TTS provider for StreamCore." with a live key using the default voice: 56 chunks, 179,968 PCM bytes, 5.62s at 16 kHz mono. (An earlier run on a Bayan voice measured 22 chunks / 3.72s with first audio 1.3s after dial; both were confirmed by ffprobe.)

Maintainer question

AGENTS.md says config.toml.example, the README provider table, and https://streamcore.ai/llms-full.txt should be updated together. llms-full.txt lives outside this repo: how is it regenerated, and should that happen as part of this PR or separately?

One pre-existing gap I did not touch

The [tts] supported-provider list in docs/configuration.md was already missing aliyun and volcengine before this change (config.toml.example lists both). I added telnyx without altering that omission to keep the diff scoped; happy to fix it in a follow-up.

@jason-shen

Copy link
Copy Markdown
Member

thank you @a692570

@jason-shen
jason-shen merged commit 7a8efdd into streamcoreai:main Sep 3, 2026
1 check passed
@jason-shen jason-shen added the enhancement New feature or request label Sep 3, 2026
@a692570

a692570 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @jason-shen, and for approving the workflow run so quickly.

Follow-up on the AGENTS.md lockstep rule: is streamcore.ai/llms-full.txt regenerated from docs/, or should I send the Telnyx section over for a manual update?

I'm also looking at Telnyx STT as a follow-up provider. There's one design wrinkle around barge-in, so I'll open an issue with the analysis first.

@jason-shen

jason-shen commented Sep 3, 2026 via email

Copy link
Copy Markdown
Member

@a692570

a692570 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Got it, thanks Jason.

One question then: where does the site source live? If it's a repo I can reach, I'll PR the Telnyx section there. If it's hand-edited, here's the addition ready to paste into section 3.4, matching the existing block style:

[telnyx]                # TTS. Telnyx streaming text-to-speech
api_key = ""            # or TELNYX_API_KEY env var
voice = ""              # defaults to Telnyx.Qwen3TTS.d9348e0d-988a-42cc-a64e-18093fe45c03 ("Delta");
                        # any id from GET /v2/text-to-speech/voices, availability varies by account
voice_speed = 1.0       # clamped to 0.8-1.2

And the TTS provider list line gains | telnyx:

provider = "cartesia"   # cartesia | deepgram | elevenlabs | speechify | vibevoice | minimax | mimo | telnyx

While in there I noticed the same list is missing volcengine and aliyun TTS (both merged around three weeks ago, #65 and #68). Happy to draft those blocks too so one paste brings the file current.

No rush at all on the #75 design question.

@jason-shen

jason-shen commented Sep 3, 2026 via email

Copy link
Copy Markdown
Member

@a692570

a692570 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Here's the complete paste, all three providers. Four edits to section 3.4:

1. Replace the TTS provider list line:

provider = "cartesia"   # aliyun | cartesia | deepgram | elevenlabs | mimo | minimax | speechify | telnyx | vibevoice | volcengine

(The STT provider list line is already current, no change needed there.)

2. Add after [mimo]:

[telnyx]                # TTS. Telnyx streaming text-to-speech
api_key = ""            # or TELNYX_API_KEY env var
voice = ""              # defaults to Telnyx.Qwen3TTS.d9348e0d-988a-42cc-a64e-18093fe45c03 ("Delta");
                        # any id from GET /v2/text-to-speech/voices, availability varies by account,
                        # a voice your key is not provisioned for fails the dial with HTTP 403
voice_speed = 1.0       # clamped to 0.8-1.2

3. Replace [aliyun] (it serves both STT and TTS now, so the header and three new fields):

[aliyun]                # STT and TTS. Alibaba Cloud Model Studio (DashScope)
api_key = ""
model = ""              # STT; defaults to paraformer-realtime-v2; fun-asr-realtime is the alternative
language = ""           # STT. "zh", "en"; empty auto-detects
vocabulary_id = ""      # STT. hotword list created in the console
tts_model = ""          # TTS; defaults to cosyvoice-v2
voice = ""              # TTS; defaults to longxiaochun_v2. Voices are tied to a model generation:
                        # a v1 voice on cosyvoice-v2 is rejected, change tts_model and voice together
url = ""                # optional; defaults to wss://dashscope.aliyuncs.com/api-ws/v1/inference

4. Replace [volcengine] (same story, plus two fields the file was missing):

[volcengine]            # STT and TTS. Doubao streaming ASR / seed-tts
api_key = ""            # console API key, sent as X-Api-Key; the app-id + access-token pair is rejected
resource_id = ""        # STT; defaults to volc.seedasr.sauc.duration (hourly billing)
model = ""              # STT; defaults to bigmodel
url = ""                # optional; defaults to wss://openspeech.bytedance.com/api/v3/sauc/bigmodel
end_window_ms = 0       # STT. silence that settles an utterance; defaults to 800
tts_resource_id = ""    # TTS; defaults to seed-tts-2.0. The 1.0 resource is a separate entitlement
                        # and answers 403 until it is activated

All fields verified against the current config.toml.example on main. Telnyx STT will follow separately once the #75 design lands, I'll send the STT list line + block addition then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants