Skip to content

Telnyx STT provider: design question, barge-in depends on partials #75

Description

@a692570

Context

#74 added Telnyx as a TTS provider. Telnyx also runs a streaming STT WebSocket (wss://api.telnyx.com/v2/speech-to-text/transcription) that already fronts Deepgram, Google, and Azure engines, plus a Telnyx in-house engine. I'd like to contribute a Telnyx STT provider as a follow-up, but one design question needs settling first because it touches barge-in.

What the Telnyx engine does (verified against the live API)

  • Client sends raw linear16 PCM as binary frames; server sends JSON frames with transcript, is_final, confidence.
  • The Telnyx engine emits exactly ONE final frame after audio stops, then closes the socket (1000). No interims, no word timestamps, no speech_started / utterance_end. Confidence is null.
  • interim_results=true suppresses ALL output from the Telnyx engine (adding the param returns zero frames). It cannot be requested.
  • transcription_engine is case-sensitive: Telnyx works, telnyx closes the socket.
  • The Deepgram engine through the same endpoint does emit interims (is_final: false frames with confidence), so the finals-only behavior is specific to the Telnyx engine.

The problem: barge-in is a hard dependency on partials

In internal/pipeline/inbound.go (line numbers at current main):

  • hasPartialText is set only by non-final results (~101-108, gate at ~76).
  • The barge-in trigger requires bargeInVAD.IsSpeaking() && agentSpeaking && hasPartialText (~196).
  • Backchannel classification ("mm-hm" filtering) reads latestPartial (~168-193), also fed only by partials.
  • Live transcript captions stream to the client from the partial path (~110-113).

With a finals-only provider, hasPartialText is never set: barge-in never fires (dead, not degraded), backchannel filtering never classifies, and the UI gets no live captions.

Options

  1. Provider + docs note (the finals-only OpenAI whisper precedent, docs/providers.md:15): ship the Telnyx-engine provider and document that barge-in and live captions do not work with it.
  2. Provider that exposes Telnyx's engine param: transcription_engine = "Telnyx" (in-house, finals-only, no barge-in) or "Deepgram" (proxied through the same endpoint, emits interims, barge-in works). Default engine becomes a product choice rather than a hard limitation.
  3. Pipeline capability flag: expose whether the active STT produces partials and degrade gracefully. More pipeline surface, so I don't want to assume that's welcome without asking.

Which way would you like it? Happy to implement any of the three.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions