fix(elevenlabs): populate SpeechData.confidence from Scribe logprobs#6461
Open
captainbanan wants to merge 1 commit into
Open
fix(elevenlabs): populate SpeechData.confidence from Scribe logprobs#6461captainbanan wants to merge 1 commit into
captainbanan wants to merge 1 commit into
Conversation
|
|
…robs The ElevenLabs STT plugin never set SpeechData.confidence, so it always defaulted to 0.0 even though Scribe returns a per-word `logprob` in its response (REST and realtime). Consumers that rely on transcription confidence get no signal. Average the spoken-word logprobs and exponentiate to a [0, 1] probability (geometric mean of the token probabilities), applied on both the REST and WebSocket paths. Returns 0.0 when per-word logprobs are unavailable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
captainbanan
force-pushed
the
fix/elevenlabs-stt-confidence
branch
from
July 17, 2026 08:39
8ba5965 to
37d4128
Compare
captainbanan
marked this pull request as ready for review
July 17, 2026 08:54
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / Why
The ElevenLabs STT plugin never populates
SpeechData.confidence, so it always stays at its0.0default — even though Scribe returns a per-wordlogprobin every response. Consumers that useconfidence(e.g. to tell a clean transcript from a low-confidence / garbled one) get no signal from ElevenLabs.Verified live: both
scribe_v1/scribe_v2(REST) andscribe_v2_realtimereturn a natural-loglogprobperwords[]token, but_process_stream_event/_transcription_to_speech_eventread onlytext+ wordstart/end, so it is discarded andconfidencestays0.0.Change
Add
_speech_confidence(words): average the spoken-word (type == "word") logprobs and exponentiate → a[0, 1]confidence (geometric mean of the token probabilities). Returns0.0when per-word logprobs aren't available (e.g. a non-timestamped commit), so behaviour is unchanged there.Wired into both the REST and WebSocket
SpeechDataconstruction.Note: this deliberately does not use Scribe's
language_probability— that is language-identification confidence, a different quantity from transcription confidence, so folding it intoSpeechData.confidencewould be misleading.Same class of fix as #5829 / #5830 (a sibling STT plugin not threading its confidence into
SpeechData).Tests
Unit tests for the helper (confident vs low-confidence logprobs, no-logprob →
0.0) and an end-to-end check that a committed transcript setsconfidence.ruff check/ruff formatclean.🤖 Generated with Claude Code