Filed after porting the turn-taking layer onto a telephony path (8 kHz µ-law, no
WebRTC). Raising it here because sip-server puts StreamCore on the same path.
Context: https://x.com/jasonshen_/status/2089925943385366962
Symptom
The energy VAD fires barge-in on the agent's own voice. It shows up as an
interruption with no caller turn behind it: the reply is cut mid-sentence,
nothing arrives to replace it, and the call sits in dead air until some other
recovery path unsticks it.
Why WebRTC hides this
A browser runs AEC before the audio ever reaches the server, so internal/vad
never sees the agent's own output coming back. Over a carrier there is no AEC
anywhere in the path. The returning audio is attenuated but structurally
identical to speech, and an RMS-vs-noise-floor test cannot separate the two —
adaptiveMinThreshold and noiseFloorMultiplier are both tuned against
background noise, which this isn't.
What worked for us
Not a higher threshold. That trades self-barging for being deaf to quiet
callers, and the adaptive floor is already doing as much as a single-signal
test can.
Instead, use the one reference the server always has: what it just sent.
- keep a rolling window (~400 ms) of the RMS of outbound audio
floor = recent_tx_rms × gain (gain ≈ 0.6 — echo can't exceed what produced it)- treat inbound as an interruption only when
inbound ≥ floor × margin (margin ≈ 1.8)
The margin is what separates double-talk from echo. Sensitivity to a quiet
caller on a clean line is untouched, because when the agent is silent the floor
is zero and the ordinary adaptive threshold governs.
One wrinkle if you keep ducking
Outbound RMS is naturally recorded at enqueue, but the duck is applied at send
(encodeAndSend). While ducked, what is actually on the wire is ~12 dB quieter,
so an un-scaled echo floor holds the confirm threshold high exactly when you
need to hear whether the caller is still talking. Scale the floor by the duck
gain while audioMuted is set.
Unrelated, same path, and it sent us down the wrong layer
Inbound went completely silent during agent playback and looked like half-duplex
media. We chased our own reader first, then transcode, then the caller's handset,
and had the PBX vendor confirm the endpoint was full-duplex by design before we
found it. Root cause was DTX — G.729 Annex B / OPUS DTX suppressing "silent" RTP,
which on that path meant all inbound during playback. Forcing G.711-only
(PCMU/PCMA) end to end fixed it.
Worth knowing before someone debugs their VAD for it: no amount of threshold
tuning helps when the frames aren't being sent.
Filed after porting the turn-taking layer onto a telephony path (8 kHz µ-law, no
WebRTC). Raising it here because
sip-serverputs StreamCore on the same path.Context: https://x.com/jasonshen_/status/2089925943385366962
Symptom
The energy VAD fires barge-in on the agent's own voice. It shows up as an
interruption with no caller turn behind it: the reply is cut mid-sentence,
nothing arrives to replace it, and the call sits in dead air until some other
recovery path unsticks it.
Why WebRTC hides this
A browser runs AEC before the audio ever reaches the server, so
internal/vadnever sees the agent's own output coming back. Over a carrier there is no AEC
anywhere in the path. The returning audio is attenuated but structurally
identical to speech, and an RMS-vs-noise-floor test cannot separate the two —
adaptiveMinThresholdandnoiseFloorMultiplierare both tuned againstbackground noise, which this isn't.
What worked for us
Not a higher threshold. That trades self-barging for being deaf to quiet
callers, and the adaptive floor is already doing as much as a single-signal
test can.
Instead, use the one reference the server always has: what it just sent.
floor = recent_tx_rms × gain(gain ≈ 0.6 — echo can't exceed what produced it)inbound ≥ floor × margin(margin ≈ 1.8)The margin is what separates double-talk from echo. Sensitivity to a quiet
caller on a clean line is untouched, because when the agent is silent the floor
is zero and the ordinary adaptive threshold governs.
One wrinkle if you keep ducking
Outbound RMS is naturally recorded at enqueue, but the duck is applied at send
(
encodeAndSend). While ducked, what is actually on the wire is ~12 dB quieter,so an un-scaled echo floor holds the confirm threshold high exactly when you
need to hear whether the caller is still talking. Scale the floor by the duck
gain while
audioMutedis set.Unrelated, same path, and it sent us down the wrong layer
Inbound went completely silent during agent playback and looked like half-duplex
media. We chased our own reader first, then transcode, then the caller's handset,
and had the PBX vendor confirm the endpoint was full-duplex by design before we
found it. Root cause was DTX — G.729 Annex B / OPUS DTX suppressing "silent" RTP,
which on that path meant all inbound during playback. Forcing G.711-only
(PCMU/PCMA) end to end fixed it.
Worth knowing before someone debugs their VAD for it: no amount of threshold
tuning helps when the frames aren't being sent.