vad: bound barge-in by an outbound echo reference, per session - #71
Draft
jason-shen wants to merge 1 commit into
Draft
vad: bound barge-in by an outbound echo reference, per session#71jason-shen wants to merge 1 commit into
jason-shen wants to merge 1 commit into
Conversation
The energy VAD fires barge-in on the agent's own voice wherever nothing in the path runs AEC. Over WebRTC the browser cancels echo before audio reaches the server; over a carrier the returning audio is attenuated but structurally identical to speech, and an RMS-vs-noise-floor test cannot separate the two. EchoGuard keeps a rolling window of the RMS the sender put on the wire and raises the barge-in threshold to floor x gain x margin while that window is live. The agent silent means a zero bound, so a quiet caller on a clean line is unaffected. The reference is sampled after duck attenuation, so a ducked talkspurt lowers the bound with the audio that produces the echo. Echo frames no longer feed the adaptive noise floor: learning them would ratchet the ordinary threshold to its cap every time the agent spoke. The bound is chosen per session rather than per server, because one instance commonly serves browsers and SIP calls at once and the two need opposite answers: a browser barge-in would have to clear the agent's own output level first, for a bound that catches nothing getUserMedia has not already removed. Clients declare a raw path with aec=none on the WHIP URL, which sip-server now sends on every call; pipeline.echo_guard defaults to "auto" and follows it. Documents aec and the previously undocumented direction parameter in the WHIP protocol reference, and DTX suppressing inbound RTP during playback as a sip-server troubleshooting entry, since it presents as half-duplex media and is not a VAD problem. Fixes #48
Member
Author
|
testing still needed before merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The energy VAD fires barge-in on the agent's own voice wherever nothing in the path runs AEC. Over WebRTC the browser cancels echo before audio reaches the server; over a carrier the returning audio is attenuated but structurally identical to speech, and an RMS-vs-noise-floor test cannot separate the two.
EchoGuard keeps a rolling window of the RMS the sender put on the wire and raises the barge-in threshold to floor x gain x margin while that window is live. The agent silent means a zero bound, so a quiet caller on a clean line is unaffected. The reference is sampled after duck attenuation, so a ducked talkspurt lowers the bound with the audio that produces the echo.
Echo frames no longer feed the adaptive noise floor: learning them would ratchet the ordinary threshold to its cap every time the agent spoke.
The bound is chosen per session rather than per server, because one instance commonly serves browsers and SIP calls at once and the two need opposite answers: a browser barge-in would have to clear the agent's own output level first, for a bound that catches nothing getUserMedia has not already removed. Clients declare a raw path with aec=none on the WHIP URL, which sip-server now sends on every call; pipeline.echo_guard defaults to "auto" and follows it.
Documents aec and the previously undocumented direction parameter in the WHIP protocol reference, and DTX suppressing inbound RTP during playback as a sip-server troubleshooting entry, since it presents as half-duplex media and is not a VAD problem.
Fixes #48