The voice-activity detector is a production-grade adaptive system designed for 24/7 continuous operation. It combines three pillars of adaptation:
- IMCRA noise estimation (Cohen 2003) — continuous noise tracking
- Dual-timescale EMA — 30-minute baseline + 2-second rapid change detection
- Hysteresis VAD — asymmetric onset/offset thresholds with hangover
This replaces static thresholds that fail as environment changes (AC on/off, people moving, time of day).
kind: "start" | "end"
start_sample: int # absolute sample index where speech began
end_sample: int | None # absolute sample index where speech ended
forced_split: bool # True if max_recording_sec triggered
SILENCE = 0
SPEECH = 1
# Noise floor tracking (dual-timescale)
_noise_floor: float # current noise floor estimate
_noise_slow_alpha: float = 0.9999 # 30-minute time constant
_noise_fast_alpha: float = 0.995 # 2-second time constant
_noise_alpha: float # current alpha (blends slow/fast)
_energy_history: deque[float] # 3-second window for percentile
# IMCRA state
_imcra_window: int = 150 # ~1.5s at 10ms frames
_imcra_power_history: deque[float] # power spectrum history
_imcra_noise_est: float # IMCRA noise estimate
# VAD state machine
_vad_state: VADState # SILENCE or SPEECH
_speech_duration_ms: int # current speech duration
_silence_duration_ms: int # current silence duration
# Thresholds (SNR-based, adaptive)
_speech_threshold_db: float = 6.0 # SNR threshold for speech
_hysteresis_up_db: float = 4.0 # onset margin (harder to start)
_hysteresis_down_db: float = 3.0 # offset margin (easier to stay)
_endpoint_timeout_ms: int = 500 # silence before speech end
# Hangover (prevents truncation)
_hang_counter: int # frames remaining in hangover
_hang_time: int # 150ms in blocks
# Streaming state
_in_speech: bool # current VAD state
_speech_start_sample: int # absolute sample of speech onset
_min_speech_samples: int # min_recording_sec * sr
_max_speech_samples: int # max_recording_sec * sr
For a mono float32 signal s of length n:
RMS(s) = sqrt( mean( s² ) )
Returns 0.0 for empty input.
The noise floor is tracked continuously using two exponential moving averages:
noise_slow = α_slow * noise_slow + (1 - α_slow) * min_energy
α_slow = 0.9999 # ~30-minute half-life
noise_fast = α_fast * noise_fast + (1 - α_fast) * min_energy
α_fast = 0.995 # ~2-second half-life
energy_history.append(rms)
min_energy = percentile(energy_history, 10) # robust to speech contamination
IF len(energy_history) >= 50:
fast_min = percentile(last_50, 10)
IF abs(fast_min - noise_floor) > 0.02:
# Rapid change detected — blend 50/50
noise_floor = 0.5 * noise_floor + 0.5 * fast_min
α = 0.99 # temporarily faster
ELSE:
# Decay back to slow adaptation
α = min(α_slow, α + 0.001)
noise_floor = α * noise_floor + (1 - α) * min_energy
noise_floor = clip(noise_floor, 0.001, 0.5)
Tracks local minimum of power spectrum over a sliding window:
power = mean(frame²)
imcra_power_history.append(power)
IF len(history) >= 150:
local_min = percentile(powers, 5) # proxy for minimum
local_mean = mean(powers)
local_var = var(powers)
bias = 1 + sqrt(2/(window-1)) * (local_var / local_mean)
imcra_noise_est = local_min * bias
Note: This is the simplified single-pass version. The full IMCRA (Cohen 2003) uses two-pass smoothing and speech-presence-probability control. Our implementation trades some accuracy for real-time performance at 100fps.
Instead of arbitrary RMS thresholds, speech is detected using SNR in dB:
snr_linear = rms / max(noise_floor, 1e-10)
snr_db = 20 * log10(snr_linear)
energy_score = snr_db / speech_threshold_db # 6dB = 2x louder = speech
When use_spectral_vad=True, combines energy with spectral features:
spectral_score = (
0.15 * flux_norm + # onset detection
0.15 * centroid_norm + # speech frequency band
0.15 * zcr_norm + # voicing pattern
0.15 * ber_norm # speech energy concentration
)
composite = (1 - w) * energy_score + w * (energy_score * 0.6 + spectral_score * 0.4)
Where w = spectral_weight (default 0.4).
When snr_db < -2.0, spectral feature computation is skipped (expensive FFT).
This saves ~30% CPU on silence-dominant audio with no accuracy loss.
| Feature | What it measures | Speech range | Noise range |
|---|---|---|---|
| Spectral flux | L1 distance between consecutive FFT frames | High (onset) | Low (steady) |
| Spectral centroid | Weighted mean frequency | 1-3 kHz | <500 Hz |
| Zero-crossing rate | Fraction of sign changes | 0.03-0.25 | <0.03 |
| Band energy ratio | Energy in 300-3400 Hz vs total | >0.5 | <0.3 |
Normalization:
flux_norm = min(flux / 50.0, 1.0)
centroid_norm = 1.0 if 300 < centroid < 4000 else max(0.2, 1.0 - |centroid - 2000| / 3000)
zcr_norm = 1.0 if 0.03 < zcr < 0.25 else 0.2
ber_norm = min(ber / 2.0, 1.0)
onset_threshold = 1.0 + hysteresis_up_db / 6.0 # = 1.67
IF vad_state == SILENCE:
IF composite > onset_threshold:
vad_state = SPEECH
speech_duration_ms = 0
silence_duration_ms = 0
emit VADEvent("start", start_sample=current_sample - pre_roll)
ELSE:
silence_duration_ms += block_duration_ms
offset_threshold = 1.0 - hysteresis_down_db / 6.0 # = 0.50
IF vad_state == SPEECH:
IF composite < offset_threshold:
silence_duration_ms += block_duration_ms
IF silence_duration_ms > endpoint_timeout_ms:
vad_state = SILENCE
ELSE:
is_speech = True # still in hangover period
ELSE:
silence_duration_ms = 0
is_speech = True
speech_duration_ms += block_duration_ms
Prevents speech truncation by extending output for 150ms after speech ends:
IF is_speech:
hang_counter = hang_time # 150ms in blocks
ELSE:
hang_counter = max(0, hang_counter - 1)
is_speech_final = hang_counter > 0 OR is_speech
| Parameter | Value | Rationale |
|---|---|---|
blocksize |
1024 samples (64ms) | Fine enough for responsive VAD, coarse enough for low CPU |
sample_rate |
16000 Hz | Whisper native rate, avoids resampling |
speech_threshold_db |
6.0 dB | SNR threshold — 2x louder than noise = speech |
hysteresis_up_db |
4.0 dB | Onset margin — harder to start (prevents false triggers) |
hysteresis_down_db |
3.0 dB | Offset margin — easier to stay (prevents cutting speech) |
endpoint_timeout_ms |
500 ms | Silence before speech end |
hang_time |
150 ms | Prevents truncation of trailing sounds |
noise_slow_alpha |
0.9999 | 30-minute baseline tracking |
noise_fast_alpha |
0.995 | 2-second rapid change detection |
imcra_window |
150 frames | ~1.5s minimum for noise estimation |
min_recording_sec |
0.5 | Ignore clicks, coughs, chair creaks |
max_recording_sec |
15.0 | Safety valve for runaway recordings |
spectral_weight |
0.4 | Balance between energy and spectral features |
pre_speech_padding_sec |
0.2 | Audio context before speech onset |
- Normalize to float32 in [-1, 1]:
audio / max(|audio|)if peak > 1.0 (ALL audio, not just clipping) - Trim trailing silence: find last sample above 0.001 (was 0.005), keep 200ms pad after it
- Noise reduction: spectral gating via
noisereducelibrary (skips short/clean signals)
IF config.backend == WHISPER_CPP:
model = cached pywhispercpp.Model(ggml_model_name)
segments = model.transcribe(audio, single_segment=True)
# Runs in subprocess for crash isolation (segfault survival)
# Frozen binary: uses in-process run_worker()
ELSE:
model = cached faster_whisper.WhisperModel(name, device, compute_type)
segments, info = model.transcribe(audio, beam_size, language, ...)
# CUDA failure → retry with device="cpu", compute_type="int8"
Layer 1: Exact regex replacements (word-boundary: (?<!\w)...\w)
Layer 2: Fuzzy phonetic matching (Levenshtein ratio >= 0.8)
Layer 3: LLM context injection (dictionary terms → prompt glossary)
IF llm_mode != OFF AND word_count > 5:
prompt = build_user_prompt(raw_text, mode)
payload = {model, messages, max_tokens, temperature, stream: true}
FOR token IN rewrite_stream(payload):
emit llm_partial event # real-time streaming to frontend
processed = clean_response(collected_tokens)
_output_text(processed, config): # ALWAYS called, regardless of hooks
type_to_focused_input → wtype (Wayland) / xdotool (X11) / keybd_event (Windows)
copy_to_clipboard → wl-copy (Wayland) / xclip (X11) / ctypes (Windows)
[parallel threads — not sequential]
_json_emit(processed event) → stdout JSON → Frontend api-tauri.ts
[debug] rms=0.034 noise=0.007 snr=16.9dB state=SPEECH
[debug] spectral: flux=43.85 centroid=1047Hz zcr=0.0811 ber=0.7267 score=1.94
rms: current chunk RMS energynoise: tracked noise floor (adapts continuously)snr: signal-to-noise ratio in dBstate: VAD state (SILENCE or SPEECH)score: composite speech probability (>1.67 = onset, <0.50 = offset)
Large-scale open VAD dataset (15GB/150GB/1.5TB) from LibriSpeech. ViT with MFCC features outperforms BDNN and ConvLSTM on seen, unseen, and OOD conditions. (arXiv:2512.17281)
3x faster inference, 6000+ languages, 2MB model. Architecture: STFT → 4×Conv1d+ReLU → LSTM(128) → Conv1d → sigmoid. Processes 512-sample chunks (32ms) in <1ms on CPU.
ResNet50 + LSTM in spectro-temporal domain with sparsity-based dimension reduction. Adapts to stationary, non-stationary, and periodic noises. (Springer MTAP 2025)
Simpler feature combinations deliver robust performance without complex architectures. Demonstrates that classical features + light ML can match deep learning for VAD.
Cache-aware streaming with FastConformer (8x downsampling). Processes only new audio "deltas" — 3x higher efficiency than buffered inference. Key insight: VAD at frontend segmenting audio into chunks for incremental encoding.
Production VAD for voice agents requires: turn detection (hardest product problem), barge-in handling, <300ms first-token latency. Cascaded pipelines (VAD → ASR → LLM → TTS) dominate over end-to-end for control.