Add live-path benchmark harness for transcription-quality tuning

.github/workflows/ci.yml

@@ -225,3 +225,15 @@ jobs:
           # pinned, so `no-deps` audits exactly those resolved versions.
           inputs: audit-requirements.txt
           no-deps: true
+          # MAL-2026-4750 is a *malicious-package* advisory in the OSV
+          # malicious-packages dataset that matches the name `fastapi`.
+          # It does NOT describe a vulnerability in the legitimate PyPI
+          # fastapi we depend on (the dataset tracks typosquats / backdoored
+          # impostors — e.g. the `fastapi-toolkit` backdoor and fake
+          # `fastapi` copies); the advisory records no fixed version, and
+          # 0.136.3 is the latest release, so there is nothing to bump to.
+          # Auditing it produces a permanent false positive on the real
+          # dependency. Drop this ignore if/when OSV scopes the advisory to
+          # the actual malicious artifact instead of the name `fastapi`.
+          ignore-vulns: |
+            MAL-2026-4750

.gitignore

@@ -23,6 +23,9 @@ recordings/
 # next run so re-launching is one Enter keystroke.
 .tapscribe-install.json
 
+# Live-benchmark results (tools/bench_live.py) — generated per run.
+bench-results/
+
 # Frontend typecheck tooling (devDep only, never shipped)
 frontend/node_modules/
 frontend/package-lock.json

docs/live-tuning-research.md

@@ -0,0 +1,175 @@
+# Live-transcription tuning: research notes
+
+Background for the live-quality investigation. The **live** path is not
+the batch path — it runs WhisperLiveKit (WlK) as a subprocess, fed by
+TapScribe's own Silero `SpeechGate`, with captions surfaced through
+`WlKRelay`. This note records what the two upstreams recommend, how our
+current defaults compare, and the concrete hypotheses the benchmark
+(`tools/bench_live.py`) should confirm or refute.
+
+It is research, not conclusions: the actual numbers come from running the
+sweep on a box that can load the models (see "Running the sweep").
+
+## The knobs and where they live
+
+| Layer | Set by | Flag / field |
+|---|---|---|
+| Model size + backend | WlK CLI | `--model`, `--backend` |
+| Streaming policy | WlK CLI | `--backend-policy` (simulstreaming / localagreement) |
+| Decode cadence | WlK CLI | `--min-chunk-size` |
+| Rolling-buffer reset | WlK CLI | `--buffer_trimming` (+ `_sec`) |
+| Token confirmation | WlK CLI | `--confidence-validation` |
+| Native VAD | WlK CLI | `--vac` / `--no-vac` |
+| TapScribe speech gate | `LiveConfig` → `SpeechGate` | `gate_speech_threshold`, `gate_hangover_ms`, `gate_pre_roll_ms`, `gate_min_speech_ms` |
+
+`tapscribe.live.build_live_cmd` builds the WlK argv; `tapscribe.speech_gate`
+runs the gate. With the default `gate_kind="tapscribe"`, build_live_cmd
+passes `--no-vac` (our gate does the gating) and `--confidence-validation`.
+
+## Our current live defaults vs upstream defaults
+
+From `tapscribe/__main__.py` and `tapscribe/live.py:LiveConfig`:
+
+| Setting | TapScribe default | Upstream default | Note |
+|---|---|---|---|
+| Whisper model | **`tiny.en`** | `base`/`medium` (WlK), version-dependent | **Smallest model — prime accuracy suspect.** |
+| `gate_kind` | `tapscribe` (→ `--no-vac`) | WlK VAC on | We replace WlK's VAC with our gate. |
+| `confidence_validation` | **`True`** (`--confidence-validation`) | off | WlK warns this is *faster but punctuation less accurate*. We opt into the speed/accuracy trade by default. |
+| `min_chunk_size` | unset → WlK default | ~1.0 s | More future context per decode = better accuracy + more lag. |
+| `buffer_trimming` | unset → WlK default | `segment` | WlK docs: `segment` performs better in their tests; `sentence` needs a segmenter. |
+| `backend-policy` (non-nb) | unset → WlK default `simulstreaming` (AlignAtt) | `simulstreaming` | nb-whisper path forces `localagreement`. Worth A/B-ing for the Whisper path. |
+| gate speech threshold | `0.5` | Silero `0.5` | Matches Silero's recommended default. |
+| gate hangover (→ `min_silence_duration_ms`) | `400 ms` | Silero `100 ms` | More conservative — avoids chopping mid-sentence pauses. |
+| gate pre-roll | `300 ms` ring flushed on open | Silero `speech_pad_ms` `30 ms` | Much larger lead-in recovery than Silero's pad — recovers leading consonants. |
+| gate `min_speech_ms` | `0` (open on first VAD "start") | — (no Silero equivalent) | `0` lets one-frame blips open the gate. |
+
+### Smells worth chasing
+
+1. **`tiny.en` is the live default.** The weakest Whisper model. The
+   single most likely cause of poor live quality.
+2. **`confidence_validation=True` by default.** Upstream explicitly says
+   it trades punctuation accuracy for speed. We enable it unconditionally.
+3. **The gate over-trims, badly, on noisy/continuous audio — confirmed
+   with real data** (see the gate-only benchmark below). On
+   `armstrong-en.wav` the default gate forwards only **~21 %** of the
+   audio and opens **once** (3.8–6.0 s), dropping the entire first phrase
+   *"That's one small step for man"* even though the clip is continuous
+   speech end-to-end. This is the strongest concrete lead for poor live
+   quality.
+
+## Gate-only benchmark (real data, model-free)
+
+The `SpeechGate` is the front half of the live path and runs entirely
+locally (Silero VAD loads with no network), so we can measure exactly how
+much audio each gate config forwards **without an ASR model**:
+
+```bash
+python tools/bench_live.py --gate-only
+```
+
+Results on the two fixtures (`fwd%` = frames forwarded to the backend;
+`segs` = silence→speech openings):
+
+| fixture | gate | thr | hang | fwd% | segs | kept_s / audio_s |
+|---|---|---|---|---|---|---|
+| armstrong-en | tapscribe | 0.50 | 400 | **20.8** | 1 | 2.5 / 12.0 |
+| armstrong-en | tapscribe | 0.30 | 400 | 22.5 | 1 | 2.7 / 12.0 |
+| armstrong-en | tapscribe | 0.70 | 400 | 20.8 | 1 | 2.5 / 12.0 |
+| armstrong-en | tapscribe | 0.50 | 800 | 24.0 | 1 | 2.9 / 12.0 |
+| armstrong-en | **backend** | — | — | **100.0** | 1 | 12.0 / 12.0 |
+| marlene-nb | tapscribe | 0.50 | 400 | **91.6** | 4 | 13.7 / 15.0 |
+| marlene-nb | backend | — | — | 100.0 | 1 | 15.0 / 15.0 |
+
+**Reading it:**
+
+- The Armstrong clip is continuous speech at −12 to −21 dBFS (no real
+  silence — verified by an RMS energy profile), yet the gate opens for a
+  single 2.2 s window at the **loudest** passage and discards the rest.
+  Lowering `gate_speech_threshold` to 0.3 barely changes it (22.5 %), so
+  this is **not** a simple threshold tweak — Silero classifies the quieter
+  (but still clearly-spoken) passages as non-speech on this noisy old
+  recording. In production a model gets only that 2.2 s, so most words are
+  never transcribed.
+- The Marlene clip (a clean studio reading) forwards ~92 % across 4
+  segments — the gate behaves well on clean speech. **So the failure is
+  audio-dependent: the gate collapses on noisy / low-level / continuous
+  speech, which is exactly what real meeting audio looks like.**
+- `gate_kind=backend` (no TapScribe gate; WlK's own VAC) forwards 100 %.
+  A strong A/B candidate, and the quickest mitigation to validate.
+
+This points the live-quality investigation squarely at the gate. Open
+questions for the code review / full sweep: is the single-open behaviour a
+Silero limitation on noisy audio, or a bug in `SpeechGate.feed()` /
+`make_silero_vad` (e.g. missing `speech_pad`, the 512-vs-320 sample
+buffering, or the gate never re-opening)? The `gate_kind` A/B in the full
+sweep quantifies the transcription cost directly (deletions).
+
+## WhisperLiveKit recommendations (upstream)
+
+- **`--buffer_trimming segment`** is the better-tested default and needs
+  no sentence segmenter; `sentence` trims on confirmed punctuation but
+  needs the segmenter installed.
+- **`--min-chunk-size`** should align with the frontend's chunk cadence;
+  larger = more context per decode = better accuracy but more lag. Tune
+  against the per-tap lag reading.
+- **`--backend auto`** picks MLX on macOS, then Faster-Whisper, then
+  Whisper. On Apple Silicon, MLX is the fast path (the harness sets
+  `--backend mlx-whisper` when `use_mlx`).
+- **`--backend-policy`**: `simulstreaming` (AlignAtt, the default) vs
+  `localagreement`. AlignAtt has `--frame-threshold` (default 25) for the
+  speed/accuracy trade.
+- General advice: **benchmark latency + accuracy for your model and VAD
+  settings on your actual device**, using `--warmup-file` and watching
+  CPU/latency. That is exactly what `tools/bench_live.py` automates.
+
+## Silero VAD recommendations (upstream)
+
+- **`threshold` 0.5** is a good lazy default; tune per dataset. (We use
+  0.5.)
+- **`min_silence_duration_ms` 100 ms** default — silence to wait before
+  closing a chunk. (We map `gate_hangover_ms=400 ms` here — more
+  conservative, fewer mid-sentence cuts.)
+- **`speech_pad_ms` 30 ms** default — pads each side of a chunk. (We don't
+  use Silero's pad; our `gate_pre_roll_ms=300 ms` ring buffer does the
+  lead-in recovery, much larger.)
+- Sampling rate must be 16 kHz for `VADIterator` (our wire format).
+
+## Hypotheses for the sweep to test
+
+1. Moving `tiny.en` → `base.en` → `small.en` drops WER substantially
+   (substitutions fall). **Most likely the main fix.**
+2. `gate_kind=backend` vs `tapscribe`: if the tapscribe gate is clipping
+   speech, the `tapscribe` rows show more **deletions** than `backend`.
+3. `confidence_validation` off improves accuracy (at some latency cost) —
+   add a sweep row toggling it once 1–2 are understood.
+4. `min_chunk_size=1.0` and/or `buffer_trimming=segment` change the
+   latency (`lag_*`, `final_delay_s`) / accuracy balance.
+
+## Running the sweep
+
+```bash
+pip install -e ".[whisper,bench]"          # whisperlivekit + faster-whisper + jiwer
+python tools/bench_live.py --sweep         # matrix over every fixture → bench-results/*.json
+# or a single config, detailed:
+python tools/bench_live.py tests/fixtures/audio/armstrong-en.wav --model base.en
+```
+
+On Apple Silicon the harness auto-selects the MLX backend; elsewhere it
+uses faster-whisper (CPU/CUDA). The metrics to read first: **WER** plus
+the **sub / del / ins** split (substitutions ⇒ weak model, deletions ⇒
+dropped/clipped speech, insertions ⇒ hallucination), then **lag** and
+**gate forward %**.
+
+> **Note on the managed dev box:** model weights download from
+> HuggingFace, which the Claude-Code-on-the-web network policy blocks
+> ("Host not in allowlist"). The harness reports this as a clean per-run
+> error rather than hanging, but the sweep itself must run on a machine
+> with model access (e.g. your Mac). The scoring, gate, and framing paths
+> were validated here without a model.
+
+## Sources
+
+- [WhisperLiveKit (GitHub)](https://github.com/QuentinFuxa/WhisperLiveKit)
+- [WhisperLiveKit default & custom models](https://github.com/quentinfuxa/whisperlivekit/blob/main/docs/default_and_custom_models.md)
+- [Silero VAD (GitHub)](https://github.com/snakers4/silero-vad)
+- [ufal/whisper_streaming (the streaming algorithm WlK builds on)](https://github.com/ufal/whisper_streaming)

pyproject.toml

@@ -142,6 +142,11 @@ parakeet = [
     "tapscribe[parakeet-cpu]; sys_platform != 'darwin' or platform_machine != 'arm64'",
 ]
 canary = ["tapscribe[canary-cpu]"]
+# Scoring deps for the live-path benchmark harness (tools/bench_live.py):
+# jiwer computes WER/CER + the hit/sub/del/ins breakdown the sweep
+# reports. Kept out of `dev` so the unit-test CI install stays lean —
+# only operators running the benchmark pull it in.
+bench = ["jiwer>=3.0"]
 dev = [
     "pytest>=8",
     "pytest-asyncio>=0.23",
@@ -199,6 +204,7 @@ markers = [
     "chaos: race-condition / error-recovery test exercising background tasks, abnormal close paths, or repeated rapid mutations. Opt out with -m 'not chaos'.",
     "browser_e2e: requires Playwright + Chromium (`pip install -e .[dev] && python -m playwright install chromium`). Opt out with -m 'not browser_e2e'.",
     "real_silero: opt out of the autouse silero stub in tests/conftest.py — exercises the real `detect_speech_silero` import path (only set on tests that need it).",
+    "real_live: drives the full live path (whisperlivekit-server child + SpeechGate + WlKRelay) on a real audio fixture and asserts a WER threshold. Skipped unless whisperlivekit + faster-whisper + jiwer are importable and the model can load. Heavy + downloads weights — not in the default CI matrix.",
 ]
 
 [tool.ruff]

tapscribe/live.py

@@ -18,6 +18,7 @@
 
 from __future__ import annotations
 
+import contextlib
 import errno
 import os
 import shutil
@@ -147,6 +148,13 @@ class LiveConfig:
     # equivalent knob — this filter lives entirely in SpeechGate.
     gate_min_speech_ms: int = 0
     confidence_validation: bool = True
+    # WlK transcription policy. None = use WlK's default ("simulstreaming",
+    # AlignAtt — commits tokens as it decodes, keeps no unvalidated buffer,
+    # so `buffer_transcription` stays empty and the dashboard's in-flight
+    # preview never shows). "localagreement" holds tokens until they agree
+    # across chunks, which populates `buffer_transcription`. Only emitted to
+    # argv for non-nb models; nb-whisper always forces localagreement.
+    backend_policy: str | None = None
     # Forwarded to whisperlivekit-server when set — see build_live_cmd.
     min_chunk_size: float | None = None
     buffer_trimming: str | None = None  # "sentence" | "segment"
@@ -219,6 +227,8 @@ def build_live_cmd(
         cmd.extend(["--model", config.model])
         if use_mlx:
             cmd.extend(["--backend", "mlx-whisper"])
+        if config.backend_policy is not None:
+            cmd.extend(["--backend-policy", config.backend_policy])
 
     if init_prompt:
         cmd.extend(["--init-prompt", init_prompt])
@@ -677,6 +687,14 @@ def _pump_logs(self, proc: subprocess.Popen) -> None:
                             promoted = True
         finally:
             rc = proc.wait()
+            # Close the stdout pipe we've drained to EOF. Popen would
+            # eventually close it on GC, but that leaks an fd per child
+            # across the dashboard's stop→start "Apply model" restarts
+            # (and emits a ResourceWarning under -W error). The pump owns
+            # this end of the pipe, so closing it here is the clean spot.
+            if proc.stdout is not None:
+                with contextlib.suppress(Exception):
+                    proc.stdout.close()
             # Only update INFO if this proc is still the active one; a fresh
             # start() may already have replaced it.
             if self._proc is proc:

tapscribe/live_relay.py

@@ -104,13 +104,6 @@ def __init__(
         # consecutive same-speaker segments — see
         # `tokens_alignment.py`) only emits the new suffix.
         self._emitted_by_key: dict[tuple, str] = {}
-        # Non-tail entries are immutable in WlK's wire format — the
-        # merger case only ever modifies the CURRENT tail in place;
-        # once a newer entry appears after a position, that position
-        # is frozen. Cache the lower bound of the scan so we don't
-        # re-walk hundreds of stable lines on every snapshot for long
-        # sessions.
-        self._last_emit_scan_upto: int = 0
         # Tail-stability bookkeeping: counts how many consecutive
         # snapshots the tail's `(key, text)` has matched while the
         # buffer was empty. Reset whenever the tail changes or buffer
@@ -245,17 +238,19 @@ async def _consume(self) -> None:
                 if not isinstance(snapshot, list):
                     continue
                 self._last_snapshot = snapshot
-                # Emit every newly-non-tail entry. Non-tail positions
-                # are immutable in WlK's wire format, so we only need
-                # to scan from the last upto we processed. The dedup
-                # in `_consider_emit_line` covers a tail-becoming-
-                # non-tail whose text already settled (no re-emit) and
-                # the rare same-key text-change case (emits the diff).
+                # Emit every non-tail entry (the tail is held back for the
+                # stability flush below). We re-scan ALL non-tail positions
+                # each snapshot rather than caching a lower bound: although
+                # WlK normally only mutates the current tail, a late
+                # LocalAgreement correction can grow an already-committed
+                # line, and freezing non-tail positions would silently drop
+                # that suffix until close (then deliver it out of order).
+                # `_consider_emit_line` is keyed + idempotent, so re-walking
+                # settled lines is a no-op beyond a dict lookup per line —
+                # negligible at WlK's few-Hz snapshot rate.
                 upto = max(0, len(snapshot) - 1)
-                for i in range(self._last_emit_scan_upto, upto):
+                for i in range(upto):
                     self._consider_emit_line(snapshot[i])
-                if upto > self._last_emit_scan_upto:
-                    self._last_emit_scan_upto = upto
                 # Tail-stability flush: once the tail text has held
                 # steady (with the in-flight buffer empty) for enough
                 # consecutive snapshots, WlK has nothing more to

tapscribe/tap_fan_out.py

@@ -149,6 +149,12 @@ async def write_frame(self, buf: bytes) -> None:
             if current_open != self._gate_open_last:
                 self._gate_open_last = current_open
                 await self._recorder.streams.update_gate_open(self._conn_id, current_open)
+                if not current_open:
+                    # Tap just went idle — drop any stale lag immediately so
+                    # the dashboard stops showing a backlog for a speaker who
+                    # stopped talking, rather than waiting for _on_metrics to
+                    # next fire (and it now suppresses while closed anyway).
+                    await self._recorder.streams.update_lag(self._conn_id, None)
         else:
             frames_to_send = (buf,)
 
@@ -277,7 +283,19 @@ async def _open(self) -> None:
 
     async def _on_metrics(self, lag_s: float) -> None:
         """Push the relay's latest reported lag to this tap's row so the
-        dashboard can render a per-tap backlog indicator."""
+        dashboard can render a per-tap backlog indicator.
+
+        Suppressed while the TapScribe gate is closed: WlK keeps emitting
+        `remaining_time_transcription` even after we stop feeding it, but
+        that value is `wall_clock - last_processed_audio` — it climbs purely
+        because time passes during the silence we're gating out, not because
+        there's a real decode backlog. Reporting it would show a phantom,
+        ever-growing lag for a tap whose speaker has gone quiet. When the
+        gate is open (actively forwarding, hangover included) the number is
+        genuine. Backend-gate mode (`self._gate is None`) feeds WlK
+        continuously, so its lag stays meaningful and is always forwarded."""
+        if self._gate is not None and not self._gate.is_open:
+            return
         await self._recorder.streams.update_lag(self._conn_id, lag_s)
 
     def _on_buffer(self, text: str) -> None: