From 543ec10afb3b95abf6e9f192bb721610b16d83e7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Fri, 9 Oct 2026 15:28:59 +0800 Subject: [PATCH] =?UTF-8?q?docs:=20WO30=20negative=20=E2=80=94=20teacher?= =?UTF-8?q?=20bottleneck=20diagnosed;=20WO31=20(Hebrew-tuned=20phoneme=20A?= =?UTF-8?q?SR)=20spec'd?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 3 ++- TODO.sota/31-heb-phoneme-asr-v1.md | 17 +++++++++++++++++ docs/RESULTS.md | 17 +++++++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 TODO.sota/31-heb-phoneme-asr-v1.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index d3829abd..5c5513d7 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -65,7 +65,8 @@ hf jobs run --flavor a100-large -d \ | 25 | 25-ar-news-specialist.md | P1 | 17 | VERDICT: OOD 10.13/8.98 (crown), ID 5.50; export staged | | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | -| 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | RUNNING (run-032, FLEURS audio-supervised student) | +| 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | ## Schedule diff --git a/TODO.sota/31-heb-phoneme-asr-v1.md b/TODO.sota/31-heb-phoneme-asr-v1.md new file mode 100644 index 00000000..8d9fa312 --- /dev/null +++ b/TODO.sota/31-heb-phoneme-asr-v1.md @@ -0,0 +1,17 @@ +# WO31 — Hebrew-tuned phoneme ASR (the v0 lesson, spec'd) + +WO30 v0 proved the pipeline but the teacher is the bottleneck: the +universal espeak-cv-ft CTC phonemizes Hebrew poorly. The fix (the +ReNikud recipe's first stage): + +1. Fine-tune wav2vec2-lv-60-espeak-cv-ft on FLEURS he_il (~10h, + ~3.6K train utterances) with espeak-ng reference phonemizations of + the TRANSCRIPTS as targets (self-supervised bootstrap: espeak reads + the text; the CTC learns to hear it in real audio). +2. Re-run the stage-2 alignment probe with the tuned ASR (expect PER + well under 0.395's already-passing mark). +3. Relabel FLEURS + retrain the v1 student; gate unchanged (<0.2393). +4. Only then does ivrit.ai scale ($500) make sense. + +Cost: one fine-tune (~2-4 GPU-hours) + relabel + v1 train. Owner +green-light for the block. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 35bca029..cdd7118c 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1380,3 +1380,20 @@ three closed levers: crown **8.18** (data scale ✗ at 50K→90K, capacity ✗ byt5-large 8.48, noisy-student ✗ 8.72). The remaining Hebrew lever is the AUDIO campaign — WO30 v0 student RUNNING now on the GO-grade supervision. + +## WO30 verdict — Hebrew learned-IPA v0 FAILED: CER 1.18; the weak link is the ASR teacher (2026-10-09) + +run-032 v0 (FLEURS 4,359 audio-labeled pairs → byt5-small): training +converged on its labels (loss 0.11) but scores **CER 1.18** on the +phonikud board (gate <0.2393). Diagnosis, recorded before any scale +spend: (1) format divergence — the student faithfully reproduces the +ASR's space-separated phone style vs the board's compact stressed +strings; (2) the deeper cause — the universal phoneme CTC +(wav2vec2-espeak-cv-ft) is a weak Hebrew teacher (stage-2 PER 0.395 +was alignment-viability, not label quality; its FLEURS outputs showed +non-Hebrew phonemizations). ReNikud's pipeline builds a Hebrew-capable +phoneme recognizer FIRST. The $500 ivrit.ai scale decision is therefore +NOT triggered — scaling a weak teacher wastes it. The audio path +remains open but requires: fine-tune the phoneme CTC on Hebrew audio +(FLEURS he ~10h is exactly that training set), then relabel, then +v1. Spec'd as WO31; needs owner green-light for the next compute block.