diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index d3829ab..5c5513d 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -65,7 +65,8 @@ hf jobs run --flavor a100-large -d \ | 25 | 25-ar-news-specialist.md | P1 | 17 | VERDICT: OOD 10.13/8.98 (crown), ID 5.50; export staged | | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | -| 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | RUNNING (run-032, FLEURS audio-supervised student) | +| 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | ## Schedule diff --git a/TODO.sota/31-heb-phoneme-asr-v1.md b/TODO.sota/31-heb-phoneme-asr-v1.md new file mode 100644 index 0000000..8d9fa31 --- /dev/null +++ b/TODO.sota/31-heb-phoneme-asr-v1.md @@ -0,0 +1,17 @@ +# WO31 — Hebrew-tuned phoneme ASR (the v0 lesson, spec'd) + +WO30 v0 proved the pipeline but the teacher is the bottleneck: the +universal espeak-cv-ft CTC phonemizes Hebrew poorly. The fix (the +ReNikud recipe's first stage): + +1. Fine-tune wav2vec2-lv-60-espeak-cv-ft on FLEURS he_il (~10h, + ~3.6K train utterances) with espeak-ng reference phonemizations of + the TRANSCRIPTS as targets (self-supervised bootstrap: espeak reads + the text; the CTC learns to hear it in real audio). +2. Re-run the stage-2 alignment probe with the tuned ASR (expect PER + well under 0.395's already-passing mark). +3. Relabel FLEURS + retrain the v1 student; gate unchanged (<0.2393). +4. Only then does ivrit.ai scale ($500) make sense. + +Cost: one fine-tune (~2-4 GPU-hours) + relabel + v1 train. Owner +green-light for the block. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 35bca02..cdd7118 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1380,3 +1380,20 @@ three closed levers: crown **8.18** (data scale ✗ at 50K→90K, capacity ✗ byt5-large 8.48, noisy-student ✗ 8.72). The remaining Hebrew lever is the AUDIO campaign — WO30 v0 student RUNNING now on the GO-grade supervision. + +## WO30 verdict — Hebrew learned-IPA v0 FAILED: CER 1.18; the weak link is the ASR teacher (2026-10-09) + +run-032 v0 (FLEURS 4,359 audio-labeled pairs → byt5-small): training +converged on its labels (loss 0.11) but scores **CER 1.18** on the +phonikud board (gate <0.2393). Diagnosis, recorded before any scale +spend: (1) format divergence — the student faithfully reproduces the +ASR's space-separated phone style vs the board's compact stressed +strings; (2) the deeper cause — the universal phoneme CTC +(wav2vec2-espeak-cv-ft) is a weak Hebrew teacher (stage-2 PER 0.395 +was alignment-viability, not label quality; its FLEURS outputs showed +non-Hebrew phonemizations). ReNikud's pipeline builds a Hebrew-capable +phoneme recognizer FIRST. The $500 ivrit.ai scale decision is therefore +NOT triggered — scaling a weak teacher wastes it. The audio path +remains open but requires: fine-tune the phoneme CTC on Hebrew audio +(FLEURS he ~10h is exactly that training set), then relabel, then +v1. Spec'd as WO31; needs owner green-light for the next compute block.