diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 5c5513d..f28b413 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -67,6 +67,7 @@ hf jobs run --flavor a100-large -d \ | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | | 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | +| 32 | 32-matched-architecture-arms.md | P1 | 17 | RUNNING — run-033 plane-large + run-034 BiLSTM on their corpus | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | ## Schedule diff --git a/TODO.sota/32-matched-architecture-arms.md b/TODO.sota/32-matched-architecture-arms.md new file mode 100644 index 0000000..13999a1 --- /dev/null +++ b/TODO.sota/32-matched-architecture-arms.md @@ -0,0 +1,16 @@ +# WO32 — the matched-architecture program (same corpus MUST win) + +Audit: no contamination either side; seq2seq converged at 10.03 → gap +is architecture + teacher. Arms: + +- run-033: byt5-large plane + full 900K silver (news-pure, K=3) — + their recipe class, stronger backbone. Gate: WN-2024 multiref < 10 + (stretch: single digits). +- run-034: literal BiLSTM tagger on the same corpus — reproduces their + class from scratch. Gate: WN-2024 < 15 validates the class at our + scale; comparison anchor. +- SOTA-stack layer above both: K-pass conditioning, byte-pretrained + backbones, preserve mode (already in the plane family). + +Ship rule: whichever arm dominates WikiNews-2024 becomes the +ara-diac-news successor; seq2seq line is closed (converged at 10.03). diff --git a/docs/RESULTS.md b/docs/RESULTS.md index cdd7118..1b45e52 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1397,3 +1397,25 @@ NOT triggered — scaling a weak teacher wastes it. The audio path remains open but requires: fine-tune the phoneme CTC on Hebrew audio (FLEURS he ~10h is exactly that training set), then relabel, then v1. Spec'd as WO31; needs owner green-light for the next compute block. + +## The "same corpus must win" program — convergence CLOSED, both matched arms in flight (2026-10-09) + +**Contamination audit: CLEAN** — 0/356 WikiNews-2024 test sentences +share even one 8-gram with the QCRI 5M-word corpus; their 2.70 is +honest generalization, ours is too. **run-029c (+2 epochs, seq2seq +specialist): 10.03/8.95** (from 10.13) — the seq2seq line has +CONVERGED; doubling training bought 0.10 WER. The remaining 7.3-point +gap is therefore ARCHITECTURE (tagging vs generation) + teacher quality, +and two arms now attack it directly on their exact recipe: + +- **run-033** (byt5-large PLANE = our tagging family + full 900K-unit + corpus, news-pure, 3ep, K=3, gradient checkpointing) — their class + with a far stronger backbone. +- **run-034** (literal BiLSTM tagger, Fadel/QCRI-style: char-emb → + 3×BiLSTM-512 → per-position haraqat head, from scratch, same corpus) + — their architecture, rebuilt and run at our data scale. + +Fixes en route (train#128-#131): haraqat split_planes mark-only crash +(QCRI silver has pure-combining-mark windows — latent library bug, now +tested); byt5-large OOM → PLANE_CHECKPOINT gradient checkpointing; +LSTM OOM → BIL_BS; padded-label loss smearing → ignore_index=-100.