From 6930e032a460d0dc0434007a997e45c77d3a66f7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 03:52:39 +0800 Subject: [PATCH] =?UTF-8?q?docs:=20WO32=20BiLSTM=20verdict=20=E2=80=94=201?= =?UTF-8?q?4.30/10.10;=20architecture-class=20hypothesis=20refuted,=20lexi?= =?UTF-8?q?cal=20channel=20is=20the=20battleground?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- docs/RESULTS.md | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index f28b413..41dd9c2 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -67,7 +67,7 @@ hf jobs run --flavor a100-large -d \ | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | | 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | -| 32 | 32-matched-architecture-arms.md | P1 | 17 | RUNNING — run-033 plane-large + run-034 BiLSTM on their corpus | +| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | ## Schedule diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 1b45e52..b8e2440 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1419,3 +1419,20 @@ Fixes en route (train#128-#131): haraqat split_planes mark-only crash (QCRI silver has pure-combining-mark windows — latent library bug, now tested); byt5-large OOM → PLANE_CHECKPOINT gradient checkpointing; LSTM OOM → BIL_BS; padded-label loss smearing → ignore_index=-100. + +## WO32 verdict 1 — the literal BiLSTM: architecture-class hypothesis REFUTED (2026-10-10) + +run-034 (char-embedding → 3×BiLSTM-512 → per-position haraqat head, +from scratch, news-pure 900K silver; fp32 after the bf16 NaN collapse): +WikiNews-2024 multiref **14.30 / 10.10**, SadeedDiac-25 **5.27**. + +Reading: reproducing their architecture CLASS on their corpus does NOT +reproduce their 2.70 — our BiLSTM is 5× their number and loses to our +own seq2seq (10.03/8.95). Their edge is not "BiLSTM" but the +components we did not replicate: the WORD-identity channel (Fadel's +design: word embeddings + char-CNN — lexical memorization is the +strongest diacritization feature), labels self-consistent with their +evaluation conventions (the corpus IS their model's output), and +tuning maturity. Implication: lexical knowledge is the battleground — +and byt5 byte pretraining (run-033, RUNNING) is our version of that +channel. The char-only BiLSTM closes as an anchor, not a contender.