Skip to content

docs: WO32 BiLSTM verdict #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 9, 2026
Merged

docs: WO32 BiLSTM verdict #291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion TODO.sota/00-README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ hf jobs run --flavor a100-large -d \
| 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic |
| 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck |
| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) |
| 32 | 32-matched-architecture-arms.md | P1 | 17 | RUNNING — run-033 plane-large + run-034 BiLSTM on their corpus |
| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING |
| 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed |

## Schedule
Expand Down
17 changes: 17 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1419,3 +1419,20 @@ Fixes en route (train#128-#131): haraqat split_planes mark-only crash
(QCRI silver has pure-combining-mark windows — latent library bug, now
tested); byt5-large OOM → PLANE_CHECKPOINT gradient checkpointing;
LSTM OOM → BIL_BS; padded-label loss smearing → ignore_index=-100.

## WO32 verdict 1 — the literal BiLSTM: architecture-class hypothesis REFUTED (2026-10-10)

run-034 (char-embedding → 3×BiLSTM-512 → per-position haraqat head,
from scratch, news-pure 900K silver; fp32 after the bf16 NaN collapse):
WikiNews-2024 multiref **14.30 / 10.10**, SadeedDiac-25 **5.27**.

Reading: reproducing their architecture CLASS on their corpus does NOT
reproduce their 2.70 — our BiLSTM is 5× their number and loses to our
own seq2seq (10.03/8.95). Their edge is not "BiLSTM" but the
components we did not replicate: the WORD-identity channel (Fadel's
design: word embeddings + char-CNN — lexical memorization is the
strongest diacritization feature), labels self-consistent with their
evaluation conventions (the corpus IS their model's output), and
tuning maturity. Implication: lexical knowledge is the battleground —
and byt5 byte pretraining (run-033, RUNNING) is our version of that
channel. The char-only BiLSTM closes as an anchor, not a contender.
Loading