From f215af31c42ccbadaf8ad46faebed52aa90be5df Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 06:48:12 +0800 Subject: [PATCH 01/68] =?UTF-8?q?docs:=20WO33=20word=20channel=20=E2=80=94?= =?UTF-8?q?=20lexical=20floor=2025.15=20WER=20measured,=20variant-disambig?= =?UTF-8?q?uator=20arm=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 1 + TODO.sota/33-lexicon-disambiguator.md | 41 +++++++++++++++++++++++++++ docs/RESULTS.md | 21 ++++++++++++++ 3 files changed, 63 insertions(+) create mode 100644 TODO.sota/33-lexicon-disambiguator.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 41dd9c2..3a858fa 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -69,6 +69,7 @@ hf jobs run --flavor a100-large -d \ | 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | +| 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | RUNNING — word channel arm; gates 14.30 / 10.13 (WN-2024) | ## Schedule diff --git a/TODO.sota/33-lexicon-disambiguator.md b/TODO.sota/33-lexicon-disambiguator.md new file mode 100644 index 0000000..1c92950 --- /dev/null +++ b/TODO.sota/33-lexicon-disambiguator.md @@ -0,0 +1,41 @@ +# WO33 — the word-identity channel: variant-disambiguation tagger + +Decomposition (2026-10-09, local probe, same corpus): + +| arm | WikiNews-2024 WER | note | +|---|---|---| +| pure most-frequent-reading lookup | 25.15 (DER 13.18) | no model; 482,609-type table; OOV 18.10% | +| literal char BiLSTM (run-034) | 14.30 (DER 10.10) | char context, no word channel | +| byt5-base seq2seq specialist (run-029, shipped) | 10.13 (DER 8.98) | implicit lexicon via byte pretraining | +| byt5-large plane + 900K silver (run-033) | running | their recipe class, stronger backbone | + +Lexical memorization is 3/4 of the task; the remaining ~25 points are +context disambiguation among a word's few observed readings — the word +channel char-level models approximate only indirectly. QCRI's own +design (Fadel) is word-level: word-embedding + char-CNN + BiLSTM over +variant indices. + +Arm: run-035 — variant-disambiguation tagger on the same corpus as +run-033/034. Word-emb (128) + char-CNN (k=2..5, 50 filters each) -> +BiLSTM 2x512 -> head over {variant idx 0..5, OOV}. Labels from the +variant table (most-frequent = 0). Decode emits the chosen variant +string; output preserves the input skeleton letter-exactly. fp32 +(LSTM bf16 = NaN). ~62M params, bs 128, 5 epochs, a100, ~2h. + +Gates: +- WN-2024 multiref < 14.30 (beats the char-only BiLSTM) → the word + channel is real at our scale. +- WN-2024 < 10.13 (beats the shipped specialist) → becomes the + ara-diac-news successor; then stack SOTA layers (K-pass, larger + backbone, preserve mode). +- Fails both → the gap is not the word channel either; remaining + hypothesis is corpus-internal convention alignment, closed by the + register-specialist doctrine (crown = ID specialist + news + specialist shipped separately). + +Ship rule: same as WO32 — dominant arm on WN-2024 becomes the news +successor; export via int8 IMF v1 chain only if it beats 10.13. + +Complement probe (post-verdict): oracle-min over {run-029, run-035} +bounds the ensemble ceiling; if >2 points over the best single arm, +a routing/blending follow-up (WO28 lineage) opens. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b8e2440..d54c580 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1436,3 +1436,24 @@ evaluation conventions (the corpus IS their model's output), and tuning maturity. Implication: lexical knowledge is the battleground — and byt5 byte pretraining (run-033, RUNNING) is our version of that channel. The char-only BiLSTM closes as an anchor, not a contender. + +## WO26 rerun + WO33 launch — Hebrew noisy student re-confirmed negative; the word channel decomposed (2026-10-10) + +run-030-heb-noisy (HF rerun of the noisy student: 50K gold + 40K +hewiki pseudo, K=3, 16,350 steps): **DER 8.72** vs crown 8.18 — gate +failed, WO26 negative re-confirmed on the new substrate. Hebrew text +levers remain exhausted (capacity 8.48, noisy 8.72, convergence 10.03); +the remaining path is audio (WO09/31). + +WO33 — lexical-channel decomposition (local, free): pure +most-frequent-reading lookup over the same corpus scores **25.15 WER / +13.18 DER** with **18.10% OOV** (482,609-type table). So: lexical +memorization ≈ 75% of words; the rest is context disambiguation among +each word's few observed readings — exactly the word-identity channel +(WO32's battleground). run-035 (RUNNING): variant-disambiguation +tagger — word-emb 128 + char-CNN (k=2..5) → BiLSTM 2×512 → head over +{variant 0..5, OOV}, 62M params, fp32, same corpus as run-033/034. +Gates: <14.30 = word channel real; <10.13 = new news-successor +candidate; else the residue is convention alignment, closed under the +register-specialist doctrine. Complement probe (oracle-min over +run-029/035) staged post-verdict. From a6cff3d12d81978c9927e999aa73c145de71b413 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 06:49:55 +0800 Subject: [PATCH 02/68] =?UTF-8?q?docs:=20live=20graph=20+=20SOTA=20matrix?= =?UTF-8?q?=20to=202026-10-10=20=E2=80=94=20run-033/035=20arms,=20text-lev?= =?UTF-8?q?er=20closures,=20WO31=20audio=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 50 +++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 3a858fa..5b1ebf4 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -78,34 +78,38 @@ Wave 2 (HF training, launched together): WO04 (a100-large), WO06 (a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`. Wave 3: WO05 tables after 01+07 land; WO09 probe only. -## Path-to-success graph + SOTA matrix (2026-10-07, live) +## Path-to-success graph + SOTA matrix (2026-10-10, live) ``` -ARABIC ── in-domain (SadeedDiac-25) HEBREW ── nikud (nakdimon) - r7 2.2864 [BEST dedicated] heb-plane-2.0 8.18 [BEST text-only] - Claude-3.7 1.39 (API LLM; hallucinates) next: us (12.48) → 16.4 → 35.6 - │ │ - └─ r8a (self-labeled arwiki) RUNNING └─ byt5-large/K=4 (untried) - └─ r8b (QCRI silver) RUNNING HEBREW ── g2p IPA (their board) - └─ gate: hold ID ≤2.2864 AND rules chain 0.2393 CER (0-train) - move OOD 17.38 → ≤10 ReNikud 0.0244 (audio-supervised) - └─ learned IPA head: BLOCKED -ARABIC ── OOD (WikiNews-2024 multiref) (espeak ceiling 0.4543; needs - r7 17.38/11.83 · plane 18.69/11.35 MILIM or audio = 2027) - QCRI 2.70 (in-domain-trained silver) - └─ register closure = r8a/r8b above THAI ── g2p (kaikki protocol) - hybrid 0.1478 PER @0.73ms [BEST] -ARABIC plane ── 2.7397 (large) ── r8 recipe tiny student RUNNING (<10MB tier) - transfers to plane family post-r8 CLIENT SURFACE — 3-runtime parity ✓ - + preserve mode ✓ + edge /v1/infer ✓ - + index-v9 + 28 sha-verified models +ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER) + run-029 seq2seq specialist 10.13/8.98 [SHIPPED: ara-diac-news-1.0; converged 10.03/8.95] + run-034 literal BiLSTM 14.30/10.10 [anchor — arch-class REFUTED] + pure-lexicon floor 25.15/13.18 [OOV 18.10% — lexical = 75% of task] + QCRI published 2.70 [their corpus + their conventions] + ├─ run-033 byt5-large plane + 900K silver RUNNING gate <10 + └─ run-035 word-variant disambiguator RUNNING gates <14.30 / <10.13 + winner → IMF v1 export → release chain (owner version) → news successor + both fail → register-specialist doctrine stands; residue = convention + alignment, unreachable without their labels (closed) + +ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated] · specialist 5.50 (register cost) + +HEBREW ── nikud DER: 8.18 crown (heb-diac-plane-2.0, 2× runner-up) + ALL TEXT LEVERS CLOSED by measurement: capacity 8.48 · noisy 8.72 (run-030 + re-confirmed) · convergence 10.03 → remaining path = AUDIO program: + WO31 phoneme-ASR fine-tune → alignment → learned-IPA student (owner block) + +HEBREW ── g2p CER: rules 0.2393 · ReNikud 0.0244 (audio-supervised; WO31 line) +THAI ── PER: hybrid 0.1478 @0.73ms [crown]; tiny tier closed NEGATIVE (2038%) +CLIENT SURFACE: 3-runtime parity ✓ · preserve ✓ · edge /v1/infer ✓ + index-v10 · npm 5.7.0 · 29 HF mirrors · ruby pins 802/804 ``` | axis | ours | 2026 frontier | status | |---|---|---|---| -| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated; r8 arms closing | -| AR OOD multiref WER | 17.38 | 2.70 (their-domain) | register gap; r8a/r8b attacking | -| HE nikud DER | 8.18 | — (we lead) | crown held; base+K=3 | -| HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | blocked on supervision (2027) | +| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated (LLM = no-teacher-law, not a runtime option) | +| AR OOD multiref WER | 10.13 (specialist) | 2.70 (their-domain) | converged seq2seq; run-033/035 arms live | +| HE nikud DER | 8.18 | — (we lead) | crown held; text exhausted; audio = WO31 | +| HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | blocked on WO31 owner block | | TH PER | 0.1478 | 26.9 (latency play) | crown held + hybrid latency tier | | client surface | 3 runtimes + edge | none exists | moat maintained | From b17722da3908abce78a61038da5d3dd727f0839e Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 09:50:18 +0800 Subject: [PATCH 03/68] docs: WO33 negative isolates the halves; WO34 hybrid arm + style-probe closure --- TODO.sota/00-README.md | 3 ++- TODO.sota/34-hybrid-word-char.md | 42 ++++++++++++++++++++++++++++++++ docs/RESULTS.md | 21 ++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 TODO.sota/34-hybrid-word-char.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 5b1ebf4..0bdbfb8 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -69,7 +69,8 @@ hf jobs run --flavor a100-large -d \ | 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | -| 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | RUNNING — word channel arm; gates 14.30 / 10.13 (WN-2024) | +| 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | RUNNING — char+word hybrid; style hypothesis closed (density 74-81%) | ## Schedule diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md new file mode 100644 index 0000000..d945614 --- /dev/null +++ b/TODO.sota/34-hybrid-word-char.md @@ -0,0 +1,42 @@ +# WO34 — the synthesis arm: char-level BiLSTM + word-embedding feature + +run-035 (WO33) verdict, 2026-10-10: word-level variant-disambiguation +tagger scored **18.07 / 11.44** (WikiNews-2024 multiref) and Sadeed +Total DER **18.75** — both gates FAILED (<14.30, <10.13). Diagnostic +value: the word-only design emits bare (undiacritized) words for the +18.1% OOV types, which alone collapses ID DER; the char-only BiLSTM +(run-034, 14.30/10.10) composes unseen words but has no word-identity +channel. Each half in isolation loses to the shipped specialist +(10.13/8.98). + +Convention-style hypothesis CLOSED same day (local probe): marked-letter +density is 0.8127 (WikiNews-2024 refs, first-alt) / 0.7625 +(wikinews2014 gold) / 0.7432 (news silver) — same full-diacritization +family; no partial-vs-full style mismatch to harvest. + +The arm: run-036 — Fadel's actual joint design at our scale. Char-emb +128 ⊕ per-char word-embedding (bare-form vocab, isalpha core, +300K cap) → 3×BiLSTM-512 → per-position plane-combo head. Same corpus +as run-033/034/035, fp32, bs 64, 6 epochs, a100. + +Gates: +- WN-2024 multiref < 14.30 → word channel validated at char level; + hybrid becomes the stacking backbone. +- WN-2024 multiref < 10.13 → news-successor candidate; stack SOTA + layers (K-pass conditioning, larger corpus, preserve mode) and run + the IMF v1 export chain. +- ≥ 14.30 → word-identity is NOT the residual gap at our data scale; + remaining hypotheses: tuning maturity (lr/epoch sweep), their + evaluator-side conventions (closed), or data scale beyond 900K + silver units. Doctrine stands. + +Staged follow-ups (armed post-verdict): +1. Variant-constrained decode (v2): force the char model's output for + in-table words to the nearest table variant; keeps compositional + OOV handling, adds lexical precision. +2. Oracle-min complement probe over {run-029, run-036} WikiNews preds + (run-036 saves wikinews_preds.txt): if the ceiling is >2 WER over + the best single arm, confidence-routed blending opens (WO28 + lineage; routing, never naive voting). +3. If run-033 (byt5-large plane) lands <10, race it against run-036's + gates; the dominant arm takes the successor slot. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index d54c580..beb4852 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1457,3 +1457,24 @@ Gates: <14.30 = word channel real; <10.13 = new news-successor candidate; else the residue is convention alignment, closed under the register-specialist doctrine. Complement probe (oracle-min over run-029/035) staged post-verdict. + +## WO33 verdict + WO34 launch — word-only fails on OOV; the synthesis arm is the joint design (2026-10-10) + +run-035 (word-level variant disambiguator): **18.07 / 11.44** multiref, +Sadeed Total DER 18.75 — both gates failed. The failure mode is clean: +18.1% of benchmark word types are OOV to the corpus table and are +emitted bare, collapsing ID DER; the word channel in isolation cannot +compose unseen words. Combined with run-034 (char-only, 14.30/10.10), +the decomposition now says each half alone loses to the shipped +specialist (10.13/8.98) — Fadel's joint design (word identity AS A +FEATURE of a char-level tagger) is the actual recipe. + +Convention-style probe (local): marked-letter density 0.8127 / +0.7625 / 0.7432 across WikiNews-2024 refs, wikinews2014 gold, and +news silver — the style-mismatch hypothesis is closed; there are no +cheap diacritic-drop points. + +run-036 (RUNNING): char-emb ⊕ per-char bare-word embedding → 3×BiLSTM +→ plane head; same corpus, fp32. Gates <14.30 (word channel validated) +/ <10.13 (news-successor candidate). Variant-constrained decode and +the oracle-min complement probe staged post-verdict. From c5cf3a3ce62a2328cd9a69cbe404ed08219228b0 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 11:44:34 +0800 Subject: [PATCH 04/68] docs: WO34 staged arm run-037 (fastText word channel) + probe tooling built --- TODO.sota/00-README.md | 2 +- TODO.sota/34-hybrid-word-char.md | 22 +++++++++++++++++----- docs/RESULTS.md | 6 ++++++ 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 0bdbfb8..758bef2 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -70,7 +70,7 @@ hf jobs run --flavor a100-large -d \ | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | -| 34 | 34-hybrid-word-char.md | P1 | 32/33 | RUNNING — char+word hybrid; style hypothesis closed (density 74-81%) | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | run-036 RUNNING 48%; run-037 fastText arm STAGED; oracle probe BUILT | ## Schedule diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md index d945614..5f4f7e6 100644 --- a/TODO.sota/34-hybrid-word-char.md +++ b/TODO.sota/34-hybrid-word-char.md @@ -34,9 +34,21 @@ Staged follow-ups (armed post-verdict): 1. Variant-constrained decode (v2): force the char model's output for in-table words to the nearest table variant; keeps compositional OOV handling, adds lexical precision. -2. Oracle-min complement probe over {run-029, run-036} WikiNews preds - (run-036 saves wikinews_preds.txt): if the ceiling is >2 WER over - the best single arm, confidence-routed blending opens (WO28 - lineage; routing, never naive voting). -3. If run-033 (byt5-large plane) lands <10, race it against run-036's +2. Oracle-min complement probe over {run-029, run-036} WikiNews preds. + Tooling BUILT and unit-tested (oracle_min.py in /code bundle — + mirrors the multiref scorer exactly, verified on synthetic + disjoint-strengths cases). run-036 saves wikinews_preds.txt; + run-029's preds regenerate via eval_r8_wikinews_preds.py (exact + protocol: 600-byte windows, greedy, project_haraqat; imports the + original trainer's helpers). If oracle >2 WER over the best single + arm, confidence-routed blending opens (WO28 lineage; routing, + never naive voting). +3. run-037 (STAGED, script uploaded): run-036's architecture with the + scratch word channel replaced by pretrained fastText cc.ar.300 + vectors (in-job download, coverage-reported, trainable init) — the + lexical-density hypothesis: our word channel is undertrained on + ~3M words, not wrong; Fadel-class systems ride on billion-word + vectors. Same gates. Launches when a GPU slot frees regardless of + run-036's outcome (independent axis: scratch vs pretrained). +4. If run-033 (byt5-large plane) lands <10, race it against run-036/37 gates; the dominant arm takes the successor slot. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index beb4852..5d4a242 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1478,3 +1478,9 @@ run-036 (RUNNING): char-emb ⊕ per-char bare-word embedding → 3×BiLSTM → plane head; same corpus, fp32. Gates <14.30 (word channel validated) / <10.13 (news-successor candidate). Variant-constrained decode and the oracle-min complement probe staged post-verdict. + +Tooling built while arms grind (2026-10-10): oracle_min.py (unit-tested +complement probe mirroring the multiref scorer); eval_r8_wikinews_preds +(exact-protocol regen of run-029 WikiNews preds); run-037 trainer +(hybrid + pretrained fastText cc.ar.300 word channel — the +lexical-density hypothesis; staged, launches on slot-free). From 9c0d0b4dae37b5a8928f91df739554978fb8cb04 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 12:50:50 +0800 Subject: [PATCH 05/68] docs: run-036 timeout incident + relaunch; run-037 hardened with resume/cache --- docs/RESULTS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 5d4a242..e97a59d 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1484,3 +1484,12 @@ complement probe mirroring the multiref scorer); eval_r8_wikinews_preds (exact-protocol regen of run-029 WikiNews preds); run-037 trainer (hybrid + pretrained fastText cc.ar.300 word channel — the lexical-density hypothesis; staged, launches on slot-free). + +Launch incident (2026-10-10): run-036 v1 died at the 30-minute DEFAULT +job timeout (mid-training, no intra-run ckpt — lost). Lesson added to +the launch checklist: ALWAYS pass explicit --timeout (4h standard for +a100 arms). Relaunched as run-036-hybrid-r2 with 4h. run-037 hardened +pre-launch: ft_init cache (skips 1.2GB re-parse), periodic step-ckpt + +seeded-generator resume. Monitor coverage hole fixed: poll ps --all +(terminal states) — the ps-only monitors stayed silent on +disappearance. From 3552b0e1f342c0633a8ff2b69d8c1a8124a59a19 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:16:56 +0800 Subject: [PATCH 06/68] docs: WO34 variant-constrained decoder implemented; sweep knobs; all follow-ups built --- TODO.sota/34-hybrid-word-char.md | 21 ++++++++++++++------- docs/RESULTS.md | 8 ++++++++ 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md index 5f4f7e6..7910843 100644 --- a/TODO.sota/34-hybrid-word-char.md +++ b/TODO.sota/34-hybrid-word-char.md @@ -31,9 +31,12 @@ Gates: silver units. Doctrine stands. Staged follow-ups (armed post-verdict): -1. Variant-constrained decode (v2): force the char model's output for - in-table words to the nearest table variant; keeps compositional - OOV handling, adds lexical precision. +1. Variant-constrained decode — IMPLEMENTED + uploaded + (variant_constrained_decode.py): loads run-036's best, scores every + observed variant's plane-combo sequence under per-position + log-softmax vs the free-greedy candidate, emits argmax; OOV and + non-Arabic tokens keep free decode. Runs on slot-free after + run-036-r2 lands. 2. Oracle-min complement probe over {run-029, run-036} WikiNews preds. Tooling BUILT and unit-tested (oracle_min.py in /code bundle — mirrors the multiref scorer exactly, verified on synthetic @@ -43,12 +46,16 @@ Staged follow-ups (armed post-verdict): original trainer's helpers). If oracle >2 WER over the best single arm, confidence-routed blending opens (WO28 lineage; routing, never naive voting). -3. run-037 (STAGED, script uploaded): run-036's architecture with the - scratch word channel replaced by pretrained fastText cc.ar.300 - vectors (in-job download, coverage-reported, trainable init) — the +3. run-037 (STAGED, script uploaded + hardened): run-036's + architecture with the scratch word channel replaced by pretrained + fastText cc.ar.300 vectors (in-job download, coverage-reported, + trainable init; ft_init cache + step-checkpoint resume) — the lexical-density hypothesis: our word channel is undertrained on ~3M words, not wrong; Fadel-class systems ride on billion-word vectors. Same gates. Launches when a GPU slot frees regardless of run-036's outcome (independent axis: scratch vs pretrained). -4. If run-033 (byt5-large plane) lands <10, race it against run-036/37 +4. Tuning-maturity sweep — READY: hybrid trainer now exposes + HYB_LR / HYB_HID env knobs; grid {lr 1e-3/5e-4} × {hid 512/768} × + {epochs 6/10} queued after the three primary arms read out. +5. If run-033 (byt5-large plane) lands <10, race it against run-036/37 gates; the dominant arm takes the successor slot. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index e97a59d..e7820d4 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1493,3 +1493,11 @@ pre-launch: ft_init cache (skips 1.2GB re-parse), periodic step-ckpt + seeded-generator resume. Monitor coverage hole fixed: poll ps --all (terminal states) — the ps-only monitors stayed silent on disappearance. + +WO34 tooling complete (2026-10-10): variant-constrained decoder +implemented + uploaded (scores observed variants under the hybrid's +per-position log-softmax vs free-greedy; OOV keeps free decode); +sweep knobs HYB_LR/HYB_HID wired into the hybrid trainer (grid queued +behind the three primary arms). All five follow-ups are now built or +running: run-033 (~98%), run-036-r2 (30%), run-037 staged, vcd + eval +jobs queued on slot-free. From 2e9d09012dcf81532e397dae81229fa6f8259ff7 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:49:17 +0800 Subject: [PATCH 07/68] =?UTF-8?q?docs:=20run-033=20verdict=20=E2=80=94=201?= =?UTF-8?q?0.78/9.16=20OOD=20(gate=20miss),=20specialist=20dominates=20bot?= =?UTF-8?q?h=20surfaces?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index e7820d4..fef24dd 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1501,3 +1501,14 @@ sweep knobs HYB_LR/HYB_HID wired into the hybrid trainer (grid queued behind the three primary arms). All five follow-ups are now built or running: run-033 (~98%), run-036-r2 (30%), run-037 staged, vcd + eval jobs queued on slot-free. + +## run-033 verdict — plane-large route closes Pareto-loser (2026-10-10) + +run-033 (byt5-large plane + full 900K silver, news-pure, 3ep, K=3): +WikiNews-2024 multiref **10.78 / 9.16** (from 18.69/11.35 — large +backbone + dose moved OOD by 7.9 WER), Sadeed windowed **DER 6.80 / +WER 23.80**. Gate <10 missed by 0.78; the shipped seq2seq specialist +DOMINATES on both surfaces (10.13/8.98; ID 5.50). The plane family's +large route closes; the contested axis is the word channel class +(run-036-r2 hybrid + run-037 fastText). run-029 preds-regen launched +for the oracle probe. From aa7f1df476847373085140092d34b2aa487e4a5d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:49:36 +0800 Subject: [PATCH 08/68] =?UTF-8?q?docs:=20README=20row=2032=20=E2=80=94=20r?= =?UTF-8?q?un-033=20gate=20miss,=20specialist=20dominates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 758bef2..d767195 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -67,7 +67,7 @@ hf jobs run --flavor a100-large -d \ | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | | 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | -| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM DONE (14.30/10.10 — arch-class refuted); run-033 plane RUNNING | +| 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | | 34 | 34-hybrid-word-char.md | P1 | 32/33 | run-036 RUNNING 48%; run-037 fastText arm STAGED; oracle probe BUILT | From 7cbc2eaf62a4ed91ff0c7e7173396f831f4585fb Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:57:07 +0800 Subject: [PATCH 09/68] =?UTF-8?q?docs:=20run-036=20verdict=20=E2=80=94=20w?= =?UTF-8?q?ord=20channel=20validated,=20fastText=20density=20arm=20launche?= =?UTF-8?q?d?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index fef24dd..beec7e0 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1512,3 +1512,16 @@ DOMINATES on both surfaces (10.13/8.98; ID 5.50). The plane family's large route closes; the contested axis is the word channel class (run-036-r2 hybrid + run-037 fastText). run-029 preds-regen launched for the oracle probe. + +## run-036 verdict — word channel VALIDATED (+0.77 WER), density hypothesis sharpens (2026-10-10) + +run-036-r2 (char-emb ⊕ per-char word-emb → 3×BiLSTM → plane head, +36.2M params, 20,460 steps): WikiNews multiref **13.53 / 9.92**, +Sadeed DER 5.90. Gate 1 PASS (<14.30): the word channel is real at +char level — 0.77 WER over the char-only anchor (14.30). Gate 2 +FAIL (>10.13): scratch 128-d embeddings over 156K types carry too +little lexical density. Hierarchy now: specialist 10.13 > plane-large +10.78 > hybrid 13.53 > char-only 14.30 > word-only 18.07. Data note: +silver pool = 202,680 units — the 900K cap never binds; corpus scale +is exhausted at what we hold. run-037 (fastText cc.ar.300, billion-word +lexical density) LAUNCHED — the decisive density arm. From e7d87a8564f8017dcfc643cde0a5923be21e3307 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:57:27 +0800 Subject: [PATCH 10/68] =?UTF-8?q?docs:=20README=20row=2034=20=E2=80=94=20c?= =?UTF-8?q?hannel=20validated,=20run-037=20running?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index d767195..73ccb5c 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -70,7 +70,7 @@ hf jobs run --flavor a100-large -d \ | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | -| 34 | 34-hybrid-word-char.md | P1 | 32/33 | run-036 RUNNING 48%; run-037 fastText arm STAGED; oracle probe BUILT | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | run-036 13.53/9.92 — channel VALIDATED (+0.77); run-037 fastText RUNNING; vcd+probe queued | ## Schedule From 7d281d5f8c67a23c96f22dad68754104054d0fca Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:58:40 +0800 Subject: [PATCH 11/68] =?UTF-8?q?docs:=20oracle=20probe=209.33=20=E2=80=94?= =?UTF-8?q?=20routed=20blend=20closed=20negative;=20vcd=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index beec7e0..dcdb126 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1525,3 +1525,14 @@ little lexical density. Hierarchy now: specialist 10.13 > plane-large silver pool = 202,680 units — the 900K cap never binds; corpus scale is exhausted at what we hold. run-037 (fastText cc.ar.300, billion-word lexical density) LAUNCHED — the decisive density arm. + +## Oracle probe — routed blend closes NEGATIVE; run-037 is the last live arm (2026-10-10) + +run-029 preds regenerated under the exact protocol: 10.032/8.9521 — +matches the converged number bit-for-bit at reporting precision. +Oracle-min over {run-029 specialist, run-036 hybrid}: **9.33 WER / +8.77 DER** — only **0.71 WER** over the specialist solo (decision +threshold was >2). Their errors overlap; the hybrid's word channel +adds nothing the specialist lacks on this benchmark. Confidence-routed +blending closes (WO28 lineage). vcd launched on run-036's ckpt +(last WO34 follow-up in flight). From b4c9f3c797ae4c200f8a0c87983d3973e5a7d288 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 13:59:01 +0800 Subject: [PATCH 12/68] =?UTF-8?q?docs:=20WO34=20follow-up=202=20closed=20?= =?UTF-8?q?=E2=80=94=20oracle=209.33,=20routing=20negative?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/34-hybrid-word-char.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md index 7910843..14040c7 100644 --- a/TODO.sota/34-hybrid-word-char.md +++ b/TODO.sota/34-hybrid-word-char.md @@ -37,15 +37,10 @@ Staged follow-ups (armed post-verdict): log-softmax vs the free-greedy candidate, emits argmax; OOV and non-Arabic tokens keep free decode. Runs on slot-free after run-036-r2 lands. -2. Oracle-min complement probe over {run-029, run-036} WikiNews preds. - Tooling BUILT and unit-tested (oracle_min.py in /code bundle — - mirrors the multiref scorer exactly, verified on synthetic - disjoint-strengths cases). run-036 saves wikinews_preds.txt; - run-029's preds regenerate via eval_r8_wikinews_preds.py (exact - protocol: 600-byte windows, greedy, project_haraqat; imports the - original trainer's helpers). If oracle >2 WER over the best single - arm, confidence-routed blending opens (WO28 lineage; routing, - never naive voting). +2. Oracle-min complement probe — DONE, CLOSED NEGATIVE (2026-10-10): + regen of run-029 reproduced 10.032/8.9521 exactly; oracle over + {specialist, hybrid} = 9.33/8.77 — 0.71 WER over best solo, + below the >2 threshold. Errors overlap; routing opens nothing. 3. run-037 (STAGED, script uploaded + hardened): run-036's architecture with the scratch word channel replaced by pretrained fastText cc.ar.300 vectors (in-job download, coverage-reported, From d950057673ff4303b47bfbd3ed7d3d5d6079e2cd Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 14:05:35 +0800 Subject: [PATCH 13/68] =?UTF-8?q?docs:=20vcd=20verdict=20=E2=80=94=20ID-on?= =?UTF-8?q?ly=20tweak,=20OOD=20bit-identical;=20run-037=20sole=20live=20ar?= =?UTF-8?q?m?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index dcdb126..3071b03 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1536,3 +1536,12 @@ threshold was >2). Their errors overlap; the hybrid's word channel adds nothing the specialist lacks on this benchmark. Confidence-routed blending closes (WO28 lineage). vcd launched on run-036's ckpt (last WO34 follow-up in flight). + +## vcd verdict — constraint is an ID-only tweak, OOD untouched (2026-10-10) + +run-036+vcd: Sadeed DER 5.90→5.75 (−0.16), multiref **13.5268/9.9211 — +bit-identical to free decode**. On every in-table word the free-greedy +candidate already wins the variant scoring; the hybrid's OOD residual +is not lexical-variant selection (consistent with the 0.71 oracle +gap). vcd closes as not-worth-integrating; WO34's live surface is +run-037 alone. From 46da6520cbc4fbc8b956d03ace121759d77c0a1f Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 14:06:04 +0800 Subject: [PATCH 14/68] =?UTF-8?q?docs:=20WO34=20follow-up=201=20closed=20?= =?UTF-8?q?=E2=80=94=20vcd=20ID-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/34-hybrid-word-char.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/TODO.sota/34-hybrid-word-char.md b/TODO.sota/34-hybrid-word-char.md index 14040c7..fa4894e 100644 --- a/TODO.sota/34-hybrid-word-char.md +++ b/TODO.sota/34-hybrid-word-char.md @@ -31,12 +31,10 @@ Gates: silver units. Doctrine stands. Staged follow-ups (armed post-verdict): -1. Variant-constrained decode — IMPLEMENTED + uploaded - (variant_constrained_decode.py): loads run-036's best, scores every - observed variant's plane-combo sequence under per-position - log-softmax vs the free-greedy candidate, emits argmax; OOV and - non-Arabic tokens keep free decode. Runs on slot-free after - run-036-r2 lands. +1. Variant-constrained decode — DONE, closed (2026-10-10): ID-only + tweak (Sadeed 5.90→5.75), multiref bit-identical to free decode + (13.5268/9.9211). Not worth integrating; the OOD residual is not + lexical-variant selection. 2. Oracle-min complement probe — DONE, CLOSED NEGATIVE (2026-10-10): regen of run-029 reproduced 10.032/8.9521 exactly; oracle over {specialist, hybrid} = 9.33/8.77 — 0.71 WER over best solo, From a3915b96b3c87fb47d49836cf206380f47c680dc Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 15:06:40 +0800 Subject: [PATCH 15/68] =?UTF-8?q?docs:=20run-037=20refutes=20density;=20WO?= =?UTF-8?q?34=20closes=20=E2=80=94=20OOD=20terminal=20at=2010.03,=20delta?= =?UTF-8?q?=20is=20external=20data=20scale?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 3071b03..2e58a0b 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1545,3 +1545,26 @@ candidate already wins the variant scoring; the hybrid's OOD residual is not lexical-variant selection (consistent with the 0.71 oracle gap). vcd closes as not-worth-integrating; WO34's live surface is run-037 alone. + +## run-037 verdict + WO34 CLOSES — the OOD front reaches its measured terminal state (2026-10-10) + +run-037 (hybrid + fastText cc.ar.300, 93.9% coverage, 63.8M params): +multiref **13.59 / 9.89** — statistically identical to the scratch +channel (13.53/9.92), ID worse (6.50 vs 5.90). **Lexical density is +REFUTED as the bottleneck.** The tuning sweep is closed as +not-worth-compute: the hybrid class sits 3.5 WER behind the specialist +— no optimizer setting closes that. + +WO34 final ledger — every lever measured, none moves OOD past the +specialist: converged training (10.03) · plane-large (10.78, dominated) +· char-only (14.30) · word-only (18.07) · joint hybrid (13.53) · +billion-word vectors (13.59) · constrained decode (±0.00 OOD) · routed +blend (0.71 oracle ceiling). Corpus scale exhausted at 202,680 units. + +TERMINAL READING: the shipped specialist (ara-diac-news-1.0, 10.13 +shipped / 10.03 converged) is the crown on the comparable surface at +our data scale. The 2.70 delta lives in QCRI's FULL silver release + +their self-consistent label conventions — an external dependency +(their complete corpus is theirs to share), not a lever we hold. +Even the literal reconstruction of their architecture lands at 14.3 +at our scale: the gap is data, not code. From a0ca796699bfd5f9a6057479e057b8e43faacb61 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 15:07:47 +0800 Subject: [PATCH 16/68] =?UTF-8?q?docs:=20terminal=20graph=20+=20matrix=20?= =?UTF-8?q?=E2=80=94=20OOD=20closes=20at=2010.03,=20delta=20external?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 33 ++++++++++++++------------------- 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 73ccb5c..9d5c71d 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -70,7 +70,7 @@ hf jobs run --flavor a100-large -d \ | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | -| 34 | 34-hybrid-word-char.md | P1 | 32/33 | run-036 13.53/9.92 — channel VALIDATED (+0.77); run-037 fastText RUNNING; vcd+probe queued | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59 (density refuted), vcd ±0, oracle 0.71; specialist stands; delta = external data scale | ## Schedule @@ -79,37 +79,32 @@ Wave 2 (HF training, launched together): WO04 (a100-large), WO06 (a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`. Wave 3: WO05 tables after 01+07 land; WO09 probe only. -## Path-to-success graph + SOTA matrix (2026-10-10, live) +## Path-to-success graph + SOTA matrix (2026-10-10, terminal for this campaign wave) ``` -ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER) - run-029 seq2seq specialist 10.13/8.98 [SHIPPED: ara-diac-news-1.0; converged 10.03/8.95] - run-034 literal BiLSTM 14.30/10.10 [anchor — arch-class REFUTED] - pure-lexicon floor 25.15/13.18 [OOV 18.10% — lexical = 75% of task] - QCRI published 2.70 [their corpus + their conventions] - ├─ run-033 byt5-large plane + 900K silver RUNNING gate <10 - └─ run-035 word-variant disambiguator RUNNING gates <14.30 / <10.13 - winner → IMF v1 export → release chain (owner version) → news successor - both fail → register-specialist doctrine stands; residue = convention - alignment, unreachable without their labels (closed) +ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER) ── MEASURED TERMINAL + CROWN on comparable surface: ara-diac-news-1.0 10.13/8.98 (converged 10.03/8.95) + full ladder: specialist 10.03 > plane-large 10.78 > hybrid 13.53 ≈ fastText 13.59 + > char-only 14.30 > word-only 18.07; oracle ceiling 9.33; vcd ±0.00 + closed by measurement: training, backbone scale, word channel (3 forms), + lexical density, constrained decode, blending, corpus dose (202,680 units) + remaining 2.70 delta = QCRI's FULL silver + self-consistent conventions + → EXTERNAL: their corpus is theirs to share (owner decision, if ever) ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated] · specialist 5.50 (register cost) HEBREW ── nikud DER: 8.18 crown (heb-diac-plane-2.0, 2× runner-up) - ALL TEXT LEVERS CLOSED by measurement: capacity 8.48 · noisy 8.72 (run-030 - re-confirmed) · convergence 10.03 → remaining path = AUDIO program: - WO31 phoneme-ASR fine-tune → alignment → learned-IPA student (owner block) - + ALL TEXT LEVERS CLOSED (8.48/8.72/10.03) → WO31 audio program (owner block) HEBREW ── g2p CER: rules 0.2393 · ReNikud 0.0244 (audio-supervised; WO31 line) -THAI ── PER: hybrid 0.1478 @0.73ms [crown]; tiny tier closed NEGATIVE (2038%) +THAI ── PER: hybrid 0.1478 @0.73ms [crown]; tiny tier closed NEGATIVE CLIENT SURFACE: 3-runtime parity ✓ · preserve ✓ · edge /v1/infer ✓ index-v10 · npm 5.7.0 · 29 HF mirrors · ruby pins 802/804 ``` | axis | ours | 2026 frontier | status | |---|---|---|---| -| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated (LLM = no-teacher-law, not a runtime option) | -| AR OOD multiref WER | 10.13 (specialist) | 2.70 (their-domain) | converged seq2seq; run-033/035 arms live | +| AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated (LLM = no-teacher law) | +| AR OOD multiref WER | 10.13 (10.03 converged) | 2.70 (full-corpus scale) | crown at our scale; delta external | | HE nikud DER | 8.18 | — (we lead) | crown held; text exhausted; audio = WO31 | | HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | blocked on WO31 owner block | | TH PER | 0.1478 | 26.9 (latency play) | crown held + hybrid latency tier | From d5be413259be82ecc652ee72814e7a59613d53e3 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 20:13:35 +0800 Subject: [PATCH 17/68] =?UTF-8?q?docs:=20WO35=20alternate-format=20artifac?= =?UTF-8?q?t=20=E2=80=94=20OOD=20corrects=20to=204.10/1.56,=20runtime=20fi?= =?UTF-8?q?xed=20in=203=20runtimes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 2e58a0b..d104bc8 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1568,3 +1568,36 @@ their self-consistent label conventions — an external dependency (their complete corpus is theirs to share), not a lever we hold. Even the literal reconstruction of their architecture lands at 14.3 at our scale: the gap is data, not code. + +## WO35 — THE ALTERNATE-FORMAT SCORING ARTIFACT: OOD corrects 10.03 → 4.10/1.56 (2026-10-10) + +Error-typography probe of the specialist's 1,065 wrong words found 74% +were LENGTH mismatches — not reading errors. Root cause: the model, +trained on QCRI multi-reference silver, emits '/'-separated alternate +sets per token (7.44% of tokens; e.g. وِكَالَةُ/وِكَالَةُ); the multiref +scorer treated each such token as one word, breaking letter alignment +— every alternate-emitting word scored wrong even when the primary +choice was correct. + +Same preds, same benchmark, first-alternate decode (deterministic, +shippable): + +| protocol | WER | DER | +|---|---|---| +| prior scorer | 10.032 | 8.9521 | +| **first-alternate** | **4.0976** | **1.5606** | +| symmetric any-vs-any | 3.1839 | — | + +run-036 hybrid re-scores 13.53 → 8.85/2.84 — the specialist remains +dominant. The 2.70 frontier is now a 1.4-point WER gap (and our DER +1.56 likely leads on letter-level). WO32-34 verdict waves were scored +on the broken surface; rankings hold (dominance unchanged) but every +absolute number above shifts under the corrected protocol. + +Runtime contract FIXED in all three runtimes (TDD, parity): +interscript-py#35, interscript-ts#108, interscript-ruby#805 — +`first_alternates` strips to the primary choice in seq2seq translate; +plane family unaffected. Follow-ups: Sadeed rescore under first-alt +(12.6% of ID rows emit alternates; DER 5.50 is inflated), models.yaml +metric correction + card text, and the re-opened 1.4-point program +(sweep/stack arms now worth compute again). From a73ee0866d2ee6c8ebdee919544ff688a6982eb2 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 20:13:52 +0800 Subject: [PATCH 18/68] docs: ara-diac-news-1.0 metrics corrected to first-alternate protocol (4.10/1.56) --- models.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models.yaml b/models.yaml index cfff270..a83f754 100644 --- a/models.yaml +++ b/models.yaml @@ -419,8 +419,8 @@ models: sha256: e5a52b1530dfa6d6c7ef8970443206ce33af865d0337a58d145945adab33484e size: 399018923 metrics: - - {name: wer_wikinews2024_multiref, value: 10.1262, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} - - {name: der_wikinews2024_multiref, value: 8.9818, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} + - {name: wer_wikinews2024_multiref, value: 4.0976, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} + - {name: der_wikinews2024_multiref, value: 1.5606, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} - {name: der_total_greedy, value: 5.5008, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} license: BSD-3-Clause heb-diac-plane-1.0: From 7d6576724a816e18f2a6037cfd96c386b0b1d8ed Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 20:14:29 +0800 Subject: [PATCH 19/68] =?UTF-8?q?docs:=20WO35=20spec=20+=20index=20?= =?UTF-8?q?=E2=80=94=20corrected=20protocol,=20re-opened=201.4-point=20pro?= =?UTF-8?q?gram?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 3 +- TODO.sota/35-alternate-protocol.md | 45 ++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) create mode 100644 TODO.sota/35-alternate-protocol.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 9d5c71d..bc962bf 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -70,7 +70,8 @@ hf jobs run --flavor a100-large -d \ | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | -| 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59 (density refuted), vcd ±0, oracle 0.71; specialist stands; delta = external data scale | +| 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | +| 35 | 35-alternate-protocol.md | P1 | 34 | SCORING ARTIFACT FIXED — OOD 4.10/1.56 (was 10.03/8.95); runtime fixed py#35/ts#108/ruby#805; 1.4-pt program open | ## Schedule diff --git a/TODO.sota/35-alternate-protocol.md b/TODO.sota/35-alternate-protocol.md new file mode 100644 index 0000000..670b4bc --- /dev/null +++ b/TODO.sota/35-alternate-protocol.md @@ -0,0 +1,45 @@ +# WO35 — the alternate-format scoring artifact + the re-opened 1.4-point program + +## Discovery (2026-10-10) + +Error-typography of the specialist's 1,065 wrong WikiNews words: 74.2% +length mismatches, 1.4% vowel-class — inverted from expectation. Root +cause: models trained on QCRI multi-reference silver EMIT '/' +alternates per token (7.44% of tokens; Sadeed rows too: 12.6%); +scorers counted each as a single word → letter misalignment → all +scored wrong despite correct primary readings. + +Corrected protocol (first-alternate decode — deterministic, +shippable, same benchmark): + +| arm | prior | first-alt | +|---|---|---| +| run-029 specialist (shipped ara-diac-news-1.0) | 10.03/8.95 | **4.10/1.56** | +| run-036 hybrid | 13.53/9.92 | 8.85/2.84 | +| symmetric any-vs-any (specialist) | — | 3.18 WER | + +Rankings from WO32-34 hold (dominance unchanged); absolute numbers +shift. models.yaml metrics corrected. Runtime contract fixed in all +three runtimes with TDD parity: interscript-py#35, interscript-ts#108, +interscript-ruby#805 (`first_alternates`/`firstAlternates` in seq2seq +translate; plane family unaffected). + +## Answer to "how do we get their silver + conventions" + +We already HAVE the conventions — the corpus taught our model to emit +them (that was the artifact). Their full silver release is only worth +requesting if the re-opened program below stalls; today it is an +owner-to-owner ask to QCRI, not a technical dependency. + +## Remaining work + +1. Sadeed rescore under first-alt (preds saved; DER 5.50 inflated). +2. Re-open the OOD program at 4.10 vs 2.70 (1.4 WER): + - hybrid-2 stack: run-036/037 class + the specialist's corpus mix + (joint seq2seq + word channel), sweep lr/hid/epochs (knobs wired). + - oracle probe re-run under corrected protocol once new arms land. +3. Card text for ara-diac-news-1.0 mirror (HF repo 29) + release + notes: decode-contract change → runtime version bumps are the + OWNER's decision (artifact bytes unchanged; sha valid). +4. Derive the 1.56 DER vs published frontier DER comparisons for the + external-benchmark table (WO05) once Sadeed rescore lands. From 6f115ff09b8bfaeeed5e6f6d9da4f135912b2701 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 21:25:04 +0800 Subject: [PATCH 20/68] =?UTF-8?q?docs:=20WO31=20green-lit=20=E2=80=94=20st?= =?UTF-8?q?age=201=20running,=20stage=202=20staged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- TODO.sota/31-heb-phoneme-asr-v1.md | 12 ++++++++++++ docs/RESULTS.md | 12 ++++++++++++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index bc962bf..e035c0d 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -66,7 +66,7 @@ hf jobs run --flavor a100-large -d \ | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | -| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | SPEC'D — tune the phoneme CTC on Hebrew first (owner block) | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | RUNNING (green-lit) — run-038 ASR fine-tune in flight; run-039 v1 staged | | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | diff --git a/TODO.sota/31-heb-phoneme-asr-v1.md b/TODO.sota/31-heb-phoneme-asr-v1.md index 8d9fa31..72b04cc 100644 --- a/TODO.sota/31-heb-phoneme-asr-v1.md +++ b/TODO.sota/31-heb-phoneme-asr-v1.md @@ -15,3 +15,15 @@ ReNikud recipe's first stage): Cost: one fine-tune (~2-4 GPU-hours) + relabel + v1 train. Owner green-light for the block. + +## Execution (green-light received 2026-10-10) + +- run-038-heb-asr-ft RUNNING (stage 1): fine-tune as spec'd; espeak-ng + targets (do_phonemize disabled — targets are pre-phonemized; the + en-us default would silently mis-label), in-memory corpus, per-phone + dev PER gate < 0.395, step-ckpts every 2000, --timeout 4h. +- run-039-heb-ipa-v1 STAGED (stage 2, script uploaded): relabel FLEURS + with the TUNED ASR (audio-derived labels — the whole point), v0 + student recipe with the byte-correct collate, gate CER < 0.2393 on + phonikud heb-g2p-benchmark (gt.tsv from run-032). Launches when + run-038's gate passes. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index d104bc8..5c6f4c4 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1601,3 +1601,15 @@ plane family unaffected. Follow-ups: Sadeed rescore under first-alt (12.6% of ID rows emit alternates; DER 5.50 is inflated), models.yaml metric correction + card text, and the re-opened 1.4-point program (sweep/stack arms now worth compute again). + +## WO31 GREEN-LIT — the Hebrew audio program launches (2026-10-10) + +Owner green-light received. run-038-heb-asr-ft RUNNING (stage 1): +espeak-ng transcript targets (do_phonemize disabled — targets arrive +pre-phonemized; the tokenizer's en-us default would have silently +mislabeled every utterance), per-phone dev PER gate < 0.395, +step-checkpoints, 4h timeout. run-039-heb-ipa-v1 STAGED: audio-derived +relabel with the tuned teacher + v0 student recipe (byte-correct +collate), gate CER < 0.2393 vs the rules layer on the phonikud +benchmark. This is the only open path to the HE g2p frontier +(ReNikud 0.0244). From 12a6783e23eff726375107b2b0a74369fb8ef696 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 21:25:30 +0800 Subject: [PATCH 21/68] =?UTF-8?q?docs:=20graph=20+=20matrix=20post-WO35=20?= =?UTF-8?q?=E2=80=94=204.10/1.56=20OOD,=20WO31=20running?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 48 ++++++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index e035c0d..a828d21 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -80,33 +80,39 @@ Wave 2 (HF training, launched together): WO04 (a100-large), WO06 (a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`. Wave 3: WO05 tables after 01+07 land; WO09 probe only. -## Path-to-success graph + SOTA matrix (2026-10-10, terminal for this campaign wave) +## Path-to-success graph + SOTA matrix (2026-10-10, post-WO35) ``` -ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER) ── MEASURED TERMINAL - CROWN on comparable surface: ara-diac-news-1.0 10.13/8.98 (converged 10.03/8.95) - full ladder: specialist 10.03 > plane-large 10.78 > hybrid 13.53 ≈ fastText 13.59 - > char-only 14.30 > word-only 18.07; oracle ceiling 9.33; vcd ±0.00 - closed by measurement: training, backbone scale, word channel (3 forms), - lexical density, constrained decode, blending, corpus dose (202,680 units) - remaining 2.70 delta = QCRI's FULL silver + self-consistent conventions - → EXTERNAL: their corpus is theirs to share (owner decision, if ever) - -ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated] · specialist 5.50 (register cost) - -HEBREW ── nikud DER: 8.18 crown (heb-diac-plane-2.0, 2× runner-up) - ALL TEXT LEVERS CLOSED (8.48/8.72/10.03) → WO31 audio program (owner block) -HEBREW ── g2p CER: rules 0.2393 · ReNikud 0.0244 (audio-supervised; WO31 line) -THAI ── PER: hybrid 0.1478 @0.73ms [crown]; tiny tier closed NEGATIVE +ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER, first-alt protocol) + ara-diac-news-1.0 4.10 WER / 1.56 DER [CROWN-tier; DER likely leads] + frontier 2.70 WER → gap = 1.4 points, program RE-OPENED (WO35): + hybrid-2 stack + sweep (knobs wired); Sadeed rescore pending + pre-WO35 numbers (broken scorer, for the record): specialist 10.03 + > plane-large 10.78 > hybrid 13.53 ≈ fastText 13.59 > char 14.30 + > word-only 18.07; rankings hold, absolutes shifted + convention alignment SOLVED in-run: the model already emits their + alternate format; runtimes now return the primary choice + (py#35 / ts#108 / ruby#805) + +ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated]; specialist + 5.50 old-protocol (rescore under first-alt pending; 12.6% of rows + emitted alternates) + +HEBREW ── nikud DER: 8.18 crown (2× runner-up); text levers closed + → WO31 AUDIO PROGRAM GREEN-LIT + RUNNING (2026-10-10): + run-038 ASR fine-tune (dev PER gate < 0.395) → run-039 v1 student + (gate CER < 0.2393 rules layer) → only then ivrit.ai scale ($500) +HEBREW ── g2p CER: rules 0.2393 · ReNikud 0.0244 (the WO31 target) +THAI ── PER: hybrid 0.1478 @0.73ms [crown] CLIENT SURFACE: 3-runtime parity ✓ · preserve ✓ · edge /v1/infer ✓ - index-v10 · npm 5.7.0 · 29 HF mirrors · ruby pins 802/804 + index-v10 · npm 5.7.0 · 29 HF mirrors ``` | axis | ours | 2026 frontier | status | |---|---|---|---| | AR in-domain DER | 2.2864 | 1.39 (LLM API) | #1 dedicated (LLM = no-teacher law) | -| AR OOD multiref WER | 10.13 (10.03 converged) | 2.70 (full-corpus scale) | crown at our scale; delta external | -| HE nikud DER | 8.18 | — (we lead) | crown held; text exhausted; audio = WO31 | -| HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | blocked on WO31 owner block | -| TH PER | 0.1478 | 26.9 (latency play) | crown held + hybrid latency tier | +| AR OOD multiref WER | **4.10** (DER **1.56**) | 2.70 | 1.4-pt program re-opened (WO35) | +| HE nikud DER | **8.18** | — (we lead) | crown held; audio = WO31 running | +| HE g2p CER | 0.2393 (rules) | 0.0244 (audio) | WO31 stages 1-2 in flight | +| TH PER | **0.1478** | 26.9 (latency play) | crown held + hybrid latency tier | | client surface | 3 runtimes + edge | none exists | moat maintained | From d500f4a99128ecda09decfdb4f350a59546a3249 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 21:27:20 +0800 Subject: [PATCH 22/68] =?UTF-8?q?docs:=20Sadeed=20first-alt=20rescore=20?= =?UTF-8?q?=E2=80=94=20ID=20unaffected=20(5.48),=20artifact=20was=20OOD-on?= =?UTF-8?q?ly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 6 ++++++ models.yaml | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 5c6f4c4..afe93c9 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1613,3 +1613,9 @@ relabel with the tuned teacher + v0 student recipe (byte-correct collate), gate CER < 0.2393 vs the rules layer on the phonikud benchmark. This is the only open path to the HE g2p frontier (ReNikud 0.0244). + +Sadeed rescore under first-alt (WO35 follow-up 1, closed): Total DER +5.5008 → **5.4814**, Morphological DER 4.4458 — the ID surface was NOT +materially inflated (the evaluator letter-aligns tolerantly; alternate +words skipped rather than poisoning alignment). The artifact lived +almost entirely on the multiref surface: OOD −5.9 WER vs ID −0.02 DER. diff --git a/models.yaml b/models.yaml index a83f754..00e1f24 100644 --- a/models.yaml +++ b/models.yaml @@ -421,7 +421,7 @@ models: metrics: - {name: wer_wikinews2024_multiref, value: 4.0976, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} - {name: der_wikinews2024_multiref, value: 1.5606, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} - - {name: der_total_greedy, value: 5.5008, source: interscript/interscript-models docs/RESULTS.md#sota-verdict-wave-4--ood-crown-1013-capacity-closed-on-arabic-too-audio-alignment-go-2026-10-09} + - {name: der_total_greedy, value: 5.4814, source: interscript/interscript-models docs/RESULTS.md#wo35--the-alternate-format-scoring-artifact-ood-corrects-1003--410156-2026-10-10} license: BSD-3-Clause heb-diac-plane-1.0: task: diacritization From 194be5ec78f03b959802b503ee2f827aea418a79 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 21:38:38 +0800 Subject: [PATCH 23/68] =?UTF-8?q?docs:=20WO31=20stage-1=20debugging=20ledg?= =?UTF-8?q?er=20=E2=80=94=20espeak=20segmentation=20bug,=20tokenizer-drive?= =?UTF-8?q?n=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index afe93c9..5546077 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1619,3 +1619,13 @@ Sadeed rescore under first-alt (WO35 follow-up 1, closed): Total DER materially inflated (the evaluator letter-aligns tolerantly; alternate words skipped rather than poisoning alignment). The artifact lived almost entirely on the multiref surface: OOD −5.9 WER vs ID −0.02 DER. + +WO31 stage-1 debugging ledger (2026-10-10): r1 missing protobuf +(vocab parse), r2 missing phonemizer (backend init runs even for +pre-phonemized input), r3 aborted by the unk-rate guard — root cause +was MY espeak --ipa segmentation assumption: espeak emits contiguous +phone strings per word; my space-split produced one giant token per +word → unk 0.9996. Fix: phonemize through the tokenizer itself +(init_backend("he")), whose separator conventions ARE the vocab's — +verified locally first (unk 0.0000 on a Hebrew sentence). r4 running. +The guard did its job: no garbage-trained checkpoint shipped. From a2ebf66d477d64c4ba27f384d6bb4fe061a6242f Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 22:41:15 +0800 Subject: [PATCH 24/68] =?UTF-8?q?docs:=20WO31=20stage-1=20gate=20pass=20?= =?UTF-8?q?=E2=80=94=20tuned=20ASR=20PER=200.2306;=20stage=202=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 5546077..44033b6 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1629,3 +1629,13 @@ word → unk 0.9996. Fix: phonemize through the tokenizer itself (init_backend("he")), whose separator conventions ARE the vocab's — verified locally first (unk 0.0000 on a Hebrew sentence). r4 running. The guard did its job: no garbage-trained checkpoint shipped. + +## WO31 stage-1 verdict — GATE PASS: tuned ASR hears Hebrew at PER 0.2306 (2026-10-10) + +run-038-r4 (30 epochs, 12,150 steps, 47m on a100): dev **PER 0.2306** +vs the universal model's 0.395 — a 42% teacher improvement. The +teacher bottleneck from WO30 v0 (CER 1.18) is broken. Stage 2 LAUNCHED +(run-039): relabel FLEURS with the tuned teacher (audio-derived +labels) → byt5-small v1 student → gate CER < 0.2393 on the phonikud +heb-g2p benchmark (rules layer). ReNikud 0.0244 is the frontier +beyond; ivrit.ai scale stays parked until this gate reads out. From 4024c8e2334cbf4e3fcd860702c05607400d778b Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 22:41:30 +0800 Subject: [PATCH 25/68] docs: WO31 stage 1 gate pass; stage 2 running --- TODO.sota/00-README.md | 2 +- TODO.sota/31-heb-phoneme-asr-v1.md | 11 ++++++----- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index a828d21..31d46e1 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -66,7 +66,7 @@ hf jobs run --flavor a100-large -d \ | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | -| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | RUNNING (green-lit) — run-038 ASR fine-tune in flight; run-039 v1 staged | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | stage 1 GATE PASS (PER 0.2306); stage 2 RUNNING (CER gate <0.2393) | | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | diff --git a/TODO.sota/31-heb-phoneme-asr-v1.md b/TODO.sota/31-heb-phoneme-asr-v1.md index 72b04cc..996b362 100644 --- a/TODO.sota/31-heb-phoneme-asr-v1.md +++ b/TODO.sota/31-heb-phoneme-asr-v1.md @@ -22,8 +22,9 @@ green-light for the block. targets (do_phonemize disabled — targets are pre-phonemized; the en-us default would silently mis-label), in-memory corpus, per-phone dev PER gate < 0.395, step-ckpts every 2000, --timeout 4h. -- run-039-heb-ipa-v1 STAGED (stage 2, script uploaded): relabel FLEURS - with the TUNED ASR (audio-derived labels — the whole point), v0 - student recipe with the byte-correct collate, gate CER < 0.2393 on - phonikud heb-g2p-benchmark (gt.tsv from run-032). Launches when - run-038's gate passes. +- run-038 stage-1 VERDICT (r4, 47m): dev PER 0.2306 < 0.395 — GATE + PASS (teacher bottleneck broken; 42% over universal). +- run-039-heb-ipa-v1 RUNNING (stage 2): relabel FLEURS with the TUNED + ASR (audio-derived labels — the whole point), v0 student recipe with + the byte-correct collate, gate CER < 0.2393 on phonikud + heb-g2p-benchmark (gt.tsv from run-032). From a139eecabd6f66e499fa0d77177da62160741990 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 22:41:42 +0800 Subject: [PATCH 26/68] =?UTF-8?q?docs:=20graph=20=E2=80=94=20Sadeed=20resc?= =?UTF-8?q?ore=20done,=20artifact=20was=20OOD-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 31d46e1..197d412 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -95,8 +95,8 @@ ARABIC ── WikiNews-2024 multiref (OOD news, WER/DER, first-alt protocol) (py#35 / ts#108 / ruby#805) ARABIC ── SadeedDiac-25 (ID): r7 2.2864 [BEST dedicated]; specialist - 5.50 old-protocol (rescore under first-alt pending; 12.6% of rows - emitted alternates) + 5.48 first-alt rescore DONE — ID surface was never inflated (the + alternate artifact was OOD-only: −5.9 WER there vs −0.02 DER here) HEBREW ── nikud DER: 8.18 crown (2× runner-up); text levers closed → WO31 AUDIO PROGRAM GREEN-LIT + RUNNING (2026-10-10): From 20be9242dc3c66593fbc09f2977fe61cb23f5de2 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 22:47:49 +0800 Subject: [PATCH 27/68] =?UTF-8?q?docs:=20WO35=20arm=20run-040=20launched?= =?UTF-8?q?=20=E2=80=94=20specialist=20on=20stripped=20corpus,=20both-prot?= =?UTF-8?q?ocol=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/35-alternate-protocol.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/TODO.sota/35-alternate-protocol.md b/TODO.sota/35-alternate-protocol.md index 670b4bc..98be202 100644 --- a/TODO.sota/35-alternate-protocol.md +++ b/TODO.sota/35-alternate-protocol.md @@ -33,10 +33,19 @@ owner-to-owner ask to QCRI, not a technical dependency. ## Remaining work -1. Sadeed rescore under first-alt (preds saved; DER 5.50 inflated). +1. DONE — Sadeed rescore: 5.4814 (was 5.5008); ID was never inflated; + the artifact was OOD-only (−5.9 WER there vs −0.02 DER here). 2. Re-open the OOD program at 4.10 vs 2.70 (1.4 WER): - - hybrid-2 stack: run-036/037 class + the specialist's corpus mix - (joint seq2seq + word channel), sweep lr/hid/epochs (knobs wired). + - run-040 RUNNING: the specialist retrained on first-alt-STRIPPED + corpus (identical recipe to run-029 — R8_EPOCHS=4, GOLD=none, + MIX=qcri, cap 900K, init r7-best — plus R8_STRIP_ALT=1; targets + and inputs both stripped; in-job eval reports raw AND first-alt + with project_haraqat on the WikiNews path). Gate: first-alt + WER < 4.10. If the alternate emission was wasted capacity, this + wins directly; if the model NEEDS the alternates as implicit + uncertainty, it loses — either way the mechanism is measured. + - next arms: byt5-large seq2seq specialist (untried in the dominant + family); hybrid-2 + sweep (knobs wired). - oracle probe re-run under corrected protocol once new arms land. 3. Card text for ara-diac-news-1.0 mirror (HF repo 29) + release notes: decode-contract change → runtime version bumps are the From c8d9cea88184967663b1c3d77ee04619f8ed5455 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 23:25:39 +0800 Subject: [PATCH 28/68] =?UTF-8?q?docs:=20WO31=20stage-2=20negative=20?= =?UTF-8?q?=E2=80=94=20convention=20mismatch=20diagnosed,=20pointed-espeak?= =?UTF-8?q?=20fix=20staged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 44033b6..c9a4bfa 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1639,3 +1639,22 @@ teacher bottleneck from WO30 v0 (CER 1.18) is broken. Stage 2 LAUNCHED labels) → byt5-small v1 student → gate CER < 0.2393 on the phonikud heb-g2p benchmark (rules layer). ReNikud 0.0244 is the frontier beyond; ivrit.ai scale stays parked until this gate reads out. + +## WO31 stage-2 verdict — CER 0.9142, gate FAILED; root cause is a CONVENTION, not the teacher (2026-10-10) + +run-039 (tuned-teacher labels, v0 student recipe): CER **0.9142** +(v0: 1.18; gate < 0.2393). The samples decode the failure: the student +learned the teacher's inventory — espeak-he over UNPOINTED text emits +consonant skeletons (`h u t s f h`), while the benchmark gold is +espeak over POINTED (nikud) text — vowel-full with stress +(`hˈuʔ tsˈafah bəsˈeret`). Verified locally in one command: +unpointed → `hˈu tsfh vsrt`; pointed → `hˈuʔ tsˈafah bəsˈeret`. + +The gold's generator was nikud-then-espeak. Our fix uses our own +crown asset (heb-diac-plane-2.0, DER 8.18; sha-verified release +artifact) to point the FLEURS transcripts before espeak — no LLM +teachers, no external dependency. run-041 (stage 1b): re-fine-tune +with pointed-espeak targets → stage 2 relabel + student re-run. +Prediction if the convention reading is right: stage-1 PER will move +only modestly (targets get denser), stage-2 CER collapses toward the +gate. From d2e0e3bbc1ed0a1c68b7d3d2f36260ca8a8dc568 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 10 Oct 2026 23:52:05 +0800 Subject: [PATCH 29/68] =?UTF-8?q?docs:=20run-041=20launched=20=E2=80=94=20?= =?UTF-8?q?pointed-espeak=20targets=20via=20our=20nikud=20crown?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index c9a4bfa..1dda480 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1658,3 +1658,10 @@ with pointed-espeak targets → stage 2 relabel + student re-run. Prediction if the convention reading is right: stage-1 PER will move only modestly (targets get denser), stage-2 CER collapses toward the gate. + +run-041 LAUNCHED (stage 1b): all 4,335 FLEURS transcripts pointed by +our own crown nikud model (100% coverage, spot-checked correct), +espeak now sees pointed text → vowel-full + stress targets matching +the benchmark's convention. Stage-2 rerun (run-042) fires on gate +pass, reusing the proven relabel+student script against the new +teacher. From 7e9aa470b38960d5308104db70506e820d198db4 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 00:41:41 +0800 Subject: [PATCH 30/68] docs: run-041 gate pass; run-042 v2 student launched --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 1dda480..b59b266 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1665,3 +1665,11 @@ espeak now sees pointed text → vowel-full + stress targets matching the benchmark's convention. Stage-2 rerun (run-042) fires on gate pass, reusing the proven relabel+student script against the new teacher. + +run-041 stage-1b GATE PASS: dev PER 0.319 (< 0.395) on the vowel-full +inventory (denser target space + nikud noise account for the rise +from the consonantal 0.2306 — different inventories, not comparable). +The teacher now hears vowels in the benchmark's convention. +run-042 LAUNCHED (stage 2 rerun): relabel with the pointed teacher → +v2 student → gate CER < 0.2393. This is the decisive read on the +convention hypothesis. From e0ecdb619e350871cba4711dca1437f35877772d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 01:44:03 +0800 Subject: [PATCH 31/68] docs: run-042 CER is a space-convention artifact; normalized gate running --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b59b266..8a0d0ea 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1673,3 +1673,13 @@ The teacher now hears vowels in the benchmark's convention. run-042 LAUNCHED (stage 2 rerun): relabel with the pointed teacher → v2 student → gate CER < 0.2393. This is the decisive read on the convention hypothesis. + +run-042 verdict: raw CER 1.3021 — GATE-METRIC ARTIFACT, third of the +campaign. The v2 student predicts vowels in the right convention +(`tsafeh beseret` ≈ `tsafˈa besˈeʁet`) but its labels are +space-per-phone while gt.tsv gold is space-per-word; the string CER +counts every inter-phone space as an error (v0/v1 carried the same +mismatch — their CERs were also inflated). run-043 RUNNING: the fair +gate — space-and-stress-stripped char CER on both sides, the same +contiguous convention the rules layer's 0.2393 was measured under — +on the saved v2 checkpoint, with truncation flagging. From 8a2afa225a6f27df8c56938865e1f559d656136d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 01:50:59 +0800 Subject: [PATCH 32/68] docs: run-043 honest gate 0.5463; run-044 label-filtered v3 launched --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 8a0d0ea..b698ea0 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1683,3 +1683,13 @@ mismatch — their CERs were also inflated). run-043 RUNNING: the fair gate — space-and-stress-stripped char CER on both sides, the same contiguous convention the rules layer's 0.2393 was measured under — on the saved v2 checkpoint, with truncation flagging. + +run-043 normalized gate: **CER 0.5463** — honest miss (rules 0.2393), +but the error anatomy is clean: repetition loops (byt5-small on 3.2K +noisy utterances) + vowel drift, labels carrying the teacher's 0.319 +PER. WO31 iteration 3 LAUNCHED (run-044): pseudo-label quality gate +(audio-decode must agree with espeak(pointed text) within 0.35 nCER — +the WO06 filtering pattern; caught+fixed a phonemize-mode bug in the +reference path pre-launch) + no-repeat generation + normalized CER in +the in-job verdict. Prediction: filtering trades data volume for +label precision; the loop failures should collapse. From 23ffc3684363432c902235f0add849a6c3ab1150 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 02:40:29 +0800 Subject: [PATCH 33/68] =?UTF-8?q?docs:=20v3=20filter=20negative;=20run-045?= =?UTF-8?q?=20byt5-base=20arm=20=E2=80=94=20last=20free=20lever=20before?= =?UTF-8?q?=20ivrit=20decision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b698ea0..0461788 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1693,3 +1693,14 @@ the WO06 filtering pattern; caught+fixed a phonemize-mode bug in the reference path pre-launch) + no-repeat generation + normalized CER in the in-job verdict. Prediction: filtering trades data volume for label precision; the loop failures should collapse. + +run-044 (v3) verdict: normalized CER **0.8851 — filtering NEGATIVE** +(v2 unfiltered: 0.5463). The agreement gate selected an easy-utterance +subset (3,452 pairs, 2,685 train) whose distribution shift cost more +than the noise removal bought — selection bias beats cleaning at this +data scale. run-045 LAUNCHED: v2's unfiltered pointed recipe at +**byt5-base** (repetition loops are the small-model symptom; +base-vs-small unmeasured in this regime). This is the last free lever +before the ivrit.ai scale decision ($500, owner-gated): if base also +misses < 0.2393, the 10-hour FLEURS ceiling is the binding constraint +and the rules layer (0.2393) stays shipped. From a5305a12d904b76ef56ae10bd7cbd9870fa0c40c Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 04:28:50 +0800 Subject: [PATCH 34/68] docs: no_repeat confound identified; clean base A/B launched --- docs/RESULTS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 0461788..c0bdd54 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1704,3 +1704,12 @@ base-vs-small unmeasured in this regime). This is the last free lever before the ivrit.ai scale decision ($500, owner-gated): if base also misses < 0.2393, the 10-hour FLEURS ceiling is the binding constraint and the rules layer (0.2393) stays shipped. + +run-045 (byt5-base + no_repeat): normalized CER 0.8817 — CONFOUNDED +READOUT, my error: the no_repeat_ngram_size=3 added in v3 blocks +legitimate phone-sequence 3-grams (IPA recurringly repeats syllable +n-grams) → early-EOS truncation visible in the samples. v3's +"filter-negative" verdict is likewise confounded (two variables at +once). Clean A/B LAUNCHED (run-046): byt5-base, unfiltered labels, NO +repetition suppression — vs v2-small's 0.5463 this isolates capacity +alone. Lesson logged: one variable per arm, always. From 07f114a355de4f52436a455721567ea830849999 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 06:16:31 +0800 Subject: [PATCH 35/68] =?UTF-8?q?docs:=20WO31=20stage-2=20closes=20at=200.?= =?UTF-8?q?546=20=E2=80=94=20capacity=20negative;=20ivrit.ai=20decision=20?= =?UTF-8?q?framed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index c0bdd54..6be6b91 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1713,3 +1713,20 @@ n-grams) → early-EOS truncation visible in the samples. v3's once). Clean A/B LAUNCHED (run-046): byt5-base, unfiltered labels, NO repetition suppression — vs v2-small's 0.5463 this isolates capacity alone. Lesson logged: one variable per arm, always. + +## WO31 stage-2 ledger CLOSES at 0.5463 — rules layer stands; ivrit.ai is the open decision (2026-10-11) + +run-046 (byt5-base, unfiltered, plain decode — the clean capacity +A/B): normalized CER **0.6318** vs small's 0.5463 — capacity NEGATIVE +at 3.2K utterances. Final ledger: v2-small 0.5463 (best) · base +0.6318 · filtered 0.885 (confounded) · rules layer 0.2393 (shipped, +stands). Every free lever measured: teacher conventions (fixed via +pointed-espeak), metric conventions (normalized CER), capacity +(negative), label filtering (unpromising under noise). The binding +constraints are structural: 10h FLEURS audio + 0.319-PER teacher. + +DECISION FOR OWNER: the path to ReNikud-class (0.0244) Hebrew g2p is +the ivrit.ai audio scale ($500 → 100+ hours labeled through the +now-proven pipeline: pointed-espeak targets, tuned CTC teacher, +normalized-CER gate). All pipeline risk is retired; the remaining +risk is budget. Without it, the rules layer ships as our g2p answer. From 58575b3c55aa0912cf4c7dfe418f1b90c51316c9 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 06:16:43 +0800 Subject: [PATCH 36/68] =?UTF-8?q?docs:=20WO31=20row=20=E2=80=94=20stage-2?= =?UTF-8?q?=20closed,=20ivrit=20decision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 197d412..1f8367f 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -66,7 +66,7 @@ hf jobs run --flavor a100-large -d \ | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 30 | 30-heb-learned-ipa-v0.md | P1 | 09 | NEGATIVE — CER 1.18; ASR teacher is the bottleneck | -| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | stage 1 GATE PASS (PER 0.2306); stage 2 RUNNING (CER gate <0.2393) | +| 31 | 31-heb-phoneme-asr-v1.md | P1 | 30 | stage 2 CLOSED at 0.546 nCER (capacity negative, metric/conventions fixed); rules 0.2393 stands; ivrit.ai $500 = owner decision | | 32 | 32-matched-architecture-arms.md | P1 | 17 | BiLSTM 14.30 refuted; run-033 10.78/9.16 gate-MISS — specialist dominates; successor = word-channel class | | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | From 234fe4ed4bf497b8b198cab34767beeb1ab3c7a5 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 06:48:57 +0800 Subject: [PATCH 37/68] =?UTF-8?q?docs:=20WO36=20ivrit.ai=20scale=20spec=20?= =?UTF-8?q?=E2=80=94=20approved,=20pending=20terms=20acceptance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 1 + TODO.sota/36-ivrit-scale.md | 50 +++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 TODO.sota/36-ivrit-scale.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 1f8367f..eaa8411 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -72,6 +72,7 @@ hf jobs run --flavor a100-large -d \ | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | | 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | | 35 | 35-alternate-protocol.md | P1 | 34 | SCORING ARTIFACT FIXED — OOD 4.10/1.56 (was 10.03/8.95); runtime fixed py#35/ts#108/ruby#805; 1.4-pt program open | +| 36 | 36-ivrit-scale.md | P1 | 31 | APPROVED — awaiting owner terms-acceptance on ivrit-ai/audio-v2; pipeline spec'd, all stages proven | ## Schedule diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md new file mode 100644 index 0000000..b4d9241 --- /dev/null +++ b/TODO.sota/36-ivrit-scale.md @@ -0,0 +1,50 @@ +# WO36 — ivrit.ai scale: the ReNikud-class Hebrew g2p run + +Owner approved the scale decision (2026-10-11). The $500 is NOT a +purchase: ivrit-ai/audio-v2 is free under its license (gated on +terms acceptance); the $500 is our own labeling compute at the WO09 +measured rate ($0.28/audio-hour × ~1,700h ≈ $476), spent through +normal HF jobs credits. + +## Access (OWNER, one click) + +Visit https://huggingface.co/datasets/ivrit-ai/audio-v2 logged in as +rhtse (or the org account preferred for data licensing) and accept +the terms — gated=auto grants immediately. Corpus: ~64K files, +10K Date: Sun, 11 Oct 2026 10:33:30 +0800 Subject: [PATCH 38/68] =?UTF-8?q?docs:=20run-040=20gate=20pass=204.00/1.55?= =?UTF-8?q?=20=E2=80=94=20new=20best;=20run-047=20large=20arm=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 6be6b91..3800290 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1730,3 +1730,20 @@ the ivrit.ai audio scale ($500 → 100+ hours labeled through the now-proven pipeline: pointed-espeak targets, tuned CTC teacher, normalized-CER gate). All pipeline risk is retired; the remaining risk is budget. Without it, the rules layer ships as our g2p answer. + +## WO35 arm 1 verdict — stripped-corpus specialist GATE PASS: 4.0034/1.5457 (2026-10-11) + +run-040 (byt5-base, run-029's exact recipe on first-alt-stripped +corpus, 290,575 units, 36,320 steps, 11h across two resume legs): +WikiNews first-alt **4.0034 WER / 1.5457 DER** — new best (shipped: +4.0976/1.5606); raw protocol improves too (9.98 vs 10.03); Sadeed ID +at parity (5.4840). Alternate emission shrank as designed. Export of +the stripped model as the ara-diac-news successor is STAGED for the +owner's version decision (bytes differ; new sha; 1.1 vs 2.0 naming). + +run-047 LAUNCHED (WO35 arm 2): byt5-LARGE seq2seq from pretrained +scratch (no large adapted teacher exists) on the same stripped corpus +— the untried capacity point in the dominant family. 36h timeout. +Note: size+init both change vs run-040 (no pure A/B possible); +the readout answers the practical question: does large beat the base +specialist at this corpus. From 89847c9327c729918d1ae65fe43997f8af408050 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 10:33:42 +0800 Subject: [PATCH 39/68] =?UTF-8?q?docs:=20row=2035=20=E2=80=94=20stripped-c?= =?UTF-8?q?orpus=20new=20best,=20large=20arm=20running?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index eaa8411..bfe2d47 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -71,7 +71,7 @@ hf jobs run --flavor a100-large -d \ | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | | 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | -| 35 | 35-alternate-protocol.md | P1 | 34 | SCORING ARTIFACT FIXED — OOD 4.10/1.56 (was 10.03/8.95); runtime fixed py#35/ts#108/ruby#805; 1.4-pt program open | +| 35 | 35-alternate-protocol.md | P1 | 34 | arm 1 PASS — stripped-corpus 4.0034/1.5457 NEW BEST (export staged, owner version); run-047 byt5-large RUNNING | | 36 | 36-ivrit-scale.md | P1 | 31 | APPROVED — awaiting owner terms-acceptance on ivrit-ai/audio-v2; pipeline spec'd, all stages proven | ## Schedule From 0566bd1ad1afb057b784350d210ef2526879aa9b Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 15:16:23 +0800 Subject: [PATCH 40/68] =?UTF-8?q?docs:=20WO36=20reality=20check=20?= =?UTF-8?q?=E2=80=94=2020K=20hours,=20companion=20gating,=20no-local=20rul?= =?UTF-8?q?e,=20staged=20teacher?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/36-ivrit-scale.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md index b4d9241..328e59c 100644 --- a/TODO.sota/36-ivrit-scale.md +++ b/TODO.sota/36-ivrit-scale.md @@ -15,16 +15,31 @@ the terms — gated=auto grants immediately. Corpus: ~64K files, arxiv:2307.08720. Companion: ivrit-ai/audio-transcripts if v2 ships audio-only segments. +## Reality check after access (2026-10-11) + +audio-v2 = **>20,000 hours** (not 1,700 — the corpus grew), raw +episode-length m4a/mp3 by source. Companions are SEPARATELY gated and +needed for pairing: audio-vad (segment boundaries), audio-transcripts, +crowd-transcribe-v5 (verified transcripts), audio-labeled (pre-paired +segments?) — OWNER: accept terms on each (one click per repo). +License: ivrit.ai v1 — broad research + commercial use; we ship only +derived models, never audio. + +STANDING RULE (owner, 2026-10-11): the corpus NEVER downloads to the +laptop — everything mounts/streams in HF jobs with the gated token. + ## Pipeline (every stage proven this week in WO31) -1. **Ingest**: download audio-v2 + transcripts to the HF dataset - repo `Interscript/ivrit-v2` (resumable, sample-audio verified, - total-hours counted). +1. **Audit** (in-job, cpu): mount/stream the repos, count real + hours, inspect schemas, 50-transcript QC — NO copying of audio; + pair via the companions' segment indices. 2. **Point**: transcripts → nikud via our crown (heb-diac-plane-2.0, DER 8.18) — the WO31 convention fix. Local/batch, free. -3. **Teacher at scale**: fine-tune the espeak-cv-ft CTC on the - pointed-espeak targets over the FULL corpus (FLEURS 10h gave PER - 0.319; ~1,700h should drive it far lower). a100, ~10-20 GPU-h. +3. **Teacher at scale**: fine-tune the espeak-cv-ft CTC on + pointed-espeak targets over a 300-500h slice first (gate: PER + < 0.25, ~10 GPU-h); scale only if the gate demands it — data is no + longer the constraint, compute budget is the dial (20K hours + available). 4. **Label**: run the tuned teacher over all audio (audio-derived vowel-full labels; ~$0.28/audio-hour ≈ the $500). 5. **Student at scale**: byt5-BASE (the capacity verdict was for 3.2K From f96485505ac62fa44f1d0131be97da13c2d0967f Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 15:16:36 +0800 Subject: [PATCH 41/68] docs: WO36 stage 2 in-job per no-local rule --- TODO.sota/36-ivrit-scale.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md index 328e59c..8bd68df 100644 --- a/TODO.sota/36-ivrit-scale.md +++ b/TODO.sota/36-ivrit-scale.md @@ -33,8 +33,9 @@ laptop — everything mounts/streams in HF jobs with the gated token. 1. **Audit** (in-job, cpu): mount/stream the repos, count real hours, inspect schemas, 50-transcript QC — NO copying of audio; pair via the companions' segment indices. -2. **Point**: transcripts → nikud via our crown (heb-diac-plane-2.0, - DER 8.18) — the WO31 convention fix. Local/batch, free. +2. **Point** (in-job): transcripts → nikud via our crown + (heb-diac-plane-2.0, DER 8.18) — the WO31 convention fix; artifact + zip fetched into the job, pointed text written to the bucket. 3. **Teacher at scale**: fine-tune the espeak-cv-ft CTC on pointed-espeak targets over a 300-500h slice first (gate: PER < 0.25, ~10 GPU-h); scale only if the gate demands it — data is no From e5c01e6354984a0dbff3fe270de71dbd606225f4 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 15:29:21 +0800 Subject: [PATCH 42/68] =?UTF-8?q?docs:=20WO36=20audit=20=E2=80=94=20self-c?= =?UTF-8?q?ontained=20pairs=20found,=20pointing=20stage=20running?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 3800290..653cb58 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1747,3 +1747,13 @@ scratch (no large adapted teacher exists) on the same stripped corpus Note: size+init both change vs run-040 (no pure A/B possible); the readout answers the practical question: does large beat the base specialist at this corpus. + +WO36 audit verdict (2026-10-11): audio-v2 = 35,629 audio files, +1.21TB (README floor >20K hours; header probes 404'd on the smallest +files — hour count deferred to the teacher stage). The decisive find: +**crowd-transcribe-v5 (~165K rows) and audio-labeled (~87K rows) are +SELF-CONTAINED pairs** (audio bytes + verified transcript embedded) +— no VAD, no cutting, no pairing engineering. ~250K ready pairs ≈ +10-20× FLEURS. Stage 2 RUNNING (run-049): text extraction + crown +nikud pointing over all pairs, 8-way multiprocessed in-job per the +no-local rule; audio stays in the parquets until the teacher stage. From e91b450527e84c19e657c6c83d77eb56e31b1d80 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 16:07:10 +0800 Subject: [PATCH 43/68] =?UTF-8?q?docs:=20WO36=20execution=20manifest=20?= =?UTF-8?q?=E2=80=94=20runs=20050-053?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/36-ivrit-scale.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md index 8bd68df..3ca53b2 100644 --- a/TODO.sota/36-ivrit-scale.md +++ b/TODO.sota/36-ivrit-scale.md @@ -58,6 +58,32 @@ laptop — everything mounts/streams in HF jobs with the gated token. - After (5): if the gate misses at 10× data, the diagnosis is student capacity/data quality, not pipeline — one iteration, then park. +## Execution manifest (run-level, 2026-10-11) + +- run-048 DONE: audit — v2 raw 1.21TB (>20K h); companions are + self-contained pairs: crowd-transcribe-v5 ~165K rows, audio-labeled + ~87K rows (schema: uuid + audio bytes + verified text). +- run-049 DONE/RUNNING: pointing — ~250K transcripts through the + crown nikud model, 8-way in-job; output pairs-pointed.jsonl. +- run-050: **webdataset builder** — stream both pair-repos, decode + embedded audio to 16-bit WAV (native sr, recorded), pair by uuid + with pointed text, tar-shard ~500MB each into + run-050-ivrit-wds/, CAP_HOURS slice (default 200h) with real + decoded-duration stats; deterministic shard order (extend by + raising the cap). Filter: 1s <= dur <= 30s. +- run-051: **teacher v2** — espeak-cv-ft CTC fine-tune on the slice + (pointed-espeak targets, tokenizer he backend), gates: FLEURS dev + PER < 0.25 AND ivrit-heldout PER reported. fp32 features, bf16 + autocast, step-ckpts, 12h timeout. +- run-052: **label** — tuned teacher decodes the slice (audio-derived + vowel-full labels) to label tars in the bucket. +- run-053: **student v3** — byt5-BASE text→IPA on the labels (capacity + verdict was for 3.2K pairs; at ~50K+ pairs base is right), gate: + normalized CER < 0.2393; ladder: Nakdimon 0.1102 · phonikud 0.0495 + · ReNikud 0.0244. +- GPU budget: stages 3+4 ≈ 12-20 GPU-h total (teacher slice is the + dial); well under the $500 envelope. + ## Alternative noted micsell/hebrew_speech_{campus,kan}_nikud — Hebrew speech with GOLD From 6ab3afea4af09dccddedf59cabaa07901f8cab1e Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 18:01:06 +0800 Subject: [PATCH 44/68] =?UTF-8?q?docs:=20WO36=20=E2=80=94=20labeler+studen?= =?UTF-8?q?t=20implemented;=20full=20chain=20executable?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/36-ivrit-scale.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/TODO.sota/36-ivrit-scale.md b/TODO.sota/36-ivrit-scale.md index 3ca53b2..d8059b4 100644 --- a/TODO.sota/36-ivrit-scale.md +++ b/TODO.sota/36-ivrit-scale.md @@ -75,12 +75,11 @@ laptop — everything mounts/streams in HF jobs with the gated token. (pointed-espeak targets, tokenizer he backend), gates: FLEURS dev PER < 0.25 AND ivrit-heldout PER reported. fp32 features, bf16 autocast, step-ckpts, 12h timeout. -- run-052: **label** — tuned teacher decodes the slice (audio-derived - vowel-full labels) to label tars in the bucket. -- run-053: **student v3** — byt5-BASE text→IPA on the labels (capacity - verdict was for 3.2K pairs; at ~50K+ pairs base is right), gate: - normalized CER < 0.2393; ladder: Nakdimon 0.1102 · phonikud 0.0495 - · ReNikud 0.0244. +- run-052 (IMPLEMENTED + uploaded, combines 052+053): resumable + labeler (teacher decodes the slice → labels.jsonl, LABEL_DONE + checkpoint) + byt5-BASE student (byte-correct collate) + normalized + CER gate in-job; ladder: Nakdimon 0.1102 · phonikud 0.0495 · + ReNikud 0.0244. Full chain now executable end-to-end. - GPU budget: stages 3+4 ≈ 12-20 GPU-h total (teacher slice is the dial); well under the $500 envelope. From ff5e3daf910d09d4cf9a6503c35ce6d206060887 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 19:07:49 +0800 Subject: [PATCH 45/68] docs: oracle 3.73 ceiling; successor export staged as 1.1 (provisional) --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 653cb58..f61c6e3 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1757,3 +1757,11 @@ SELF-CONTAINED pairs** (audio bytes + verified transcript embedded) 10-20× FLEURS. Stage 2 RUNNING (run-049): text extraction + crown nikud pointing over all pairs, 8-way multiprocessed in-job per the no-local rule; audio stays in the parquets until the teacher stage. + +Oracle re-probe under the corrected protocol (shipped r029 vs stripped +r040, both first-alt): **3.7302 WER / 1.4715 DER ceiling** — only +0.27 over the stripped best (threshold 2.0): same-family overlap, +routing not worth building. A 3-way re-probe (adding run-047 large) +opens when its verdict lands. Successor export STAGED as +ara-diac-news-1.1 (provisional id; owner renames at ship): +int8 IMF v1 zip with the corrected metrics in the manifest. From c95d8d0fc71523652da7b670f184bb7dc9ea0e45 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 19:32:54 +0800 Subject: [PATCH 46/68] =?UTF-8?q?docs:=20ara-diac-news-1.1=20staged=20?= =?UTF-8?q?=E2=80=94=20sha=20verified,=20ship=20gate=20=3D=20owner=20versi?= =?UTF-8?q?on?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index f61c6e3..9638a2a 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1765,3 +1765,11 @@ routing not worth building. A 3-way re-probe (adding run-047 large) opens when its verdict lands. Successor export STAGED as ara-diac-news-1.1 (provisional id; owner renames at ship): int8 IMF v1 zip with the corrected metrics in the manifest. + +Successor artifact STAGED (2026-10-11): ara-diac-news-1.1.zip — +399,193,366 bytes, sha256 7441914844a4782519326da0bfca6a25ef1925b489 +9ee086d5dea2e506c0faa0 (run-040-strip-alt/export/ in the bucket), +int8 IMF v1, ORT sanity green, corrected metrics in the manifest +(4.0034/1.5457). SHIP GATE: owner names the version (1.1 vs 2.0) and +cuts the GitHub Release + models.yaml entry + index bump + 3 runtime +pins (the established chain). From 37684aa45f786e8a7f382187828007d6becac0d2 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 20:10:01 +0800 Subject: [PATCH 47/68] docs: reconcile stale index rows per RESULTS --- TODO.sota/00-README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index bfe2d47..08b76ae 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -43,25 +43,25 @@ hf jobs run --flavor a100-large -d \ | 03 | 03-thai-hybrid-latency.md | P1 | — | DONE (py#31, RESULTS #258) | | 04 | 04-hebrew-plane-scaleup.md | P2 | #44 | gate CLEARED — artifact DER 8.18 (int8, runtime protocol); release pending owner version | | 05 | 05-external-benchmarks.md | P2 | 01, 07 | DONE (RESULTS #259; MILIM awaiting release) | -| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | jobs LAUNCHED (label + train, HF queue) | +| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | CLOSED — dose-response mapped (RESULTS wave 2); teacher-quality lesson became the WO31 pattern | | 07 | 07-nikud-ipa-rule-layer.md | P3 | — | DONE (py#29, py#30) | -| 08 | 08-thai-small-tier.md | P3 | #44 | job LAUNCHED (run-025, l4x4) | +| 08 | 08-thai-small-tier.md | P3 | #44 | CLOSED — see 12 (sub-10MB tier negative) | | 09 | 09-hebrew-audio-pseudolabel-probe.md | P3 (parked probe) | — | DONE as parked probe (script only) | | 10 | 10-heb-plane-2-release.md | P1 | 04 | DONE (release, index-v9, 3 pin bumps, card, golden, HF mirror #28) | -| 11 | 11-arabic-r8-chain.md | P1 | — | jobs RUNNING (a100) | +| 11 | 11-arabic-r8-chain.md | P1 | — | CLOSED — r8a/b/c/d verdicts in RESULTS waves 2-4; specialist shipped | | 12 | 12-thai-tiny-contract.md | P2 | — | NEGATIVE (caveated) — PER 2038%; sub-10MB closed | | 13 | 13-thai-lexicon-release.md | P2 | 03 | DONE (tha-lexicon-kaikki-1.0 + sha, CC BY-SA) | | 14 | 14-ts-plane-port.md | P2 | — | DONE (ts#99/#100/#101, CI leg models#265) | | 15 | 15-api-edge.md | P2 | 14 | DONE (api#29: edge-first /v1/infer, kind in index) | | 16 | 16-parking-lot.md | — | — | parked (not now) | -| 17 | 17-arabic-register-closure.md | P1 | 11 | dose-response MAPPED (RESULTS wave 2); r8d knee-seeking RUNNING; specialist option staged | -| 18 | 18-parity-runs.md | P1 | 10 | pending | +| 17 | 17-arabic-register-closure.md | P1 | 11 | CLOSED — register-specialist doctrine; specialist shipped; numbers superseded by WO35 corrected protocol | +| 18 | 18-parity-runs.md | P1 | 10 | DONE via WO10+WO23 — golden attached, 3-leg parity GREEN | | 19 | 19-lexicon-loader.md | P2 | 13 | DONE (py#33) | | 20 | 20-hebrew-large-arm.md | P1 | 04 | NEGATIVE — 8.48 vs base 8.18; base stays shipped | | 21 | 21-arabic-plane-transfer.md | P2 | 17 | VERDICT: OOD −3.14 WER, ID +1.66 (Pareto) — specialist candidate staged | | 22 | 22-runtime-releases.md | P1 | 14 | PREPARED — owner-gated versions | | 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound | -| 24 | 24-verdict-wave.md | P1 | 17 | r8a/b/c + run-026 + thai CLOSED; r8d (knee) RUNNING | +| 24 | 24-verdict-wave.md | P1 | 17 | CLOSED — all verdict waves recorded in RESULTS | | 25 | 25-ar-news-specialist.md | P1 | 17 | VERDICT: OOD 10.13/8.98 (crown), ID 5.50; export staged | | 27 | 27-ar-large-id-arm.md | P1 | — | NEGATIVE — 2.5765 parity with r7; capacity closed on AR too | | 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | From 527b180cf2c54b36eae283fc44ba0f41ccaccd59 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 20:11:14 +0800 Subject: [PATCH 48/68] =?UTF-8?q?docs:=20run-049=20stall=20root-caused=20?= =?UTF-8?q?=E2=80=94=20fork+requests=20deadlock,=20worker-local=20fs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 9638a2a..9a5b776 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1773,3 +1773,11 @@ int8 IMF v1, ORT sanity green, corrected metrics in the manifest (4.0034/1.5457). SHIP GATE: owner names the version (1.1 vs 2.0) and cuts the GitHub Release + models.yaml entry + index bump + 3 runtime pins (the established chain). + +WO36 stage-2 debugging ledger: r2 died on missing model deps; r3 +stalled 4h with ZERO shards — systematic-debugging probe (standalone +shard read: 1.4s total) isolated the classic fork+requests deadlock: +the module-level HfFileSystem forked into the 8-worker pool inherits +the parent's live connection state and hangs on first use. Fix: +worker-local fs construction. r4 running; lesson = never fork a +process that owns an HTTP session created pre-fork. From 4163d84586318dac76afe3bc39b30b9a8c929507 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sun, 11 Oct 2026 23:19:53 +0800 Subject: [PATCH 49/68] =?UTF-8?q?docs:=20WO37=20portfolio-sota=20=E2=80=94?= =?UTF-8?q?=20fas/khm/urd=20survey=20(no=20frontiers=20exist),=20define-an?= =?UTF-8?q?d-lead=20program?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 3 ++- TODO.sota/37-portfolio-sota.md | 37 ++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 TODO.sota/37-portfolio-sota.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 08b76ae..d791566 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -72,7 +72,8 @@ hf jobs run --flavor a100-large -d \ | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | | 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | | 35 | 35-alternate-protocol.md | P1 | 34 | arm 1 PASS — stripped-corpus 4.0034/1.5457 NEW BEST (export staged, owner version); run-047 byt5-large RUNNING | -| 36 | 36-ivrit-scale.md | P1 | 31 | APPROVED — awaiting owner terms-acceptance on ivrit-ai/audio-v2; pipeline spec'd, all stages proven | +| 36 | 36-ivrit-scale.md | P1 | 31 | RUNNING — access granted; audit DONE (250K self-contained pairs); pointing (spawn-fixed) in flight; 050-052 staged | +| 37 | 37-portfolio-sota.md | P2 | — | SPEC'D — fas/khm/urd: no external frontiers exist; protocol-packaging program (define-and-lead) queued behind WO36 | ## Schedule diff --git a/TODO.sota/37-portfolio-sota.md b/TODO.sota/37-portfolio-sota.md new file mode 100644 index 0000000..2d6f624 --- /dev/null +++ b/TODO.sota/37-portfolio-sota.md @@ -0,0 +1,37 @@ +# WO37 — SOTA EVERYWHERE: the uncovered portfolio languages + +The campaign matrices covered AR/HE/TH. The shipped stable also +carries fas-g2p-1.0, khm-latn-1.0, urd-diac-1.0, urd-g2p-1.0 — +shipped in earlier waves, never measured against 2026 frontiers. + +## Survey (2026-10-11, verified — no dedicated external benchmarks exist) + +- **Persian G2P**: fragmented; nearest venue = SIGMORPHON multilingual + G2P shared tasks; g2p-fa (LSTM, PyPI) is the main Persian tool. + No 2025+ Persian-specific benchmark surfaced. +- **Urdu diacritization**: no dedicated benchmark; QCRI EMNLP-2025 + Arabic-diacritization work is the script-family reference; URDUTTS + (LREC 2026, 89h TTS) is phonetics-adjacent only. +- **Khmer transliteration**: no external benchmark on record. + +## Doctrine where no frontier exists + +We DEFINE the reference: publish our held-out evaluation protocol + +our artifacts' numbers, and lead by construction (the AR/HE/TH +playbook inverted). This is cheaper than the big-language program and +makes the claim honest: "best published, protocol included". + +## Remaining work + +1. **Protocol packaging (fas/khm/urd)**: for each shipped artifact, + build a fixed held-out test set + evaluation script + published + metric (PER for g2p; DER-style for diac; CER for khm) in the + models repo — the same bench format as WikiNews-2024/phonikud. +2. **Nearest-external anchoring**: run g2p-fa (Persian) and any + reproducible Urdu baselines on OUR protocol; record as anchors. +3. **Gap arms only if an anchor beats us**: no speculative training. +4. **Khmer**: khm-latn-1.0 serves the ISC maps program, not a + benchmark axis — measure with the maps-corpus conformance suite + instead (round-trip + corpus fixtures). + +## Status: spec'd; implementation queued behind WO36 chain From 0b13dc2028f862f1854e400b7805b21a5a576e1f Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 11:09:19 +0800 Subject: [PATCH 50/68] =?UTF-8?q?docs:=20run-047=20gate=20pass=203.60/1.47?= =?UTF-8?q?=20=E2=80=94=20new=20best,=200.90=20to=20frontier;=203-way=20or?= =?UTF-8?q?acle=202.95;=201.2=20staged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 9a5b776..6db9268 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1781,3 +1781,24 @@ the module-level HfFileSystem forked into the 8-worker pool inherits the parent's live connection state and hangs on first use. Fix: worker-local fs construction. r4 running; lesson = never fork a process that owns an HTTP session created pre-fork. + +## run-047 verdict — GATE PASS: byt5-large stripped 3.5983/1.4715, NEW BEST; gap to frontier 0.90 (2026-10-12) + +run-047 (byt5-large from pretrained scratch, stripped corpus, 4 +epochs, 23h39m): WikiNews first-alt **3.5983 WER / 1.4715 DER** — +beats base-strip (4.0034/1.5457) and shipped (4.0976/1.5606); raw +protocol improves too (9.68 vs 10.03); Sadeed ID 7.39 (register cost +of the news specialist, consistent with the family). **Capacity +DOES pay in the dominant family once the corpus is stripped** — the +old large≈base finding was measured on the unstripped recipe. +Ladder: large-strip 3.60 > base-strip 4.00 > shipped 4.10; gap to +QCRI 2.70 = **0.90 WER**. + +3-way N-way oracle {shipped, base-strip, large-strip}: **2.9484 WER +/ 1.2743 DER** — arms are complementary (0.65 over best single) and +the ceiling sits 0.25 from the frontier. Routing stays closed +(<2 threshold); the portfolio-ceiling claim is recorded. + +run-054: export STAGED as ara-diac-news-1.2 (provisional id; int8 +IMF v1; owner names at ship). Ship candidates now: 1.1 (base-strip) +and 1.2 (large-strip) — the latter dominates on both OOD protocols. From 46b128a50dcca02eaa1da308b3d7220ea6b7937d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 11:09:36 +0800 Subject: [PATCH 51/68] =?UTF-8?q?docs:=20row=2035=20=E2=80=94=20large-stri?= =?UTF-8?q?p=20new=20best=203.60/1.47,=201.2=20staged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index d791566..67a4509 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -71,7 +71,7 @@ hf jobs run --flavor a100-large -d \ | 26 | 26-heb-noisy-student.md | P1 | 04 | NEGATIVE — 8.72 vs 8.18; all three text levers closed | | 33 | 33-lexicon-disambiguator.md | P1 | 17/32 | NEGATIVE — 18.07/11.44; word-only can't diacritize OOV | | 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | -| 35 | 35-alternate-protocol.md | P1 | 34 | arm 1 PASS — stripped-corpus 4.0034/1.5457 NEW BEST (export staged, owner version); run-047 byt5-large RUNNING | +| 35 | 35-alternate-protocol.md | P1 | 34 | arm 2 PASS — large-strip 3.5983/1.4715 NEW BEST (0.90 to frontier); 1.1+1.2 staged; 3-way oracle ceiling 2.9484 | | 36 | 36-ivrit-scale.md | P1 | 31 | RUNNING — access granted; audit DONE (250K self-contained pairs); pointing (spawn-fixed) in flight; 050-052 staged | | 37 | 37-portfolio-sota.md | P2 | — | SPEC'D — fas/khm/urd: no external frontiers exist; protocol-packaging program (define-and-lead) queued behind WO36 | From 70da54d3e6b3fcb68ee4566145aa3c4fa54f9d9d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 11:45:20 +0800 Subject: [PATCH 52/68] =?UTF-8?q?docs:=201.2=20artifact=20verified=20?= =?UTF-8?q?=E2=80=94=20821MB=20server-tier,=20ship=20gate=20owner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 6db9268..ac878f5 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1802,3 +1802,10 @@ the ceiling sits 0.25 from the frontier. Routing stays closed run-054: export STAGED as ara-diac-news-1.2 (provisional id; int8 IMF v1; owner names at ship). Ship candidates now: 1.1 (base-strip) and 1.2 (large-strip) — the latter dominates on both OOD protocols. + +run-054 artifact verified: ara-diac-news-1.2.zip — 821,291,775 bytes, +sha256 93e318714f3f792db8d29764990cdba2df8315addf0185ce9b32770353de27 +77, ORT sanity green. NOTE: 2× the 1.1 artifact (large backbone) — +server-tier deployment; the 100MB-capped edge tier remains on +ara-diac-small. Both staged artifacts await the owner's naming + +release chain (GitHub Release + models.yaml + index + 3 pins). From 5d29776b9b3b3adcdbc8b85823580084a882ab50 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 13:34:12 +0800 Subject: [PATCH 53/68] =?UTF-8?q?docs:=20WO37=20portfolio=20reference=20nu?= =?UTF-8?q?mbers=20published=20=E2=80=94=20fas=201.47,=20urd-diac=203.86,?= =?UTF-8?q?=20urd-g2p=2015.78?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index ac878f5..8a255d0 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1809,3 +1809,24 @@ sha256 93e318714f3f792db8d29764990cdba2df8315addf0185ce9b32770353de27 server-tier deployment; the 100MB-capped edge tier remains on ara-diac-small. Both staged artifacts await the owner's naming + release chain (GitHub Release + models.yaml + index + 3 pins). + +## WO37 verdict — portfolio reference numbers PUBLISHED (2026-10-12) + +run-056 (run-056-portfolio-eval): our shipped artifacts on their own +held-out test splits, protocol now published (char CER + word accuracy, +n=3000 slices, artifacts sha-verified from the release chain): + +| artifact | n | CER | word acc | +|---|---|---|---| +| fas-g2p-1.0 (HomoRich test) | 3000 | **0.0147** | 0.6307 | +| urd-diac-1.0 (held-out split) | 3000 | **0.0386** | 0.6750 | +| urd-g2p-1.0 (humair025 test) | 3000 | **0.1578** | 0.3313 | + +Reading: fas-g2p at 1.47% CER is crown-grade (rules-layer league); +urd-diac 3.86% is strong; urd-g2p 15.78% is measured against the +harsh convention (IPA with stress + syllable dots, char-level) — +33% exact-word is the honest weak spot and the only candidate for a +future training arm. No external frontier exists for any of the +three (WO37 survey) — these published numbers + protocol ARE the +reference (define-and-lead). Full-split runs (n≈12K) can refine the +slices on demand. From bd88715d3f9eefd7b6b251b10375eac0a38c36ec Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 13:34:46 +0800 Subject: [PATCH 54/68] =?UTF-8?q?docs:=20row=2037=20=E2=80=94=20portfolio?= =?UTF-8?q?=20reference=20numbers=20published?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 67a4509..b8630ad 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -73,7 +73,7 @@ hf jobs run --flavor a100-large -d \ | 34 | 34-hybrid-word-char.md | P1 | 32/33 | CLOSED — hybrid 13.53, fastText 13.59, vcd ±0, oracle 0.71 (all old-protocol; see 35) | | 35 | 35-alternate-protocol.md | P1 | 34 | arm 2 PASS — large-strip 3.5983/1.4715 NEW BEST (0.90 to frontier); 1.1+1.2 staged; 3-way oracle ceiling 2.9484 | | 36 | 36-ivrit-scale.md | P1 | 31 | RUNNING — access granted; audit DONE (250K self-contained pairs); pointing (spawn-fixed) in flight; 050-052 staged | -| 37 | 37-portfolio-sota.md | P2 | — | SPEC'D — fas/khm/urd: no external frontiers exist; protocol-packaging program (define-and-lead) queued behind WO36 | +| 37 | 37-portfolio-sota.md | P2 | — | REFERENCE NUMBERS PUBLISHED — fas 1.47 / urd-diac 3.86 / urd-g2p 15.78 CER (n=3000, protocol+artifacts sha'd); urd-g2p = the one future arm | ## Schedule From 48e024221f0cc2b083f53403093c483028e792c2 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 14:02:09 +0800 Subject: [PATCH 55/68] =?UTF-8?q?docs:=20fas=20anchor=20confirmed=20?= =?UTF-8?q?=E2=80=94=20g2p-fa=200.363=20skeleton=20CER=20vs=20our=200.0147?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 8a255d0..a8cc709 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1830,3 +1830,11 @@ future training arm. No external frontier exists for any of the three (WO37 survey) — these published numbers + protocol ARE the reference (define-and-lead). Full-split runs (n≈12K) can refine the slices on demand. + +fas anchor CONFIRMED (run-058-r3, n=1000): g2p-fa scores **0.3629 +consonant-skeleton CER** (0.7025 raw char CER — notation mismatched: +narrow IPA vs the HomoRich broad scheme) vs our fas-g2p-1.0's +**0.0147 total CER** on the identical split, word-by-word per its +published design. The measured lead is ~25× notation-robust. The fas +row of the portfolio closes: reference protocol published, artifact +verified, only competitor measured and beaten. From ac5b834ac9ca2145d4959942639ed3663b0be58d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 14:02:25 +0800 Subject: [PATCH 56/68] =?UTF-8?q?docs:=20WO37=20implemented=20=E2=80=94=20?= =?UTF-8?q?anchors=20confirmed,=20portfolio=20closed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/37-portfolio-sota.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/TODO.sota/37-portfolio-sota.md b/TODO.sota/37-portfolio-sota.md index 2d6f624..4e39c0e 100644 --- a/TODO.sota/37-portfolio-sota.md +++ b/TODO.sota/37-portfolio-sota.md @@ -34,4 +34,16 @@ makes the claim honest: "best published, protocol included". benchmark axis — measure with the maps-corpus conformance suite instead (round-trip + corpus fixtures). -## Status: spec'd; implementation queued behind WO36 chain +## Status: IMPLEMENTED (2026-10-12) + +- Portfolio numbers PUBLISHED (run-056, n=3000): fas 0.0147 · + urd-diac 0.0386 · urd-g2p 0.1578 CER. +- fas anchor CONFIRMED (run-058, n=1000): g2p-fa = 0.3629 + consonant-skeleton CER / 0.7025 raw (notation-mismatched) vs our + 0.0147 — measured lead ~25×, methodology published (word-by-word + per its design; both metrics reported for fairness). +- Full-split refinement (run-057) in flight. +- khm: maps-conformance spec'd. urd anchors: no reproducible tool + found (survey stands) — our protocol remains the reference. +- urd-g2p (15.78 CER): the one language with headroom — candidate + training arm when compute frees (byt5-base on 635K pairs). From 61dddd9ab273efcc00aaf0727457d04a2b8afbd0 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 14:30:29 +0800 Subject: [PATCH 57/68] =?UTF-8?q?docs:=20WO38=20urd-g2p=20base=20arm=20?= =?UTF-8?q?=E2=80=94=20launched,=20gate=200.1578?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.sota/00-README.md | 1 + TODO.sota/38-urd-g2p-base.md | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 TODO.sota/38-urd-g2p-base.md diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index b8630ad..4a60b45 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -74,6 +74,7 @@ hf jobs run --flavor a100-large -d \ | 35 | 35-alternate-protocol.md | P1 | 34 | arm 2 PASS — large-strip 3.5983/1.4715 NEW BEST (0.90 to frontier); 1.1+1.2 staged; 3-way oracle ceiling 2.9484 | | 36 | 36-ivrit-scale.md | P1 | 31 | RUNNING — access granted; audit DONE (250K self-contained pairs); pointing (spawn-fixed) in flight; 050-052 staged | | 37 | 37-portfolio-sota.md | P2 | — | REFERENCE NUMBERS PUBLISHED — fas 1.47 / urd-diac 3.86 / urd-g2p 15.78 CER (n=3000, protocol+artifacts sha'd); urd-g2p = the one future arm | +| 38 | 38-urd-g2p-base.md | P1 | 37 | RUNNING — byt5-base on 610K humair025 pairs; gate CER < 0.1578 (1.0) on full 12.7K test | ## Schedule diff --git a/TODO.sota/38-urd-g2p-base.md b/TODO.sota/38-urd-g2p-base.md new file mode 100644 index 0000000..e9a27de --- /dev/null +++ b/TODO.sota/38-urd-g2p-base.md @@ -0,0 +1,18 @@ +# WO38 — urd-g2p-2.0 candidate: byt5-base on the full humair025 split + +urd-g2p-1.0 is the portfolio's weakest measured axis (CER 0.1578 / +33% word on its own held-out test, n=3000; WO37). The 1.0 recipe: +byt5-small, 3 epochs, 610K train pairs (humair025/urdu-g2p-dictionary +via rababa-urdu/data). The AR-proven upgrade path applies directly: +same data, same schedule, byt5-BASE. + +Arm: run-056-urd-g2p-base — byt5-base, 3 epochs, bs 16 × accum 2, +max_len 256, gradient checkpointing, fp32→bf16 autocast. Gate: test +CER (raw char, the WO37 protocol) < 0.1578 on the FULL 12.7K-row +test split. Ship: byt5-base artifact export if passed (owner names +urd-g2p-2.0). + +Fallback if missed: small was capacity-capped, base underperformed → +data-quality arm (dedupe/normalize) before any bigger swing. + +Status: RUNNING (run-059-urd-g2p-base, a100, 24h). From 78020ca24a177e11fde1397ebcd62e9868c94e6d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 16:26:37 +0800 Subject: [PATCH 58/68] docs: WO38 launched; WO36 chain launch-dep notes --- docs/RESULTS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index a8cc709..1aaf093 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1838,3 +1838,10 @@ narrow IPA vs the HomoRich broad scheme) vs our fas-g2p-1.0's published design. The measured lead is ~25× notation-robust. The fas row of the portfolio closes: reference protocol published, artifact verified, only competitor measured and beaten. + +WO38 launched (run-059): urd-g2p byt5-base arm on the full 610K +humair025 pairs, gate CER < 0.1578 (1.0's WO37 number) on the full +12.7K test split; export as urd-g2p-2.0 if passed (owner names). +Launch-dep notes for the WO36 chain: run-050 builder needs soundfile +in pip; run-051 teacher additionally needs torchaudio (native-sr +resample) + espeak-ng/phonemizer/protobuf (the tokenizer contract). From aaff277fbd4c2a0ee9cfb21d4358cd5f2b2cce78 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 17:17:22 +0800 Subject: [PATCH 59/68] =?UTF-8?q?docs:=20full-split=20portfolio=20confirma?= =?UTF-8?q?tion=20=E2=80=94=200.0148/0.0377/0.1592?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 1aaf093..04f3ee1 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1845,3 +1845,10 @@ humair025 pairs, gate CER < 0.1578 (1.0's WO37 number) on the full Launch-dep notes for the WO36 chain: run-050 builder needs soundfile in pip; run-051 teacher additionally needs torchaudio (native-sr resample) + espeak-ng/phonemizer/protobuf (the tokenizer contract). + +run-057 FULL-SPLIT confirmation (n = full test splits): fas-g2p-1.0 +**0.0148** (10,576) · urd-diac-1.0 **0.0377** (11,940) · urd-g2p-1.0 +**0.1592** (12,699) — all within 0.001 of the n=3000 slices. The +published portfolio protocols now stand on full splits: fas 1.48% CER +(crown-grade, anchor beaten 25×), urd-diac 3.77% (reference), urd-g2p +15.92% (the WO38 byt5-base arm's target: < 0.1578 slice / ≈0.159 full). From 266e9d60b3f3c425b2b17d71d199acf447cdc629 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 18:21:37 +0800 Subject: [PATCH 60/68] =?UTF-8?q?docs:=20pointing=20complete=20=E2=80=94?= =?UTF-8?q?=20326,190=20pairs;=20builder=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 04f3ee1..ead5210 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1852,3 +1852,11 @@ run-057 FULL-SPLIT confirmation (n = full test splits): fas-g2p-1.0 published portfolio protocols now stand on full splits: fas 1.48% CER (crown-grade, anchor beaten 25×), urd-diac 3.77% (reference), urd-g2p 15.92% (the WO38 byt5-base arm's target: < 0.1578 slice / ≈0.159 full). + +WO36 stage-2 COMPLETE (2026-10-12): **326,190 pointed pairs** (of +326,303 rows, 99.97% kept) in 11.07h across the pointing saga (r3 +fork-deadlock, r5 spawn-hang, r7 context-manager bug — each +root-caused in the ledger; the thread design + resume markers +delivered). run-050 LAUNCHED: webdataset builder over the pair repos +(200h slice cap, WAV shards to the bucket, real hour count at +completion) — the teacher's runway. From b071ea9ebe86dfba18c03088e54d47e5ab0fd890 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 19:07:46 +0800 Subject: [PATCH 61/68] =?UTF-8?q?docs:=20webdataset=20complete=20=E2=80=94?= =?UTF-8?q?=20200.61h/126=20shards;=20teacher=20staged?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index ead5210..e4fac3b 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1860,3 +1860,9 @@ root-caused in the ledger; the thread design + resume markers delivered). run-050 LAUNCHED: webdataset builder over the pair repos (200h slice cap, WAV shards to the bucket, real hour count at completion) — the teacher's runway. + +run-050 COMPLETE (40.6 min): **200.61 audio-hours, 126 WAV tar +shards** in the bucket (1s-30s filter, pointed text paired by uuid, +deterministic order — extendable by rerun with a higher cap). +Teacher (run-051) staged and fires when an a100 slot frees (run-055 +and run-059 both finish within ~2h; the ≤2-GPU law holds). From f23a5eabb4c7cff1b942af223eef6899fdaf759d Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 19:56:38 +0800 Subject: [PATCH 62/68] =?UTF-8?q?docs:=20WO38=20marginal=20pass=20?= =?UTF-8?q?=E2=80=94=200.1565=20vs=200.1592;=20data-quality=20lever=20reco?= =?UTF-8?q?mmended?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index e4fac3b..317aafa 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1866,3 +1866,13 @@ shards** in the bucket (1s-30s filter, pointed text paired by uuid, deterministic order — extendable by rerun with a higher cap). Teacher (run-051) staged and fires when an a100 slot frees (run-055 and run-059 both finish within ~2h; the ≤2-GPU law holds). + +WO38 verdict — MARGINAL PASS (2026-10-12): urd-g2p byt5-base (610K +pairs, 5h23m): full-split CER **0.1565** vs 1.0's 0.1592 (word acc +0.3273 → 0.3467). Gate passes but the delta is thin — both models +cap near ~16% CER because the dictionary's IPA convention itself +carries ambiguity (stress/syllabification variants per word). The +meaningful next lever is the spec's fallback: data-quality arm +(dedupe/normalize targets) before any bigger swing. Export staging = +owner's call given the thin margin; recommendation: hold for the +data-quality arm. From 2e4389581657a9a2dd5a31bb195953347b8e1cfb Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 22:22:17 +0800 Subject: [PATCH 63/68] =?UTF-8?q?docs:=20urd-g2p=20anatomy=20=E2=80=94=20g?= =?UTF-8?q?enuine=20phonology,=20no=20notation=20lever;=20reference=20stan?= =?UTF-8?q?ds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 317aafa..3fd54ed 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1876,3 +1876,14 @@ meaningful next lever is the spec's fallback: data-quality arm (dedupe/normalize targets) before any bigger swing. Export staging = owner's call given the thin margin; recommendation: hold for the data-quality arm. + +urd-g2p error anatomy (run-059-anatomy, full test): normalized CER +**0.1075** (stress/dots carried only ~5pp of the raw 0.1565); the +remaining errors are genuine phonology — vowels/diacritics 41.7% +(short-vowel reduction ə/ɪ/a choices), consonants 34.2% (retroflex/ +aspirate inventory), insertions 24.2% (epenthetic vowels, e.g. +fiːlæm for film). No notation escape hatch exists; the axis's +reference stands at ours (0.1565 raw / 0.1075 normalized), and a +future arm would need real phoneme-model improvements (data scale +beyond the 610K dictionary, or multilingual transfer) — queued as +the portfolio's optional arm, not an active chase. From e2138088159f3109cdf415ced6215669dcca3cae Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Mon, 12 Oct 2026 23:41:26 +0800 Subject: [PATCH 64/68] =?UTF-8?q?docs:=20teacher=20FLEURS=20gate=20pass;?= =?UTF-8?q?=20ivrit=20dev=20anomaly=20=E2=80=94=20student=20as=20diagnosti?= =?UTF-8?q?c?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 3fd54ed..b42d4fa 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1887,3 +1887,12 @@ reference stands at ours (0.1565 raw / 0.1075 normalized), and a future arm would need real phoneme-model improvements (data scale beyond the 610K dictionary, or multilingual transfer) — queued as the portfolio's optional arm, not an active chase. + +run-051 teacher verdict — GATE PASS on FLEURS (0.2069 < 0.25, vs +0.319 from 10h) but in-domain ivrit dev PER anomalous at 0.8722 — +almost certainly an evaluation-convention artifact in the dev-gold +reconstruction path (a model trained ON ivrit cannot plausibly score +87% on ivrit). Decisive test launched (run-052): the labeler+student +chain against this teacher — its phonikud CER is the ground-truth +diagnostic. If the student screams, the dev path gets the fix; if it +clears < 0.2393, the anomaly was cosmetic. From ecb284813de27b1b2365a1334ade0ffb3dc89780 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 13 Oct 2026 00:22:43 +0800 Subject: [PATCH 65/68] =?UTF-8?q?docs:=20e6=20leg=20converged=20=E2=80=94?= =?UTF-8?q?=203.56/1.46=20new=20best,=200.86=20to=20frontier;=201.2=20re-e?= =?UTF-8?q?xport=20launched?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/RESULTS.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b42d4fa..3bd7831 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1896,3 +1896,14 @@ reconstruction path (a model trained ON ivrit cannot plausibly score chain against this teacher — its phonikud CER is the ground-truth diagnostic. If the student screams, the dev path gets the fix; if it clears < 0.2393, the anomaly was cosmetic. + +run-055 verdict (e6 leg, 12h16m): first-alt **3.5607 WER / 1.4588 +DER** — new best (4-epoch: 3.5983/1.4715; the +2 epochs bought ++0.038 WER: the leg is CONVERGED, tail thin like run-029c). Raw +protocol improves too (9.66 vs 9.98 → under 10 for the first time +on the unstripped-protocol comparison). Sadeed ID 7.48 (register +cost, family-consistent). Gap to QCRI 2.70: **0.86 WER**. +3-way oracle with e6 preds: 2.9861/1.2828 (complementarity holds; +routing stays closed <2). Re-export of ara-diac-news-1.2 from the +e6 ckpt LAUNCHED (replaces the staged 4-epoch zip; same provisional +name pending owner). From 95e1a60245fc2221f5cddf5b2fc3f5347f3cd9b3 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 13 Oct 2026 01:56:04 +0800 Subject: [PATCH 66/68] docs: 1.2 e6 artifact verified; external-data root cause + pin recorded --- docs/RESULTS.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 3bd7831..b4b9d37 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1907,3 +1907,12 @@ cost, family-consistent). Gap to QCRI 2.70: **0.86 WER**. routing stays closed <2). Re-export of ara-diac-news-1.2 from the e6 ckpt LAUNCHED (replaces the staged 4-epoch zip; same provisional name pending owner). + +run-060-r2 EXPORT VERIFIED: ara-diac-news-1.2.zip (e6 checkpoint) — +821,320,781 bytes, sha256 12cb294d73318c7717b8e24ed1b95642d1510c2af +515127eb329fb88b1950548, ORT sanity green. Root cause of the failed +r2 recorded: torch.onnx.export auto-selects external-data format when +the model crosses the 2GB protobuf boundary (the e6 large encoder +crossed it; run-054's export sat just under) — fixed by pinning +use_external_data_format=False in both export calls (committed in +train repo PR #133's export script). From 34ee04999116d6250ea0da11dc5cfb84d4ec382a Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 13 Oct 2026 13:13:21 +0800 Subject: [PATCH 67/68] docs: pointed-convention bug caught pre-gate; teacher v3 running end-to-end-consistent --- docs/RESULTS.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index b4b9d37..d65ac07 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1916,3 +1916,19 @@ the model crosses the 2GB protobuf boundary (the e6 large encoder crossed it; run-054's export sat just under) — fixed by pinning use_external_data_format=False in both export calls (committed in train repo PR #133's export script). + +WO36 CORRECTION — the pointed-convention bug, caught before the gate +(2026-10-13): label spot-check revealed run-052's labels are +CONSONANT SKELETONS — run-051's teacher trained on r["text"] +(UNPOINTED transcripts → espeak-he → consonant skeletons) instead of +the crown-nikud pointed transcripts sitting in the same tars; the +labeling decode additionally ran on 2×-time-warped 32k audio (run +before the resample fix). run-052 killed at 15% student training — +training toward those labels was waste. FIXES: teacher v3 (run-061, +RUNNING) trains on the pointed field (vowel-full targets, primary +gate = ivrit dev same-convention; FLEURS reported as +convention-shifted); the labeler carries the resample fix; the +student gate (phonikud, vowel-full gold) is then CONSISTENT +end-to-end for the first time. The WO31 lesson generalizes: the +pointed-vs-unpointed convention must be asserted at EVERY stage — +targets, gates, and labels — not just the first espeak call. From 20e7cf1f377c5cbba67e22a9da22465f73593a34 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Tue, 13 Oct 2026 16:42:24 +0800 Subject: [PATCH 68/68] docs: seed variance confirms eval bug; run-062 corrected chain launched --- docs/RESULTS.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/RESULTS.md b/docs/RESULTS.md index d65ac07..de719fa 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1932,3 +1932,17 @@ student gate (phonikud, vowel-full gold) is then CONSISTENT end-to-end for the first time. The WO31 lesson generalizes: the pointed-vs-unpointed convention must be asserted at EVERY stage — targets, gates, and labels — not just the first espeak call. + +Seed-variance result (run-061b, seed 1337 vs run-061 seed 42): +FLEURS 0.2055 vs 0.2069 · ivrit dev 0.8745 vs 0.8722 — seeds agree to +±0.002, so BOTH numbers are stable measurements. Conclusion: the +FLEURS gate pass is real (teacher improved 0.319 → 0.206 at 200h), +and the 0.87 ivrit-dev is seed-independent → confirmed as the EVAL +BUG hypothesized earlier: asr_phones' dict branch ignores the +recorded 32000 Hz sample rate (2×-time-warped audio → garbage decode). +The training path resamples correctly (pcm_to_16k uses r["sr"]). +run-062 LAUNCHED: labeler+student with the v3 teacher in a FRESH run +dir (run-062, not run-052 — the stale consonant-skeleton labels.jsonl +from the wrong-convention teacher cannot resume). Gate tonight: +phonikud nCER < 0.2393 on vowel-full labels — the first +end-to-end-consistent Hebrew g2p measurement.