diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md new file mode 100644 index 0000000..8feb0f1 --- /dev/null +++ b/TODO.sota/00-README.md @@ -0,0 +1,56 @@ +# TODO.sota — 2026 SOTA campaign (client-usable quality) + +Derived from the 2026-10-05 SOTA sweep (Arabic / Hebrew / Thai). +Goal: close the measured gaps to the 2026 frontier on all three +languages while keeping every artifact client-usable (int8, sha-verified, +multi-runtime, versioned). + +## Substrate: HF (NOT Modal) + +All training and eval jobs run on **Hugging Face Jobs** (`hf` CLI, v1.26+): + +```bash +hf jobs run --flavor a100-large -d \ + -v hf://datasets/Interscript/hebrew-v4:/data:ro \ + -v hf://buckets/Interscript/isx-training:/ckpt:rw \ + python:3.12 python train.py +``` + +- Datasets: `hf://datasets/Interscript/` (mounted read-only). +- Checkpoints/artifacts: `hf://buckets/Interscript/isx-training` (read-write). +- Detached always (`-d`); poll via `hf jobs ls` / `hf jobs logs`. +- Modal is retired for compute. It may be used ONLY to export corpora + off Modal volumes during the one-time migration (#44). + +## Standing rules that shape this campaign + +- **NO LLM TEACHERS** (memory law, 2026-08): never use Claude/GPT/ + Gemini/any LLM as a label source for haraqat/niqqud/Thai IPA. + WO06 is therefore noisy-student/self-ensemble — NOT the + "LLM-ensemble teacher" phrasing used in the sweep notes. +- Versions/releases/cards/index are owner decisions at ship time; + specs define gates, not version numbers. +- Data channel for shipped artifacts stays GitHub Releases + sha256; + HF dataset repos here are TRAINING inputs, not the shipping channel. +- PRs only; no tags without explicit instruction. + +## Work orders + +| WO | File | Priority | Depends | Status | +|----|------|----------|---------|--------| +| 01 | 01-wikinews-multiref-reconciliation.md | P1 | — | pending | +| 02 | 02-user-diacritics-preservation.md | P1 | — | pending | +| 03 | 03-thai-hybrid-latency.md | P1 | — | pending | +| 04 | 04-hebrew-plane-scaleup.md | P2 | #44 | pending | +| 05 | 05-external-benchmarks.md | P2 | 01, 07 | pending | +| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | pending | +| 07 | 07-nikud-ipa-rule-layer.md | P3 | — | pending | +| 08 | 08-thai-small-tier.md | P3 | #44 | pending | +| 09 | 09-hebrew-audio-pseudolabel-probe.md | P3 (parked probe) | — | pending | + +## Schedule + +Wave 1 (local, days): WO01, WO02, WO03, WO07 + infra #44. +Wave 2 (HF training, launched together): WO04 (a100-large), WO06 +(a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`. +Wave 3: WO05 tables after 01+07 land; WO09 probe only. diff --git a/TODO.sota/01-wikinews-multiref-reconciliation.md b/TODO.sota/01-wikinews-multiref-reconciliation.md new file mode 100644 index 0000000..f582b94 --- /dev/null +++ b/TODO.sota/01-wikinews-multiref-reconciliation.md @@ -0,0 +1,41 @@ +# WO01 — WikiNews-2024 multiref reconciliation (Arabic) + +**Priority:** P1 · **Cost:** hours · **Gate:** a protocol-matched statement +of where our models stand on WikiNews-2024, in RESULTS.md. + +## Problem + +Our OOD number on WikiNews-2024 is 17.38 WER / 11.83 DER while the EMNLP +2025 QCRI paper (Mohamed & Mubarak, 2025.emnlp-main.846) reports +**2.70% WER** with a BiLSTM on the same benchmark name. A 6× gap on the +same named benchmark is almost certainly protocol divergence, not model +divergence — but we cannot quote ANY WikiNews number until the delta is +explained. Suspects: + +1. Different test split (our slice may be the harder OOD half). +2. Different reference policy (multiref semantics: correct if the word + matches ANY reference; ref count per word may differ). +3. Different WER definition (undiacritized word identity vs + fully-diacritic-exact match). +4. Normalization (ta-marbuta/ha, alef variants, tatweel, punct). +5. Different model class being scored (our number may come from a plane + or seq2seq student, theirs from a task-dedicated BiLSTM). + +## Deliverables + +1. Obtain the official WikiNews-2024 release (paper's repo/QCRI) — + test paragraphs + per-word references + (if published) their scorer. + Record provenance + sha256 of the files we use. +2. Reconcile harnesses: run `interscript-train/eval_wikinews_multiref.py` + (ours) and their protocol on THE SAME predictions; diff every metric + definition line-by-line; write a delta table. +3. Score, under the reconciled protocol: released ara-diac-2.0, + ara-diac-plane-1.0, ara-diac-plane-large-1.0 (+ r7 if weights are + reachable). Greedy, no conditioning, runtime protocol. +4. RESULTS.md entry: one table, protocol column explicit, plus a + one-paragraph verdict (mismatch explained or real gap owned). + +## Out of scope + +Retraining; changing the shipped artifacts; quoting any WikiNews number +externally before this WO lands. diff --git a/TODO.sota/02-user-diacritics-preservation.md b/TODO.sota/02-user-diacritics-preservation.md new file mode 100644 index 0000000..d90ad9b --- /dev/null +++ b/TODO.sota/02-user-diacritics-preservation.md @@ -0,0 +1,48 @@ +# WO02 — User-diacritics preservation in plane runtimes + +**Priority:** P1 · **Cost:** hours–days · **Gate:** `preserve` mode with +byte-identical user diacritics, all tests green, no behavior change when +off. + +## Why + +QCRI's EMNLP 2025 paper ships "a model that preserves user-specified +diacritics" as a flagship client feature. For us it is nearly free: the +plane's K-pass conditioning already takes a plane tensor as input, so a +partially-vocalized input is just a partially-filled plane — pin user +classes, mask only the empty positions. + +## Contract + +`PlaneModel.translate(text, preserve_diacritics=True)`: + +1. `split_planes(text)` on the INPUT (skeleton + user classes; Hebrew: + `nikud_planes`, Arabic: `haraqat_planes`). +2. Initial plane tensor = user class ids where non-empty, MASK_ID where + empty. K-pass decode runs as usual. +3. Positions the user supplied are NEVER overwritten by the model — + final classes = user classes at those positions, model predictions + elsewhere. +4. Byte-exact guarantee: every diacritic the user typed round-trips + verbatim (cluster order preserved as written — the Hebrew canon + lesson applies to user input too). +5. Default `preserve_diacritics=False` — behavior and parity goldens + UNCHANGED. + +## Deliverables + +- `interscript-py`: `PlaneModel.translate(..., preserve_diacritics=...)` + (TDD: fully-labeled input → identity output; partial input → user + classes intact, empty positions filled; unlabeled → identical to + current path). +- `interscript-ruby-ml`: same flag on `Interscript::ML::PlaneModel#translate` + (same test matrix against the tiny-plane fixture). +- ts port: blocked on the documented ts plane loader gap — note it, do + not unblock here. +- RESULTS.md / model card note: preservation mode + the byte-exact + guarantee. + +## Explicit non-goals + +No seq2seq (IMF) preservation mode — that decode cannot pin positions +without a new contract; out of scope. diff --git a/TODO.sota/03-thai-hybrid-latency.md b/TODO.sota/03-thai-hybrid-latency.md new file mode 100644 index 0000000..5fcb9ef --- /dev/null +++ b/TODO.sota/03-thai-hybrid-latency.md @@ -0,0 +1,43 @@ +# WO03 — Thai hybrid fast-path + latency table + +**Priority:** P1 · **Cost:** days · **Gate:** published latency × accuracy +table; hybrid within 1.2× of dict-only latency and within +0.3pp PER of +model-only on our benchmark. + +## Why + +FastThaiG2P (arXiv 2608.12814) owns the latency narrative (0.15 ms/utt) +but is 26.9% PER on its own set — dictionary+rules. We own quality +(2.85% PER, greedy, runtime protocol) at neural latency. The client +answer is not a race but a hybrid tier: dictionary fast-path for +in-vocab words, our neural model as the OOV fallback (their own profile: +58% of latency is fallbacks — that fallback is us). + +## Design + +1. **Lexicon**: word→IPA from Kaikki Thai (the same lineage as training + data). Multiple readings per word → keep the most frequent; store + count provenance. Build script emits `tha-lexicon.jsonl` + (word, ipa, count) — a data artifact, versioned with the model. +2. **Fast-path**: Thai has no spaces → longest-prefix (maximal-munch) + match over the lexicon; unmatched spans go to the neural model + (windowed 1400-byte protocol, unchanged); spliced back. +3. **Bench** on `thai-kaikki-g2p/test.jsonl` (1,219 sentences, CPU): + - dict-only PER + latency/utt + - model-only PER + latency/utt + - hybrid PER + latency/utt (+ OOV-fallback rate) +4. **RESULTS.md**: the table, protocol column explicit, one-paragraph + positioning vs FastThaiG2P (never quote their PER against ours — + different set; quote ours and cite theirs as self-reported). + +## Deliverables + +- `interscript-py`: hybrid translate path + lexicon builder script. +- Artifact: lexicon released alongside the model (GitHub Releases + + sha256, per data-channel law). +- Bench JSON + RESULTS.md table. + +## Non-goals + +PyThaiNLP dependency (our maximal-munch over Kaikki covers it); +retraining anything; ts/ruby ports of the hybrid (py first, note ports). diff --git a/TODO.sota/04-hebrew-plane-scaleup.md b/TODO.sota/04-hebrew-plane-scaleup.md new file mode 100644 index 0000000..d5c7daa --- /dev/null +++ b/TODO.sota/04-hebrew-plane-scaleup.md @@ -0,0 +1,39 @@ +# WO04 — Hebrew plane scale-up (HF training) + +**Priority:** P2 · **Cost:** ~10–15 h on `a100-large` · **Gate:** +**≤ 10.0 DER** on the nakdimon test (exact run-021 protocol: v4 test, +windowed 1400 bytes, original-separator stitching, K-pass greedy, +seq2seq_der). Shipped baseline: heb-diac-plane-1.0 at 12.48. + +## Recipe (run-022) + +- Encoder: **byt5-base** (up from small). Plane embedding + per-position + head unchanged (the ReNikud-validated factorization). +- Corpus: hebrew-v4 (rebuilt locally from + nakdimon+sefaria+distilled-v1/v2+expanded-v2 per + `train_hebrew_v4.py`), uploaded to + `hf://datasets/Interscript/hebrew-v4`, mounted ro. +- Epochs 4 (up from 2), warmup 500, cosine, bs 16 (grad-accum if + memory-bound on base), lr 1e-4, bf16 autocast, K_PASSES=3 at eval. +- Checkpoints → `hf://buckets/Interscript/isx-training` every 500 steps + + `labels.sha` resume contract + per-marker eval progress file + (run-019 lesson) + original-separator stitching (run-021 lesson, + already correct in the trainer). + +## Deliverables + +1. `train_hebrew_plane_hf.py` — the HF Jobs port of run-021's trainer + (paths: /data, /ckpt; volumes; no Modal). +2. Launch on `a100-large -d`; monitor via `hf jobs logs`. +3. Export plane.onnx int8 from the best checkpoint; package plane zip + (metadata kind=plane + sha256s + classes.json); golden rows vs the + py runtime. +4. **If gate passes**: full ship chain — release (version = owner + decision, ASK before publishing), index entry, card, HF mirror. + If it fails: verdict in RESULTS.md, keep 1.0 shipped. + +## Honest risks + +byt5-base on 50K units may overfit by epoch 3 — the gate decides, not +the hope. 12.48 → 10 is ambitious; a plateau at ~11 is a documented +negative, not a failure of the campaign. diff --git a/TODO.sota/05-external-benchmarks.md b/TODO.sota/05-external-benchmarks.md new file mode 100644 index 0000000..d3dd7fd --- /dev/null +++ b/TODO.sota/05-external-benchmarks.md @@ -0,0 +1,35 @@ +# WO05 — External benchmark adoption + +**Priority:** P2 · **Depends:** WO01 (multiref protocol), WO07 (nikud→IPA +for g2p comparability) · **Gate:** RESULTS.md tables with explicit +protocol columns for at least: heb plane on heb-g2p-benchmark, Arabic +models on reconciled WikiNews-2024. + +## Targets + +1. **heb-g2p-benchmark** (Phonikud line; ReNikud reports 85.1% word + accuracy on it): adopt their test set. Our plane emits nikud; with + WO07's nikud→IPA rules we can report both (a) nikud-restoration + DER/WER on their vocalized test text and (b) composed IPA word + accuracy — each labeled with its protocol. If their set is + nikud-free (plain Hebrew → IPA), report (b) only. +2. **WikiNews-2024 multiref** (from WO01): becomes the OFFICIAL Arabic + OOD protocol alongside SadeedDiac-25; every future Arabic RESULTS.md + entry carries both. +3. **MILIM** (ReNikud's spoken benchmark): track its release; evaluate + only when data is actually obtainable — do not fabricate numbers + (memory law). + +## Deliverables + +- Eval scripts under `interscript-models/benchmarks/` (reproducible, + pinned inputs + sha256s). +- RESULTS.md: new "External benchmarks" section; every number carries + protocol + input provenance; missing data (MILIM) listed as "awaiting + release", never estimated. + +## Non-goals + +Protocol shopping: we adopt THEIR definitions where feasible; where a +definition is ambiguous we publish BOTH numbers labeled, never pick the +flattering one silently. diff --git a/TODO.sota/06-arabic-r8-noisy-student.md b/TODO.sota/06-arabic-r8-noisy-student.md new file mode 100644 index 0000000..98dea01 --- /dev/null +++ b/TODO.sota/06-arabic-r8-noisy-student.md @@ -0,0 +1,44 @@ +# WO06 — Arabic r8: noisy-student / self-ensemble (HF training) + +**Priority:** P2 · **Cost:** teacher labeling (~hours on GPU) + training +(~6–8 h `a100-large`) · **Gate:** **beat 2.2864 DER** (r7/ara-diac-2.0) +on SadeedDiac-25 under the exact zero-skip haraqat-projected protocol; +stretch ≤ 2.0. + +## PIVOT (standing rule) + +The sweep notes floated an "LLM-ensemble teacher". **Forbidden** — +memory law: NEVER use Claude/GPT/Gemini/any LLM as a diacritization +label source (hallucinated haraqat bake into the student). r8 uses only +signals we control: + +1. **Self-training (noisy student):** our best dedicated model + pseudo-labels UNLABELED Arabic text (`data/arwiki`, 115 MB local); + keep predictions where the model is confident (per-position class + margin above threshold; span-keep, not char-spaghetti). +2. **Mixture:** gold tashkeela-full + pseudo-arwiki (cap pseudo at ~50% + of gold tokens; ablate the ratio only if the first run misses the + gate). +3. **Self-ensemble (optional second arm):** per-position vote across + OUR OWN checkpoints (r6+r7 lineages + plane-large) as pseudo-labels — + model diversity without external teachers. +4. Excluded: yallamorph-aux (run-009 measured NEGATIVE — do not re-add). + +## Deliverables + +1. `pseudo_label_arabic_hf.py` — teacher inference + margin filter + (ONNX int8 teacher on GPU is fine; greedy; emit arwiki-pseudo.jsonl + with kept-token stats). +2. `train_arabic_r8_hf.py` — the proven dedicated seq2seq lineage + (byt5-small, IMF contract), mixture loader, same eval protocol as + run-019's corrected harness (per-marker progress, exact separators). +3. HF Jobs launch + monitoring; export/package per release system. +4. **If gate passes:** ship chain with owner-confirmed version; else + RESULTS.md verdict with kept-token stats and margin-threshold + sensitivity. + +## Honesty anchors + +Single-seed caveat (2608.27729): a gain under ~0.15 DER is within seed +noise — mark it as such rather than claiming SOTA movement. Keep-token +rate is a first-class reported number. diff --git a/TODO.sota/07-nikud-ipa-rule-layer.md b/TODO.sota/07-nikud-ipa-rule-layer.md new file mode 100644 index 0000000..1224c99 --- /dev/null +++ b/TODO.sota/07-nikud-ipa-rule-layer.md @@ -0,0 +1,44 @@ +# WO07 — nikud→IPA rule layer + +**Priority:** P3 · **Cost:** days · **Gate:** deterministic, documented, +tested rule table over the v4 corpus cluster inventory; composes with +the plane runtime (`heb plane → nikud text → IPA`). + +## Why + +ReNikud's target surface is IPA for TTS (lexical stress, spoken norms). +We are not ReNikud — but a deterministic nikud→IPA layer gives TTS +clients a working Hebrew phonemizer TODAY, on top of our shipped plane +model, and unlocks WO05's comparability with heb-g2p-benchmark. + +## Design + +1. **Rules over the actual corpus inventory** (not theoretical Hebrew): + enumerate the nikud clusters that occur in hebrew-v4 (the 135-class + inventory); write letter×cluster→IPA per standard reading. + - Special cases handled explicitly: שׁ/שׂ dots; dagesh lene in + בג"ד כפ"ת; dagesh chazaq (doubling) conservative rule; sheva + (initial → ə; otherwise ə/e per position heuristics, documented as + heuristic); matres lectionis (vav/yod with holam/hiriq/male); + final gutturals; qamats qatan (documented as lexicon-resolved). +2. **Lexicon overrides**: a table for the highest-frequency exceptions + (qamats qatan words, proper names); rules lose to lexicon. +3. **Composition API**: `nikud_to_ipa(text)` in + `interscript-py` + a `PlaneModel.translate_to_ipa()` convenience + that chains plane restoration → rules. Stateless, deterministic, + no ONNX involved. +4. **Tests**: cluster-inventory coverage table (every class → explicit + rule or explicit lexicon fallback — no silent falls-through); + round-trip spot checks against our labeled corpus pairs; golden + examples for each special case. + +## Deliverables + +- `nikud_ipa.py` + rule doc (the table itself is the doc). +- Tests (TDD, real corpus strings, no doubles). +- Documented limitation: no lexical stress, no spoken-norm variation — + that is ReNikud's audio-supervised territory; stated in the card/README. + +## Non-goals + +Learning anything (rules only); stress assignment; teamim. diff --git a/TODO.sota/08-thai-small-tier.md b/TODO.sota/08-thai-small-tier.md new file mode 100644 index 0000000..67050ee --- /dev/null +++ b/TODO.sota/08-thai-small-tier.md @@ -0,0 +1,43 @@ +# WO08 — Thai small-tier student (HF training) + +**Priority:** P3 · **Cost:** ~2–4 h `l4x1` · **Gate:** **≤ 3.5% PER +(greedy, corpus-level, the runtime protocol)** at **< 10 MB** int8. +Ship only if both hold; else documented negative. + +## Why + +FastThaiG2P wins voice-agent deals on sub-ms latency. A sub-10MB +neural student that holds ~2.85%-class PER gives us a latency tier to +quote next to theirs (and WO03's hybrid stacks on top of it). + +## Recipe + +1. **Student**: char-level encoder-decoder transformer, d=256, 4+4 + layers, ~10–15M params, BOS/EOS IPA-target vocab (small, ~500) — + NOT byte-level (byte seq2seq needs ByT5 capacity to phonotactically + decode; a tiny model needs the compact target vocab; CTC is ruled + out by the Thai CTC limitation verdict). +2. **Data**: teacher-generated pairs — umt5-thai-g2p-v2-0.5k teacher + (public HF, 4.43% PER lineage) or the existing tha-g2p-small + (2.85% greedy, local artifact) labeling Thai text (Kaikki train + side + hewiki-style Thai wiki text if needed). Sequence-level KD: + train student on teacher OUTPUTS (hard labels) — the recipe that + already worked for tha-g2p-small. +3. **Training**: HF Jobs `l4x1` (small model), bf16, cosine. +4. **Export**: int8 static ONNX; measure size + greedy PER with the + exact runtime harness (1,219 Kaikki test, corpus-level, true + Levenshtein, same numbers as RESULTS.md). + +## Deliverables + +- `train_thai_tiny_hf.py` + distill-data builder. +- Verdict in RESULTS.md either way (size table: 219MB student vs tiny). +- **If gate passes**: ship chain, owner-confirmed version; the hybrid + (WO03) gets a `--tiny` model option. + +## Honest risks + +10–15M params may fail phonotactic decoding entirely (mode collapse +lineage in this repo's history is real). Gate is cheap; a miss is a +clean negative that closes the sub-10MB idea rather than an open +question. diff --git a/TODO.sota/09-hebrew-audio-pseudolabel-probe.md b/TODO.sota/09-hebrew-audio-pseudolabel-probe.md new file mode 100644 index 0000000..10dcc6f --- /dev/null +++ b/TODO.sota/09-hebrew-audio-pseudolabel-probe.md @@ -0,0 +1,32 @@ +# WO09 — Hebrew audio pseudolabel POC (PARKED probe) + +**Priority:** P3 · **Status: PARKED — probe only.** No full campaign, +no GPU-scale labeling, without an explicit owner decision. + +## Why parked + +ReNikud's real moat is thousands of hours of Hebrew audio + a phoneme +ASR pseudo-labeling pipeline — a multi-week, multi-GPU program. The +2026-10-05 sweep verdict: park as the 2027 bet; probe viability only. + +## Probe (what "implement all" means here) + +1. `probe_hebrew_audio_pseudolabel.py`: + - Input: a small dir of Hebrew audio clips (any public sample set; + the script takes `--audio-dir` and does NOT fetch anything itself). + - Stage 1: universal phoneme CTC ASR + (`facebook/wav2vec2-lv-60-espeak-cv-ft`) → IPA phoneme sequence. + - Stage 2: forced alignment viability check — espeak-style reference + phonemization of our plane-restored nikud text vs the ASR phoneme + stream; measure token-level agreement on a small sample. + - Output: a viability report (agreement stats, per-hour labeling + cost estimate on HF flavors, failure modes). +2. A short probe report in RESULTS.md (or a TODO note if audio isn't + locally available), ending with the go/no-go recommendation for a + 2027 campaign. + +## Explicit non-goals + +No large-scale labeling; no trainer changes; no corpus construction; no +dependency on Dicta or any external API for labels (self-hosted models +only).