Add TODO.sota campaign specs (9 WOs, HF substrate)

TODO.sota/00-README.md

@@ -0,0 +1,56 @@
+# TODO.sota — 2026 SOTA campaign (client-usable quality)
+
+Derived from the 2026-10-05 SOTA sweep (Arabic / Hebrew / Thai).
+Goal: close the measured gaps to the 2026 frontier on all three
+languages while keeping every artifact client-usable (int8, sha-verified,
+multi-runtime, versioned).
+
+## Substrate: HF (NOT Modal)
+
+All training and eval jobs run on **Hugging Face Jobs** (`hf` CLI, v1.26+):
+
+```bash
+hf jobs run --flavor a100-large -d \
+  -v hf://datasets/Interscript/hebrew-v4:/data:ro \
+  -v hf://buckets/Interscript/isx-training:/ckpt:rw \
+  python:3.12 python train.py
+```
+
+- Datasets: `hf://datasets/Interscript/<name>` (mounted read-only).
+- Checkpoints/artifacts: `hf://buckets/Interscript/isx-training` (read-write).
+- Detached always (`-d`); poll via `hf jobs ls` / `hf jobs logs`.
+- Modal is retired for compute. It may be used ONLY to export corpora
+  off Modal volumes during the one-time migration (#44).
+
+## Standing rules that shape this campaign
+
+- **NO LLM TEACHERS** (memory law, 2026-08): never use Claude/GPT/
+  Gemini/any LLM as a label source for haraqat/niqqud/Thai IPA.
+  WO06 is therefore noisy-student/self-ensemble — NOT the
+  "LLM-ensemble teacher" phrasing used in the sweep notes.
+- Versions/releases/cards/index are owner decisions at ship time;
+  specs define gates, not version numbers.
+- Data channel for shipped artifacts stays GitHub Releases + sha256;
+  HF dataset repos here are TRAINING inputs, not the shipping channel.
+- PRs only; no tags without explicit instruction.
+
+## Work orders
+
+| WO | File | Priority | Depends | Status |
+|----|------|----------|---------|--------|
+| 01 | 01-wikinews-multiref-reconciliation.md | P1 | — | pending |
+| 02 | 02-user-diacritics-preservation.md | P1 | — | pending |
+| 03 | 03-thai-hybrid-latency.md | P1 | — | pending |
+| 04 | 04-hebrew-plane-scaleup.md | P2 | #44 | pending |
+| 05 | 05-external-benchmarks.md | P2 | 01, 07 | pending |
+| 06 | 06-arabic-r8-noisy-student.md | P2 | #44 | pending |
+| 07 | 07-nikud-ipa-rule-layer.md | P3 | — | pending |
+| 08 | 08-thai-small-tier.md | P3 | #44 | pending |
+| 09 | 09-hebrew-audio-pseudolabel-probe.md | P3 (parked probe) | — | pending |
+
+## Schedule
+
+Wave 1 (local, days): WO01, WO02, WO03, WO07 + infra #44.
+Wave 2 (HF training, launched together): WO04 (a100-large), WO06
+(a100-large), WO08 (l4x1) — run concurrently, watch via `hf jobs ls`.
+Wave 3: WO05 tables after 01+07 land; WO09 probe only.

TODO.sota/01-wikinews-multiref-reconciliation.md

@@ -0,0 +1,41 @@
+# WO01 — WikiNews-2024 multiref reconciliation (Arabic)
+
+**Priority:** P1 · **Cost:** hours · **Gate:** a protocol-matched statement
+of where our models stand on WikiNews-2024, in RESULTS.md.
+
+## Problem
+
+Our OOD number on WikiNews-2024 is 17.38 WER / 11.83 DER while the EMNLP
+2025 QCRI paper (Mohamed & Mubarak, 2025.emnlp-main.846) reports
+**2.70% WER** with a BiLSTM on the same benchmark name. A 6× gap on the
+same named benchmark is almost certainly protocol divergence, not model
+divergence — but we cannot quote ANY WikiNews number until the delta is
+explained. Suspects:
+
+1. Different test split (our slice may be the harder OOD half).
+2. Different reference policy (multiref semantics: correct if the word
+   matches ANY reference; ref count per word may differ).
+3. Different WER definition (undiacritized word identity vs
+   fully-diacritic-exact match).
+4. Normalization (ta-marbuta/ha, alef variants, tatweel, punct).
+5. Different model class being scored (our number may come from a plane
+   or seq2seq student, theirs from a task-dedicated BiLSTM).
+
+## Deliverables
+
+1. Obtain the official WikiNews-2024 release (paper's repo/QCRI) —
+   test paragraphs + per-word references + (if published) their scorer.
+   Record provenance + sha256 of the files we use.
+2. Reconcile harnesses: run `interscript-train/eval_wikinews_multiref.py`
+   (ours) and their protocol on THE SAME predictions; diff every metric
+   definition line-by-line; write a delta table.
+3. Score, under the reconciled protocol: released ara-diac-2.0,
+   ara-diac-plane-1.0, ara-diac-plane-large-1.0 (+ r7 if weights are
+   reachable). Greedy, no conditioning, runtime protocol.
+4. RESULTS.md entry: one table, protocol column explicit, plus a
+   one-paragraph verdict (mismatch explained or real gap owned).
+
+## Out of scope
+
+Retraining; changing the shipped artifacts; quoting any WikiNews number
+externally before this WO lands.

TODO.sota/02-user-diacritics-preservation.md

@@ -0,0 +1,48 @@
+# WO02 — User-diacritics preservation in plane runtimes
+
+**Priority:** P1 · **Cost:** hours–days · **Gate:** `preserve` mode with
+byte-identical user diacritics, all tests green, no behavior change when
+off.
+
+## Why
+
+QCRI's EMNLP 2025 paper ships "a model that preserves user-specified
+diacritics" as a flagship client feature. For us it is nearly free: the
+plane's K-pass conditioning already takes a plane tensor as input, so a
+partially-vocalized input is just a partially-filled plane — pin user
+classes, mask only the empty positions.
+
+## Contract
+
+`PlaneModel.translate(text, preserve_diacritics=True)`:
+
+1. `split_planes(text)` on the INPUT (skeleton + user classes; Hebrew:
+   `nikud_planes`, Arabic: `haraqat_planes`).
+2. Initial plane tensor = user class ids where non-empty, MASK_ID where
+   empty. K-pass decode runs as usual.
+3. Positions the user supplied are NEVER overwritten by the model —
+   final classes = user classes at those positions, model predictions
+   elsewhere.
+4. Byte-exact guarantee: every diacritic the user typed round-trips
+   verbatim (cluster order preserved as written — the Hebrew canon
+   lesson applies to user input too).
+5. Default `preserve_diacritics=False` — behavior and parity goldens
+   UNCHANGED.
+
+## Deliverables
+
+- `interscript-py`: `PlaneModel.translate(..., preserve_diacritics=...)`
+  (TDD: fully-labeled input → identity output; partial input → user
+  classes intact, empty positions filled; unlabeled → identical to
+  current path).
+- `interscript-ruby-ml`: same flag on `Interscript::ML::PlaneModel#translate`
+  (same test matrix against the tiny-plane fixture).
+- ts port: blocked on the documented ts plane loader gap — note it, do
+  not unblock here.
+- RESULTS.md / model card note: preservation mode + the byte-exact
+  guarantee.
+
+## Explicit non-goals
+
+No seq2seq (IMF) preservation mode — that decode cannot pin positions
+without a new contract; out of scope.

TODO.sota/03-thai-hybrid-latency.md

@@ -0,0 +1,43 @@
+# WO03 — Thai hybrid fast-path + latency table
+
+**Priority:** P1 · **Cost:** days · **Gate:** published latency × accuracy
+table; hybrid within 1.2× of dict-only latency and within +0.3pp PER of
+model-only on our benchmark.
+
+## Why
+
+FastThaiG2P (arXiv 2608.12814) owns the latency narrative (0.15 ms/utt)
+but is 26.9% PER on its own set — dictionary+rules. We own quality
+(2.85% PER, greedy, runtime protocol) at neural latency. The client
+answer is not a race but a hybrid tier: dictionary fast-path for
+in-vocab words, our neural model as the OOV fallback (their own profile:
+58% of latency is fallbacks — that fallback is us).
+
+## Design
+
+1. **Lexicon**: word→IPA from Kaikki Thai (the same lineage as training
+   data). Multiple readings per word → keep the most frequent; store
+   count provenance. Build script emits `tha-lexicon.jsonl`
+   (word, ipa, count) — a data artifact, versioned with the model.
+2. **Fast-path**: Thai has no spaces → longest-prefix (maximal-munch)
+   match over the lexicon; unmatched spans go to the neural model
+   (windowed 1400-byte protocol, unchanged); spliced back.
+3. **Bench** on `thai-kaikki-g2p/test.jsonl` (1,219 sentences, CPU):
+   - dict-only PER + latency/utt
+   - model-only PER + latency/utt
+   - hybrid PER + latency/utt (+ OOV-fallback rate)
+4. **RESULTS.md**: the table, protocol column explicit, one-paragraph
+   positioning vs FastThaiG2P (never quote their PER against ours —
+   different set; quote ours and cite theirs as self-reported).
+
+## Deliverables
+
+- `interscript-py`: hybrid translate path + lexicon builder script.
+- Artifact: lexicon released alongside the model (GitHub Releases +
+  sha256, per data-channel law).
+- Bench JSON + RESULTS.md table.
+
+## Non-goals
+
+PyThaiNLP dependency (our maximal-munch over Kaikki covers it);
+retraining anything; ts/ruby ports of the hybrid (py first, note ports).

TODO.sota/04-hebrew-plane-scaleup.md

@@ -0,0 +1,39 @@
+# WO04 — Hebrew plane scale-up (HF training)
+
+**Priority:** P2 · **Cost:** ~10–15 h on `a100-large` · **Gate:**
+**≤ 10.0 DER** on the nakdimon test (exact run-021 protocol: v4 test,
+windowed 1400 bytes, original-separator stitching, K-pass greedy,
+seq2seq_der). Shipped baseline: heb-diac-plane-1.0 at 12.48.
+
+## Recipe (run-022)
+
+- Encoder: **byt5-base** (up from small). Plane embedding + per-position
+  head unchanged (the ReNikud-validated factorization).
+- Corpus: hebrew-v4 (rebuilt locally from
+  nakdimon+sefaria+distilled-v1/v2+expanded-v2 per
+  `train_hebrew_v4.py`), uploaded to
+  `hf://datasets/Interscript/hebrew-v4`, mounted ro.
+- Epochs 4 (up from 2), warmup 500, cosine, bs 16 (grad-accum if
+  memory-bound on base), lr 1e-4, bf16 autocast, K_PASSES=3 at eval.
+- Checkpoints → `hf://buckets/Interscript/isx-training` every 500 steps
+  + `labels.sha` resume contract + per-marker eval progress file
+  (run-019 lesson) + original-separator stitching (run-021 lesson,
+  already correct in the trainer).
+
+## Deliverables
+
+1. `train_hebrew_plane_hf.py` — the HF Jobs port of run-021's trainer
+   (paths: /data, /ckpt; volumes; no Modal).
+2. Launch on `a100-large -d`; monitor via `hf jobs logs`.
+3. Export plane.onnx int8 from the best checkpoint; package plane zip
+   (metadata kind=plane + sha256s + classes.json); golden rows vs the
+   py runtime.
+4. **If gate passes**: full ship chain — release (version = owner
+   decision, ASK before publishing), index entry, card, HF mirror.
+   If it fails: verdict in RESULTS.md, keep 1.0 shipped.
+
+## Honest risks
+
+byt5-base on 50K units may overfit by epoch 3 — the gate decides, not
+the hope. 12.48 → 10 is ambitious; a plateau at ~11 is a documented
+negative, not a failure of the campaign.

TODO.sota/05-external-benchmarks.md

@@ -0,0 +1,35 @@
+# WO05 — External benchmark adoption
+
+**Priority:** P2 · **Depends:** WO01 (multiref protocol), WO07 (nikud→IPA
+for g2p comparability) · **Gate:** RESULTS.md tables with explicit
+protocol columns for at least: heb plane on heb-g2p-benchmark, Arabic
+models on reconciled WikiNews-2024.
+
+## Targets
+
+1. **heb-g2p-benchmark** (Phonikud line; ReNikud reports 85.1% word
+   accuracy on it): adopt their test set. Our plane emits nikud; with
+   WO07's nikud→IPA rules we can report both (a) nikud-restoration
+   DER/WER on their vocalized test text and (b) composed IPA word
+   accuracy — each labeled with its protocol. If their set is
+   nikud-free (plain Hebrew → IPA), report (b) only.
+2. **WikiNews-2024 multiref** (from WO01): becomes the OFFICIAL Arabic
+   OOD protocol alongside SadeedDiac-25; every future Arabic RESULTS.md
+   entry carries both.
+3. **MILIM** (ReNikud's spoken benchmark): track its release; evaluate
+   only when data is actually obtainable — do not fabricate numbers
+   (memory law).
+
+## Deliverables
+
+- Eval scripts under `interscript-models/benchmarks/` (reproducible,
+  pinned inputs + sha256s).
+- RESULTS.md: new "External benchmarks" section; every number carries
+  protocol + input provenance; missing data (MILIM) listed as "awaiting
+  release", never estimated.
+
+## Non-goals
+
+Protocol shopping: we adopt THEIR definitions where feasible; where a
+definition is ambiguous we publish BOTH numbers labeled, never pick the
+flattering one silently.

TODO.sota/06-arabic-r8-noisy-student.md

@@ -0,0 +1,44 @@
+# WO06 — Arabic r8: noisy-student / self-ensemble (HF training)
+
+**Priority:** P2 · **Cost:** teacher labeling (~hours on GPU) + training
+(~6–8 h `a100-large`) · **Gate:** **beat 2.2864 DER** (r7/ara-diac-2.0)
+on SadeedDiac-25 under the exact zero-skip haraqat-projected protocol;
+stretch ≤ 2.0.
+
+## PIVOT (standing rule)
+
+The sweep notes floated an "LLM-ensemble teacher". **Forbidden** —
+memory law: NEVER use Claude/GPT/Gemini/any LLM as a diacritization
+label source (hallucinated haraqat bake into the student). r8 uses only
+signals we control:
+
+1. **Self-training (noisy student):** our best dedicated model
+   pseudo-labels UNLABELED Arabic text (`data/arwiki`, 115 MB local);
+   keep predictions where the model is confident (per-position class
+   margin above threshold; span-keep, not char-spaghetti).
+2. **Mixture:** gold tashkeela-full + pseudo-arwiki (cap pseudo at ~50%
+   of gold tokens; ablate the ratio only if the first run misses the
+   gate).
+3. **Self-ensemble (optional second arm):** per-position vote across
+   OUR OWN checkpoints (r6+r7 lineages + plane-large) as pseudo-labels —
+   model diversity without external teachers.
+4. Excluded: yallamorph-aux (run-009 measured NEGATIVE — do not re-add).
+
+## Deliverables
+
+1. `pseudo_label_arabic_hf.py` — teacher inference + margin filter
+   (ONNX int8 teacher on GPU is fine; greedy; emit arwiki-pseudo.jsonl
+   with kept-token stats).
+2. `train_arabic_r8_hf.py` — the proven dedicated seq2seq lineage
+   (byt5-small, IMF contract), mixture loader, same eval protocol as
+   run-019's corrected harness (per-marker progress, exact separators).
+3. HF Jobs launch + monitoring; export/package per release system.
+4. **If gate passes:** ship chain with owner-confirmed version; else
+   RESULTS.md verdict with kept-token stats and margin-threshold
+   sensitivity.
+
+## Honesty anchors
+
+Single-seed caveat (2608.27729): a gain under ~0.15 DER is within seed
+noise — mark it as such rather than claiming SOTA movement. Keep-token
+rate is a first-class reported number.

TODO.sota/07-nikud-ipa-rule-layer.md

@@ -0,0 +1,44 @@
+# WO07 — nikud→IPA rule layer
+
+**Priority:** P3 · **Cost:** days · **Gate:** deterministic, documented,
+tested rule table over the v4 corpus cluster inventory; composes with
+the plane runtime (`heb plane → nikud text → IPA`).
+
+## Why
+
+ReNikud's target surface is IPA for TTS (lexical stress, spoken norms).
+We are not ReNikud — but a deterministic nikud→IPA layer gives TTS
+clients a working Hebrew phonemizer TODAY, on top of our shipped plane
+model, and unlocks WO05's comparability with heb-g2p-benchmark.
+
+## Design
+
+1. **Rules over the actual corpus inventory** (not theoretical Hebrew):
+   enumerate the nikud clusters that occur in hebrew-v4 (the 135-class
+   inventory); write letter×cluster→IPA per standard reading.
+   - Special cases handled explicitly: שׁ/שׂ dots; dagesh lene in
+     בג"ד כפ"ת; dagesh chazaq (doubling) conservative rule; sheva
+     (initial → ə; otherwise ə/e per position heuristics, documented as
+     heuristic); matres lectionis (vav/yod with holam/hiriq/male);
+     final gutturals; qamats qatan (documented as lexicon-resolved).
+2. **Lexicon overrides**: a table for the highest-frequency exceptions
+   (qamats qatan words, proper names); rules lose to lexicon.
+3. **Composition API**: `nikud_to_ipa(text)` in
+   `interscript-py` + a `PlaneModel.translate_to_ipa()` convenience
+   that chains plane restoration → rules. Stateless, deterministic,
+   no ONNX involved.
+4. **Tests**: cluster-inventory coverage table (every class → explicit
+   rule or explicit lexicon fallback — no silent falls-through);
+   round-trip spot checks against our labeled corpus pairs; golden
+   examples for each special case.
+
+## Deliverables
+
+- `nikud_ipa.py` + rule doc (the table itself is the doc).
+- Tests (TDD, real corpus strings, no doubles).
+- Documented limitation: no lexical stress, no spoken-norm variation —
+  that is ReNikud's audio-supervised territory; stated in the card/README.
+
+## Non-goals
+
+Learning anything (rules only); stress assignment; teamim.