diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index 23101e7..7247f4e 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -63,6 +63,8 @@ hf jobs run --flavor a100-large -d \ | 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound | | 24 | 24-verdict-wave.md | P1 | 17 | r8a/b/c + run-026 + thai CLOSED; r8d (knee) RUNNING | | 25 | 25-ar-news-specialist.md | P1 | 17 | RUNNING (run-029, register-pure full scale) | +| 27 | 27-ar-large-id-arm.md | P1 | — | RUNNING (run-031, byt5-large dedicated, r5 corpus) | +| 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic | | 26 | 26-heb-noisy-student.md | P1 | 04 | stage 1 RUNNING (labeling); stage 2 staged | ## Schedule diff --git a/TODO.sota/28-oracle-complementarity.md b/TODO.sota/28-oracle-complementarity.md new file mode 100644 index 0000000..661b3f8 --- /dev/null +++ b/TODO.sota/28-oracle-complementarity.md @@ -0,0 +1,21 @@ +# WO28 — cross-system complementarity: oracle ceiling real, naive voting catastrophic + +Measured on SadeedDiac-25 word-exact over 52,906 words (r7, r8b, +plane-large stitched predictions): + +| selection | exact-word | +|---|---| +| best solo (r8b) | 93.36% | +| **oracle any-of-3** | **96.78%** | +| naive 2-of-3 word vote | (word-level 93.81%) → **DER 28.77** | + +Findings: (1) +3.42 points of words are covered by at least one system +— the complementarity ceiling is REAL and large. (2) Naive voting is +CATASTROPHIC (DER 28.77, 74.6% under-diacritized): the systems' +diacritization CONVENTIONS conflict at sentence level — mixing words +from different conventions produces incoherent output under DER even +when each word is individually "exact". (3) Harvesting the ceiling +requires a LEARNED, convention-aware selection model (train a router +on the agreement features against Sadeed gold) — candidate future WO, +not free. This empirically closes generalist mixing in OUTPUT space +exactly as WO17 closed it in TRAINING space. diff --git a/docs/RESULTS.md b/docs/RESULTS.md index 2cbe701..edbb1b2 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -1318,3 +1318,19 @@ materially. **Generalist register blending is closed** — the doctrine is register SPECIALISTS: ara-diac-2.0 (ID crown) + ara-diac-news-1.0 candidate (run-029, register-pure full-scale, RUNNING) + plane transfer (run-028, RUNNING). Hebrew noisy student (WO26) also running. + +## WO27/28 — the large dedicated arm launched; cross-system oracle measured, naive voting closed (2026-10-08) + +**run-031 (byt5-large dedicated, r7 lineage corpus, no silver)** — the +heb-large negative does not transfer: Arabic's 400K+ unit corpus is a +different regime. Gate: ID < 2.2864. + +**WO28**: complementarity across our three families measured on word- +exact (52,906 words): r7 93.22 / r8b 93.36 / plane-large 91.81 solo; +**oracle any-of-3 = 96.78%**. The ceiling is real — but naive 2-of-3 +word voting is CATASTROPHIC (DER 28.77; 74.6% under-diacritized): the +families' conventions conflict at sentence level, so mixing words from +different conventions is incoherent under DER. Harvesting the ceiling +needs a learned convention-aware router — staged as a candidate WO. +Generalist mixing is now closed in BOTH training space (WO17) and +output space (WO28); the specialist doctrine is total.