Skip to content

docs: WO27 launch + WO28 oracle/vote findings #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 8, 2026
Merged

docs: WO27 launch + WO28 oracle/vote findings #283

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions TODO.sota/00-README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ hf jobs run --flavor a100-large -d \
| 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound |
| 24 | 24-verdict-wave.md | P1 | 17 | r8a/b/c + run-026 + thai CLOSED; r8d (knee) RUNNING |
| 25 | 25-ar-news-specialist.md | P1 | 17 | RUNNING (run-029, register-pure full scale) |
| 27 | 27-ar-large-id-arm.md | P1 | — | RUNNING (run-031, byt5-large dedicated, r5 corpus) |
| 28 | 28-oracle-complementarity.md | P2 | — | CLOSED — ceiling 96.78 real; naive voting catastrophic |
| 26 | 26-heb-noisy-student.md | P1 | 04 | stage 1 RUNNING (labeling); stage 2 staged |

## Schedule
Expand Down
21 changes: 21 additions & 0 deletions TODO.sota/28-oracle-complementarity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# WO28 — cross-system complementarity: oracle ceiling real, naive voting catastrophic

Measured on SadeedDiac-25 word-exact over 52,906 words (r7, r8b,
plane-large stitched predictions):

| selection | exact-word |
|---|---|
| best solo (r8b) | 93.36% |
| **oracle any-of-3** | **96.78%** |
| naive 2-of-3 word vote | (word-level 93.81%) → **DER 28.77** |

Findings: (1) +3.42 points of words are covered by at least one system
— the complementarity ceiling is REAL and large. (2) Naive voting is
CATASTROPHIC (DER 28.77, 74.6% under-diacritized): the systems'
diacritization CONVENTIONS conflict at sentence level — mixing words
from different conventions produces incoherent output under DER even
when each word is individually "exact". (3) Harvesting the ceiling
requires a LEARNED, convention-aware selection model (train a router
on the agreement features against Sadeed gold) — candidate future WO,
not free. This empirically closes generalist mixing in OUTPUT space
exactly as WO17 closed it in TRAINING space.
16 changes: 16 additions & 0 deletions docs/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1318,3 +1318,19 @@ materially. **Generalist register blending is closed** — the doctrine
is register SPECIALISTS: ara-diac-2.0 (ID crown) + ara-diac-news-1.0
candidate (run-029, register-pure full-scale, RUNNING) + plane
transfer (run-028, RUNNING). Hebrew noisy student (WO26) also running.

## WO27/28 — the large dedicated arm launched; cross-system oracle measured, naive voting closed (2026-10-08)

**run-031 (byt5-large dedicated, r7 lineage corpus, no silver)** — the
heb-large negative does not transfer: Arabic's 400K+ unit corpus is a
different regime. Gate: ID < 2.2864.

**WO28**: complementarity across our three families measured on word-
exact (52,906 words): r7 93.22 / r8b 93.36 / plane-large 91.81 solo;
**oracle any-of-3 = 96.78%**. The ceiling is real — but naive 2-of-3
word voting is CATASTROPHIC (DER 28.77; 74.6% under-diacritized): the
families' conventions conflict at sentence level, so mixing words from
different conventions is incoherent under DER. Harvesting the ceiling
needs a learned convention-aware router — staged as a candidate WO.
Generalist mixing is now closed in BOTH training space (WO17) and
output space (WO28); the specialist doctrine is total.
Loading