@@ -1111,3 +1111,36 @@ reconstruction mismatches. Same lesson as run-019's E2: every verdict
11111111number earns a mechanism check before it enters the ledger. nikud
11121112cluster order is preserved as written (the corpus has no uniform canon:
11131113בְּ is sheva-then-dagesh; שָׁ is dot-then-qamats).
1114+
1115+ ## WO01 WikiNews-2024 multiref reconciliation — harness verified against EvalDiac.java; the QCRI gap is register, not vocabulary (2026-10-07)
1116+
1117+ Fidelity (WO01, TODO.sota): our multiref scorer is a line-by-line port of
1118+ QCRI's ` Evaluation/EvalDiac.java `
1119+ (Abubakr17/advancing-arabic-diacritization). Normalization tables
1120+ (removeDefaultDiac), cluster codes, empty-ref acceptance, shadda-only
1121+ acceptance, and ANY-alternate word matching all match. Two deltas found
1122+ and closed/recorded: (1) the cross-letter FATHATAN swap was missing from
1123+ the port — added in interscript-train #107 ; (2) we keep our own
1124+ letter-accounting for failed multi-ref words (once, max-scoring
1125+ alternate) instead of Java's once-per-attempted-alternate — WER is
1126+ identical either way, DER differs only in the third decimal for
1127+ multi-ref-heavy sets. Input provenance: our bench copy is byte-identical
1128+ to the official release
1129+ (` sha256 f750fa11ad7c8b775dff7a5054e24b36bf6587881ffa02c2919cf7321be87587 ` ).
1130+
1131+ | model (protocol: WikiNews-2024 multiref, greedy, zero-skip) | WER | DER |
1132+ | ---| ---| ---|
1133+ | r7 / ara-diac-2.0 (dedicated teacher) | ** 17.38** | 11.83 |
1134+ | ara-diac-plane-1.0 (int8, on-device) — NEW measurement | 18.69 | ** 11.35** |
1135+ | QCRI BiLSTM (EMNLP 2025, self-reported, in-domain training) | 2.70 | — |
1136+
1137+ Reconciliation verdict: the 2.70-vs-17.38 gap is NOT vocabulary
1138+ coverage — WikiNews-2024 token OOV against our training vocabulary
1139+ (tashkeela-full + arwiki, 982,798 types) is only ** 1.61%** . The model
1140+ has seen these word forms; it fails to select the news-register READING
1141+ (contextual sense, named-entity conventions, gold-set diacritization
1142+ style). This refines the paper's "classical hadith vs news text" domain
1143+ framing: the gap is register/reading-selection, deeper than OOV, and
1144+ closes only with in-register gold supervision — which is why the
1145+ teacher-labeled news mix in r7 moved in-domain DER but not this number.
1146+ Never quote QCRI's 2.70 against our numbers without this paragraph.
0 commit comments