Skip to content

Commit f558125

Browse files
ronaldtseRonald Tse
andauthored
docs: WO01 WikiNews-2024 reconciliation — EvalDiac fidelity, plane OOD numbers, register-not-vocab verdict (#257)
Co-authored-by: Ronald Tse <ronald.tse@gmail.com>
1 parent e310b52 commit f558125

1 file changed

Lines changed: 33 additions & 0 deletions

File tree

‎docs/RESULTS.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1111,3 +1111,36 @@ reconstruction mismatches. Same lesson as run-019's E2: every verdict
11111111
number earns a mechanism check before it enters the ledger. nikud
11121112
cluster order is preserved as written (the corpus has no uniform canon:
11131113
בְּ is sheva-then-dagesh; שָׁ is dot-then-qamats).
1114+
1115+
## WO01 WikiNews-2024 multiref reconciliation — harness verified against EvalDiac.java; the QCRI gap is register, not vocabulary (2026-10-07)
1116+
1117+
Fidelity (WO01, TODO.sota): our multiref scorer is a line-by-line port of
1118+
QCRI's `Evaluation/EvalDiac.java`
1119+
(Abubakr17/advancing-arabic-diacritization). Normalization tables
1120+
(removeDefaultDiac), cluster codes, empty-ref acceptance, shadda-only
1121+
acceptance, and ANY-alternate word matching all match. Two deltas found
1122+
and closed/recorded: (1) the cross-letter FATHATAN swap was missing from
1123+
the port — added in interscript-train#107; (2) we keep our own
1124+
letter-accounting for failed multi-ref words (once, max-scoring
1125+
alternate) instead of Java's once-per-attempted-alternate — WER is
1126+
identical either way, DER differs only in the third decimal for
1127+
multi-ref-heavy sets. Input provenance: our bench copy is byte-identical
1128+
to the official release
1129+
(`sha256 f750fa11ad7c8b775dff7a5054e24b36bf6587881ffa02c2919cf7321be87587`).
1130+
1131+
| model (protocol: WikiNews-2024 multiref, greedy, zero-skip) | WER | DER |
1132+
|---|---|---|
1133+
| r7 / ara-diac-2.0 (dedicated teacher) | **17.38** | 11.83 |
1134+
| ara-diac-plane-1.0 (int8, on-device) — NEW measurement | 18.69 | **11.35** |
1135+
| QCRI BiLSTM (EMNLP 2025, self-reported, in-domain training) | 2.70 | — |
1136+
1137+
Reconciliation verdict: the 2.70-vs-17.38 gap is NOT vocabulary
1138+
coverage — WikiNews-2024 token OOV against our training vocabulary
1139+
(tashkeela-full + arwiki, 982,798 types) is only **1.61%**. The model
1140+
has seen these word forms; it fails to select the news-register READING
1141+
(contextual sense, named-entity conventions, gold-set diacritization
1142+
style). This refines the paper's "classical hadith vs news text" domain
1143+
framing: the gap is register/reading-selection, deeper than OOV, and
1144+
closes only with in-register gold supervision — which is why the
1145+
teacher-labeled news mix in r7 moved in-domain DER but not this number.
1146+
Never quote QCRI's 2.70 against our numbers without this paragraph.

0 commit comments

Comments
 (0)