Skip to content

Commit 02cd8d4

Browse files
author
Ronald Tse
committed
docs(paper): glm-4.7-flash leaderboard row — 13.2256 zero-skip; axis complete
Fourth row of the GLM family regression on SadeedDiac-25; the only family member regressing on both axes (missing 6.67%, wrong 9.01%); bootstrap vs GLM-5.2 +7.945pp CI [+7.516, +8.370]. Full record: rababa#75.
1 parent aa07078 commit 02cd8d4

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

‎docs/paper.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,14 +208,15 @@ External claims are made only where a public benchmark exists and the full proto
208208
|GLM-5.2 (reproduced, zero-skip) |— |2.6911
209209
|GLM-5.3-Flash (reproduced, zero-skip, reasoning_effort=low) |— |8.7978
210210
|GLM-5.3 (reproduced, zero-skip, reasoning_effort=low) |— |9.8971
211+
|glm-4.7-flash (reproduced, zero-skip, thinking-disabled) |— |13.2256
211212
|Gemini Flash 2.0 (published) |— |3.1926
212213
|GPT-4 (published) |— |3.8645
213214
|Sadeed (published) |1.5B |7.2915
214215
|**our client student 2.0 (ByT5-small)** |300M |**4.822**
215216
|our client student 1.0 (ByT5-small) |300M |8.259
216217
|===
217218

218-
A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp.
219+
A protocol note attaches to the newest generalist: GLM-5.3-Flash's API rejects disabled thinking outright (HTTP 400, code 1210 — valid reasoning_effort values are exactly low/high/max), so its row carries `reasoning_effort=low`, the nearest expressible analog of the plain-completion protocol the other LLM rows used. It scores 8.7978 zero-skip / 8.5721 raw — ~3.4x the DER of its predecessor GLM-5.2 and behind Sadeed-1.5B. Attribution over the same 1,200 paragraphs (convention-normalized, with our teacher and GLM-5.2 as <=0.01pp controls) locates the regression in wrong haraqat: 10.05% of positions vs GLM-5.2's 2.64% — which matches our dedicated teacher's 2.62% to within 0.02pp — while missing marks run 1.01%, extra 0.20%, and the entire dagger-alif (U+0670) convention effect is 0.125pp. The frontier's newest generalist lost core classical-Arabic mark knowledge its predecessor had, while our 580M teacher improved; we record it rather than average it away. The regression is family-wide and axis-split: the full GLM-5.3 scores 9.8971 with the loss concentrated in *missing* marks (3.75% of positions vs Flash's 1.01%, whose loss is *wrong* marks at 10.05%) — both decisively behind GLM-5.2 by paired bootstrap, whose wrong-mark rate matches our dedicated teacher's to within 0.02pp. The axis completes with glm-4.7-flash, the last plain-completion GLM row (thinking-disabled still accepted on 4.x): 13.2256 zero-skip, the worst frontier DER, and the only family member regressing on both axes at once (missing 6.67% — worst of the family — plus wrong 9.01%); +7.945pp behind GLM-5.2 by paired bootstrap (CI [+7.516, +8.370]). Every measurable GLM successor lost classical-Arabic mark knowledge its predecessor had.
219220

220221
The teacher is the best dedicated (task-trained, runnable-locally) model measured under this protocol — second only to a frontier proprietary LLM's published number, ahead of a clean GLM-5.2 reproduction, Gemini Flash, and GPT-4 — and it does so at 580M parameters against Sadeed's 1.5B. The current rung (r7) adds a teacher-labeled news-domain mix to the morphological-auxiliary teacher (r6) and improves both surfaces at once: in-domain 2.5793 → 2.2864, and out-of-domain WikiNews-2024 multi-reference 19.82/12.46 → 17.38/11.83 (WER/DER) — the paragraph-specialization trade-off of the earlier lineage is gone.
221222

0 commit comments

Comments
 (0)