Skip to content

Commit ac9f300

Browse files
authored
Merge pull request #49 from interscript/docs/paper-final
paper: teacher at greedy — same-protocol shrink +1.60pp
2 parents 026136a + 168d701 commit ac9f300

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

‎docs/paper.adoc‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ All fp32 artifacts measured 0.0pp parity (torch reference vs ONNX, ≥500 sample
180180

181181
=== Capacity costs are real and are disclosed
182182

183-
The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization but larger on G2P (Thai +7.63pp beam-4 / teacher-relative 2.85% greedy). Users choosing a tier choose a measured trade, not a mystery box.
183+
The Hebrew student met its pre-registered budget (+5.58pp against a ≈5.6pp allowance, the widened budget reflecting the teacher's own headroom). The Arabic client student missed its strict +0.5pp budget by +2.34pp and ships with that number in its metadata, its model card, and the results log. The measured pattern across languages: ByT5-small → ByT5-base shrink costs are single-digit percentage points on diacritization, and — measured same-protocol at greedy — +1.60pp on Thai G2P. Users choosing a tier choose a measured trade, not a mystery box.
184184

185185
=== The Persian teacher is competitive with published state of the art
186186

@@ -233,6 +233,14 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on
233233

234234
Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right.
235235

236+
The *teacher* is also affected, though less severely: measured at
237+
greedy on the same harness and split, the umt5 teacher scores 1.25% PER
238+
(EM 95.16%) against its published beam-4 4.43%. Same-protocol, the
239+
client tier's true shrink cost is +1.60pp — not the +7.63pp the
240+
beam-vs-beam comparison suggested. Both numbers in that comparison were
241+
decode artifacts; the distillation itself was substantially better than
242+
either published figure implied.
243+
236244
[[fig-decode]]
237245
.(a) Top-1 log-probabilities at successive positions of a representative greedy decode against the uniform baseline; the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset: no normalization exponent recovers greedy quality.
238246
[image::paper-assets/decode.png[Decode distributions and alpha sweep]]

0 commit comments

Comments
 (0)