Skip to content

Commit e3bf218

Browse files
authored
Merge pull request #47 from interscript/docs/paper-ci
paper: bootstrap CI for the decode comparison
2 parents d153f67 + d8508cb commit e3bf218

5 files changed

Lines changed: 48 additions & 15 deletions

File tree

‎docs/RESULTS.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -85,10 +85,15 @@ better than the beam-4 harness numbers.** Re-measured on the shipped
8585
int4 zip through the Python runtime (the exact ONNX KV decode users
8686
get), true Levenshtein, full 1,219-sentence set:
8787

88-
| Decode | PER | Exact match |
89-
|---|---|---|
90-
| beam-4 (published, torch harness) | 12.06% | 87.94% |
91-
| **greedy (runtime protocol)** | **2.85%** | **88.93%** |
88+
| Decode | Teacher PER | Student PER | Student EM |
89+
|---|---|---|---|
90+
| beam-4 (published, torch harness) | 4.43% | 12.06% | 87.94% |
91+
| **greedy (runtime protocol)** | **1.25%** | **2.85%** | **88.93%** |
92+
93+
The teacher is also affected by the beam pathology (4.43 beam-4 → 1.25
94+
greedy, measured 2026-08-25 through the same harness at num_beams=1):
95+
same-protocol, the client tier's true shrink cost is **+1.60pp**, not
96+
the +7.63pp the beam-vs-beam comparison suggested.
9297

9398
The beam-4 numbers are inflated by length-normalized beam preferring
9499
long garbage on this model's flat per-token distributions (top-1

‎docs/paper-assets/frontier.png‎

1.36 KB
Loading

‎docs/paper.adoc‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,9 @@ short ones), while greedy decoding rides the argmax margins to a 4.2x
8383
lower error rate. Published evaluations that decode such models with
8484
beams risk reporting the decode, not the model.
8585

86-
image:paper-assets/architecture.png[The IMF v1 pipeline,width=580,align=center]
87-
.Caption: The phonological-layer pipeline. Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output.
86+
[[fig-architecture]]
87+
.The phonological-layer pipeline: frozen teachers, gated students, one checksummed artifact, three runtimes, byte-identical output.
88+
[image::paper-assets/architecture.png[The IMF v1 pipeline]] Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output.
8889

8990
== The artifact contract (IMF v1)
9091
[[section-imf]]
@@ -201,8 +202,9 @@ The campaign's central engineering question: how small can a client artifact be?
201202
|ByT5-base (server tier) |pretrained |580M |1.2 GiB fp32 |9.19
202203
|===
203204

204-
image:paper-assets/frontier.png[Client-tier size-quality frontier,width=580,align=center]
205-
.Caption: The client-tier frontier on Thai G2P (identical harness). Randomly initialized students collapse at every capacity tested; the pretrained rung (green) is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention — see <<section-decode>>.
205+
[[fig-frontier]]
206+
.The client-tier frontier on Thai G2P (identical harness): randomly initialized students collapse at every capacity tested; the pretrained rung is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention (<<section-decode>>).
207+
[image::paper-assets/frontier.png[Client-tier size-quality frontier]]
206208

207209
Three findings:
208210

@@ -231,8 +233,9 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on
231233

232234
Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right.
233235

234-
image:paper-assets/decode.png[Decode distributions and alpha sweep,width=620,align=center]
235-
.Caption: (a) Top-1 log-probabilities at successive positions of a representative greedy decode, against the uniform baseline ln(259); the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset; no normalization exponent recovers greedy quality.
236+
[[fig-decode]]
237+
.(a) Top-1 log-probabilities at successive positions of a representative greedy decode against the uniform baseline; the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset: no normalization exponent recovers greedy quality.
238+
[image::paper-assets/decode.png[Decode distributions and alpha sweep]]
236239

237240
=== The mechanism
238241

@@ -241,7 +244,7 @@ Distilled byte-level students have *flat* per-token distributions. On the measur
241244
Beam search ranks *completions* by accumulated likelihood. On flat distributions this is pathological in both directions:
242245

243246
* with length normalization (`score / len^α`), longer completions divide away their accumulated cost — a 30-token sequence at −0.40/token outranks the correct 15-token answer at −0.53/token, and generated text runs long;
244-
* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี");
247+
* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี"). Measured as a paired comparison on 300 sentences (10,000 bootstrap resamples): the raw-score beam implementation scores 72.6% PER at 0.7% exact match against greedy's 2.6% / 89.0% — a difference of 69.9pp, 95% CI [67.6, 72.1];
245248
* an α sweep (0, 0.3, 0.6, 1.0) produced PERs of 69.8, 70.1, 70.4, and 45.9 on a 150-sentence subset where greedy scores 3.20 — *no* normalization exponent recovers greedy quality.
246249

247250
A concrete example from the shipped artifact, input Thai for "hello":
@@ -255,7 +258,7 @@ A concrete example from the shipped artifact, input Thai for "hello":
255258
|beam-4, raw |`sa˨˩` (truncates)
256259
|===
257260

258-
The published beam-4 figures were not wrong measurements; they were measurements of the decode. Under a protocol users never execute, they overstated error 4.2x.
261+
The published beam-4 figures (the length-normalized harness convention: 12.06% PER) were not wrong measurements; they were measurements of the decode — and the two beam variants fail in opposite directions. Under a protocol users never execute, they overstated error 4.2x.
259262

260263
=== Implications
261264

@@ -291,6 +294,24 @@ The general lesson: when a file "reads corrupt remotely and clean locally," susp
291294

292295
The phonological layer ships: eleven models, four languages, two tiers, one artifact contract, zero untraceable numbers. The deterministic guarantees of map-based transliteration — same bytes everywhere, verifiable artifacts, public protocols — carry over to learned conversion. The two negative results (no sub-100M rung without pretraining; beam decode counterproductive on distilled byte students) are released with the same rigor as the positive ones, because in an infrastructure whose value is trust, a measured failure is a deliverable.
293296

297+
== Data and compute statement
298+
299+
*Datasets.* Thai: Kaikki-derived Thai IPA pairs (train on the expanded
300+
corpus with epitran augmentation; 1,219 held-out test sentences).
301+
Hebrew: hebrew-v4 gold corpus with a phonikud weak-pretraining stage;
302+
Nakdimon test split. Arabic: r5-units domain and replay corpora of
303+
fully-voweled hadith text; SadeedDiac-25 evaluation paragraphs.
304+
Persian: the v1 g2p corpus (release-frozen); SentenceBench for homograph
305+
accuracy. No dataset contains personal data; all are derived from
306+
published textual corpora under their respective licenses.
307+
308+
*Compute.* Students train on a single 24 GB A10G: 3-11 hours per
309+
student (Thai 4 h; Hebrew 3 h; Arabic 11 h including labeling; the
310+
Arabic client tier trained three times across the debugging saga).
311+
Teacher labeling is the dominant one-off cost per language (2-6 h).
312+
Exports, validation, and parity gates run on CPU. Total campaign GPU
313+
budget is on the order of 100 A10G-hours.
314+
294315
== Availability and provenance
295316

296317
* Artifact contract, index, validators, harnesses, results log: `github.com/interscript/interscript-ml` (BSD-3-Clause, code and weights).

‎models/metrics-sources.yaml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,13 +69,14 @@ heb-diac-small-1.0:
6969
- {row: "Student (ByT5-small, gate)", column: DER, as: der_student_greedy}
7070
tha-g2p-small-1.0:
7171
repo: interscript/interscript-ml
72-
ref: docs/greedy-correction
72+
ref: docs/paper-ci
7373
path: docs/RESULTS.md
7474
anchor: tha-g2p-small-10-thai-g2p-client-tier-2026-08-22
7575
protocol: "greedy decode via the Python runtime (the shipped ONNX KV path), corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences"
7676
tables:
77-
- {row: "beam-4 (published, torch harness)", column: PER, as: per_student_beam4}
78-
- {row: "greedy (runtime protocol)", column: PER, as: per_student}
77+
- {row: "greedy (runtime protocol)", column: Teacher PER, as: per_teacher_greedy}
78+
- {row: "beam-4 (published, torch harness)", column: Student PER, as: per_student_beam4}
79+
- {row: "greedy (runtime protocol)", column: Student PER, as: per_student}
7980
ara-diac-small-1.0:
8081
repo: interscript/interscript-ml
8182
ref: release/ara-diac-small-1.0

‎models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ trained_from: 'sequence-level KD from the B-K/umt5-thai-g2p-v2-0.5k teacher (4.4
1717
this is the smallest rung that does not collapse (docs/RESULTS.md frontier table,
1818
~300MB int8).'
1919
metrics:
20+
- name: per_teacher_greedy
21+
value: 1.25
22+
protocol: greedy decode (num_beams=1) through the same harness; 1,219 Kaikki Thai
23+
test sentences; exact match 95.16%; measured 2026-08-25 — the published beam-4
24+
teacher figure 4.43 was itself inflated by the decode pathology
25+
source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-small-1.0
2026
- name: per_student_beam4
2127
value: 12.06
2228
protocol: beam-4, corpus-level PER, same harness as the teacher (src/gpu/ modal_distill.py::evaluate_per,

0 commit comments

Comments
 (0)