You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e3bf218
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/paper.adoc
+29-8Lines changed: 29 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,8 +83,9 @@ short ones), while greedy decoding rides the argmax margins to a 4.2x
83
83
lower error rate. Published evaluations that decode such models with
84
84
beams risk reporting the decode, not the model.
85
85
86
-
image:paper-assets/architecture.png[The IMF v1 pipeline,width=580,align=center]
87
-
.Caption: The phonological-layer pipeline. Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output.
86
+
[[fig-architecture]]
87
+
.The phonological-layer pipeline: frozen teachers, gated students, one checksummed artifact, three runtimes, byte-identical output.
88
+
[image::paper-assets/architecture.png[The IMF v1 pipeline]] Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output.
88
89
89
90
== The artifact contract (IMF v1)
90
91
[[section-imf]]
@@ -201,8 +202,9 @@ The campaign's central engineering question: how small can a client artifact be?
.Caption: The client-tier frontier on Thai G2P (identical harness). Randomly initialized students collapse at every capacity tested; the pretrained rung (green) is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention — see <<section-decode>>.
205
+
[[fig-frontier]]
206
+
.The client-tier frontier on Thai G2P (identical harness): randomly initialized students collapse at every capacity tested; the pretrained rung is the whole quality cliff. The hollow point marks the same ByT5-small student scored under the beam-4 evaluation convention (<<section-decode>>).
@@ -231,8 +233,9 @@ Porting beam search into the runtimes forced an apples-to-apples measurement on
231
233
232
234
Exact match barely moves; edit distance collapses by 4.2x. The signature is diagnostic: beam decoding is mangling precisely the sentences the model did not get exactly right.
233
235
234
-
image:paper-assets/decode.png[Decode distributions and alpha sweep,width=620,align=center]
235
-
.Caption: (a) Top-1 log-probabilities at successive positions of a representative greedy decode, against the uniform baseline ln(259); the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset; no normalization exponent recovers greedy quality.
236
+
[[fig-decode]]
237
+
.(a) Top-1 log-probabilities at successive positions of a representative greedy decode against the uniform baseline; the argmax is barely distinguished from the field. (b) PER by decode on a 150-sentence subset: no normalization exponent recovers greedy quality.
238
+
[image::paper-assets/decode.png[Decode distributions and alpha sweep]]
236
239
237
240
=== The mechanism
238
241
@@ -241,7 +244,7 @@ Distilled byte-level students have *flat* per-token distributions. On the measur
241
244
Beam search ranks *completions* by accumulated likelihood. On flat distributions this is pathological in both directions:
242
245
243
246
* with length normalization (`score / len^α`), longer completions divide away their accumulated cost — a 30-token sequence at −0.40/token outranks the correct 15-token answer at −0.53/token, and generated text runs long;
244
-
* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี");
247
+
* with raw cumulative scores, short completions win — the maximum-likelihood complete hypothesis on this model genuinely is a truncation (`sa˨˩` for "สวัสดี"). Measured as a paired comparison on 300 sentences (10,000 bootstrap resamples): the raw-score beam implementation scores 72.6% PER at 0.7% exact match against greedy's 2.6% / 89.0% — a difference of 69.9pp, 95% CI [67.6, 72.1];
245
248
* an α sweep (0, 0.3, 0.6, 1.0) produced PERs of 69.8, 70.1, 70.4, and 45.9 on a 150-sentence subset where greedy scores 3.20 — *no* normalization exponent recovers greedy quality.
246
249
247
250
A concrete example from the shipped artifact, input Thai for "hello":
@@ -255,7 +258,7 @@ A concrete example from the shipped artifact, input Thai for "hello":
255
258
|beam-4, raw |`sa˨˩` (truncates)
256
259
|===
257
260
258
-
The published beam-4 figures were not wrong measurements; they were measurements of the decode. Under a protocol users never execute, they overstated error 4.2x.
261
+
The published beam-4 figures (the length-normalized harness convention: 12.06% PER) were not wrong measurements; they were measurements of the decode — and the two beam variants fail in opposite directions. Under a protocol users never execute, they overstated error 4.2x.
259
262
260
263
=== Implications
261
264
@@ -291,6 +294,24 @@ The general lesson: when a file "reads corrupt remotely and clean locally," susp
291
294
292
295
The phonological layer ships: eleven models, four languages, two tiers, one artifact contract, zero untraceable numbers. The deterministic guarantees of map-based transliteration — same bytes everywhere, verifiable artifacts, public protocols — carry over to learned conversion. The two negative results (no sub-100M rung without pretraining; beam decode counterproductive on distilled byte students) are released with the same rigor as the positive ones, because in an infrastructure whose value is trust, a measured failure is a deliverable.
293
296
297
+
== Data and compute statement
298
+
299
+
*Datasets.* Thai: Kaikki-derived Thai IPA pairs (train on the expanded
300
+
corpus with epitran augmentation; 1,219 held-out test sentences).
301
+
Hebrew: hebrew-v4 gold corpus with a phonikud weak-pretraining stage;
302
+
Nakdimon test split. Arabic: r5-units domain and replay corpora of
protocol: "greedy decode via the Python runtime (the shipped ONNX KV path), corpus-level PER, true Levenshtein; 1,219 held-out Kaikki Thai test sentences"
0 commit comments