Evaluated results for models produced in this repository. Each section
anchor is the provenance target referenced by model metadata
(models/*/ metadata.yaml) and models/metrics-sources.yaml.
Sequence-level KD: B-K/umt5-thai-g2p-v2-0.5k teacher -> ByT5-base
student over 48,757 beam-4 teacher-generated labels (Kaikki + epitran
Wikipedia corpus, deduplicated, degenerate outputs filtered). Harness:
beam-4, corpus-level PER (total_ed / total_gold over characters of
joined-piece decode), 1,219 held-out Kaikki Thai test sentences
(src/gpu/modal_distill.py::evaluate_per).
| Model | PER (beam-4) | PER (greedy) | EM (greedy) |
|---|---|---|---|
| Teacher (B-K/umt5 hub base) | 4.43% | 1.25% | 95.16% |
| Student (ByT5-base, gate) | 9.19% | 3.53% | 90.48% |
Same-protocol distillation cost: +2.28pp greedy-to-greedy (was +4.76pp beam-vs-beam), comfortably inside the +5pp budget. Greedy measured 2026-08-26 through the same harness at num_beams=1 — see the tha-g2p-small correction for the decode pathology.
Tier inversion at greedy: the client tier (ByT5-small int4, 2.85% through the runtime ONNX path) outperforms this server-tier student (3.53% through the torch harness); the 0.08pp ONNX parity delta cannot account for a 0.68pp gap, so the ordering is real on this harness. The beam-4 figures had the tiers reversed. (docs/DISTILL-SOURCE-PROMPT.md). ByT5-small ablations for reference: 12.63% on 23K labels, 12.06% on 48.7K labels (capacity-limited, both rejected by the gate).
Context: the secryst-published 2.32% umt5 teacher is unrecoverable from saved artifacts (transformers 5.15 save drops the untied umt5 lm_head) and the volume's epitran augmentation corpus is tone-less; this release distills the best verified teacher available. A repaired 2.32%-tier teacher re-enters this pipeline when secryst regenerates it.
The 1,219-sentence test set is now published as a citable benchmark:
benchmarks/thai-kaikki-g2p/ (data, protocol, reference points). No
external Thai comparison exists to date; systems evaluating on the
published benchmark can be ranked against the reference points above.
The v1 ByT5-small teacher shipped directly (byte-level, client-tier
size — no distillation step applies). REF teacher: persian-g2p-checkpoints
persian_g2p/run-001/best, RELEASE-FROZEN per rababa
docs/DISTILL-SOURCE-PROMPT.md (RL variants and the v5/mapped
representation line are closed negative; v1 is final).
| Metric | Value |
|---|---|
| CER (v1 test split, greedy, editdistance) | ≈1.6% |
| SentenceBench homograph (ezafe-normalized) | 77.34% |
Published reference: Homo-GE2PE homograph 76.89% — v1 is above the published best on this benchmark. Claim scope (2026-08-26): SentenceBench homograph accuracy only. Concurrent Persian G2P lines report on their own PER benchmarks — prompted LLMs with post-processing (arXiv 2409.08554, best 8.30% PER) and intermediate-language transliteration trained on LLM-generated data (arXiv 2505.06599) — none shares an evaluation set with SentenceBench, so no cross-paper ranking is claimed.
Logit KD from the s43 teacher (rababa_hebrew_byt5_s43/run-001/best):
KL + CE on hebrew-v4, ByT5-small init. Harness: greedy decode, Nakdimon
IMF test split (1,864 long sentences), same harness for both models
(src/gpu/modal_distill.py::evaluate).
| Model | DER | CER |
|---|---|---|
| Teacher (s43, ByT5-base) | 24.79% | 22.14% |
| Student (ByT5-small, gate) | 30.37% | 24.47% |
Shrink cost +5.58pp — inside the ~5.6pp budget pre-accepted for this pair (rababa docs/DISTILL-SOURCE-PROMPT.md section 2).
The client-tier release of the Thai G2P distillation: run-003,
ByT5-small student on the full label set (48,757 usable beam-4 labels
from the B-K/umt5-thai-g2p-v2-0.5k teacher). Same harness as
tha-g2p-base-1.0 (beam-4, corpus-level PER, 1,219 held-out Kaikki Thai
test sentences, src/gpu/modal_distill.py::evaluate_per; checkpoint
re-measured 2026-08-22 for this release).
| Model | PER | Exact match |
|---|---|---|
| Teacher (B-K/umt5 hub base) | 4.43% | 95.57% |
| Student (ByT5-small, client rung) | 12.06% | 87.94% |
Shrink cost +7.63pp — outside the +5pp server-tier gate (that gate is met by tha-g2p-base-1.0 at 9.19%): shipped anyway per the frontier below, as the smallest artifact that does not collapse. Exported at int8 (~300MB); see the frontier table for why no smaller rung exists today.
Correction (2026-08-24): greedy is the real decode, and it is far better than the beam-4 harness numbers. Re-measured on the shipped int4 zip through the Python runtime (the exact ONNX KV decode users get), true Levenshtein, full 1,219-sentence set:
| Decode | Teacher PER | Student PER | Student EM |
|---|---|---|---|
| beam-4 (published, torch harness) | 4.43% | 12.06% | 87.94% |
| greedy (runtime protocol) | 1.25% | 2.85% | 88.93% |
The teacher is also affected by the beam pathology (4.43 beam-4 → 1.25 greedy, measured 2026-08-25 through the same harness at num_beams=1): same-protocol, the client tier's true shrink cost is +1.60pp, not the +7.63pp the beam-vs-beam comparison suggested.
The beam-4 numbers are inflated by length-normalized beam preferring long garbage on this model's flat per-token distributions (top-1 logprob ≈ -4.6 vs uniform -5.6): exact-match barely moves but every non-exact output runs long, multiplying edit distance. Beam decode is COUNTERPRODUCTIVE for these students; the runtimes ship greedy and that is optimal. The runtime exposes num_beams as an opt-in (verified correct against per-beam batch-1 references); the published beam-4 figures stand as measurements under that decode, not as quality claims. All future gates decode greedy (the Arabic harness already does).
Thai G2P, same harness (beam-4 corpus PER, 1,219 Kaikki sentences; teacher B-K umt5 4.43%):
| Student | Init | Params | Artifact (int8) | PER |
|---|---|---|---|---|
| custom 8+8 d384 | random | 33M | ~30MB | 75.80 (collapsed) |
| custom 8+8 d384 + bridges | random | 33M | ~30MB | 71.12 |
| custom 10+10 d512 + bridges | random | 70M | ~70MB | 78.51 |
| ByT5-small | pretrained | 300M | ~300MB | 12.06 |
| ByT5-base (server tier) | pretrained | 580M | 1.2GB fp32 | 9.19 |
Findings: (1) random-init byte-level seq2seq collapses regardless of capacity at this scale — the microkimi bridges improve structure (75.8 → 71.1) but cannot rescue G2P accuracy; enlarging without pretraining does not help (70M = 78.5). (2) ByT5-small's width (d=1472) dominates its parameter count — depth-pruning yields no useful intermediate rung (263M). (3) The pretrained rung is the whole quality cliff: 300M at 12.06% (run-003, full labels; 12.63% on the 23K subset) vs 70M at 78.5%.
Conclusion: G2P client tier ships at the ByT5-small rung — 246MB at int8, 202MB at int4 (parity 0.0734pp, quality cost ~0.17pp CER; PRs #30/#31) — today; a 30–70MB G2P tier requires byte-level pretraining of the small model first (future work). Copy-task languages are evaluated separately below.
The Arabic copy-task hypothesis test: a 33M-parameter custom byte-level student (d384, 8+8) trained CE on 11,792 r6-teacher labels for 3 epochs (train CE converged to 0.46). Gate harness: windowed DER-CE at the 1400-byte r5 window, greedy, haraqat-projected, Misraj evaluator — identical to rababa's eval_sadeed_windowed; validated by the teacher reproducing its documented tier on this replication.
| Model | DER-CE (300 Sadeed paragraphs) |
|---|---|
| Teacher (r6, run-006-morph) | 1.32% |
| Student (33M from-scratch) | 83.08% — REJECTED |
Gate ≤ teacher + 0.5pp: the student misses by two orders of magnitude.
RETRACTION (2026-08-24): this verdict is CONFOUNDED — every Arabic
label generated before the byt5 decode_joined fix was mojibake
(double-encoded targets); both Arabic students trained on corrupted
labels, and their identical DER scores are the bare-text constant, not
a capacity result. The numbers stand as measured but the capacity
conclusion for Arabic is UNPROVEN pending a clean-label re-run. The
Thai tiny verdict is unaffected (umt5/sentencepiece labels were
byte-exact); the pretrained-backbone law rests on Thai evidence.
The spec intended to rerun the tiny tier on clean labels silently
consumed the Aug-23 label snapshot (pre decode_joined fix). Evidence
chain:
- run-004/best decodes
كتابasÙÙاتÙب— UTF-8-as-Latin1 mojibake, the exact label corruption the retraction describes - the 300-paragraph windowed gate (n=300, teacher reproduces its documented 1.3205) scores the student 83.0797 vs the retracted 83.08 — identical to four decimals: the poisoned-data constant, not a capacity result
final_eval.jsonin the run dir is the durable provenance
Verdict: run-004 says nothing about from-scratch capacity; the retraction stands. run-005 (fresh teacher labels, no snapshot, 2026-08-29) is the actual clean-label falsification test — in flight.
The falsification test the Aug-24 retraction called for: same 33M from-scratch student (d384, 8+8), labels regenerated live from the r6 teacher (11,793 units, no snapshot), 4,422 steps / 3 epochs, final CE ~0.9. Windowed gate, 300 SadeedDiac-25 paragraphs, teacher reproduces 1.3205:
| Model | DER-CE (300) |
|---|---|
| Teacher (r6) | 1.3205% |
| Tiny, mojibake labels (run-004) | 83.08% |
| Tiny, clean labels (run-005) | 74.68% — REJECTED |
Clean labels recover ~8pp of the collapse — the student learns real signal — but remains two orders off the <= 3.07 gate. The pretrained-backbone law now rests on Arabic evidence as well as Thai: from-scratch byte-level students at this width do not work. The viable path to a sub-100MB browser tier is width reduction FROM a pretrained ByT5-small (closed-form stitch across widths, microkimi protocol), not from-scratch training.
[CORRECTED 2026-09-05: the 4.8218 figure below did not reproduce; the corrected 2.0 number is 5.08 (see the correction entry above).]
Sequence-level KD from the r6 teacher (rababa_arabic_byt5/run-006-morph, 2.5793 windowed DER-CE full-protocol): 29,322 greedy labels on r5-units (domain + replay, 1400-byte windows), ByT5-small init, 3 epochs. Same windowed harness as the ara-diac-tiny verdict (300 SadeedDiac-25 paragraphs, Misraj evaluator, haraqat projection):
| Model | DER-CE (300-para subset) |
|---|---|
| Teacher (r6) | 1.3205% |
| Student (ByT5-small, client rung) | 3.6580% |
Full-set correction (2026-08-26): the subset was not representative. Re-measured on the full 1,200-paragraph SadeedDiac-25 benchmark (same harness; teacher reproduces its documented value at 2.5815 vs 2.5793, confirming protocol consistency):
| Model | DER-CE (full 1,200) |
|---|---|
| Teacher (r6, full-set) | 2.5815% |
| Student (ByT5-small, full-set) | 8.2590% |
The first 300 paragraphs sit in the student's training-domain neighborhood; the remaining 900 expose a domain-generalization gap the subset hid. The student's catalog number is the full-set 8.26; the 300-para figures above stand as measurements of that subset only.
Gate discussion: against the full set the strict budget (teacher +0.5pp) is missed by +5.68pp — the capacity-plus-domain cost of ByT5-small trained on 29K r5-unit labels, consistent in direction with the Thai client tier. Shipped as the Arabic client rung with that number disclosed: the student generates real, well-voweled Arabic at a fraction of the teacher's artifact (1.3 vs 2.6 GiB) and the strict gate is met by the teacher release (ara-diac-1.0, 2.58 full-set). On the SadeedDiac-25 leaderboard the student at 8.26 sits just behind Sadeed-1.5B (7.2915 published) and ahead of nothing measured below it — the earlier "between Gemini-Flash and GPT-4" reading was an artifact of the unrepresentative subset and is withdrawn.
Training notes: this is the third training of run-002 — the first on mojibake labels (byt5 decode_joined bug), the second silently resumed from the poisoned lineage's checkpoints (now guarded by labels.sha digest matching), this one clean end-to-end. CE plateaued at ~0.016.
The qwen-next capacity experiment (EXPERIMENTS.md E2): identical to run-002 except three product-key memory layers (+85.9M lookup params, zero-init gates) on the ByT5-small decoder — single-variable.
| Model | DER-CE (full 1,200) |
|---|---|
| Teacher (r6, full-set) | 2.5815% |
| Student + PKM memory (run-003-pkm) | 7.5553% |
| Student vanilla (run-002) | 8.2590% |
0.704pp of the 5.677pp teacher-student gap closed (12.4% relative) at near-zero added compute — below the pre-registered ≥1.0pp win bar, so the memory axis is real but not the dominant term of the gap. Gates verified engaged (0.034-0.053 at completion). Not shipped; the vanilla client rung stands.
Identical to run-003-pkm except the optimizer (Muon on 2D hidden matrices, AdamW group for embedding-like params; EXPERIMENTS.md E3):
| Model | DER-CE (full 1,200) |
|---|---|
| Teacher (r6, this container) | 2.5997% |
| ByT5-small + PKM + Muon (run-004) | 4.8287% |
| ByT5-small + PKM + AdamW (run-003) | 7.5553% |
| ByT5-small vanilla + AdamW (run-002) | 8.2590% |
−2.727pp from the optimizer alone — the adopt gate (≥0.3pp) exceeded 9x; 3.430pp of the 5.677pp canonical gap closed (60.4%) combining memory + optimizer. Training CE ~0.007 vs ~0.02 at equal steps; ~1.2s/step vs ~3.4s; no stability events. The teacher-student gap at this rung decomposes: ~0.70pp capacity + ~2.73pp optimization
- ~2.25pp residual (domain coverage). The vanilla+Muon factorial cell (run-005-muon) completes the decomposition.
| Model | DER-CE (full 1,200) |
|---|---|
| ByT5-small + Muon (run-005) | 5.2945% |
| ByT5-small + PKM + Muon (run-004) | 4.8287% |
| ByT5-small + PKM + AdamW (run-003) | 7.5553% |
| ByT5-small vanilla + AdamW (run-002) | 8.2590% |
The 2x2 closes cleanly: optimizer alone −2.96pp; memory alone −0.70pp (−0.47 under Muon); combined −3.43pp (60.4% of the 5.677pp canonical gap) — roughly additive, slightly sub-additive on memory. Residual ~2.2pp is domain coverage. Optimization is the dominant recoverable term of the distillation gap at the ByT5-small rung.
The two measured wins compounded on the vanilla architecture: r7 canonical teacher (fresh greedy labels) + Muon optimizer, same corpus/limits/seed family. Pre-registered E4 gate ≤ 6.26 (prediction 4.3–5.0):
| Model | DER-CE (full 1,200) |
|---|---|
| Teacher (r7, in-run) | 2.2890% |
| ByT5-small, r7 labels + Muon (run-006) | 4.8218% |
| ByT5-small, r6 labels + Muon (run-005) | 5.2945% |
| ByT5-small, r6 labels + AdamW (run-002, shipped 1.0) | 8.2590% |
4.8218 — gate passed; −3.44pp / 42% relative vs the shipped 1.0 at identical architecture and artifact size. Matches the PKM arm's 4.829 without the memory layers. Release: ara-diac-small-2.0 (run-006 checkpoint; strict teacher+0.5pp still missed at +2.53pp, disclosed).
Leaderboard context (SadeedDiac-25, Misraj evaluator, zero-skip, harakat-projected DER-CE): the teacher tier (r6, 580M) at 2.5793 (reproduced at 2.5815, 2026-08-26) is the best dedicated model measured under this protocol — second only to Claude-3.7-Sonnet's published 1.3941, ahead of GLM-5.2 zero-skip (2.6911), Gemini-Flash-2.0 (3.1926), GPT-4 (3.8645), Sadeed-1.5B (7.2915; source table in rababa docs/RESULTS.md), and GLM-5.3-Flash (8.5721 raw / 8.7978 zero-skip, effort=low, 2026-08-31 — behind Sadeed-1.5B; protocol note: the API rejects disabled thinking combined with reasoning_effort, HTTP 400 code 1210, so effort is pinned per row). The client student's full-set 8.26 lands behind Sadeed-1.5B; see the correction above.
Paper-C evaluation axis (benchmarks/imf-runtime; SPEC.md defines tiers x environments x metrics). First measurements, node tier (Apple Silicon, node 24, interscript@4.1.0, production Release path):
| tier | cold resolve+fetch+verify | warm cache-hit | sha256 tax | session create | decode (short/long) | peak RSS |
|---|---|---|---|---|---|---|
| ara-diac-small-1.0-int8 (257MB) | 25.0s (network) | 528ms | 110ms | 13.1s | 994ms / 2.76s | 953MB |
| tha-g2p-small-1.0 (int8, 202MB) | — | 397ms | 114ms | 12.1s | 85ms / 3.18s | 983MB |
| tha-g2p-small-1.0-int4 (202MB) | — | 369ms | 87ms | 7.3s | 213ms / 6.48s | 711MB |
Headline: the integrity discipline is free — whole-file sha256 is ~0.1s against 7-13s session creation; the verified-index + cache-hit path is ~0.4s. int4 halves load time but decodes ~2.5x slower than int8; int8 is the client default. E2 (Modal 4-vCPU / 8 GiB — the production serving shape, 2026-08-29):
| tier | cold load (zip + verify + ORT) | decode short/med/long |
|---|---|---|
| tha-g2p-small-1.0 (257MB) | 3.92s | 130 / 409 / 671 ms |
| ara-diac-small-1.0-int8 (257MB) | 4.04s | 395 / 525 / 973 ms |
Server vs node-laptop tier: cold load 4s vs 13s session create, decode ~2.5-3x faster — the serving tier trades network for speed. E3 (browser WASM/WebGPU) pending.
Encoder 12->6 from pretrained ByT5-small, layers copied VERBATIM (no projection - the width-cut rungs failed at 74.68/82.96), Muon, same clean r6 labels, 10,995 steps, final CE 0.016 (the scratch rung converged near 0.9 - 50x lower train loss at the same step count). 300-paragraph subset gate (teacher reproduces 1.3205):
| rung | params | subset DER-CE |
|---|---|---|
| ByT5-small 1.0 (AdamW, full) | 300M | 3.658 |
| layerdrop (enc 6, Muon) | ~190M | 3.8088 |
Halving encoder depth costs 0.15pp on the subset - the pretrained representation survives a depth cut that width surgery destroyed. Full-set gate (1,200 paragraphs) in flight; int8 ~190MB, int4 ~95MB (the browser-budget artifact). Survived two infra failures en route (eviction without watchdog; a regressed d_kv derivation) - both fixed.
The 1,200-paragraph gate (teacher reproduces 2.5815):
| rung | params | full-set DER-CE | subset DER-CE |
|---|---|---|---|
| 1.0 (full depth, AdamW, r6) | 300M | 8.259 | 3.658 |
| layerdrop (enc 6, Muon, r6) | ~190M (63%) | 7.4413 | 3.8088 |
| r6 + Muon (full depth) | 300M | 5.2945 | — |
| 2.0 (r7 + Muon, full depth) | 300M | 4.8218 | — |
| scratch d384 | 33M | 74.68 | 83.08 |
| SVD width-stitch d384 | 29M | 82.96 | — |
Reading: halving encoder depth + Muon BEATS full-depth AdamW (7.44 vs 8.26) at 63% of the parameters — but the depth cut costs 2.15pp against its optimizer-matched peer (5.29). Strict gate (teacher+0.5) failed. This is the THIRD instance of the first-300 subset overstating quality (3.66 vs 8.26; 3.81 vs 7.44) — the subset sits in the training-domain neighborhood; full-set-only stands as the publication rule, now with a quantified repeat rate. The size-quality frontier is complete and monotone: 33M/74.7 - 29M/83.0 - 190M/7.4 - 300M/5.3 - 300M/4.8 (teacher 2.28-2.58). Browser-tier decision (user): ship layerdrop-int4 (~95MB, ~7.5 DER with int4 flip risk ungated) as the lite rung, or hold the tier at 2.0-int8 (264MB, 4.82).
The untested 2x ratio (d1472->d768, ~105M, Muon, same r7 labels, 10,995 steps, train CE recovered to 0.95): 300-subset DER 78.23 (teacher in-run 1.2887). SVD width-stitching now fails at BOTH tested ratios (3.8x: 82.96; 2x: 78.23) while the depth cut works (7.44). The law sharpens: the pretrained WIDTH is load-bearing - projection destroys the representation at any compression; depth is the compressible axis. First verdict carrying the label-provenance hash (labels sha256 e70ce991..., 137.2MB recorded in final_eval.json).
Same as the 2.0 rung (r7 labels, Muon, vanilla ByT5-small) at 6 epochs instead of 3 (21,990 steps, final CE 0.0013). 300-subset verdict: teacher in-run 1.2887, student 2.0062 — gate_delta 0.72, within striking distance of the strict teacher+0.5 gate, vs 3.66 (1.0) and 3.81 (layerdrop) on the same subset. This overturns the E2/E3 residual attribution: the 2.25pp residual was mostly optimization (undertraining), not domain coverage. Full-set gate running - given three prior subset-overstatement instances, the honest number waits there. Provenance: labels sha256 e70ce991 (137.2MB) recorded.
Full 1,200-paragraph gate (teacher in-run 2.289): student 4.5701, gate_delta 2.28. Doubling epochs bought 0.25pp over the 3-epoch 2.0 rung (4.8218 -> 4.5701) - the residual is NOT mostly undertraining; the E2/E3 domain attribution substantially stands (~0.25pp epochs, ~2.0pp domain/other). The subset had said 2.0062 / delta 0.72 - a near-overturn of the decomposition that the full set corrects: the FOURTH subset-overstatement instance and the most dramatic (subset delta 0.72 -> full delta 2.28, a 3.2x inflation). The full-set-only publication rule earns its keep here: this entry is the paper's strongest measurement-discipline exhibit.
A2 (TODO.improve-compare): the lite rung with G2a's full lever set (6 epochs, Muon, r7 labels, half depth). 300-subset: teacher in-run 1.3014, student 2.6495, paired bootstrap delta 1.4842 [1.079, 1.975], p=0. The epochs lever moved the lite rung 3.81 -> 2.65 (full-depth moved 3.66 -> 2.01): most of the optimization gain transfers at half depth; the depth cost widened from 0.15pp (3ep) to 0.64pp (6ep) on this subset. Full-set gate in flight - the honest number per the four-instance subset-overstatement record. First lite verdict carrying its own CI.
Full 1,200-paragraph gate (teacher r7 in-run 2.2921): student 5.784, paired bootstrap delta 3.2455 [3.033, 3.491], p=0. The complete full-set frontier: 1.0 (3ep AdamW, 300M) 8.259 -> lite (6ep Muon, 190M) 5.784 -> full (6ep Muon, 300M) 4.5701 -> teacher 2.29. Depth cost at 6 epochs: 1.21pp (CIs non-overlapping vs G2a [1.91, 2.35] - statistically real). Subset said 0.64pp: the FIFTH subset-overstatement instance. The lite tier is shippable: 28% smaller artifact for 1.21pp; int4 (~95MB) export is the browser-budget item.
The 1.0 baseline's retrofit lands the final interval: student 8.2576, delta CI [4.596, 5.651] (n=1200). The full-set table, all rows with non-overlapping sequential intervals:
| rung | full-set DER | delta CI95 vs teacher |
|---|---|---|
| 1.0 (AdamW, r6, 3ep) | 8.2576 | [4.596, 5.651] |
| lite (Muon, r7, 6ep, half-depth) | 5.784 | [3.033, 3.491] |
| 2.0 (Muon, r7, 3ep) | 4.8218 | [2.358, 2.823] |
| G2a (Muon, r7, 6ep) | 4.5701 | [1.911, 2.352] |
Every lever claim in paper B now carries its interval; the monotone separation between rungs is statistically real end to end.
The 30M class with EVERY lever (full 24k+6k corpus, r7 labels, Muon, 6 epochs, 21,990 steps, CE ~0.51): full-set 73.9489, delta CI [70.222, 71.093], p=0. The maximal run improves on the handicapped scratch collapse (74.68) by 0.7pp: data quality and optimization do NOT rescue 30M from-scratch. Conclusion from evidence: the "30M-parameter encoder" title claim is closed; the thesis repositions per PAPER-ALIGNMENT.md.
ara-diac-small-2-6ep-tashkeela (G2b) — full-set 4.8231: the add-direction test is FLAT-NEGATIVE (2026-09-04)
The domain-residual causal test (TODO.publish-client 01): the G2a recipe with the FULL cleaned Tashkeela corpus added (5x classical coverage, 48k units, 39,018 steps, final CE ~0.002). Full 1,200-para gate: teacher in-run 2.289, student 4.8231, paired bootstrap delta 2.3717 [2.194, 2.554], p=0 — vs G2a's 4.5701 [1.91, 2.35]. The point estimate moved the WRONG direction by 0.25pp with overlapping CIs: statistically flat. Gate (>=0.5pp improvement) FAILED.
Verdict: combined with E6 (register swap at constant budget, -0.98pp), BOTH directions of the classical-corpus manipulation fail to close the residual — the student-tier gap is NOT a classical-domain coverage deficit. The domain-coverage attribution is rejected in the add direction and the swap direction; the residual reframes as a teacher-student interaction the corpus cannot reach (candidate next levers: on-policy distillation — GKD in flight; label-distribution effects). G2a (2.1) remains the best student; G2b stands as the closing negative of the data-side program. Provenance: labels sha256 b59e2f56 (235.0MB).
Item 04 (TODO.publish-client): the Arabic width/depth finding replicated on Hebrew — encoder 12->6 verbatim layer copy from pretrained ByT5-small, single variable vs run-002-s46 (same s46 teacher, hebrew-v4 corpus, 3 epochs, logit-KD recipe). Training converged normally (val_loss 0.550); the full Nakdimon gate did not:
| Model | DER (n=1864) |
|---|---|
| teacher s46 | 23.72 (reproduces exactly) |
| full-depth student (run-002, 1.1) | 30.38 |
| layerdrop student (run-003) | 77.48 — collapse |
Paired delta +53.77pp [51.64, 55.92], p=0.
Verdict: the "depth is the compressible axis" finding does NOT transfer as-is. Confound, stated honestly: the Arabic rung used sequence-KD (teacher labels, Muon, 6 epochs); this run used the Hebrew lineage's logit-KD (alpha-KL + CE) at its native 3 epochs — so the collapse may be recipe-dependent (depth-cut + logit-KD), not purely linguistic. Either way the cross-lingual generalization claim is closed as a negative: depth-compressibility is NOT a universal property of pretrained ByT5-small; it held under one recipe on one language. Paper B's depth paragraph is scoped accordingly (this entry is its counterexample).
The published 2.0 number (2026-08-30, in this ledger and the shipped metadata) does not reproduce. Two independent later measurements of the same checkpoint agree and disagree with it:
| Measurement | Path | DER-CE |
|---|---|---|
| published (2026-08-30) | harness, in-run | 4.8218 |
| harness re-eval (final_eval.json, bootstrap CI [2.358, 2.823]) | torch, same protocol | 5.0821 |
| artifact-level (this entry) | shipped zip sha d9aa95d0 (= index pin = release bytes), windowed ONNX runtime decode, sadeedbench scoring | 5.0321-class (5.0329) |
The artifact-level measurement is the governing one: it scores the exact bytes users download. The 4.8218 figure is withdrawn; the 2.0 rung's catalog number is 5.08 (harness) / 5.03 (runtime path). The cause of the original reading is not reconstructed; both later measurements postdate it and agree to 0.05pp across independent decode paths.
Consequences, stated plainly:
- the E4 error-reduction claim becomes 8.259 -> 5.08 (38%, not 42%)
- the "matches the PKM arm (4.829)" statement is wrong: the vanilla 2.0 rung (5.08) does NOT match the PKM arm; the PKM arm was better by 0.25pp at its measurement
- frontier ordering 1.0 -> lite -> 2.0 -> 2.1 is unchanged; G2b (4.8231) sits between 2.0 and 2.1 rather than above 2.0
- every other frontier row re-verified exactly by the same artifact/preds-level tooling (8.2576 / 4.5701 / 5.784 / 4.8231)
Predictions for all five frontier runs publish alongside this entry (release frontier-predictions-v1) so the numbers above are re-derivable by anyone.
The on-policy rung (GKD: student-generated mistakes scored by the teacher, 10,995 steps, labels sha e70ce991): teacher reproduces 2.289, student 6.0036 full-set, paired bootstrap delta 3.4083 [3.109, 3.743], p=0 — gate FAILED, and 1.43pp WORSE than the off-policy sequence-KD rung it was meant to improve (G2a 4.5701).
This completes the residual-attribution program with a clean pattern: every lever tested fails to close the teacher-student gap —
- classical corpus add (G2b): flat-negative
- register swap (E6): negative
- on-policy distillation (GKD): negative, worse than off-policy
- memory layers (PKM): real but small (0.70pp), below bar
- epochs: small (0.25pp) The residual is not data, not domain coverage, not an off/on-policy deficit. It is a property of the compression itself at this rung — the honest open question for the paper. On-policy stays listed in Paper B's future work with its measured negative attached.
The golden matrix (12 models x 25 rows, generated from the released zips on x86/ORT-1.23.2) exposes the true boundary of the byte-parity contract, measured three ways (Python-x86 generator, Python-arm64, TS-arm64; ORT 1.23.2 everywhere):
- fp32-class artifacts: byte-stable. ara-diac-1.0 reproduced exactly on arm64 before the run reached the quantized models.
- Quantized artifacts (int8/int4): prefix-consistent, stop-point-unstable. Every divergent output is an exact PREFIX of the golden (no contradictory content anywhere); the divergence is exclusively WHERE decode emits EOS — the stop decision sits at a near-tie on these flat distributions, and architecture-level float accumulation differences (x86 container vs arm64, ORT-web vs ORT-native) flip it. Same mechanism family as the beam-decode pathology: likelihood ranking on near-uniform distributions.
Contract, scoped and honest: byte-identical across runtimes and hardware for fp32/fp16; for quantized artifacts, quality parity (the published per-model cer_delta gates) plus prefix-consistency. golden-v1's quantized rows are reference outputs (documented hardware/ORT provenance), not byte-assertions.
The shipped artifacts form a drafter/verifier pair: ara-diac-layerdrop-1.0-int4 (190M) drafts, ara-diac-small-2.1-int8 (300M) verifies. Measured over all 25 golden-v1 Arabic rows (CPU, K=8, greedy verification):
- acceptance 0.9886 mean (min 0.955, median 0.991) — the int4 drafter's argmax matches the int8 verifier's at ~99% of positions
- 8.85 tokens per verifier pass (K=8 plus the bonus token on full acceptance): ~9x fewer verifier invocations than token-by-token decode
- output preservation holds by construction and by measurement: every exactness-checked row (18 of 25; the O(T^2) plain-path reference is capped to <=600B rows) is byte-identical to the verifier's plain-path greedy; plain and KV paths agreed on all checked rows
Source: TODO.qwen-next/10 (DeepSeek-V4.1-Flash learnings, DSpark pattern), probe at scripts/probe_speculative.py. Runtime: interscript-ts SpeculativeModel (PR #77). The lite tier stays the standalone fast path; this is a middle tier — 2.1 outputs at a fraction of the decode calls.
Follow-up measurement to the acceptance entry (bench: interscript-ts/scripts/bench-speculative.mts, 5 golden rows, warm, arm64 CPU / ORT-native):
| path | total s | tokens/s |
|---|---|---|
| 2.1 int8 plain (KV greedy) | 19.8 | 67 |
| lite int4 plain (KV greedy) | 61.5 | 22 |
| 2.1 via speculative | 113.8 | 12 |
Acceptance held at the bench (0.9880, matching the probe's 0.9886) — the acceptance math is sound. The wall-clock is not, for two measured reasons:
- Both speculative paths rebuild KV from zero every block — the drafter re-prefills [PAD]+prefix per block and the verifier runs the full sequence with zero pasts per review: O(T^2) against the plain path's O(T). This is an implementation defect, not a property of the method; the production design carries pasts across blocks.
- int4 decode is slower than int8 on this CPU (61.5s vs 19.8s for the plain paths): ORT's CPU int4 path (MatMulNBits decompression) loses to the int8 kernels on arm64 — the drafter is the expensive model here, inverting the small-drafter premise on this hardware.
Consequence for positioning: the tier's published claims ("~9x fewer verifier invocations", output-preserving) stand; no wall-clock speedup is claimed anywhere, and none should be until the KV-carrying implementation is measured. On hardware with fast int4 (or GPU verifier batches) the cost model inverts in the tier's favor.
The KV-carrying rewrite of the runtime's speculative decode exposed a mechanism the cross-hardware study (2026-09-07) did not cover:
Dynamic-int8 ONNX graphs compute activation quantization scales per fed tensor. Decode framing — how many tokens share one decoder call — therefore changes the numerics materially, on the SAME machine, same runtime, same artifact:
- batched feed (K tokens with pasts) vs single-step feed: present values differ up to ~0.03 per element (fp32 KV would be ~1e-6); inner-position argmax flips are routine
- the batched-framing greedy is a DIFFERENT DECODE than the single-step one: drafter==verifier self-acceptance measured 33/66 despite identical final strings via self-correction; the int4->int8 pair's batched-verifier output lost a word ("امُ عليكم" vs "السلام عليكم")
- measured pair acceptance under runtime framing (single-step drafting vs batched verification): 0.4614, vs the probe's 0.9886 measured under uniform plain-path framing — the probe number is framing-relative, and the runtime number is the real one
Wall-clock (arm64 CPU, warm, 5 golden rows): plain 2.1 int8 21.4s; lite int4 75.7s (int4 CPU kernels lose to int8 — MatMulNBits); the O(T) speculative pair 155.4s at 0.46 acceptance. Verdict: the speculative tier is not viable on quantized CPU artifacts — the int4 drafter is the expensive model AND framing divergence collapses acceptance. The technique's domain is fp-class artifacts or serving paths with consistent framing. The playground tier was pulled accordingly; the runtime keeps SpeculativeModel as measurement infrastructure with the constraint documented.
50/50 weight average of the two measured Arabic teachers (580M, run-006-morph and run-007-news), scored under the windowed protocol on all 1200 rows: 2.4188 vs r6's 2.5997 and r7's 2.289. Per domain: classical 1.38 (r7 1.36), news 3.31 (r7 3.21), wiki 2.66 (r7 2.08) — strictly between the parents everywhere; r7 remains the best available teacher and the supervision choice is unchanged.
Two conclusions: (a) the checkpoints are same-basin (the soup is a functional model, confirming linear connectivity between the two teacher lineages — model-soup mechanics apply), and (b) at this pair and scale the soup buys nothing over the better parent. The axis closes negative; recorded so it is not re-derived.
The lite cell's one untested variable was the layer-drop INIT SOURCE (TODO.impl/04): run-009 (5.78) drops from generic pretrained byt5-small; lite2 drops the same layers from the TRAINED 2.1 student, then runs the identical 6-epoch sequence-KD distill (canonical r7 labels, sha e70ce991; teacher re-scores 2.2921 on the same run).
Result: 7.1402 full-set (n=1200), paired-bootstrap gap to teacher 4.29pp [3.83, 4.78]. The trained init is 1.36pp WORSE than the generic init, not better.
Reading: generic pretraining keeps encoder layers redundant and interchangeable, so every-other-layer deletion survives; task adaptation prunes that redundancy — the layers become co-specialized, and deleting half of a co-adapted stack breaks more learned computation. Depth compression on this family survives on generic init and degrades on adapted init, from either direction (the Hebrew layerdrop collapsed from generic init under a weaker recipe; the Arabic adapted-init collapses under the strong one). The lite tier remains run-009 (5.78); init-source closes negative and the depth axis now reads 2-of-3 negative. Remaining architecture lever: TODO.impl/10 (lexical memory), gated as before.
The same single-vs-batched greedy test (5 golden rows x 96 steps, same machine, ORT 1.23 everywhere), run to its endpoints:
| runtime | precision | single==batched |
|---|---|---|
| Python ORT | fp32 | 480/480 |
| Python ORT | dynamic int8 | 480/480 |
| Python ORT | static int8 | 480/480 |
| onnxruntime-node | fp32 | 101/485 |
| onnxruntime-node | dynamic int8 | 97/485 |
| onnxruntime-node | static int8 | 89/485 |
The framing instability is NOT a quantization property: the node build diverges across batch shapes at fp32, and static activation scales (pre-computed into the graph — TODO.impl/11's proposed fix) do not repair it. It is a property of the onnxruntime-node kernel paths (multi-token inputs take numerically different code paths than single-token inputs), absent from the Python build at every precision.
Contract consequences, final form:
- cross-FRAMING parity holds under the Python reference at all precisions and does not hold under onnxruntime-node at ANY precision
- the shipped TS runtime is unaffected in practice: every shipped path (translate, worker, CLI) is single-framing; framing becomes a parity variable exactly when a runtime mixes batch shapes (speculative decode, batched serving) — which is why the speculative tier degraded and was pulled
- static int8 remains a POSITIVE byproduct: quality-clean vs fp32 on this sample (0/480 drift) and ~8% faster than dynamic on CPU — candidate for the export path on its own merits, decided by full-set quality, not framing
The static-activation artifact (TODO.impl/11's byproduct: quantize_static over 365 real decode feeds, MatMul-only, head fp32, QUInt8 acts, remainder addressing) scored under the windowed protocol on all 1200 rows: 4.6241 vs the shipped dynamic int8's 4.5701 — a +0.054pp delta, the same order as artifact-vs-checkpoint drift (the 2.0 artifact measured 5.0329 vs the checkpoint's 5.08). Together with the framing matrix's speed leg (+8% tok/s vs dynamic on CPU: 78 vs 72), static int8 clears quality and wins speed.
Decision state: the RE-EXPORT of all quantized artifacts through the static path (modal_export gains the calibration stage; index-v6; golden rows regenerate) is a release-scale operation and carries a release-scale bar: re-run this gate with --out so the delta ships with a paired-bootstrap CI. The point estimate stands recorded; the export path change is small and the calibration corpus recipe is in scripts/static_int8_experiment.py.
Both artifacts scored full-set with per-row predictions saved (dynamic 4.5619; static 4.5952 — its second run, inside the drift band with the first's 4.6241). Sentence-level paired bootstrap (seed 42, n=1000, the campaign's standard) on the per-item DER delta:
static − dynamic = +2.71pp, CI95 [−1.54, +7.11] (n=1195) — the interval crosses zero: not separated. Micro aggregates differ by 0.03pp; the macro point is dominated by short rows where a few haraqat swings are a large per-sentence percentage.
Verdict by the standing rule: static and dynamic are quality-indistinguishable at our measurement power, and static carries +8% CPU decode speed. The re-export decision (index-v6, golden regen, browser-size composition gated separately) is now fully informed; the export path is one command (modal_export::static, merged in PR #217).