Skip to content

Commit 8d0018d

Browse files
author
Ronald Tse
committed
feat(tha): tha-g2p-small-1.0 client tier — run-003 byt5-small registration
Re-measured run-003 on the beam-4 harness (12.06% PER, EM 87.94%, +7.63pp over the 4.43% teacher, n=1219). Capacity-limited but the smallest non-collapsing rung; exported at int8 (~300MB) per the frontier table. Frontier row updated 12.63 -> 12.06 (full-label run).
1 parent 7190901 commit 8d0018d

6 files changed

Lines changed: 85 additions & 23 deletions

File tree

‎docs/RESULTS.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,26 @@ IMF test split (1,864 long sentences), same harness for both models
6060
Shrink cost +5.58pp — inside the ~5.6pp budget pre-accepted for this
6161
pair (rababa docs/DISTILL-SOURCE-PROMPT.md section 2).
6262

63+
## tha-g2p-small-1.0 — Thai G2P client tier (2026-08-22)
64+
65+
The client-tier release of the Thai G2P distillation: run-003,
66+
ByT5-small student on the full label set (48,757 usable beam-4 labels
67+
from the B-K/umt5-thai-g2p-v2-0.5k teacher). Same harness as
68+
tha-g2p-base-1.0 (beam-4, corpus-level PER, 1,219 held-out Kaikki Thai
69+
test sentences, `src/gpu/modal_distill.py::evaluate_per`; checkpoint
70+
re-measured 2026-08-22 for this release).
71+
72+
| Model | PER | Exact match |
73+
|---|---|---|
74+
| Teacher (B-K/umt5 hub base) | 4.43% | 95.57% |
75+
| **Student (ByT5-small, client rung)** | **12.06%** | 87.94% |
76+
77+
Shrink cost +7.63pp — outside the +5pp server-tier gate (that gate is
78+
met by tha-g2p-base-1.0 at 9.19%): shipped anyway per the frontier
79+
below, as the smallest artifact that does not collapse. Exported at
80+
int8 (~300MB); see the frontier table for why no smaller rung exists
81+
today.
82+
6383
## Client-tier size–quality frontier (2026-08-22)
6484

6585
Thai G2P, same harness (beam-4 corpus PER, 1,219 Kaikki sentences; teacher
@@ -70,7 +90,7 @@ B-K umt5 4.43%):
7090
| custom 8+8 d384 | random | 33M | ~30MB | 75.80 (collapsed) |
7191
| custom 8+8 d384 + bridges | random | 33M | ~30MB | 71.12 |
7292
| custom 10+10 d512 + bridges | random | 70M | ~70MB | 78.51 |
73-
| ByT5-small | pretrained | 300M | ~300MB | 12.63 |
93+
| ByT5-small | pretrained | 300M | ~300MB | 12.06 |
7494
| ByT5-base (server tier) | pretrained | 580M | 1.2GB fp32 | 9.19 |
7595

7696
Findings: (1) random-init byte-level seq2seq collapses regardless of
@@ -79,7 +99,7 @@ capacity at this scale — the microkimi bridges improve structure (75.8 →
7999
not help (70M = 78.5). (2) ByT5-small's width (d=1472) dominates its
80100
parameter count — depth-pruning yields no useful intermediate rung
81101
(263M). (3) The pretrained rung is the whole quality cliff: 300M at
82-
12.63% vs 70M at 78.5%.
102+
12.06% (run-003, full labels; 12.63% on the 23K subset) vs 70M at 78.5%.
83103

84104
Conclusion: G2P client tier ships at the ByT5-small rung (~300MB int8)
85105
today; a 30–70MB G2P tier requires byte-level pretraining of the small

‎models/metrics-sources.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,3 +67,12 @@ heb-diac-small-1.0:
6767
tables:
6868
- {row: "Teacher (s43, ByT5-base)", column: DER, as: der_teacher_greedy}
6969
- {row: "Student (ByT5-small, gate)", column: DER, as: der_student_greedy}
70+
tha-g2p-small-1.0:
71+
repo: interscript/interscript-ml
72+
ref: release/tha-g2p-small-1.0
73+
path: docs/RESULTS.md
74+
anchor: tha-g2p-small-10-thai-g2p-client-tier-2026-08-22
75+
protocol: "beam-4, corpus-level PER (total_ed/total_gold over chars of joined-piece decode); 1,219 held-out Kaikki Thai test sentences"
76+
tables:
77+
- {row: "Teacher (B-K/umt5 hub base)", column: PER, as: per_teacher}
78+
- {row: "Student (ByT5-small, client rung)", column: PER, as: per_student}

‎models/tha-g2p-small/tha-g2p-small-1.0.README.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
11
# tha-g2p-small-1.0
22

33
Thai grapheme-to-phoneme (IPA). Client-tier ByT5-small (300M) student
4-
distilled from the secryst umt5 Thai teacher (2.32% PER, public baseline
5-
6.37%) via sequence-level KD: the teacher generated 23,295 labels with
6-
its own sentencepiece tokenizer and the student trained CE on them with
7-
the canonical byte table.
4+
for the int8 release (~300MB): sequence-level KD from the verified
5+
B-K/umt5-thai-g2p-v2-0.5k teacher over 48,757 beam-4 labels, trained
6+
with the canonical byte table.
87

9-
First model of the distillation campaign
10-
(docs/DISTILL-SOURCE-PROMPT.md); identical IMF v1 contract to the
11-
server-tier models — dynamic fetch, sha256-verified, KV decode.
8+
Capacity-limited at +7.63pp over the teacher (12.06% vs 4.43% PER) —
9+
inside the +5pp server-tier gate is tha-g2p-base-1.0; this is the
10+
smallest artifact that does not collapse (from-scratch 33M/70M students
11+
and pruned rungs all fail — see the frontier table in
12+
docs/RESULTS.md). Identical IMF v1 contract to every other model:
13+
dynamic fetch, sha256-verified, KV decode.
1214

1315
```python
1416
from interscript_ml import Model

‎models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml‎

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -9,22 +9,27 @@ decoder: kv
99
precision: fp32
1010
license: BSD-3-Clause
1111
trained_from: >-
12-
distilled from the recovered Thai umt5 teacher (transformers 5.14.1
13-
re-finetune of B-K/umt5-thai-g2p-v2-0.5k on the 60K Kaikki+epitran
14-
corpus; the secryst-saved umt5 artifacts are unusable — 5.15 dropped
15-
the untied lm_head) via sequence-level KD; ByT5-small init
16-
google/byt5-small; checkpoint
17-
secryst-checkpoints:/secryst_thai_g2p_distill_small/run-002/best
12+
sequence-level KD from the B-K/umt5-thai-g2p-v2-0.5k teacher (4.43%
13+
PER on this harness; the secryst-saved umt5 artifacts are unusable —
14+
transformers 5.15 dropped the untied lm_head); 48,757 usable beam-4
15+
teacher-generated labels (51,321 raw, deduplicated, degenerate outputs
16+
filtered); ByT5-small init google/byt5-small; checkpoint
17+
secryst-checkpoints:/secryst_thai_g2p_distill_small/run-003/best.
18+
Client tier: capacity-limited at +7.63pp over the teacher — the +5pp
19+
gate is met by tha-g2p-base-1.0; this is the smallest rung that does
20+
not collapse (docs/RESULTS.md frontier table, ~300MB int8).
1821
metrics:
1922
- name: per_teacher
20-
value: 2.32
23+
value: 4.43
2124
protocol: >-
22-
greedy decode; 1,219 Kaikki Thai test sentences; umt5 teacher,
23-
sentencepiece tokenizer; secryst RESULTS.md protocol
24-
source: secryst/docs/RESULTS.md#thai-g2p
25+
beam-4, corpus-level PER (total_ed/total_gold over chars of
26+
joined-piece decode); 1,219 Kaikki Thai test sentences; B-K/
27+
umt5-thai-g2p-v2-0.5k teacher; exact match 95.57%
28+
source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-small-1.0
2529
- name: per_student
26-
value: PLACEHOLDER_STUDENT_PER
30+
value: 12.06
2731
protocol: >-
28-
greedy decode; same 1,219 test sentences, same harness as the
29-
teacher (interscript/ml-models src/gpu/modal_distill.py::evaluate_per)
30-
source: interscript/ml-models release tha-g2p-small-1.0
32+
beam-4, corpus-level PER, same harness as the teacher (src/gpu/
33+
modal_distill.py::evaluate_per, run-003 checkpoint re-measured
34+
2026-08-22); exact match 87.94%; +7.63pp client-tier shrink cost
35+
source: interscript/interscript-ml docs/RESULTS.md#tha-g2p-small-1.0

‎src/gpu/modal_distill.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,23 @@
8383
"mode": "sequence", # cross-tokenizer: teacher generates, student trains CE
8484
"note": "umt5 (sentencepiece) teacher -> ByT5-small byte student; +5pp PER gate",
8585
},
86+
"tha-g2p-client": {
87+
# the shipped client rung: run-003, ByT5-small on the full label
88+
# set — capacity-limited (+7.6pp) but the smallest artifact that
89+
# does not collapse (see docs/RESULTS.md frontier table)
90+
"teacher": "B-K/umt5-thai-g2p-v2-0.5k",
91+
"teacher_is_hub": "true",
92+
"teacher_volume": "secryst",
93+
"student_init": "google/byt5-small",
94+
"train": "thai-ipa-expanded/train.jsonl",
95+
"train_extra": ["thai-ipa/train.jsonl", "thai-ipa/augmented_epitran.jsonl"],
96+
"val": "thai-ipa-expanded/val.jsonl",
97+
"test": "thai-ipa-expanded/test.jsonl",
98+
"eval_test": "thai-ipa/test.jsonl",
99+
"out": "secryst_thai_g2p_distill_small/run-003",
100+
"mode": "sequence",
101+
"note": "eval-only spec for tha-g2p-small-1.0 (client tier)",
102+
},
86103
"ara-diac-small": {
87104
# r5 paragraph-context teacher (2.68 DER-CE windowed @1400B,
88105
# RELEASE-FROZEN) -> ByT5-small student. Contract decode is

‎src/gpu/modal_export.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,15 @@
113113
"test_data": "thai-ipa/test.jsonl",
114114
"probe": "สวัสดี",
115115
},
116+
"tha-g2p-small": {
117+
"volume": "/volumes/secryst-checkpoints",
118+
"checkpoint": "secryst_thai_g2p_distill_small/run-003/best",
119+
"metadata": "models/tha-g2p-small/tha-g2p-small-1.0.metadata.yaml",
120+
"readme": "models/tha-g2p-small/tha-g2p-small-1.0.README.md",
121+
"test_volume": "/datasets/secryst",
122+
"test_data": "thai-ipa/test.jsonl",
123+
"probe": "สวัสดี",
124+
},
116125
"fas-g2p": {
117126
"volume": "/volumes/persian-checkpoints",
118127
"checkpoint": "persian_g2p/run-001/best",

0 commit comments

Comments
 (0)