Skip to content

Commit fe14708

Browse files
author
Ronald Tse
committed
feat(specs): heb-diac-small-s46-layerdrop — cross-lingual depth-cut replication
Item 04: the Arabic width/depth finding replicated on Hebrew — encoder 12->6 verbatim layer copy from pretrained ByT5-small, single variable vs run-002-s46 (same s46 teacher, hebrew-v4 corpus, 3 epochs). Wires _maybe_stitch into the logit-KD custom-config branch so depth-cut students keep the pretrained init there too (previously random-init, which the capacity law rejects).
1 parent 00d4efa commit fe14708

2 files changed

Lines changed: 26 additions & 0 deletions

File tree

‎src/gpu/distill_specs.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -527,3 +527,25 @@ heb-diac-small-s46:
527527
train: hebrew-v4/train.jsonl
528528
val: hebrew-v4/val.jsonl
529529
out: rababa_hebrew_distill_small/run-002-s46
530+
heb-diac-small-s46-layerdrop:
531+
# Item 04 (TODO.publish-client): cross-lingual depth-cut test — the
532+
# Arabic width/depth finding replicated on Hebrew. Single variable
533+
# vs run-002-s46: encoder 12->6 via verbatim layer copy (layer_drop),
534+
# same teacher, corpus, and epochs. Gate: the Arabic depth premium
535+
# (1.21pp at 6ep) predicts a comparable lite-vs-full gap here.
536+
teacher: rababa_hebrew/run-s46-phonikud-plus/run-002-gold-ft/best
537+
teacher_volume: rababa
538+
student_init: google/byt5-small
539+
layer_drop: 'true'
540+
student_config:
541+
d_model: 1472
542+
d_kv: 64
543+
d_ff: 3584
544+
num_heads: 6
545+
enc_layers: 6
546+
dec_layers: 4
547+
feed_forward_proj: gated-gelu
548+
train: hebrew-v4/train.jsonl
549+
val: hebrew-v4/val.jsonl
550+
out: rababa_hebrew_distill_small/run-003-s46-layerdrop
551+
note: Hebrew lite rung; cross-lingual depth-cut replication

‎src/gpu/modal_distill.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -311,6 +311,10 @@ def distill(spec_id: str, epochs: int = 3, alpha: float = 0.5, temperature: floa
311311

312312
cfg = T5Config(**spec["student_config"])
313313
student = T5ForConditionalGeneration(cfg).to(device)
314+
if spec.get("layer_drop"):
315+
# depth-cut students keep the pretrained init: verbatim
316+
# layer copy, not the random init the tiny tier uses
317+
_maybe_stitch(spec_id, spec, student)
314318
else:
315319
student = AutoModelForSeq2SeqLM.from_pretrained(spec["student_init"]).to(device)
316320
student.train()

0 commit comments

Comments
 (0)