Skip to content

Commit 0027253

Browse files
author
Ronald Tse
committed
docs(RESULTS): lite2 verdict - trained-init depth cut is worse (7.14 vs 5.78)
1 parent 10fb95e commit 0027253

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

‎docs/RESULTS.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -728,3 +728,27 @@ functional model, confirming linear connectivity between the two
728728
teacher lineages — model-soup mechanics apply), and (b) at this pair
729729
and scale the soup buys nothing over the better parent. The axis
730730
closes negative; recorded so it is not re-derived.
731+
732+
## ara-diac-small-lite2 — trained-init depth cut is WORSE: 7.1402 (2026-09-12)
733+
734+
The lite cell's one untested variable was the layer-drop INIT SOURCE
735+
(TODO.impl/04): run-009 (5.78) drops from generic pretrained
736+
byt5-small; lite2 drops the same layers from the TRAINED 2.1 student,
737+
then runs the identical 6-epoch sequence-KD distill (canonical r7
738+
labels, sha e70ce991; teacher re-scores 2.2921 on the same run).
739+
740+
Result: **7.1402** full-set (n=1200), paired-bootstrap gap to teacher
741+
4.29pp [3.83, 4.78]. The trained init is 1.36pp WORSE than the
742+
generic init, not better.
743+
744+
Reading: generic pretraining keeps encoder layers redundant and
745+
interchangeable, so every-other-layer deletion survives; task
746+
adaptation prunes that redundancy — the layers become co-specialized,
747+
and deleting half of a co-adapted stack breaks more learned
748+
computation. Depth compression on this family survives on generic
749+
init and degrades on adapted init, from either direction (the Hebrew
750+
layerdrop collapsed from generic init under a weaker recipe; the
751+
Arabic adapted-init collapses under the strong one). The lite tier
752+
remains run-009 (5.78); init-source closes negative and the depth
753+
axis now reads 2-of-3 negative. Remaining architecture lever:
754+
TODO.impl/10 (lexical memory), gated as before.

0 commit comments

Comments
 (0)