Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified docs/paper-arabic/main.pdf
Binary file not shown.
44 changes: 26 additions & 18 deletions docs/paper-arabic/main.tex
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
every GLM successor we measured is 3--5$\times$ worse (5.3-Flash 8.80,
5.3 9.90, 4.7-flash 13.23 zero-skip), with per-position attribution
locating the loss in wrong haraqat rather than writing convention.
A distilled 300M client-tier student reaches 4.82\% (42\% error
A distilled 300M client-tier student reaches 4.57\% (45\% error
reduction) through a controlled lever ladder whose two registered
negatives---multi-token-prediction auxiliary and constant-budget
register mixing---mirror the RL negative: at SFT convergence every
Expand Down Expand Up @@ -85,15 +85,16 @@ \section{Introduction}
which matches our 580M teacher's 2.62\%), not writing
convention (the dagger-alif effect is 0.125pp).
\item A distilled client tier at 300M parameters: 8.26\%
$\rightarrow$ \textbf{4.82\%} (42\% error reduction at
$\rightarrow$ \textbf{4.57\%} (45\% error reduction at
identical architecture) via a controlled lever ladder
(optimizer $-2.96$pp, fresher teacher labels $-0.47$pp,
longer training $-0.25$pp), plus four registered negatives
(multi-token-prediction auxiliary $+0.26$pp; register swap
$+0.98$pp; register add flat-with-epochs-cancelled at 4.82;
on-policy GKD $+1.18$pp, the worst rung) showing that neither
frontier-training techniques, register mixing in either
direction, nor on-policy exposure substitute
(optimizer $-2.96$pp, fresher teacher labels $-0.21$pp,
longer training $-0.51$pp), plus four registered negatives
(multi-token-prediction auxiliary level with control and far
from its gate; register swap $+0.72$pp; register add flat
against news-only training; on-policy GKD $+1.43$pp over the
rung it was meant to improve, the worst measured) showing that
neither frontier-training techniques, register mixing in
either direction, nor on-policy exposure substitute
for supervision quality.
\item A three-way RL negative result (RAFT, sequence-GRPO on Persian,
entropy-weighted GTPO-GRPO on Arabic): all flat or negative at
Expand Down Expand Up @@ -224,21 +225,28 @@ \subsection{The frontier regression axis}
\subsection{The distilled client tier}
The same benchmark disciplines a 300M distilled student shipped for
browser/edge inference. From the 1.0 rung (8.26\%), a pre-registered
lever ladder reached \textbf{4.82\%} (42\% error reduction at
lever ladder reached \textbf{4.57\%} (45\% error reduction at
identical architecture and artifact size): the Muon optimizer~\cite{muon} alone
contributes $-2.96$pp (a controlled 2$\times$2 factorial closes
additively: optimizer $-2.96$, memory-layer capacity $-0.70$,
combined $-3.43$), fresher r7 teacher labels $-0.47$pp, and longer
training a further $-0.25$pp (4.57\%). Four registered negatives
combined $-3.43$), fresher r7 teacher labels $-0.21$pp, and longer
training a further $-0.51$pp. (The 2.0 rung's originally published
4.8218 did not reproduce---two independent later measurements,
including one at the artifact level on the exact released bytes,
agree on 5.08; the figure was withdrawn and the ladder restated on
the corrected number.) Four registered negatives
bracket the ladder: multi-token-prediction~\cite{mtp} as a training auxiliary
scored 5.09\% ($+0.26$pp vs control, with a disclosed preemption
scored 5.09\% (level with the corrected control and far from its
pre-registered 4.52 gate, with a disclosed preemption
confound); swapping news-domain training units for classical
Tashkeela at constant budget scored 5.81\% ($+0.98$pp); adding
18k classical units on top at matched epochs scored 4.82\%---flat
against the 3-epoch control and 0.25pp behind news-only training at
the same epochs, the add canceling the epoch gain entirely; and
Tashkeela at constant budget scored 5.81\% ($+0.72$pp over the
corrected control); adding 18k classical units on top at matched
epochs scored 4.82\%---statistically flat against news-only training
at the same epochs (4.57, intervals overlapping) and sitting between
the corrected 3-epoch control and the news-only rung; and
on-policy generalized distillation~\cite{gkd} (reverse-KL on student-sampled
sequences scored by the frozen teacher) scored 6.00\% ($+1.18$pp,
sequences scored by the frozen teacher) scored 6.00\% ($+1.43$pp over
the off-policy rung it was meant to improve,
the worst rung measured)---the domain-coverage hypothesis failing
in every direction it was tested. Together with the RL negatives,
the pattern is one line: at SFT convergence, supervision quality
Expand Down
Binary file modified docs/paper-umbrella/main.pdf
Binary file not shown.
8 changes: 4 additions & 4 deletions docs/paper-umbrella/main.tex
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
GLM successor we measured is 3--5$\times$ worse (5.3-Flash 8.80,
5.3 9.90, 4.7-flash 13.23 zero-skip), the loss located by per-position
attribution in wrong haraqat, not writing convention; a 300M distilled
client tier at 4.82\% (42\% error reduction) whose lever ladder brackets
client tier at 4.57\% (45\% error reduction) whose lever ladder brackets
two registered negatives (MTP-auxiliary, register swap) mirroring the
three-way RL negative (RAFT, sequence-GRPO, entropy-weighted GRPO); a
contamination audit showing the benchmark's public source corpus leaks
Expand Down Expand Up @@ -86,10 +86,10 @@ \section{The bottleneck taxonomy}
halved DER; a 30M encoder matches a 1.5B LM; supervision-side
moves (paragraph context, morphological auxiliary, teacher
labeled domain mix) then carried the 580M teacher from 2.81 to
2.29\%; a distilled 300M client tier reached 4.82\% (42\%
2.29\%; a distilled 300M client tier reached 4.57\% (45\%
error reduction) through a controlled lever ladder whose two
registered negatives (MTP-auxiliary $+0.26$pp, register swap
$+0.98$pp) bracket it---supervision quality, not technique,
registered negatives (MTP-auxiliary level with control and far
from its gate; register swap $+0.72$pp) bracket it---supervision quality, not technique,
moves this task.
\item \textbf{Hebrew --- data domain}: the SOTA model collapses
cross-domain (9$\times$); mixed-domain training wins; the teamim
Expand Down