Skip to content

Commit 382b8c8

Browse files
author
Ronald Tse
committed
docs(E2): PKM student verdict — 7.5553 vs 8.259 (0.704pp closed, below the 1.0pp bar)
Full 1,200-paragraph windowed zero-skip: teacher reproduces 2.5815. Positive but not the pre-registered win; capacity is real (12.4% relative) yet not the dominant term of the teacher-student gap. Engagement verified independent of outcome. E3 (Muon arm) tests the optimization half.
1 parent 5c6092d commit 382b8c8

1 file changed

Lines changed: 16 additions & 3 deletions

File tree

‎docs/EXPERIMENTS.md‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,9 +72,22 @@ All rows passed the CER parity gate at release. Readings:
7272

7373
## E2 — PKM memory-layer student (ara-diac-small run-003-pkm)
7474

75-
- **Status:** in flight (launched 2026-08-27, A10G, labels reused from
76-
run-002; single-variable design vs run-002). Survived one mid-run
77-
eviction: resumed from step-2000 via the checkpoint guard.
75+
- **Status:** COMPLETE (2026-08-28). **Outcome: positive but below the
76+
pre-registered win bar.**
77+
- **Measured (full 1,200 paragraphs, windowed zero-skip, greedy):**
78+
PKM student **7.5553** DER-CE vs run-002 vanilla ByT5-small **8.259**
79+
(teacher reproduces 2.5815) — 0.704pp of the 5.677pp teacher-student
80+
gap closed (12.4% relative) at +85.9M lookup params and near-zero
81+
added compute (gathers, not matmuls).
82+
- **Verdict per the pre-agreed rule (≥1.0pp = win): NOT met.** The
83+
capacity axis is real but not the dominant term of the gap; the rest
84+
is modeling/optimization (E3 tests the optimization half) and domain
85+
coverage (the subset lesson from run-002). Reported exactly as
86+
measured — no threshold-moving.
87+
- Engagement was verified independent of outcome: gates moved off zero
88+
by step-500 (0.0008-0.0028), CE 2.07 → 0.02 over 10,995 steps.
89+
- Launched 2026-08-27, A10G, labels reused from run-002 (single-variable
90+
design). Survived one mid-run eviction: resumed from step-2000.
7891
- **Hypothesis:** the 5.68pp teacher→student gap (r6 2.5815 → ByT5-small
7992
8.259 full-set) is partly a *capacity* gap that lookup memory closes
8093
at near-zero compute — parameters and compute are separable (arXiv

0 commit comments

Comments
 (0)