docs: DeepSeek-V4.1-Flash learnings + TODO.impl work ledger

TODO.arabic/10-engram-episodic-memory.md

@@ -46,3 +46,29 @@ Engram store is updated during training (FIFO + importance sampling).
 
 ## Open questions
 - Capacity: 10K is from the paper. For our smaller model, sweep 1K/5K/10K.
+
+## CORRECTION (2026-09-11, from the DeepSeek-V4.1-Flash report §2.4.2)
+
+The mechanism described above (retrieving similar past training
+examples) is NOT Engram. The actual design (arXiv:2601.07372):
+hash-addressed n-gram embedding tables summed into the forward pass —
+no example retrieval, no similarity search, no FIFO store.
+
+- addresses = n-gram hashes of the input token sequence
+- orders {2,3,4}, 8 hash heads, 2048-dim per order
+- table sizes are distinct primes (~16M entries per head in DS's
+  196B-parameter deployment)
+- FP8 storage, modules placed at layers 1 and 14
+- 5x learning rate for the tables; table updates use the
+  Sinkhorn-balanced momentum rule (momentum + alternating row/col
+  normalization), not Adam
+- effect: memorization decoupled from compute — deterministic lookup,
+  prefetchable, no bandwidth interaction with attention
+
+The class-imbalance rationale above (rare haraqat reinforced by
+retrieval) does not transfer to the real mechanism; the applicable
+rationale is lexical idiom: haraqat are strongly n-gram-driven, and a
+lookup table captures idiosyncratic lexical patterns without
+compute. Any wire-in must be rebuilt against the real design; the
+browser size ceiling (int8 4M x 64-dim ~ 64 MB) bounds the table.
+Tracked at TODO.impl/10.

TODO.arabic/18-per-head-muon-wirein.md.md

@@ -1,17 +1,21 @@
 # Per-Head Muon wire-in
 
-## Status
-Module is built (see v0.5.0 shipped). Wire-in needs:
-- Config flag in YAML
-- Dispatch in training loop or model builder
-- End-to-end smoke test
-- Spec for the wire-in
+## Status (2026-09-11)
+Implemented in ml-models (this note was stale — no module had shipped
+in v0.5.0): `src/gpu/muon.py` carries the headwise group
+(`add_headwise_group`, per-slice Newton-Schulz, `qk_named` selector),
+config-gated by the `headwise_muon` spec flag, off by default. Specs:
+`ml-models/tests/test_muon_headwise.py` (5 tests, green) — per-head
+update equals vanilla Muon on each head slice; heads=1 equals vanilla
+whole; off-state routing unchanged. Externally validated by
+DeepSeek-V4.1-Flash sec 2.5, GLM-5, Kimi-K3 (TODO.impl/03).
+Remaining: exercise on the next teacher run (flag on).
 
 ## Acceptance
-- [ ] Feature is enabled by config flag, off by default
-- [ ] Spec covers the wire-in dispatch
+- [x] Feature is enabled by config flag, off by default
+- [x] Spec covers the wire-in dispatch (tests + flag wiring)
 - [ ] End-to-end training run completes with feature enabled
-- [ ] No regression on baseline (feature off → identical results)
+- [x] No regression on baseline (feature off → identical results)
 
 ## Files
 - (TBD based on feature)

TODO.arabic/19-n-stream-mhc-wirein.md.md

@@ -15,3 +15,13 @@ Module is built (see v0.5.0 shipped). Wire-in needs:
 
 ## Files
 - (TBD based on feature)
+
+## Deployable variant (added 2026-09-11, DeepSeek-V4.1-Flash §2.4.1)
+
+Single-Pass mHC: consume the PREVIOUS block's input-mixing coefficients
+(A_{l-1}) instead of the current block's. This removes the reduction
+dependency so residual update, input mixing, and coefficient prediction
+fuse into one kernel — halving activation memory traffic at "negligible
+degradation." If n-stream mHC resumes, train with the shift from the
+start so the network adapts to it (their pattern: architecture change
+plus training-aware adaptation, never architecture change alone).

TODO.hebrew/11-engram-wirein.md

@@ -1,7 +1,10 @@
 # Engram wire-in
 
-Same pattern as Arabic equivalent. See Arabic TODO.arabic/ for shared
-infrastructure.
+Same pattern as Arabic equivalent — and the same correction applies:
+see TODO.arabic/10 (2026-09-11). Engram is a hash-addressed n-gram
+embedding table summed into the forward pass, not example retrieval;
+any Hebrew wire-in must be built against the real mechanism
+(TODO.impl/10).
 
 ## Acceptance
 - [ ] Feature enabled for Hebrew

TODO.impl/01-speculative-runtime.md

@@ -0,0 +1,55 @@
+# 01 — Speculative decoding across the tier ladder (P0)
+
+Source: TODO.qwen-next/10 §1 (unparks TODO.qwen-next/04). The parked
+objection — "the draft model would itself need training" — expired
+when ara-diac-layerdrop-1.0-int4 (190M, 95 MB) shipped: drafter and
+verifier (ara-diac-small-2.1-int8, 300M) share the byte table and
+greedy decode, and greedy verification is output-preserving, so the
+artifacts need no parity story — only a speed one.
+
+## Why
+
+Byte-level decode is one token per step; the browser tier pays it in
+wall-clock. DSpark's shape (draft K, verify all K in one pass,
+confidence-scheduled K) maps directly onto two shipped models. Early
+probe rows: acceptance 0.98–1.00, 8.5–8.9 tokens per verifier pass at
+K=8 — the int4 drafter matches the int8 verifier's argmax almost
+everywhere, consistent with the 0.036% near-tie flip measurement.
+
+## Deliverables
+
+1. **Acceptance probe** (CPU, golden-v1 Arabic rows): per-row
+   acceptance rate, tokens/verify, exactness of the speculative loop
+   vs the verifier's plain-path greedy (same execution path — the
+   theorem check). Script:
+   `ml-models/scripts/probe_speculative.py`, results to
+   `~/ml-logs/spec_probe/results.json`.
+2. **Runtime implementation** (interscript-ts): a
+   `SpeculativeSession` composing drafter + verifier IMFModel
+   instances — OCP: `translate()` untouched, the strategy is a new
+   class; block size is config data, not code. Unit tests on the
+   acceptance algorithm (synthetic logits are data, not model mocks);
+   e2e opt-in with real zips.
+3. **Tier decision**: if full-probe acceptance clears ~0.9, expose
+   "2.1 quality at lite-ish latency" as a runtime option (not a new
+   artifact). Publish the acceptance measurement next to the tier in
+   docs.
+
+## Acceptance
+
+- [ ] Probe: 25 rows measured; exactness holds on every checked row
+      (plain-path reference; long rows may skip the O(T^2) reference,
+      recorded as unchecked)
+- [ ] Probe bug class closed: EOS-block acceptance appends pre-EOS
+      tokens; bonus index computed post-extend
+- [ ] TS `SpeculativeSession` with unit + opt-in e2e tests, PR merged
+- [ ] Numbers recorded in TODO.qwen-next/10 §1 and RESULTS.md if the
+      tier ships
+- [ ] Probe script landed in ml-models with its results
+
+## Status
+
+- [x] Probe v1 written; EOS-acceptance and index bugs found and fixed
+- [ ] Probe rerun to completion
+- [ ] Runtime implementation
+- [ ] Tier exposure decision

TODO.impl/02-multi-teacher-domain-routing.md

@@ -0,0 +1,64 @@
+# 02 — Multi-teacher domain-routed distillation (P0)
+
+Source: TODO.qwen-next/10 §2. DeepSeek's final OPD stage distills from
+40+ heterogeneous teachers, best-per-domain; we have always used one
+teacher per run. Our residual program closed corpus volume, domain
+swap, and on-policy GKD — a teacher mixture is the cheapest untested
+probe of the open attribution ("the residual lives in the
+teacher–student interaction").
+
+## Measured so far (2026-09-10, released run-007 predictions)
+
+Protocol check: re-aggregated slices reproduce the published 2.289
+(teacher) / 4.5701 (student) exactly.
+
+| domain | n | teacher r7 | student 2.1 | gap |
+|---|---|---|---|---|
+| Fadel_test (classical) | 600 | 1.36 | 2.36 | 1.00 |
+| Our-Benchmark (news) | 454 | 3.21 | 6.08 | 2.87 |
+| WikiNewsTruth | 146 | 2.08 | 6.37 | 4.29 |
+
+The old attribution inverts: the residual is NOT classical coverage —
+the student tracks r7 tightly on classical. The gap lives on news and
+wiki rows, which also explains why classical-corpus swaps measured
+flat: they targeted the domain where the student is already closest.
+
+## Deliverables
+
+1. r6 teacher predictions on the same 1200 rows under the published
+   windowed protocol (`ml-models/src/gpu/modal_teacher_sadeed.py`,
+   output `/checkpoints/probes/r6_sadeed_preds.jsonl`).
+2. r6 per-domain slice vs r7's; record both profiles in the memo.
+3. Verdict: if r6 beats r7 on the news/wiki slices by a margin the
+   paired bootstrap separates, arm a routed-teacher distill spec
+   (supervision per row from that domain's winner) and launch;
+   otherwise record the negative and close the axis.
+
+## Acceptance
+
+- [ ] r6 preds fetched and sliced; both teacher profiles in
+      TODO.qwen-next/10 §2
+- [ ] Routed arm launched with an explicit single-variable spec
+      (same student shape, same epochs as 2.1), or the negative
+      recorded with intervals
+- [ ] No subset-overstatement: any routed-run claim is full-set
+
+## Verdict (2026-09-11): NEGATIVE — axis closed for the r6/r7 pair
+
+| domain | n | r6 | r7 | winner |
+|---|---|---|---|---|
+| Fadel_test (classical) | 600 | 1.37 | 1.36 | tie |
+| Our-Benchmark (news) | 454 | 3.52 | 3.21 | r7 |
+| WikiNewsTruth | 146 | 3.29 | 2.08 | r7 |
+
+r7 dominates everywhere it wins full-set. No routed mixture of the
+available teachers can beat plain r7 supervision. The residual is
+student-side (news/wiki rows), not teacher-selection. Remaining
+levers: 04 (trained-lite), 10 (lexical memory), 01 (runtime).
+
+## Status
+
+- [x] r7 per-domain slice (from released predictions, zero GPU)
+- [x] r6 preds on volume, fetched, sliced
+- [x] Verdict: negative, recorded in TODO.qwen-next/10 §2
+- [x] No routed launch (nothing to route)

TODO.impl/03-headwise-muon.md

@@ -0,0 +1,39 @@
+# 03 — Per-head Muon wire-in for Q/K (P1)
+
+Source: TODO.qwen-next/10 §3. DeepSeek-V4.1-Flash splits Query and
+Key weight matrices by head before the Muon update ("head-wise
+Muon"), reporting it over vanilla Muon; GLM-5 and Kimi-K3 validate
+the same choice. Our module exists (v0.5.0); the wire-in never ran
+(TODO.arabic/18 acceptance list is unchecked). Three-lab convergence
+upgrades it to the default optimizer treatment for Q/K in the next
+teacher run.
+
+## Why
+
+Muon's single preconditioner spans all attention heads; per-head
+splitting gives each head its own preconditioner, matching head
+heterogeneity. Cost: one reshape per step. It is a recipe change,
+not an architecture change — single-variable, cheap to ablate.
+
+## Deliverables
+
+1. `src/gpu/muon.py`: per-head view application for 2-D Q/K weights
+   (reshape to [heads, d_head * in, ...] per the module's existing
+   convention), config-gated, off by default.
+2. Unit test with real tensors: per-head update equals vanilla Muon
+   applied per head slice; off-state bit-identical to today's path.
+3. Wire-in spec for the next teacher run (r8-class): flag on, log
+   line proves the per-head path executed.
+
+## Acceptance
+
+- [ ] Config flag exists, default off; no behavior change when off
+- [ ] Unit tests green on real tensors (not doubles)
+- [ ] Flag-on path exercised end-to-end in a smoke run
+- [ ] TODO.arabic/18 updated to point here
+
+## Status
+
+- [ ] Inspect muon.py for the built module's exact API
+- [ ] Implementation + tests
+- [ ] Smoke run

TODO.impl/04-trained-lite.md

@@ -0,0 +1,43 @@
+# 04 — lite-2.0: init the depth cut from the trained student (P1)
+
+Source: TODO.qwen-next/10 §4. CORRECTED 2026-09-11 (spec audit):
+the shipped lite rung (run-009-layerdrop-6ep, 5.78) IS a trained
+6-epoch sequence-KD distill — its layer-drop INIT copies kept layers
+verbatim from generic pretrained byt5-small. The untested variable in
+the lite cell is the INIT SOURCE: DeepSeek's repeated pattern is
+compress-from-TRAINED-weights then adapt (CSA2 modes are trained from
+the start; Single-Pass mHC shifts a dependency and eats it in
+training). Dropping layers from the trained 2.1 student instead of
+generic byt5-small is the single-variable variant.
+
+## Experiment
+
+`ara-diac-small-lite2`: byte-identical to the run-009 spec except
+`student_init: /checkpoints/rababa_arabic_distill_small/run-007-r7-muon-6ep/best`
+(the shipped 2.1) — the layer_drop bridge then copies its kept layers.
+
+- Gate: full-set DER vs 5.78 (run-009) with paired bootstrap; also
+  compare against 2.1 (4.57) to locate the lite ceiling.
+- The Hebrew collapse (77.48) is a separate cross-lingual replication
+  failure with a known recipe confound (logit-KD 3ep vs Arabic
+  sequence-KD 6ep) — not evidence about this axis.
+
+## Why this ordering
+
+Init source is the one lever in the lite cell with a mechanism story
+(task-adapted features survive the drop better than generic ones) and
+a cost of one run. If it fails, the lite frontier is recipe-bound and
+/10 (lexical memory) becomes the remaining architecture lever.
+
+## Acceptance
+
+- [ ] Spec entry committed (the existing layer_drop bridge is the init builder)
+- [ ] Run launched detached with retry watchdog; one watcher
+- [ ] Full-set verdict with intervals recorded in RESULTS.md
+- [ ] Decision: lite tier replaced or negative recorded
+
+## Status
+
+- [ ] Build init on volume
+- [ ] Launch
+- [ ] Verdict

TODO.impl/05-kv-int8-runtime.md

@@ -0,0 +1,43 @@
+# 05 — int8 KV cache in the decode path (P2, contract-gated)
+
+Source: TODO.qwen-next/10 §5. DeepSeek's cache precision hierarchy —
+main KV at FP4, the sensitive SWA branch at FP8, head fp32 — matches
+our head-fp32 discipline and points at the one runtime tensor we
+still keep at fp32: the decode-path KV cache. Caching K/V at int8
+with per-head scales halves decode memory bandwidth; in the browser,
+bandwidth is the binding cost.
+
+## Design constraints
+
+- This changes the IMF artifact surface (decoder-kv.onnx past/present
+  dtypes, or scale tensors beside them) — it is an IMF v1 → v2
+  question, not a silent tweak. All three runtimes (TS, Python,
+  Ruby) must load both old and new zips, or the index pins a
+  minimum-runtime version.
+- The claim is quality-level parity for quantized artifacts (the
+  golden-v1 scoping): gate with per-model cer_delta + decode-health
+  smoke, NOT byte parity — the int8-weights story already established
+  that contract.
+- Keep head fp32 (unchanged discipline).
+
+## Deliverables
+
+1. Export-side experiment: one model exported with int8 KV pasts +
+   per-head scales; measure cer_delta and decode speed on the
+   artifact scorer.
+2. If delta clears the existing quantized gates: spec the zip
+   manifest fields (`kv_quant: int8-head`), runtime loader support
+   behind a capability flag, and an index/runtime pin plan.
+3. Browser wall-clock measurement before/after on the playground
+   tier (the 95 MB lite model first).
+
+## Acceptance
+
+- [ ] Experiment artifact scored; cer_delta recorded
+- [ ] Speed measurement recorded (decode tokens/sec, before/after)
+- [ ] Contract decision documented (v2 fields or reject) — owner gate
+- [ ] If adopted: all-runtime loading + golden rows smoke
+
+## Status
+
+- [ ] Awaiting 01/02 results before spending export effort; spec only

TODO.impl/06-sinkhorn-update.md

@@ -0,0 +1,41 @@
+# 06 — Sinkhorn-balanced momentum update for embedding/head (P2)
+
+Source: TODO.qwen-next/10 §6, DeepSeek Alg. 1. Replaces Adam for
+embedding tables and the prediction head: Nesterov momentum,
+alternating row/column L2 normalization (odd K steps), near-zero row
+masking, sqrt(n) RMS conversion, learning-rate correction gamma=0.18,
+no weight decay. Momentum-only state (vs Adam's two moments) and
+"empirically outperforming Adam" at their scale — unproven at ours,
+which is the point of measuring it.
+
+We already own log-domain Sinkhorn machinery from the mHC work
+(direct division NaNs for ~10% of inits — the log-domain lesson
+carries to the row/col normalizations here if magnitudes get small).
+
+## Deliverables
+
+1. `src/gpu/sinkhorn_update.py`: the update as a pure function over
+   (weight, grad, momentum state) + an optimizer wrapper selecting it
+   for embedding/head parameter patterns by name — OCP: the training
+   loop keeps AdamW/Muon dispatch, this registers as another
+   treatment.
+2. Unit tests with real tensors: masked rows stay masked; unit
+   row-RMS after balancing; momentum state evolution; the sqrt(n)
+   and gamma factors against a hand-computed 3x2 case.
+3. Config-gated use in a distill spec (one run) if the owner wants
+   the quality datapoint; otherwise the implementation + tests land
+   ready.
+
+## Acceptance
+
+- [ ] Update implemented exactly per Alg. 1 (K=11, tau=1e-3,
+      eps=1e-20, gamma=0.18, Nesterov beta, no weight decay)
+- [ ] Unit tests green on real tensors
+- [ ] Spec entry (off by default) for the optional measurement run
+- [ ] No change to any shipped run's reproducibility
+
+## Status
+
+- [ ] Implementation
+- [ ] Tests
+- [ ] Optional run decision