diff --git a/TODO.arabic/10-engram-episodic-memory.md b/TODO.arabic/10-engram-episodic-memory.md index bd23bb9..30c89c1 100644 --- a/TODO.arabic/10-engram-episodic-memory.md +++ b/TODO.arabic/10-engram-episodic-memory.md @@ -46,3 +46,29 @@ Engram store is updated during training (FIFO + importance sampling). ## Open questions - Capacity: 10K is from the paper. For our smaller model, sweep 1K/5K/10K. + +## CORRECTION (2026-09-11, from the DeepSeek-V4.1-Flash report §2.4.2) + +The mechanism described above (retrieving similar past training +examples) is NOT Engram. The actual design (arXiv:2601.07372): +hash-addressed n-gram embedding tables summed into the forward pass — +no example retrieval, no similarity search, no FIFO store. + +- addresses = n-gram hashes of the input token sequence +- orders {2,3,4}, 8 hash heads, 2048-dim per order +- table sizes are distinct primes (~16M entries per head in DS's + 196B-parameter deployment) +- FP8 storage, modules placed at layers 1 and 14 +- 5x learning rate for the tables; table updates use the + Sinkhorn-balanced momentum rule (momentum + alternating row/col + normalization), not Adam +- effect: memorization decoupled from compute — deterministic lookup, + prefetchable, no bandwidth interaction with attention + +The class-imbalance rationale above (rare haraqat reinforced by +retrieval) does not transfer to the real mechanism; the applicable +rationale is lexical idiom: haraqat are strongly n-gram-driven, and a +lookup table captures idiosyncratic lexical patterns without +compute. Any wire-in must be rebuilt against the real design; the +browser size ceiling (int8 4M x 64-dim ~ 64 MB) bounds the table. +Tracked at TODO.impl/10. diff --git a/TODO.arabic/18-per-head-muon-wirein.md.md b/TODO.arabic/18-per-head-muon-wirein.md.md index 44ee18d..16df79c 100644 --- a/TODO.arabic/18-per-head-muon-wirein.md.md +++ b/TODO.arabic/18-per-head-muon-wirein.md.md @@ -1,17 +1,21 @@ # Per-Head Muon wire-in -## Status -Module is built (see v0.5.0 shipped). Wire-in needs: -- Config flag in YAML -- Dispatch in training loop or model builder -- End-to-end smoke test -- Spec for the wire-in +## Status (2026-09-11) +Implemented in ml-models (this note was stale — no module had shipped +in v0.5.0): `src/gpu/muon.py` carries the headwise group +(`add_headwise_group`, per-slice Newton-Schulz, `qk_named` selector), +config-gated by the `headwise_muon` spec flag, off by default. Specs: +`ml-models/tests/test_muon_headwise.py` (5 tests, green) — per-head +update equals vanilla Muon on each head slice; heads=1 equals vanilla +whole; off-state routing unchanged. Externally validated by +DeepSeek-V4.1-Flash sec 2.5, GLM-5, Kimi-K3 (TODO.impl/03). +Remaining: exercise on the next teacher run (flag on). ## Acceptance -- [ ] Feature is enabled by config flag, off by default -- [ ] Spec covers the wire-in dispatch +- [x] Feature is enabled by config flag, off by default +- [x] Spec covers the wire-in dispatch (tests + flag wiring) - [ ] End-to-end training run completes with feature enabled -- [ ] No regression on baseline (feature off → identical results) +- [x] No regression on baseline (feature off → identical results) ## Files - (TBD based on feature) diff --git a/TODO.arabic/19-n-stream-mhc-wirein.md.md b/TODO.arabic/19-n-stream-mhc-wirein.md.md index 88a5eef..2a25359 100644 --- a/TODO.arabic/19-n-stream-mhc-wirein.md.md +++ b/TODO.arabic/19-n-stream-mhc-wirein.md.md @@ -15,3 +15,13 @@ Module is built (see v0.5.0 shipped). Wire-in needs: ## Files - (TBD based on feature) + +## Deployable variant (added 2026-09-11, DeepSeek-V4.1-Flash §2.4.1) + +Single-Pass mHC: consume the PREVIOUS block's input-mixing coefficients +(A_{l-1}) instead of the current block's. This removes the reduction +dependency so residual update, input mixing, and coefficient prediction +fuse into one kernel — halving activation memory traffic at "negligible +degradation." If n-stream mHC resumes, train with the shift from the +start so the network adapts to it (their pattern: architecture change +plus training-aware adaptation, never architecture change alone). diff --git a/TODO.hebrew/11-engram-wirein.md b/TODO.hebrew/11-engram-wirein.md index 07a1f28..ed243e6 100644 --- a/TODO.hebrew/11-engram-wirein.md +++ b/TODO.hebrew/11-engram-wirein.md @@ -1,7 +1,10 @@ # Engram wire-in -Same pattern as Arabic equivalent. See Arabic TODO.arabic/ for shared -infrastructure. +Same pattern as Arabic equivalent — and the same correction applies: +see TODO.arabic/10 (2026-09-11). Engram is a hash-addressed n-gram +embedding table summed into the forward pass, not example retrieval; +any Hebrew wire-in must be built against the real mechanism +(TODO.impl/10). ## Acceptance - [ ] Feature enabled for Hebrew diff --git a/TODO.impl/01-speculative-runtime.md b/TODO.impl/01-speculative-runtime.md new file mode 100644 index 0000000..173f119 --- /dev/null +++ b/TODO.impl/01-speculative-runtime.md @@ -0,0 +1,55 @@ +# 01 — Speculative decoding across the tier ladder (P0) + +Source: TODO.qwen-next/10 §1 (unparks TODO.qwen-next/04). The parked +objection — "the draft model would itself need training" — expired +when ara-diac-layerdrop-1.0-int4 (190M, 95 MB) shipped: drafter and +verifier (ara-diac-small-2.1-int8, 300M) share the byte table and +greedy decode, and greedy verification is output-preserving, so the +artifacts need no parity story — only a speed one. + +## Why + +Byte-level decode is one token per step; the browser tier pays it in +wall-clock. DSpark's shape (draft K, verify all K in one pass, +confidence-scheduled K) maps directly onto two shipped models. Early +probe rows: acceptance 0.98–1.00, 8.5–8.9 tokens per verifier pass at +K=8 — the int4 drafter matches the int8 verifier's argmax almost +everywhere, consistent with the 0.036% near-tie flip measurement. + +## Deliverables + +1. **Acceptance probe** (CPU, golden-v1 Arabic rows): per-row + acceptance rate, tokens/verify, exactness of the speculative loop + vs the verifier's plain-path greedy (same execution path — the + theorem check). Script: + `ml-models/scripts/probe_speculative.py`, results to + `~/ml-logs/spec_probe/results.json`. +2. **Runtime implementation** (interscript-ts): a + `SpeculativeSession` composing drafter + verifier IMFModel + instances — OCP: `translate()` untouched, the strategy is a new + class; block size is config data, not code. Unit tests on the + acceptance algorithm (synthetic logits are data, not model mocks); + e2e opt-in with real zips. +3. **Tier decision**: if full-probe acceptance clears ~0.9, expose + "2.1 quality at lite-ish latency" as a runtime option (not a new + artifact). Publish the acceptance measurement next to the tier in + docs. + +## Acceptance + +- [ ] Probe: 25 rows measured; exactness holds on every checked row + (plain-path reference; long rows may skip the O(T^2) reference, + recorded as unchecked) +- [ ] Probe bug class closed: EOS-block acceptance appends pre-EOS + tokens; bonus index computed post-extend +- [ ] TS `SpeculativeSession` with unit + opt-in e2e tests, PR merged +- [ ] Numbers recorded in TODO.qwen-next/10 §1 and RESULTS.md if the + tier ships +- [ ] Probe script landed in ml-models with its results + +## Status + +- [x] Probe v1 written; EOS-acceptance and index bugs found and fixed +- [ ] Probe rerun to completion +- [ ] Runtime implementation +- [ ] Tier exposure decision diff --git a/TODO.impl/02-multi-teacher-domain-routing.md b/TODO.impl/02-multi-teacher-domain-routing.md new file mode 100644 index 0000000..72530f0 --- /dev/null +++ b/TODO.impl/02-multi-teacher-domain-routing.md @@ -0,0 +1,64 @@ +# 02 — Multi-teacher domain-routed distillation (P0) + +Source: TODO.qwen-next/10 §2. DeepSeek's final OPD stage distills from +40+ heterogeneous teachers, best-per-domain; we have always used one +teacher per run. Our residual program closed corpus volume, domain +swap, and on-policy GKD — a teacher mixture is the cheapest untested +probe of the open attribution ("the residual lives in the +teacher–student interaction"). + +## Measured so far (2026-09-10, released run-007 predictions) + +Protocol check: re-aggregated slices reproduce the published 2.289 +(teacher) / 4.5701 (student) exactly. + +| domain | n | teacher r7 | student 2.1 | gap | +|---|---|---|---|---| +| Fadel_test (classical) | 600 | 1.36 | 2.36 | 1.00 | +| Our-Benchmark (news) | 454 | 3.21 | 6.08 | 2.87 | +| WikiNewsTruth | 146 | 2.08 | 6.37 | 4.29 | + +The old attribution inverts: the residual is NOT classical coverage — +the student tracks r7 tightly on classical. The gap lives on news and +wiki rows, which also explains why classical-corpus swaps measured +flat: they targeted the domain where the student is already closest. + +## Deliverables + +1. r6 teacher predictions on the same 1200 rows under the published + windowed protocol (`ml-models/src/gpu/modal_teacher_sadeed.py`, + output `/checkpoints/probes/r6_sadeed_preds.jsonl`). +2. r6 per-domain slice vs r7's; record both profiles in the memo. +3. Verdict: if r6 beats r7 on the news/wiki slices by a margin the + paired bootstrap separates, arm a routed-teacher distill spec + (supervision per row from that domain's winner) and launch; + otherwise record the negative and close the axis. + +## Acceptance + +- [ ] r6 preds fetched and sliced; both teacher profiles in + TODO.qwen-next/10 §2 +- [ ] Routed arm launched with an explicit single-variable spec + (same student shape, same epochs as 2.1), or the negative + recorded with intervals +- [ ] No subset-overstatement: any routed-run claim is full-set + +## Verdict (2026-09-11): NEGATIVE — axis closed for the r6/r7 pair + +| domain | n | r6 | r7 | winner | +|---|---|---|---|---| +| Fadel_test (classical) | 600 | 1.37 | 1.36 | tie | +| Our-Benchmark (news) | 454 | 3.52 | 3.21 | r7 | +| WikiNewsTruth | 146 | 3.29 | 2.08 | r7 | + +r7 dominates everywhere it wins full-set. No routed mixture of the +available teachers can beat plain r7 supervision. The residual is +student-side (news/wiki rows), not teacher-selection. Remaining +levers: 04 (trained-lite), 10 (lexical memory), 01 (runtime). + +## Status + +- [x] r7 per-domain slice (from released predictions, zero GPU) +- [x] r6 preds on volume, fetched, sliced +- [x] Verdict: negative, recorded in TODO.qwen-next/10 §2 +- [x] No routed launch (nothing to route) diff --git a/TODO.impl/03-headwise-muon.md b/TODO.impl/03-headwise-muon.md new file mode 100644 index 0000000..7dd3239 --- /dev/null +++ b/TODO.impl/03-headwise-muon.md @@ -0,0 +1,39 @@ +# 03 — Per-head Muon wire-in for Q/K (P1) + +Source: TODO.qwen-next/10 §3. DeepSeek-V4.1-Flash splits Query and +Key weight matrices by head before the Muon update ("head-wise +Muon"), reporting it over vanilla Muon; GLM-5 and Kimi-K3 validate +the same choice. Our module exists (v0.5.0); the wire-in never ran +(TODO.arabic/18 acceptance list is unchecked). Three-lab convergence +upgrades it to the default optimizer treatment for Q/K in the next +teacher run. + +## Why + +Muon's single preconditioner spans all attention heads; per-head +splitting gives each head its own preconditioner, matching head +heterogeneity. Cost: one reshape per step. It is a recipe change, +not an architecture change — single-variable, cheap to ablate. + +## Deliverables + +1. `src/gpu/muon.py`: per-head view application for 2-D Q/K weights + (reshape to [heads, d_head * in, ...] per the module's existing + convention), config-gated, off by default. +2. Unit test with real tensors: per-head update equals vanilla Muon + applied per head slice; off-state bit-identical to today's path. +3. Wire-in spec for the next teacher run (r8-class): flag on, log + line proves the per-head path executed. + +## Acceptance + +- [ ] Config flag exists, default off; no behavior change when off +- [ ] Unit tests green on real tensors (not doubles) +- [ ] Flag-on path exercised end-to-end in a smoke run +- [ ] TODO.arabic/18 updated to point here + +## Status + +- [ ] Inspect muon.py for the built module's exact API +- [ ] Implementation + tests +- [ ] Smoke run diff --git a/TODO.impl/04-trained-lite.md b/TODO.impl/04-trained-lite.md new file mode 100644 index 0000000..c8c4c38 --- /dev/null +++ b/TODO.impl/04-trained-lite.md @@ -0,0 +1,43 @@ +# 04 — lite-2.0: init the depth cut from the trained student (P1) + +Source: TODO.qwen-next/10 §4. CORRECTED 2026-09-11 (spec audit): +the shipped lite rung (run-009-layerdrop-6ep, 5.78) IS a trained +6-epoch sequence-KD distill — its layer-drop INIT copies kept layers +verbatim from generic pretrained byt5-small. The untested variable in +the lite cell is the INIT SOURCE: DeepSeek's repeated pattern is +compress-from-TRAINED-weights then adapt (CSA2 modes are trained from +the start; Single-Pass mHC shifts a dependency and eats it in +training). Dropping layers from the trained 2.1 student instead of +generic byt5-small is the single-variable variant. + +## Experiment + +`ara-diac-small-lite2`: byte-identical to the run-009 spec except +`student_init: /checkpoints/rababa_arabic_distill_small/run-007-r7-muon-6ep/best` +(the shipped 2.1) — the layer_drop bridge then copies its kept layers. + +- Gate: full-set DER vs 5.78 (run-009) with paired bootstrap; also + compare against 2.1 (4.57) to locate the lite ceiling. +- The Hebrew collapse (77.48) is a separate cross-lingual replication + failure with a known recipe confound (logit-KD 3ep vs Arabic + sequence-KD 6ep) — not evidence about this axis. + +## Why this ordering + +Init source is the one lever in the lite cell with a mechanism story +(task-adapted features survive the drop better than generic ones) and +a cost of one run. If it fails, the lite frontier is recipe-bound and +/10 (lexical memory) becomes the remaining architecture lever. + +## Acceptance + +- [ ] Spec entry committed (the existing layer_drop bridge is the init builder) +- [ ] Run launched detached with retry watchdog; one watcher +- [ ] Full-set verdict with intervals recorded in RESULTS.md +- [ ] Decision: lite tier replaced or negative recorded + +## Status + +- [ ] Build init on volume +- [ ] Launch +- [ ] Verdict diff --git a/TODO.impl/05-kv-int8-runtime.md b/TODO.impl/05-kv-int8-runtime.md new file mode 100644 index 0000000..39b22d0 --- /dev/null +++ b/TODO.impl/05-kv-int8-runtime.md @@ -0,0 +1,43 @@ +# 05 — int8 KV cache in the decode path (P2, contract-gated) + +Source: TODO.qwen-next/10 §5. DeepSeek's cache precision hierarchy — +main KV at FP4, the sensitive SWA branch at FP8, head fp32 — matches +our head-fp32 discipline and points at the one runtime tensor we +still keep at fp32: the decode-path KV cache. Caching K/V at int8 +with per-head scales halves decode memory bandwidth; in the browser, +bandwidth is the binding cost. + +## Design constraints + +- This changes the IMF artifact surface (decoder-kv.onnx past/present + dtypes, or scale tensors beside them) — it is an IMF v1 → v2 + question, not a silent tweak. All three runtimes (TS, Python, + Ruby) must load both old and new zips, or the index pins a + minimum-runtime version. +- The claim is quality-level parity for quantized artifacts (the + golden-v1 scoping): gate with per-model cer_delta + decode-health + smoke, NOT byte parity — the int8-weights story already established + that contract. +- Keep head fp32 (unchanged discipline). + +## Deliverables + +1. Export-side experiment: one model exported with int8 KV pasts + + per-head scales; measure cer_delta and decode speed on the + artifact scorer. +2. If delta clears the existing quantized gates: spec the zip + manifest fields (`kv_quant: int8-head`), runtime loader support + behind a capability flag, and an index/runtime pin plan. +3. Browser wall-clock measurement before/after on the playground + tier (the 95 MB lite model first). + +## Acceptance + +- [ ] Experiment artifact scored; cer_delta recorded +- [ ] Speed measurement recorded (decode tokens/sec, before/after) +- [ ] Contract decision documented (v2 fields or reject) — owner gate +- [ ] If adopted: all-runtime loading + golden rows smoke + +## Status + +- [ ] Awaiting 01/02 results before spending export effort; spec only diff --git a/TODO.impl/06-sinkhorn-update.md b/TODO.impl/06-sinkhorn-update.md new file mode 100644 index 0000000..bc35c52 --- /dev/null +++ b/TODO.impl/06-sinkhorn-update.md @@ -0,0 +1,41 @@ +# 06 — Sinkhorn-balanced momentum update for embedding/head (P2) + +Source: TODO.qwen-next/10 §6, DeepSeek Alg. 1. Replaces Adam for +embedding tables and the prediction head: Nesterov momentum, +alternating row/column L2 normalization (odd K steps), near-zero row +masking, sqrt(n) RMS conversion, learning-rate correction gamma=0.18, +no weight decay. Momentum-only state (vs Adam's two moments) and +"empirically outperforming Adam" at their scale — unproven at ours, +which is the point of measuring it. + +We already own log-domain Sinkhorn machinery from the mHC work +(direct division NaNs for ~10% of inits — the log-domain lesson +carries to the row/col normalizations here if magnitudes get small). + +## Deliverables + +1. `src/gpu/sinkhorn_update.py`: the update as a pure function over + (weight, grad, momentum state) + an optimizer wrapper selecting it + for embedding/head parameter patterns by name — OCP: the training + loop keeps AdamW/Muon dispatch, this registers as another + treatment. +2. Unit tests with real tensors: masked rows stay masked; unit + row-RMS after balancing; momentum state evolution; the sqrt(n) + and gamma factors against a hand-computed 3x2 case. +3. Config-gated use in a distill spec (one run) if the owner wants + the quality datapoint; otherwise the implementation + tests land + ready. + +## Acceptance + +- [ ] Update implemented exactly per Alg. 1 (K=11, tau=1e-3, + eps=1e-20, gamma=0.18, Nesterov beta, no weight decay) +- [ ] Unit tests green on real tensors +- [ ] Spec entry (off by default) for the optional measurement run +- [ ] No change to any shipped run's reproducibility + +## Status + +- [ ] Implementation +- [ ] Tests +- [ ] Optional run decision diff --git a/TODO.impl/07-diacritization-depth-knob.md b/TODO.impl/07-diacritization-depth-knob.md new file mode 100644 index 0000000..08091dc --- /dev/null +++ b/TODO.impl/07-diacritization-depth-knob.md @@ -0,0 +1,37 @@ +# 07 — Diacritization-depth conditioning (P3, product decision) + +Source: TODO.qwen-next/10 §6. DeepSeek trains a scalar reasoning +effort (1–100) as an input-side signal; one checkpoint then serves +low/high/max tiers by conditioning. The analogous product surface for +us: one Arabic model with a depth control — none / light / full +tashkeel — instead of users post-filtering full output (which is what +"light" consumers do today). + +## Why gated + +- Needs a new student run with conditioning baked in (input-side + marker or reserved control bytes) and a training corpus labeled at + both depths; classical sources are fully voweled, so the "light" + labels must be derived (strip a haraqat subset), which is a + labeling decision with quality consequences. +- Product question first: do users ask for light tashkel from us, or + is full output + client-side filtering enough? + +## If built + +- Control as data: a reserved input token (never a code path). +- Training: same recipe as the next student run, plus a depth field + in the batch; eval protocol gains per-depth DER (light rows scored + after the same projection). +- Runtime: an optional second argument, defaulted to full — OCP at + the API boundary. + +## Acceptance (only after the owner approves the product surface) + +- [ ] Depth-label derivation rule documented and spot-checked +- [ ] One conditioned run, full-set per-depth verdicts +- [ ] API surface spec across TS/Python/Ruby + +## Status + +- [ ] Owner decision requested (registered from the memo, not built) diff --git a/TODO.impl/08-golden-closeout.md b/TODO.impl/08-golden-closeout.md new file mode 100644 index 0000000..6f3cdb5 --- /dev/null +++ b/TODO.impl/08-golden-closeout.md @@ -0,0 +1,36 @@ +# 08 — golden-v1 closeout (P0) + +The remaining loose ends of the golden-matrix work (measured +2026-09-07): the cross-runtime test's final form, the parity wording +in the paper, and the release annotation. All three say the same +thing — the guarantee is precision-scoped: fp32/fp16 byte-stable +cross-hardware; int8/int4 quality-level (cer_delta gates) with +near-tie flips across architectures at stop points AND mid-text. + +## Deliverables + +1. `ml-models/runtime/tests/test_model.py::test_golden_matrix` final + form: fp32 rows byte-exact; quantized rows decode-health only + (length floor). One comment block explaining the scoping (the + staged fix had two overlapping comments — deduplicate). Verified + green locally with GOLDEN_DIR before the PR. +2. `docs/paper-c.adoc`: parity claim scoped by precision (byte parity + for fp32/fp16; quality parity for quantized), matching + How-we-measure and RESULTS.md. +3. golden-v1 release notes: one-line annotation of the same scoping + so the release states its own guarantee. + +## Acceptance + +- [ ] Test green locally over all 12 golden files +- [ ] PR merged (ml-models) with test + paper wording +- [ ] Release body annotated +- [ ] No OTHER claim text contradicts the scoping (grep the repo for + "byte-identical" near quantized artifacts) + +## Status + +- [x] Test fix written (this session, uncommitted) +- [ ] Dedupe comment, local green run +- [ ] PR +- [ ] Release annotation diff --git a/TODO.impl/09-doc-corrections.md b/TODO.impl/09-doc-corrections.md new file mode 100644 index 0000000..2aae371 --- /dev/null +++ b/TODO.impl/09-doc-corrections.md @@ -0,0 +1,31 @@ +# 09 — External-reference doc corrections (P2) + +Two rababa notes describe mechanisms incorrectly relative to their +sources. Both were written from summaries, not the papers; the +DeepSeek V4.1-Flash report quotes the actual designs. + +## Fixes + +1. **Engram (TODO.arabic/10, TODO.hebrew/11)**: described as + "retrieving similar past training examples" (episodic replay). + The actual mechanism (arXiv:2601.07372; DS V4.1-Flash §2.4.2) is + hash-addressed n-gram embedding tables summed into the forward + pass: orders {2,3,4}, 8 hash heads, 2048-dim per order, distinct + prime table sizes, FP8 storage, modules at layers 1 and 14, 5x LR, + tables updated by the Sinkhorn-balanced rule. Memorization + decoupled from compute — no example retrieval at all. +2. **mHC single-pass shift (TODO.arabic/19)**: add the report's + deployable variant — each block consumes the PREVIOUS block's + input-mixing coefficients (A_{l-1}), killing the kernel dependency + and halving activation memory traffic at "negligible degradation" + (§2.4.1). Adopt if n-stream mHC resumes. + +## Acceptance + +- [ ] Both Engram notes corrected with the real mechanism + config +- [ ] mHC note carries the single-pass shift +- [ ] Both land in the same rababa PR as the TODO.impl ledger + +## Status + +- [ ] Edits diff --git a/TODO.impl/10-engram-lexical-memory.md b/TODO.impl/10-engram-lexical-memory.md new file mode 100644 index 0000000..905cf76 --- /dev/null +++ b/TODO.impl/10-engram-lexical-memory.md @@ -0,0 +1,39 @@ +# 10 — Engram-style lexical memory for diacritization (P3, gated) + +Source: TODO.qwen-next/10 §6. The one architecture-level lever the +DeepSeek pass surfaced that fits our open question — the residual is +interaction-level and the corpus axis is closed, so architecture is +the remaining category. Haraqat are heavily lexical/n-gram driven; +a hashed n-gram lookup table is a direct fit for the idiom memory a +300M byte model lacks. + +## Why gated + +- Changes export and runtime: gather ops over a table inside the ONNX + graph, table shipped in the zip. Browser size caps the table — + int8 4M-entry x 64-dim (~64 MB) is the ceiling before the lite + tier's size story breaks. +- Sequencing: only worth building if 02 (teacher mixture) and 04 + (trained lite) also fail to move the frontier — otherwise it + spends the architecture budget before the cheap cells are closed. +- Corrected mechanism per 09: hash-addressed tables in the forward + pass (DeepSeek config), not example retrieval. + +## If built + +- Table as a parameter module (addresses = n-gram hashes of the input + bytes; embeddings summed at the two chosen layers); trained with + the distill recipe; Sinkhorn-balanced table updates (06) are the + natural pairing, as in the report. +- Export probe first: a table-augmented student exported to ONNX with + decode-health smoke before any quality run. + +## Acceptance (only after the gate) + +- [ ] Size/latency budget for the browser tier documented +- [ ] Table module + export probe +- [ ] One run, full-set verdict with intervals + +## Status + +- [ ] Gated behind 02/04 results; not built diff --git a/TODO.impl/README.md b/TODO.impl/README.md new file mode 100644 index 0000000..e8f39fc --- /dev/null +++ b/TODO.impl/README.md @@ -0,0 +1,31 @@ +# TODO.impl — remaining implementation work from the DeepSeek V4.1-Flash pass + +Source: TODO.qwen-next/10 (the report memo) plus the session's +open closeouts. One file per workstream, MECE by deliverable. Every +file carries Why / Acceptance / Status; nothing ships without its +measurement recorded. + +Priorities: + +- **P0** — in flight or closes an open published gap +- **P1** — validated recipe for the next training run +- **P2** — ready to build, no contract change without a gate +- **P3** — gated candidates; need an owner decision or a P0/P1 result + +| # | file | deliverable | priority | status | +|---|---|---|---|---| +| 01 | speculative-runtime.md | acceptance probe + TS SpeculativeSession + tier decision | P0 | probe running; runtime pending | +| 02 | multi-teacher-domain-routing.md | r6/r7 per-domain slice + routed distill verdict | P0 | r6 preds running on Modal | +| 03 | headwise-muon.md | per-head Muon wire-in for Q/K | P1 | module built, wire-in pending | +| 04 | trained-lite.md | lite-2.0: distill-trained 6-layer encoder | P1 | spec + launch | +| 05 | kv-int8-runtime.md | int8 KV cache in the decode path | P2 | spec; IMF contract gate | +| 06 | sinkhorn-update.md | Sinkhorn-balanced optimizer for embedding/head | P2 | implement + unit spec | +| 07 | diacritization-depth-knob.md | depth-conditioned vocalization | P3 | product decision | +| 08 | golden-closeout.md | golden test PR, paper-c wording, release README note | P0 | fix staged locally | +| 09 | doc-corrections.md | Engram mechanism fix, mHC single-pass note | P2 | quick edits | +| 10 | engram-lexical-memory.md | hashed n-gram memory module | P3 | gated behind 02/04 | + +Standing rules that apply to every item: full-set measurement or no +claim; quantized parity is quality-level, not byte-level; LLM teachers +for haraqat labels remain forbidden; GPU launches go detached with a +retry watchdog, one watcher per job. diff --git a/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md b/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md new file mode 100644 index 0000000..253ffc0 --- /dev/null +++ b/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md @@ -0,0 +1,167 @@ +# 10 — DeepSeek-V4.1-Flash learnings, mapped to our stack + +Source: DeepSeek_V41_Tech_Report.pdf +(huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash, fetched 2026-09-10). +Model class: 552B multimodal MoE, 8B active prefill / 16B decode, +1M context. The report's subject is KV-cache compression at serving +scale — not our problem — but five of its components and two of its +recipe claims map onto levers we have open, and one unparks a parked +probe. Recorded so we don't re-derive it. + +## 1. Speculative decoding — UNPARK 04 (the parked objection is void) + +TODO 04 parked draft/verify because "the draft model would itself need +training + a parity story — not free." That objection expired: we now +ship the drafter. ara-diac-layerdrop-1.0-int4 (190M, 95 MB) and +ara-diac-small-2.1-int8 (300M) share the byte vocabulary, byte table, +and greedy decode; both are released, checksummed, and parity-gated. + +DeepSeek's DSpark (§2.4.3) adds the design shape worth copying: +a small drafter emits K tokens in one pass, the verifier checks all K +in one pass, and a confidence estimate schedules K adaptively. With +greedy verification the verifier's argmax is authoritative, so outputs +are byte-identical to non-speculative greedy — no parity story needed, +just a speed one. + +- Step 1 (no training, CPU-only): acceptance probe on golden-v1 rows. + Run both models, measure per-position argmax agreement and the + distribution of first-divergence offsets. Consonant positions should + agree ~100% (copy-through dominates); disagreement concentrates on + haraqat near-ties — the same margins that flip under quantization. + Net speedup = agreement rate × (drafter cost / verifier cost); the + int4 190M vs int8 300M ratio is favorable. +- Step 2 (if step 1 clears ~80% agreement): K-draft batch-verify in + the runtime KV decoder (TS worker first, it has the latency story). +- Honest scope: this speeds the 2.1 tier. The lite tier stays as the + standalone fast path; speculative decode is a middle tier that only + pays when users want 2.1 quality at lower latency. + +## 2. Multi-teacher domain-routed distillation — cheap untested lever + +§5.2.4: the final OPD stage trains from 40+ architecturally +heterogeneous teachers, selecting the best teacher *per domain* ("the +best teacher for each domain may come from a different stage of model +development"). This is standard practice for them, not research. + +Our residual program closed corpus volume, domain swap, and on-policy +GKD — but never a teacher *mixture*. r6 and r7 are both measured +full-set (2.5793 / 2.29), and frontier-predictions-v1 already ships +both teachers' raw predictions. + +- Step 1 result (2026-09-11, measured): r6 was run over the same + 1200 rows under the windowed protocol (Modal probe, preds at + /checkpoints/probes/ on rababa-checkpoints) and sliced by the + benchmark's filename domains. r7 wins EVERY domain — classical a + tie (r6 1.37 / r7 1.36), news r7 (3.52/3.21), wiki r7 strongly + (3.29/2.08). r6 full-set re-scores 2.5997 vs the published 2.5793 + (0.02pp protocol-window variance, same model). The routing + hypothesis is NEGATIVE for the available teacher pair: no domain + exists where routing to r6 helps. Combined with the student-side + slice (gap concentrated on news/wiki, NOT classical), the residual + is neither classical coverage nor teacher selection — the remaining + category is student-side: capacity/interaction (TODO.impl/04 + trained-lite, /10 lexical memory) or runtime (TODO.impl/01). + +## 3. Head-wise Muon — external validation, wire-in already pending + +§2.5: Q/K weights split per head before the Muon update, "outperforms +vanilla Muon," independently validated in GLM 5 and Kimi-K3. We built +the module in v0.5.0; the wire-in is still pending acceptance criteria +(TODO.arabic/18). Three-lab external validation upgrades this from +"nice to have" to "default for the next teacher run." No new code — +flip the config flag, run the spec. + +## 4. Layer compression must be trained, not pruned — reframes the Hebrew collapse + +CSA2's Reuse-mode layers (§2.3.1) share KV computed by other layers — +functionally a depth cut — but the mode assignment is static *during +training*, so the network adapts around what it will and won't +compute. CORRECTION (2026-09-11, spec audit): our lite rung is NOT an +untrained copy — run-009-layerdrop-6ep is a full 6-epoch sequence-KD +distill whose INIT is a verbatim kept-layer copy from byt5-small. +"Verbatim layer copy" describes the init bridge, not absent training. +What is actually untested in the lite cell: the INIT SOURCE. run-009 +drops layers from generic pretrained byt5-small; DeepSeek's pattern +(compress from TRAINED weights, then adapt) suggests dropping from the +trained 2.1 student instead. + +- Candidate: lite-2.0 = layer-drop init from + run-007-r7-muon-6ep/best (the shipped 2.1), then the identical + 6-epoch distill. Single variable vs run-009's 5.78: init source. +- The Hebrew collapse (77.48) stands as a cross-lingual depth + replication failure with a known recipe confound (logit-KD 3ep vs + Arabic sequence-KD 6ep), not as evidence about trained vs untrained + compression. + +## 5. KV-cache int8 in the runtime decoder — one runtime lever + +§2.4.4's precision hierarchy: main KV → FP4 (QAT), SWA KV → FP8 +("sensitivity"), head fp32. Our head-fp32 discipline already matches +their reasoning. Unexplored on our side: the runtime decoder's KV +cache is fp32 today; caching K/V at int8 with per-head scales would +halve decode memory bandwidth in the browser, where bandwidth is the +binding cost. Gate with the existing quantized contract (cer_delta + +decode-health smoke), not byte parity — same class of claim as +weights-quantization, already scoped by golden-v1. + +## 6. Smaller items + +- **Sinkhorn-balanced update for embedding + head (Alg. 1)**: replaces + Adam with momentum + Sinkhorn row/col normalization, momentum-only + buffer, "empirically outperforming Adam." K=11, τ=1e-3, γ=0.18. Our + byte-vocab embedding and head are tiny, so the memory saving is + irrelevant; the quality claim is untested at our scale. Our + log-domain Sinkhorn code (mHC work) transfers. Candidate + single-variable run, low priority. +- **mHC single-pass shift (§2.4.1)**: consume the previous block's + mixing coefficients (A_{l-1}) to kill the kernel dependency, + halving activation traffic; "negligible degradation." Adopt if we + ever resume n-stream mHC students (TODO.arabic/19). +- **Engram is a hash lookup, not retrieval (§2.4.2)**: TODO.arabic/10 + and TODO.hebrew/11 describe Engram as retrieving similar past + *training examples*. The actual mechanism is n-gram-hash-addressed + embedding tables summed into the forward pass (orders {2,3,4}, + 8 hash heads, 2048-dim per order, distinct prime table sizes, + FP8 storage, modules at layers 1 and 14, 5× LR, tables updated with + the Sinkhorn-balanced rule). 196B of their 748B parameters live + there — memorization decoupled from compute. For diacritization the + lexical-idiom story is real (haraqat are n-gram-driven), but browser + size caps the table (int8 4M-entry × 64-dim ≈ 64 MB is the ceiling). + Architecture-level lever; only worth it if §2's teacher-mixture also + fails, since it changes export and runtime. +- **Diacritization-depth conditioning**: the reasoning-effort scalar + (§5.1.4, effort 1–100 → low/high/max API tiers) suggests training a + depth control (none/light/full tashkeel) as an input-side signal — + one model, one knob, interpolated at inference. Product feature, + needs a new student run; user decision, registered for the roadmap. + +## 7. External validation for decisions already made + +- §4.1: model-generated content is treated as "implicit duplication" + and filtered from pretraining — machine-labeled data demoted, not + promoted. Validates the knesset-v6 (Dicta labels) weak-pretrain-only + decision for Hebrew. +- §5.1: "the marginal return of engineering the data and environment + pipeline substantially exceeds that of algorithmic novelty in + post-training." Consistent with our closed program: corpus and data + levers measured; what remains is interaction-level, which is why + §2 (teacher mixture) is the probe that fits. + +## 8. Not applicable, with reasons + +| Report component | Why not | +|---|---| +| CED, CSA2, cross-layer KV reuse | Long-context serving scale; our windows are ≤1400 B, KV storage is not a cost | +| SWA Bounded Replay, persistent KV mgmt, EPD disaggregation | LLM serving infra; no prefix-reuse workload | +| Hierarchical Sparse Indexer | Reduces indexer cost at 1M context; nothing to index at ours | +| FP4/MXFP4 formats | Hardware-GEMM formats; our targets are CPU/web ONNX (int8/int4 weights) | +| 1M-context pretraining, 45T tokens | Avoiding that scale is the point of the client tier | +| Agent task synthesis, DSec, multi-agent, effort-RL machinery | No RL anywhere in our pipeline; RL-for-diacritization is closed negative (knowledge-limited at SFT convergence) | +| Async OPD infrastructure | Our distillation is offline sequence/logit KD; no rollout phase | + +## Status + +- [ ] §1 step 1: acceptance probe on golden-v1 (CPU, no training) +- [ ] §2 step 1: per-domain teacher slice from released predictions +- [ ] §3: head-wise Muon wire-in spec executed on next teacher run +- [ ] §4-§6: registered as candidates, costed, awaiting user prioritization