From 75457effffe7a8ae95582b9fcb13fd7204894c94 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 12 Sep 2026 15:17:26 +0800 Subject: [PATCH] docs(TODO.impl): closeouts - lite2 negative, soup negative, framing pre-registered 04 closed negative (7.14 vs 5.78, init-source axis closed; depth reads 2-of-3 negative). 02 gains the soup postscript (2.4188, same-basin, no gain). 11 pre-registered: static activation scales as the framing fix, gates defined before measurement. README statuses synced. --- TODO.impl/01-speculative-runtime.md | 13 ++++-- TODO.impl/02-multi-teacher-domain-routing.md | 8 ++++ TODO.impl/04-trained-lite.md | 19 ++++++-- TODO.impl/11-static-int8-framing.md | 46 +++++++++++++++++++ TODO.impl/README.md | 17 +++---- .../10-deepseek-v4-1-flash-learnings.md | 20 ++++---- 6 files changed, 100 insertions(+), 23 deletions(-) create mode 100644 TODO.impl/11-static-int8-framing.md diff --git a/TODO.impl/01-speculative-runtime.md b/TODO.impl/01-speculative-runtime.md index 173f119..46360f9 100644 --- a/TODO.impl/01-speculative-runtime.md +++ b/TODO.impl/01-speculative-runtime.md @@ -50,6 +50,13 @@ everywhere, consistent with the 0.036% near-tie flip measurement. ## Status - [x] Probe v1 written; EOS-acceptance and index bugs found and fixed -- [ ] Probe rerun to completion -- [ ] Runtime implementation -- [ ] Tier exposure decision +- [x] Probe rerun to completion: 25/25 rows, mean acceptance 0.9886 + (min 0.955), 8.85 tokens/verify at K=8; all 18 exactness-checked + rows byte-identical to the verifier's plain-path greedy + (results: ~/ml-logs/spec_probe/results.json) +- [x] Runtime implementation: interscript-ts PR #77 (merged) - + SpeculativeModel + acceptBlock exported from interscript/ml; + unit tests + tiny-fixture session + real-pair e2e green +- [ ] Tier exposure: expose SpeculativeModel in the playground/API + surface (acceptance threshold cleared; site wiring pending) +- [x] Numbers recorded in TODO.qwen-next/10 and RESULTS.md diff --git a/TODO.impl/02-multi-teacher-domain-routing.md b/TODO.impl/02-multi-teacher-domain-routing.md index 72530f0..0faf801 100644 --- a/TODO.impl/02-multi-teacher-domain-routing.md +++ b/TODO.impl/02-multi-teacher-domain-routing.md @@ -56,9 +56,17 @@ available teachers can beat plain r7 supervision. The residual is student-side (news/wiki rows), not teacher-selection. Remaining levers: 04 (trained-lite), 10 (lexical memory), 01 (runtime). +## Soup postscript (2026-09-12): also negative + +The r6+r7 50/50 weight soup scored 2.4188 full-set (r6 2.5997, +r7 2.289) — same-basin (functional model, linear connectivity +confirmed) but strictly between the parents on every domain slice. +r7 remains the best available teacher; supervision unchanged. + ## Status - [x] r7 per-domain slice (from released predictions, zero GPU) - [x] r6 preds on volume, fetched, sliced - [x] Verdict: negative, recorded in TODO.qwen-next/10 §2 - [x] No routed launch (nothing to route) +- [x] Soup probe: negative (RESULTS.md 2026-09-12) diff --git a/TODO.impl/04-trained-lite.md b/TODO.impl/04-trained-lite.md index c8c4c38..6b9ae67 100644 --- a/TODO.impl/04-trained-lite.md +++ b/TODO.impl/04-trained-lite.md @@ -36,8 +36,21 @@ a cost of one run. If it fails, the lite frontier is recipe-bound and - [ ] Full-set verdict with intervals recorded in RESULTS.md - [ ] Decision: lite tier replaced or negative recorded +## Verdict (2026-09-12): NEGATIVE — 7.1402 vs run-009's 5.78 + +Same recipe, same canonical labels (sha e70ce991), same teacher +(re-scores 2.2921 in-run); the single variable - layer-drop init from +the trained 2.1 instead of generic byt5-small - made the student +1.36pp WORSE. Reading: generic pretraining keeps layers redundant; +task adaptation co-specializes them, and deleting half a co-adapted +stack breaks more computation. The depth axis now reads 2-of-3 +negative (Hebrew generic-init collapse; Arabic adapted-init collapse; +run-009's generic-init 5.78 stands). The lite tier is unchanged; the +remaining architecture lever is /10 (lexical memory), still gated. + ## Status -- [ ] Build init on volume -- [ ] Launch -- [ ] Verdict +- [x] Build init on volume (the layer_drop bridge, student_init=2.1) +- [x] Launch (checkpoint-resumed once; canonical labels seeded after + catching the re-labeling hazard) +- [x] Verdict: negative, recorded in RESULTS.md diff --git a/TODO.impl/11-static-int8-framing.md b/TODO.impl/11-static-int8-framing.md new file mode 100644 index 0000000..a76ef84 --- /dev/null +++ b/TODO.impl/11-static-int8-framing.md @@ -0,0 +1,46 @@ +# 11 — Static activation scales: fix decode framing at the source (P0) + +Pre-registered 2026-09-12, from the framing finding (RESULTS.md: +dynamic-int8 graphs compute activation quantization scales per fed +tensor, so single-step and batched decode framings produce materially +different decodes). The shipped int8 uses `quantize_dynamic` +(DynamicQuantizeLinear at runtime); static quantization +(`quantize_static` + calibration) bakes activation scales into the +graph and removes the per-run computation entirely. + +## Why this is the top lever + +If framing stabilizes, three separate findings improve at once: +- the speculative tier's acceptance collapse (0.99 → 0.46 purely by + framing) may reverse — its domain would reopen on CPU +- the cross-hardware near-tie flips (RESULTS 2026-09-07) should shrink: + one fewer runtime-computed quantity to diverge across machines +- the golden test could return to byte-exact assertions for int8 + +## Experiment (pre-registered gates) + +`scripts/static_int8_experiment.py`: re-quantize the 2.1 decoder-kv +graph with QUInt8 static activations (MatMul-only, head fp32 — same +node policy as the shipped dynamic int8), calibrated on 365 real decode +feeds sampled across BOTH framings (prefills, incremental steps, +8-token windows). + +- **A. framing equality**: the same greedy decode driven token-by-token + vs in 8-token batched calls — GATE: identical trajectories +- **B. drift vs fp32** (quality proxy): GATE: no worse than the + dynamic int8 artifact's relationship to fp32; full-set DER via the + artifact scorer if A clears +- **C. speed**: static vs dynamic tok/s on CPU — record, not gate + (u8s8 static kernels are usually comparable or faster) + +If A and B clear: re-export all quantized artifacts through the static +path (modal_export gains the calibration stage), index-v6, golden rows +regenerate, framing claim added to the contract. + +## Status + +- [x] Pre-registered (this file) +- [x] fp32 artifact fetched, sha-verified against index-v5 +- [x] Calibration corpus collected (365 feeds, both framings) +- [ ] quantize_static result measured (A/B/C) +- [ ] Verdict recorded in RESULTS.md; ledger cross-links updated diff --git a/TODO.impl/README.md b/TODO.impl/README.md index e8f39fc..cdf4493 100644 --- a/TODO.impl/README.md +++ b/TODO.impl/README.md @@ -14,16 +14,17 @@ Priorities: | # | file | deliverable | priority | status | |---|---|---|---|---| -| 01 | speculative-runtime.md | acceptance probe + TS SpeculativeSession + tier decision | P0 | probe running; runtime pending | -| 02 | multi-teacher-domain-routing.md | r6/r7 per-domain slice + routed distill verdict | P0 | r6 preds running on Modal | -| 03 | headwise-muon.md | per-head Muon wire-in for Q/K | P1 | module built, wire-in pending | -| 04 | trained-lite.md | lite-2.0: distill-trained 6-layer encoder | P1 | spec + launch | +| 01 | speculative-runtime.md | probe + runtime + framing verdict (tier pulled) | P0 | closed 2026-09-12: framing finding; see 11 | +| 02 | multi-teacher-domain-routing.md | r6/r7 per-domain slice + routed distill verdict | P0 | closed negative 2026-09-11 | +| 03 | headwise-muon.md | per-head Muon wire-in for Q/K | P1 | implemented + tested; awaits next teacher run | +| 04 | trained-lite.md | lite-2.0: init the cut from the trained student | P1 | closed negative 2026-09-12 (7.14 vs 5.78) | | 05 | kv-int8-runtime.md | int8 KV cache in the decode path | P2 | spec; IMF contract gate | -| 06 | sinkhorn-update.md | Sinkhorn-balanced optimizer for embedding/head | P2 | implement + unit spec | +| 06 | sinkhorn-update.md | Sinkhorn-balanced optimizer for embedding/head | P2 | implemented + tested; run optional | | 07 | diacritization-depth-knob.md | depth-conditioned vocalization | P3 | product decision | -| 08 | golden-closeout.md | golden test PR, paper-c wording, release README note | P0 | fix staged locally | -| 09 | doc-corrections.md | Engram mechanism fix, mHC single-pass note | P2 | quick edits | -| 10 | engram-lexical-memory.md | hashed n-gram memory module | P3 | gated behind 02/04 | +| 08 | golden-closeout.md | golden test PR, paper-c wording, release README note | P0 | closed 2026-09-12 (PR #211 merged) | +| 09 | doc-corrections.md | Engram mechanism fix, mHC single-pass note | P2 | landed in PR #95 | +| 10 | engram-lexical-memory.md | hashed n-gram memory module | P3 | gated behind 04 | +| 11 | static-int8-framing.md | static activation scales: fix framing at the source | P0 | pre-registered; experiment running | Standing rules that apply to every item: full-set measurement or no claim; quantized parity is quality-level, not byte-level; LLM teachers diff --git a/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md b/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md index 253ffc0..38e2c89 100644 --- a/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md +++ b/TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md @@ -23,18 +23,20 @@ greedy verification the verifier's argmax is authoritative, so outputs are byte-identical to non-speculative greedy — no parity story needed, just a speed one. -- Step 1 (no training, CPU-only): acceptance probe on golden-v1 rows. - Run both models, measure per-position argmax agreement and the - distribution of first-divergence offsets. Consonant positions should - agree ~100% (copy-through dominates); disagreement concentrates on - haraqat near-ties — the same margins that flip under quantization. - Net speedup = agreement rate × (drafter cost / verifier cost); the - int4 190M vs int8 300M ratio is favorable. -- Step 2 (if step 1 clears ~80% agreement): K-draft batch-verify in - the runtime KV decoder (TS worker first, it has the latency story). - Honest scope: this speeds the 2.1 tier. The lite tier stays as the standalone fast path; speculative decode is a middle tier that only pays when users want 2.1 quality at lower latency. +- Step 1 RESULT (2026-09-11, complete, 25/25 golden rows): + mean acceptance **0.9886** (min 0.955, median 0.991), + **8.85 tokens per verifier pass** at K=8 (incl. bonus tokens). + Exactness: all 18 rows checked against the verifier's plain-path + greedy are byte-identical (spec_loop == verifier greedy, the + output-preservation theorem holds on real quantized artifacts); + plain==KV on every checked row. Gate (>=0.9 acceptance) cleared + decisively. Runtime implementation: interscript-ts PR #77 (merged) + - SpeculativeModel exported from interscript/ml, tiny-fixture + + real-pair e2e green. Remaining: playground/API tier exposure. + ## 2. Multi-teacher domain-routed distillation — cheap untested lever