Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions TODO.impl/01-speculative-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ everywhere, consistent with the 0.036% near-tie flip measurement.
## Status

- [x] Probe v1 written; EOS-acceptance and index bugs found and fixed
- [ ] Probe rerun to completion
- [ ] Runtime implementation
- [ ] Tier exposure decision
- [x] Probe rerun to completion: 25/25 rows, mean acceptance 0.9886
(min 0.955), 8.85 tokens/verify at K=8; all 18 exactness-checked
rows byte-identical to the verifier's plain-path greedy
(results: ~/ml-logs/spec_probe/results.json)
- [x] Runtime implementation: interscript-ts PR #77 (merged) -
SpeculativeModel + acceptBlock exported from interscript/ml;
unit tests + tiny-fixture session + real-pair e2e green
- [ ] Tier exposure: expose SpeculativeModel in the playground/API
surface (acceptance threshold cleared; site wiring pending)
- [x] Numbers recorded in TODO.qwen-next/10 and RESULTS.md
8 changes: 8 additions & 0 deletions TODO.impl/02-multi-teacher-domain-routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,17 @@ available teachers can beat plain r7 supervision. The residual is
student-side (news/wiki rows), not teacher-selection. Remaining
levers: 04 (trained-lite), 10 (lexical memory), 01 (runtime).

## Soup postscript (2026-09-12): also negative

The r6+r7 50/50 weight soup scored 2.4188 full-set (r6 2.5997,
r7 2.289) — same-basin (functional model, linear connectivity
confirmed) but strictly between the parents on every domain slice.
r7 remains the best available teacher; supervision unchanged.

## Status

- [x] r7 per-domain slice (from released predictions, zero GPU)
- [x] r6 preds on volume, fetched, sliced
- [x] Verdict: negative, recorded in TODO.qwen-next/10 §2
- [x] No routed launch (nothing to route)
- [x] Soup probe: negative (RESULTS.md 2026-09-12)
19 changes: 16 additions & 3 deletions TODO.impl/04-trained-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,21 @@ a cost of one run. If it fails, the lite frontier is recipe-bound and
- [ ] Full-set verdict with intervals recorded in RESULTS.md
- [ ] Decision: lite tier replaced or negative recorded

## Verdict (2026-09-12): NEGATIVE — 7.1402 vs run-009's 5.78

Same recipe, same canonical labels (sha e70ce991), same teacher
(re-scores 2.2921 in-run); the single variable - layer-drop init from
the trained 2.1 instead of generic byt5-small - made the student
1.36pp WORSE. Reading: generic pretraining keeps layers redundant;
task adaptation co-specializes them, and deleting half a co-adapted
stack breaks more computation. The depth axis now reads 2-of-3
negative (Hebrew generic-init collapse; Arabic adapted-init collapse;
run-009's generic-init 5.78 stands). The lite tier is unchanged; the
remaining architecture lever is /10 (lexical memory), still gated.

## Status

- [ ] Build init on volume
- [ ] Launch
- [ ] Verdict
- [x] Build init on volume (the layer_drop bridge, student_init=2.1)
- [x] Launch (checkpoint-resumed once; canonical labels seeded after
catching the re-labeling hazard)
- [x] Verdict: negative, recorded in RESULTS.md
46 changes: 46 additions & 0 deletions TODO.impl/11-static-int8-framing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# 11 — Static activation scales: fix decode framing at the source (P0)

Pre-registered 2026-09-12, from the framing finding (RESULTS.md:
dynamic-int8 graphs compute activation quantization scales per fed
tensor, so single-step and batched decode framings produce materially
different decodes). The shipped int8 uses `quantize_dynamic`
(DynamicQuantizeLinear at runtime); static quantization
(`quantize_static` + calibration) bakes activation scales into the
graph and removes the per-run computation entirely.

## Why this is the top lever

If framing stabilizes, three separate findings improve at once:
- the speculative tier's acceptance collapse (0.99 → 0.46 purely by
framing) may reverse — its domain would reopen on CPU
- the cross-hardware near-tie flips (RESULTS 2026-09-07) should shrink:
one fewer runtime-computed quantity to diverge across machines
- the golden test could return to byte-exact assertions for int8

## Experiment (pre-registered gates)

`scripts/static_int8_experiment.py`: re-quantize the 2.1 decoder-kv
graph with QUInt8 static activations (MatMul-only, head fp32 — same
node policy as the shipped dynamic int8), calibrated on 365 real decode
feeds sampled across BOTH framings (prefills, incremental steps,
8-token windows).

- **A. framing equality**: the same greedy decode driven token-by-token
vs in 8-token batched calls — GATE: identical trajectories
- **B. drift vs fp32** (quality proxy): GATE: no worse than the
dynamic int8 artifact's relationship to fp32; full-set DER via the
artifact scorer if A clears
- **C. speed**: static vs dynamic tok/s on CPU — record, not gate
(u8s8 static kernels are usually comparable or faster)

If A and B clear: re-export all quantized artifacts through the static
path (modal_export gains the calibration stage), index-v6, golden rows
regenerate, framing claim added to the contract.

## Status

- [x] Pre-registered (this file)
- [x] fp32 artifact fetched, sha-verified against index-v5
- [x] Calibration corpus collected (365 feeds, both framings)
- [ ] quantize_static result measured (A/B/C)
- [ ] Verdict recorded in RESULTS.md; ledger cross-links updated
17 changes: 9 additions & 8 deletions TODO.impl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,17 @@ Priorities:

| # | file | deliverable | priority | status |
|---|---|---|---|---|
| 01 | speculative-runtime.md | acceptance probe + TS SpeculativeSession + tier decision | P0 | probe running; runtime pending |
| 02 | multi-teacher-domain-routing.md | r6/r7 per-domain slice + routed distill verdict | P0 | r6 preds running on Modal |
| 03 | headwise-muon.md | per-head Muon wire-in for Q/K | P1 | module built, wire-in pending |
| 04 | trained-lite.md | lite-2.0: distill-trained 6-layer encoder | P1 | spec + launch |
| 01 | speculative-runtime.md | probe + runtime + framing verdict (tier pulled) | P0 | closed 2026-09-12: framing finding; see 11 |
| 02 | multi-teacher-domain-routing.md | r6/r7 per-domain slice + routed distill verdict | P0 | closed negative 2026-09-11 |
| 03 | headwise-muon.md | per-head Muon wire-in for Q/K | P1 | implemented + tested; awaits next teacher run |
| 04 | trained-lite.md | lite-2.0: init the cut from the trained student | P1 | closed negative 2026-09-12 (7.14 vs 5.78) |
| 05 | kv-int8-runtime.md | int8 KV cache in the decode path | P2 | spec; IMF contract gate |
| 06 | sinkhorn-update.md | Sinkhorn-balanced optimizer for embedding/head | P2 | implement + unit spec |
| 06 | sinkhorn-update.md | Sinkhorn-balanced optimizer for embedding/head | P2 | implemented + tested; run optional |
| 07 | diacritization-depth-knob.md | depth-conditioned vocalization | P3 | product decision |
| 08 | golden-closeout.md | golden test PR, paper-c wording, release README note | P0 | fix staged locally |
| 09 | doc-corrections.md | Engram mechanism fix, mHC single-pass note | P2 | quick edits |
| 10 | engram-lexical-memory.md | hashed n-gram memory module | P3 | gated behind 02/04 |
| 08 | golden-closeout.md | golden test PR, paper-c wording, release README note | P0 | closed 2026-09-12 (PR #211 merged) |
| 09 | doc-corrections.md | Engram mechanism fix, mHC single-pass note | P2 | landed in PR #95 |
| 10 | engram-lexical-memory.md | hashed n-gram memory module | P3 | gated behind 04 |
| 11 | static-int8-framing.md | static activation scales: fix framing at the source | P0 | pre-registered; experiment running |

Standing rules that apply to every item: full-set measurement or no
claim; quantized parity is quality-level, not byte-level; LLM teachers
Expand Down
20 changes: 11 additions & 9 deletions TODO.qwen-next/10-deepseek-v4-1-flash-learnings.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,20 @@ greedy verification the verifier's argmax is authoritative, so outputs
are byte-identical to non-speculative greedy — no parity story needed,
just a speed one.

- Step 1 (no training, CPU-only): acceptance probe on golden-v1 rows.
Run both models, measure per-position argmax agreement and the
distribution of first-divergence offsets. Consonant positions should
agree ~100% (copy-through dominates); disagreement concentrates on
haraqat near-ties — the same margins that flip under quantization.
Net speedup = agreement rate × (drafter cost / verifier cost); the
int4 190M vs int8 300M ratio is favorable.
- Step 2 (if step 1 clears ~80% agreement): K-draft batch-verify in
the runtime KV decoder (TS worker first, it has the latency story).
- Honest scope: this speeds the 2.1 tier. The lite tier stays as the
standalone fast path; speculative decode is a middle tier that only
pays when users want 2.1 quality at lower latency.
- Step 1 RESULT (2026-09-11, complete, 25/25 golden rows):
mean acceptance **0.9886** (min 0.955, median 0.991),
**8.85 tokens per verifier pass** at K=8 (incl. bonus tokens).
Exactness: all 18 rows checked against the verifier's plain-path
greedy are byte-identical (spec_loop == verifier greedy, the
output-preservation theorem holds on real quantized artifacts);
plain==KV on every checked row. Gate (>=0.9 acceptance) cleared
decisively. Runtime implementation: interscript-ts PR #77 (merged)
- SpeculativeModel exported from interscript/ml, tiny-fixture +
real-pair e2e green. Remaining: playground/API tier exposure.


## 2. Multi-teacher domain-routed distillation — cheap untested lever

Expand Down