multi-token-prediction

llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).

PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery.

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx

[NeurIPS 2026] DMax: Aggressive Parallel Decoding for dLLMs

Qwen4-Exp (Qwen3.8-Flash-Next) SWA + MTP + PLE-on-iswa — bounded deep-context decode w/ long-range recall, TBQ4 4.25 bpv KV, DSpark + RotorQuant. Main (master) = Qwen4-Exp build; the Qwen35 SWA hybrid lives on the qwen35 branch. RTX 4090.

Official codebase for "Multi-Token Prediction Done Right"

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

Curated collection of research on the limitations of next-token prediction and methods that go beyond it.

Run Hermes Agent + Claude Code locally on llama.cpp — zero API costs. A 4h / 7M-token session that would have cost $94 on Claude Opus 4.7

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Docker image. Decode-optimized to ~91–99 tok/s (min ~91 chat, max ~99 math), lossless (full GPU residency + ECC-off).

A C++23 library for local LLMs at the metal — inference and training, built from explicit neural-network components you can read and understand.

Rust + CUDA inference engine for Crow on one RTX 5090: GLM-5.3-Flash, Qwen3.8-Flash-Next (MoE) and the dense Qwen3.8-27B as CNQ containers, own kernels, MTP speculative decoding, vision, OpenAI-compatible serve. Windows and Linux.

Preregistered, DOI-archived llama.cpp study: Qwen3.8-27B on one RTX 3090 (sm_86). draft-mtp at --spec-draft-n-max 2 is +59.8% [+57.0, +62.8] server-reported decode over 25 purposive prompts; draft-dflash (PR #27342) +51.9%, not separated from it. Decode energy -37.1%, uncalibrated. 23-25 of 25 prompts diverge from serial greedy at 1600 tokens.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

Reproducible Qwen3.8-27B MTPLX benchmark on Apple M1 Max

Faithful from-scratch reimplementation of DeepSeek-V3 (MLA + aux-loss-free MoE + MTP + speculative decoding), ~412M params, Chinchilla-optimal 8.4B-token training on a single A100 80GB.

Qwen3.8-27B on Ampere (RTX 3090 / 3090 Ti): TurboQuant+ llama.cpp fork with SM86 kernel and memory work, 245K context with the model's own MTP head. Model: sjakek/Qwen3.8-27B-ATX-4-XS-GGUF on Hugging Face.

MTP draft-head research harness for Qwen3.5-9B. Verdict: function cuts (pruning, low-rank, distillation) pay a 10-19pt off-distribution tax fidelity cuts do not — the zero-training trimmed-vocab NVFP4 recipe wins at 1.8-2.7x end-to-end in memra. Negative-result ledgers included.

A controlled 180-run study of DeepSeek-inspired MLA, sparse MoE routing, V3-style load balancing, and multi-token prediction under constrained compute.

PureDrive: a 35B-A3B MoE coding model at 110 tok/s on one Apple M4 Pro (63 tok/s at 128K), lossless. Exact MTP speculative decoding + custom Metal attention kernels for MLX/oMLX. Local OpenAI-compatible server; 73.3% on a Terminal-Bench 2.1 subset.