llama-cpp-optimizer

Empirical sweep-driven optimizer for llama.cpp server scripts on consumer NVIDIA hardware (2× RTX 3060 12GB primary target).

Encodes findings from 50+ benchmark rounds across three model categories:

Model	Spec	KV Cache
REAP20 (barozp Q4_K_S)	ngram-mod	turbo4 / q8_0
28B MTP (tminh Q4_K_M)	draft-mtp	turbo4 / q4_0
35B MTP (unsloth Q3_K_XL)	draft-mtp	turbo4 / q8_0

Contents

.
├── skill/                     # opencode skill (SKILL.md)
│   └── SKILL.md
├── sweeps/                    # Python sweep drivers (reproducible)
│   ├── sweep-mtp2.py          #   28B MTP single-param
│   ├── sweep-mtp-combo.py     #   28B MTP combo
│   ├── sweep-turbo-28a3.py    #   28A3 turbo4 single-param
│   ├── sweep-turbo-35a3.py    #   35A3-MTP turbo4 single-param
│   └── sweep-turbo-35a3-combo.py
├── results/                   # Raw benchmark JSON
│   └── *.json
├── findings/                  # Final reports
│   ├── MTP-FINDINGS.md
│   └── TURBO-FINDINGS.md
├── scripts/                   # Sanitized sample server scripts (HF tokens redacted)
│   └── qwen-*-server.sh
└── README.md

Key Findings (TL;DR)

REAP20 + ngram-mod (hand-tuned, already near-optimal)

--cache-type-k q8_0 --cache-type-v q8_0  # or turbo4
--batch-size 4096 --ubatch-size 2048
--spec-ngram-mod-n-match 16 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
--ctx-size 128000

Expected: pp ≈ 1334 t/s, tg ≈ 130 t/s. All variations within ±1.5%. AVOID bs=8192 (−3.8% tg).

28B MTP (the big win is nmax=1)

--spec-draft-n-max 1              # CRITICAL: was 3, +68% tg
--batch-size 2048 --ubatch-size 1024
--cache-type-k q4_0 --cache-type-v q4_0
--ctx-size 32000
--fit on                          # REQUIRED (without → cudaMalloc OOM)

Expected: pp ≈ 1185, tg ≈ 57 t/s (+42% / +141% vs baseline).

Reverse correlation: --spec-draft-n-max is INVERSELY correlated with tg for MTP. nmax=16 is catastrophic (tg=6 t/s).

35B MTP (nmax not critical, q8_0 KV slightly better)

--cache-type-k q8_0 --cache-type-v q8_0
--spec-draft-n-max 5              # vs 3: marginal
--threads-batch 4

Expected: pp ≈ 1334, tg_warm ≈ 132 t/s (+1.1% / +2.3%).

Difference from 28B MTP: nmax=1 is NOT critical for 35B — bigger model, draft overhead proportionally smaller.

Anti-Patterns (HARD BLOCKS)

1. pkill -9 -f llama-server — hangs the shell 60+ sec
2. bs=8192 for MTP — OOM, crashes
3. q5_0 KV — kernels broken in current builds
4. f16 KV for MTP — counter-intuitively SLOWER than q8_0
5. nmax > 3 for 28B MTP — catastrophic tg degradation
6. --fit on for non-MTP — reduces pp peak ~20%

Hardware Target

• 2× RTX 3060 12GB (24 GiB VRAM)
• i7-8700K (6 cores), 62 GiB RAM
• Tested with vanilla /root/cuda-12.8/llama-server AND TurboQuant build

Reproduce

# 1. Start server (REAP20 + ngram-mod example)
LD_LIBRARY_PATH=/root/cuda-12.8 \
  bash scripts/qwen-28A3-server.sh &

# 2. Wait ~30s, then run sweep
python3 sweeps/sweep-turbo-28a3.py

Use as opencode skill

cp -r skill/llama-cpp-optimizer /root/.cache/opencode/skills/

Then trigger via chat:

оптимизируй параметры llama.cpp для нового qwen-сервера

See skill/SKILL.md for full documentation.

Credits

Empirical data gathered Aug 9 2026 on hardware described above. Assisted-by: Sisyphus.