hybrid MoE strategy 20-40x slower than offload on an offload-heavy box (24 GB card, PCIe gen 3)

[gberasmus87]

Summary

On a single 24 GB card with the routed experts offloaded, --moe-strategy hybrid decodes at 0.17-1.02 tok/s. The same tree, same checkpoint, same box, with only --moe-strategy offload instead, decodes at 23.07 tok/s — about 20-40x faster. GPU utilisation during generation is 0% under hybrid and 85% under offload.

hybrid is not a niche choice here: --moe-strategy auto resolves a MoE model to hybrid when a ft bench bw profile recommends it, which it does on this box, and hybrid is what made this configuration fast in the first place — the same box on v0.1.2 serves this checkpoint at 33.5 tok/s under hybrid.

Numbers

Single RTX PRO 4000 Blackwell 24 GB (sm_120), PCIe gen 3 x16, EPYC 7302, CUDA 13.0, torch 2.11.0+cu130, TP=1, lovedheart/Qwen3.8-Flash-Next-NVFP4-FP8 (123 GiB, NVFP4 routed experts + block-FP8 dense), --moe-cache-auto, --kv-reserve-tokens 65536, --moe-prefill-hit-d2d.

tree	--moe-strategy	decode (end-to-end, every streamed delta)	engine-reported decode	GPU util
v0.1.2 + local patches	hybrid	33.5 tok/s	33.4 - 38.9	74-95%
main + #427 + #428 (b16877b)	hybrid	never completed a 3-prompt run	0.17 / 0.43 / 0.55 / 0.56 / 0.74 / 1.02	0%
main + #427 + #428 (b16877b)	offload	23.07 tok/s	21.3 - 25.9	85%

The hybrid figures come from three independent launches of the new tree, so they are not a cold-start artifact. Both new-tree rows are the same install; the only difference is the strategy flag.

What it looks like while it is slow

• GPU 0% during generation, weights resident at 21,962 MiB.
• One CPU thread pinned at ~100% (state=R, wchan=0, ~1,034 s of CPU accumulated) while the other 76 threads sit idle — including the whole CPU MoE executor ready: threads=15 (pinned to cores 0..14) pool. So neither the GPU nor the CPU expert pool is doing the work; something is spinning on the engine's main thread.
• Under offload on the identical install that same thread reads 3.0% CPU, state=Ssl.

That pattern points at the hybrid split itself. Startup logs --moe-hybrid-max-fetch auto: fetching 27.8% of each decode step's expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU — and it is exactly the "rest on the CPU" half that appears not to reach the pool.

Ruled out

• Not JIT compilation. ~/.triton/cache stopped growing (4,015 files / 145 MB, unchanged over a 30 s sample) while that thread burned CPU.
• Not expert-cache starvation. --moe-cache-auto resolves moe_cache_size=3741 num_pages=1025, versus 4041 on the old build — 7% down, nowhere near enough.
• Not the deprecated flags. Re-ran with --nvfp4-backend triton removed entirely (quant_backend=None); it still resolves MoE experts: nvfp4 via triton and is still 0.55 tok/s. The offload run above also has that flag removed.
• Not the loader. feat(qwen4_exp): load block-fp8 dense projections natively #428's reader loads this checkpoint correctly — all buffers match, output is correct, and the offload run proves the same install serves it fine.

Bisect status

I have not bisected. My baseline is v0.1.2 plus local patches rather than upstream main, because main cannot load this checkpoint at all without #428, so the window is v0.1.2..b16877b — roughly 34 commits including the #418 quantization refactor. Each cycle here is a ~5 minute install plus load, so it is a few hours; happy to do it if that is the fastest way to close this out, or to run any specific commit, flag set, or instrumented build you want to point at.

I could not get a Python stack: the box is ptrace_scope=1 and running the engine under py-spy record --subprocesses attached and sampled but never flushed before the window closed.

Why this may not show elsewhere

hybrid only carries real load when the experts genuinely do not fit. On an 80 GB H100 or 2 x 48 GB Ada this model is close to resident and the CPU-assist half of the split barely engages, so the same code path is nearly free. Here it carries 123 GiB of weights against 24 GB of VRAM on a gen-3 board, which is the regime the offload family exists for.

Context: first reported on #428 (#428 (comment)), narrowed to the strategy flag afterwards.

[jason-fxz]

Thanks for the write-up. Two things would help us narrow this down:

1. The ft bench bw output for this box (or the profile file it wrote, ~/.cache/freetoken/benchbw.json). The 27.8% fetch split comes from it, and we would like to see the CPU / PCIe numbers behind that.
2. The full startup log of a hybrid launch on b16877b, up to API server is ready to serve. The lines we are after are the ones from CPU MoE executor ready: ... (isa / threads), whether cpu-moe flag handshake unavailable: ... appears, and PLE disk backend: ..., wait-sync|launch-gating.

For reference, the same tree and checkpoint on an H100, with --moe-cache-rate 0.152 to match your 3741-slot residency, decodes at 56 tok/s on hybrid vs 50 tok/s on offload, so this is specific to something on your platform rather than the hybrid path itself.

[gberasmus87]

All three, plus one comparison that I think narrows it a long way.

1. ~/.cache/freetoken/benchbw.json

Written 2026-08-25T00:23:21+00:00, version 4. Ceilings and the nvfp4 entry (the one this checkpoint's experts use):

"cpu": { "physical_cores": 16, "threads_used": 16 },
"threshold": 2.0,
"ceilings": {
  "cpu_stream_read_gbs": 31.75,
  "pcie_linear_h2d_gbs": 14.21,
  "pcie_linear_d2h_gbs": 14.34
},
"dtypes": { "bf16": "hybrid", "nvfp4": "hybrid", "fp8_block": "offload",
            "mxfp4_triton": "offload", "ds_fp4": "hybrid" },
"dtype_kernels": {
  "nvfp4": {
    "expert_bytes": 7974912, "synth_experts": 128,
    "cpu_moe_gbs": 29.97, "cpu_moe_isa": "avx2", "isa_sweep": null,
    "pcie_gather_gbs": 13.09,
    "cpu_moe_overlap_gbs": 21.89,
    "pcie_gather_overlap_gbs": 8.41,
    "ratio": 2.29, "recommended": "hybrid", "note": null
  }
}

ratio 2.29 against threshold 2.0 is where the hybrid recommendation and the 27.8% split come from. isa_sweep is null on every entry. Happy to attach the whole file if useful — bf16 is ratio 2.352, ds_fp4 2.329, mxfp4_triton 1.38 -> offload, fp8_block has "CPU MoE has no fp8_block weight path; hybrid unavailable".

Note the date: this profile predates the problem and both builds below read the same one, so it is a constant across the comparison rather than a variable.

2. Startup, hybrid on b16877b, to "ready to serve"

Resolved config: moe_strategy='hybrid', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
Free memory before loading model: 23.13 GiB
PLE disk backend: io_uring, O_DIRECT, wait-sync
MoE experts: nvfp4 via triton
expert banks: slow path (parallel build)
--moe-cache-auto resolved moe_cache_size=3741 num_pages=1025 (prefill_overlap=True)
--moe-hybrid-max-fetch auto: fetching 27.8% of each decode step's expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU
torch intra-op threads: 16 -> 1 (cores reserved for the pinned CPU MoE pool)
CPU MoE executor ready: threads=15 (pinned to cores 0..14) isa=avx2 fmt=nvfp4 H=2560 I=640 experts=512 layers=48 top_k=10 act=silu max_tokens=4
Allocating 65600 tokens for KV cache, K + V = 1.55 GiB
Free memory after initialization: 2.18 GiB
Start capturing CUDA graphs with sizes: [1, 2, 4]
Free GPU memory after capturing CUDA graphs: 2.03 GiB
Scheduler is idle, waiting for new reqs...
API server is ready to serve on 127.0.0.1:1919

PLE disk backend is io_uring, O_DIRECT, **wait-sync**.

3. cpu-moe flag handshake unavailable

Never appears. journalctl | grep -ci "cpu-moe flag handshake" over every run since 2026-09-09 returns 0.

The comparison I think matters

My 33.5 tok/s baseline is v0.1.2 + local patches. I pulled its startup log and put it beside the b16877b one. Every logged parameter is identical:

	v0.1.2 (33.5 tok/s)	b16877b (0.5 tok/s)
strategy / backend / cache	hybrid, qsa_sparse, hybrid_radix, page 64	same
moe_cache_auto	3741 slots, num_pages=1025	same
hybrid split	27.8% PCIe	same
torch intra-op	16 -> 1	same
CPU MoE executor	threads=15, cores 0..14, isa=avx2, fmt=nvfp4	same
PLE backend	io_uring, O_DIRECT, wait-sync	same

Same box, same checkpoint, same flags, same cached bench profile, same resolved everything — 60x apart. So whatever this is, it is not the split, the ISA, the thread layout, the cache sizing or the PLE backend being resolved differently. It is in the code between the two.

Two other things that rule out the obvious:

• PLE disk backend is wait-sync on the offload run too, and that run is fine at 23 tok/s. So the PLE path is not the differentiator between the two strategies on this box.
• The box is an EPYC 7302, 16 physical cores / 32 threads. torch intra-op threads: 16 -> 1 with 15 cores pinned to the MoE pool leaves torch exactly one thread. I suspected that, but the working build does exactly the same thing, so it is not the cause on its own — though it may be what makes this platform sensitive to a change that is free on a box with more cores.

Given your H100 numbers (56 hybrid vs 50 offload at matched residency), agreed that it is platform-specific rather than the hybrid path as such. The window is v0.1.2..b16877b. I am happy to bisect it here — roughly 34 commits at a ~5 minute install-plus-load cycle, so a few hours — or to run any specific commit or instrumented build you would rather point at first. Say which and I will get you numbers.