[Model] DeepSeek-V4.1-Flash (deepseek_v41) — hybrid CPU-MoE makes it viable on a single GPU, and Engram is already isolated in 2 shards

[angelbaba01L]

Before you start

Checked the Roadmap (#79 — no mention of deepseek / v4.1 / engram), the FAQ (#84), and existing issues: searching deepseek_v41, engram, V4.1 and v41 across this repo returns nothing as of 2026-09-11.

Hugging Face link

https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

Is the model architecture already supported

No — deepseek_v4 is supported, but this is a new architecture: model_type: deepseek_v41, architectures: ["DeepseekV41ForCausalLM"].

Is the quantization already supported

Partly. Top-level quantization_config is {quant_method: fp8, activation_scheme: dynamic, weight_block_size: [32, 32], scale_fmt: ue8m0}, and the reference config declares dtype: fp8 with expert_dtype: fp4 — i.e. FP4 routed experts with FP8 dense. FP8 and NVFP4 are listed as supported today, so the open question is whether this FP4 expert layout maps onto an existing bank schema or needs a new one.

What happens when you load it

ERROR   Backend supervisor: ValueError: Model architecture DeepseekV41ForCausalLM not supported
Traceback (most recent call last):
    return _MODEL_REGISTRY[model_architecture]
KeyError: 'DeepseekV41ForCausalLM'

(Registry lookup fails before anything else, so this is an architecture gap rather than a checkpoint/quant issue.)

Anything else

I think FreeToken is in an unusually strong position for this model, and the checkpoint turns out to be laid out in a way that helps. Sharing the measurements in case they are useful for scoping.

Why the hybrid MoE backend matters here

The checkpoint is 475.3 GiB. Engines whose offload path copies weights back to the GPU each forward pass are PCIe-bound; on a 32 GiB GPU that means streaming ~256 GiB per token, i.e. fractions of a token/s. ft bench bw on my box (2026-08-29):

path	measured
PCIe H2D, linear	57.8 GB/s
host DRAM stream read	266.9 GB/s
hybrid ds_fp4: CPU MoE, overlapped	114.5 GB/s
hybrid ds_fp4: PCIe gather, overlapped	53.1 GB/s
hybrid combined	167.7 GB/s

Keeping expert weights in host RAM and computing them on CPU workers — only activations crossing the bus — is ~3x the effective bandwidth of weight streaming. As far as I can tell it is the only architecture that makes this checkpoint viable on a single consumer GPU.

The checkpoint is already partitioned along the useful line

Of 96,085 tensors, only 12 are Engram (layers.1.engram.* and layers.14.engram.*), and they occupy exactly the last two shards:

model-00047-of-00048.safetensors   94.6 GiB  ┐  Engram, 196B params
model-00048-of-00048.safetensors   94.6 GiB  ┘  = 189.2 GiB
model-00001..00046-of-00048        286.1 GiB    backbone, 552B params
                                   ─────────
                                   475.3 GiB

So the 196B "conditional memory" can be handled entirely separately from the 552B backbone with no repacking.

Engram does not need to be resident

From inference/engram.py and the reference config.json:

• engram_layer_ids = [1, 14], engram_max_ngram_size = 4, engram_n_heads = 8, engram_head_dim = 256, fp8
• engram_num_embeddings = [384006168, 384016682] (~384M rows each)
• per token: 2 layers x (4-1) n-gram sizes x 8 heads = 48 row gathers, 256 B each

It is a token-hash lookup, not a GEMM — an ideal candidate for staying on NVMe. I benchmarked that exact access pattern against a PCIe 5.0 SSD (Samsung 9100 PRO), using O_DIRECT so neither the page cache nor the host's VHDX cache can flatter the result:

random 4 KiB read, QD=1:  median 80.6 us   p95 117.6 us      (through a WSL2 VHDX)
48 gathers/token, serial: 3.78 ms
48 gathers/token, 8-way:  1.14 ms   <-- best

Against a 30 ms/token decode budget that is ~3.7% overhead, and it is a pessimistic bound: every gather is assumed to miss, the I/O crosses a virtualized block device, and there is no batching (io_uring or a batched gather would cut it further). N-gram frequencies are Zipf-distributed, so spare host RAM acting as page cache should absorb a large share in practice.

What that does to the memory budget

Config: 512 GiB host RAM, one RTX 5090 (32.6 GiB), PCIe 5.0 NVMe.

Engram placement	host RAM needed
resident in host RAM	~482 GiB — does not fit
read from NVMe	~293 GiB — fits, ~160 GiB to spare

The GPU side is where V4.1 is surprisingly friendly:

dense/attn                ~23.6 GiB
KV cache @ 1M context       0.89 GiB    (CSA2: 890 B/token, global)
CUDA graphs                  ~1 GiB
                          ---------
expert slot cache            ~5 GiB     (~263 slots at 16.88 MiB/expert)

For comparison, DeepSeek-V4-Flash-0731 on the same card spends 4.95 GiB of VRAM on KV for a 150K window (Allocating 150016 tokens for DSV4 KV cache, total = 4.95 GiB). V4.1 gets its full 1M context for 890 MB.

Architecture delta vs deepseek_v4

Likely reusable: the hybrid CPU-MoE machinery (384 routed + 1 shared, top-6 — same shape as V4, just larger: 3 x 5120 x 2304 FP4 = 16.88 MiB/expert vs V4's 12.00 MiB), DSpark (dspark_block_size: 5, 3 MTP layers, dspark_target_layer_ids: [37,38,39], dspark_n_routed_experts: 128), and the FP8/FP4 quant paths.

New:

• CSA2 attention — each layer is assigned one of Full / Reindex / Reuse; kv_source_layers: [2,8,14,20] and index_source_layers: [2,8,14,20,24,28,32,36] share main KV and indexer K across layers; FP4 main KV (E2M1, one E4M3 scale per 16 channels); index_topk: 512, window_size: 128, compress_ratios per layer. This is what gets the global KV footprint to 890 B/token, and it is the main departure from dsv4_compress / dsv4_sparse / dsv4_indexer / the DSV4 paged pool.
• Engram — as above; small in code terms.
• Nested text_config + vision_config; the vision tower is optional for text-only serving (shards 1-2, 2.1 GiB).

Other engines

SGLang has a preview (lmsysorg/sglang:dev-dsv41) with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1, which moves the two Engram tables to a shared host copy — documented as bitwise unchanged, but loaded into host RAM rather than memory-mapped, and its verified configs are 4x GB300 / 4x MI350X. vLLM has an open PR. Neither addresses the single-GPU + large-host-RAM case that FreeToken's hybrid backend already targets.

Offer

Happy to help with the parts I can carry: I can implement and test the Engram-from-NVMe path, and I can supply ft bench bw profiles and end-to-end numbers from a single-GPU + 512 GiB host configuration.