FreeToken Roadmap (2026)

[jason-fxz]

FreeToken RoadMap - 2026

New Model Support

• GLM-5.3-Flash @andy-yang-1 feat(glm5_next): support GLM-5.3-Flash #332
• Qwen3.8-Flash-Next @jason-fxz
  • pin PLE in RAM — landed in feat(qwen4_exp): support Qwen3.8-Flash-Next #257
  • offload PLE to Disk feat(qwen4_exp): stream the PLE n-gram table from disk #311
• DeepSeek-V4.1-Flash

Hardware Support

• macOS: a native Metal engine on Apple Silicon Macs.
• AMD: support for AMD GPUs via ROCm.
• DGX Spark: aarch64 wheels, sm_121 kernels, and a unified-memory mode.

Multimodal

• Support Image input for vision-language models
  • Qwen3.5 / Qwen3.6 / Qwen3.8 family @jason-fxz
  • Qwen3.8-Flash-Next
  • Gemma-4 family

Multi-GPU

• DGX Spark: multi-GPU optimisation for linked DGX Spark units.
• Tensor parallelism (TP): serve one model across multiple GPUs on one machine.

Quantization

• Quant Layer Refactor:
  • refactor(quant): config, scheme and method layers for quantization #418
  • refactor(quant): hand the checkpoint QuantConfig to the weight readers #427
• GGUF: support GGUF checkpoints and their quantization types across model architectures.

Speculative Decoding

• Speculative decoding, including MTP / DFlash / Dspark.

---

Any contribution is welcome. Join the Developer Slack if you want to get involved.

---

TODO: This is still WIP.

[dusty44]

Any thoughts about enabling experts to stream from NVME drives? with the cached experts optimised in three tiers vram, system ram then streamed from disk? I've built it in a dwarfstar derived engine and it can work shockingly well.

[Aladincz]

Is there any chance that this technology could be usable and give some benefits to old macs running small LLM on CPU? Would it be technically applicable?

[Headmaster218]

Interesting timing — I’ve independently been working on a fairly similar MoE serving problem on AMD RDNA3/Vulkan, mainly around expert caching/paging, heterogeneous CPU–GPU execution and large-model offloading.
It’s still very much WIP and I’m cleaning up the benchmarks/docs, but the current code is here: [infr]
Running on a single RX 7900 XTX + 64GB DDR4. Happy to compare notes on the AMD side.
Successfully running with
Qwen3.6-35B-A3B with decode at 150 ~ 40tk/s, prefill at 3000 ~ 400tk/s,
Qwen3.5-122B-A10B with decode at ~23tk/s
Ling 3.0 flash with decode at 36tk/s,
Deepseek V4 Flash at ~4tk/s

[orangelightening]

Looking forward to the qwen3.8 27b model support. I am using the qwen3.6 35b in the freetoken engine and using deepseek harness to serve it via the openai compatible endpoint. I have an rx3060 and 128gb of ram on an old xeon server and this is the best performing local ai system I have owned. Absolutely wonderful.

[Headmaster218]

Different from MoE, 27B is a kind of dense model, that doesnt suit this kind of cache architecture. But, Qwen team said they will public a new middle size MoE model this week, probably Qwen3.8-122B-A10B, (so thats why now im optimizing Qwen3.5-122B-A10B ;) ), and probably it will be more powerful than Qwen3.8 27B, so just, wait for it.

[erikfenriz]

Would be so awesome to see it on macOS!

[McL-M]

First, let me just say this is very interesting work. Keep it up.

Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.

But they can use Vulkan. And so can virtually everything else.

Please consider supporting Vulkan!

[Headmaster218]

First, let me just say this is very interesting work. Keep it up.

Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.

But they can use Vulkan. And so can virtually everything else.

Please consider supporting Vulkan!

https://github.com/Headmaster218/infr
Try MoE4All ;) , it uses Vulkan

[Abdul-Sen]

First, let me just say this is very interesting work. Keep it up.
Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.
But they can use Vulkan. And so can virtually everything else.
Please consider supporting Vulkan!

https://github.com/Headmaster218/infr Try MoE4All ;) , it uses Vulkan

Tried with Qwen3.8-Flash-Next?

[Headmaster218]

First, let me just say this is very interesting work. Keep it up.
Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.
But they can use Vulkan. And so can virtually everything else.
Please consider supporting Vulkan!

https://github.com/Headmaster218/infr Try MoE4All ;) , it uses Vulkan

Tried with Qwen3.8-Flash-Next?

yes!!! just had an early success!!!!! Qwen3.8-Flash-Next!!!

[McL-M]

First, let me just say this is very interesting work. Keep it up.
Unfortunately, I can't try it. Lot's of older AMD GPUs can't run ROCm.
But they can use Vulkan. And so can virtually everything else.
Please consider supporting Vulkan!

https://github.com/Headmaster218/infr Try MoE4All ;) , it uses Vulkan

Yeah, I'll check it out! Thanks! 😁

[radubalanpro]

Any plans for AMD gfx1151 ? I have a GMKtec EVO-X2, Ryzen AI Max+ 395 3.0GHz, 128GB DDR5, Radeon 8060S. I am running llama.cpp with ROCm on linux - custom build (actually wsl2) with pretty good results in inference, better then Vulkan in Windows, but i look forward to FreeToken, As for a test i get 24 t/s on Qwen3.8-27 Q4KM gguf and around 35-40 t/s on qwen-coder-next UD Q4KL unsloth

[LeonardoDaviti]

Any plans for AMD gfx1151 ? I have a GMKtec EVO-X2, Ryzen AI Max+ 395 3.0GHz, 128GB DDR5, Radeon 8060S. I am running llama.cpp with ROCm on linux - custom build (actually wsl2) with pretty good results in inference, better then Vulkan in Windows, but i look forward to FreeToken, As for a test i get 24 t/s on Qwen3.8-27 Q4KM gguf and around 35-40 t/s on qwen-coder-next UD Q4KL unsloth

Same, but Vulkan for support will be much easier no?