Build software better, together

giannisanni / pulsar

SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7 tok/s on two consumer 16GB GPUs. Zero-config multi-GPU: measures PCIe bandwidth, places attention and hot experts where they fit.

rust cuda moe glm quantization multi-gpu inference-engine mixture-of-experts llm local-llm gguf ssd-streaming

Updated Sep 1, 2026
Rust

andreaborio / hebrus

Star

Metal-first MoE inference for Apple Silicon with bounded SSD expert streaming and local OpenAI Chat, Responses, and Anthropic Messages endpoints.

c macos metal moe glm inference-engine mixture-of-experts on-device-ai apple-silicon local-llm llm-inference local-ai qwen gguf deepseek openai-compatible ssd-streaming

Updated Aug 31, 2026
C

giannisanni / neutronstar

Star

Giant MoE models on a single consumer GPU by streaming experts from SSD. CUDA fork of antirez/ds4: runs GLM-5.2 (743B), Tencent Hy3 (295B), and DeepSeek 4 Flash, with io_uring expert streaming, LFU host cache, cross-layer expert prefetch, and the first MTP speculative decoding for GLM-5.2.

cuda glm mixture-of-experts llm-inference gguf speculative-decoding deepseek consumer-gpu ssd-streaming

Updated Jul 12, 2026
C

Ninnix / q36

Star

Qwen3.8-27b and Qwen3.6-35B-A3B local inference engine for Vulkan and Metal

c metal vulkan moe quantization inference-engine apple-silicon bc-250 local-llm llm-inference qwen gguf openai-compatible coding-agent ssd-streaming

Updated Sep 1, 2026
C

szibis / mlx-flash

Star

Run AI models too large for your Mac's memory — at near-full speed. Intelligent expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon.

python macos rust machine-learning inference moe quantization mlx speculative-execution mixture-of-experts memory-optimization apple-silicon llm metal-gpu ssd-streaming expert-caching

Updated May 11, 2026
Python

doramirdor / mbolt

Star

Profile-guided layout optimization for llama.cpp GGUF MoE models — BOLT/PGO for LLM weights. Trace expert routing, rewrite the file so co-activated experts sit together, prefetch as merged reads.

metal moe bolt quantization model-serving pgo profile-guided-optimization inference-optimization mixture-of-experts edge-ai on-device-ai apple-silicon llama-cpp local-llm llm-inference gguf ssd-streaming expert-parallelism

Updated Jul 22, 2026
Python

outerreaches / moonshine

Star

Run the official Kimi K3 MoE checkpoint on one 128 GB AMD Strix Halo box. ROCm-resident static weights, MXFP4 experts streamed from NVMe via io_uring. C engine, chat client, OpenAI-compatible server. Very experimental.