llm-inference-optimization

[NeurIPS 2025] NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

｢ICCAD 2026｣ MeltKV: Memory-Aligned KV Cache Compression via Variable-Bit Packed Vector Quantization for Long-Context LLMs

Code for HDA-MoE (IEEE TCAD): topology- and hardware-aware deployment, scheduling, and gating for efficient MoE inference on 3D near-memory systems.

Transparent LLM inference experiments on Apple Metal

Optimizing LLM throughput via binned padding, sequence packing, and Flash Attention.

This project replaces broken tokenization heuristics for Roman Urdu-English text with a single-feature Ridge model, achieving 2.5× higher accuracy at an ultra-low latency of 0.003 ms. By eliminating subword underestimation, our ML-driven scheduler drops misrouted requests to 3.4%, cutting queue wait times by 28% over standard word heuristics.

Qwen3-4B FP8 推理引擎 | 手写 CUDA 融合算子 | 支持 RTX 5060 本地部署

Production-oriented LLM serving examples covering KV-cache decoding, batching, quantization, LoRA, multi-LoRA, testing, benchmarking, and reproducible MLOps workflows.

Anllm is an llm inference engine from scratch powered by mlx on mac devices that supports the Qwen3 and Llama3.x family Int 4bit quantized models

Layer-wise weight offloading with profiling-driven optimizations.

QTIP and QUIP Quantization from First Principles. Most people understand basic INT8 quantization, but don't know exactly why lower bit-widths like INT4 completely destroy model quality, and how we fix it. To understand how we can run massive models on consumer hardware, we need to understand the evolution of quantization up to QTIP.

Energy characterisation of LLM inference across heterogeneous CPU islands, measured with RAPL counters. Finding: the efficiency cores cost 2.3× the joules per token — the penalty is duration, not power.