cuda-graph

PINNs-Torch, Physics-informed Neural Networks (PINNs) implemented in PyTorch.

Foundry materializes CUDA graphs along with its execution context to disk to support fast cold start of serving engines.

A comprehensive guide to using CUDA Graphs effectively with PyTorch, covering CUDA fundamentals, PyTorch integration, Megatron-LM implementations, and practical troubleshooting.

A light, transparent, and modular inference & quantization engine for studying LLMs.

Lightweight white-box LLM inference engine & Triton kernel benchmark for legacy GPUs (Turing sm_75) — CUDA Graph + custom Flash-Decoding/Prefill, 3.34x vLLM decode TPS on RTX 2080 Ti.

A vLLM-class inference and RL-rollout engine for embodied models with cross-environment batching, episode-affine data parallelism, and CUDA-graph capture.

PyTorch CUDA Graph Capture Examples

This repository contains the source code for Grape.

A minimalist, production-ready framework for training small language models through 4 sequential stages (Pretraining, SFT, Alignment, Reasoning). Optimized for extreme performance with a hardware-aware inference engine featuring CUDA Graphs, Static KV Caching, and Topic-Aware Conversational Memory.

Compression-aware Qwen3-VL inference engine with torch.compile, CUDA Graph, scaled-FP8 KV, visual compaction, and multimodal prefix caching.

🚀 从零手写的 Qwen3 高性能推理引擎 —— 1,500 行纯 Python 实现 Paged KV Cache · 连续批处理 · FlashAttention-2 · CUDA Graph,零框架依赖(不基于 vLLM/SGLang),单卡 256 并发 1,200+ tok/s

Enhancing CUDA Intra-Streaming-Multiprocessor Parallelism for Large Language Models via Fine-Grained Task Graph

Lightweight LLM inference framework inspired by DeepInfer: paged KV cache, continuous batching, CUDA Graph, pure PyTorch, Windows-friendly. Runs Qwen2.5-1.5B on 6GB VRAM.

Speculative decoding (n-gram / draft model, dynamic k) and vLLM V1-style mixed batching on nano-vllm, benchmarked against vLLM on A100

Triton kernels and a fused QK-norm + RoPE + paged-KV-write kernel for nano-vllm

Qwen3.8-27B 双 RTX 2080 Ti 22GB + NVLink 的 vLLM 部署（上游 main 路线）：SM75 移植、MTP 下保住 FULL cudagraph（单并发解码 54.1→37.5 ms/步）、256K/500K 上下文与 128K tiered offload profile

Profiling CUDA Graph vs torch.compile vs eager decode on RTX 2070 — measuring per-token latency reduction across batch sizes, sequence lengths, and GPT-2 model sizes.

Single-stream measurements of three local inference engines (vLLM, NInfer, LM Studio) on one RTX 5090: context depth 0-96K, KV cache dtypes, CUDA graphs, speculative decoding.

Speculative decoding in nano-vLLM with DFlash and DSpark block drafts: 2.08–2.20× at C=1, 1.15× at C=64 on Qwen3-8B, RTX PRO 6000.

实时语音转写系统：FireRedASR2S + CUDA Graph 解码加速（RTF 0.024，20 秒音频 0.47 秒）+ 声纹说话人分离（在线句内声学分段 + 离线定版两遍式）。Real-time ASR with voiceprint diarization.