#

inference-engine

Here are 729 public repositories matching this topic...

halfrost / Halfrost-Field

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记

Updated Aug 31, 2026
Go

FareedKhan-dev / kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Updated Aug 26, 2026
C

FedML-AI / FedML

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

machine-learning deep-learning inference-engine model-deployment model-serving distributed-training federated-learning mlops edge-ai ai-agent on-device-training

Updated Oct 28, 2025
Python

zjhellofss / KuiperInfer

校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step

deep-neural-networks caffe deep-learning graph-algorithms inference pytorch yolo convolution diy resnet maxpooling sigmoid inference-engine ncnn relu yolov5 pnnx

Updated Jun 22, 2025
C++

hyperjumptech / grule-rule-engine

Rule engine implementation in Golang

golang rule-engine rule hacktoberfest inference-engine rule-based rule-based-engine hacktoberfest2021

Updated Feb 10, 2026
Go

onediff

siliconflow / onediff

OneDiff: An out-of-the-box acceleration library for diffusion models.

cuda pytorch lora lcm performance-optimization inference-engine diffusion-models stable-diffusion diffusers sd-webui comfyui sdxl aigc-serving lcm-lora stable-video-diffusion sdxl-turbo comfyui-workflow

Updated Dec 4, 2025
Jupyter Notebook

youssofal / MTPLX

3x faster speeds on MLX | Qwen 3.8 27B | Native MTP Speculative Decoding On Apple Silicon With No External Drafter.

metal mtp mlx inference-engine apple-silicon local-ai qwen speculative-decoding speculative-sampling openai-compatible qwen3-next anthropic-compatible native-mtp mtplx

Updated Sep 2, 2026
Python

sonar

dphnAI / sonar

Large-scale LLM inference engine

machine-learning cuda intel api-rest lora rocm inference-engine tpu inferentia speculative-decoding

Updated Aug 13, 2026
C++

xllm

xLLM-AI / xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

inference glm inference-engine large-language-models llm-inference qwen deepseek

Updated Sep 1, 2026
C++

ovg-project / kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

serverless inference-engine llm llm-serving vllm llm-inference ollama llm-framework sglang kvcache gpu-sharing kvcached gpu-mutiplexing kvcache-optimization elastic-kvcache online-offline-coserve

Updated Aug 23, 2026
Python

Tencent / FeatherCNN

FeatherCNN is a high performance inference engine for convolutional neural networks.

android ios caffe convolutional-neural-networks arm-neon inference-engine

Updated Sep 24, 2019
C++

qualcomm / ai-hub-models

Qualcomm® AI Hub Models is our collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.

machine-learning inference pytorch machinelearning deeplearning demos inference-engine onnx tensorflow-lite qnn inference-api

Updated Sep 2, 2026
Python

PaddlePaddle / Paddle.js

Paddle.js is a web project for Baidu PaddlePaddle, which is an open source deep learning framework running in the browser. Paddle.js can either load a pre-trained model, or transforming a model from paddle-hub with model transforming tools provided by Paddle.js. It could run in every browser with WebGL/WebGPU/WebAssembly supported. It could also…

webgl ocr deep-learning model webassembly webgpu inference-engine paddlepaddle

Updated Apr 3, 2024
JavaScript

nobodywho-ooo / nobodywho

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

kotlin python swift react-native ai tts godot godot-engine flutter slm stt expo inference-engine godot-plugin on-device-ai godot4 llm python-llm

Updated Sep 2, 2026
Rust

jjang-ai / mlxstudio

MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)

ai macbook inference mlx ai-agents inference-engine apple-silicon openai-api llm anthropic lmstudio anthropic-api mlxllm mlxstudio macstudio omlx omlx-alternative

Updated Sep 1, 2026

zhihu / ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

cuda pytorch llama gpt inference-engine model-serving llm llm-serving llm-inference deepseek-r1

Updated Mar 18, 2026
C++

insight-platform / Savant

Python Computer Vision & Video Analytics Framework With Batteries Included

opencv machine-learning video computer-vision deep-learning cuda nvidia yolo object-detection deepstream tensorrt inference-engine instance-segmentation edge-computing peoplenet nvidia-deepstream-sdk yolov5-face yolov8 yolov8-face

Updated Sep 1, 2026
Python

Adlik / Adlik

Adlik: Toolkit for Accelerating Deep Learning Inference

deep-learning compiler inference docker-images tensorflow-serving tensorrt inference-engine openvino model-optimizer

Updated Dec 27, 2023
C++

msnh2012 / Msnhnet

🔥 (yolov3 yolov4 yolov5 unet ...)A mini pytorch inference framework which inspired from darknet.

pytorch darknet inference-engine mobilenetyolo mobilenetv2 yolov3 yolov4 yolov5 jetson-nx

Updated Apr 23, 2023
C++

pegainfer

pegainfer-project / pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Updated Sep 1, 2026
Rust

Add this topic to your repo

To associate your repository with the inference-engine topic, visit your repo's landing page and select "manage topics."