From c9a84b2b058f7066c3ce7dee2c9146a919d0618b Mon Sep 17 00:00:00 2001 From: john-rocky Date: Thu, 10 Sep 2026 13:24:40 +0900 Subject: [PATCH 1/7] docs: LiteRT-LM cookbook and Agent Skill for MiniCPM5-2B / MiniCPM5-1B (Android / iOS / desktop) - docs/deployment/litert.md: the litert-lm CLI (download from the Hub, CPU / GPU, thinking on / off / budget, sampling), Android through the AI Edge Gallery app and the Kotlin API, an iOS pointer, the measured tables from the litert-community card, pitfalls. Every command was run with litert-lm 0.17.0; the Kotlin path on a Galaxy S26 with litertlm-android 0.17.0. - skills/minicpm5-deploy-litert/SKILL.md: the paired Agent Skill (same shape as minicpm5-deploy-mlx). - skills/minicpm5-deploy/SKILL.md: one checkpoint row, one decision-matrix row, one default pick, one line on the sanity check. - README.md / README-cn.md: one row in each deployment table, `litert` in the router's route list, backend count 7 -> 8. --- README-cn.md | 6 +- README.md | 6 +- docs/deployment/litert.md | 146 +++++++++++++++++++++++++ skills/minicpm5-deploy-litert/SKILL.md | 75 +++++++++++++ skills/minicpm5-deploy/SKILL.md | 5 + 5 files changed, 234 insertions(+), 4 deletions(-) create mode 100644 docs/deployment/litert.md create mode 100644 skills/minicpm5-deploy-litert/SKILL.md diff --git a/README-cn.md b/README-cn.md index 7d0273e..ef62b64 100644 --- a/README-cn.md +++ b/README-cn.md @@ -299,6 +299,7 @@ MiniCPM5-2B 使用**标准** `LlamaForCausalLM` **架构**，主流推理引擎 | LM Studio | GGUF，Mac 桌面应用与 OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit，Apple Silicon 本地推理 | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | +| LiteRT-LM | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU（社区转换） | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### 微调 @@ -561,7 +562,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | 顶层 Skill | 作用 | 路由到 | | --- | --- | --- | -| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | 推理路由 | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` | +| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | 推理路由 | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` · `litert` | | **[`minicpm5-finetune`](./skills/minicpm5-finetune/SKILL.md)** | 微调路由 | `trl` · `llamafactory` · `ms-swift` · `unsloth` · `xtuner` | 在 Cursor / Claude Code 中可以这样调用：agent 会读取顶层 Skill，并根据目标后端、硬件和数据路径选择对应的子 Skill 与 cookbook，再执行命令并回报结果。 @@ -573,7 +574,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 下方按推理后端和微调框架列出对应的 cookbook 与子 Skill。量化模型不单独作为后端列出，而是在可加载对应格式的推理后端中说明。 -**推理部署**（7 个后端） +**推理部署**（8 个后端） | 后端 | 适用模型 / 场景 | Cookbook | 对应 Agent Skill | | --- | --- | --- | --- | @@ -584,6 +585,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | [Ollama](https://github.com/ollama/ollama) | GGUF，本地端侧运行 | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF，Mac 桌面应用与 OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit，Apple Silicon 本地推理 | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU（社区转换） | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **微调**（5 个框架） diff --git a/README.md b/README.md index 2b6e0f2..39781f4 100644 --- a/README.md +++ b/README.md @@ -297,6 +297,7 @@ MiniCPM5-2B uses the **standard `LlamaForCausalLM` architecture**, so mainstream | LM Studio | GGUF Mac desktop app and OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit local inference on Apple Silicon | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF local on-device, CPU, Desktop & Server | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | +| LiteRT-LM | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU (community conversion) | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### Fine-tuning @@ -559,7 +560,7 @@ The two top-level skills cover deployment and fine-tuning: | Top-level skill | What it does | Routes to | | --- | --- | --- | -| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | Inference router | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` | +| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | Inference router | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` · `litert` | | **[`minicpm5-finetune`](./skills/minicpm5-finetune/SKILL.md)** | Fine tuning router | `trl` · `llamafactory` · `ms-swift` · `unsloth` · `xtuner` | In Cursor / Claude Code, you can call them like this: the agent reads the top-level skill, selects the matching sub-skill and cookbook based on the target backend, hardware, and data path, then runs the command and reports back. @@ -571,7 +572,7 @@ In Cursor / Claude Code, you can call them like this: the agent reads the top-le The tables below list the cookbook and sub-skill for each inference backend and fine-tuning framework. Quantized models are not listed as standalone backends; they are described under the inference backend that can load each format. -**Inference Deployment** (7 backends) +**Inference Deployment** (8 backends) | Backend | Model format / use case | Cookbook | Paired Agent Skill | | --- | --- | --- | --- | @@ -581,6 +582,7 @@ The tables below list the cookbook and sub-skill for each inference backend and | [Ollama](https://github.com/ollama/ollama) | GGUF local on-device runtime | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF Mac desktop app and OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit local inference on Apple Silicon | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU (community conversion) | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF local on-device, CPU, Desktop & Server | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **Fine tuning** (5 frameworks) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md new file mode 100644 index 0000000..b89b8d1 --- /dev/null +++ b/docs/deployment/litert.md @@ -0,0 +1,146 @@ +# Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) + +[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models (LiteRT is the runtime formerly called TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones and desktops, through a command-line tool, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone or an app**. For a Python process on a Mac use [MLX](./mlx.md); for a GGUF workflow use [llama.cpp](./llama_cpp.md) or [Ollama](./ollama.md). + +The MiniCPM5-2B bundles below are **community conversions** (not an OpenBMB or Google release), hosted in [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B); that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). Every command on this page was run with `litert-lm` 0.17.0 on an Apple M4 Max; the Android check used `litertlm-android` 0.17.0 on a Galaxy S26. + +## TL;DR + +```bash +uv tool install litert-lm # the LiteRT-LM CLI (0.17.0 at the time of writing) + +# Downloads the 1.55 GB int4 bundle on first use, runs it on the CPU: +litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm \ + --prompt "What is the capital of France?" +# [thought] … The capital of France is Paris. I should provide a clear and concise answer. [/thought] +# The capital of France is Paris. +``` + +The file lands in `~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/`; later runs skip the download. GPU, and the thinking switches: + +```bash +MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_int4.litertlm + +litert-lm run "$MODEL" --backend gpu --prompt "1+1=?" # reasoning on the thought channel, then: 2 +litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?" # direct answer, no reasoning +litert-lm run "$MODEL" --backend gpu --thinking-budget 2048 --prompt "1+1=?" +litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?" # sampled (see Thinking) +``` + +## Pre-converted bundles + +| Model | File | Recipe | Size | Runs on | +| --- | --- | --- | --- | --- | +| MiniCPM5-2B | [`MiniCPM5-2B_int4.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int4.litertlm) | int4 blockwise-32 linears, int8 embedding | 1.55 GB | CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro | +| MiniCPM5-2B | [`MiniCPM5-2B_int8.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int8.litertlm) | int8 dynamic linears + embedding, fp32 activations declared | 2.60 GB | CPU + GPU: Mac, Galaxy S26 (not iOS, see pitfalls) | +| MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32.litertlm) | int4 block-32 linears, int8 embedding and LM head, fp32 activations | 0.79 GB | CPU (Mac) | +| MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm) | same recipe, graph laid out for the GPU | 0.79 GB | GPU (Mac) | +| MiniCPM5-1B | [`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/MiniCPM5-1B_dynamic_wi8_afp32.litertlm) | int8 dynamic, fp32 activations | 1.11 GB | CPU (Mac) | + +Which 2B file: **int4 is the phone file** (smaller, fastest GPU decode on every device measured) and the right one for direct answers or short reasoning. **int8 is the file when the reasoning has to complete**: on the same questions its thinking chains are 3–4× shorter than int4's and they terminate where int4 runs into the token budget. Both 2B files embed the checkpoint's own `chat_template.jinja`, so `enable_thinking` and the tool-calling format work unchanged, and both declare the `thought` channel (next section). The 2B repo also carries two CPU-only files described in its card. + +```bash +# int8 (2.60 GB), when the reasoning has to finish; the GPU here is the Mac's Metal: +litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int8.litertlm --backend gpu \ + --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?" + +# MiniCPM5-1B: +litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32.litertlm --prompt "1+1=?" +litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm --backend gpu --prompt "1+1=?" +``` + +## Thinking + +Thinking is the model's default: with no thinking flag it decides for itself and, in practice, reasons before every answer. The bundles declare the reasoning as a `thought` channel, so the CLI prints it between `[thought]` and `[/thought]` and the Kotlin / Swift APIs hand it to you separately (`channels["thought"]`); the answer text stays clean. + +- `--thinking false`: direct answers. Two- to seven-token replies on trivial questions, about 10× faster turns. +- `--thinking-budget N`: caps the reasoning. Keep N at 2048 or more (4096 for math): a chain cut mid-thought yields no final answer at all. +- Sampling: OpenBMB recommends `temperature 1.0`, `top_p 0.95`. The CLI's default top-k is 1, which is greedy whatever the temperature, so pass all three: `--top-k 40 --top-p 0.95 --temperature 1.0` (with `--top-p --temperature` alone the output stays byte-identical across seeds). The card's correctness numbers are greedy. +- The KV cache is 4096 tokens by default (`--max-num-tokens` to change it); the prompt format is the checkpoint's ChatML template. + +## Android + +**AI Edge Gallery (no code).** Install [Google AI Edge Gallery](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery), open **Model manager**, tap **+** and choose **Import from HF**, then paste the Hugging Face link of the `.litertlm` file (or download the file to the phone first and use **From local model file**). The [litert-community card](https://huggingface.co/litert-community/MiniCPM5-2B#edge-gallery-app-android) and the [Gallery wiki](https://github.com/google-ai-edge/gallery/wiki) have the details. + +**Kotlin API (your own app).** The runtime is one dependency from Google Maven ([Android guide](https://developers.google.com/edge/litert-lm/android)): + +```kotlin +// build.gradle.kts — repositories { google() } +implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0") +``` + +The GPU backend needs the OpenCL library declared in `AndroidManifest.xml` (Android 12+ hides vendor libraries from apps otherwise; without it the first message fails with `Can not find OpenCL library on this device`): + +```xml + + + + + +``` + +```kotlin +import com.google.ai.edge.litertlm.* + +val engine = Engine(EngineConfig( + modelPath = modelFile.absolutePath, // MiniCPM5-2B_int4.litertlm in your app's storage + backend = Backend.GPU(), // or Backend.CPU() + cacheDir = context.cacheDir.absolutePath, // compiled-kernel cache; the first load is slower +)) +engine.initialize() + +engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation -> + val reply = conversation.sendMessage("Explain on-device AI in simple terms.") + println(reply) // the answer + println(reply.channels["thought"]) // the reasoning, kept out of the answer +} +engine.close() +``` + +Checked on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17.0 and the int4 file: the engine initializes on `Backend.GPU()` with every node of every signature delegated to OpenCL (1873 of 1873 on the 1024-token prefill, 1692 of 1692 on decode; only the externalized embedding lookup runs on the CPU), and the reply arrives with the reasoning in `channels["thought"]`. `ConversationConfig` also takes `thinkingConfig = ThinkingConfig(enableThinking = false)` or `ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048)` and a `samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0)`, as in the Android guide. `conversation.sendMessageAsync(...)` streams. + +## iOS + +The int4 file passes the card's 8-question check on an iPhone 17 Pro on both backends (Metal GPU and CPU, init 5.7 s and 2.2 s). The int8 file is a desktop / Android build: its main weight section is 2.33 GB, above the single-section memory-map budget of an iOS app with default entitlements. The Swift API is documented in the [LiteRT-LM Swift guide](https://developers.google.com/edge/litert-lm/swift) (Swift Package Manager, `https://github.com/google-ai-edge/LiteRT-LM`); this page does not cover the Xcode steps. + +## Measured (from the litert-community card) + +Apple M4 Max, `litert-lm benchmark` 0.17.0, 256-token prefill / 256-token decode, 3 runs, no compiled-kernel cache: + +| File | Backend | Prefill | Decode | Time to first token | Init | +| --- | --- | --- | --- | --- | --- | +| int4 | GPU (Metal) | 1699 tok/s | 92.8 tok/s | 0.16 s | 3.7 s | +| int4 | CPU | 149 tok/s | 31.1 tok/s | 1.76 s | 4.5 s | +| int8 (fp32 activations) | GPU (Metal) | 1405 tok/s | 74.7 tok/s | 0.20 s | 3.0 s | +| int8 (fp32 activations) | CPU | 161 tok/s | 30.0 tok/s | 1.62 s | 15.0 s | + +Galaxy S26 (Snapdragon SM8850, Adreno), the LiteRT-LM v0.16.0 release binary, 205-token prompt, 2 runs per cell (ranges shown; a reasoning model decodes its own full response, so decode lengths vary): + +| File | Backend | Prefill | Decode | Time to first token | Peak RSS | +| --- | --- | --- | --- | --- | --- | +| int4 | GPU (OpenCL) | 401–411 tok/s | 16.1–18.6 tok/s | 0.56 s | 1.14 GB | +| int4 | CPU | 39–72 tok/s | 15.6–15.8 tok/s | 2.9–5.3 s | 2.12 GB | +| int8 (fp32 activations) | GPU (OpenCL) | 150–160 tok/s | 10.9–12.8 tok/s | 1.4 s | 1.10 GB | +| int8 (fp32 activations) | CPU | 103–157 tok/s | 11.7 tok/s | 1.4–2.1 s | 2.90 GB | + +Accuracy, GSM8K first 100 test questions, greedy, thinking off (the protocol of OpenBMB's MiniCPM5 cards), 2048 new tokens: bf16 PyTorch 92 %, int8 CPU 91 %, int4 CPU 86 %, int4 GPU 87 %. Both 2B files score 8/8 on the card's 8-question sanity gate on CPU and GPU. + +## Common pitfalls + +- **int4 with thinking on may never close its reasoning.** Quantization costs this 42-layer model its thinking discipline first: on the card's 10-question thinking-on subset the int4 file closes 0/10 on the CPU (it keeps re-checking until the budget), while int8 closes 9/10 like the bf16 model. The same shows on the CLI: `1+1=?` on the CPU and `Explain GQA in one sentence.` on the GPU ran into the 4096-token budget with no answer. When the answer matters, use `--thinking false` on int4, or the int8 file. +- **int8 declares fp32 activations in the bundle.** With the GPU's default fp16 activations the int8 model's reasoning on one gate question ran 2000+ tokens without closing; with fp32 it closes in ~450 tokens. The cost is about 14 % of GPU decode speed, and on Adreno the int8 GPU decode ends up level with the same phone's CPU. +- **int8 on iOS**: not loadable by a default-entitlement app (2.33 GB single section, above). Use int4 on iPhone. +- **Budget cuts leave no answer**: a `--thinking-budget` below ~2048 (4096 for math) truncates the chain and the model emits nothing after it. +- **First run per backend is slow**: the CPU run writes a 1.3 GB XNNPACK cache beside the file and the GPU run compiles its kernels (`--cache no` skips the disk cache; the Kotlin `cacheDir` is the same mechanism). +- **GPU in your own Android app**: without the `` entries above, `initialize()` succeeds and the first message fails with `Can not find OpenCL library on this device`. +- **Temperature without top-k does nothing**: `--temperature` and `--top-p` only take effect together with `--top-k` above 1 (default 1 = greedy). The Kotlin `SamplerConfig` takes the same three fields. +- **Scripts**: the CLI reads stdin; when it runs from a script or a job with an open pipe on stdin, redirect it (`< /dev/null`) or it waits for end-of-input before generating. + +## See also + +- [`ollama.md`](./ollama.md) — one-line CLI path on a laptop, GGUF +- [`mlx.md`](./mlx.md) — Python on Apple Silicon +- [`llama_cpp.md`](./llama_cpp.md) — GGUF on CPU / CUDA +- [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B) — recipe, correctness, all measurements, `litertlm_manifest.json` +- [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B) +- LiteRT-LM guides: [CLI](https://developers.google.com/edge/litert-lm/cli), [Android](https://developers.google.com/edge/litert-lm/android), [Swift](https://developers.google.com/edge/litert-lm/swift), [Python](https://developers.google.com/edge/litert-lm/python) diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md new file mode 100644 index 0000000..a4493e2 --- /dev/null +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -0,0 +1,75 @@ +--- +name: minicpm5-deploy-litert +description: Run MiniCPM5-2B or MiniCPM5-1B on-device with Google's LiteRT-LM runtime — the litert-lm CLI on a desktop, the Kotlin API or the AI Edge Gallery app on Android, the same .litertlm bundle on CPU or GPU. Use when the user says "LiteRT", "LiteRT-LM", "litertlm", ".litertlm", "Android", "Edge Gallery", "on-device app", or wants one bundle for phones and desktops. +--- + +# Deploy MiniCPM5-2B / MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) + +Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` bundle runs on CPU and GPU through a CLI, Python, Kotlin and Swift. The MiniCPM5-2B bundles are community conversions hosted in `litert-community` (not an OpenBMB or Google release). Human-readable reference: [`docs/deployment/litert.md`](../../docs/deployment/litert.md). + +## Required input + +| Var | Example | Default | +| --- | --- | --- | +| `LITERT_REPO` | `litert-community/MiniCPM5-2B` or `litert-community/MiniCPM5-1B` | `litert-community/MiniCPM5-2B` | +| `FILE` | `MiniCPM5-2B_int4.litertlm` (1.55 GB, the phone file) or `MiniCPM5-2B_int8.litertlm` (2.60 GB, reasoning that has to finish); 1B: `minicpm_wi4b32_wi8_afp32.litertlm` (CPU) or `minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm` (GPU) | `MiniCPM5-2B_int4.litertlm` | +| `BACKEND` | `cpu` or `gpu` | `cpu` | +| `THINKING` | `true` (reason first) or `false` (direct answer) | the model decides; in practice it reasons | +| `PROMPT` | `1+1=?` | `1+1=?` | + +## Steps + +### 1. Install (once) + +```bash +uv tool install litert-lm # litert-lm 0.17.0 at the time of writing +``` + +### 2A. Run a pre-converted bundle (desktop CLI) + +```bash +litert-lm run --from-huggingface-repo="${LITERT_REPO}" "${FILE}" \ + --backend "${BACKEND}" --thinking "${THINKING}" --prompt "${PROMPT}" +``` + +The first run downloads the file into `~/.litert-lm/cache/huggingface//`; later runs skip the download. With thinking on, the reasoning prints between `[thought]` and `[/thought]` and the answer follows; `--thinking-budget 2048` caps it (never lower: a chain cut mid-thought yields no answer). Sampling as OpenBMB recommends: `--top-k 40 --top-p 0.95 --temperature 1.0` (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing). + +### 2B. Android + +- **No code**: the [AI Edge Gallery](https://play.google.com/store/apps/details?id=com.google.ai.edge.gallery) app, Model manager → **+** → **Import from HF** → paste the file's Hugging Face link. +- **Your own app**: `implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0")` from Google Maven, the `libOpenCL.so` `` entries in the manifest for the GPU, then: + +```kotlin +val engine = Engine(EngineConfig(modelPath = path, backend = Backend.GPU(), cacheDir = context.cacheDir.absolutePath)) +engine.initialize() +engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation -> + val reply = conversation.sendMessage("1+1=?") + println(reply); println(reply.channels["thought"]) // answer; reasoning +} +``` + +`ConversationConfig(thinkingConfig = ThinkingConfig(enableThinking = false))` turns thinking off. Full snippet, manifest and the Galaxy S26 check: the cookbook. + +### 3. Validate + +The reply contains `2` for `1+1=?`. With thinking on it comes after `[/thought]`; with `--thinking false` it is the whole reply. On the int4 file prefer `--thinking false` or `--backend gpu` for this check: on the CPU with thinking on, `1+1=?` can run to the token budget without an answer (pitfalls). + +## Common pitfalls + +- **int4 + thinking on may not close its reasoning** (CPU especially): the card's 10-question thinking-on subset closes 0/10 on int4 CPU, 9/10 on int8. Use `--thinking false` on int4, or the int8 file, when the answer matters. +- **int8 declares fp32 activations** (a correctness fix for the GPU's fp16 default); GPU decode is ~14 % slower for it, and on Adreno int8 GPU decode is level with the CPU. int4 is the phone file. +- **int8 is not for iOS**: its 2.33 GB weight section exceeds a default-entitlement app's single-section memory-map budget. iPhone: int4 (checked on iPhone 17 Pro, GPU and CPU). +- **First run per backend is slow**: a 1.3 GB XNNPACK cache is written beside the file on the CPU, kernels are compiled on the GPU (`--cache no` to skip the disk cache). +- **Android GPU without the manifest entries**: `initialize()` succeeds, the first message fails with `Can not find OpenCL library on this device`. +- **Temperature without top-k does nothing**: pass `--top-k` above 1 together with `--top-p` / `--temperature` (default top-k 1 = greedy). +- **Scripts**: the CLI reads stdin; redirect it (`< /dev/null`) when running from a job with an open pipe. + +## When NOT to use + +- Python on a Mac, no app → `minicpm5-deploy-mlx` +- GGUF, one-line CLI on a laptop → `minicpm5-deploy-ollama`; CPU / CUDA build → `minicpm5-deploy-llama-cpp` +- Server with an OpenAI-compatible endpoint → `minicpm5-deploy-vllm` or `minicpm5-deploy-sglang` + +## Reference + +[`docs/deployment/litert.md`](../../docs/deployment/litert.md) diff --git a/skills/minicpm5-deploy/SKILL.md b/skills/minicpm5-deploy/SKILL.md index 301369e..d120614 100644 --- a/skills/minicpm5-deploy/SKILL.md +++ b/skills/minicpm5-deploy/SKILL.md @@ -24,6 +24,7 @@ Before picking a backend, you MUST know: | **HF fp16 (recommended)** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) or [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) | `transformers` / `vllm` (no `--quantization`) / `vllm-ascend` / `sglang` / any `minicpm5-finetune-*` | | GGUF F16 / Q8_0 / Q4_K_M | [`openbmb/MiniCPM5-2B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF) or [`openbmb/MiniCPM5-1B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) | `minicpm5-deploy-llama-cpp` / `-ollama` / `-lmstudio` | | MLX (Apple Silicon) | [`openbmb/MiniCPM5-2B-MLX`](https://huggingface.co/openbmb/MiniCPM5-2B-MLX) or [`openbmb/MiniCPM5-1B-MLX`](https://huggingface.co/openbmb/MiniCPM5-1B-MLX) | `minicpm5-deploy-mlx` | +| LiteRT-LM `.litertlm` (Android / iOS / desktop, CPU + GPU; community conversion) | [`litert-community/MiniCPM5-2B`](https://huggingface.co/litert-community/MiniCPM5-2B) or [`litert-community/MiniCPM5-1B`](https://huggingface.co/litert-community/MiniCPM5-1B) | `minicpm5-deploy-litert` | If the user has a local copy, accept any directory path that contains `config.json` and `model.safetensors` (or the equivalent GGUF / MLX layout). @@ -39,6 +40,7 @@ If the user has a local copy, accept any directory path that contains `config.js | "Ollama" / "ollama run" / "Modelfile" | macOS / Linux laptop | GGUF | **`minicpm5-deploy-ollama`** | | "LM Studio" / "desktop GUI" | macOS / Windows / Linux | GGUF or MLX | **`minicpm5-deploy-lmstudio`** | | "MLX" / "Apple Silicon native" / "fastest on Mac" | Apple Silicon | MLX | **`minicpm5-deploy-mlx`** | +| "Android" / "on-device app" / "Edge Gallery" / "LiteRT" / "LiteRT-LM" / "litertlm" | Android phone, iPhone, or a desktop app (CPU or GPU) | LiteRT-LM `.litertlm` | **`minicpm5-deploy-litert`** | If the user **has not specified** any of the above and asks "how do I run this?": @@ -46,6 +48,7 @@ If the user **has not specified** any of the above and asks "how do I run this?" - **Ascend NPU, want an OpenAI-compatible server**: pick `minicpm5-deploy-vllm-ascend`. - **CUDA box, want minimal Python**: pick `minicpm5-deploy-transformers`. - **Apple Silicon laptop**: pick `minicpm5-deploy-ollama` (easiest) or `minicpm5-deploy-mlx` (fastest). +- **Android phone, or an on-device app (Android / iOS)**: pick `minicpm5-deploy-litert`. - **CPU only / Windows / low-VRAM**: pick `minicpm5-deploy-llama-cpp` (Q4_K_M). ## 3. Invocation contract @@ -70,6 +73,8 @@ curl http://localhost:PORT/v1/chat/completions \ Expected: HTTP 200 with `choices[0].message.content` containing `"2"`. +`minicpm5-deploy-litert` runs in-process with no HTTP server; its own step 3 (`litert-lm run … --prompt "1+1=?"`, the answer contains `2`) is the equivalent check. + ## 5. Known cross-backend pitfalls These are common to multiple backends — surface to the user up front: From f92d4fa09ab3a38fc1cd9b35c59a42d00d13f134 Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 14:10:26 +0900 Subject: [PATCH 2/7] docs: drop the community-conversion wording from the LiteRT-LM cookbook, skill and README rows --- README-cn.md | 4 ++-- README.md | 4 ++-- docs/deployment/litert.md | 2 +- skills/minicpm5-deploy-litert/SKILL.md | 2 +- skills/minicpm5-deploy/SKILL.md | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README-cn.md b/README-cn.md index ef62b64..f309860 100644 --- a/README-cn.md +++ b/README-cn.md @@ -299,7 +299,7 @@ MiniCPM5-2B 使用**标准** `LlamaForCausalLM` **架构**，主流推理引擎 | LM Studio | GGUF，Mac 桌面应用与 OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit，Apple Silicon 本地推理 | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | -| LiteRT-LM | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU（社区转换） | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | +| LiteRT-LM | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### 微调 @@ -585,7 +585,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | [Ollama](https://github.com/ollama/ollama) | GGUF，本地端侧运行 | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF，Mac 桌面应用与 OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit，Apple Silicon 本地推理 | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | -| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU（社区转换） | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **微调**（5 个框架） diff --git a/README.md b/README.md index 39781f4..f0ee33f 100644 --- a/README.md +++ b/README.md @@ -297,7 +297,7 @@ MiniCPM5-2B uses the **standard `LlamaForCausalLM` architecture**, so mainstream | LM Studio | GGUF Mac desktop app and OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit local inference on Apple Silicon | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF local on-device, CPU, Desktop & Server | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | -| LiteRT-LM | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU (community conversion) | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | +| LiteRT-LM | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### Fine-tuning @@ -582,7 +582,7 @@ The tables below list the cookbook and sub-skill for each inference backend and | [Ollama](https://github.com/ollama/ollama) | GGUF local on-device runtime | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF Mac desktop app and OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit local inference on Apple Silicon | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | -| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU (community conversion) | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF local on-device, CPU, Desktop & Server | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **Fine tuning** (5 frameworks) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index b89b8d1..d2f9d78 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -2,7 +2,7 @@ [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models (LiteRT is the runtime formerly called TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones and desktops, through a command-line tool, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone or an app**. For a Python process on a Mac use [MLX](./mlx.md); for a GGUF workflow use [llama.cpp](./llama_cpp.md) or [Ollama](./ollama.md). -The MiniCPM5-2B bundles below are **community conversions** (not an OpenBMB or Google release), hosted in [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B); that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). Every command on this page was run with `litert-lm` 0.17.0 on an Apple M4 Max; the Android check used `litertlm-android` 0.17.0 on a Galaxy S26. +The MiniCPM5-2B bundles are hosted in [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B); that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). Every command on this page was run with `litert-lm` 0.17.0 on an Apple M4 Max; the Android check used `litertlm-android` 0.17.0 on a Galaxy S26. ## TL;DR diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index a4493e2..1daa4dd 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -5,7 +5,7 @@ description: Run MiniCPM5-2B or MiniCPM5-1B on-device with Google's LiteRT-LM ru # Deploy MiniCPM5-2B / MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) -Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` bundle runs on CPU and GPU through a CLI, Python, Kotlin and Swift. The MiniCPM5-2B bundles are community conversions hosted in `litert-community` (not an OpenBMB or Google release). Human-readable reference: [`docs/deployment/litert.md`](../../docs/deployment/litert.md). +Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` bundle runs on CPU and GPU through a CLI, Python, Kotlin and Swift. The bundles are hosted in `litert-community`. Human-readable reference: [`docs/deployment/litert.md`](../../docs/deployment/litert.md). ## Required input diff --git a/skills/minicpm5-deploy/SKILL.md b/skills/minicpm5-deploy/SKILL.md index d120614..371a962 100644 --- a/skills/minicpm5-deploy/SKILL.md +++ b/skills/minicpm5-deploy/SKILL.md @@ -24,7 +24,7 @@ Before picking a backend, you MUST know: | **HF fp16 (recommended)** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) or [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) | `transformers` / `vllm` (no `--quantization`) / `vllm-ascend` / `sglang` / any `minicpm5-finetune-*` | | GGUF F16 / Q8_0 / Q4_K_M | [`openbmb/MiniCPM5-2B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF) or [`openbmb/MiniCPM5-1B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) | `minicpm5-deploy-llama-cpp` / `-ollama` / `-lmstudio` | | MLX (Apple Silicon) | [`openbmb/MiniCPM5-2B-MLX`](https://huggingface.co/openbmb/MiniCPM5-2B-MLX) or [`openbmb/MiniCPM5-1B-MLX`](https://huggingface.co/openbmb/MiniCPM5-1B-MLX) | `minicpm5-deploy-mlx` | -| LiteRT-LM `.litertlm` (Android / iOS / desktop, CPU + GPU; community conversion) | [`litert-community/MiniCPM5-2B`](https://huggingface.co/litert-community/MiniCPM5-2B) or [`litert-community/MiniCPM5-1B`](https://huggingface.co/litert-community/MiniCPM5-1B) | `minicpm5-deploy-litert` | +| LiteRT-LM `.litertlm` (Android / iOS / desktop, CPU + GPU) | [`litert-community/MiniCPM5-2B`](https://huggingface.co/litert-community/MiniCPM5-2B) or [`litert-community/MiniCPM5-1B`](https://huggingface.co/litert-community/MiniCPM5-1B) | `minicpm5-deploy-litert` | If the user has a local copy, accept any directory path that contains `config.json` and `model.safetensors` (or the equivalent GGUF / MLX layout). From 2be21b481cf51223581cd7966c7ad3f33ea8ce48 Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 14:28:15 +0900 Subject: [PATCH 3/7] docs: link LiteRT, drop the pointers to other backends, label the bundle table as tested devices --- docs/deployment/litert.md | 7 ++----- skills/minicpm5-deploy-litert/SKILL.md | 3 +-- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index d2f9d78..591337b 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -1,6 +1,6 @@ # Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) -[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models (LiteRT is the runtime formerly called TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones and desktops, through a command-line tool, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone or an app**. For a Python process on a Mac use [MLX](./mlx.md); for a GGUF workflow use [llama.cpp](./llama_cpp.md) or [Ollama](./ollama.md). +[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models, built on [LiteRT](https://github.com/google-ai-edge/litert) (formerly TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones and desktops, through a command-line tool, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone or an app**. The MiniCPM5-2B bundles are hosted in [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B); that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). Every command on this page was run with `litert-lm` 0.17.0 on an Apple M4 Max; the Android check used `litertlm-android` 0.17.0 on a Galaxy S26. @@ -29,7 +29,7 @@ litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 - ## Pre-converted bundles -| Model | File | Recipe | Size | Runs on | +| Model | File | Recipe | Size | Tested on | | --- | --- | --- | --- | --- | | MiniCPM5-2B | [`MiniCPM5-2B_int4.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int4.litertlm) | int4 blockwise-32 linears, int8 embedding | 1.55 GB | CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro | | MiniCPM5-2B | [`MiniCPM5-2B_int8.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int8.litertlm) | int8 dynamic linears + embedding, fp32 activations declared | 2.60 GB | CPU + GPU: Mac, Galaxy S26 (not iOS, see pitfalls) | @@ -138,9 +138,6 @@ Accuracy, GSM8K first 100 test questions, greedy, thinking off (the protocol of ## See also -- [`ollama.md`](./ollama.md) — one-line CLI path on a laptop, GGUF -- [`mlx.md`](./mlx.md) — Python on Apple Silicon -- [`llama_cpp.md`](./llama_cpp.md) — GGUF on CPU / CUDA - [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B) — recipe, correctness, all measurements, `litertlm_manifest.json` - [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B) - LiteRT-LM guides: [CLI](https://developers.google.com/edge/litert-lm/cli), [Android](https://developers.google.com/edge/litert-lm/android), [Swift](https://developers.google.com/edge/litert-lm/swift), [Python](https://developers.google.com/edge/litert-lm/python) diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index 1daa4dd..09205b9 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -66,8 +66,7 @@ The reply contains `2` for `1+1=?`. With thinking on it comes after `[/thought]` ## When NOT to use -- Python on a Mac, no app → `minicpm5-deploy-mlx` -- GGUF, one-line CLI on a laptop → `minicpm5-deploy-ollama`; CPU / CUDA build → `minicpm5-deploy-llama-cpp` +- GGUF files already in hand → `minicpm5-deploy-llama-cpp` or `minicpm5-deploy-ollama` - Server with an OpenAI-compatible endpoint → `minicpm5-deploy-vllm` or `minicpm5-deploy-sglang` ## Reference From 140620a5b84051aaeba6c87450a93eabd9259d83 Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 14:29:49 +0900 Subject: [PATCH 4/7] docs: int8 on iOS is an entitlement setting, not a wall --- docs/deployment/litert.md | 6 +++--- skills/minicpm5-deploy-litert/SKILL.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index 591337b..14f7028 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -32,7 +32,7 @@ litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 - | Model | File | Recipe | Size | Tested on | | --- | --- | --- | --- | --- | | MiniCPM5-2B | [`MiniCPM5-2B_int4.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int4.litertlm) | int4 blockwise-32 linears, int8 embedding | 1.55 GB | CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro | -| MiniCPM5-2B | [`MiniCPM5-2B_int8.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int8.litertlm) | int8 dynamic linears + embedding, fp32 activations declared | 2.60 GB | CPU + GPU: Mac, Galaxy S26 (not iOS, see pitfalls) | +| MiniCPM5-2B | [`MiniCPM5-2B_int8.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int8.litertlm) | int8 dynamic linears + embedding, fp32 activations declared | 2.60 GB | CPU + GPU: Mac, Galaxy S26 (iOS: see below) | | MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32.litertlm) | int4 block-32 linears, int8 embedding and LM head, fp32 activations | 0.79 GB | CPU (Mac) | | MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm) | same recipe, graph laid out for the GPU | 0.79 GB | GPU (Mac) | | MiniCPM5-1B | [`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/MiniCPM5-1B_dynamic_wi8_afp32.litertlm) | int8 dynamic, fp32 activations | 1.11 GB | CPU (Mac) | @@ -101,7 +101,7 @@ Checked on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17 ## iOS -The int4 file passes the card's 8-question check on an iPhone 17 Pro on both backends (Metal GPU and CPU, init 5.7 s and 2.2 s). The int8 file is a desktop / Android build: its main weight section is 2.33 GB, above the single-section memory-map budget of an iOS app with default entitlements. The Swift API is documented in the [LiteRT-LM Swift guide](https://developers.google.com/edge/litert-lm/swift) (Swift Package Manager, `https://github.com/google-ai-edge/LiteRT-LM`); this page does not cover the Xcode steps. +The int4 file passes the card's 8-question check on an iPhone 17 Pro on both backends (Metal GPU and CPU, init 5.7 s and 2.2 s). The int8 file's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; the `com.apple.developer.kernel.increased-memory-limit` and `com.apple.developer.kernel.extended-virtual-addressing` entitlements lift that limit (sections larger than this have loaded and run on an iPhone 17 Pro with other models; not re-checked with this file). The Swift API is documented in the [LiteRT-LM Swift guide](https://developers.google.com/edge/litert-lm/swift) (Swift Package Manager, `https://github.com/google-ai-edge/LiteRT-LM`); this page does not cover the Xcode steps. ## Measured (from the litert-community card) @@ -129,7 +129,7 @@ Accuracy, GSM8K first 100 test questions, greedy, thinking off (the protocol of - **int4 with thinking on may never close its reasoning.** Quantization costs this 42-layer model its thinking discipline first: on the card's 10-question thinking-on subset the int4 file closes 0/10 on the CPU (it keeps re-checking until the budget), while int8 closes 9/10 like the bf16 model. The same shows on the CLI: `1+1=?` on the CPU and `Explain GQA in one sentence.` on the GPU ran into the 4096-token budget with no answer. When the answer matters, use `--thinking false` on int4, or the int8 file. - **int8 declares fp32 activations in the bundle.** With the GPU's default fp16 activations the int8 model's reasoning on one gate question ran 2000+ tokens without closing; with fp32 it closes in ~450 tokens. The cost is about 14 % of GPU decode speed, and on Adreno the int8 GPU decode ends up level with the same phone's CPU. -- **int8 on iOS**: not loadable by a default-entitlement app (2.33 GB single section, above). Use int4 on iPhone. +- **int8 on iOS needs two entitlements.** With the default entitlements the 2.33 GB weight section does not map; the increased-memory-limit and extended-virtual-addressing entitlements lift that (see iOS). int4 needs neither. - **Budget cuts leave no answer**: a `--thinking-budget` below ~2048 (4096 for math) truncates the chain and the model emits nothing after it. - **First run per backend is slow**: the CPU run writes a 1.3 GB XNNPACK cache beside the file and the GPU run compiles its kernels (`--cache no` skips the disk cache; the Kotlin `cacheDir` is the same mechanism). - **GPU in your own Android app**: without the `` entries above, `initialize()` succeeds and the first message fails with `Can not find OpenCL library on this device`. diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index 09205b9..f207d53 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -58,7 +58,7 @@ The reply contains `2` for `1+1=?`. With thinking on it comes after `[/thought]` - **int4 + thinking on may not close its reasoning** (CPU especially): the card's 10-question thinking-on subset closes 0/10 on int4 CPU, 9/10 on int8. Use `--thinking false` on int4, or the int8 file, when the answer matters. - **int8 declares fp32 activations** (a correctness fix for the GPU's fp16 default); GPU decode is ~14 % slower for it, and on Adreno int8 GPU decode is level with the CPU. int4 is the phone file. -- **int8 is not for iOS**: its 2.33 GB weight section exceeds a default-entitlement app's single-section memory-map budget. iPhone: int4 (checked on iPhone 17 Pro, GPU and CPU). +- **int8 on iOS needs the increased-memory-limit and extended-virtual-addressing entitlements** (2.33 GB weight section; the default entitlements do not map it). int4 needs neither (checked on iPhone 17 Pro, GPU and CPU). - **First run per backend is slow**: a 1.3 GB XNNPACK cache is written beside the file on the CPU, kernels are compiled on the GPU (`--cache no` to skip the disk cache). - **Android GPU without the manifest entries**: `initialize()` succeeds, the first message fails with `Can not find OpenCL library on this device`. - **Temperature without top-k does nothing**: pass `--top-k` above 1 together with `--top-p` / `--temperature` (default top-k 1 = greedy). From 4f07606725ca62339aed8b54d8684cddc2919a2a Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 14:37:14 +0900 Subject: [PATCH 5/7] docs: trim the comments in the code blocks --- docs/deployment/litert.md | 28 ++++++++++++-------------- skills/minicpm5-deploy-litert/SKILL.md | 4 ++-- 2 files changed, 15 insertions(+), 17 deletions(-) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index 14f7028..0adf3da 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -7,13 +7,11 @@ The MiniCPM5-2B bundles are hosted in [litert-community/MiniCPM5-2B](https://hug ## TL;DR ```bash -uv tool install litert-lm # the LiteRT-LM CLI (0.17.0 at the time of writing) +uv tool install litert-lm -# Downloads the 1.55 GB int4 bundle on first use, runs it on the CPU: +# first run downloads the 1.55 GB int4 bundle litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm \ --prompt "What is the capital of France?" -# [thought] … The capital of France is Paris. I should provide a clear and concise answer. [/thought] -# The capital of France is Paris. ``` The file lands in `~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/`; later runs skip the download. GPU, and the thinking switches: @@ -21,10 +19,10 @@ The file lands in `~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/` ```bash MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_int4.litertlm -litert-lm run "$MODEL" --backend gpu --prompt "1+1=?" # reasoning on the thought channel, then: 2 -litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?" # direct answer, no reasoning +litert-lm run "$MODEL" --backend gpu --prompt "1+1=?" +litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?" litert-lm run "$MODEL" --backend gpu --thinking-budget 2048 --prompt "1+1=?" -litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?" # sampled (see Thinking) +litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?" ``` ## Pre-converted bundles @@ -40,7 +38,7 @@ litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 - Which 2B file: **int4 is the phone file** (smaller, fastest GPU decode on every device measured) and the right one for direct answers or short reasoning. **int8 is the file when the reasoning has to complete**: on the same questions its thinking chains are 3–4× shorter than int4's and they terminate where int4 runs into the token budget. Both 2B files embed the checkpoint's own `chat_template.jinja`, so `enable_thinking` and the tool-calling format work unchanged, and both declare the `thought` channel (next section). The 2B repo also carries two CPU-only files described in its card. ```bash -# int8 (2.60 GB), when the reasoning has to finish; the GPU here is the Mac's Metal: +# int8 litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int8.litertlm --backend gpu \ --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?" @@ -65,7 +63,7 @@ Thinking is the model's default: with no thinking flag it decides for itself and **Kotlin API (your own app).** The runtime is one dependency from Google Maven ([Android guide](https://developers.google.com/edge/litert-lm/android)): ```kotlin -// build.gradle.kts — repositories { google() } +// build.gradle.kts implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0") ``` @@ -83,21 +81,21 @@ The GPU backend needs the OpenCL library declared in `AndroidManifest.xml` (Andr import com.google.ai.edge.litertlm.* val engine = Engine(EngineConfig( - modelPath = modelFile.absolutePath, // MiniCPM5-2B_int4.litertlm in your app's storage - backend = Backend.GPU(), // or Backend.CPU() - cacheDir = context.cacheDir.absolutePath, // compiled-kernel cache; the first load is slower + modelPath = modelFile.absolutePath, + backend = Backend.GPU(), + cacheDir = context.cacheDir.absolutePath, )) engine.initialize() engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation -> val reply = conversation.sendMessage("Explain on-device AI in simple terms.") - println(reply) // the answer - println(reply.channels["thought"]) // the reasoning, kept out of the answer + println(reply) + println(reply.channels["thought"]) } engine.close() ``` -Checked on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17.0 and the int4 file: the engine initializes on `Backend.GPU()` with every node of every signature delegated to OpenCL (1873 of 1873 on the 1024-token prefill, 1692 of 1692 on decode; only the externalized embedding lookup runs on the CPU), and the reply arrives with the reasoning in `channels["thought"]`. `ConversationConfig` also takes `thinkingConfig = ThinkingConfig(enableThinking = false)` or `ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048)` and a `samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0)`, as in the Android guide. `conversation.sendMessageAsync(...)` streams. +Checked on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17.0 and the int4 file: the engine initializes on `Backend.GPU()` (`Backend.CPU()` selects the CPU) with every node of every signature delegated to OpenCL (1873 of 1873 on the 1024-token prefill, 1692 of 1692 on decode; only the externalized embedding lookup runs on the CPU), and the reply arrives with the reasoning in `channels["thought"]`. `ConversationConfig` also takes `thinkingConfig = ThinkingConfig(enableThinking = false)` or `ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048)` and a `samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0)`, as in the Android guide. `conversation.sendMessageAsync(...)` streams. ## iOS diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index f207d53..f1cd8a9 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -22,7 +22,7 @@ Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` b ### 1. Install (once) ```bash -uv tool install litert-lm # litert-lm 0.17.0 at the time of writing +uv tool install litert-lm ``` ### 2A. Run a pre-converted bundle (desktop CLI) @@ -44,7 +44,7 @@ val engine = Engine(EngineConfig(modelPath = path, backend = Backend.GPU(), cach engine.initialize() engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation -> val reply = conversation.sendMessage("1+1=?") - println(reply); println(reply.channels["thought"]) // answer; reasoning + println(reply); println(reply.channels["thought"]) } ``` From ed6a888ef20e900599144e91c19cdf16af5341e1 Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 15:27:45 +0900 Subject: [PATCH 6/7] docs: IoT in the LiteRT-LM rows, OpenAI-compatible server section and skill step, thinking and iOS wording from the sources --- README-cn.md | 6 +- README.md | 4 +- docs/deployment/litert.md | 77 ++++++++++++++++---------- skills/minicpm5-deploy-litert/SKILL.md | 39 +++++++------ skills/minicpm5-deploy/SKILL.md | 6 +- 5 files changed, 77 insertions(+), 55 deletions(-) diff --git a/README-cn.md b/README-cn.md index f309860..6e23d94 100644 --- a/README-cn.md +++ b/README-cn.md @@ -299,7 +299,7 @@ MiniCPM5-2B 使用**标准** `LlamaForCausalLM` **架构**，主流推理引擎 | LM Studio | GGUF，Mac 桌面应用与 OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit，Apple Silicon 本地推理 | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | -| LiteRT-LM | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | +| LiteRT-LM | `.litertlm` 端侧运行时：Android / iOS / 桌面 / IoT，CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### 微调 @@ -574,7 +574,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 下方按推理后端和微调框架列出对应的 cookbook 与子 Skill。量化模型不单独作为后端列出，而是在可加载对应格式的推理后端中说明。 -**推理部署**（8 个后端） +**推理部署**（9 个后端） | 后端 | 适用模型 / 场景 | Cookbook | 对应 Agent Skill | | --- | --- | --- | --- | @@ -585,7 +585,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | [Ollama](https://github.com/ollama/ollama) | GGUF，本地端侧运行 | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF，Mac 桌面应用与 OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit，Apple Silicon 本地推理 | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | -| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` 端侧运行时：Android / iOS / 桌面，CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` 端侧运行时：Android / iOS / 桌面 / IoT，CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **微调**（5 个框架） diff --git a/README.md b/README.md index f0ee33f..1058d8b 100644 --- a/README.md +++ b/README.md @@ -297,7 +297,7 @@ MiniCPM5-2B uses the **standard `LlamaForCausalLM` architecture**, so mainstream | LM Studio | GGUF Mac desktop app and OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit local inference on Apple Silicon | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | | ArcLight | GGUF local on-device, CPU, Desktop & Server | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | -| LiteRT-LM | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | +| LiteRT-LM | `.litertlm` on-device runtime: Android / iOS / desktop / IoT, CPU + GPU | [litert.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/litert.md) | [minicpm5-deploy-litert](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-litert/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | #### Fine-tuning @@ -582,7 +582,7 @@ The tables below list the cookbook and sub-skill for each inference backend and | [Ollama](https://github.com/ollama/ollama) | GGUF local on-device runtime | [`docs/deployment/ollama.md`](./docs/deployment/ollama.md) | [`minicpm5-deploy-ollama`](./skills/minicpm5-deploy-ollama/SKILL.md) | | [LM Studio](https://lmstudio.ai) | GGUF Mac desktop app and OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit local inference on Apple Silicon | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | -| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` on-device runtime: Android / iOS / desktop, CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | +| [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) | `.litertlm` on-device runtime: Android / iOS / desktop / IoT, CPU + GPU | [`docs/deployment/litert.md`](./docs/deployment/litert.md) | [`minicpm5-deploy-litert`](./skills/minicpm5-deploy-litert/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF local on-device, CPU, Desktop & Server | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | **Fine tuning** (5 frameworks) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index 0adf3da..0cd4234 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -1,6 +1,6 @@ -# Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) +# Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop / IoT) -[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models, built on [LiteRT](https://github.com/google-ai-edge/litert) (formerly TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones and desktops, through a command-line tool, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone or an app**. +[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) is Google's on-device runtime for language models, built on [LiteRT](https://github.com/google-ai-edge/litert) (formerly TensorFlow Lite). A model is packed once into a `.litertlm` bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones, desktops and IoT boards such as the Raspberry Pi, through a command-line tool, an OpenAI-compatible local server, a Python package, and Kotlin / Swift APIs. Use this path when the target is a **phone, an app, or a local server on a laptop**. The MiniCPM5-2B bundles are hosted in [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B); that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B). Every command on this page was run with `litert-lm` 0.17.0 on an Apple M4 Max; the Android check used `litertlm-android` 0.17.0 on a Galaxy S26. @@ -14,7 +14,7 @@ litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_i --prompt "What is the capital of France?" ``` -The file lands in `~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/`; later runs skip the download. GPU, and the thinking switches: +The file lands in `~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/`; later runs skip the download. GPU, the thinking switches, and sampling: ```bash MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_int4.litertlm @@ -22,39 +22,58 @@ MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_in litert-lm run "$MODEL" --backend gpu --prompt "1+1=?" litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?" litert-lm run "$MODEL" --backend gpu --thinking-budget 2048 --prompt "1+1=?" -litert-lm run "$MODEL" --backend gpu --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?" +litert-lm run "$MODEL" --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?" ``` ## Pre-converted bundles | Model | File | Recipe | Size | Tested on | | --- | --- | --- | --- | --- | -| MiniCPM5-2B | [`MiniCPM5-2B_int4.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int4.litertlm) | int4 blockwise-32 linears, int8 embedding | 1.55 GB | CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro | +| MiniCPM5-2B | [`MiniCPM5-2B_int4.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int4.litertlm) | int4 blockwise-32 + OCTAV on linears, int8 embedding | 1.55 GB | CPU + GPU: Mac, Galaxy S26, iPhone 17 Pro | | MiniCPM5-2B | [`MiniCPM5-2B_int8.litertlm`](https://huggingface.co/litert-community/MiniCPM5-2B/blob/main/MiniCPM5-2B_int8.litertlm) | int8 dynamic linears + embedding, fp32 activations declared | 2.60 GB | CPU + GPU: Mac, Galaxy S26 (iOS: see below) | | MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32.litertlm) | int4 block-32 linears, int8 embedding and LM head, fp32 activations | 0.79 GB | CPU (Mac) | -| MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm) | same recipe, graph laid out for the GPU | 0.79 GB | GPU (Mac) | +| MiniCPM5-1B | [`minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm) | same recipe, optimized for GPU execution | 0.79 GB | GPU (Mac) | | MiniCPM5-1B | [`MiniCPM5-1B_dynamic_wi8_afp32.litertlm`](https://huggingface.co/litert-community/MiniCPM5-1B/blob/main/MiniCPM5-1B_dynamic_wi8_afp32.litertlm) | int8 dynamic, fp32 activations | 1.11 GB | CPU (Mac) | -Which 2B file: **int4 is the phone file** (smaller, fastest GPU decode on every device measured) and the right one for direct answers or short reasoning. **int8 is the file when the reasoning has to complete**: on the same questions its thinking chains are 3–4× shorter than int4's and they terminate where int4 runs into the token budget. Both 2B files embed the checkpoint's own `chat_template.jinja`, so `enable_thinking` and the tool-calling format work unchanged, and both declare the `thought` channel (next section). The 2B repo also carries two CPU-only files described in its card. +Which 2B file: **int4 is the phone file** (smaller, fastest GPU decode on both devices measured) and the right one for direct answers or short reasoning. **int8 is the file when the reasoning has to complete**: on the same questions its thinking chains are 3–4× shorter than int4's and close where int4's keep going. Both 2B files embed the checkpoint's own `chat_template.jinja`, so `enable_thinking` and the tool-calling format work unchanged, and both declare the `thought` channel (next section). The 2B repo also carries two CPU-only files described in its card. + +The int8 file, here on the Mac's Metal GPU: ```bash -# int8 litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int8.litertlm --backend gpu \ --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?" +``` + +MiniCPM5-1B: -# MiniCPM5-1B: +```bash litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32.litertlm --prompt "1+1=?" litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm --backend gpu --prompt "1+1=?" ``` ## Thinking -Thinking is the model's default: with no thinking flag it decides for itself and, in practice, reasons before every answer. The bundles declare the reasoning as a `thought` channel, so the CLI prints it between `[thought]` and `[/thought]` and the Kotlin / Swift APIs hand it to you separately (`channels["thought"]`); the answer text stays clean. +Thinking is the default of the 2B files and the two 1B int4 files: with no thinking flag the model decides for itself and, in practice, reasons before every answer (`MiniCPM5-1B_dynamic_wi8_afp32` is packaged with thinking off by default). The bundles declare the reasoning as a `thought` channel, so the CLI prints it between `[thought]` and `[/thought]` and the Kotlin / Swift APIs hand it to you separately (`channels["thought"]`); the answer text stays clean. + +- `--thinking false`: direct answers with no reasoning block; on `1+1=?` (GPU) the turn took 3.0 s against 24.7 s with thinking on. +- `--thinking-budget N`: caps the reasoning at N tokens. This model's chains run long, so start at 2048 and use 4096 for math. +- Sampling: OpenBMB recommends `temperature 1.0`, `top_p 0.95`. The bundles set no sampler and the engine's default is greedy whatever the temperature, so pass all three: `--top-k 40 --top-p 0.95 --temperature 1.0` (with `--top-p --temperature` alone the output stays byte-identical across seeds). The card's correctness numbers are greedy. +- The KV cache takes the bundle's setting: 4096 tokens for the two 2B files and `MiniCPM5-1B_dynamic_wi8_afp32`, 1024 for the two 1B int4 files (`--max-num-tokens` to change it); the prompt format is ChatML, from the template embedded in each bundle. + +## OpenAI-compatible server + +`litert-lm serve` exposes every bundle in the local registry on `/v1/models` and `/v1/chat/completions` ([server guide](https://developers.google.com/edge/litert-lm/cli/openai_server)). Import the file once under a name, start the server, then call it like any OpenAI endpoint: + +```bash +litert-lm import --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm minicpm5-2b +litert-lm serve --host 127.0.0.1 --port 9379 + +curl http://127.0.0.1:9379/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model": "minicpm5-2b", "messages": [{"role": "user", "content": "1+1=?"}], "reasoning_effort": "none"}' +``` -- `--thinking false`: direct answers. Two- to seven-token replies on trivial questions, about 10× faster turns. -- `--thinking-budget N`: caps the reasoning. Keep N at 2048 or more (4096 for math): a chain cut mid-thought yields no final answer at all. -- Sampling: OpenBMB recommends `temperature 1.0`, `top_p 0.95`. The CLI's default top-k is 1, which is greedy whatever the temperature, so pass all three: `--top-k 40 --top-p 0.95 --temperature 1.0` (with `--top-p --temperature` alone the output stays byte-identical across seeds). The card's correctness numbers are greedy. -- The KV cache is 4096 tokens by default (`--max-num-tokens` to change it); the prompt format is the checkpoint's ChatML template. +`"reasoning_effort": "none"` gives a direct answer; any other value (`low` to `xhigh`) turns thinking on, and the reasoning is counted in `usage.completion_tokens_details.reasoning_tokens`. With thinking on, the reasoning needs room (the int4 file filled the 4096-token context on `1+1=?` and returned no content); when the reasoning has to complete, use the int8 file. `max_completion_tokens` caps the reply, `temperature`, `top_p`, `top_k` and `seed` are honored, and `"stream": true` streams. ## Android @@ -67,13 +86,12 @@ Thinking is the model's default: with no thinking flag it decides for itself and implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0") ``` -The GPU backend needs the OpenCL library declared in `AndroidManifest.xml` (Android 12+ hides vendor libraries from apps otherwise; without it the first message fails with `Can not find OpenCL library on this device`): +The GPU backend needs these entries in `AndroidManifest.xml` ([Android guide](https://developers.google.com/edge/litert-lm/android)); apps targeting Android 12 (API 31) or higher cannot load vendor libraries they do not declare, and without them `initialize()` succeeds and the first message fails with `Can not find OpenCL library on this device`: ```xml - + + - - ``` @@ -95,15 +113,15 @@ engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conve engine.close() ``` -Checked on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17.0 and the int4 file: the engine initializes on `Backend.GPU()` (`Backend.CPU()` selects the CPU) with every node of every signature delegated to OpenCL (1873 of 1873 on the 1024-token prefill, 1692 of 1692 on decode; only the externalized embedding lookup runs on the CPU), and the reply arrives with the reasoning in `channels["thought"]`. `ConversationConfig` also takes `thinkingConfig = ThinkingConfig(enableThinking = false)` or `ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048)` and a `samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0)`, as in the Android guide. `conversation.sendMessageAsync(...)` streams. +Tested on a Galaxy S26 (Snapdragon SM8850, Adreno) with `litertlm-android` 0.17.0 and the int4 file on `Backend.GPU()` (`Backend.CPU()` selects the CPU): the engine initializes with every node of every signature delegated to OpenCL, and the reply arrives with the reasoning in `channels["thought"]`. `ConversationConfig` also takes `thinkingConfig = ThinkingConfig(enableThinking = false)` or `ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048)` and a `samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0)`, as in the Android guide. `conversation.sendMessageAsync(...)` streams. ## iOS -The int4 file passes the card's 8-question check on an iPhone 17 Pro on both backends (Metal GPU and CPU, init 5.7 s and 2.2 s). The int8 file's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; the `com.apple.developer.kernel.increased-memory-limit` and `com.apple.developer.kernel.extended-virtual-addressing` entitlements lift that limit (sections larger than this have loaded and run on an iPhone 17 Pro with other models; not re-checked with this file). The Swift API is documented in the [LiteRT-LM Swift guide](https://developers.google.com/edge/litert-lm/swift) (Swift Package Manager, `https://github.com/google-ai-edge/LiteRT-LM`); this page does not cover the Xcode steps. +Tested on an iPhone 17 Pro with the int4 file on both backends (Metal GPU and CPU; init 5.7 s and 2.2 s). The int8 file's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; the `com.apple.developer.kernel.increased-memory-limit` entitlement lifts that limit (single sections of 2.87 GB and 4.24 GB from other models have loaded and run on an iPhone 17 Pro with it), not checked with this file. The int4 file's main section is 1.28 GB. The Swift API is documented in the [LiteRT-LM Swift guide](https://developers.google.com/edge/litert-lm/swift) (Swift Package Manager, `https://github.com/google-ai-edge/LiteRT-LM`); this page does not cover the Xcode steps. -## Measured (from the litert-community card) +## Speed and accuracy -Apple M4 Max, `litert-lm benchmark` 0.17.0, 256-token prefill / 256-token decode, 3 runs, no compiled-kernel cache: +All numbers are from the [litert-community card](https://huggingface.co/litert-community/MiniCPM5-2B). Apple M4 Max, `litert-lm benchmark` 0.17.0, 256-token prefill / 256-token decode, 3 runs, 1024-token KV cache, no compiled-model cache (`--cache no`): | File | Backend | Prefill | Decode | Time to first token | Init | | --- | --- | --- | --- | --- | --- | @@ -121,21 +139,20 @@ Galaxy S26 (Snapdragon SM8850, Adreno), the LiteRT-LM v0.16.0 release binary, 20 | int8 (fp32 activations) | GPU (OpenCL) | 150–160 tok/s | 10.9–12.8 tok/s | 1.4 s | 1.10 GB | | int8 (fp32 activations) | CPU | 103–157 tok/s | 11.7 tok/s | 1.4–2.1 s | 2.90 GB | -Accuracy, GSM8K first 100 test questions, greedy, thinking off (the protocol of OpenBMB's MiniCPM5 cards), 2048 new tokens: bf16 PyTorch 92 %, int8 CPU 91 %, int4 CPU 86 %, int4 GPU 87 %. Both 2B files score 8/8 on the card's 8-question sanity gate on CPU and GPU. +Accuracy, GSM8K first 100 test questions, greedy, thinking off, up to 2048 new tokens: bf16 PyTorch 92 %, int8 CPU 91 %, int4 CPU 86 %, int4 GPU 87 %. Both 2B files score 8/8 on the card's 8-question sanity gate on the Mac, CPU and GPU. ## Common pitfalls -- **int4 with thinking on may never close its reasoning.** Quantization costs this 42-layer model its thinking discipline first: on the card's 10-question thinking-on subset the int4 file closes 0/10 on the CPU (it keeps re-checking until the budget), while int8 closes 9/10 like the bf16 model. The same shows on the CLI: `1+1=?` on the CPU and `Explain GQA in one sentence.` on the GPU ran into the 4096-token budget with no answer. When the answer matters, use `--thinking false` on int4, or the int8 file. -- **int8 declares fp32 activations in the bundle.** With the GPU's default fp16 activations the int8 model's reasoning on one gate question ran 2000+ tokens without closing; with fp32 it closes in ~450 tokens. The cost is about 14 % of GPU decode speed, and on Adreno the int8 GPU decode ends up level with the same phone's CPU. -- **int8 on iOS needs two entitlements.** With the default entitlements the 2.33 GB weight section does not map; the increased-memory-limit and extended-virtual-addressing entitlements lift that (see iOS). int4 needs neither. -- **Budget cuts leave no answer**: a `--thinking-budget` below ~2048 (4096 for math) truncates the chain and the model emits nothing after it. -- **First run per backend is slow**: the CPU run writes a 1.3 GB XNNPACK cache beside the file and the GPU run compiles its kernels (`--cache no` skips the disk cache; the Kotlin `cacheDir` is the same mechanism). +- **Thinking on the int4 file** runs long chains: on the card's ten-question thinking-on subset (3584-token budget) it closes 0/10 on the CPU, where int8 closes 9/10 like the bf16 model. Pass `--thinking false` for direct answers; when the reasoning has to complete, use the int8 file. +- **int8 on iOS needs the increased-memory-limit entitlement.** With the default entitlements the 2.33 GB weight section does not map; add `com.apple.developer.kernel.increased-memory-limit` (see iOS). int4's 1.28 GB main section is under the limit. +- **The output cap is what leaves no answer**: when the reasoning runs into the context (`--max-num-tokens`, 4096 for the 2B files) or the app's `maxOutputToken`, the chain is cut and nothing follows. Keep those at 2048 or more (4096 for math). +- **First run per backend is slow**: the CPU run writes an XNNPACK weight cache beside the file and the GPU run compiles its kernels (`--cache no` skips the disk cache; the Kotlin `cacheDir` is the same mechanism). - **GPU in your own Android app**: without the `` entries above, `initialize()` succeeds and the first message fails with `Can not find OpenCL library on this device`. -- **Temperature without top-k does nothing**: `--temperature` and `--top-p` only take effect together with `--top-k` above 1 (default 1 = greedy). The Kotlin `SamplerConfig` takes the same three fields. +- **Temperature without top-k does nothing**: `--temperature` and `--top-p` only take effect together with `--top-k` above 1 (the default is greedy). The Kotlin `SamplerConfig` takes the same three fields. - **Scripts**: the CLI reads stdin; when it runs from a script or a job with an open pipe on stdin, redirect it (`< /dev/null`) or it waits for end-of-input before generating. ## See also - [litert-community/MiniCPM5-2B](https://huggingface.co/litert-community/MiniCPM5-2B) — recipe, correctness, all measurements, `litertlm_manifest.json` - [litert-community/MiniCPM5-1B](https://huggingface.co/litert-community/MiniCPM5-1B) -- LiteRT-LM guides: [CLI](https://developers.google.com/edge/litert-lm/cli), [Android](https://developers.google.com/edge/litert-lm/android), [Swift](https://developers.google.com/edge/litert-lm/swift), [Python](https://developers.google.com/edge/litert-lm/python) +- LiteRT-LM guides: [CLI](https://developers.google.com/edge/litert-lm/cli), [OpenAI-compatible server](https://developers.google.com/edge/litert-lm/cli/openai_server), [CLI configuration](https://developers.google.com/edge/litert-lm/cli/configuration), [model management](https://developers.google.com/edge/litert-lm/cli/model_management), [Android](https://developers.google.com/edge/litert-lm/android), [Swift](https://developers.google.com/edge/litert-lm/swift), [Python](https://developers.google.com/edge/litert-lm/python) diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index f1cd8a9..a56e8d6 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -1,11 +1,11 @@ --- name: minicpm5-deploy-litert -description: Run MiniCPM5-2B or MiniCPM5-1B on-device with Google's LiteRT-LM runtime — the litert-lm CLI on a desktop, the Kotlin API or the AI Edge Gallery app on Android, the same .litertlm bundle on CPU or GPU. Use when the user says "LiteRT", "LiteRT-LM", "litertlm", ".litertlm", "Android", "Edge Gallery", "on-device app", or wants one bundle for phones and desktops. +description: Run MiniCPM5-2B or MiniCPM5-1B on-device with Google's LiteRT-LM runtime — the litert-lm CLI or its OpenAI-compatible server on a desktop, the Kotlin API or the AI Edge Gallery app on Android, the same .litertlm bundle on CPU or GPU. Use when the user says "LiteRT", "LiteRT-LM", "litertlm", ".litertlm", "Android", "Edge Gallery", "on-device app", or wants one bundle for phones and desktops. --- -# Deploy MiniCPM5-2B / MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop) +# Deploy MiniCPM5-2B / MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop / IoT) -Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` bundle runs on CPU and GPU through a CLI, Python, Kotlin and Swift. The bundles are hosted in `litert-community`. Human-readable reference: [`docs/deployment/litert.md`](../../docs/deployment/litert.md). +[LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM), Google's on-device runtime built on [LiteRT](https://github.com/google-ai-edge/litert) (formerly TensorFlow Lite). One `.litertlm` bundle, CPU or GPU, through a CLI, an OpenAI-compatible server, Python, Kotlin and Swift. The bundles are hosted in `litert-community`. Human-readable reference: [`docs/deployment/litert.md`](../../docs/deployment/litert.md). ## Required input @@ -14,7 +14,7 @@ Google's on-device runtime (LiteRT, formerly TensorFlow Lite). One `.litertlm` b | `LITERT_REPO` | `litert-community/MiniCPM5-2B` or `litert-community/MiniCPM5-1B` | `litert-community/MiniCPM5-2B` | | `FILE` | `MiniCPM5-2B_int4.litertlm` (1.55 GB, the phone file) or `MiniCPM5-2B_int8.litertlm` (2.60 GB, reasoning that has to finish); 1B: `minicpm_wi4b32_wi8_afp32.litertlm` (CPU) or `minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm` (GPU) | `MiniCPM5-2B_int4.litertlm` | | `BACKEND` | `cpu` or `gpu` | `cpu` | -| `THINKING` | `true` (reason first) or `false` (direct answer) | the model decides; in practice it reasons | +| `THINKING` | `true` (reason first) or `false` (direct answer) | `false` | | `PROMPT` | `1+1=?` | `1+1=?` | ## Steps @@ -29,10 +29,10 @@ uv tool install litert-lm ```bash litert-lm run --from-huggingface-repo="${LITERT_REPO}" "${FILE}" \ - --backend "${BACKEND}" --thinking "${THINKING}" --prompt "${PROMPT}" + --backend "${BACKEND}" --thinking "${THINKING:-false}" --prompt "${PROMPT}" ``` -The first run downloads the file into `~/.litert-lm/cache/huggingface//`; later runs skip the download. With thinking on, the reasoning prints between `[thought]` and `[/thought]` and the answer follows; `--thinking-budget 2048` caps it (never lower: a chain cut mid-thought yields no answer). Sampling as OpenBMB recommends: `--top-k 40 --top-p 0.95 --temperature 1.0` (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing). +The first run downloads the file into `~/.litert-lm/cache/huggingface//`; later runs skip the download. With thinking on, the reasoning prints between `[thought]` and `[/thought]` and the answer follows; `--thinking-budget 2048` caps the reasoning (4096 for math); leave `--thinking` out and the model decides for itself, and in practice it reasons before every answer. Sampling as OpenBMB recommends: `--top-k 40 --top-p 0.95 --temperature 1.0` (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing). ### 2B. Android @@ -50,25 +50,30 @@ engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conve `ConversationConfig(thinkingConfig = ThinkingConfig(enableThinking = false))` turns thinking off. Full snippet, manifest and the Galaxy S26 check: the cookbook. +### 2C. OpenAI-compatible server (desktop) + +```bash +litert-lm import --from-huggingface-repo="${LITERT_REPO}" "${FILE}" minicpm5-2b +litert-lm serve --host 127.0.0.1 --port 9379 +curl http://127.0.0.1:9379/v1/chat/completions -H "Content-Type: application/json" \ + -d '{"model": "minicpm5-2b", "messages": [{"role": "user", "content": "1+1=?"}], "reasoning_effort": "none"}' +``` + +`reasoning_effort` `none` = direct answer, any other value = thinking on; the cap field is `max_completion_tokens`; `top_k` is accepted alongside `temperature` / `top_p` / `seed`; `"stream": true` streams ([server guide](https://developers.google.com/edge/litert-lm/cli/openai_server)). + ### 3. Validate -The reply contains `2` for `1+1=?`. With thinking on it comes after `[/thought]`; with `--thinking false` it is the whole reply. On the int4 file prefer `--thinking false` or `--backend gpu` for this check: on the CPU with thinking on, `1+1=?` can run to the token budget without an answer (pitfalls). +The reply contains `2` for `1+1=?`: with `--thinking false` it is the whole reply; with thinking on it follows `[/thought]`. Through the server (2C), the same check is HTTP 200 with `choices[0].message.content` containing `2`, and `GET /v1/models` lists `minicpm5-2b`. ## Common pitfalls -- **int4 + thinking on may not close its reasoning** (CPU especially): the card's 10-question thinking-on subset closes 0/10 on int4 CPU, 9/10 on int8. Use `--thinking false` on int4, or the int8 file, when the answer matters. -- **int8 declares fp32 activations** (a correctness fix for the GPU's fp16 default); GPU decode is ~14 % slower for it, and on Adreno int8 GPU decode is level with the CPU. int4 is the phone file. -- **int8 on iOS needs the increased-memory-limit and extended-virtual-addressing entitlements** (2.33 GB weight section; the default entitlements do not map it). int4 needs neither (checked on iPhone 17 Pro, GPU and CPU). -- **First run per backend is slow**: a 1.3 GB XNNPACK cache is written beside the file on the CPU, kernels are compiled on the GPU (`--cache no` to skip the disk cache). +- **Thinking on the int4 file** runs long chains: pass `--thinking false` for direct answers, or use the int8 file when the reasoning has to complete. +- **int8 on iOS needs the increased-memory-limit entitlement** (2.33 GB weight section; the default entitlements do not map it). int4's 1.28 GB main section is under the limit (checked on iPhone 17 Pro, GPU and CPU). +- **First run per backend is slow**: an XNNPACK weight cache is written beside the file on the CPU, kernels are compiled on the GPU (`--cache no` to skip the disk cache). - **Android GPU without the manifest entries**: `initialize()` succeeds, the first message fails with `Can not find OpenCL library on this device`. -- **Temperature without top-k does nothing**: pass `--top-k` above 1 together with `--top-p` / `--temperature` (default top-k 1 = greedy). +- **Temperature without top-k does nothing**: pass `--top-k` above 1 together with `--top-p` / `--temperature` (the default is greedy). - **Scripts**: the CLI reads stdin; redirect it (`< /dev/null`) when running from a job with an open pipe. -## When NOT to use - -- GGUF files already in hand → `minicpm5-deploy-llama-cpp` or `minicpm5-deploy-ollama` -- Server with an OpenAI-compatible endpoint → `minicpm5-deploy-vllm` or `minicpm5-deploy-sglang` - ## Reference [`docs/deployment/litert.md`](../../docs/deployment/litert.md) diff --git a/skills/minicpm5-deploy/SKILL.md b/skills/minicpm5-deploy/SKILL.md index 371a962..a4a8412 100644 --- a/skills/minicpm5-deploy/SKILL.md +++ b/skills/minicpm5-deploy/SKILL.md @@ -24,7 +24,7 @@ Before picking a backend, you MUST know: | **HF fp16 (recommended)** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) or [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) | `transformers` / `vllm` (no `--quantization`) / `vllm-ascend` / `sglang` / any `minicpm5-finetune-*` | | GGUF F16 / Q8_0 / Q4_K_M | [`openbmb/MiniCPM5-2B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF) or [`openbmb/MiniCPM5-1B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) | `minicpm5-deploy-llama-cpp` / `-ollama` / `-lmstudio` | | MLX (Apple Silicon) | [`openbmb/MiniCPM5-2B-MLX`](https://huggingface.co/openbmb/MiniCPM5-2B-MLX) or [`openbmb/MiniCPM5-1B-MLX`](https://huggingface.co/openbmb/MiniCPM5-1B-MLX) | `minicpm5-deploy-mlx` | -| LiteRT-LM `.litertlm` (Android / iOS / desktop, CPU + GPU) | [`litert-community/MiniCPM5-2B`](https://huggingface.co/litert-community/MiniCPM5-2B) or [`litert-community/MiniCPM5-1B`](https://huggingface.co/litert-community/MiniCPM5-1B) | `minicpm5-deploy-litert` | +| LiteRT-LM `.litertlm` (Android / iOS / desktop / IoT, CPU + GPU) | [`litert-community/MiniCPM5-2B`](https://huggingface.co/litert-community/MiniCPM5-2B) or [`litert-community/MiniCPM5-1B`](https://huggingface.co/litert-community/MiniCPM5-1B) | `minicpm5-deploy-litert` | If the user has a local copy, accept any directory path that contains `config.json` and `model.safetensors` (or the equivalent GGUF / MLX layout). @@ -40,7 +40,7 @@ If the user has a local copy, accept any directory path that contains `config.js | "Ollama" / "ollama run" / "Modelfile" | macOS / Linux laptop | GGUF | **`minicpm5-deploy-ollama`** | | "LM Studio" / "desktop GUI" | macOS / Windows / Linux | GGUF or MLX | **`minicpm5-deploy-lmstudio`** | | "MLX" / "Apple Silicon native" / "fastest on Mac" | Apple Silicon | MLX | **`minicpm5-deploy-mlx`** | -| "Android" / "on-device app" / "Edge Gallery" / "LiteRT" / "LiteRT-LM" / "litertlm" | Android phone, iPhone, or a desktop app (CPU or GPU) | LiteRT-LM `.litertlm` | **`minicpm5-deploy-litert`** | +| "Android" / "on-device app" / "Edge Gallery" / "LiteRT" / "LiteRT-LM" / "litertlm" / "Raspberry Pi" | Android phone, iPhone, desktop or IoT board (CPU or GPU) | LiteRT-LM `.litertlm` | **`minicpm5-deploy-litert`** | If the user **has not specified** any of the above and asks "how do I run this?": @@ -73,7 +73,7 @@ curl http://localhost:PORT/v1/chat/completions \ Expected: HTTP 200 with `choices[0].message.content` containing `"2"`. -`minicpm5-deploy-litert` runs in-process with no HTTP server; its own step 3 (`litert-lm run … --prompt "1+1=?"`, the answer contains `2`) is the equivalent check. +`minicpm5-deploy-litert` also serves this endpoint: after `litert-lm import … minicpm5-2b`, `litert-lm serve` listens on port 9379 ([guide](https://developers.google.com/edge/litert-lm/cli/openai_server)); use `"model": "minicpm5-2b"`, add `"reasoning_effort": "none"` for a direct answer, and cap with `max_completion_tokens`. ## 5. Known cross-backend pitfalls From d860be83cdfe63ca6b7e13182061210cf0fd5c5f Mon Sep 17 00:00:00 2001 From: Daisuke Majima Date: Thu, 10 Sep 2026 15:42:01 +0900 Subject: [PATCH 7/7] docs: three wording fixes from a final read (sampler default, output cap scope, thinking flag in the skill) --- docs/deployment/litert.md | 4 ++-- skills/minicpm5-deploy-litert/SKILL.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deployment/litert.md b/docs/deployment/litert.md index 0cd4234..0f71953 100644 --- a/docs/deployment/litert.md +++ b/docs/deployment/litert.md @@ -57,7 +57,7 @@ Thinking is the default of the 2B files and the two 1B int4 files: with no think - `--thinking false`: direct answers with no reasoning block; on `1+1=?` (GPU) the turn took 3.0 s against 24.7 s with thinking on. - `--thinking-budget N`: caps the reasoning at N tokens. This model's chains run long, so start at 2048 and use 4096 for math. -- Sampling: OpenBMB recommends `temperature 1.0`, `top_p 0.95`. The bundles set no sampler and the engine's default is greedy whatever the temperature, so pass all three: `--top-k 40 --top-p 0.95 --temperature 1.0` (with `--top-p --temperature` alone the output stays byte-identical across seeds). The card's correctness numbers are greedy. +- Sampling: OpenBMB recommends `temperature 1.0`, `top_p 0.95`. The bundles ship a greedy sampler (temperature 0, no top-k), which stays greedy whatever the temperature, so pass all three: `--top-k 40 --top-p 0.95 --temperature 1.0` (with `--top-p --temperature` alone the output stays byte-identical across seeds). The card's correctness numbers are greedy. - The KV cache takes the bundle's setting: 4096 tokens for the two 2B files and `MiniCPM5-1B_dynamic_wi8_afp32`, 1024 for the two 1B int4 files (`--max-num-tokens` to change it); the prompt format is ChatML, from the template embedded in each bundle. ## OpenAI-compatible server @@ -145,7 +145,7 @@ Accuracy, GSM8K first 100 test questions, greedy, thinking off, up to 2048 new t - **Thinking on the int4 file** runs long chains: on the card's ten-question thinking-on subset (3584-token budget) it closes 0/10 on the CPU, where int8 closes 9/10 like the bf16 model. Pass `--thinking false` for direct answers; when the reasoning has to complete, use the int8 file. - **int8 on iOS needs the increased-memory-limit entitlement.** With the default entitlements the 2.33 GB weight section does not map; add `com.apple.developer.kernel.increased-memory-limit` (see iOS). int4's 1.28 GB main section is under the limit. -- **The output cap is what leaves no answer**: when the reasoning runs into the context (`--max-num-tokens`, 4096 for the 2B files) or the app's `maxOutputToken`, the chain is cut and nothing follows. Keep those at 2048 or more (4096 for math). +- **The output cap is what leaves no answer**: when the reasoning runs into the context (`--max-num-tokens`, 4096 for the 2B files) or the app's `maxOutputToken`, the chain is cut and nothing follows. When the reasoning has to complete, keep those at 2048 or more (4096 for math); 1024, as in the snippets, is enough for short answers. - **First run per backend is slow**: the CPU run writes an XNNPACK weight cache beside the file and the GPU run compiles its kernels (`--cache no` skips the disk cache; the Kotlin `cacheDir` is the same mechanism). - **GPU in your own Android app**: without the `` entries above, `initialize()` succeeds and the first message fails with `Can not find OpenCL library on this device`. - **Temperature without top-k does nothing**: `--temperature` and `--top-p` only take effect together with `--top-k` above 1 (the default is greedy). The Kotlin `SamplerConfig` takes the same three fields. diff --git a/skills/minicpm5-deploy-litert/SKILL.md b/skills/minicpm5-deploy-litert/SKILL.md index a56e8d6..470a1ff 100644 --- a/skills/minicpm5-deploy-litert/SKILL.md +++ b/skills/minicpm5-deploy-litert/SKILL.md @@ -32,7 +32,7 @@ litert-lm run --from-huggingface-repo="${LITERT_REPO}" "${FILE}" \ --backend "${BACKEND}" --thinking "${THINKING:-false}" --prompt "${PROMPT}" ``` -The first run downloads the file into `~/.litert-lm/cache/huggingface//`; later runs skip the download. With thinking on, the reasoning prints between `[thought]` and `[/thought]` and the answer follows; `--thinking-budget 2048` caps the reasoning (4096 for math); leave `--thinking` out and the model decides for itself, and in practice it reasons before every answer. Sampling as OpenBMB recommends: `--top-k 40 --top-p 0.95 --temperature 1.0` (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing). +The first run downloads the file into `~/.litert-lm/cache/huggingface//`; later runs skip the download. With thinking on, the reasoning prints between `[thought]` and `[/thought]` and the answer follows; `--thinking-budget 2048` caps the reasoning (4096 for math); `THINKING=true` makes the model reason first. Sampling as OpenBMB recommends: `--top-k 40 --top-p 0.95 --temperature 1.0` (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing). ### 2B. Android