diff --git a/README-cn.md b/README-cn.md index 7d0273e..973c334 100644 --- a/README-cn.md +++ b/README-cn.md @@ -298,6 +298,7 @@ MiniCPM5-2B 使用**标准** `LlamaForCausalLM` **架构**，主流推理引擎 | Ollama | GGUF，本地端侧运行 | [ollama.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/ollama.md) | [minicpm5-deploy-ollama](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-ollama/SKILL.md) | | LM Studio | GGUF，Mac 桌面应用与 OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit，Apple Silicon 本地推理 | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | +| Core AI | Apple `.aimodel`，Swift 应用内端侧推理：iPhone / iPad / Mac（社区转换） | [coreai.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/coreai.md) | [minicpm5-deploy-coreai](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-coreai/SKILL.md) | | ArcLight | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | @@ -561,7 +562,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | 顶层 Skill | 作用 | 路由到 | | --- | --- | --- | -| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | 推理路由 | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` | +| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | 推理路由 | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` · `coreai` | | **[`minicpm5-finetune`](./skills/minicpm5-finetune/SKILL.md)** | 微调路由 | `trl` · `llamafactory` · `ms-swift` · `unsloth` · `xtuner` | 在 Cursor / Claude Code 中可以这样调用：agent 会读取顶层 Skill，并根据目标后端、硬件和数据路径选择对应的子 Skill 与 cookbook，再执行命令并回报结果。 @@ -573,7 +574,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 下方按推理后端和微调框架列出对应的 cookbook 与子 Skill。量化模型不单独作为后端列出，而是在可加载对应格式的推理后端中说明。 -**推理部署**（7 个后端） +**推理部署**（8 个后端） | 后端 | 适用模型 / 场景 | Cookbook | 对应 Agent Skill | | --- | --- | --- | --- | @@ -585,6 +586,7 @@ MiniCPM5-1B 使用**标准 `LlamaForCausalLM` 架构**，主流推理引擎可 | [LM Studio](https://lmstudio.ai) | GGUF，Mac 桌面应用与 OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit，Apple Silicon 本地推理 | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF 本地端侧 / CPU / 桌面 / 服务器 | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | +| Core AI | Apple `.aimodel`，Swift 应用内端侧推理：iPhone / iPad / Mac（社区转换） | [`docs/deployment/coreai.md`](./docs/deployment/coreai.md) | [`minicpm5-deploy-coreai`](./skills/minicpm5-deploy-coreai/SKILL.md) | **微调**（5 个框架） diff --git a/README.md b/README.md index 2b6e0f2..17aa744 100644 --- a/README.md +++ b/README.md @@ -296,6 +296,7 @@ MiniCPM5-2B uses the **standard `LlamaForCausalLM` architecture**, so mainstream | Ollama | GGUF local on-device runtime | [ollama.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/ollama.md) | [minicpm5-deploy-ollama](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-ollama/SKILL.md) | | LM Studio | GGUF Mac desktop app and OpenAI server | [lmstudio.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/lmstudio.md) | [minicpm5-deploy-lmstudio](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-lmstudio/SKILL.md) | | MLX | MLX / 4bit local inference on Apple Silicon | [mlx.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/mlx.md) | [minicpm5-deploy-mlx](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-mlx/SKILL.md) | +| Core AI | Apple `.aimodel`, on-device in Swift apps: iPhone / iPad / Mac (community conversion) | [coreai.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/coreai.md) | [minicpm5-deploy-coreai](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-coreai/SKILL.md) | | ArcLight | GGUF local on-device, CPU, Desktop & Server | [arclight.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/arclight.md) | [minicpm5-deploy-arclight](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-arclight/SKILL.md) | | vLLM Ascend | BF16 / FP16 OpenAI server | [vllm_ascend.md](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm_ascend.md) | [minicpm5-deploy-vllm-ascend](https://github.com/OpenBMB/MiniCPM/blob/main/skills/minicpm5-deploy-vllm-ascend/SKILL.md) | @@ -559,7 +560,7 @@ The two top-level skills cover deployment and fine-tuning: | Top-level skill | What it does | Routes to | | --- | --- | --- | -| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | Inference router | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` | +| **[`minicpm5-deploy`](./skills/minicpm5-deploy/SKILL.md)** | Inference router | `transformers` · `vllm` · `sglang` · `llama-cpp` · `ollama` · `lmstudio` · `mlx` · `coreai` | | **[`minicpm5-finetune`](./skills/minicpm5-finetune/SKILL.md)** | Fine tuning router | `trl` · `llamafactory` · `ms-swift` · `unsloth` · `xtuner` | In Cursor / Claude Code, you can call them like this: the agent reads the top-level skill, selects the matching sub-skill and cookbook based on the target backend, hardware, and data path, then runs the command and reports back. @@ -571,7 +572,7 @@ In Cursor / Claude Code, you can call them like this: the agent reads the top-le The tables below list the cookbook and sub-skill for each inference backend and fine-tuning framework. Quantized models are not listed as standalone backends; they are described under the inference backend that can load each format. -**Inference Deployment** (7 backends) +**Inference Deployment** (8 backends) | Backend | Model format / use case | Cookbook | Paired Agent Skill | | --- | --- | --- | --- | @@ -582,6 +583,7 @@ The tables below list the cookbook and sub-skill for each inference backend and | [LM Studio](https://lmstudio.ai) | GGUF Mac desktop app and OpenAI server | [`docs/deployment/lmstudio.md`](./docs/deployment/lmstudio.md) | [`minicpm5-deploy-lmstudio`](./skills/minicpm5-deploy-lmstudio/SKILL.md) | | [MLX](https://github.com/ml-explore/mlx-lm) | MLX / 4bit local inference on Apple Silicon | [`docs/deployment/mlx.md`](./docs/deployment/mlx.md) | [`minicpm5-deploy-mlx`](./skills/minicpm5-deploy-mlx/SKILL.md) | | [ArcLight](https://github.com/OpenBMB/ArcLight) | GGUF local on-device, CPU, Desktop & Server | [`docs/deployment/arclight.md`](./docs/deployment/arclight.md) | [`minicpm5-deploy-arclight`](./skills/minicpm5-deploy-arclight/SKILL.md) | +| Core AI | Apple `.aimodel`, on-device in Swift apps: iPhone / iPad / Mac (community conversion) | [`docs/deployment/coreai.md`](./docs/deployment/coreai.md) | [`minicpm5-deploy-coreai`](./skills/minicpm5-deploy-coreai/SKILL.md) | **Fine tuning** (5 frameworks) diff --git a/docs/deployment/coreai.md b/docs/deployment/coreai.md new file mode 100644 index 0000000..d5c6728 --- /dev/null +++ b/docs/deployment/coreai.md @@ -0,0 +1,121 @@ +# Deploy MiniCPM5-2B / MiniCPM5-1B with Apple Core AI (iPhone / iPad / Mac) + +[Core AI](https://developer.apple.com/documentation/coreai) is Apple's on-device ML runtime in iOS 27 / macOS 27. A model is exported once to an `.aimodel` bundle and then runs inside a Swift app on iPhone, iPad and Apple Silicon Mac, with no Python at runtime. The bundle below runs on the GPU through Core AI's pipelined engine. Use this path when the target is an **app** (Swift / Xcode). For Python on a Mac, use [MLX](./mlx.md). + +The bundles below are **community conversions**, maintained in the [Core AI Model Zoo](https://github.com/john-rocky/coreai-model-zoo) (not an OpenBMB or Apple release). Each Hugging Face card carries the conversion recipe and the measurements quoted here. This page covers MiniCPM5-2B and MiniCPM5-1B; the catalog ids are `minicpm5-2b` and `minicpm5-1b`. + +## TL;DR + +```bash +# Requires macOS 27 + Xcode 27 (CoreAIKit 0.4.1 targets Xcode 27 beta 5). +git clone https://github.com/john-rocky/coreai-kit +cd coreai-kit/Examples/ChatDemo + +# Headless run on the Mac (downloads the bundle on first use, then loads from cache): +swift run -c release chat-cli --model minicpm5-2b --prompt "1+1=?" +# → 1 + 1 = 2. +swift run -c release chat-cli --model minicpm5-1b --prompt "1+1=?" # the 1B: same command, 1.1 GB + +# On an iPhone: open ChatDemo.xcodeproj, pick your device as the run destination, +# set your signing team, Run, then choose "MiniCPM5 2B" (or "MiniCPM5 1B") in the model picker. +``` + +## Pre-converted bundle + +| Model | Hugging Face repo | Quantization | Size | Numerics vs HF fp32 | +| --- | --- | --- | --- | --- | +| MiniCPM5-2B | [mlboydaisuke/MiniCPM5-2B-CoreAI](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-CoreAI) | int8 weight-only, per-block-32 | 2.7 GB | 24/24 + 24/24 greedy tokens exact on iPhone, 16/16 on Mac | +| MiniCPM5-1B | [mlboydaisuke/MiniCPM5-1B-CoreAI](https://huggingface.co/mlboydaisuke/MiniCPM5-1B-CoreAI) (revision b8a6ac397ccd5fb815f97336f8a8b1800b110da1 or newer) | int8 weight-only, per-block-32 | 1.1 GB | 24/24 greedy tokens exact + the stop on iPhone, 16/16 on Mac | + +Both are dynamic-shape bundles: one file runs unchanged on macOS and iOS through Core AI's pipelined engine. SDPA, RoPE and RMSNorm stay in full precision; the chat template's end-of-turn token `<|im_end|>` (id 130073) is set as the bundle's `eos_token`, and the model stops there (checked on `1+1=?` with both runtimes below, for both sizes). + +Use the 1B revision named above or newer: the earlier 1B revision (`5ad650f`, per-channel int8) did not stop at the end of a turn, and its card says why. + +## Measured (greedy, 128-token random prompt, Release build) + +| | iPhone 17 Pro decode | iPhone 17 Pro prefill | M4 Max decode | +| --- | --- | --- | --- | +| MiniCPM5-2B int8 | 22.4 tok/s | 27.3 tok/s | 127.6 tok/s | +| MiniCPM5-1B int8 | 61.7 tok/s | 65.6 tok/s | 246.6 tok/s | + +Per-block-32 int8 lands on the Mac GPU's quantized-matmul path; the per-channel int8 sibling of the 2B decoded 5× slower on the Mac (25.6 tok/s) and the same on the phone, which is why both bundles ship per-block-32 scales. + +## Swift API (CoreAIKit) + +[CoreAIKit](https://github.com/john-rocky/coreai-kit) is a community Swift package over Core AI: it downloads a catalog model from Hugging Face on first use, caches it in Application Support, and exposes a chat session. Add the package in Xcode (**File → Add Package Dependencies…**, exact version `0.4.1`, product **CoreAIKit**), or in `Package.swift`: + +```swift +.package(url: "https://github.com/john-rocky/coreai-kit", exact: "0.4.1") +// target dependency: +.product(name: "CoreAIKit", package: "coreai-kit") +``` + +```swift +import CoreAIKit + +let chat = try await ChatSession(catalog: "minicpm5-2b") // or "minicpm5-1b" +let reply = try await chat.respond(to: "1+1=?") +print(reply) // final answer only + +// Streaming: answer and reasoning arrive as separate events. +for try await event in await chat.streamResponse(to: "Explain GQA in one sentence.") { + switch event { + case .response(let delta): print(delta, terminator: "") + case .thinking(let delta): _ = delta // the trace, if you want it + default: break + } +} +``` + +- The session keeps the conversation history; call `respond(to:)` again for the next turn. +- `ChatSession.Configuration`: `temperature` (default 0.7; `nil` = greedy), `maxResponseTokens` (default 2048), `systemPrompt`. Version 0.4.1 exposes temperature only, not top-p. +- Thinking is on by default for MiniCPM5, as in the released chat template. The trace is surfaced as `.thinking` events and on `Message.thinking`; `respond(to:)` returns the answer without it. The trace alone can run several hundred tokens, so keep `maxResponseTokens` generous. +- `KitLanguageModel(model:)` plugs the same bundles into Apple's `FoundationModels` `LanguageModelSession`, with tool calling on ChatML-style models. See the kit README. + +## Apple's own runtime package (no CoreAIKit) + +The bundle also loads with Apple's Swift package from [apple/coreai-models](https://github.com/apple/coreai-models). Download the bundle folder, then: + +```bash +hf download mlboydaisuke/MiniCPM5-2B-CoreAI --include "int8/*" --local-dir ./MiniCPM5-2B-CoreAI +# ./MiniCPM5-2B-CoreAI/int8/ holds metadata.json, the .aimodel, and tokenizer/ +# 1B: hf download mlboydaisuke/MiniCPM5-1B-CoreAI --include "int8/*" --local-dir ./MiniCPM5-1B-CoreAI +``` + +```swift +import FoundationModels +import CoreAILanguageModels + +let model = try await CoreAILanguageModel(resourcesAt: bundleFolderURL) // the int8/ folder +let session = LanguageModelSession(model: model) +print(try await session.respond(to: "1+1=?")) +``` + +The same repo's CLI runs the folder on a Mac: `swift run -c release llm-runner --model ./MiniCPM5-2B-CoreAI/int8 --prompt "1+1=?"`. + +## Building the bundle from your own checkpoint (advanced) + +The export uses Apple's `coreai-torch` (`coreai.llm.export`) with two adjustments that the zoo's recipe applies for you: + +- MiniCPM5 is a plain `LlamaForCausalLM`; the exporter's Mistral graph builder is architecturally identical (GQA, no qkv bias, no qk-norm, explicit `head_dim`), so the recipe maps `llama → mistral`. +- int8 weight-only symmetric quantization via `--compression-config` (per-block-32 scales), and `eos_token` set to `<|im_end|>` in the bundle's tokenizer. + +```bash +git clone https://github.com/john-rocky/coreai-model-zoo && cd coreai-model-zoo +python3 conversion/zoo_convert.py show minicpm5-2b # prints the exact export command (minicpm5-1b for the 1B) +python3 conversion/zoo_convert.py run minicpm5-2b # export_minicpm5.py --hf-id openbmb/MiniCPM5-2B --qconfig minicpm5_int8sym_b32.yaml +python3 cli/coreai_verify.py -n 16 # greedy token check against the fp32 HF reference +python3 cli/coreai_verify.py --chat no-think --prompt "Reply with only the number: 1+1=?" -n 16 --must-stop-within 16 +# ^ the end-of-turn check: the fp32 reference answers `2` and stops; a bundle that runs past that stop fails +``` + +Point `--hf-id` at a local fine-tuned checkpoint to convert your own weights. Keep per-block-32 scales: a per-channel int8 export of the 1B produced a bundle whose LM head scored every vocab id above ~65024 at zero through the engine, `<|im_end|>` included, while a short greedy parity check still passed — the stop check above is what catches it. Full notes: [`models/minicpm5-2b`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/minicpm5-2b/README.md) and [`models/minicpm5-1b`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/minicpm5-1b/README.md) in the zoo. + +## Common pitfalls + +- **Toolchain**: macOS 27 / iOS 27 and Xcode 27 are required. CoreAIKit 0.4.1 is pinned to Xcode 27 beta 5; if `swift` picks another Xcode, export `DEVELOPER_DIR=/Applications/Xcode-27.0.0-Beta.5.app/Contents/Developer` first. +- **First launch on iPhone**: the bundle specializes on the phone once (2B: 28.9 s on iPhone 17 Pro; 1B: 7.3 s), then the cache persists. The 2B needs about 3 GB of free storage and the `com.apple.developer.kernel.increased-memory-limit` entitlement (ChatDemo has it). A full phone fails with `No space left on device`. +- **Context on iPhone**: prompt + generated tokens must stay under 1024 on iOS (the shipped pipelined engine caps growing-KV capacity there). Trim or chunk the history on the phone; macOS has no cap. +- **Debug builds** are about 3× slower per token on host-side work. Measure in Release. +- **macOS App Sandbox**: enable **Outgoing Connections (Client)** for the first-use download. +- **Model runs past the end of turn**: not expected with these bundles (the 1B from revision b8a6ac397ccd5fb815f97336f8a8b1800b110da1 on), since `eos_token` is `<|im_end|>` and the stop was checked end to end. If you re-export yourself, keep that setting, keep per-block-32 scales, and run the stop check above (`Reply with only the number: 1+1=?` should end at `2`). diff --git a/skills/minicpm5-deploy-coreai/SKILL.md b/skills/minicpm5-deploy-coreai/SKILL.md new file mode 100644 index 0000000..38d6a71 --- /dev/null +++ b/skills/minicpm5-deploy-coreai/SKILL.md @@ -0,0 +1,105 @@ +--- +name: minicpm5-deploy-coreai +description: Run MiniCPM5-2B or MiniCPM5-1B on-device inside a Swift app with Apple Core AI (iOS 27 / macOS 27). Use when the user targets an iPhone, iPad or Mac app, or says "iOS", "iPhone", "Swift", "Xcode", "Core AI", "CoreAIKit", ".aimodel", "on-device in my app", or wants a Foundation Models LanguageModelSession backed by MiniCPM5. +--- + +# Deploy MiniCPM5-2B / MiniCPM5-1B with Apple Core AI (iPhone / iPad / Mac) + +Apple's on-device runtime in iOS 27 / macOS 27. The model is an `.aimodel` bundle loaded from Swift; no Python at runtime. The bundles are community conversions from the [Core AI Model Zoo](https://github.com/john-rocky/coreai-model-zoo) (not an OpenBMB or Apple release). Covers MiniCPM5-2B and MiniCPM5-1B. Human-readable reference: [`docs/deployment/coreai.md`](../../docs/deployment/coreai.md). + +## Required input + +| Var | Example | Default | +| --- | --- | --- | +| `MODEL` | `minicpm5-2b` or `minicpm5-1b` (CoreAIKit catalog id) | `minicpm5-2b` | +| `TARGET` | `mac` (headless CLI) or `iphone` (Xcode app) | `mac` | +| `PROMPT` | `1+1=?` | `1+1=?` | + +Bundles: [`mlboydaisuke/MiniCPM5-2B-CoreAI`](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-CoreAI) (int8, 2.7 GB) and [`mlboydaisuke/MiniCPM5-1B-CoreAI`](https://huggingface.co/mlboydaisuke/MiniCPM5-1B-CoreAI) (int8, 1.1 GB; revision b8a6ac397ccd5fb815f97336f8a8b1800b110da1 or newer). Download on first use. + +## Steps + +### 0. Check the toolchain (once) + +macOS 27 and Xcode 27 are required; CoreAIKit 0.4.1 is pinned to Xcode 27 beta 5. iPhone runs need iOS 27 on the device. + +```bash +xcodebuild -version # expect Xcode 27 +# If another Xcode is selected: +export DEVELOPER_DIR=/Applications/Xcode-27.0.0-Beta.5.app/Contents/Developer +``` + +### 1. Get the runner + +```bash +git clone https://github.com/john-rocky/coreai-kit +cd coreai-kit/Examples/ChatDemo +``` + +### 2A. `TARGET=mac`: headless run + +```bash +swift run -c release chat-cli --model "${MODEL}" --prompt "${PROMPT}" +``` + +Progress (download, load) goes to stderr; stdout carries only the final answer, so assert on stdout. First run downloads the bundle into Application Support; later runs load from cache. + +### 2B. `TARGET=iphone`: run the app on the device + +```bash +open ChatDemo.xcodeproj +``` + +In Xcode: choose the iPhone as the run destination, set your signing team under **Signing & Capabilities**, Run, then pick "MiniCPM5 2B" or "MiniCPM5 1B" in the model picker. The app already carries the `com.apple.developer.kernel.increased-memory-limit` entitlement the bundle needs. + +### 2C. Integrate into your own app + +Add the package (Xcode: **File → Add Package Dependencies…**, `https://github.com/john-rocky/coreai-kit`, exact `0.4.1`, product **CoreAIKit**), then: + +```swift +import CoreAIKit + +let chat = try await ChatSession(catalog: "minicpm5-2b") // or "minicpm5-1b" +let reply = try await chat.respond(to: "1+1=?") // answer only; thinking is separate +``` + +`chat.streamResponse(to:)` yields `.response` and `.thinking` deltas. `ChatSession.Configuration` has `temperature` (default 0.7; `nil` = greedy), `maxResponseTokens` (default 2048) and `systemPrompt`; 0.4.1 exposes no top-p. + +### 3. Validate + +stdout (or the app's reply) contains `2` for `1+1=?`, and generation stops on its own. Measured on an M4 Max with the bundle already cached: the 2B answers `1 + 1 = 2.` in 18 s wall time including engine load; Apple's `llm-runner` on the same bundles thinks, answers `2`, and stops at `<|im_end|>` (2B after 190 tokens, 1B after 171). + +## Without CoreAIKit (Apple's own package) + +```bash +hf download mlboydaisuke/MiniCPM5-2B-CoreAI --include "int8/*" --local-dir ./MiniCPM5-2B-CoreAI +# 1B: hf download mlboydaisuke/MiniCPM5-1B-CoreAI --include "int8/*" --local-dir ./MiniCPM5-1B-CoreAI +``` + +```swift +import FoundationModels +import CoreAILanguageModels // Swift package: https://github.com/apple/coreai-models + +let model = try await CoreAILanguageModel(resourcesAt: folderURL) // the int8/ folder +let session = LanguageModelSession(model: model) +print(try await session.respond(to: "1+1=?")) +``` + +Mac CLI from the same repo: `swift run -c release llm-runner --model ./MiniCPM5-2B-CoreAI/int8 --prompt "1+1=?"`. + +## Common pitfalls + +- **Wrong Xcode selected**: `swift` builds against whichever Xcode `xcode-select` points at. Set `DEVELOPER_DIR` as in step 0. +- **First launch on iPhone**: one-time on-device specialization (2B: 28.9 s on iPhone 17 Pro; 1B: 7.3 s), then cached. The 2B needs about 3 GB free storage; a full phone fails with `No space left on device`. +- **1B revision**: use `b8a6ac397ccd5fb815f97336f8a8b1800b110da1` or newer. The earlier 1B revision (`5ad650f`) did not stop at the end of a turn. +- **Context on iPhone**: prompt + generated tokens must stay under 1024 on iOS. Trim or chunk history on the phone; macOS has no cap. +- **Thinking is on by default** (as in the released chat template) and can run several hundred tokens before the answer. Keep `maxResponseTokens` generous; read the trace from `.thinking` events if needed. +- **Debug builds** are about 3× slower per token. Measure in Release (`-c release`). +- **macOS App Sandbox**: enable **Outgoing Connections (Client)** for the first-use download. + +## When NOT to use + +- Python on a Mac, no app → `minicpm5-deploy-mlx` (fastest on Apple Silicon in Python) +- Desktop GUI, no code → `minicpm5-deploy-lmstudio` +- Linux / Windows / CPU-only → `minicpm5-deploy-llama-cpp` +- Server with an OpenAI-compatible endpoint → `minicpm5-deploy-vllm` or `minicpm5-deploy-sglang` diff --git a/skills/minicpm5-deploy/SKILL.md b/skills/minicpm5-deploy/SKILL.md index 301369e..f7f6306 100644 --- a/skills/minicpm5-deploy/SKILL.md +++ b/skills/minicpm5-deploy/SKILL.md @@ -24,6 +24,7 @@ Before picking a backend, you MUST know: | **HF fp16 (recommended)** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) or [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) | `transformers` / `vllm` (no `--quantization`) / `vllm-ascend` / `sglang` / any `minicpm5-finetune-*` | | GGUF F16 / Q8_0 / Q4_K_M | [`openbmb/MiniCPM5-2B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF) or [`openbmb/MiniCPM5-1B-GGUF`](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) | `minicpm5-deploy-llama-cpp` / `-ollama` / `-lmstudio` | | MLX (Apple Silicon) | [`openbmb/MiniCPM5-2B-MLX`](https://huggingface.co/openbmb/MiniCPM5-2B-MLX) or [`openbmb/MiniCPM5-1B-MLX`](https://huggingface.co/openbmb/MiniCPM5-1B-MLX) | `minicpm5-deploy-mlx` | +| Core AI `.aimodel` (iPhone / iPad / Mac apps, Swift; community conversion, MiniCPM5-2B and MiniCPM5-1B) | [`mlboydaisuke/MiniCPM5-2B-CoreAI`](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-CoreAI) | `minicpm5-deploy-coreai` | If the user has a local copy, accept any directory path that contains `config.json` and `model.safetensors` (or the equivalent GGUF / MLX layout). @@ -39,6 +40,7 @@ If the user has a local copy, accept any directory path that contains `config.js | "Ollama" / "ollama run" / "Modelfile" | macOS / Linux laptop | GGUF | **`minicpm5-deploy-ollama`** | | "LM Studio" / "desktop GUI" | macOS / Windows / Linux | GGUF or MLX | **`minicpm5-deploy-lmstudio`** | | "MLX" / "Apple Silicon native" / "fastest on Mac" | Apple Silicon | MLX | **`minicpm5-deploy-mlx`** | +| "iPhone" / "iOS app" / "Swift" / "Xcode" / "Core AI" | iPhone / iPad / Apple Silicon Mac | Core AI `.aimodel` (MiniCPM5-2B / 1B) | **`minicpm5-deploy-coreai`** | If the user **has not specified** any of the above and asks "how do I run this?": @@ -46,6 +48,7 @@ If the user **has not specified** any of the above and asks "how do I run this?" - **Ascend NPU, want an OpenAI-compatible server**: pick `minicpm5-deploy-vllm-ascend`. - **CUDA box, want minimal Python**: pick `minicpm5-deploy-transformers`. - **Apple Silicon laptop**: pick `minicpm5-deploy-ollama` (easiest) or `minicpm5-deploy-mlx` (fastest). +- **iPhone / iPad, or a Swift / Xcode app on any Apple device**: pick `minicpm5-deploy-coreai` (MiniCPM5-2B or MiniCPM5-1B). - **CPU only / Windows / low-VRAM**: pick `minicpm5-deploy-llama-cpp` (Q4_K_M). ## 3. Invocation contract @@ -70,6 +73,8 @@ curl http://localhost:PORT/v1/chat/completions \ Expected: HTTP 200 with `choices[0].message.content` containing `"2"`. +`minicpm5-deploy-coreai` runs in-process with no HTTP server; its own step 3 (`chat-cli --prompt "1+1=?"`, stdout contains `2`) is the equivalent check. + ## 5. Known cross-backend pitfalls These are common to multiple backends — surface to the user up front: