You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Robot.cpp is a lightweight on-device robot model inference framework built on top of llama.cpp. It inherits llama.cpp's zero-dependency and lightweight philosophy: robot model inference can run without complex Python dependency setup or PyTorch environment configuration. This makes Robot.cpp especially useful for cross-platform deployment and edge devices where environment setup is often painful.
18
10
19
-
```sh
11
+
The core concept in Robot.cpp is [`model-server`](robot_server/README.md), which provides the main unified model interface. In practice, you start `model-server`; it listens for robot inference requests, receives observations from the robot, runs the underlying model forward pass, and returns the generated action. To keep the system lightweight and dependency-free, the communication layer uses a custom TCP protocol.
12
+
13
+
For robot-side usage, this repository provides examples for both simulation and real hardware: LIBERO as a simulation template, and the low-cost SO-101 as a real-robot template. The project is organized around the following concepts:
14
+
15
+
*`model-client`: a client for communicating with `model-server`. It wraps the communication protocol and sends requests to `model-server`. We provide both C++ and Python clients.
16
+
*`policy`: an abstraction layer that uses `model-client` and connects it to a specific robot or simulation system. A policy receives observations from the robot platform, processes them, sends them through `model-client`, and returns the final action output. This layer hides the communication details and is easier to use.
17
+
*`platform`: a concrete robot platform responsible for sensor management and robot control.
18
+
19
+
We also provide two tools to support robot model development:
20
+
21
+
*[`hf2gguf`](tools/hf2gguf/README.md): converts safetensors checkpoints into the GGUF files used by this project.
22
+
*[`quant`](tools/quant/README_zh.md): selectively quantizes arbitrary tensor groups in a model. Users only need to adjust a YAML plan to generate quantized GGUF files.
We use SmolVLA GGUF files as the example for starting `model-server`.
43
+
44
+
#### Step 0: Download a GGUF model
45
+
46
+
Download a sample GGUF model from Hugging Face: [huggingface.co/rrobottt/smolvla-so101-fp32](https://huggingface.co/rrobottt/smolvla-so101-fp32)
37
47
38
-
The hook set follows the same shape as `llama.cpp`'s pre-commit setup, but is
39
-
kept local and lightweight: Python fatal checks through `ruff`, and C/C++
40
-
format checks through `clang-format`. GitHub Actions runs the same hooks on all
41
-
tracked files in pushes and pull requests.
48
+
#### Step 1: Start model-server
42
49
43
-
## SmolVLA Server
50
+
There are two ways to start `model-server`.
44
51
45
-
```sh
46
-
./build/bin/model-server \
52
+
##### Option 1: Download a prebuilt binary
53
+
54
+
For several platforms and configurations, we provide prebuilt `model-server` binaries on the release page.
55
+
56
+
After downloading, run `model-server` like this:
57
+
58
+
```bash
59
+
./model-server \
47
60
--model-type smolvla \
48
61
--llm /path/to/smolvla-llm-f32.gguf \
49
62
--mmproj /path/to/mmproj-smolvla-f32.gguf \
@@ -53,68 +66,167 @@ tracked files in pushes and pull requests.
53
66
--port 5555
54
67
```
55
68
56
-
The helper scripts under `robot_server/shell/` wrap the same command for common
57
-
local setups. Use `MODEL_TYPE=smolvla` or `MODEL_TYPE=pi0` with the Linux
58
-
launchers when switching models.
69
+
##### Option 2: Build locally
59
70
60
-
## pi0 Server
71
+
For general local setups, we provide ready-to-use build-and-launch shells for three platforms. You can modify the environment variables inside the scripts, or override them directly with `export`. See [robot_server/README.md](robot_server/README.md) for details.
61
72
62
-
Converted pi0 checkpoints are expected as split GGUF components:
| CUDA | - |`robot_server/shell/launch_robot_server_linux_cuda.sh`|`robot_server/shell/launch_robot_server_windows_cuda.bat`|
76
+
| CPU |`robot_server/shell/launch_robot_server_mac_cpu.sh`|`robot_server/shell/launch_robot_server_linux_cpu.sh`|`robot_server/shell/launch_robot_server_windows_cpu.bat`|
77
+
| Metal |`robot_server/shell/launch_robot_server_mac_metal.sh`| - | - |
#### Step 2: Send one dummy request to model-server
86
+
87
+
After the server starts, it listens for requests. We provide a minimal example that sends a random observation request. You can use either Python or C++.
|`BUILD_DIR`|`${ROBOT_CPP_ROOT}/build_robot_client`| C++ client CMake build directory. |
104
+
|`PORT`|`5555`| Server port used by the client. |
105
+
|`BUILD_CLIENT`|`0`| Whether to force rebuild the client. Set to `1` to rebuild even if the binary already exists. |
106
+
|`CMAKE_BIN`|`cmake`| CMake command path, useful for selecting a custom CMake binary. |
107
+
108
+
Then run:
109
+
110
+
```bash
111
+
bash robot_client/shell/cpp_client_example.sh
112
+
```
113
+
114
+
### Using model-server in simulation, using LIBERO as the example
115
+
116
+
See the [LIBERO simulation evaluation guide](eval/libero/README.md).
117
+
118
+
### Using model-server on real hardware, using SO-101 as the example
119
+
120
+
See the [SO-101 deployment guide](eval/lerobot_so101/README.md). A video tutorial is also planned (bilibili link).
121
+
122
+
---
123
+
124
+
## Performance
125
+
126
+
We benchmark Robot.cpp on several platforms. Each measurement uses 5 warmup runs and 100 loop runs. The reported latency is the average time from receiving the image, through preprocessing and forward inference, to producing a usable action chunk, measured in milliseconds. All state projectors remain in f32 precision.
127
+
128
+
For the LIBERO setting, the input contains two 256x256 images and an 8-dimensional state. For the SO-101 real-robot setting, the input contains one 224x224 image and a 6-dimensional state.
129
+
130
+
For SmolVLA preprocessing, we follow the official default setting: images are first resized to 512x512.
131
+
132
+
| Model | Mac M4 Pro (CPU) | Mac M4 Pro (Metal) | RTX 4090 | RTX 3060 | A100 | Jetson AGX Orin |
> `bf16*`: on Mac, f16 results are used in place of bf16 because current Mac bf16 support is not ideal.
142
+
143
+
---
144
+
145
+
## Model Zoo
146
+
147
+
This section lists converted GGUF models that can be used directly with `model-server` for smoke tests and quick starts. For your own real-world scenarios, we recommend using [`hf2gguf`](tools/hf2gguf/README.md) to generate your own GGUF models. Different components can also use different precisions; in practice, the best precision choice is often component-specific. In our examples, the state projector always stays in f32, while the other GGUF files follow the listed precision. You can mix and match them to explore better accuracy/performance tradeoffs.
│ ├── base_platform.py # Shared base class for real-robot platforms
193
+
│ ├── libero/ # LIBERO simulation evaluation
194
+
│ └── lerobot_so101/ # SO-101 real-robot scripts and examples
195
+
└── third_party/
196
+
├── llama.cpp/ # ggml / llama.cpp backend
197
+
└── lerobot/ # LeRobot dependency or reference code
198
+
```
199
+
200
+
---
201
+
202
+
## Extension and Contribution
203
+
204
+
Robot.cpp welcomes community contributions for new model runtimes, platform adapters, evaluation flows, model conversion tools, and performance optimization. We aim to keep the core inference framework lightweight, cross-platform, and easy to reproduce, while allowing different robot models and platforms to connect through a unified interface.
205
+
206
+
If you want to extend this project, start with these documents:
207
+
208
+
*[How to add a new model](src/readme_zh.md)
209
+
* How to add a new platform: [real robot](eval/README.md), [simulation](eval/HOW_TO_ADD_NEW_SIM.md).
210
+
211
+
Issues and PRs are welcome. For larger model-architecture changes, protocol changes, or platform abstraction changes, we recommend opening an issue first to align on the interface boundary.
212
+
213
+
---
214
+
215
+
## License
216
+
217
+
Robot.cpp source code is released under the Apache License, Version 2.0. See [LICENSE](LICENSE) for the full license text.
218
+
219
+
This repository also includes third-party open-source components, each under its own license. See the license files under `third_party/` for details.
220
+
221
+
---
222
+
223
+
## Acknowledgements
110
224
111
-
## Evaluation
225
+
Robot.cpp's design and implementation benefit from several excellent open-source projects:
*[llama.cpp](https://github.com/ggerganov/llama.cpp): provides lightweight local inference, the GGML/GGUF ecosystem, and cross-platform backend foundations. This project continues building robot model inference capabilities on top of its engineering philosophy and low-level runtime.
228
+
*[LeRobot](https://github.com/huggingface/lerobot): provides reference implementations for robot data, policy training, and real-robot integration. The SO-101 real-robot example and parts of the evaluation flow in this project are inspired by the LeRobot ecosystem.
229
+
*[LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO): provides robot simulation tasks and evaluation benchmarks. The LIBERO simulation evaluation flow in this project is based on its task environments and benchmark design.
230
+
*[OpenPI](https://github.com/Physical-Intelligence/openpi): provides the pi0 policy model and related open-source implementation. The pi0 runtime, conversion, and evaluation work in this project references OpenPI's model design.
0 commit comments