Skip to content

Commit 46dd77c

Browse files
JJJYmmmJJJYmmm
committed
docs: add StarVLA setup and model zoo
1 parent 3cd5d82 commit 46dd77c

4 files changed

Lines changed: 291 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 85 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,13 @@ We also provide two tools to support robot model development:
5454
git clone https://github.com/Robot-cpp/robot.cpp
5555
cd robot.cpp
5656
git submodule update --init --recursive
57+
./tools/apply_patches.sh
5758
```
5859

60+
The launch scripts below configure and build `model-server` automatically. For
61+
a manual StarVLA build, enable `ROBOT_CPP_BUILD_STARVLA`; see the
62+
[Robot Server build instructions](robot_server/README.md#manual-build).
63+
5964
This section introduces three usage paths to help you quickly understand the repository:
6065

6166
* Starting `model-server` and connecting it to a minimal dummy `model-client`.
@@ -95,6 +100,14 @@ After downloading, run `model-server` like this:
95100

96101
For general local setups, we provide ready-to-use build-and-launch shells for three platforms. You can modify the environment variables inside the scripts, or override them directly with `export`. See [robot_server/README.md](robot_server/README.md) for details.
97102

103+
For example, from the repository root on Linux with CUDA:
104+
105+
```bash
106+
export ROBOT_CPP_ROOT="$PWD"
107+
export GGUF_DIR=/path/to/smolvla-so101-fp32
108+
bash robot_server/shell/launch_robot_server_linux_cuda.sh
109+
```
110+
98111
| Backend | macOS | Linux | Windows |
99112
| ------- | ------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------- |
100113
| CUDA | - | `robot_server/shell/launch_robot_server_linux_cuda.sh` | `robot_server/shell/launch_robot_server_windows_cuda.bat` |
@@ -127,7 +140,7 @@ We provide a build-to-run example in `robot_client/shell/cpp_client_example.sh`.
127140
| `ROBOT_CPP_ROOT` | unset; required | Repository root. |
128141
| `BUILD_DIR` | `${ROBOT_CPP_ROOT}/build_robot_client` | C++ client CMake build directory. |
129142
| `PORT` | `5555` | Server port used by the client. |
130-
| `BUILD_CLIENT` | `0` | Whether to force rebuild the client. Set to`1` to rebuild even if the binary already exists. |
143+
| `BUILD_CLIENT` | `0` | Whether to force rebuild the client. Set to `1` to rebuild even if the binary already exists. |
131144
| `CMAKE_BIN` | `cmake` | CMake command path, useful for selecting a custom CMake binary. |
132145

133146
Then run:
@@ -150,20 +163,33 @@ See the [SO-101 deployment guide](eval/lerobot_so101/README.md).
150163

151164
## ⚡ Performance
152165

153-
We benchmark Robot.cpp on several platforms. Each measurement uses 5 warmup runs and 100 loop runs. The reported latency is the average time from receiving the image, through preprocessing and forward inference, to producing a usable action chunk, measured in milliseconds. All state projectors remain in f32 precision.
166+
We benchmark Robot.cpp on several platforms. Each measurement uses 5 warmup runs and 100 loop runs. The reported latency is the average time from receiving the image, through preprocessing and forward inference, to producing a usable action chunk, measured in milliseconds. State projectors, where present, remain in f32 precision.
154167

155168
For the LIBERO setting, the input contains two 256x256 images and an 8-dimensional state. For the SO-101 real-robot setting, the input contains one 224x224 image and a 6-dimensional state.
156169

157170
For SmolVLA preprocessing, we follow the official default setting: images are first resized to 512x512.
158171

159-
| Model | Mac M4 Pro (CPU) | Mac M4 Pro (Metal) | RTX 4090 | RTX 3060 | A100 | Jetson AGX Orin |
160-
| ---------------------- | ---------------: | -----------------: | -------: | ----------: | ---: | --------------: |
161-
| smolvla@libero (bf16*) | 527 | 216 | 28 | 116 | 43 | 282 |
162-
| smolvla@libero (f32) | 577 | 236 | 32 | 142 | 42 | 299 |
163-
| smolvla@so-101 (bf16*) | 339 | 145 | 23 | 77 | 36 | 184 |
164-
| smolvla@so-101 (f32) | 396 | 158 | 24 | 92 | 34 | 200 |
165-
| pi0@libero (f32) | 1839 | 710 | 83 | OOM/offload | 71 | 956 |
166-
| pi0@libero (bf16*) | 1954 | 635 | 57 | 267 | 66 | 498 |
172+
For StarVLA, the input contains one 224x224 image and no robot state. Qwen and
173+
the multimodal projector use bf16; OFT, GR00T, PI, and PI_v3 policies use f32.
174+
FAST stores its action codec in the policy GGUF.
175+
The StarVLA A100 results use an A100-PCIE-40GB with 8 CPU threads,
176+
`n_ctx=2048`, `n_batch=2048`, and noise seed 0.
177+
178+
| Model | Mac M4 Pro (CPU) | Mac M4 Pro (Metal) | RTX 4090 | RTX 3060 | A100 | Jetson AGX Orin |
179+
| ----------------------------- | ---------------: | -----------------: | -------: | ----------: | ---: | --------------: |
180+
| smolvla@libero (bf16*) | 527 | 216 | 28 | 116 | 43 | 282 |
181+
| smolvla@libero (f32) | 577 | 236 | 32 | 142 | 42 | 299 |
182+
| smolvla@so-101 (bf16*) | 339 | 145 | 23 | 77 | 36 | 184 |
183+
| smolvla@so-101 (f32) | 396 | 158 | 24 | 92 | 34 | 200 |
184+
| pi0@libero (f32) | 1839 | 710 | 83 | OOM/offload | 71 | 956 |
185+
| pi0@libero (bf16*) | 1954 | 635 | 57 | 267 | 66 | 498 |
186+
| starvla/oft@bridge | - | - | - | - | 50 | - |
187+
| starvla/groot@bridge | - | - | - | - | 54 | - |
188+
| starvla/pi_v3@bridge | - | - | - | - | 112 | - |
189+
| starvla/qwen25_oft@bridge | - | - | - | - | 42 | - |
190+
| starvla/qwen25_groot@bridge | - | - | - | - | 51 | - |
191+
| starvla/qwen25_pi@bridge | - | - | - | - | 101 | - |
192+
| starvla/qwen25_fast@bridge | - | - | - | - | 386 | - |
167193

168194
> `bf16*`: on Mac, f16 results are used in place of bf16 because current Mac bf16 support is not ideal.
169195
> `OOM/offload`: pi0@libero (f32) runs out of memory on RTX 3060 and triggers offload, so we do not report a latency number for now.
@@ -230,6 +256,55 @@ This section lists converted GGUF models that can be used directly with `model-s
230256
<td>f32</td>
231257
<td><a href="https://huggingface.co/robotcpp/pi0-libero-f32">pi0-libero-f32</a></td>
232258
</tr>
259+
<tr>
260+
<td>StarVLA Qwen3-VL OFT</td>
261+
<td>Bridge</td>
262+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-OFT-Bridge-RT-1">StarVLA/Qwen3VL-OFT-Bridge-RT-1</a></td>
263+
<td>bf16 + f32 policy</td>
264+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-oft-bridge-bf16">starvla-qwen3-oft-bridge-bf16</a></td>
265+
</tr>
266+
<tr>
267+
<td>StarVLA Qwen3-VL GR00T</td>
268+
<td>Bridge</td>
269+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-GR00T-Bridge-RT-1">StarVLA/Qwen3VL-GR00T-Bridge-RT-1</a></td>
270+
<td>bf16 + f32 policy</td>
271+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-groot-bridge-bf16">starvla-qwen3-groot-bridge-bf16</a></td>
272+
</tr>
273+
<tr>
274+
<td>StarVLA Qwen3-VL PI_v3</td>
275+
<td>Bridge</td>
276+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-PI_v3-Bridge-RT_1">StarVLA/Qwen3VL-PI_v3-Bridge-RT_1</a></td>
277+
<td>bf16 + f32 policy</td>
278+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-pi-v3-bridge-bf16">starvla-qwen3-pi-v3-bridge-bf16</a></td>
279+
</tr>
280+
<tr>
281+
<td>StarVLA Qwen2.5-VL OFT</td>
282+
<td>Bridge</td>
283+
<td><a href="https://huggingface.co/StarVLA/Qwen-OFT-Bridge-RT-1">StarVLA/Qwen-OFT-Bridge-RT-1</a></td>
284+
<td>bf16 + f32 policy</td>
285+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-oft-bridge-bf16">starvla-qwen25-oft-bridge-bf16</a></td>
286+
</tr>
287+
<tr>
288+
<td>StarVLA Qwen2.5-VL GR00T</td>
289+
<td>Bridge</td>
290+
<td><a href="https://huggingface.co/StarVLA/Qwen-GR00T-Bridge-RT-1">StarVLA/Qwen-GR00T-Bridge-RT-1</a></td>
291+
<td>bf16 + f32 policy</td>
292+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-groot-bridge-bf16">starvla-qwen25-groot-bridge-bf16</a></td>
293+
</tr>
294+
<tr>
295+
<td>StarVLA Qwen2.5-VL PI</td>
296+
<td>Bridge</td>
297+
<td><a href="https://huggingface.co/StarVLA/Qwen-PI-Bridge-RT-1">StarVLA/Qwen-PI-Bridge-RT-1</a></td>
298+
<td>bf16 + f32 policy</td>
299+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-pi-bridge-bf16">starvla-qwen25-pi-bridge-bf16</a></td>
300+
</tr>
301+
<tr>
302+
<td>StarVLA Qwen2.5-VL FAST</td>
303+
<td>Bridge</td>
304+
<td><a href="https://huggingface.co/StarVLA/Qwen-FAST-Bridge-RT-1">StarVLA/Qwen-FAST-Bridge-RT-1</a></td>
305+
<td>bf16 + codec</td>
306+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-fast-bridge-bf16">starvla-qwen25-fast-bridge-bf16</a></td>
307+
</tr>
233308
</tbody>
234309
</table>
235310

‎README_ZH.md‎

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,13 @@ Robot.cpp是一个轻量化的on-device机器人模型推理框架，在llama.cp
5454
git clone https://github.com/Robot-cpp/robot.cpp
5555
cd robot.cpp
5656
git submodule update --init --recursive
57+
./tools/apply_patches.sh
5758
```
5859

60+
下文的启动脚本会自动配置并编译 `model-server`。手动构建 StarVLA 时需要开启
61+
`ROBOT_CPP_BUILD_STARVLA`，详见
62+
[Robot Server 构建说明](robot_server/README_ZH.md#手动构建)。
63+
5964
我们介绍三类使用案例来帮助你快速了解本仓库：
6065

6166
* model-server的启动，其与最小dummy model-client通信的案例。
@@ -95,6 +100,14 @@ git submodule update --init --recursive
95100

96101
对于更加一般的情况，我们也提供了三个平台的开箱即用编译+启动的shell，可以通过修改shell里的环境变量，或者直接export的形式来快速在本机实现启动。详情参见 [robot_server/README_ZH.md](robot_server/README_ZH.md)
97102

103+
例如，在 Linux CUDA 环境中从仓库根目录运行：
104+
105+
```bash
106+
export ROBOT_CPP_ROOT="$PWD"
107+
export GGUF_DIR=/path/to/smolvla-so101-fp32
108+
bash robot_server/shell/launch_robot_server_linux_cuda.sh
109+
```
110+
98111
| Backend | macOS | Linux | Windows |
99112
| ------- | ------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------- |
100113
| CUDA | - | `robot_server/shell/launch_robot_server_linux_cuda.sh` | `robot_server/shell/launch_robot_server_windows_cuda.bat` |
@@ -127,7 +140,7 @@ python robot_client/examples/python/minimal_example.py
127140
| `ROBOT_CPP_ROOT` | 无，必须设置 | 仓库根目录。 |
128141
| `BUILD_DIR` | `${ROBOT_CPP_ROOT}/build_robot_client` | C++ client 的 CMake build 目录 |
129142
| `PORT` | `5555` | client 连接的 server port |
130-
| `BUILD_CLIENT` | `0` | 是否强制重新build client。设为`1` 时即使 binary 已存在也会重新 build |
143+
| `BUILD_CLIENT` | `0` | 是否强制重新build client。设为 `1` 时即使 binary 已存在也会重新 build |
131144
| `CMAKE_BIN` | `cmake` | 使用的 CMake 命令路径，可用于指定自定义 CMake |
132145

133146
然后运行下面的bash：
@@ -156,6 +169,11 @@ bash robot_client/shell/cpp_client_example.sh
156169

157170
其中对于smolvla的preprocess设定，参考官方的基本设定，即首先会将图片变成512*512。
158171

172+
StarVLA 使用一张 224x224 图像且不输入 robot state。Qwen 和 multimodal projector
173+
使用 bf16，OFT、GR00T、PI 和 PI_v3 policy 使用 f32；FAST 的 policy GGUF 保存 action
174+
codec。A100 数据在 A100-PCIE-40GB、8 个 CPU 线程、`n_ctx=2048`、`n_batch=2048` 和
175+
noise seed 0 下测得。
176+
159177
| Model | Mac M4 Pro (CPU) | Mac M4 Pro (Metal) | RTX 4090 | RTX 3060 | A100 | Jetson AGX Orin |
160178
| ---------------------- | ---------------: | -----------------: | -------: | ----------: | ---: | --------------: |
161179
| smolvla@libero (bf16*) | 527 | 216 | 28 | 116 | 43 | 282 |
@@ -164,15 +182,24 @@ bash robot_client/shell/cpp_client_example.sh
164182
| smolvla@so-101 (f32) | 396 | 158 | 24 | 92 | 34 | 200 |
165183
| pi0@libero (f32) | 1839 | 710 | 83 | OOM/offload | 71 | 956 |
166184
| pi0@libero (bf16*) | 1954 | 635 | 57 | 267 | 66 | 498 |
185+
| starvla/oft@bridge | - | - | - | - | 50 | - |
186+
| starvla/groot@bridge | - | - | - | - | 54 | - |
187+
| starvla/pi_v3@bridge | - | - | - | - | 112 | - |
188+
| starvla/qwen25_oft@bridge | - | - | - | - | 42 | - |
189+
| starvla/qwen25_groot@bridge | - | - | - | - | 51 | - |
190+
| starvla/qwen25_pi@bridge | - | - | - | - | 101 | - |
191+
| starvla/qwen25_fast@bridge | - | - | - | - | 386 | - |
167192

168193
> `bf16*`：在 Mac上使用 f16 结果替代 bf16，因为当前 Mac对 bf16 的支持不够好。
169194
> `OOM/offload`：pi0@libero (f32) 在 RTX 3060 上会 OOM 并触发 offload，因此暂时不报告 latency 数值。
170195
171196
---
172197

173-
## 🧩 model-zoo
198+
## 🧩 Model Zoo
174199

175-
这里整理一些已经转换好的 GGUF 模型，可以直接配合 `model-server` 做smoke test，以方便quick start！但针对自己的实际场景，我们推荐使用[hf2gguf](tools/hf2gguf/README_ZH.md)来生成自己的GGUF model！并且对于不同的部分，您还可以自定义不同的精度，来实现不同部分的精度组合（事实上，不同部分的最优精度通常是不同的），我们的例子中，state proj始终保持f32精度，其他的gguf随着precision精度变化而变化，您可以自行组合，探索更好更高效的性能tradeoff！
200+
下表列出可直接配合 `model-server` 使用的 GGUF 模型。实际部署时，建议使用
201+
[`hf2gguf`](tools/hf2gguf/README_ZH.md) 转换自己的 checkpoint。各组件可以分别选择
202+
精度；表中示例的 state projector 固定为 f32，其余组件采用标注的精度。
176203

177204
<table>
178205
<thead>
@@ -230,6 +257,55 @@ bash robot_client/shell/cpp_client_example.sh
230257
<td>f32</td>
231258
<td><a href="https://huggingface.co/robotcpp/pi0-libero-f32">pi0-libero-f32</a></td>
232259
</tr>
260+
<tr>
261+
<td>StarVLA Qwen3-VL OFT</td>
262+
<td>Bridge</td>
263+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-OFT-Bridge-RT-1">StarVLA/Qwen3VL-OFT-Bridge-RT-1</a></td>
264+
<td>bf16 + f32 policy</td>
265+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-oft-bridge-bf16">starvla-qwen3-oft-bridge-bf16</a></td>
266+
</tr>
267+
<tr>
268+
<td>StarVLA Qwen3-VL GR00T</td>
269+
<td>Bridge</td>
270+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-GR00T-Bridge-RT-1">StarVLA/Qwen3VL-GR00T-Bridge-RT-1</a></td>
271+
<td>bf16 + f32 policy</td>
272+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-groot-bridge-bf16">starvla-qwen3-groot-bridge-bf16</a></td>
273+
</tr>
274+
<tr>
275+
<td>StarVLA Qwen3-VL PI_v3</td>
276+
<td>Bridge</td>
277+
<td><a href="https://huggingface.co/StarVLA/Qwen3VL-PI_v3-Bridge-RT_1">StarVLA/Qwen3VL-PI_v3-Bridge-RT_1</a></td>
278+
<td>bf16 + f32 policy</td>
279+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen3-pi-v3-bridge-bf16">starvla-qwen3-pi-v3-bridge-bf16</a></td>
280+
</tr>
281+
<tr>
282+
<td>StarVLA Qwen2.5-VL OFT</td>
283+
<td>Bridge</td>
284+
<td><a href="https://huggingface.co/StarVLA/Qwen-OFT-Bridge-RT-1">StarVLA/Qwen-OFT-Bridge-RT-1</a></td>
285+
<td>bf16 + f32 policy</td>
286+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-oft-bridge-bf16">starvla-qwen25-oft-bridge-bf16</a></td>
287+
</tr>
288+
<tr>
289+
<td>StarVLA Qwen2.5-VL GR00T</td>
290+
<td>Bridge</td>
291+
<td><a href="https://huggingface.co/StarVLA/Qwen-GR00T-Bridge-RT-1">StarVLA/Qwen-GR00T-Bridge-RT-1</a></td>
292+
<td>bf16 + f32 policy</td>
293+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-groot-bridge-bf16">starvla-qwen25-groot-bridge-bf16</a></td>
294+
</tr>
295+
<tr>
296+
<td>StarVLA Qwen2.5-VL PI</td>
297+
<td>Bridge</td>
298+
<td><a href="https://huggingface.co/StarVLA/Qwen-PI-Bridge-RT-1">StarVLA/Qwen-PI-Bridge-RT-1</a></td>
299+
<td>bf16 + f32 policy</td>
300+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-pi-bridge-bf16">starvla-qwen25-pi-bridge-bf16</a></td>
301+
</tr>
302+
<tr>
303+
<td>StarVLA Qwen2.5-VL FAST</td>
304+
<td>Bridge</td>
305+
<td><a href="https://huggingface.co/StarVLA/Qwen-FAST-Bridge-RT-1">StarVLA/Qwen-FAST-Bridge-RT-1</a></td>
306+
<td>bf16 + codec</td>
307+
<td><a href="https://huggingface.co/robotcpp/starvla-qwen25-fast-bridge-bf16">starvla-qwen25-fast-bridge-bf16</a></td>
308+
</tr>
233309
</tbody>
234310
</table>
235311

0 commit comments

Comments
 (0)