eval: add SimplerEnv StarVLA evaluation

Author: JJJYmmm

README.md

@@ -151,7 +151,9 @@ bash robot_client/shell/cpp_client_example.sh
 
 ### 🧪 Using model-server in simulation, using LIBERO as the example
 
-See the [LIBERO simulation evaluation guide](eval/libero/README.md).
+See the [LIBERO simulation evaluation guide](eval/libero/README.md). To run
+local StarVLA GGUF models on the WidowX Bridge tasks, see the
+[SimplerEnv Bridge guide](eval/simpler_env/README.md).
 
 ### 🦾 Using model-server on real hardware, using SO-101 as the example
 
@@ -344,6 +346,7 @@ robot.cpp/
 ├── eval/
 │   ├── base_platform.py           # Shared base class for real-robot platforms
 │   ├── libero/                    # LIBERO simulation evaluation
+│   ├── simpler_env/               # SimplerEnv WidowX / Bridge evaluation
 │   └── lerobot_so101/             # SO-101 real-robot scripts and examples
 └── third_party/
     ├── llama.cpp/                 # ggml / llama.cpp backend
@@ -400,6 +403,7 @@ Robot.cpp's design and implementation benefit from several excellent open-source
 * [llama.cpp](https://github.com/ggerganov/llama.cpp): provides lightweight local inference, the GGML/GGUF ecosystem, and cross-platform backend foundations. This project continues building robot model inference capabilities on top of its engineering philosophy and low-level runtime.
 * [LeRobot](https://github.com/huggingface/lerobot): provides reference implementations for robot data, policy training, and real-robot integration. The SO-101 real-robot example and parts of the evaluation flow in this project are inspired by the LeRobot ecosystem.
 * [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO): provides robot simulation tasks and evaluation benchmarks. The LIBERO simulation evaluation flow in this project is based on its task environments and benchmark design.
+* [SimplerEnv](https://github.com/simpler-env/SimplerEnv): provides real-to-sim robot evaluation environments. The StarVLA Bridge success-rate evaluation uses its WidowX task suite and official visual-matching assets.
 * [OpenPI](https://github.com/Physical-Intelligence/openpi): provides the pi0 policy model and related open-source implementation. The pi0 runtime, conversion, and evaluation work in this project references OpenPI's model design.
 
 Thanks to these projects and communities for their contributions to robot learning and on-device inference.

README_ZH.md

@@ -151,7 +151,9 @@ bash robot_client/shell/cpp_client_example.sh
 
 ### 🧪 model-server在仿真平台上的使用（以LIBERO为例）
 
-详见 [LIBERO 仿真评测说明](eval/libero/README_ZH.md)。
+详见 [LIBERO 仿真评测说明](eval/libero/README_ZH.md)。在 WidowX Bridge 任务上运行
+StarVLA，并比较 Python checkpoint 与 GGUF 的方法见
+[SimplerEnv Bridge 说明](eval/simpler_env/README_ZH.md)。
 
 ### 🦾 model-server在真机平台上的使用（以SO-101为例）
 
@@ -345,6 +347,7 @@ robot.cpp/
 ├── eval/
 │   ├── base_platform.py           # 真机 platform 的统一基类
 │   ├── libero/                    # LIBERO 仿真评测
+│   ├── simpler_env/               # SimplerEnv WidowX / Bridge 仿真评测
 │   └── lerobot_so101/             # SO-101 真机相关脚本与示例
 └── third_party/
     ├── llama.cpp/                 # ggml / llama.cpp 后端
@@ -401,6 +404,7 @@ robot.cpp 的设计与实现受益于多个优秀的开源项目：
 * [llama.cpp](https://github.com/ggerganov/llama.cpp)：提供了轻量化本地推理、GGML/GGUF 生态与跨平台后端基础，本项目在其工程哲学和底层能力上继续构建机器人模型推理框架。
 * [LeRobot](https://github.com/huggingface/lerobot)：提供了机器人数据、策略训练与真实机器人接入的参考实现，本项目的 SO-101 真机示例与部分评测流程参考了 LeRobot 生态。
 * [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO)：提供了机器人仿真任务与评测基准，本项目的 LIBERO 仿真评测流程基于其任务环境与 benchmark 设计。
+* [SimplerEnv](https://github.com/simpler-env/SimplerEnv)：提供了 real-to-sim 机器人评测环境，本项目的 StarVLA Bridge 成功率评测使用其 WidowX 任务集与官方 visual-matching 资产。
 * [OpenPI](https://github.com/Physical-Intelligence/openpi)：提供了pi0策略模型与相关开源实现，本项目的 pi0 相关 runtime、转换与评测工作参考了 OpenPI 的模型设计。
 
 感谢这些项目和社区为机器人学习与端侧推理生态做出的贡献。

eval/README.md

@@ -22,6 +22,7 @@ The repo root [README.md](../README.md) describes the three-layer layout:
 eval/
 ├── base_platform.py          # Shared base class for real-robot / sim platforms
 ├── libero/                   # LIBERO sim benchmark (multi-camera, batch rollout)
+├── simpler_env/              # SimplerEnv WidowX / Bridge closed-loop benchmark
 └── lerobot_so101/            # SO-101 real-robot sync closed-loop example
 ```
 
@@ -30,12 +31,16 @@ eval/
 | Directory | Scenario | Notes |
 |---|---|---|
 | [`libero/`](libero/README.md) | Sim eval | LIBERO benchmark with C++ policy rollout and LeRobot baseline. [中文](libero/README_ZH.md) |
+| [`simpler_env/`](simpler_env/README.md) | Sim eval | Runs StarVLA Python and GGUF on the SimplerEnv WidowX Bridge tasks. [中文](simpler_env/README_ZH.md) |
 | [`lerobot_so101/`](lerobot_so101/README.md) | Real robot | SO-101 follower + single-camera observe → predict → act loop. [中文](lerobot_so101/README_ZH.md) |
 
 The two examples are organized slightly differently:
 
 - **SO-101** follows the standard `BasePlatform` + `RobotPolicy` + `SyncControlLoop` path—use it as the template for new real-robot platforms.
 - **LIBERO** implements a dedicated observation adapter under `eval/libero/policy/` (multi-camera, state packing, sim rollout) and does **not** inherit `BasePlatform`—use it as a reference for new **sim benchmarks**.
+- **SimplerEnv** follows the same dedicated runner pattern and implements the
+  WidowX action transform, temporal ensemble, normalization profile, and Bridge
+  task settings for both Python and C++ runs.
 
 ## Standard closed-loop data flow
 
@@ -237,6 +242,7 @@ LIBERO’s [`ModelServerPolicy`](libero/policy/model_server.py) is an example of
 
 - [SO-101 real-robot guide](lerobot_so101/README.md) · [中文](lerobot_so101/README_ZH.md)
 - [LIBERO sim eval](libero/README.md) · [中文](libero/README_ZH.md)
+- [SimplerEnv Bridge eval](simpler_env/README.md) · [中文](simpler_env/README_ZH.md)
 - [robot_server launch and protocol](../robot_server/README.md)
 - [robot_client and policy](../robot_client/README.md)
 - [Adding a new model runtime](../src/README.md)

eval/README_ZH.md

@@ -24,6 +24,7 @@
 eval/
 ├── base_platform.py          # 真机 / 仿真 platform 的统一基类
 ├── libero/                   # LIBERO 仿真 benchmark（多相机、批量 rollout）
+├── simpler_env/              # SimplerEnv WidowX / Bridge 闭环 benchmark
 └── lerobot_so101/            # SO-101 真机同步闭环示例
 ```
 
@@ -33,13 +34,15 @@ eval/
 | 目录                                             | 场景   | 说明                                                                                         |
 | ---------------------------------------------- | ---- | ------------------------------------------------------------------------------------------ |
 | `[libero/](libero/README_ZH.md)`               | 仿真评测 | 面向 LIBERO benchmark，含 C++ policy rollout 与 LeRobot baseline 对比。[English](libero/README.md) |
+| `[simpler_env/](simpler_env/README_ZH.md)`     | 仿真评测 | 在 SimplerEnv WidowX Bridge 任务上运行并比较 StarVLA Python 与 GGUF。[English](simpler_env/README.md) |
 | `[lerobot_so101/](lerobot_so101/README_ZH.md)` | 真机闭环 | SO-101 follower + 单相机的 observe → predict → act 同步控制。[English](lerobot_so101/README.md)     |
 
 
 两类示例的组织方式略有不同：
 
 - **SO-101** 走标准 `BasePlatform` + `RobotPolicy` + `SyncControlLoop` 路径，适合作为新增真机 platform 的模板。
 - **LIBERO** 在 `eval/libero/policy/` 里实现了专用的 observation 适配（多相机、state 拼接、仿真 rollout），不继承 `BasePlatform`，适合作为新增 **仿真 benchmark** 的参考。
+- **SimplerEnv** 沿用专用 runner 结构，为 Python 和 C++ 实现相同的 WidowX action 变换、时序集成、normalization profile 与 Bridge 任务设置。
 
 ## 标准闭环数据流
 
@@ -244,6 +247,7 @@ LIBERO 的 `[ModelServerPolicy](libero/policy/model_server.py)` 即为自定义
 
 - [SO-101 真机使用说明](lerobot_so101/README_ZH.md)
 - [LIBERO 仿真评测说明](libero/README_ZH.md)
+- [SimplerEnv Bridge 仿真评测说明](simpler_env/README_ZH.md)
 - [robot_server 启动与协议](../robot_server/README_ZH.md)
 - [robot_client 与 policy](../robot_client/README.md)
 - [新增模型 runtime](../src/README_ZH.md)

eval/simpler_env/README.md

@@ -0,0 +1,97 @@
+# SimplerEnv WidowX Bridge Eval
+
+This directory evaluates the robot.cpp StarVLA GGUF runtime on the SimplerEnv
+WidowX Bridge tasks.
+
+## Protocol
+
+- Four Bridge tasks with object episodes `0..23`
+- At most 120 steps per episode at 5 Hz
+- Visual-matching RGB overlay resized to 224x224 with OpenCV `INTER_AREA`
+- One action chunk per step with the official seven-prediction adaptive ensemble
+
+A full run contains 96 rollouts. The result reports overall and per-task success
+rates; subset runs use `partial` coverage.
+
+## Setup
+
+Convert a checkpoint as described in the
+[StarVLA guide](../../tools/hf2gguf/starvla/README.md) and build the CUDA runtime.
+
+The environment uses these revisions:
+
+```text
+SimplerEnv:          06accaca93535902d408da4855f21cece12bceb7
+ManiSkill2_real2sim: ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682
+```
+
+```bash
+conda env create -f eval/simpler_env/environment.yaml
+conda activate robotcpp-simpler-env
+
+git clone --recurse-submodules https://github.com/simpler-env/SimplerEnv \
+  ckpts/simpler_env/source/SimplerEnv
+git -C ckpts/simpler_env/source/SimplerEnv checkout \
+  06accaca93535902d408da4855f21cece12bceb7
+git -C ckpts/simpler_env/source/SimplerEnv submodule update --init --recursive
+
+pip install -e ckpts/simpler_env/source/SimplerEnv/ManiSkill2_real2sim
+pip install -e ckpts/simpler_env/source/SimplerEnv
+```
+
+Headless simulation requires a working Vulkan ICD. Run SimplerEnv's environment
+test first to confirm that SAPIEN can find a rendering device.
+
+## Run
+
+`VARIANT` accepts `oft`, `groot`, `pi_v3`, `qwen25_oft`, `qwen25_groot`,
+`qwen25_pi`, and `qwen25_fast`.
+
+Run the full profile:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+VARIANT=oft \
+OUTPUT=ckpts/starvla/results/oft/bridge.json \
+bash eval/simpler_env/scripts/run_model_server.sh
+```
+
+Run one smoke episode:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+VARIANT=groot TASK_IDS=0 EPISODE_IDS=0 \
+bash eval/simpler_env/scripts/run_model_server.sh
+```
+
+The script reads three GGUF files from `ckpts/starvla/gguf/<variant>` and uses
+`build_cuda/bin/model-server` by default. Common overrides are `GGUF_DIR`,
+`SERVER_BIN`, `PYTHON`, `SIMPLER_ENV_ROOT`, `TASK_IDS`,
+`EPISODE_IDS`, `REPEATS`, and `OUTPUT`.
+
+Each task/repeat starts a fresh model-server. Results include checkpoint
+identity, rollout records, success rates, and timing summaries.
+
+## Latency
+
+Benchmark the official PyTorch checkpoint directly:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python -m eval.simpler_env.runners.latency_starvla \
+  --variant oft --compile-model
+```
+
+The runner selects the checkpoint, Qwen assets, and Bridge normalization from
+the StarVLA catalog. It reports policy, action unnormalization, and total
+latency after 5 warmup calls and 20 measured calls. Policy latency includes
+StarVLA's image/text preprocessing and model forward. Omit `--compile-model`
+for eager PyTorch. Compilation is lazy; the first FAST warmup can take several
+minutes and is not included in the reported measurements.
+
+For the robot.cpp model-server path, use the common server benchmark:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 N_BATCH=2048 SKIP_BUILD=1 \
+GGUF_DIR="$PWD/ckpts/starvla/gguf/oft" \
+bash robot_server/test/test_server_latency.sh starvla linux-cuda starvla-bridge
+```

eval/simpler_env/README_ZH.md

@@ -0,0 +1,97 @@
+# SimplerEnv WidowX Bridge 评测
+
+本目录使用 robot.cpp 的 StarVLA GGUF runtime 运行 SimplerEnv WidowX Bridge 任务。
+
+## 评测设置
+
+- 四个 Bridge 任务，每个任务包含 object episode `0..23`
+- 每个 episode 最多 120 步，控制频率 5 Hz
+- visual-matching RGB overlay 使用 OpenCV `INTER_AREA` 缩放到 224x224
+- 每步预测一个 action chunk，并对最近七次预测做自适应集成
+
+完整评测包含 96 个 rollout。结果文件同时记录总体和各任务成功率；子集运行会标记为
+`partial` coverage。
+
+## 安装
+
+先按 [StarVLA 转换说明](../../tools/hf2gguf/starvla/README.md) 生成 GGUF，并完成 CUDA
+构建。
+
+SimplerEnv 使用以下 revision：
+
+```text
+SimplerEnv:          06accaca93535902d408da4855f21cece12bceb7
+ManiSkill2_real2sim: ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682
+```
+
+```bash
+conda env create -f eval/simpler_env/environment.yaml
+conda activate robotcpp-simpler-env
+
+git clone --recurse-submodules https://github.com/simpler-env/SimplerEnv \
+  ckpts/simpler_env/source/SimplerEnv
+git -C ckpts/simpler_env/source/SimplerEnv checkout \
+  06accaca93535902d408da4855f21cece12bceb7
+git -C ckpts/simpler_env/source/SimplerEnv submodule update --init --recursive
+
+pip install -e ckpts/simpler_env/source/SimplerEnv/ManiSkill2_real2sim
+pip install -e ckpts/simpler_env/source/SimplerEnv
+```
+
+无头运行需要可用的 Vulkan ICD。请先运行 SimplerEnv 自带的环境测试，确认 SAPIEN 能找到
+渲染设备。
+
+## 运行
+
+`VARIANT` 支持：
+
+```text
+oft groot pi_v3 qwen25_oft qwen25_groot qwen25_pi qwen25_fast
+```
+
+完整运行：
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+VARIANT=oft \
+OUTPUT=ckpts/starvla/results/oft/bridge.json \
+bash eval/simpler_env/scripts/run_model_server.sh
+```
+
+快速检查一个 episode：
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+VARIANT=groot TASK_IDS=0 EPISODE_IDS=0 \
+bash eval/simpler_env/scripts/run_model_server.sh
+```
+
+脚本默认从 `ckpts/starvla/gguf/<variant>` 读取三个 GGUF，并使用
+`build_cuda/bin/model-server`。常用覆盖项包括 `GGUF_DIR`、`SERVER_BIN`、`PYTHON`、
+`SIMPLER_ENV_ROOT`、`TASK_IDS`、`EPISODE_IDS`、`REPEATS` 和 `OUTPUT`。
+
+每个 task/repeat 会启动新的 model-server。结果包含 checkpoint 标识、rollout 明细、成功率
+和各阶段耗时。
+
+## 延迟测试
+
+直接测试官方 PyTorch checkpoint：
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python -m eval.simpler_env.runners.latency_starvla \
+  --variant oft --compile-model
+```
+
+runner 会根据 StarVLA catalog 选择 checkpoint、Qwen 资源和 Bridge 归一化配置。默认先预热
+5 次，再统计 20 次推理，并分别报告 policy、action 反归一化和总耗时。policy 耗时包含
+StarVLA 的图像/文本预处理和模型 forward。去掉 `--compile-model` 即可测试 eager
+PyTorch。`torch.compile` 为惰性编译；FAST 第一次预热可能需要几分钟，这部分不会计入
+最终统计。
+
+robot.cpp model-server 使用统一的服务端测试脚本：
+
+```bash
+CUDA_VISIBLE_DEVICES=0 N_BATCH=2048 SKIP_BUILD=1 \
+GGUF_DIR="$PWD/ckpts/starvla/gguf/oft" \
+bash robot_server/test/test_server_latency.sh starvla linux-cuda starvla-bridge
+```

eval/simpler_env/__init__.py

@@ -0,0 +1 @@
+"""SimplerEnv evaluation integration."""

eval/simpler_env/environment.yaml

@@ -0,0 +1,19 @@
+name: robotcpp-simpler-env
+channels:
+  - conda-forge
+dependencies:
+  - python=3.10
+  - pip
+  - ffmpeg
+  - pip:
+      - numpy==1.24.4
+      - scipy==1.11.4
+      - opencv-python==4.11.0.86
+      - opencv-python-headless==4.11.0.86
+      - setuptools<81
+      - transforms3d
+      - matplotlib
+      - mediapy
+      - tyro
+      - msgpack
+      - websockets

eval/simpler_env/policy/__init__.py

@@ -0,0 +1 @@
+"""Policies used by the SimplerEnv runners."""