Skip to content

README.md

Latest commit

 

History

History
97 lines (71 loc) · 3.12 KB

README.md

File metadata and controls

97 lines (71 loc) · 3.12 KB
Raw

SimplerEnv WidowX Bridge Eval

This directory evaluates the robot.cpp StarVLA GGUF runtime on the SimplerEnv WidowX Bridge tasks.

Protocol

  • Four Bridge tasks with object episodes 0..23
  • At most 120 steps per episode at 5 Hz
  • Visual-matching RGB overlay resized to 224x224 with OpenCV INTER_AREA
  • One action chunk per step with the official seven-prediction adaptive ensemble

A full run contains 96 rollouts. The result reports overall and per-task success rates; subset runs use partial coverage.

Setup

Convert a checkpoint as described in the StarVLA guide and build the CUDA runtime.

The environment uses these revisions:

SimplerEnv:          06accaca93535902d408da4855f21cece12bceb7
ManiSkill2_real2sim: ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682

conda env create -f eval/simpler_env/environment.yaml
conda activate robotcpp-simpler-env

git clone --recurse-submodules https://github.com/simpler-env/SimplerEnv \
  ckpts/simpler_env/source/SimplerEnv
git -C ckpts/simpler_env/source/SimplerEnv checkout \
  06accaca93535902d408da4855f21cece12bceb7
git -C ckpts/simpler_env/source/SimplerEnv submodule update --init --recursive

pip install -e ckpts/simpler_env/source/SimplerEnv/ManiSkill2_real2sim
pip install -e ckpts/simpler_env/source/SimplerEnv

Headless simulation requires a working Vulkan ICD. Run SimplerEnv's environment test first to confirm that SAPIEN can find a rendering device.

Run

VARIANT accepts oft, groot, pi_v3, qwen25_oft, qwen25_groot, qwen25_pi, and qwen25_fast.

Run the full profile:

CUDA_VISIBLE_DEVICES=0 \
VARIANT=oft \
OUTPUT=ckpts/starvla/results/oft/bridge.json \
bash eval/simpler_env/scripts/run_model_server.sh

Run one smoke episode:

CUDA_VISIBLE_DEVICES=0 \
VARIANT=groot TASK_IDS=0 EPISODE_IDS=0 \
bash eval/simpler_env/scripts/run_model_server.sh

The script reads three GGUF files from ckpts/starvla/gguf/<variant> and uses build_cuda/bin/model-server by default. Common overrides are GGUF_DIR, SERVER_BIN, PYTHON, SIMPLER_ENV_ROOT, TASK_IDS, EPISODE_IDS, REPEATS, and OUTPUT.

Each task/repeat starts a fresh model-server. Results include checkpoint identity, rollout records, success rates, and timing summaries.

Latency

Benchmark the official PyTorch checkpoint directly:

CUDA_VISIBLE_DEVICES=0 python -m eval.simpler_env.runners.latency_starvla \
  --variant oft --compile-model

The runner selects the checkpoint, Qwen assets, and Bridge normalization from the StarVLA catalog. It reports policy, action unnormalization, and total latency after 5 warmup calls and 20 measured calls. Policy latency includes StarVLA's image/text preprocessing and model forward. Omit --compile-model for eager PyTorch. Compilation is lazy; the first FAST warmup can take several minutes and is not included in the reported measurements.

For the robot.cpp model-server path, use the common server benchmark:

CUDA_VISIBLE_DEVICES=0 N_BATCH=2048 SKIP_BUILD=1 \
GGUF_DIR="$PWD/ckpts/starvla/gguf/oft" \
bash robot_server/test/test_server_latency.sh starvla linux-cuda starvla-bridge