AR/VR companion to the OpenMuscle ecosystem — turning a Meta Quest 3 headset into a labeling rig, live-prediction demo, and field-capture device for the muscle-signal → finger-pose machine learning model.
This repo is the discoverability home and future expansion point for OpenMuscle AR/VR work. The current WebXR client lives in a sibling repo (see below) because it's tightly coupled to the FastAPI server there; this repo will grow native APK / OpenXR / BLE-direct work as those land.
Right now, the working WebXR client is at:
github.com/Open-Muscle/OpenMuscle-Software
└── pc/src/openmuscle/web/static/vr/
├── index.html
├── app.js # XRHand capture, ray-pointer menu, ghost predicted hand,
│ # passthrough, sync slate, drag handles, ...
└── styles.css
It runs in Quest Browser, served by openmuscle web (FastAPI). Why it isn't here yet: the client posts hand-joint frames to a /ws/quest WebSocket on the Python server, which synthesizes them into in-process OpenMusclePacket(device_type="quest_hand") and routes them through the same packet pipeline UDP devices use. That coupling makes it a static-files appendage to the server, not an independent product.
When the architecture genuinely diverges (native APK in Unity, BLE-direct bracelet→headset, etc.) the new code lands here. See ROADMAP.md for the planned phases.
You want to run the current WebXR client. Steps live in OpenMuscle-Software:
- One-click launcher (Windows):
pc/start-vr.bat - Full operator guide:
docs/vr-setup.md(includes the mkcert + Quest cert install procedure — non-obvious because Meta's Horizon OS hides the standard Android cert UI) - Web UI architecture:
pc/src/openmuscle/web/README.md— "VR companion" section
The active PR for the WebXR work: Open-Muscle/OpenMuscle-Software#1.
- Two modes via URL param:
?mode=vr(immersive, deliberate gesture training) and?mode=ar(passthrough, real-world visible, field capture during natural activities) - Hand tracking as ML ground truth: 25 joints × 7 floats per frame from Quest's
XRHandAPI, paired with FlexGrid sensor frames via the existing temporal matcher - Live heatmap panel: floats in front of you, shows FlexGrid pressure matrix in real time
- Ghost predicted hand: amber semi-transparent joint spheres overlaid on your real hand showing model predictions (wrist-aligned position + orientation)
- REAL vs PRED finger-curl bars: per-finger (index/middle/ring/pinky) visualization of model error
- In-VR menu: REC / SESSION / PREDICT / TRAIN / RECENTER / EXIT VR — operate the whole capture-train-predict loop without taking the headset off
- Movable panels: drag handles on the data display and menu clusters, off-hand ray + pinch to grab
- Collapse-to-STOP while recording: full menu collapses to a single big red STOP button during capture sessions so it doesn't dominate your view during real work
- Sync slate: 2.5-second high-contrast splash at REC press, visible in Quest's built-in screen recording for frame-accurate video↔CSV pairing post-hoc
OpenMuscle-AR/
├── README.md # this file
├── ROADMAP.md # planned phases (WebXR → native APK → BLE-direct)
├── docs/
│ └── design-explorations/ # scoping docs, architecture sketches
└── samples/ # tiny WebXR snippets for extension / experiments
For now this repo is mostly documentation + planning. The first executable code that lands here will be the native Quest APK when that phase starts.
- OpenMuscle-Software — Python PC tools, FastAPI server, current WebXR client, training/inference pipeline
- OpenMuscle-FlexGrid — Hardware design files (KiCad) for the 60-sensor flex-rigid pressure bracelet
- FlexGridV3-Firmware — MicroPython firmware for the current FlexGrid revision
- OpenMuscle-Hub — Central docs and roadmap for the whole ecosystem
MIT (for code that lands here in the future). Documentation: CC BY-SA 4.0.