Quest 3 + WebXR companion (quest_hand device type, /vr, ML loop closed)

.gitignore

@@ -28,3 +28,7 @@ settings.json
 .venv/
 venv/
 env/
+
+# Local TLS certs for the VR /vr page (mkcert-generated, per-machine,
+# not secret in the cryptographic sense but no reason to ship them)
+*.pem

README.md

@@ -58,6 +58,23 @@ openmuscle predict -m data/models/random_forest_*/model.pkl
 openmuscle simulate --device-type flexgrid
 ```
 
+### VR companion (Meta Quest 3)
+
+A WebXR client lets you use a Quest 3's hand tracking as ML ground truth (richer than the LASK5 4-piston labeler) and visualize the model's predictions live as a ghost hand overlaid on your real hand.
+
+```bash
+# Windows one-click launcher: starts the server, sets up adb-reverse,
+# opens Quest Browser to /vr automatically
+pc/start-vr.bat                 # right arm (default)
+pc/start-vr.bat left            # left arm
+
+# Or manually: openmuscle web, then in Quest Browser go to
+# http://localhost:8000/vr   (via `adb reverse tcp:8000 tcp:8000`)
+# https://<lan-ip>:8000/vr   (via `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem`)
+```
+
+Full setup + per-session walkthrough: [`docs/vr-setup.md`](docs/vr-setup.md).
+
 ### Firmware (ESP32-S3 + MicroPython)
 
 ```bash
@@ -80,8 +97,9 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/
 | Command | Description |
 |---------|-------------|
 | `openmuscle receive` | Live heatmap of sensor data (matplotlib) |
-| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management — see [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) |
+| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management. Also serves the VR companion at `/vr` (see [`docs/vr-setup.md`](docs/vr-setup.md)). Full docs: [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) |
 | `openmuscle web --model M.pkl --hand IP` | Same UI plus live inference, with optional UDP forwarding of predictions to an OpenHand device |
+| `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem` | Same UI over HTTPS (required for the VR `/vr` page over LAN, since Quest Browser refuses WebXR hand-tracking on plain HTTP) |
 | `openmuscle record -o file.csv` | Record paired data to CSV |
 | `openmuscle train data.csv` | Train ML model (RandomForest) |
 | `openmuscle predict -m model.pkl` | Real-time inference (matplotlib) |
@@ -91,9 +109,11 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/
 ## Documentation
 
 - [Architecture Overview](docs/architecture.md)
-- [Packet Protocol Spec](docs/protocol.md)
+- [Packet Protocol Spec](docs/protocol.md) — includes the `quest_hand` device type
 - [Adding a New Device](docs/adding-a-device.md)
 - [CLI Usage Guide](docs/pc-cli.md)
+- [VR Setup & Operation](docs/vr-setup.md) — mkcert + Quest cert install + per-session walkthrough
+- [VR Testing Scenarios](docs/vr-testing-scenarios.md) — bring-up order, 2-minute smoke test, per-feature test runbook
 
 ## Adding a New Device
 
@@ -114,6 +134,9 @@ See [docs/adding-a-device.md](docs/adding-a-device.md) for details.
 **Standalone firmware (promoted from this repo's `embedded/devices/`):**
 - [FlexGridV3-Firmware](https://github.com/Open-Muscle/FlexGridV3-Firmware) — current FlexGrid revision, with the sensor-scan techniques writeup
 
+**AR / VR:**
+- [OpenMuscle-AR](https://github.com/Open-Muscle/OpenMuscle-AR) — AR/VR companion. The current WebXR client lives here in `pc/src/openmuscle/web/static/vr/` (tight coupling to the FastAPI server), but the AR repo is the discoverability anchor and the future home for the planned native Quest APK / BLE-direct work. See its [ROADMAP](https://github.com/Open-Muscle/OpenMuscle-AR/blob/main/ROADMAP.md).
+
 **Coordination / docs:**
 - [OpenMuscle-Hub](https://github.com/Open-Muscle/OpenMuscle-Hub) — central docs and roadmap
 

docs/protocol.md

@@ -56,10 +56,44 @@ All OpenMuscle devices communicate over **UDP port 3141** using **JSON-encoded U
 
 4 piston sensor values. Joystick is optional.
 
+### Quest hand tracking (`type: "quest_hand"`)
+
+Synthesized server-side from WebSocket frames sent by the WebXR client at `/vr` (browsers can't speak UDP). The payload represents one tracked hand sampled from `XRHand` each XRFrame:
+
+```json
+{
+    "values":     [px,py,pz, rx,ry,rz,rw,  ...]    // flat, 7 floats per joint
+    "handedness": "left" | "right",
+    "joint_names": ["wrist", "thumb-metacarpal", ...],
+    "hands": {
+        "handedness": "left" | "right",
+        "joints": [
+            {"name": "wrist", "pos": [x,y,z], "rot": [x,y,z,w], "radius": 0.02, "valid": true},
+            ...
+        ]
+    }
+}
+```
+
+- `values` follows the same convention as LASK5 — flat, in canonical joint × channel order — so the recorder and matcher pair `quest_hand` frames with FlexGrid frames identically to LASK5. The order is `[px, py, pz, rx, ry, rz, rw] × N joints`.
+- **Fixed-length contract.** A well-behaved client MUST send all joints in canonical `joint_names` order every frame, so the `values` length and per-slot meaning are stable across frames. When an individual joint pose is momentarily unavailable, send a zero position + identity quaternion with `"valid": false` rather than omitting the joint — omitting it would shift every later joint into the wrong CSV column and silently misalign the labels. The server defends against violations by locking the label width at the first label packet and padding/truncating later rows to keep the CSV rectangular (it counts and logs any mismatch), but clients should not rely on that.
+- `valid` (per joint, in `hands.joints`) marks whether the pose was actually tracked this frame. It is preserved in the JSONL sidecar so offline analysis can filter zero-filled joints; the flat `values` / CSV carry only the numbers.
+- `joint_names` lists the joints in the same order they're flattened into `values`. The standard set is the W3C WebXR Hand Input spec (25 joints: wrist + 4 thumb + 5 each for index/middle/ring/pinky).
+- `hands` is the structured per-joint form, kept in the JSONL sidecar for offline analysis. It's redundant with `values` + `joint_names` but easier to diff by eye.
+- Empty payloads (the headset reports tracking-lost for the whole hand this frame) are dropped silently by the server — they'd otherwise produce zero-rows that mislead training.
+
+A per-capture `<name>.labels.schema.json` sidecar is written on the first `quest_hand` packet of a recording. It maps `label_0..label_N` columns in the CSV back to `(joint, channel)`, so the wide label vector is self-describing.
+
+**Future-proofing note:** v1 captures one hand per recording (`handedness` is a single `"left"` or `"right"` string). A future `handedness: "both"` extension — payload carries both hands in `data.values` with the schema sidecar growing a parallel `joint_names_left` / `joint_names_right` (or a per-column `hand` field) — would be backward-compatible. Old consumers see a wider but still-flat values vector; new consumers can split it via the schema. Don't accidentally close that door in any future refactor of `_flatten_quest_joint` / `_write_labels_schema`.
+
 ### Adding a New Device Type
 
 Define a new `type` string and document the `data` shape. The PC-side parser auto-discovers devices by their `type` field.
 
+## Versioning policy
+
+Adding a new `type` string (e.g. `quest_hand`) is **non-breaking under v1.0** — existing parsers ignore unknown types, the schema envelope is unchanged, and devices that don't speak the new type are unaffected. Bump the `"v"` field to `"1.1"` (or beyond) only when changing the envelope itself (required-field set, semantics of `ts`/`id`, etc.).
+
 ## Backward Compatibility
 
 The PC parser (`openmuscle.protocol.parser`) auto-detects three formats:

docs/vr-setup.md

@@ -0,0 +1,171 @@
+# OpenMuscle VR — setup & operation
+
+Wear FlexGrid + a Meta Quest 3S together. The headset's hand tracking becomes the ground-truth label source for training a muscle-signal → finger-pose model. The desktop OpenMuscle UI is unchanged; this guide just adds the VR companion.
+
+Companion code: [`pc/src/openmuscle/web/static/vr/`](../pc/src/openmuscle/web/static/vr/) (client) and [`pc/src/openmuscle/web/app.py`](../pc/src/openmuscle/web/app.py) (server endpoints). The Quest is integrated as a synthetic device with `device_type: "quest_hand"` — see [`protocol.md`](protocol.md) for the wire format.
+
+## Why HTTPS is non-optional
+
+Quest Browser refuses to grant WebXR hand-tracking outside a secure context. You have two ways to get one:
+
+| Mode | Pros | Cons |
+|---|---|---|
+| **HTTPS over LAN** via mkcert | untethered, real use case | one-time cert install on the headset |
+| **`adb reverse` over USB** (localhost) | zero certs, fast for debugging | cable on the headset distorts natural arm motion |
+
+Day-to-day capture work should use HTTPS. The USB fallback is for proving the loop end-to-end before you've finished cert setup.
+
+## One-time setup
+
+### 1. mkcert on the PC
+
+```powershell
+# Windows (choco). On macOS: `brew install mkcert`. On Linux: distro package.
+choco install mkcert
+
+mkcert -install                       # installs the mkcert root CA into Windows
+mkcert -cert-file vr-cert.pem `
+       -key-file  vr-key.pem `
+       192.168.1.42 localhost          # <-- replace with your PC's LAN IP
+```
+
+The two PEM files land in your current directory. Keep them next to the captures dir, or anywhere the `openmuscle web` command can read them.
+
+### 2. Install the mkcert root CA on the Quest
+
+**Heads up:** this is the painful step. Meta's Horizon OS consumer shell **hides** the standard Android "Settings → Security → Install a certificate" path. The Settings panel you see when tapping the gear icon in VR is Meta's Horizon Settings, not AOSP Settings — different surface, no cert-install button. You have to launch the AOSP Settings activity via ADB.
+
+Procedure validated on Quest 3S, Horizon OS as of May 2026:
+
+```powershell
+# (a) Push the root CA from your PC to the headset's Downloads folder
+adb push "$env:LOCALAPPDATA\mkcert\rootCA.pem" /sdcard/Download/openmuscle-vr-rootCA.pem
+
+# (b) Launch the AOSP Security Dashboard activity directly via ADB.
+#     The $ MUST be backslash-escaped through PowerShell single-quotes so it
+#     survives BOTH the host shell AND the Android shell. Without the escape,
+#     $ gets eaten as a variable expansion and the activity name truncates to
+#     just '.Settings' which is not exported (Permission Denial).
+adb shell 'am start -n com.android.settings/.Settings\$SecurityDashboardActivity'
+```
+
+On the Quest after running those:
+
+1. **The 2D Settings panel may not appear immediately in your view.** Horizon shell aggressively suppresses unfamiliar 2D activities. **Press the Meta button** on your right controller, look for "Settings" (or similar) in the universal-menu app switcher, and click to bring it forward.
+2. In the AOSP Security dashboard: **Trusted credentials** → **Encryption** section → **Install a certificate** → **CA certificate**.
+3. **"Your data won't be private"** warning → tap **Install anyway** (your own root CA, the warning is generic Android boilerplate).
+4. **If no screen-lock PIN is set**, Quest refuses and prompts you to set one. Set a PIN, then re-do step 2.
+5. **File picker gotcha:** defaults to "Recent" view which is **empty** on a fresh install. Tap the **hamburger menu (≡)** at the top of the picker → navigate to **Internal Storage → Download** → tap **openmuscle-vr-rootCA.pem**.
+6. Success: "CA certificate installed" or similar.
+
+You only do this once per headset. The CA stays trusted across Horizon OS updates.
+
+**Fallback if you can't find the AOSP Settings panel in the app switcher:** open the headset's **Files** app, navigate to **Download/openmuscle-vr-rootCA.pem**, tap it. Some Horizon versions trigger the install dialog from the file-open intent directly.
+
+**Activities that work on Horizon OS** (useful for future Quest dev when standard menus are hidden):
+- `com.android.settings/.Settings$SecurityDashboardActivity` — security menu (the one you just used)
+- `com.android.settings/.Settings$TrustedCredentialsSettingsActivity` — view installed certs (for verifying yours is there)
+- `com.android.settings/.security.CredentialStorage` — direct install activity, but Horizon often suppresses its panel
+- `com.android.settings/.Settings$NetworkDashboardActivity`
+- `com.android.settings/.Settings$DevelopmentSettingsDashboardActivity`
+
+**Activities that DON'T exist on Horizon** (don't try — Error type 3):
+- `com.android.settings/.Settings$SecuritySettingsActivity` (older Android name, removed)
+- `com.android.settings/.Settings$EncryptionAndCredentialActivity` (renamed)
+
+**Non-exported, can't launch via ADB** (Permission Denial from null uid):
+- `com.android.settings/.Settings` (the bare main activity)
+
+### 3. Start the server with HTTPS
+
+```powershell
+cd D:\path\to\OpenMuscle-Software\pc
+pip install -e .                       # first time only
+
+openmuscle web --ssl-certfile vr-cert.pem `
+               --ssl-keyfile  vr-key.pem
+```
+
+You'll see:
+
+```
+OpenMuscle web UI: https://localhost:8000
+Listening for devices on UDP 3141
+TLS: cert=vr-cert.pem  key=vr-key.pem
+WebXR URL for the Quest: https://<your-LAN-ip>:8000/vr
+```
+
+## Per-session flow
+
+1. **Boot FlexGrid** — it joins your Wi-Fi and starts streaming UDP to port 3141. Confirm in the desktop UI's Devices panel.
+2. **Open `/vr` in Quest Browser** at `https://<your-LAN-ip>:8000/vr`. (The desktop UI lives at `/`; the VR companion at `/vr`.)
+3. The landing page runs three preflight checks. All three should be green checkmarks:
+   - HTTPS / secure context
+   - WebXR + immersive-vr supported
+   - Server reachable
+4. **Pick the FlexGrid arm** in the dropdown (`right` is default). Only the joints of this hand get pushed to `/ws/quest`. The other hand stays free for the record button.
+5. Tap **Enter VR**. Grant hand-tracking when Quest asks.
+6. A floating panel ~70cm in front shows the live FlexGrid heatmap. Header strip above it shows per-device Hz when idle, or `REC · N rows · match X%` when recording.
+7. Hold your captured hand up; you'll see small blue spheres on each tracked joint.
+8. **Start recording**:
+   - **Off-hand button**: tap the gray sphere below the heatmap with your other hand's index finger. It turns red.
+   - **Pinch gesture**: pinch index + thumb on the captured hand and hold for ~1 second. A yellow ring on the index tip fills during the hold; release triggers the toggle.
+9. Do your gesture set. For the first FlexGrid V3 training run we focus on **index / middle / ring / pinky finger curls** — the thumb is driven by intrinsic hand muscles the FlexGrid can't see, so it's excluded.
+10. **Stop recording** the same way.
+11. Take the headset off. The capture is in `data/raw/merged/`:
+
+    ```
+    capture_<ts>.csv                  # paired (sensor + 182 joint floats)
+    capture_<ts>.sensor.jsonl         # raw FlexGrid packets
+    capture_<ts>.label.jsonl          # raw Quest packets
+    capture_<ts>.labels.schema.json   # column -> (joint, channel) map
+    capture_<ts>.meta.json            # auto.label_source="quest_hand" + your tags
+    ```
+
+12. **Train**: in the desktop UI's Captures panel, tick your new capture(s) and hit **Train selected**. Or from CLI: `openmuscle train data/raw/merged/capture_<ts>.csv`.
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---|---|---|
+| "Enter VR" button stays disabled | Page opened in the in-headset Library panel, not Quest Browser | Open Quest Browser app explicitly; navigate to the URL |
+| "INSECURE (http:)" in the HTTPS checkmark | Reached the page over plain HTTP | Either switch to `https://` (LAN + mkcert) or use `adb reverse` + `http://localhost:8000/vr` |
+| Cert install on Quest fails silently | Quest doesn't have a screen-lock PIN | Settings → Security → Lock screen → set a PIN, then re-try CA install |
+| Heatmap stays "waiting for FlexGrid…" inside VR | FlexGrid not streaming, or wrong UDP port | Open the desktop UI; verify the device is in the Devices panel and packets > 0 |
+| `match_rate` is very low (< 30%) in recording header | Quest WS connection dropping, or sensor and label streams have wildly different rates | Check WS reconnect logs in the desktop UI's Log panel; try increasing `window_ms` via the recording start body if needed (default for Quest is 175 ms) |
+| Joints visible but no `quest-<arm>` device in the Devices panel | `/ws/quest` failed to connect (e.g. CORS, cert mismatch) | Check the Quest Browser dev-tools (chrome://inspect on a connected PC) for WS errors |
+
+## Architecture, in one diagram
+
+```
+                            ┌──────────────────────┐
+                            │   Meta Quest 3S       │
+                            │                       │
+   ┌───────────┐            │  Quest Browser /vr   │
+   │ FlexGrid  │            │    │                 │
+   │ (ESP32-   │   UDP      │    │ WebXR XRHand    │
+   │  S3 WiFi) │ ─────────→ │    │ sample joints   │
+   └───────────┘   :3141    │    ▼                 │
+        ▲                   │  WS /ws/quest        │
+        │                   │    │                 │
+        │                   │    │   WS /ws/live   │
+        │                   │    │   ◀──── heatmap │
+        │                   │    │                 │
+        │                   └────┼─────────────────┘
+        │                        │
+        │                        ▼
+        │            ┌──────────────────────────┐
+        │            │  PC: openmuscle web      │
+        │            │                          │
+        │  UDP       │   UDPListener  ─→ ┐      │
+        └────────────┤                   │      │
+                     │   ingest_quest    ─→ AppState   ─→ paired CSV
+                     │   (synthesizes a    matcher        + JSONL +
+                     │    quest_hand       recorder       labels.schema.json
+                     │    packet)                         + meta.json
+                     │                                    │
+                     │   /ws/live ◀── snapshot ───────────┘
+                     └──────────────────────────┘
+```
+
+The whole integration trick is that `ingest_quest_packet` synthesizes an `OpenMusclePacket` of type `quest_hand` in-process, then hands it to `_handle_packet` — the **same** function the UDP listener feeds into. From the recorder's view the Quest is just another device. Zero special-casing downstream.