diff --git a/.gitignore b/.gitignore index aa836db..3602d5e 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,7 @@ settings.json .venv/ venv/ env/ + +# Local TLS certs for the VR /vr page (mkcert-generated, per-machine, +# not secret in the cryptographic sense but no reason to ship them) +*.pem diff --git a/README.md b/README.md index 392e572..f2f1c77 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,23 @@ openmuscle predict -m data/models/random_forest_*/model.pkl openmuscle simulate --device-type flexgrid ``` +### VR companion (Meta Quest 3) + +A WebXR client lets you use a Quest 3's hand tracking as ML ground truth (richer than the LASK5 4-piston labeler) and visualize the model's predictions live as a ghost hand overlaid on your real hand. + +```bash +# Windows one-click launcher: starts the server, sets up adb-reverse, +# opens Quest Browser to /vr automatically +pc/start-vr.bat # right arm (default) +pc/start-vr.bat left # left arm + +# Or manually: openmuscle web, then in Quest Browser go to +# http://localhost:8000/vr (via `adb reverse tcp:8000 tcp:8000`) +# https://:8000/vr (via `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem`) +``` + +Full setup + per-session walkthrough: [`docs/vr-setup.md`](docs/vr-setup.md). + ### Firmware (ESP32-S3 + MicroPython) ```bash @@ -80,8 +97,9 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/ | Command | Description | |---------|-------------| | `openmuscle receive` | Live heatmap of sensor data (matplotlib) | -| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management — see [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) | +| `openmuscle web` | Browser UI: live heatmap, LASK5 piston bars, ML inference panel, recording, captures management. Also serves the VR companion at `/vr` (see [`docs/vr-setup.md`](docs/vr-setup.md)). Full docs: [`pc/src/openmuscle/web/README.md`](pc/src/openmuscle/web/README.md) | | `openmuscle web --model M.pkl --hand IP` | Same UI plus live inference, with optional UDP forwarding of predictions to an OpenHand device | +| `openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem` | Same UI over HTTPS (required for the VR `/vr` page over LAN, since Quest Browser refuses WebXR hand-tracking on plain HTTP) | | `openmuscle record -o file.csv` | Record paired data to CSV | | `openmuscle train data.csv` | Train ML model (RandomForest) | | `openmuscle predict -m model.pkl` | Real-time inference (matplotlib) | @@ -91,9 +109,11 @@ mpremote cp embedded/devices/flexgrid_v1/config/defaults.json :/config/ ## Documentation - [Architecture Overview](docs/architecture.md) -- [Packet Protocol Spec](docs/protocol.md) +- [Packet Protocol Spec](docs/protocol.md) — includes the `quest_hand` device type - [Adding a New Device](docs/adding-a-device.md) - [CLI Usage Guide](docs/pc-cli.md) +- [VR Setup & Operation](docs/vr-setup.md) — mkcert + Quest cert install + per-session walkthrough +- [VR Testing Scenarios](docs/vr-testing-scenarios.md) — bring-up order, 2-minute smoke test, per-feature test runbook ## Adding a New Device @@ -114,6 +134,9 @@ See [docs/adding-a-device.md](docs/adding-a-device.md) for details. **Standalone firmware (promoted from this repo's `embedded/devices/`):** - [FlexGridV3-Firmware](https://github.com/Open-Muscle/FlexGridV3-Firmware) — current FlexGrid revision, with the sensor-scan techniques writeup +**AR / VR:** +- [OpenMuscle-AR](https://github.com/Open-Muscle/OpenMuscle-AR) — AR/VR companion. The current WebXR client lives here in `pc/src/openmuscle/web/static/vr/` (tight coupling to the FastAPI server), but the AR repo is the discoverability anchor and the future home for the planned native Quest APK / BLE-direct work. See its [ROADMAP](https://github.com/Open-Muscle/OpenMuscle-AR/blob/main/ROADMAP.md). + **Coordination / docs:** - [OpenMuscle-Hub](https://github.com/Open-Muscle/OpenMuscle-Hub) — central docs and roadmap diff --git a/docs/protocol.md b/docs/protocol.md index c500381..7231af3 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -56,10 +56,44 @@ All OpenMuscle devices communicate over **UDP port 3141** using **JSON-encoded U 4 piston sensor values. Joystick is optional. +### Quest hand tracking (`type: "quest_hand"`) + +Synthesized server-side from WebSocket frames sent by the WebXR client at `/vr` (browsers can't speak UDP). The payload represents one tracked hand sampled from `XRHand` each XRFrame: + +```json +{ + "values": [px,py,pz, rx,ry,rz,rw, ...] // flat, 7 floats per joint + "handedness": "left" | "right", + "joint_names": ["wrist", "thumb-metacarpal", ...], + "hands": { + "handedness": "left" | "right", + "joints": [ + {"name": "wrist", "pos": [x,y,z], "rot": [x,y,z,w], "radius": 0.02, "valid": true}, + ... + ] + } +} +``` + +- `values` follows the same convention as LASK5 — flat, in canonical joint × channel order — so the recorder and matcher pair `quest_hand` frames with FlexGrid frames identically to LASK5. The order is `[px, py, pz, rx, ry, rz, rw] × N joints`. +- **Fixed-length contract.** A well-behaved client MUST send all joints in canonical `joint_names` order every frame, so the `values` length and per-slot meaning are stable across frames. When an individual joint pose is momentarily unavailable, send a zero position + identity quaternion with `"valid": false` rather than omitting the joint — omitting it would shift every later joint into the wrong CSV column and silently misalign the labels. The server defends against violations by locking the label width at the first label packet and padding/truncating later rows to keep the CSV rectangular (it counts and logs any mismatch), but clients should not rely on that. +- `valid` (per joint, in `hands.joints`) marks whether the pose was actually tracked this frame. It is preserved in the JSONL sidecar so offline analysis can filter zero-filled joints; the flat `values` / CSV carry only the numbers. +- `joint_names` lists the joints in the same order they're flattened into `values`. The standard set is the W3C WebXR Hand Input spec (25 joints: wrist + 4 thumb + 5 each for index/middle/ring/pinky). +- `hands` is the structured per-joint form, kept in the JSONL sidecar for offline analysis. It's redundant with `values` + `joint_names` but easier to diff by eye. +- Empty payloads (the headset reports tracking-lost for the whole hand this frame) are dropped silently by the server — they'd otherwise produce zero-rows that mislead training. + +A per-capture `.labels.schema.json` sidecar is written on the first `quest_hand` packet of a recording. It maps `label_0..label_N` columns in the CSV back to `(joint, channel)`, so the wide label vector is self-describing. + +**Future-proofing note:** v1 captures one hand per recording (`handedness` is a single `"left"` or `"right"` string). A future `handedness: "both"` extension — payload carries both hands in `data.values` with the schema sidecar growing a parallel `joint_names_left` / `joint_names_right` (or a per-column `hand` field) — would be backward-compatible. Old consumers see a wider but still-flat values vector; new consumers can split it via the schema. Don't accidentally close that door in any future refactor of `_flatten_quest_joint` / `_write_labels_schema`. + ### Adding a New Device Type Define a new `type` string and document the `data` shape. The PC-side parser auto-discovers devices by their `type` field. +## Versioning policy + +Adding a new `type` string (e.g. `quest_hand`) is **non-breaking under v1.0** — existing parsers ignore unknown types, the schema envelope is unchanged, and devices that don't speak the new type are unaffected. Bump the `"v"` field to `"1.1"` (or beyond) only when changing the envelope itself (required-field set, semantics of `ts`/`id`, etc.). + ## Backward Compatibility The PC parser (`openmuscle.protocol.parser`) auto-detects three formats: diff --git a/docs/vr-setup.md b/docs/vr-setup.md new file mode 100644 index 0000000..677f084 --- /dev/null +++ b/docs/vr-setup.md @@ -0,0 +1,171 @@ +# OpenMuscle VR — setup & operation + +Wear FlexGrid + a Meta Quest 3S together. The headset's hand tracking becomes the ground-truth label source for training a muscle-signal → finger-pose model. The desktop OpenMuscle UI is unchanged; this guide just adds the VR companion. + +Companion code: [`pc/src/openmuscle/web/static/vr/`](../pc/src/openmuscle/web/static/vr/) (client) and [`pc/src/openmuscle/web/app.py`](../pc/src/openmuscle/web/app.py) (server endpoints). The Quest is integrated as a synthetic device with `device_type: "quest_hand"` — see [`protocol.md`](protocol.md) for the wire format. + +## Why HTTPS is non-optional + +Quest Browser refuses to grant WebXR hand-tracking outside a secure context. You have two ways to get one: + +| Mode | Pros | Cons | +|---|---|---| +| **HTTPS over LAN** via mkcert | untethered, real use case | one-time cert install on the headset | +| **`adb reverse` over USB** (localhost) | zero certs, fast for debugging | cable on the headset distorts natural arm motion | + +Day-to-day capture work should use HTTPS. The USB fallback is for proving the loop end-to-end before you've finished cert setup. + +## One-time setup + +### 1. mkcert on the PC + +```powershell +# Windows (choco). On macOS: `brew install mkcert`. On Linux: distro package. +choco install mkcert + +mkcert -install # installs the mkcert root CA into Windows +mkcert -cert-file vr-cert.pem ` + -key-file vr-key.pem ` + 192.168.1.42 localhost # <-- replace with your PC's LAN IP +``` + +The two PEM files land in your current directory. Keep them next to the captures dir, or anywhere the `openmuscle web` command can read them. + +### 2. Install the mkcert root CA on the Quest + +**Heads up:** this is the painful step. Meta's Horizon OS consumer shell **hides** the standard Android "Settings → Security → Install a certificate" path. The Settings panel you see when tapping the gear icon in VR is Meta's Horizon Settings, not AOSP Settings — different surface, no cert-install button. You have to launch the AOSP Settings activity via ADB. + +Procedure validated on Quest 3S, Horizon OS as of May 2026: + +```powershell +# (a) Push the root CA from your PC to the headset's Downloads folder +adb push "$env:LOCALAPPDATA\mkcert\rootCA.pem" /sdcard/Download/openmuscle-vr-rootCA.pem + +# (b) Launch the AOSP Security Dashboard activity directly via ADB. +# The $ MUST be backslash-escaped through PowerShell single-quotes so it +# survives BOTH the host shell AND the Android shell. Without the escape, +# $ gets eaten as a variable expansion and the activity name truncates to +# just '.Settings' which is not exported (Permission Denial). +adb shell 'am start -n com.android.settings/.Settings\$SecurityDashboardActivity' +``` + +On the Quest after running those: + +1. **The 2D Settings panel may not appear immediately in your view.** Horizon shell aggressively suppresses unfamiliar 2D activities. **Press the Meta button** on your right controller, look for "Settings" (or similar) in the universal-menu app switcher, and click to bring it forward. +2. In the AOSP Security dashboard: **Trusted credentials** → **Encryption** section → **Install a certificate** → **CA certificate**. +3. **"Your data won't be private"** warning → tap **Install anyway** (your own root CA, the warning is generic Android boilerplate). +4. **If no screen-lock PIN is set**, Quest refuses and prompts you to set one. Set a PIN, then re-do step 2. +5. **File picker gotcha:** defaults to "Recent" view which is **empty** on a fresh install. Tap the **hamburger menu (≡)** at the top of the picker → navigate to **Internal Storage → Download** → tap **openmuscle-vr-rootCA.pem**. +6. Success: "CA certificate installed" or similar. + +You only do this once per headset. The CA stays trusted across Horizon OS updates. + +**Fallback if you can't find the AOSP Settings panel in the app switcher:** open the headset's **Files** app, navigate to **Download/openmuscle-vr-rootCA.pem**, tap it. Some Horizon versions trigger the install dialog from the file-open intent directly. + +**Activities that work on Horizon OS** (useful for future Quest dev when standard menus are hidden): +- `com.android.settings/.Settings$SecurityDashboardActivity` — security menu (the one you just used) +- `com.android.settings/.Settings$TrustedCredentialsSettingsActivity` — view installed certs (for verifying yours is there) +- `com.android.settings/.security.CredentialStorage` — direct install activity, but Horizon often suppresses its panel +- `com.android.settings/.Settings$NetworkDashboardActivity` +- `com.android.settings/.Settings$DevelopmentSettingsDashboardActivity` + +**Activities that DON'T exist on Horizon** (don't try — Error type 3): +- `com.android.settings/.Settings$SecuritySettingsActivity` (older Android name, removed) +- `com.android.settings/.Settings$EncryptionAndCredentialActivity` (renamed) + +**Non-exported, can't launch via ADB** (Permission Denial from null uid): +- `com.android.settings/.Settings` (the bare main activity) + +### 3. Start the server with HTTPS + +```powershell +cd D:\path\to\OpenMuscle-Software\pc +pip install -e . # first time only + +openmuscle web --ssl-certfile vr-cert.pem ` + --ssl-keyfile vr-key.pem +``` + +You'll see: + +``` +OpenMuscle web UI: https://localhost:8000 +Listening for devices on UDP 3141 +TLS: cert=vr-cert.pem key=vr-key.pem +WebXR URL for the Quest: https://:8000/vr +``` + +## Per-session flow + +1. **Boot FlexGrid** — it joins your Wi-Fi and starts streaming UDP to port 3141. Confirm in the desktop UI's Devices panel. +2. **Open `/vr` in Quest Browser** at `https://:8000/vr`. (The desktop UI lives at `/`; the VR companion at `/vr`.) +3. The landing page runs three preflight checks. All three should be green checkmarks: + - HTTPS / secure context + - WebXR + immersive-vr supported + - Server reachable +4. **Pick the FlexGrid arm** in the dropdown (`right` is default). Only the joints of this hand get pushed to `/ws/quest`. The other hand stays free for the record button. +5. Tap **Enter VR**. Grant hand-tracking when Quest asks. +6. A floating panel ~70cm in front shows the live FlexGrid heatmap. Header strip above it shows per-device Hz when idle, or `REC · N rows · match X%` when recording. +7. Hold your captured hand up; you'll see small blue spheres on each tracked joint. +8. **Start recording**: + - **Off-hand button**: tap the gray sphere below the heatmap with your other hand's index finger. It turns red. + - **Pinch gesture**: pinch index + thumb on the captured hand and hold for ~1 second. A yellow ring on the index tip fills during the hold; release triggers the toggle. +9. Do your gesture set. For the first FlexGrid V3 training run we focus on **index / middle / ring / pinky finger curls** — the thumb is driven by intrinsic hand muscles the FlexGrid can't see, so it's excluded. +10. **Stop recording** the same way. +11. Take the headset off. The capture is in `data/raw/merged/`: + + ``` + capture_.csv # paired (sensor + 182 joint floats) + capture_.sensor.jsonl # raw FlexGrid packets + capture_.label.jsonl # raw Quest packets + capture_.labels.schema.json # column -> (joint, channel) map + capture_.meta.json # auto.label_source="quest_hand" + your tags + ``` + +12. **Train**: in the desktop UI's Captures panel, tick your new capture(s) and hit **Train selected**. Or from CLI: `openmuscle train data/raw/merged/capture_.csv`. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| "Enter VR" button stays disabled | Page opened in the in-headset Library panel, not Quest Browser | Open Quest Browser app explicitly; navigate to the URL | +| "INSECURE (http:)" in the HTTPS checkmark | Reached the page over plain HTTP | Either switch to `https://` (LAN + mkcert) or use `adb reverse` + `http://localhost:8000/vr` | +| Cert install on Quest fails silently | Quest doesn't have a screen-lock PIN | Settings → Security → Lock screen → set a PIN, then re-try CA install | +| Heatmap stays "waiting for FlexGrid…" inside VR | FlexGrid not streaming, or wrong UDP port | Open the desktop UI; verify the device is in the Devices panel and packets > 0 | +| `match_rate` is very low (< 30%) in recording header | Quest WS connection dropping, or sensor and label streams have wildly different rates | Check WS reconnect logs in the desktop UI's Log panel; try increasing `window_ms` via the recording start body if needed (default for Quest is 175 ms) | +| Joints visible but no `quest-` device in the Devices panel | `/ws/quest` failed to connect (e.g. CORS, cert mismatch) | Check the Quest Browser dev-tools (chrome://inspect on a connected PC) for WS errors | + +## Architecture, in one diagram + +``` + ┌──────────────────────┐ + │ Meta Quest 3S │ + │ │ + ┌───────────┐ │ Quest Browser /vr │ + │ FlexGrid │ │ │ │ + │ (ESP32- │ UDP │ │ WebXR XRHand │ + │ S3 WiFi) │ ─────────→ │ │ sample joints │ + └───────────┘ :3141 │ ▼ │ + ▲ │ WS /ws/quest │ + │ │ │ │ + │ │ │ WS /ws/live │ + │ │ │ ◀──── heatmap │ + │ │ │ │ + │ └────┼─────────────────┘ + │ │ + │ ▼ + │ ┌──────────────────────────┐ + │ │ PC: openmuscle web │ + │ │ │ + │ UDP │ UDPListener ─→ ┐ │ + └────────────┤ │ │ + │ ingest_quest ─→ AppState ─→ paired CSV + │ (synthesizes a matcher + JSONL + + │ quest_hand recorder labels.schema.json + │ packet) + meta.json + │ │ + │ /ws/live ◀── snapshot ───────────┘ + └──────────────────────────┘ +``` + +The whole integration trick is that `ingest_quest_packet` synthesizes an `OpenMusclePacket` of type `quest_hand` in-process, then hands it to `_handle_packet` — the **same** function the UDP listener feeds into. From the recorder's view the Quest is just another device. Zero special-casing downstream. diff --git a/docs/vr-testing-scenarios.md b/docs/vr-testing-scenarios.md new file mode 100644 index 0000000..81d579c --- /dev/null +++ b/docs/vr-testing-scenarios.md @@ -0,0 +1,173 @@ +# VR Testing Scenarios + Smoke-Test Checklist + +A practical runbook for validating the OpenMuscle VR app against the hardware. Use this when you sit down to test: it gives you a fixed bring-up order, a 2-minute smoke test to confirm the basics, and per-feature scenarios with expected results so you can validate each piece fast instead of rediscovering how it all works. + +Companion docs: +- [`vr-setup.md`](vr-setup.md) -- one-time setup (mkcert, Quest cert install, launchers) +- [OpenMuscle-AR wiki](https://github.com/Open-Muscle/OpenMuscle-AR/wiki) -- troubleshooting + community successes log +- [`pc/src/openmuscle/web/README.md`](../pc/src/openmuscle/web/README.md) -- architecture of the web/VR surface + +--- + +## Bring-up order (do this every session, in order) + +Order matters. Each step depends on the previous one being live. + +1. **Power the FlexGrid.** Wait for its OLED to show it joined Wi-Fi. It streams UDP to the PC on port 3141. +2. **Start the server on the PC.** Pick the path that matches how you'll test: + - **USB tethered** (fast iteration, no certs): `cd pc` then `start-vr.bat` (VR) or `start-vr.bat right ar` (AR). It auto-runs `adb reverse` and opens Quest Browser for you. + - **Cordless HTTPS** (walk around, real field capture): `cd pc` then `start-vr-https.bat`. It prints the headset URL. Requires the one-time mkcert + Quest cert install. +3. **Confirm the device shows up.** On the PC, open the desktop UI (`http://localhost:8000/` or `https://:8000/`). The Devices panel should list `flexgrid-v3-*` with a non-zero Hz and a live heatmap. +4. **Put the headset on. Set both controllers down** on a flat surface. Quest auto-switches to hand tracking within 2 to 5 seconds. (It stays in controller mode as long as it can see them being held.) +5. **Open `/vr` in Quest Browser** (the USB launcher does this for you; for HTTPS, type the printed URL). Confirm the three preflight checkmarks are green, then tap Enter VR / Start AR. + +If any step fails, the [troubleshooting wiki page](https://github.com/Open-Muscle/OpenMuscle-AR/wiki/Troubleshooting) lists the common causes per symptom. + +--- + +## 2-minute smoke test (no FlexGrid required) + +The fastest "is anything obviously broken" check. Only needs the headset + PC + server. Do this first whenever you pick the project back up after code changes. + +| # | Action | Expected | +|---|---|---| +| 1 | Open `/vr` (or `/vr?mode=ar`), look at the preflight page | Three green checkmarks: secure context, WebXR supported, server reachable | +| 2 | Tap Enter VR / Start AR | Scene loads. VR = dark background; AR = passthrough (you see your room) | +| 3 | Look forward | Heatmap panel (says "waiting for FlexGrid"), 3x2 menu below it, status strip below that | +| 4 | Hold up your captured arm (the FlexGrid arm) | Blue joint spheres on that hand | +| 5 | Hold up your other hand | Green joint spheres + a blue ray pointer from the index | +| 6 | Point the ray at a menu button | Ray turns amber, button highlights | +| 7 | Pinch (off-hand index + thumb) while hovering RECENTER | Brief white flash on the button; panels re-anchor in front of you | +| 8 | Point at a panel's drag handle (small cube, top-left corner) and pinch-hold | Handle turns green; panel follows your hand. Release: panel drops + faces you | +| 9 | Tap REC | Yellow SYNC slate flashes ~2.5s; menu collapses to one big red STOP button; header shows a live ms clock + filename + a capture-quality dot (gray while warming up) | +| 10 | Tap STOP | Menu returns to the full 3x2 grid; status strip shows "saved: ..." | + +If all 10 pass, the core interaction loop is healthy. If FlexGrid is connected, the heatmap in step 3 animates and the REC in step 9 actually pairs sensor frames. + +### Reading capture quality (live, while recording) + +The recording header is also a live data-quality gauge, so you can tell mid-capture whether the data is good without taking the headset off: + +- **Quality dot color** (tracks the sensor-to-label match rate): + - **gray** = warming up (fewer than ~10 sensor frames seen yet) + - **green** = match rate >= 70% (good) + - **amber** = 40 to 70% (marginal -- check that both FlexGrid and Quest are streaming) + - **red** = below 40% (poor pairing -- something is off; see Troubleshooting) +- **`NN%`** in the header is the live match rate. +- **`JOINTS DROPPING`** appears (and the dot goes amber) when your hand is partially out of the cameras' view and the headset is sending incomplete joint frames. Those frames get zero-filled joint columns -- keep your capture hand more fully in view if you see this a lot. The final count is also reported in the stop result (`label_width_mismatch`). + +--- + +## Full test scenarios + +Each scenario is independent. Run the ones relevant to what you changed. + +### Scenario A: Gesture-training capture (VR mode) + +The deliberate-gesture workflow for building training data. + +1. Launch VR mode: `start-vr.bat right` (or left). Enter VR. +2. Tap **SESSION** (turns red). All captures now group under one session. +3. Tap **REC**. Curl your index finger open/close ~10 to 15 times. Tap **STOP**. +4. Repeat step 3 for middle, ring, pinky. (Thumb is intentionally excluded -- FlexGrid can't see it.) +5. Tap **SESSION** again to end it. +6. Tap **TRAIN**. Wait ~10 to 30s. + +**Expected:** +- Each REC shows a rising row count + match rate in the header while recording. +- Status strip after each STOP: `saved: capture_.csv (N rows, match X%)`. Match rate should be well above 0 if both FlexGrid and Quest are streaming. +- TRAIN status: `trained: R2= ... model loaded ... predict ON`. +- On the PC, `pc/data/raw/merged/` has the capture CSVs + `.labels.schema.json` + `.meta.json` sidecars, and `pc/data/raw/sessions/` has the session JSON. + +**Watch for:** match rate near 0 (Quest WebSocket not connected, or hands out of camera FOV during capture). Very low R2 is normal for a first small dataset; record more and retrain. + +### Scenario B: Field capture (AR passthrough mode) + +The natural-activity workflow. You see your real workspace and do real tasks. + +1. Launch AR mode: `start-vr-https.bat`, open `https://:8000/vr?mode=ar` (cordless), or `start-vr.bat right ar` (tethered for a quick check). Start AR. +2. Confirm passthrough: you should see your real room behind the panels. (If it is black, the AR session did not grant passthrough -- check the browser console for the blend-mode log line.) +3. **Start the Quest's built-in screen recorder** (press and hold the Meta button, or use the camera shortcut) so the video is captured for later labeling. +4. Drag the panels out of your central view using the handles, so they sit in your periphery and do not block your work. +5. Tap **REC**. Do a real task with your hands in view (type, stir, assemble something). Tap **STOP** when done. +6. Stop the screen recorder. + +**Expected:** +- SYNC slate flashes at REC with `capture_.csv` + a Unix-ms timestamp. This is your video-to-CSV pairing anchor. +- Header strip shows a live `HH:MM:SS.mmm` clock + filename throughout the recording, visible in the screen recording. +- Captured-arm joints stream to the CSV whenever the hand is in the headset cameras' FOV; frames where the hand is out of view are dropped (gaps in the data, by design). + +**Watch for:** panels snapping back to center after you moved them -- if that happens on reload, the localStorage layout persistence regressed (see Scenario E). + +### Scenario C: Pause / boundary redraw recovery + +This is the v1.11 fix. It used to glitch the app. + +1. Enter either mode. Tap **REC** to start a recording. +2. Trigger a pause: press the **Meta button** (opens the universal menu) OR walk past your Guardian boundary so Quest prompts a redraw. +3. Observe the status strip, then close the menu / finish the redraw. + +**Expected:** +- During pause: status strip reads `session paused (visible-blurred) -- boundary redraw or system UI`. No joint frames are sent during this time (the recorder timeline stays clean). +- On resume: status reads `session resumed after Xs -- re-anchoring UI`, and the panels pop back in front of your current head position. +- If you were mid-drag when the pause hit, the panel is released cleanly (not stuck following a frozen controller). +- Tap **STOP**: the recording closes normally with all the valid pre/post-pause frames. + +### Scenario D: Collapse-to-STOP while recording + +1. Enter either mode. Confirm the full 3x2 menu is visible. +2. Tap **REC**. + +**Expected:** the menu collapses to a single large red STOP button at the same location. The status strip stays visible. Pointing + pinching STOP ends the recording and the full menu returns. + +**Why it matters:** during real field capture you do not want six action buttons cluttering your view -- just an obvious way to stop. + +### Scenario E: Panel layout persistence + +This is the v1.12 feature. + +1. Enter a mode. Drag the heatmap cluster and the menu to custom positions. +2. Exit VR (or just reload the Quest Browser tab) and re-enter the same mode. + +**Expected:** the panels come back where you left them. VR and AR remember their layouts independently (per-mode localStorage key). + +3. Now tap **RECENTER**. + +**Expected:** panels reset to defaults AND the saved layout is cleared, so a subsequent reload also starts from defaults until you drag again. + +### Scenario F: Predict + ghost hand + +1. Complete Scenario A at least once so a model is trained and loaded. +2. Confirm **PREDICT** is red (TRAIN auto-enables it; otherwise tap it). +3. Hold your captured hand up in view and curl fingers. + +**Expected:** +- The REAL-vs-PRED bar panel appears (four fingers, green = real curl, amber = predicted). +- An amber ghost hand overlays your real hand, anchored to your real wrist position + orientation, showing what the model predicts. +- As the model improves with more training data, the ghost should track your real hand more closely. + +--- + +## What "good" looks like (acceptance bar) + +Before calling the app "ready to demo or hand to a tester," these should all be true: + +- [ ] 2-minute smoke test passes end to end +- [ ] Scenario A produces a trainable CSV with match rate > 50% +- [ ] Scenario B passthrough is visible and the sync slate + header clock are legible in the screen recording +- [ ] Scenario C recovers gracefully from a Meta-button pause with no app glitch +- [ ] Scenario D collapses and restores the menu correctly +- [ ] Scenario E remembers panel layout across a reload and RECENTER clears it +- [ ] Scenario F shows the ghost hand tracking once a model is trained + +--- + +## Reporting a problem + +If a scenario fails, capture: +- Which scenario + step +- Mode (vr / ar) and transport (USB / HTTPS) +- Quest model + Horizon OS version +- Browser console output (via `chrome://inspect/#devices` with the Quest USB-connected) and the `openmuscle web` server console + +File it as an issue on [OpenMuscle-AR](https://github.com/Open-Muscle/OpenMuscle-AR/issues) (AR/VR-specific) or [OpenMuscle-Software](https://github.com/Open-Muscle/OpenMuscle-Software/issues) (server/pipeline). Add a [Troubleshooting wiki](https://github.com/Open-Muscle/OpenMuscle-AR/wiki/Troubleshooting) entry once you resolve it so the next person benefits. diff --git a/embedded/devices/openhand_v2/boot.py b/embedded/devices/openhand_v2/boot.py index ee3f002..6aeb1db 100644 --- a/embedded/devices/openhand_v2/boot.py +++ b/embedded/devices/openhand_v2/boot.py @@ -348,22 +348,38 @@ def espnow_listen(): pass frint('ESP-NOW ready') frint('SEL=exit') - while True: - # Non-blocking recv with 100ms timeout so buttons stay responsive - msg = e.recv(100) - if msg and msg[1]: - print(f'msg[1]:{msg[1]}') - result = parse_packet(msg[1]) - if result: - device_id, values = result - apply_packet(device_id, values) - if select_btn.value() == 0: - time.sleep(0.2) # debounce - break - e.active(False) + try: + while True: + # Non-blocking recv with 100ms timeout so buttons stay responsive + msg = e.recv(100) + if msg and msg[1]: + print('msg[1]:' + str(msg[1])) + result = parse_packet(msg[1]) + if result: + device_id, values = result + apply_packet(device_id, values) + if select_btn.value() == 0: + time.sleep(0.2) # debounce + break + except KeyboardInterrupt: + frint('ESPNow -> REPL') + try: release_all() + except Exception: pass + raise + finally: + try: e.active(False) + except Exception: pass frint('ESP-NOW stopped') +# Idle-sleep threshold: after this many seconds with no incoming UDP packet +# we release all servos. They stop humming/holding torque, and the OLED +# shows "Sleeping..." so the operator knows the hand is intentionally +# limp (not crashed). The next packet wakes it up — set_finger() implicitly +# re-energizes the servo, so wake is just "go back to applying packets". +UDP_IDLE_SLEEP_S = 30 + + def udp_listen(): # Reuse the boot-time STA connection. If for some reason we don't have # one yet (auto_mode=menu user navigated here manually before boot @@ -382,20 +398,62 @@ def udp_listen(): s.setblocking(False) frint('UDP :' + str(UDP_PORT)) frint('SEL=exit') - while True: - try: - data, addr = s.recvfrom(256) - if data: - result = parse_packet(data) - if result: - device_id, values = result - apply_packet(device_id, values) - except OSError: - pass # no data available (non-blocking) - if select_btn.value() == 0: - time.sleep(0.2) # debounce - break - s.close() + + last_packet_t = time.time() # grace period: 30s before first sleep + is_asleep = False + + # KeyboardInterrupt-aware loop: a Ctrl-C from the REPL (mpremote, Thonny, + # serial monitor) now exits this loop cleanly so the operator can land + # in the REPL and edit settings / firmware without having to power-cycle. + try: + while True: + got_packet = False + try: + data, addr = s.recvfrom(256) + if data: + got_packet = True + result = parse_packet(data) + if result: + device_id, values = result + apply_packet(device_id, values) + except OSError: + pass # no data available (non-blocking) + + if got_packet: + last_packet_t = time.time() + if is_asleep: + # Wake transition: brief OLED note, then resume listening. + # apply_packet() above already drove this packet's values, + # which implicitly re-energized the servos -- no extra work. + is_asleep = False + frint('Awake') + else: + # No packet this iteration -- check idle timeout. + if not is_asleep and (time.time() - last_packet_t) > UDP_IDLE_SLEEP_S: + release_all() # stop all 16 channels (servos go limp) + is_asleep = True + frint('Sleeping') + + if select_btn.value() == 0: + time.sleep(0.2) # debounce + break + + # Small yield so the non-blocking recv loop doesn't peg the CPU + # (also lets the REPL Ctrl-C have a chance to interrupt). + if is_asleep: + time.sleep(0.05) # asleep: be lazy, 20 Hz wake-check + else: + time.sleep(0.002) # awake: tight loop, ~500 Hz + except KeyboardInterrupt: + frint('UDP -> REPL') + try: release_all() + except Exception: pass + # Re-raise so the outer boot-sequence handler can also drop cleanly. + raise + finally: + try: s.close() + except Exception: pass + # Keep STA connected on exit so ESPNow still has its channel locked. frint('UDP stopped') @@ -485,20 +543,26 @@ def run_menu(): selected = 0 n_items = len(MENU_ITEMS) draw_menu(selected) - while True: - if up_btn.value() == 0: - selected = (selected - 1) % n_items - draw_menu(selected) - time.sleep(0.25) - if down_btn.value() == 0: - selected = (selected + 1) % n_items - draw_menu(selected) - time.sleep(0.25) - if start_btn.value() == 0: - time.sleep(0.2) # debounce - MENU_ACTIONS[selected]() - draw_menu(selected) - time.sleep(0.05) + try: + while True: + if up_btn.value() == 0: + selected = (selected - 1) % n_items + draw_menu(selected) + time.sleep(0.25) + if down_btn.value() == 0: + selected = (selected + 1) % n_items + draw_menu(selected) + time.sleep(0.25) + if start_btn.value() == 0: + time.sleep(0.2) # debounce + MENU_ACTIONS[selected]() + draw_menu(selected) + time.sleep(0.05) + except KeyboardInterrupt: + frint('Menu -> REPL') + try: release_all() + except Exception: pass + raise # ============================================================================= @@ -518,19 +582,33 @@ def run_menu(): # Hold Select at boot to force the menu, regardless of auto_mode -- escape # hatch in case auto-mode is set to something that crashes or hangs. -if select_btn.value() == 0: - frint('Auto skipped') - time.sleep(0.5) -else: - mode = settings.get('auto_mode', 'menu') - if mode == 'espnow': - frint('Auto: ESP-NOW') - time.sleep(0.4) - espnow_listen() # returns when Select pressed - elif mode == 'udp': - frint('Auto: UDP') - time.sleep(0.4) - udp_listen() # returns when Select pressed - -blink(2) -run_menu() +# +# The whole runtime is wrapped in try/except KeyboardInterrupt so a Ctrl-C +# from the REPL (mpremote, Thonny, serial monitor) drops back to an +# interactive prompt cleanly. Without this, the device locks out the host +# while a listen loop is running and we can't edit firmware/settings +# without a physical power-cycle. Servos are released on the way out so +# fingers don't hold torque while you're hacking. +try: + if select_btn.value() == 0: + frint('Auto skipped') + time.sleep(0.5) + else: + mode = settings.get('auto_mode', 'menu') + if mode == 'espnow': + frint('Auto: ESP-NOW') + time.sleep(0.4) + espnow_listen() # returns when Select pressed + elif mode == 'udp': + frint('Auto: UDP') + time.sleep(0.4) + udp_listen() # returns when Select pressed + + blink(2) + run_menu() +except KeyboardInterrupt: + try: release_all() + except Exception: pass + frint('REPL') + print('\nCtrl-C received -- dropped to REPL.') + print('To resume: run_menu() or exec(open("boot.py").read())') diff --git a/pc/src/openmuscle/cli.py b/pc/src/openmuscle/cli.py index 7967c6d..597c62d 100644 --- a/pc/src/openmuscle/cli.py +++ b/pc/src/openmuscle/cli.py @@ -43,11 +43,25 @@ def receive(port, save_dir): "10.0.0.55:3145). When set together with --model, inference " "outputs are forwarded over UDP as 'PC,a1,a2,a3,a4,a5'. " "Default port if omitted: 3145.") -def web(host, port, udp_port, captures_dir, model_path, hand): +@click.option("--ssl-certfile", default=None, type=click.Path(exists=True, dir_okay=False), + help="Path to TLS cert (PEM). Required for WebXR over LAN -- Quest " + "Browser refuses hand-tracking on plain HTTP. Generate with " + "`mkcert ` and install the mkcert root CA " + "on the headset (Settings -> Security -> Install a certificate).") +@click.option("--ssl-keyfile", default=None, type=click.Path(exists=True, dir_okay=False), + help="Path to TLS private key (PEM). Pair with --ssl-certfile.") +def web(host, port, udp_port, captures_dir, model_path, hand, ssl_certfile, ssl_keyfile): """Launch the browser-based UI with live heatmap, recording, and captures.""" from openmuscle.web.app import serve - click.echo(f"OpenMuscle web UI: http://{host if host != '0.0.0.0' else 'localhost'}:{port}") + # mkcert produces a cert AND key; we need both or neither. + if bool(ssl_certfile) != bool(ssl_keyfile): + raise click.BadParameter("--ssl-certfile and --ssl-keyfile must be used together") + scheme = "https" if ssl_certfile else "http" + click.echo(f"OpenMuscle web UI: {scheme}://{host if host != '0.0.0.0' else 'localhost'}:{port}") click.echo(f"Listening for devices on UDP {udp_port}") + if ssl_certfile: + click.echo(f"TLS: cert={ssl_certfile} key={ssl_keyfile}") + click.echo(f"WebXR URL for the Quest: {scheme}://:{port}/vr") if model_path: click.echo(f"Inference model: {model_path}") hand_target = None @@ -62,7 +76,8 @@ def web(host, port, udp_port, captures_dir, model_path, hand): hand_target = (hand, 3145) click.echo(f"Forwarding inference to robot hand at {hand_target[0]}:{hand_target[1]}") serve(host=host, port=port, udp_port=udp_port, captures_dir=captures_dir, - model_path=model_path, hand_target=hand_target) + model_path=model_path, hand_target=hand_target, + ssl_certfile=ssl_certfile, ssl_keyfile=ssl_keyfile) @main.command() diff --git a/pc/src/openmuscle/data/converter.py b/pc/src/openmuscle/data/converter.py index f159d2b..c31c39c 100644 --- a/pc/src/openmuscle/data/converter.py +++ b/pc/src/openmuscle/data/converter.py @@ -1,16 +1,28 @@ -"""Convert legacy capture formats to standard CSV.""" +"""Convert legacy capture formats to standard CSV with packet matching.""" import ast import csv import os +from collections import deque from pathlib import Path +# Legacy device IDs for SensorBand (3 banks × 4 sensors = 12 total) +SENSOR_IDS = ['OM-SB-V1-C.0', 'OM-SB-V1-C.1', 'OM-SB-V1-C.2'] +LABEL_ID = 'OM-LASK5' + + def convert_legacy_capture(input_path: str, output_path: str) -> int: - """Convert a legacy capture_*.txt file to standard CSV format. + """Convert a legacy capture_*.txt file to matched CSV format. + + Matches sensor packets from 3 SensorBand banks with LASK5 label packets + by temporal proximity, producing rows with 12 sensor values + 4 labels. Legacy format: one Python dict repr per line, e.g.: - {'id': 'OM-LASK5', 'ticks': 164587, 'time': (2000, 1, 1, ...), 'data': [-30, -35, -30, -37]} + {'id': 'OM-SB-V1-C.0', 'ticks': 164587, 'data': [2686.1, 2926.1, 2519.7, 2653.1], ...} + + Output CSV columns: + Sensor_0..Sensor_11, Sensor_Timestamp, Label_0..Label_3, Label_Timestamp Args: input_path: path to legacy .txt capture file @@ -20,11 +32,19 @@ def convert_legacy_capture(input_path: str, output_path: str) -> int: Number of rows written """ Path(output_path).parent.mkdir(parents=True, exist_ok=True) + + buffers = {sid: deque(maxlen=5) for sid in SENSOR_IDS} + buffers[LABEL_ID] = deque(maxlen=5) + + header = ([f"Sensor_{i}" for i in range(12)] + ['Sensor_Timestamp'] + + [f"Label_{i}" for i in range(4)] + ['Label_Timestamp']) + rows_written = 0 + last_record = None with open(input_path, "r") as f_in, open(output_path, "w", newline="") as f_out: writer = csv.writer(f_out) - header_written = False + writer.writerow(header) for line in f_in: line = line.strip() @@ -38,17 +58,71 @@ def convert_legacy_capture(input_path: str, output_path: str) -> int: if not isinstance(pkt, dict) or "data" not in pkt: continue - device_id = pkt.get("id", "unknown") - ticks = pkt.get("ticks", 0) - data = pkt["data"] + pkt_id = pkt.get("id") + if pkt_id not in buffers: + continue + + buffers[pkt_id].append(pkt) + + # Try to match: need at least one packet from each sensor bank + labels + if not all(buffers[sid] for sid in SENSOR_IDS): + continue + if not buffers[LABEL_ID]: + continue + + # Combine latest from each sensor bank + sensor_values = [] + sensor_times = [] + for sid in SENSOR_IDS: + latest = buffers[sid][-1] + sensor_values.extend(latest.get('data', [])) + sensor_times.append(latest.get('rec_time', 0)) + + label_pkt = buffers[LABEL_ID][-1] + labels = label_pkt.get('data', []) + label_time = label_pkt.get('rec_time', 0) + sensor_time = min(sensor_times) - if not header_written: - n = len(data) - header = ["device_id", "ticks"] + [f"value_{i}" for i in range(n)] - writer.writerow(header) - header_written = True + record = sensor_values + [sensor_time] + labels + [label_time] - writer.writerow([device_id, ticks] + data) - rows_written += 1 + # Skip duplicate records (same data as last write) + if record == last_record: + continue + + if len(record) == 18: + writer.writerow(record) + rows_written += 1 + last_record = record return rows_written + + +def combine_csvs(csv_paths: list[str], output_path: str) -> int: + """Concatenate multiple CSVs (same schema) into one file. + + Args: + csv_paths: list of CSV file paths to combine + output_path: path for the combined output CSV + + Returns: + Total number of data rows written + """ + Path(output_path).parent.mkdir(parents=True, exist_ok=True) + total = 0 + header_written = False + + with open(output_path, "w", newline="") as f_out: + writer = csv.writer(f_out) + + for csv_path in csv_paths: + with open(csv_path, "r") as f_in: + reader = csv.reader(f_in) + header = next(reader) + if not header_written: + writer.writerow(header) + header_written = True + for row in reader: + writer.writerow(row) + total += 1 + + return total diff --git a/pc/src/openmuscle/data/dataset.py b/pc/src/openmuscle/data/dataset.py index 2b48fe9..b23d951 100644 --- a/pc/src/openmuscle/data/dataset.py +++ b/pc/src/openmuscle/data/dataset.py @@ -19,8 +19,12 @@ def detect_columns(df: pd.DataFrame) -> tuple[list[str], list[str]]: Returns: (sensor_columns, label_columns) """ - sensor_cols = [c for c in df.columns if c.startswith("R") or c.startswith("Sensor_")] - label_cols = [c for c in df.columns if c.startswith("label_") or c.startswith("Label_")] + sensor_cols = [c for c in df.columns + if (c.startswith("R") or c.startswith("Sensor_")) + and "Timestamp" not in c] + label_cols = [c for c in df.columns + if (c.startswith("label_") or c.startswith("Label_")) + and "Timestamp" not in c] if not sensor_cols: raise ValueError("No sensor columns found (expected R*C* or Sensor_* prefix)") diff --git a/pc/src/openmuscle/data/storage.py b/pc/src/openmuscle/data/storage.py index e8938ca..d488a74 100644 --- a/pc/src/openmuscle/data/storage.py +++ b/pc/src/openmuscle/data/storage.py @@ -4,17 +4,25 @@ import os import time from pathlib import Path +from typing import Optional class CaptureWriter: """Writes matched FlexGrid + label data to CSV files. The CSV format uses columns: - timestamp, R0C0, R0C1, ..., R3C15, label_0, label_1, label_2, label_3 + timestamp, R0C0, R0C1, ..., R3C15, label_0, label_1, ..., label_N + + Header is written lazily on the first `write_row` call so the number + of label columns can be inferred from that row's label_values length. + Callers that know the count up front can pass it via `label_count`; + callers that don't (e.g. Quest hand tracking, whose joint vector is + not known until the first label packet arrives) pass `label_count=None` + and the writer derives it from the first row. """ def __init__(self, output_path: str = None, matrix_rows: int = 4, - matrix_cols: int = 16, label_count: int = 4): + matrix_cols: int = 16, label_count: Optional[int] = 4): if output_path is None: os.makedirs("data/raw/merged", exist_ok=True) output_path = f"data/raw/merged/capture_{int(time.time())}.csv" @@ -22,15 +30,28 @@ def __init__(self, output_path: str = None, matrix_rows: int = 4, self.path = Path(output_path) self.path.parent.mkdir(parents=True, exist_ok=True) - sensor_cols = [f"R{r}C{c}" for r in range(matrix_rows) for c in range(matrix_cols)] - label_cols = [f"label_{i}" for i in range(label_count)] + self._matrix_rows = matrix_rows + self._matrix_cols = matrix_cols + self._label_count: Optional[int] = label_count # None = infer on first row self._file = open(self.path, "w", newline="") self._writer = csv.writer(self._file) - self._writer.writerow(["timestamp"] + sensor_cols + label_cols) + self._header_written = False self._count = 0 + def _write_header(self, label_count: int) -> None: + sensor_cols = [f"R{r}C{c}" for r in range(self._matrix_rows) + for c in range(self._matrix_cols)] + label_cols = [f"label_{i}" for i in range(label_count)] + self._writer.writerow(["timestamp"] + sensor_cols + label_cols) + self._label_count = label_count + self._header_written = True + def write_row(self, timestamp: float, sensor_values: list, label_values: list): + if not self._header_written: + count = (self._label_count if self._label_count is not None + else len(label_values)) + self._write_header(count) self._writer.writerow([timestamp] + sensor_values + label_values) self._count += 1 @@ -38,7 +59,19 @@ def write_row(self, timestamp: float, sensor_values: list, label_values: list): def row_count(self) -> int: return self._count + @property + def label_count(self) -> Optional[int]: + """How many label columns the CSV has. None until the first row + is written (or close() is called on an empty capture, which + falls back to the constructor hint or 0).""" + return self._label_count + def close(self): + if not self._header_written: + # No row was ever paired -- still emit a header so consumers + # don't trip on a zero-byte file. Use the constructor hint + # if given, else 0 label columns (sensor-only capture). + self._write_header(self._label_count if self._label_count is not None else 0) self._file.close() def __enter__(self): diff --git a/pc/src/openmuscle/web/README.md b/pc/src/openmuscle/web/README.md index 5b76a7f..dadce5c 100644 --- a/pc/src/openmuscle/web/README.md +++ b/pc/src/openmuscle/web/README.md @@ -207,12 +207,55 @@ The WS snapshot already exposes `matrix`, `values`, and `joystick` for any devic Today there's exactly one `UDPListener` per `AppState`, on `--udp-port`. If you need to listen on more (e.g. legacy LASK5 on 3145 while keeping FlexGrid on 3141), the cleanest extension is to instantiate multiple `UDPListener`s in `AppState.__init__` and merge their queues in `run_broadcaster`. As of v0.2.0 we've standardized everything on 3141 instead, so this hasn't been needed yet. +## VR companion (`/vr`) + +The same FastAPI process also serves a WebXR client that turns a Meta Quest 3 into a labeling rig and live demo for the muscle→finger model. Operator guide: [`docs/vr-setup.md`](../../../../../docs/vr-setup.md). Wire format for the new device type: [`docs/protocol.md`](../../../../../docs/protocol.md#quest-hand-tracking-type-quest_hand). + +### What's added vs the bare web UI + +| Surface | Purpose | +|---|---| +| `GET /vr` | Serves `static/vr/index.html` — landing page + WebXR client (Three.js + XRHand). | +| `WS /ws/quest` | Inbound — accepts XRHand joint frames from the headset. Each frame is synthesized in-process into an `OpenMusclePacket(device_type="quest_hand")` and fed through the same `_handle_packet` UDP devices use. From the recorder/matcher/snapshot's view the Quest is just another device. | +| `start-vr.bat` (in `pc/`) | One-click launcher: ADB sanity check → start server → poll until up → `adb reverse` → open Quest Browser to `/vr`. Optional arm arg (`right`\|`left`). | +| `--ssl-certfile` / `--ssl-keyfile` | Serve HTTPS so the headset can hit `/vr` over LAN. WebXR refuses hand-tracking on plain HTTP (localhost is the only exception, via `adb reverse`). | + +### Why a WebSocket inbound (when everything else is UDP) + +Browsers can't speak UDP. WebXR therefore can't run as a UDP-emitting device. The chosen workaround is one synthesizer method (`AppState.ingest_quest_packet`) that builds the same `OpenMusclePacket` shape the UDP listener emits, so the rest of the pipeline — `_handle_packet` → `DeviceInfo.update` → `_record_packet` → `TemporalMatcher` → `CaptureWriter` — is unchanged. The integration cost was one new endpoint plus one synthesizer; everything downstream rides for free. + +### `quest_hand` recording specifics + +- **Match window default** is 175 ms for `quest_hand` label sources vs 100 ms for `lask5`, set per-device-type in `AppState.DEFAULT_WINDOW_MS_BY_TYPE`. Quest WebXR has higher end-to-end latency than LASK5's ESP-NOW path, so a tighter window dropped too many sensor frames as unpaired. +- **`label_count` is None / lazy-inferred** for `quest_hand` — `CaptureWriter` defers the CSV header write until the first label packet so the column count is derived from `len(values)`. Quest 3 sends 25 joints × 7 floats = 175 floats per frame; the hardcoded LASK5 `label_count=4` doesn't fit. +- **Per-capture `.labels.schema.json` sidecar** lists joint names + channel order so the wide CSV is self-describing. Only emitted for label sources whose meaning isn't obvious from `device_type` alone (today: `quest_hand`). +- **`meta.json` `auto.label_source`** is tagged `"quest_hand"` so the Captures panel filter and any downstream training pipeline can cleanly separate Quest-labeled from LASK5-labeled datasets. +- **Auto-pick** in `start_recording` walks `AUTO_LABEL_TYPE_PREFERENCE = ("quest_hand", "lask5")` — Quest wins if both are connected, since it's the richer label source. + +### WebXR client structure (`static/vr/`) + +| File | Role | +|---|---| +| `index.html` | Landing page with a 3-checkmark preflight (HTTPS, WebXR support, server reachable), arm selector, VRButton mount. Script tag uses `?v=N` cache-buster (see gotcha below). | +| `app.js` | Scene + XR session lifecycle, per-frame joint capture → `/ws/quest`, real-time hand visualizer (blue captured-arm + green off-hand spheres), heatmap panel painted from `/ws/live`, 3×2 menu (REC / SESSION / PREDICT / TRAIN / RECENTER / EXIT VR), ray pointers + select-event button activation, REAL-vs-PRED finger-curl bars, ghost-hand overlay anchored at the real wrist when inference is on. | +| `styles.css` | Pre-VR landing page only (the XR session paints WebGL directly, CSS doesn't apply inside). | + +### Cache-bust contract + +The HTML references the JS as `app.js?v=N`. **Bump N every time `app.js` changes.** Quest Browser ignores `Cache-Control: no-store` for ES modules in some configurations — the querystring forces a fresh fetch. The no-cache middleware on `/`, `/vr`, and `/static/*` is correct, but the version-querystring is the actual cache-busting mechanism in practice. + +### Auto-enable inference on train (VR-only) + +The desktop UI's server-side default is **paused-on-load** for inference (see commit `bd1b68a` rationale). In VR there's no obvious second click to enable it, so `runTrain` in `app.js` POSTs `{enabled: true}` to `/api/inference/enabled` after a successful activate — pressing TRAIN implies "I want predictions running." The status strip surfaces `trained: R²=X · model loaded ✓ · predict ON` so the change is visible. + ## Known gotchas - **`from __future__ import annotations` breaks FastAPI body inference.** Don't add it back to `app.py` — the lazy-string annotations make FastAPI treat Pydantic-model parameters as query fields. (Bit us once, documented in `app.py` header.) - **`Pin.init(Pin.OUT, value=0)` quirks** are in the firmware, not here — but if you ever rewrite the matrix scan, see the firmware repo's "Sensor scan techniques" section first. -- **Browser cache during dev**: the no-cache middleware on `/` and `/static/*` makes JS/CSS edits land on plain F5. If you ever serve this off a CDN or behind a cache, remove or scope down that middleware. +- **Browser cache during dev**: the no-cache middleware on `/`, `/vr`, and `/static/*` makes JS/CSS edits land on plain F5 in desktop browsers. Quest Browser ignores it for ES modules — see the cache-bust contract in the VR section above (`?v=N` on the script src). +- **`/vr` was missing from the no-cache middleware** until commit `fb83f82`. If you add another HTML entry point, remember to whitelist it too. - **mpremote and the LASK5 don't coexist** on the same serial port — `openmuscle web` only uses UDP, but Thonny / PuTTY / another mpremote session will lock the COM port. Symptom: `mpremote: failed to access COMxx`. +- **WebXR requires a secure context.** localhost (`http://`) counts; LAN HTTPS via mkcert is the untethered path. Plain HTTP over LAN will silently refuse to grant hand-tracking and the user sees "WebXR not available" with no specific reason. The landing-page preflight surfaces this. ## v0.2.0 history (the LASK5 expansion) diff --git a/pc/src/openmuscle/web/app.py b/pc/src/openmuscle/web/app.py index 68f1c7f..dd17b93 100644 --- a/pc/src/openmuscle/web/app.py +++ b/pc/src/openmuscle/web/app.py @@ -9,6 +9,9 @@ # string and falls back to treating the param as a query string field). import asyncio +import shutil +import subprocess +import sys from contextlib import asynccontextmanager from pathlib import Path from typing import Optional @@ -24,6 +27,48 @@ STATIC_DIR = Path(__file__).parent / "static" +def _reveal_path_in_file_manager(path: Path, select_file: bool) -> None: + """Open `path` in the OS file manager. If `select_file=True` and the + platform supports it, highlight the file inside its parent folder + rather than just opening the folder. Raises RuntimeError on failure. + + Whitelist-guarded by the caller: this function does NOT verify that + `path` is inside captures_dir. That check happens in the route. + """ + if not path.exists(): + raise RuntimeError(f"Path does not exist: {path}") + + # Explorer / Finder / xdg-open all need *absolute* paths -- they don't + # inherit our CWD predictably and a relative path like + # "data/raw/merged/foo.csv" silently fails with "location not found". + path = path.resolve() + + try: + if sys.platform.startswith("win"): + if select_file and path.is_file(): + # explorer /select,"C:\full\path\file.csv" -- highlights file + subprocess.Popen(["explorer", f"/select,{path}"]) + else: + # Open the folder itself + folder = path if path.is_dir() else path.parent + subprocess.Popen(["explorer", str(folder)]) + elif sys.platform == "darwin": + if select_file and path.is_file(): + subprocess.Popen(["open", "-R", str(path)]) + else: + folder = path if path.is_dir() else path.parent + subprocess.Popen(["open", str(folder)]) + else: + # Linux / other -- xdg-open only opens directories cleanly + opener = shutil.which("xdg-open") or shutil.which("gio") + if opener is None: + raise RuntimeError("No file-manager opener found (xdg-open / gio)") + folder = path if path.is_dir() else path.parent + subprocess.Popen([opener, str(folder)]) + except Exception as e: + raise RuntimeError(f"Failed to open file manager: {e}") + + def create_app(udp_port: int = 3141, captures_dir: Optional[str] = None, model_path: Optional[str] = None, hand_target: Optional[tuple] = None) -> FastAPI: @@ -59,7 +104,11 @@ async def lifespan(_app: FastAPI): async def no_cache_static(request: Request, call_next): response = await call_next(request) path = request.url.path - if path == "/" or path.startswith("/static/"): + # Cache-bust the HTML entry points + every static asset. Without + # /vr in this list, Quest Browser cached the old VR HTML (which + # pointed at app.js without the version querystring), so refreshes + # kept loading stale JS even after the file changed on disk. + if path == "/" or path == "/vr" or path.startswith("/static/"): response.headers["Cache-Control"] = "no-store, no-cache, must-revalidate, max-age=0" response.headers["Pragma"] = "no-cache" response.headers["Expires"] = "0" @@ -69,6 +118,14 @@ async def no_cache_static(request: Request, call_next): async def index(): return FileResponse(STATIC_DIR / "index.html") + # WebXR companion page served at /vr. Quest Browser loads this URL, + # negotiates 'hand-tracking', opens /ws/quest, and streams XRHand frames. + # WebXR requires a secure context -- HTTPS over LAN (mkcert) or + # http://localhost via `adb reverse tcp:8000 tcp:8000` over USB. + @app.get("/vr") + async def vr_page(): + return FileResponse(STATIC_DIR / "vr" / "index.html") + app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static") # ----- live data over WebSocket ----- @@ -89,6 +146,40 @@ async def ws_live(websocket: WebSocket): finally: state.ws_clients.discard(websocket) + # Inbound WS from the Quest headset. Browsers can't speak UDP so the + # WebXR client opens this socket and pushes XRHand joint frames as + # JSON. We feed each frame through ingest_quest_packet, which + # synthesizes a device_type="quest_hand" OpenMusclePacket and routes + # it through the same _handle_packet path as UDP devices. Net effect: + # the Quest looks like any other label-producing device to the + # recorder, matcher, snapshot, and meta-sidecar code. + @app.websocket("/ws/quest") + async def ws_quest(websocket: WebSocket): + await websocket.accept() + client = (f"{websocket.client.host}:{websocket.client.port}" + if websocket.client else "unknown") + state.log_buffer.info("quest", f"connected: {client}") + frame_count = 0 + try: + while True: + payload = await websocket.receive_json() + try: + state.ingest_quest_packet(payload) + frame_count += 1 + except Exception as e: + # Per-frame errors shouldn't kill the socket -- a single + # malformed frame happens; the next one is usually fine. + state.log_buffer.warn( + "quest", f"ingest failed at frame {frame_count}: " + f"{type(e).__name__}: {e}") + except WebSocketDisconnect: + state.log_buffer.info( + "quest", f"disconnected: {client} after {frame_count} frames") + except Exception as e: + state.log_buffer.error( + "quest", f"socket error from {client}: " + f"{type(e).__name__}: {e}") + # ----- REST: devices ----- @app.get("/api/devices") @@ -104,7 +195,8 @@ class StartRecordingBody(BaseModel): sensor_device_id: Optional[str] = None label_device_id: Optional[str] = None filename: Optional[str] = None - window_ms: int = 100 + # If None, AppState picks per-device-type (lask5=100, quest_hand=175). + window_ms: Optional[int] = None @app.post("/api/recording") async def start_recording(body: StartRecordingBody): @@ -168,6 +260,31 @@ async def download_capture(name: str): raise HTTPException(status_code=404, detail="Capture not found") return FileResponse(p, filename=p.name, media_type="text/csv") + class RevealBody(BaseModel): + # If empty/None -> just open captures_dir. Otherwise must be a + # capture name whitelisted by state.capture_path(). + name: Optional[str] = None + + @app.post("/api/reveal") + async def reveal_in_folder(body: RevealBody): + """Open the captures folder (and optionally highlight a specific + capture) in the OS file manager. Local-only convenience; the server + is intended for localhost use.""" + if body.name: + p = state.capture_path(body.name) + if p is None: + raise HTTPException(status_code=404, detail="Capture not found") + target = p + select = True + else: + target = state.captures_dir + select = False + try: + _reveal_path_in_file_manager(target, select_file=select) + except RuntimeError as e: + raise HTTPException(status_code=500, detail=str(e)) + return {"opened": str(target), "selected": select} + @app.delete("/api/captures/{name}") async def delete_capture(name: str): ok = state.delete_capture(name) @@ -358,14 +475,98 @@ async def delete_session_endpoint(session_id: str, unlink_captures: bool = True) raise HTTPException(status_code=404, detail="Session not found") return {"deleted": session_id} + # ----- REST: retroactive session<->capture linking ----- + # + # A capture made *outside* an active session can be added to one + # afterwards, and vice versa removed. This is the "I forgot to start a + # session before recording" recovery path. We update both: + # 1. the session JSON's `captures` list (authoritative) + # 2. the capture's meta sidecar (tag `session:` + auto.session_id) + # so the captures-panel filter, the past-sessions expansion, and any + # future export all agree on which session a capture belongs to. + + class LinkCapturesBody(BaseModel): + capture_names: list[str] # bulk add + + @app.post("/api/sessions/{session_id}/captures") + async def add_captures_to_session(session_id: str, body: LinkCapturesBody): + s = state.get_session(session_id) + if s is None: + raise HTTPException(status_code=404, detail="Session not found") + + tag = "session:" + session_id + added, skipped = [], [] + for name in body.capture_names: + if state.capture_path(name) is None: + skipped.append({"name": name, "reason": "capture not found"}) + continue + if name in s.get("captures", []): + skipped.append({"name": name, "reason": "already in session"}) + continue + try: + state.link_capture_to_session(session_id, name) + # Update the capture's meta so the tag-based filter + any + # future export sees this capture as part of the session. + meta = state.read_capture_meta(name) or {} + tags = list(meta.get("tags") or []) + if tag not in tags: + tags.append(tag) + state.write_capture_meta(name, { + "tags": tags, + "auto": {"session_id": session_id}, + }) + added.append(name) + except Exception as e: + skipped.append({"name": name, "reason": str(e)}) + + return { + "added": added, + "skipped": skipped, + "session": state.get_session(session_id), + } + + @app.delete("/api/sessions/{session_id}/captures/{capture_name}") + async def remove_capture_from_session(session_id: str, capture_name: str): + s = state.get_session(session_id) + if s is None: + raise HTTPException(status_code=404, detail="Session not found") + if capture_name not in s.get("captures", []): + raise HTTPException(status_code=404, detail="Capture not in session") + try: + state.unlink_capture_from_session(session_id, capture_name) + # Strip the session tag + clear auto.session_id, but ONLY for this + # session (leave any other 'session:xxx' tags alone -- though by + # the data model a capture should only ever belong to one session). + tag = "session:" + session_id + meta = state.read_capture_meta(capture_name) or {} + new_tags = [t for t in (meta.get("tags") or []) if t != tag] + state.write_capture_meta(capture_name, { + "tags": new_tags, + "auto": {"session_id": None}, + }) + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + return { + "removed": capture_name, + "session": state.get_session(session_id), + } + return app def serve(host: str = "0.0.0.0", port: int = 8000, udp_port: int = 3141, captures_dir: Optional[str] = None, model_path: Optional[str] = None, - hand_target: Optional[tuple] = None): - """Run the web UI server (blocks).""" + hand_target: Optional[tuple] = None, + ssl_certfile: Optional[str] = None, + ssl_keyfile: Optional[str] = None): + """Run the web UI server (blocks). + + Pass ssl_certfile + ssl_keyfile to serve HTTPS -- required for the + WebXR /vr page since Quest Browser refuses hand-tracking on plain + HTTP. Generate certs locally with mkcert and install the root CA + on the headset (see README). + """ import uvicorn app = create_app( udp_port=udp_port, @@ -373,4 +574,5 @@ def serve(host: str = "0.0.0.0", port: int = 8000, udp_port: int = 3141, model_path=model_path, hand_target=hand_target, ) - uvicorn.run(app, host=host, port=port, log_level="info") + uvicorn.run(app, host=host, port=port, log_level="info", + ssl_certfile=ssl_certfile, ssl_keyfile=ssl_keyfile) diff --git a/pc/src/openmuscle/web/state.py b/pc/src/openmuscle/web/state.py index 4dde98a..fb251a5 100644 --- a/pc/src/openmuscle/web/state.py +++ b/pc/src/openmuscle/web/state.py @@ -22,7 +22,38 @@ import socket from openmuscle.data.storage import CaptureWriter -from openmuscle.protocol.schema import OpenMusclePacket +from openmuscle.protocol.schema import CURRENT_VERSION, OpenMusclePacket + + +# Canonical channel ordering for a single quest_hand joint. This is the +# coupling point between two places: +# - `AppState.ingest_quest_packet` packs joints into `data.values` in +# this order via `_flatten_quest_joint`. +# - `AppState._write_labels_schema` emits the column->(joint, channel) +# map using `_quest_label_column` which assumes this same order. +# Changing one without the other makes the labels-schema sidecar lie +# about what's in the CSV. Both callers route through this tuple + the +# two helpers below to make the coupling structural rather than two +# aspirational comments. +QUEST_JOINT_CHANNEL_ORDER = ("px", "py", "pz", "rx", "ry", "rz", "rw") + + +def _flatten_quest_joint(pos, rot): + """Pack one joint's (pos, rot) into the canonical channel order. + + `pos` is [x, y, z]; `rot` is a unit quaternion [x, y, z, w]. Returns a + flat list of `len(QUEST_JOINT_CHANNEL_ORDER)` floats. + """ + components = {"px": pos[0], "py": pos[1], "pz": pos[2], + "rx": rot[0], "ry": rot[1], "rz": rot[2], "rw": rot[3]} + return [components[ch] for ch in QUEST_JOINT_CHANNEL_ORDER] + + +def _quest_label_column(joint_index: int, channel_index: int) -> int: + """Index of the CSV column corresponding to (joint, channel) for a + quest_hand recording. Inverse view of the layout produced by + `_flatten_quest_joint` applied N times.""" + return joint_index * len(QUEST_JOINT_CHANNEL_ORDER) + channel_index from openmuscle.receiver.matcher import TemporalMatcher from openmuscle.receiver.udp_listener import UDPListener from openmuscle.web.inference import InferenceEngine @@ -146,6 +177,22 @@ class ActiveCapture: matcher: TemporalMatcher sensor_jsonl: Optional[IO] = None label_jsonl: Optional[IO] = None + # Path for a per-capture labels-schema sidecar. Written on the first + # label packet (lazy, like the CSV header) so the schema reflects + # whatever the device actually sent rather than what we expected. + # Only populated for label sources whose label width / structure is + # not derivable from device_type alone -- e.g. quest_hand. + labels_schema_path: Optional[Path] = None + labels_schema_written: bool = False + # Label-width lock. quest_hand frames can vary in joint count when hand + # tracking is partial; writing those varying lengths straight to the CSV + # produces ragged rows (different column count per row) that break + # pandas.read_csv and corrupt the capture for training. We lock the + # expected width at the first label packet (same width the labels-schema + # sidecar describes) and pad/truncate every paired row to it, so the CSV + # stays rectangular and consistent with the schema. None until locked. + locked_label_count: Optional[int] = None + label_width_mismatch_count: int = 0 # Stats surfaced in the WS snapshot sensor_frames_seen: int = 0 label_packets_seen: int = 0 @@ -307,6 +354,77 @@ def _handle_packet(self, pkt: OpenMusclePacket): if self.hand_target: self._forward_to_hand(pred) + def ingest_quest_packet(self, payload: dict) -> None: + """Synthesize an OpenMusclePacket from a Quest WebSocket frame and + route it through the standard packet path. + + From the recorder's perspective the Quest is just another device: + once we build an OpenMusclePacket with `device_type="quest_hand"` + and hand it to `_handle_packet`, the DeviceInfo registry, the + TemporalMatcher, the JSONL sidecars, and the WS snapshot all + treat it identically to LASK5. This is why the Quest never needs + to learn UDP -- the JS in the headset only has WebSocket. + + Expected payload shape (one hand for v1, per the team's + "FlexGrid-arm only" decision): + + { + "device_id": "quest-01", # optional + "ts": 12345, # device-local ms (optional) + "handedness": "left" | "right", # which hand this frame is for + "joints": [ + {"name": "wrist", "pos": [x,y,z], "rot": [x,y,z,w], "radius": 0.02}, + {"name": "thumb-metacarpal", "pos": [...], "rot": [...]}, + ... 26 entries total, in OpenXR canonical order + ], + "meta": {...} # optional, e.g. tracking confidence + } + + We flatten joints into `data.values = [px,py,pz, rx,ry,rz,rw] * N` + so it matches LASK5's `data.values` convention (the recorder pulls + from this field). The structured per-joint form is preserved under + `data.hands` for the JSONL sidecar -- if you later want to know + WHICH joint a column corresponds to, the sidecar tells you, while + the trainable CSV stays a plain matrix of floats. + + Empty payloads (e.g. headset reports tracking lost this frame) + are silently dropped -- we want gaps in the data, not zero rows + that would mislead the model. + """ + joints = payload.get("joints") or [] + if not joints: + return + + flat: list[float] = [] + joint_names: list[str] = [] + for j in joints: + pos = j.get("pos") or [0.0, 0.0, 0.0] + rot = j.get("rot") or [0.0, 0.0, 0.0, 1.0] + # Route through _flatten_quest_joint so the channel order is + # taken from QUEST_JOINT_CHANNEL_ORDER -- same source-of-truth + # the labels-schema sidecar reads via _quest_label_column. + flat.extend(float(v) for v in _flatten_quest_joint(pos, rot)) + joint_names.append(j.get("name", "")) + + pkt = OpenMusclePacket( + version=CURRENT_VERSION, + device_type="quest_hand", + device_id=payload.get("device_id") or "quest-01", + timestamp_ms=int(payload.get("ts") or 0), + data={ + "values": flat, + "handedness": payload.get("handedness") or "unknown", + "joint_names": joint_names, + "hands": { + "handedness": payload.get("handedness") or "unknown", + "joints": joints, + }, + }, + metadata=payload.get("meta") or {}, + receive_time=time.time(), + ) + self._handle_packet(pkt) + # Flush JSONL sidecars every N frames to bound crash-loss to ~3 s of # data while keeping syscalls ~50× cheaper than line-buffered writes. # At 33 Hz sensor + 25 Hz label rates, 100 ≈ 3 s. @@ -323,6 +441,12 @@ def _record_packet(self, pkt: OpenMusclePacket): rec.matcher.add_label(pkt) rec.label_packets_seen += 1 self._write_jsonl(rec.label_jsonl, pkt) + # Lazy: emit the labels-schema sidecar on the first label packet + # for label sources whose column layout is opaque from device_type + # alone. v1: quest_hand only. The sidecar gives consumers a map + # from the CSV's label_0..label_N columns back to (joint, channel). + if rec.labels_schema_path is not None and not rec.labels_schema_written: + self._write_labels_schema(rec, pkt) # Bounded crash-loss flush if (rec.label_packets_seen % self.JSONL_FLUSH_EVERY == 0 and rec.label_jsonl is not None): @@ -361,6 +485,25 @@ def _record_packet(self, pkt: OpenMusclePacket): rec.matched_count += 1 label_values = list(matched.data.get("values", [])) + # Guarantee a rectangular CSV. If the label width was locked (quest_hand) + # and this matched label has a different length, pad with zeros or + # truncate to the locked width. Without this, variable-length payloads + # (partial hand tracking, or a misbehaving client) would write ragged + # rows that break pandas.read_csv and corrupt the whole capture. The + # locked width matches the labels-schema sidecar, so consumers can + # still map every column to a (joint, channel). NB: a well-behaved + # client sends a fixed-length array every frame, so this rarely fires; + # the mismatch counter surfaces it in the stop-recording stats + log + # if it does. + if rec.locked_label_count is not None and label_values: + n = rec.locked_label_count + if len(label_values) != n: + rec.label_width_mismatch_count += 1 + if len(label_values) < n: + label_values = list(label_values) + [0.0] * (n - len(label_values)) + else: + label_values = list(label_values[:n]) + # Flatten as-sent [cols][rows] matrix row-major. Header in CaptureWriter # is R0C0..R0Cn, R1C0.., so iterating rows-then-cols here keeps the # column meaning correct (cf. the col-major bug we fixed in 245cb8f). @@ -369,6 +512,78 @@ def _record_packet(self, pkt: OpenMusclePacket): flat = [mat[c][r] for r in range(rows) for c in range(cols)] rec.writer.write_row(pkt.receive_time, flat, label_values) + def _write_labels_schema(self, rec: "ActiveCapture", pkt: OpenMusclePacket) -> None: + """Emit the per-capture labels-schema sidecar. + + Maps the CSV's label_0..label_N columns back to the underlying + (joint, channel) coordinates for a quest_hand recording. Without + this a wide-label CSV is opaque -- you'd have to know the joint + ordering by convention. With it, any consumer can deserialize + label columns into named joint poses. + + TODO(wrist-relative-labels): joint positions are stored in absolute + world coordinates as captured by the headset, so a model trained + on them learns to predict positions where the recordings happened + to be -- generally not where the user is at inference time. The + VR ghost-hand viz works around this by anchoring predicted joints + to the real wrist each frame, but the right long-term fix is to + subtract the wrist position (and optionally rotate into the + wrist's frame) before writing, and reverse the transform at + inference. This changes the CSV semantics, so it deserves its + own scope. Track in the OpenMuscle-Software repo issue list + ("Wrist-relative label coordinates for portable quest_hand + models") before any model that needs to generalize across + capture-locations. + """ + if rec.labels_schema_path is None: + return + joint_names = list(pkt.data.get("joint_names") or []) + handedness = pkt.data.get("handedness") or "unknown" + # Pull the channel order from the module-level constant so the schema + # is guaranteed consistent with how ingest_quest_packet flattened + # the values into the CSV (see QUEST_JOINT_CHANNEL_ORDER docstring). + ordering = list(QUEST_JOINT_CHANNEL_ORDER) + n_floats = len(ordering) + # Build the explicit column->(joint, channel) map so consumers + # don't have to re-derive it from joint order. + columns = [] + for ji, jn in enumerate(joint_names): + for ci, ch in enumerate(ordering): + columns.append({ + "name": f"label_{_quest_label_column(ji, ci)}", + "joint": jn, + "channel": ch, + }) + n_label_columns = len(joint_names) * n_floats + # Lock the CSV label width to what this first label packet described, + # so _record_packet can pad/truncate every paired row to match the + # schema and keep the CSV rectangular. (Well-behaved clients send a + # fixed-length joint array every frame, so this almost never triggers + # a pad/truncate -- it's a safety net against partial-tracking frames + # and any client that emits a variable-length payload.) + rec.locked_label_count = n_label_columns + schema = { + "label_source": "quest_hand", + "handedness": handedness, + "ordering": ordering, + "floats_per_joint": n_floats, + "n_joints": len(joint_names), + "n_label_columns": n_label_columns, + "joint_names": joint_names, + "columns": columns, + } + try: + with open(rec.labels_schema_path, "w") as f: + json.dump(schema, f, indent=2) + rec.labels_schema_written = True + self.log_buffer.info( + "recording", + f"labels-schema written: {rec.labels_schema_path.name} " + f"({len(joint_names)} joints, {len(columns)} columns)") + except OSError as e: + self.log_buffer.warn( + "recording", f"labels-schema write failed: {e}") + @staticmethod def _write_jsonl(stream: Optional[IO], pkt: OpenMusclePacket): """Append one packet as a JSONL line. No-op if stream is None. @@ -400,24 +615,22 @@ def _write_jsonl(stream: Optional[IO], pkt: OpenMusclePacket): def _forward_to_hand(self, pred: list): """Send the prediction to the robot hand as a `PC,...` UDP datagram. - Builds 5 servo angles in 0..179 from the 4 piston predictions (assumed - normalized 0..1; clamped) plus the most recent LASK5 joystick X as the - 5th. The hand's `'PC'` device config uses linear 0..179 -> 0..179 - mapping, so values land directly on servo angles. + Builds 5 servo angles in 0..179. Channel order on the hand + (FINGER_CHANNELS = [1, 3, 5, 7, 9]) is anatomically: + channel 1 -> thumb + channel 3 -> index + channel 5 -> middle + channel 7 -> ring + channel 9 -> pinky + + The LASK5 has 4 pistons (the 4 closing fingers) and a joystick. + We map joystick X -> thumb, pistons 0..3 -> index..pinky. + The hand's 'PC' device config uses linear 0..179 -> 0..179, so values + land directly on servo angles. """ - # Pistons -> 0..179, assuming model output is normalized 0..1. - # Anything else gets clamped, which is the right failure mode -- - # bracelet finger goes to extreme rather than 4000-degree angle. - angles = [] - for v in pred[:4]: - try: - v = max(0.0, min(1.0, float(v))) - except Exception: - v = 0.0 - angles.append(int(v * 179)) - - # 5th finger = joystick X from the most recent LASK5 packet. Range - # 0..4095 -> 0..179. Default to center (90) if no LASK5 has been seen. + # Thumb (channel 1) = joystick X from the most recent LASK5 packet. + # Range 0..4095 -> 0..179. Default to center (90) if no LASK5 has been + # seen yet (so the thumb sits in a neutral pose instead of slamming open). joy_x = None for d in self.devices.values(): if d.device_type == "lask5" and d.last_joystick: @@ -425,23 +638,57 @@ def _forward_to_hand(self, pred: list): if isinstance(jx, (int, float)): joy_x = jx break - if joy_x is None: - angles.append(90) - else: - angles.append(max(0, min(179, int((joy_x / 4095.0) * 179)))) + thumb_angle = 90 if joy_x is None else max(0, min(179, int((joy_x / 4095.0) * 179))) + + # Index..pinky (channels 3, 5, 7, 9) from pistons 0..3. + # Model output is assumed normalized 0..1; anything else gets clamped, + # which is the right failure mode -- finger goes to extreme rather + # than commanding a 4000-degree servo angle. + finger_angles = [] + for v in pred[:4]: + try: + v = max(0.0, min(1.0, float(v))) + except Exception: + v = 0.0 + finger_angles.append(int(v * 179)) + + # The hand's 'PC' device config has reverse=False, but the LASK5's + # native ESPNow path uses the 'default' / 'L5' config (reverse=True) + # which flips the piston order before mapping to FINGER_CHANNELS. + # To match that mapping from our PC path, we reverse the pistons + # ourselves: P1 -> index, P2 -> middle, P3 -> ring, P4 -> pinky. + # (Documented in DEVICES of the hand firmware, archived 2026-05-14.) + angles = [thumb_angle] + finger_angles[::-1] # [thumb, P4, P3, P2, P1] # Build the CSV the hand expects: 'PC,a1,a2,a3,a4,a5' payload = ("PC," + ",".join(str(a) for a in angles)).encode("utf-8") + # Rate-limited log so we can SEE whether forwarding is working. + # First time + every 500th packet: log success/failure to the buffer + # so the operator can debug without strace. + self._hand_forward_count = getattr(self, "_hand_forward_count", 0) + 1 + log_now = (self._hand_forward_count == 1 + or self._hand_forward_count % 500 == 0) try: if self._hand_sock is None: self._hand_sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self._hand_sock.setblocking(False) - self._hand_sock.sendto(payload, self.hand_target) - except Exception: - # Non-fatal: hand might be offline / on a different subnet. - # We don't spam logs since this fires per FlexGrid packet. - pass + n = self._hand_sock.sendto(payload, self.hand_target) + if log_now: + self.log_buffer.info("inference", + "hand forward #{}: sent {} bytes to {}:{} -> {!r}".format( + self._hand_forward_count, n, + self.hand_target[0], self.hand_target[1], + payload.decode("utf-8", errors="replace"))) + except Exception as e: + # Always log the first failure so the operator sees it; rate-limit + # subsequent ones (every 500) so we don't spam. + if log_now or not getattr(self, "_hand_forward_error_logged", False): + self.log_buffer.warn("inference", + "hand forward #{} FAILED: {} ({!r}) -> target={}".format( + self._hand_forward_count, type(e).__name__, str(e), + self.hand_target)) + self._hand_forward_error_logged = True async def _broadcast_latest_frames(self): """Push the latest frame for each device to all WS clients.""" @@ -498,6 +745,11 @@ def _snapshot(self) -> dict: "sensor_frames_seen": r.sensor_frames_seen, "label_packets_seen": r.label_packets_seen, "match_rate": round(r.match_rate, 3), + # Rows that had to be padded/truncated to the locked label + # width (variable-length label frames, e.g. partial hand + # tracking). Surfaced live so the in-VR header can warn the + # operator that joints are dropping mid-capture. + "label_width_mismatch": r.label_width_mismatch_count, } return { "type": "tick", @@ -575,18 +827,34 @@ def _auto_pick_sensor(self) -> Optional[str]: return d.device_id return None + # Order in which auto-pick prefers label-producing device types when + # the operator doesn't pick one explicitly. Quest first because it's + # the richer ground-truth source -- if both are connected during a + # comparison session, we want the wider label vector by default. + AUTO_LABEL_TYPE_PREFERENCE = ("quest_hand", "lask5") + def _auto_pick_label(self) -> Optional[str]: - """First connected label-producing device (LASK5), if any.""" - for d in self.devices.values(): - if d.device_type == "lask5": - return d.device_id + """First connected label-producing device, by type preference.""" + for preferred_type in self.AUTO_LABEL_TYPE_PREFERENCE: + for d in self.devices.values(): + if d.device_type == preferred_type: + return d.device_id return None + # Default match windows per label-device family. Quest WebXR has higher + # end-to-end latency than LASK5's ESP-NOW path (browser -> WS -> server), + # so a tighter window would reject too many sensor frames as unpaired. + DEFAULT_WINDOW_MS_BY_TYPE = { + "lask5": 100, + "quest_hand": 175, + } + DEFAULT_WINDOW_MS_FALLBACK = 100 + def start_recording(self, sensor_device_id: Optional[str] = None, label_device_id: Optional[str] = None, filename: Optional[str] = None, - window_ms: int = 100, + window_ms: Optional[int] = None, label_count: int = 4) -> ActiveCapture: """Start a paired recording. @@ -599,9 +867,13 @@ def start_recording(self, record sensor frames only (the paired CSV will have no label columns). filename: CSV name. JSONL sidecars derived from it. - window_ms: temporal match window in milliseconds (default 100). + window_ms: temporal match window in milliseconds. If None, picked + per-device-type from DEFAULT_WINDOW_MS_BY_TYPE + (lask5=100, quest_hand=175, else 100). label_count: how many label_* columns to write per row (default 4 - for the standard LASK5 piston count). + for the standard LASK5 piston count). Ignored when + the label device is quest_hand -- in that case the + writer infers from the first packet's values length. """ if self.recording is not None: raise RuntimeError("Already recording -- stop the current capture first") @@ -632,6 +904,22 @@ def start_recording(self, effective_label_count = label_count if label_device_id else 0 + # Quest hand tracking sends a wide joint vector whose width depends on + # the headset / WebXR implementation (Quest 3S = 26 joints * 7 floats = + # 182 per hand). Rather than hardcode it, pass None so CaptureWriter + # derives the column count from the first label packet. + label_dev_for_width = self.devices.get(label_device_id) if label_device_id else None + label_device_type = label_dev_for_width.device_type if label_dev_for_width else None + if label_device_type == "quest_hand": + effective_label_count = None + + # Pick the match window: explicit arg wins; otherwise per-device-type + # default (Quest needs a wider window than LASK5 because WebXR + # latency is higher than ESP-NOW). + if window_ms is None: + window_ms = self.DEFAULT_WINDOW_MS_BY_TYPE.get( + label_device_type or "", self.DEFAULT_WINDOW_MS_FALLBACK) + # Build paths name = filename or f"capture_{int(time.time())}.csv" if not name.endswith(".csv"): @@ -641,6 +929,12 @@ def start_recording(self, stem = csv_path.with_suffix("") # data/raw/merged/foo (no .csv) sensor_sidecar = stem.with_suffix(".sensor.jsonl") label_sidecar = stem.with_suffix(".label.jsonl") + # Labels-schema sidecar is only written for label sources whose + # column meaning isn't obvious from device_type alone -- v1: Quest. + labels_schema_sidecar: Optional[Path] = ( + Path(str(stem) + ".labels.schema.json") + if label_device_type == "quest_hand" else None + ) writer = CaptureWriter( output_path=str(csv_path), @@ -677,6 +971,7 @@ def start_recording(self, matcher=matcher, sensor_jsonl=sensor_stream, label_jsonl=label_stream, + labels_schema_path=labels_schema_sidecar, ) self.log_buffer.info("recording", "started: {} (sensor={}, label={}, window={}ms)".format( @@ -692,6 +987,7 @@ def start_recording(self, auto = { "sensor_device_id": sensor_device_id, "label_device_id": label_device_id, + "label_source": label_device_type, # "lask5" | "quest_hand" | None "window_ms": window_ms, "sensor_shape": [sensor_dev.rows, sensor_dev.cols], "started_at": self.recording.started_at, @@ -747,6 +1043,17 @@ def stop_recording(self) -> Optional[dict]: "stopped: {} -- {} matched / {} sensor frames ({}%), {}s".format( rec.path.name, rec.matched_count, rec.sensor_frames_seen, round(rec.match_rate * 100, 1), round(rec.duration_s, 1))) + # If any rows had to be padded/truncated to the locked label width, + # call it out -- it means the label source sent variable-length + # frames (e.g. partial hand tracking), and those rows have some + # zero-filled joint columns. + if rec.label_width_mismatch_count: + self.log_buffer.warn("recording", + "{}: {} row(s) padded/truncated to locked label width {} " + "(variable-length label frames -- some joint columns are " + "zero-filled)".format( + rec.path.name, rec.label_width_mismatch_count, + rec.locked_label_count)) result = { "filename": rec.path.name, "rows": rec.row_count, @@ -759,10 +1066,14 @@ def stop_recording(self) -> Optional[dict]: "unpaired_sensor": rec.unpaired_sensor_count, "sensor_frames_seen": rec.sensor_frames_seen, "label_packets_seen": rec.label_packets_seen, + "label_width_mismatch": rec.label_width_mismatch_count, "match_rate": round(rec.match_rate, 3), "sidecars": { "sensor": str(rec.path.with_suffix("")) + ".sensor.jsonl", "label": (str(rec.path.with_suffix("")) + ".label.jsonl") if rec.label_jsonl else None, + "labels_schema": (str(rec.labels_schema_path) + if (rec.labels_schema_path and rec.labels_schema_written) + else None), }, } self.recording = None @@ -819,7 +1130,7 @@ def delete_capture(self, name: str) -> bool: p.unlink() # Also delete sidecars if present stem = p.with_suffix("") - for suffix in (".sensor.jsonl", ".label.jsonl", ".meta.json"): + for suffix in (".sensor.jsonl", ".label.jsonl", ".meta.json", ".labels.schema.json"): sidecar = Path(str(stem) + suffix) if sidecar.exists(): try: diff --git a/pc/src/openmuscle/web/static/app.js b/pc/src/openmuscle/web/static/app.js index 474637c..14062c6 100644 --- a/pc/src/openmuscle/web/static/app.js +++ b/pc/src/openmuscle/web/static/app.js @@ -18,10 +18,31 @@ const selStatus = document.getElementById('captures-sel-status'); const checkAll = document.getElementById('captures-check-all'); const modelsBody = document.getElementById('models-body'); const modelsCount = document.getElementById('models-count'); +const openFolderBtn = document.getElementById('captures-open-folder'); + +// Ask the server to open the captures folder in the OS file manager. +// If `name` is given, highlight that capture file inside the folder. +async function revealCaptureFolder(name) { + try { + const r = await fetch('/api/reveal', { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({name: name || null}), + }); + if (!r.ok) throw new Error(await readError(r)); + } catch (e) { + alert('Could not open folder: ' + e.message); + } +} + +if (openFolderBtn) { + openFolderBtn.onclick = () => revealCaptureFolder(null); +} // Per-user pick preferences that survive a refresh const STORE_SENSOR = 'om.sensor_device_id'; const STORE_LABEL = 'om.label_device_id'; +const STORE_HAND = 'om.hand_target'; // last successfully-applied "host:port" — auto-restored on next launch // Set of capture filenames currently checked in the table const selectedCaptures = new Set(); @@ -41,6 +62,11 @@ function connectWS() { ws.onopen = () => { wsStatus.textContent = 'connected'; wsStatus.className = 'badge online'; + // Re-arm the hand-target auto-restore: every fresh WS connect (which + // includes server restarts) gets a chance to re-apply the saved hand + // target. Otherwise the operator has to remember to click Apply + // after every `openmuscle web` restart. + handTargetRestoreAttempted = false; }; ws.onclose = () => { wsStatus.textContent = 'disconnected'; @@ -84,6 +110,59 @@ function handleTick(msg) { const lask = lastDevices.find(d => d.device_type === 'lask5'); renderLask(lask); renderInference(msg.inference); + // Comparator + top-bar pipeline strip are Studio-shell additions. + // They derive everything from the per-tick snapshot, so they update + // in lockstep with the underlying bars and the WS message. + renderResiduals(lask, msg.inference); + renderPipelinePills(msg, lask); + // quest_hand 3D viewer: when a hand label source is streaming, swap the + // LASK5 piston comparator for a live 3D hand (the pistons are zeros for + // a hand source). No-op when no quest_hand device is present. + renderHandViewer(lastDevices.find(d => d.device_type === 'quest_hand'), + msg.inference); +} + +// ---------- quest_hand 3D viewer ---------- + +// Drives the Three.js hand viewer (window.OMHandViewer, loaded as a module). +// Shows the REAL captured hand from the live quest_hand device's flat joint +// `values`, plus the model's PREDICTED hand from inference.piston_values when +// a quest-trained model (>= 25 joints * 7 floats) is running. Toggles the +// .hand-mode class on .comparator so CSS hides the LASK5 pistons in favor of +// the viewer. +function renderHandViewer(questDev, inference) { + const comparator = document.querySelector('.comparator'); + const viewerReady = window.OMHandViewer && window.OMHandViewer.isReady; + if (!questDev) { + if (comparator) comparator.classList.remove('hand-mode'); + if (viewerReady && window.OMHandViewer.isReady()) window.OMHandViewer.setVisible(false); + return; + } + // Lazy-init the viewer on first quest_hand sighting (the module may still + // be loading right at page open; guard with isReady). + if (window.OMHandViewer && !window.OMHandViewer.isReady()) { + const el = document.getElementById('hand-viewer-canvas'); + if (el) window.OMHandViewer.init(el); + } + if (!(window.OMHandViewer && window.OMHandViewer.isReady())) return; + + if (comparator) comparator.classList.add('hand-mode'); + window.OMHandViewer.setVisible(true); + + const realFlat = Array.isArray(questDev.values) ? questDev.values : null; + // Predicted hand: only when the live model emits a full hand vector. + let predFlat = null; + const pv = inference && inference.piston_values; + if (Array.isArray(pv) && pv.length >= 25 * 7) predFlat = pv; + window.OMHandViewer.update(realFlat, predFlat); + + // Reuse the existing GT meta slot to label the hand source. + const gtMeta = document.getElementById('lask-meta'); + if (gtMeta) { + const hz = (typeof questDev.hz === 'number') ? questDev.hz.toFixed(0) : '0'; + const nJoints = realFlat ? Math.floor(realFlat.length / 7) : 0; + gtMeta.textContent = `Quest hand · ${nJoints} joints · ${hz} Hz`; + } } // ---------- device list ---------- @@ -370,6 +449,7 @@ function renderActiveSession() { ${armBit}${subj} · ${s.capture_count || 0} captures · ${formatUptime(dur)}${gestures}
+
@@ -377,6 +457,8 @@ function renderActiveSession() { ${s.notes ? `
${escapeHtml(s.notes)}
` : ''} `; document.getElementById('session-end-btn').onclick = endSession; + const addBtn = document.getElementById('active-session-add-btn'); + if (addBtn) addBtn.onclick = () => openLinkModal(activeSession); sessionStartBtn.disabled = true; sessionStartBtn.title = 'End the current session before starting a new one'; capturesFilterLabel.textContent = `· filtered to ${s.name || s.id}`; @@ -400,6 +482,137 @@ async function refreshPastSessions() { } catch (e) { /* best-effort */ } } +// Sessions whose capture list is currently expanded in the UI. Persisted +// across re-renders (refreshPastSessions can fire on its own) so a poll +// doesn't collapse what the user just opened. +const expandedSessions = new Set(); + +// ---------- Add-captures-to-session picker modal ---------- +// +// Lets the operator retroactively assign past recordings (made without an +// active session) to a session. The picker shows every capture NOT +// currently linked to the target session, with checkboxes for bulk add. +// +// Wires up: +// - "+ Add captures" button in each past-session card +// - "×" remove button on each capture in the expanded view + +const linkModal = document.getElementById('link-modal'); +const linkSessionName = document.getElementById('link-session-name'); +const linkCaptureList = document.getElementById('link-capture-list'); +const linkAddBtn = document.getElementById('link-add-btn'); +let linkSessionId = null; // current session being edited +const linkSelected = new Set(); // capture names currently checked + +function openLinkModal(session) { + linkSessionId = session.id; + linkSelected.clear(); + linkSessionName.textContent = session.name || session.id; + linkAddBtn.disabled = true; + linkAddBtn.textContent = 'Add 0 captures'; + linkCaptureList.innerHTML = '
Loading captures…
'; + linkModal.classList.add('open'); + linkModal.setAttribute('aria-hidden', 'false'); + + // Fetch the full capture list, filter out ones already in this session. + fetch('/api/captures') + .then(r => r.ok ? r.json() : Promise.reject('fetch failed')) + .then(list => { + const alreadyLinked = new Set(session.captures || []); + const candidates = list.filter(c => !alreadyLinked.has(c.name)); + if (!candidates.length) { + linkCaptureList.innerHTML = '
All captures are already in this session.
'; + return; + } + // Render rows with checkbox + name + meta summary + (if linked + // to a different session) an annotation so the operator doesn't + // accidentally yank a capture out of another session. + linkCaptureList.innerHTML = candidates.map(c => { + const meta = c.meta || {}; + const otherSession = (meta.tags || []).find(t => t.startsWith('session:')); + const otherNote = otherSession + ? `⚠ ${escapeHtml(otherSession)}` + : ''; + const kb = (c.size_bytes / 1024).toFixed(1); + return ``; + }).join(''); + linkCaptureList.querySelectorAll('input[type=checkbox]').forEach(cb => { + cb.onchange = () => { + if (cb.checked) linkSelected.add(cb.dataset.name); + else linkSelected.delete(cb.dataset.name); + const n = linkSelected.size; + linkAddBtn.disabled = (n === 0); + linkAddBtn.textContent = `Add ${n} capture${n === 1 ? '' : 's'}`; + }; + }); + }) + .catch(err => { + linkCaptureList.innerHTML = '
Could not load captures.
'; + console.warn('link picker fetch:', err); + }); +} + +function closeLinkModal() { + linkModal.classList.remove('open'); + linkModal.setAttribute('aria-hidden', 'true'); + linkSessionId = null; + linkSelected.clear(); +} + +linkModal.querySelectorAll('[data-close]').forEach(el => { + el.addEventListener('click', closeLinkModal); +}); +document.addEventListener('keydown', e => { + if (e.key === 'Escape' && linkModal.classList.contains('open')) closeLinkModal(); +}); + +linkAddBtn.onclick = async () => { + if (!linkSessionId || linkSelected.size === 0) return; + linkAddBtn.disabled = true; + linkAddBtn.textContent = 'Adding…'; + try { + const r = await fetch(`/api/sessions/${encodeURIComponent(linkSessionId)}/captures`, { + method: 'POST', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({capture_names: [...linkSelected]}), + }); + if (!r.ok) throw new Error(await readError(r)); + const result = await r.json(); + if ((result.skipped || []).length) { + // Surface skips inline -- e.g. "already in another session" + console.warn('some captures skipped:', result.skipped); + } + closeLinkModal(); + await refreshPastSessions(); + await refreshCaptures(); + } catch (e) { + alert('Add failed: ' + (e.message || e)); + linkAddBtn.disabled = false; + const n = linkSelected.size; + linkAddBtn.textContent = `Add ${n} capture${n === 1 ? '' : 's'}`; + } +}; + +async function removeCaptureFromSession(sessionId, captureName) { + if (!confirm(`Remove ${captureName} from this session?\n(The capture file itself stays — just the link is cleared.)`)) return; + try { + const r = await fetch( + `/api/sessions/${encodeURIComponent(sessionId)}/captures/${encodeURIComponent(captureName)}`, + {method: 'DELETE'} + ); + if (!r.ok) throw new Error(await readError(r)); + await refreshPastSessions(); + await refreshCaptures(); + } catch (e) { + alert('Remove failed: ' + (e.message || e)); + } +} + function renderPastSessions() { if (!pastSessions.length) { pastSessionsList.innerHTML = '
No past sessions yet.
'; @@ -408,21 +621,59 @@ function renderPastSessions() { pastSessionsList.innerHTML = pastSessions.map(s => { const dur = (s.ended_at && s.started_at) ? Math.floor(s.ended_at - s.started_at) : null; const armBit = s.arm ? escapeHtml(s.arm) + ' arm' : '—'; - const captures = s.capture_count || (s.captures || []).length; - return `
-
+ const captureList = Array.isArray(s.captures) ? s.captures : []; + const captureCount = s.capture_count != null ? s.capture_count : captureList.length; + const isOpen = expandedSessions.has(s.id); + const caret = captureList.length ? (isOpen ? '▾' : '▸') : '·'; + // The captures sub-list is a sibling div, toggled by .hidden. We + // render it eagerly (with .hidden if closed) so the open/close + // animation isn't required and so screen readers can find it. + const capturesInner = captureList.length + ? captureList.map(name => ` +
  • + ${escapeHtml(name)} + + + + download + + +
    • `).join('') + : '
  • No captures linked to this session.
    • '; + + return `
    +
    + ${caret} ${escapeHtml(s.name || s.id)} - ${armBit} · ${escapeHtml(s.subject || '—')} · ${captures} captures${dur != null ? ' · ' + formatUptime(dur) : ''} + ${armBit} · ${escapeHtml(s.subject || '—')} · ${captureCount} captures${dur != null ? ' · ' + formatUptime(dur) : ''}
    +
    ${s.notes ? `
    ${escapeHtml(s.notes)}
    ` : ''} +
      ${capturesInner}
    `; }).join(''); + + // Stop session-action buttons from triggering the row-toggle handler + pastSessionsList.querySelectorAll('.session-actions button').forEach(btn => { + btn.addEventListener('click', e => e.stopPropagation()); + }); + + // Toggle expand/collapse when the session header row is clicked + pastSessionsList.querySelectorAll('[data-toggle-session]').forEach(head => { + head.onclick = () => { + const sid = head.dataset.toggleSession; + if (expandedSessions.has(sid)) expandedSessions.delete(sid); + else expandedSessions.add(sid); + renderPastSessions(); + }; + }); + pastSessionsList.querySelectorAll('button[data-delete-session]').forEach(btn => { btn.onclick = async () => { const sid = btn.dataset.deleteSession; @@ -435,6 +686,34 @@ function renderPastSessions() { } catch (e) { alert('Delete failed: ' + e.message); } }; }); + + // Per-capture actions inside the expanded list + pastSessionsList.querySelectorAll('button[data-reveal-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + revealCaptureFolder(btn.dataset.revealCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-edit-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + openMetaModal(btn.dataset.editCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-unlink-cap]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + removeCaptureFromSession(btn.dataset.fromSession, btn.dataset.unlinkCap); + }; + }); + pastSessionsList.querySelectorAll('button[data-add-to-session]').forEach(btn => { + btn.onclick = (e) => { + e.stopPropagation(); + const sid = btn.dataset.addToSession; + const session = pastSessions.find(s => s.id === sid); + if (session) openLinkModal(session); + }; + }); } pastSessionsToggle.onclick = () => { @@ -643,6 +922,7 @@ function renderCaptures(list) { ${escapeHtml(date)} + download @@ -669,6 +949,9 @@ function renderCaptures(list) { capturesBody.querySelectorAll('button[data-edit]').forEach(btn => { btn.onclick = () => openMetaModal(btn.dataset.edit); }); + capturesBody.querySelectorAll('button[data-reveal]').forEach(btn => { + btn.onclick = () => revealCaptureFolder(btn.dataset.reveal); + }); updateSelectionStatus(); } @@ -1035,7 +1318,43 @@ function renderInference(inf) { }); } +// One-shot: if the server has no hand_target on first snapshot but we have +// one saved in localStorage, auto-apply it so launching `openmuscle web` +// doesn't lose the address every time. UDP-only (the only protocol we +// support); port defaults to 3145. +let handTargetRestoreAttempted = false; +function maybeRestoreHandTarget(inf) { + if (handTargetRestoreAttempted) return; + if (!inf) return; // wait for first inference snapshot + handTargetRestoreAttempted = true; // one-shot regardless of outcome + if (inf.hand_target) return; // server already has one (e.g. --hand on CLI) + const saved = localStorage.getItem(STORE_HAND); + if (!saved) return; + autoApplyHandTarget(saved); +} + +async function autoApplyHandTarget(raw) { + let host = raw, port = 3145; + if (raw.includes(':')) { + const idx = raw.lastIndexOf(':'); + host = raw.slice(0, idx); + const portN = parseInt(raw.slice(idx + 1), 10); + if (Number.isFinite(portN) && portN > 0 && portN < 65536) port = portN; + } + try { + await fetch('/api/inference/hand', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ host, port }), + }); + } catch (e) { + console.warn('hand target auto-restore failed', e); + } +} + function renderInferenceControls(inf) { + maybeRestoreHandTarget(inf); + const hasModel = !!(inf && inf.model); const enabled = !!(inf && inf.enabled); @@ -1111,6 +1430,10 @@ async function applyHandTarget() { body: JSON.stringify({ host, port }), }); if (!r.ok) throw new Error(await readError(r)); + // Persist so next launch auto-restores. Clear on explicit empty + // so the operator can "forget" the target deliberately. + if (host) localStorage.setItem(STORE_HAND, raw); + else localStorage.removeItem(STORE_HAND); // Force the snapshot side to refresh by clearing the cache so the // next tick syncs the (possibly normalized) value back into the input. lastSnapshotHand = undefined; @@ -1124,6 +1447,110 @@ inferHandInput.addEventListener('keydown', (e) => { if (e.key === 'Enter') applyHandTarget(); }); +// ---------- Studio shell: comparator residuals (Δ) ---------- + +// Compute per-piston residual (predicted - ground_truth) and write it into +// the .delta-row elements in the comparator. Color-codes by direction so +// the operator can see at a glance whether the model is over- or under- +// shooting each finger. +// +// CLOSE_THRESHOLD picked at 0.05 (5% of the 0..1 scale) — below that, the +// difference is below the noise floor of the LASK5 measurement itself. +const RESIDUAL_CLOSE_THRESHOLD = 0.05; + +function renderResiduals(laskDev, inf) { + const deltaRows = document.querySelectorAll('#comparator-deltas .delta-row'); + if (!deltaRows.length) return; + const gt = laskDev && Array.isArray(laskDev.values) ? laskDev.values : null; + const pred = inf && Array.isArray(inf.piston_values) ? inf.piston_values : null; + + deltaRows.forEach((row, i) => { + const valEl = row.querySelector('.delta-val'); + row.classList.remove('over', 'under', 'close'); + if (!gt || !pred || i >= gt.length || i >= pred.length) { + if (valEl) valEl.textContent = '--'; + return; + } + const g = pistonFraction(gt[i]); + const p = pistonFraction(pred[i]); + const d = p - g; + valEl.textContent = (d >= 0 ? '+' : '') + d.toFixed(2); + if (Math.abs(d) < RESIDUAL_CLOSE_THRESHOLD) row.classList.add('close'); + else if (d > 0) row.classList.add('over'); + else row.classList.add('under'); + }); +} + +// ---------- Studio shell: top-bar pipeline status strip ---------- + +// Set a pipe-pill's status + value text. State controls colour: +// 'live' -- blue accent (data flowing) +// 'ok' -- green (idle but healthy) +// 'warn' -- orange +// 'bad' -- red +// '' -- neutral grey +function setPipePill(id, state, valText) { + const el = document.getElementById(id); + if (!el) return; + el.classList.remove('ok', 'warn', 'bad', 'live'); + if (state) el.classList.add(state); + const valEl = el.querySelector('.pipe-val'); + if (valEl) valEl.textContent = valText; +} + +function renderPipelinePills(msg, laskDev) { + // SENSOR pill = the active flexgrid (the one driving the heatmap) + const dev = selectedDevice(); + if (dev && dev.device_type === 'flexgrid') { + const stale = dev.last_seen_age > 2.0; + setPipePill('pipe-sensor', stale ? 'warn' : 'live', `${dev.hz.toFixed(0)}Hz`); + } else { + setPipePill('pipe-sensor', '', '--'); + } + + // LABEL pill = LASK5 stream + if (laskDev) { + const stale = laskDev.last_seen_age > 2.0; + setPipePill('pipe-label', stale ? 'warn' : 'live', `${laskDev.hz.toFixed(0)}Hz`); + } else { + setPipePill('pipe-label', '', '--'); + } + + // CAPTURE pill + if (recordingState) { + const matchRate = recordingState.match_rate ?? 0; + const cls = matchRate < 0.5 ? 'bad' : (matchRate < 0.9 ? 'warn' : 'live'); + setPipePill('pipe-capture', cls, `REC ${recordingState.rows ?? 0}r`); + } else if (activeSession) { + setPipePill('pipe-capture', 'ok', `session: ${activeSession.name || activeSession.id}`); + } else { + setPipePill('pipe-capture', '', 'idle'); + } + + // MODEL pill + const inf = msg.inference; + if (inf && inf.model && inf.enabled) setPipePill('pipe-model', 'live', inf.model); + else if (inf && inf.model && !inf.enabled) setPipePill('pipe-model', 'ok', inf.model + ' (paused)'); + else setPipePill('pipe-model', '', 'none'); + + // HAND pill = UDP forwarding target + if (inf && inf.hand_target) setPipePill('pipe-hand', 'live', inf.hand_target); + else setPipePill('pipe-hand', '', 'off'); +} + +// ---------- Studio shell: diagnostics drawer ---------- + +const diagToggle = document.getElementById('diag-toggle'); +const diagBody = document.getElementById('diag-body'); +if (diagToggle && diagBody) { + diagToggle.onclick = () => { + const isHidden = diagBody.classList.toggle('hidden'); + diagToggle.setAttribute('aria-expanded', isHidden ? 'false' : 'true'); + diagToggle.textContent = (isHidden ? '▸' : '▾') + ' Diagnostics & logs'; + // Logs poll runs unconditionally; we just hide the DOM. Cheap. + }; +} + // ---------- utils ---------- function escapeHtml(s) { diff --git a/pc/src/openmuscle/web/static/hand-viewer.js b/pc/src/openmuscle/web/static/hand-viewer.js new file mode 100644 index 0000000..1c8bb74 --- /dev/null +++ b/pc/src/openmuscle/web/static/hand-viewer.js @@ -0,0 +1,217 @@ +// OpenMuscle desktop 3D hand viewer. +// +// Renders quest_hand joint data as a 3D skeleton in the Studio "Live" stage, +// replacing the LASK5 4-piston comparator (which shows zeros for a hand +// label source). Shows the REAL captured hand and, when a quest-trained +// model is running, the model's PREDICTED hand overlaid -- the desktop +// counterpart to the VR ghost hand. +// +// Loaded as an ES module from index.html; exposes a small imperative API on +// window.OMHandViewer so the plain (non-module) app.js can drive it: +// OMHandViewer.init(containerEl) +// OMHandViewer.update(realFlat, predFlat) // flat [px,py,pz,rx,ry,rz,rw]*N +// OMHandViewer.setVisible(bool) +// +// Both hands are transformed into WRIST-LOCAL space (subtract the wrist +// position, rotate by the inverse wrist orientation) before drawing, so the +// hand always appears in a canonical palm orientation regardless of how it +// was held, and REAL vs PRED is a direct shape comparison. (This is the same +// wrist-relative idea tracked for training in issue #2, used here purely for +// visualization.) + +import * as THREE from 'three'; + +// Canonical WebXR hand joint order (25 joints). Index i in a flat values +// array occupies [i*7 .. i*7+6] = px,py,pz, rx,ry,rz,rw. +const N_JOINTS = 25; +const FLOATS_PER_JOINT = 7; + +// Bone connectivity as [parentIdx, childIdx] pairs. Wrist = 0; then 4 thumb +// joints (1..4), then 5 each for index/middle/ring/pinky. +const BONES = [ + // thumb + [0, 1], [1, 2], [2, 3], [3, 4], + // index + [0, 5], [5, 6], [6, 7], [7, 8], [8, 9], + // middle + [0, 10], [10, 11], [11, 12], [12, 13], [13, 14], + // ring + [0, 15], [15, 16], [16, 17], [17, 18], [18, 19], + // pinky + [0, 20], [20, 21], [21, 22], [22, 23], [23, 24], +]; + +// Fingertip joint indices, for slightly larger tip markers. +const TIPS = new Set([4, 9, 14, 19, 24]); + +const COLOR_REAL = 0x34d399; // emerald +const COLOR_PRED = 0xfbbf24; // amber + +let scene, camera, renderer, container; +let raf = null; +let visible = false; +let autoRotate = true; +let yaw = 0.6, pitch = -0.25; // view angles (radians) +let dragging = false, lastX = 0, lastY = 0; + +// One reusable hand rig = 25 joint spheres + bone line-segments + a label. +function makeHandRig(color, opacity) { + const group = new THREE.Group(); + const jointMat = new THREE.MeshBasicMaterial({ + color, transparent: opacity < 1, opacity, + }); + const tipGeo = new THREE.SphereGeometry(0.007, 10, 8); + const jointGeo = new THREE.SphereGeometry(0.0045, 8, 6); + const joints = []; + for (let i = 0; i < N_JOINTS; i++) { + const m = new THREE.Mesh(TIPS.has(i) ? tipGeo : jointGeo, jointMat); + group.add(m); + joints.push(m); + } + // Bones: one BufferGeometry with 2 vertices per bone, updated each frame. + const positions = new Float32Array(BONES.length * 2 * 3); + const boneGeo = new THREE.BufferGeometry(); + boneGeo.setAttribute('position', new THREE.BufferAttribute(positions, 3)); + const boneMat = new THREE.LineBasicMaterial({ + color, transparent: opacity < 1, opacity: Math.min(1, opacity + 0.1), + }); + const bones = new THREE.LineSegments(boneGeo, boneMat); + group.add(bones); + return { group, joints, bones, positions }; +} + +let realRig, predRig; + +// Scratch objects reused per update (no per-frame allocation). +const _wristPos = new THREE.Vector3(); +const _wristQuatInv = new THREE.Quaternion(); +const _p = new THREE.Vector3(); + +// Transform a flat values array into wrist-local joint positions and write +// them into the rig. Returns false (and hides the rig) if the data is +// missing/degenerate. `out` is an array of N_JOINTS THREE.Vector3 to fill. +function layoutHand(flat, rig, outPositions) { + if (!flat || flat.length < N_JOINTS * FLOATS_PER_JOINT) { + rig.group.visible = false; + return false; + } + _wristPos.set(flat[0], flat[1], flat[2]); + // Wrist quaternion -> inverse, with a degeneracy guard (a bad model can + // emit a near-zero quat; inverting that yields NaN). + const qx = flat[3], qy = flat[4], qz = flat[5], qw = flat[6]; + const qLenSq = qx * qx + qy * qy + qz * qz + qw * qw; + let useRot = false; + if (qLenSq > 1e-6) { + _wristQuatInv.set(qx, qy, qz, qw).normalize().invert(); + useRot = true; + } + for (let i = 0; i < N_JOINTS; i++) { + const b = i * FLOATS_PER_JOINT; + _p.set(flat[b], flat[b + 1], flat[b + 2]).sub(_wristPos); + if (useRot) _p.applyQuaternion(_wristQuatInv); + rig.joints[i].position.copy(_p); + outPositions[i].copy(_p); + } + // Update bone vertices from the laid-out joint positions. + const pos = rig.positions; + for (let k = 0; k < BONES.length; k++) { + const [a, c] = BONES[k]; + const pa = outPositions[a], pc = outPositions[c]; + const o = k * 6; + pos[o] = pa.x; pos[o + 1] = pa.y; pos[o + 2] = pa.z; + pos[o + 3] = pc.x; pos[o + 4] = pc.y; pos[o + 5] = pc.z; + } + rig.bones.geometry.attributes.position.needsUpdate = true; + rig.group.visible = true; + return true; +} + +const _realOut = Array.from({ length: N_JOINTS }, () => new THREE.Vector3()); +const _predOut = Array.from({ length: N_JOINTS }, () => new THREE.Vector3()); + +function resize() { + if (!container || !renderer) return; + const w = container.clientWidth || 320; + const h = container.clientHeight || 240; + renderer.setSize(w, h, false); + camera.aspect = w / h; + camera.updateProjectionMatrix(); +} + +function animate() { + raf = requestAnimationFrame(animate); + if (!visible) return; + if (autoRotate && !dragging) yaw += 0.005; + // Orbit camera around the origin (hands are wrist-centered at origin). + const r = 0.34; + camera.position.set( + r * Math.cos(pitch) * Math.sin(yaw), + r * Math.sin(pitch), + r * Math.cos(pitch) * Math.cos(yaw), + ); + camera.lookAt(0, 0, 0); + renderer.render(scene, camera); +} + +const OMHandViewer = { + init(containerEl) { + if (renderer) return; // idempotent + container = containerEl; + scene = new THREE.Scene(); + camera = new THREE.PerspectiveCamera(45, 1, 0.01, 10); + renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true }); + renderer.setPixelRatio(window.devicePixelRatio || 1); + container.appendChild(renderer.domElement); + + realRig = makeHandRig(COLOR_REAL, 1.0); + predRig = makeHandRig(COLOR_PRED, 0.55); + predRig.group.visible = false; + scene.add(realRig.group); + scene.add(predRig.group); + + // Drag to rotate (pauses auto-rotate while dragging). + const el = renderer.domElement; + el.style.cursor = 'grab'; + el.addEventListener('pointerdown', (e) => { + dragging = true; lastX = e.clientX; lastY = e.clientY; + el.style.cursor = 'grabbing'; el.setPointerCapture(e.pointerId); + }); + el.addEventListener('pointermove', (e) => { + if (!dragging) return; + yaw -= (e.clientX - lastX) * 0.01; + pitch = Math.max(-1.4, Math.min(1.4, pitch + (e.clientY - lastY) * 0.01)); + lastX = e.clientX; lastY = e.clientY; + }); + const endDrag = () => { dragging = false; el.style.cursor = 'grab'; }; + el.addEventListener('pointerup', endDrag); + el.addEventListener('pointercancel', endDrag); + // Double-click resets to auto-rotate. + el.addEventListener('dblclick', () => { autoRotate = true; }); + + window.addEventListener('resize', resize); + resize(); + animate(); + }, + + // realFlat: live captured hand. predFlat: model prediction (or null/short + // for a non-quest model -> predicted hand hidden). + update(realFlat, predFlat) { + if (!renderer) return; + layoutHand(realFlat, realRig, _realOut); + if (predFlat && predFlat.length >= N_JOINTS * FLOATS_PER_JOINT) { + layoutHand(predFlat, predRig, _predOut); + } else { + predRig.group.visible = false; + } + }, + + setVisible(v) { + visible = !!v; + if (container) container.style.display = v ? '' : 'none'; + if (v) resize(); + }, + + isReady() { return !!renderer; }, +}; + +window.OMHandViewer = OMHandViewer; diff --git a/pc/src/openmuscle/web/static/index.html b/pc/src/openmuscle/web/static/index.html index b795c34..6e1de5c 100644 --- a/pc/src/openmuscle/web/static/index.html +++ b/pc/src/openmuscle/web/static/index.html @@ -3,179 +3,313 @@ - OpenMuscle Live + OpenMuscle Studio + + + -
    -

    OpenMuscle Live

    -
    +
    +
    +

    OpenMuscle Studio

    +
    + + + + +
    disconnected
    -
    -
    -

    Devices

    -
      -
    • Waiting for a device to send a packet…
      • -
    -
    +
    -
    -
    -

    Heatmap

    - -
    - -
    + +
    +
    +

    1 Live

    + what the band feels · what the model thinks +
    -
    -
    -

    LASK5 — Ground Truth

    - no device -
    -
    -
    P1
    --
    -
    P2
    --
    -
    P3
    --
    -
    P4
    --
    -
    -
    - joystick - - --, -- -
    -
    +
    + + -
    -
    -

    LASK Inference — Predicted

    - no model loaded -
    -
    -
    P1̂
    --
    -
    P2̂
    --
    -
    P3̂
    --
    -
    P4̂
    --
    -
    -
    - - - - -
    -
    + +
    +
    + +
    + +
    -
    -

    Record

    -
    - - - - + +
    +
    +

    Ground truth vs Predicted

    +
    + GT: no device + MODEL: no model loaded +
    +
    + +
    + +
    +
    P1
    --
    +
    P2
    --
    +
    P3
    --
    +
    P4
    --
    +
    + + +
    +
    Δ--
    +
    Δ--
    +
    Δ--
    +
    Δ--
    +
    + + +
    +
    P1̂
    --
    +
    P2̂
    --
    +
    P3̂
    --
    +
    P4̂
    --
    +
    +
    + + +
    +
    +
    + ● real + ● predicted + drag to rotate +
    +
    + + +
    + + --, -- +
    + + +
    + + predictions to a robot hand: see Stage 4 +
    +
    -
    -
    -
    -

    Sessions

    -
    - + +
    +
    +

    2 Capture

    + session + record · paired sensor/label rows go to disk +
    + +
    + +
    +
    + Session + +
    +
    +
    No active session — recordings won't be grouped. Click "New session" to start one.
    +
    +
    + + +
    +
    + Record +
    +
    + + + + +
    +
    -
    -
    No active session — recordings won't be grouped. Click "New session" to start one.
    -
    + +
    -
    -
    -

    Captures

    -
    - 0 selected - + +
    +
    +

    3 Data & Models

    + captures become models · audit honestly · iterate +
    + +
    + +
    +
    +

    Captures

    +
    + 0 selected + +
    +
    + + + + + + + + + + + + + + +
    NameMetaSizeModified
    No captures saved yet.
    +
    + + +
    + +
    +
    + + +
    +
    +

    Models

    + +
    + + + + + + + + + + + + + + +
    NameCreatedR²MSEFeatures × Labels
    No models trained yet.
    - - - - - - - - - - - - - - -
    NameMetaSizeModified
    No captures saved yet.
    -
    -
    -
    -

    Models

    - + +
    +
    +

    4 Output

    + where the predictions are sent +
    +
    + + + no hand target
    - - - - - - - - - - - - - - -
    NameCreatedR²MSEFeatures × Labels
    No models trained yet.
    -
    -
    -

    Logs

    -
    - - - -
    + +
    + +
    - Open-Muscle · FlexGrid · github + Open-Muscle · FlexGrid Studio · github
    - + - + + + +