Quest 3 + WebXR companion (quest_hand device type, /vr, ML loop closed)#1
Open
turfptax wants to merge 47 commits into
Open
Quest 3 + WebXR companion (quest_hand device type, /vr, ML loop closed)#1turfptax wants to merge 47 commits into
turfptax wants to merge 47 commits into
Conversation
The legacy capture .txt converter previously emitted one row per packet with (device_id, ticks, data) columns -- not directly trainable. Rewrite it to deque-match the 3 SensorBand banks against LASK5 label packets by recency, producing the standard 12-sensor + 4-label paired-CSV shape plus per-stream timestamp columns. Dedup back-to-back identical records. Also lift combine_csvs() into this module so cli `train` and the web `/api/train` route can share one row-wise concatenation helper. dataset.py: detect_columns() now excludes any column with "Timestamp" in its name so the new Sensor_Timestamp / Label_Timestamp columns don't get fed to the model as features. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrite _forward_to_hand() to match the OpenHand firmware's anatomical channel layout (FINGER_CHANNELS = [1, 3, 5, 7, 9] -> thumb, index, middle, ring, pinky). The previous implementation appended joystick X as the 5th angle, which lined up with channel 9 (pinky) instead of channel 1 (thumb) -- the hand was getting "thumb=predicted_pinky" and "pinky=joystick_x", silently inverted. Also reverse the piston order (P4..P1) on the way out so the PC path matches the LASK5 ESP-NOW path, whose default 'L5' device config has reverse=True. Without this reversal the same prediction array drives different fingers depending on whether the packet came via the model or via direct ESP-NOW from the LASK5. Add rate-limited logs (first hit + every 500th) so the operator can verify forwarding is actually happening without strace. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two additive endpoints for the Captures and Sessions panels:
POST /api/reveal {name?}
Opens captures_dir in the OS file manager. With a capture name,
highlights that specific .csv inside its folder (Explorer /select on
Windows, open -R on macOS, xdg-open on Linux). Whitelist-guarded
via state.capture_path().
POST/DELETE /api/sessions/{id}/captures{/name}
The "I forgot to start a session before recording" recovery path.
Bulk-add or remove existing captures from a session after the fact.
Updates both the session JSON's `captures` list AND the capture's
.meta.json (tag `session:<id>` + auto.session_id) so the captures
filter and session expansion stay consistent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rebrand the web UI from "OpenMuscle Live" to "OpenMuscle Studio" and restructure the page around the data-pipeline narrative: SENSOR -> LABEL -> CAPTURE -> MODEL -> HAND A topbar pipeline-strip shows live status for each stage with click- to-scroll anchors. Body splits into numbered stages: (1) Live -- heatmap + GT-vs-Predicted comparator hero, (2) Capture, (3) Models, (4) Output. The Devices list collapses into a thin left rail. styles.css gets the matching grid-template + new pipe-pill / stage-* classes. app.js adds renderPipelinePills() (called every WS tick) and moves the per-panel renderers into the new stage containers. No behavior changes -- same WS contract, same REST surface. Pure restructure + visual rework. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ease
Two ergonomics fixes to the boot.py listener loops:
(1) KeyboardInterrupt handling
Both espnow_listen() and udp_listen() now catch Ctrl-C from the REPL
(mpremote / Thonny / serial), release_all() the servos, close the
socket cleanly, and re-raise so the boot orchestrator can also bail
out. Previously a Ctrl-C left the servos energized in whatever
angle they were last commanded -- annoying and a power drain.
(2) UDP idle-sleep
udp_listen() tracks last_packet_t. After UDP_IDLE_SLEEP_S (30s) with
no incoming packets, release_all() puts the servos in low-torque
state and the OLED shows "Sleeping..." so the operator knows the
hand is intentionally limp rather than crashed. The next incoming
packet wakes it (apply_packet implicitly re-energizes), so wake is
just "resume listening".
Loop yield drops from 500 Hz (2 ms) to 20 Hz (50 ms) while asleep,
so the ESP32 isn't pegged when nobody's streaming.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…cket The Quest hand-tracking pipeline (coming next) doesn't know how many label columns it'll have until the first XRHand frame arrives -- joint count varies by Quest model and XRHand implementation (Quest 3S sends ~26 joints * 7 floats = 182 per hand). Hardcoding 4 (LASK5) in CaptureWriter's constructor doesn't fit. Refactor CaptureWriter to defer the header row until the first write_row call, and let label_count be either an explicit hint (today: 4 for LASK5) or None (infer from len(label_values) of the first row). Preserve the "empty CSV still has a header" invariant via a check in close(): if no row was ever paired, emit the header using the hint or 0 label columns. So consumers (combine_csvs, train, pandas readers) never see a zero-byte file. state.py: start_recording() detects label_device_type == "quest_hand" and passes label_count=None so the writer infers. LASK5 callers are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hesis
Browsers can't speak UDP, so the WebXR client in the Quest headset will
open a WebSocket and push XRHand frames as JSON. Rather than introduce
a parallel ingest path, we synthesize each frame into an
OpenMusclePacket(device_type="quest_hand") and route it through the
same _handle_packet that UDP listeners use. From the recorder, matcher,
snapshot, and meta-sidecar code's perspective, the Quest is just
another device.
state.py: new AppState.ingest_quest_packet(payload) builds
data.values = flat [px,py,pz, rx,ry,rz,rw] per joint (matches
LASK5 convention so the recorder needs no
special-case)
data.handedness = "left" | "right"
data.joint_names = canonical OpenXR-ish names for the column layout
data.hands = structured per-joint form preserved for JSONL
sidecar / offline analysis
Empty payloads (Quest reports tracking lost) drop silently -- we want
gaps in the data, not zero rows that mislead the model.
app.py: new /ws/quest endpoint accepts the inbound socket, calls
ingest_quest_packet per frame, logs connect/disconnect/per-frame errors
to the existing log buffer. Per-frame errors don't kill the socket --
one bad frame happens; the next one is usually fine.
Smoke-tested end-to-end with a 5-frame fake Quest client. Device
appears in /api/devices as type="quest_hand" with values_len=182.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Server-side prep for the WebXR client. Recording with the Quest as the
label source now Just Works:
* /vr route serves static/vr/index.html (placeholder for now; the real
WebXR client lands in the next commit). WebXR needs a secure context
-- HTTPS via mkcert or localhost via `adb reverse`.
* start_recording's window_ms argument is now Optional[int]. None means
"pick per-device-type" via a new DEFAULT_WINDOW_MS_BY_TYPE map
(lask5=100, quest_hand=175, fallback=100). Quest WebXR has higher
end-to-end latency than LASK5 ESP-NOW, so a tighter window was
dropping too many sensor frames as unpaired.
* _auto_pick_label now walks an explicit type-preference list
("quest_hand", "lask5"). Previously only LASK5 was auto-pickable, so
starting a recording with only a Quest device connected fell through
to sensor-only mode silently.
* New per-capture <name>.labels.schema.json sidecar, written lazily on
the first quest_hand label packet. Maps each CSV column
(label_0..label_N) back to (joint_name, channel) so consumers can
deserialize the wide label vector without reverse-engineering the
joint ordering. LASK5 captures don't write the sidecar (column
meaning is obvious from device_type).
* Auto-meta gains label_source ("lask5" | "quest_hand" | None) so the
Captures panel filter and any downstream training pipeline can
cleanly separate the two label families.
* delete_capture extended to also remove .labels.schema.json.
* stop_recording sidecars response gains labels_schema path.
* StartRecordingBody.window_ms becomes Optional[int] (default None ->
per-device-type) so the JS client doesn't have to know the Quest
default value.
End-to-end verified: prime two devices, POST /api/recording with no
overrides, server picks quest-test as label_device_id, window=175,
streams 30 paired frames, schema sidecar lands with 26 joints / 182
label columns, meta.auto.label_source = "quest_hand", CSV has the
expected 243 columns (1 + 60 + 182).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quest Browser refuses to grant WebXR hand-tracking on plain HTTP --
WebXR requires a secure context. Two paths get you one:
1. mkcert + LAN HTTPS (untethered, the real use case)
2. adb reverse over USB (Quest sees http://localhost, tethered)
This adds path 1. mkcert produces a cert/key pair; install the mkcert
root CA on the headset via Settings -> Security -> Install a
certificate, then:
openmuscle web --ssl-certfile cert.pem --ssl-keyfile key.pem
uvicorn handles the TLS termination directly -- no nginx etc. Both
flags must be passed together (or neither, for plain HTTP local
development).
The CLI prints the LAN VR URL pattern on startup so the operator
doesn't have to compose it by hand.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… record The /vr page now actually does something. Pure HTML + ES modules from the three.js CDN, no build step. Companion to the server-side ingestion that landed in 95556fb / 7b1d2b4. Architecture in the headset: XRSession ('hand-tracking' optional feature requested) | v per XRFrame: 1. capture XRHand joints (25 per OpenXR spec) for the chosen arm -> POST {device_id, ts, handedness, joints[]} to /ws/quest (throttled to ~30 Hz so the WS queue stays drained) 2. update small joint-sphere visualizer so user sees what we see 3. detect pinch (index-tip <-> thumb-tip < 2.5 cm, hold >= 1 s) -> toggle recording. Yellow ring on the index tip fills during hold. 4. raycast: if the OFF-hand's index tip touches the floating button, toggle recording too. Off-hand on purpose -- if you tap with the same fingers we're trying to capture, you smear the gesture. 5. paint heatmap from latest /ws/live snapshot onto a CanvasTexture on a 40 x 12 cm world-anchored panel placed at session start. The matching desktop UI colour ramp is reproduced inline so we don't depend on its CSS. 6. update header strip: live device Hz when idle, "REC ... rows ... match X%" when recording. The heatmap + button are WORLD-anchored at session-start (not head-locked) -- locking to the head is the canonical nausea recipe. Pre-VR landing page has a checklist (HTTPS, WebXR support, server reachable) so failure modes are obvious before the operator pulls the headset on. URL params: ?arm=right (default) | ?arm=left -- which Quest hand is the FlexGrid arm. We capture only this one; the other stays free for the record button. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
protocol.md: document the quest_hand payload (flat values + structured hands form + handedness + joint_names), plus the per-capture .labels.schema.json sidecar contract. Sibling LASK5 / FlexGrid sections left untouched. docs/vr-setup.md (new): operator guide for the Quest 3S companion -- mkcert setup + headset CA install (Settings -> Security -> Install a certificate), HTTPS server invocation, per-session walkthrough, and a troubleshooting table for the failure modes we hit during smoke testing. Ends with the architecture diagram showing how the synthetic quest_hand packet path collapses into the existing UDP-listener pipeline (no parallel ingest, no special-casing downstream). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The single record-toggle sphere becomes a row of three labeled buttons
so the user can drive a full record -> train cycle without taking the
headset off. Per first-headset test feedback: "We may want to design
buttons on the UI so i can record and stop and train through the VR app."
Layout (anchored at session start, world-locked):
[ HEATMAP ]
[ 25 Hz · 30 Hz · idle ] <- header (unchanged)
⬤ REC ⬤ TRAIN ⬤ SESSION <- new row, off-hand tap
[ trained: R²=0.81 ✓ ] <- new status strip (fades)
Architecture:
* createButton(name, isActive, onActivate) factory builds a small sphere
plus a billboard text label canvas-textured plane. Each button lives in
the global `buttons` map keyed by name. The off-hand touch raycast now
iterates the map, with per-button hover state so adjacent presses don't
cascade.
* uiState centralizes recording / sessionActive / sessionId / training
flags previously scattered across recordingState + ad-hoc. Server-side
state mirrors in from /ws/live snapshots each frame, so the SESSION
button's "active" color reflects whatever the server thinks, even if
another client (e.g. the desktop UI) started or ended the session.
* runTrain() decides what to train on with this priority:
1. active session's captures
2. fallback: most recent capture only
Edge cases (no captures, training in flight, recording in flight) all
surface human-readable messages in the status strip rather than failing
silently.
* drawStatus()/setStatus() paints the status strip with an alpha-fade
over STATUS_FADE_MS (6s) so old messages don't pile up but you have
time to read the result of a training run.
* REC button's label flips between "REC" (idle) and "STOP" (recording),
so glance-readable from the headset.
Pinch-to-record stays as a hands-free fallback for REC only (TRAIN and
SESSION are deliberately tap-only -- you don't want to accidentally
start a session by clenching your fist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v1.1 floating spheres turned out wrong in practice: no visible pointer
to aim with, and the user had to physically reach to where each sphere was
anchored. Real-headset feedback: "usually there is a pointer from the
controller or hand which I can use to navigate, which is gone in our app.
We need a proper menu within the vr app."
This rewrites the in-VR UI to the standard WebXR pattern:
* Flat menu panel (~46 x 20 cm, translucent dark plate) below the heatmap,
tilted ~18° toward the head so it's readable from a natural viewing angle.
Holds a 2x2 grid of rectangular labeled buttons:
REC SESSION
TRAIN EXIT VR
Plenty of room for more rows later (PREDICT toggle, gesture-label dropdown,
etc.) without re-layout.
* Ray pointers from each XR controller, via the canonical Three.js pattern:
renderer.xr.getController(i) returns an Object3D whose transform tracks
the corresponding XRInputSource's targetRaySpace. We attach a Line geometry
forward (3 m default, truncated to the hit point on hover) and a
selectstart listener that fires the hovered button's action.
* Pinch = select. With hand tracking, Quest maps a pinch to OpenXR's "select"
action, which surfaces as the standard XRInputSource select event. No more
custom proximity detection for buttons.
* Only the OFF-hand ray is rendered + active. The captured-arm ray would
clutter the view and risk hovering buttons while you perform a gesture.
The captured arm's pinch-to-record (1-second hold) still works as the
hands-free shortcut for REC.
* Hover-glow + active-state coloring drawn into each button's CanvasTexture:
gray idle, blue when the ray is over it, red when its underlying state
is "on" (recording for REC, session open for SESSION, training in flight
for TRAIN). REC's label flips REC <-> STOP per state.
* New EXIT VR button calls session.end() -- clean way out without taking
the headset off.
Cache-bust: script src is now `/static/vr/app.js?v=2`. Quest Browser
ignores `Cache-Control: no-store` for ES modules in some configurations,
which is why v1.1 didn't show up after a refresh. Bump this counter for
every future app.js change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The no-cache middleware whitelisted "/" and "/static/*" but missed the /vr route added in 7b1d2b4, so Quest Browser cached the VR HTML and refreshes kept loading stale JS (this is why v1.2 didn't appear on refresh -- the cached HTML still pointed at app.js with no version querystring). Add /vr to the whitelist. Bump app.js?v=3 to force a fresh fetch on whoever still has the stale HTML. Also: the off-hand had no visualizer in v1.2, just a ray emanating from nowhere. Per first-headset feedback ("I can't see the shape of my left hand to see which i'm grabbing"), add a parallel green-sphere visualizer for the off-hand alongside the captured-arm's blue spheres. Slightly larger spheres (8.5 mm vs 6 mm) so the pointing hand reads as more solid in the user's peripheral vision even when most of the ray is behind the fingertip. Refactor: updateArmVisualizer -> updateHandVisualizer(... , meshesMap) so the same logic drives both hands. Same for the hide path. Color legend now: BLUE = captured arm (the hand whose pose we're recording for ML) GREEN = off-hand (the hand that drives the menu) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…train Menu grows from a 2x2 to a 3x2 grid so the data->model->system workflow fits cleanly on one panel: Row 0 (toggles): REC SESSION PREDICT Row 1 (actions): TRAIN RECENTER EXIT VR PREDICT toggles inference at the server (POST /api/inference/enabled) and reflects whatever the server actually thinks via the inference field in /ws/live snapshots. Refuses cleanly with "no model loaded -- train one first" if no model is loaded. TRAIN now auto-enables inference after a successful activate (which already hot-swaps the new model into the engine). Server-side default is paused-on-load (commit bd1b68a) but in VR there's no obvious second click to do this -- pressing TRAIN already implies "I want this model running". Status strip surfaces "trained: R²=X · model loaded ✓ · predict ON" so the user knows the loop closed. RECENTER re-anchors the heatmap + menu + status strip to wherever the head is right now. Setting `placed = false` makes the next XRFrame re-run placeAnchors with the current viewer pose. Useful when the user shifts in their chair or moves around the room. Grid layout math now driven by MENU_COLS / MENU_ROWS constants so adding or removing buttons stays a one-line change. Bump app.js?v=4. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per first-headset feedback: "we need to see the inference after training to predict or infer what the gestures are and to see how big the difference is between reality and the prediction." Small panel between heatmap and menu, four columns (INDEX / MIDDLE / RING / PINKY -- thumb omitted since FlexGrid can't see it) each with a pair of vertical bars: GREEN R = ground-truth curl from XRHand joints on the captured arm AMBER P = model-predicted curl from the inference engine "Curl" is normalized [0..1] derived from wrist <-> finger-tip distance: when extended the tip is far from the wrist, when closed it's close. The per-finger max-extended distance is tracked dynamically so the metric adapts to the user's hand size without calibration -- first time you fully extend a finger, that becomes that finger's 0.0 reference. The model's predicted output (inference.piston_values in the WS snapshot) uses the same 25-joint x 7-float ordering the WebXR client sends in. So we extract predicted wrist + tip positions at the same indices XRHand uses and apply the same curl formula. Apples-to-apples. Panel is hidden when (a) no captured hand is tracked or (b) inference is paused / no model loaded -- nothing meaningful to compare. The moment PREDICT goes on the panel pops in, so you can train -> see-error-live without taking the headset off. Layout: menu pushed down 5cm (MENU_OFFSET_DOWN 0.23 -> 0.28) to make room for the 5cm-tall compare panel in the gap. Heatmap unchanged. Bump app.js?v=5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small UX wins from observing v1.5 in the headset: * Ray idle length 3 m -> 0.8 m, and a more muted blue. At 3 m the ray shot off into the room and read like a laser pointer; at 0.8 m it reads more as "where my finger is aimed." On hover the ray still extends to the hit point and turns amber, so the hover affordance isn't lost. Rename RAY_DEFAULT_LEN_M -> RAY_IDLE_LEN_M to make the intent obvious at the use sites. * Brief white flash on button activation (BUTTON_FLASH_MS = 180 ms). Priority is flash > active > hovered > idle in drawMenuButton so the confirmation lands regardless of what state-change the action triggers (which may not be visible immediately if the server is slow to respond). Label text inverts to dark on the white flash for legibility. * Pinch progress ring grows from 2.5-3.0 cm to 3.8-4.6 cm radius and brighter (0xfacc15 vs 0xfbbf24). Easy to spot at arm's length while you're focusing on the gesture itself. Bump app.js?v=6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a third joint visualizer alongside the existing real-arm (blue) and off-hand (green) -- amber, semi-transparent spheres rendered at the model's predicted joint positions. Visible only while PREDICT is on AND the captured hand is tracked. The ghost is anchored to the user's real wrist each frame: we offset every predicted joint by (real_wrist - predicted_wrist) so the visualization shows predicted SHAPE rather than absolute model output. This sidesteps the "model trained on absolute world positions, predicts positions where the recordings happened to be" confound -- the user sees their real hand and a ghost hand at the same place, the difference between them = the model's shape error in 3D rather than the abstract 0..1 curl scalar from the comparison bars. Rotation isn't aligned yet. A future iteration could match wrist orientations too (compute the quat that takes predicted-wrist-orientation to real-wrist-orientation, apply to all predicted joints). For now, shape-only is dramatically more useful than the raw absolute output and already gives the visceral "see how wrong the model is" feedback the user asked for. Bump app.js?v=7. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Windows .bat that takes you from "everything powered down" to "in VR
with /vr loaded" in one click. Useful both for development iteration
and for new contributors who want to try the VR side without learning
the adb-reverse incantation.
What it does, in order:
1. Locate adb (PATH first, then %LOCALAPPDATA%\Android SDK fallback)
2. Parse `adb devices` -- distinguishes "device" / "unauthorized" /
"offline" so the failure message tells you what to fix
3. Verify `openmuscle` CLI is installed (`pip install -e .`-ready)
4. Start `openmuscle web` in its own console window
5. Poll localhost:8000 for up to 20s until the server responds
6. `adb reverse tcp:8000 tcp:8000` so Quest sees localhost
7. `adb shell am start -a android.intent.action.VIEW -d <url>`
to open Quest Browser straight to /vr
Portable: uses %~dp0 for self-location, no hardcoded paths. Works
from any clone path. ADB lookup falls through PATH then Android SDK
default so it works with MQDH installs and chocolatey installs alike.
Optional first arg picks arm: `start-vr.bat right` (default) or
`start-vr.bat left`. The arg flows into `?arm=...` on the /vr URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The VR/Quest companion shipped across 14 commits but none of them updated the doc surfaces a contributor checks first. Close that gap: * README.md (root): add a VR-companion subsection under Quick Start pointing at start-vr.bat + the manual paths, add the --ssl flags row to the CLI table, add docs/vr-setup.md to the Documentation links. * pc/src/openmuscle/web/README.md: new "VR companion (/vr)" section documenting the architecture (why WebSocket-inbound when everything else is UDP), the new endpoints (/vr, /ws/quest, /ws/quest payload), the quest_hand recording specifics (window_ms default, lazy label_count, labels-schema sidecar, label_source meta tag, auto-pick preference), the WebXR client file structure, the ?v=N cache-bust contract, and the auto-enable-inference-on-train policy. Update Known Gotchas with the cache-related lessons learned (/vr was missing from middleware; Quest Browser ignores no-cache for ES modules; WebXR needs a secure context). * docs/protocol.md: clarify the versioning policy -- adding a new `type` like quest_hand is non-breaking under v1.0, since existing parsers ignore unknown types and the envelope is unchanged. Bump to "1.1" only when the envelope itself changes. No code changes in this commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review nit-fix: the joint-flatten ordering was implicit in two places
(`ingest_quest_packet` did `extend(pos); extend(rot)`, and
`_write_labels_schema` reconstructed the column index as
`ji * n_floats + ci` with its own local `ordering` list). Changing
one would silently break the other -- the sidecar would lie about
what's in the CSV.
Centralize:
* `QUEST_JOINT_CHANNEL_ORDER = ("px","py","pz","rx","ry","rz","rw")`
at module level, with a docstring naming both call sites.
* `_flatten_quest_joint(pos, rot)` -- used by ingest_quest_packet to
pack one joint in the canonical order.
* `_quest_label_column(ji, ci)` -- the inverse view, used by
_write_labels_schema to compute column indices.
Both call sites now route through these helpers, so the coupling is
structural rather than two aspirational comments. Verified the flatten
output is byte-identical to the old direct-extend code with a quick
smoke test (1,2,3 + 4,5,6,7 -> [1..7]).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Team feedback on the v1.7 ghost-hand: "the wrist-position-only ghost
looks broken in demo recordings. Ship in this PR if it's really 3 lines.
If it turns into a rabbit hole, defer to v1.8." This is v1.8 -- about
12 lines of additional Three.js quaternion math, not a rabbit hole.
Transform each predicted joint from "predicted wrist frame" into "real
wrist frame":
pos_in_real_frame = real_wrist_pos +
delta_quat * (pred_pos - pred_wrist_pos)
delta_quat = real_wrist_quat * inverse(pred_wrist_quat)
Same convention WebXR uses (XRJointPose.transform.orientation is a
quaternion in (x, y, z, w) order, matching Three.js's THREE.Quaternion
constructor). Allocations are reused per-frame (six module-level
Vector3 / Quaternion temporaries) so the XR loop doesn't churn.
Graceful degradation: if the wrist quat is missing for a frame (tracking
hiccup), falls back to position-only alignment, same as v1.7.
Bump app.js?v=8. Visual validation requires the headset -- the math is
the standard delta-quaternion approach but the ghost may need fine-
tuning if Quest's joint orientation convention is rotated relative to
the world (e.g., wrist forward axis differs from what we expect). If
demo recordings show the ghost mirrored or rotated 90°, swap the
multiply order or apply a fixed correction quat -- but typical Quest
hand-tracking joints follow the OpenXR convention which matches this.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review: three pytest cases requested before merge.
tests/test_storage.py::TestCaptureWriterLazyHeader
* test_label_count_inferred_from_first_row -- None hint, write_row with
175-element label_values, header gains 175 label_N columns.
* test_explicit_label_count_still_works -- LASK5 path unchanged
(label_count=4 -> 4 label columns).
* test_close_on_empty_capture_writes_header -- empty capture still
produces a header-only file (preserves "consumers never see a
zero-byte CSV" invariant from before the lazy refactor).
* test_close_on_empty_capture_with_none_count -- empty + None hint
falls back to 0 label columns rather than crashing.
tests/test_quest_ingest.py::TestIngestQuestPacket
* test_full_payload_registers_device -- 25-joint frame -> device in
AppState.devices with type "quest_hand" and 175-element values.
* test_data_values_use_canonical_channel_order -- pins the layout
against QUEST_JOINT_CHANNEL_ORDER + _flatten_quest_joint so a
future re-order of one immediately breaks the test.
* test_empty_joints_drops_silently -- tracking-lost frames (empty
or missing 'joints' key) don't create zombie devices.
* test_missing_pos_rot_defaults_to_identity -- partial joint dicts
still produce 7 floats (zero pos + identity quat).
* test_default_device_id_when_missing -- payload without device_id
falls back to "quest-01".
* test_repeated_packets_increment_packet_count -- packet counter
advances per frame.
Full suite: 23 passed (4 matcher + 9 protocol + 4 storage + 6 ingest).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two forward-looking notes from review: 1. TODO comment in `_write_labels_schema` docstring naming the "wrist-relative label coordinates" follow-up explicitly. Right time to do that work is when we want a portable model that generalizes across capture-locations, not now. Will track as a GitHub issue in the OpenMuscle-Software repo separately. 2. `docs/protocol.md` notes that the current single-hand `handedness` string can grow to `"both"` in a backward-compatible way: payload widens, schema sidecar grows a per-column hand field or parallel joint-name lists. Captured here so a future refactor of `_flatten_quest_joint` / `_write_labels_schema` doesn't accidentally shape itself in a way that prevents the extension. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mkcert-generated certs for the VR /vr page over LAN are per-machine (they encode your specific LAN IP and hostname) and not useful to anyone else. They're not secret in the cryptographic sense either -- the mkcert root CA only trusts certs signed by your local install -- but there's no reason to push them. Catch the whole class with *.pem since nothing in the repo today is a checked-in PEM. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validated end-to-end on real hardware 2026-05-29. The original docs assumed the standard Android "Settings -> Security -> Install a certificate" path was reachable from Horizon OS's consumer Settings panel. It isn't -- Meta has hidden it. The Settings UI you see when tapping the gear icon in VR is Horizon Settings, not AOSP Settings. Document the actual workaround we hard-won: * Push the cert via `adb push`, then launch the AOSP Security Dashboard activity directly via ADB with the right backslash- escaped activity name (the $ must survive both PowerShell single quotes AND the Android shell, hence \$). * Activity name is SecurityDashboardActivity, not SecuritySettings- Activity (that's the older Android name and doesn't exist on current Horizon OS -- enumerated via `dumpsys package com.android.settings`). * The 2D panel often launches but isn't auto-foregrounded -- find it in the Meta-button universal-menu app switcher. * The file picker that opens for cert selection defaults to "Recent" which is empty on first install; user has to hit the hamburger menu and navigate to Internal Storage -> Download manually. * Include the table of working / non-working Settings activities so future Quest dev (when other AOSP menus are similarly hidden) doesn't have to re-enumerate from scratch. Also a cortex note logged with the same procedure (tags: openmuscle,quest3,webxr,mkcert,adb,horizon-os,gotchas) so the knowledge survives outside this repo too. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Field-capture mode. Wear the headset while doing real activities (cooking, typing, woodworking) and capture FlexGrid + Quest hand-tracking + a video of what you're doing, so the model trains on natural movement rather than deliberate in-VR gestures. Three pieces: 1. ?mode=ar|vr URL param toggles between immersive-vr (existing default, black background, deliberate-gesture training) and immersive-ar (passthrough, real world visible behind our floating panels, field capture). XR_SESSION_TYPE flows through preflight check, scene background nulling, and the VRButton/ARButton swap. local-floor goes from required (VR) to optional (AR -- passthrough gives us the floor we need without explicit anchoring). 2. Sync slate: SYNC_SLATE_MS (2.5 s) high-contrast splash that pops up when REC starts, centered 55 cm in front of the user. Big yellow plate with black text -- visible in passthrough AND in the headset's screen recording. Shows the capture filename + Unix-ms timestamp so a re-watched video can be paired to a CSV row by timestamp without guessing. 3. Header strip filename + ms clock while recording: REC * 14:32:05.123 * capture_X.csv (previously was REC * N rows * match X%). The new format gives the operator a running sync marker every frame they happen to glance at the heatmap during a real-world capture session. Idle header text (FlexGrid Hz / Quest Hz) is unchanged. start-vr.bat gains a second optional arg: `start-vr.bat right ar` for field capture, `start-vr.bat right` (or no args) for the VR default. URL is properly quoted in echo so the & between query params doesn't trip cmd. Bump app.js?v=9. The "tap REC * see SYNC splash * do your activity * tap STOP * pull MP4 off headset * pair by filename" loop is now a thing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…posites
v1.9 shipped with renderer alpha disabled, so even when Quest granted
immersive-ar the WebGL framebuffer was opaque and passthrough had
nothing to composite behind. User report: "still shows black background"
after loading ?mode=ar.
* WebGLRenderer({ alpha: true }) unconditionally (cheap in VR mode too,
we just paint a solid background ourselves)
* renderer.setClearAlpha(0) when MODE === 'ar' as a belt-and-suspenders
explicit transparent clear
* Log the granted XRSession's environmentBlendMode on sessionstart so
next time we can tell "Quest fell back to VR" vs "renderer setup wrong"
from one console line ('opaque' = VR, 'alpha-blend'/'additive' = AR)
Bump app.js?v=10. Refresh ?mode=ar on the headset to pick up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First-headset report on passthrough mode: "the buttons are super big
and in the center" -- which is exactly what you'd expect when the same
panel sizes that read as "comfortably in front of me" in VR (no
real-world reference) are dropped against an actual kitchen / desk /
workshop. In passthrough the user has scale anchors, and 40cm panels at
70cm distance feel enormous.
Two coupled adjustments only when MODE === 'ar':
HEATMAP_FORWARD_M : 0.70 -> 1.10 (panels further away, halves
angular size in the user's view)
UI_SCALE : 1.0 -> 0.65 (uniform scale on heatmap, header,
compare, status, menu groups)
UI_SCALE also multiplies the inter-panel offsets in placeAnchors
(MENU_OFFSET_DOWN, COMPARE_OFFSET_DOWN, STATUS_ROW_DOWN, and the
heatmap's drop-below-eye-height) so gaps shrink proportionally --
without that, small panels would have weirdly large empty space
between them and look broken.
The SYNC slate (centered 55cm in front of head at REC press) stays
full-size deliberately -- it's *supposed* to dominate the 2.5s
video-sync frame so a re-watched screen recording has an obvious
sync marker.
Reminder for the user: the RECENTER button in the menu re-anchors the
panel cluster to wherever the head is pointing right now, which is
useful in AR mode if you've moved around your workspace since session
start.
Bump app.js?v=11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Field-capture feedback: "we will need to be able to move the menu and
also collapse it when recording... it will just be one button for
stopping the recording. Also we will need to be able to move the
sensor array visualization too. These pieces will be integeral to
being able to do work with the headset on and recording."
Three coupled changes:
1. Group restructuring
- infoGroup (new Group) wraps heatmapMesh + headerMesh + compareMesh.
The "data display cluster" moves as one unit. Children positions are
now local-space offsets from the group origin (= heatmap center).
- menuRoot (new Group) wraps menuPanel + statusMesh so the action UI
moves as one. statusMesh's vertical offset captured as a fixed
local offset; placeAnchors no longer positions it independently.
- UI_SCALE is now applied to the two groups directly in placeAnchors
rather than to each leaf mesh. Cleaner; children inherit.
2. Drag handles + drag logic
- HANDLE_SIZE_M (3 cm) cubes attached to top-left corner of infoGroup
and menuRoot, exposed as raycast targets. Hover-amber, grabbed-emerald.
- Off-hand ray + pinch on a handle starts a drag. Pinch release ends
the drag and re-aims the panel toward the head's current position
(so a panel dragged behind your back lands facing you, not away).
- Reusing the existing select / selectend XR events on the controllers;
dragState is mutually exclusive with button activation -- if you
pinch while hovering a handle, you drag; if hovering a button, you
click; raycast picks geometrically nearest hit.
3. Collapse-to-STOP while recording
- New stopButtonMesh (18 x 10 cm big red button) lives as a child of
menuRoot at the same world position as menuPanel.
- setRecordingCollapsedUI(isRecording) called every frame from the XR
loop: menuPanel.visible = !recording, stopButtonMesh.visible =
recording. Mutually exclusive.
- STOP button is a first-class raycast target -- pinch fires
toggleRecording (= stop). No need to find a small button in a 3x2
grid mid-activity.
- Status strip stays visible in both states; useful save/train
feedback applies regardless.
Behavior of the existing RECENTER button is unchanged but more useful
now: it sets placed=false, which re-runs placeAnchors and resets BOTH
group positions + the slate position to defaults relative to the head's
current pose. Use it if you've dragged things into a corner and want a
clean slate.
Bump app.js?v=12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Created Open-Muscle/OpenMuscle-AR as the discoverability anchor + future home for AR/VR work. The current WebXR client stays here in pc/src/openmuscle/web/static/vr/ for now (tight coupling to the FastAPI server), but the new repo gives the AR side a findable URL and a clear roadmap for v2 (native Quest APK with BLE-direct bracelet) and beyond. Add it to the Related Repositories section under a new AR/VR heading, linking to its ROADMAP so contributors looking at this repo see the AR work exists and where it's headed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
start-vr.bat already covers the USB-tethered + adb-reverse + plain
HTTP path (fast iteration). This new launcher covers the other half:
HTTPS over LAN, no cable, for real field-capture sessions.
What it does:
1. Verify openmuscle CLI on PATH
2. Verify vr-cert.pem + vr-key.pem exist next to it (helpful error
linking to the mkcert wiki page if not)
3. Sniff a non-loopback LAN IPv4 via PowerShell and print the
exact headset URLs (both /vr and /vr?mode=ar) so the operator
doesn't have to look up the IP separately
4. cd into pc/ and run `openmuscle web --ssl-certfile vr-cert.pem
--ssl-keyfile vr-key.pem`
No adb-reverse, no Quest Browser auto-open -- the whole appeal of
HTTPS-LAN is being away from the PC, so user types the printed URL
into Quest Browser themselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First-headset failure mode report: "I moved from where I was and the headset wanted to define a new boundary. When this happened it seemed to glitch our code and mess up." What was happening: Quest pauses the WebXR session whenever system UI takes over (boundary redraw, universal menu, notifications). Visibility state goes 'visible' -> 'visible-blurred'. Our frame loop kept firing but viewer/joint poses can return null or stale data, so we kept feeding garbage frames to /ws/quest and the recorder accumulated bad rows. On resume, our world-anchored panels stayed put even though the user's actual position in the room had shifted, leaving the menu floating somewhere the user wasn't. Three additive fixes: 1. Track xrPaused via session.visibilitychange. While paused, the frame loop short-circuits before pose sampling -- just renders the static scene + returns. No joints sent, no hand visualizers updated, no raycast or button activations. Skipping these is the difference between clean pause + resume vs corrupted-recording-then-glitch. 2. On resume to 'visible', force placed=false so the next frame re-runs placeAnchors with the current head pose. The panel cluster pops to wherever the user is looking now -- exactly what you'd want after walking ten feet to redraw a boundary. 3. On sessionend with an active recording, send DELETE /api/recording so the server's ActiveCapture closes cleanly. Without this the server thinks recording is still going forever (or until the next server restart), and a subsequent REC tap would fail "Already recording -- stop the current capture first" with no obvious way to recover from inside the headset. Fire-and-forget; if the server is unreachable simultaneously (Wi-Fi went out at the same time) we log and move on. Status strip surfaces both transitions: "session paused (visible-blurred) -- boundary redraw or system UI" "session resumed after 12.3s -- re-anchoring UI" So the operator scrubbing the screen recording later can see exactly when Quest paused and how long it took, paired against the CSV's gap in timestamps. Bump app.js?v=13. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Field-capture sessions want a stable workstation layout. Previously every page reload, every Quest cert reset, every session resume after a long pause would snap the heatmap + menu back to their default positions and the user had to re-drag everything into their preferred workspace arrangement. Now: every drag-end serializes infoGroup + menuRoot pose to localStorage under a per-MODE key (openmuscle-vr-layout-vr or -ar). On next placeAnchors call -- whether that's a fresh session start or a post-pause re-anchor -- we apply the saved layout if present, else fall back to default positioning. The user's preferred field-capture station persists across sessions, browser reloads, Quest reboots, mkcert renewals, anything. Per-MODE key because VR and AR have meaningfully different ergonomics: in VR you might want the menu front-and-center; in AR field capture you probably want it shoved to the periphery so it doesn't block your real-world view of the stove / desk / whatever you're capturing. RECENTER button now clears the saved layout in addition to setting placed=false, so it's a real "I'm lost, start over" gesture. Without that, a user who hit RECENTER, then closed the tab, would come back next session to the same lost layout. Slate position is computed fresh per-session inside placeAnchors regardless of saved layout, since the sync slate should always pop up in front of wherever the user is at REC press, not at some previously-saved location. Bump app.js?v=14. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three real edge cases found reviewing the field-capture work, all of which Tory could plausibly hit during a first real test session: 1. Orphaned drag state. endDrag() was only called from the controller selectend event. If the controller disconnected mid-drag (hand tracking lost while holding a panel) or the session paused (boundary redraw) mid-drag, selectend never fired -- so dragState stayed set, the handle stayed stuck GRAB-green, and updateDrag() kept gluing the panel to a frozen/stale controller position. Added cancelDrag() and call it from both the controller 'disconnected' handler and the visibilitychange pause-start path. It releases the panel in place (no head re-orientation, since head pose is unreliable during a pause) and persists the layout. 2. Dishonest sessionend logging. The auto-stop-recording-on-sessionend path (added in v1.11) awaited the DELETE but never checked r.ok, so it logged "auto-stopped active recording" even when the server returned 400 (state drift -> "not currently recording"). Now checks r.ok and logs the HTTP status honestly when it isn't. 3. NaN panel positions from corrupt localStorage. _applyPoseJSON only checked that pose.p / pose.q existed, not that they were finite numbers of the right length. A corrupt / partially-written / schema- drifted layout payload would set NaN positions and fling a panel to an unreachable spot the user couldn't even find to drag back. Added _allFinite() validation (length + Number.isFinite on every element), reject degenerate near-zero quaternions, normalize the quat, and require a positive finite scale. Invalid payloads fall through to default positioning instead of corrupting the scene. Also swept the last 3 em-dashes out of app.js string literals / comments (house style: no em-dashes in shipped files). Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Tory's goal for the VR app is "an easy to use application he can test
with the hardware and testing scenarios." This is the runbook that
serves that directly: a fixed bring-up order, a 10-step 2-minute smoke
test that needs only the headset (no FlexGrid), and six independent
per-feature scenarios (gesture training, AR field capture, pause/
boundary-redraw recovery, collapse-to-STOP, panel persistence, predict
+ ghost hand) each with explicit expected results and what-to-watch-for.
Ends with an acceptance-bar checklist ("what good looks like") and a
problem-reporting template pointing at the right repo + the wiki
troubleshooting page.
Linked from the root README docs index. No em-dashes (house style).
Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Self-review (iteration 4) found two canvases re-uploading to the GPU every frame when their content rarely changes. Each `tex.needsUpdate = true` forces three.js to re-upload the whole canvas next render. On Quest's mobile GPU at 72fps this is wasted bandwidth competing with hand tracking -- and it happens during recording, exactly when smooth tracking matters most. 1. drawStopButton(): the collapsed STOP button is visible for the entire recording but its look only flips on hover enter/leave. It was redrawing + re-uploading a 512x280 texture every frame. Now guarded by _stopButtonLastHovered -- only redraws when the hover state actually changes (so ~0 uploads/sec instead of 72). 2. drawStatus(): called every frame from updateFromSnapshot to animate the fade, but it kept clearing + uploading the 800x90 texture forever, even long after the text fully faded to nothing. Now guarded by _statusFadeDone -- once alpha hits 0 we do one final cleared upload and then no-op until the text changes. setStatus() resets the flag so re-issuing even an identical message re-flashes it. No behavior change, purely fewer GPU uploads. drawHeatmap (signature- gated) and drawCompare (live data, legitimately per-frame) were already fine; drawHeader's idle-Hz redraw is lower impact and left for a later pass. Bump app.js?v=16. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Real correctness bug found in iteration 5's state.py review, the kind that would quietly poison field-capture training data. Root cause (client): captureAndSend skipped joints whose pose was momentarily unavailable (hand partly out of camera FOV -- common during real activities). That made the quest_hand payload a VARIABLE length frame to frame, and worse: when a middle joint dropped, every later joint shifted into the wrong slot. The server writes those slots straight into CSV columns, so this produced (a) ragged CSVs (different column count per row -> breaks pandas.read_csv) and (b) silently misaligned ground-truth labels (column N means different joints in different rows). Fix, two layers: * Client (app.js): emit ALL joints in canonical JOINT_NAMES order every frame. For an unavailable joint, send zero pos + identity quat with valid:false instead of omitting it, so the array length and per-slot meaning are stable. Drop the whole frame only when NO joint is valid. The validity flag rides in hands.joints (preserved in the JSONL sidecar); the flat values / CSV carry only the numbers. * Server (state.py): defense in depth. Lock the label width at the first label packet (same width the labels-schema sidecar describes) and pad/truncate every paired row to it, so the CSV stays rectangular even if some client sends variable-length frames. Count mismatches (label_width_mismatch) and surface them in the stop-recording result + a warn log so a partial-tracking session is visible, not silent. docs/protocol.md: document the fixed-length contract + the valid field. Tests: new tests/test_quest_recording.py drives interleaved 25/20/25-joint frames through a real recording and asserts the CSV is rectangular, the schema sidecar matches the CSV width, and exactly one width-mismatch was counted. Full suite 25 passed. Bump app.js?v=17. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Completes the planned server-side coverage so the quest_hand recording contract is regression-guarded: * test_quest_window_defaults_to_175ms / test_lask5_window_defaults_to_100ms -- per-device-type match window picked when window_ms is None. * test_explicit_window_overrides_default -- caller arg wins. * test_auto_pick_prefers_quest_over_lask5 -- AUTO_LABEL_TYPE_PREFERENCE picks the richer Quest label vector when both sources are connected. * test_label_source_meta_tag_quest / _lask5 -- the capture meta sidecar's auto.label_source + auto.window_ms reflect the actual label source, so the Captures panel and downstream training can keep Quest-labeled and LASK5-labeled datasets separable. Each test calls stop_recording() before the temp dir is cleaned up -- on Windows an open CSV/JSONL handle blocks TemporaryDirectory deletion. Full suite now 31 passed (4 matcher + 9 protocol + 6 ingest + 8 recording + 4 storage). Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
So the operator can judge data quality mid-session without taking the
headset off or checking the PC -- directly serves the "easy to test"
goal.
While recording, the floating header now shows:
* a quality DOT colored by the live sensor<->label match rate:
gray = warming up (< 10 sensor frames seen)
green = match rate >= 70% (good)
amber = 40-70% (marginal)
red = < 40% (poor pairing -- something's off)
* the match percentage appended to the clock + filename
* a "JOINTS DROPPING" suffix (and amber dot) when the server has had
to zero-pad joint columns because partial hand tracking sent short
frames -- the live counterpart to the v1.15 fix, so the operator
SEES tracking degrade instead of finding zero-filled joints later.
Server: _snapshot's recording dict now carries label_width_mismatch so
the client can surface it live (it was previously only in the stop
result).
Two supporting touches in drawHeader:
* auto-fit font size (shrinks 32->16px) so long capture filenames no
longer overflow the canvas edges.
* a redraw guard keyed on (text, recording, quality) -- the idle
"X Hz · Y Hz" header was re-uploading its texture every frame even
though it changes ~1/sec. (The recording header still redraws every
frame because its ms clock genuinely changes; that's intended.)
Bump app.js?v=18. 31 tests still pass.
Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Add a "Reading capture quality" subsection + a note in the smoke-test step 9, so a tester knows what the recording-header quality dot colors mean (gray warming-up / green good / amber marginal / red poor) and what "JOINTS DROPPING" indicates. Keeps the runbook in sync with the v1.16 in-VR quality gauge. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Contributor
Author
Update: robustness + field-capture hardening (v1.11 → v1.16)Picking up after the second-round real-headset testing, this batch focused on making the app safe to hand to a tester and run real field-capture sessions. All on Correctness / data integrity
Robustness / UX for testing
Tests + docs
Two notes for review
|
Read-through (iteration 10) of the comparison/ghost-hand math found two issues that bite exactly when the model is bad -- i.e. early in training, which is most of the time during testing. 1. Shared curl auto-calibration. curlFromDistance auto-learns each finger's "fully extended" wrist->tip distance, but realCurls and predictedCurls fed the SAME fingerMaxExtended array. Real distances are physical (~0.10 m); a poorly-trained model's predicted distances can be wildly off-scale, and one bad predicted frame would pollute the shared max and distort the REAL bars too -- right when you need the real bars to stay trustworthy to judge the model. Split into separate fingerMaxExtendedReal / fingerMaxExtendedPred arrays; each curl source now takes its calibration array explicitly. 2. Ghost-hand degenerate quaternion. The wrist-frame alignment inverts the model's predicted wrist quaternion. A bad model can output a near-zero quaternion, and inverting that divides by ~zero -> NaN -> the whole ghost hand jumps to NaN positions. Guard qLenSq > 1e-6: below that, fall back to position-only alignment (still useful). Also normalize the predicted quat and use a dedicated inverse temp (_ghostPredQuatInv) instead of .clone().invert(), dropping a per-frame allocation in the XR loop. No behavior change for a well-trained model; for a bad one the REAL bars stay accurate and the ghost hand stays on-screen instead of distorting/vanishing. Bump app.js?v=19. 31 tests still pass. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Final read-through (iteration 11) found a real leak. The /ws/quest and /ws/live sockets auto-reconnect 1.5s after any `close` event so a network blip doesn't kill the session. But sessionend DELIBERATELY closed both sockets -- which fired their close handlers, which reconnected them. Net effect: after you exit VR the page keeps a live /ws/quest + /ws/live connection open forever, reconnecting on a loop, and each enter/exit-VR cycle stacked ANOTHER reconnect loop on top. Server-side this shows up as endless quest connect/disconnect log spam after the headset comes off. Fix: standard want-open flag. connectQuestWS/connectLiveWS set _questWsWantOpen/_liveWsWantOpen = true; the close handler only reconnects while the flag is set. New closeQuestWS/closeLiveWS clear the flag first, then close + drop the reference, so an intentional teardown stays torn down. sessionend now calls these helpers instead of closing the sockets by hand. The error handlers are unchanged (error -> close -> reconnect is still correct -- a real drop should reconnect). Bump app.js?v=20. 31 tests still pass. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Two finds from iteration 12's review of detectPinchAndToggle + the menu button rendering. 1. Pinch repeat-toggle (functional bug). The captured-arm pinch-hold record toggle tried to "require release before next" by setting pinchStart = -Infinity after firing. But that makes `held` = now - (-Infinity) = Infinity, so the only thing stopping a re-fire was a 1.5s cooldown -- meaning a CONTINUOUS pinch-hold re-toggled recording every 1.5 seconds. Replace the -Infinity/cooldown hack with a pinchToggled flag that fires exactly once per hold and is only cleared by an actual release (a non-pinching frame). Dropped the now-unused uiState.lastPinchToggleAt. 2. Menu-button redraw waste (perf). updateButtonVisual calls drawMenuButton for all 6 buttons every frame, each doing a full canvas redraw + GPU texture upload -- but a button's look only changes on active/hover/flash transitions. That's 6 uploads/frame at 72fps for nothing. Add a per-button _lastDrawKey guard (same fix v1.14 applied to the STOP button + status strip; the menu buttons were missed). Flash still works: the flashing bool is time-derived so its true->false expiry flips the key and forces the clearing redraw. Bump app.js?v=21. 31 tests still pass. Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
…ze it Iteration 13 review: placeAnchors AR-scaling + slate timing checked out correct, but onXRFrame had no error guard. A single throwing frame (malformed /ws/live snapshot, an unexpected null pose, a bad predicted vector) propagating out of the setAnimationLoop callback can stop the loop and FREEZE the headset mid-session. On an untethered field-capture rig that means losing a real-work recording that can't easily be redone -- the worst failure mode for Tory's use case. Wrap the frame body (renamed onXRFrameImpl) in onXRFrame: catch, rate- limit a console.error (first error immediately, then <=1 per 2s, with a running count so a persistent error is obvious without flooding at 72fps), and ALWAYS render so the view stays live and the user can still reach EXIT / STOP. The error is logged, not silently swallowed -- it's just no longer fatal to the session. Bump app.js?v=22. 31 tests still pass (server unchanged). Co-Authored-By: turfptax-claude O4.8 <noreply@anthropic.com>
Contributor
Author
Update 2: review-pass bug fixes (v1.17 → v1.20)A systematic read-through of the WebXR client (which can't be unit-tested) surfaced several real bugs, all now fixed on
Net theme: the client was built fast across many iterations and had accumulated correctness + lifecycle bugs that only show up under real field-capture conditions (partial tracking, exit/re-enter, bad early models). These passes were specifically about making it safe to hand to a tester. Happy to walk through any of them. |
Closes the deferred item the team flagged in the first review ("the
4-piston comparator UI will need a future 'if quest_hand, show 3D hand
viewer' mode"). When a quest_hand device is streaming, the Live stage's
LASK5 piston comparator (which shows zeros for a hand source) is swapped
for a live 3D hand skeleton -- the desktop counterpart to the VR app's
ghost hand.
New static/hand-viewer.js (ES module, exposes window.OMHandViewer):
* Three.js scene (same CDN + version the VR client uses), 25-joint
skeleton with bones, drawn from the device's flat `values` array.
* Renders the REAL captured hand (green) and, when a quest-trained
model is running (inference.piston_values has >= 25*7 floats), the
PREDICTED hand (amber) overlaid.
* Both hands transformed into WRIST-LOCAL space (subtract wrist pos,
rotate by inverse wrist quat, with a degeneracy guard for bad-model
quaternions) so the hand shows in a canonical palm orientation and
REAL vs PRED is a direct shape comparison. (Same wrist-relative idea
tracked for training in issue #2, used here just for viz.)
* Auto-rotates; drag to orbit, double-click to resume auto-spin.
Per-frame allocation avoided (reused scratch vec/quat).
Wiring:
* index.html: importmap + module script + #hand-viewer container in
the comparator.
* app.js: renderHandViewer() in handleTick detects the quest_hand
device, lazy-inits the viewer, toggles .comparator.hand-mode (CSS
hides the pistons, shows the viewer), feeds real+pred flats, and
relabels the GT meta line "Quest hand - N joints - X Hz".
* styles.css: viewer panel + .hand-mode toggle + legend.
No server change -- the /ws/live snapshot already exposes a quest_hand
device's device_type + flat `values`, and inference.piston_values for
predictions.
Verified server-side end-to-end: all assets serve, the page references
the module, and a fake quest_hand device yields a snapshot with a
175-float `values` array that renderHandViewer consumes. The actual 3D
RENDER is unverified by me (no display in this environment) -- needs a
visual check on a real browser.
Co-Authored-By: turfptax-claude O4.8 <turfptax-claude@openmuscle.org>
The desktop Studio 3D hand viewer reads each device's flat `values` from the /ws/live snapshot. Add a test asserting a quest_hand device surfaces in _snapshot() as device_type 'quest_hand' with its full 25*7 flat joint vector, so a future snapshot refactor can't silently break the viewer's only data source. Suite now 32 passing. Co-Authored-By: turfptax-claude O4.8 <turfptax-claude@openmuscle.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a WebXR client at
/vrthat turns a Meta Quest 3 into a labeling rig, live demo, and field-capture device for the muscle→finger model. Quest hand-tracking joints become ML ground truth (25 joints × 7 floats ≈ 175-dim label vector, vs LASK5's 4-dim). The training pipeline, recorder, matcher, sessions, and captures system are unchanged — the Quest is integrated as a syntheticdevice_type: "quest_hand"via one synthesizer method (AppState.ingest_quest_packet).The loop now closes end-to-end for both training and field capture:
Validated on a real Quest 3S. Branch has 30 single-purpose commits — review trail intentionally preserved (per @-team feedback "your messages are good enough that the log is the changelog").
Day-1 requirements (from initial design review)
--ssl-certfile/--ssl-keyfile+ full operator procedure indocs/vr-setup.md)CaptureWriterlazy header-writewindow_msdefaults per device type (quest_hand=175,lask5=100)_handle_packet)<name>.labels.schema.jsonsidecarmeta.auto.label_source = "quest_hand"tagReview nit-fixes (addressed before second review round)
_joint_to_column_indexhelper —QUEST_JOINT_CHANNEL_ORDERconstant +_flatten_quest_joint+_quest_label_column, both call sites coupled structurallytest_storage.py+test_quest_ingest.py, full suite: 23 passed_write_labels_schemafor wrist-relative labels follow-updocs/protocol.mdField-capture mode (added after second-round real-headset testing)
The bigger second round — turning the WebXR client from a deliberate-gesture training rig into a wearable data-capture device for natural activities.
?mode=ar):immersive-arsession with WebGLRendereralpha: trueso passthrough composites behind our floating panels. URL-toggleable; VR mode preserved for deliberate gesture training.ARButtonswap in three.js boot.SYNC: capture_X.csv · t = <unix_ms>visible for 2.5s, centered in user's primary view. Frame-accurate video↔CSV pairing point when the operator reviews the headset's screen recording later.REC · N rows · match X%withREC · 14:32:05.123 · capture_X.csvso any frame where the user happens to look at the heatmap during a real-world session is a self-contained sync anchor.infoGroup, menu+status wrapped inmenuRoot. Each gets a drag handle (3 cm cube, top-left of each cluster); off-hand ray + pinch grabs, release drops and re-faces toward user. While recording, the full 3×2 action menu collapses to a single big red STOP button anchored at the menu position. Status strip stays visible in both modes.Architecture in one line
Browsers can't speak UDP, so the WebXR client posts hand-joint frames to a new
/ws/questWebSocket. The server synthesizes each frame into anOpenMusclePacket(device_type="quest_hand")and hands it to the existing_handle_packet. From the recorder/matcher/snapshot's view, the Quest is just another device. One new endpoint, one synthesizer; the rest of the pipeline rides for free.What's in the diff (by surface)
pc/src/openmuscle/web/state.pyingest_quest_packet,_write_labels_schema, Quest-awarestart_recording,QUEST_JOINT_CHANNEL_ORDER+ helpers (structural coupling between flatten/schema)pc/src/openmuscle/web/app.py/vrroute,/ws/questinbound,/vradded to no-cache middleware (caught Quest Browser cache bug)pc/src/openmuscle/data/storage.pylabel_count: Optional[int]so 175-dim Quest captures Just Workpc/src/openmuscle/cli.py,web/app.py::serve--ssl-certfile/--ssl-keyfileflags plumbed into uvicornpc/src/openmuscle/web/static/vr/{index.html, app.js, styles.css}web/README.mdpc/start-vr.batarmandmodeargs. Portable (no hardcoded paths).pc/tests/test_storage.py,pc/tests/test_quest_ingest.pydocs/protocol.md,docs/vr-setup.md, rootREADME.md,pc/src/openmuscle/web/README.md*.pemso dev TLS certs don't accidentally shipmkcert procedure (the unexpected gotcha)
Documenting because it was hard-won (full procedure in
docs/vr-setup.md):$MUST be backslash-escaped through both PowerShell single-quotes AND the Android shell.docs/vr-setup.mdso future Quest dev doesn't have to re-discover viadumpsys.How to test
In VR (gesture training): set both controllers down → 3-checkmark preflight → Enter VR → tap SESSION → REC → curl a finger → STOP → repeat → TRAIN → ghost hand + bars appear.
In AR (field capture): same flow but with passthrough visible. Drag panels via the handles to wherever fits your workspace. Tap REC to start a real-activity session — menu collapses to a single STOP button, sync slate flashes for video pairing, header strip shows running filename + ms clock. Quest's built-in screen recorder captures the video; pair by filename post-hoc.
Full operator walkthrough:
docs/vr-setup.md.Where to start review
pc/src/openmuscle/web/state.py— synthesizer + helpers (~200 LOC of real change)pc/src/openmuscle/data/storage.py— lazy header refactor (~25 LOC)pc/src/openmuscle/web/README.mdnew "VR companion" section — architecture writeup; read before diving into the WebXR clientpc/src/openmuscle/web/static/vr/app.js— WebXR client (~1700 LOC). Sections: scene init, drag/raycast, button factory, frame loop, REST handlers, sync slate, ghost hand, comparison panel. README section names every subsystem.docs/vr-setup.md— operator guide. Worth reading because the mkcert procedure is non-obvious.Test plan
pytestpasses (10 new tests for lazy writer + ingest synthesis)start-vr.batlaunches end-to-end on a clean clonedocs/vr-setup.mdend-to-end on a clean headsetDeferred (with owner's blessing)
_write_labels_schema, GitHub issue to file separately. Right time is when training a portable model that needs to generalize across capture-locations.quest_handcaptures — its own scope, Studio v1.x enhancement.pc/start-vr.batis Windows-only..shmirror is a 5-min add when needed.Branch state
🤖 Generated with Claude Code