Problem
quest_hand recordings currently store joint positions in absolute world coordinates as captured by the headset's XRReferenceSpace. A model trained on these labels learns to predict positions where the recordings happened to be — generally not where the user is at inference time.
The VR ghost-hand visualization papers over this by anchoring the predicted joints to the real wrist each frame, but the underlying CSV semantics are not portable across:
- Different recording locations (the user moves their VR play area)
- Different recording sessions (the user re-centers between sessions)
- Inference on a fresh location at all
In other words: today's quest_hand captures are demo-friendly but not training-portable.
Proposed fix
Before writing each label row to the CSV, transform every joint pose from world space into the wrist's local frame:
- Subtract the wrist position so all positions are relative to the wrist (translation invariance)
- Optionally rotate into the wrist's frame so all rotations are relative to the wrist's orientation (rotation invariance)
At inference, reverse the transform: the model predicts wrist-relative poses, and the renderer applies the live wrist transform to place the predicted hand in world space.
Scope
This is not a backward-compatible change to the CSV format — every previously captured quest_hand CSV is in world coordinates. Options:
- A. Land the change behind an opt-in flag (
coord_frame: "world" | "wrist_relative") and stamp it in meta.auto.label_coord_frame so trainers can filter. Old captures still trainable as-is (just not portable). New captures default to wrist_relative.
- B. Add a one-shot post-processing tool (
openmuscle convert-quest-to-wrist-relative <capture>) that rewrites the CSV in place and updates the labels-schema sidecar to flag the transformation. Lossless if the wrist pose is also recorded.
- C. Both: ship A as the new default, ship B as the migration path for existing captures.
I'd recommend C so we can finally use the captures already on disk.
Where the workaround lives today
The ghost-hand viz handles this in JS at render time: it reads the predicted joints from the inference output and re-anchors them to the live wrist transform each frame. That logic stays useful even after the fix lands — it becomes the inverse transform applied to wrist-relative predictions instead of an ad-hoc world-coords correction.
Source pointer
TODO comment lives in pc/src/openmuscle/web/state.py inside _write_labels_schema (right above the schema-writing logic). Link this issue from there once we have a number.
When to do this
Not blocking the v1 Quest companion shipping. Right time to do this work is when we train a model intended to generalize across recording locations or run inference at a fresh location — none of those is on the immediate roadmap. Track this so it doesn't get lost.
Labels
enhancement, quest_hand, ml-pipeline
Problem
quest_handrecordings currently store joint positions in absolute world coordinates as captured by the headset's XRReferenceSpace. A model trained on these labels learns to predict positions where the recordings happened to be — generally not where the user is at inference time.The VR ghost-hand visualization papers over this by anchoring the predicted joints to the real wrist each frame, but the underlying CSV semantics are not portable across:
In other words: today's
quest_handcaptures are demo-friendly but not training-portable.Proposed fix
Before writing each label row to the CSV, transform every joint pose from world space into the wrist's local frame:
At inference, reverse the transform: the model predicts wrist-relative poses, and the renderer applies the live wrist transform to place the predicted hand in world space.
Scope
This is not a backward-compatible change to the CSV format — every previously captured
quest_handCSV is in world coordinates. Options:coord_frame: "world" | "wrist_relative") and stamp it inmeta.auto.label_coord_frameso trainers can filter. Old captures still trainable as-is (just not portable). New captures default towrist_relative.openmuscle convert-quest-to-wrist-relative <capture>) that rewrites the CSV in place and updates the labels-schema sidecar to flag the transformation. Lossless if the wrist pose is also recorded.I'd recommend C so we can finally use the captures already on disk.
Where the workaround lives today
The ghost-hand viz handles this in JS at render time: it reads the predicted joints from the inference output and re-anchors them to the live wrist transform each frame. That logic stays useful even after the fix lands — it becomes the inverse transform applied to wrist-relative predictions instead of an ad-hoc world-coords correction.
Source pointer
TODO comment lives in
pc/src/openmuscle/web/state.pyinside_write_labels_schema(right above the schema-writing logic). Link this issue from there once we have a number.When to do this
Not blocking the v1 Quest companion shipping. Right time to do this work is when we train a model intended to generalize across recording locations or run inference at a fresh location — none of those is on the immediate roadmap. Track this so it doesn't get lost.
Labels
enhancement,quest_hand,ml-pipeline