[wip] mujoco xr as a quick robot visualizer - #869
Draft
jiwenc-nv wants to merge 7 commits into
Draft
Conversation
Render a MuJoCo scene stereoscopically into a Televiz XR session, with the operator's controllers drawn as markers. Single process, single thread, one OpenXR session: VizSession(kXr) owns the graphics-bound session and hands its handles to TeleopSession, so device trackers attach to the same session. The C++ renderer links no viz target. It borrows viz's VkDevice through the integer handles session_bindings.cpp already exposes "for callers wiring Televiz into a foreign Vulkan / OpenXR app", exports its per-eye colour and depth images as CUDA-linear buffers, and feeds ProjectionLayer.submit() over __cuda_array_interface__. Coupling is by protocol, not linkage, so the module needs only Vulkan, CUDA, MuJoCo and pybind11, and src/viz is unchanged. Controller poses are drawn as markers only -- no IK, no clutch, no rate limiting. A frame-convention bug and a control bug produce the identical symptom, and separating them is what makes the first one debuggable. The mujoco wheel supplies both the headers and libmujoco, so one library serves C++ and Python; the extension carries no RPATH and __init__.py imports mujoco first, making a version mismatch a clean ImportError rather than two copies loaded silently. Also adds glslang-tools, pkg-config and patchelf to the build prerequisites -- pkg-config is a hard configure failure under BUILD_VIZ=ON via GLFW, and patchelf was already documented but missing from the devcontainer. Verified: 313 ctest entries with the 4 pre-existing [window] failures (Tegra/Xvfb, unrelated) and 8 [xr] skips; 36 pytest cases; clang-format and pre-commit clean. The XR path itself -- the frame loop, session sharing via oxr_handles, depth-layer acceptance and controllers on a shared session -- is not verified anywhere: it needs a headset and a CloudXR runtime. The frame loop below the XR layer is exercised by --mode offscreen and by the Vulkan-to-CUDA-to-submit test, neither of which needs one. Planned and reviewed across three rounds by a six-persona panel. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
`uv pip install .` at the repository root never delivered this example: the
root pyproject's install.components filters out everything that is not
isaacteleop_wheel or isaacteleop_binaries, and neither of the example's
install rules named a component. Folding it in was the wrong fix -- _mujoco_xr
links libmujoco, so the isaacteleop wheel's contents would have depended on
whether the build host happened to have mujoco installed, and a MuJoCo-linked
.so would have shipped to everyone.
So the example becomes its own wheel, and the wheel is now the only way to run
it:
uv pip install ./examples/mujoco_xr
python -m mujoco_xr --mode offscreen
mujoco is a declared dependency rather than a build-host accident. This is
only possible because the module links no viz target -- just Vulkan, CUDA and
libmujoco.
The scene XML moves into the package (python/mujoco_xr/assets/) so it ships as
package data and one .parent resolves it both in the wheel and in the source
tree the tests import. All install() rules are gone, and the rig invokes
{python} -m mujoco_xr, which also gains a real preflight check: {python}
expands to an absolute sys.executable, so _check_commands_exist validates it
where the old bare `uv` token was skipped. It does NOT gain the
find_runtime_footguns() warning -- that requires `-m isaacteleop` or a .py
token -- so --no-launch-cloudxr-runtime remains load-bearing and unguarded.
The CMakeLists is dual-mode: add_subdirectory'd from the root, or top-level
under scikit-build-core. The in-tree path stays because ctest runs against it,
so the extension compiles twice. That is deliberate; collapsing it needs
isaacteleop on an index so the test environment can resolve it.
Note sdist.exclude, not wheel.exclude, keeps the in-place .so out of the
wheel. scikit-build-core reads .gitignore relative to the project root, so the
repo-wide *.so never applied, and wheel.exclude is applied again at zip time
where it would have deleted the real extension instead.
Verified: 313 ctest entries with the 4 pre-existing [window] failures and 8
[xr] skips, 36 pytest cases, clang-format, ruff and pre-commit clean. A cold
install into a fresh venv yields exactly one .so and runs to a clean exit.
Reviewed by tech-lead, bar-raiser and YAGNI.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
A bare top-level `mujoco_xr` in site-packages is a namespace claim staked by
an example. Move it under a PEP 420 namespace so the import path says where it
came from, and so future examples have somewhere to live:
dist: isaacteleop-examples-mujoco-xr (was mujoco-xr-example)
import: isaacteleop_examples.mujoco_xr (was mujoco_xr)
run: python -m isaacteleop_examples.mujoco_xr
`isaacteleop_examples/` deliberately has no __init__.py. That is what lets a
second example distribution join the same namespace later; a regular package
would own it exclusively.
Only one other example carries an isaacteleop- dist prefix
(isaacteleop-haptic-feedback-examples), so this diverges from the bare names
the other eleven use. It is deliberate: this is the only example that installs
an importable package, so it is the only one whose name can collide with
anything.
The C++ side is untouched -- namespace mujoco_xr, PYBIND11_MODULE(_mujoco_xr),
the <mujoco_xr/shaders/...> include prefix, the mujoco_xr_py target, the ctest
label and the [mujoco_xr] log prefix all stay. The extension is reached by a
relative import, so the namespace does not appear in C++ at all.
sdist.exclude moved with the package. It is coupled to wheel.packages by path,
and leaving it behind would have silently re-admitted the stale in-place .so
that the previous commit's note exists to prevent -- verified cross-ABI, since
a same-ABI count cannot distinguish the two.
One behaviour change fell out of the rename rather than being designed in:
find_runtime_footguns() tests for the substring `-m isaacteleop`, which
`-m isaacteleop_examples.mujoco_xr` now contains and `-m mujoco_xr` did not.
The rig's warning about a dropped --no-launch-cloudxr-runtime is therefore
live where it previously could not fire. Measured, and the comment now says so.
Verified: 313 ctest entries with the 4 pre-existing [window] failures and 8
[xr] skips, 36 pytest cases, clang-format, ruff and pre-commit clean. A cold
install into a fresh venv ships exactly one .so, no namespace __init__.py, and
runs to a clean exit.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Contributor
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
jiwenc-nv
marked this pull request as draft
August 2, 2026 01:01
Contributor
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
The arm now follows the XR controller. Ports MuJoCoXR's DLS IK, per-robot tables and squeeze-clutch to Python, where control belongs: the C++ side owns mjvScene and rendering, Python owns mjModel/mjData/mj_step, and control writes d.ctrl. That placement also makes the whole control stack unit-testable with no GPU, no headset and no XR session -- a ScriptedSource drives the real frame loop from synthetic poses. SO-101 additionally carries a leader-gripper ghost: the three leader-specific parts (Handle, Trigger, Wrist_Roll, Apache-2.0 from TheRobotStudio/SO-ARM100) as a 50%-transparent mocap body locked to the controller pose. All three share one CAD datum with Menagerie's follower, so every transform is derived rather than tuned -- the assembled gap is 0.3 mm shank-to-handle. No renderer change was needed: mjv_updateScene emits in geom-id order, so a mocap body declared after the robot draws last and blends correctly. Values that are wrong if derived are tabulated, with their upstream measurements: joint order, the NAMED jaw actuator, jaw endpoints including polarity, and ctrl limits as ctrlrange INTERSECT jnt_range -- clamping to ctrlrange alone parks SO-101 wrist_roll at 2.940000 N.m, exactly its actuator_forcerange, against a live joint limit indefinitely. Reproduced here to six figures. ns_gain is 0 on the SO-101 because its Jacobian is full column rank; at 0.1 a commanded pure translation leaks 4.73 degrees of tool roll, which a position median cannot see. Menagerie is fetched, not vendored (55 MB); the three leader STLs are vendored via LFS. A MuJoCo wrapper must be a sibling of the XML it includes, so the fetch unpacks into the scene directory and must precede the wheel install. Fixes a latent defect found while testing: the app never loaded the home keyframe, so a scene opened at qpos0 with the clutch target latched below the table. Every earlier test missed it by resetting first. Verified: 317 ctest entries with the same 4 pre-existing [window] failures (named, not numbered -- catch_discover_tests re-enumerates on relink) and 8 [xr] skips; 113 pytest cases, 61 pass / 52 skip on an unfetched checkout; clang-format, ruff and pre-commit clean; cold wheel install runs all three scenes. The XR path itself remains unverified -- no headset on this host. Planned by a four-persona panel; four plan contradictions reported and fixed. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
The mujoco_xr example draws world-locked content at a known height above the floor, but VizSession never set a reference space, so OpenXrSession fell through to its XR_REFERENCE_SPACE_TYPE_LOCAL default -- an origin at head height. The example now asks for kLocalFloor. The default stays kLocal, so no other app changes behaviour, and an unavailable space throws naming itself rather than silently substituting a different origin. That was a real gap but not the cause of the scene tracking the head. The CloudXR WebXR client's own defaults were: `auto` prefers local-floor while a -155 cm vertical offset is applied whichever space it lands on. Setting local-floor with a zero offset in the client UI fixes it. The README records that pairing; NVIDIA#871 asks why it is not the default. _HeadTravelProbe exists because those two failures are indistinguishable through a headset. It reports peak head displacement, so a pinned rotation-only pose stream gets named rather than guessed at. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
The --mode flag offered window and offscreen alongside xr, and neither earned its cost. window never worked on any machine anyone checked. offscreen rendered into memory, displayed nothing, could not terminate on its own (OffscreenBackend never overrides should_close), and was invoked by no CI job, no rig and no script -- only by a human copy-pasting from the README. Both forced a fabricated debug camera, a second clock, a variable view count, and a run() branch that built everything except the TeleopSession. The GPU-backed kOffscreen test goes too. It needed a Vulkan + CUDA device, so it skipped everywhere except a workstation, and since nothing in .github/workflows installs mujoco, no test here has ever run in CI anyway. Examples have no test infrastructure yet (NVIDIA#880); a headless path should come back with the job that runs it, not before. That leaves the Vulkan -> CUDA -> ProjectionLayer.submit() path verified nowhere, which is a deliberate gap rather than an oversight -- the README status table, the "Not verified anywhere" section and tests/CMakeLists.txt each say so and point at NVIDIA#880. ctest -L mujoco_xr: 7/7 pass (requires a re-configure; the test glob runs at configure time). pre-commit clean, clang-format-14 clean. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
The branch is 9,051 insertions across 53 files, and the reason is not that any one piece is bloated -- four unrelated changes with four different review audiences are stacked in it. Two commits are 96% of the diff, and one of them (the Franka/SO-101 teleop) is 46% on its own. Splits into: the viz reference-space API change, which carries the branch's only regression risk and is currently buried under an example; an MVP that renders a MuJoCo scene into XR with the SO-101 leader gripper locked to the controller, chosen because a frames bug is then visible on first run; the two robot scenes; and the clutch/IK teleop. The teleop PR is the one worth reading closely. examples/.../teleop.py implements a clutch and a jaw mapping from scratch, and both already exist as unit-tested retargeters in the shipped package -- SO101ClutchRetargeter and SO101GripperRetargeter. The plan calls for using those and records the integration risks, chiefly that the retargeter wants controller poses already in the robot base frame while this example defines that transform independently in cpp/frames.hpp, with nothing to catch a disagreement. Scaffolding: to be deleted before the final PR lands. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #(issue)
Type of change
Testing
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO