From 084bd71d0488f3498113b4647cef9f7ac5ab16eb Mon Sep 17 00:00:00 2001 From: Farbod Motlagh Date: Tue, 6 Oct 2026 14:42:11 -0700 Subject: [PATCH 1/5] update new documentation Signed-off-by: Farbod Motlagh --- docs/source/references/camera_streaming.rst | 225 +++++++++++++++++--- 1 file changed, 199 insertions(+), 26 deletions(-) diff --git a/docs/source/references/camera_streaming.rst b/docs/source/references/camera_streaming.rst index 35c7d0ce41..aa8f192500 100644 --- a/docs/source/references/camera_streaming.rst +++ b/docs/source/references/camera_streaming.rst @@ -12,7 +12,7 @@ data ` — a video file replayed in place of a live c to a desktop window instead of the headset. The sample lives at :code-dir:`examples/camera_viz/ `. This page walks you -from setup and a hardware-free first run to real cameras and the robot → workstation split mode. +from setup and a camera-free first run to real cameras and the robot → workstation split mode. For the exact command surface and flags, see the :code-file:`README `. @@ -30,17 +30,72 @@ For the exact command surface and flags, see the Requirements ------------ -- A workstation meeting the :doc:`system requirements ` (Ubuntu, NVIDIA - GPU, CUDA driver) — every source hands frames to the renderer GPU-resident via CuPy. -- For Jetson platforms: +Required +^^^^^^^^ + +- **GPU host:** an Ubuntu workstation meeting the :doc:`system requirements + `, or a Jetson with the appropriate JetPack installation: + **Orin:** `JetPack 6.2.x `__ or + `7.2.1 `__; + **Thor:** `JetPack 7.x `__, including 7.2.1. + Every source uses GPU-resident frames through CuPy, including recordings and synthetic + patterns. Window mode also requires the NVIDIA GPU. +- **CUDA development components:** the CUDA runtime, NVRTC, and development headers matching + the installed CUDA version. CuPy compiles kernels at runtime; a working driver alone is + insufficient. For JetPack 7.2.1 / CUDA 13.2, follow :ref:`camera-streaming-jetson-cuda` below. +- **Display:** for XR, a supported headset running the CloudXR client, with network access to + the GPU host and the :ref:`CloudXR firewall ports ` open. Follow + :ref:`connect-xr-headset` for headset setup. Alternatively, ``--mode window`` needs a local + desktop display or a video-capable remote desktop. The viewer starts CloudXR when needed. +- **Sample and dependencies:** a repository checkout, Git LFS for the bundled replay clip, + and network access for setup and the first CloudXR launch. Setup creates a Python 3.12 + environment and installs the sample's Python dependencies. Have sudo access available if + it prompts to install missing system packages. + +Optional, recommended hardware +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- **Camera:** recommended for checking the actual live view and capture latency, but optional + for the first run. Use a USB / UVC webcam, OAK-D, or supported ZED camera; see + :ref:`camera-streaming-sources`. Connect it to the viewer host for direct mode, with the + camera's required USB connection, power, device permissions, and SDK. OAK-D and UVC support + are installed by default; ZED needs its SDK and ``setup --with-zed``. A recording or synthetic + pattern lets you complete the initial display checks without a camera. +- **XR controllers:** optional for viewing, recommended for testing the live placement and + display controls in :ref:`camera-streaming-controls`. + +.. _camera-streaming-jetson-cuda: + +Jetson CUDA prerequisites +^^^^^^^^^^^^^^^^^^^^^^^^^ + +On JetPack 7.2.1, use the CUDA 13.2 development components from the matching NVIDIA package +repositories. Check the BSP and toolkit before running sample setup: + +.. code-block:: bash + + cat /etc/nv_tegra_release + readlink -f /usr/local/cuda + /usr/local/cuda/bin/nvcc --version + +JetPack 7.2.1 reports L4T ``R39``, ``REVISION: 2.1``; the toolkit should report CUDA 13.2. +If the development components are missing, follow NVIDIA's +`JetPack installation instructions +`__ for your +BSP. The full ``nvidia-jetpack`` package includes them; for just CUDA, NVIDIA documents +`nvidia-cuda-dev +`__: + +.. code-block:: bash + + sudo apt update + sudo apt install nvidia-cuda-dev - - **Orin:** `JetPack 6.2.x `__ or - `7.2.x `__ - - **Thor:** `JetPack 7.x `__ -- For the default XR mode, a headset to connect as the CloudXR client — follow the - :doc:`quick start ` step :ref:`connect-xr-headset`. The viewer - launches the CloudXR runtime itself; nothing to start separately. No headset handy? - ``--mode window`` renders to a desktop window instead. +The runtime-only ``nvidia-cuda`` package is insufficient. Check that ``/usr/local/cuda`` points +at the intended toolkit before setup: the sample uses it to select ``cupy-cuda12x`` or +``cupy-cuda13x``. On JetPack 6.2.x, keep the matching CUDA 12.x components. The sample's +``--jetson`` flag handles CUDA library discovery; it does not install the full toolkit or +upgrade JetPack. Setup ----- @@ -53,15 +108,18 @@ run the sample's one-time setup: examples/camera_viz/camera_viz.sh setup source examples/camera_viz/.venv/bin/activate +On a Jetson, add ``--jetson`` to the setup command. Activating the environment is optional +for ``camera_viz.sh`` commands; the wrapper selects the sample's environment explicitly. + There is no need to install the ``isaaccapture`` pip package yourself. ``setup`` builds the sample's own environment: ``isaaccapture[cloudxr]`` — the ``cloudxr`` extra is not optional here, since XR is the default display mode and the viewer launches the runtime itself — plus every -other Python dependency, into ``.venv/`` via ``uv``. It resolves a version new enough for the -sample on its own, falling back to a release candidate and then, after asking, to a source build -of the surrounding checkout (:code-file:`scripts/_install_deps.sh -` holds the minimum; ``--wheel`` and -``--build-from-source`` override the choice). Finally it probes the system packages it needs and -prints the exact ``apt-get`` line to approve — declining, or a non-interactive run, aborts. +other Python dependency, into ``.venv/`` via ``uv``. It resolves a release matching the checkout's +major/minor version, falling back to a release candidate and then, after asking, to a source build +of the surrounding checkout (see :code-file:`scripts/_install_deps.sh +`; ``--wheel`` and ``--build-from-source`` override +the choice). When required system packages are missing, it prints the exact ``apt-get`` line +to approve — declining, or a non-interactive run, aborts. By default ``setup`` provisions the direct-mode path — USB / UVC and OAK-D camera support. Split mode (RTP) and ZED support are opt-in, since both pull in dependencies the direct path never @@ -87,9 +145,10 @@ needs. Flags trim or extend that: * - ``--sender-only`` - Split mode only — robot-side install of just the sender's dependencies. * - ``--jetson`` - - Split mode only — extra CUDA wiring JetPack images need on the robot. + - Enable Jetson CUDA library discovery for direct or split mode. With RTP enabled, also + check for the matching NVRTC package. Install CUDA development components first. * - ``--venv PATH`` - - Install into an existing virtual environment instead of creating ``.venv/``. + - Use ``PATH`` for the environment and link the sample's ``.venv`` to it; see below. * - ``--wheel PATH`` - Install a locally built ``isaaccapture`` wheel instead of resolving one from the index — for developing Isaac Teleop itself. @@ -98,6 +157,40 @@ needs. Flags trim or extend that: prompting — a full C++ / CUDA / Vulkan build (see :doc:`/getting_started/build_from_source/index`). +Using an external virtual environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To reuse an existing Python 3.12 environment, pass its path explicitly, from the repository root: + +.. code-block:: bash + + examples/camera_viz/camera_viz.sh setup --venv /absolute/path/to/venv + +Add ``--jetson`` on Jetson. Setup installs dependencies into that environment and creates a +symlink at ``examples/camera_viz/.venv``. If that path already exists as a real directory, +move it aside before using ``--venv``; setup refuses to replace it. Subsequent ``run`` and +``loopback`` commands use this link automatically. Activating another environment, or setting +``VIRTUAL_ENV``, does not change the wrapper's choice. + +Check CUDA execution +^^^^^^^^^^^^^^^^^^^^ + +Setup's final check verifies imports. Before opening the viewer, run this small GPU operation +from the repository root to also check CuPy kernel compilation and execution: + +.. code-block:: bash + + examples/camera_viz/.venv/bin/python - <<'PY' + import cupy as cp + + values = cp.arange(4, dtype=cp.float32) + assert cp.asnumpy(values + 1).tolist() == [1.0, 2.0, 3.0, 4.0] + print("CUDA kernel check passed") + PY + +Missing NVRTC libraries or CUDA headers here mean the CUDA installation needs attention; +see :ref:`camera-streaming-troubleshooting` before continuing. + .. _recorded-camera-streaming: First run — a recording, no camera required @@ -107,7 +200,14 @@ The recorded-camera source (``type: video``) replays a video file through exactl source → layer → Televiz path a live camera uses. It is the supported way to stream recorded camera data, it needs no extra setup, and it is the quickest end-to-end check. A test clip ships with the repo, and :code-file:`configs/replay.yaml ` -already points at it: +already points at it. Fetch the clip with Git LFS if your checkout contains only its pointer: + +.. code-block:: bash + + git lfs pull --include="examples/camera_viz/test_data/recording.mp4" + +Keep the default setup's OpenCV dependency for video replay (do not use ``--no-v4l2`` for this +first run). From the repository root, choose one of these launch commands: .. code-block:: bash @@ -116,8 +216,11 @@ already points at it: ./camera_viz.sh run configs/replay.yaml --mode window # desktop window instead In XR mode the viewer first brings up the CloudXR runtime (accept the EULA on first launch, or -pass ``--accept-eula``), then **you should see** the terminal report the session and the source -coming up:: +pass ``--accept-eula``). Open the printed client URL in your headset browser, accept the host's +certificate, and click **Connect**, following :ref:`connect-xr-headset`. On Jetson Orin, select +**H.264** in the client's **Video Codec** setting before connecting. + +**You should see** the terminal report the session and the source coming up:: camera_viz: source=local, mode=xr, xr=True, shapes=quad, 1 layer(s) [video] opening... @@ -127,6 +230,41 @@ coming up:: and the clip looping on a plane in the headset once it connects — or in a desktop window (``mode=window, xr=False``) with the ``--mode window`` override, which starts no runtime. +.. _camera-streaming-first-session: + +First-session checklist +^^^^^^^^^^^^^^^^^^^^^^^ + +Use the unmodified ``replay.yaml`` for these checks; it starts with one mono plane in ``lazy`` +lock mode. For ``--mode window``, check visibility and image quality, then stop and restart; +head tracking, recentering, and XR controller checks require a headset. + +1. **Confirm the plane is visible.** The clip should play and loop on one plane in front of + you. A connected client or a positive frame rate alone does not confirm that the image is + visible. For a black plane on Orin, apply the ``compositor: televiz`` workaround in + :ref:`camera-streaming-troubleshooting` and restart. +2. **Move your head.** Make small turns and lean slightly. In the default ``lazy`` mode, the + plane should stay anchored in the room for small movements while your viewpoint changes. +3. **Test lazy recentering.** Turn to look well away from the plane (more than 45 degrees) and + hold that direction for about a second. The plane should move smoothly back in front of + you. The defaults are a 0.5-second delay and a 0.3-second transition; the placement settings + below control these values. +4. **Check image quality.** Watch while still and while moving your head. Look for freezing, + flicker, torn or corrupted regions, and unexpected colors. Replay is mono, so both eyes + should see the same source image. To separate a recording problem from the display path, + stop the viewer and try ``./camera_viz.sh run configs/synthetic.yaml``: its color pattern + should animate continuously. +5. **Try the controllers, if available.** Click the right stick to recenter, press ``A`` to + change lock mode, and press ``Y`` to restore the YAML settings. Confirm the change appears + in the view and status panel. ``B`` only affects stereo sources, so it has no effect on + this mono replay. See :ref:`camera-streaming-controls` for the full bindings. +6. **Stop and repeat.** Press ``Ctrl+C`` in the viewer terminal, launch the same command again, + and reconnect the client if needed. Confirm the plane is visible and animating again. + +If a check fails, use :ref:`camera-streaming-troubleshooting` before adding a real camera or +the split-mode network path. For a bug report, include the failing check, config, host / GPU, +JetPack and CUDA versions where applicable, client device, and relevant logs. + Replaying your own recording ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -160,6 +298,8 @@ rejected, since the sender needs per-eye streams. A mono recording is otherwise side for :ref:`split mode ` and for ``loopback``, which exercises the whole encode → UDP → decode path with no camera attached. +.. _camera-streaming-sources: + Supported sources ----------------- @@ -183,13 +323,17 @@ The source kind is selected by the ``type`` field of each entry in the YAML ``ca by default; ``stereo: true`` splits side-by-side recordings into eyes (viewer only). See :ref:`recorded-camera-streaming`. * - ``synthetic`` - - Debugging tool — GPU-generated test pattern, no hardware or file. + - Debugging tool — GPU-generated test pattern, no camera or file required. + +.. _camera-streaming-live-camera: Running with a real camera -------------------------- Attach the camera to the machine that runs the viewer, keep ``source: local`` in the config, and -run with the matching config: +run with the matching config. First edit its device, resolution, and frame rate to match your +camera. In particular, ``v4l2.yaml`` defaults to a ZED Mini's 2560×720 UVC mode; a regular webcam +needs values it supports (inspect them with ``v4l2-ctl --list-formats-ext``): .. code-block:: bash @@ -232,6 +376,8 @@ In XR, how a plane follows the operator's head is the per-camera ``lock_mode`` u Lazy-mode knobs live under ``placements.``: ``look_away_angle_deg``, ``reposition_distance``, ``reposition_delay_s``, ``transition_duration_s``. +.. _camera-streaming-controls: + Controller bindings ^^^^^^^^^^^^^^^^^^^ @@ -471,6 +617,8 @@ its own plane (and, in split mode, its own RTP port). Abbreviated: See the :code-dir:`configs/ ` directory for a complete, commented YAML per source kind. +.. _camera-streaming-troubleshooting: + Troubleshooting --------------- @@ -487,8 +635,15 @@ Troubleshooting you're sitting at, or use a video-capable remote desktop. - **"video source: no such file"** — relative ``path:`` values resolve against the YAML's directory (``configs/``), not the directory you launched from. -- **A source fails asking for CuPy / CUDA** — check ``nvidia-smi`` works and setup completed; - all sources allocate their frame buffers on the GPU. +- **A source fails asking for CuPy / CUDA** — rerun the CUDA execution check above with the + sample's ``.venv/bin/python``. ``nvidia-smi`` checks the driver; its CUDA version does not + establish that the matching toolkit, NVRTC, or headers are installed. For missing + ``libnvrtc.so``, ``cuda_runtime.h``, or ``cuda_fp16.h``, check the development components and + ``/usr/local/cuda`` target (:ref:`camera-streaming-jetson-cuda` on Jetson), then rerun setup + with ``--jetson`` on Jetson. Setup selects CuPy for that toolkit's major version. +- **Setup succeeded but another Python cannot import the dependencies** — the wrapper uses + ``examples/camera_viz/.venv/bin/python`` regardless of the active shell environment. Use + that interpreter for diagnostics, or rerun ``setup --venv PATH`` to select your environment. - **No video on Orin.** When the CloudXR runtime runs on Jetson Orin, set **Video Codec** to **H.264** in the CloudXR web client. See :ref:`connect-xr-headset`. @@ -568,3 +723,21 @@ extensions and forwards the handles through ``TeleopSessionConfig.oxr_handles``: See the *Sharing the XR session* section of :doc:`/getting_started/televiz` for the full pattern (imports, frame loop, and how the two sessions' lifecycles relate), and :doc:`/getting_started/teleop_session` for the ``TeleopSession`` side. + +Next steps +---------- + +After the :ref:`camera-streaming-first-session` checks pass, choose the next step for your use case: + +- **See a live camera:** follow :ref:`camera-streaming-live-camera`, then repeat the visibility + and image-quality checks with motion in front of the camera. With a stereo source, also + check both eyes and the ``B`` mono / stereo toggle. +- **Tune the view:** adjust placement, size, and lock mode using the controller bindings and + YAML settings above. Add entries to ``cameras`` for multiple feeds, or use + :ref:`recorded-camera-streaming` to replay your own recording. +- **Stream from a robot:** use :ref:`split-mode` when capture and viewing run on different + machines, after validating direct mode locally. +- **Integrate camera views into teleoperation:** follow :ref:`sharing-the-xr-session` to share + Televiz's OpenXR session with :doc:`/getting_started/teleop_session`, and use the + :doc:`retargeting interface ` to map tracking inputs to robot + actions. The camera viewer itself does not command a robot. From 6c2d25bb7eb365fb63ecac50193c8948af01a6b2 Mon Sep 17 00:00:00 2001 From: Farbod Motlagh Date: Tue, 6 Oct 2026 14:54:29 -0700 Subject: [PATCH 2/5] make it more concise Signed-off-by: Farbod Motlagh --- docs/AGENTS.md | 6 + docs/source/references/camera_streaming.rst | 803 ++++++-------------- 2 files changed, 259 insertions(+), 550 deletions(-) diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 9f7f806fa1..0510cf0180 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -97,3 +97,9 @@ and `logo_dark`. Keep cache-removal documentation focused on migration impact. Omit asset-fetch implementation walkthroughs and alternative override instructions. + +## Developer-facing guides + +Lead with concise prerequisites, a runnable example, and observable success checks. +Keep optional setup and diagnostics outside the first-run path. Link to existing +reference material instead of repeating implementation details. diff --git a/docs/source/references/camera_streaming.rst b/docs/source/references/camera_streaming.rst index aa8f192500..c3d620b1ae 100644 --- a/docs/source/references/camera_streaming.rst +++ b/docs/source/references/camera_streaming.rst @@ -4,355 +4,158 @@ Camera Streaming ================ -``camera_viz`` is the reference camera-streaming sample built on :doc:`Televiz -` (``isaaccapture.viz``). It captures frames from one or more cameras -and streams them to an XR headset — one plane per camera, aspect-fit — either directly or from a -robot to a workstation over the network (split mode). It can also stream :ref:`recorded camera -data ` — a video file replayed in place of a live camera — and render -to a desktop window instead of the headset. - -The sample lives at :code-dir:`examples/camera_viz/ `. This page walks you -from setup and a camera-free first run to real cameras and the robot → workstation split mode. -For the exact command surface and flags, see the -:code-file:`README `. +Use :code-dir:`camera_viz ` to stream live cameras or recordings to an XR +headset or desktop window. The sample uses :doc:`Televiz ` and supports +cameras attached locally or streamed from a robot over RTP. .. figure:: ../_static/televiz_2d.gif - :alt: camera_viz in window mode with synthetic multi-camera feeds + :alt: camera_viz with multiple camera feeds :width: 420px :class: no-image-zoom - ``camera_viz`` in XR mode — one plane per camera. + One plane per camera. .. contents:: On this page :local: - :depth: 2 + :depth: 1 Requirements ------------ -Required -^^^^^^^^ - -- **GPU host:** an Ubuntu workstation meeting the :doc:`system requirements - `, or a Jetson with the appropriate JetPack installation: - **Orin:** `JetPack 6.2.x `__ or - `7.2.1 `__; - **Thor:** `JetPack 7.x `__, including 7.2.1. - Every source uses GPU-resident frames through CuPy, including recordings and synthetic - patterns. Window mode also requires the NVIDIA GPU. -- **CUDA development components:** the CUDA runtime, NVRTC, and development headers matching - the installed CUDA version. CuPy compiles kernels at runtime; a working driver alone is - insufficient. For JetPack 7.2.1 / CUDA 13.2, follow :ref:`camera-streaming-jetson-cuda` below. -- **Display:** for XR, a supported headset running the CloudXR client, with network access to - the GPU host and the :ref:`CloudXR firewall ports ` open. Follow - :ref:`connect-xr-headset` for headset setup. Alternatively, ``--mode window`` needs a local - desktop display or a video-capable remote desktop. The viewer starts CloudXR when needed. -- **Sample and dependencies:** a repository checkout, Git LFS for the bundled replay clip, - and network access for setup and the first CloudXR launch. Setup creates a Python 3.12 - environment and installs the sample's Python dependencies. Have sudo access available if - it prompts to install missing system packages. - -Optional, recommended hardware -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- **Camera:** recommended for checking the actual live view and capture latency, but optional - for the first run. Use a USB / UVC webcam, OAK-D, or supported ZED camera; see - :ref:`camera-streaming-sources`. Connect it to the viewer host for direct mode, with the - camera's required USB connection, power, device permissions, and SDK. OAK-D and UVC support - are installed by default; ZED needs its SDK and ``setup --with-zed``. A recording or synthetic - pattern lets you complete the initial display checks without a camera. -- **XR controllers:** optional for viewing, recommended for testing the live placement and - display controls in :ref:`camera-streaming-controls`. - -.. _camera-streaming-jetson-cuda: - -Jetson CUDA prerequisites -^^^^^^^^^^^^^^^^^^^^^^^^^ - -On JetPack 7.2.1, use the CUDA 13.2 development components from the matching NVIDIA package -repositories. Check the BSP and toolkit before running sample setup: - -.. code-block:: bash - - cat /etc/nv_tegra_release - readlink -f /usr/local/cuda - /usr/local/cuda/bin/nvcc --version - -JetPack 7.2.1 reports L4T ``R39``, ``REVISION: 2.1``; the toolkit should report CUDA 13.2. -If the development components are missing, follow NVIDIA's -`JetPack installation instructions -`__ for your -BSP. The full ``nvidia-jetpack`` package includes them; for just CUDA, NVIDIA documents -`nvidia-cuda-dev -`__: - -.. code-block:: bash - - sudo apt update - sudo apt install nvidia-cuda-dev - -The runtime-only ``nvidia-cuda`` package is insufficient. Check that ``/usr/local/cuda`` points -at the intended toolkit before setup: the sample uses it to select ``cupy-cuda12x`` or -``cupy-cuda13x``. On JetPack 6.2.x, keep the matching CUDA 12.x components. The sample's -``--jetson`` flag handles CUDA library discovery; it does not install the full toolkit or -upgrade JetPack. +- **Host:** Ubuntu with an NVIDIA GPU; see :doc:`system requirements `. + Jetson: Orin with JetPack 6.2.x or 7.2.1; Thor with JetPack 7.x. +- **CUDA:** runtime, NVRTC, and development headers matching the installed toolkit. + JetPack 7.2.1 uses CUDA 13.2; see :ref:`camera-streaming-jetson-cuda`. + A GPU is required even for recordings and window mode. +- **Display:** a :ref:`supported XR headset ` with network access to the host + and :ref:`CloudXR ports ` open, or a desktop display for window mode. +- **Camera — optional, recommended:** USB / UVC, OAK-D, or supported ZED camera for a live feed. + Start with the bundled recording if no camera is available. ZED requires its SDK. +- **Controllers — optional:** for adjusting the view in XR. Setup ----- -Clone the repository if you haven't already (quick start step :ref:`check-out-code-base`), then -run the sample's one-time setup: - -.. code-block:: bash - - examples/camera_viz/camera_viz.sh setup - source examples/camera_viz/.venv/bin/activate - -On a Jetson, add ``--jetson`` to the setup command. Activating the environment is optional -for ``camera_viz.sh`` commands; the wrapper selects the sample's environment explicitly. - -There is no need to install the ``isaaccapture`` pip package yourself. ``setup`` builds the -sample's own environment: ``isaaccapture[cloudxr]`` — the ``cloudxr`` extra is not optional here, -since XR is the default display mode and the viewer launches the runtime itself — plus every -other Python dependency, into ``.venv/`` via ``uv``. It resolves a release matching the checkout's -major/minor version, falling back to a release candidate and then, after asking, to a source build -of the surrounding checkout (see :code-file:`scripts/_install_deps.sh -`; ``--wheel`` and ``--build-from-source`` override -the choice). When required system packages are missing, it prints the exact ``apt-get`` line -to approve — declining, or a non-interactive run, aborts. - -By default ``setup`` provisions the direct-mode path — USB / UVC and OAK-D camera support. Split -mode (RTP) and ZED support are opt-in, since both pull in dependencies the direct path never -needs. Flags trim or extend that: - -.. list-table:: - :header-rows: 1 - :widths: 22 78 - - * - Flag - - Effect - * - ``--no-v4l2`` - - Skip USB / UVC webcam support (``opencv-python``). - * - ``--no-oakd`` - - Skip OAK-D support (``depthai``). - * - ``--with-rtp`` - - Also install the split-mode dependencies: the GStreamer system packages, PyGObject, and the - native NVENC/NVDEC codec build. Required for ``loopback`` and for any config with - ``source: rtp``; implied by ``--sender-only``. Direct mode does not need it. - * - ``--with-zed`` - - Also build + install the ZED SDK's Python API (``pyzed``). Requires the ZED SDK on the - machine (default ``/usr/local/zed``; override with ``--zed-sdk PATH``). - * - ``--sender-only`` - - Split mode only — robot-side install of just the sender's dependencies. - * - ``--jetson`` - - Enable Jetson CUDA library discovery for direct or split mode. With RTP enabled, also - check for the matching NVRTC package. Install CUDA development components first. - * - ``--venv PATH`` - - Use ``PATH`` for the environment and link the sample's ``.venv`` to it; see below. - * - ``--wheel PATH`` - - Install a locally built ``isaaccapture`` wheel instead of resolving one from the index — for - developing Isaac Teleop itself. - * - ``--build-from-source`` - - Skip the package index and build ``isaaccapture`` from the surrounding checkout without - prompting — a full C++ / CUDA / Vulkan build (see - :doc:`/getting_started/build_from_source/index`). - -Using an external virtual environment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To reuse an existing Python 3.12 environment, pass its path explicitly, from the repository root: +From a :ref:`repository checkout `, run: .. code-block:: bash - examples/camera_viz/camera_viz.sh setup --venv /absolute/path/to/venv - -Add ``--jetson`` on Jetson. Setup installs dependencies into that environment and creates a -symlink at ``examples/camera_viz/.venv``. If that path already exists as a real directory, -move it aside before using ``--venv``; setup refuses to replace it. Subsequent ``run`` and -``loopback`` commands use this link automatically. Activating another environment, or setting -``VIRTUAL_ENV``, does not change the wrapper's choice. - -Check CUDA execution -^^^^^^^^^^^^^^^^^^^^ - -Setup's final check verifies imports. Before opening the viewer, run this small GPU operation -from the repository root to also check CuPy kernel compilation and execution: - -.. code-block:: bash - - examples/camera_viz/.venv/bin/python - <<'PY' - import cupy as cp - - values = cp.arange(4, dtype=cp.float32) - assert cp.asnumpy(values + 1).tolist() == [1.0, 2.0, 3.0, 4.0] - print("CUDA kernel check passed") - PY + cd examples/camera_viz + ./camera_viz.sh setup -Missing NVRTC libraries or CUDA headers here mean the CUDA installation needs attention; -see :ref:`camera-streaming-troubleshooting` before continuing. +Setup creates a Python 3.12 ``.venv`` with ``isaaccapture[cloudxr]``, UVC, and OAK-D support. +It needs network access and may prompt for sudo to install system dependencies. +On Jetson, install the :ref:`CUDA prerequisites ` first and add +``--jetson``. To reuse an environment, use :ref:`setup --venv `. -.. _recorded-camera-streaming: +All commands below run from ``examples/camera_viz``. No environment activation is needed. First run — a recording, no camera required ------------------------------------------- -The recorded-camera source (``type: video``) replays a video file through exactly the same -source → layer → Televiz path a live camera uses. It is the supported way to stream recorded -camera data, it needs no extra setup, and it is the quickest end-to-end check. A test clip ships -with the repo, and :code-file:`configs/replay.yaml ` -already points at it. Fetch the clip with Git LFS if your checkout contains only its pointer: +Fetch the bundled clip with Git LFS, then launch the viewer: .. code-block:: bash git lfs pull --include="examples/camera_viz/test_data/recording.mp4" + ./camera_viz.sh run configs/replay.yaml -Keep the default setup's OpenCV dependency for video replay (do not use ``--no-v4l2`` for this -first run). From the repository root, choose one of these launch commands: +The viewer starts CloudXR automatically; accept the EULA on first launch. Open the printed +client URL in your headset browser, accept the certificate, and click **Connect** +(:ref:`connection instructions `). -.. code-block:: bash - - cd examples/camera_viz - ./camera_viz.sh run configs/replay.yaml # XR headset (default) - ./camera_viz.sh run configs/replay.yaml --mode window # desktop window instead - -In XR mode the viewer first brings up the CloudXR runtime (accept the EULA on first launch, or -pass ``--accept-eula``). Open the printed client URL in your headset browser, accept the host's -certificate, and click **Connect**, following :ref:`connect-xr-headset`. On Jetson Orin, select -**H.264** in the client's **Video Codec** setting before connecting. - -**You should see** the terminal report the session and the source coming up:: - - camera_viz: source=local, mode=xr, xr=True, shapes=quad, 1 layer(s) - [video] opening... - [video] connected - [video] streaming - -and the clip looping on a plane in the headset once it connects — or in a desktop window -(``mode=window, xr=False``) with the ``--mode window`` override, which starts no runtime. +For a desktop preview instead, add ``--mode window`` to the run command. +On Orin, select **H.264** in the client and apply the +:ref:`black-plane workaround ` below. .. _camera-streaming-first-session: First-session checklist ^^^^^^^^^^^^^^^^^^^^^^^ -Use the unmodified ``replay.yaml`` for these checks; it starts with one mono plane in ``lazy`` -lock mode. For ``--mode window``, check visibility and image quality, then stop and restart; -head tracking, recentering, and XR controller checks require a headset. - -1. **Confirm the plane is visible.** The clip should play and loop on one plane in front of - you. A connected client or a positive frame rate alone does not confirm that the image is - visible. For a black plane on Orin, apply the ``compositor: televiz`` workaround in - :ref:`camera-streaming-troubleshooting` and restart. -2. **Move your head.** Make small turns and lean slightly. In the default ``lazy`` mode, the - plane should stay anchored in the room for small movements while your viewpoint changes. -3. **Test lazy recentering.** Turn to look well away from the plane (more than 45 degrees) and - hold that direction for about a second. The plane should move smoothly back in front of - you. The defaults are a 0.5-second delay and a 0.3-second transition; the placement settings - below control these values. -4. **Check image quality.** Watch while still and while moving your head. Look for freezing, - flicker, torn or corrupted regions, and unexpected colors. Replay is mono, so both eyes - should see the same source image. To separate a recording problem from the display path, - stop the viewer and try ``./camera_viz.sh run configs/synthetic.yaml``: its color pattern - should animate continuously. -5. **Try the controllers, if available.** Click the right stick to recenter, press ``A`` to - change lock mode, and press ``Y`` to restore the YAML settings. Confirm the change appears - in the view and status panel. ``B`` only affects stereo sources, so it has no effect on - this mono replay. See :ref:`camera-streaming-controls` for the full bindings. -6. **Stop and repeat.** Press ``Ctrl+C`` in the viewer terminal, launch the same command again, - and reconnect the client if needed. Confirm the plane is visible and animating again. - -If a check fails, use :ref:`camera-streaming-troubleshooting` before adding a real camera or -the split-mode network path. For a bug report, include the failing check, config, host / GPU, -JetPack and CUDA versions where applicable, client device, and relevant logs. +With the default replay config: -Replaying your own recording -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +- **Playback:** one plane is visible and the clip loops. +- **Head movement:** small turns and leaning change your viewpoint; the plane stays anchored. +- **Lazy recentering:** look more than 45° away for about a second; the plane moves smoothly + back in front of you. +- **Image quality:** no freezing, flicker, torn regions, or unexpected colors, including while + moving your head. This mono clip should show the same source image in both eyes. +- **Controllers, if available:** right-stick click recenters; ``A`` changes lock mode; + ``Y`` restores the config. +- **Restart:** press ``Ctrl+C``, rerun, and reconnect if needed; playback resumes. -Point ``path`` at any file OpenCV's FFmpeg backend reads — mp4 / mkv / webm carrying H.264, -HEVC, AV1, and so on. The file is probed once at startup for its size and frame rate (a missing -or absurd rate falls back to 30 fps); a missing or unreadable file is a configuration error and -fails immediately instead of retrying like a camera would. A complete config: +In window mode, check playback, image quality, and restart. If a check fails, see +:ref:`camera-streaming-troubleshooting`. -.. code-block:: yaml +.. _camera-streaming-live-camera: - source: local +Running with a real camera +-------------------------- - cameras: - - name: replay - enabled: true - type: video - path: ../test_data/recording.mp4 # required; relative to this YAML's directory - loop: true # false = hold the last frame at end of file - fps: 0 # 0 = the file's native rate - stereo: false # true = split a side-by-side recording into eyes - # width/height default to the file's native size (per eye when stereo) +Attach the camera to the viewer host and keep ``source: local``. Edit the matching config's +device, resolution, and frame rate, then run: - display: - mode: xr # or: window - placements: - replay: { lock_mode: lazy, distance: 1.5 } +.. code-block:: bash -With ``source: rtp``, ``width``, ``height``, and ``fps`` become required — the sender paces its -encode loop at ``fps`` and the receiver sizes its decoder from the config — and ``stereo`` is -rejected, since the sender needs per-eye streams. A mono recording is otherwise a fine capture -side for :ref:`split mode ` and for ``loopback``, which exercises the whole -encode → UDP → decode path with no camera attached. + ./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml + +``v4l2.yaml`` defaults to a ZED Mini's 2560×720 UVC mode. For a regular webcam, use modes reported +by ``v4l2-ctl --list-formats-ext``. ZED SDK capture also needs ``setup --with-zed``. .. _camera-streaming-sources: Supported sources ------------------ +^^^^^^^^^^^^^^^^^ -The source kind is selected by the ``type`` field of each entry in the YAML ``cameras`` list: +Set ``type`` in each ``cameras`` entry. Multiple entries produce one plane per camera. +See :code-dir:`configs/ ` for complete examples. .. list-table:: :header-rows: 1 :widths: 16 84 - * - ``type:`` - - Notes + * - Type + - Source * - ``v4l2`` - - USB / UVC cameras — anything ``v4l2-ctl --list-formats-ext`` reports. + - USB / UVC camera. * - ``oakd`` - - OAK-D RGB / LEFT / RIGHT; mono or ``stereo: true``. Stereo ships GRAY8 over USB to halve - bandwidth and is broadcast to RGBA on the GPU; ``stereo_rgb`` keeps color. + - OAK-D RGB / LEFT / RIGHT; ``stereo: true`` for grayscale stereo. * - ``zed`` - - ZED 2 / Mini / X One; mono or ``stereo: true`` (per-eye SDK retrieve, zero-copy on the GPU). + - ZED 2 / Mini / X One through the ZED SDK; mono or stereo. * - ``video`` - - Recorded camera data — video-file replay (anything OpenCV's FFmpeg backend reads). Loops - by default; ``stereo: true`` splits side-by-side recordings into eyes (viewer only). See - :ref:`recorded-camera-streaming`. + - :ref:`Recorded video `, including side-by-side stereo. * - ``synthetic`` - - Debugging tool — GPU-generated test pattern, no camera or file required. + - Animated GPU test pattern; no camera or file needed. -.. _camera-streaming-live-camera: +.. _recorded-camera-streaming: -Running with a real camera --------------------------- +Replaying your own recording +---------------------------- -Attach the camera to the machine that runs the viewer, keep ``source: local`` in the config, and -run with the matching config. First edit its device, resolution, and frame rate to match your -camera. In particular, ``v4l2.yaml`` defaults to a ZED Mini's 2560×720 UVC mode; a regular webcam -needs values it supports (inspect them with ``v4l2-ctl --list-formats-ext``): +Edit :code-file:`configs/replay.yaml `: -.. code-block:: bash +.. code-block:: yaml - ./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml + cameras: + - name: replay + type: video + path: /path/to/clip.mp4 + loop: true + fps: 0 # use the file's frame rate + stereo: false # true for side-by-side recordings in direct mode -**You should see** the same startup lines as above with the camera's tag (``[v4l2]``, -``[oakd]``, ``[zed]``) and the live feed. Multiple entries in the ``cameras`` list render as one -plane each. +Relative paths resolve from the YAML's directory. OpenCV's FFmpeg backend must support the +file's format. Size and frame rate default to the recording's values; ``loop: false`` holds +the last frame. Keep setup's OpenCV dependency for replay (``--no-v4l2`` skips it). Display modes ------------- -XR is the default: each camera renders as its own plane in the headset via the active OpenXR -runtime, and stereo sources (``stereo: true``) render true side-by-side stereo. Pass -``--mode window`` (or set ``display.mode: window`` in the YAML) to render to a desktop window -instead — no headset or runtime needed; stereo shows the left eye. +XR is the default. Use ``--mode window`` or ``display.mode: window`` for a desktop preview; +stereo sources show only the left eye in window mode. -In XR, how a plane follows the operator's head is the per-camera ``lock_mode`` under -``display.placements.``: +In XR, set ``display.placements..lock_mode``: .. list-table:: :header-rows: 1 @@ -361,28 +164,23 @@ In XR, how a plane follows the operator's head is the per-camera ``lock_mode`` u * - Mode - Behavior * - ``world`` - - Placed once in front of you and stays put. + - Stays where initially placed. * - ``head`` - - Follows your head every frame. Head-locked content updates its pose at application - rate, so it trails fast head motion by roughly a frame — expected for any head-locked - OpenXR layer; prefer ``lazy`` unless you need a true HUD. + - Follows your head every frame; may trail fast motion. * - ``gimbal`` - - Follows your position but not your rotation: the surface stays pointed where you first - looked, walks with you, and turning your head looks around it. The natural mode for - wide cylinder feeds (a "virtual gimbal"). - * - ``lazy`` - - World-locked, but re-snaps in front of you when you look away (default). + - Follows your position while keeping its initial heading. + * - ``lazy`` (default) + - Stays anchored until you look or move away, then recenters smoothly. -Lazy-mode knobs live under ``placements.``: ``look_away_angle_deg``, -``reposition_distance``, ``reposition_delay_s``, ``transition_duration_s``. +Lazy-mode settings: ``look_away_angle_deg``, ``reposition_distance``, ``reposition_delay_s``, +and ``transition_duration_s`` under the camera's placement. .. _camera-streaming-controls: Controller bindings ^^^^^^^^^^^^^^^^^^^ -In XR the controllers retune the view live, without editing the YAML and restarting. The right -hand changes how the feed looks; the left, what surface it is mapped onto: +Changes apply to all cameras and appear in the terminal and headset status panels. .. Use concise titles and Televiz branding; name camera_viz in the prose. @@ -391,8 +189,6 @@ hand changes how the feed looks; the left, what surface it is mapped onto: right controller adjusts depth and placement. :width: 100% - Bindings at a glance. The table below is the same thing in words. - .. list-table:: :header-rows: 1 :widths: 22 78 @@ -400,344 +196,251 @@ hand changes how the feed looks; the left, what surface it is mapped onto: * - Input - Effect * - Right stick ←/→ - - Stereo plane gap — how far apart the two eyes' planes sit - (``placements..stereo_plane_distance_cm``). Widening it pushes the scene back - instead of packing it into the space in front of the planes. Stereo cameras only; on - ``equirect``, which has no gap to set, the same stick pans the panorama. + - Adjust stereo plane gap; pan the panorama on ``equirect``. * - Right stick click - - Recenter on your view: an ``equirect`` yaws so the middle of the panorama lands dead - ahead, and a placed surface re-snaps its anchor. + - Recenter the surface or panorama. * - ``A`` - - Cycle the lock mode: ``world`` → ``head`` → ``gimbal`` → ``lazy``. + - Cycle ``world`` → ``head`` → ``gimbal`` → ``lazy``. * - ``B`` - - Toggle mono / stereo. Stereo cameras only. + - Toggle mono / stereo for stereo sources. * - ``X`` - - Cycle the shape: ``quad`` → ``cylinder`` → ``equirect``. + - Cycle ``quad`` → ``cylinder`` → ``equirect``. * - ``Y`` - - Reset everything to the YAML values. + - Reset to YAML settings. * - Left stick - - Retunes the active shape: ``quad`` size / vertical position, ``cylinder`` arc width / - vertical position, ``equirect`` horizontal / vertical span. - -Changes apply to every camera at once and appear both on the terminal status panel and on a -head-locked panel in the headset that auto-hides shortly after. Neither toggle reallocates: -``B`` sends the left frame to both eyes, and ``X`` flips visibility between shapes built at -startup, so the extra shapes cost VRAM (reported at startup) rather than a stall on the press. + - Adjust size / vertical position; arc width for cylinders, span for panoramas. -The stereo gap is bounded below **divergent parallax** — a gap wider than your IPD would need -the eyes to splay outward — and the headset's measured IPD sets that ceiling, not the config. -The HUD suggests a gap derived from the plane distance and that IPD. - -Bindings, rates and limits live under ``display.controls``; see the -:code-file:`README ` for the full set. +The stereo gap is limited by the headset's measured IPD. Configure bindings and rates under +``display.controls``; see the :code-file:`README `. Display surfaces ----------------- +^^^^^^^^^^^^^^^^ -By default each camera renders on a flat plane, which suits normal-FOV feeds. Wide-FOV and -panoramic sources look better on a curved surface: set ``shape`` per camera under -``display.placements.``: +Set ``shape`` under ``display.placements.``: .. list-table:: :header-rows: 1 - :widths: 30 70 + :widths: 24 76 * - Shape - - Behavior + - Use * - ``quad`` (default) - - Flat plane. All lock modes apply. + - Flat camera feed. Set ``size: [width_m, height_m]``. * - ``cylinder`` - - Curved arc facing the operator, so the image stays at a constant viewing distance edge - to edge. ``cylinder_radius_m`` (default 2.0) sets that distance, ``cylinder_angle_deg`` - (default 90) the arc width. All lock modes apply (``head`` = gaze-tracking curved - visor). + - Wide-FOV feed. Set ``cylinder_radius_m`` and ``cylinder_angle_deg``. * - ``equirect`` - - Full 360°×180° sphere around the operator, for equirectangular panorama / VR-video - sources. The middle of the texture points along ``equirect_yaw_deg`` (0 = the - direction the headset last recentered on, positive to the left), which is how a camera - mounted facing another way gets aimed. Lock modes don't apply — the sphere is centred - on the operator and follows them everywhere. - -Curved shapes exist only in XR mode — the viewer exits with an error in window mode. Stereo -sources render per-eye textures on the same surface, and ``stereo_baseline_mm`` adds a per-eye -pose shift (no effect on the equirect sphere at its default infinite radius). - -Surfaces are composited by the OpenXR runtime, which keeps them sharp under head motion and lets -CloudXR stream them efficiently (see -:ref:`OpenXR composition layers `). For flat planes only, -``compositor: televiz`` opts a camera back into Televiz's built-in compositor. On Jetson Orin -(CloudXR experimental runtime), set ``compositor: televiz`` so the plane is not left black; see -Troubleshooting below. On the Televiz 2D API the same switch is -``openxr_composition = False`` (see :ref:`Jetson Orin, XR `). + - 360°×180° panorama. Set ``equirect_yaw_deg`` to adjust heading; lock modes do not apply. + +Curved surfaces require XR. The OpenXR runtime composites surfaces by default. Flat planes can +use ``compositor: televiz`` instead; this is required with the experimental runtime on Orin. +See :ref:`openxr-composition-layers` for compositor details. + +Setup options +------------- + +.. list-table:: + :header-rows: 1 + :widths: 26 74 + + * - Flag + - Effect + * - ``--jetson`` + - Configure CUDA library discovery on Jetson; with RTP, also check for NVRTC. + * - ``--venv PATH`` + - Install into ``PATH`` and link the sample's ``.venv`` to it. + * - ``--with-rtp`` + - Install GStreamer, PyGObject, and native codec dependencies for split mode or loopback. + * - ``--with-zed`` + - Install the ZED Python API. Requires the SDK at ``/usr/local/zed`` or ``--zed-sdk PATH``. + * - ``--no-v4l2`` / ``--no-oakd`` + - Skip OpenCV / DepthAI respectively. Video replay also needs OpenCV. + * - ``--sender-only`` + - Install only the robot sender's dependencies; implies ``--with-rtp``. + * - ``--wheel PATH`` + - Install a locally built ``isaaccapture`` wheel. + * - ``--build-from-source`` + - Build from this checkout; requires the :doc:`build environment `. + +.. _camera-streaming-external-venv: + +External virtual environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To reuse an existing Python 3.12 environment: + +.. code-block:: bash + + ./camera_viz.sh setup --venv /absolute/path/to/venv + +Setup installs dependencies there and links ``.venv`` to it. If ``.venv`` is already a real +directory, move it aside first. The wrapper always uses ``.venv/bin/python``, regardless of +the environment activated in your shell. + +.. _camera-streaming-jetson-cuda: + +Jetson CUDA prerequisites +^^^^^^^^^^^^^^^^^^^^^^^^^ + +JetPack 7.2.1 (L4T R39.2.1) needs CUDA 13.2 development components, including headers and NVRTC. +With the matching NVIDIA package repositories configured, install them using the +`JetPack CUDA development package +`__: + +.. code-block:: bash + + sudo apt update + sudo apt install nvidia-cuda-dev + readlink -f /usr/local/cuda + /usr/local/cuda/bin/nvcc --version + +On JetPack 7.2.1, verify that ``/usr/local/cuda`` selects CUDA 13.2. On JetPack 6.2.x, keep the +matching CUDA 12.x toolkit. Setup selects CuPy from this path; ``--jetson`` configures library +discovery but does not install the full toolkit. See NVIDIA's +`JetPack installation guide +`__ for BSP setup. CloudXR runtime flags --------------------- -In XR mode the viewer attaches to the running CloudXR runtime, or starts one if none is -serving. Useful flags: +The viewer attaches to an existing CloudXR service or starts one. Common run flags: -- ``--accept-eula`` — accept the CloudXR EULA non-interactively (first run only). -- ``--cloudxr-device-profile PROFILE`` — ``NV_DEVICE_PROFILE`` (default ``Quest3``). -- ``--no-host-client`` — disable serving CloudXR.js at - ``https://:48322/client/``. Only applies when this run starts the - service; stop and restart the service to change hosting. +- ``--accept-eula`` — accept the EULA non-interactively. +- ``--cloudxr-device-profile PROFILE`` — device profile (default ``Quest3``). +- ``--no-host-client`` — disable the hosted client at ``https://:48322/client/`` when + starting a service. Stop an existing service first to change hosting. -When the viewer starts a CloudXR service with hosting enabled (the default; -``--no-host-client`` disables it), it prints the local ``/client/`` URL. To -inspect or stop an already-running service, activate the example venv (or any -environment with ``isaaccapture``) and run:: +Inspect or stop the service with the sample's interpreter: - python -m isaaccapture.cloudxr.service status - python -m isaaccapture.cloudxr.service stop +.. code-block:: bash -Run ``camera_viz.py --help`` for the rest (install dir, env-config file, WSS proxy toggle). + .venv/bin/python -m isaaccapture.cloudxr.service status + .venv/bin/python -m isaaccapture.cloudxr.service stop -Runtime settings themselves are declared in the config rather than exported. The viewer ships -defaults (pose wait off, runtime foveation on), and ``display.cloudxr`` overrides them per -deployment — list only what you want to change: +Set runtime overrides in YAML: .. code-block:: yaml display: cloudxr: NV_DEVICE_PROFILE: apple-vision-pro - NV_ENABLE_POSE_WAIT: null # null drops a viewer default - -The viewer writes these to a generated ``--cloudxr-env-config`` file, which outranks the process -environment — so a value left in your shell (for example after sourcing -``~/.cloudxr/run/cloudxr.env``, as ``--no-launch-cloudxr-runtime`` suggests) cannot silently -override the config, and neither can it override ``--cloudxr-device-profile``. Booleans are -written in the lowercase spelling the runtime's parser recognises; the launcher-computed keys -(``XR_RUNTIME_JSON``, ``NV_CXR_RUNTIME_DIR``, ``NV_CXR_OUTPUT_DIR``, ``XRT_NO_STDIN``) are -rejected by name. Passing ``--cloudxr-env-config`` yourself takes precedence over the generated -file. - -An unknown variable name warns with a suggestion before the runtime launches, and is passed -through regardless — the runtime has more settings than the viewer lists. To confirm what the -runtime resolved, read the settings dump at the top of the newest -``~/.cloudxr/logs/cxr_server.*.log``. + NV_ENABLE_POSE_WAIT: null # remove a viewer default + +These settings override shell environment variables; an explicit ``--cloudxr-env-config`` +takes precedence over the generated config. Resolved settings appear in +``~/.cloudxr/logs/cxr_server.*.log``. See :doc:`/references/cloudxr` and +``.venv/bin/python camera_viz.py --help`` for more options. .. _split-mode: Split mode — robot → workstation over RTP ----------------------------------------- -Split mode runs the capture side on the robot (``camera_streamer.py``) and ships RTP H.264 to -the viewer on the workstation (``source: rtp``). - -.. warning:: - - Split mode is **not recommended in most cases**. It exists for one situation: the cameras are - on the robot, but Isaac Teleop runs on a workstation, so the frames must be streamed to where - Isaac Teleop is running. That costs a full extra encode/decode hop — NVENC on the robot, UDP, - NVDEC on the workstation — so whenever a camera can attach directly to the machine running - Isaac Teleop, run direct mode instead. Wired networks only: there is no retransmit or FEC — one lost packet corrupts one - frame until the next IDR (default every 5 s). - -In split mode every camera entry must pin ``width``, ``height``, and ``fps`` in the YAML — the -receiver sizes its decoder from the config, not from the wire. - -The workstation needs the RTP dependencies, which ``setup`` does not install by default — re-run -it with ``--with-rtp`` if you provisioned for direct mode first. The robot side is handled by -``deploy``, which implies the flag. +Use split mode when cameras are attached to a robot and the viewer runs on a workstation. +Prefer a direct connection when possible: RTP adds an encode/decode hop. Use a wired network; +there is no retransmission or FEC, so packet loss can corrupt video until the next IDR frame +(default interval: 5 seconds). -Set ``source: rtp`` in the config, export the robot/streaming credentials once per shell, then -deploy the sender and run the viewer: +In the YAML, set ``source: rtp`` and specify ``width``, ``height``, and ``fps`` for each camera. +Then, on the workstation: .. code-block:: bash + ./camera_viz.sh setup --with-rtp export REMOTE_HOST=10.0.0.5 REMOTE_USER=nvidia export STREAMING_HOST=10.0.0.42 # workstation IP + ./camera_viz.sh deploy configs/v4l2.yaml + ./camera_viz.sh run configs/v4l2.yaml - ./camera_viz.sh deploy configs/v4l2.yaml # full deploy + systemd unit on the robot - ./camera_viz.sh run configs/v4l2.yaml # viewer on the workstation - -``deploy`` rsyncs the source to the robot, installs sender dependencies, renders a -``camera-streamer.service`` systemd user unit (injecting ``--host`` from ``$STREAMING_HOST`` -without editing the YAML on disk), and enables it. Operate the running unit with -``./camera_viz.sh service-{status,logs,restart}``. The sender retries forever across unplug, SDK -errors, and network blips. +``deploy`` installs the sender and starts a ``camera-streamer`` systemd user service on the +robot. Use ``./camera_viz.sh service-{status,logs,restart}`` to manage it. The workstation's +UDP receive ports (default 5000+) must be reachable from the robot. Loopback ^^^^^^^^ -Loopback is a testing / debugging aid, not a deployment mode: ``./camera_viz.sh loopback -configs/v4l2.yaml`` runs the sender and viewer together on ``127.0.0.1`` — the quickest way to -smoke-test the RTP path on one machine. It also works camera-free with a mono ``type: video`` -entry (set ``width`` / ``height`` / ``fps``). +Test RTP on one machine with ``./camera_viz.sh loopback configs/v4l2.yaml``. This also supports +mono video recordings with explicit ``width``, ``height``, and ``fps``; stereo video replay is +unsupported over RTP. Configuration ------------- -A single YAML drives both capture and visualization. Each entry in the ``cameras`` list becomes -its own plane (and, in split mode, its own RTP port). Abbreviated: +Start from a :code-dir:`source config ` and adjust the camera +settings for your device. Placement keys must match the camera's ``name``: .. code-block:: yaml - source: local | rtp - streaming: - host: 192.168.1.100 # workstation IP (overridden at deploy time) - encoder: auto | native | gstreamer - - cameras: - - name: cam - enabled: true - type: v4l2 # v4l2 | oakd | zed | video | synthetic - width: 2560 # video: optional — defaults to the file's size - height: 720 # (required when source: rtp) - fps: 30 - stereo: false # zed / video / synthetic — per-eye capture + SBS in XR - path: clip.mp4 # video only — file to replay, relative to this YAML - loop: true # video only — rewind at end of file - rtp: - port: 5000 # left eye when stereo - port_right: 5001 # required when stereo + source: rtp - bitrate_mbps: 15 - display: - mode: xr | window # default: xr - window: { width, height } - xr: - near_z: 0.05 # default - far_z: 100.0 # default - system_wait_seconds: -1 # default; wait forever (0 fails fast) - clear_color: [r, g, b, a] + mode: xr placements: cam: - lock_mode: lazy # world | head | lazy | gimbal + lock_mode: lazy distance: 1.5 - # size: [w_m, h_m] - # stereo_plane_distance_cm: 0 # gap between the eyes' planes - # shape: quad # quad | cylinder | equirect (cylinder/equirect are XR-only) - # equirect_yaw_deg: 0.0 # equirect: heading the middle of the feed points at - # compositor: openxr # openxr (default) | televiz — quads only - # cylinder_radius_m: 2.0 - # cylinder_angle_deg: 90 + shape: quad + size: [1.0, 0.56] # meters -See the :code-dir:`configs/ ` directory for a complete, commented -YAML per source kind. +Add entries to ``cameras`` and ``display.placements`` for multiple feeds. In split mode, assign +each stream its own RTP port; stereo sources also require ``rtp.port_right``. +The :code-file:`README ` documents the full schema. .. _camera-streaming-troubleshooting: Troubleshooting --------------- -- **The XR session fails to create** — check ``~/.cloudxr/logs/cxr_server.*.log`` and - ``~/.cloudxr/logs/runtime_worker_stderr.log`` (``runtime_stderr.log`` with - ``ISAACCAPTURE_LOGGING=off``) for the startup failure. - ``XR_ERROR_FORM_FACTOR_UNAVAILABLE`` - (-35) means the runtime is up but no headset has clicked CONNECT yet; XR mode waits - according to ``display.xr.system_wait_seconds`` (default ``-1`` to wait - indefinitely; ``0`` fails fast; positive values wait that many seconds; - override with ``--xr-wait``). Pass ``--mode window`` to - render to a desktop window instead (no runtime involved). -- **No window appears over SSH** — ``--mode window`` needs a local display; run on the machine - you're sitting at, or use a video-capable remote desktop. -- **"video source: no such file"** — relative ``path:`` values resolve against the YAML's - directory (``configs/``), not the directory you launched from. -- **A source fails asking for CuPy / CUDA** — rerun the CUDA execution check above with the - sample's ``.venv/bin/python``. ``nvidia-smi`` checks the driver; its CUDA version does not - establish that the matching toolkit, NVRTC, or headers are installed. For missing - ``libnvrtc.so``, ``cuda_runtime.h``, or ``cuda_fp16.h``, check the development components and - ``/usr/local/cuda`` target (:ref:`camera-streaming-jetson-cuda` on Jetson), then rerun setup - with ``--jetson`` on Jetson. Setup selects CuPy for that toolkit's major version. -- **Setup succeeded but another Python cannot import the dependencies** — the wrapper uses - ``examples/camera_viz/.venv/bin/python`` regardless of the active shell environment. Use - that interpreter for diagnostics, or rerun ``setup --venv PATH`` to select your environment. -- **No video on Orin.** When the CloudXR runtime runs on Jetson Orin, set - **Video Codec** to **H.264** in the CloudXR web client. See - :ref:`connect-xr-headset`. -- **Black camera plane on Orin** — with the CloudXR experimental runtime on Orin, the default - OpenXR compositor shows the camera plane but leaves its contents black. Under the camera's - ``display.placements`` entry, set ``compositor: televiz`` (quads only). The same feed displays - correctly with this workaround: +- **XR session does not start:** connect the client first. ``XR_ERROR_FORM_FACTOR_UNAVAILABLE`` + (-35) means no headset is connected. The viewer waits indefinitely by default; set + ``display.xr.system_wait_seconds`` or ``--xr-wait`` to change the timeout. + Check ``~/.cloudxr/logs/cxr_server.*.log`` and ``~/.cloudxr/logs/runtime_worker_stderr.log`` + (``runtime_stderr.log`` with ``ISAACCAPTURE_LOGGING=off``). +- **No video on Orin:** set the client's **Video Codec** to **H.264**. +- **Black camera plane on Orin:** set ``compositor: televiz`` under the camera's placement + and restart. This workaround supports quads only; for the replay config: .. code-block:: yaml display: placements: - cam: + replay: compositor: televiz -- **Split mode renders nothing** — check the sender is up (``./camera_viz.sh service-status``), - ``$STREAMING_HOST`` was the workstation's IP at deploy time, and UDP ports (default 5000+) - aren't firewalled. -- **Not sure which side is stuck?** — set ``verbose: true`` at the top of the YAML for periodic - per-source breadcrumbs on both ends. +- **Corrupted or frozen video:** stop the viewer and run + ``./camera_viz.sh run configs/synthetic.yaml`` to check the display path without a camera + or recording. Add ``--mode window`` to check local rendering separately from XR streaming. +- **No window over SSH:** window mode needs a desktop display or video-capable remote desktop. +- **Video file fails to open:** paths are relative to the YAML directory. For the bundled clip, + run the Git LFS command in the first-run instructions. +- **Python cannot find dependencies:** use ``.venv/bin/python`` or rerun ``setup --venv PATH``. +- **Split mode has no frames:** check ``./camera_viz.sh service-status``, ``STREAMING_HOST``, + and the workstation's UDP ports. Set ``verbose: true`` in YAML for per-source logs. -How it works ------------- - -The sample is organized so that capture, transport, and visualization are cleanly separated, with -Televiz as the compositor at the end of the chain: - -.. code-block:: text - :class: code-100col - - camera_viz/ - ├── camera_viz.sh — CLI: setup / loopback / run / deploy / service-* - ├── camera_viz.py — receiver / viewer (drives a Televiz VizSession) - ├── camera_streamer.py — robot-side RTP sender (per-camera supervisor) - ├── pipeline/ — source ABC + threaded runner - ├── placements/ — XR lock-mode strategies (world / head / lazy / gimbal) - ├── sources/ — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264 - ├── transports/ — RTP sender + receiver (native + GStreamer) - ├── codec/ — native NVENC / NVDEC pybind module - ├── configs/ — one YAML per source kind - ├── test_data/ — sample replay clip (Git LFS) - └── scripts/ — installer + systemd unit template - -- **Sources** (:code-dir:`sources/ `) implement a common source ABC and - hand frames to a threaded runner in :code-dir:`pipeline/ `. Each - source produces GPU frames where possible — e.g. the ZED source uses ``retrieve_image(MEM.GPU)`` so - BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip. -- **The viewer** (:code-file:`camera_viz.py `) creates a - ``VizSession`` and adds one ``QuadLayer`` per enabled camera, then submits each frame to its layer - and calls ``render()`` once per frame. Stereo cameras submit both eyes. -- **Transport** (:code-dir:`transports/ `) carries the split mode: - an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec - module (:code-dir:`codec/ `) or a GStreamer fallback. -- **Placement** (:code-dir:`placements/ `) holds the XR lock-mode - strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app - sets each frame. - -Sharing the XR session with TeleopSession ------------------------------------------ +CuPy / CUDA errors +^^^^^^^^^^^^^^^^^^ -Only one OpenXR session is allowed per process, so when this sample runs alongside teleoperation the -``VizSession`` owns the session and hands its handles to ``TeleopSession`` / ``DeviceIOSession``, -which then skip creating their own. The viewer builds the session with the trackers' required -extensions and forwards the handles through ``TeleopSessionConfig.oxr_handles``: +Setup checks imports, not GPU execution. Test a small kernel with the sample's interpreter: -.. code-block:: python +.. code-block:: bash - cfg.required_extensions = DeviceIOSession.get_required_extensions(trackers) - viz_session = televiz.VizSession.create(cfg) + .venv/bin/python - <<'PY' + import cupy as cp - config = TeleopSessionConfig( - app_name="MyApp", - pipeline=pipeline, - oxr_handles=OpenXRSessionHandles(*viz_session.get_oxr_handles()), - ) + values = cp.arange(4, dtype=cp.float32) + assert cp.asnumpy(values + 1).tolist() == [1.0, 2.0, 3.0, 4.0] + print("CUDA kernel check passed") + PY -See the *Sharing the XR session* section of :doc:`/getting_started/televiz` for the full pattern -(imports, frame loop, and how the two sessions' lifecycles relate), and -:doc:`/getting_started/teleop_session` for the ``TeleopSession`` side. +Missing ``libnvrtc.so``, ``cuda_runtime.h``, or ``cuda_fp16.h`` indicates missing or mismatched +CUDA development components. Check ``/usr/local/cuda`` and rerun setup (``--jetson`` on Jetson). +The CUDA version reported by ``nvidia-smi`` describes driver support, not installed headers. +See :ref:`camera-streaming-jetson-cuda`. Next steps ---------- -After the :ref:`camera-streaming-first-session` checks pass, choose the next step for your use case: - -- **See a live camera:** follow :ref:`camera-streaming-live-camera`, then repeat the visibility - and image-quality checks with motion in front of the camera. With a stereo source, also - check both eyes and the ``B`` mono / stereo toggle. -- **Tune the view:** adjust placement, size, and lock mode using the controller bindings and - YAML settings above. Add entries to ``cameras`` for multiple feeds, or use - :ref:`recorded-camera-streaming` to replay your own recording. -- **Stream from a robot:** use :ref:`split-mode` when capture and viewing run on different - machines, after validating direct mode locally. -- **Integrate camera views into teleoperation:** follow :ref:`sharing-the-xr-session` to share - Televiz's OpenXR session with :doc:`/getting_started/teleop_session`, and use the - :doc:`retargeting interface ` to map tracking inputs to robot - actions. The camera viewer itself does not command a robot. +- :ref:`Connect a live camera ` or + :ref:`replay your own recording `. +- :ref:`Stream from a robot ` when capture and viewing run on separate machines. +- :ref:`Integrate with teleoperation ` by sharing Televiz's OpenXR + session with ``TeleopSession``. Only one OpenXR session is allowed per process. +- :doc:`Build a custom viewer ` using the Televiz API and + :code-file:`camera_viz.py ` as a reference. From ca26a6f1e32524b6882a378ebc950131b40e0f3d Mon Sep 17 00:00:00 2001 From: Farbod Motlagh Date: Tue, 6 Oct 2026 15:54:38 -0700 Subject: [PATCH 3/5] fix cupy cuda mismatch Signed-off-by: Farbod Motlagh --- docs/AGENTS.md | 11 +- docs/source/references/camera_streaming.rst | 719 ++++++++++++------ examples/camera_viz/README.md | 16 + examples/camera_viz/pipeline/interface.py | 3 +- examples/camera_viz/scripts/_install_deps.sh | 25 +- examples/camera_viz/scripts/check_cuda.py | 121 +++ examples/camera_viz/sources/_helpers.py | 4 +- examples/camera_viz/sources/oakd.py | 3 +- examples/camera_viz/sources/rtp_h264.py | 4 +- examples/camera_viz/sources/synthetic.py | 44 +- examples/camera_viz/sources/zed.py | 4 +- tests/AGENTS.md | 2 + tests/python/examples/camera_viz/conftest.py | 8 + .../examples/camera_viz/test_cuda_setup.py | 141 ++++ .../examples/camera_viz/test_runner_health.py | 15 + .../camera_viz/test_synthetic_failures.py | 80 ++ .../camera_viz/test_synthetic_offscreen.py | 10 +- 17 files changed, 938 insertions(+), 272 deletions(-) create mode 100644 examples/camera_viz/scripts/check_cuda.py create mode 100644 tests/python/examples/camera_viz/test_cuda_setup.py create mode 100644 tests/python/examples/camera_viz/test_synthetic_failures.py diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 0510cf0180..238ccdf895 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -98,8 +98,11 @@ and `logo_dark`. Keep cache-removal documentation focused on migration impact. Omit asset-fetch implementation walkthroughs and alternative override instructions. -## Developer-facing guides +## Scope of documentation revisions -Lead with concise prerequisites, a runnable example, and observable success checks. -Keep optional setup and diagnostics outside the first-run path. Link to existing -reference material instead of repeating implementation details. +When asked to shorten additions, edit only the branch's additions. Preserve the +existing reference content, examples, and structure. Propose broader cleanup +separately; do not apply it without agreement. Keep new prerequisites and checks concise. + +Distinguish runtime dependencies from source-build tools. Make installation steps +conditional when existing environments may already satisfy the requirements. diff --git a/docs/source/references/camera_streaming.rst b/docs/source/references/camera_streaming.rst index c3d620b1ae..be030ddd62 100644 --- a/docs/source/references/camera_streaming.rst +++ b/docs/source/references/camera_streaming.rst @@ -4,83 +4,201 @@ Camera Streaming ================ -Use :code-dir:`camera_viz ` to stream live cameras or recordings to an XR -headset or desktop window. The sample uses :doc:`Televiz ` and supports -cameras attached locally or streamed from a robot over RTP. +``camera_viz`` is the reference camera-streaming sample built on :doc:`Televiz +` (``isaaccapture.viz``). It captures frames from one or more cameras +and streams them to an XR headset — one plane per camera, aspect-fit — either directly or from a +robot to a workstation over the network (split mode). It can also stream :ref:`recorded camera +data ` — a video file replayed in place of a live camera — and render +to a desktop window instead of the headset. + +The sample lives at :code-dir:`examples/camera_viz/ `. This page walks you +from setup and a camera-free first run to real cameras and the robot → workstation split mode. +For the exact command surface and flags, see the +:code-file:`README `. .. figure:: ../_static/televiz_2d.gif - :alt: camera_viz with multiple camera feeds + :alt: camera_viz in window mode with synthetic multi-camera feeds :width: 420px :class: no-image-zoom - One plane per camera. + ``camera_viz`` in XR mode — one plane per camera. .. contents:: On this page :local: - :depth: 1 + :depth: 2 Requirements ------------ -- **Host:** Ubuntu with an NVIDIA GPU; see :doc:`system requirements `. - Jetson: Orin with JetPack 6.2.x or 7.2.1; Thor with JetPack 7.x. -- **CUDA:** runtime, NVRTC, and development headers matching the installed toolkit. +- A workstation meeting the :doc:`system requirements ` (Ubuntu, NVIDIA + GPU, CUDA driver) — every source hands frames to the renderer GPU-resident via CuPy. +- For Jetson platforms: + + - **Orin:** `JetPack 6.2.x `__ or + `7.2.1 `__ + - **Thor:** `JetPack 7.x `__ +- For the default XR mode, a headset to connect as the CloudXR client — follow the + :doc:`quick start ` step :ref:`connect-xr-headset`. The viewer + launches the CloudXR runtime itself; nothing to start separately. No headset handy? + ``--mode window`` renders to a desktop window instead. + +- **CUDA development components:** matching runtime, NVRTC, and headers for CuPy kernels. JetPack 7.2.1 uses CUDA 13.2; see :ref:`camera-streaming-jetson-cuda`. - A GPU is required even for recordings and window mode. -- **Display:** a :ref:`supported XR headset ` with network access to the host - and :ref:`CloudXR ports ` open, or a desktop display for window mode. - **Camera — optional, recommended:** USB / UVC, OAK-D, or supported ZED camera for a live feed. - Start with the bundled recording if no camera is available. ZED requires its SDK. -- **Controllers — optional:** for adjusting the view in XR. + A recording or synthetic pattern works without a camera; ZED SDK capture needs ``setup --with-zed``. +- **XR controllers — optional:** for adjusting the view with :ref:`camera-streaming-controls`. +- **CloudXR setup:** see Quick Start for :ref:`first-launch requirements ` + and :ref:`firewall configuration `. Setup ----- -From a :ref:`repository checkout `, run: +Clone the repository if you haven't already (quick start step :ref:`check-out-code-base`), then +run the sample's one-time setup: .. code-block:: bash - cd examples/camera_viz - ./camera_viz.sh setup + examples/camera_viz/camera_viz.sh setup + source examples/camera_viz/.venv/bin/activate + +On Jetson, add ``--jetson``. If CUDA components are missing, see +:ref:`camera-streaming-jetson-cuda` below. + +There is no need to install the ``isaaccapture`` pip package yourself. ``setup`` builds the +sample's own environment: ``isaaccapture[cloudxr]`` — the ``cloudxr`` extra is not optional here, +since XR is the default display mode and the viewer launches the runtime itself — plus every +other Python dependency, into ``.venv/`` via ``uv``. It resolves a release matching the checkout's +major/minor version, falling back to a release candidate and then, after asking, to a source build +of the surrounding checkout (see :code-file:`scripts/_install_deps.sh +`; ``--wheel`` and ``--build-from-source`` override +the choice). When required system packages are missing, it prints the exact ``apt-get`` line +to approve — declining, or a non-interactive run, aborts. + +By default ``setup`` provisions the direct-mode path — USB / UVC and OAK-D camera support. Split +mode (RTP) and ZED support are opt-in, since both pull in dependencies the direct path never +needs. Flags trim or extend that: + +.. list-table:: + :header-rows: 1 + :widths: 22 78 + + * - Flag + - Effect + * - ``--no-v4l2`` + - Skip USB / UVC webcam support (``opencv-python``). + * - ``--no-oakd`` + - Skip OAK-D support (``depthai``). + * - ``--with-rtp`` + - Also install the split-mode dependencies: the GStreamer system packages, PyGObject, and the + native NVENC/NVDEC codec build. Required for ``loopback`` and for any config with + ``source: rtp``; implied by ``--sender-only``. Direct mode does not need it. + * - ``--with-zed`` + - Also build + install the ZED SDK's Python API (``pyzed``). Requires the ZED SDK on the + machine (default ``/usr/local/zed``; override with ``--zed-sdk PATH``). + * - ``--sender-only`` + - Split mode only — robot-side install of just the sender's dependencies. + * - ``--jetson`` + - Enable Jetson CUDA library discovery for direct or split mode. With RTP enabled, also + check for the matching NVRTC package. + * - ``--venv PATH`` + - Use ``PATH`` for the environment and link the sample's ``.venv`` to it; see below. + * - ``--wheel PATH`` + - Install a locally built ``isaaccapture`` wheel instead of resolving one from the index — for + developing Isaac Teleop itself. + * - ``--build-from-source`` + - Skip the package index and build ``isaaccapture`` from the surrounding checkout without + prompting — a full C++ / CUDA / Vulkan build (see + :doc:`/getting_started/build_from_source/index`). + +.. _camera-streaming-external-venv: + +Using an external virtual environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To reuse a Python 3.12 environment, run from the repository root: + +.. code-block:: bash + + examples/camera_viz/camera_viz.sh setup --venv /absolute/path/to/venv + +Setup installs dependencies there and links ``examples/camera_viz/.venv`` to it. Move an existing +``.venv`` directory aside first. The wrapper uses ``.venv/bin/python`` regardless of the active +shell environment; activation is optional. Add ``--jetson`` on Jetson. + +.. _camera-streaming-jetson-cuda: + +Jetson CUDA prerequisites +^^^^^^^^^^^^^^^^^^^^^^^^^ + +CuPy compiles GPU kernels at runtime using NVRTC and CUDA headers +(`CuPy requirements `__). +**If these components are already installed, skip this step.** + +On JetPack 7.2.1 / CUDA 13.2, the `JetPack CUDA development package +`__ +provides them. With the matching NVIDIA repositories configured: + +.. code-block:: bash + + sudo apt update + sudo apt install nvidia-cuda-dev -Setup creates a Python 3.12 ``.venv`` with ``isaaccapture[cloudxr]``, UVC, and OAK-D support. -It needs network access and may prompt for sudo to install system dependencies. -On Jetson, install the :ref:`CUDA prerequisites ` first and add -``--jetson``. To reuse an environment, use :ref:`setup --venv `. +Setup selects CuPy from ``CUDA_PATH``, the toolkit containing ``nvcc`` on ``PATH``, or +``/usr/local/cuda``, in that order, and verifies GPU kernel execution. ``--jetson`` configures +library discovery. A separate ``nvcc`` compiler is needed for CUDA source builds, such as the native +RTP codec, rather than ordinary CuPy kernel execution. -All commands below run from ``examples/camera_viz``. No environment activation is needed. +.. _recorded-camera-streaming: First run — a recording, no camera required ------------------------------------------- -Fetch the bundled clip with Git LFS, then launch the viewer: +The recorded-camera source (``type: video``) replays a video file through exactly the same +source → layer → Televiz path a live camera uses. It is the supported way to stream recorded +camera data, it needs no extra setup, and it is the quickest end-to-end check. A test clip ships +with the repo, and :code-file:`configs/replay.yaml ` +already points at it. Fetch the clip with Git LFS if your checkout contains only its pointer: .. code-block:: bash git lfs pull --include="examples/camera_viz/test_data/recording.mp4" - ./camera_viz.sh run configs/replay.yaml -The viewer starts CloudXR automatically; accept the EULA on first launch. Open the printed -client URL in your headset browser, accept the certificate, and click **Connect** -(:ref:`connection instructions `). +Keep OpenCV installed for replay (``--no-v4l2`` skips it). From the repository root: + +.. code-block:: bash + + cd examples/camera_viz + ./camera_viz.sh run configs/replay.yaml # XR headset (default) + ./camera_viz.sh run configs/replay.yaml --mode window # desktop window instead + +In XR mode the viewer first brings up the CloudXR runtime (accept the EULA on first launch, or +pass ``--accept-eula``). Open the printed client URL in your headset browser, accept the host's +certificate, and click **Connect**, following :ref:`connect-xr-headset`. On Jetson Orin, select +**H.264** in the client's **Video Codec** setting before connecting. + +**You should see** the terminal report the session and the source coming up:: + + camera_viz: source=local, mode=xr, xr=True, shapes=quad, 1 layer(s) + [video] opening... + [video] connected + [video] streaming -For a desktop preview instead, add ``--mode window`` to the run command. -On Orin, select **H.264** in the client and apply the -:ref:`black-plane workaround ` below. +and the clip looping on a plane in the headset once it connects — or in a desktop window +(``mode=window, xr=False``) with the ``--mode window`` override, which starts no runtime. .. _camera-streaming-first-session: First-session checklist ^^^^^^^^^^^^^^^^^^^^^^^ -With the default replay config: +With the default ``replay.yaml``: - **Playback:** one plane is visible and the clip loops. - **Head movement:** small turns and leaning change your viewpoint; the plane stays anchored. - **Lazy recentering:** look more than 45° away for about a second; the plane moves smoothly back in front of you. -- **Image quality:** no freezing, flicker, torn regions, or unexpected colors, including while - moving your head. This mono clip should show the same source image in both eyes. +- **Image quality:** no freezing, flicker, torn regions, or unexpected colors while still or + moving. This mono clip should show the same source image in both eyes. - **Controllers, if available:** right-stick click recenters; ``A`` changes lock mode; ``Y`` restores the config. - **Restart:** press ``Ctrl+C``, rerun, and reconnect if needed; playback resumes. @@ -88,74 +206,94 @@ With the default replay config: In window mode, check playback, image quality, and restart. If a check fails, see :ref:`camera-streaming-troubleshooting`. -.. _camera-streaming-live-camera: +Replaying your own recording +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Running with a real camera --------------------------- +Point ``path`` at any file OpenCV's FFmpeg backend reads — mp4 / mkv / webm carrying H.264, +HEVC, AV1, and so on. The file is probed once at startup for its size and frame rate (a missing +or absurd rate falls back to 30 fps); a missing or unreadable file is a configuration error and +fails immediately instead of retrying like a camera would. A complete config: -Attach the camera to the viewer host and keep ``source: local``. Edit the matching config's -device, resolution, and frame rate, then run: +.. code-block:: yaml -.. code-block:: bash + source: local - ./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml + cameras: + - name: replay + enabled: true + type: video + path: ../test_data/recording.mp4 # required; relative to this YAML's directory + loop: true # false = hold the last frame at end of file + fps: 0 # 0 = the file's native rate + stereo: false # true = split a side-by-side recording into eyes + # width/height default to the file's native size (per eye when stereo) + + display: + mode: xr # or: window + placements: + replay: { lock_mode: lazy, distance: 1.5 } -``v4l2.yaml`` defaults to a ZED Mini's 2560×720 UVC mode. For a regular webcam, use modes reported -by ``v4l2-ctl --list-formats-ext``. ZED SDK capture also needs ``setup --with-zed``. +With ``source: rtp``, ``width``, ``height``, and ``fps`` become required — the sender paces its +encode loop at ``fps`` and the receiver sizes its decoder from the config — and ``stereo`` is +rejected, since the sender needs per-eye streams. A mono recording is otherwise a fine capture +side for :ref:`split mode ` and for ``loopback``, which exercises the whole +encode → UDP → decode path with no camera attached. .. _camera-streaming-sources: Supported sources -^^^^^^^^^^^^^^^^^ +----------------- -Set ``type`` in each ``cameras`` entry. Multiple entries produce one plane per camera. -See :code-dir:`configs/ ` for complete examples. +The source kind is selected by the ``type`` field of each entry in the YAML ``cameras`` list: .. list-table:: :header-rows: 1 :widths: 16 84 - * - Type - - Source + * - ``type:`` + - Notes * - ``v4l2`` - - USB / UVC camera. + - USB / UVC cameras — anything ``v4l2-ctl --list-formats-ext`` reports. * - ``oakd`` - - OAK-D RGB / LEFT / RIGHT; ``stereo: true`` for grayscale stereo. + - OAK-D RGB / LEFT / RIGHT; mono or ``stereo: true``. Stereo ships GRAY8 over USB to halve + bandwidth and is broadcast to RGBA on the GPU; ``stereo_rgb`` keeps color. * - ``zed`` - - ZED 2 / Mini / X One through the ZED SDK; mono or stereo. + - ZED 2 / Mini / X One; mono or ``stereo: true`` (per-eye SDK retrieve, zero-copy on the GPU). * - ``video`` - - :ref:`Recorded video `, including side-by-side stereo. + - Recorded camera data — video-file replay (anything OpenCV's FFmpeg backend reads). Loops + by default; ``stereo: true`` splits side-by-side recordings into eyes (viewer only). See + :ref:`recorded-camera-streaming`. * - ``synthetic`` - - Animated GPU test pattern; no camera or file needed. + - Debugging tool — GPU-generated test pattern, no camera or file required. -.. _recorded-camera-streaming: +.. _camera-streaming-live-camera: -Replaying your own recording ----------------------------- +Running with a real camera +-------------------------- -Edit :code-file:`configs/replay.yaml `: +Attach the camera to the machine that runs the viewer, keep ``source: local`` in the config, and +run with the matching config. First edit its device, resolution, and frame rate to match your +camera. In particular, ``v4l2.yaml`` defaults to a ZED Mini's 2560×720 UVC mode; a regular webcam +needs values it supports (inspect them with ``v4l2-ctl --list-formats-ext``): -.. code-block:: yaml +.. code-block:: bash - cameras: - - name: replay - type: video - path: /path/to/clip.mp4 - loop: true - fps: 0 # use the file's frame rate - stereo: false # true for side-by-side recordings in direct mode + ./camera_viz.sh run configs/v4l2.yaml # or oakd.yaml / zed.yaml -Relative paths resolve from the YAML's directory. OpenCV's FFmpeg backend must support the -file's format. Size and frame rate default to the recording's values; ``loop: false`` holds -the last frame. Keep setup's OpenCV dependency for replay (``--no-v4l2`` skips it). +**You should see** the same startup lines as above with the camera's tag (``[v4l2]``, +``[oakd]``, ``[zed]``) and the live feed. Multiple entries in the ``cameras`` list render as one +plane each. Display modes ------------- -XR is the default. Use ``--mode window`` or ``display.mode: window`` for a desktop preview; -stereo sources show only the left eye in window mode. +XR is the default: each camera renders as its own plane in the headset via the active OpenXR +runtime, and stereo sources (``stereo: true``) render true side-by-side stereo. Pass +``--mode window`` (or set ``display.mode: window`` in the YAML) to render to a desktop window +instead — no headset or runtime needed; stereo shows the left eye. -In XR, set ``display.placements..lock_mode``: +In XR, how a plane follows the operator's head is the per-camera ``lock_mode`` under +``display.placements.``: .. list-table:: :header-rows: 1 @@ -164,23 +302,28 @@ In XR, set ``display.placements..lock_mode``: * - Mode - Behavior * - ``world`` - - Stays where initially placed. + - Placed once in front of you and stays put. * - ``head`` - - Follows your head every frame; may trail fast motion. + - Follows your head every frame. Head-locked content updates its pose at application + rate, so it trails fast head motion by roughly a frame — expected for any head-locked + OpenXR layer; prefer ``lazy`` unless you need a true HUD. * - ``gimbal`` - - Follows your position while keeping its initial heading. - * - ``lazy`` (default) - - Stays anchored until you look or move away, then recenters smoothly. + - Follows your position but not your rotation: the surface stays pointed where you first + looked, walks with you, and turning your head looks around it. The natural mode for + wide cylinder feeds (a "virtual gimbal"). + * - ``lazy`` + - World-locked, but re-snaps in front of you when you look away (default). -Lazy-mode settings: ``look_away_angle_deg``, ``reposition_distance``, ``reposition_delay_s``, -and ``transition_duration_s`` under the camera's placement. +Lazy-mode knobs live under ``placements.``: ``look_away_angle_deg``, +``reposition_distance``, ``reposition_delay_s``, ``transition_duration_s``. .. _camera-streaming-controls: Controller bindings ^^^^^^^^^^^^^^^^^^^ -Changes apply to all cameras and appear in the terminal and headset status panels. +In XR the controllers retune the view live, without editing the YAML and restarting. The right +hand changes how the feed looks; the left, what surface it is mapped onto: .. Use concise titles and Televiz branding; name camera_viz in the prose. @@ -189,6 +332,8 @@ Changes apply to all cameras and appear in the terminal and headset status panel right controller adjusts depth and placement. :width: 100% + Bindings at a glance. The table below is the same thing in words. + .. list-table:: :header-rows: 1 :widths: 22 78 @@ -196,251 +341,349 @@ Changes apply to all cameras and appear in the terminal and headset status panel * - Input - Effect * - Right stick ←/→ - - Adjust stereo plane gap; pan the panorama on ``equirect``. + - Stereo plane gap — how far apart the two eyes' planes sit + (``placements..stereo_plane_distance_cm``). Widening it pushes the scene back + instead of packing it into the space in front of the planes. Stereo cameras only; on + ``equirect``, which has no gap to set, the same stick pans the panorama. * - Right stick click - - Recenter the surface or panorama. + - Recenter on your view: an ``equirect`` yaws so the middle of the panorama lands dead + ahead, and a placed surface re-snaps its anchor. * - ``A`` - - Cycle ``world`` → ``head`` → ``gimbal`` → ``lazy``. + - Cycle the lock mode: ``world`` → ``head`` → ``gimbal`` → ``lazy``. * - ``B`` - - Toggle mono / stereo for stereo sources. + - Toggle mono / stereo. Stereo cameras only. * - ``X`` - - Cycle ``quad`` → ``cylinder`` → ``equirect``. + - Cycle the shape: ``quad`` → ``cylinder`` → ``equirect``. * - ``Y`` - - Reset to YAML settings. + - Reset everything to the YAML values. * - Left stick - - Adjust size / vertical position; arc width for cylinders, span for panoramas. + - Retunes the active shape: ``quad`` size / vertical position, ``cylinder`` arc width / + vertical position, ``equirect`` horizontal / vertical span. + +Changes apply to every camera at once and appear both on the terminal status panel and on a +head-locked panel in the headset that auto-hides shortly after. Neither toggle reallocates: +``B`` sends the left frame to both eyes, and ``X`` flips visibility between shapes built at +startup, so the extra shapes cost VRAM (reported at startup) rather than a stall on the press. + +The stereo gap is bounded below **divergent parallax** — a gap wider than your IPD would need +the eyes to splay outward — and the headset's measured IPD sets that ceiling, not the config. +The HUD suggests a gap derived from the plane distance and that IPD. -The stereo gap is limited by the headset's measured IPD. Configure bindings and rates under -``display.controls``; see the :code-file:`README `. +Bindings, rates and limits live under ``display.controls``; see the +:code-file:`README ` for the full set. Display surfaces -^^^^^^^^^^^^^^^^ +---------------- -Set ``shape`` under ``display.placements.``: +By default each camera renders on a flat plane, which suits normal-FOV feeds. Wide-FOV and +panoramic sources look better on a curved surface: set ``shape`` per camera under +``display.placements.``: .. list-table:: :header-rows: 1 - :widths: 24 76 + :widths: 30 70 * - Shape - - Use + - Behavior * - ``quad`` (default) - - Flat camera feed. Set ``size: [width_m, height_m]``. + - Flat plane. All lock modes apply. * - ``cylinder`` - - Wide-FOV feed. Set ``cylinder_radius_m`` and ``cylinder_angle_deg``. + - Curved arc facing the operator, so the image stays at a constant viewing distance edge + to edge. ``cylinder_radius_m`` (default 2.0) sets that distance, ``cylinder_angle_deg`` + (default 90) the arc width. All lock modes apply (``head`` = gaze-tracking curved + visor). * - ``equirect`` - - 360°×180° panorama. Set ``equirect_yaw_deg`` to adjust heading; lock modes do not apply. - -Curved surfaces require XR. The OpenXR runtime composites surfaces by default. Flat planes can -use ``compositor: televiz`` instead; this is required with the experimental runtime on Orin. -See :ref:`openxr-composition-layers` for compositor details. - -Setup options -------------- - -.. list-table:: - :header-rows: 1 - :widths: 26 74 - - * - Flag - - Effect - * - ``--jetson`` - - Configure CUDA library discovery on Jetson; with RTP, also check for NVRTC. - * - ``--venv PATH`` - - Install into ``PATH`` and link the sample's ``.venv`` to it. - * - ``--with-rtp`` - - Install GStreamer, PyGObject, and native codec dependencies for split mode or loopback. - * - ``--with-zed`` - - Install the ZED Python API. Requires the SDK at ``/usr/local/zed`` or ``--zed-sdk PATH``. - * - ``--no-v4l2`` / ``--no-oakd`` - - Skip OpenCV / DepthAI respectively. Video replay also needs OpenCV. - * - ``--sender-only`` - - Install only the robot sender's dependencies; implies ``--with-rtp``. - * - ``--wheel PATH`` - - Install a locally built ``isaaccapture`` wheel. - * - ``--build-from-source`` - - Build from this checkout; requires the :doc:`build environment `. - -.. _camera-streaming-external-venv: - -External virtual environment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To reuse an existing Python 3.12 environment: - -.. code-block:: bash - - ./camera_viz.sh setup --venv /absolute/path/to/venv - -Setup installs dependencies there and links ``.venv`` to it. If ``.venv`` is already a real -directory, move it aside first. The wrapper always uses ``.venv/bin/python``, regardless of -the environment activated in your shell. - -.. _camera-streaming-jetson-cuda: - -Jetson CUDA prerequisites -^^^^^^^^^^^^^^^^^^^^^^^^^ - -JetPack 7.2.1 (L4T R39.2.1) needs CUDA 13.2 development components, including headers and NVRTC. -With the matching NVIDIA package repositories configured, install them using the -`JetPack CUDA development package -`__: - -.. code-block:: bash - - sudo apt update - sudo apt install nvidia-cuda-dev - readlink -f /usr/local/cuda - /usr/local/cuda/bin/nvcc --version - -On JetPack 7.2.1, verify that ``/usr/local/cuda`` selects CUDA 13.2. On JetPack 6.2.x, keep the -matching CUDA 12.x toolkit. Setup selects CuPy from this path; ``--jetson`` configures library -discovery but does not install the full toolkit. See NVIDIA's -`JetPack installation guide -`__ for BSP setup. + - Full 360°×180° sphere around the operator, for equirectangular panorama / VR-video + sources. The middle of the texture points along ``equirect_yaw_deg`` (0 = the + direction the headset last recentered on, positive to the left), which is how a camera + mounted facing another way gets aimed. Lock modes don't apply — the sphere is centred + on the operator and follows them everywhere. + +Curved shapes exist only in XR mode — the viewer exits with an error in window mode. Stereo +sources render per-eye textures on the same surface, and ``stereo_baseline_mm`` adds a per-eye +pose shift (no effect on the equirect sphere at its default infinite radius). + +Surfaces are composited by the OpenXR runtime, which keeps them sharp under head motion and lets +CloudXR stream them efficiently (see +:ref:`OpenXR composition layers `). For flat planes only, +``compositor: televiz`` opts a camera back into Televiz's built-in compositor. On Jetson Orin +(CloudXR experimental runtime), set ``compositor: televiz`` so the plane is not left black; see +Troubleshooting below. On the Televiz 2D API the same switch is +``openxr_composition = False`` (see :ref:`Jetson Orin, XR `). CloudXR runtime flags --------------------- -The viewer attaches to an existing CloudXR service or starts one. Common run flags: +In XR mode the viewer attaches to the running CloudXR runtime, or starts one if none is +serving. Useful flags: -- ``--accept-eula`` — accept the EULA non-interactively. -- ``--cloudxr-device-profile PROFILE`` — device profile (default ``Quest3``). -- ``--no-host-client`` — disable the hosted client at ``https://:48322/client/`` when - starting a service. Stop an existing service first to change hosting. +- ``--accept-eula`` — accept the CloudXR EULA non-interactively (first run only). +- ``--cloudxr-device-profile PROFILE`` — ``NV_DEVICE_PROFILE`` (default ``Quest3``). +- ``--no-host-client`` — disable serving CloudXR.js at + ``https://:48322/client/``. Only applies when this run starts the + service; stop and restart the service to change hosting. -Inspect or stop the service with the sample's interpreter: +When the viewer starts a CloudXR service with hosting enabled (the default; +``--no-host-client`` disables it), it prints the local ``/client/`` URL. To +inspect or stop an already-running service, activate the example venv (or any +environment with ``isaaccapture``) and run:: -.. code-block:: bash + python -m isaaccapture.cloudxr.service status + python -m isaaccapture.cloudxr.service stop - .venv/bin/python -m isaaccapture.cloudxr.service status - .venv/bin/python -m isaaccapture.cloudxr.service stop +Run ``camera_viz.py --help`` for the rest (install dir, env-config file, WSS proxy toggle). -Set runtime overrides in YAML: +Runtime settings themselves are declared in the config rather than exported. The viewer ships +defaults (pose wait off, runtime foveation on), and ``display.cloudxr`` overrides them per +deployment — list only what you want to change: .. code-block:: yaml display: cloudxr: NV_DEVICE_PROFILE: apple-vision-pro - NV_ENABLE_POSE_WAIT: null # remove a viewer default - -These settings override shell environment variables; an explicit ``--cloudxr-env-config`` -takes precedence over the generated config. Resolved settings appear in -``~/.cloudxr/logs/cxr_server.*.log``. See :doc:`/references/cloudxr` and -``.venv/bin/python camera_viz.py --help`` for more options. + NV_ENABLE_POSE_WAIT: null # null drops a viewer default + +The viewer writes these to a generated ``--cloudxr-env-config`` file, which outranks the process +environment — so a value left in your shell (for example after sourcing +``~/.cloudxr/run/cloudxr.env``, as ``--no-launch-cloudxr-runtime`` suggests) cannot silently +override the config, and neither can it override ``--cloudxr-device-profile``. Booleans are +written in the lowercase spelling the runtime's parser recognises; the launcher-computed keys +(``XR_RUNTIME_JSON``, ``NV_CXR_RUNTIME_DIR``, ``NV_CXR_OUTPUT_DIR``, ``XRT_NO_STDIN``) are +rejected by name. Passing ``--cloudxr-env-config`` yourself takes precedence over the generated +file. + +An unknown variable name warns with a suggestion before the runtime launches, and is passed +through regardless — the runtime has more settings than the viewer lists. To confirm what the +runtime resolved, read the settings dump at the top of the newest +``~/.cloudxr/logs/cxr_server.*.log``. .. _split-mode: Split mode — robot → workstation over RTP ----------------------------------------- -Use split mode when cameras are attached to a robot and the viewer runs on a workstation. -Prefer a direct connection when possible: RTP adds an encode/decode hop. Use a wired network; -there is no retransmission or FEC, so packet loss can corrupt video until the next IDR frame -(default interval: 5 seconds). +Split mode runs the capture side on the robot (``camera_streamer.py``) and ships RTP H.264 to +the viewer on the workstation (``source: rtp``). + +.. warning:: + + Split mode is **not recommended in most cases**. It exists for one situation: the cameras are + on the robot, but Isaac Teleop runs on a workstation, so the frames must be streamed to where + Isaac Teleop is running. That costs a full extra encode/decode hop — NVENC on the robot, UDP, + NVDEC on the workstation — so whenever a camera can attach directly to the machine running + Isaac Teleop, run direct mode instead. Wired networks only: there is no retransmit or FEC — one lost packet corrupts one + frame until the next IDR (default every 5 s). + +In split mode every camera entry must pin ``width``, ``height``, and ``fps`` in the YAML — the +receiver sizes its decoder from the config, not from the wire. + +The workstation needs the RTP dependencies, which ``setup`` does not install by default — re-run +it with ``--with-rtp`` if you provisioned for direct mode first. The robot side is handled by +``deploy``, which implies the flag. -In the YAML, set ``source: rtp`` and specify ``width``, ``height``, and ``fps`` for each camera. -Then, on the workstation: +Set ``source: rtp`` in the config, export the robot/streaming credentials once per shell, then +deploy the sender and run the viewer: .. code-block:: bash - ./camera_viz.sh setup --with-rtp export REMOTE_HOST=10.0.0.5 REMOTE_USER=nvidia export STREAMING_HOST=10.0.0.42 # workstation IP - ./camera_viz.sh deploy configs/v4l2.yaml - ./camera_viz.sh run configs/v4l2.yaml -``deploy`` installs the sender and starts a ``camera-streamer`` systemd user service on the -robot. Use ``./camera_viz.sh service-{status,logs,restart}`` to manage it. The workstation's -UDP receive ports (default 5000+) must be reachable from the robot. + ./camera_viz.sh deploy configs/v4l2.yaml # full deploy + systemd unit on the robot + ./camera_viz.sh run configs/v4l2.yaml # viewer on the workstation + +``deploy`` rsyncs the source to the robot, installs sender dependencies, renders a +``camera-streamer.service`` systemd user unit (injecting ``--host`` from ``$STREAMING_HOST`` +without editing the YAML on disk), and enables it. Operate the running unit with +``./camera_viz.sh service-{status,logs,restart}``. The sender retries forever across unplug, SDK +errors, and network blips. Loopback ^^^^^^^^ -Test RTP on one machine with ``./camera_viz.sh loopback configs/v4l2.yaml``. This also supports -mono video recordings with explicit ``width``, ``height``, and ``fps``; stereo video replay is -unsupported over RTP. +Loopback is a testing / debugging aid, not a deployment mode: ``./camera_viz.sh loopback +configs/v4l2.yaml`` runs the sender and viewer together on ``127.0.0.1`` — the quickest way to +smoke-test the RTP path on one machine. It also works camera-free with a mono ``type: video`` +entry (set ``width`` / ``height`` / ``fps``). Configuration ------------- -Start from a :code-dir:`source config ` and adjust the camera -settings for your device. Placement keys must match the camera's ``name``: +A single YAML drives both capture and visualization. Each entry in the ``cameras`` list becomes +its own plane (and, in split mode, its own RTP port). Abbreviated: .. code-block:: yaml + source: local | rtp + streaming: + host: 192.168.1.100 # workstation IP (overridden at deploy time) + encoder: auto | native | gstreamer + + cameras: + - name: cam + enabled: true + type: v4l2 # v4l2 | oakd | zed | video | synthetic + width: 2560 # video: optional — defaults to the file's size + height: 720 # (required when source: rtp) + fps: 30 + stereo: false # zed / video / synthetic — per-eye capture + SBS in XR + path: clip.mp4 # video only — file to replay, relative to this YAML + loop: true # video only — rewind at end of file + rtp: + port: 5000 # left eye when stereo + port_right: 5001 # required when stereo + source: rtp + bitrate_mbps: 15 + display: - mode: xr + mode: xr | window # default: xr + window: { width, height } + xr: + near_z: 0.05 # default + far_z: 100.0 # default + system_wait_seconds: -1 # default; wait forever (0 fails fast) + clear_color: [r, g, b, a] placements: cam: - lock_mode: lazy + lock_mode: lazy # world | head | lazy | gimbal distance: 1.5 - shape: quad - size: [1.0, 0.56] # meters + # size: [w_m, h_m] + # stereo_plane_distance_cm: 0 # gap between the eyes' planes + # shape: quad # quad | cylinder | equirect (cylinder/equirect are XR-only) + # equirect_yaw_deg: 0.0 # equirect: heading the middle of the feed points at + # compositor: openxr # openxr (default) | televiz — quads only + # cylinder_radius_m: 2.0 + # cylinder_angle_deg: 90 -Add entries to ``cameras`` and ``display.placements`` for multiple feeds. In split mode, assign -each stream its own RTP port; stereo sources also require ``rtp.port_right``. -The :code-file:`README ` documents the full schema. +See the :code-dir:`configs/ ` directory for a complete, commented +YAML per source kind. .. _camera-streaming-troubleshooting: Troubleshooting --------------- -- **XR session does not start:** connect the client first. ``XR_ERROR_FORM_FACTOR_UNAVAILABLE`` - (-35) means no headset is connected. The viewer waits indefinitely by default; set - ``display.xr.system_wait_seconds`` or ``--xr-wait`` to change the timeout. - Check ``~/.cloudxr/logs/cxr_server.*.log`` and ``~/.cloudxr/logs/runtime_worker_stderr.log`` - (``runtime_stderr.log`` with ``ISAACCAPTURE_LOGGING=off``). -- **No video on Orin:** set the client's **Video Codec** to **H.264**. -- **Black camera plane on Orin:** set ``compositor: televiz`` under the camera's placement - and restart. This workaround supports quads only; for the replay config: +- **The XR session fails to create** — check ``~/.cloudxr/logs/cxr_server.*.log`` and + ``~/.cloudxr/logs/runtime_worker_stderr.log`` (``runtime_stderr.log`` with + ``ISAACCAPTURE_LOGGING=off``) for the startup failure. + ``XR_ERROR_FORM_FACTOR_UNAVAILABLE`` + (-35) means the runtime is up but no headset has clicked CONNECT yet; XR mode waits + according to ``display.xr.system_wait_seconds`` (default ``-1`` to wait + indefinitely; ``0`` fails fast; positive values wait that many seconds; + override with ``--xr-wait``). Pass ``--mode window`` to + render to a desktop window instead (no runtime involved). +- **No window appears over SSH** — ``--mode window`` needs a local display; run on the machine + you're sitting at, or use a video-capable remote desktop. +- **"video source: no such file"** — relative ``path:`` values resolve against the YAML's + directory (``configs/``), not the directory you launched from. +- **A source fails asking for CuPy / CUDA** — check the matching CUDA headers, NVRTC, and + ``/usr/local/cuda`` target (:ref:`camera-streaming-jetson-cuda` on Jetson), then rerun setup. + ``nvidia-smi`` checks driver support, not installed headers. To rerun the kernel check + and print CUDA diagnostics on failure, run from ``examples/camera_viz``: + + .. code-block:: bash + + .venv/bin/python scripts/check_cuda.py + +- **Thor: ``CUDA_ERROR_NO_BINARY_FOR_GPU``** — an older NVRTC compiler can produce kernels + incompatible with Thor. Select your JetPack's CUDA 13.x toolkit and rerun setup. As a + temporary workaround for CuPy kernels, try: + + .. code-block:: bash + + CUPY_COMPILE_WITH_PTX=1 ./camera_viz.sh run configs/synthetic.yaml + + This lets the driver compile the kernel for the GPU (`CuPy PTX option + `__). + +- **Python cannot find dependencies** — use the sample's ``.venv/bin/python``, or select an + environment with :ref:`setup --venv `. +- **No video on Orin.** When the CloudXR runtime runs on Jetson Orin, set + **Video Codec** to **H.264** in the CloudXR web client. See + :ref:`connect-xr-headset`. +- **Black camera plane on Orin** — with the CloudXR experimental runtime on Orin, the default + OpenXR compositor shows the camera plane but leaves its contents black. Under the camera's + ``display.placements`` entry, set ``compositor: televiz`` (quads only). The same feed displays + correctly with this workaround: .. code-block:: yaml display: placements: - replay: + cam: compositor: televiz -- **Corrupted or frozen video:** stop the viewer and run - ``./camera_viz.sh run configs/synthetic.yaml`` to check the display path without a camera - or recording. Add ``--mode window`` to check local rendering separately from XR streaming. -- **No window over SSH:** window mode needs a desktop display or video-capable remote desktop. -- **Video file fails to open:** paths are relative to the YAML directory. For the bundled clip, - run the Git LFS command in the first-run instructions. -- **Python cannot find dependencies:** use ``.venv/bin/python`` or rerun ``setup --venv PATH``. -- **Split mode has no frames:** check ``./camera_viz.sh service-status``, ``STREAMING_HOST``, - and the workstation's UDP ports. Set ``verbose: true`` in YAML for per-source logs. +- **Split mode renders nothing** — check the sender is up (``./camera_viz.sh service-status``), + ``$STREAMING_HOST`` was the workstation's IP at deploy time, and UDP ports (default 5000+) + aren't firewalled. +- **Not sure which side is stuck?** — set ``verbose: true`` at the top of the YAML for periodic + per-source breadcrumbs on both ends. -CuPy / CUDA errors -^^^^^^^^^^^^^^^^^^ +How it works +------------ + +The sample is organized so that capture, transport, and visualization are cleanly separated, with +Televiz as the compositor at the end of the chain: + +.. code-block:: text + :class: code-100col + + camera_viz/ + ├── camera_viz.sh — CLI: setup / loopback / run / deploy / service-* + ├── camera_viz.py — receiver / viewer (drives a Televiz VizSession) + ├── camera_streamer.py — robot-side RTP sender (per-camera supervisor) + ├── pipeline/ — source ABC + threaded runner + ├── placements/ — XR lock-mode strategies (world / head / lazy / gimbal) + ├── sources/ — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264 + ├── transports/ — RTP sender + receiver (native + GStreamer) + ├── codec/ — native NVENC / NVDEC pybind module + ├── configs/ — one YAML per source kind + ├── test_data/ — sample replay clip (Git LFS) + └── scripts/ — installer + systemd unit template + +- **Sources** (:code-dir:`sources/ `) implement a common source ABC and + hand frames to a threaded runner in :code-dir:`pipeline/ `. Each + source produces GPU frames where possible — e.g. the ZED source uses ``retrieve_image(MEM.GPU)`` so + BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip. +- **The viewer** (:code-file:`camera_viz.py `) creates a + ``VizSession`` and adds one ``QuadLayer`` per enabled camera, then submits each frame to its layer + and calls ``render()`` once per frame. Stereo cameras submit both eyes. +- **Transport** (:code-dir:`transports/ `) carries the split mode: + an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec + module (:code-dir:`codec/ `) or a GStreamer fallback. +- **Placement** (:code-dir:`placements/ `) holds the XR lock-mode + strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app + sets each frame. + +Sharing the XR session with TeleopSession +----------------------------------------- -Setup checks imports, not GPU execution. Test a small kernel with the sample's interpreter: +Only one OpenXR session is allowed per process, so when this sample runs alongside teleoperation the +``VizSession`` owns the session and hands its handles to ``TeleopSession`` / ``DeviceIOSession``, +which then skip creating their own. The viewer builds the session with the trackers' required +extensions and forwards the handles through ``TeleopSessionConfig.oxr_handles``: -.. code-block:: bash +.. code-block:: python - .venv/bin/python - <<'PY' - import cupy as cp + cfg.required_extensions = DeviceIOSession.get_required_extensions(trackers) + viz_session = televiz.VizSession.create(cfg) - values = cp.arange(4, dtype=cp.float32) - assert cp.asnumpy(values + 1).tolist() == [1.0, 2.0, 3.0, 4.0] - print("CUDA kernel check passed") - PY + config = TeleopSessionConfig( + app_name="MyApp", + pipeline=pipeline, + oxr_handles=OpenXRSessionHandles(*viz_session.get_oxr_handles()), + ) -Missing ``libnvrtc.so``, ``cuda_runtime.h``, or ``cuda_fp16.h`` indicates missing or mismatched -CUDA development components. Check ``/usr/local/cuda`` and rerun setup (``--jetson`` on Jetson). -The CUDA version reported by ``nvidia-smi`` describes driver support, not installed headers. -See :ref:`camera-streaming-jetson-cuda`. +See the *Sharing the XR session* section of :doc:`/getting_started/televiz` for the full pattern +(imports, frame loop, and how the two sessions' lifecycles relate), and +:doc:`/getting_started/teleop_session` for the ``TeleopSession`` side. Next steps ---------- - :ref:`Connect a live camera ` or :ref:`replay your own recording `. +- :ref:`Adjust the view ` and save placement settings in YAML. - :ref:`Stream from a robot ` when capture and viewing run on separate machines. -- :ref:`Integrate with teleoperation ` by sharing Televiz's OpenXR - session with ``TeleopSession``. Only one OpenXR session is allowed per process. -- :doc:`Build a custom viewer ` using the Televiz API and - :code-file:`camera_viz.py ` as a reference. +- :ref:`Share the XR session with teleoperation ` and map inputs with + the :doc:`retargeting interface `. diff --git a/examples/camera_viz/README.md b/examples/camera_viz/README.md index f16f519512..649b99a398 100644 --- a/examples/camera_viz/README.md +++ b/examples/camera_viz/README.md @@ -371,3 +371,19 @@ with TeleopSession(config) as session: ``` `viz_session.get_oxr_handles()` returns `(instance, session, space, proc_addr)` as raw `uint64`s, or `None` outside `kXr`. + +## Validate a Jetson installation + +On each supported Thor and Orin JetPack image, run setup in a fresh environment. +Setup must pass its CuPy kernel check using the image's matching CUDA toolkit. +Then run from the repository root: + +```bash +uv pip install --python examples/camera_viz/.venv/bin/python pytest +examples/camera_viz/.venv/bin/python -m pytest --require-gpu -v \ + tests/python/examples/camera_viz/test_synthetic_offscreen.py +``` + +This requires mono and stereo sources to publish frames and render non-black output; +missing GPU support fails instead of skipping. Follow with replay and the +[first-session checklist](https://nvidia.github.io/IsaacCapture/main/references/camera_streaming.html#camera-streaming-first-session). diff --git a/examples/camera_viz/pipeline/interface.py b/examples/camera_viz/pipeline/interface.py index 1ab83659f2..27cc9ff079 100644 --- a/examples/camera_viz/pipeline/interface.py +++ b/examples/camera_viz/pipeline/interface.py @@ -61,7 +61,8 @@ def start(self) -> None: ... def latest(self) -> Optional[Frame]: """Return the freshest frame, or None if no new frame since the last call. Must be non-blocking — the render loop polls this - every frame and skips submission on None.""" + every frame and skips submission on None. Raise on a fatal + producer failure so the runner can stop and report it.""" @abstractmethod def stop(self) -> None: ... diff --git a/examples/camera_viz/scripts/_install_deps.sh b/examples/camera_viz/scripts/_install_deps.sh index a1adfb3caf..dd46149e40 100755 --- a/examples/camera_viz/scripts/_install_deps.sh +++ b/examples/camera_viz/scripts/_install_deps.sh @@ -88,19 +88,10 @@ if [[ "$MODE" == sender ]]; then WITH_RTP=true fi -# major picks the cupy wheel (cupy-cuda12x / cupy-cuda13x). -# major.minor picks the apt nvrtc package (cuda-nvrtc-12-6 on Orin/JP6, -# cuda-nvrtc-13-0 on Thor/JP7); JetPack only publishes the exact-minor. -cuda_major=12 -cuda_minor=0 -if [[ -e /usr/local/cuda ]]; then - cuda_resolved=$(readlink -f /usr/local/cuda 2>/dev/null) - full=$(echo "$cuda_resolved" | grep -oE 'cuda-[0-9]+\.[0-9]+' | head -1 | sed 's/cuda-//') - if [[ -n "$full" ]]; then - cuda_major=$(echo "$full" | cut -d. -f1) - cuda_minor=$(echo "$full" | cut -d. -f2) - fi -fi +# Select the toolkit CuPy will discover, without guessing from the driver. +cuda_info=$(python3 "$HERE/check_cuda.py" --detect) || die "CUDA toolkit detection failed" +IFS=$'\t' read -r cuda_root cuda_major cuda_minor <<< "$cuda_info" +note "CUDA $cuda_major.$cuda_minor at $cuda_root" # System-dep check. apt-installable bits are NOT auto-installed — we # only probe what's present and, if anything is missing, print the @@ -219,10 +210,10 @@ check_cuda_symlinks() { if ! $JETSON; then return 0 fi - if [[ ! -d /usr/local/cuda/lib64 ]]; then + if [[ ! -d "$cuda_root/lib64" ]]; then return 0 fi - local lib64=/usr/local/cuda/lib64 + local lib64="$cuda_root/lib64" local cmds=() for stem in libnvrtc.so libnvrtc-builtins.so libcudart.so; do if [[ ! -e "$lib64/$stem" ]]; then @@ -639,6 +630,10 @@ for m, why in fail: sys.exit(0 if not fail else 1) PY +step "verifying CUDA kernel execution" +"$PY" "$HERE/check_cuda.py" --cuda-major "$cuda_major" \ + || die "CUDA kernel execution failed — see the diagnostics above" + # Close on what the user can do next, not on the fact that the script ended. step "setup complete" if [[ "$MODE" == full ]]; then diff --git a/examples/camera_viz/scripts/check_cuda.py b/examples/camera_viz/scripts/check_cuda.py new file mode 100644 index 0000000000..d536469501 --- /dev/null +++ b/examples/camera_viz/scripts/check_cuda.py @@ -0,0 +1,121 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Select a CUDA toolkit and exercise CuPy before declaring setup complete.""" + +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +import re +import shutil +import sys +import tempfile + + +def detect_toolkit() -> tuple[Path, tuple[int, int]]: + # Select a system toolkit; check_kernel verifies what CuPy actually loads. + explicit = os.environ.get("CUDA_PATH") + nvcc = shutil.which("nvcc") + root = ( + Path(explicit) + if explicit + else (Path(nvcc).resolve().parent.parent if nvcc else Path("/usr/local/cuda")) + ) + root = root.resolve() + version = "" + if (root / "version.json").is_file(): + version = json.loads((root / "version.json").read_text())["cuda"]["version"] + elif (root / "version.txt").is_file(): + version = (root / "version.txt").read_text() + elif root.is_dir(): + version = root.name if root.name.startswith("cuda-") else "" + match = re.search(r"(\d+)\.(\d+)", version) + if match is None: + raise RuntimeError( + f"Cannot determine the CUDA toolkit version at {root}. " + "Install the toolkit matching your JetPack/driver and set CUDA_PATH " + "to its directory, then rerun setup." + ) + major, minor = map(int, match.groups()) + if major not in (12, 13): + raise RuntimeError( + f"Unsupported CUDA toolkit {major}.{minor}; use CUDA 12 or 13." + ) + return root, (major, minor) + + +def check_kernel(expected_major: int | None = None) -> str: + import cupy as cp + + device = cp.cuda.Device() + props = cp.cuda.runtime.getDeviceProperties(device.id) + name = props["name"] + if isinstance(name, bytes): + name = name.decode() + nvrtc = cp.cuda.nvrtc.getVersion() + if expected_major is not None and nvrtc[0] != expected_major: + raise RuntimeError( + f"Selected CUDA {expected_major}, but CuPy loaded NVRTC {nvrtc[0]}.{nvrtc[1]}. " + "Check CUDA_PATH, LD_LIBRARY_PATH, and CUDA packages in the virtual environment." + ) + # Allocation/import alone does not compile or execute a kernel. + values = cp.arange(4, dtype=cp.float32) * 2 + 1 + cp.cuda.Stream.null.synchronize() + if values.get().tolist() != [1.0, 3.0, 5.0, 7.0]: + raise RuntimeError("CuPy kernel returned incorrect results") + return f"CuPy kernel check passed: {name}, CuPy {cp.__version__}, NVRTC {nvrtc[0]}.{nvrtc[1]}" + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--detect", action="store_true", help="print toolkit path, major, minor" + ) + parser.add_argument("--cuda-major", type=int, help="expected NVRTC major version") + args = parser.parse_args(argv) + try: + if args.detect: + root, (major, minor) = detect_toolkit() + print(f"{root}\t{major}\t{minor}") + else: + # A warm cache must not hide missing headers or a broken compiler. + previous_cache = os.environ.get("CUPY_CACHE_DIR") + try: + with tempfile.TemporaryDirectory(prefix="camera-viz-cupy-") as cache: + os.environ["CUPY_CACHE_DIR"] = cache + print(check_kernel(args.cuda_major)) + finally: + if previous_cache is None: + os.environ.pop("CUPY_CACHE_DIR", None) + else: + os.environ["CUPY_CACHE_DIR"] = previous_cache + except Exception as exc: + print(f"CUDA check failed: {exc}", file=sys.stderr) + if not args.detect: + print( + "Install matching CUDA runtime, NVRTC, headers, and CuPy. On Jetson, use " + "your JetPack's CUDA packages and rerun camera_viz.sh setup --jetson. " + "nvidia-smi reports driver support, not the installed toolkit.", + file=sys.stderr, + ) + if "CUDA_ERROR_NO_BINARY_FOR_GPU" in str(exc): + print( + "For this kernel-binary mismatch, CUPY_COMPILE_WITH_PTX=1 may " + "work around an older NVRTC compiler; prefer a toolkit/CuPy " + "combination that supports your GPU.", + file=sys.stderr, + ) + try: + import cupy as cp + + cp.show_config() + except Exception: + pass # Preserve the original failure even if diagnostics fail. + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/camera_viz/sources/_helpers.py b/examples/camera_viz/sources/_helpers.py index 93cf6efba9..2ba8fd94ff 100644 --- a/examples/camera_viz/sources/_helpers.py +++ b/examples/camera_viz/sources/_helpers.py @@ -164,8 +164,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - f"{self._kind} source requires CuPy (cupy-cuda12x). " - "Install via `uv pip install cupy-cuda12x`." + f"{self._kind} source requires CuPy. Run `camera_viz.sh setup` " + "to install the package matching your CUDA toolkit." ) from e self._cp = cp diff --git a/examples/camera_viz/sources/oakd.py b/examples/camera_viz/sources/oakd.py index 405ea1488b..11b1c78865 100644 --- a/examples/camera_viz/sources/oakd.py +++ b/examples/camera_viz/sources/oakd.py @@ -136,7 +136,8 @@ def __init__(self, device_id: str, streams: List[_StreamSpec]) -> None: except ImportError as e: raise RuntimeError( "OakdSource requires CuPy + depthai. " - "Install via `uv pip install cupy-cuda12x depthai`." + "Run `camera_viz.sh setup` to install the camera dependencies " + "and CuPy matching your CUDA toolkit." ) from e self._device_id = device_id diff --git a/examples/camera_viz/sources/rtp_h264.py b/examples/camera_viz/sources/rtp_h264.py index 4b957f9db1..6ec5fa67a8 100644 --- a/examples/camera_viz/sources/rtp_h264.py +++ b/examples/camera_viz/sources/rtp_h264.py @@ -52,8 +52,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "RtpH264Source requires CuPy. Install via " - "`uv pip install cupy-cuda12x`." + "RtpH264Source requires CuPy. Run `camera_viz.sh setup --with-rtp` " + "to install the package matching your CUDA toolkit." ) from e self._spec = SourceSpec( diff --git a/examples/camera_viz/sources/synthetic.py b/examples/camera_viz/sources/synthetic.py index e0c30fd03f..1d8125c2d4 100644 --- a/examples/camera_viz/sources/synthetic.py +++ b/examples/camera_viz/sources/synthetic.py @@ -38,8 +38,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "SyntheticSource requires CuPy (cupy-cuda12x). Install via " - "`uv pip install cupy-cuda12x` or skip this source." + "SyntheticSource requires CuPy. Run `camera_viz.sh setup` " + "to install the package matching your CUDA toolkit." ) from e self._cp = cp @@ -57,6 +57,7 @@ def __init__( self._publish_idx: int = -1 # -1 = nothing published yet self._consumed_idx: int = -2 # track what `latest()` last returned self._lock = threading.Lock() + self._error: Optional[BaseException] = None self._stop = threading.Event() self._thread: Optional[threading.Thread] = None @@ -70,6 +71,10 @@ def start(self) -> None: if self._thread is not None: return self._stop.clear() + with self._lock: + self._error = None + self._publish_idx = -1 + self._consumed_idx = -2 self._t0_ns = time.monotonic_ns() self._thread = threading.Thread( target=self._produce_loop, name=f"synth_{self._spec.name}", daemon=False @@ -84,6 +89,11 @@ def stop(self) -> None: def latest(self) -> Optional[Frame]: with self._lock: + if self._error is not None: + raise RuntimeError( + f"Source {self._spec.name!r} failed: {self._error}. " + "Run camera_viz.sh setup to check the CUDA environment." + ) from self._error if self._publish_idx < 0 or self._publish_idx == self._consumed_idx: return None idx = self._publish_idx @@ -96,6 +106,14 @@ def latest(self) -> Optional[Frame]: ) def _produce_loop(self) -> None: + try: + self._produce_frames() + except BaseException as exc: + # latest() forwards the producer failure to VizRunner.wait(). + with self._lock: + self._error = exc + + def _produce_frames(self) -> None: cp = self._cp h, w = self._spec.height, self._spec.width # Pin this producer thread to the GPU the pre-allocated buffers @@ -161,8 +179,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "SyntheticStereoSource requires CuPy (cupy-cuda12x). Install via " - "`uv pip install cupy-cuda12x` or skip this source." + "SyntheticStereoSource requires CuPy. Run `camera_viz.sh setup` " + "to install the package matching your CUDA toolkit." ) from e self._cp = cp @@ -180,6 +198,7 @@ def __init__( self._publish_idx: int = -1 self._consumed_idx: int = -2 self._lock = threading.Lock() + self._error: Optional[BaseException] = None self._stop = threading.Event() self._thread: Optional[threading.Thread] = None @@ -193,6 +212,10 @@ def start(self) -> None: if self._thread is not None: return self._stop.clear() + with self._lock: + self._error = None + self._publish_idx = -1 + self._consumed_idx = -2 self._t0_ns = time.monotonic_ns() self._thread = threading.Thread( target=self._produce_loop, @@ -209,6 +232,11 @@ def stop(self) -> None: def latest(self) -> Optional[Frame]: with self._lock: + if self._error is not None: + raise RuntimeError( + f"Source {self._spec.name!r} failed: {self._error}. " + "Run camera_viz.sh setup to check the CUDA environment." + ) from self._error if self._publish_idx < 0 or self._publish_idx == self._consumed_idx: return None idx = self._publish_idx @@ -222,6 +250,14 @@ def latest(self) -> Optional[Frame]: ) def _produce_loop(self) -> None: + try: + self._produce_frames() + except BaseException as exc: + # latest() forwards the producer failure to VizRunner.wait(). + with self._lock: + self._error = exc + + def _produce_frames(self) -> None: cp = self._cp h, w = self._spec.height, self._spec.width with cp.cuda.Device(int(self._left[0].device.id)): diff --git a/examples/camera_viz/sources/zed.py b/examples/camera_viz/sources/zed.py index e649d569a3..f3fc67506f 100644 --- a/examples/camera_viz/sources/zed.py +++ b/examples/camera_viz/sources/zed.py @@ -142,8 +142,8 @@ def __init__( except ImportError as e: raise RuntimeError( "ZedSource requires CuPy + pyzed. " - "Install via `uv pip install cupy-cuda12x` and follow the ZED " - "SDK's Python install instructions for pyzed." + "Run `camera_viz.sh setup --with-zed` with the ZED SDK installed " + "to install pyzed and CuPy matching your CUDA toolkit." ) from e self._resolution_name = _resolution_for_dims(width, height) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index a3ad52b9b9..8acee24b2a 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -47,6 +47,8 @@ Shared C++ fixtures (not executables) live under `tests/cpp/viz/support/` multiline layout. - Format changed Python tests before staging; if pre-commit rewrites a file, restage the result and rerun the full hook set. +- Use raw strings for intentional regexes in `pytest.raises(match=...)`; escape + literal punctuation such as version-number dots (Ruff RUF043). - CTest runs one `pytest` invocation per `test_*.py` with `WORKING_DIRECTORY` set to that leaf. - Use [`repo_paths.py`](python/repo_paths.py) for paths into `src/python/` or diff --git a/tests/python/examples/camera_viz/conftest.py b/tests/python/examples/camera_viz/conftest.py index 2c1ec96f82..4cec3ca556 100644 --- a/tests/python/examples/camera_viz/conftest.py +++ b/tests/python/examples/camera_viz/conftest.py @@ -16,3 +16,11 @@ from repo_paths import repo_root # noqa: E402 sys.path.insert(0, str(repo_root() / "examples" / "camera_viz")) + + +def pytest_addoption(parser): + parser.addoption( + "--require-gpu", + action="store_true", + help="Fail camera rendering tests if CUDA/CuPy/Vulkan is unavailable", + ) diff --git a/tests/python/examples/camera_viz/test_cuda_setup.py b/tests/python/examples/camera_viz/test_cuda_setup.py new file mode 100644 index 0000000000..26ceaf8840 --- /dev/null +++ b/tests/python/examples/camera_viz/test_cuda_setup.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""CUDA discovery and failure diagnostics, without requiring a GPU.""" + +import json +import os +from pathlib import Path +import sys +from types import SimpleNamespace + +import numpy as np +import pytest + +from scripts import check_cuda + + +@pytest.mark.parametrize("version", ["12.6.10", "13.0.0", "13.2.0"]) +def test_explicit_toolkit_takes_precedence(tmp_path, monkeypatch, version): + toolkit = tmp_path / "selected" + toolkit.mkdir() + (toolkit / "version.json").write_text(json.dumps({"cuda": {"version": version}})) + monkeypatch.setenv("CUDA_PATH", str(toolkit)) + monkeypatch.setattr( + check_cuda.shutil, "which", lambda _: "/wrong/cuda-12.0/bin/nvcc" + ) + assert check_cuda.detect_toolkit() == ( + toolkit, + tuple(map(int, version.split(".")[:2])), + ) + + +def test_toolkit_on_path(tmp_path, monkeypatch): + toolkit = tmp_path / "cuda-13.0" + toolkit.mkdir() + monkeypatch.delenv("CUDA_PATH", raising=False) + monkeypatch.setattr(check_cuda.shutil, "which", lambda _: str(toolkit / "bin/nvcc")) + assert check_cuda.detect_toolkit() == (toolkit, (13, 0)) + + +def test_toolkit_symlink(tmp_path, monkeypatch): + toolkit = tmp_path / "cuda-12.6" + toolkit.mkdir() + alias = tmp_path / "cuda" + alias.symlink_to(toolkit) + monkeypatch.setenv("CUDA_PATH", str(alias)) + assert check_cuda.detect_toolkit() == (toolkit, (12, 6)) + + +@pytest.mark.parametrize("exists", [False, True]) +def test_unknown_toolkit_does_not_guess_cuda12(tmp_path, monkeypatch, exists): + toolkit = tmp_path / "unknown" + if exists: + toolkit.mkdir() + monkeypatch.setenv("CUDA_PATH", str(toolkit)) + with pytest.raises(RuntimeError, match=r"Cannot determine.*CUDA toolkit"): + check_cuda.detect_toolkit() + + +class _Array(np.ndarray): + def get(self): + return np.asarray(self) + + +@pytest.fixture +def cupy(monkeypatch): + cp = SimpleNamespace( + __version__="test-version", + float32=np.float32, + arange=lambda *a, **kw: np.arange(*a, **kw).view(_Array), + cuda=SimpleNamespace( + Device=lambda: SimpleNamespace(id=0), + runtime=SimpleNamespace( + getDeviceProperties=lambda _: {"name": b"NVIDIA Thor"} + ), + nvrtc=SimpleNamespace(getVersion=lambda: (13, 0)), + Stream=SimpleNamespace(null=SimpleNamespace(synchronize=lambda: None)), + ), + show_config=lambda: print("CUDA diagnostic configuration"), + ) + monkeypatch.setitem(sys.modules, "cupy", cp) + return cp + + +@pytest.mark.parametrize( + "name, version", [("Orin", (12, 6)), ("Thor", (13, 0)), ("Thor", (13, 2))] +) +def test_matching_runtime_executes_kernel(cupy, name, version): + cupy.cuda.nvrtc.getVersion = lambda: version + cupy.cuda.runtime.getDeviceProperties = lambda _: {"name": name.encode()} + assert name in check_cuda.check_kernel(version[0]) + + +def test_mismatched_nvrtc_is_rejected(cupy): + cupy.cuda.nvrtc.getVersion = lambda: (12, 6) + with pytest.raises( + RuntimeError, match=r"Selected CUDA 13, but CuPy loaded NVRTC 12\.6" + ): + check_cuda.check_kernel(13) + + +@pytest.mark.parametrize("at_sync", [False, True]) +def test_kernel_failure_returns_nonzero_and_diagnostics(cupy, capsys, at_sync): + def fail(*a, **kw): + raise RuntimeError("CUDA_ERROR_NO_BINARY_FOR_GPU") + + if at_sync: + cupy.cuda.Stream.null.synchronize = fail + else: + cupy.arange = fail + assert check_cuda.main(["--cuda-major", "13"]) == 1 + output = capsys.readouterr() + assert "CUDA_ERROR_NO_BINARY_FOR_GPU" in output.err + assert "CUPY_COMPILE_WITH_PTX=1" in output.err + assert "CUDA diagnostic configuration" in output.out + assert "passed" not in output.out + + +def test_incorrect_kernel_result_is_rejected(cupy): + cupy.arange = lambda *a, **kw: np.zeros(4).view(_Array) + with pytest.raises(RuntimeError, match="incorrect results"): + check_cuda.check_kernel(13) + + +def test_probe_uses_cold_cache_and_restores_environment(cupy, monkeypatch, tmp_path): + original = str(tmp_path / "existing-cache") + monkeypatch.setenv("CUPY_CACHE_DIR", original) + paths = [] + arange = cupy.arange + + def checked_arange(*a, **kw): + path = Path(os.environ["CUPY_CACHE_DIR"]) + assert str(path) != original + assert path.is_dir() + paths.append(path) + return arange(*a, **kw) + + cupy.arange = checked_arange + assert check_cuda.main([]) == 0 + assert os.environ["CUPY_CACHE_DIR"] == original + assert len(paths) == 1 + assert not paths[0].exists() diff --git a/tests/python/examples/camera_viz/test_runner_health.py b/tests/python/examples/camera_viz/test_runner_health.py index 903e1ad781..48ad7783e6 100644 --- a/tests/python/examples/camera_viz/test_runner_health.py +++ b/tests/python/examples/camera_viz/test_runner_health.py @@ -31,3 +31,18 @@ def health_check() -> None: runner.wait(health_check=health_check) assert checks == 1 + + +def test_source_failure_stops_runner_and_reaches_wait() -> None: + failure = RuntimeError("camera producer: CUDA_ERROR_NO_BINARY_FOR_GPU") + + class FailedSource: + def latest(self): + raise failure + + runner = VizRunner(object(), [FailedSource()], [object()]) + runner._submit_loop() + assert runner._stop.is_set() + with pytest.raises(RuntimeError, match="CUDA_ERROR_NO_BINARY_FOR_GPU") as error: + runner.wait() + assert error.value is failure diff --git a/tests/python/examples/camera_viz/test_synthetic_failures.py b/tests/python/examples/camera_viz/test_synthetic_failures.py new file mode 100644 index 0000000000..744c65d4a9 --- /dev/null +++ b/tests/python/examples/camera_viz/test_synthetic_failures.py @@ -0,0 +1,80 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +"""Producer failures reach the consumer instead of leaving an empty mailbox.""" + +from contextlib import nullcontext +import sys +from types import SimpleNamespace + +import numpy as np +import pytest + +from sources.synthetic import SyntheticSource, SyntheticStereoSource + + +class _Buffer(np.ndarray): + @property + def device(self): + return SimpleNamespace(id=0) + + +@pytest.fixture(params=[SyntheticSource, SyntheticStereoSource]) +def source(request, monkeypatch): + cp = SimpleNamespace( + zeros=lambda shape, dtype: np.zeros(shape, dtype=dtype).view(_Buffer), + arange=np.arange, + sin=np.sin, + uint8=np.uint8, + float32=np.float32, + cuda=SimpleNamespace( + Device=lambda _: nullcontext(), + Stream=SimpleNamespace(null=SimpleNamespace(synchronize=lambda: None)), + ), + ) + monkeypatch.setitem(sys.modules, "cupy", cp) + instance = request.param("test-camera", 8, 4) + yield instance + instance.stop() + + +@pytest.mark.parametrize("at_sync", [False, True]) +def test_cuda_failure_reaches_latest_and_survives_stop(source, at_sync): + failure = RuntimeError("CUDA_ERROR_NO_BINARY_FOR_GPU") + + def fail(*args, **kwargs): + raise failure + + if at_sync: + source._cp.cuda.Stream.null.synchronize = fail + else: + source._cp.arange = fail + source.start() + source._thread.join(timeout=2) + assert not source._thread.is_alive(), "producer did not terminate" + for _ in range(2): + with pytest.raises( + RuntimeError, match=r"test-camera.*CUDA_ERROR_NO_BINARY_FOR_GPU" + ) as error: + source.latest() + assert error.value.__cause__ is failure + source.stop() + + +def test_restart_clears_failure_and_stale_frame(source): + failure = RuntimeError("producer failed after publishing") + + def fail(): + with source._lock: + source._publish_idx = 0 + raise failure + + source._produce_frames = fail + source.start() + source._thread.join(timeout=2) + assert not source._thread.is_alive() + with pytest.raises(RuntimeError, match="producer failed after publishing"): + source.latest() + source.stop() + source._produce_frames = source._stop.wait + source.start() + assert source.latest() is None diff --git a/tests/python/examples/camera_viz/test_synthetic_offscreen.py b/tests/python/examples/camera_viz/test_synthetic_offscreen.py index 1eea586a6a..3a951b4f4c 100644 --- a/tests/python/examples/camera_viz/test_synthetic_offscreen.py +++ b/tests/python/examples/camera_viz/test_synthetic_offscreen.py @@ -47,9 +47,13 @@ def _gpu_and_cupy_available() -> bool: return True -pytestmark = pytest.mark.skipif( - not _gpu_and_cupy_available(), reason="no Vulkan/CUDA GPU or cupy missing" -) +@pytest.fixture(scope="module", autouse=True) +def require_rendering_environment(request): + if not _gpu_and_cupy_available(): + reason = "no Vulkan/CUDA GPU or cupy missing" + if request.config.getoption("--require-gpu"): + pytest.fail(reason) + pytest.skip(reason) # Bounded wait helper so a stuck producer fails fast instead of hanging CI. From c8e319bfcb1c63bc40a4b5fbee02d3bfa05a2c3d Mon Sep 17 00:00:00 2001 From: Farbod Motlagh Date: Tue, 6 Oct 2026 16:07:02 -0700 Subject: [PATCH 4/5] docs: keep camera onboarding separate from CUDA fix Signed-off-by: Farbod Motlagh --- docs/source/references/camera_streaming.rst | 21 +-- examples/camera_viz/README.md | 16 -- examples/camera_viz/pipeline/interface.py | 3 +- examples/camera_viz/scripts/_install_deps.sh | 25 ++-- examples/camera_viz/scripts/check_cuda.py | 121 --------------- examples/camera_viz/sources/_helpers.py | 4 +- examples/camera_viz/sources/oakd.py | 3 +- examples/camera_viz/sources/rtp_h264.py | 4 +- examples/camera_viz/sources/synthetic.py | 44 +----- examples/camera_viz/sources/zed.py | 4 +- tests/AGENTS.md | 2 - tests/python/examples/camera_viz/conftest.py | 8 - .../examples/camera_viz/test_cuda_setup.py | 141 ------------------ .../examples/camera_viz/test_runner_health.py | 15 -- .../camera_viz/test_synthetic_failures.py | 80 ---------- .../camera_viz/test_synthetic_offscreen.py | 10 +- 16 files changed, 34 insertions(+), 467 deletions(-) delete mode 100644 examples/camera_viz/scripts/check_cuda.py delete mode 100644 tests/python/examples/camera_viz/test_cuda_setup.py delete mode 100644 tests/python/examples/camera_viz/test_synthetic_failures.py diff --git a/docs/source/references/camera_streaming.rst b/docs/source/references/camera_streaming.rst index be030ddd62..c236054c63 100644 --- a/docs/source/references/camera_streaming.rst +++ b/docs/source/references/camera_streaming.rst @@ -143,9 +143,7 @@ provides them. With the matching NVIDIA repositories configured: sudo apt update sudo apt install nvidia-cuda-dev -Setup selects CuPy from ``CUDA_PATH``, the toolkit containing ``nvcc`` on ``PATH``, or -``/usr/local/cuda``, in that order, and verifies GPU kernel execution. ``--jetson`` configures -library discovery. A separate ``nvcc`` compiler is needed for CUDA source builds, such as the native +Use the CUDA toolkit matching your JetPack. ``--jetson`` configures library discovery. A separate ``nvcc`` compiler is needed for CUDA source builds, such as the native RTP codec, rather than ordinary CuPy kernel execution. .. _recorded-camera-streaming: @@ -578,23 +576,12 @@ Troubleshooting directory (``configs/``), not the directory you launched from. - **A source fails asking for CuPy / CUDA** — check the matching CUDA headers, NVRTC, and ``/usr/local/cuda`` target (:ref:`camera-streaming-jetson-cuda` on Jetson), then rerun setup. - ``nvidia-smi`` checks driver support, not installed headers. To rerun the kernel check - and print CUDA diagnostics on failure, run from ``examples/camera_viz``: + ``nvidia-smi`` checks driver support, not installed headers. To test GPU execution, + run from ``examples/camera_viz`` and expect ``[1 2 3 4]``: .. code-block:: bash - .venv/bin/python scripts/check_cuda.py - -- **Thor: ``CUDA_ERROR_NO_BINARY_FOR_GPU``** — an older NVRTC compiler can produce kernels - incompatible with Thor. Select your JetPack's CUDA 13.x toolkit and rerun setup. As a - temporary workaround for CuPy kernels, try: - - .. code-block:: bash - - CUPY_COMPILE_WITH_PTX=1 ./camera_viz.sh run configs/synthetic.yaml - - This lets the driver compile the kernel for the GPU (`CuPy PTX option - `__). + .venv/bin/python -c 'import cupy as cp; print((cp.arange(4) + 1).get())' - **Python cannot find dependencies** — use the sample's ``.venv/bin/python``, or select an environment with :ref:`setup --venv `. diff --git a/examples/camera_viz/README.md b/examples/camera_viz/README.md index 649b99a398..f16f519512 100644 --- a/examples/camera_viz/README.md +++ b/examples/camera_viz/README.md @@ -371,19 +371,3 @@ with TeleopSession(config) as session: ``` `viz_session.get_oxr_handles()` returns `(instance, session, space, proc_addr)` as raw `uint64`s, or `None` outside `kXr`. - -## Validate a Jetson installation - -On each supported Thor and Orin JetPack image, run setup in a fresh environment. -Setup must pass its CuPy kernel check using the image's matching CUDA toolkit. -Then run from the repository root: - -```bash -uv pip install --python examples/camera_viz/.venv/bin/python pytest -examples/camera_viz/.venv/bin/python -m pytest --require-gpu -v \ - tests/python/examples/camera_viz/test_synthetic_offscreen.py -``` - -This requires mono and stereo sources to publish frames and render non-black output; -missing GPU support fails instead of skipping. Follow with replay and the -[first-session checklist](https://nvidia.github.io/IsaacCapture/main/references/camera_streaming.html#camera-streaming-first-session). diff --git a/examples/camera_viz/pipeline/interface.py b/examples/camera_viz/pipeline/interface.py index 27cc9ff079..1ab83659f2 100644 --- a/examples/camera_viz/pipeline/interface.py +++ b/examples/camera_viz/pipeline/interface.py @@ -61,8 +61,7 @@ def start(self) -> None: ... def latest(self) -> Optional[Frame]: """Return the freshest frame, or None if no new frame since the last call. Must be non-blocking — the render loop polls this - every frame and skips submission on None. Raise on a fatal - producer failure so the runner can stop and report it.""" + every frame and skips submission on None.""" @abstractmethod def stop(self) -> None: ... diff --git a/examples/camera_viz/scripts/_install_deps.sh b/examples/camera_viz/scripts/_install_deps.sh index dd46149e40..a1adfb3caf 100755 --- a/examples/camera_viz/scripts/_install_deps.sh +++ b/examples/camera_viz/scripts/_install_deps.sh @@ -88,10 +88,19 @@ if [[ "$MODE" == sender ]]; then WITH_RTP=true fi -# Select the toolkit CuPy will discover, without guessing from the driver. -cuda_info=$(python3 "$HERE/check_cuda.py" --detect) || die "CUDA toolkit detection failed" -IFS=$'\t' read -r cuda_root cuda_major cuda_minor <<< "$cuda_info" -note "CUDA $cuda_major.$cuda_minor at $cuda_root" +# major picks the cupy wheel (cupy-cuda12x / cupy-cuda13x). +# major.minor picks the apt nvrtc package (cuda-nvrtc-12-6 on Orin/JP6, +# cuda-nvrtc-13-0 on Thor/JP7); JetPack only publishes the exact-minor. +cuda_major=12 +cuda_minor=0 +if [[ -e /usr/local/cuda ]]; then + cuda_resolved=$(readlink -f /usr/local/cuda 2>/dev/null) + full=$(echo "$cuda_resolved" | grep -oE 'cuda-[0-9]+\.[0-9]+' | head -1 | sed 's/cuda-//') + if [[ -n "$full" ]]; then + cuda_major=$(echo "$full" | cut -d. -f1) + cuda_minor=$(echo "$full" | cut -d. -f2) + fi +fi # System-dep check. apt-installable bits are NOT auto-installed — we # only probe what's present and, if anything is missing, print the @@ -210,10 +219,10 @@ check_cuda_symlinks() { if ! $JETSON; then return 0 fi - if [[ ! -d "$cuda_root/lib64" ]]; then + if [[ ! -d /usr/local/cuda/lib64 ]]; then return 0 fi - local lib64="$cuda_root/lib64" + local lib64=/usr/local/cuda/lib64 local cmds=() for stem in libnvrtc.so libnvrtc-builtins.so libcudart.so; do if [[ ! -e "$lib64/$stem" ]]; then @@ -630,10 +639,6 @@ for m, why in fail: sys.exit(0 if not fail else 1) PY -step "verifying CUDA kernel execution" -"$PY" "$HERE/check_cuda.py" --cuda-major "$cuda_major" \ - || die "CUDA kernel execution failed — see the diagnostics above" - # Close on what the user can do next, not on the fact that the script ended. step "setup complete" if [[ "$MODE" == full ]]; then diff --git a/examples/camera_viz/scripts/check_cuda.py b/examples/camera_viz/scripts/check_cuda.py deleted file mode 100644 index d536469501..0000000000 --- a/examples/camera_viz/scripts/check_cuda.py +++ /dev/null @@ -1,121 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -"""Select a CUDA toolkit and exercise CuPy before declaring setup complete.""" - -from __future__ import annotations - -import argparse -import json -import os -from pathlib import Path -import re -import shutil -import sys -import tempfile - - -def detect_toolkit() -> tuple[Path, tuple[int, int]]: - # Select a system toolkit; check_kernel verifies what CuPy actually loads. - explicit = os.environ.get("CUDA_PATH") - nvcc = shutil.which("nvcc") - root = ( - Path(explicit) - if explicit - else (Path(nvcc).resolve().parent.parent if nvcc else Path("/usr/local/cuda")) - ) - root = root.resolve() - version = "" - if (root / "version.json").is_file(): - version = json.loads((root / "version.json").read_text())["cuda"]["version"] - elif (root / "version.txt").is_file(): - version = (root / "version.txt").read_text() - elif root.is_dir(): - version = root.name if root.name.startswith("cuda-") else "" - match = re.search(r"(\d+)\.(\d+)", version) - if match is None: - raise RuntimeError( - f"Cannot determine the CUDA toolkit version at {root}. " - "Install the toolkit matching your JetPack/driver and set CUDA_PATH " - "to its directory, then rerun setup." - ) - major, minor = map(int, match.groups()) - if major not in (12, 13): - raise RuntimeError( - f"Unsupported CUDA toolkit {major}.{minor}; use CUDA 12 or 13." - ) - return root, (major, minor) - - -def check_kernel(expected_major: int | None = None) -> str: - import cupy as cp - - device = cp.cuda.Device() - props = cp.cuda.runtime.getDeviceProperties(device.id) - name = props["name"] - if isinstance(name, bytes): - name = name.decode() - nvrtc = cp.cuda.nvrtc.getVersion() - if expected_major is not None and nvrtc[0] != expected_major: - raise RuntimeError( - f"Selected CUDA {expected_major}, but CuPy loaded NVRTC {nvrtc[0]}.{nvrtc[1]}. " - "Check CUDA_PATH, LD_LIBRARY_PATH, and CUDA packages in the virtual environment." - ) - # Allocation/import alone does not compile or execute a kernel. - values = cp.arange(4, dtype=cp.float32) * 2 + 1 - cp.cuda.Stream.null.synchronize() - if values.get().tolist() != [1.0, 3.0, 5.0, 7.0]: - raise RuntimeError("CuPy kernel returned incorrect results") - return f"CuPy kernel check passed: {name}, CuPy {cp.__version__}, NVRTC {nvrtc[0]}.{nvrtc[1]}" - - -def main(argv=None) -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--detect", action="store_true", help="print toolkit path, major, minor" - ) - parser.add_argument("--cuda-major", type=int, help="expected NVRTC major version") - args = parser.parse_args(argv) - try: - if args.detect: - root, (major, minor) = detect_toolkit() - print(f"{root}\t{major}\t{minor}") - else: - # A warm cache must not hide missing headers or a broken compiler. - previous_cache = os.environ.get("CUPY_CACHE_DIR") - try: - with tempfile.TemporaryDirectory(prefix="camera-viz-cupy-") as cache: - os.environ["CUPY_CACHE_DIR"] = cache - print(check_kernel(args.cuda_major)) - finally: - if previous_cache is None: - os.environ.pop("CUPY_CACHE_DIR", None) - else: - os.environ["CUPY_CACHE_DIR"] = previous_cache - except Exception as exc: - print(f"CUDA check failed: {exc}", file=sys.stderr) - if not args.detect: - print( - "Install matching CUDA runtime, NVRTC, headers, and CuPy. On Jetson, use " - "your JetPack's CUDA packages and rerun camera_viz.sh setup --jetson. " - "nvidia-smi reports driver support, not the installed toolkit.", - file=sys.stderr, - ) - if "CUDA_ERROR_NO_BINARY_FOR_GPU" in str(exc): - print( - "For this kernel-binary mismatch, CUPY_COMPILE_WITH_PTX=1 may " - "work around an older NVRTC compiler; prefer a toolkit/CuPy " - "combination that supports your GPU.", - file=sys.stderr, - ) - try: - import cupy as cp - - cp.show_config() - except Exception: - pass # Preserve the original failure even if diagnostics fail. - return 1 - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/examples/camera_viz/sources/_helpers.py b/examples/camera_viz/sources/_helpers.py index 2ba8fd94ff..93cf6efba9 100644 --- a/examples/camera_viz/sources/_helpers.py +++ b/examples/camera_viz/sources/_helpers.py @@ -164,8 +164,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - f"{self._kind} source requires CuPy. Run `camera_viz.sh setup` " - "to install the package matching your CUDA toolkit." + f"{self._kind} source requires CuPy (cupy-cuda12x). " + "Install via `uv pip install cupy-cuda12x`." ) from e self._cp = cp diff --git a/examples/camera_viz/sources/oakd.py b/examples/camera_viz/sources/oakd.py index 11b1c78865..405ea1488b 100644 --- a/examples/camera_viz/sources/oakd.py +++ b/examples/camera_viz/sources/oakd.py @@ -136,8 +136,7 @@ def __init__(self, device_id: str, streams: List[_StreamSpec]) -> None: except ImportError as e: raise RuntimeError( "OakdSource requires CuPy + depthai. " - "Run `camera_viz.sh setup` to install the camera dependencies " - "and CuPy matching your CUDA toolkit." + "Install via `uv pip install cupy-cuda12x depthai`." ) from e self._device_id = device_id diff --git a/examples/camera_viz/sources/rtp_h264.py b/examples/camera_viz/sources/rtp_h264.py index 6ec5fa67a8..4b957f9db1 100644 --- a/examples/camera_viz/sources/rtp_h264.py +++ b/examples/camera_viz/sources/rtp_h264.py @@ -52,8 +52,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "RtpH264Source requires CuPy. Run `camera_viz.sh setup --with-rtp` " - "to install the package matching your CUDA toolkit." + "RtpH264Source requires CuPy. Install via " + "`uv pip install cupy-cuda12x`." ) from e self._spec = SourceSpec( diff --git a/examples/camera_viz/sources/synthetic.py b/examples/camera_viz/sources/synthetic.py index 1d8125c2d4..e0c30fd03f 100644 --- a/examples/camera_viz/sources/synthetic.py +++ b/examples/camera_viz/sources/synthetic.py @@ -38,8 +38,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "SyntheticSource requires CuPy. Run `camera_viz.sh setup` " - "to install the package matching your CUDA toolkit." + "SyntheticSource requires CuPy (cupy-cuda12x). Install via " + "`uv pip install cupy-cuda12x` or skip this source." ) from e self._cp = cp @@ -57,7 +57,6 @@ def __init__( self._publish_idx: int = -1 # -1 = nothing published yet self._consumed_idx: int = -2 # track what `latest()` last returned self._lock = threading.Lock() - self._error: Optional[BaseException] = None self._stop = threading.Event() self._thread: Optional[threading.Thread] = None @@ -71,10 +70,6 @@ def start(self) -> None: if self._thread is not None: return self._stop.clear() - with self._lock: - self._error = None - self._publish_idx = -1 - self._consumed_idx = -2 self._t0_ns = time.monotonic_ns() self._thread = threading.Thread( target=self._produce_loop, name=f"synth_{self._spec.name}", daemon=False @@ -89,11 +84,6 @@ def stop(self) -> None: def latest(self) -> Optional[Frame]: with self._lock: - if self._error is not None: - raise RuntimeError( - f"Source {self._spec.name!r} failed: {self._error}. " - "Run camera_viz.sh setup to check the CUDA environment." - ) from self._error if self._publish_idx < 0 or self._publish_idx == self._consumed_idx: return None idx = self._publish_idx @@ -106,14 +96,6 @@ def latest(self) -> Optional[Frame]: ) def _produce_loop(self) -> None: - try: - self._produce_frames() - except BaseException as exc: - # latest() forwards the producer failure to VizRunner.wait(). - with self._lock: - self._error = exc - - def _produce_frames(self) -> None: cp = self._cp h, w = self._spec.height, self._spec.width # Pin this producer thread to the GPU the pre-allocated buffers @@ -179,8 +161,8 @@ def __init__( import cupy as cp except ImportError as e: raise RuntimeError( - "SyntheticStereoSource requires CuPy. Run `camera_viz.sh setup` " - "to install the package matching your CUDA toolkit." + "SyntheticStereoSource requires CuPy (cupy-cuda12x). Install via " + "`uv pip install cupy-cuda12x` or skip this source." ) from e self._cp = cp @@ -198,7 +180,6 @@ def __init__( self._publish_idx: int = -1 self._consumed_idx: int = -2 self._lock = threading.Lock() - self._error: Optional[BaseException] = None self._stop = threading.Event() self._thread: Optional[threading.Thread] = None @@ -212,10 +193,6 @@ def start(self) -> None: if self._thread is not None: return self._stop.clear() - with self._lock: - self._error = None - self._publish_idx = -1 - self._consumed_idx = -2 self._t0_ns = time.monotonic_ns() self._thread = threading.Thread( target=self._produce_loop, @@ -232,11 +209,6 @@ def stop(self) -> None: def latest(self) -> Optional[Frame]: with self._lock: - if self._error is not None: - raise RuntimeError( - f"Source {self._spec.name!r} failed: {self._error}. " - "Run camera_viz.sh setup to check the CUDA environment." - ) from self._error if self._publish_idx < 0 or self._publish_idx == self._consumed_idx: return None idx = self._publish_idx @@ -250,14 +222,6 @@ def latest(self) -> Optional[Frame]: ) def _produce_loop(self) -> None: - try: - self._produce_frames() - except BaseException as exc: - # latest() forwards the producer failure to VizRunner.wait(). - with self._lock: - self._error = exc - - def _produce_frames(self) -> None: cp = self._cp h, w = self._spec.height, self._spec.width with cp.cuda.Device(int(self._left[0].device.id)): diff --git a/examples/camera_viz/sources/zed.py b/examples/camera_viz/sources/zed.py index f3fc67506f..e649d569a3 100644 --- a/examples/camera_viz/sources/zed.py +++ b/examples/camera_viz/sources/zed.py @@ -142,8 +142,8 @@ def __init__( except ImportError as e: raise RuntimeError( "ZedSource requires CuPy + pyzed. " - "Run `camera_viz.sh setup --with-zed` with the ZED SDK installed " - "to install pyzed and CuPy matching your CUDA toolkit." + "Install via `uv pip install cupy-cuda12x` and follow the ZED " + "SDK's Python install instructions for pyzed." ) from e self._resolution_name = _resolution_for_dims(width, height) diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 8acee24b2a..a3ad52b9b9 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -47,8 +47,6 @@ Shared C++ fixtures (not executables) live under `tests/cpp/viz/support/` multiline layout. - Format changed Python tests before staging; if pre-commit rewrites a file, restage the result and rerun the full hook set. -- Use raw strings for intentional regexes in `pytest.raises(match=...)`; escape - literal punctuation such as version-number dots (Ruff RUF043). - CTest runs one `pytest` invocation per `test_*.py` with `WORKING_DIRECTORY` set to that leaf. - Use [`repo_paths.py`](python/repo_paths.py) for paths into `src/python/` or diff --git a/tests/python/examples/camera_viz/conftest.py b/tests/python/examples/camera_viz/conftest.py index 4cec3ca556..2c1ec96f82 100644 --- a/tests/python/examples/camera_viz/conftest.py +++ b/tests/python/examples/camera_viz/conftest.py @@ -16,11 +16,3 @@ from repo_paths import repo_root # noqa: E402 sys.path.insert(0, str(repo_root() / "examples" / "camera_viz")) - - -def pytest_addoption(parser): - parser.addoption( - "--require-gpu", - action="store_true", - help="Fail camera rendering tests if CUDA/CuPy/Vulkan is unavailable", - ) diff --git a/tests/python/examples/camera_viz/test_cuda_setup.py b/tests/python/examples/camera_viz/test_cuda_setup.py deleted file mode 100644 index 26ceaf8840..0000000000 --- a/tests/python/examples/camera_viz/test_cuda_setup.py +++ /dev/null @@ -1,141 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -"""CUDA discovery and failure diagnostics, without requiring a GPU.""" - -import json -import os -from pathlib import Path -import sys -from types import SimpleNamespace - -import numpy as np -import pytest - -from scripts import check_cuda - - -@pytest.mark.parametrize("version", ["12.6.10", "13.0.0", "13.2.0"]) -def test_explicit_toolkit_takes_precedence(tmp_path, monkeypatch, version): - toolkit = tmp_path / "selected" - toolkit.mkdir() - (toolkit / "version.json").write_text(json.dumps({"cuda": {"version": version}})) - monkeypatch.setenv("CUDA_PATH", str(toolkit)) - monkeypatch.setattr( - check_cuda.shutil, "which", lambda _: "/wrong/cuda-12.0/bin/nvcc" - ) - assert check_cuda.detect_toolkit() == ( - toolkit, - tuple(map(int, version.split(".")[:2])), - ) - - -def test_toolkit_on_path(tmp_path, monkeypatch): - toolkit = tmp_path / "cuda-13.0" - toolkit.mkdir() - monkeypatch.delenv("CUDA_PATH", raising=False) - monkeypatch.setattr(check_cuda.shutil, "which", lambda _: str(toolkit / "bin/nvcc")) - assert check_cuda.detect_toolkit() == (toolkit, (13, 0)) - - -def test_toolkit_symlink(tmp_path, monkeypatch): - toolkit = tmp_path / "cuda-12.6" - toolkit.mkdir() - alias = tmp_path / "cuda" - alias.symlink_to(toolkit) - monkeypatch.setenv("CUDA_PATH", str(alias)) - assert check_cuda.detect_toolkit() == (toolkit, (12, 6)) - - -@pytest.mark.parametrize("exists", [False, True]) -def test_unknown_toolkit_does_not_guess_cuda12(tmp_path, monkeypatch, exists): - toolkit = tmp_path / "unknown" - if exists: - toolkit.mkdir() - monkeypatch.setenv("CUDA_PATH", str(toolkit)) - with pytest.raises(RuntimeError, match=r"Cannot determine.*CUDA toolkit"): - check_cuda.detect_toolkit() - - -class _Array(np.ndarray): - def get(self): - return np.asarray(self) - - -@pytest.fixture -def cupy(monkeypatch): - cp = SimpleNamespace( - __version__="test-version", - float32=np.float32, - arange=lambda *a, **kw: np.arange(*a, **kw).view(_Array), - cuda=SimpleNamespace( - Device=lambda: SimpleNamespace(id=0), - runtime=SimpleNamespace( - getDeviceProperties=lambda _: {"name": b"NVIDIA Thor"} - ), - nvrtc=SimpleNamespace(getVersion=lambda: (13, 0)), - Stream=SimpleNamespace(null=SimpleNamespace(synchronize=lambda: None)), - ), - show_config=lambda: print("CUDA diagnostic configuration"), - ) - monkeypatch.setitem(sys.modules, "cupy", cp) - return cp - - -@pytest.mark.parametrize( - "name, version", [("Orin", (12, 6)), ("Thor", (13, 0)), ("Thor", (13, 2))] -) -def test_matching_runtime_executes_kernel(cupy, name, version): - cupy.cuda.nvrtc.getVersion = lambda: version - cupy.cuda.runtime.getDeviceProperties = lambda _: {"name": name.encode()} - assert name in check_cuda.check_kernel(version[0]) - - -def test_mismatched_nvrtc_is_rejected(cupy): - cupy.cuda.nvrtc.getVersion = lambda: (12, 6) - with pytest.raises( - RuntimeError, match=r"Selected CUDA 13, but CuPy loaded NVRTC 12\.6" - ): - check_cuda.check_kernel(13) - - -@pytest.mark.parametrize("at_sync", [False, True]) -def test_kernel_failure_returns_nonzero_and_diagnostics(cupy, capsys, at_sync): - def fail(*a, **kw): - raise RuntimeError("CUDA_ERROR_NO_BINARY_FOR_GPU") - - if at_sync: - cupy.cuda.Stream.null.synchronize = fail - else: - cupy.arange = fail - assert check_cuda.main(["--cuda-major", "13"]) == 1 - output = capsys.readouterr() - assert "CUDA_ERROR_NO_BINARY_FOR_GPU" in output.err - assert "CUPY_COMPILE_WITH_PTX=1" in output.err - assert "CUDA diagnostic configuration" in output.out - assert "passed" not in output.out - - -def test_incorrect_kernel_result_is_rejected(cupy): - cupy.arange = lambda *a, **kw: np.zeros(4).view(_Array) - with pytest.raises(RuntimeError, match="incorrect results"): - check_cuda.check_kernel(13) - - -def test_probe_uses_cold_cache_and_restores_environment(cupy, monkeypatch, tmp_path): - original = str(tmp_path / "existing-cache") - monkeypatch.setenv("CUPY_CACHE_DIR", original) - paths = [] - arange = cupy.arange - - def checked_arange(*a, **kw): - path = Path(os.environ["CUPY_CACHE_DIR"]) - assert str(path) != original - assert path.is_dir() - paths.append(path) - return arange(*a, **kw) - - cupy.arange = checked_arange - assert check_cuda.main([]) == 0 - assert os.environ["CUPY_CACHE_DIR"] == original - assert len(paths) == 1 - assert not paths[0].exists() diff --git a/tests/python/examples/camera_viz/test_runner_health.py b/tests/python/examples/camera_viz/test_runner_health.py index 48ad7783e6..903e1ad781 100644 --- a/tests/python/examples/camera_viz/test_runner_health.py +++ b/tests/python/examples/camera_viz/test_runner_health.py @@ -31,18 +31,3 @@ def health_check() -> None: runner.wait(health_check=health_check) assert checks == 1 - - -def test_source_failure_stops_runner_and_reaches_wait() -> None: - failure = RuntimeError("camera producer: CUDA_ERROR_NO_BINARY_FOR_GPU") - - class FailedSource: - def latest(self): - raise failure - - runner = VizRunner(object(), [FailedSource()], [object()]) - runner._submit_loop() - assert runner._stop.is_set() - with pytest.raises(RuntimeError, match="CUDA_ERROR_NO_BINARY_FOR_GPU") as error: - runner.wait() - assert error.value is failure diff --git a/tests/python/examples/camera_viz/test_synthetic_failures.py b/tests/python/examples/camera_viz/test_synthetic_failures.py deleted file mode 100644 index 744c65d4a9..0000000000 --- a/tests/python/examples/camera_viz/test_synthetic_failures.py +++ /dev/null @@ -1,80 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -"""Producer failures reach the consumer instead of leaving an empty mailbox.""" - -from contextlib import nullcontext -import sys -from types import SimpleNamespace - -import numpy as np -import pytest - -from sources.synthetic import SyntheticSource, SyntheticStereoSource - - -class _Buffer(np.ndarray): - @property - def device(self): - return SimpleNamespace(id=0) - - -@pytest.fixture(params=[SyntheticSource, SyntheticStereoSource]) -def source(request, monkeypatch): - cp = SimpleNamespace( - zeros=lambda shape, dtype: np.zeros(shape, dtype=dtype).view(_Buffer), - arange=np.arange, - sin=np.sin, - uint8=np.uint8, - float32=np.float32, - cuda=SimpleNamespace( - Device=lambda _: nullcontext(), - Stream=SimpleNamespace(null=SimpleNamespace(synchronize=lambda: None)), - ), - ) - monkeypatch.setitem(sys.modules, "cupy", cp) - instance = request.param("test-camera", 8, 4) - yield instance - instance.stop() - - -@pytest.mark.parametrize("at_sync", [False, True]) -def test_cuda_failure_reaches_latest_and_survives_stop(source, at_sync): - failure = RuntimeError("CUDA_ERROR_NO_BINARY_FOR_GPU") - - def fail(*args, **kwargs): - raise failure - - if at_sync: - source._cp.cuda.Stream.null.synchronize = fail - else: - source._cp.arange = fail - source.start() - source._thread.join(timeout=2) - assert not source._thread.is_alive(), "producer did not terminate" - for _ in range(2): - with pytest.raises( - RuntimeError, match=r"test-camera.*CUDA_ERROR_NO_BINARY_FOR_GPU" - ) as error: - source.latest() - assert error.value.__cause__ is failure - source.stop() - - -def test_restart_clears_failure_and_stale_frame(source): - failure = RuntimeError("producer failed after publishing") - - def fail(): - with source._lock: - source._publish_idx = 0 - raise failure - - source._produce_frames = fail - source.start() - source._thread.join(timeout=2) - assert not source._thread.is_alive() - with pytest.raises(RuntimeError, match="producer failed after publishing"): - source.latest() - source.stop() - source._produce_frames = source._stop.wait - source.start() - assert source.latest() is None diff --git a/tests/python/examples/camera_viz/test_synthetic_offscreen.py b/tests/python/examples/camera_viz/test_synthetic_offscreen.py index 3a951b4f4c..1eea586a6a 100644 --- a/tests/python/examples/camera_viz/test_synthetic_offscreen.py +++ b/tests/python/examples/camera_viz/test_synthetic_offscreen.py @@ -47,13 +47,9 @@ def _gpu_and_cupy_available() -> bool: return True -@pytest.fixture(scope="module", autouse=True) -def require_rendering_environment(request): - if not _gpu_and_cupy_available(): - reason = "no Vulkan/CUDA GPU or cupy missing" - if request.config.getoption("--require-gpu"): - pytest.fail(reason) - pytest.skip(reason) +pytestmark = pytest.mark.skipif( + not _gpu_and_cupy_available(), reason="no Vulkan/CUDA GPU or cupy missing" +) # Bounded wait helper so a stuck producer fails fast instead of hanging CI. From 5ae55f589a483ee9f6903dc865363e69dc9b71d4 Mon Sep 17 00:00:00 2001 From: Farbod Motlagh Date: Tue, 6 Oct 2026 16:41:50 -0700 Subject: [PATCH 5/5] docs: remove documentation revision scope guidance Signed-off-by: Farbod Motlagh --- docs/AGENTS.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 238ccdf895..9f7f806fa1 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -97,12 +97,3 @@ and `logo_dark`. Keep cache-removal documentation focused on migration impact. Omit asset-fetch implementation walkthroughs and alternative override instructions. - -## Scope of documentation revisions - -When asked to shorten additions, edit only the branch's additions. Preserve the -existing reference content, examples, and structure. Propose broader cleanup -separately; do not apply it without agreement. Keep new prerequisites and checks concise. - -Distinguish runtime dependencies from source-build tools. Make installation steps -conditional when existing environments may already satisfy the requirements.