Skip to content

examples/egocap: add Ego Capture UI example - #1403

Merged
gareth-morgan-nv merged 11 commits into
mainfrom
gmorgan/egocap-transfer
Oct 31, 2026
Merged

gareth-morgan-nv merged 11 commits into
mainfrom
gmorgan/egocap-transfer

Conversation

@gareth-morgan-nv

@gareth-morgan-nv gareth-morgan-nv commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Imports the Ego Capture UI example: a recording module library, a standalone CaptureServer, and a web backend/frontend for ego-centric capture sessions. The source layout and behavior are preserved from the upstream example; destination-required formatting, file-mode, and license-header fixes were applied on top.

Notes for reviewers

  • Content is imported from the upstream source, which passed its own public-release scan before export.
  • Destination-required fixes applied: examples/egocap/recording/tools/replay_episode.py made executable (shebang-executable check), NOTICE.md/config.yaml trailing-whitespace/EOF fixes, and standard SPDX headers added across the tree (third-party LICENSE.*/NOTICE.md provenance text itself was left untouched).
  • A local run of this repo's REUSE license check crashed with a parser error unrelated to this change (triggered by pre-existing untracked content in the local checkout, not by anything in this PR); the equivalent CI check ran cleanly against the real checkout and its required header additions are included in this PR.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a complete Ego Capture experience for live capture previews, tracking and camera status, recording controls, voice commands, and optional 3D visualization.
    • Added configurable station setup, task catalogs, recording quality checks, recovery workflows, and a separate recordings browser.
    • Added capture-server support for mock, live hardware, GMSL camera feeds, and recorded-session replay.
  • Documentation
    • Added setup, configuration, and usage guides for the capture UI, recording tools, and capture server.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@gareth-morgan-nv gareth-morgan-nv self-assigned this Oct 30, 2026
@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 49a2181 and 186f352.

Files ignored due to path filters: 10
  • examples/egocap/assets/design/browser-connections.svg is excluded by !**/*.svg
  • examples/egocap/assets/design/system-overview.svg is excluded by !**/*.svg
  • examples/egocap/data/taskfiles/Kitchen.csv is excluded by !**/*.csv
  • examples/egocap/frontend/package-lock.json is excluded by !**/package-lock.json
  • examples/egocap/recordings_frontend/package-lock.json is excluded by !**/package-lock.json
  • examples/egocap/tests/recording/fixtures/mid_train/grasp_tasks.csv is excluded by !**/*.csv
  • examples/egocap/tests/recording/fixtures/task_library/Dining_Room.csv is excluded by !**/*.csv
  • examples/egocap/tests/recording/fixtures/task_library/Kitchen.csv is excluded by !**/*.csv
  • examples/egocap/tests/recording/fixtures/task_library/Summary.csv is excluded by !**/*.csv
  • examples/egocap/uv.lock is excluded by !**/*.lock
Files selected for processing: 217
  • examples/egocap/.gitignore
  • examples/egocap/AGENTS.md
  • examples/egocap/DESIGN.md
  • examples/egocap/README.md
  • examples/egocap/assets/viser/LICENSE.headset.md
  • examples/egocap/assets/viser/NOTICE.md
  • examples/egocap/assets/viser/headset.glb
  • examples/egocap/assets/viser/pico-4-ultra/LICENSE.md
  • examples/egocap/assets/viser/pico-4-ultra/build_headset.py
  • examples/egocap/assets/viser/pico-4-ultra/headset.glb
  • examples/egocap/assets/viser/pico-4-ultra/left.glb
  • examples/egocap/assets/viser/pico-4-ultra/right.glb
  • examples/egocap/assets/viser/pico-4/LICENSE.md
  • examples/egocap/assets/viser/pico-4/left.glb
  • examples/egocap/assets/viser/pico-4/right.glb
  • examples/egocap/backend/README.md
  • examples/egocap/backend/egocap_web/__init__.py
  • examples/egocap/backend/egocap_web/app.py
  • examples/egocap/backend/egocap_web/capture_preview.py
  • examples/egocap/backend/egocap_web/capture_status.py
  • examples/egocap/backend/egocap_web/config.py
  • examples/egocap/backend/egocap_web/preview_protocol.py
  • examples/egocap/backend/egocap_web/preview_routes.py
  • examples/egocap/backend/egocap_web/recording_adapter.py
  • examples/egocap/backend/egocap_web/recording_command.py
  • examples/egocap/backend/egocap_web/recording_session.py
  • examples/egocap/backend/egocap_web/recordings.py
  • examples/egocap/backend/egocap_web/station_config.py
  • examples/egocap/backend/egocap_web/task_file_routes.py
  • examples/egocap/backend/egocap_web/task_files.py
  • examples/egocap/backend/egocap_web/viser_scene.py
  • examples/egocap/build_support.py
  • examples/egocap/capture_server/README.md
  • examples/egocap/capture_server/capture_server/__init__.py
  • examples/egocap/capture_server/capture_server/__main__.py
  • examples/egocap/capture_server/capture_server/acquisition/__init__.py
  • examples/egocap/capture_server/capture_server/acquisition/_thread_runner.py
  • examples/egocap/capture_server/capture_server/acquisition/acquisition.py
  • examples/egocap/capture_server/capture_server/acquisition/clock.py
  • examples/egocap/capture_server/capture_server/acquisition/depthai.py
  • examples/egocap/capture_server/capture_server/acquisition/errors.py
  • examples/egocap/capture_server/capture_server/acquisition/isaac_capture.py
  • examples/egocap/capture_server/capture_server/acquisition/isaac_sdk.py
  • examples/egocap/capture_server/capture_server/acquisition/tensor_indices.py
  • examples/egocap/capture_server/capture_server/client.py
  • examples/egocap/capture_server/capture_server/gmsl_ipc/__init__.py
  • examples/egocap/capture_server/capture_server/gmsl_ipc/cuda_slots.py
  • examples/egocap/capture_server/capture_server/gmsl_ipc/feed.py
  • examples/egocap/capture_server/capture_server/gmsl_ipc/source.py
  • examples/egocap/capture_server/capture_server/launch.py
  • examples/egocap/capture_server/capture_server/mock/mock_depthai.py
  • examples/egocap/capture_server/capture_server/mock/mock_isaac_capture.py
  • examples/egocap/capture_server/capture_server/protocol.py
  • examples/egocap/capture_server/capture_server/replay/__init__.py
  • examples/egocap/capture_server/capture_server/replay/episode.py
  • examples/egocap/capture_server/capture_server/replay/playback.py
  • examples/egocap/capture_server/capture_server/replay/replay_depthai.py
  • examples/egocap/capture_server/capture_server/replay/replay_isaac_capture.py
  • examples/egocap/capture_server/capture_server/replay/tracking_map.py
  • examples/egocap/capture_server/capture_server/server.py
  • examples/egocap/data/taskfiles/README.md
  • examples/egocap/frontend/.gitignore
  • examples/egocap/frontend/README.md
  • examples/egocap/frontend/index.html
  • examples/egocap/frontend/package.json
  • examples/egocap/frontend/src/App.liveRecordingWire.test.tsx
  • examples/egocap/frontend/src/App.test.tsx
  • examples/egocap/frontend/src/App.tsx
  • examples/egocap/frontend/src/CameraGrid.test.tsx
  • examples/egocap/frontend/src/CameraGrid.tsx
  • examples/egocap/frontend/src/CaptureTree.tsx
  • examples/egocap/frontend/src/PoseViserPanel.test.tsx
  • examples/egocap/frontend/src/PoseViserPanel.tsx
  • examples/egocap/frontend/src/RecordingPanel.test.tsx
  • examples/egocap/frontend/src/RecordingPanel.tsx
  • examples/egocap/frontend/src/RecordingPanel.voice.test.tsx
  • examples/egocap/frontend/src/cameraRates.test.ts
  • examples/egocap/frontend/src/cameraRates.ts
  • examples/egocap/frontend/src/captureProtocol.test.ts
  • examples/egocap/frontend/src/captureProtocol.ts
  • examples/egocap/frontend/src/captureStatus.test.ts
  • examples/egocap/frontend/src/captureStatus.ts
  • examples/egocap/frontend/src/main.tsx
  • examples/egocap/frontend/src/realDescriptor.test.ts
  • examples/egocap/frontend/src/recordingActions.ts
  • examples/egocap/frontend/src/recordingEligibility.ts
  • examples/egocap/frontend/src/recordingTypes.ts
  • examples/egocap/frontend/src/recordingUiReducer.ts
  • examples/egocap/frontend/src/recordingWire.test.ts
  • examples/egocap/frontend/src/recordingWire.ts
  • examples/egocap/frontend/src/styles.css
  • examples/egocap/frontend/src/trackingDecoder.test.ts
  • examples/egocap/frontend/src/trackingDecoder.ts
  • examples/egocap/frontend/src/useCaptureSession.hook.test.ts
  • examples/egocap/frontend/src/useCaptureSession.test.ts
  • examples/egocap/frontend/src/useCaptureSession.ts
  • examples/egocap/frontend/src/useVoiceRecognition.test.ts
  • examples/egocap/frontend/src/useVoiceRecognition.ts
  • examples/egocap/frontend/src/vite-env.d.ts
  • examples/egocap/frontend/src/voiceCommands.test.ts
  • examples/egocap/frontend/src/voiceCommands.ts
  • examples/egocap/frontend/tsconfig.json
  • examples/egocap/frontend/vite.config.ts
  • examples/egocap/frontend/vitest.config.ts
  • examples/egocap/pyproject.toml
  • examples/egocap/recording/README.md
  • examples/egocap/recording/egocap/__init__.py
  • examples/egocap/recording/egocap/analytics/__init__.py
  • examples/egocap/recording/egocap/analytics/count_daily_trajectories.py
  • examples/egocap/recording/egocap/analytics/station_status_log.py
  • examples/egocap/recording/egocap/config/data_check/data_check_config.yaml
  • examples/egocap/recording/egocap/config/data_recording/config.yaml
  • examples/egocap/recording/egocap/config/data_recording/pico_manus_config.yaml
  • examples/egocap/recording/egocap/core/__init__.py
  • examples/egocap/recording/egocap/core/config_io.py
  • examples/egocap/recording/egocap/core/config_utils.py
  • examples/egocap/recording/egocap/core/data_root.py
  • examples/egocap/recording/egocap/data_processing/update_episode_task_description.py
  • examples/egocap/recording/egocap/egocap_stations.example.json
  • examples/egocap/recording/egocap/eod/__init__.py
  • examples/egocap/recording/egocap/eod/eod_studio_session_move.py
  • examples/egocap/recording/egocap/recording/__init__.py
  • examples/egocap/recording/egocap/recording/capture_measure.py
  • examples/egocap/recording/egocap/recording/capture_qc.py
  • examples/egocap/recording/egocap/recording/constants.py
  • examples/egocap/recording/egocap/recording/dataset_check.py
  • examples/egocap/recording/egocap/recording/dataset_reader.py
  • examples/egocap/recording/egocap/recording/episode_error_log.py
  • examples/egocap/recording/egocap/recording/episode_layout.py
  • examples/egocap/recording/egocap/recording/episode_store.py
  • examples/egocap/recording/egocap/recording/route.py
  • examples/egocap/recording/egocap/recording/video_writer_backends.py
  • examples/egocap/recording/egocap/stations/ego_stations.example.yaml
  • examples/egocap/recording/egocap/studio/__init__.py
  • examples/egocap/recording/egocap/studio/mid_train_data.py
  • examples/egocap/recording/egocap/studio/studio_lookup.py
  • examples/egocap/recording/egocap/studio/studio_path_names.py
  • examples/egocap/recording/egocap/studio/task_library_data.py
  • examples/egocap/recording/recording_service/README.md
  • examples/egocap/recording/recording_service/__init__.py
  • examples/egocap/recording/recording_service/models.py
  • examples/egocap/recording/recording_service/recording_service.py
  • examples/egocap/recording/tools/replay_episode.py
  • examples/egocap/recordings_frontend/.gitignore
  • examples/egocap/recordings_frontend/README.md
  • examples/egocap/recordings_frontend/index.html
  • examples/egocap/recordings_frontend/package.json
  • examples/egocap/recordings_frontend/src/api.ts
  • examples/egocap/recordings_frontend/src/main.ts
  • examples/egocap/recordings_frontend/src/styles.css
  • examples/egocap/recordings_frontend/tsconfig.json
  • examples/egocap/recordings_frontend/vite.config.ts
  • examples/egocap/tests/__init__.py
  • examples/egocap/tests/backend/__init__.py
  • examples/egocap/tests/backend/conftest.py
  • examples/egocap/tests/backend/test_capture_status.py
  • examples/egocap/tests/backend/test_config.py
  • examples/egocap/tests/backend/test_preview_routes.py
  • examples/egocap/tests/backend/test_recording_adapter.py
  • examples/egocap/tests/backend/test_recording_command.py
  • examples/egocap/tests/backend/test_recording_session.py
  • examples/egocap/tests/backend/test_recordings.py
  • examples/egocap/tests/backend/test_station_config.py
  • examples/egocap/tests/backend/test_station_yaml_startup.py
  • examples/egocap/tests/backend/test_task_file_routes.py
  • examples/egocap/tests/backend/test_task_files.py
  • examples/egocap/tests/backend/test_viser_scene.py
  • examples/egocap/tests/capture_server/__init__.py
  • examples/egocap/tests/capture_server/conftest.py
  • examples/egocap/tests/capture_server/test_acquisition_adapters.py
  • examples/egocap/tests/capture_server/test_cli.py
  • examples/egocap/tests/capture_server/test_client.py
  • examples/egocap/tests/capture_server/test_gmsl_ipc.py
  • examples/egocap/tests/capture_server/test_mock_sdks.py
  • examples/egocap/tests/capture_server/test_protocol.py
  • examples/egocap/tests/capture_server/test_replay.py
  • examples/egocap/tests/capture_server/test_server.py
  • examples/egocap/tests/capture_server/test_setup_selection.py
  • examples/egocap/tests/capture_server/test_thread_runner.py
  • examples/egocap/tests/recording/integration/test_package_install.py
  • examples/egocap/tests/recording/unit/test_capture_qc.py
  • examples/egocap/tests/recording/unit/test_capture_qc_devices.py
  • examples/egocap/tests/recording/unit/test_config_io.py
  • examples/egocap/tests/recording/unit/test_config_utils.py
  • examples/egocap/tests/recording/unit/test_count_daily_trajectories.py
  • examples/egocap/tests/recording/unit/test_data_root.py
  • examples/egocap/tests/recording/unit/test_dataset_check_and_reader.py
  • examples/egocap/tests/recording/unit/test_eod_studio_session_move.py
  • examples/egocap/tests/recording/unit/test_episode_error_log.py
  • examples/egocap/tests/recording/unit/test_episode_store.py
  • examples/egocap/tests/recording/unit/test_episode_store_failure_paths.py
  • examples/egocap/tests/recording/unit/test_mid_train_data.py
  • examples/egocap/tests/recording/unit/test_route.py
  • examples/egocap/tests/recording/unit/test_station_status_log.py
  • examples/egocap/tests/recording/unit/test_studio_lookup.py
  • examples/egocap/tests/recording/unit/test_studio_path_names.py
  • examples/egocap/tests/recording/unit/test_task_library_data.py
  • examples/egocap/tests/recording/unit/test_update_episode_task_description.py
  • examples/egocap/tests/recording/unit/test_video_writer_backends.py
  • examples/egocap/tests/recording/unit/test_write_yaml_atomic.py
  • examples/egocap/tests/recording_service/conftest.py
  • examples/egocap/tests/recording_service/crash_child.py
  • examples/egocap/tests/recording_service/real.py
  • examples/egocap/tests/recording_service/test_capture_qc_service.py
  • examples/egocap/tests/recording_service/test_config_digest.py
  • examples/egocap/tests/recording_service/test_configuration.py
  • examples/egocap/tests/recording_service/test_dependency_boundary.py
  • examples/egocap/tests/recording_service/test_failures_and_recovery.py
  • examples/egocap/tests/recording_service/test_quality_check.py
  • examples/egocap/tests/recording_service/test_real_flow.py
  • examples/egocap/tests/recording_service/test_replay_tool.py
  • examples/egocap/tests/recording_service/test_routing_service.py
  • examples/egocap/tests/recording_service/test_state_guards.py
  • examples/egocap/tests/recording_service/test_station_config.py
  • examples/egocap/tests/support/__init__.py
  • examples/egocap/tests/support/capture_server.py
  • examples/egocap/tests/support/test_capture_server.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

[!NOTE]
Walkthrough couldn't be generated for this run and will be retried on the next review.

Merge Risk: Minimal · previous review 186f3

This change adds a self-contained example under examples/egocap without modifying existing code paths. No concrete merge-blocking defect was identified in the supplied context.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 64.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly summarizes the main change: adding the Ego Capture UI example and its supporting components.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 64.13% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 460 functions across 50 files. (155 skipped: 30 unsupported, 125 over the file limit.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>

@jiwenc-nv jiwenc-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1/ do we really need those pico glb files in the code base? can we populate them from http?

At least put them into git lfs: currently glb is not registered for git lfs: https://github.com/NVIDIA/IsaacCapture/blob/main/.gitattributes

please fix.

2/ we can refactor in a next PR: it's a bit weird that frontend and recordings_backend are splatted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
CaptureServer's websocket is a native-client-only endpoint (CaptureClient,
replay tools); the web backend proxies browser access through its own
same-origin /api/session instead. Setting origins=[None] rejects any
handshake carrying an Origin header while still accepting the
missing-Origin native clients this server actually serves.

Verified with real handshakes: no-Origin clients connect; a hostile
Origin and a literal "null" Origin are both rejected with HTTP 403.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@gareth-morgan-nv

Copy link
Copy Markdown
Contributor Author

🤖 Agent-generated (Claude Code · claude-sonnet-5-5)

Re: @jiwenc-nv's review comments.

1/ GLB files not in git-lfs -- fixed. Confirmed: .glb was genuinely missing from .gitattributes's "3D assets are tracked by default" section (right next to .obj/.stl/.usd*), so the 6 pico/headset .glb files (~1.1 MB total) were committed as raw blobs. Added *.glb filter=lfs diff=lfs merge=lfs -text to .gitattributes, then used git lfs migrate import to rewrite this branch's history so all 6 files are now LFS pointers rather than raw blobs. Verified: git lfs ls-files shows all 6 as LFS objects, git lfs pull + file confirms the real glTF bytes still materialize correctly, and the full test suite still passes (1236 passed, 103 skipped) after the rewrite. Pushed as 2f1e826 (force-push, since this rewrote the branch's history).

On "do we really need these in the codebase / can we populate from HTTP instead" -- leaving that as-is for now since it's a design question about this example's asset-provisioning approach, not a mechanical fix; happy to follow up if you want to pursue a fetch-at-build-time approach instead.

2/ frontend/recordings_frontend layout -- noted, no action taken. You already scoped this to a follow-up PR ("we can refactor in a next PR"), so leaving it out of this one as you suggested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@gareth-morgan-nv
gareth-morgan-nv enabled auto-merge (squash) October 31, 2026 00:11
Nothing reads the module-level alias of protocol.HAND_JOINT_NAMES.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Gareth Morgan <gmorgan@nvidia.com>
@gareth-morgan-nv
gareth-morgan-nv merged commit 71f22fd into main Oct 31, 2026
38 checks passed
@gareth-morgan-nv
gareth-morgan-nv deleted the gmorgan/egocap-transfer branch October 31, 2026 19:09

This branch was successfully deployed

1 active deployment
dev — 2b7e1743 Deployed Oct 31, 2026 by gareth-morgan-nv via publish-wheel #6370
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants