fix(cloudxr): refuse to start over a live runtime; explain -35 - #911
Open
jiwenc-nv wants to merge 1 commit into
Open
fix(cloudxr): refuse to start over a live runtime; explain -35#911jiwenc-nv wants to merge 1 commit into
-35#911jiwenc-nv wants to merge 1 commit into
Conversation
Contributor
📝 WalkthroughWalkthroughCloudXR runtime startup now probes Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant CloudXRLauncher
participant RuntimeIPC
participant ExistingRuntime
CLI->>CloudXRLauncher: Start with optional --force
CloudXRLauncher->>RuntimeIPC: Probe ipc_cloudxr
RuntimeIPC-->>CloudXRLauncher: Return liveness
alt Existing runtime is live
CloudXRLauncher->>ExistingRuntime: Send SIGTERM when forced
CloudXRLauncher->>RuntimeIPC: Wait for termination
else Socket is stale
CloudXRLauncher->>RuntimeIPC: Remove stale socket
end
CloudXRLauncher-->>CLI: Start runtime or report error
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/python/isaacteleop/cloudxr/launcher.py`:
- Around line 594-636: The launch flow currently relies on the point-in-time
is_runtime_live check and force-mode socket disappearance, so runtime ownership
is not atomic and shutdown is not confirmed. Add an exclusive run-directory
ownership lock that remains held from startup through the runtime lifetime,
acquire it before checking or removing stale state, and ensure it is released
only when the launcher exits. Update _terminate_live_runtime to identify and
wait for the prior owner process to exit, not merely until is_runtime_live
returns false, before cleanup and replacement startup proceed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 90cb1f88-3f35-4695-a691-bdb2dabd7a7d
📒 Files selected for processing (11)
docs/source/getting_started/quick_start.rstdocs/source/getting_started/televiz.rstdocs/source/references/cloudxr.rstsrc/core/cloudxr_tests/python/test_launcher.pysrc/core/cloudxr_tests/python/test_runtime.pysrc/core/oxr/cpp/oxr_session.cppsrc/python/isaacteleop/cloudxr/__main__.pysrc/python/isaacteleop/cloudxr/env_config.pysrc/python/isaacteleop/cloudxr/launcher.pysrc/python/isaacteleop/cloudxr/runtime.pysrc/viz/xr/cpp/openxr_session.cpp
Comment on lines
+594
to
+636
| if is_runtime_live(run_dir): | ||
| if not force: | ||
| raise RuntimeError( | ||
| f"A CloudXR runtime is already serving {run_dir}; starting a " | ||
| "second one would drop the live session. To use the running " | ||
| f"runtime: source {env_cfg.env_filepath()} and pass " | ||
| "--no-launch-cloudxr-runtime. To take it over: " | ||
| "python -m isaacteleop.cloudxr --force, or " | ||
| "CloudXRLauncher(force=True)." | ||
| ) | ||
| if result.returncode == 0: | ||
| time.sleep(1) | ||
| logger.info("Sent SIGTERM to processes holding stale IPC socket") | ||
| except (FileNotFoundError, subprocess.TimeoutExpired): | ||
| pass | ||
| CloudXRLauncher._terminate_live_runtime(run_dir, ipc_socket) | ||
|
|
||
| for name in ("ipc_cloudxr", "runtime_started", "monado.pid", "cloudxr.pid"): | ||
| path = os.path.join(run_dir, name) | ||
| try: | ||
| os.remove(ipc_socket) | ||
| os.remove(path) | ||
| except FileNotFoundError: | ||
| pass | ||
| continue | ||
| logger.warning("Removed stale CloudXR runtime file %s", path) | ||
|
|
||
| for name in ("runtime_started", "monado.pid", "cloudxr.pid"): | ||
| try: | ||
| os.remove(os.path.join(run_dir, name)) | ||
| except FileNotFoundError: | ||
| pass | ||
| @staticmethod | ||
| def _terminate_live_runtime(run_dir: str, ipc_socket: str) -> None: | ||
| """SIGTERM whoever holds ``ipc_socket`` and wait for the socket to go dead.""" | ||
| logger.warning( | ||
| "--force: terminating the CloudXR runtime serving %s", ipc_socket | ||
| ) | ||
| try: | ||
| subprocess.run( | ||
| ["fuser", "-k", "-TERM", ipc_socket], capture_output=True, timeout=5 | ||
| ) | ||
| except (FileNotFoundError, subprocess.TimeoutExpired) as exc: | ||
| raise RuntimeError( | ||
| f"Cannot take over the runtime serving {ipc_socket}: 'fuser' is " | ||
| "unavailable (install psmisc), so the running process cannot be " | ||
| "identified. Stop it manually and retry." | ||
| ) from exc | ||
|
|
||
| deadline = time.monotonic() + RUNTIME_TERMINATE_TIMEOUT_SEC | ||
| while time.monotonic() < deadline: | ||
| if not is_runtime_live(run_dir): | ||
| logger.warning("Previous CloudXR runtime stopped") | ||
| return | ||
| time.sleep(0.2) |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Make runtime ownership atomic and confirm shutdown.
is_runtime_live() is only a point-in-time check. Two launchers can both observe no listener while one runtime starts. They can then remove markers and start against the same run directory. A later unlink can detach the first runtime's live socket name.
In force mode, a closed IPC socket does not prove that the prior process exited. Its signal handler can close the listener before it releases other runtime resources.
Use an exclusive ownership lock that spans startup and runtime lifetime. When force=True, wait for the identified owner process to exit before removing files or starting the replacement runtime.
🧰 Tools
🪛 ast-grep (0.45.0)
[error] 620-622: Command coming from incoming request
Context: subprocess.run(
["fuser", "-k", "-TERM", ipc_socket], capture_output=True, timeout=5
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/python/isaacteleop/cloudxr/launcher.py` around lines 594 - 636, The
launch flow currently relies on the point-in-time is_runtime_live check and
force-mode socket disappearance, so runtime ownership is not atomic and shutdown
is not confirmed. Add an exclusive run-directory ownership lock that remains
held from startup through the runtime lifetime, acquire it before checking or
removing stale state, and ensure it is released only when the launcher exits.
Update _terminate_live_runtime to identify and wait for the prior owner process
to exit, not merely until is_runtime_live returns false, before cleanup and
replacement startup proceed.
jiwenc-nv
force-pushed
the
jiwenc-nv/IT908
branch
from
August 5, 2026 22:55
14c8c2e to
5810c72
Compare
A second launch treated the existence of `ipc_cloudxr` as proof of a stale runtime and `fuser -k -TERM`ed whoever held it — including a healthy runtime mid-session, which the first session saw as a broken pipe. When `fuser` was missing the error was swallowed and the socket unlinked anyway, leaving the old process running and the operator to clean `~/.cloudxr/run/` by hand. Liveness is now decided by connecting to the socket rather than by stat-ing it. A live runtime makes the launcher refuse, pointing at the env file and `--no-launch-cloudxr-runtime`; replacing it means stopping it yourself. A dead socket is cleaned with a warning. Ambiguous probe errors count as live, since refusing is recoverable and clobbering a session is not. `-35` is thrown after `xrCreateInstance` succeeds, so the runtime was found and only the headset is missing, but library defaults are fail-fast and a direct consumer got a bare `Failed to get OpenXR system: -35`. Both throw sites now name `XR_ERROR_FORM_FACTOR_UNAVAILABLE` and point at the docs. The device profile is the usual culprit, so the startup banner prints the resolved value and the launcher/env-config defaults agree on one constant — the docs claimed `auto-webrtc` where `Quest3` always won. Overriding it stays an env-file or `CloudXRLauncher(device_profile=...)` job; Apple Vision Pro needs `auto-native`. Closes #908 Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
jiwenc-nv
force-pushed
the
jiwenc-nv/IT908
branch
from
August 5, 2026 23:02
5810c72 to
f3f830e
Compare
| The CloudXR runtime uses the ``auto-webrtc`` device profile by default | ||
| (Pico & Quest). For Apple Vision Pro it defaults to ``auto-native``. To | ||
| override settings, write a ``KEY=value`` env file and pass it to the example | ||
| The CloudXR runtime uses the ``Quest3`` device profile by default; Apple Vision |
Collaborator
There was a problem hiding this comment.
we should use auto-webrtc tho?
Collaborator
Author
There was a problem hiding this comment.
The problem with auto-webrtc is that you have to connect a device first start running OpenXR session, which has been a hotspot for getting people blocked... (the -35 error).
@nv-jakob already switched the default profile to Quest3 a while ago, but we just haven't update the doc yet.
Collaborator
Author
There was a problem hiding this comment.
Ideally, we should just rename Quest3 to generic-webxr, wdyt?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Items 1 and 2 of #736.
_cleanup_stale_runtimetreated an existingipc_cloudxras proof of staleness andfuser -k -TERMed its holder — including a healthy runtime mid-session, which the first session saw as a broken pipe. Withfuserabsent the error was swallowed and the socket unlinked anyway, so the old process survived; hence the manualrm.Liveness is now a
connect(). Live → refuse, naming the resolved env file and--no-launch-cloudxr-runtime; replacing it means stopping it yourself. Dead → clean, at WARNING. Ambiguous probe errors count as live. Nothing signals another process now, so the undeclaredfuser/psmiscdependency goes too.Both
-35throw sites now nameXR_ERROR_FORM_FACTOR_UNAVAILABLEand point at the one new docs section, carrying the-51contrast and the checklist. The banner prints the resolved device profile, and the two device-profile defaults collapse onto one constant — docs saidauto-webrtc,Quest3always won.#908 also proposed
--forceand--cloudxr-device-profile; both dropped as unnecessary.Fixes #908
Type of change
Testing
src/core/cloudxr_tests/python/), covering the probe and the refusal.SKIP=check-copyright-year pre-commit run --all-filesclean.Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO