Skip to content

[Interactive Drive] Add physics via Physx- #423 - #430

Open
ArielG-NV wants to merge 25 commits into
NVIDIA:mainfrom
ArielG-NV:physics-final
Open

[Interactive Drive] Add physics via Physx- #423#430
ArielG-NV wants to merge 25 commits into
NVIDIA:mainfrom
ArielG-NV:physics-final

Conversation

@ArielG-NV

Copy link
Copy Markdown
Collaborator

Fixes: #313
Changes:

  • Added physx to drive the backend of physics. This requires maintaining our own physx python bindings.
  • Removed Vulkan backend from ludus-renderer due to significant overhaul causing this backend to be a maintenance burden that requires a complete rewrite. If there is need for the backend in the future we will just rewrite the code then.
  • Physics are generally obeying physx as the main-controller, but some hacks are added to prevent out-of-distribution behavior (yaw-lock on vehicles, speed lock on non-ego vehicles, etc...)
  • Changed the BEV render to be an orthographic top-down view for reasons that it makes driving+navigation easier. Added a green car to identify the ego on the BEV.
  • Made all collision logic (and other game-like-frills) under the --game-mode flag for the interactive-drive demos
  • Added a new view when clicking 3, a physx-debug view. This view enabled viewing the colliders we setup on the backend of physx: vehicles, pedestrians, road-boundaries as walls, etc....
  • Added a visual-flare when hitting/getting-hit by objects with enough force.
  • Tried to fit all objects into a very 'hacked-up' scene graph structure for ease of future feature additions

@copy-pr-bot

copy-pr-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps

greptile-apps Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a PhysX-backed physics system and scene graph to interactive drive, introduces game-mode collisions and debugging views, removes the Vulkan renderer backend, and updates compilation and benchmark infrastructure.

  • Builds and loads pinned PhysX bindings into a platform-local cache.
  • Integrates vehicle, pedestrian, and static-map physics with game-mode effects.
  • Adds top-down BEV and PhysX-debug presentation modes.
  • Updates Inductor cache handling and cross-platform benchmark process cleanup.

Confidence Score: 3/5

The PR does not yet appear safe to merge because two PhysX build-lock races can still block initialization or allow concurrent mutation of the shared native-build cache.

The heartbeat refresh and stale-lock reclamation remain non-atomic, so a waiter can evict a live owner and start a second build in the same CMake and module directories. Separately, a heartbeat followed by holder exit can postpone stale eligibility beyond an already-running waiter’s fixed deadline, causing first-use PhysX initialization to time out with no build left active.

Files Needing Attention: integrations/omnidreams/ludus-renderer/ludus_renderer/_physx_native.py

Important Files Changed

Filename Overview
integrations/omnidreams/ludus-renderer/ludus_renderer/_physx_native.py Adds first-use PhysX download, native compilation, caching, and heartbeat-based build locking; two previously reported lock lifecycle failures remain outstanding.
integrations/omnidreams/ludus-renderer/ludus_renderer/physx.py Exposes the native PhysX simulation API used by interactive drive.
integrations/omnidreams/omnidreams/interactive_drive/simulation/game_physics.py Adds game-mode collision behavior and physics-driven scene updates.
integrations/omnidreams/omnidreams/interactive_drive/runtime/loop.py Integrates physics, scene state, rendering, and presentation into the interactive runtime.
flashdreams/flashdreams/infra/compile.py Moves Inductor artifacts into the FlashDreams cache and patches static Triton bundle collection.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  CLI[Interactive-drive CLI] --> Runtime[Runtime loop]
  Runtime --> Scene[Dynamic scene graph]
  Scene --> Physics[PhysX simulation]
  Physics --> Native[Cached native PhysX module]
  Physics --> Raster[CUDA rasterizer]
  Raster --> Views[Camera / BEV / debug views]
  Physics --> Effects[Game-mode collisions and visual flare]
Loading

Reviews (11): Last reviewed commit: "push test fix" | Re-trigger Greptile

Comment thread integrations/omnidreams/ludus-renderer/ludus_renderer/_physx_native.py Outdated
@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test e425c34

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test c97940a

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test 5210855

Comment on lines +152 to +154
if stale:
lock_path.unlink(missing_ok=True)
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Heartbeat stale-lock eviction race

If an active build's heartbeat touches build.lock after a waiter reads its old mtime, the waiter still unlinks the lock using that stale result. Another process can then acquire the lock and concurrently modify the shared PhysX source, CMake build, and module-output directories, causing corrupted build state or native-module build and loading failures.

Knowledge Base Used: Integrations

# Start the wait timeout only after observing the competing lock.
# A lock created between entering this function and the first open
# attempt must get a full stale interval before we time out.
deadline = time.monotonic() + _BUILD_LOCK_TIMEOUT_SECONDS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Heartbeat extends lock beyond deadline

If a lock holder emits a heartbeat after another process starts waiting and then exits, the refreshed mtime postpones stale eligibility without extending the waiter's fixed deadline. The waiter therefore raises TimeoutError after 1,800 seconds even though no build remains active and the lock becomes reclaimable shortly afterward.

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test 4dbe024

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test 493592d

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

/ok to test

@ArielG-NV, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@ArielG-NV

Copy link
Copy Markdown
Collaborator Author

/ok to test cb82d2c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[interactive-drive] better physics

1 participant