[https://nvbugs/6426852][fix] Add explicit `gc.collect()` between `del weights` and… by trtllm-agent · Pull Request #16144 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-08T21:27:43Z

Summary

Root cause: Per-RPC ipc_collect() couldn't reclaim IPC mappings because del weights alone left the mapped storages in unreachable cycles; many partial-update RPCs on MoE model exhausted CUDA driver's IPC handle table.
Fix: Add explicit gc.collect() between del weights and torch.cuda.ipc_collect() in the per-RPC branch so IPC-mapped storages are released in the same iteration, complementing the prior producer-side lazy-replica fix.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6426852

Summary by CodeRabbit

Bug Fixes
- Improved weight update reliability during repeated requests by ensuring GPU memory handles are released more consistently.
- Reduced the chance of failures when updating model weights across multiple GPU operations.
Tests
- Updated integration waivers and related GPU test behavior to reflect the new weight-handling flow.
- Adjusted test helpers to create GPU weight copies more lazily.

coderabbitai · 2026-07-08T21:30:39Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 448e27d6-f7eb-42c7-8225-25bccd60b1cf

📥 Commits

Reviewing files that changed from the base of the PR and between 5ce293f and 9821ab7.

📒 Files selected for processing (3)

tensorrt_llm/llmapi/rlhf_utils.py
tests/integration/test_lists/waives.txt
tests/unittest/_torch/ray_orchestrator/single_gpu/test_llm_update_weights.py

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

Adds an explicit gc.collect() call before torch.cuda.ipc_collect() in the weight update path to release CUDA IPC storages. Removes an associated test waiver and refactors a test helper to lazily replicate model weights per device instead of eagerly copying to all devices.

Changes

CUDA IPC weight update fix

Layer / File(s)	Summary
gc.collect() before IPC sweep `tensorrt_llm/llmapi/rlhf_utils.py`	Adds an unconditional `gc.collect()` call after weight reload and before `torch.cuda.ipc_collect()` to ensure IPC storages are released.
Lazy weight replication and waiver removal `tests/unittest/_torch/ray_orchestrator/single_gpu/test_llm_update_weights.py`, `tests/integration/test_lists/waives.txt`	Test helper now only clones weights for the owning device and lazily populates other devices' weights on demand; removes the obsolete waiver for the multi-gpu update-weights test.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#16093: Both PRs modify the same waives.txt entry for the multi_gpu test_llm_update_weights_multi_gpu.py -m "part1" case.
NVIDIA/TensorRT-LLM#16109: Both PRs adjust CI waiver entries in tests/integration/test_lists/waives.txt.

Suggested reviewers: brb-nv, shuyixiong, mingyangHao

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main fix: adding explicit gc.collect() for CUDA IPC handle cleanup, though it is truncated.
Description check	✅ Passed	The description covers summary, root cause, fix, and test plan, but it omits the template's explicit Description and PR Checklist sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Partial-update loops on MoE models (Qwen3-30B-A3B) issue many per-suffix update_weights RPCs. Each RPC's worker side reconstructs CUDA IPC tensors via rebuild_cuda_tensor (cudaIpcOpenMemHandle) into 'weights', then drops the dict. Without an explicit gc.collect(), the underlying storages (backed by IPC mappings) can linger in unreachable cycles until the next full GC, so the following torch.cuda.ipc_collect() finds nothing to reclaim and the CUDA driver's IPC handle table saturates before the finalize step, producing cudaErrorMapBufferObjectFailed on later RPCs. Add gc.collect() between 'del weights' and 'torch.cuda.ipc_collect()' in WorkerExtension.update_weights so IPC-mapped storages are actually released before the CUDA IPC pool sweep in each iteration. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

allisonlim-nv

LGTM; forces GC before ipc_collect() so IPC weight mappings are released in the same RPC.

trtllm-agent · 2026-08-04T14:39:52Z

NVBug 6426852 is closed as Bug - Fixed. The linked bug appears resolved elsewhere or for a reason that does not prove this PR is redundant. This PR should be judged on its own merits; repair-bot is not auto-closing it.

brnguyen2

The same failure was fixed on main by #16226 (merged 2026-07-20), which took a different approach (resource-cleanup wait in ray_executor.py) and removed the waiver plus updated the tests. This PR predates that and hasn't been rebased. Please confirm the per-RPC gc.collect() is still needed on top of #16226 — with a reproduction on the same MoE config — or close it. If it is kept: a full gc.collect() on every partial-update RPC is not free; note the per-RPC cost, since the finalize branch was deliberately written to collect once.

brnguyen2 · 2026-08-10T21:43:56Z

+                # already gone; force GC so cudaIpcOpenMemHandle storages are
+                # released before the sweep, or the IPC handle table saturates
+                # across repeated RPCs.
+                gc.collect()


del weights isn't the only live reference at this point: the loop above leaves tensor, func, args, list_args, and all_handles bound in the frame, so the last IPC-mapped storage survives this gc.collect()/ipc_collect() pair and is only released when the method returns. If the handle-table pressure is real, drop those too:

Suggested change

gc.collect()

del weights, tensor, list_args, all_handles

gc.collect()

trtllm-agent requested a review from a team as a code owner July 8, 2026 21:27

trtllm-agent assigned shuyixiong Jul 8, 2026

trtllm-agent requested a review from schetlur-nv July 8, 2026 21:27

github-actions Bot assigned trtllm-agent Jul 8, 2026

trtllm-agent force-pushed the repair-bot-bug6426852 branch from 9821ab7 to ff85e5d Compare July 13, 2026 09:39

trtllm-agent force-pushed the repair-bot-bug6426852 branch from ff85e5d to 8a94d16 Compare July 20, 2026 11:49

trtllm-agent requested review from a team as code owners July 20, 2026 11:49

trtllm-agent requested review from ZhanruiSunCh, asfiyab-nvidia, brnguyen2, jieli-matrix and tburt-nv July 20, 2026 11:49

mikeiovine requested review from mikeiovine and removed request for schetlur-nv July 20, 2026 15:35

jieli-matrix approved these changes Jul 21, 2026

View reviewed changes

trtllm-agent force-pushed the repair-bot-bug6426852 branch from 8a94d16 to 701d5eb Compare July 22, 2026 14:01

allisonlim-nv approved these changes Jul 29, 2026

View reviewed changes

brnguyen2 reviewed Aug 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

[https://nvbugs/6426852][fix] Add explicit `gc.collect()` between `del weights` and… - #16144

[https://nvbugs/6426852][fix] Add explicit `gc.collect()` between `del weights` and…#16144
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6426852

trtllm-agent commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 8, 2026

Walkthrough

Changes

Uh oh!

allisonlim-nv left a comment

Uh oh!

trtllm-agent commented Aug 4, 2026

Uh oh!

brnguyen2 left a comment

Uh oh!

brnguyen2 Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	gc.collect()
	del weights, tensor, list_args, all_handles
	gc.collect()

Uh oh!

Conversation

trtllm-agent commented Jul 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 8, 2026

Walkthrough

Changes

Uh oh!

allisonlim-nv left a comment

Choose a reason for hiding this comment

Uh oh!

trtllm-agent commented Aug 4, 2026

Uh oh!

brnguyen2 left a comment

Choose a reason for hiding this comment

Uh oh!

brnguyen2 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

trtllm-agent commented Jul 8, 2026 •

edited by coderabbitai Bot

Loading