[https://nvbugs/6418103][fix] Clamp the post-allreduce quota by the pre-allreduce quota (`quota = min(quota… by trtllm-agent · Pull Request #15991 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-07-06T18:27:36Z

Summary

Root cause: _get_max_tokens_from_quota and _get_quota_from_max_tokens are not inverses when full_attn_size_per_token == 0 (all layers SWA per test's max_attention_window=[128, 32768]), so the intended cross-rank normalization inflates the KV cache GPU quota ~10× (115.83 → 1156.5 GiB) and cuMemCreate OOMs.
Fix: Clamp the post-allreduce quota by the pre-allreduce quota (quota = min(quota, _get_quota_from_max_tokens(max_tokens))) — allreduce(MIN) must never raise the local quota. Also remove the corresponding waiver from tests/integration/test_lists/waives.txt.
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure
Check for regressions in related tests

Links

Bug: https://nvbugs/6418103

Summary by CodeRabbit

Bug Fixes
- Improved GPU memory quota calculation for cache management to better handle mixed layer configurations and avoid quota over-allocation.
- Added safeguards so available token capacity is capped more conservatively, improving stability in edge cases.

coderabbitai · 2026-07-06T18:32:24Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 23a086b3-2373-4572-872c-cc3b6874ecec

📥 Commits

Reviewing files that changed from the base of the PR and between 011e849 and 87521a8.

📒 Files selected for processing (2)

tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

tests/integration/test_lists/waives.txt

📝 Walkthrough

Walkthrough

This PR modifies the GPU quota calculation in KVCacheManagerV2 to clamp quota using min() against a max-token-derived value, guarding against quota inflation when SWA layers dominate. A related test waiver entry is also removed from waives.txt.

Changes

KV Cache Quota Fix

Layer / File(s)	Summary
Quota clamping and waiver cleanup `tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py`, `tests/integration/test_lists/waives.txt`	Quota is now clamped via `min(quota, _get_quota_from_max_tokens(max_tokens))` instead of direct assignment, with updated comments explaining SWA-related token/quota mismatch; the corresponding test skip entry is removed from the waiver list.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#15959: Both PRs modify tests/integration/test_lists/waives.txt to remove waiver entries for integration tests.
NVIDIA/TensorRT-LLM#15960: Directly relates to the same test_kv_cache_v2_nixl_python[cache_mgr_v2] waiver entry being removed here.

Suggested reviewers: yiqingy0, Superjomn, mzweilz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required ticket/type pattern and clearly summarizes the quota-clamping fix.
Description check	✅ Passed	The description covers the issue, fix, test plan, and bug link, though the PR checklist section is omitted.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

jiaganc

LGTM

jiaganc · 2026-07-07T07:07:25Z

@yizhang-nv Could you please review this PR?

jiaganc · 2026-07-07T07:07:36Z

/bot run

tensorrt-cicd · 2026-07-07T07:13:32Z

PR_Github #57965 [ run ] triggered by Bot. Commit: 87521a8 Link to invocation

tensorrt-cicd · 2026-07-07T12:47:23Z

PR_Github #57965 [ run ] completed with state SUCCESS. Commit: 87521a8
/LLM/main/L0_MergeRequest_PR pipeline #46641 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jiaganc · 2026-07-08T02:44:42Z

/bot run

jiaganc · 2026-07-08T05:33:20Z

/bot run

tensorrt-cicd · 2026-07-08T05:39:56Z

PR_Github #58157 [ run ] triggered by Bot. Commit: 87521a8 Link to invocation

tensorrt-cicd · 2026-07-08T08:40:25Z

PR_Github #58157 [ run ] completed with state SUCCESS. Commit: 87521a8
/LLM/main/L0_MergeRequest_PR pipeline #46808 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

The token<->quota round-trip in KVCacheManagerV2.__init__ is not an identity when SWA layers dominate the layer set (i.e. full_attn_size_per_token == 0). For gpt-oss-120b disagg with max_attention_window=[128, 32768], all layers are treated as SWA and the reverse map inflates size_per_batch by max_batch_size, blowing quota from 115.83 GiB up to 1156.5 GiB. cuMemCreate then fails with CU_ERROR_OUT_OF_MEMORY. allreduce(MIN) must never raise the local quota. Clamp the post-sync value by the pre-sync value to preserve that invariant. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>

jiaganc · 2026-07-08T14:27:31Z

/bot run

tensorrt-cicd · 2026-07-08T14:33:02Z

PR_Github #58246 [ run ] triggered by Bot. Commit: 95dbcc0 Link to invocation

tensorrt-cicd · 2026-07-09T00:55:34Z

PR_Github #58246 [ run ] completed with state SUCCESS. Commit: 95dbcc0
/LLM/main/L0_MergeRequest_PR pipeline #46887 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

jiaganc · 2026-07-09T02:41:34Z

/bot run

jiaganc · 2026-07-09T02:43:14Z

/bot run --extra-stage "DGX_B300-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-09T02:47:52Z

PR_Github #58356 [ run ] triggered by Bot. Commit: 95dbcc0 Link to invocation

tensorrt-cicd · 2026-07-09T02:48:41Z

PR_Github #58357 [ run ] triggered by Bot. Commit: 95dbcc0 Link to invocation

tensorrt-cicd · 2026-07-09T02:48:45Z

PR_Github #58356 [ run ] completed with state ABORTED. Commit: 95dbcc0

Link to invocation

tensorrt-cicd · 2026-07-09T03:32:49Z

PR_Github #58357 [ run ] completed with state SUCCESS. Commit: 95dbcc0
/LLM/main/L0_MergeRequest_PR pipeline #46981 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nvpohanh · 2026-07-09T07:29:31Z

[by Codex] @yizhang-nv Could you review this PR? Thanks!

jiaganc · 2026-07-09T09:30:48Z

Wait for #16186 to waive tests before rerun pipeline.

jiaganc · 2026-07-09T15:40:36Z

/bot run --extra-stage "DGX_B300-4_GPUs-PyTorch-Post-Merge-1"

tensorrt-cicd · 2026-07-09T15:46:28Z

PR_Github #58484 [ run ] triggered by Bot. Commit: 95dbcc0 Link to invocation

tensorrt-cicd · 2026-07-09T19:00:21Z

PR_Github #58484 [ run ] completed with state SUCCESS. Commit: 95dbcc0
/LLM/main/L0_MergeRequest_PR pipeline #47093 completed with status: 'SUCCESS'

CI Report

Link to invocation

The KV Cache Manager V2 quota-inflation OOM on multi-GPU VSWA (variable sliding-window attention) configurations -- a regression from NVIDIA#15633 -- was fixed on main by a parallel effort, NVIDIA#15991 (nvbugs/6418103), which clamps the post-allreduce quota by the pre-allreduce quota so the cross-rank normalization can only reduce, never over-commit. That fix landed without a test guarding the V2 + VSWA path. This change amends the missing coverage. Parametrize TestGPTOSS::test_eagle3_vswa_reuse_4gpus over v2_kv_cache so the previously-uncovered V2 path (GPT-OSS-120B, TP=4, max_attention_window=[128, 32768], free_gpu_memory_fraction=0.4) runs and guards against regressing the fix -- this variant OOM'd before NVIDIA#15991. The v2_kv_cache + two-model combination is skipped (V2 is not compatible with two-model overlap scheduling). Register the new variants in llm_function_core.txt and llm_function_rtx6k.txt. Signed-off-by: Yueh-Ting Chen <yuehtingc@nvidia.com>

trtllm-agent assigned Shixiaowei02 Jul 6, 2026

trtllm-agent requested a review from a team as a code owner July 6, 2026 18:27

github-actions Bot assigned trtllm-agent Jul 6, 2026

jiaganc approved these changes Jul 7, 2026

View reviewed changes

jiaganc assigned jiaganc and unassigned Shixiaowei02 Jul 7, 2026

trtllm-agent force-pushed the repair-bot-bug6418103 branch from 87521a8 to 03be757 Compare July 8, 2026 08:40

trtllm-agent force-pushed the repair-bot-bug6418103 branch from 03be757 to 5ddd0da Compare July 8, 2026 14:20

jiaganc reviewed Jul 8, 2026

View reviewed changes

Comment thread tests/integration/test_lists/waives.txt

trtllm-agent force-pushed the repair-bot-bug6418103 branch from 5ddd0da to 95dbcc0 Compare July 8, 2026 14:25

nvpohanh requested a review from yizhang-nv July 9, 2026 06:29

yizhang-nv approved these changes Jul 9, 2026

View reviewed changes

jiaganc enabled auto-merge (squash) July 9, 2026 15:40

jiaganc merged commit 8e7b98c into NVIDIA:main Jul 9, 2026
8 checks passed

coderabbitai Bot mentioned this pull request Jul 10, 2026

[None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend #16218

Merged

3 tasks

eopXD mentioned this pull request Jul 10, 2026

[None][test] KV cache manager v2: add V2 + VSWA multi-GPU test coverage #16114

Merged

1 task

Uh oh!

Conversation

trtllm-agent commented Jul 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 6, 2026

Walkthrough

Changes

Uh oh!

jiaganc left a comment

Choose a reason for hiding this comment

Uh oh!

jiaganc commented Jul 7, 2026

Uh oh!

jiaganc commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

tensorrt-cicd commented Jul 7, 2026

Uh oh!

jiaganc commented Jul 8, 2026

Uh oh!

jiaganc commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

Uh oh!

jiaganc commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 8, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

jiaganc commented Jul 9, 2026

Uh oh!

jiaganc commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

nvpohanh commented Jul 9, 2026

Uh oh!

jiaganc commented Jul 9, 2026

Uh oh!

jiaganc commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

tensorrt-cicd commented Jul 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

trtllm-agent commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading