[https://nvbugs/6248648][fix] Wrap warmup+capture in try/except to graph.reset() the orphan then re-raise… - #14914
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughCUDA graph capture now restores temporary state and resets partial graphs on failure. Cleanup uses guarded resets that log errors and continue. Graph metadata is committed only after successful capture. New tests cover decoder, encoder, and piecewise runner cleanup paths. ChangesCUDA Graph Error Resilience
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant CUDAGraphRunner
participant PiecewiseRunner
participant CUDA Graph
CUDAGraphRunner->>CUDA Graph: capture decoder or encoder execution
PiecewiseRunner->>CUDA Graph: capture piecewise execution
CUDA Graph-->>CUDAGraphRunner: return capture result
CUDA Graph-->>PiecewiseRunner: return capture result
CUDAGraphRunner->>CUDA Graph: reset partial graph on failure
PiecewiseRunner->>CUDA Graph: reset partial graph on failure
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py (1)
565-573: ⚡ Quick winLog exception details for debugging.
While the broad
Exceptioncatch is justified here for defensive teardown (and the comment explains this well), losing the exception details makes debugging harder when resets do fail. Include the exception in the log.♻️ Suggested improvement
`@staticmethod` def _safe_reset_graph(graph: torch.cuda.CUDAGraph, context: str): # graph.reset() can raise (e.g. a stale CUDA generator state inside # ~CUDAGraph()); swallow it so one failing reset cannot abort the rest # of the teardown or mask an earlier, more relevant error. try: graph.reset() - except Exception: - logger.warning("Failed to reset CUDA graph %s.", context) + except Exception as e: + logger.warning("Failed to reset CUDA graph %s: %s", context, e)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py` around lines 565 - 573, The _safe_reset_graph function swallows exceptions without details; modify the except block in _safe_reset_graph(graph: torch.cuda.CUDAGraph, context: str) to capture the exception (e.g., except Exception as exc:) and include the exception information in the warning log (for example by passing exc or exc_info to logger.warning or including exc in the message) so failures to reset the CUDA graph log the exception details for debugging.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py`:
- Around line 565-573: The _safe_reset_graph function swallows exceptions
without details; modify the except block in _safe_reset_graph(graph:
torch.cuda.CUDAGraph, context: str) to capture the exception (e.g., except
Exception as exc:) and include the exception information in the warning log (for
example by passing exc or exc_info to logger.warning or including exc in the
message) so failures to reset the CUDA graph log the exception details for
debugging.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c461e243-7204-4aa9-adcb-1602ba1fc8bb
📒 Files selected for processing (1)
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
| try: | ||
| graph.reset() | ||
| except Exception: | ||
| logger.warning("Failed to reset CUDA graph %s.", context) |
There was a problem hiding this comment.
Exact exception is not logged
| for _ in range(self.WARMUP_STEPS): | ||
| _setup_spec_decoding_and_forward(key, forward_fn, | ||
| capture_inputs) | ||
| if postprocess_fn is not None: |
There was a problem hiding this comment.
postprocess_fn is pulled inside the loop, is that intended?
There was a problem hiding this comment.
Yes, this is intentional and predates this PR. It was added in a49cfb3e68f for NVBug 5516666 so speculative-decoding input state is postprocessed / restored after each warmup forward, before the next iteration. This PR preserves that behavior while ensuring it also runs on failure and outside the graph-capture context.
b99805e to
0c47d85
Compare
0c47d85 to
a05aafa
Compare
a05aafa to
a8b1071
Compare
a8b1071 to
d327421
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py (1)
861-869: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse a narrower exception contract for
_safe_reset_graph.Line 868 catches
Exception. This hides unrelated programming errors during teardown and triggers Ruff BLE001. Catch the specific exception type or types raised bytorch.cuda.CUDAGraph.reset()for the supported PyTorch versions. If a broad catch is required for destructor safety, document that invariant. Add-> Noneto_safe_reset_graph.PyTorch documents
CUDAGraph.reset()as deleting the graph held by the instance. (docs.pytorch.org) Verify the repository’s supported PyTorch versions before selecting the exception type.As per coding guidelines,
**/*.py: “Catch specific exceptions instead of using broad or bareexcept:handlers” and “Annotate every function.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py` around lines 861 - 869, Update the static method _safe_reset_graph to declare -> None and replace the broad Exception handler with the specific exception type(s) that CUDAGraph.reset() raises across the repository’s supported PyTorch versions; if destructor safety requires retaining a broad catch, document that invariant directly in the method. Keep the existing warning and teardown behavior unchanged.Sources: Coding guidelines, MCP tools, Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py`:
- Around line 535-551: Make CUDA graph creation transactional across warmup and
capture: move the warmup loop and capture/postprocess operations into one
exception handler, and on any failure safely reset the local graph, remove the
pending self.graph_metadata[key], and restore saved_kv_lens_cuda without masking
the original exception. Defer publishing graph metadata until self.graphs[key]
has been successfully committed, and add regression coverage for both warmup and
capture failures.
---
Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py`:
- Around line 861-869: Update the static method _safe_reset_graph to declare ->
None and replace the broad Exception handler with the specific exception type(s)
that CUDAGraph.reset() raises across the repository’s supported PyTorch
versions; if destructor safety requires retaining a broad catch, document that
invariant directly in the method. Keep the existing warning and teardown
behavior unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 96604267-3204-43b4-826b-834128f09181
📒 Files selected for processing (4)
tensorrt_llm/_torch/auto_deploy/compile/backends/torch_cudagraph.pytensorrt_llm/_torch/auto_deploy/compile/piecewise_runner.pytensorrt_llm/_torch/auto_deploy/shim/ad_executor.pytensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
d327421 to
63b8c8f
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py (1)
531-547: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftMake capture setup transactional.
The
trystarts after warmup. A failure in Lines 519-525 leavesself.graph_metadata[key]published and can leavesaved_kv_lens_cudaunrestored. A later lookup can return metadata for a graph that was never committed.Wrap warmup, capture, and postprocessing in one failure path. Restore capture-scoped state without replacing the original exception. Remove or defer
graph_metadata[key]untilself.graphs[key]andself.graph_outputs[key]are committed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py` around lines 531 - 547, Make the capture flow transactional by extending the existing try/except around graph creation to include warmup and setup operations before the current try boundary, including the code that publishes graph_metadata[key]. On any failure, restore saved_kv_lens_cuda and other capture-scoped state without masking the original exception, safely reset the partially captured graph, and avoid publishing graph_metadata[key] until self.graphs[key] and self.graph_outputs[key] are committed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/auto_deploy/compile/backends/torch_cudagraph.py`:
- Around line 71-74: Replace broad Exception handlers with RuntimeError handlers
in the CUDA graph reset cleanup blocks at
tensorrt_llm/_torch/auto_deploy/compile/backends/torch_cudagraph.py:71-74,
tensorrt_llm/_torch/auto_deploy/compile/piecewise_runner.py:418-425,
tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py:557-560, and
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py:861-864; preserve the
existing warning behavior for reset failures.
---
Duplicate comments:
In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py`:
- Around line 531-547: Make the capture flow transactional by extending the
existing try/except around graph creation to include warmup and setup operations
before the current try boundary, including the code that publishes
graph_metadata[key]. On any failure, restore saved_kv_lens_cuda and other
capture-scoped state without masking the original exception, safely reset the
partially captured graph, and avoid publishing graph_metadata[key] until
self.graphs[key] and self.graph_outputs[key] are committed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3e800bf3-df0f-434e-bee0-57bfb9beccd9
📒 Files selected for processing (4)
tensorrt_llm/_torch/auto_deploy/compile/backends/torch_cudagraph.pytensorrt_llm/_torch/auto_deploy/compile/piecewise_runner.pytensorrt_llm/_torch/auto_deploy/shim/ad_executor.pytensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py
|
AutoDeploy is deprecated and being removed in September. If this change isn't needed for correctness / compatabilty with the non-AD change then probably better to skip it. Otherwise I can take a final pass and approve it when the usage in main TRT-LLM is approved. |
| # does not abort cleanup of the remaining graphs. | ||
| for graph in self.graphs.values(): | ||
| graph.reset() | ||
| self._safe_reset_graph(graph, "during cleanup") |
There was a problem hiding this comment.
EncoderCUDAGraphRunner in this same file has the identical unguarded pattern — capture() at ~1223 and clear() at ~1316 — and the TypeError raised at line 1239 after a successful capture orphans the graph deterministically. PiecewiseRunner in _torch/compilation/piecewise_optimizer.py is the same (capture ~233, clear_cuda_graphs ~186). Given AutoDeploy is being removed in September, could the AD changes be dropped in favor of covering these two mainline paths instead?
There was a problem hiding this comment.
Agreed. I dropped the AD changes and moved the coverage to the active pytorch paths.
EncoderCUDAGraphRunner now resets any graph that fails before commit (including the post-capture nested-output TypeError) and uses best-effort cleanup.
PiecewiseRunner now does the same for capture and clear. I added focused regression tests for both paths.
| """ | ||
| try: | ||
| graph.reset() | ||
| except Exception: |
There was a problem hiding this comment.
Following up on @achartier's note on the other site: the swallowed exception is dropped at all four sites. Please bind it (except Exception as e: … logger.warning("... %s", e), or exc_info=True) — losing the reset error reintroduces exactly the "teardown failure masks the real problem" behavior this PR is fixing.
There was a problem hiding this comment.
Agreed. The shared reset helper now logs the caught exception (Failed to reset CUDA graph ...: <error>) while preserving the original failure/continuing teardown.
| # registered to the state"), masking the real capture-time | ||
| # error. | ||
| try: | ||
| graph.reset() |
There was a problem hiding this comment.
This reimplements the _safe_reset_cuda_graph() helper already added in backends/torch_cudagraph.py (same package), with the explanatory comment duplicated near-verbatim. ad_executor.py is a third copy. Please import and reuse the single helper.
There was a problem hiding this comment.
The AD changes (and their duplicate helpers) were removed. The retained pytorch paths share one internal _safe_reset_cuda_graph helper.
| key, forward_fn, capture_inputs) | ||
| if postprocess_fn is not None: | ||
| postprocess_fn(capture_inputs) | ||
| _restore_spec_decode_capture_state(attn_metadata, |
There was a problem hiding this comment.
The try also covers postprocess_fn and _restore_spec_decode_capture_state, which run outside the graph context. If either raises the capture actually succeeded, so the "after a capture failure" comment is misleading — and _restore_spec_decode_capture_state is then skipped, leaving attn_metadata in its capture state. Either narrow the try to the with block, or move the restore into a finally.
There was a problem hiding this comment.
Agreed. State restoration now runs after every forward via a scoped context manager, outside torch.cuda.graph. Postprocessing is also outside the graph context. Any failure before graph commit resets the local graph, and a restoration failure no longer masks an active forward / postprocess exception.
| """Releases all captured graphs and the associated memory pool.""" | ||
| # Reset each graph independently so a failure tearing down one graph | ||
| # does not abort cleanup of the remaining graphs. | ||
| for graph in self.graphs.values(): |
There was a problem hiding this comment.
The clear() half is unit-testable without a GPU: inject a fake graph whose reset() raises into self.graphs, then assert the remaining graphs still get reset and all the dicts end up cleared. Worth adding — this resilience property will otherwise silently regress.
| -padding_size] | ||
|
|
||
| @staticmethod | ||
| def _safe_reset_graph(graph: torch.cuda.CUDAGraph, context: str): |
There was a problem hiding this comment.
Missing -> None, unlike its twin _safe_reset_cuda_graph added in torch_cudagraph.py.
63b8c8f to
77351d1
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py (2)
516-530: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDetect the in-flight exception from the
yield, not fromsys.exc_info().
sys.exc_info()returns the innermost exception that is being handled in the current thread. If a caller invokescapture()from inside anexceptblock,active_erroris notNoneeven when the forward pass succeeded. A real restore failure is then downgraded to a warning and hidden. Catch the exception around theyieldinstead. That makes the decision local and deterministic.♻️ Proposed refactor
`@contextlib.contextmanager` def _restore_state_after_forward() -> Iterator[None]: + active_error = None try: yield - finally: - active_error = sys.exc_info()[1] + except BaseException as forward_error: + active_error = forward_error + raise + finally: try: _restore_spec_decode_capture_state(attn_metadata, saved_kv_lens_cuda) except RuntimeError as restore_error: if active_error is None: raise logger.warning( "Failed to restore speculative-decoding state after " "CUDA graph forward failure: %s", restore_error)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py` around lines 516 - 530, Update _restore_state_after_forward to track whether the context-managed yield itself raises by catching the yield exception locally, rather than deriving active_error from sys.exc_info(). Preserve re-raising restore failures when forward succeeds, while logging restore failures only when the forward pass is already failing.
1-3: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the copyright header in this directory.
Files under
tensorrt_llm/_torch/pyexecutor/intentionally omit the NVIDIA copyright header. Drop lines 1-2 to keep this directory consistent.Based on learnings: "Do not add NVIDIA copyright headers to Python files under tensorrt_llm/_torch/pyexecutor/".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py` around lines 1 - 3, Remove the SPDX copyright and license header lines from cuda_graph_runner.py so Python files under tensorrt_llm/_torch/pyexecutor/ consistently omit NVIDIA copyright headers.Source: Learnings
tests/unittest/_torch/executor/test_cuda_graph_cleanup.py (1)
156-178: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAssert the encoder and decoder capture paths also leave
memory_pooluntouched.The test confirms
graphs,graph_outputs, andgraph_metadatastay empty.self.memory_poolis the fourth field committed at Line 569 incuda_graph_runner.py. Add an assertion for it so a future regression that assigns the pool before the commit is caught.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/executor/test_cuda_graph_cleanup.py` around lines 156 - 178, Extend test_decoder_capture_failure_resets_graph_without_publishing_metadata to assert runner.memory_pool remains unchanged after capture fails, alongside the existing graphs, graph_outputs, and graph_metadata assertions. Add equivalent memory_pool assertions to the encoder and decoder capture failure tests so both capture paths verify no pool is published.tests/unittest/_torch/compilation/test_piecewise_cuda_graph_cleanup.py (1)
73-96: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for
extra_attrs["global_stream"]after a capture failure.The test patches
get_model_extra_attrsto return{}and never inspects it. The failure path inpiecewise_optimizer.pyskips theglobal_streamrestore, and this test cannot detect that. Assert the dictionary contents after the failure so the restore behavior is pinned.💚 Proposed test change
+ extra_attrs = {"global_stream": object()} + eager_stream = object() with ( piecewise_cuda_graph(True), capture_piecewise_cuda_graph(True), patch("torch.cuda.CUDAGraph", return_value=graph), patch("torch.cuda.graph", return_value=nullcontext()), - patch("torch.cuda.current_stream", return_value=object()), + patch("torch.cuda.current_stream", return_value=eager_stream), patch( "tensorrt_llm._torch.compilation.piecewise_optimizer.get_model_extra_attrs", - return_value={}, + return_value=extra_attrs, ), pytest.raises(ValueError, match="capture failed"), ): runner() assert graph.reset_count == 1 assert runner.entries[1].cuda_graph is None + assert extra_attrs["global_stream"] is eager_stream🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/compilation/test_piecewise_cuda_graph_cleanup.py` around lines 73 - 96, Update test_capture_failure_resets_graph_before_entry_commit to retain the patched get_model_extra_attrs dictionary, then assert that extra_attrs["global_stream"] is restored after runner() raises ValueError. Keep the existing graph reset and cuda_graph assertions unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/_torch/compilation/piecewise_optimizer.py`:
- Around line 240-262: Update the graph-capture exception handling around
`entry.callable(*args)` so `extra_attrs["global_stream"]` is restored in a
`finally` block, regardless of whether capture succeeds or raises. Keep
`_safe_reset_cuda_graph` and exception propagation in the existing `except`
path, while ensuring the success path still proceeds with the restored stream.
In `@tensorrt_llm/_torch/utils.py`:
- Around line 391-396: Preformat warning messages because tensorrt_llm.logger
joins arguments instead of applying printf-style interpolation: update
_safe_reset_cuda_graph in tensorrt_llm/_torch/utils.py (391-396) and the warning
call in tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py (528-530) to pass a
single f-string argument; update
tests/unittest/_torch/executor/test_cuda_graph_cleanup.py (124-124) to assert
the rendered message via warning.call_args.args[0] rather than the placeholder
argument.
---
Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/cuda_graph_runner.py`:
- Around line 516-530: Update _restore_state_after_forward to track whether the
context-managed yield itself raises by catching the yield exception locally,
rather than deriving active_error from sys.exc_info(). Preserve re-raising
restore failures when forward succeeds, while logging restore failures only when
the forward pass is already failing.
- Around line 1-3: Remove the SPDX copyright and license header lines from
cuda_graph_runner.py so Python files under tensorrt_llm/_torch/pyexecutor/
consistently omit NVIDIA copyright headers.
In `@tests/unittest/_torch/compilation/test_piecewise_cuda_graph_cleanup.py`:
- Around line 73-96: Update
test_capture_failure_resets_graph_before_entry_commit to retain the patched
get_model_extra_attrs dictionary, then assert that extra_attrs["global_stream"]
is restored after runner() raises ValueError. Keep the existing graph reset and
cuda_graph assertions unchanged.
In `@tests/unittest/_torch/executor/test_cuda_graph_cleanup.py`:
- Around line 156-178: Extend
test_decoder_capture_failure_resets_graph_without_publishing_metadata to assert
runner.memory_pool remains unchanged after capture fails, alongside the existing
graphs, graph_outputs, and graph_metadata assertions. Add equivalent memory_pool
assertions to the encoder and decoder capture failure tests so both capture
paths verify no pool is published.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 32e4a977-5df4-495a-93d6-6a3bb4041bbe
📒 Files selected for processing (5)
tensorrt_llm/_torch/compilation/piecewise_optimizer.pytensorrt_llm/_torch/pyexecutor/cuda_graph_runner.pytensorrt_llm/_torch/utils.pytests/unittest/_torch/compilation/test_piecewise_cuda_graph_cleanup.pytests/unittest/_torch/executor/test_cuda_graph_cleanup.py
77351d1 to
25de6d6
Compare
…uctor abort Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
4f292ed to
43af684
Compare
|
/bot run |
|
PR_Github #64826 [ run ] triggered by Bot. Commit: |
Signed-off-by: William Zhang <williamz@nvidia.com>
43af684 to
5aadd73
Compare
|
/bot run |
|
PR_Github #64826 [ run ] completed with state
|
7 participants
Summary
Test plan
Links
Dev Engineer Review
RuntimeError._safe_reset_cuda_graphlogs reset failures instead of propagating them.clearandclear_cuda_graphsnow returnNone.QA Engineer Review
Added test functions:
test_clear_continues_after_reset_failuretest_capture_failure_resets_graph_before_entry_committest_decoder_clear_continues_after_reset_failuretest_decoder_warmup_failure_restores_state_without_publishing_metadatatest_decoder_capture_failure_resets_graph_without_publishing_metadatatest_encoder_rejects_nested_output_without_orphaning_graphThese are test-code changes outside
tests/integration/test_lists/. Notest-db/orqa/coverage entries were changed. Coverage status is unavailable. Verdict: needs follow-up.