[TRTLLM-13409][feat] hard-kill all ranks when one rank's executor loop crashes - #16592
Conversation
03e5e9e to
cf7fdc1
Compare
|
/bot run --disable-fail-fast |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds configurable rank-crash hard-kill handling with a grace period, optional watchdog, and defensive error handling. Executor crash cleanup now starts the watchdog before cleanup and invokes direct world termination afterward, with tests covering policy and ordering. ChangesRank crash handling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant PyExecutor_event_loop_wrapper
participant RankCrashKillWatchdog
participant executor_loop_cleanup
participant hard_kill_on_rank_crash
PyExecutor_event_loop_wrapper->>RankCrashKillWatchdog: Arm after event-loop crash
PyExecutor_event_loop_wrapper->>executor_loop_cleanup: Execute local cleanup
RankCrashKillWatchdog->>hard_kill_on_rank_crash: Trigger after grace period
PyExecutor_event_loop_wrapper->>hard_kill_on_rank_crash: Invoke after cleanup with original deadline
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
PR_Github #60270 [ run ] triggered by Bot. Commit: |
|
PR_Github #60270 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60547 [ run ] triggered by Bot. Commit: |
|
PR_Github #60547 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60580 [ run ] triggered by Bot. Commit: |
nv-xtf
left a comment
nv-xtf
left a comment
There was a problem hiding this comment.
LGTM — One non-blocking question inline about kill reachability if cleanup blocks on a PP send handle.
|
PR_Github #60580 [ run ] completed with state
|
cf7fdc1 to
ec5a475
Compare
ec5a475 to
c0c1a43
Compare
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
|
PR_Github #64515 [ run ] triggered by Bot. Commit: |
|
PR_Github #64509 [ run ] completed with state |
|
PR_Github #64517 [ run ] triggered by Bot. Commit: |
|
PR_Github #64515 [ run ] completed with state |
|
PR_Github #64517 [ run ] completed with state
|
3cf73d7 to
d2f37e0
Compare
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
1 similar comment
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
|
PR_Github #64580 [ run ] triggered by Bot. Commit: |
|
PR_Github #64581 [ run ] triggered by Bot. Commit: |
|
PR_Github #64580 [ run ] completed with state |
|
PR_Github #64581 [ run ] completed with state
|
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
The core mechanism looks right: the completion sentinel with its AST guard, arming the watchdog before cleanup, and the deadline handover are all carefully done, and the test coverage (including the real 2-rank MPI_Abort tests) is well above the bar. Two things to address before merge, detailed inline:
- The delivery gate is set optimistically on the proxy/IPC path (
[base_worker.py:1163](https://github.com/NVIDIA/TensorRT-LLM/pull/16592/files#diff-0ebb3195d987a59ab750ef38e06a2e519b0cc100ed9965c4223c77e8c3154349R1163)). In the default spawned-worker deployment, a leader-only crash marks itself delivered without the client actually being woken, which disarms the 10s kill and regresses that case back to the 300s HangDetector. TLLM_RANK_CRASH_HARD_KILL_GRACE=nanslips past validation and produces a zero-grace kill ([hang_detector.py:110](https://github.com/NVIDIA/TensorRT-LLM/pull/16592/files#diff-45f89712de7a99902beeffb4dbedd76e66eddf672d06da8dc08bc446e317a4f4R110)).
Separately: this ships a default behavior change (world-kill-on-crash enabled for every multi-rank run) under a [fix] title. The description now flags it, which helps, but the new env var and the new default aren't documented anywhere under docs/. A short note in the docs or release notes would let users find the -1 escape hatch without reading the source.
JunyiXu-nv
changed the title
…p crashes When a rank's executor loop dies on an exception, that rank stops participating in collectives but nothing tells its peers. Every peer blocks in its next collective until its own HangDetector fires 300s later, so the whole multi-GPU session burns that long for an error that was already known. The A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop. A rank whose executor loop crashes now tears the job down (MPI_Abort, falling back to self-SIGKILL) after a grace period. Single-rank worlds are exempt -- there are no peers to strand -- and the kill helper never raises, since it runs in a `finally` where an exception would mask the loop's original error. A cancellable RankCrashKillWatchdog is armed before cleanup so the kill stays reachable when cleanup itself blocks or raises, and the original fire deadline survives handover between waiters. The grace exists so cleaner paths can win the race: if the crash is surfaced to a client first, the caller gets the real traceback and the kill stands down rather than replacing it with a bare exit 137. That gate is set only on verified delivery -- the helper must be on the single-process path AND a readable queue must have taken the error -- because on the proxy/IPC path the broadcast cannot reach the client at all. Setting it there would disarm the kill while the peers were still stranded, which is the case this change exists for, in the default spawned-worker deployment. The stand-down is not guaranteed for symmetric crashes: only rank 0 can set the gate, so subordinate kills still fire at crash+grace and the "N tracebacks" outcome holds only if each subordinate exits within the grace. destroy_process_group() on a wedged NCCL communicator can exceed it. That is deliberate -- the timer doubles as protection against wedged teardown. This enables world-kill-on-crash by default for every multi-rank run. TLLM_RANK_CRASH_HARD_KILL_GRACE tunes the grace; any negative value (e.g. -1) disables the kill entirely and restores the previous hang-detector behavior. Unset, unparsable and non-finite values use the 10s default -- nan in particular would otherwise slip past the `grace < 0` check and collapse to a zero grace, and inf would produce a watchdog that never fires. Both the default change and the escape hatch are documented in docs/source/developer-guide/overview.md. Tests: test_hang_detector_kill.py (l0_sanity_check) covers the exemption, grace timing and ordering, disable/invalid/non-finite env handling, non-raising behavior, watchdog arm/cancel/deadline-handover and the crash-vs-clean-exit wiring, including two real 2-rank MPI_Abort tests; test_event_loop_error_broadcast.py (l0_cpu) pins the delivery gate, including that it stays clear on ipc_batched. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
5b5050c to
9c7cd06
Compare
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
|
PR_Github #64928 [ run ] triggered by Bot. Commit: |
brnguyen2
left a comment
brnguyen2
left a comment
There was a problem hiding this comment.
Approving — the comments below are optional touch-ups, not blockers.
The escalation design holds together well: the _event_loop_completed sentinel is the right crash predicate, the delivery gate consistently fails toward killing, and the deadline handover between the watchdog and the inline kill avoids the double-grace trap. The AST guards on the break sites and the two unmocked 2-rank mpirun tests close the gaps a purely mocked suite would leave.
One remaining behavior worth a look before merge (inline comment on hang_detector.py): the post-cleanup kill path sleeps the full remaining grace uninterruptibly even when the delivery gate is already set, which adds up to the grace to shutdown latency in the symmetric-crash case since shutdown() joins the event-loop thread.
Description nits: the QA test list has drifted from the final code — several test names changed (e.g. test_watchdog_cancel_prevents_the_kill is now test_watchdog_cancel_disarms_this_timer), the delivery-gate tests and the real-MPI tests are missing, and tests/unittest/executor/test_proxy_fast_death.py is listed as modified but is not in the diff. Worth refreshing so the merge-commit description matches what ships.
|
PR_Github #64928 [ run ] completed with state
|
|
/bot run --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
|
/bot help |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
|
/bot run --disable-fail-fast --add-multi-gpu-test --extra-stage "DGX_B200-4_GPUs-PyTorch-3, DGX_H100-4_GPUs-PyTorch-DeepSeek-1" |
|
PR_Github #65054 [ run ] triggered by Bot. Commit: |
|
PR_Github #65054 [ run ] completed with state |
8 participants
Dev Engineer Review
TLLM_RANK_CRASH_HARD_KILL_GRACE), where invalid/unset values default to 10s and negative values disable world hard-kill.RankCrashKillWatchdogand integrated it intoPyExecutor._event_loop_wrapperso the watchdog is armed before executor cleanup and the hard-kill is attempted afterward, preserving the original fire deadline.QA Engineer Review
Test changes (unit)
tests/unittest/_torch/executor/test_hang_detector_kill.pytest_rank_crash_kill_single_rank_is_nooptest_rank_crash_kill_fires_for_multi_ranktest_rank_crash_kill_sleeps_grace_before_killtest_rank_crash_kill_disabled_by_negative_gracetest_rank_crash_kill_invalid_grace_uses_defaulttest_rank_crash_kill_never_raisestest_watchdog_kills_while_caller_blockstest_watchdog_not_armed_for_single_ranktest_watchdog_not_armed_when_disabledtest_watchdog_cancel_prevents_the_killtest_kill_keeps_original_deadline_on_handovertest_event_loop_wrapper_kills_world_on_crashtest_event_loop_wrapper_kills_world_when_cleanup_raisestest_event_loop_wrapper_kills_world_when_watchdog_cannot_armtest_event_loop_wrapper_no_kill_on_clean_exittest_event_loop_wrapper_no_kill_when_loop_raises_after_shutdowntest_event_loop_wrapper_no_kill_when_enclosing_context_manager_raisestests/integration/test_lists/test-db/l0_sanity_check.yml(moduleunittest/_torch/executor/test_hang_detector_kill.py)tests/unittest/executor/test_proxy_fast_death.pytest_pool_session_shutdown_never_blocks_after_releasetests/integration/test_lists/test-db/l0_a10.yml(moduleunittest/executor/test_proxy_fast_death.py)Verdict
test-dbvia the test list entries above).Description
When a rank's executor loop dies on an exception, the rank stops participating in collectives but nothing tells its peers: every peer blocks in its next collective until its own HangDetector fires 300s later, and the whole multi-GPU test session burns that long for an error that was already known (the A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop).
Single-rank worlds are exempt (no peers to unblock), and the kill helper never raises (it runs in a finally where an exception would mask the loop's original error).
Behavior change
This enables world-kill-on-crash by default for every multi-rank run. The escape hatch is
TLLM_RANK_CRASH_HARD_KILL_GRACE=-1(any negative value), which restores the previous behavior of waiting for the hang detector. Unset/unparsable/non-finite values use the 10s default. Both are now documented indocs/source/developer-guide/overview.md.The grace period, and what it does and does not guarantee
The kill waits out a grace period first so that cleaner paths can win the race. If the crash is surfaced to a client during the grace — the caller gets the real traceback — the kill stands down, so the failure is reported rather than replaced by a bare exit 137.
That stand-down is not guaranteed for symmetric crashes, and the description should not be read as promising it. Only rank 0 can set the delivery gate: both delivery sites require a client consumer, and subordinate ranks block in
wait_shutdown()with no response thread. So when every rank raises the same deterministic error, the subordinates' kills still fire at crash+grace, and the "N clean tracebacks" outcome holds only if each subordinate finishes cleanup and exits within the grace.destroy_process_group()on a wedged NCCL communicator can exceed it. This is the intended tradeoff: the timer doubles as protection against wedged teardown.The gate is also set only when a client verifiably woke. On the proxy/IPC path (
ipc_batched, the default when the LLM spawns MPI workers) the broadcast cannot reach the client, so the gate stays clear and the kill proceeds — otherwise a leader-only crash would disarm the kill while the peers were still stranded.Test Coverage
tests/unittest/_torch/executor/test_hang_detector_kill.py(CI:l0_sanity_check.yml) — single-rank exemption, grace timing and ordering, disable/invalid/non-finite env handling, non-raising behavior, watchdog arm/cancel/deadline-handover, and the crash-vs-clean-exit wiring, including two real 2-rankMPI_Aborttests.tests/unittest/executor/test_event_loop_error_broadcast.py(CI:l0_cpu.yml) —TestEventLoopErrorDeliveryGatepins the delivery gate: set on verified single-process delivery, left clear onipc_batched, left clear when there was nobody to wake, and left clear whennotify_many()fails.tests/unittest/executor/test_proxy_fast_death.py(CI:l0_a10.yml) —test_pool_session_shutdown_never_blocks_after_release.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.