[https://nvbugs/6480621][feat] disagg serving lifecycle fixes and orchestrator diagnostics by chienchunhung · Pull Request #16880 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-07-26T17:47:05Z

Note

Superseded draft — do not merge as-is.

This work is mostly covered by newer PRs. Those successor PRs are still open, so the lifecycle gaps are not yet resolved on main.

Scope disposition

Change in this PR	Current owner / action
Stop retrying worker HTTP errors	Mostly #16402. It covers the motivating GEN failure and timeout cases. Decide the remaining CTX 4xx/5xx retry policy there.
Cancel the surviving GEN-first task	#16402. Its cancellation-safe cleanup supersedes this implementation.
Detect client disconnects	#16402. It provides stronger streaming and non-streaming ASGI disconnect handling with bounded cleanup. Port the disconnect metric there if still useful.
Emit orchestrator lifecycle diagnostics	#16872. Port useful serve-layer events into its shared, bounded diagnostics framework rather than keeping a second event schema.
Coordinate CTX and GEN outcomes	#16909. That PR owns cross-side grants, obligations, abort propagation, and terminal-state coordination; this PR only cancels local orchestrator tasks.

Recommendation

Do not merge or cherry-pick this PR wholesale. Close it after the remaining small deltas are either incorporated into the successor PRs or explicitly tracked:

decide whether CTX HTTP 4xx/5xx responses should ever be retried;
add a disconnect counter at the point where an actual ASGI disconnect is observed, if needed;
add orchestrator events through the [https://nvbugs/6480621][feat] add disaggregated lifecycle diagnostics #16872 diagnostics framework.

…hestrator diagnostics Three pull-forward fixes from the Fable disaggregated lifecycle design §13, plus a serve-layer orchestrator lifecycle emitter (WS1 serve layer). **Fix 1 — Stop retrying HTTP 4xx/5xx as network errors** (`openai_client.py`) `aiohttp.ClientResponseError` (raised for any HTTP error response) is a subclass of `aiohttp.ClientError`. Previously all `ClientError` exceptions were retried, which meant a GEN worker returning 503 (over capacity) or 422 (bad request) consumed the full retry budget against the same server instead of surfacing the error immediately. Now `ClientResponseError` is re-raised on the first attempt; only true TCP-level errors continue to be retried. **Fix 2 — Cancel surviving leg in gen-first non-streaming gather** (`openai_disagg_service.py`) In the `_send_disagg_request_gen_first` non-streaming path both legs were launched with `asyncio.gather`. If one failed, the surviving leg kept running to completion, holding a KV-receive session or transfer-block slot until its own timeout. The gather is now wrapped in a try/except that cancels all pending tasks on the first exception. **Fix 3 — Disconnect watcher for non-streaming disagg requests** (`openai_disagg_server.py`) The non-streaming disagg handler had no mechanism to detect client disconnect. `_run_with_disconnect_guard` races the handler coroutine against `_poll_disconnect` (polling `raw_request.is_disconnected()` at 0.5 s intervals) and cancels the handler on disconnect. The cancelled request is converted to an HTTP 499 response; `perf_metrics.DisaggPerfMetricsCollector` gains a `cancelled_requests` counter. Workers are not explicitly cancelled here (that requires WS4 abort RPCs — a later workstream). `tensorrt_llm/serve/disagg_lifecycle.py` adds a lightweight, opt-in per-process lifecycle event stream for the disagg orchestrator, enabled by `TRTLLM_DISAGG_ORCHESTRATOR_DIAGNOSTICS=1`. Events: `ctx_dispatch`, `ctx_complete`, `ctx_error`, `gen_dispatch`, `gen_complete`, `gen_error`, `gen_rejected` (HTTP 4xx/5xx), `client_disconnect`, `abort`. Each record is a JSON line on stdout with `(clock_id, seq, wall_ns, rid, elapsed_ms)` for cross-host correlation. `OrchestratorRequestTracer` (obtained via `DisaggOrchestratorLifecycle.tracer()`) provides paired dispatch/complete helpers that measure leg-level elapsed time from the orchestrator's perspective. Both ctx-first and gen-first flows are instrumented. Correlated with executor-layer events (`TRTLLM_DISAGG_TRANSFER_DIAGNOSTICS`) via `disagg_request_id`. - `test_disagg_openai_client.py`: verifies that `ClientResponseError` (5 status codes: 400, 422, 429, 500, 503) is not retried and that plain `ClientError` still is. - `test_openai_disagg_service.py`: two new tests verify that the surviving task is cancelled when CTX fails and when GEN fails in the gen-first non-streaming gather. - `test_disagg_lifecycle.py` (new): 16 tests covering disabled/enabled emitter, field presence, sequence increment, error truncation, tracer paired events, and `from_environment` with multiple env-var values. Signed-off-by: Chien-Chun Hung <chienchunh@nvidia.com> Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-07-26T18:10:03Z

/bot help

github-actions · 2026-07-26T18:10:10Z

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label. Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the ci: post-merge approved PR label applied by an active member of NVIDIA/trt-llm-ci-approvers. The approval label remains in place when new commits are pushed.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge". The patterns "*", "*Post-Merge*", and "*PerfSanity*", including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the ci: post-merge approved PR label.

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

chienchunhung · 2026-07-26T18:14:59Z

/bot run --extra-stage "DGX_H100-8_GPUs-Disaggregated-1, DGX_H100-8_GPUs-Disaggregated-2, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1"

tensorrt-cicd · 2026-07-26T18:21:22Z

PR_Github #61749 [ run ] triggered by Bot. Commit: 5c9a7ec Link to invocation

tensorrt-cicd · 2026-07-26T18:46:27Z

PR_Github #61749 [ run ] completed with state FAILURE. Commit: 5c9a7ec
/LLM/main/L0_MergeRequest_PR pipeline #49951 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-08-04T22:40:07Z

Most of the work in this PR are split into other PRs.

github-actions Bot assigned chienchunhung Jul 26, 2026

chienchunhung force-pushed the fable-disagg-lifecycle branch from d6134f2 to 5c9a7ec Compare July 26, 2026 18:05

chienchunhung closed this Aug 4, 2026

Navigation Menu

Uh oh!

[https://nvbugs/6480621][feat] disagg serving lifecycle fixes and orchestrator diagnostics - #16880

[https://nvbugs/6480621][feat] disagg serving lifecycle fixes and orchestrator diagnostics#16880
chienchunhung wants to merge 1 commit into
NVIDIA:mainfrom
chienchunhung:fable-disagg-lifecycle

chienchunhung commented Jul 26, 2026 •

edited

Loading

Uh oh!

chienchunhung commented Jul 26, 2026

Uh oh!

github-actions Bot commented Jul 26, 2026

kill

skip

reuse-pipeline

Uh oh!

chienchunhung commented Jul 26, 2026

Uh oh!

tensorrt-cicd commented Jul 26, 2026

Uh oh!

tensorrt-cicd commented Jul 26, 2026

Uh oh!

chienchunhung commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

chienchunhung commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope disposition

Recommendation

Uh oh!

chienchunhung commented Jul 26, 2026

Uh oh!

github-actions Bot commented Jul 26, 2026

GitHub Bot Help

kill

skip

reuse-pipeline

Uh oh!

chienchunhung commented Jul 26, 2026

Uh oh!

tensorrt-cicd commented Jul 26, 2026

Uh oh!

tensorrt-cicd commented Jul 26, 2026

Uh oh!

chienchunhung commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chienchunhung commented Jul 26, 2026 •

edited

Loading