fix(agent-runtime): rebuild the shared provider transport after unanswered failures

docs/adr/0271-provider-transport-rebuild.md

@@ -0,0 +1,88 @@
+# ADR 0271: Rebuild the shared provider transport after repeated unanswered failures
+
+- Status: Accepted for implementation
+- Date: 2026-09-16
+- Related: [Error codes](../spec/03-runtime/08-error-codes.md) · [Agent runtime](../spec/03-runtime/02-agent-runtime.md) · [ADR 0212](0212-remove-diagnostic-timing-log-streams.md) · issue #234
+
+## Context
+
+One turn on a long session retried a Codex request ten times over ~112 seconds and
+reported `NETWORK_ERROR: fetch failed` for every attempt; a new turn on the same
+provider, model, and network succeeded immediately. The log record carried
+`phase=stream`, `streamMs=1~2`, and `retryAttempt=10`, and nothing else.
+
+Two readings of that evidence were possible, and the first one was wrong:
+
+- `phase=stream` with a two-millisecond stream looks like a failure *after* the
+  response headers arrived, which would mean the connection worked and the retry
+  hypothesis is empty.
+- The pair is actually the fingerprint of the opposite: pi-agent-core emits
+  `message_start` for a stream that ended **without** ever emitting `start`, so
+  `streamMs` measures the synthetic start/end pair, not a started stream. With
+  `providerWaitMs=112442` covering the whole retry loop and each attempt spending
+  seconds in `phase=request` retries, every one of the ten attempts died before
+  any response.
+
+The consequence of the wrong reading was that the shared transport was never
+touched: `node-proxy.ts` installs one process-wide undici dispatcher (a
+`ProxyAgent`, a SOCKS5 agent, or the plain agent) and `closeActiveDispatchers()`
+ran only when settings changed. A pooled connection that died without the pool
+noticing therefore stayed in use for the whole retry budget, and the first-hand
+`error.cause` — where node and undici keep `ENOTFOUND`, `ECONNRESET`,
+`UND_ERR_SOCKET`, or a TLS code — was already flattened into an `errorMessage`
+string by pi-ai before classification, so the log could only say
+`networkCategory: unknown`.
+
+## Decision
+
+1. Describe the rejected provider fetch where the original Error still exists:
+   the fetch wrapper in `provider-retry.ts` runs `describeNetworkFailure` on the
+   live cause chain and reports the same validated fields a directly classified
+   network error carries (`networkCategory`, `networkCode`, `networkSyscall`,
+   `networkHost`) instead of letting a bare `fetch failed` fall back to
+   `unknown`. Field shapes, bounds, and redaction stay owned by
+   `agent-errors.ts`; no new error code and no new locale string is introduced.
+2. Report the transport route (`networkRoute`: `direct`, `environment-proxy`,
+   `http-proxy`, `socks5-proxy`) with the diagnosis, so a failure at the proxy
+   hop is readable without inferring it from an errno.
+3. Report `phase: request` for a failure that never received a response. The
+   phase describes what actually happened to the attempt, not which message
+   lifecycle surfaced it.
+4. Rebuild the shared transport when one origin fails this way **repeatedly**:
+   after the second consecutive unanswered failure for the same origin inside a
+   turn, once per streak, at most once every 30 seconds process-wide, never for a
+   `dns` failure (name resolution happens before a socket exists). Rebuild means
+   rebuilding the configured route from the settings already applied.
+5. Swap before closing: the replacement dispatcher is installed, then the
+   previous one is closed with `close()` — never `destroy()` — and the captured
+   original-dispatcher reference is repointed if the closed pool was it. In-flight
+   requests finish on the pool they were dispatched on.
+
+## Consequences
+
+- The next occurrence of issue #234 is diagnosable: the log names the layer and
+  the route, and the retry indicator names the errno.
+- A turn can now spend one of its retry attempts on a fresh pool, so a transport
+  that never recovers on its own stops being reused. A rebuild does not change
+  the retry budget, the backoff curve, or the error classification.
+- Other sessions can observe a rebuild: their idle pooled sockets are closed, so
+  their next request pays a new TCP/TLS handshake. That is a latency cost, never
+  a correctness cost, and the 30-second throttle plus the per-origin streak bound
+  it.
+- A transport failure during a `dns` outage triggers no rebuild, which is
+  deliberate: throttling the pool would spend the cost for no possible benefit.
+
+## Alternatives considered
+
+- **Rebuild on every failure.** Rejected: the dispatcher is process-wide, so one
+  session's single transient failure would churn every other session's pool, and
+  a long outage would rebuild once per attempt.
+- **Never rebuild.** Rejected: it leaves a repeated unanswered failure on the
+  same origin with no recovery path, which is exactly the reported behavior.
+- **Per-session dispatchers.** Rejected for this change: it changes the ownership
+  model of the transport and the proxy/bypass composition for every provider
+  request, which is a larger and separate architectural decision.
+- **Retain the raw cause message and its `name`.** Rejected: it would put
+  free-form provider text (URLs, header values, query strings) into logs and the
+  transcript for a small diagnostic gain over the errno, the category, and the
+  route.

docs/adr/README.md

@@ -295,5 +295,6 @@ Each ADR includes:
 | 0266 | [Plugin fs roots follow the calling session](0266-plugin-fs-root-follows-the-calling-session.md) | Accepted (D093) |
 | 0267 | [Plugin labels follow the app language](0267-plugin-labels-follow-the-app-language.md) | Accepted (amends 0160; ADR 0182) |
 | 0270 | [Builtin subagents can be switched off](0270-builtin-subagents-can-be-disabled.md) | Accepted for implementation (amends ADR 0063; ADR 0112) |
+| 0271 | [Rebuild the shared provider transport after repeated unanswered failures](0271-provider-transport-rebuild.md) | Accepted for implementation (issue #234) |
 | 0269 | [Move capability documents between the global and project levels](0269-capability-level-transfer.md) | Accepted for implementation (amends ADR 0112) |
 | 0268 | Remove quotes, annotations, and side chats | Accepted (supersedes message-quotes-and-side-chats / response-annotations / floating-annotation-index) |

docs/spec/03-runtime/01-ipc-protocol.md

@@ -735,8 +735,9 @@ context. Manual compaction never silently falls back.
 Provider `error` events may include bounded diagnostic fields in
 `AppError.details`: `phase` (`request` or `stream`), `providerStatus`,
 `providerCode`, `providerWaitMs`, `streamMs`, `retryAttempt`, and, for a
-network failure, `networkCategory`, `networkCode`, `networkSyscall` and
-`networkHost` plus the request correlation fields `requestMessages`,
+network failure, `networkCategory`, `networkCode`, `networkSyscall`,
+`networkHost` and `networkRoute` plus the request correlation fields
+`requestMessages`,
 `requestBytes` and `compactionGeneration`. These fields are additive and
 redacted; they never carry credentials or an unrestricted provider response,
 and the request fields are counts and byte sizes only. A transient stream

docs/spec/03-runtime/02-agent-runtime.md

@@ -229,13 +229,23 @@ When the retry budget is exhausted, the final assistant error and lifecycle
 `AppError.details` when available: `phase` (`request` or `stream`),
 `providerStatus`, `providerCode`, `providerWaitMs`, `streamMs`,
 `retryAttempt`, the network diagnosis (`networkCategory`, `networkCode`,
-`networkSyscall`, `networkHost`) and the request correlation fields
+`networkSyscall`, `networkHost`, `networkRoute`) and the request correlation
 (`requestMessages`, `requestBytes`, `compactionGeneration`). For a persistent
 429 or non-429 transient failure,
 `retryAttempt` is `10`. Credentials and unrestricted response bodies never
 enter the event or log. The active-turn status shows the remaining backoff and
 the retry budget as `Retrying in 0s · attempt 9/10` in English.
 
+Each retry builds a new request, stream, and `AbortController`; the one piece of
+state a retry shares is the process-wide undici dispatcher. When the same origin
+fails twice in a row without any response and no fresh attempt reaches it, the
+transport is rebuilt once (throttled to one rebuild every 30 seconds, never for
+`dns`) before the next attempt, so the retry does not replay into a pool whose
+connection is already dead. The rebuild installs the replacement before closing
+the previous dispatcher and closes it gracefully, and it reproduces the
+configured route, so another session's in-flight request finishes on the pool it
+started on and a proxy is never silently dropped.
+
 ### 5e. Silent-turn recovery
 
 A turn that ends with no tool call and no visible assistant text is invisible

docs/spec/03-runtime/08-error-codes.md

@@ -309,6 +309,28 @@ omitted when it would repeat `networkCode`. Per-layer codes (`DNS_ERROR`,
 splits the layers without adding user-visible codes and locale strings for
 each of them.
 
+The diagnosis is read from the live cause chain at the fetch boundary, not only
+from the provider message. pi-ai flattens a rejected request into
+`errorMessage`, so by the time classification runs the errno undici keeps in
+`error.cause` is already gone and a bare `fetch failed` can only be reported as
+`networkCategory: unknown`; the fetch wrapper still holds the original Error and
+supplies the same validated fields from it. A captured cause also settles the
+phase: the fault is reported as `phase: request` because no response ever
+arrived, which is what distinguishes it from a stream that ended mid-response.
+`networkRoute` (`direct`, `environment-proxy`, `http-proxy`, `socks5-proxy`)
+names the hop the request was taking, so a failure at the proxy is readable
+without guessing from an errno.
+
+When one origin fails this way repeatedly inside a turn — twice in a row,
+without any response — the provider transport is rebuilt before the next attempt
+instead of replaying into the same undici pool. The rebuild is process-wide and
+deliberately bounded: one rebuild per streak, at most one every 30 seconds, and
+never for a `dns` failure, which a fresh pool cannot change. The replacement is
+installed before the previous dispatcher is closed, and the previous one is
+closed gracefully, so a request another session already dispatched finishes on
+the pool it started on. The route in effect is reproduced, never downgraded to a
+direct connection.
+
 ### Permission timeout
 UI/host timeout emits `PERMISSION_TIMEOUT` internally, tool result presented as denied (`TOOL_DENIED`) to agent.
 
@@ -343,7 +365,8 @@ common credential/header values are redacted before event emission or
 persistence. When available, the details disclosure may also show bounded
 `phase`, `providerStatus`, `providerCode`, `providerWaitMs`, `streamMs`,
 `retryAttempt`, `networkCategory`, `networkCode`, `networkSyscall`,
-`networkHost`, `requestMessages`, `requestBytes`, and `compactionGeneration`
+`networkHost`, `networkRoute`, `requestMessages`, `requestBytes`, and
+`compactionGeneration`
 fields. The request fields are counts and byte sizes only and the compaction
 field is the checkpoint generation counter; none of them carries message
 content. While a transient provider failure retries, the activity indicator's

docs/zh-CN/spec/03-runtime/01-ipc-protocol.md

@@ -617,8 +617,8 @@ type AgentEvent =
 提供程序 `error` 事件可能包括以下中的有限诊断字段：
 `AppError.details`：`phase`（`request` 或 `stream`）、`providerStatus`、
 `providerCode`、`providerWaitMs`、`streamMs`、`retryAttempt`，以及网络故障
-时的 `networkCategory`、`networkCode`、`networkSyscall`、`networkHost` 和请求
-关联字段 `requestMessages`、`requestBytes`、`compactionGeneration`。这些字段
+时的 `networkCategory`、`networkCode`、`networkSyscall`、`networkHost`、
+`networkRoute` 和请求关联字段 `requestMessages`、`requestBytes`、`compactionGeneration`。这些字段
 都是新增且经过编辑的；它们从不携带凭据或不受限制的提供商响应，请求字段
 只有计数与字节大小。瞬时流故障可能会在内部重播
 同一回合，没有终端 `error` 事件或重复的辅助消息。

docs/zh-CN/spec/03-runtime/02-agent-runtime.md

@@ -187,11 +187,17 @@ HTTP 429 处理是一个逻辑回合策略。此路径禁用了 pi-ai 的嵌套
 提供程序故障在可用时于 `AppError.details` 中携带有界诊断：
 `phase`（`request` 或 `stream`）、`providerStatus`、`providerCode`、
 `providerWaitMs`、`streamMs`、`retryAttempt`、网络诊断
-（`networkCategory`、`networkCode`、`networkSyscall`、`networkHost`）以及
-请求关联字段（`requestMessages`、`requestBytes`、`compactionGeneration`）。
+（`networkCategory`、`networkCode`、`networkSyscall`、`networkHost`、
+`networkRoute`）以及请求关联字段（`requestMessages`、`requestBytes`、
+`compactionGeneration`）。
 对于持续的 429，
 `retryAttempt` 为 `5`；对于持续的非 429 瞬时故障，它为 `4`。凭据与不受限制的
-响应正文永远不会进入事件或日志。
+响应正文永远不会进入事件或日志。每次重试都会新建请求、流和 `AbortController`；
+重试唯一共享的状态是进程级 undici dispatcher。当同一来源在一轮内连续两次没有
+任何响应、且新尝试仍无法到达它时，下一次尝试前会重建一次传输（每 30 秒最多一次，
+`dns` 除外），避免重试继续复用连接已死的连接池。重建先安装替换、再优雅关闭旧
+的 dispatcher，并复现已配置的链路，因此其他会话正在进行的请求仍会在原连接池上
+完成，代理也不会被悄悄丢弃。
 
 ### 5e。静默回合恢复
 

docs/zh-CN/spec/03-runtime/08-error-codes.md

@@ -301,6 +301,22 @@ Node sidecar 将提供商 SDK 错误映射到：
 `TLS_ERROR`、`SOCKET_RESET` 等）：分类字段已能区分这些层，而无需为每一层
 增加用户可见的错误码与本地化文案。
 
+诊断来自 fetch 边界处的实时 cause 链，而不只是提供程序消息：pi-ai 会把被
+拒绝的请求摊平成 `errorMessage`，等到分类运行时 undici 存放在 `error.cause`
+里的 errno 已经消失，裸 `fetch failed` 只能被记为 `networkCategory: unknown`；
+而 fetch 包装层仍持有原始 Error，并从它给出同一组经过校验的字段。捕获到的
+cause 同时确定了阶段：没有任何响应到达时故障记为 `phase: request`，这正是它
+与「响应中途断流」的区别。`networkRoute`（`direct`、`environment-proxy`、
+`http-proxy`、`socks5-proxy`）指出请求实际走的链路，代理这一跳失败无需再从
+errno 猜测。
+
+同一来源在一轮内连续两次这样失败（完全没有响应）时，下一次尝试前会重建提供
+程序传输，而不是继续复用同一个 undici 连接池。重建是进程级的、且有明确边界：
+每个连续失败序列只重建一次，每 30 秒最多一次，且 `dns` 永不触发重建（新连接池
+无法改变名字解析结果）。替换在关闭旧 dispatcher 之前安装，旧的 dispatcher 采
+用优雅关闭，因此其他会话已派发的请求仍会在它原本使用的连接池上完成。生效的
+链路会被原样复现，绝不会悄悄降级为直连。
+
 ### 权限超时
 UI/host 超时在内部发出 `PERMISSION_TIMEOUT`，工具结果向代理显示为拒绝 (`TOOL_DENIED`)。
 
@@ -332,7 +348,7 @@ UI/host 超时在内部发出 `PERMISSION_TIMEOUT`，工具结果向代理显示
 公共 credential/header 值在事件发射或持久化之前进行编辑。
 详细信息披露也可能显示有界的 `phase`、`providerStatus`、`providerCode`、
 `providerWaitMs`、`streamMs`、`retryAttempt`、`networkCategory`、
-`networkCode`、`networkSyscall`、`networkHost`、`requestMessages`、
+`networkCode`、`networkSyscall`、`networkHost`、`networkRoute`、`requestMessages`、
 `requestBytes` 和 `compactionGeneration` 字段。请求字段只有计数与字节大小，
 压缩字段是检查点世代计数器，均不携带消息内容。当瞬时提供商故障正在重试时，
 活动指示器的原因气泡会显示本地化摘要、稳定错误码，并在网络故障时显示传输层

packages/agent-runtime/src/agent-errors.ts

@@ -315,6 +315,26 @@ export function describeNetworkFailure(
   };
 }
 
+/**
+ * Network diagnosis for a caller that still holds the rejected request's Error.
+ *
+ * `classifyAgentError` normally sees pi-ai's flattened `errorMessage`, so an
+ * errno that only lives in `error.cause` is invisible to it and `fetch failed`
+ * collapses to `networkCategory: "unknown"` (issue #234). The fetch wrapper in
+ * `provider-retry.ts` is the last layer that still sees the original object, so
+ * it describes the cause chain there and hands these validated fields to the
+ * runtime. The fields come from `describeNetworkFailure`, so a captured cause is
+ * exactly as narrow as a classified one: an errno-shaped code, a lowercase
+ * syscall, and a bare hostname — never a URL, port, path, query, or credential.
+ */
+export function networkFailureDiagnostics(
+  err: unknown,
+  message: string,
+): { category: NetworkFailureCategory; fields: Record<string, unknown> } {
+  const network = describeNetworkFailure(err, message);
+  return { category: network.category, fields: networkDetailFields(network) };
+}
+
 export function classifyAgentError(err: unknown): ClassifiedAgentError {
   const rawMessage =
     typeof err === "string"