Skip to content

Fail Turns when the model host is unreachable - #704

Merged
SaladDay merged 2 commits into
mainfrom
fail-turns-when-the-model-host-is-unreachable
Oct 14, 2026
Merged

SaladDay merged 2 commits into
mainfrom
fail-turns-when-the-model-host-is-unreachable

Conversation

@SaladDay

@SaladDay SaladDay commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

What

A Turn whose model host can't be resolved or reached now fails in bounded time with a classified reason. Before this change it could stay in_progress forever.

  • Codex adapter. Disables Codex 0.153.4's unbounded_connection_retries feature (Stable, on by default), so Codex falls back to its own bounded provider defaults. It also deletes the RequestMaxRetries/StreamMaxRetries provider fields. They had no producer, and only a test set them.
  • Claude bridge. Claude Code 2.1.269 reports two different failures as server_error with a null api_error_status: a request that got no HTTP response (DNS failure, refused connection, timeout) and a server error in a response that was already streaming. Only the first comes right after an api_retry whose error_status is null, with no model output in between. The bridge classifies server_error as connection_failed only on that structured signal; every other server_error, mid-response included, stays server_error. No error text is matched.
  • Web. The connection_failed next step (en, zh-CN) still starts with checking the provider's Base URL. Only when the failure has no HTTP status does it add: check that the URL's host resolves and is reachable from the Session's sandbox, or from the host of a self-hosted Session. A failure with an HTTP status already reached a server, so it keeps the Base URL step alone.
  • Docs.
    • docs/getting-started/nodes.md (EN and zh): replaces "needs a policy change on the node", which promised a setting that doesn't exist, with what is true:
      • sandbox DNS queries go to the nameservers in the host's /etc/resolv.conf; microsandbox never reads /etc/hosts;
      • DNS rebind protection drops answers that hold a private address;
      • installer-added nodes have no setting for this.
        It also lists the working options: make those nameservers return public addresses (an /etc/hosts entry helps only when /etc/resolv.conf points at a local stub that serves it, such as systemd-resolved at 127.0.0.53, as on mx2), use a Docker node, or register the node manually with an address-only network rule.
    • Harness onboarding (EN and zh) gets the shared rule: bound native retries, classify the final connection error, and add no Core timeout.
    • The Codex, Claude and MiniMax adapter guides each state their bound.

Why

On mx2, the VPC resolver returned private addresses for api.minimaxi.com. microsandbox's rebind protection turned those answers into NXDOMAIN. Codex then retried about every 60 s forever, and the Turn showed no events and no diagnostics.

Per-Harness behavior (pinned versions, unresolvable host, measured locally)

Harness Native retry policy Bound before After this PR
Codex 0.153.4 With unbounded_connection_retries on: 5 s doubling to a 60 s cap, forever ("Reconnecting... waiting for network"). With it off: request_max_retries 4 and stream_max_retries 5, with stream idle timeout 300 s unchanged none (>3 min, no events) connection_failed within about a minute (25.8 s here, 46.5 s in review), depending on lookup and connect time
Claude Code 2.1.269 10 retries, api_retry with error_status: null, backoff to about 35 s 177 s, classified server_error about 3 min, connection_failed; a mid-response server error stays server_error
MiniMax Code 0.4.12 5 retries, 1 s base, 30 s max delay, 120 s elapsed cap; retries stay internal 116 s, unclassified unchanged: errors reach ACP only as prose JSON-RPC -32603, so it stays unclassified

Why these bounds are safe:

  • Codex's bounded values are its own maintained defaults.
  • Retries start only after a request has failed, so slow or long responses are unaffected.
  • The stream idle timeout keeps its native 300 s.
  • No new setting, field or Core timeout is added.

Residual: Codex has no connect timeout. Against a host that silently drops SYNs, each of the 30 attempts waits for the OS connect timeout. microsandbox resets denied traffic and returns NXDOMAIN for rebinding answers, so the case that motivated this PR fails fast.

Not in this PR (proposals)

  • Retry progress ("Reconnecting to the model provider"). There is no typed path for it today:

    • proto.TypeError is terminal;
    • Turn diagnostics have no progress field;
    • Codex retry notifications are only logged.

    Adding it needs a non-terminal Runtime envelope (attempt, max, next retry time), fed by Codex error notifications with willRetry and by Claude api_retry. Core would keep the latest value per running Turn and expose it in Turn diagnostics. MiniMax emits no ACP signal.

  • MiniMax classification. Needs structured errors from MiniMax Code upstream.

Verification

  • go test ./apps/daemon/internal/agent/codex (the whole package; it includes the new bound assertion and the existing httpConnectionFailed classification tests).
  • go vet ./...
  • Opt-in native test TestExecutorNativeUnreachableModelHost with the real Codex 0.153.4 (OAC_TEST_CODEX_BINARY), base URL https://model-host.invalid/v1:
    • with the fix: connection_failed after 25.8 s;
    • with the feature left on: no event within the 3-minute limit.
  • Claude bridge:
    • pnpm --dir packages/claude-sdk-adapter build;
    • node --test on native_failure, messages, command_observer and classified_execution (26 pass).
    • The real SDK 0.3.269 stream, fed through the built NativeFailure (probe not committed; logs in the lane evidence directory):
      • unresolvable host: 10 api_retry with error_status null, connection_failed at 187 s;
      • refused connection (127.0.0.1:1): same retries, connection_failed at 176 s;
      • silent server with API_TIMEOUT_MS=2000: same retries, connection_failed at 201 s;
      • local SSE server that streams text and then an error event: no retry, server_error at 0.5 s.
  • MiniMax: a temporary probe with the real 0.4.12 CLI, not committed, settled after 116 s with an unclassified error.
  • Web: pnpm --filter @oac/web typecheck, plus vitest run on session-diagnostics, provider-observations and i18n tests (19 pass).
  • Docs: make check-docs and pnpm --dir website check-translations.

Skipped:

  • full make check;
  • full Claude adapter suite, claudesdk Go tests (no Go change there) and the website build;
  • Web screenshots (copy-only change to an existing next-step string, no layout change).

Residual (Claude): api_retry carries no parent_tool_use_id. A subagent's connection retry directly followed by a root server_error without a status, with no stream event or assistant message between them, would read as connection_failed. Any model output resets the signal.

@SaladDay
SaladDay force-pushed the fail-turns-when-the-model-host-is-unreachable branch from 04f1461 to 0902c94 Compare October 14, 2026 10:23
@SaladDay
SaladDay merged commit 283f14e into main Oct 14, 2026
24 checks passed
@SaladDay
SaladDay deleted the fail-turns-when-the-model-host-is-unreachable branch October 14, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant