fix(codex): keep shared network failures account-neutral - #922
Conversation
📝 WalkthroughWalkthroughChangesThe PR adds bounded classification for account-neutral DNS and network reachability failures. Compact and regular Codex Responses flows preserve account health and affinity, release owned probe leases, and return Account-neutral network failure handling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Codex client
participant Responses flow
participant Account pool
participant Probe lease
Codex client->>Responses flow: Send request
Responses flow->>Account pool: Classify transport failure
Account pool-->>Responses flow: Preserve health and affinity
Responses flow->>Probe lease: Release owned lease
Responses flow-->>Codex client: Return 502
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 888c9558b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | `accountPoolStrategy?` | `"quota" \| "round-robin" \| "fill-first"` | `"quota"` | Assignment strategy for new/unbound Codex requests. A request is unbound when it has no live (parent thread id, quota scope) affinity; a visible existing task can become unbound after proxy restart or affinity reset. `quota` picks the lowest-usage eligible account when no active account exists, keeps an eligible active account below `autoSwitchThreshold`, and after the threshold may move an unbound request or proactively rebind a bound task to a lower-usage eligible account. `round-robin` distributes unbound requests evenly; `fill-first` keeps assigning unbound requests to the active account until cooldown, unavailability, or the configured drain threshold. | | ||
| | `accountPoolStickyLimit?` | `number` | `1` | New/unbound task assignments retained on one round-robin selection before advancing; the counter advances when a task is bound, not after an upstream success. Range 1–100. | | ||
| | `upstreamFailoverThreshold?` | `number` | `3` | Consecutive transient failures before future new sessions fail over. Set `0` to disable. | | ||
| | `upstreamFailoverThreshold?` | `number` | `3` | Consecutive account-scoped transient outcomes before future new sessions fail over. Host-wide DNS and network reachability failures are account-neutral and do not count. Set `0` to disable. | |
There was a problem hiding this comment.
When a routed request uses a pool-backed OpenAI web-search or vision sidecar, a DNS/reachability exception still reaches recordOutcome as connect_error in src/web-search/executor.ts:96-98 and src/vision/describe.ts:116-118; the callback in src/providers/openai-sidecar.ts:100-108 then increments that account's failure streak and can rotate it. Thus the documented guarantee is false for sidecar-enabled requests. Apply the new classifier and probe-lease handling to these sidecar catches as well, or narrow this claim.
AGENTS.md reference: docs-site/AGENTS.md:L7-L10
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/configuration/providers.md`:
- Line 25: Update the upstreamFailoverThreshold documentation to explicitly
describe account-neutral failures as runtime-classified exact pre-connect DNS
and reachability errors within a bounded cause chain, and state that timeouts,
aborts, resets, socket closures, HTTP failures, and semantic upstream failures
remain account-scoped. Apply equivalent wording in
docs-site/src/content/docs/reference/configuration/providers.md lines 25-25,
docs-site/src/content/docs/ja/reference/configuration/providers.md lines 24-24,
docs-site/src/content/docs/ko/reference/configuration/providers.md lines 24-24,
docs-site/src/content/docs/ru/reference/configuration/providers.md lines 25-25,
and docs-site/src/content/docs/zh-cn/reference/configuration/providers.md lines
24-24, translating the qualifiers appropriately for each locale.
In `@src/lib/upstream-retry.ts`:
- Around line 32-53: Update the ACCOUNT_NEUTRAL_NETWORK_ERROR_CODES and
ACCOUNT_SCOPED_CONNECTION_ERROR_CODES sets to match Bun 1.3.14: retain ENOTFOUND
and the existing supported codes, and remove DNSResolveFailed,
DNSResolutionFailed, Timeout, Aborted, AbortedBeforeConnecting, and
ClientAborted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8aa7417e-a90c-4725-b3f6-750bc7b24b6d
📒 Files selected for processing (10)
docs-site/src/content/docs/ja/reference/configuration/providers.mddocs-site/src/content/docs/ko/reference/configuration/providers.mddocs-site/src/content/docs/reference/configuration/providers.mddocs-site/src/content/docs/ru/reference/configuration/providers.mddocs-site/src/content/docs/zh-cn/reference/configuration/providers.mdsrc/lib/upstream-retry.tssrc/server/responses/compact.tssrc/server/responses/core.tstests/codex-account-neutral-network.test.tstests/server-auth.test.ts
| | `accountPoolStrategy?` | `"quota" \| "round-robin" \| "fill-first"` | `"quota"` | Assignment strategy for new/unbound Codex requests. A request is unbound when it has no live (parent thread id, quota scope) affinity; a visible existing task can become unbound after proxy restart or affinity reset. `quota` picks the lowest-usage eligible account when no active account exists, keeps an eligible active account below `autoSwitchThreshold`, and after the threshold may move an unbound request or proactively rebind a bound task to a lower-usage eligible account. `round-robin` distributes unbound requests evenly; `fill-first` keeps assigning unbound requests to the active account until cooldown, unavailability, or the configured drain threshold. | | ||
| | `accountPoolStickyLimit?` | `number` | `1` | New/unbound task assignments retained on one round-robin selection before advancing; the counter advances when a task is bound, not after an upstream success. Range 1–100. | | ||
| | `upstreamFailoverThreshold?` | `number` | `3` | Consecutive transient failures before future new sessions fail over. Set `0` to disable. | | ||
| | `upstreamFailoverThreshold?` | `number` | `3` | Consecutive account-scoped transient outcomes before future new sessions fail over. Host-wide DNS and network reachability failures are account-neutral and do not count. Set `0` to disable. | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Keep the neutral-failure boundary explicit in every locale.
The five lines broaden the account-neutral category. The runtime applies it only to exact pre-connect DNS and reachability errors found within a bounded cause chain. Timeouts, aborts, resets, socket closures, HTTP failures, and semantic upstream failures remain account-scoped.
docs-site/src/content/docs/reference/configuration/providers.md#L25-L25: update the canonical English wording with “pre-connect” and “runtime-classified exact errors,” then state that other failures remain account-scoped.docs-site/src/content/docs/ja/reference/configuration/providers.md#L24-L24: translate the same qualifiers into Japanese.docs-site/src/content/docs/ko/reference/configuration/providers.md#L24-L24: translate the same qualifiers into Korean.docs-site/src/content/docs/ru/reference/configuration/providers.md#L25-L25: translate the same qualifiers into Russian.docs-site/src/content/docs/zh-cn/reference/configuration/providers.md#L24-L24: translate the same qualifiers into Simplified Chinese.
As per path instructions, keep all localized provider documentation synchronized with actual runtime behavior.
📍 Affects 5 files
docs-site/src/content/docs/reference/configuration/providers.md#L25-L25(this comment)docs-site/src/content/docs/ja/reference/configuration/providers.md#L24-L24docs-site/src/content/docs/ko/reference/configuration/providers.md#L24-L24docs-site/src/content/docs/ru/reference/configuration/providers.md#L25-L25docs-site/src/content/docs/zh-cn/reference/configuration/providers.md#L24-L24
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs-site/src/content/docs/reference/configuration/providers.md` at line 25,
Update the upstreamFailoverThreshold documentation to explicitly describe
account-neutral failures as runtime-classified exact pre-connect DNS and
reachability errors within a bounded cause chain, and state that timeouts,
aborts, resets, socket closures, HTTP failures, and semantic upstream failures
remain account-scoped. Apply equivalent wording in
docs-site/src/content/docs/reference/configuration/providers.md lines 25-25,
docs-site/src/content/docs/ja/reference/configuration/providers.md lines 24-24,
docs-site/src/content/docs/ko/reference/configuration/providers.md lines 24-24,
docs-site/src/content/docs/ru/reference/configuration/providers.md lines 25-25,
and docs-site/src/content/docs/zh-cn/reference/configuration/providers.md lines
24-24, translating the qualifiers appropriately for each locale.
Source: Path instructions
|
Reviewing this against a runtime probe, because I spent four audit rounds on the same problem in a planning unit and want to hand over what killed each attempt rather than have you rediscover it. Including But there is a consequence worth deciding deliberately. On Bun 1.3.14 a nonexistent hostname and a refused port are indistinguishable from the error object: Same code, same errno, no Two cases my audit found that are worth adding to your test matrix, both of which put a rejection on this path after the credential was seen:
One more, from the same audit: a transient 5xx that exhausts The analysis that produced all of this is in |
2 participants
Summary
upstreamFailoverThresholdin every shipped documentation localeRoot cause
Regular Responses and native compact mapped every non-timeout transport exception to an account-specific
connect_error. A host-wide DNS outage therefore incremented each selected credential's failure streak and could clear affinity or rotate accounts even though every account used the same unreachable ChatGPT host.The classifier is deliberately narrow: it trusts exact own-data
codeandcauseproperties through a bounded chain, and any abort, timeout, reset, or socket-close blocker wins.User impact
An existing Codex task remains on its affined account during shared DNS or reachability failures. The request still receives a retryable 502, but OpenCodex no longer treats the outage as evidence that another credential is healthier.
This does not retry a partially committed request, suppress real upstream rejections, or change quota-based selection for new/unbound work.
Validation
bun test tests/codex-account-neutral-network.test.tsbun test tests/server-auth.test.ts --timeout 15000bun run typecheckbun run privacy:scangit diff --check origin/dev...HEADnode_modules/bun/bin/bun install --frozen-lockfile && node_modules/bun/bin/bun run buildfromdocs-site/A prior local full-suite run exercised 7,442 tests. Its remaining Windows/runtime-provenance, fixed-timeout, and local dependency failures were reproduced or isolated on clean
origin/dev; the one stale expectation affected by this change was corrected and the completeserver-auth.test.tssuite now passes. Hosted CI on the finaldevbase remains required.Review gates
This touches authentication/account-routing behavior. Per the current contribution policy it requires the
maintainer-sponsoredlabel and explicit maintainer security review before merge. The PR is otherwise complete and ready for review.Fixes #914.
Related: #919 covers a distinct post-header mid-stream body failure; this PR handles only pre-header DNS and network reachability failures.
Summary by CodeRabbit
Bug Fixes
Documentation
upstreamFailoverThresholdbehavior across supported languages, including which failures count and how to disable the threshold.