fix(anthropic): surface upstream error bodies through sidecar bridges

devlog/_plan/260805_bug_stack_campaign/000_plan.md

@@ -0,0 +1,99 @@
+# 000 — Bug stack campaign: triage the open bug surface, fix the 400 regression, stack the repairs
+
+## Objective
+
+As of 2026-08-05 (KST), bring the open bug surface of lidge-jun/opencodex to
+the state where every item is either landed, closed with evidence, or carried
+by a stacked PR grounded in code somebody actually read. Three fronts:
+
+1. PR #988 — review against the GUI design system, small fixes if needed,
+   merge to `dev` (user authorized this merge/push).
+2. Full triage of open bug issues + bug/code-improvement PRs against
+   `origin/dev` HEAD `e44d234f08e03dd4dbf0c4aa13af43046d86b0a6`. Items
+   already fixed on `dev` get closed with evidence; the rest enter the stack.
+3. New regression: 400 `invalid_request_error` when a new Codex session/thread
+   spawns against `anthropic/claude-opus-5` (2026-08-05 00:22 KST, request
+   `ocx-mset3rk6-1i6`, conversation `2c0f87c664e130b3927c4884fafd8283`).
+   Root-cause and fix with a regression test.
+
+Items opened after 2026-08-05 00:00 KST are out of scope. Feature programs
+(provider batches, RI router-intelligence stack, localization programs,
+large infrastructure PRs) are out of scope.
+
+## Loop spec
+
+- Loop archetype: spec-satisfaction repair (each fix has a checkable verifier).
+- Trigger: user-directed campaign, HOTL goal loop (goalplan
+  `opencodex-pr-988-988-uiux-bug-pr-dev-head-close`).
+- Goal: open bug surface fully dispositioned; #988 landed; 400 fixed.
+- Non-goals: feature PRs, new providers, RI stack, docs-only improvements
+  without code defect, anything opened after the campaign cutoff.
+- Verifier: `bun run typecheck`, focused `bun test` per fix, full
+  `bun run test` on `ssh lidge` (Linux baseline recorded below),
+  `bun run privacy:scan`, plus per-fix activation evidence.
+- Stop condition: every in-scope issue/PR dispositioned (landed / closed with
+  evidence / stacked PR open with CI).
+- Memory artifact: this unit + goalplan ledger.
+- Resource bounds: sol-medium explorer subagents for research/verification
+  (unlimited); B phases owned by the main session; wall-clock unbounded but
+  checkpoint every B step with commits (LOOP-GIT-01).
+- Expected terminal outcomes: DONE (all dispositioned), or per-item
+  BLOCKED/NEEDS_HUMAN named with evidence.
+- Escalation: push/merge only within the user-approved scope (#988 merge,
+  stack PR creation, evidence-based closes).
+
+## Environment baselines
+
+- `origin/dev` HEAD: `e44d234f08e03dd4dbf0c4aa13af43046d86b0a6`.
+- Worktree: `/Users/jun/.codex/worktrees/250c/opencodex`, branch
+  `codex/bug-stack-campaign` (forked from `origin/dev`; the previously
+  detached other-unit HEAD `9289891a5` remains untouched, contained in local
+  `dev`).
+- lidge Linux baseline (`bun run test` on dev HEAD, 2026-08-05): the first
+  cold-cache run showed 12 fails with a phantom
+  `ROLLUP_COST_SEMANTICS_VERSION` import error (an identifier that exists
+  nowhere in the repo) plus storage/keyring suites — a first-run module-cache
+  race, not repo state. Warm-cache rerun: **8212 pass / 10 skip / 0 fail,
+  EXIT=0** (`/tmp/ocx-baseline-test2.log` on lidge). Campaign deltas are
+  measured against the green warm baseline.
+
+## Work-phase map
+
+Honest dependency structure (amended after audit round 1): only three real
+dependencies exist — phase 2 builds on the 003 probes, phase 3's core.ts
+cluster (stack 02/03/05) must be internally ordered, and each phase-4
+disposition waits for its corresponding landing. Everything else is
+independent and does not inherit merge blockage from earlier items.
+
+| Phase | Doc | Content | Depends on |
+|-------|-----|---------|------------|
+| 0 | this doc + 001/002/003 | triage + research (docs-only) | — |
+| 1 | 010 | PR #988 design review + merge | none (independent GUI slice) |
+| 2 | 020 | anthropic sidecar-bridge error fidelity (`formatErrorBody`) | 003 probes + bridge analysis |
+| 3 | 030 | #914 DNS transport attribution | 001/002 triage |
+| 4 | 040 | #893 sparse snapshot repair | none (independent) |
+| 5 | 050 | #875 DeepSeek Flash stall | none (independent) |
+| 6 | 060 | #938 UUID item IDs | 050 (shared JSON→event boundary) |
+| 7 | 070 | #907 jawcode prices (+ external jawcode source write) | none |
+| 8 | 080 | #1007 login URL flush | none |
+| 9 | 090 | #1001 forced-answer validation | none |
+| 10 | 100 | #992 routed context_window | none |
+| 11 | 110 | #993 Kiro profileArn | none |
+| 12 | 120 | #959 provider headers (adopt PR #961) | none |
+| 13 | 130 | dispositions: #806 close (anytime); supersede-closes per landing | respective phases |
+
+Dependency honesty (audit round 3): 030, 040, 050 are semantically
+independent — none consumes another's artifact. Only 050→060 is a real
+dependency (060 extends 050's bounded-JSON event boundary). To control
+`core.ts` merge conflicts the RECOMMENDED merge order is 030 → 040 → 050 →
+060, then the independent lanes 070-120 in any order, each branching from
+`origin/dev` (or the campaign base when doc files ride along). Each phase is
+one full PABCD cycle whose P re-verifies its decade doc against the
+then-current tree.
+
+## Triage inputs
+
+- `001_issue_triage.md` — 38 open issues (sol-medium lane A).
+- `002_pr_triage.md` — 30 open PRs (sol-medium lane B).
+- `003_anthropic_400_research.md` — 400 regression research (lane C + local
+  log analysis).

devlog/_plan/260805_bug_stack_campaign/001_issue_triage.md

@@ -0,0 +1,43 @@
+# 001 — Open issue triage (2026-08-05, vs origin/dev e44d234f0)
+
+Source: sol-medium explorer lane A. 38 open issues. Classification: 14 bug
+(10 unresolved, 4 needs-verification), 4 code-level improvement (1 already
+resolved on dev), 20 out of scope (feature/provider/tracker/upstream).
+
+## Confirmed unresolved bugs — stack candidates
+
+| Issue | Defect | Code evidence |
+|------:|--------|---------------|
+| #1007 | `ocx account login` withholds the authorization URL under non-TTY stdout | `src/cli/account-auth.ts:84-101` — console.log then long polling, no explicit flush |
+| #1001 | Hosted web-search forced-answer pass accepts a malformed tool call and completes without an assistant message | `src/web-search/loop.ts:72-75,681-705` — empty tool name judged real; forced-answer ends without visible message check |
+| #993 | Kiro provider: profileArn required 400 for Builder ID accounts on gated models | `src/adapters/kiro.ts:1723-1729` — profileArn silently omitted, CLI wire used; no per-account ARN acquisition |
+| #992 | Routed models inherit native template context_window when /models omits context metadata | `src/codex/catalog/sync.ts:205-228`, `src/codex/catalog/parsing.ts:290-301` |
+| #959 | No management-plane support for provider `headers` | `src/server/management/provider-routes.ts:210-289` — PATCH mask lacks `headers`, unknown fields 400; CLI lacks the option (`src/cli/provider-runtime.ts:16-55`) |
+| #938 | Responses passthrough UUID item IDs leave Codex stuck on Thinking | `src/server/responses-item-id-repair.ts:84-94,220-223` — repair only handles pre-registered placeholders / missing terminal IDs, never validates arbitrary UUID prefixes |
+| #914 | DNS/network reachability failures incorrectly rotate Codex pool accounts | `src/server/responses/core.ts:1737`, `src/server/responses/compact.ts:419` — fetch rejections still recorded as `connect_error` against account health; candidate fix `fe693ae62` not an ancestor of origin/dev |
+| #907 | Bundled jawcode prices for gpt-5.6-terra/luna are pre-price-cut rates | `src/generated/jawcode-model-metadata.ts:47` — stale Luna 1/6/0.1/1.25, Terra 2.5/15/0.25/3.125 |
+| #893 | Responses-compatible gateways can return sparse lifecycle snapshots Codex clients do not commit | no `responsesSnapshotRepair`; current SSE assembly only repairs images and item IDs (`src/server/responses/core.ts:1967-1976`) |
+| #875 | DeepSeek V4 Flash Responses route stalls after tool calls | `5dd965a13` only disabled WebSocket upstream streaming (`src/providers/registry.ts:1146`); reporter reproduced on default HTTP/SSE path at e44d234f0, still SSE relay (`src/server/responses/core.ts:1880-1881,1967`) |
+
+## Needs verification before stack entry
+
+| Issue | Why unresolved status is uncertain |
+|------:|-----------------------------------|
+| #994 | `reasoning_content` replay is model-allowlisted (`src/providers/registry.ts:918-958,1637-1655`); report lacks provider/model + wire capture |
+| #904 | `eeef7a32a` fixed astral-surrogate boundaries but does not explain the original Hangul-only corruption; needs the failing wire capture |
+| #796 | Structured empty-content fix `d3abf4345` + regression test landed (`src/adapters/openai-chat.ts:366-379`), but no live Ark credential verification |
+| #418 | Current code provides catalog/roster guidance (`src/server/responses/collaboration.ts:243-304`); latest same-run trace does not reproduce the original failure |
+
+## Already resolved on dev — close candidates
+
+| Issue | Evidence |
+|------:|----------|
+| #806 | `d52b387db` (ancestor of origin/dev) split/fixed GUI+CLI+docs wording; current GUI distinguishes "Usage-based proactive switching" from cache warnings (`gui/src/i18n/en.ts:1296-1315`) |
+
+## Out of scope (20)
+
+Feature/provider/tracker/upstream items: #974, #823, #822, #821, #755, #695,
+#657, #572, #561, #540, #241, #201, #178, #177, #95, #92, #417, #415, #414,
+#386. (Improvement #820 and #809 are code-level and stay in the campaign's
+improvement bucket; #820 is a larger architecture epic — deferred to its own
+unit, not this stack.)

devlog/_plan/260805_bug_stack_campaign/002_pr_triage.md

@@ -0,0 +1,47 @@
+# 002 — Open PR triage (2026-08-05, vs origin/dev e44d234f0)
+
+Source: sol-medium explorer lane B. 30 open PRs, all targeting `dev`.
+Classification: 15 bug, 3 code/docs improvement, 12 feature/program.
+No bug/improvement PR has an equivalent fix already merged on dev.
+
+## Bug PRs
+
+| PR | Author | Subject | Draft | CI | Merge state |
+|---:|--------|---------|:-----:|----|-------------|
+| #988 | Wibias | GUI providers quota/auth, Claude pool toggle, combos/models layout, dev session bootstrap | no | full PASS | MERGEABLE/CLEAN — only clean bug PR |
+| #983 | DevMello | stop counting base64 attachments as raw characters in token estimates | no | policy-only | MERGEABLE/UNSTABLE |
+| #978 | DevMello | send thinkingLevel for any model with an effort ladder (google) | no | policy-only | MERGEABLE/UNSTABLE |
+| #985 | DevMello | deliver structured output to routed openai-chat models | yes | policy-only | MERGEABLE/UNSTABLE |
+| #1006 | Michael-Han0608 | bounded JSON policy on HTTP SSE for DeepSeek Flash | no | policy-only | MERGEABLE/UNSTABLE |
+| #1000 | WZBbiao | avoid false project config warnings | no | policy-only | MERGEABLE/UNSTABLE |
+| #947 | WZBbiao | prevent Darwin rewrite stalls (streaming) | no | policy-only | MERGEABLE/UNSTABLE |
+| #997 | Yuxin-Qiao | isolate usage-log fixtures from the real OpenCodex home | no | policy-only | MERGEABLE/UNSTABLE |
+| #966 | Yuxin-Qiao | keep pre-connection DNS/network failures off account health (#914) | yes | policy-only | CONFLICTING/DIRTY; prior audit left mixed-5xx/redirect counterexamples |
+| #928 | 0xWinner98 | repair sparse Responses snapshots (streaming) | yes | policy-only | CONFLICTING/DIRTY; changes requested |
+| #933 | IMHinnG | enforce type:"object" on all tool parameters (openai-chat) | yes | FAIL (Ubuntu/Windows/macOS + enforce-target) | MERGEABLE/UNSTABLE |
+| #940 | mouzhi | DeepSeek Responses UUID item ids for Codex | yes | policy-only | CONFLICTING/DIRTY; changes requested |
+| #922 | luvs01 | isolate provider host transport health | no | policy-only | MERGEABLE/UNSTABLE; changes requested; #966 supersedes but is unfinished |
+| #936 | lidge-jun | harden credential and runtime trust boundaries (rebase of #916) | yes | full PASS | CONFLICTING/DIRTY; needs explicit security review |
+| #557 | lidge-jun | harden npm cache recovery preflight logs (update) | no | FAIL (Ubuntu/Windows; macOS cancelled) | MERGEABLE/UNSTABLE |
+
+## Improvement PRs
+
+| PR | Author | Subject | Note |
+|---:|--------|---------|------|
+| #999 | Yuxin-Qiao | docs: Desktop remote allowlist limitation (#241) | docs-only, draft |
+| #569 | diegocantarero | post-sync readiness endpoint + bounded `ocx ready` wait | draft, CI PASS but CONFLICTING/DIRTY |
+| #1002 | hanjianjun | configurable sidecar reasoning (vision) | includes ignored-CLI-setting defect |
+
+## Feature programs — out of scope (12)
+
+#811, #1003, #1004, #1005, #715, #961, #581, #998, #937, #872, #870, #812.
+
+## Issue↔PR coupling for the stack
+
+- #914 ↔ PR #966 (conflicting, audit counterexamples) and PR #922 (superseded
+  but unfinished) — the stack should land its own clean fix and route both
+  PRs to a disposition.
+- #938 ↔ PR #940 (conflicting, changes requested) — same handling.
+- #893 ↔ PR #928 (conflicting, changes requested).
+- #875 ↔ PR #1006 (mergeable but policy-only CI; unverified against live
+  DeepSeek).

devlog/_plan/260805_bug_stack_campaign/003_anthropic_400_research.md

@@ -0,0 +1,119 @@
+# 003 — Anthropic 400 on new-session spawn: research
+
+## Symptom
+
+2026-08-05 00:21-00:23 KST: three consecutive 400 `invalid_request_error`
+responses against `anthropic/claude-opus-5` (effort high), all in conversation
+`2c0f87c664e130b3927c4884fafd8283` (`ocx-mset3lcs-1i1`, `ocx-mset3rk6-1i6`,
+`ocx-mset4i2p-1ik`), plus one in `7f8dbd4a06336b01da0d476fb65b38f2` at 00:23.
+Same model+effort returned 200 for other conversations from 00:26 onward,
+including 319K-620K-token inputs. Model alias failure, auth failure, and
+context overflow are excluded.
+
+Local evidence gap: `usage.jsonl` records only `Provider error 400` — the
+upstream error body was not retained for these responses-inbound failures
+(compare 2026-08-01 rows, which preserved the full Anthropic error JSON).
+
+## Root cause — REFUTED hypothesis and what the live probes proved
+
+The initial lane-C hypothesis (unsigned foreign tool_use + adaptive thinking →
+400) is **refuted by two live probes** run through the local proxy against the
+real Anthropic API on 2026-08-05 ~01:00 KST:
+
+- Probe 1 (synthetic): inline `custom_tool_call` + `custom_tool_call_output`
+  history, `anthropic/claude-opus-5`, effort high → **200** (134 tokens).
+- Probe 2 (exact parent state): the full 923,656-byte item list of
+  `resp_050cd54528dd6d7f…` (the last successful state at 00:21:12, 46s before
+  the first failure) + a new user message, same model/effort → **200**
+  (297,376 input tokens).
+
+Anthropic's current extended-thinking contract matches probe 1: adaptive
+thinking does not require a thinking preface on replayed tool-use turns.
+The history shape is exonerated. (A-gate audit round 1, blocker 1.)
+
+Transport-branch analysis (audit round 2, verified in code): the bare
+`Provider error 400` does NOT come from the direct path — that path appends
+`: <body>` even when the body is empty (`src/server/responses/core.ts:2822`,
+trailing colon preserved). The exact bare shape is produced by the
+**web-search bridge** (`src/web-search/loop.ts:460-484`): a body-read failure
+returns status-only, and the formatted suffix requires
+`prepared.responseAdapter.formatErrorBody`, **which the anthropic adapter
+does not implement** — so on this path even a well-formed Anthropic JSON
+error envelope is discarded and the client + ledger see only
+`Provider error 400`. The images bridge has analogous status-only behavior.
+Thread-spawn requests from the Codex app carry the `web_search` tool, and
+routed (non-OpenAI) models dispatch through `runWithWebSearch`
+(`src/server/responses/core.ts:2295-2310`), which returns before the normal
+recovery loop.
+
+Conclusion: the failing requests almost certainly went through the web-search
+bridge, and the 400's own cause (validation vs edge) is **unknowable from
+surviving evidence because the bridge discarded the upstream body**. The
+diagnosability defect is precisely located; the underlying 400 is reproduced
+only if it recurs after the observability fix. Anthropic's error contract
+(every API error carries a JSON envelope) makes "proxy discarded the body"
+the more probable reading than "empty-body edge rejection", though both
+remain possible.
+
+## Original research record (kept for provenance)
+
+Each failure happened exactly when resuming the next turn after a
+`custom_tool_call_output`. Reconstruction of the failing body from saved
+Responses state + rollout items shows:
+
+```json
+{ "thinking": { "type": "adaptive" }, "output_config": { "effort": "high" }, "max_tokens": 24576 }
+```
+
+with history containing assistant turns of shape `[text, tool_use]` with **no
+thinking/redacted_thinking block**, followed by `user: [tool_result]`.
+
+Mechanism (later refuted by probes 1-2 above):
+
+1. `previous_response_id` expansion merges stored input regardless of which
+   provider produced the earlier turns (`src/responses/state.ts:857-860`).
+2. The Responses parser restores foreign `custom_tool_call` items as assistant
+   tool calls but cannot fabricate a valid Anthropic-signed thinking block
+   from OpenAI/Grok reasoning (`src/responses/parser.ts:510-517`, envelope
+   handling at 439-475; unsigned fallbacks fail
+   `isLikelyRealAnthropicThinkingSignature`, `src/adapters/anthropic.ts:240-244`).
+3. The Anthropic adapter drops unsigned thinking parts
+   (`isLikelyRealAnthropicThinkingSignature`, `src/adapters/anthropic.ts:601`)
+   but still emits the `tool_use` blocks (`src/adapters/anthropic.ts:604-607`),
+   then turns adaptive thinking on for the whole request
+   (`src/adapters/anthropic.ts:834-835`).
+4. Anthropic 400s: with thinking enabled, an assistant tool-use turn must be
+   preceded by a thinking/redacted_thinking block.
+
+Secondary hypothesis (prefixed model ID missing the adaptive gate on older
+runtimes, fixed by `930efdf60`/`f728dc0fb`) does not fit: the running process
+started 2026-08-04 22:45 KST from a checkout containing both fixes, and the
+reconstructed body was already `adaptive`.
+
+Latent-path activation: `787bd1541` exposed Opus 5 in the catalog
+(`src/providers/registry.ts:240`); the regression is not the alias but the
+first time an adaptive-thinking model replayed foreign tool history.
+
+## "New session" interpretation
+
+`conversationId` is a log-correlation hash, not a continuation key
+(`src/server/request-log-conversation.ts:30-38,64-75`). History replay is decided by
+`previous_response_id`. The failing "new prompt/spawn" inherited the parent's
+Responses chain — the proxy must not clear state on thread-spawn headers
+(that would break intentional context inheritance).
+
+## Fix direction
+
+Revised after the probes (see `020_phase2_anthropic_400_fix.md`):
+
+1. Observability (phase 2): implement `formatErrorBody` on the anthropic
+   adapter so the web-search/images bridges surface the upstream error
+   message instead of discarding it; persisted `upstreamError` then flows
+   through the existing capture (`src/server/relay.ts:429` →
+   `src/server/request-log.ts:648`) with no production change there.
+2. Resilience (deferred): any 400 retry decision waits until the observability
+   fix distinguishes empty-body edge rejections from discarded JSON bodies in
+   the wild. Premature retry design was audit-blocked twice and is dropped
+   from phase 2.
+3. NO history flattening (refuted hypothesis); no signature fabrication; no
+   state clearing on thread spawn.