Add codex-setup and docs/faq, and correct the --agent list

README.md

@@ -44,7 +44,7 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha
 /benchmark provider=bedrock model=claude-opus-5 dataset=dataset/mcp-gateway-registry-v2.yaml agent=omp
 ```
 
-`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set.
+`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set.
 
 ## Step 2 — Developers install the skill
 
@@ -120,6 +120,9 @@ Where to read more, by topic:
 | [benchmarks/docs/path-anthropic-on-bedrock.md](benchmarks/docs/path-anthropic-on-bedrock.md) | Path 1 setup: benchmarking the Anthropic family (Claude Opus/Sonnet/Haiku) directly on Amazon Bedrock. |
 | [benchmarks/docs/path-open-weight-on-bedrock-litellm.md](benchmarks/docs/path-open-weight-on-bedrock-litellm.md) | Path 2 setup: open-weight models on Amazon Bedrock through the LiteLLM proxy. |
 | [benchmarks/docs/path-self-hosted-vllm.md](benchmarks/docs/path-self-hosted-vllm.md) | Path 3 setup: self-hosting a model on vLLM and pointing the harness at it. |
+| [docs/faq/](docs/faq/) | Wiring each agent to a model, one page per agent (Claude Code, omp, codex), same format throughout: every provider route with the exact command. |
+| [docs/omp-setup.md](docs/omp-setup.md) | The omp harness: install, the provider flags, auto-approve, and the JSON event stream the harness reads for metrics. |
+| [docs/codex-setup.md](docs/codex-setup.md) | The codex harness: install, `codex exec` headless use, the provider block an endpoint run needs, and why the sandbox must be bypassed on a benchmark host. |
 | [docs/kiro-cli-setup.md](docs/kiro-cli-setup.md) | The kiro-cli harness: install, sign-in, headless use, and the Bedrock-managed-only constraint. |
 | [benchmarks/docs/end-to-end-self-hosted-run.md](benchmarks/docs/end-to-end-self-hosted-run.md) | The full manual run-book for an end-to-end self-hosted benchmark. |
 | [self-hosted/vllm/README.md](self-hosted/vllm/README.md) | Standing up a vLLM server: install, tensor parallelism, tool-call parsers, and the serving-config reference. |

benchmarks/scripts/run-e2e-benchmark.sh

@@ -81,7 +81,7 @@ REPO_ROOT="$(dirname "$BENCHMARKS_DIR")"
 VLLM_DIR="$REPO_ROOT/self-hosted/vllm"
 
 # Defaults
-AGENT="claude"            # coding agent: claude (Claude Code) or pi (pi agent)
+AGENT="claude"            # coding agent: claude, pi, omp, kiro, or codex
 SKILL="swe3"              # SWE skill: swe3 (single-agent, default) or swe2 (multi-agent)
 PROVIDER=""
 MODEL=""

docs/codex-setup.md

@@ -0,0 +1,69 @@
+# Installing codex and pointing it at a model
+
+`codex` does two jobs here, and they are configured separately. It is the **judge** that scores every benchmark run, always against Amazon Bedrock, and it is one of five **coding harnesses** (`--agent codex`) that can drive a task on any hosting path. This page installs it, proves it works, and states the one constraint that decides which models it can reach. The per-path wiring recipes live in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md)
+
+## Install
+
+```bash
+npm install -g @openai/codex
+codex --version    # >= 0.144 for the native amazon-bedrock provider
+```
+
+The `setup-machine` skill installs it as part of the core stack, so on a box bootstrapped with [setup-machine.sh](../.claude/skills/setup-machine/setup-machine.sh) it is already present. Installing the CLI does **not** wire it to Bedrock.
+
+## Wire it to Bedrock and prove it
+
+codex ships a native `amazon-bedrock` provider that authenticates from the AWS credential chain. No proxy, no bearer token; the LiteLLM route older notes describe is legacy. The config lives at `~/.codex/config.toml`:
+
+```toml
+model_provider = "amazon-bedrock"
+model_providers.amazon-bedrock.aws.region = "us-east-2"
+model = "openai.gpt-5.6-sol"
+```
+
+Full walk-through, including the `claude` side: [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md).
+
+**Prove it with a real call before starting a long run.** Working AWS credentials are not sufficient: an unconfigured codex ignores them and 401s against `api.openai.com`, and the failure surfaces only after the harness has finished generating.
+
+```bash
+codex exec --skip-git-repo-check "Reply with exactly: JUDGE OK"
+```
+
+## The constraint that decides everything else
+
+**codex speaks the OpenAI Responses API and nothing else.** codex 0.153.4 removed the chat-completions wire and rejects the fallback:
+
+```
+Error loading config.toml: `wire_api = "chat"` is no longer supported.
+How to fix: set `wire_api = "responses"` in your provider config.
+```
+
+Amazon Bedrock answers `/v1/responses` for the **`openai.*` family only**. Everything downstream follows:
+
+| Model you want to drive | Provider | Reachable directly? |
+|---|---|---|
+| `openai.*` on Bedrock | `bedrock` | Yes, this same native provider |
+| Any model on your own vLLM server | `endpoint` (`--provider vllm`) | Yes, if its tool parser accepts Responses-shaped tools |
+| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `endpoint` (`--provider litellm`) | No. Bedrock rejects Responses for these; a LiteLLM bridge is required |
+| Anthropic models | `bedrock` | Yes, though Claude Code is the better-measured harness for them |
+
+The commands for each row are in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md)
+
+## Known failure modes
+
+Each of these cost real debugging time. The FAQ carries the detail and the fix.
+
+| Symptom | Cause |
+|---|---|
+| Run hangs before the first request, log stops at `Reading additional input from stdin...` | `codex exec` blocks on an open, empty stdin. Launch every run with `< /dev/null` |
+| `404 The model '<name>' does not exist` on an endpoint run | codex 0.153.4 ignores `OPENAI_BASE_URL` and used the config's provider, which on a judge-configured box is Bedrock (issue #183) |
+| Every request retried five times, stream ends with no `response.completed` | The vLLM tool parser reads the nested chat-completions tool shape and aborts on the flat Responses shape. Use `qwen3_coder` or `hermes` |
+| `Model metadata not found. Defaulting to fallback metadata` | codex does not know a self-hosted model's window. Pass `--context-window` |
+| `does not support the '/openai/v1/responses' API` | An open-weight Bedrock model reached without the bridge |
+| `total_cost_usd` is null | The model has no row in [bedrock_pricing.py](../benchmarks/scripts/bedrock_pricing.py). codex reports tokens but no billed cost, so an unpriced model records null rather than a misleading zero |
+
+## Related
+
+- [faq/wiring-codex-to-models.md](faq/wiring-codex-to-models.md) -- the per-path recipes, the parser table, and why prompt caching does not happen on the Bedrock bridge.
+- [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock.
+- [harness-reference.md](../benchmarks/docs/harness-reference.md#choosing-the-agent) -- every supported agent, its providers, and its cost basis.

docs/faq/README.md

@@ -0,0 +1,17 @@
+# FAQ
+
+Answers to questions that come up while running this benchmark. One file per question, listed here.
+
+| Question | Answer |
+|----------|--------|
+| How do I wire codex to a model: an open-weight model on Bedrock, one I serve on vLLM, or an OpenAI model on Bedrock? | [wiring-codex-to-models.md](wiring-codex-to-models.md) |
+| How do I wire omp to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-omp-to-models.md](wiring-omp-to-models.md) |
+| How do I wire Claude Code to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-claude-code-to-models.md](wiring-claude-code-to-models.md) |
+
+Add an entry by writing the file next to this one and adding a row above. Keep the question in the reader's words, not the code's: someone searching this index knows what they are trying to do, not what the harness calls it.
+
+For setup and reference material rather than questions, start at the [documentation map](../../README.md#documentation-map) or [AGENTS.md](../../AGENTS.md).
+
+---
+
+[< Back to the README](../../README.md)

docs/faq/wiring-claude-code-to-models.md

@@ -0,0 +1,125 @@
+# How do I wire Claude Code to a model?
+
+Three ways to point `claude` at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or a model on Bedrock directly. Each ends with a `claude -p` call you can run by hand.
+
+For wiring `claude` and the `codex` judge to Bedrock on a fresh box, see [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md). This page assumes `claude --version` already answers.
+
+> [!NOTE]
+> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with Claude Code `2.1.266`. Each `claude -p` example returned `OK`: the Bedrock one against `claude-haiku-4-5`, the vLLM one against a live `qwen3.6-35b-fp8` server, and the open-weight-on-Bedrock one against `qwen.qwen3-coder-30b-a3b-instruct` through the LiteLLM mantle proxy.
+
+## The three facts everything follows from
+
+**Claude Code speaks the Anthropic Messages API.** Anything else must answer `POST /v1/messages`. vLLM does, which is why a self-hosted model needs no bridge here; Bedrock's non-Anthropic models do not, which is why they need one.
+
+**`--settings` wins, and you always want to pass it.** A settings object's `env` block takes precedence over process environment variables, including anything in your global `~/.claude/settings.json`. A global file pinning `CLAUDE_CODE_USE_BEDROCK=1` will otherwise redirect a run meant for a local endpoint straight to Bedrock, which rejects the local model id. Passing `--settings` is what reliably wins.
+
+**Claude Code cannot detect a custom model's context window.** Against a custom `ANTHROPIC_BASE_URL` it has no window to compact against, so on a long task the conversation grows until the endpoint rejects it, and Claude Code treats that 500 as transient and retries it forever. Set `CLAUDE_CODE_AUTO_COMPACT_WINDOW` to the served window on every endpoint run.
+
+## Open-weight model you serve with vLLM
+
+**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`:
+
+```bash
+cd self-hosted/vllm/scripts
+MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \
+SERVED_NAME="qwen3.6-35b-fp8" \
+TP=1 \
+PORT=8000 \
+MAX_MODEL_LEN=262144 \
+GPU_MEM_UTIL=0.92 \
+TOOL_PARSER="qwen3_coder" \
+EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \
+  ./vllm-serve.sh
+```
+
+vLLM answers the Anthropic Messages API as well as the OpenAI one, so nothing sits between Claude Code and the server. Confirm it before blaming the agent:
+
+```bash
+curl -s http://127.0.0.1:8000/v1/messages \
+  -H 'content-type: application/json' -H 'x-api-key: local' \
+  -H 'anthropic-version: 2023-06-01' \
+  -d '{"model":"qwen3.6-35b-fp8","max_tokens":16,"messages":[{"role":"user","content":"Reply with exactly: OK"}]}'
+```
+
+**2. Run it.**
+
+```bash
+claude -p "Reply with exactly: OK" \
+  --model qwen3.6-35b-fp8 \
+  --output-format json \
+  --settings '{"apiKeyHelper":"echo local","env":{
+      "CLAUDE_CODE_USE_BEDROCK":"0",
+      "ANTHROPIC_BASE_URL":"http://127.0.0.1:8000",
+      "ANTHROPIC_API_KEY":"local",
+      "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144",
+      "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null
+```
+
+**`apiKeyHelper` is required even though the server ignores the value.** Without a token source Claude Code refuses with "Not logged in". `echo local` satisfies it.
+
+**Do not trust `total_cost_usd` on this route.** The verified run against a local server reported `"total_cost_usd": 0.1138` for a model that costs nothing per token, because Claude Code prices from its own assumptions for an unrecognized model. It also reported `"contextWindow": 200000` while the server was serving 262,144. For a self-hosted model the repo derives cost from GPU-seconds instead, never from this field ([cost-per-task-methodology.md](../cost-per-task-methodology.md)).
+
+## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek)
+
+Bedrock's non-Anthropic models do not answer the Messages API. Worse, sent through Bedrock's Converse path they return their native tool-call tokens as plain text, so Claude Code never sees a structured `tool_use` block and an agentic run stalls at one turn with zero artifacts. The fix is a proxy that translates Anthropic Messages to OpenAI chat completions and on to Bedrock's OpenAI-compatible `bedrock-mantle` endpoint, which parses those tokens into real tool calls. The repo ships it.
+
+**1. Start the proxy.**
+
+```bash
+cd benchmarks
+./scripts/bedrock-mantle-proxy.sh          # installs deps, mints a 12h token, listens on :4000
+./scripts/bedrock-mantle-proxy.sh --status
+```
+
+It mints the bearer token from your ambient AWS credentials and holds it server-side; clients send a throwaway key. Models come from [config/litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml), and `curl -s http://127.0.0.1:4000/v1/models` lists them.
+
+**2. Run it.** Same shape as the vLLM route, pointed at the proxy:
+
+```bash
+claude -p "Reply with exactly: OK" \
+  --model qwen.qwen3-coder-30b-a3b-instruct \
+  --output-format json \
+  --settings '{"apiKeyHelper":"echo local","env":{
+      "CLAUDE_CODE_USE_BEDROCK":"0",
+      "ANTHROPIC_BASE_URL":"http://127.0.0.1:4000",
+      "ANTHROPIC_API_KEY":"local",
+      "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144",
+      "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null
+```
+
+**`DISABLE_NON_ESSENTIAL_MODEL_CALLS=1` is doing real work here.** Without it, the same command emitted `[claude-code:unrecognized_model] {"model":"qwen.qwen3-coder-30b-a3b-instruct","query_source":"generate_session_title"}` before the answer: Claude Code makes side calls, such as generating a session title, and an unrecognized model id makes them fail noisily. With the flag set the run returned `"result":"OK","subtype":"success","num_turns":1`.
+
+Setting `ANTHROPIC_API_KEY` also prints a warning that claude.ai connectors are disabled because an auth source takes precedence over your claude.ai login. That is expected on this route and harmless.
+
+## Model on Bedrock directly
+
+Anthropic models need no proxy. Flip Bedrock mode on and name the inference profile:
+
+```bash
+claude -p "Reply with exactly: OK" \
+  --model us.anthropic.claude-haiku-4-5-20251001-v1:0 \
+  --output-format json \
+  --settings '{"env":{"CLAUDE_CODE_USE_BEDROCK":"1","AWS_REGION":"us-east-2"}}' < /dev/null
+```
+
+No `apiKeyHelper` and no base URL: Bedrock mode authenticates from the ambient AWS credential chain. Pin the region in the settings block, not just the environment, so a global settings file cannot flip routing.
+
+**This route reports real costs and real caching.** The verified run returned `"total_cost_usd": 0.0253` with `"cache_creation_input_tokens": 20064` -- Claude Code inserts its own cache-control markers, so a repeated prefix is written once and read back cheaply on later turns. That is why published Claude Code runs on Bedrock show millions of cache-read tokens against a tiny fresh-input count.
+
+## Which route serves which model
+
+| Model | Route | Extra service |
+|---|---|---|
+| Anthropic on Bedrock | `CLAUDE_CODE_USE_BEDROCK=1` + inference profile | none |
+| Any model on your own vLLM server | `ANTHROPIC_BASE_URL` at `:8000` | none, vLLM answers `/v1/messages` |
+| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `ANTHROPIC_BASE_URL` at the mantle proxy | `bedrock-mantle-proxy.sh` |
+
+## Driving a benchmark with this wiring
+
+The harness builds the settings object and the environment per run, so a benchmark run needs only `--provider` and a model; `claude` is the default agent. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for its flags, the permission model, and the auto-compaction detail, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow.
+
+## Related
+
+- [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock and proving it with a live call.
+- [faq/wiring-codex-to-models.md](wiring-codex-to-models.md) and [faq/wiring-omp-to-models.md](wiring-omp-to-models.md) -- the same three routes for the other two harnesses.
+- [cost-per-task-methodology.md](../cost-per-task-methodology.md) -- why `total_cost_usd` is trustworthy on Bedrock and meaningless against a self-hosted endpoint.