From 2d03c8fb58d4e10ca3ac9cc065ce3a0dd07b6529 Mon Sep 17 00:00:00 2001 From: shekharprateek Date: Tue, 15 Sep 2026 06:50:58 -0600 Subject: [PATCH] Add codex-setup and docs/faq, and correct the --agent list The fork gained these after the earlier import, so they were missing here. docs/codex-setup.md -- codex is fully wired in the harness (68 references in run-swe-headless.py, and run-e2e-benchmark.sh accepts --agent codex) but had no setup page, so a user had no way to learn it was an option. docs/faq/ (4 files) -- one page per agent for wiring it to a model, same format throughout, covering Claude Code, omp and codex. Corrected the --agent list in two places, both of which omitted codex: - README.md: --agent claude|pi|omp|kiro -> claude|pi|omp|kiro|codex - run-e2e-benchmark.sh line 84: the inline comment still read "claude (Claude Code) or pi (pi agent)", predating omp, kiro and codex. The script's own validation at line 167 already listed all five. README documentation map gained rows for docs/faq/, docs/codex-setup.md, and docs/omp-setup.md (the last was imported earlier but never listed). Still not imported: docs/release-notes/0.2.0.md, like 0.1.0.md, sources its figures from docs/metrics/pareto-frontier-*.json and quotes per-model scores and per-tier breakdowns directly. The per-harness guides (harness-pi-swe2.md and friends) are results reports, not onboarding docs -- harness-pi-swe2.md opens "# Results: pi harness (swe2)" and is a per-model table of scores, costs and token counts. Onboarding for those harnesses is already covered by omp-setup.md, kiro-cli-setup.md, codex-setup.md, docs/faq/ and benchmarks/docs/path-*.md. Verified: every relative link in README.md resolves, and every internal link across docs/ including docs/faq/ resolves. --- README.md | 5 +- benchmarks/scripts/run-e2e-benchmark.sh | 2 +- docs/codex-setup.md | 69 ++++++++++ docs/faq/README.md | 17 +++ docs/faq/wiring-claude-code-to-models.md | 125 +++++++++++++++++ docs/faq/wiring-codex-to-models.md | 164 +++++++++++++++++++++++ docs/faq/wiring-omp-to-models.md | 149 ++++++++++++++++++++ 7 files changed, 529 insertions(+), 2 deletions(-) create mode 100644 docs/codex-setup.md create mode 100644 docs/faq/README.md create mode 100644 docs/faq/wiring-claude-code-to-models.md create mode 100644 docs/faq/wiring-codex-to-models.md create mode 100644 docs/faq/wiring-omp-to-models.md diff --git a/README.md b/README.md index 891d8d5..da3f554 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha /benchmark provider=bedrock model=claude-opus-5 dataset=dataset/mcp-gateway-registry-v2.yaml agent=omp ``` -`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set. +`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set. ## Step 2 — Developers install the skill @@ -120,6 +120,9 @@ Where to read more, by topic: | [benchmarks/docs/path-anthropic-on-bedrock.md](benchmarks/docs/path-anthropic-on-bedrock.md) | Path 1 setup: benchmarking the Anthropic family (Claude Opus/Sonnet/Haiku) directly on Amazon Bedrock. | | [benchmarks/docs/path-open-weight-on-bedrock-litellm.md](benchmarks/docs/path-open-weight-on-bedrock-litellm.md) | Path 2 setup: open-weight models on Amazon Bedrock through the LiteLLM proxy. | | [benchmarks/docs/path-self-hosted-vllm.md](benchmarks/docs/path-self-hosted-vllm.md) | Path 3 setup: self-hosting a model on vLLM and pointing the harness at it. | +| [docs/faq/](docs/faq/) | Wiring each agent to a model, one page per agent (Claude Code, omp, codex), same format throughout: every provider route with the exact command. | +| [docs/omp-setup.md](docs/omp-setup.md) | The omp harness: install, the provider flags, auto-approve, and the JSON event stream the harness reads for metrics. | +| [docs/codex-setup.md](docs/codex-setup.md) | The codex harness: install, `codex exec` headless use, the provider block an endpoint run needs, and why the sandbox must be bypassed on a benchmark host. | | [docs/kiro-cli-setup.md](docs/kiro-cli-setup.md) | The kiro-cli harness: install, sign-in, headless use, and the Bedrock-managed-only constraint. | | [benchmarks/docs/end-to-end-self-hosted-run.md](benchmarks/docs/end-to-end-self-hosted-run.md) | The full manual run-book for an end-to-end self-hosted benchmark. | | [self-hosted/vllm/README.md](self-hosted/vllm/README.md) | Standing up a vLLM server: install, tensor parallelism, tool-call parsers, and the serving-config reference. | diff --git a/benchmarks/scripts/run-e2e-benchmark.sh b/benchmarks/scripts/run-e2e-benchmark.sh index 37271a3..a883c7c 100755 --- a/benchmarks/scripts/run-e2e-benchmark.sh +++ b/benchmarks/scripts/run-e2e-benchmark.sh @@ -81,7 +81,7 @@ REPO_ROOT="$(dirname "$BENCHMARKS_DIR")" VLLM_DIR="$REPO_ROOT/self-hosted/vllm" # Defaults -AGENT="claude" # coding agent: claude (Claude Code) or pi (pi agent) +AGENT="claude" # coding agent: claude, pi, omp, kiro, or codex SKILL="swe3" # SWE skill: swe3 (single-agent, default) or swe2 (multi-agent) PROVIDER="" MODEL="" diff --git a/docs/codex-setup.md b/docs/codex-setup.md new file mode 100644 index 0000000..8e3f94c --- /dev/null +++ b/docs/codex-setup.md @@ -0,0 +1,69 @@ +# Installing codex and pointing it at a model + +`codex` does two jobs here, and they are configured separately. It is the **judge** that scores every benchmark run, always against Amazon Bedrock, and it is one of five **coding harnesses** (`--agent codex`) that can drive a task on any hosting path. This page installs it, proves it works, and states the one constraint that decides which models it can reach. The per-path wiring recipes live in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md) + +## Install + +```bash +npm install -g @openai/codex +codex --version # >= 0.144 for the native amazon-bedrock provider +``` + +The `setup-machine` skill installs it as part of the core stack, so on a box bootstrapped with [setup-machine.sh](../.claude/skills/setup-machine/setup-machine.sh) it is already present. Installing the CLI does **not** wire it to Bedrock. + +## Wire it to Bedrock and prove it + +codex ships a native `amazon-bedrock` provider that authenticates from the AWS credential chain. No proxy, no bearer token; the LiteLLM route older notes describe is legacy. The config lives at `~/.codex/config.toml`: + +```toml +model_provider = "amazon-bedrock" +model_providers.amazon-bedrock.aws.region = "us-east-2" +model = "openai.gpt-5.6-sol" +``` + +Full walk-through, including the `claude` side: [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md). + +**Prove it with a real call before starting a long run.** Working AWS credentials are not sufficient: an unconfigured codex ignores them and 401s against `api.openai.com`, and the failure surfaces only after the harness has finished generating. + +```bash +codex exec --skip-git-repo-check "Reply with exactly: JUDGE OK" +``` + +## The constraint that decides everything else + +**codex speaks the OpenAI Responses API and nothing else.** codex 0.153.4 removed the chat-completions wire and rejects the fallback: + +``` +Error loading config.toml: `wire_api = "chat"` is no longer supported. +How to fix: set `wire_api = "responses"` in your provider config. +``` + +Amazon Bedrock answers `/v1/responses` for the **`openai.*` family only**. Everything downstream follows: + +| Model you want to drive | Provider | Reachable directly? | +|---|---|---| +| `openai.*` on Bedrock | `bedrock` | Yes, this same native provider | +| Any model on your own vLLM server | `endpoint` (`--provider vllm`) | Yes, if its tool parser accepts Responses-shaped tools | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `endpoint` (`--provider litellm`) | No. Bedrock rejects Responses for these; a LiteLLM bridge is required | +| Anthropic models | `bedrock` | Yes, though Claude Code is the better-measured harness for them | + +The commands for each row are in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md) + +## Known failure modes + +Each of these cost real debugging time. The FAQ carries the detail and the fix. + +| Symptom | Cause | +|---|---| +| Run hangs before the first request, log stops at `Reading additional input from stdin...` | `codex exec` blocks on an open, empty stdin. Launch every run with `< /dev/null` | +| `404 The model '' does not exist` on an endpoint run | codex 0.153.4 ignores `OPENAI_BASE_URL` and used the config's provider, which on a judge-configured box is Bedrock (issue #183) | +| Every request retried five times, stream ends with no `response.completed` | The vLLM tool parser reads the nested chat-completions tool shape and aborts on the flat Responses shape. Use `qwen3_coder` or `hermes` | +| `Model metadata not found. Defaulting to fallback metadata` | codex does not know a self-hosted model's window. Pass `--context-window` | +| `does not support the '/openai/v1/responses' API` | An open-weight Bedrock model reached without the bridge | +| `total_cost_usd` is null | The model has no row in [bedrock_pricing.py](../benchmarks/scripts/bedrock_pricing.py). codex reports tokens but no billed cost, so an unpriced model records null rather than a misleading zero | + +## Related + +- [faq/wiring-codex-to-models.md](faq/wiring-codex-to-models.md) -- the per-path recipes, the parser table, and why prompt caching does not happen on the Bedrock bridge. +- [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock. +- [harness-reference.md](../benchmarks/docs/harness-reference.md#choosing-the-agent) -- every supported agent, its providers, and its cost basis. diff --git a/docs/faq/README.md b/docs/faq/README.md new file mode 100644 index 0000000..3c903cf --- /dev/null +++ b/docs/faq/README.md @@ -0,0 +1,17 @@ +# FAQ + +Answers to questions that come up while running this benchmark. One file per question, listed here. + +| Question | Answer | +|----------|--------| +| How do I wire codex to a model: an open-weight model on Bedrock, one I serve on vLLM, or an OpenAI model on Bedrock? | [wiring-codex-to-models.md](wiring-codex-to-models.md) | +| How do I wire omp to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-omp-to-models.md](wiring-omp-to-models.md) | +| How do I wire Claude Code to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-claude-code-to-models.md](wiring-claude-code-to-models.md) | + +Add an entry by writing the file next to this one and adding a row above. Keep the question in the reader's words, not the code's: someone searching this index knows what they are trying to do, not what the harness calls it. + +For setup and reference material rather than questions, start at the [documentation map](../../README.md#documentation-map) or [AGENTS.md](../../AGENTS.md). + +--- + +[< Back to the README](../../README.md) diff --git a/docs/faq/wiring-claude-code-to-models.md b/docs/faq/wiring-claude-code-to-models.md new file mode 100644 index 0000000..dcd5686 --- /dev/null +++ b/docs/faq/wiring-claude-code-to-models.md @@ -0,0 +1,125 @@ +# How do I wire Claude Code to a model? + +Three ways to point `claude` at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or a model on Bedrock directly. Each ends with a `claude -p` call you can run by hand. + +For wiring `claude` and the `codex` judge to Bedrock on a fresh box, see [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md). This page assumes `claude --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with Claude Code `2.1.266`. Each `claude -p` example returned `OK`: the Bedrock one against `claude-haiku-4-5`, the vLLM one against a live `qwen3.6-35b-fp8` server, and the open-weight-on-Bedrock one against `qwen.qwen3-coder-30b-a3b-instruct` through the LiteLLM mantle proxy. + +## The three facts everything follows from + +**Claude Code speaks the Anthropic Messages API.** Anything else must answer `POST /v1/messages`. vLLM does, which is why a self-hosted model needs no bridge here; Bedrock's non-Anthropic models do not, which is why they need one. + +**`--settings` wins, and you always want to pass it.** A settings object's `env` block takes precedence over process environment variables, including anything in your global `~/.claude/settings.json`. A global file pinning `CLAUDE_CODE_USE_BEDROCK=1` will otherwise redirect a run meant for a local endpoint straight to Bedrock, which rejects the local model id. Passing `--settings` is what reliably wins. + +**Claude Code cannot detect a custom model's context window.** Against a custom `ANTHROPIC_BASE_URL` it has no window to compact against, so on a long task the conversation grows until the endpoint rejects it, and Claude Code treats that 500 as transient and retries it forever. Set `CLAUDE_CODE_AUTO_COMPACT_WINDOW` to the served window on every endpoint run. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +vLLM answers the Anthropic Messages API as well as the OpenAI one, so nothing sits between Claude Code and the server. Confirm it before blaming the agent: + +```bash +curl -s http://127.0.0.1:8000/v1/messages \ + -H 'content-type: application/json' -H 'x-api-key: local' \ + -H 'anthropic-version: 2023-06-01' \ + -d '{"model":"qwen3.6-35b-fp8","max_tokens":16,"messages":[{"role":"user","content":"Reply with exactly: OK"}]}' +``` + +**2. Run it.** + +```bash +claude -p "Reply with exactly: OK" \ + --model qwen3.6-35b-fp8 \ + --output-format json \ + --settings '{"apiKeyHelper":"echo local","env":{ + "CLAUDE_CODE_USE_BEDROCK":"0", + "ANTHROPIC_BASE_URL":"http://127.0.0.1:8000", + "ANTHROPIC_API_KEY":"local", + "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144", + "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null +``` + +**`apiKeyHelper` is required even though the server ignores the value.** Without a token source Claude Code refuses with "Not logged in". `echo local` satisfies it. + +**Do not trust `total_cost_usd` on this route.** The verified run against a local server reported `"total_cost_usd": 0.1138` for a model that costs nothing per token, because Claude Code prices from its own assumptions for an unrecognized model. It also reported `"contextWindow": 200000` while the server was serving 262,144. For a self-hosted model the repo derives cost from GPU-seconds instead, never from this field ([cost-per-task-methodology.md](../cost-per-task-methodology.md)). + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +Bedrock's non-Anthropic models do not answer the Messages API. Worse, sent through Bedrock's Converse path they return their native tool-call tokens as plain text, so Claude Code never sees a structured `tool_use` block and an agentic run stalls at one turn with zero artifacts. The fix is a proxy that translates Anthropic Messages to OpenAI chat completions and on to Bedrock's OpenAI-compatible `bedrock-mantle` endpoint, which parses those tokens into real tool calls. The repo ships it. + +**1. Start the proxy.** + +```bash +cd benchmarks +./scripts/bedrock-mantle-proxy.sh # installs deps, mints a 12h token, listens on :4000 +./scripts/bedrock-mantle-proxy.sh --status +``` + +It mints the bearer token from your ambient AWS credentials and holds it server-side; clients send a throwaway key. Models come from [config/litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml), and `curl -s http://127.0.0.1:4000/v1/models` lists them. + +**2. Run it.** Same shape as the vLLM route, pointed at the proxy: + +```bash +claude -p "Reply with exactly: OK" \ + --model qwen.qwen3-coder-30b-a3b-instruct \ + --output-format json \ + --settings '{"apiKeyHelper":"echo local","env":{ + "CLAUDE_CODE_USE_BEDROCK":"0", + "ANTHROPIC_BASE_URL":"http://127.0.0.1:4000", + "ANTHROPIC_API_KEY":"local", + "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144", + "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null +``` + +**`DISABLE_NON_ESSENTIAL_MODEL_CALLS=1` is doing real work here.** Without it, the same command emitted `[claude-code:unrecognized_model] {"model":"qwen.qwen3-coder-30b-a3b-instruct","query_source":"generate_session_title"}` before the answer: Claude Code makes side calls, such as generating a session title, and an unrecognized model id makes them fail noisily. With the flag set the run returned `"result":"OK","subtype":"success","num_turns":1`. + +Setting `ANTHROPIC_API_KEY` also prints a warning that claude.ai connectors are disabled because an auth source takes precedence over your claude.ai login. That is expected on this route and harmless. + +## Model on Bedrock directly + +Anthropic models need no proxy. Flip Bedrock mode on and name the inference profile: + +```bash +claude -p "Reply with exactly: OK" \ + --model us.anthropic.claude-haiku-4-5-20251001-v1:0 \ + --output-format json \ + --settings '{"env":{"CLAUDE_CODE_USE_BEDROCK":"1","AWS_REGION":"us-east-2"}}' < /dev/null +``` + +No `apiKeyHelper` and no base URL: Bedrock mode authenticates from the ambient AWS credential chain. Pin the region in the settings block, not just the environment, so a global settings file cannot flip routing. + +**This route reports real costs and real caching.** The verified run returned `"total_cost_usd": 0.0253` with `"cache_creation_input_tokens": 20064` -- Claude Code inserts its own cache-control markers, so a repeated prefix is written once and read back cheaply on later turns. That is why published Claude Code runs on Bedrock show millions of cache-read tokens against a tiny fresh-input count. + +## Which route serves which model + +| Model | Route | Extra service | +|---|---|---| +| Anthropic on Bedrock | `CLAUDE_CODE_USE_BEDROCK=1` + inference profile | none | +| Any model on your own vLLM server | `ANTHROPIC_BASE_URL` at `:8000` | none, vLLM answers `/v1/messages` | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `ANTHROPIC_BASE_URL` at the mantle proxy | `bedrock-mantle-proxy.sh` | + +## Driving a benchmark with this wiring + +The harness builds the settings object and the environment per run, so a benchmark run needs only `--provider` and a model; `claude` is the default agent. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for its flags, the permission model, and the auto-compaction detail, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock and proving it with a live call. +- [faq/wiring-codex-to-models.md](wiring-codex-to-models.md) and [faq/wiring-omp-to-models.md](wiring-omp-to-models.md) -- the same three routes for the other two harnesses. +- [cost-per-task-methodology.md](../cost-per-task-methodology.md) -- why `total_cost_usd` is trustworthy on Bedrock and meaningless against a self-hosted endpoint. diff --git a/docs/faq/wiring-codex-to-models.md b/docs/faq/wiring-codex-to-models.md new file mode 100644 index 0000000..436c040 --- /dev/null +++ b/docs/faq/wiring-codex-to-models.md @@ -0,0 +1,164 @@ +# How do I wire codex to a model? + +Three ways to point `codex` at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or an OpenAI model on Bedrock. Each ends with a `codex exec` call you can run by hand. + +For installing codex and wiring it to Bedrock, see [codex-setup.md](../codex-setup.md). This page assumes `codex --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with `codex-cli 0.153.4`. Each `codex exec` example returned `OK`: the Bedrock one against `openai.gpt-5.6-luna`, the vLLM one against a live `qwen3.6-35b-fp8` server at a 262,144-token window, and the bridge one against `qwen.qwen3-coder-30b-a3b-v1:0` through LiteLLM on port 4002. Where a command emits a warning anyway, this page says so. + +## The one fact everything follows from + +**codex speaks the OpenAI Responses API and nothing else.** codex 0.153.4 removed the chat-completions wire and rejects the fallback: + +``` +Error loading config.toml: `wire_api = "chat"` is no longer supported. +How to fix: set `wire_api = "responses"` in your provider config. +``` + +So whatever serves the model must answer `POST /v1/responses`. Bedrock does that for the `openai.*` family only, which is why an open-weight Bedrock model needs a bridge and your own vLLM server does not. + +**Close stdin on every call.** `codex exec` reads stdin even when the prompt arrives as an argument, and blocks forever on an open, empty one. Every example below ends with `< /dev/null`. + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +Bedrock will not answer Responses for these: + +``` +The model 'qwen.qwen3-coder-30b-a3b-v1:0' does not support the '/openai/v1/responses' API +``` + +The model works there; it just does not answer that API. Put a LiteLLM proxy in front that speaks Responses to codex and Converse to Bedrock, using LiteLLM's native `bedrock/` provider, which authenticates from ambient AWS credentials with no bearer token. + +**1. Write the proxy config.** + +```yaml +# litellm-responses-bedrock.yaml +model_list: + - model_name: qwen.qwen3-coder-30b-a3b-instruct + litellm_params: + model: bedrock/qwen.qwen3-coder-30b-a3b-v1:0 + aws_region_name: us-east-2 + +litellm_settings: + drop_params: true +``` + +**2. Start it.** + +```bash +uv run --with 'litellm[proxy]' litellm --config litellm-responses-bedrock.yaml \ + --host 127.0.0.1 --port 4002 +``` + +**3. Point codex at it.** + +```bash +export OPENAI_API_KEY=local # the proxy ignores the value; codex requires the variable + +codex exec --json --skip-git-repo-check \ + --model qwen.qwen3-coder-30b-a3b-instruct \ + -c model_provider=litellm \ + -c model_providers.litellm.name=litellm \ + -c model_providers.litellm.base_url=http://127.0.0.1:4002/v1 \ + -c model_providers.litellm.wire_api=responses \ + -c model_providers.litellm.env_key=OPENAI_API_KEY \ + -- "Reply with exactly: OK" < /dev/null +``` + +**Two traps.** + +The committed [litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml) **cannot** serve this path. It registers each model as `openai/` against the `bedrock-mantle` endpoint, so LiteLLM forwards `/v1/responses` untouched and mantle rejects it for a non-`openai.*` model. That config exists for Claude Code, which speaks the Anthropic Messages API. Write a separate config with the `bedrock/` provider. + +Pin nothing older than current LiteLLM. The range the mantle script uses, `litellm[proxy]>=1.72,<1.84`, fails the bridge with `500 'NoneType' object has no attribute 'encode'`. + +### Prompt caching does not happen on this route + +Bedrock Converse caching is opt-in per request: the caller must place `cachePoint` blocks in the message content. codex has no cache-control concept and LiteLLM's bridge adds none, so `cache_read` is 0 and every token is billed fresh. + +Two separate limits hide behind that zero: + +| Model | `cachePoint` on Converse | +|---|---| +| `qwen.qwen3-coder-30b-a3b-v1:0` | `AccessDeniedException: You invoked an unsupported model or your request did not allow prompt caching` | +| `us.anthropic.claude-haiku-4-5` | cold call writes `cacheWriteInputTokens: 5204`; warm call reads `cacheReadInputTokens: 5204` | + +Qwen cannot cache on Bedrock at all, and a model that can still will not through this bridge. A self-hosted vLLM server is the opposite: its prefix caching is automatic and server-side, needing nothing from the client. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +**2. Point codex at it.** + +```bash +export OPENAI_API_KEY=local # vLLM ignores the value; codex requires the variable + +codex exec --json --skip-git-repo-check \ + --model qwen3.6-35b-fp8 \ + -c model_provider=vllm \ + -c model_providers.vllm.name=vllm \ + -c model_providers.vllm.base_url=http://127.0.0.1:8000/v1 \ + -c model_providers.vllm.wire_api=responses \ + -c model_providers.vllm.env_key=OPENAI_API_KEY \ + -c model_context_window=262144 \ + -- "Reply with exactly: OK" < /dev/null +``` + +**The base URL must travel in the provider block.** codex 0.153.4 ignores `OPENAI_BASE_URL` and takes its base URL from whichever provider its config selects. Exporting that variable alone sends the call wherever `~/.codex/config.toml` points, which on a judge-configured machine is Bedrock, producing the misleading `404 The model 'minicpm5-2b' does not exist` (issue #183). + +**The tool-call parser must accept Responses-shaped tools.** The Responses API sends a flat tool definition, `{"type": "function", "name": ...}`; chat completions nests it under `"function"`. Seven of vLLM 0.29.0's parsers read only the nested shape, abort tool extraction on the flat one, end the stream with no `response.completed`, and make codex retry every request five times before failing the turn: + +| Works with codex | Breaks with codex | +|---|---| +| `qwen3_coder`, `hermes` | `minicpm5xml`, `dots`, `hy_v3`, `hy_v4`, `rust`, `step3`, `step3p5` | + +**Pass `model_context_window`.** A self-hosted model is unknown to codex, so it sizes the conversation from fallback metadata unless told the real window. Passing the flag does **not** silence the warning -- the run above still logged `Model metadata for 'qwen3.6-35b-fp8' not found. Defaulting to fallback metadata` with `model_context_window=262144` set. Treat that line as noise; what matters is that the window codex plans against matches the one vLLM booted. + +## OpenAI model on Bedrock + +Nothing to bridge. Bedrock answers Responses natively for the `openai.*` family, so codex needs only its own provider and a region: + +```bash +codex exec --json --skip-git-repo-check \ + --model openai.gpt-5.6-luna \ + -c model_provider=amazon-bedrock \ + -c model_providers.amazon-bedrock.aws.region=us-east-2 \ + -- "Reply with exactly: OK" < /dev/null +``` + +Authentication comes from the ambient AWS credential chain: no proxy, no bearer token. Set the same two values in `~/.codex/config.toml` to make them the default for every call, which is how the judge is configured ([codex-setup.md](../codex-setup.md)). + +**This is the one route where codex gets prompt caching.** The call above reported `cache_write_input_tokens: 8761` on a cold run, so a repeated prefix is read back cheaply on the next call. Bedrock's Responses implementation handles the cache itself, with nothing for the caller to place -- unlike Converse behind the LiteLLM bridge, where caching needs `cachePoint` blocks that nothing in the chain inserts. + +## Which route serves which model + +| Model | Route | Bridge needed | +|---|---|---| +| `openai.*` on Bedrock | native `amazon-bedrock` provider | no | +| Any model on your own vLLM server | provider block at `:8000` | no, but the tool parser must be Responses-safe | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | LiteLLM with the native `bedrock/` provider | yes | +| Anthropic models on Bedrock | native `amazon-bedrock` provider | no | + +## Driving a benchmark with this wiring + +The harness builds these provider blocks itself, so a benchmark run needs only `--agent codex` and a provider. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for the agent's flags and cost accounting, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [codex-setup.md](../codex-setup.md) -- installing codex, wiring it to Bedrock, and its failure-mode table. +- [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock and proving it with a live call. +- [self-hosted/vllm/README.md](../../self-hosted/vllm/README.md) -- installing vLLM and the serving-config reference. diff --git a/docs/faq/wiring-omp-to-models.md b/docs/faq/wiring-omp-to-models.md new file mode 100644 index 0000000..5549bad --- /dev/null +++ b/docs/faq/wiring-omp-to-models.md @@ -0,0 +1,149 @@ +# How do I wire omp to a model? + +Three ways to point `omp` (oh-my-pi) at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or a model on Bedrock directly. Each ends with an `omp -p` call you can run by hand. + +For installing omp and the three ways it differs from `pi`, see [omp-setup.md](../omp-setup.md). This page assumes `omp --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with `omp/18.1.15`. Each `omp -p` example returned `OK`: the vLLM one against a live `qwen3.6-35b-fp8` server at a 262,144-token window, the Bedrock one against `claude-haiku-4-5`, and the open-weight-on-Bedrock one against `qwen.qwen3-coder-30b-a3b-instruct` through the LiteLLM mantle proxy. + +## The two facts everything follows from + +**omp's config is YAML, and it lives wherever `PI_CODING_AGENT_DIR` points.** omp is a fork of pi and honours that variable, but where pi reads `models.json`, omp reads `models.yml` for providers and `config.yml` for settings. Pointing the variable at a scratch directory keeps an experiment away from your own `~/.omp`, which is what the harness does per run. + +**A model is named `provider/model`.** The provider is a key in `models.yml` for a custom endpoint, or the built-in `amazon-bedrock` for Bedrock. So `--model vllm/qwen3.6-35b-fp8` and `--model amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0`. + +Two smaller ones that bite: + +- **omp hangs on an inherited stdin.** Redirect on every call: the examples below all end with `< /dev/null`. +- **Put `--` before the prompt.** omp parses options until it sees `--`. A prompt that starts with `---`, which any inlined `SKILL.md` does because of its YAML frontmatter, is otherwise read as an unknown flag. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +**2. Declare the server as a provider.** Write `models.yml` into a scratch agent dir: + +```yaml +# /tmp/omp-vllm/.omp/models.yml +providers: + vllm: + baseUrl: http://127.0.0.1:8000/v1 + api: openai-completions + apiKey: local + models: + - id: qwen3.6-35b-fp8 + name: "vLLM: qwen3.6-35b-fp8" + contextWindow: 262144 + maxTokens: 16000 + cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0} +``` + +**3. Size compaction to the window.** Without this omp fills the context to within its default reserve, and one capped response overflows the endpoint mid-run. Reserve a full response plus about 8K of headroom -- for a 262,144 window and a 16,000-token cap, that is `262144 - (16000 + 8192) = 237952`: + +```yaml +# /tmp/omp-vllm/.omp/config.yml +compaction: + enabled: true + thresholdTokens: 237952 +``` + +**4. Run it.** + +```bash +PI_CODING_AGENT_DIR=/tmp/omp-vllm/.omp \ + omp -p --mode json --no-session --auto-approve \ + --model vllm/qwen3.6-35b-fp8 \ + -- "Reply with exactly: OK" < /dev/null +``` + +`--mode json` gives the pi-shaped event stream; the final `message_end` event carries usage. The verified run reported `"provider":"vllm","model":"qwen3.6-35b-fp8"` with 18,108 input and 6 output tokens. + +**`--no-session` keeps the run ephemeral** so nothing lands in your session history, and **`--auto-approve`** runs tools without an approval gate, which an unattended run needs. + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +omp speaks the OpenAI chat-completions wire through `api: openai-completions`, so it needs an OpenAI-compatible endpoint in front of Bedrock. The repo ships one: [bedrock-mantle-proxy.sh](../../benchmarks/scripts/bedrock-mantle-proxy.sh) starts a LiteLLM proxy over Bedrock's OpenAI-compatible `bedrock-mantle` endpoint, minting a 12-hour bearer token from your ambient AWS credentials and holding it server-side. + +**1. Start the proxy.** + +```bash +cd benchmarks +./scripts/bedrock-mantle-proxy.sh # installs deps, mints the token, listens on :4000 +./scripts/bedrock-mantle-proxy.sh --status +``` + +Models come from [config/litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml); `curl -s http://127.0.0.1:4000/v1/models` lists them. + +**2. Declare it as a provider, with real rates.** + +```yaml +# /tmp/omp-mantle/.omp/models.yml +providers: + mantle: + baseUrl: http://127.0.0.1:4000/v1 + api: openai-completions + apiKey: local + models: + - id: qwen.qwen3-coder-30b-a3b-instruct + name: "Bedrock: qwen3-coder-30b" + contextWindow: 262144 + maxTokens: 16000 + cost: {input: 0.1545, output: 0.618, cacheRead: 0, cacheWrite: 0} +``` + +**3. Run it.** + +```bash +PI_CODING_AGENT_DIR=/tmp/omp-mantle/.omp \ + omp -p --mode json --no-session --auto-approve \ + --model mantle/qwen.qwen3-coder-30b-a3b-instruct \ + -- "Reply with exactly: OK" < /dev/null +``` + +**The `cost` block is not decoration: omp prices the run from it.** The verified run reported `"cost":{"input":0.002824878,"output":0.000001236,...,"total":0.002826114}` for 18,284 input and 2 output tokens, which is exactly those per-1M rates applied. Leave the zeros in for a self-hosted model, where per-token cost is meaningless and the repo derives cost from GPU-seconds instead ([cost-per-task-methodology.md](../cost-per-task-methodology.md)). Put the real Bedrock rates in for a metered model, and omp's own numbers are usable. + +The rates above are Qwen3-Coder-30B's Standard tier, the same ones in [bedrock_pricing.py](../../benchmarks/scripts/bedrock_pricing.py). + +## Model on Bedrock directly + +Bedrock is built in, so no `models.yml` is needed. Name the inference profile after `amazon-bedrock/`: + +```bash +omp -p --mode json --no-session --auto-approve \ + --model amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 \ + -- "Reply with exactly: OK" < /dev/null +``` + +Authentication comes from the ambient AWS credential chain; set `AWS_REGION` if your default region does not host the model. Nothing is written to disk and no key is passed on the command line. + +## Which route serves which model + +| Model | Route | Extra service | +|---|---|---| +| Anthropic (or any native Bedrock model) | `--model amazon-bedrock/` | none | +| Any model on your own vLLM server | a `models.yml` provider at `:8000` | none | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | a `models.yml` provider at the mantle proxy | `bedrock-mantle-proxy.sh` | + +## Driving a benchmark with this wiring + +The harness writes `models.yml` and `config.yml` per run into a scratch `PI_CODING_AGENT_DIR`, so a benchmark run needs only `--agent omp` and a provider. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for the agent's flags and cost accounting, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [omp-setup.md](../omp-setup.md) -- installing omp and the three ways it differs from pi. +- [faq/wiring-codex-to-models.md](wiring-codex-to-models.md) -- the same three routes for codex, which needs the Responses API and therefore a different bridge. +- [cost-per-task-methodology.md](../cost-per-task-methodology.md) -- why a metered Bedrock bill and a hardware-derived self-hosted figure do not compare as raw dollars.