diff --git a/README.md b/README.md index 891d8d5..da3f554 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ One command per model. The `/benchmark` skill runs the pre-flight checks, the ha /benchmark provider=bedrock model=claude-opus-5 dataset=dataset/mcp-gateway-registry-v2.yaml agent=omp ``` -`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set. +`agent` names the coding agent that drives the task and defaults to `claude`. The same flow runs headless from [`run-e2e-benchmark.sh`](benchmarks/scripts/run-e2e-benchmark.sh) (`--provider bedrock|litellm|vllm --model ... --dataset ... --agent claude|pi|omp|kiro|codex --skill swe2|swe3`). Repeat across your model list, then the generators plot the cost/quality frontier for your repo and model set. ## Step 2 — Developers install the skill @@ -120,6 +120,9 @@ Where to read more, by topic: | [benchmarks/docs/path-anthropic-on-bedrock.md](benchmarks/docs/path-anthropic-on-bedrock.md) | Path 1 setup: benchmarking the Anthropic family (Claude Opus/Sonnet/Haiku) directly on Amazon Bedrock. | | [benchmarks/docs/path-open-weight-on-bedrock-litellm.md](benchmarks/docs/path-open-weight-on-bedrock-litellm.md) | Path 2 setup: open-weight models on Amazon Bedrock through the LiteLLM proxy. | | [benchmarks/docs/path-self-hosted-vllm.md](benchmarks/docs/path-self-hosted-vllm.md) | Path 3 setup: self-hosting a model on vLLM and pointing the harness at it. | +| [docs/faq/](docs/faq/) | Wiring each agent to a model, one page per agent (Claude Code, omp, codex), same format throughout: every provider route with the exact command. | +| [docs/omp-setup.md](docs/omp-setup.md) | The omp harness: install, the provider flags, auto-approve, and the JSON event stream the harness reads for metrics. | +| [docs/codex-setup.md](docs/codex-setup.md) | The codex harness: install, `codex exec` headless use, the provider block an endpoint run needs, and why the sandbox must be bypassed on a benchmark host. | | [docs/kiro-cli-setup.md](docs/kiro-cli-setup.md) | The kiro-cli harness: install, sign-in, headless use, and the Bedrock-managed-only constraint. | | [benchmarks/docs/end-to-end-self-hosted-run.md](benchmarks/docs/end-to-end-self-hosted-run.md) | The full manual run-book for an end-to-end self-hosted benchmark. | | [self-hosted/vllm/README.md](self-hosted/vllm/README.md) | Standing up a vLLM server: install, tensor parallelism, tool-call parsers, and the serving-config reference. | diff --git a/benchmarks/scripts/run-e2e-benchmark.sh b/benchmarks/scripts/run-e2e-benchmark.sh index 37271a3..a883c7c 100755 --- a/benchmarks/scripts/run-e2e-benchmark.sh +++ b/benchmarks/scripts/run-e2e-benchmark.sh @@ -81,7 +81,7 @@ REPO_ROOT="$(dirname "$BENCHMARKS_DIR")" VLLM_DIR="$REPO_ROOT/self-hosted/vllm" # Defaults -AGENT="claude" # coding agent: claude (Claude Code) or pi (pi agent) +AGENT="claude" # coding agent: claude, pi, omp, kiro, or codex SKILL="swe3" # SWE skill: swe3 (single-agent, default) or swe2 (multi-agent) PROVIDER="" MODEL="" diff --git a/docs/codex-setup.md b/docs/codex-setup.md new file mode 100644 index 0000000..8e3f94c --- /dev/null +++ b/docs/codex-setup.md @@ -0,0 +1,69 @@ +# Installing codex and pointing it at a model + +`codex` does two jobs here, and they are configured separately. It is the **judge** that scores every benchmark run, always against Amazon Bedrock, and it is one of five **coding harnesses** (`--agent codex`) that can drive a task on any hosting path. This page installs it, proves it works, and states the one constraint that decides which models it can reach. The per-path wiring recipes live in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md) + +## Install + +```bash +npm install -g @openai/codex +codex --version # >= 0.144 for the native amazon-bedrock provider +``` + +The `setup-machine` skill installs it as part of the core stack, so on a box bootstrapped with [setup-machine.sh](../.claude/skills/setup-machine/setup-machine.sh) it is already present. Installing the CLI does **not** wire it to Bedrock. + +## Wire it to Bedrock and prove it + +codex ships a native `amazon-bedrock` provider that authenticates from the AWS credential chain. No proxy, no bearer token; the LiteLLM route older notes describe is legacy. The config lives at `~/.codex/config.toml`: + +```toml +model_provider = "amazon-bedrock" +model_providers.amazon-bedrock.aws.region = "us-east-2" +model = "openai.gpt-5.6-sol" +``` + +Full walk-through, including the `claude` side: [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md). + +**Prove it with a real call before starting a long run.** Working AWS credentials are not sufficient: an unconfigured codex ignores them and 401s against `api.openai.com`, and the failure surfaces only after the harness has finished generating. + +```bash +codex exec --skip-git-repo-check "Reply with exactly: JUDGE OK" +``` + +## The constraint that decides everything else + +**codex speaks the OpenAI Responses API and nothing else.** codex 0.153.4 removed the chat-completions wire and rejects the fallback: + +``` +Error loading config.toml: `wire_api = "chat"` is no longer supported. +How to fix: set `wire_api = "responses"` in your provider config. +``` + +Amazon Bedrock answers `/v1/responses` for the **`openai.*` family only**. Everything downstream follows: + +| Model you want to drive | Provider | Reachable directly? | +|---|---|---| +| `openai.*` on Bedrock | `bedrock` | Yes, this same native provider | +| Any model on your own vLLM server | `endpoint` (`--provider vllm`) | Yes, if its tool parser accepts Responses-shaped tools | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `endpoint` (`--provider litellm`) | No. Bedrock rejects Responses for these; a LiteLLM bridge is required | +| Anthropic models | `bedrock` | Yes, though Claude Code is the better-measured harness for them | + +The commands for each row are in the FAQ: [How do I wire codex to a model?](faq/wiring-codex-to-models.md) + +## Known failure modes + +Each of these cost real debugging time. The FAQ carries the detail and the fix. + +| Symptom | Cause | +|---|---| +| Run hangs before the first request, log stops at `Reading additional input from stdin...` | `codex exec` blocks on an open, empty stdin. Launch every run with `< /dev/null` | +| `404 The model '' does not exist` on an endpoint run | codex 0.153.4 ignores `OPENAI_BASE_URL` and used the config's provider, which on a judge-configured box is Bedrock (issue #183) | +| Every request retried five times, stream ends with no `response.completed` | The vLLM tool parser reads the nested chat-completions tool shape and aborts on the flat Responses shape. Use `qwen3_coder` or `hermes` | +| `Model metadata not found. Defaulting to fallback metadata` | codex does not know a self-hosted model's window. Pass `--context-window` | +| `does not support the '/openai/v1/responses' API` | An open-weight Bedrock model reached without the bridge | +| `total_cost_usd` is null | The model has no row in [bedrock_pricing.py](../benchmarks/scripts/bedrock_pricing.py). codex reports tokens but no billed cost, so an unpriced model records null rather than a misleading zero | + +## Related + +- [faq/wiring-codex-to-models.md](faq/wiring-codex-to-models.md) -- the per-path recipes, the parser table, and why prompt caching does not happen on the Bedrock bridge. +- [agent-cli-bedrock-setup.md](../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock. +- [harness-reference.md](../benchmarks/docs/harness-reference.md#choosing-the-agent) -- every supported agent, its providers, and its cost basis. diff --git a/docs/faq/README.md b/docs/faq/README.md new file mode 100644 index 0000000..3c903cf --- /dev/null +++ b/docs/faq/README.md @@ -0,0 +1,17 @@ +# FAQ + +Answers to questions that come up while running this benchmark. One file per question, listed here. + +| Question | Answer | +|----------|--------| +| How do I wire codex to a model: an open-weight model on Bedrock, one I serve on vLLM, or an OpenAI model on Bedrock? | [wiring-codex-to-models.md](wiring-codex-to-models.md) | +| How do I wire omp to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-omp-to-models.md](wiring-omp-to-models.md) | +| How do I wire Claude Code to a model: an open-weight model on Bedrock, one I serve on vLLM, or a model on Bedrock directly? | [wiring-claude-code-to-models.md](wiring-claude-code-to-models.md) | + +Add an entry by writing the file next to this one and adding a row above. Keep the question in the reader's words, not the code's: someone searching this index knows what they are trying to do, not what the harness calls it. + +For setup and reference material rather than questions, start at the [documentation map](../../README.md#documentation-map) or [AGENTS.md](../../AGENTS.md). + +--- + +[< Back to the README](../../README.md) diff --git a/docs/faq/wiring-claude-code-to-models.md b/docs/faq/wiring-claude-code-to-models.md new file mode 100644 index 0000000..dcd5686 --- /dev/null +++ b/docs/faq/wiring-claude-code-to-models.md @@ -0,0 +1,125 @@ +# How do I wire Claude Code to a model? + +Three ways to point `claude` at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or a model on Bedrock directly. Each ends with a `claude -p` call you can run by hand. + +For wiring `claude` and the `codex` judge to Bedrock on a fresh box, see [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md). This page assumes `claude --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with Claude Code `2.1.266`. Each `claude -p` example returned `OK`: the Bedrock one against `claude-haiku-4-5`, the vLLM one against a live `qwen3.6-35b-fp8` server, and the open-weight-on-Bedrock one against `qwen.qwen3-coder-30b-a3b-instruct` through the LiteLLM mantle proxy. + +## The three facts everything follows from + +**Claude Code speaks the Anthropic Messages API.** Anything else must answer `POST /v1/messages`. vLLM does, which is why a self-hosted model needs no bridge here; Bedrock's non-Anthropic models do not, which is why they need one. + +**`--settings` wins, and you always want to pass it.** A settings object's `env` block takes precedence over process environment variables, including anything in your global `~/.claude/settings.json`. A global file pinning `CLAUDE_CODE_USE_BEDROCK=1` will otherwise redirect a run meant for a local endpoint straight to Bedrock, which rejects the local model id. Passing `--settings` is what reliably wins. + +**Claude Code cannot detect a custom model's context window.** Against a custom `ANTHROPIC_BASE_URL` it has no window to compact against, so on a long task the conversation grows until the endpoint rejects it, and Claude Code treats that 500 as transient and retries it forever. Set `CLAUDE_CODE_AUTO_COMPACT_WINDOW` to the served window on every endpoint run. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +vLLM answers the Anthropic Messages API as well as the OpenAI one, so nothing sits between Claude Code and the server. Confirm it before blaming the agent: + +```bash +curl -s http://127.0.0.1:8000/v1/messages \ + -H 'content-type: application/json' -H 'x-api-key: local' \ + -H 'anthropic-version: 2023-06-01' \ + -d '{"model":"qwen3.6-35b-fp8","max_tokens":16,"messages":[{"role":"user","content":"Reply with exactly: OK"}]}' +``` + +**2. Run it.** + +```bash +claude -p "Reply with exactly: OK" \ + --model qwen3.6-35b-fp8 \ + --output-format json \ + --settings '{"apiKeyHelper":"echo local","env":{ + "CLAUDE_CODE_USE_BEDROCK":"0", + "ANTHROPIC_BASE_URL":"http://127.0.0.1:8000", + "ANTHROPIC_API_KEY":"local", + "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144", + "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null +``` + +**`apiKeyHelper` is required even though the server ignores the value.** Without a token source Claude Code refuses with "Not logged in". `echo local` satisfies it. + +**Do not trust `total_cost_usd` on this route.** The verified run against a local server reported `"total_cost_usd": 0.1138` for a model that costs nothing per token, because Claude Code prices from its own assumptions for an unrecognized model. It also reported `"contextWindow": 200000` while the server was serving 262,144. For a self-hosted model the repo derives cost from GPU-seconds instead, never from this field ([cost-per-task-methodology.md](../cost-per-task-methodology.md)). + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +Bedrock's non-Anthropic models do not answer the Messages API. Worse, sent through Bedrock's Converse path they return their native tool-call tokens as plain text, so Claude Code never sees a structured `tool_use` block and an agentic run stalls at one turn with zero artifacts. The fix is a proxy that translates Anthropic Messages to OpenAI chat completions and on to Bedrock's OpenAI-compatible `bedrock-mantle` endpoint, which parses those tokens into real tool calls. The repo ships it. + +**1. Start the proxy.** + +```bash +cd benchmarks +./scripts/bedrock-mantle-proxy.sh # installs deps, mints a 12h token, listens on :4000 +./scripts/bedrock-mantle-proxy.sh --status +``` + +It mints the bearer token from your ambient AWS credentials and holds it server-side; clients send a throwaway key. Models come from [config/litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml), and `curl -s http://127.0.0.1:4000/v1/models` lists them. + +**2. Run it.** Same shape as the vLLM route, pointed at the proxy: + +```bash +claude -p "Reply with exactly: OK" \ + --model qwen.qwen3-coder-30b-a3b-instruct \ + --output-format json \ + --settings '{"apiKeyHelper":"echo local","env":{ + "CLAUDE_CODE_USE_BEDROCK":"0", + "ANTHROPIC_BASE_URL":"http://127.0.0.1:4000", + "ANTHROPIC_API_KEY":"local", + "CLAUDE_CODE_AUTO_COMPACT_WINDOW":"262144", + "DISABLE_NON_ESSENTIAL_MODEL_CALLS":"1"}}' < /dev/null +``` + +**`DISABLE_NON_ESSENTIAL_MODEL_CALLS=1` is doing real work here.** Without it, the same command emitted `[claude-code:unrecognized_model] {"model":"qwen.qwen3-coder-30b-a3b-instruct","query_source":"generate_session_title"}` before the answer: Claude Code makes side calls, such as generating a session title, and an unrecognized model id makes them fail noisily. With the flag set the run returned `"result":"OK","subtype":"success","num_turns":1`. + +Setting `ANTHROPIC_API_KEY` also prints a warning that claude.ai connectors are disabled because an auth source takes precedence over your claude.ai login. That is expected on this route and harmless. + +## Model on Bedrock directly + +Anthropic models need no proxy. Flip Bedrock mode on and name the inference profile: + +```bash +claude -p "Reply with exactly: OK" \ + --model us.anthropic.claude-haiku-4-5-20251001-v1:0 \ + --output-format json \ + --settings '{"env":{"CLAUDE_CODE_USE_BEDROCK":"1","AWS_REGION":"us-east-2"}}' < /dev/null +``` + +No `apiKeyHelper` and no base URL: Bedrock mode authenticates from the ambient AWS credential chain. Pin the region in the settings block, not just the environment, so a global settings file cannot flip routing. + +**This route reports real costs and real caching.** The verified run returned `"total_cost_usd": 0.0253` with `"cache_creation_input_tokens": 20064` -- Claude Code inserts its own cache-control markers, so a repeated prefix is written once and read back cheaply on later turns. That is why published Claude Code runs on Bedrock show millions of cache-read tokens against a tiny fresh-input count. + +## Which route serves which model + +| Model | Route | Extra service | +|---|---|---| +| Anthropic on Bedrock | `CLAUDE_CODE_USE_BEDROCK=1` + inference profile | none | +| Any model on your own vLLM server | `ANTHROPIC_BASE_URL` at `:8000` | none, vLLM answers `/v1/messages` | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | `ANTHROPIC_BASE_URL` at the mantle proxy | `bedrock-mantle-proxy.sh` | + +## Driving a benchmark with this wiring + +The harness builds the settings object and the environment per run, so a benchmark run needs only `--provider` and a model; `claude` is the default agent. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for its flags, the permission model, and the auto-compaction detail, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock and proving it with a live call. +- [faq/wiring-codex-to-models.md](wiring-codex-to-models.md) and [faq/wiring-omp-to-models.md](wiring-omp-to-models.md) -- the same three routes for the other two harnesses. +- [cost-per-task-methodology.md](../cost-per-task-methodology.md) -- why `total_cost_usd` is trustworthy on Bedrock and meaningless against a self-hosted endpoint. diff --git a/docs/faq/wiring-codex-to-models.md b/docs/faq/wiring-codex-to-models.md new file mode 100644 index 0000000..436c040 --- /dev/null +++ b/docs/faq/wiring-codex-to-models.md @@ -0,0 +1,164 @@ +# How do I wire codex to a model? + +Three ways to point `codex` at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or an OpenAI model on Bedrock. Each ends with a `codex exec` call you can run by hand. + +For installing codex and wiring it to Bedrock, see [codex-setup.md](../codex-setup.md). This page assumes `codex --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with `codex-cli 0.153.4`. Each `codex exec` example returned `OK`: the Bedrock one against `openai.gpt-5.6-luna`, the vLLM one against a live `qwen3.6-35b-fp8` server at a 262,144-token window, and the bridge one against `qwen.qwen3-coder-30b-a3b-v1:0` through LiteLLM on port 4002. Where a command emits a warning anyway, this page says so. + +## The one fact everything follows from + +**codex speaks the OpenAI Responses API and nothing else.** codex 0.153.4 removed the chat-completions wire and rejects the fallback: + +``` +Error loading config.toml: `wire_api = "chat"` is no longer supported. +How to fix: set `wire_api = "responses"` in your provider config. +``` + +So whatever serves the model must answer `POST /v1/responses`. Bedrock does that for the `openai.*` family only, which is why an open-weight Bedrock model needs a bridge and your own vLLM server does not. + +**Close stdin on every call.** `codex exec` reads stdin even when the prompt arrives as an argument, and blocks forever on an open, empty one. Every example below ends with `< /dev/null`. + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +Bedrock will not answer Responses for these: + +``` +The model 'qwen.qwen3-coder-30b-a3b-v1:0' does not support the '/openai/v1/responses' API +``` + +The model works there; it just does not answer that API. Put a LiteLLM proxy in front that speaks Responses to codex and Converse to Bedrock, using LiteLLM's native `bedrock/` provider, which authenticates from ambient AWS credentials with no bearer token. + +**1. Write the proxy config.** + +```yaml +# litellm-responses-bedrock.yaml +model_list: + - model_name: qwen.qwen3-coder-30b-a3b-instruct + litellm_params: + model: bedrock/qwen.qwen3-coder-30b-a3b-v1:0 + aws_region_name: us-east-2 + +litellm_settings: + drop_params: true +``` + +**2. Start it.** + +```bash +uv run --with 'litellm[proxy]' litellm --config litellm-responses-bedrock.yaml \ + --host 127.0.0.1 --port 4002 +``` + +**3. Point codex at it.** + +```bash +export OPENAI_API_KEY=local # the proxy ignores the value; codex requires the variable + +codex exec --json --skip-git-repo-check \ + --model qwen.qwen3-coder-30b-a3b-instruct \ + -c model_provider=litellm \ + -c model_providers.litellm.name=litellm \ + -c model_providers.litellm.base_url=http://127.0.0.1:4002/v1 \ + -c model_providers.litellm.wire_api=responses \ + -c model_providers.litellm.env_key=OPENAI_API_KEY \ + -- "Reply with exactly: OK" < /dev/null +``` + +**Two traps.** + +The committed [litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml) **cannot** serve this path. It registers each model as `openai/` against the `bedrock-mantle` endpoint, so LiteLLM forwards `/v1/responses` untouched and mantle rejects it for a non-`openai.*` model. That config exists for Claude Code, which speaks the Anthropic Messages API. Write a separate config with the `bedrock/` provider. + +Pin nothing older than current LiteLLM. The range the mantle script uses, `litellm[proxy]>=1.72,<1.84`, fails the bridge with `500 'NoneType' object has no attribute 'encode'`. + +### Prompt caching does not happen on this route + +Bedrock Converse caching is opt-in per request: the caller must place `cachePoint` blocks in the message content. codex has no cache-control concept and LiteLLM's bridge adds none, so `cache_read` is 0 and every token is billed fresh. + +Two separate limits hide behind that zero: + +| Model | `cachePoint` on Converse | +|---|---| +| `qwen.qwen3-coder-30b-a3b-v1:0` | `AccessDeniedException: You invoked an unsupported model or your request did not allow prompt caching` | +| `us.anthropic.claude-haiku-4-5` | cold call writes `cacheWriteInputTokens: 5204`; warm call reads `cacheReadInputTokens: 5204` | + +Qwen cannot cache on Bedrock at all, and a model that can still will not through this bridge. A self-hosted vLLM server is the opposite: its prefix caching is automatic and server-side, needing nothing from the client. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +**2. Point codex at it.** + +```bash +export OPENAI_API_KEY=local # vLLM ignores the value; codex requires the variable + +codex exec --json --skip-git-repo-check \ + --model qwen3.6-35b-fp8 \ + -c model_provider=vllm \ + -c model_providers.vllm.name=vllm \ + -c model_providers.vllm.base_url=http://127.0.0.1:8000/v1 \ + -c model_providers.vllm.wire_api=responses \ + -c model_providers.vllm.env_key=OPENAI_API_KEY \ + -c model_context_window=262144 \ + -- "Reply with exactly: OK" < /dev/null +``` + +**The base URL must travel in the provider block.** codex 0.153.4 ignores `OPENAI_BASE_URL` and takes its base URL from whichever provider its config selects. Exporting that variable alone sends the call wherever `~/.codex/config.toml` points, which on a judge-configured machine is Bedrock, producing the misleading `404 The model 'minicpm5-2b' does not exist` (issue #183). + +**The tool-call parser must accept Responses-shaped tools.** The Responses API sends a flat tool definition, `{"type": "function", "name": ...}`; chat completions nests it under `"function"`. Seven of vLLM 0.29.0's parsers read only the nested shape, abort tool extraction on the flat one, end the stream with no `response.completed`, and make codex retry every request five times before failing the turn: + +| Works with codex | Breaks with codex | +|---|---| +| `qwen3_coder`, `hermes` | `minicpm5xml`, `dots`, `hy_v3`, `hy_v4`, `rust`, `step3`, `step3p5` | + +**Pass `model_context_window`.** A self-hosted model is unknown to codex, so it sizes the conversation from fallback metadata unless told the real window. Passing the flag does **not** silence the warning -- the run above still logged `Model metadata for 'qwen3.6-35b-fp8' not found. Defaulting to fallback metadata` with `model_context_window=262144` set. Treat that line as noise; what matters is that the window codex plans against matches the one vLLM booted. + +## OpenAI model on Bedrock + +Nothing to bridge. Bedrock answers Responses natively for the `openai.*` family, so codex needs only its own provider and a region: + +```bash +codex exec --json --skip-git-repo-check \ + --model openai.gpt-5.6-luna \ + -c model_provider=amazon-bedrock \ + -c model_providers.amazon-bedrock.aws.region=us-east-2 \ + -- "Reply with exactly: OK" < /dev/null +``` + +Authentication comes from the ambient AWS credential chain: no proxy, no bearer token. Set the same two values in `~/.codex/config.toml` to make them the default for every call, which is how the judge is configured ([codex-setup.md](../codex-setup.md)). + +**This is the one route where codex gets prompt caching.** The call above reported `cache_write_input_tokens: 8761` on a cold run, so a repeated prefix is read back cheaply on the next call. Bedrock's Responses implementation handles the cache itself, with nothing for the caller to place -- unlike Converse behind the LiteLLM bridge, where caching needs `cachePoint` blocks that nothing in the chain inserts. + +## Which route serves which model + +| Model | Route | Bridge needed | +|---|---|---| +| `openai.*` on Bedrock | native `amazon-bedrock` provider | no | +| Any model on your own vLLM server | provider block at `:8000` | no, but the tool parser must be Responses-safe | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | LiteLLM with the native `bedrock/` provider | yes | +| Anthropic models on Bedrock | native `amazon-bedrock` provider | no | + +## Driving a benchmark with this wiring + +The harness builds these provider blocks itself, so a benchmark run needs only `--agent codex` and a provider. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for the agent's flags and cost accounting, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [codex-setup.md](../codex-setup.md) -- installing codex, wiring it to Bedrock, and its failure-mode table. +- [agent-cli-bedrock-setup.md](../../benchmarks/docs/agent-cli-bedrock-setup.md) -- wiring both CLIs to Bedrock and proving it with a live call. +- [self-hosted/vllm/README.md](../../self-hosted/vllm/README.md) -- installing vLLM and the serving-config reference. diff --git a/docs/faq/wiring-omp-to-models.md b/docs/faq/wiring-omp-to-models.md new file mode 100644 index 0000000..5549bad --- /dev/null +++ b/docs/faq/wiring-omp-to-models.md @@ -0,0 +1,149 @@ +# How do I wire omp to a model? + +Three ways to point `omp` (oh-my-pi) at a model: an open-weight model on Amazon Bedrock, an open-weight model you serve yourself with vLLM, or a model on Bedrock directly. Each ends with an `omp -p` call you can run by hand. + +For installing omp and the three ways it differs from `pi`, see [omp-setup.md](../omp-setup.md). This page assumes `omp --version` already answers. + +> [!NOTE] +> **Every command below was run before this page shipped**, on a `g6e.4xlarge` (1x L40S) with `omp/18.1.15`. Each `omp -p` example returned `OK`: the vLLM one against a live `qwen3.6-35b-fp8` server at a 262,144-token window, the Bedrock one against `claude-haiku-4-5`, and the open-weight-on-Bedrock one against `qwen.qwen3-coder-30b-a3b-instruct` through the LiteLLM mantle proxy. + +## The two facts everything follows from + +**omp's config is YAML, and it lives wherever `PI_CODING_AGENT_DIR` points.** omp is a fork of pi and honours that variable, but where pi reads `models.json`, omp reads `models.yml` for providers and `config.yml` for settings. Pointing the variable at a scratch directory keeps an experiment away from your own `~/.omp`, which is what the harness does per run. + +**A model is named `provider/model`.** The provider is a key in `models.yml` for a custom endpoint, or the built-in `amazon-bedrock` for Bedrock. So `--model vllm/qwen3.6-35b-fp8` and `--model amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0`. + +Two smaller ones that bite: + +- **omp hangs on an inherited stdin.** Redirect on every call: the examples below all end with `< /dev/null`. +- **Put `--` before the prompt.** omp parses options until it sees `--`. A prompt that starts with `---`, which any inlined `SKILL.md` does because of its YAML frontmatter, is otherwise read as an unknown flag. + +## Open-weight model you serve with vLLM + +**1. Serve it.** Take `MODEL`, `SERVED_NAME`, `TP`, `MAX_MODEL_LEN` and `TOOL_PARSER` from the model's guide under [self-hosted/vllm/models/](../../self-hosted/vllm/models/) rather than inventing them. For `qwen3.6-35b-fp8`: + +```bash +cd self-hosted/vllm/scripts +MODEL="Qwen/Qwen3.6-35B-A3B-FP8" \ +SERVED_NAME="qwen3.6-35b-fp8" \ +TP=1 \ +PORT=8000 \ +MAX_MODEL_LEN=262144 \ +GPU_MEM_UTIL=0.92 \ +TOOL_PARSER="qwen3_coder" \ +EXTRA_ARGS="--max-num-seqs 32 --kv-cache-dtype fp8" \ + ./vllm-serve.sh +``` + +**2. Declare the server as a provider.** Write `models.yml` into a scratch agent dir: + +```yaml +# /tmp/omp-vllm/.omp/models.yml +providers: + vllm: + baseUrl: http://127.0.0.1:8000/v1 + api: openai-completions + apiKey: local + models: + - id: qwen3.6-35b-fp8 + name: "vLLM: qwen3.6-35b-fp8" + contextWindow: 262144 + maxTokens: 16000 + cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0} +``` + +**3. Size compaction to the window.** Without this omp fills the context to within its default reserve, and one capped response overflows the endpoint mid-run. Reserve a full response plus about 8K of headroom -- for a 262,144 window and a 16,000-token cap, that is `262144 - (16000 + 8192) = 237952`: + +```yaml +# /tmp/omp-vllm/.omp/config.yml +compaction: + enabled: true + thresholdTokens: 237952 +``` + +**4. Run it.** + +```bash +PI_CODING_AGENT_DIR=/tmp/omp-vllm/.omp \ + omp -p --mode json --no-session --auto-approve \ + --model vllm/qwen3.6-35b-fp8 \ + -- "Reply with exactly: OK" < /dev/null +``` + +`--mode json` gives the pi-shaped event stream; the final `message_end` event carries usage. The verified run reported `"provider":"vllm","model":"qwen3.6-35b-fp8"` with 18,108 input and 6 output tokens. + +**`--no-session` keeps the run ephemeral** so nothing lands in your session history, and **`--auto-approve`** runs tools without an approval gate, which an unattended run needs. + +## Open-weight model on Bedrock (Qwen, Kimi, DeepSeek) + +omp speaks the OpenAI chat-completions wire through `api: openai-completions`, so it needs an OpenAI-compatible endpoint in front of Bedrock. The repo ships one: [bedrock-mantle-proxy.sh](../../benchmarks/scripts/bedrock-mantle-proxy.sh) starts a LiteLLM proxy over Bedrock's OpenAI-compatible `bedrock-mantle` endpoint, minting a 12-hour bearer token from your ambient AWS credentials and holding it server-side. + +**1. Start the proxy.** + +```bash +cd benchmarks +./scripts/bedrock-mantle-proxy.sh # installs deps, mints the token, listens on :4000 +./scripts/bedrock-mantle-proxy.sh --status +``` + +Models come from [config/litellm-mantle.yaml](../../benchmarks/config/litellm-mantle.yaml); `curl -s http://127.0.0.1:4000/v1/models` lists them. + +**2. Declare it as a provider, with real rates.** + +```yaml +# /tmp/omp-mantle/.omp/models.yml +providers: + mantle: + baseUrl: http://127.0.0.1:4000/v1 + api: openai-completions + apiKey: local + models: + - id: qwen.qwen3-coder-30b-a3b-instruct + name: "Bedrock: qwen3-coder-30b" + contextWindow: 262144 + maxTokens: 16000 + cost: {input: 0.1545, output: 0.618, cacheRead: 0, cacheWrite: 0} +``` + +**3. Run it.** + +```bash +PI_CODING_AGENT_DIR=/tmp/omp-mantle/.omp \ + omp -p --mode json --no-session --auto-approve \ + --model mantle/qwen.qwen3-coder-30b-a3b-instruct \ + -- "Reply with exactly: OK" < /dev/null +``` + +**The `cost` block is not decoration: omp prices the run from it.** The verified run reported `"cost":{"input":0.002824878,"output":0.000001236,...,"total":0.002826114}` for 18,284 input and 2 output tokens, which is exactly those per-1M rates applied. Leave the zeros in for a self-hosted model, where per-token cost is meaningless and the repo derives cost from GPU-seconds instead ([cost-per-task-methodology.md](../cost-per-task-methodology.md)). Put the real Bedrock rates in for a metered model, and omp's own numbers are usable. + +The rates above are Qwen3-Coder-30B's Standard tier, the same ones in [bedrock_pricing.py](../../benchmarks/scripts/bedrock_pricing.py). + +## Model on Bedrock directly + +Bedrock is built in, so no `models.yml` is needed. Name the inference profile after `amazon-bedrock/`: + +```bash +omp -p --mode json --no-session --auto-approve \ + --model amazon-bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 \ + -- "Reply with exactly: OK" < /dev/null +``` + +Authentication comes from the ambient AWS credential chain; set `AWS_REGION` if your default region does not host the model. Nothing is written to disk and no key is passed on the command line. + +## Which route serves which model + +| Model | Route | Extra service | +|---|---|---| +| Anthropic (or any native Bedrock model) | `--model amazon-bedrock/` | none | +| Any model on your own vLLM server | a `models.yml` provider at `:8000` | none | +| Open-weight on Bedrock (Qwen, Kimi, DeepSeek) | a `models.yml` provider at the mantle proxy | `bedrock-mantle-proxy.sh` | + +## Driving a benchmark with this wiring + +The harness writes `models.yml` and `config.yml` per run into a scratch `PI_CODING_AGENT_DIR`, so a benchmark run needs only `--agent omp` and a provider. See [harness-reference.md](../../benchmarks/docs/harness-reference.md#choosing-the-agent) for the agent's flags and cost accounting, and the [benchmark skill](../../.claude/skills/benchmark/SKILL.md) for the end-to-end flow. + +## Related + +- [omp-setup.md](../omp-setup.md) -- installing omp and the three ways it differs from pi. +- [faq/wiring-codex-to-models.md](wiring-codex-to-models.md) -- the same three routes for codex, which needs the Responses API and therefore a different bridge. +- [cost-per-task-methodology.md](../cost-per-task-methodology.md) -- why a metered Bedrock bill and a hardware-derived self-hosted figure do not compare as raw dollars.