From 94adeb818e68cef8edd4990119ff3ae1d855fc90 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Fri, 14 Aug 2026 00:47:13 +0000 Subject: [PATCH 1/2] Replace the model allowlist with pi-ai catalog passthrough MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CUA_MODEL_ANNOTATIONS decided which models were allowed to run. It was the last gate between a caller and pi-ai's catalog, and it refused models that needed no work at all: grok-4.3 was already in pi's registry and still rejected, and grok-4.6 could not be reached the day xAI shipped it. Every model pi-ai carries is now selectable — 37 providers, ~1,150 models — and an id the registry has not caught up with is synthesized from its nearest sibling. Only an unqualified ref or a provider pi-ai does not carry is refused. Two tables replace the allowlist, and neither decides whether a model may run: - CUA_NATIVE_SURFACES records which models have a provider-native computer or browser tool, so the tool menu can offer it. `cua models` shows it in a NATIVE column, and CuaModelInfo carries it alongside a vision flag. - CUA_MODEL_QUIRKS records request-shape limits, each carrying the documented limit or observed failure that justifies it. Capabilities default to permissive: a limit we cannot evidence becomes a provider-side error rather than a refusal cua invents. Synthesis follows the sibling sharing the longest id prefix, preferring the latest. Providers migrate transports mid-generation — xAI carries grok-4.3 on chat completions and grok-4.5 on Responses — so a new id has to follow its nearest, newest relative rather than whichever model comes first. Two consequences the opened catalog forces: - A bare model id that several providers carry resolves to the first-party provider, since gateways resell the same ids. It is a disambiguation preference for bare ids only, never a gate. - The CLI's API-key preflight runs only where CUA documents the variable names. For any other provider, pi resolves the credential when it streams; failing up front would refuse a model that works. --- .agents/skills/update-models/SKILL.md | 72 +++--- README.md | 8 +- packages/agent/CHANGELOG.md | 6 + packages/agent/src/agent.ts | 2 - packages/agent/test/agent.test.ts | 8 +- packages/ai/CHANGELOG.md | 24 ++ packages/ai/README.md | 4 +- packages/ai/docs/supported-models.md | 192 +++++--------- packages/ai/src/api-keys.ts | 37 +-- packages/ai/src/models.ts | 309 +++++++++++++---------- packages/ai/src/tool-catalog.ts | 3 +- packages/ai/test/models.test.ts | 197 +++++++++------ packages/cli/CHANGELOG.md | 16 ++ packages/cli/src/cli-harness.ts | 18 +- packages/cli/src/harness-models.ts | 36 ++- packages/cli/src/harness.ts | 41 ++- packages/cli/test/harness-models.test.ts | 18 +- packages/cli/test/model-picker.test.ts | 6 +- 18 files changed, 556 insertions(+), 441 deletions(-) diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md index 1cad940..b2bc092 100644 --- a/.agents/skills/update-models/SKILL.md +++ b/.agents/skills/update-models/SKILL.md @@ -1,11 +1,19 @@ --- name: update-models -description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. +description: Detect drift in CUA's provider-native tool surfaces and per-model request limits. Use when a provider ships a new native computer/browser tool version, when a model starts rejecting a tool schema CUA sends, or when auditing provider-native action vocabularies against official docs and examples. --- # Update Models -Use this workflow to keep CUA current with provider model releases and computer-use support. Do not trust a static model list: combine provider metadata, official docs, official example repos, and live non-destructive smoke tests. +Use this workflow to keep CUA's two model tables honest. Note what it is **not** for: CUA has no model allowlist. Every model pi-ai carries is selectable, and an id pi-ai has not caught up with is synthesized from its nearest sibling, so a newly released model needs no repo change to be usable. + +What still needs maintaining is narrow: + +- `CUA_NATIVE_SURFACES` — which models have a provider-native computer or browser tool CUA can offer. +- `CUA_MODEL_QUIRKS` — request-shape limits, each justified by a documented limit or an observed failure. +- The provider adapters themselves, when a native action vocabulary or tool version changes. + +Do not trust a static list for any of those: combine provider metadata, official docs, official example repos, and live non-destructive smoke tests. ## Quick Start @@ -62,7 +70,7 @@ Treat example repos as strongest when they are provider-owned or linked from off There are two enumeration layers: - Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access. -- CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use. +- Selectable refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` returns pi-ai's whole catalog, each entry marked with the native surfaces CUA can offer for it. `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` answer those two questions for a single model. When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("")`. @@ -145,14 +153,18 @@ The probe does not execute browser actions. It elicits tool calls for screenshot ## Decision Rules -Recommend a model as CUA-supported only if: +Add a `CUA_NATIVE_SURFACES` entry only if: + +- The provider documents a native computer or browser tool for that model. +- A live probe elicits a native tool call: `npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider

--model `. +- The local adapter can execute the actions the probe emits; otherwise the adapter needs updating first. + +Add a `CUA_MODEL_QUIRKS` entry only if you can state the failure it prevents. A quirk is not a preference — it is +a request CUA must not send because the provider rejects it. Record the evidence in the entry's `reason`, and +prefer the narrowest scope that covers it: a single model id over a family, a family over a whole provider. -- It appears in the provider metadata API for the available key. -- Its model-specific docs do not rule out required CUA runtime features such as streaming. -- Its provider-appropriate computer-use smoke test passes: native tools where offered, or supplied function tools for Meta and xAI. -- Its local cua-ai smoke test emits a computer tool call: `CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai` returns a `toolCall` block. -- Official docs or examples support the same tool mechanism, or the smoke result clearly supersedes stale docs. -- The model is annotated in `CUA_MODEL_ANNOTATIONS` in `packages/ai/src/models.ts`, resolved from `pi-ai`'s registry or backed by a `CUA_MODEL_OVERRIDES` entry. +Remove a quirk when the provider lifts the limit. A stale quirk silently denies a model a tool it now accepts, +which is harder to notice than the reverse. Recommend adapter updates when: @@ -160,29 +172,29 @@ Recommend adapter updates when: - Official examples handle response fields the local adapter ignores. - Smoke probes emit native actions not present in local constants. -Do not print API keys. Keep smoke tests non-destructive. Do not edit repo defaults or adapters unless the user explicitly asks after reviewing the report. +Do not print API keys. Keep smoke tests non-destructive. Do not edit repo defaults or adapters unless the user +explicitly asks after reviewing the report. ## Updating CUA Support -All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). When a new model is discovered, decide which layer needs changing: - -- New model ID, same provider/tool surface: - - Add a `CUA_MODEL_ANNOTATIONS` entry in `packages/ai/src/models.ts` under the correct provider, citing the official source that documents computer-use support. Use a `family` match for a root that covers numeric revisions and dated snapshots (e.g. `claude-opus-4`), or an `exact` match for a single ID. A model already covered by an existing family annotation needs no change. - - If `pi-ai`'s registry does not carry the ID yet (`pi_ai_registry: "missing"` in the discovery report), add a `CUA_MODEL_OVERRIDES` entry so `getCuaModel()` can return a provider-shaped model. When the ID is already in the registry, the annotation alone is enough. - - Update the snapshot in `packages/ai/docs/supported-models.md` to match. - -- New provider-native action, response field, or tool version: - - OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes. - - Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package. - - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed. - - xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility. - - Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`. - - Shared canonical action semantics go in `packages/ai/src/providers/common.ts`. - -- New provider or routing rule: - - Update `CuaProvider`, `CUA_PROVIDERS`, `CUA_MODEL_ANNOTATIONS`, and `CUA_MODEL_OVERRIDES` in `packages/ai/src/models.ts`, plus the provider-module wiring in `packages/ai/src/providers.ts`. - -After changing support, run `npm run typecheck`, `npm test --workspace @onkernel/cua-ai`, and at least one live smoke per changed provider, for example `CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai`. +All model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). + +- **New model id, no native surface, no quirk**: nothing to do. It already works. +- **New model with a provider-native tool**: add a `CUA_NATIVE_SURFACES` entry in `packages/ai/src/models.ts` + citing first-party documentation. Use a `family` match to cover numeric revisions and dated snapshots, or an + `exact` match for a single id. Anthropic's surfaces are version-gated in + `providers/anthropic/capabilities.ts` instead. +- **A model rejects a tool CUA sends**: add a `CUA_MODEL_QUIRKS` entry with the observed error as its `reason`. +- **New provider-native action, response field, or tool version**: update that provider's adapter under + `packages/ai/src/providers/`. Anthropic's computer tool version and `computer-use-*` beta header are selected + by pi-ai per model, so a new dated version usually means bumping `@earendil-works/pi-ai`. +- **A provider pi-ai does not carry**: it is not selectable. Adding one means registering a provider in + `packages/ai/src/providers.ts`, which is a deliberate decision — the repo has removed four such providers + rather than carry them unused. + +Update `packages/ai/docs/supported-models.md` to match either table change. Then run `npm run typecheck`, +`npm test --workspace @onkernel/cua-ai`, and at least one live smoke per changed provider, for example +`CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai`. ## Reference Files diff --git a/README.md b/README.md index 6e3d9be..7970f58 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ packages/ (`@onkernel/cua-agent`) — `CuaAgent`/`CuaAgentHarness` run the full computer-use loop against a Kernel browser. It sits on [`packages/ai`](packages/ai) (`@onkernel/cua-ai`), the model layer with the -curated computer-use model catalog, canonical tool schemas, and per-provider +pi-ai model catalog, canonical tool schemas, and per-provider adapters on top of pi-ai; reach for cua-ai directly only when you bring your own execution. @@ -101,7 +101,7 @@ export KERNEL_API_KEY=sk_... # always required # single-shot cua -p "Open https://news.ycombinator.com and tell me the top story" -# list supported model ids +# list selectable model ids cua models # Claude @@ -141,7 +141,7 @@ cua -p -o jsonl "open example.com and tell me the heading" ## How it works -1. **Model layer** — `@onkernel/cua-ai` owns the curated model catalog, +1. **Model layer** — `@onkernel/cua-ai` opens pi-ai's whole model catalog, stable tool identities, explicit tool factories/toolsets, compatibility checks, and provider declarations/headers/payload transforms. 2. **Execution layer** — `@onkernel/cua-agent` composes around @@ -171,7 +171,7 @@ Highlights: - `-p`/`--print` for single-shot mode; `-o jsonl` for structured output. - `cua models` to list supported `-m`/`--model` values and their providers. -- `-m`/`--model ` to choose one of those supported models. +- `-m`/`--model ` to choose any model pi-ai carries. - `/model` in the TUI for a searchable model picker; `/model ` still switches directly. - `/tools` in the TUI to enable or disable tools for the current session. diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 0672ec1..2025f15 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 0.14.0 - 2026-08-14 + +- `CuaAgentHarness` no longer refuses a model ref that is absent from its + supplied `Models` collection: it falls back to the registry, and an id the + registry lacks is synthesized. Update `@onkernel/cua-ai` to 0.14.0. + ## 0.13.0 - 2026-08-13 Breaking: Tzafon and Yutori support is removed. diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 6dd045e..5dcf0fd 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -27,7 +27,6 @@ import { cuaModels, type CuaIncomingToolPlan, type CuaModelRef, - findCuaAnnotation, getCuaModel, parseCuaModelRef, type CuaSimpleStreamOptions, @@ -499,7 +498,6 @@ const defaultCuaStream: StreamFn = (model, context, options) => cuaModels().stre function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model { const { provider, model: id } = parseCuaModelRef(ref); - if (!findCuaAnnotation(provider, id)) throw new Error(`unsupported CUA model "${ref}"`); return models.getModel(provider, id) ?? getCuaModel(ref); } diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index ed459a4..742ecf5 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -451,7 +451,7 @@ describe("CuaAgent explicit tools", () => { }); describe("CuaAgentHarness explicit tools", () => { - it("resolves annotated refs from supplied models for construction and setModel", async () => { + it("resolves refs from supplied models for construction and setModel", async () => { const models = createCuaModels(); const openai = models.getProvider("openai")!; const first = { ...getCuaModel("openai:gpt-5.5"), baseUrl: "https://first.example" }; @@ -462,7 +462,11 @@ describe("CuaAgentHarness explicit tools", () => { expect(harness.getModel()).toBe(first); await harness.setModel("openai:gpt-5.6-sol"); expect(harness.getModel()).toBe(second); - expect(() => new CuaAgentHarness({ ...services, browser, client, models, model: "openai:gpt-4o", tools: [] })).toThrow(/unsupported CUA model/); + + // A ref the supplied collection does not carry falls back to the registry + // rather than being refused: the provider decides what exists. + const fallback = new CuaAgentHarness({ ...services, browser, client, models, model: "openai:gpt-5.4", tools: [] }); + expect(fallback.getModel().id).toBe("gpt-5.4"); }); it("uses composition, hides active-tool APIs, and supports an empty catalog", async () => { diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index f1e3d6a..5f373c6 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## 0.14.0 - 2026-08-14 + +Breaking: the model allowlist is removed. + +- `listCuaModels()` returns pi-ai's whole catalog — 37 providers, ~1,150 models — + instead of a curated subset, and each entry now carries `nativeSurfaces` and + `vision` so callers can render what a model can do. +- `getCuaModel(ref)` resolves any model pi-ai carries, and synthesizes one for + an id the registry has not caught up with, using the sibling that shares the + longest id prefix and preferring the latest such sibling. Providers migrate + transports mid-generation, so a new id follows its nearest, newest relative. + Only an unqualified ref or a provider pi-ai does not carry is refused. +- `CUA_MODEL_ANNOTATIONS`, `CUA_PROVIDERS`, `isCuaProvider`, and the + `CuaProvider` union are gone; `CuaProvider` is now a provider id string and + `cuaProviders()` returns what pi-ai carries. `providerForModel` no longer + throws. +- Two tables replace the allowlist, neither of which decides whether a model may + run: `CUA_NATIVE_SURFACES` (which models have a provider-native computer or + browser tool, with first-party sources) and `CUA_MODEL_QUIRKS` (request-shape + limits, each carrying the documented limit or observed failure that justifies + it). `cuaModelCapabilities` reads the quirk table and defaults to permissive; + `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` are exported for menus + and diagnostics. + ## 0.13.0 - 2026-08-13 - Remove the Meta provider. pi-ai ships no `meta` provider, so cua hand-wrote a diff --git a/packages/ai/README.md b/packages/ai/README.md index d7e7b20..5d6ec5c 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -32,8 +32,8 @@ console.table(listCuaModels("google")); ``` `gemini:` aliases `google:` and `moonshot:` aliases `moonshotai:`. The package -does not export a default model. See [supported models](docs/supported-models.md) -for the curated list. +does not export a default model. See [models and native surfaces](docs/supported-models.md) +for which models have provider-native tools and which have known request limits. ## Explicit tools diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 6c80220..589fc72 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -1,130 +1,62 @@ -# Supported CUA Models - -`@onkernel/cua-ai` accepts any pi-ai model whose ID is annotated as -CUA-supporting in `CUA_MODEL_ANNOTATIONS` (see -[`src/models.ts`](https://github.com/kernel/cua/blob/main/packages/ai/src/models.ts)). -Annotations are either a `family` -match or an `exact` ID match. A family match covers the family root plus -suffixes made of hyphen-separated numeric segments — revisions and dated -snapshots such as `claude-opus-4-7`, `gpt-5.5-2026-04-23`, or -`claude-3-7-sonnet-20250219`. Named sibling variants like `gpt-5.4-mini` -are distinct models that may not support computer use, so they need their -own annotation. Each annotation cites the provider's CUA docs. - -The list below is the current snapshot. Run -`listCuaModels(provider?)` for the live list — it merges pi-ai's registry -with CUA-only entries that pi-ai does not ship yet. - -## `openai` - -CLI default interaction: CUA browser primitives plus the explicit -`browser_act` verified-plan tool. The optional native computer tool uses pixel -coordinates. - -Exact IDs: - -- `gpt-5.6-sol` ([docs](https://developers.openai.com/api/docs/models/gpt-5.6-sol)) - -Family matches (root + numeric revision/dated-snapshot suffixes): - -- `gpt-5.4` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4)) -- `gpt-5.4-mini` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4-mini)) -- `gpt-5.5` ([docs](https://developers.openai.com/api/docs/models/gpt-5.5)) - -## `anthropic` - -CLI default interaction: native `browser_20260701` on supported model families, -with CUA browser primitives plus explicit `browser_act` as the -model-compatibility fallback. If the active -credential cannot access the native browser beta, CUA uses its equivalent -function-tool transport. The optional native computer tool uses pixel -coordinates. - -Family matches (root + numeric revision/dated-snapshot suffixes): - -- `claude-3-7-sonnet` -- `claude-opus-4` -- `claude-opus-5` -- `claude-sonnet-4` -- `claude-sonnet-5` -- `claude-haiku-4` -- `claude-fable-5` - -Source: [Anthropic computer use docs](https://docs.anthropic.com/en/docs/build-with-claude/computer-use). - -## `google` - -Coordinates: normalized 0–999 - -Model refs use the `google:` prefix; `gemini:` is accepted as an alias. - -Exact IDs: - -- `gemini-3.6-flash` (recommended) -- `gemini-3.5-flash` -- `gemini-3.5-flash-lite` - -Google computer use is configured explicitly with -`cua.providers.google.toolsets.browser()`, which emits Google's native -`tools.computer_use` declaration and current predefined browser actions. -Partial catalogs exclude every unselected current action. Legacy and preview -model/tool surfaces are intentionally not exposed. - -Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use). - -## `xai` - -CLI default interaction: CUA browser primitives plus explicit `browser_act`. - -Exact IDs: - -- `grok-4.5` - -Grok 4.5 uses xAI's OpenAI-compatible Responses API with ordinary CUA browser -function tools. xAI does not define a native computer tool. Tool loops continue -through `previous_response_id`. -CUA adds xAI's doubled token-price tier above 200k input tokens to pi-ai's -Grok 4.5 model metadata. - -Source: [Grok 4.5 docs](https://docs.x.ai/developers/grok-4-5), [function calling](https://docs.x.ai/developers/tools/function-calling), and [image understanding](https://docs.x.ai/developers/model-capabilities/images/understanding). - -## `moonshotai` - -CLI default interaction: CUA browser primitives only. Moonshot's API accepts the -complex `browser_wait_for` schema but rejects a request outright once the much -larger `browser_act` schema is attached, so `browser_act` is unavailable on -Moonshot models and the catalog rejects it explicitly. - -Model refs use the `moonshotai:` prefix; `moonshot:` is accepted as an alias. - -Exact IDs: - -- `kimi-k3` - -Kimi K3 uses Moonshot's OpenAI-compatible chat completions API with ordinary -CUA browser function tools. Moonshot does not define a native computer tool. -K3 launched with max-only thinking effort—other levels are clamped away until -Moonshot ships them. - -Source: [Kimi K3 announcement](https://www.kimi.com/blog/kimi-k3), [tool use](https://platform.kimi.ai/docs/api/tool-use), and [vision input](https://platform.kimi.ai/docs/guide/use-kimi-vision-model). - -## `openrouter` - -CLI default interaction: per model, not per provider. OpenRouter fronts several -model families, so the CLI asks each model whether it accepts `browser_act`'s -schema rather than assuming one answer for the whole provider. - -Model refs use the `openrouter:` prefix: - -- `moonshotai/kimi-k3` — browser primitives only: accepts complex function - schemas but rejects the larger `browser_act` schema. State mutations are - serialized. -- `meta/muse-spark-1.1` — browser primitives plus explicit `browser_act`. State - mutations are serialized. - -Both use OpenRouter's OpenAI-compatible chat completions API with ordinary CUA -browser function tools. OpenRouter does not expose the provider-native computer -tools declared by other CUA providers. - -Sources: [Kimi K3](https://openrouter.ai/moonshotai/kimi-k3), -[Muse Spark 1.1](https://openrouter.ai/meta/muse-spark-1.1). +# Models and native surfaces + +`@onkernel/cua-ai` accepts **any model pi-ai carries**, and any model id its +registry has not caught up with yet. There is no allowlist: a model id you pass +resolves, and the provider decides whether it exists. Run +`listCuaModels(provider?)` for the live catalog. + +Two small tables in [`src/models.ts`](https://github.com/kernel/cua/blob/main/packages/ai/src/models.ts) +describe what is *different* about particular models. Neither decides whether a +model may run. + +## Native surfaces + +`CUA_NATIVE_SURFACES` records which models have a provider-native computer or +browser tool, so the tool menu can offer it. Entries match either an exact id or +a `family` — the family root plus suffixes made of hyphen-separated numeric +segments, covering revisions and dated snapshots such as `claude-opus-4-7` or +`gpt-5.5-2026-04-23`. Named sibling variants like `gpt-5.4-mini` are distinct +models and need their own entry. Each cites first-party documentation. + +| provider | models | surfaces | +| --- | --- | --- | +| `anthropic` | `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser | +| `anthropic` | `claude-fable-5` family | computer | +| `openai` | `gpt-5.6-sol`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5` | computer | +| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` | browser | + +Anthropic's entries live in `providers/anthropic/capabilities.ts`, which is +version-gated separately; `cuaNativeSurfaces(model)` reads both sources. + +A model with no native surface is not restricted — it drives a Kernel browser +with CUA's own CDP tools, which is the default for every provider. + +## Quirks + +`CUA_MODEL_QUIRKS` records request-shape limits. Anything absent gets the +permissive default and is allowed to try; the provider's own error is the +feedback. Every entry exists because of a documented limit or an observed +failure, and carries its reason inline. + +| provider | models | limit | +| --- | --- | --- | +| `google` | all | rejects `browser_wait_for`'s schema shape; the Gemini API accepts a subset of JSON Schema for function declarations | +| `moonshotai` | `kimi-k3` | rejects the request once `browser_act`'s schema is attached; serializes state mutations | +| `openrouter` | `moonshotai/kimi-k3` | the same Kimi limit, reached through OpenRouter | +| `openrouter` | `meta/muse-spark-1.1` | serializes state mutations | +| `xai` | all | serializes state mutations | + +`cuaModelCapabilities(model)` applies provider-wide quirks first, then +model-specific ones. `cuaModelQuirks(model)` returns the entries that applied, +for diagnostics and menu hints. + +## Model ids pi-ai does not carry + +A ref whose id is missing from the registry is synthesized from the sibling +sharing the longest id prefix, preferring the latest such sibling. Providers +migrate transports mid-generation — xAI carries `grok-4.3` on chat completions +and `grok-4.5` on Responses — so a new id follows its nearest, newest relative. +This is what lets a model work the day the provider ships it rather than when +models.dev catches up. + +Only an unqualified ref or a provider pi-ai does not carry is refused. diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts index 70f9138..fba7ee8 100644 --- a/packages/ai/src/api-keys.ts +++ b/packages/ai/src/api-keys.ts @@ -1,14 +1,16 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { parseCuaModelRef, providerForModel, type CuaModelRef, type CuaProvider } from "./models"; +import { parseCuaModelRef, providerForModel, type CuaModelRef } from "./models"; /** - * Environment variables accepted for each CUA provider. + * Environment variables for the providers CUA documents, in precedence order. * - * This mirrors pi-ai's approach: model lookup is pure, while auth is resolved - * when streaming. These helpers let callers share one readable convention for - * explicit `getApiKey` wiring (especially useful for `google` vs `gemini`). + * Every provider pi-ai carries is selectable, and pi resolves each one's own + * credential when streaming. This table exists only so callers and the CLI can + * name the variable to set up front; a provider absent from it is not + * unsupported, it just has no CUA-side preflight. pi-ai does not export its + * own env-var registry, or this would read from that. */ -const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { +const CUA_PROVIDER_API_KEY_ENV_VARS: Readonly> = { openai: ["OPENAI_API_KEY"], anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], @@ -17,20 +19,16 @@ const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { openrouter: ["OPENROUTER_API_KEY"], }; +/** Provider prefixes accepted as aliases for a pi-ai provider id. */ +const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; + /** * List the environment variables checked for a provider's API key, in - * precedence order. Accepts `"gemini"` as an alias for `"google"` and - * `"moonshot"` as an alias for `"moonshotai"`; returns an empty list for - * unknown providers. + * precedence order. Returns an empty list for a provider CUA does not document, + * whose credential pi resolves at request time instead. */ export function cuaApiKeyEnvVarsForProvider(provider: string): readonly string[] { - if (provider === "gemini") { - return CUA_PROVIDER_API_KEY_ENV_VARS.google; - } - if (provider === "moonshot") { - return CUA_PROVIDER_API_KEY_ENV_VARS.moonshotai; - } - return CUA_PROVIDER_API_KEY_ENV_VARS[provider as keyof typeof CUA_PROVIDER_API_KEY_ENV_VARS] ?? []; + return CUA_PROVIDER_API_KEY_ENV_VARS[PROVIDER_ALIASES[provider] ?? provider] ?? []; } /** Read a provider's API key from the environment, or return undefined when unset. */ @@ -42,7 +40,12 @@ export function getCuaEnvApiKey(provider: string): string | undefined { return undefined; } -/** Read a provider's API key from the environment, or throw naming the variables to set. */ +/** + * Read a provider's API key from the environment, or throw naming the variables + * to set. Throws for a provider CUA documents no variables for — callers that + * accept any pi-ai provider should use {@link cuaApiKeyEnvVarsForProvider} to + * decide whether a preflight is possible at all. + */ export function requireCuaEnvApiKey(provider: string): string { const apiKey = getCuaEnvApiKey(provider); if (apiKey) return apiKey; diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index e023085..a7feb6e 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -1,11 +1,15 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; +import { getBuiltinModel, getBuiltinModels, getBuiltinProviders } from "@earendil-works/pi-ai/providers/all"; +import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer } from "./providers/anthropic/capabilities"; -/** Providers with curated computer-use model support. */ -export type CuaProvider = "openai" | "anthropic" | "google" | "xai" | "moonshotai" | "openrouter"; +/** A pi-ai provider id. Any provider pi-ai carries can be selected. */ +export type CuaProvider = string; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ -export type CuaModelRef = `${CuaProvider}:${string}`; +export type CuaModelRef = `${string}:${string}`; + +/** A provider-native tool surface CUA can offer for a model. */ +export type CuaNativeSurface = "computer" | "browser"; /** One entry returned by {@link listCuaModels}. */ export interface CuaModelInfo { @@ -16,13 +20,14 @@ export interface CuaModelInfo { model: string; /** Human-readable model name. */ name: string; + /** Provider-native tool surfaces available for this model, if any. */ + nativeSurfaces: readonly CuaNativeSurface[]; + /** Whether the model accepts image input, i.e. whether screenshot-based tools are usable. */ + vision: boolean; } -/** All providers this package curates computer-use models for. */ -export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; - /** - * How a {@link CuaModelAnnotation} matches model ids. + * How a model-id table entry matches. * * - `exact`: `id === match.id` * - `family`: `id === match.family`, or `match.family` plus hyphen-separated @@ -41,84 +46,96 @@ export interface CuaModelCapabilities { readonly serializesStateMutations: boolean; } -/** One CUA-support annotation: a model-id match plus the official source documenting support. */ -export interface CuaModelAnnotation { +/** + * A model or provider whose request handling differs from the permissive + * default, with the evidence for it. Entries exist to prevent a request the + * provider would reject — never to express a preference. + */ +export interface CuaModelQuirk { + readonly provider: CuaProvider; + /** Omit to apply the quirk to every model from the provider. */ + readonly match?: CuaModelMatch; + readonly capabilities: Partial; + /** Why this quirk exists: the documented limit or the observed failure. */ + readonly reason: string; +} + +/** + * Models with a provider-native computer or browser tool, and the first-party + * documentation for it. This table answers "can CUA offer a native tool for + * this model", not "may this model run" — every model pi-ai carries runs, with + * CUA's own CDP browser tools. + * + * Anthropic is absent deliberately: its native surfaces are version-gated in + * `providers/anthropic/capabilities.ts`, which {@link cuaNativeSurfaces} reads. + */ +export const CUA_NATIVE_SURFACES: readonly { + readonly provider: CuaProvider; readonly match: CuaModelMatch; - /** URL of the provider documentation establishing computer-use support. */ + readonly surfaces: readonly CuaNativeSurface[]; readonly source: string; - /** Optional tool-catalog capabilities that describe which CUA schemas and state mutations the model supports. */ - readonly capabilities?: CuaModelCapabilities; -} +}[] = [ + { provider: "openai", match: { kind: "exact", id: "gpt-5.6-sol" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.6-sol" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.4" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.4" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.4-mini" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.4-mini" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.5" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.5" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.6-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash-lite" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, +]; -// Muse Spark accepts the full CUA schema set; OpenRouter's provider-level -// defaults are conservative because the proxy fronts many model families. -const MUSE_SPARK_CAPABILITIES: CuaModelCapabilities = Object.freeze({ - acceptsComplexSchemas: true, - acceptsLargeSchemas: true, - serializesStateMutations: true, -}); +/** + * Known request-shape limits. Anything absent from this table gets the + * permissive default and is allowed to try; a provider-side error is the + * feedback. Every entry below is a limit we have documentation for or have + * observed against the live API. + */ +export const CUA_MODEL_QUIRKS: readonly CuaModelQuirk[] = [ + { + provider: "google", + capabilities: { acceptsComplexSchemas: false, acceptsLargeSchemas: false }, + reason: "The Gemini API accepts a subset of JSON Schema for function declarations and rejects browser_wait_for's shape.", + }, + { + provider: "moonshotai", + match: { kind: "exact", id: "kimi-k3" }, + capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, + reason: "Kimi K3 rejects the request outright once browser_act's schema is attached.", + }, + { + provider: "openrouter", + match: { kind: "exact", id: "moonshotai/kimi-k3" }, + capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, + reason: "Same Kimi K3 limit, reached through OpenRouter.", + }, + { + provider: "openrouter", + match: { kind: "exact", id: "meta/muse-spark-1.1" }, + capabilities: { serializesStateMutations: true }, + reason: "Muse Spark's computer-use cookbook disables parallel tool calls.", + }, + { + provider: "xai", + capabilities: { serializesStateMutations: true }, + reason: "Grok's computer-use guidance disables parallel tool calls for state-mutating catalogs.", + }, +]; -const KIMI_K3_CAPABILITIES: CuaModelCapabilities = Object.freeze({ +const PERMISSIVE_CAPABILITIES: CuaModelCapabilities = Object.freeze({ acceptsComplexSchemas: true, - acceptsLargeSchemas: false, - serializesStateMutations: true, + acceptsLargeSchemas: true, + serializesStateMutations: false, }); -/** - * Per-provider computer-use support annotations. - * - * pi-ai's model registry is generated from models.dev (see - * node_modules/@earendil-works/pi-ai/scripts/generate-models.ts) and lists every - * model a provider offers. Only some of those models support computer-use, so - * this table layers per-provider CUA-support annotations on top of the - * registry. Each entry cites the official source documenting CUA support. - * - * To verify support and add new entries, follow the `update-models` skill at - * .agents/skills/update-models/SKILL.md. - */ -export const CUA_MODEL_ANNOTATIONS: Record = { - openai: [ - { match: { kind: "exact", id: "gpt-5.6-sol" }, source: "https://developers.openai.com/api/docs/models/gpt-5.6-sol" }, - { match: { kind: "family", family: "gpt-5.4" }, source: "https://developers.openai.com/api/docs/models/gpt-5.4" }, - { match: { kind: "family", family: "gpt-5.4-mini" }, source: "https://developers.openai.com/api/docs/models/gpt-5.4-mini" }, - { match: { kind: "family", family: "gpt-5.5" }, source: "https://developers.openai.com/api/docs/models/gpt-5.5" }, - ], - anthropic: [ - { match: { kind: "family", family: "claude-3-7-sonnet" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-opus-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-opus-5" }, source: "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool" }, - { match: { kind: "family", family: "claude-sonnet-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-sonnet-5" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-haiku-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-fable-5" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - ], - google: [ - { match: { kind: "exact", id: "gemini-3.6-flash" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - { match: { kind: "exact", id: "gemini-3.5-flash-lite" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - { match: { kind: "exact", id: "gemini-3.5-flash" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - ], - xai: [ - { match: { kind: "exact", id: "grok-4.5" }, source: "https://docs.x.ai/developers/grok-4-5" }, - ], - // Kimi computer use is custom-function-tool support over Moonshot's - // OpenAI-compatible API, not a provider-native computer tool. K3 ships - // native vision plus screenshot-grounded agentic tool use. - moonshotai: [ - { match: { kind: "exact", id: "kimi-k3" }, source: "https://www.kimi.com/blog/kimi-k3", capabilities: KIMI_K3_CAPABILITIES }, - ], - openrouter: [ - { match: { kind: "exact", id: "moonshotai/kimi-k3" }, source: "https://openrouter.ai/moonshotai/kimi-k3", capabilities: KIMI_K3_CAPABILITIES }, - { match: { kind: "exact", id: "meta/muse-spark-1.1" }, source: "https://openrouter.ai/meta/muse-spark-1.1", capabilities: MUSE_SPARK_CAPABILITIES }, - ], -}; +/** Provider prefixes accepted as aliases for a pi-ai provider id. */ +const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; /** * Split a provider-qualified ref like `"openai:gpt-5.6-sol"` into its parts. * - * `"gemini:"` is accepted as an alias for the canonical `"google:"` prefix - * and normalizes to provider `"google"`; `"moonshot:"` likewise normalizes - * to `"moonshotai"`. Throws when the ref is unqualified or names an - * unsupported provider. + * `"gemini:"` is accepted as an alias for the canonical `"google:"` prefix and + * `"moonshot:"` for `"moonshotai"`. Throws when the ref is unqualified or names + * a provider pi-ai does not carry. */ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: string } { const idx = ref.indexOf(":"); @@ -126,10 +143,10 @@ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: s throw new Error(`CUA model ref must be provider-qualified as ":"; got "${ref}"`); } const prefix = ref.slice(0, idx); - const provider = prefix === "gemini" ? "google" : prefix === "moonshot" ? "moonshotai" : prefix; + const provider = PROVIDER_ALIASES[prefix] ?? prefix; const model = ref.slice(idx + 1); - if (!isCuaProvider(provider)) { - throw new Error(`unsupported CUA provider "${prefix}" (expected one of: ${CUA_PROVIDERS.join(", ")})`); + if (!cuaProviders().includes(provider)) { + throw new Error(`unknown provider "${prefix}" (pi-ai carries: ${cuaProviders().join(", ")})`); } return { provider, model }; } @@ -139,18 +156,21 @@ export function formatCuaModelRef(provider: CuaProvider, model: string): CuaMode return `${provider}:${model}` as CuaModelRef; } +/** Every provider id pi-ai carries. */ +export function cuaProviders(): readonly CuaProvider[] { + return getBuiltinProviders(); +} + /** - * List the computer-use-capable models this package curates, optionally - * filtered to one provider. Merges pi-ai's registry with local overrides and - * keeps only models annotated in {@link CUA_MODEL_ANNOTATIONS}. + * List the models pi-ai carries, optionally filtered to one provider, each + * annotated with the provider-native surfaces CUA can offer for it. */ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { - const providers = provider ? [provider] : [...CUA_PROVIDERS]; + const providers = provider ? [PROVIDER_ALIASES[provider] ?? provider] : [...cuaProviders()]; const byRef = new Map(); for (const p of providers) { for (const model of getBuiltinModels(p as never) as Model[]) { - if (!supportsCuaProvider(p, model.id)) continue; const ref = formatCuaModelRef(p, model.id); if (byRef.has(ref)) continue; byRef.set(ref, { @@ -158,6 +178,8 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { provider: p, model: model.id, name: model.name, + nativeSurfaces: cuaNativeSurfaces(model), + vision: model.input.includes("image"), }); } } @@ -168,68 +190,101 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { /** * Resolve a {@link CuaModelRef} to a concrete pi-ai model. * - * Throws when the ref is unqualified, names an unsupported provider, or names - * a model without a CUA-support annotation. `"gemini:"` refs are accepted as - * an alias for `"google:"` (see {@link parseCuaModelRef}). + * A ref pi-ai's registry does not carry is synthesized from the provider's + * other models, so a model id works the day the provider ships it rather than + * when models.dev catches up. Throws only for an unqualified ref or a provider + * pi-ai does not carry. */ export function getCuaModel(ref: CuaModelRef): Model { const { provider, model: modelId } = parseCuaModelRef(ref); - if (!supportsCuaProvider(provider, modelId)) { - throw new Error(`unsupported CUA model "${ref}"`); - } const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; if (fromRegistry) return fromRegistry; - throw new Error(`CUA model "${ref}" is supported but not carried by pi-ai's registry`); + return synthesizeCuaModel(provider, modelId); } +/** + * Build a model entry for an id pi-ai's registry does not carry, using another + * model from the same provider for the transport, base URL, and compatibility + * fields it cannot know from the id alone. + * + * The template is the sibling sharing the longest id prefix, and the latest + * such sibling when several tie. Providers migrate transports mid-generation — + * xAI carries grok-4.3 on chat completions and grok-4.5 on Responses — so a new + * id should follow its nearest, newest relative rather than whichever model + * happens to come first. + */ +function synthesizeCuaModel(provider: CuaProvider, modelId: string): Model { + const siblings = getBuiltinModels(provider as never) as Model[]; + if (siblings.length === 0) { + throw new Error(`provider "${provider}" carries no models to infer "${modelId}" from`); + } + let template = siblings[0]!; + let bestPrefix = -1; + siblings.forEach((sibling, index) => { + const prefix = sharedPrefixLength(sibling.id.toLowerCase(), modelId.toLowerCase()); + if (prefix >= bestPrefix) { + bestPrefix = prefix; + template = siblings[index]!; + } + }); + return { ...template, id: modelId, name: modelId }; +} + +function sharedPrefixLength(a: string, b: string): number { + let length = 0; + while (length < a.length && length < b.length && a[length] === b[length]) length += 1; + return length; +} -/** Return the {@link CuaProvider} for a concrete model, or throw when it is not a CUA provider. */ +/** Return the provider id for a concrete model. */ export function providerForModel(model: Model): CuaProvider { - if (!isCuaProvider(model.provider)) { - throw new Error(`unsupported CUA model provider "${model.provider}" (expected one of: ${CUA_PROVIDERS.join(", ")})`); - } return model.provider; } -/** Narrow an arbitrary string to {@link CuaProvider}. */ -export function isCuaProvider(value: string): value is CuaProvider { - return (CUA_PROVIDERS as readonly string[]).includes(value); +/** Provider-native tool surfaces available for a model, if any. */ +export function cuaNativeSurfaces(model: Model): readonly CuaNativeSurface[] { + if (model.provider === "anthropic") { + const surfaces: CuaNativeSurface[] = []; + if (supportsAnthropicNativeComputer(model.id)) surfaces.push("computer"); + if (supportsAnthropicNativeBrowser(model.id)) surfaces.push("browser"); + return surfaces; + } + for (const entry of CUA_NATIVE_SURFACES) { + if (entry.provider === model.provider && matchesModelId(model.id, entry.match)) return entry.surfaces; + } + return []; +} + +/** + * Tool-catalog capabilities for a model: permissive unless a quirk says + * otherwise. Provider-wide quirks apply first, then model-specific ones. + */ +export function cuaModelCapabilities(model: Model): CuaModelCapabilities { + let capabilities = PERMISSIVE_CAPABILITIES; + for (const quirk of CUA_MODEL_QUIRKS) { + if (quirk.provider !== model.provider) continue; + if (quirk.match && !matchesModelId(model.id, quirk.match)) continue; + capabilities = { ...capabilities, ...quirk.capabilities }; + } + return capabilities; } -function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean { - return findCuaAnnotation(provider, modelId) !== undefined; +/** Find the quirks that apply to a model, for diagnostics and menu hints. */ +export function cuaModelQuirks(model: Model): readonly CuaModelQuirk[] { + return CUA_MODEL_QUIRKS.filter( + (quirk) => quirk.provider === model.provider && (!quirk.match || matchesModelId(model.id, quirk.match)), + ); } -/** Return tool-catalog capabilities for a model, using annotation or provider defaults. */ -export function cuaModelCapabilities(model: Model): CuaModelCapabilities { - const annotation = isCuaProvider(model.provider) ? findCuaAnnotation(model.provider, model.id) : undefined; - if (annotation?.capabilities) return annotation.capabilities; - const acceptsComplexSchemas = ["openai", "anthropic", "xai", "moonshotai"].includes(model.provider); - return { - acceptsComplexSchemas, - acceptsLargeSchemas: acceptsComplexSchemas && model.provider !== "moonshotai", - serializesStateMutations: ["xai", "moonshotai"].includes(model.provider), - }; -} - -/** Find the CUA-support annotation covering a model id, if any. */ -export function findCuaAnnotation(provider: CuaProvider, modelId: string): CuaModelAnnotation | undefined { +function matchesModelId(modelId: string, match: CuaModelMatch): boolean { const id = modelId.toLowerCase(); - for (const annotation of CUA_MODEL_ANNOTATIONS[provider]) { - if (annotation.match.kind === "exact") { - if (id === annotation.match.id.toLowerCase()) return annotation; - } else if (isCuaFamilyMatch(id, annotation.match.family.toLowerCase())) { - return annotation; - } - } - return undefined; + return match.kind === "exact" ? id === match.id.toLowerCase() : isCuaFamilyMatch(id, match.family.toLowerCase()); } -// A family annotation covers its root id plus suffixes made of -// hyphen-separated numeric segments: revisions like "claude-opus-4-7" and -// dated snapshots like "gpt-5.5-2026-04-23" or "claude-3-7-sonnet-20250219". -// Named sibling variants ("gpt-5.4-mini") may not support computer use and -// must be annotated explicitly. +// A family entry covers its root id plus suffixes made of hyphen-separated +// numeric segments: revisions like "claude-opus-4-7" and dated snapshots like +// "gpt-5.5-2026-04-23". Named sibling variants ("gpt-5.4-mini") are distinct +// models and need their own entry. function isCuaFamilyMatch(id: string, family: string): boolean { if (id === family) return true; if (!id.startsWith(`${family}-`)) return false; @@ -240,6 +295,6 @@ function isCuaFamilyMatch(id: string, family: string): boolean { } function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { - if (a.provider !== b.provider) return CUA_PROVIDERS.indexOf(a.provider) - CUA_PROVIDERS.indexOf(b.provider); + if (a.provider !== b.provider) return a.provider.localeCompare(b.provider); return a.model.localeCompare(b.model); } diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index 206dea2..9a1bd47 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -1,7 +1,7 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; import type { CuaAction } from "./actions/index"; import type { CuaModelRef } from "./models"; -import { cuaModelCapabilities, getCuaModel, providerForModel } from "./models"; +import { cuaModelCapabilities, getCuaModel } from "./models"; import { anthropicAdaptiveThinkingOnPayload } from "./providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, @@ -417,7 +417,6 @@ function validateToolsetCompatibility(model: Model, entries: readonly CuaCa if (nativeProviderKinds.size > 1) { throw new Error(`selected tools contribute incompatible native provider transports: ${[...nativeProviderKinds].join(", ")}`); } - providerForModel(model); const requiresApis = new Set(entries.flatMap((entry) => bindingRequiresApi(entry.providerBinding))); if (requiresApis.size > 1) { diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index 41a55a9..7529fa2 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from "vitest"; import { - CUA_MODEL_ANNOTATIONS, - CUA_PROVIDERS, + CUA_MODEL_QUIRKS, + CUA_NATIVE_SURFACES, type CuaModelRef, - findCuaAnnotation, + cuaModelCapabilities, + cuaNativeSurfaces, + cuaProviders, formatCuaModelRef, getCuaModel, listCuaModels, @@ -16,16 +18,16 @@ describe("CUA model refs", () => { expect(formatCuaModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); }); - it("rejects unqualified and unsupported refs", () => { + it("rejects unqualified refs and unknown providers, but not unknown models", () => { expect(() => getCuaModel("gpt-5.5" as never)).toThrow(/provider-qualified/); - expect(() => getCuaModel("bogus:model" as never)).toThrow(/unsupported CUA provider/); - expect(() => getCuaModel("openai:gpt-3.5" as never)).toThrow(/unsupported CUA model/); + expect(() => getCuaModel("bogus:model" as never)).toThrow(/unknown provider/); + // A model id pi-ai's registry has not caught up with still resolves: the + // provider decides whether it exists, not a table in this package. + expect(getCuaModel("openai:gpt-3.5" as never).id).toBe("gpt-3.5"); }); - it("names the valid providers in the unsupported-provider error", () => { - expect(() => parseCuaModelRef("bogus:model")).toThrow( - 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, xai, moonshotai, openrouter)', - ); + it("names pi-ai's providers in the unknown-provider error", () => { + expect(() => parseCuaModelRef("bogus:model")).toThrow(/unknown provider "bogus" \(pi-ai carries: /); }); it("accepts gemini: as an alias for google:", () => { @@ -120,85 +122,118 @@ describe("CUA model refs", () => { expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); }); - it("rejects supported model IDs that pi-ai does not carry", () => { - // Dated snapshots match the family annotation but pi-ai's registry - // (generated from models.dev) only carries family roots. - expect(() => getCuaModel("openai:gpt-5.5-2026-04-23")).toThrow( - /not carried by pi-ai's registry/, - ); + it("synthesizes models pi-ai's registry does not carry", () => { + // pi-ai's registry (generated from models.dev) carries family roots, not + // dated snapshots, and lags a provider's newest ids. Both still resolve, + // inheriting the transport and base URL from the provider's other models. + const snapshot = getCuaModel("openai:gpt-5.5-2026-04-23"); + expect(snapshot.id).toBe("gpt-5.5-2026-04-23"); + expect(snapshot.provider).toBe("openai"); + expect(snapshot.api).toBe(getCuaModel("openai:gpt-5.5").api); + expect(snapshot.baseUrl).toBe(getCuaModel("openai:gpt-5.5").baseUrl); + + // The motivating case: a model the provider has shipped and models.dev + // has not picked up yet. + expect(getCuaModel("xai:grok-4.6").id).toBe("grok-4.6"); }); -}); -describe("CUA support annotations", () => { - it("covers every provider", () => { - for (const provider of CUA_PROVIDERS) { - expect(CUA_MODEL_ANNOTATIONS[provider].length).toBeGreaterThan(0); - } + it("synthesizes from the nearest, newest sibling", () => { + // xAI carries grok-4.3 on chat completions and grok-4.5 on Responses, so + // picking the wrong sibling would send a new Grok to the wrong transport. + expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); + expect(getCuaModel("xai:grok-4.6").api).toBe("openai-responses"); + expect(getCuaModel("xai:grok-4.6").baseUrl).toBe(getCuaModel("xai:grok-4.5").baseUrl); + expect(getCuaModel("anthropic:claude-opus-6").api).toBe("anthropic-messages"); }); +}); - it("cites an official source for every annotation", () => { - for (const provider of CUA_PROVIDERS) { - for (const annotation of CUA_MODEL_ANNOTATIONS[provider]) { - expect(annotation.source).toMatch(/^https?:\/\//); - } +describe("native surfaces", () => { + it("cites first-party documentation for every entry", () => { + for (const entry of CUA_NATIVE_SURFACES) { + expect(entry.source).toMatch(/^https?:\/\//); + expect(entry.surfaces.length).toBeGreaterThan(0); } }); it("matches family roots, dated snapshots, and numeric revisions", () => { - expect(findCuaAnnotation("openai", "gpt-5.5")?.match).toEqual({ kind: "family", family: "gpt-5.5" }); - expect(findCuaAnnotation("openai", "gpt-5.5-2026-04-23")?.match).toEqual({ kind: "family", family: "gpt-5.5" }); - expect(findCuaAnnotation("openai", "gpt-5.4-mini")?.match).toEqual({ kind: "family", family: "gpt-5.4-mini" }); - expect(findCuaAnnotation("openai", "gpt-5.4-mini-2026-03-17")?.match).toEqual({ kind: "family", family: "gpt-5.4-mini" }); - expect(findCuaAnnotation("anthropic", "claude-opus-4-7")).toBeDefined(); - expect(findCuaAnnotation("anthropic", "claude-opus-5")?.match).toEqual({ kind: "family", family: "claude-opus-5" }); - expect(findCuaAnnotation("anthropic", "claude-opus-5-20260724")?.match).toEqual({ kind: "family", family: "claude-opus-5" }); - expect(findCuaAnnotation("anthropic", "claude-3-7-sonnet-20250219")).toBeDefined(); - }); - - it("does not match adjacent families", () => { - expect(findCuaAnnotation("openai", "gpt-5.55-foo")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.6")).toBeUndefined(); - expect(findCuaAnnotation("anthropic", "claude-3-5-sonnet")).toBeUndefined(); - }); - - it("does not match named sibling variants of a family", () => { - expect(findCuaAnnotation("openai", "gpt-5.4-nano")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.4-pro")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.5-pro")).toBeUndefined(); - const openaiModels = listCuaModels("openai").map((model) => model.model); - expect(openaiModels).not.toContain("gpt-5.4-nano"); - expect(openaiModels).not.toContain("gpt-5.4-pro"); - expect(openaiModels).toContain("gpt-5.5"); - }); - - it("matches exact-id annotations", () => { - expect(findCuaAnnotation("openai", "gpt-5.6-sol")?.match).toEqual({ kind: "exact", id: "gpt-5.6-sol" }); - expect(findCuaAnnotation("openai", "gpt-5.6-sol-20260728")).toBeUndefined(); - expect(findCuaAnnotation("google", "gemini-3.6-flash")).toBeDefined(); - expect(findCuaAnnotation("openrouter", "meta/muse-spark-1.1")).toBeDefined(); - expect(findCuaAnnotation("xai", "grok-4.5")).toBeDefined(); - expect(findCuaAnnotation("xai", "grok-4.5-latest")).toBeUndefined(); - expect(findCuaAnnotation("xai", "grok-4.3")).toBeUndefined(); - expect(findCuaAnnotation("moonshotai", "kimi-k3")).toBeDefined(); - expect(findCuaAnnotation("moonshotai", "kimi-k2.5")).toBeUndefined(); - expect(findCuaAnnotation("moonshotai", "kimi-latest")).toBeUndefined(); - expect(findCuaAnnotation("google", "gemini-3.5-flash-lite")).toBeDefined(); - }); - - it("advertises only Google's current documented computer-use models", () => { - expect(listCuaModels("google").map((model) => model.model)).toEqual([ - "gemini-3.5-flash", - "gemini-3.5-flash-lite", - "gemini-3.6-flash", - ]); - for (const retired of [ - "gemini-2.5-computer-use-preview-10-2025", - "gemini-3-flash-preview", - "gemini-3.1-flash-lite", - "gemini-3-pro-preview", - ]) { - expect(findCuaAnnotation("google", retired)).toBeUndefined(); - expect(() => getCuaModel(`google:${retired}` as CuaModelRef)).toThrow(/unsupported CUA model/); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5-2026-04-23"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-mini"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5"))).toEqual(["computer", "browser"]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5-20260724"))).toEqual(["computer", "browser"]); + }); + + it("does not match adjacent families or named sibling variants", () => { + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-nano"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-pro"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-3-5-sonnet"))).toEqual([]); + }); + + it("reports no native surface for models that have none, without refusing them", () => { + expect(cuaNativeSurfaces(getCuaModel("moonshotai:kimi-k3"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("xai:grok-4.5"))).toEqual([]); + // A model with no native surface still resolves and runs on CUA's own tools. + expect(getCuaModel("xai:grok-4.5").provider).toBe("xai"); + }); + + it("surfaces the flag on catalog listings", () => { + const google = listCuaModels("google"); + const flash = google.find((model) => model.model === "gemini-3.6-flash"); + expect(flash?.nativeSurfaces).toEqual(["browser"]); + expect(flash?.vision).toBe(true); + expect(google.some((model) => model.nativeSurfaces.length === 0)).toBe(true); + }); +}); + +describe("model quirks", () => { + it("explains why every quirk exists", () => { + for (const quirk of CUA_MODEL_QUIRKS) { + expect(quirk.reason.length).toBeGreaterThan(20); + expect(Object.keys(quirk.capabilities).length).toBeGreaterThan(0); } }); + + it("defaults to permissive for a model with no quirk", () => { + expect(cuaModelCapabilities(getCuaModel("openai:gpt-5.6-sol"))).toEqual({ + acceptsComplexSchemas: true, + acceptsLargeSchemas: true, + serializesStateMutations: false, + }); + // Including a model pi-ai's registry does not carry. + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).acceptsComplexSchemas).toBe(true); + }); + + it("keeps the limits we have evidence for", () => { + // Observed live: the Gemini API rejects browser_wait_for's schema shape. + expect(cuaModelCapabilities(getCuaModel("google:gemini-3.6-flash")).acceptsComplexSchemas).toBe(false); + // Observed live: Kimi K3 rejects the request once browser_act is attached. + expect(cuaModelCapabilities(getCuaModel("moonshotai:kimi-k3")).acceptsLargeSchemas).toBe(false); + expect(cuaModelCapabilities(getCuaModel("openrouter:moonshotai/kimi-k3")).acceptsLargeSchemas).toBe(false); + // Muse Spark accepts the large schema but serializes state mutations. + const muse = cuaModelCapabilities(getCuaModel("openrouter:meta/muse-spark-1.1")); + expect(muse.acceptsLargeSchemas).toBe(true); + expect(muse.serializesStateMutations).toBe(true); + }); + + it("applies a provider-wide quirk to every model from that provider", () => { + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.5")).serializesStateMutations).toBe(true); + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).serializesStateMutations).toBe(true); + }); +}); + +describe("catalog passthrough", () => { + it("exposes every provider pi-ai carries", () => { + expect(cuaProviders().length).toBeGreaterThan(20); + expect(cuaProviders()).toContain("openai"); + expect(cuaProviders()).toContain("groq"); + expect(cuaProviders()).toContain("zai"); + }); + + it("lists models no CUA table mentions", () => { + const all = listCuaModels(); + expect(all.length).toBeGreaterThan(100); + // Previously refused for want of a table entry. + expect(all.some((model) => model.ref === "xai:grok-4.3")).toBe(true); + expect(all.some((model) => model.provider === "groq")).toBe(true); + }); }); diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 68ba217..cb4559d 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## 0.13.0 - 2026-08-14 + +- `cua models` lists every model pi-ai carries, not a curated subset, and `-p` + accepts any provider it carries. +- `-m` accepts any model id. A bare id that several providers carry now resolves + to the first-party provider rather than erroring, since gateways resell the + same ids; pass a qualified `provider:model` ref to reach a specific one. +- The API-key preflight now runs only for providers CUA documents variable + names for. Any other pi-ai provider is still selectable; pi resolves its + credential when it streams, and failing up front would refuse a model that + works. +- The default interaction toolset is chosen from the model rather than its + provider: a model with a native browser surface gets it, and everything else + gets CUA's CDP tools, with `browser_act` included only where the model accepts + its schema. + ## 0.12.0 - 2026-08-13 - `-m meta:muse-spark-1.1` is removed; use `-m openrouter:meta/muse-spark-1.1`. diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index 5193c0a..0f99c03 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -7,6 +7,7 @@ import { type Skill, } from "@onkernel/cua-agent"; import { + cuaApiKeyEnvVarsForProvider, type CuaModelRef, parseCuaModelRef, requireCuaEnvApiKey, @@ -48,7 +49,7 @@ import { import { type ContextFile, discoverCuaSkills } from "./harness-skills"; import { runPrint } from "./print"; -const MODELS_HELP = `cua models — list supported -m/--model values +const MODELS_HELP = `cua models — list selectable -m/--model values Usage: cua models @@ -57,7 +58,7 @@ Usage: cua models --json Options: - -p, --provider Filter by provider: openai | anthropic | google | gemini | xai | moonshotai | openrouter + -p, --provider Filter by provider id (any pi-ai provider; gemini/moonshot are aliases) --json Output JSON -h, --help Show this help `; @@ -124,14 +125,16 @@ function formatModelsTable(models: ReturnType): stri provider: entry.provider, model: entry.model, default: entry.ref === DEFAULT_CUA_MODEL_REF ? "yes" : "", + native: entry.nativeSurfaces.join(","), name: entry.name, })); - const headers = { ref: "REF", provider: "PROVIDER", model: "MODEL", default: "DEFAULT", name: "NAME" }; + const headers = { ref: "REF", provider: "PROVIDER", model: "MODEL", default: "DEFAULT", native: "NATIVE", name: "NAME" }; const widths = { ref: columnWidth(headers.ref, rows.map((r) => r.ref)), provider: columnWidth(headers.provider, rows.map((r) => r.provider)), model: columnWidth(headers.model, rows.map((r) => r.model)), default: columnWidth(headers.default, rows.map((r) => r.default)), + native: columnWidth(headers.native, rows.map((r) => r.native)), name: columnWidth(headers.name, rows.map((r) => r.name)), }; const lines = [ @@ -140,6 +143,7 @@ function formatModelsTable(models: ReturnType): stri headers.provider.padEnd(widths.provider), headers.model.padEnd(widths.model), headers.default.padEnd(widths.default), + headers.native.padEnd(widths.native), headers.name, ].join(" "), [ @@ -147,6 +151,7 @@ function formatModelsTable(models: ReturnType): stri "-".repeat(widths.provider), "-".repeat(widths.model), "-".repeat(widths.default), + "-".repeat(widths.native), "-".repeat(widths.name), ].join(" "), ]; @@ -157,6 +162,7 @@ function formatModelsTable(models: ReturnType): stri row.provider.padEnd(widths.provider), row.model.padEnd(widths.model), row.default.padEnd(widths.default), + row.native.padEnd(widths.native), row.name, ].join(" "), ); @@ -214,8 +220,10 @@ function resolveAuth(flags: HarnessCliFlags): ResolvedAuth { const { apiKey, baseUrl } = requireKernelApiKey(); const modelRef = resolveCuaModelRef(flags.model); const { provider } = parseCuaModelRef(modelRef); - // Throws naming the env vars the user must set (`requireCuaEnvApiKey`). - requireCuaEnvApiKey(provider); + // Preflight only where CUA documents the variable names; for any other + // pi-ai provider the credential is pi's to resolve when it streams, and + // failing here would refuse a model that works. + if (cuaApiKeyEnvVarsForProvider(provider).length > 0) requireCuaEnvApiKey(provider); return { kernelApiKey: apiKey, kernelBaseUrl: baseUrl, modelRef }; } diff --git a/packages/cli/src/harness-models.ts b/packages/cli/src/harness-models.ts index b691ca6..28ecec2 100644 --- a/packages/cli/src/harness-models.ts +++ b/packages/cli/src/harness-models.ts @@ -4,7 +4,7 @@ import { type CuaProvider, formatCuaModelRef, getCuaModel, - isCuaProvider, + cuaProviders, listCuaModels, parseCuaModelRef, } from "@onkernel/cua-ai"; @@ -13,9 +13,21 @@ import { export const DEFAULT_CUA_MODEL_REF: CuaModelRef = "openai:gpt-5.6-sol"; /** - * Resolve a model ref from CLI input. Accepts either a provider-qualified - * `provider:model` ref or a bare model id when it matches exactly one - * catalog entry. Throws when bare ids are ambiguous or unknown. + * Providers preferred when a bare model id is carried by several of them, in + * order. Gateways and aggregators resell the same ids as the provider that + * trained the model, so `-m gpt-5.5` should mean OpenAI's. + * + * This is a disambiguation preference for bare ids only. It never decides + * whether a model may run, and any provider is still reachable by passing a + * qualified `provider:model` ref. + */ +const BARE_ID_PROVIDER_PREFERENCE: readonly string[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; + +/** + * Resolve a model ref from CLI input. Accepts a provider-qualified + * `provider:model` ref, or a bare model id when exactly one provider carries it + * or one of the preferred providers does. Throws when a bare id is unknown, or + * ambiguous among providers none of which is preferred. */ export function resolveCuaModelRef(input: string | undefined): CuaModelRef { if (!input || !input.trim()) return DEFAULT_CUA_MODEL_REF; @@ -32,22 +44,24 @@ export function resolveCuaModelRef(input: string | undefined): CuaModelRef { throw new Error(`unknown model "${value}" (run \`cua models\` to list supported -m/--model values)`); } if (matches.length > 1) { + const preferred = BARE_ID_PROVIDER_PREFERENCE.map((provider) => matches.find((m) => m.provider === provider)).find(Boolean); + if (preferred) return preferred.ref; const refs = matches.map((m) => m.ref).join(", "); - throw new Error(`ambiguous model "${value}" (matches: ${refs}); pass a provider-qualified ref like "openai:${value}"`); + throw new Error(`ambiguous model "${value}" (matches: ${refs}); pass a provider-qualified ref`); } return matches[0]!.ref; } /** - * List supported models, optionally filtered to a provider. Accepts the - * canonical `"google"`/`"moonshotai"` ids or the CLI-friendly `"gemini"`/ - * `"moonshot"` aliases. + * List selectable models, optionally filtered to a provider. Accepts any + * provider pi-ai carries, plus the CLI-friendly `"gemini"`/`"moonshot"` + * aliases. */ export function listSupportedModels(provider?: string): CuaModelInfo[] { if (!provider) return listCuaModels(); const normalized = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; - if (!isCuaProvider(normalized)) { - throw new Error(`unknown provider "${provider}"`); + if (!cuaProviders().includes(normalized)) { + throw new Error(`unknown provider "${provider}" (pi-ai carries: ${cuaProviders().join(", ")})`); } - return listCuaModels(normalized as CuaProvider); + return listCuaModels(normalized); } diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index f9f71dd..a21a5de 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -12,6 +12,7 @@ import { type Api, cua, cuaModelCapabilities, + cuaNativeSurfaces, type CuaModelRef, getCuaModel, type Model, @@ -93,29 +94,27 @@ function structuredBrowserTools(): CuaCliTool[] { return [...cua.toolsets.browser(), cua.tools.browser.act()]; } -/** CLI policy is explicit application composition, not a CuaAgent default. */ +/** + * CLI interaction policy, asked of the model rather than switched on its + * provider: a model with a provider-native browser surface gets that surface, + * and everything else gets CUA's CDP browser tools, with `browser_act` included + * only where the model accepts its schema. + * + * OpenAI's native computer tool is deliberately not a default: it is a distinct + * interaction style callers opt into through `--tools` or the `/tools` picker. + */ export function defaultInteractionTools(model: CuaModelRef): CuaCliTool[] { - const { provider, model: modelId } = parseCuaModelRef(model); - switch (provider) { - case "openai": - return structuredBrowserTools(); - case "anthropic": - return cua.providers.anthropic.supports.browser(modelId) - ? [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })] - : structuredBrowserTools(); - case "google": - return cua.providers.google.toolsets.browser(); - case "xai": - return structuredBrowserTools(); - case "moonshotai": - case "openrouter": - // Kimi's API rejects the request outright once `browser_act`'s schema - // is attached. OpenRouter fronts several model families, so this is a - // per-model capability question rather than a per-provider one. - return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas - ? structuredBrowserTools() - : cua.toolsets.browser(); + const { provider } = parseCuaModelRef(model); + const resolved = getCuaModel(model); + if (cuaNativeSurfaces(resolved).includes("browser")) { + if (provider === "anthropic") { + return [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; + } + if (provider === "google") return cua.providers.google.toolsets.browser(); } + return cuaModelCapabilities(resolved).acceptsLargeSchemas + ? structuredBrowserTools() + : cua.toolsets.browser(); } function composeSystemPrompt(skills: Skill[], contextFiles: ContextFile[]): string { diff --git a/packages/cli/test/harness-models.test.ts b/packages/cli/test/harness-models.test.ts index 2f87f79..b27c4d5 100644 --- a/packages/cli/test/harness-models.test.ts +++ b/packages/cli/test/harness-models.test.ts @@ -30,14 +30,22 @@ describe("resolveCuaModelRef", () => { expect(() => resolveCuaModelRef("does-not-exist")).toThrow(/unknown model/); }); - it("filters custom provider catalogs", () => { - expect(listSupportedModels("xai").map((model) => model.ref)).toEqual(["xai:grok-4.5"]); - expect(listSupportedModels("moonshotai").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("moonshot").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:meta/muse-spark-1.1", "openrouter:moonshotai/kimi-k3"]); + it("filters to a provider's whole catalog", () => { + // No allowlist: every model the provider carries is listed, including the + // ones no CUA table mentions. + const xai = listSupportedModels("xai").map((model) => model.ref); + expect(xai).toContain("xai:grok-4.5"); + expect(xai).toContain("xai:grok-4.3"); + expect(listSupportedModels("moonshotai").map((model) => model.ref)).toContain("moonshotai:kimi-k3"); + expect(listSupportedModels("moonshot").map((model) => model.ref)).toContain("moonshotai:kimi-k3"); + expect(listSupportedModels("openrouter").map((model) => model.ref)).toContain("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("openrouter:moonshotai/kimi-k3")).toBe("openrouter:moonshotai/kimi-k3"); }); + it("rejects a provider pi-ai does not carry", () => { + expect(() => listSupportedModels("bogus")).toThrow(/unknown provider "bogus"/); + }); + it("treats 'gemini' as an alias for google when filtering", () => { const fromGemini = listSupportedModels("gemini"); const fromGoogle = listSupportedModels("google"); diff --git a/packages/cli/test/model-picker.test.ts b/packages/cli/test/model-picker.test.ts index 621a212..664c309 100644 --- a/packages/cli/test/model-picker.test.ts +++ b/packages/cli/test/model-picker.test.ts @@ -39,8 +39,10 @@ describe("filterModelsForPicker", () => { const google = catalog.filter((m) => m.provider === "google"); expect(google.length).toBeGreaterThan(0); const filtered = filterModelsForPicker(catalog, "google"); - expect(filtered.length).toBeGreaterThan(0); - expect(filtered.every((m) => m.provider === "google")).toBe(true); + // The full pi-ai catalog contains related providers (google-vertex), so + // the query narrows rather than isolating one provider. + expect(filtered.some((m) => m.provider === "google")).toBe(true); + expect(filtered.some((m) => m.provider === "anthropic")).toBe(false); }); it("returns an empty list when nothing matches", () => { From 05e0181d889880bcc03893cb75a0e3e6e7ffcc22 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Fri, 14 Aug 2026 01:44:50 +0000 Subject: [PATCH 2/2] Retire the update-models skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill existed to maintain the model allowlist: enumerate every provider's models, smoke-test each one, decide whether it counted as CUA-supported, and write the verdict into a table. With the allowlist gone there is no verdict to reach and no table to sweep, so the periodic audit it prescribed has nothing to service. What remains is reactive and belongs next to the tables it describes, so `supported-models.md` gains a short section covering the four cases: a provider released a model (nothing to do), the catalog looks stale (bump pi-ai), a provider changed a native tool (probe it, update the adapter, adjust the surface entry), and a model rejects a tool we send (add a quirk with the error as its reason). `native-action-probe.ts` survives as `packages/ai/scripts/` — eliciting what a model actually emits is how you find out an adapter broke. The other three scripts were evidence-gathering for allowlist entries: enumerating provider metadata, cloning provider example repos, and diffing documented action names against local constants. Also documents that cua does not read pi's `models.json`, since that is a pi-coding-agent config file and cua builds its collection from pi-ai directly. --- .agents/skills/update-models/SKILL.md | 206 ----- .../skills/update-models/reference/README.md | 56 -- .../reference/audit-official-examples.ts | 265 ------ .../reference/discover-models.ts | 757 ------------------ .../reference/provider-doc-drift.ts | 199 ----- .../update-models/reference/report-schema.md | 100 --- packages/ai/docs/supported-models.md | 36 + .../ai/scripts}/native-action-probe.ts | 6 +- 8 files changed, 39 insertions(+), 1586 deletions(-) delete mode 100644 .agents/skills/update-models/SKILL.md delete mode 100644 .agents/skills/update-models/reference/README.md delete mode 100644 .agents/skills/update-models/reference/audit-official-examples.ts delete mode 100644 .agents/skills/update-models/reference/discover-models.ts delete mode 100644 .agents/skills/update-models/reference/provider-doc-drift.ts delete mode 100644 .agents/skills/update-models/reference/report-schema.md rename {.agents/skills/update-models/reference => packages/ai/scripts}/native-action-probe.ts (97%) diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md deleted file mode 100644 index b2bc092..0000000 --- a/.agents/skills/update-models/SKILL.md +++ /dev/null @@ -1,206 +0,0 @@ ---- -name: update-models -description: Detect drift in CUA's provider-native tool surfaces and per-model request limits. Use when a provider ships a new native computer/browser tool version, when a model starts rejecting a tool schema CUA sends, or when auditing provider-native action vocabularies against official docs and examples. ---- - -# Update Models - -Use this workflow to keep CUA's two model tables honest. Note what it is **not** for: CUA has no model allowlist. Every model pi-ai carries is selectable, and an id pi-ai has not caught up with is synthesized from its nearest sibling, so a newly released model needs no repo change to be usable. - -What still needs maintaining is narrow: - -- `CUA_NATIVE_SURFACES` — which models have a provider-native computer or browser tool CUA can offer. -- `CUA_MODEL_QUIRKS` — request-shape limits, each justified by a documented limit or an observed failure. -- The provider adapters themselves, when a native action vocabulary or tool version changes. - -Do not trust a static list for any of those: combine provider metadata, official docs, official example repos, and live non-destructive smoke tests. - -## Quick Start - -1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. -2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them: - -```bash -eval "$(python3 - <<'PY' -import pathlib, re, shlex -text = pathlib.Path('~/AGENTS.md').expanduser().read_text() -for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: - m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text) - if m: - print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}') -PY -)" -``` - -3. From the repo root, run the all-provider probe: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json -``` - -4. Audit official examples for tool shape drift: - -```bash -npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json -``` - -5. Compare docs, examples, live probes, and local adapter constants: - -```bash -npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -``` - -6. Summarize findings with the template in `reference/report-schema.md`. Only recommend repo changes after checking the decision rules below. - -## Evidence Order - -Use all four evidence sources when possible: - -- Provider metadata APIs: tells us what models are available to this API key. -- Official docs: tells us intended tool names, dated beta headers, and documented action vocabularies. -- Model-specific docs: tells us endpoint, streaming, feature, and tool support for a specific model ID. -- Official example repos: shows real response parsing, action execution, safety handling, and follow-up payload shapes. -- Live smoke tests: confirms the current model/API combination can emit provider-native computer-use tool calls. -- Local cua-ai smoke tests: confirms `@onkernel/cua-ai` resolves the model through `getCuaModel()` and its provider adapter emits executable CUA tool calls. - -Treat example repos as strongest when they are provider-owned or linked from official docs. If discovered through search only, mark them lower confidence until verified. - -## Model Enumeration - -There are two enumeration layers: - -- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access. -- Selectable refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` returns pi-ai's whole catalog, each entry marked with the native surfaces CUA can offer for it. `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` answer those two questions for a single model. - -When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("")`. - -## Provider Checks - -Meta: - -- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response output contains a `function_call` for one of the supplied browser actions. -- Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`. -- Set `parallel_tool_calls: false` because browser actions mutate shared state. -- Treat Meta computer use as custom-function-tool support, not a provider-native `{ type: "computer" }` tool. - -OpenAI: - -- Discover with `OpenAI().models.list()` and optionally `models.retrieve(modelId)`. -- OpenAI model metadata is sparse (`id`, `created`, `owned_by`), so computer-use support must be smoke-tested. -- Check the model-specific docs page at `https://developers.openai.com/api/docs/models/` before adding support. For aliases/snapshots, check the canonical family page too, e.g. `gpt-5.5-pro-2026-04-23` -> `gpt-5.5-pro`. -- For CUA support, require `Responses` endpoint support, `Streaming` support, and `Function calling` support. Do not list models like `gpt-5.5-pro` that say `Streaming: Not supported`. -- For provider-native OpenAI computer use, require `Computer use: Supported`. If a model supports function calling but not native `computer`, label it custom-tool-only and do not treat it as provider-native computer-use support. -- Smoke-test `responses.create` with `tools: [{ type: "computer" }]` and `tool_choice: { type: "computer" }`. -- Pass condition: response output contains `type: "computer_call"` with `actions[]` or legacy `action`. -- Audit official examples for `computer_call`, `actions`, `computer_call_output`, `pending_safety_checks`, and screenshot payload handling. - -Anthropic: - -- Discover with `Anthropic().models.list({ limit: 1000 })`. -- Record `id`, `display_name`, `created_at`, token limits, and `capabilities`. -- Smoke-test `client.beta.messages.create` with discovered computer tool and beta pairs, newest first. -- Pass condition: `stop_reason === "tool_use"` and a `tool_use` block named `computer`. -- For CUA support, the passing pair should match the Anthropic tool version and beta header the cua-ai runtime (via `pi-ai`) sends for that model; `discover-models.ts` reports this as `runtime_compatible`. A pass on a different pair is provider support that needs a `pi-ai` bump before the runtime can use it. -- Watch for dated drift: `computer_YYYYMMDD` tool names and `computer-use-YYYY-MM-DD` beta headers. - -Google/Gemini: - -- Discover with `GoogleGenAI().models.list()` and `models.get(...)`. -- Filter models that support `generateContent`, then test official `computer_use`. -- Pass condition: response contains provider-native `functionCall.name` values such as `open_web_browser`, `click_at`, or `type_text_at`. -- Do not infer official computer-use support from CUA's custom Gemini `functionDeclarations`; those are a separate compatibility path. - -xAI: - -- Discover with the OpenAI SDK against `https://api.x.ai/v1` using `XAI_API_KEY`. -- Record aliases, context length, standard and long-context token prices, and the 200k long-context threshold returned by `models.list()`. -- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response output contains a `function_call` for one of the supplied browser actions. -- Treat Grok computer use as custom-function-tool support, not a provider-native computer tool. xAI currently documents image understanding and function calling but no native coordinate protocol. -- Use CUA's normalized 0-1000 coordinate instructions, `parallel_tool_calls: false`, `store: true`, and `previous_response_id` for browser loops. xAI accepts encrypted reasoning replay in these requests. -- Use `reasoning: { effort: "low" }` for low-latency smoke tests; Grok 4.5 also supports `medium` and `high`, cannot disable reasoning, and defaults to `high`. - -Moonshot: - -- Discover with the OpenAI SDK against `https://api.moonshot.ai/v1` using `MOONSHOT_API_KEY`. -- Smoke-test the chat completions API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response `choices[0].message.tool_calls[]` contains one of the supplied browser actions. -- Treat Kimi computer use as custom-function-tool support, not a provider-native computer tool. Moonshot documents vision input and function calling but no coordinate protocol. -- Kimi grounding emits width/height fractions from 0 to 1 regardless of prompt or schema wording; keep CUA's fractional coordinate contract and verify emitted values stay in 0-1. -- Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn. -- Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes. - -## Native Action Discovery - -Run action probes when updating adapters or when docs/examples show drift: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 -``` - -The probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare: - -- `documented_actions`: extracted from provider docs or SDK source. -- `example_repo_actions`: extracted from official examples. -- `observed_actions`: emitted by live smoke probes. -- `repo_supported_actions`: local adapter constants. -- `unknown_observed_actions`: actions emitted by providers but not supported locally. - -## Decision Rules - -Add a `CUA_NATIVE_SURFACES` entry only if: - -- The provider documents a native computer or browser tool for that model. -- A live probe elicits a native tool call: `npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider

--model `. -- The local adapter can execute the actions the probe emits; otherwise the adapter needs updating first. - -Add a `CUA_MODEL_QUIRKS` entry only if you can state the failure it prevents. A quirk is not a preference — it is -a request CUA must not send because the provider rejects it. Record the evidence in the entry's `reason`, and -prefer the narrowest scope that covers it: a single model id over a family, a family over a whole provider. - -Remove a quirk when the provider lifts the limit. A stale quirk silently denies a model a tool it now accepts, -which is harder to notice than the reverse. - -Recommend adapter updates when: - -- A provider exposes a newer dated tool version or beta header. -- Official examples handle response fields the local adapter ignores. -- Smoke probes emit native actions not present in local constants. - -Do not print API keys. Keep smoke tests non-destructive. Do not edit repo defaults or adapters unless the user -explicitly asks after reviewing the report. - -## Updating CUA Support - -All model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). - -- **New model id, no native surface, no quirk**: nothing to do. It already works. -- **New model with a provider-native tool**: add a `CUA_NATIVE_SURFACES` entry in `packages/ai/src/models.ts` - citing first-party documentation. Use a `family` match to cover numeric revisions and dated snapshots, or an - `exact` match for a single id. Anthropic's surfaces are version-gated in - `providers/anthropic/capabilities.ts` instead. -- **A model rejects a tool CUA sends**: add a `CUA_MODEL_QUIRKS` entry with the observed error as its `reason`. -- **New provider-native action, response field, or tool version**: update that provider's adapter under - `packages/ai/src/providers/`. Anthropic's computer tool version and `computer-use-*` beta header are selected - by pi-ai per model, so a new dated version usually means bumping `@earendil-works/pi-ai`. -- **A provider pi-ai does not carry**: it is not selectable. Adding one means registering a provider in - `packages/ai/src/providers.ts`, which is a deliberate decision — the repo has removed four such providers - rather than carry them unused. - -Update `packages/ai/docs/supported-models.md` to match either table change. Then run `npm run typecheck`, -`npm test --workspace @onkernel/cua-ai`, and at least one live smoke per changed provider, for example -`CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai`. - -## Reference Files - -- `reference/README.md`: script usage and output overview. -- `reference/discover-models.ts`: provider metadata plus smoke-test orchestration. -- `reference/native-action-probe.ts`: live provider-native action elicitation. -- `reference/audit-official-examples.ts`: clone/update official examples and extract implementation evidence. -- `reference/provider-doc-drift.ts`: compare docs/examples/local constants for drift. -- `reference/report-schema.md`: normalized report fields and Markdown summary template. diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md deleted file mode 100644 index 2c0bca4..0000000 --- a/.agents/skills/update-models/reference/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Update Models Reference - -These scripts support the `update-models` skill. Run them from the repository root. - -## Requirements - -- Node 20+ -- Repository dependencies installed with `npm install` -- TypeScript runner available through `npx tsx` or another local TS runner -- Provider API keys as needed: - - `OPENAI_API_KEY` - - `ANTHROPIC_API_KEY` - - `GOOGLE_API_KEY` or `GEMINI_API_KEY` - - `XAI_API_KEY` - - `MOONSHOT_API_KEY` - -The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools. - -## Common Commands - -Discover all providers and smoke-test likely candidates: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json -``` - -Probe native action vocabularies for a specific provider/model: - -```bash -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json -``` - -Clone/update official examples and extract tool-handling evidence: - -```bash -npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json -``` - -Examples are cached under `/tmp/cua-update-models/examples` by default so cloned upstream repos do not appear as untracked files in this repository. - -Compare official docs, examples, and local adapter constants: - -```bash -npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -``` - -## Evidence Types - -- `metadata`: provider model-list APIs. -- `docs`: provider docs or SDK source fetched live. -- `examples`: provider-owned or doc-linked example repos. -- `smoke`: live API response shape for provider-native computer use. -- `local`: constants in this repo's provider adapters. - -Use `report-schema.md` when summarizing results for humans. diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts deleted file mode 100644 index 330b53b..0000000 --- a/.agents/skills/update-models/reference/audit-official-examples.ts +++ /dev/null @@ -1,265 +0,0 @@ -#!/usr/bin/env tsx -import { existsSync } from "node:fs"; -import { mkdir, readFile, readdir, writeFile } from "node:fs/promises"; -import { basename, join, resolve } from "node:path"; -import { spawnSync } from "node:child_process"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface ExampleRepo { - provider: Provider; - name: string; - repo: string; - confidence: string; - pathHint?: string; - patterns: string[]; -} - -interface Args { - cache: string; - out: string; - noUpdate: boolean; -} - -interface GitResult { - status: number | null; - stdout: string; - stderr: string; -} - -const EXAMPLES: ExampleRepo[] = [ - { - provider: "openai", - name: "openai-cua-sample-app", - repo: "https://github.com/openai/openai-cua-sample-app.git", - confidence: "provider-owned", - patterns: ["computer_call", "computer_call_output", "actions", "pending_safety_checks", "computer_use_preview"], - }, - { - provider: "anthropic", - name: "anthropic-quickstarts", - repo: "https://github.com/anthropics/anthropic-quickstarts.git", - confidence: "provider-owned", - pathHint: "computer-use-demo", - patterns: ["computer_", "computer-use-", "tool_use", "tool_result", "input.action"], - }, - { - provider: "gemini", - name: "computer-use-preview", - repo: "https://github.com/google/computer-use-preview.git", - confidence: "provider-owned", - patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"], - }, - { - provider: "xai", - name: "xai-sdk-python", - repo: "https://github.com/xai-org/xai-sdk-python.git", - confidence: "provider-owned", - pathHint: "examples", - patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"], - }, -]; - -const ACTION_REGEXES: Record = { - openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g], - anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g], - gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g], - xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], - moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], -}; - -function parseArgs(argv: string[]): Args { - const out: Args = { - cache: "/tmp/cua-update-models/examples", - out: "", - noUpdate: false, - }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--cache" && next) { - out.cache = next; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--no-update") { - out.noUpdate = true; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json - -Options: - --cache

Clone/update examples here. Default: /tmp/cua-update-models/examples - --no-update Do not git pull existing repos. - --out Write JSON report to file. -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const cacheDir = resolve(process.cwd(), args.cache); - await mkdir(cacheDir, { recursive: true }); - - const repos = []; - for (const example of EXAMPLES) { - repos.push(await auditRepo(example, cacheDir, args)); - } - - const report = { - generated_at: new Date().toISOString(), - cache_dir: cacheDir, - repos, - by_provider: groupByProvider(repos), - }; - await emitJson(report, args.out); -} - -async function auditRepo(example: ExampleRepo, cacheDir: string, args: Args): Promise> { - const dir = join(cacheDir, example.name); - const cloneOrUpdate = ensureRepo(example.repo, dir, args.noUpdate); - const commit = git(["rev-parse", "HEAD"], dir).stdout.trim() || null; - const files = await collectFiles(example.pathHint ? join(dir, example.pathHint) : dir); - const matches = []; - const toolVersions = new Set(); - const betaHeaders = new Set(); - const actionNames = new Set(); - const responseFields = new Set(); - - for (const file of files) { - const text = await readFile(file, "utf8").catch(() => ""); - if (!text) continue; - const foundPatterns = example.patterns.filter((p) => text.includes(p)); - if (foundPatterns.length === 0) continue; - - extractAll(text, /computer_\d{8}/g).forEach((v) => toolVersions.add(v)); - extractAll(text, /computer-use-\d{4}-\d{2}-\d{2}/g).forEach((v) => betaHeaders.add(v)); - for (const regex of ACTION_REGEXES[example.provider] ?? []) { - extractAll(text, regex).forEach((v) => actionNames.add(v)); - } - for (const field of ["computer_call", "actions", "action", "pending_safety_checks", "tool_use", "tool_result", "tool_calls", "function_call", "functionCall", "FunctionResponse", "safety_decision", "previous_response_id", "parallel_tool_calls", "reasoning_effort"]) { - if (text.includes(field)) responseFields.add(field); - } - - matches.push({ - file: relativePath(dir, file), - patterns: foundPatterns, - snippets: snippets(text, foundPatterns), - }); - } - - return { - provider: example.provider, - name: example.name, - repo: example.repo, - confidence: example.confidence, - local_path: dir, - commit, - clone_or_update: cloneOrUpdate, - tool_versions: sorted(toolVersions), - beta_headers: sorted(betaHeaders), - action_names: sorted(actionNames), - response_fields: sorted(responseFields), - matches, - }; -} - -function ensureRepo(repo: string, dir: string, noUpdate: boolean): Record { - if (!existsSync(dir)) { - const res = git(["clone", "--depth", "1", repo, dir], process.cwd()); - return { action: "clone", ok: res.status === 0, stderr: res.stderr.trim() }; - } - if (noUpdate) return { action: "skip-update", ok: true, stderr: "" }; - const res = git(["pull", "--ff-only"], dir); - return { action: "pull", ok: res.status === 0, stderr: res.stderr.trim() }; -} - -function git(args: string[], cwd: string): GitResult { - const res = spawnSync("git", args, { cwd, encoding: "utf8" }); - return { - status: res.status, - stdout: res.stdout ?? "", - stderr: res.stderr ?? "", - }; -} - -async function collectFiles(root: string): Promise { - if (!existsSync(root)) return []; - const out: string[] = []; - await walk(root, out); - return out.filter((file) => /\.(py|ts|tsx|js|jsx|mjs|md|json|yaml|yml)$/i.test(file)); -} - -async function walk(dir: string, out: string[]): Promise { - const entries = await readdir(dir, { withFileTypes: true }).catch(() => []); - for (const entry of entries) { - if (entry.name === ".git" || entry.name === "node_modules" || entry.name === "__pycache__") continue; - const path = join(dir, entry.name); - if (entry.isDirectory()) await walk(path, out); - else if (entry.isFile()) out.push(path); - } -} - -function extractAll(text: string, regex: RegExp): string[] { - const values: string[] = []; - for (const match of text.matchAll(regex)) values.push(match[1] ?? match[0]); - return values; -} - -function snippets(text: string, patterns: string[]): Array> { - const lines = text.split(/\r?\n/); - const out: Array> = []; - for (const pattern of patterns.slice(0, 8)) { - const idx = lines.findIndex((line) => line.includes(pattern)); - if (idx < 0) continue; - out.push({ - pattern, - line: idx + 1, - text: (lines[idx] ?? "").trim().slice(0, 240), - }); - } - return out; -} - -function relativePath(root: string, file: string): string { - return file.startsWith(root) ? file.slice(root.length + 1) : basename(file); -} - -function groupByProvider(repos: Array>): Record> { - const out: Record> = {}; - for (const repo of repos) { - const provider = String(repo.provider); - out[provider] ??= { action_names: [], tool_versions: [], beta_headers: [], response_fields: [] }; - out[provider].action_names = sorted(new Set([...out[provider].action_names, ...repo.action_names])); - out[provider].tool_versions = sorted(new Set([...out[provider].tool_versions, ...repo.tool_versions])); - out[provider].beta_headers = sorted(new Set([...out[provider].beta_headers, ...repo.beta_headers])); - out[provider].response_fields = sorted(new Set([...out[provider].response_fields, ...repo.response_fields])); - } - return out; -} - -function sorted(values: Iterable): string[] { - return [...values].sort(); -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(err instanceof Error ? err.message : String(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts deleted file mode 100644 index 8411400..0000000 --- a/.agents/skills/update-models/reference/discover-models.ts +++ /dev/null @@ -1,757 +0,0 @@ -#!/usr/bin/env tsx -import { existsSync } from "node:fs"; -import { readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface Args { - provider: Provider | "all"; - out: string; - models: string[]; - candidateLimit: number; - smoke: boolean; -} - -interface SmokeResult { - status: "pass" | "inconclusive" | "unsupported" | "fail"; - tool_name?: string; - tool_version?: string | null; - beta_header?: string | null; - observed_actions: string[]; - response_item_types: string[]; - error: string | null; - [key: string]: unknown; -} - -interface ModelResult { - id: string; - display_name?: string; - name?: string | null; - created_at?: string | null; - raw?: unknown; - supports_generation?: boolean; - computer_use?: SmokeResult | Record; - model_docs?: Record; - cua?: Record; -} - -const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "xai", "moonshot"]; -const GEMINI_DOC_COMPUTER_USE_MODELS = [ - "gemini-3.5-flash", - "gemini-3-flash-preview", - "gemini-2.5-computer-use-preview-10-2025", -]; - -const OPENAI_EXCLUDE = [ - "embedding", - "moderation", - "whisper", - "tts", - "dall-e", - "image", - "audio", - "transcribe", - "realtime", -]; - -function parseArgs(argv: string[]): Args { - const out: Args = { - provider: "all", - out: "", - models: [], - candidateLimit: 20, - smoke: true, - }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--provider" && next) { - out.provider = next as Provider | "all"; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--models" && next) { - out.models = next.split(",").map((s) => s.trim()).filter(Boolean); - i++; - } else if (arg === "--candidate-limit" && next) { - out.candidateLimit = Number(next) || out.candidateLimit; - i++; - } else if (arg === "--no-smoke") { - out.smoke = false; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - if (out.provider !== "all" && !PROVIDERS.includes(out.provider)) { - throw new Error(`--provider must be one of: all, ${PROVIDERS.join(", ")}`); - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json - npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4 - -Options: - --provider - --models Smoke-test explicit models instead of inferred candidates. - --candidate-limit Max inferred candidates per provider. Default: 20. - --no-smoke Only list metadata. - --out Write JSON report to file. -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const selected = args.provider === "all" ? PROVIDERS : [args.provider]; - const report: Record = { - generated_at: new Date().toISOString(), - providers: {}, - }; - const providers = report.providers as Record; - await Promise.all(selected.map(async (provider) => { - providers[provider] = await runProvider(provider, args); - })); - await emitJson(report, args.out); -} - -async function runProvider(provider: Provider, args: Args): Promise> { - try { - if (provider === "openai") return await discoverOpenAI(args); - if (provider === "anthropic") return await discoverAnthropic(args); - if (provider === "gemini") return await discoverGemini(args); - if (provider === "xai") return await discoverXai(args); - if (provider === "moonshot") return await discoverMoonshot(args); - throw new Error(`unknown provider ${provider satisfies never}`); - } catch (err) { - return { - provider, - error: publicError(err), - }; - } -} - -async function discoverOpenAI(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels - .map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: likelyOpenAIGenerationModel(String(m.id)), - })) - .sort((a, b) => String(b.created_at ?? "").localeCompare(String(a.created_at ?? ""))); - - const candidates = explicitOrCandidates(args, models.filter((m) => m.supports_generation).map((m) => m.id)); - await annotateOpenAIModelDocs(models.filter((m) => candidates.includes(m.id))); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeOpenAI(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("openai", models); - return { provider: "openai", metadata_source: "client.models.list()", models, candidates }; -} - -async function smokeMeta(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.responses.create({ - model, - store: false, - parallel_tool_calls: false, - max_output_tokens: 512, - reasoning: { effort: "low" }, - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}` }, - ], - }], - tools: [{ - type: "function", - name: "click", - description: "Click at normalized 0-1000 screen coordinates.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }], - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "function_call"); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -async function discoverXai(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.XAI_API_KEY, baseURL: "https://api.x.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: likelyXaiGenerationModel(String(m.id)), - model_docs: { - url: String(m.id) === "grok-4.5" ? "https://docs.x.ai/developers/grok-4-5" : "https://docs.x.ai/developers/models", - responses_endpoint: "supported", - function_calling: "supported", - image_input: "verify-per-model", - coordinate_space: "CUA-defined 0-1000", - }, - })); - const candidates = explicitOrCandidates( - args, - models - .filter((model) => model.supports_generation) - .sort(compareXaiCandidates) - .map((model) => model.id), - ); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeXai(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("xai", models); - return { provider: "xai", metadata_source: "xAI models.list()", models, candidates }; -} - -async function smokeXai(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.responses.create({ - model, - store: true, - parallel_tool_calls: false, - max_output_tokens: 768, - reasoning: { effort: "low" }, - instructions: "Coordinates are normalized from 0 to 1000 relative to the screenshot.", - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}`, detail: "high" }, - ], - }], - tools: [{ - type: "function", - name: "click", - description: "Click at normalized 0-1000 screen coordinates.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }], - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "function_call"); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - reasoning_effort: "low", - coordinate_space: "0-1000", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -function likelyXaiGenerationModel(id: string): boolean { - const lower = id.toLowerCase(); - return lower.startsWith("grok-") && !lower.includes("imagine"); -} - -async function discoverMoonshot(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.MOONSHOT_API_KEY, baseURL: "https://api.moonshot.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: String(m.id).toLowerCase().startsWith("kimi-"), - model_docs: { - url: "https://platform.kimi.ai/docs/api/tool-use", - chat_completions_endpoint: "supported", - function_calling: "supported", - image_input: "verify-per-model", - coordinate_space: "CUA-defined 0-1 fractions", - }, - })); - const candidates = explicitOrCandidates( - args, - models - .filter((model) => model.supports_generation) - .sort(compareMoonshotCandidates) - .map((model) => model.id), - ); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeMoonshot(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("moonshot", models); - return { provider: "moonshot", metadata_source: "Moonshot models.list()", models, candidates }; -} - -async function smokeMoonshot(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.chat.completions.create({ - model, - parallel_tool_calls: false, - max_tokens: 8192, - messages: [ - { - role: "system", - content: "Coordinates are fractions of the screenshot, normalized from 0 to 1.", - }, - { - role: "user", - content: [ - { type: "text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "image_url", image_url: { url: `data:image/png;base64,${screenshot.toString("base64")}` } }, - ], - }, - ], - tools: [{ - type: "function", - function: { - name: "click", - description: "Click at coordinates given as 0-1 fractions of the screenshot.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }, - }], - }); - const calls: any[] = response.choices?.[0]?.message?.tool_calls ?? []; - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.function?.name).filter(Boolean)), - response_item_types: unique([response.choices?.[0]?.finish_reason].filter(Boolean)), - coordinate_space: "0-1 fractions", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -function compareMoonshotCandidates(a: ModelResult, b: ModelResult): number { - if (a.id === "kimi-k3") return -1; - if (b.id === "kimi-k3") return 1; - return String(b.created_at ?? "").localeCompare(String(a.created_at ?? "")); -} - -function compareXaiCandidates(a: ModelResult, b: ModelResult): number { - if (a.id === "grok-4.5") return -1; - if (b.id === "grok-4.5") return 1; - return String(b.created_at ?? "").localeCompare(String(a.created_at ?? "")); -} - -function likelyOpenAIGenerationModel(id: string): boolean { - const lower = id.toLowerCase(); - if (OPENAI_EXCLUDE.some((needle) => lower.includes(needle))) return false; - return lower.startsWith("gpt-") || /^o\d/.test(lower) || lower.includes("computer-use"); -} - -async function annotateOpenAIModelDocs(models: ModelResult[]): Promise { - await Promise.all(models.map(async (model) => { - model.model_docs = await fetchOpenAIModelDocs(model.id); - })); -} - -async function fetchOpenAIModelDocs(modelId: string): Promise> { - const docId = canonicalOpenAIModelDocId(modelId); - const url = `https://developers.openai.com/api/docs/models/${docId}`; - try { - const response = await fetch(url); - const text = await response.text(); - return { - url, - ok: response.ok, - streaming: supportStatus(text, "Streaming"), - function_calling: supportStatus(text, "Function calling"), - computer_use: supportStatus(text, "Computer use"), - responses_endpoint: text.includes("v1/responses") ? "supported" : "unknown", - }; - } catch (err) { - return { - url, - ok: false, - error: publicError(err), - }; - } -} - -function canonicalOpenAIModelDocId(modelId: string): string { - return modelId.replace(/-\d{4}-\d{2}-\d{2}$/, ""); -} - -function supportStatus(text: string, label: string): "supported" | "not_supported" | "unknown" { - const compact = text.replace(/<[^>]*>/g, " ").replace(/\s+/g, " "); - const re = new RegExp(`${escapeRegex(label)}\\s+(Supported|Not supported)`, "i"); - const match = compact.match(re); - if (!match) return "unknown"; - return match[1]?.toLowerCase() === "supported" ? "supported" : "not_supported"; -} - -function escapeRegex(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); -} - -async function smokeOpenAI(client: any, model: string): Promise { - try { - const response = await client.responses.create({ - model, - input: "Use the computer tool to request a screenshot. Do not answer in text.", - tools: [{ type: "computer" }], - tool_choice: { type: "computer" }, - max_output_tokens: 64, - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "computer_call"); - const actions = calls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer", - tool_version: null, - beta_header: null, - observed_actions: unique(actions.map((a) => a?.type).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "computer" }); - } -} - -async function discoverAnthropic(args: Args): Promise> { - const Anthropic = await importDefault("@anthropic-ai/sdk", "Anthropic"); - const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); - const rawModels = await collectAsync(client.models.list({ limit: 1000 })); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: m.display_name ?? m.id, - created_at: m.created_at ?? null, - raw: m, - supports_generation: m.type === "model" || String(m.id).startsWith("claude-"), - })); - const candidates = explicitOrCandidates(args, models.filter((m) => m.id.startsWith("claude-")).map((m) => m.id)); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeAnthropic(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("anthropic", models); - return { provider: "anthropic", metadata_source: "client.models.list({ limit: 1000 })", models, candidates }; -} - -type AnthropicToolPair = { tool: string; beta: string }; - -const ANTHROPIC_TOOL_PAIRS: AnthropicToolPair[] = [ - { tool: "computer_20251124", beta: "computer-use-2025-11-24" }, - { tool: "computer_20250124", beta: "computer-use-2025-01-24" }, - { tool: "computer_20241022", beta: "computer-use-2024-10-22" }, -]; - -async function smokeAnthropic(client: any, model: string): Promise> { - const attempts: Record[] = []; - const runtimePair = anthropicRuntimeToolPair(model); - for (const pair of orderAnthropicPairs(runtimePair)) { - try { - const response = await client.beta.messages.create({ - model, - max_tokens: 64, - messages: [{ role: "user", content: "Use the computer tool to take a screenshot. Do not answer in text." }], - tools: [{ - type: pair.tool, - name: "computer", - display_width_px: 1024, - display_height_px: 768, - display_number: 1, - }], - betas: [pair.beta], - }); - const content: any[] = response.content ?? []; - const calls = content.filter((block) => block?.type === "tool_use" && block?.name === "computer"); - const actions = calls.map((call) => call?.input?.action).filter(Boolean); - const result = { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer", - tool_version: pair.tool, - beta_header: pair.beta, - runtime_tool_version: runtimePair.tool, - runtime_beta_header: runtimePair.beta, - runtime_compatible: calls.length > 0 && pair.tool === runtimePair.tool && pair.beta === runtimePair.beta, - observed_actions: unique(actions), - response_item_types: unique(content.map((block) => block?.type).filter(Boolean)), - stop_reason: response.stop_reason ?? null, - error: null, - }; - if (result.status === "pass") return result; - attempts.push(result); - } catch (err) { - attempts.push(smokeError(err, { tool_name: "computer", tool_version: pair.tool, beta_header: pair.beta })); - } - } - return { status: "fail", attempts, error: attempts.at(-1)?.error ?? "all tool versions failed" }; -} - -function orderAnthropicPairs(runtimePair: AnthropicToolPair): AnthropicToolPair[] { - const rest = ANTHROPIC_TOOL_PAIRS.filter((pair) => pair.tool !== runtimePair.tool || pair.beta !== runtimePair.beta); - return [runtimePair, ...rest]; -} - -function anthropicRuntimeToolPair(model: string): AnthropicToolPair { - const id = model.toLowerCase(); - if ( - id.startsWith("claude-opus-4-8") || - id.startsWith("claude-opus-4-7") || - id.startsWith("claude-opus-4-6") || - id.startsWith("claude-opus-4-5") || - id.startsWith("claude-sonnet-4-6") || - id.startsWith("claude-fable-5") - ) { - return { tool: "computer_20251124", beta: "computer-use-2025-11-24" }; - } - return { tool: "computer_20250124", beta: "computer-use-2025-01-24" }; -} - -async function discoverGemini(args: Args): Promise> { - const { GoogleGenAI } = await import("@google/genai"); - const apiKey = process.env.GOOGLE_API_KEY || process.env.GEMINI_API_KEY; - const client = new GoogleGenAI({ apiKey }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: normalizeGeminiModelId(m.name ?? m.id ?? m.baseModelId), - name: m.name ?? null, - display_name: m.displayName ?? m.display_name ?? m.name ?? m.id, - created_at: null, - raw: m, - supports_generation: hasGenerateContent(m), - })); - const geminiCandidates = unique([ - ...GEMINI_DOC_COMPUTER_USE_MODELS, - ...rankGeminiCandidates(models.filter((m) => m.supports_generation && likelyGeminiCandidate(m.id))).map((m) => m.id), - ]); - const candidates = explicitOrCandidates(args, geminiCandidates); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id || m.name === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeGemini(client, id); - if (!models.find((m) => m.id === id || m.name === id)) models.unshift(model); - })); - } - await annotateCuaSupport("gemini", models); - return { provider: "gemini", metadata_source: "client.models.list()", models, candidates }; -} - -async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise { - const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; - const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined); - for (const model of models) { - const inRegistry = getBuiltinModel ? !!getBuiltinModel(piProvider as never, model.id as never) : false; - const localAdapterSupport = localAdapterSupportStatus(provider, model); - model.cua = { - provider_inference: provider, - pi_ai_registry: inRegistry ? "present" : "missing", - dynamic_model_fallback: "available", - local_adapter_support: localAdapterSupport, - }; - } -} - -function localAdapterSupportStatus(provider: Provider, model: ModelResult): string { - if (!model.computer_use || !("status" in model.computer_use) || model.computer_use.status !== "pass") { - return "needs-check"; - } - if (provider !== "anthropic") return "passes-smoke"; - return model.computer_use.runtime_compatible === true ? "passes-smoke" : "smoke-pass-runtime-mismatch"; -} - -function hasGenerateContent(model: any): boolean { - const actions = model.supportedActions ?? model.supported_actions ?? model.supportedGenerationMethods ?? []; - return Array.isArray(actions) && actions.some((a) => String(a).toLowerCase() === "generatecontent"); -} - -function likelyGeminiCandidate(id: string): boolean { - const lower = String(id ?? "").toLowerCase(); - return lower.includes("gemini") && !lower.includes("embedding") && !lower.includes("tts") && !lower.includes("imagen"); -} - -function rankGeminiCandidates(models: ModelResult[]): ModelResult[] { - return [...models].sort((a, b) => geminiScore(b.id) - geminiScore(a.id)); -} - -function geminiScore(id: string): number { - const lower = id.toLowerCase(); - let score = 0; - if (lower.includes("computer-use")) score += 100; - if (lower.includes("gemini-3")) score += 80; - if (lower.includes("preview")) score += 20; - if (lower.includes("flash")) score += 10; - if (lower.includes("pro")) score += 5; - return score; -} - -async function smokeGemini(client: any, model: string): Promise> { - const configVariants = [ - { tools: [{ computerUse: { environment: "ENVIRONMENT_BROWSER" } }], maxOutputTokens: 64 }, - { tools: [{ computer_use: { environment: "ENVIRONMENT_BROWSER" } }], maxOutputTokens: 64 }, - ]; - const attempts: Record[] = []; - for (const config of configVariants) { - try { - const response = await client.models.generateContent({ - model, - contents: [{ role: "user", parts: [{ text: "Use the computer-use tool to open the web browser. Do not answer in text." }] }], - config, - }); - const parts: any[] = response?.candidates?.[0]?.content?.parts ?? []; - const calls = parts.map((part) => part.functionCall ?? part.function_call).filter(Boolean); - const actions = calls.map((call) => call.name).filter(Boolean); - const result = { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer_use", - tool_version: null, - beta_header: null, - observed_actions: unique(actions), - response_item_types: unique(parts.map((part) => part.functionCall || part.function_call ? "function_call" : part.text ? "text" : Object.keys(part)[0]).filter(Boolean)), - error: null, - }; - if (result.status === "pass") return result; - attempts.push(result); - } catch (err) { - attempts.push(smokeError(err, { tool_name: "computer_use" })); - } - } - return { status: "fail", attempts, error: attempts.at(-1)?.error ?? "all config variants failed" }; -} - -function explicitOrCandidates(args: Args, ids: string[]): string[] { - return (args.models.length ? args.models : ids).slice(0, args.candidateLimit); -} - -async function importDefault(pkg: string, named: string): Promise { - try { - const mod = await import(pkg); - return mod.default ?? mod[named]; - } catch (err) { - throw new Error(`failed to import ${pkg}. Run npm install first. ${publicError(err)}`); - } -} - -async function collectAsync(value: any): Promise { - const awaited = await value; - if (Array.isArray(awaited)) return awaited; - if (Array.isArray(awaited?.data)) return awaited.data; - if (Array.isArray(awaited?.models)) return awaited.models; - if (Array.isArray(awaited?.items)) return awaited.items; - if (typeof awaited?.[Symbol.asyncIterator] === "function") { - const out: any[] = []; - for await (const item of awaited) out.push(item); - return out; - } - if (typeof awaited?.[Symbol.iterator] === "function") return Array.from(awaited); - return []; -} - -function normalizeGeminiModelId(id: unknown): string { - const value = String(id ?? ""); - return value.startsWith("models/") ? value.slice("models/".length) : value; -} - -function smokeError(err: unknown, extra: Record = {}): SmokeResult { - return { - status: isUnsupportedError(err) ? "unsupported" : "fail", - ...extra, - observed_actions: [], - response_item_types: [], - error: publicError(err), - }; -} - -function isUnsupportedError(err: unknown): boolean { - const msg = publicError(err).toLowerCase(); - return msg.includes("unsupported") || msg.includes("not support") || msg.includes("not enabled") || msg.includes("not compatible") || msg.includes("invalid tool"); -} - -function publicError(err: unknown): string { - if (err && typeof err === "object" && "status" in err && "message" in err) { - return `${String((err as { status: unknown }).status)}: ${String((err as { message: unknown }).message)}`; - } - return err instanceof Error ? err.message : String(err); -} - -function unique(values: T[]): T[] { - return [...new Set(values)]; -} - -function fixtureScreenshotPath(): string { - const path = [ - join(process.cwd(), "examples", "screenshot.png"), - join(process.cwd(), "packages", "ai", "examples", "screenshot.png"), - ].find(existsSync); - if (!path) throw new Error("could not find packages/ai/examples/screenshot.png"); - return path; -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(publicError(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts deleted file mode 100644 index 1dfef98..0000000 --- a/.agents/skills/update-models/reference/provider-doc-drift.ts +++ /dev/null @@ -1,199 +0,0 @@ -#!/usr/bin/env tsx -import { readFile, writeFile } from "node:fs/promises"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface Args { - examples: string; - out: string; -} - -interface FetchResult { - url: string; - ok: boolean; - status: number | null; - error?: string; - text: string; -} - -const DOCS: Record = { - openai: [ - "https://developers.openai.com/api/docs/guides/tools-computer-use", - "https://raw.githubusercontent.com/openai/openai-node/master/src/resources/responses/responses.ts", - ], - anthropic: [ - "https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool", - "https://console.anthropic.com/docs/en/agents-and-tools/tool-use/tool-reference", - ], - gemini: [ - "https://ai.google.dev/gemini-api/docs/computer-use", - "https://ai.google.dev/api/models", - ], - xai: [ - "https://docs.x.ai/developers/grok-4-5", - "https://docs.x.ai/developers/tools/function-calling", - "https://docs.x.ai/developers/model-capabilities/images/understanding", - ], - moonshot: [ - "https://www.kimi.com/blog/kimi-k3", - "https://platform.kimi.ai/docs/api/tool-use", - "https://platform.kimi.ai/docs/guide/use-kimi-vision-model", - ], -}; - -const LOCAL_FILES: Record = { - openai: "packages/ai/src/providers/openai/index.ts", - anthropic: "packages/ai/src/providers/anthropic/actions.ts", - gemini: "packages/ai/src/providers/gemini/index.ts", - xai: "packages/ai/src/providers/xai/index.ts", - moonshot: "packages/ai/src/providers/moonshot/index.ts", -}; - -const ACTION_REGEXES: Record = { - openai: /\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g, - anthropic: /\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g, - gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g, - xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, - moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, -}; - -function parseArgs(argv: string[]): Args { - const out: Args = { examples: "", out: "" }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--examples" && next) { - out.examples = next; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const examples = args.examples ? JSON.parse(await readFile(args.examples, "utf8")) : {}; - const report: Record = { - generated_at: new Date().toISOString(), - providers: {}, - }; - const providers = report.providers as Record; - await Promise.all((Object.keys(DOCS) as Provider[]).map(async (provider) => { - providers[provider] = await checkProvider(provider, examples); - })); - await emitJson(report, args.out); -} - -async function checkProvider(provider: Provider, examples: any): Promise> { - const docTexts = await Promise.all(DOCS[provider].map(fetchText)); - const docText = docTexts.map((r) => r.text).join("\n"); - const localText = await readFile(LOCAL_FILES[provider], "utf8").catch((err) => `/* failed to read local file: ${err.message} */`); - const example = examples?.by_provider?.[provider] ?? {}; - - const documentedActions = unique(extractAll(docText, ACTION_REGEXES[provider])); - const localActions = unique(extractAll(localText, ACTION_REGEXES[provider])); - const exampleActions: string[] = example.action_names ?? []; - const documentedToolVersions = unique(extractAll(docText, /computer_\d{8}/g)); - const localToolVersions = unique(extractAll(localText, /computer_\d{8}/g)); - const exampleToolVersions: string[] = example.tool_versions ?? []; - const documentedBetaHeaders = unique(extractAll(docText, /computer-use-\d{4}-\d{2}-\d{2}/g)); - const localBetaHeaders = unique(extractAll(localText, /computer-use-\d{4}-\d{2}-\d{2}/g)); - const exampleBetaHeaders: string[] = example.beta_headers ?? []; - - return { - provider, - doc_sources: docTexts.map(({ url, ok, status, error }) => ({ url, ok, status, error })), - documented_tool_versions: sorted(documentedToolVersions), - example_tool_versions: sorted(exampleToolVersions), - local_tool_versions: sorted(localToolVersions), - newer_tool_versions: sorted(difference(new Set([...documentedToolVersions, ...exampleToolVersions]), new Set(localToolVersions))), - documented_beta_headers: sorted(documentedBetaHeaders), - example_beta_headers: sorted(exampleBetaHeaders), - local_beta_headers: sorted(localBetaHeaders), - newer_beta_headers: sorted(difference(new Set([...documentedBetaHeaders, ...exampleBetaHeaders]), new Set(localBetaHeaders))), - documented_actions: sorted(documentedActions), - example_repo_actions: sorted(exampleActions), - repo_supported_actions: sorted(localActions), - unknown_documented_actions: provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(documentedActions), new Set(localActions))), - unknown_example_actions: provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(exampleActions), new Set(localActions))), - response_fields_from_examples: sorted(example.response_fields ?? []), - notes: notesFor(provider, documentedToolVersions, exampleToolVersions, localToolVersions), - }; -} - -async function fetchText(url: string): Promise { - try { - const response = await fetch(url); - const text = await response.text(); - return { url, ok: response.ok, status: response.status, text }; - } catch (err) { - return { url, ok: false, status: null, error: err instanceof Error ? err.message : String(err), text: "" }; - } -} - -function notesFor(provider: Provider, documentedToolVersions: string[], exampleToolVersions: string[], localToolVersions: string[]): string[] { - const notes: string[] = []; - if (provider === "openai") { - notes.push("OpenAI's GA computer tool is currently undated (`computer`); drift usually appears as action-shape changes or preview deprecations."); - } - if (provider === "anthropic") { - const newest = sorted(new Set([...documentedToolVersions, ...exampleToolVersions])).at(-1); - if (newest && !localToolVersions.includes(newest)) { - notes.push(`Anthropic docs/examples mention ${newest}, which is not in local constants.`); - } - } - if (provider === "gemini") { - notes.push("Gemini official computer use emits predefined function-call names; keep this separate from CUA custom function declarations."); - } - if (provider === "xai") { - notes.push("xAI uses developer-defined function tools and does not document a native coordinate protocol; compare Responses compatibility, reasoning controls, and CUA's normalized coordinate contract instead."); - } - if (provider === "moonshot") { - notes.push("Moonshot uses developer-defined function tools over OpenAI-compatible chat completions and does not document a coordinate protocol; Kimi grounding emits 0-1 width/height fractions, so compare against CUA's fractional coordinate contract."); - } - return notes; -} - -function extractAll(text: string, regex: RegExp): string[] { - const values: string[] = []; - for (const match of String(text).matchAll(regex)) values.push(match[1] ?? match[0]); - return values; -} - -function difference(a: Set, b: Set): string[] { - return [...a].filter((value) => !b.has(value)); -} - -function unique(values: T[]): T[] { - return [...new Set(values)]; -} - -function sorted(values: Iterable): string[] { - return [...values].sort(); -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(err instanceof Error ? err.message : String(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/report-schema.md b/.agents/skills/update-models/reference/report-schema.md deleted file mode 100644 index 9887465..0000000 --- a/.agents/skills/update-models/reference/report-schema.md +++ /dev/null @@ -1,100 +0,0 @@ -# Report Schema - -Use this shape for JSON reports and the same fields when writing a Markdown summary. - -## Top Level - -```json -{ - "generated_at": "2026-04-26T00:00:00.000Z", - "repo": "/path/to/cua", - "providers": { - "openai": {}, - "anthropic": {}, - "gemini": {}, - "xai": {}, - "moonshot": {} - }, - "example_evidence": {}, - "drift": {}, - "recommendations": [] -} -``` - -## Provider Result - -```json -{ - "provider": "openai", - "metadata_source": "models.list", - "models": [ - { - "id": "gpt-5.5", - "display_name": "GPT-5.5", - "created_at": "2026-04-01T00:00:00.000Z", - "raw": {}, - "supports_generation": true, - "model_docs": { - "url": "https://developers.openai.com/api/docs/models/gpt-5.5", - "streaming": "supported", - "function_calling": "supported", - "computer_use": "supported", - "responses_endpoint": "supported" - }, - "computer_use": { - "status": "pass", - "tool_name": "computer", - "tool_version": null, - "beta_header": null, - "observed_actions": ["screenshot"], - "response_item_types": ["computer_call"], - "accepts_image_tool_results": "assumed-from-docs", - "error": null - }, - "cua": { - "provider_inference": "openai", - "pi_ai_registry": "missing", - "dynamic_model_fallback": "available", - "local_adapter_support": "passes-smoke" - }, - "recommended_action": "candidate-default" - } - ] -} -``` - -## Drift Result - -```json -{ - "provider": "anthropic", - "documented_tool_versions": ["computer_20251124"], - "example_tool_versions": ["computer_20251124"], - "local_tool_versions": ["computer_20251124"], - "documented_actions": ["screenshot", "left_click"], - "example_repo_actions": ["screenshot", "left_click"], - "observed_actions": ["screenshot"], - "repo_supported_actions": ["screenshot", "left_click"], - "unknown_observed_actions": [], - "notes": [] -} -``` - -## Markdown Summary - -Use this order: - -1. New or changed model candidates. -2. Computer-use smoke-test pass/fail/inconclusive table. -3. Official example repo findings. -4. Drift against local adapter constants. -5. CUA support changes needed: - - default model update - - `pi-ai` registry present or dynamic fallback needed - - model-specific docs do not block required runtime features - - provider routing update - - adapter action/tool version update - - docs/config examples update -6. Recommended repo changes and blockers. - -Only recommend changing defaults when metadata discovery, official evidence, smoke tests, and CUA compatibility all line up. diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 589fc72..bfa9c19 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -60,3 +60,39 @@ This is what lets a model work the day the provider ships it rather than when models.dev catches up. Only an unqualified ref or a provider pi-ai does not carry is refused. + +## Keeping these tables current + +There is no periodic audit to run. The allowlist that once needed one is gone, +so maintenance is reactive — four cases, in rough order of how often they come up. + +**A provider released a model.** Nothing to do. If pi-ai's registry carries it, +it resolves; if not, it is synthesized from its nearest sibling. Neither needs a +change here. + +**The catalog looks stale.** Bump `@earendil-works/pi-ai`. Its registry is +generated from models.dev, so a newer pi-ai is how names, context windows, and +pricing get refreshed. Note that cua does not read pi's `models.json`: that is a +pi-coding-agent config file, and cua builds its `Models` collection from pi-ai +directly. A provider pi-ai does not ship is not selectable without registering +it in `src/providers.ts` — a deliberate decision, since the repo has removed +four such providers rather than carry them unused. + +**A provider shipped or changed a native tool.** This is real adapter work, not +a table edit. Probe what the model actually emits: + +```bash +npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --limit 3 +``` + +Update that provider's adapter under `src/providers/` to execute the actions the +probe returns, then add or adjust the `CUA_NATIVE_SURFACES` entry, citing the +provider's documentation. Anthropic's computer tool version and its +`computer-use-*` beta header are chosen by pi-ai per model, so a new dated +version there usually means bumping pi-ai rather than editing this package. + +**A model rejects a tool CUA sends.** Add a `CUA_MODEL_QUIRKS` entry with the +observed error as its `reason`, scoped as narrowly as the evidence supports: a +single model id over a family, a family over a whole provider. Remove a quirk +when the provider lifts the limit — a stale quirk silently denies a model a tool +it now accepts, which is harder to notice than the reverse. diff --git a/.agents/skills/update-models/reference/native-action-probe.ts b/packages/ai/scripts/native-action-probe.ts similarity index 97% rename from .agents/skills/update-models/reference/native-action-probe.ts rename to packages/ai/scripts/native-action-probe.ts index c3b9434..343d428 100644 --- a/.agents/skills/update-models/reference/native-action-probe.ts +++ b/packages/ai/scripts/native-action-probe.ts @@ -71,9 +71,9 @@ function parseArgs(argv: string[]): Args { function usage(): never { console.log(`Usage: - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 + npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json + npx tsx packages/ai/scripts/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 + npx tsx packages/ai/scripts/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 `); process.exit(0); }