diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md deleted file mode 100644 index 1cad940..0000000 --- a/.agents/skills/update-models/SKILL.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -name: update-models -description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. ---- - -# Update Models - -Use this workflow to keep CUA current with provider model releases and computer-use support. Do not trust a static model list: combine provider metadata, official docs, official example repos, and live non-destructive smoke tests. - -## Quick Start - -1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. -2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them: - -```bash -eval "$(python3 - <<'PY' -import pathlib, re, shlex -text = pathlib.Path('~/AGENTS.md').expanduser().read_text() -for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: - m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text) - if m: - print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}') -PY -)" -``` - -3. From the repo root, run the all-provider probe: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json -``` - -4. Audit official examples for tool shape drift: - -```bash -npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json -``` - -5. Compare docs, examples, live probes, and local adapter constants: - -```bash -npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -``` - -6. Summarize findings with the template in `reference/report-schema.md`. Only recommend repo changes after checking the decision rules below. - -## Evidence Order - -Use all four evidence sources when possible: - -- Provider metadata APIs: tells us what models are available to this API key. -- Official docs: tells us intended tool names, dated beta headers, and documented action vocabularies. -- Model-specific docs: tells us endpoint, streaming, feature, and tool support for a specific model ID. -- Official example repos: shows real response parsing, action execution, safety handling, and follow-up payload shapes. -- Live smoke tests: confirms the current model/API combination can emit provider-native computer-use tool calls. -- Local cua-ai smoke tests: confirms `@onkernel/cua-ai` resolves the model through `getCuaModel()` and its provider adapter emits executable CUA tool calls. - -Treat example repos as strongest when they are provider-owned or linked from official docs. If discovered through search only, mark them lower confidence until verified. - -## Model Enumeration - -There are two enumeration layers: - -- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access. -- CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use. - -When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("")`. - -## Provider Checks - -Meta: - -- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response output contains a `function_call` for one of the supplied browser actions. -- Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`. -- Set `parallel_tool_calls: false` because browser actions mutate shared state. -- Treat Meta computer use as custom-function-tool support, not a provider-native `{ type: "computer" }` tool. - -OpenAI: - -- Discover with `OpenAI().models.list()` and optionally `models.retrieve(modelId)`. -- OpenAI model metadata is sparse (`id`, `created`, `owned_by`), so computer-use support must be smoke-tested. -- Check the model-specific docs page at `https://developers.openai.com/api/docs/models/` before adding support. For aliases/snapshots, check the canonical family page too, e.g. `gpt-5.5-pro-2026-04-23` -> `gpt-5.5-pro`. -- For CUA support, require `Responses` endpoint support, `Streaming` support, and `Function calling` support. Do not list models like `gpt-5.5-pro` that say `Streaming: Not supported`. -- For provider-native OpenAI computer use, require `Computer use: Supported`. If a model supports function calling but not native `computer`, label it custom-tool-only and do not treat it as provider-native computer-use support. -- Smoke-test `responses.create` with `tools: [{ type: "computer" }]` and `tool_choice: { type: "computer" }`. -- Pass condition: response output contains `type: "computer_call"` with `actions[]` or legacy `action`. -- Audit official examples for `computer_call`, `actions`, `computer_call_output`, `pending_safety_checks`, and screenshot payload handling. - -Anthropic: - -- Discover with `Anthropic().models.list({ limit: 1000 })`. -- Record `id`, `display_name`, `created_at`, token limits, and `capabilities`. -- Smoke-test `client.beta.messages.create` with discovered computer tool and beta pairs, newest first. -- Pass condition: `stop_reason === "tool_use"` and a `tool_use` block named `computer`. -- For CUA support, the passing pair should match the Anthropic tool version and beta header the cua-ai runtime (via `pi-ai`) sends for that model; `discover-models.ts` reports this as `runtime_compatible`. A pass on a different pair is provider support that needs a `pi-ai` bump before the runtime can use it. -- Watch for dated drift: `computer_YYYYMMDD` tool names and `computer-use-YYYY-MM-DD` beta headers. - -Google/Gemini: - -- Discover with `GoogleGenAI().models.list()` and `models.get(...)`. -- Filter models that support `generateContent`, then test official `computer_use`. -- Pass condition: response contains provider-native `functionCall.name` values such as `open_web_browser`, `click_at`, or `type_text_at`. -- Do not infer official computer-use support from CUA's custom Gemini `functionDeclarations`; those are a separate compatibility path. - -xAI: - -- Discover with the OpenAI SDK against `https://api.x.ai/v1` using `XAI_API_KEY`. -- Record aliases, context length, standard and long-context token prices, and the 200k long-context threshold returned by `models.list()`. -- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response output contains a `function_call` for one of the supplied browser actions. -- Treat Grok computer use as custom-function-tool support, not a provider-native computer tool. xAI currently documents image understanding and function calling but no native coordinate protocol. -- Use CUA's normalized 0-1000 coordinate instructions, `parallel_tool_calls: false`, `store: true`, and `previous_response_id` for browser loops. xAI accepts encrypted reasoning replay in these requests. -- Use `reasoning: { effort: "low" }` for low-latency smoke tests; Grok 4.5 also supports `medium` and `high`, cannot disable reasoning, and defaults to `high`. - -Moonshot: - -- Discover with the OpenAI SDK against `https://api.moonshot.ai/v1` using `MOONSHOT_API_KEY`. -- Smoke-test the chat completions API with screenshot input and explicit function tools matching CUA's canonical actions. -- Pass condition: response `choices[0].message.tool_calls[]` contains one of the supplied browser actions. -- Treat Kimi computer use as custom-function-tool support, not a provider-native computer tool. Moonshot documents vision input and function calling but no coordinate protocol. -- Kimi grounding emits width/height fractions from 0 to 1 regardless of prompt or schema wording; keep CUA's fractional coordinate contract and verify emitted values stay in 0-1. -- Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn. -- Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes. - -## Native Action Discovery - -Run action probes when updating adapters or when docs/examples show drift: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 -``` - -The probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare: - -- `documented_actions`: extracted from provider docs or SDK source. -- `example_repo_actions`: extracted from official examples. -- `observed_actions`: emitted by live smoke probes. -- `repo_supported_actions`: local adapter constants. -- `unknown_observed_actions`: actions emitted by providers but not supported locally. - -## Decision Rules - -Recommend a model as CUA-supported only if: - -- It appears in the provider metadata API for the available key. -- Its model-specific docs do not rule out required CUA runtime features such as streaming. -- Its provider-appropriate computer-use smoke test passes: native tools where offered, or supplied function tools for Meta and xAI. -- Its local cua-ai smoke test emits a computer tool call: `CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai` returns a `toolCall` block. -- Official docs or examples support the same tool mechanism, or the smoke result clearly supersedes stale docs. -- The model is annotated in `CUA_MODEL_ANNOTATIONS` in `packages/ai/src/models.ts`, resolved from `pi-ai`'s registry or backed by a `CUA_MODEL_OVERRIDES` entry. - -Recommend adapter updates when: - -- A provider exposes a newer dated tool version or beta header. -- Official examples handle response fields the local adapter ignores. -- Smoke probes emit native actions not present in local constants. - -Do not print API keys. Keep smoke tests non-destructive. Do not edit repo defaults or adapters unless the user explicitly asks after reviewing the report. - -## Updating CUA Support - -All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). When a new model is discovered, decide which layer needs changing: - -- New model ID, same provider/tool surface: - - Add a `CUA_MODEL_ANNOTATIONS` entry in `packages/ai/src/models.ts` under the correct provider, citing the official source that documents computer-use support. Use a `family` match for a root that covers numeric revisions and dated snapshots (e.g. `claude-opus-4`), or an `exact` match for a single ID. A model already covered by an existing family annotation needs no change. - - If `pi-ai`'s registry does not carry the ID yet (`pi_ai_registry: "missing"` in the discovery report), add a `CUA_MODEL_OVERRIDES` entry so `getCuaModel()` can return a provider-shaped model. When the ID is already in the registry, the annotation alone is enough. - - Update the snapshot in `packages/ai/docs/supported-models.md` to match. - -- New provider-native action, response field, or tool version: - - OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes. - - Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package. - - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed. - - xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility. - - Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`. - - Shared canonical action semantics go in `packages/ai/src/providers/common.ts`. - -- New provider or routing rule: - - Update `CuaProvider`, `CUA_PROVIDERS`, `CUA_MODEL_ANNOTATIONS`, and `CUA_MODEL_OVERRIDES` in `packages/ai/src/models.ts`, plus the provider-module wiring in `packages/ai/src/providers.ts`. - -After changing support, run `npm run typecheck`, `npm test --workspace @onkernel/cua-ai`, and at least one live smoke per changed provider, for example `CUA_MODEL=: npm run example:quickstart --workspace @onkernel/cua-ai`. - -## Reference Files - -- `reference/README.md`: script usage and output overview. -- `reference/discover-models.ts`: provider metadata plus smoke-test orchestration. -- `reference/native-action-probe.ts`: live provider-native action elicitation. -- `reference/audit-official-examples.ts`: clone/update official examples and extract implementation evidence. -- `reference/provider-doc-drift.ts`: compare docs/examples/local constants for drift. -- `reference/report-schema.md`: normalized report fields and Markdown summary template. diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md deleted file mode 100644 index 2c0bca4..0000000 --- a/.agents/skills/update-models/reference/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Update Models Reference - -These scripts support the `update-models` skill. Run them from the repository root. - -## Requirements - -- Node 20+ -- Repository dependencies installed with `npm install` -- TypeScript runner available through `npx tsx` or another local TS runner -- Provider API keys as needed: - - `OPENAI_API_KEY` - - `ANTHROPIC_API_KEY` - - `GOOGLE_API_KEY` or `GEMINI_API_KEY` - - `XAI_API_KEY` - - `MOONSHOT_API_KEY` - -The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools. - -## Common Commands - -Discover all providers and smoke-test likely candidates: - -```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json -``` - -Probe native action vocabularies for a specific provider/model: - -```bash -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json -``` - -Clone/update official examples and extract tool-handling evidence: - -```bash -npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json -``` - -Examples are cached under `/tmp/cua-update-models/examples` by default so cloned upstream repos do not appear as untracked files in this repository. - -Compare official docs, examples, and local adapter constants: - -```bash -npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -``` - -## Evidence Types - -- `metadata`: provider model-list APIs. -- `docs`: provider docs or SDK source fetched live. -- `examples`: provider-owned or doc-linked example repos. -- `smoke`: live API response shape for provider-native computer use. -- `local`: constants in this repo's provider adapters. - -Use `report-schema.md` when summarizing results for humans. diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts deleted file mode 100644 index 330b53b..0000000 --- a/.agents/skills/update-models/reference/audit-official-examples.ts +++ /dev/null @@ -1,265 +0,0 @@ -#!/usr/bin/env tsx -import { existsSync } from "node:fs"; -import { mkdir, readFile, readdir, writeFile } from "node:fs/promises"; -import { basename, join, resolve } from "node:path"; -import { spawnSync } from "node:child_process"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface ExampleRepo { - provider: Provider; - name: string; - repo: string; - confidence: string; - pathHint?: string; - patterns: string[]; -} - -interface Args { - cache: string; - out: string; - noUpdate: boolean; -} - -interface GitResult { - status: number | null; - stdout: string; - stderr: string; -} - -const EXAMPLES: ExampleRepo[] = [ - { - provider: "openai", - name: "openai-cua-sample-app", - repo: "https://github.com/openai/openai-cua-sample-app.git", - confidence: "provider-owned", - patterns: ["computer_call", "computer_call_output", "actions", "pending_safety_checks", "computer_use_preview"], - }, - { - provider: "anthropic", - name: "anthropic-quickstarts", - repo: "https://github.com/anthropics/anthropic-quickstarts.git", - confidence: "provider-owned", - pathHint: "computer-use-demo", - patterns: ["computer_", "computer-use-", "tool_use", "tool_result", "input.action"], - }, - { - provider: "gemini", - name: "computer-use-preview", - repo: "https://github.com/google/computer-use-preview.git", - confidence: "provider-owned", - patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"], - }, - { - provider: "xai", - name: "xai-sdk-python", - repo: "https://github.com/xai-org/xai-sdk-python.git", - confidence: "provider-owned", - pathHint: "examples", - patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"], - }, -]; - -const ACTION_REGEXES: Record = { - openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g], - anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g], - gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g], - xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], - moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], -}; - -function parseArgs(argv: string[]): Args { - const out: Args = { - cache: "/tmp/cua-update-models/examples", - out: "", - noUpdate: false, - }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--cache" && next) { - out.cache = next; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--no-update") { - out.noUpdate = true; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/audit-official-examples.ts --out /tmp/cua-example-evidence.json - -Options: - --cache Clone/update examples here. Default: /tmp/cua-update-models/examples - --no-update Do not git pull existing repos. - --out Write JSON report to file. -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const cacheDir = resolve(process.cwd(), args.cache); - await mkdir(cacheDir, { recursive: true }); - - const repos = []; - for (const example of EXAMPLES) { - repos.push(await auditRepo(example, cacheDir, args)); - } - - const report = { - generated_at: new Date().toISOString(), - cache_dir: cacheDir, - repos, - by_provider: groupByProvider(repos), - }; - await emitJson(report, args.out); -} - -async function auditRepo(example: ExampleRepo, cacheDir: string, args: Args): Promise> { - const dir = join(cacheDir, example.name); - const cloneOrUpdate = ensureRepo(example.repo, dir, args.noUpdate); - const commit = git(["rev-parse", "HEAD"], dir).stdout.trim() || null; - const files = await collectFiles(example.pathHint ? join(dir, example.pathHint) : dir); - const matches = []; - const toolVersions = new Set(); - const betaHeaders = new Set(); - const actionNames = new Set(); - const responseFields = new Set(); - - for (const file of files) { - const text = await readFile(file, "utf8").catch(() => ""); - if (!text) continue; - const foundPatterns = example.patterns.filter((p) => text.includes(p)); - if (foundPatterns.length === 0) continue; - - extractAll(text, /computer_\d{8}/g).forEach((v) => toolVersions.add(v)); - extractAll(text, /computer-use-\d{4}-\d{2}-\d{2}/g).forEach((v) => betaHeaders.add(v)); - for (const regex of ACTION_REGEXES[example.provider] ?? []) { - extractAll(text, regex).forEach((v) => actionNames.add(v)); - } - for (const field of ["computer_call", "actions", "action", "pending_safety_checks", "tool_use", "tool_result", "tool_calls", "function_call", "functionCall", "FunctionResponse", "safety_decision", "previous_response_id", "parallel_tool_calls", "reasoning_effort"]) { - if (text.includes(field)) responseFields.add(field); - } - - matches.push({ - file: relativePath(dir, file), - patterns: foundPatterns, - snippets: snippets(text, foundPatterns), - }); - } - - return { - provider: example.provider, - name: example.name, - repo: example.repo, - confidence: example.confidence, - local_path: dir, - commit, - clone_or_update: cloneOrUpdate, - tool_versions: sorted(toolVersions), - beta_headers: sorted(betaHeaders), - action_names: sorted(actionNames), - response_fields: sorted(responseFields), - matches, - }; -} - -function ensureRepo(repo: string, dir: string, noUpdate: boolean): Record { - if (!existsSync(dir)) { - const res = git(["clone", "--depth", "1", repo, dir], process.cwd()); - return { action: "clone", ok: res.status === 0, stderr: res.stderr.trim() }; - } - if (noUpdate) return { action: "skip-update", ok: true, stderr: "" }; - const res = git(["pull", "--ff-only"], dir); - return { action: "pull", ok: res.status === 0, stderr: res.stderr.trim() }; -} - -function git(args: string[], cwd: string): GitResult { - const res = spawnSync("git", args, { cwd, encoding: "utf8" }); - return { - status: res.status, - stdout: res.stdout ?? "", - stderr: res.stderr ?? "", - }; -} - -async function collectFiles(root: string): Promise { - if (!existsSync(root)) return []; - const out: string[] = []; - await walk(root, out); - return out.filter((file) => /\.(py|ts|tsx|js|jsx|mjs|md|json|yaml|yml)$/i.test(file)); -} - -async function walk(dir: string, out: string[]): Promise { - const entries = await readdir(dir, { withFileTypes: true }).catch(() => []); - for (const entry of entries) { - if (entry.name === ".git" || entry.name === "node_modules" || entry.name === "__pycache__") continue; - const path = join(dir, entry.name); - if (entry.isDirectory()) await walk(path, out); - else if (entry.isFile()) out.push(path); - } -} - -function extractAll(text: string, regex: RegExp): string[] { - const values: string[] = []; - for (const match of text.matchAll(regex)) values.push(match[1] ?? match[0]); - return values; -} - -function snippets(text: string, patterns: string[]): Array> { - const lines = text.split(/\r?\n/); - const out: Array> = []; - for (const pattern of patterns.slice(0, 8)) { - const idx = lines.findIndex((line) => line.includes(pattern)); - if (idx < 0) continue; - out.push({ - pattern, - line: idx + 1, - text: (lines[idx] ?? "").trim().slice(0, 240), - }); - } - return out; -} - -function relativePath(root: string, file: string): string { - return file.startsWith(root) ? file.slice(root.length + 1) : basename(file); -} - -function groupByProvider(repos: Array>): Record> { - const out: Record> = {}; - for (const repo of repos) { - const provider = String(repo.provider); - out[provider] ??= { action_names: [], tool_versions: [], beta_headers: [], response_fields: [] }; - out[provider].action_names = sorted(new Set([...out[provider].action_names, ...repo.action_names])); - out[provider].tool_versions = sorted(new Set([...out[provider].tool_versions, ...repo.tool_versions])); - out[provider].beta_headers = sorted(new Set([...out[provider].beta_headers, ...repo.beta_headers])); - out[provider].response_fields = sorted(new Set([...out[provider].response_fields, ...repo.response_fields])); - } - return out; -} - -function sorted(values: Iterable): string[] { - return [...values].sort(); -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(err instanceof Error ? err.message : String(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts deleted file mode 100644 index 8411400..0000000 --- a/.agents/skills/update-models/reference/discover-models.ts +++ /dev/null @@ -1,757 +0,0 @@ -#!/usr/bin/env tsx -import { existsSync } from "node:fs"; -import { readFile, writeFile } from "node:fs/promises"; -import { join } from "node:path"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface Args { - provider: Provider | "all"; - out: string; - models: string[]; - candidateLimit: number; - smoke: boolean; -} - -interface SmokeResult { - status: "pass" | "inconclusive" | "unsupported" | "fail"; - tool_name?: string; - tool_version?: string | null; - beta_header?: string | null; - observed_actions: string[]; - response_item_types: string[]; - error: string | null; - [key: string]: unknown; -} - -interface ModelResult { - id: string; - display_name?: string; - name?: string | null; - created_at?: string | null; - raw?: unknown; - supports_generation?: boolean; - computer_use?: SmokeResult | Record; - model_docs?: Record; - cua?: Record; -} - -const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "xai", "moonshot"]; -const GEMINI_DOC_COMPUTER_USE_MODELS = [ - "gemini-3.5-flash", - "gemini-3-flash-preview", - "gemini-2.5-computer-use-preview-10-2025", -]; - -const OPENAI_EXCLUDE = [ - "embedding", - "moderation", - "whisper", - "tts", - "dall-e", - "image", - "audio", - "transcribe", - "realtime", -]; - -function parseArgs(argv: string[]): Args { - const out: Args = { - provider: "all", - out: "", - models: [], - candidateLimit: 20, - smoke: true, - }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--provider" && next) { - out.provider = next as Provider | "all"; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--models" && next) { - out.models = next.split(",").map((s) => s.trim()).filter(Boolean); - i++; - } else if (arg === "--candidate-limit" && next) { - out.candidateLimit = Number(next) || out.candidateLimit; - i++; - } else if (arg === "--no-smoke") { - out.smoke = false; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - if (out.provider !== "all" && !PROVIDERS.includes(out.provider)) { - throw new Error(`--provider must be one of: all, ${PROVIDERS.join(", ")}`); - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/discover-models.ts --provider all --out /tmp/cua-model-report.json - npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4 - -Options: - --provider - --models Smoke-test explicit models instead of inferred candidates. - --candidate-limit Max inferred candidates per provider. Default: 20. - --no-smoke Only list metadata. - --out Write JSON report to file. -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const selected = args.provider === "all" ? PROVIDERS : [args.provider]; - const report: Record = { - generated_at: new Date().toISOString(), - providers: {}, - }; - const providers = report.providers as Record; - await Promise.all(selected.map(async (provider) => { - providers[provider] = await runProvider(provider, args); - })); - await emitJson(report, args.out); -} - -async function runProvider(provider: Provider, args: Args): Promise> { - try { - if (provider === "openai") return await discoverOpenAI(args); - if (provider === "anthropic") return await discoverAnthropic(args); - if (provider === "gemini") return await discoverGemini(args); - if (provider === "xai") return await discoverXai(args); - if (provider === "moonshot") return await discoverMoonshot(args); - throw new Error(`unknown provider ${provider satisfies never}`); - } catch (err) { - return { - provider, - error: publicError(err), - }; - } -} - -async function discoverOpenAI(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels - .map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: likelyOpenAIGenerationModel(String(m.id)), - })) - .sort((a, b) => String(b.created_at ?? "").localeCompare(String(a.created_at ?? ""))); - - const candidates = explicitOrCandidates(args, models.filter((m) => m.supports_generation).map((m) => m.id)); - await annotateOpenAIModelDocs(models.filter((m) => candidates.includes(m.id))); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeOpenAI(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("openai", models); - return { provider: "openai", metadata_source: "client.models.list()", models, candidates }; -} - -async function smokeMeta(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.responses.create({ - model, - store: false, - parallel_tool_calls: false, - max_output_tokens: 512, - reasoning: { effort: "low" }, - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}` }, - ], - }], - tools: [{ - type: "function", - name: "click", - description: "Click at normalized 0-1000 screen coordinates.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }], - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "function_call"); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -async function discoverXai(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.XAI_API_KEY, baseURL: "https://api.x.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: likelyXaiGenerationModel(String(m.id)), - model_docs: { - url: String(m.id) === "grok-4.5" ? "https://docs.x.ai/developers/grok-4-5" : "https://docs.x.ai/developers/models", - responses_endpoint: "supported", - function_calling: "supported", - image_input: "verify-per-model", - coordinate_space: "CUA-defined 0-1000", - }, - })); - const candidates = explicitOrCandidates( - args, - models - .filter((model) => model.supports_generation) - .sort(compareXaiCandidates) - .map((model) => model.id), - ); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeXai(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("xai", models); - return { provider: "xai", metadata_source: "xAI models.list()", models, candidates }; -} - -async function smokeXai(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.responses.create({ - model, - store: true, - parallel_tool_calls: false, - max_output_tokens: 768, - reasoning: { effort: "low" }, - instructions: "Coordinates are normalized from 0 to 1000 relative to the screenshot.", - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}`, detail: "high" }, - ], - }], - tools: [{ - type: "function", - name: "click", - description: "Click at normalized 0-1000 screen coordinates.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }], - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "function_call"); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - reasoning_effort: "low", - coordinate_space: "0-1000", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -function likelyXaiGenerationModel(id: string): boolean { - const lower = id.toLowerCase(); - return lower.startsWith("grok-") && !lower.includes("imagine"); -} - -async function discoverMoonshot(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ apiKey: process.env.MOONSHOT_API_KEY, baseURL: "https://api.moonshot.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: String(m.id).toLowerCase().startsWith("kimi-"), - model_docs: { - url: "https://platform.kimi.ai/docs/api/tool-use", - chat_completions_endpoint: "supported", - function_calling: "supported", - image_input: "verify-per-model", - coordinate_space: "CUA-defined 0-1 fractions", - }, - })); - const candidates = explicitOrCandidates( - args, - models - .filter((model) => model.supports_generation) - .sort(compareMoonshotCandidates) - .map((model) => model.id), - ); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeMoonshot(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("moonshot", models); - return { provider: "moonshot", metadata_source: "Moonshot models.list()", models, candidates }; -} - -async function smokeMoonshot(client: any, model: string): Promise { - try { - const screenshot = await readFile(fixtureScreenshotPath()); - const response = await client.chat.completions.create({ - model, - parallel_tool_calls: false, - max_tokens: 8192, - messages: [ - { - role: "system", - content: "Coordinates are fractions of the screenshot, normalized from 0 to 1.", - }, - { - role: "user", - content: [ - { type: "text", text: "Call the click tool for the sign in link. Do not answer only in text." }, - { type: "image_url", image_url: { url: `data:image/png;base64,${screenshot.toString("base64")}` } }, - ], - }, - ], - tools: [{ - type: "function", - function: { - name: "click", - description: "Click at coordinates given as 0-1 fractions of the screenshot.", - parameters: { - type: "object", - properties: { x: { type: "number" }, y: { type: "number" } }, - required: ["x", "y"], - additionalProperties: false, - }, - }, - }], - }); - const calls: any[] = response.choices?.[0]?.message?.tool_calls ?? []; - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "function_tools", - tool_version: null, - beta_header: null, - observed_actions: unique(calls.map((call) => call?.function?.name).filter(Boolean)), - response_item_types: unique([response.choices?.[0]?.finish_reason].filter(Boolean)), - coordinate_space: "0-1 fractions", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -function compareMoonshotCandidates(a: ModelResult, b: ModelResult): number { - if (a.id === "kimi-k3") return -1; - if (b.id === "kimi-k3") return 1; - return String(b.created_at ?? "").localeCompare(String(a.created_at ?? "")); -} - -function compareXaiCandidates(a: ModelResult, b: ModelResult): number { - if (a.id === "grok-4.5") return -1; - if (b.id === "grok-4.5") return 1; - return String(b.created_at ?? "").localeCompare(String(a.created_at ?? "")); -} - -function likelyOpenAIGenerationModel(id: string): boolean { - const lower = id.toLowerCase(); - if (OPENAI_EXCLUDE.some((needle) => lower.includes(needle))) return false; - return lower.startsWith("gpt-") || /^o\d/.test(lower) || lower.includes("computer-use"); -} - -async function annotateOpenAIModelDocs(models: ModelResult[]): Promise { - await Promise.all(models.map(async (model) => { - model.model_docs = await fetchOpenAIModelDocs(model.id); - })); -} - -async function fetchOpenAIModelDocs(modelId: string): Promise> { - const docId = canonicalOpenAIModelDocId(modelId); - const url = `https://developers.openai.com/api/docs/models/${docId}`; - try { - const response = await fetch(url); - const text = await response.text(); - return { - url, - ok: response.ok, - streaming: supportStatus(text, "Streaming"), - function_calling: supportStatus(text, "Function calling"), - computer_use: supportStatus(text, "Computer use"), - responses_endpoint: text.includes("v1/responses") ? "supported" : "unknown", - }; - } catch (err) { - return { - url, - ok: false, - error: publicError(err), - }; - } -} - -function canonicalOpenAIModelDocId(modelId: string): string { - return modelId.replace(/-\d{4}-\d{2}-\d{2}$/, ""); -} - -function supportStatus(text: string, label: string): "supported" | "not_supported" | "unknown" { - const compact = text.replace(/<[^>]*>/g, " ").replace(/\s+/g, " "); - const re = new RegExp(`${escapeRegex(label)}\\s+(Supported|Not supported)`, "i"); - const match = compact.match(re); - if (!match) return "unknown"; - return match[1]?.toLowerCase() === "supported" ? "supported" : "not_supported"; -} - -function escapeRegex(value: string): string { - return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); -} - -async function smokeOpenAI(client: any, model: string): Promise { - try { - const response = await client.responses.create({ - model, - input: "Use the computer tool to request a screenshot. Do not answer in text.", - tools: [{ type: "computer" }], - tool_choice: { type: "computer" }, - max_output_tokens: 64, - }); - const output: any[] = response.output ?? []; - const calls = output.filter((item) => item?.type === "computer_call"); - const actions = calls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer", - tool_version: null, - beta_header: null, - observed_actions: unique(actions.map((a) => a?.type).filter(Boolean)), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "computer" }); - } -} - -async function discoverAnthropic(args: Args): Promise> { - const Anthropic = await importDefault("@anthropic-ai/sdk", "Anthropic"); - const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); - const rawModels = await collectAsync(client.models.list({ limit: 1000 })); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: m.display_name ?? m.id, - created_at: m.created_at ?? null, - raw: m, - supports_generation: m.type === "model" || String(m.id).startsWith("claude-"), - })); - const candidates = explicitOrCandidates(args, models.filter((m) => m.id.startsWith("claude-")).map((m) => m.id)); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeAnthropic(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("anthropic", models); - return { provider: "anthropic", metadata_source: "client.models.list({ limit: 1000 })", models, candidates }; -} - -type AnthropicToolPair = { tool: string; beta: string }; - -const ANTHROPIC_TOOL_PAIRS: AnthropicToolPair[] = [ - { tool: "computer_20251124", beta: "computer-use-2025-11-24" }, - { tool: "computer_20250124", beta: "computer-use-2025-01-24" }, - { tool: "computer_20241022", beta: "computer-use-2024-10-22" }, -]; - -async function smokeAnthropic(client: any, model: string): Promise> { - const attempts: Record[] = []; - const runtimePair = anthropicRuntimeToolPair(model); - for (const pair of orderAnthropicPairs(runtimePair)) { - try { - const response = await client.beta.messages.create({ - model, - max_tokens: 64, - messages: [{ role: "user", content: "Use the computer tool to take a screenshot. Do not answer in text." }], - tools: [{ - type: pair.tool, - name: "computer", - display_width_px: 1024, - display_height_px: 768, - display_number: 1, - }], - betas: [pair.beta], - }); - const content: any[] = response.content ?? []; - const calls = content.filter((block) => block?.type === "tool_use" && block?.name === "computer"); - const actions = calls.map((call) => call?.input?.action).filter(Boolean); - const result = { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer", - tool_version: pair.tool, - beta_header: pair.beta, - runtime_tool_version: runtimePair.tool, - runtime_beta_header: runtimePair.beta, - runtime_compatible: calls.length > 0 && pair.tool === runtimePair.tool && pair.beta === runtimePair.beta, - observed_actions: unique(actions), - response_item_types: unique(content.map((block) => block?.type).filter(Boolean)), - stop_reason: response.stop_reason ?? null, - error: null, - }; - if (result.status === "pass") return result; - attempts.push(result); - } catch (err) { - attempts.push(smokeError(err, { tool_name: "computer", tool_version: pair.tool, beta_header: pair.beta })); - } - } - return { status: "fail", attempts, error: attempts.at(-1)?.error ?? "all tool versions failed" }; -} - -function orderAnthropicPairs(runtimePair: AnthropicToolPair): AnthropicToolPair[] { - const rest = ANTHROPIC_TOOL_PAIRS.filter((pair) => pair.tool !== runtimePair.tool || pair.beta !== runtimePair.beta); - return [runtimePair, ...rest]; -} - -function anthropicRuntimeToolPair(model: string): AnthropicToolPair { - const id = model.toLowerCase(); - if ( - id.startsWith("claude-opus-4-8") || - id.startsWith("claude-opus-4-7") || - id.startsWith("claude-opus-4-6") || - id.startsWith("claude-opus-4-5") || - id.startsWith("claude-sonnet-4-6") || - id.startsWith("claude-fable-5") - ) { - return { tool: "computer_20251124", beta: "computer-use-2025-11-24" }; - } - return { tool: "computer_20250124", beta: "computer-use-2025-01-24" }; -} - -async function discoverGemini(args: Args): Promise> { - const { GoogleGenAI } = await import("@google/genai"); - const apiKey = process.env.GOOGLE_API_KEY || process.env.GEMINI_API_KEY; - const client = new GoogleGenAI({ apiKey }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: normalizeGeminiModelId(m.name ?? m.id ?? m.baseModelId), - name: m.name ?? null, - display_name: m.displayName ?? m.display_name ?? m.name ?? m.id, - created_at: null, - raw: m, - supports_generation: hasGenerateContent(m), - })); - const geminiCandidates = unique([ - ...GEMINI_DOC_COMPUTER_USE_MODELS, - ...rankGeminiCandidates(models.filter((m) => m.supports_generation && likelyGeminiCandidate(m.id))).map((m) => m.id), - ]); - const candidates = explicitOrCandidates(args, geminiCandidates); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id || m.name === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeGemini(client, id); - if (!models.find((m) => m.id === id || m.name === id)) models.unshift(model); - })); - } - await annotateCuaSupport("gemini", models); - return { provider: "gemini", metadata_source: "client.models.list()", models, candidates }; -} - -async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise { - const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; - const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined); - for (const model of models) { - const inRegistry = getBuiltinModel ? !!getBuiltinModel(piProvider as never, model.id as never) : false; - const localAdapterSupport = localAdapterSupportStatus(provider, model); - model.cua = { - provider_inference: provider, - pi_ai_registry: inRegistry ? "present" : "missing", - dynamic_model_fallback: "available", - local_adapter_support: localAdapterSupport, - }; - } -} - -function localAdapterSupportStatus(provider: Provider, model: ModelResult): string { - if (!model.computer_use || !("status" in model.computer_use) || model.computer_use.status !== "pass") { - return "needs-check"; - } - if (provider !== "anthropic") return "passes-smoke"; - return model.computer_use.runtime_compatible === true ? "passes-smoke" : "smoke-pass-runtime-mismatch"; -} - -function hasGenerateContent(model: any): boolean { - const actions = model.supportedActions ?? model.supported_actions ?? model.supportedGenerationMethods ?? []; - return Array.isArray(actions) && actions.some((a) => String(a).toLowerCase() === "generatecontent"); -} - -function likelyGeminiCandidate(id: string): boolean { - const lower = String(id ?? "").toLowerCase(); - return lower.includes("gemini") && !lower.includes("embedding") && !lower.includes("tts") && !lower.includes("imagen"); -} - -function rankGeminiCandidates(models: ModelResult[]): ModelResult[] { - return [...models].sort((a, b) => geminiScore(b.id) - geminiScore(a.id)); -} - -function geminiScore(id: string): number { - const lower = id.toLowerCase(); - let score = 0; - if (lower.includes("computer-use")) score += 100; - if (lower.includes("gemini-3")) score += 80; - if (lower.includes("preview")) score += 20; - if (lower.includes("flash")) score += 10; - if (lower.includes("pro")) score += 5; - return score; -} - -async function smokeGemini(client: any, model: string): Promise> { - const configVariants = [ - { tools: [{ computerUse: { environment: "ENVIRONMENT_BROWSER" } }], maxOutputTokens: 64 }, - { tools: [{ computer_use: { environment: "ENVIRONMENT_BROWSER" } }], maxOutputTokens: 64 }, - ]; - const attempts: Record[] = []; - for (const config of configVariants) { - try { - const response = await client.models.generateContent({ - model, - contents: [{ role: "user", parts: [{ text: "Use the computer-use tool to open the web browser. Do not answer in text." }] }], - config, - }); - const parts: any[] = response?.candidates?.[0]?.content?.parts ?? []; - const calls = parts.map((part) => part.functionCall ?? part.function_call).filter(Boolean); - const actions = calls.map((call) => call.name).filter(Boolean); - const result = { - status: calls.length > 0 ? "pass" : "inconclusive", - tool_name: "computer_use", - tool_version: null, - beta_header: null, - observed_actions: unique(actions), - response_item_types: unique(parts.map((part) => part.functionCall || part.function_call ? "function_call" : part.text ? "text" : Object.keys(part)[0]).filter(Boolean)), - error: null, - }; - if (result.status === "pass") return result; - attempts.push(result); - } catch (err) { - attempts.push(smokeError(err, { tool_name: "computer_use" })); - } - } - return { status: "fail", attempts, error: attempts.at(-1)?.error ?? "all config variants failed" }; -} - -function explicitOrCandidates(args: Args, ids: string[]): string[] { - return (args.models.length ? args.models : ids).slice(0, args.candidateLimit); -} - -async function importDefault(pkg: string, named: string): Promise { - try { - const mod = await import(pkg); - return mod.default ?? mod[named]; - } catch (err) { - throw new Error(`failed to import ${pkg}. Run npm install first. ${publicError(err)}`); - } -} - -async function collectAsync(value: any): Promise { - const awaited = await value; - if (Array.isArray(awaited)) return awaited; - if (Array.isArray(awaited?.data)) return awaited.data; - if (Array.isArray(awaited?.models)) return awaited.models; - if (Array.isArray(awaited?.items)) return awaited.items; - if (typeof awaited?.[Symbol.asyncIterator] === "function") { - const out: any[] = []; - for await (const item of awaited) out.push(item); - return out; - } - if (typeof awaited?.[Symbol.iterator] === "function") return Array.from(awaited); - return []; -} - -function normalizeGeminiModelId(id: unknown): string { - const value = String(id ?? ""); - return value.startsWith("models/") ? value.slice("models/".length) : value; -} - -function smokeError(err: unknown, extra: Record = {}): SmokeResult { - return { - status: isUnsupportedError(err) ? "unsupported" : "fail", - ...extra, - observed_actions: [], - response_item_types: [], - error: publicError(err), - }; -} - -function isUnsupportedError(err: unknown): boolean { - const msg = publicError(err).toLowerCase(); - return msg.includes("unsupported") || msg.includes("not support") || msg.includes("not enabled") || msg.includes("not compatible") || msg.includes("invalid tool"); -} - -function publicError(err: unknown): string { - if (err && typeof err === "object" && "status" in err && "message" in err) { - return `${String((err as { status: unknown }).status)}: ${String((err as { message: unknown }).message)}`; - } - return err instanceof Error ? err.message : String(err); -} - -function unique(values: T[]): T[] { - return [...new Set(values)]; -} - -function fixtureScreenshotPath(): string { - const path = [ - join(process.cwd(), "examples", "screenshot.png"), - join(process.cwd(), "packages", "ai", "examples", "screenshot.png"), - ].find(existsSync); - if (!path) throw new Error("could not find packages/ai/examples/screenshot.png"); - return path; -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(publicError(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts deleted file mode 100644 index 1dfef98..0000000 --- a/.agents/skills/update-models/reference/provider-doc-drift.ts +++ /dev/null @@ -1,199 +0,0 @@ -#!/usr/bin/env tsx -import { readFile, writeFile } from "node:fs/promises"; -import process from "node:process"; - -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; - -interface Args { - examples: string; - out: string; -} - -interface FetchResult { - url: string; - ok: boolean; - status: number | null; - error?: string; - text: string; -} - -const DOCS: Record = { - openai: [ - "https://developers.openai.com/api/docs/guides/tools-computer-use", - "https://raw.githubusercontent.com/openai/openai-node/master/src/resources/responses/responses.ts", - ], - anthropic: [ - "https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool", - "https://console.anthropic.com/docs/en/agents-and-tools/tool-use/tool-reference", - ], - gemini: [ - "https://ai.google.dev/gemini-api/docs/computer-use", - "https://ai.google.dev/api/models", - ], - xai: [ - "https://docs.x.ai/developers/grok-4-5", - "https://docs.x.ai/developers/tools/function-calling", - "https://docs.x.ai/developers/model-capabilities/images/understanding", - ], - moonshot: [ - "https://www.kimi.com/blog/kimi-k3", - "https://platform.kimi.ai/docs/api/tool-use", - "https://platform.kimi.ai/docs/guide/use-kimi-vision-model", - ], -}; - -const LOCAL_FILES: Record = { - openai: "packages/ai/src/providers/openai/index.ts", - anthropic: "packages/ai/src/providers/anthropic/actions.ts", - gemini: "packages/ai/src/providers/gemini/index.ts", - xai: "packages/ai/src/providers/xai/index.ts", - moonshot: "packages/ai/src/providers/moonshot/index.ts", -}; - -const ACTION_REGEXES: Record = { - openai: /\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g, - anthropic: /\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g, - gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g, - xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, - moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, -}; - -function parseArgs(argv: string[]): Args { - const out: Args = { examples: "", out: "" }; - for (let i = 0; i < argv.length; i++) { - const arg = argv[i]; - const next = argv[i + 1]; - if (arg === "--examples" && next) { - out.examples = next; - i++; - } else if (arg === "--out" && next) { - out.out = next; - i++; - } else if (arg === "--help" || arg === "-h") { - usage(); - } else { - throw new Error(`unknown argument: ${arg}`); - } - } - return out; -} - -function usage(): never { - console.log(`Usage: - npx tsx .agents/skills/update-models/reference/provider-doc-drift.ts --examples /tmp/cua-example-evidence.json --out /tmp/cua-drift.json -`); - process.exit(0); -} - -async function main(): Promise { - const args = parseArgs(process.argv.slice(2)); - const examples = args.examples ? JSON.parse(await readFile(args.examples, "utf8")) : {}; - const report: Record = { - generated_at: new Date().toISOString(), - providers: {}, - }; - const providers = report.providers as Record; - await Promise.all((Object.keys(DOCS) as Provider[]).map(async (provider) => { - providers[provider] = await checkProvider(provider, examples); - })); - await emitJson(report, args.out); -} - -async function checkProvider(provider: Provider, examples: any): Promise> { - const docTexts = await Promise.all(DOCS[provider].map(fetchText)); - const docText = docTexts.map((r) => r.text).join("\n"); - const localText = await readFile(LOCAL_FILES[provider], "utf8").catch((err) => `/* failed to read local file: ${err.message} */`); - const example = examples?.by_provider?.[provider] ?? {}; - - const documentedActions = unique(extractAll(docText, ACTION_REGEXES[provider])); - const localActions = unique(extractAll(localText, ACTION_REGEXES[provider])); - const exampleActions: string[] = example.action_names ?? []; - const documentedToolVersions = unique(extractAll(docText, /computer_\d{8}/g)); - const localToolVersions = unique(extractAll(localText, /computer_\d{8}/g)); - const exampleToolVersions: string[] = example.tool_versions ?? []; - const documentedBetaHeaders = unique(extractAll(docText, /computer-use-\d{4}-\d{2}-\d{2}/g)); - const localBetaHeaders = unique(extractAll(localText, /computer-use-\d{4}-\d{2}-\d{2}/g)); - const exampleBetaHeaders: string[] = example.beta_headers ?? []; - - return { - provider, - doc_sources: docTexts.map(({ url, ok, status, error }) => ({ url, ok, status, error })), - documented_tool_versions: sorted(documentedToolVersions), - example_tool_versions: sorted(exampleToolVersions), - local_tool_versions: sorted(localToolVersions), - newer_tool_versions: sorted(difference(new Set([...documentedToolVersions, ...exampleToolVersions]), new Set(localToolVersions))), - documented_beta_headers: sorted(documentedBetaHeaders), - example_beta_headers: sorted(exampleBetaHeaders), - local_beta_headers: sorted(localBetaHeaders), - newer_beta_headers: sorted(difference(new Set([...documentedBetaHeaders, ...exampleBetaHeaders]), new Set(localBetaHeaders))), - documented_actions: sorted(documentedActions), - example_repo_actions: sorted(exampleActions), - repo_supported_actions: sorted(localActions), - unknown_documented_actions: provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(documentedActions), new Set(localActions))), - unknown_example_actions: provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(exampleActions), new Set(localActions))), - response_fields_from_examples: sorted(example.response_fields ?? []), - notes: notesFor(provider, documentedToolVersions, exampleToolVersions, localToolVersions), - }; -} - -async function fetchText(url: string): Promise { - try { - const response = await fetch(url); - const text = await response.text(); - return { url, ok: response.ok, status: response.status, text }; - } catch (err) { - return { url, ok: false, status: null, error: err instanceof Error ? err.message : String(err), text: "" }; - } -} - -function notesFor(provider: Provider, documentedToolVersions: string[], exampleToolVersions: string[], localToolVersions: string[]): string[] { - const notes: string[] = []; - if (provider === "openai") { - notes.push("OpenAI's GA computer tool is currently undated (`computer`); drift usually appears as action-shape changes or preview deprecations."); - } - if (provider === "anthropic") { - const newest = sorted(new Set([...documentedToolVersions, ...exampleToolVersions])).at(-1); - if (newest && !localToolVersions.includes(newest)) { - notes.push(`Anthropic docs/examples mention ${newest}, which is not in local constants.`); - } - } - if (provider === "gemini") { - notes.push("Gemini official computer use emits predefined function-call names; keep this separate from CUA custom function declarations."); - } - if (provider === "xai") { - notes.push("xAI uses developer-defined function tools and does not document a native coordinate protocol; compare Responses compatibility, reasoning controls, and CUA's normalized coordinate contract instead."); - } - if (provider === "moonshot") { - notes.push("Moonshot uses developer-defined function tools over OpenAI-compatible chat completions and does not document a coordinate protocol; Kimi grounding emits 0-1 width/height fractions, so compare against CUA's fractional coordinate contract."); - } - return notes; -} - -function extractAll(text: string, regex: RegExp): string[] { - const values: string[] = []; - for (const match of String(text).matchAll(regex)) values.push(match[1] ?? match[0]); - return values; -} - -function difference(a: Set, b: Set): string[] { - return [...a].filter((value) => !b.has(value)); -} - -function unique(values: T[]): T[] { - return [...new Set(values)]; -} - -function sorted(values: Iterable): string[] { - return [...values].sort(); -} - -async function emitJson(value: unknown, outPath: string): Promise { - const text = `${JSON.stringify(value, null, 2)}\n`; - if (outPath) await writeFile(outPath, text); - else process.stdout.write(text); -} - -main().catch((err) => { - console.error(err instanceof Error ? err.message : String(err)); - process.exit(1); -}); diff --git a/.agents/skills/update-models/reference/report-schema.md b/.agents/skills/update-models/reference/report-schema.md deleted file mode 100644 index 9887465..0000000 --- a/.agents/skills/update-models/reference/report-schema.md +++ /dev/null @@ -1,100 +0,0 @@ -# Report Schema - -Use this shape for JSON reports and the same fields when writing a Markdown summary. - -## Top Level - -```json -{ - "generated_at": "2026-04-26T00:00:00.000Z", - "repo": "/path/to/cua", - "providers": { - "openai": {}, - "anthropic": {}, - "gemini": {}, - "xai": {}, - "moonshot": {} - }, - "example_evidence": {}, - "drift": {}, - "recommendations": [] -} -``` - -## Provider Result - -```json -{ - "provider": "openai", - "metadata_source": "models.list", - "models": [ - { - "id": "gpt-5.5", - "display_name": "GPT-5.5", - "created_at": "2026-04-01T00:00:00.000Z", - "raw": {}, - "supports_generation": true, - "model_docs": { - "url": "https://developers.openai.com/api/docs/models/gpt-5.5", - "streaming": "supported", - "function_calling": "supported", - "computer_use": "supported", - "responses_endpoint": "supported" - }, - "computer_use": { - "status": "pass", - "tool_name": "computer", - "tool_version": null, - "beta_header": null, - "observed_actions": ["screenshot"], - "response_item_types": ["computer_call"], - "accepts_image_tool_results": "assumed-from-docs", - "error": null - }, - "cua": { - "provider_inference": "openai", - "pi_ai_registry": "missing", - "dynamic_model_fallback": "available", - "local_adapter_support": "passes-smoke" - }, - "recommended_action": "candidate-default" - } - ] -} -``` - -## Drift Result - -```json -{ - "provider": "anthropic", - "documented_tool_versions": ["computer_20251124"], - "example_tool_versions": ["computer_20251124"], - "local_tool_versions": ["computer_20251124"], - "documented_actions": ["screenshot", "left_click"], - "example_repo_actions": ["screenshot", "left_click"], - "observed_actions": ["screenshot"], - "repo_supported_actions": ["screenshot", "left_click"], - "unknown_observed_actions": [], - "notes": [] -} -``` - -## Markdown Summary - -Use this order: - -1. New or changed model candidates. -2. Computer-use smoke-test pass/fail/inconclusive table. -3. Official example repo findings. -4. Drift against local adapter constants. -5. CUA support changes needed: - - default model update - - `pi-ai` registry present or dynamic fallback needed - - model-specific docs do not block required runtime features - - provider routing update - - adapter action/tool version update - - docs/config examples update -6. Recommended repo changes and blockers. - -Only recommend changing defaults when metadata discovery, official evidence, smoke tests, and CUA compatibility all line up. diff --git a/README.md b/README.md index 6e3d9be..7970f58 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ packages/ (`@onkernel/cua-agent`) — `CuaAgent`/`CuaAgentHarness` run the full computer-use loop against a Kernel browser. It sits on [`packages/ai`](packages/ai) (`@onkernel/cua-ai`), the model layer with the -curated computer-use model catalog, canonical tool schemas, and per-provider +pi-ai model catalog, canonical tool schemas, and per-provider adapters on top of pi-ai; reach for cua-ai directly only when you bring your own execution. @@ -101,7 +101,7 @@ export KERNEL_API_KEY=sk_... # always required # single-shot cua -p "Open https://news.ycombinator.com and tell me the top story" -# list supported model ids +# list selectable model ids cua models # Claude @@ -141,7 +141,7 @@ cua -p -o jsonl "open example.com and tell me the heading" ## How it works -1. **Model layer** — `@onkernel/cua-ai` owns the curated model catalog, +1. **Model layer** — `@onkernel/cua-ai` opens pi-ai's whole model catalog, stable tool identities, explicit tool factories/toolsets, compatibility checks, and provider declarations/headers/payload transforms. 2. **Execution layer** — `@onkernel/cua-agent` composes around @@ -171,7 +171,7 @@ Highlights: - `-p`/`--print` for single-shot mode; `-o jsonl` for structured output. - `cua models` to list supported `-m`/`--model` values and their providers. -- `-m`/`--model ` to choose one of those supported models. +- `-m`/`--model ` to choose any model pi-ai carries. - `/model` in the TUI for a searchable model picker; `/model ` still switches directly. - `/tools` in the TUI to enable or disable tools for the current session. diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 0672ec1..2025f15 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## 0.14.0 - 2026-08-14 + +- `CuaAgentHarness` no longer refuses a model ref that is absent from its + supplied `Models` collection: it falls back to the registry, and an id the + registry lacks is synthesized. Update `@onkernel/cua-ai` to 0.14.0. + ## 0.13.0 - 2026-08-13 Breaking: Tzafon and Yutori support is removed. diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 6dd045e..5dcf0fd 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -27,7 +27,6 @@ import { cuaModels, type CuaIncomingToolPlan, type CuaModelRef, - findCuaAnnotation, getCuaModel, parseCuaModelRef, type CuaSimpleStreamOptions, @@ -499,7 +498,6 @@ const defaultCuaStream: StreamFn = (model, context, options) => cuaModels().stre function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model { const { provider, model: id } = parseCuaModelRef(ref); - if (!findCuaAnnotation(provider, id)) throw new Error(`unsupported CUA model "${ref}"`); return models.getModel(provider, id) ?? getCuaModel(ref); } diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index ed459a4..742ecf5 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -451,7 +451,7 @@ describe("CuaAgent explicit tools", () => { }); describe("CuaAgentHarness explicit tools", () => { - it("resolves annotated refs from supplied models for construction and setModel", async () => { + it("resolves refs from supplied models for construction and setModel", async () => { const models = createCuaModels(); const openai = models.getProvider("openai")!; const first = { ...getCuaModel("openai:gpt-5.5"), baseUrl: "https://first.example" }; @@ -462,7 +462,11 @@ describe("CuaAgentHarness explicit tools", () => { expect(harness.getModel()).toBe(first); await harness.setModel("openai:gpt-5.6-sol"); expect(harness.getModel()).toBe(second); - expect(() => new CuaAgentHarness({ ...services, browser, client, models, model: "openai:gpt-4o", tools: [] })).toThrow(/unsupported CUA model/); + + // A ref the supplied collection does not carry falls back to the registry + // rather than being refused: the provider decides what exists. + const fallback = new CuaAgentHarness({ ...services, browser, client, models, model: "openai:gpt-5.4", tools: [] }); + expect(fallback.getModel().id).toBe("gpt-5.4"); }); it("uses composition, hides active-tool APIs, and supports an empty catalog", async () => { diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index f1e3d6a..5f373c6 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## 0.14.0 - 2026-08-14 + +Breaking: the model allowlist is removed. + +- `listCuaModels()` returns pi-ai's whole catalog — 37 providers, ~1,150 models — + instead of a curated subset, and each entry now carries `nativeSurfaces` and + `vision` so callers can render what a model can do. +- `getCuaModel(ref)` resolves any model pi-ai carries, and synthesizes one for + an id the registry has not caught up with, using the sibling that shares the + longest id prefix and preferring the latest such sibling. Providers migrate + transports mid-generation, so a new id follows its nearest, newest relative. + Only an unqualified ref or a provider pi-ai does not carry is refused. +- `CUA_MODEL_ANNOTATIONS`, `CUA_PROVIDERS`, `isCuaProvider`, and the + `CuaProvider` union are gone; `CuaProvider` is now a provider id string and + `cuaProviders()` returns what pi-ai carries. `providerForModel` no longer + throws. +- Two tables replace the allowlist, neither of which decides whether a model may + run: `CUA_NATIVE_SURFACES` (which models have a provider-native computer or + browser tool, with first-party sources) and `CUA_MODEL_QUIRKS` (request-shape + limits, each carrying the documented limit or observed failure that justifies + it). `cuaModelCapabilities` reads the quirk table and defaults to permissive; + `cuaNativeSurfaces(model)` and `cuaModelQuirks(model)` are exported for menus + and diagnostics. + ## 0.13.0 - 2026-08-13 - Remove the Meta provider. pi-ai ships no `meta` provider, so cua hand-wrote a diff --git a/packages/ai/README.md b/packages/ai/README.md index d7e7b20..5d6ec5c 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -32,8 +32,8 @@ console.table(listCuaModels("google")); ``` `gemini:` aliases `google:` and `moonshot:` aliases `moonshotai:`. The package -does not export a default model. See [supported models](docs/supported-models.md) -for the curated list. +does not export a default model. See [models and native surfaces](docs/supported-models.md) +for which models have provider-native tools and which have known request limits. ## Explicit tools diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 6c80220..bfa9c19 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -1,130 +1,98 @@ -# Supported CUA Models - -`@onkernel/cua-ai` accepts any pi-ai model whose ID is annotated as -CUA-supporting in `CUA_MODEL_ANNOTATIONS` (see -[`src/models.ts`](https://github.com/kernel/cua/blob/main/packages/ai/src/models.ts)). -Annotations are either a `family` -match or an `exact` ID match. A family match covers the family root plus -suffixes made of hyphen-separated numeric segments — revisions and dated -snapshots such as `claude-opus-4-7`, `gpt-5.5-2026-04-23`, or -`claude-3-7-sonnet-20250219`. Named sibling variants like `gpt-5.4-mini` -are distinct models that may not support computer use, so they need their -own annotation. Each annotation cites the provider's CUA docs. - -The list below is the current snapshot. Run -`listCuaModels(provider?)` for the live list — it merges pi-ai's registry -with CUA-only entries that pi-ai does not ship yet. - -## `openai` - -CLI default interaction: CUA browser primitives plus the explicit -`browser_act` verified-plan tool. The optional native computer tool uses pixel -coordinates. - -Exact IDs: - -- `gpt-5.6-sol` ([docs](https://developers.openai.com/api/docs/models/gpt-5.6-sol)) - -Family matches (root + numeric revision/dated-snapshot suffixes): - -- `gpt-5.4` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4)) -- `gpt-5.4-mini` ([docs](https://developers.openai.com/api/docs/models/gpt-5.4-mini)) -- `gpt-5.5` ([docs](https://developers.openai.com/api/docs/models/gpt-5.5)) - -## `anthropic` - -CLI default interaction: native `browser_20260701` on supported model families, -with CUA browser primitives plus explicit `browser_act` as the -model-compatibility fallback. If the active -credential cannot access the native browser beta, CUA uses its equivalent -function-tool transport. The optional native computer tool uses pixel -coordinates. - -Family matches (root + numeric revision/dated-snapshot suffixes): - -- `claude-3-7-sonnet` -- `claude-opus-4` -- `claude-opus-5` -- `claude-sonnet-4` -- `claude-sonnet-5` -- `claude-haiku-4` -- `claude-fable-5` - -Source: [Anthropic computer use docs](https://docs.anthropic.com/en/docs/build-with-claude/computer-use). - -## `google` - -Coordinates: normalized 0–999 - -Model refs use the `google:` prefix; `gemini:` is accepted as an alias. - -Exact IDs: - -- `gemini-3.6-flash` (recommended) -- `gemini-3.5-flash` -- `gemini-3.5-flash-lite` - -Google computer use is configured explicitly with -`cua.providers.google.toolsets.browser()`, which emits Google's native -`tools.computer_use` declaration and current predefined browser actions. -Partial catalogs exclude every unselected current action. Legacy and preview -model/tool surfaces are intentionally not exposed. - -Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use). - -## `xai` - -CLI default interaction: CUA browser primitives plus explicit `browser_act`. - -Exact IDs: - -- `grok-4.5` - -Grok 4.5 uses xAI's OpenAI-compatible Responses API with ordinary CUA browser -function tools. xAI does not define a native computer tool. Tool loops continue -through `previous_response_id`. -CUA adds xAI's doubled token-price tier above 200k input tokens to pi-ai's -Grok 4.5 model metadata. - -Source: [Grok 4.5 docs](https://docs.x.ai/developers/grok-4-5), [function calling](https://docs.x.ai/developers/tools/function-calling), and [image understanding](https://docs.x.ai/developers/model-capabilities/images/understanding). - -## `moonshotai` - -CLI default interaction: CUA browser primitives only. Moonshot's API accepts the -complex `browser_wait_for` schema but rejects a request outright once the much -larger `browser_act` schema is attached, so `browser_act` is unavailable on -Moonshot models and the catalog rejects it explicitly. - -Model refs use the `moonshotai:` prefix; `moonshot:` is accepted as an alias. - -Exact IDs: - -- `kimi-k3` - -Kimi K3 uses Moonshot's OpenAI-compatible chat completions API with ordinary -CUA browser function tools. Moonshot does not define a native computer tool. -K3 launched with max-only thinking effort—other levels are clamped away until -Moonshot ships them. - -Source: [Kimi K3 announcement](https://www.kimi.com/blog/kimi-k3), [tool use](https://platform.kimi.ai/docs/api/tool-use), and [vision input](https://platform.kimi.ai/docs/guide/use-kimi-vision-model). - -## `openrouter` - -CLI default interaction: per model, not per provider. OpenRouter fronts several -model families, so the CLI asks each model whether it accepts `browser_act`'s -schema rather than assuming one answer for the whole provider. - -Model refs use the `openrouter:` prefix: - -- `moonshotai/kimi-k3` — browser primitives only: accepts complex function - schemas but rejects the larger `browser_act` schema. State mutations are - serialized. -- `meta/muse-spark-1.1` — browser primitives plus explicit `browser_act`. State - mutations are serialized. - -Both use OpenRouter's OpenAI-compatible chat completions API with ordinary CUA -browser function tools. OpenRouter does not expose the provider-native computer -tools declared by other CUA providers. - -Sources: [Kimi K3](https://openrouter.ai/moonshotai/kimi-k3), -[Muse Spark 1.1](https://openrouter.ai/meta/muse-spark-1.1). +# Models and native surfaces + +`@onkernel/cua-ai` accepts **any model pi-ai carries**, and any model id its +registry has not caught up with yet. There is no allowlist: a model id you pass +resolves, and the provider decides whether it exists. Run +`listCuaModels(provider?)` for the live catalog. + +Two small tables in [`src/models.ts`](https://github.com/kernel/cua/blob/main/packages/ai/src/models.ts) +describe what is *different* about particular models. Neither decides whether a +model may run. + +## Native surfaces + +`CUA_NATIVE_SURFACES` records which models have a provider-native computer or +browser tool, so the tool menu can offer it. Entries match either an exact id or +a `family` — the family root plus suffixes made of hyphen-separated numeric +segments, covering revisions and dated snapshots such as `claude-opus-4-7` or +`gpt-5.5-2026-04-23`. Named sibling variants like `gpt-5.4-mini` are distinct +models and need their own entry. Each cites first-party documentation. + +| provider | models | surfaces | +| --- | --- | --- | +| `anthropic` | `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` families | computer, browser | +| `anthropic` | `claude-fable-5` family | computer | +| `openai` | `gpt-5.6-sol`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5` | computer | +| `google` | `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite` | browser | + +Anthropic's entries live in `providers/anthropic/capabilities.ts`, which is +version-gated separately; `cuaNativeSurfaces(model)` reads both sources. + +A model with no native surface is not restricted — it drives a Kernel browser +with CUA's own CDP tools, which is the default for every provider. + +## Quirks + +`CUA_MODEL_QUIRKS` records request-shape limits. Anything absent gets the +permissive default and is allowed to try; the provider's own error is the +feedback. Every entry exists because of a documented limit or an observed +failure, and carries its reason inline. + +| provider | models | limit | +| --- | --- | --- | +| `google` | all | rejects `browser_wait_for`'s schema shape; the Gemini API accepts a subset of JSON Schema for function declarations | +| `moonshotai` | `kimi-k3` | rejects the request once `browser_act`'s schema is attached; serializes state mutations | +| `openrouter` | `moonshotai/kimi-k3` | the same Kimi limit, reached through OpenRouter | +| `openrouter` | `meta/muse-spark-1.1` | serializes state mutations | +| `xai` | all | serializes state mutations | + +`cuaModelCapabilities(model)` applies provider-wide quirks first, then +model-specific ones. `cuaModelQuirks(model)` returns the entries that applied, +for diagnostics and menu hints. + +## Model ids pi-ai does not carry + +A ref whose id is missing from the registry is synthesized from the sibling +sharing the longest id prefix, preferring the latest such sibling. Providers +migrate transports mid-generation — xAI carries `grok-4.3` on chat completions +and `grok-4.5` on Responses — so a new id follows its nearest, newest relative. +This is what lets a model work the day the provider ships it rather than when +models.dev catches up. + +Only an unqualified ref or a provider pi-ai does not carry is refused. + +## Keeping these tables current + +There is no periodic audit to run. The allowlist that once needed one is gone, +so maintenance is reactive — four cases, in rough order of how often they come up. + +**A provider released a model.** Nothing to do. If pi-ai's registry carries it, +it resolves; if not, it is synthesized from its nearest sibling. Neither needs a +change here. + +**The catalog looks stale.** Bump `@earendil-works/pi-ai`. Its registry is +generated from models.dev, so a newer pi-ai is how names, context windows, and +pricing get refreshed. Note that cua does not read pi's `models.json`: that is a +pi-coding-agent config file, and cua builds its `Models` collection from pi-ai +directly. A provider pi-ai does not ship is not selectable without registering +it in `src/providers.ts` — a deliberate decision, since the repo has removed +four such providers rather than carry them unused. + +**A provider shipped or changed a native tool.** This is real adapter work, not +a table edit. Probe what the model actually emits: + +```bash +npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --limit 3 +``` + +Update that provider's adapter under `src/providers/` to execute the actions the +probe returns, then add or adjust the `CUA_NATIVE_SURFACES` entry, citing the +provider's documentation. Anthropic's computer tool version and its +`computer-use-*` beta header are chosen by pi-ai per model, so a new dated +version there usually means bumping pi-ai rather than editing this package. + +**A model rejects a tool CUA sends.** Add a `CUA_MODEL_QUIRKS` entry with the +observed error as its `reason`, scoped as narrowly as the evidence supports: a +single model id over a family, a family over a whole provider. Remove a quirk +when the provider lifts the limit — a stale quirk silently denies a model a tool +it now accepts, which is harder to notice than the reverse. diff --git a/.agents/skills/update-models/reference/native-action-probe.ts b/packages/ai/scripts/native-action-probe.ts similarity index 97% rename from .agents/skills/update-models/reference/native-action-probe.ts rename to packages/ai/scripts/native-action-probe.ts index c3b9434..343d428 100644 --- a/.agents/skills/update-models/reference/native-action-probe.ts +++ b/packages/ai/scripts/native-action-probe.ts @@ -71,9 +71,9 @@ function parseArgs(argv: string[]): Args { function usage(): never { console.log(`Usage: - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 + npx tsx packages/ai/scripts/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json + npx tsx packages/ai/scripts/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 + npx tsx packages/ai/scripts/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 `); process.exit(0); } diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts index 70f9138..fba7ee8 100644 --- a/packages/ai/src/api-keys.ts +++ b/packages/ai/src/api-keys.ts @@ -1,14 +1,16 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { parseCuaModelRef, providerForModel, type CuaModelRef, type CuaProvider } from "./models"; +import { parseCuaModelRef, providerForModel, type CuaModelRef } from "./models"; /** - * Environment variables accepted for each CUA provider. + * Environment variables for the providers CUA documents, in precedence order. * - * This mirrors pi-ai's approach: model lookup is pure, while auth is resolved - * when streaming. These helpers let callers share one readable convention for - * explicit `getApiKey` wiring (especially useful for `google` vs `gemini`). + * Every provider pi-ai carries is selectable, and pi resolves each one's own + * credential when streaming. This table exists only so callers and the CLI can + * name the variable to set up front; a provider absent from it is not + * unsupported, it just has no CUA-side preflight. pi-ai does not export its + * own env-var registry, or this would read from that. */ -const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { +const CUA_PROVIDER_API_KEY_ENV_VARS: Readonly> = { openai: ["OPENAI_API_KEY"], anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], @@ -17,20 +19,16 @@ const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { openrouter: ["OPENROUTER_API_KEY"], }; +/** Provider prefixes accepted as aliases for a pi-ai provider id. */ +const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; + /** * List the environment variables checked for a provider's API key, in - * precedence order. Accepts `"gemini"` as an alias for `"google"` and - * `"moonshot"` as an alias for `"moonshotai"`; returns an empty list for - * unknown providers. + * precedence order. Returns an empty list for a provider CUA does not document, + * whose credential pi resolves at request time instead. */ export function cuaApiKeyEnvVarsForProvider(provider: string): readonly string[] { - if (provider === "gemini") { - return CUA_PROVIDER_API_KEY_ENV_VARS.google; - } - if (provider === "moonshot") { - return CUA_PROVIDER_API_KEY_ENV_VARS.moonshotai; - } - return CUA_PROVIDER_API_KEY_ENV_VARS[provider as keyof typeof CUA_PROVIDER_API_KEY_ENV_VARS] ?? []; + return CUA_PROVIDER_API_KEY_ENV_VARS[PROVIDER_ALIASES[provider] ?? provider] ?? []; } /** Read a provider's API key from the environment, or return undefined when unset. */ @@ -42,7 +40,12 @@ export function getCuaEnvApiKey(provider: string): string | undefined { return undefined; } -/** Read a provider's API key from the environment, or throw naming the variables to set. */ +/** + * Read a provider's API key from the environment, or throw naming the variables + * to set. Throws for a provider CUA documents no variables for — callers that + * accept any pi-ai provider should use {@link cuaApiKeyEnvVarsForProvider} to + * decide whether a preflight is possible at all. + */ export function requireCuaEnvApiKey(provider: string): string { const apiKey = getCuaEnvApiKey(provider); if (apiKey) return apiKey; diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index e023085..a7feb6e 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -1,11 +1,15 @@ import type { Api, Model } from "@earendil-works/pi-ai"; -import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; +import { getBuiltinModel, getBuiltinModels, getBuiltinProviders } from "@earendil-works/pi-ai/providers/all"; +import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer } from "./providers/anthropic/capabilities"; -/** Providers with curated computer-use model support. */ -export type CuaProvider = "openai" | "anthropic" | "google" | "xai" | "moonshotai" | "openrouter"; +/** A pi-ai provider id. Any provider pi-ai carries can be selected. */ +export type CuaProvider = string; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ -export type CuaModelRef = `${CuaProvider}:${string}`; +export type CuaModelRef = `${string}:${string}`; + +/** A provider-native tool surface CUA can offer for a model. */ +export type CuaNativeSurface = "computer" | "browser"; /** One entry returned by {@link listCuaModels}. */ export interface CuaModelInfo { @@ -16,13 +20,14 @@ export interface CuaModelInfo { model: string; /** Human-readable model name. */ name: string; + /** Provider-native tool surfaces available for this model, if any. */ + nativeSurfaces: readonly CuaNativeSurface[]; + /** Whether the model accepts image input, i.e. whether screenshot-based tools are usable. */ + vision: boolean; } -/** All providers this package curates computer-use models for. */ -export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; - /** - * How a {@link CuaModelAnnotation} matches model ids. + * How a model-id table entry matches. * * - `exact`: `id === match.id` * - `family`: `id === match.family`, or `match.family` plus hyphen-separated @@ -41,84 +46,96 @@ export interface CuaModelCapabilities { readonly serializesStateMutations: boolean; } -/** One CUA-support annotation: a model-id match plus the official source documenting support. */ -export interface CuaModelAnnotation { +/** + * A model or provider whose request handling differs from the permissive + * default, with the evidence for it. Entries exist to prevent a request the + * provider would reject — never to express a preference. + */ +export interface CuaModelQuirk { + readonly provider: CuaProvider; + /** Omit to apply the quirk to every model from the provider. */ + readonly match?: CuaModelMatch; + readonly capabilities: Partial; + /** Why this quirk exists: the documented limit or the observed failure. */ + readonly reason: string; +} + +/** + * Models with a provider-native computer or browser tool, and the first-party + * documentation for it. This table answers "can CUA offer a native tool for + * this model", not "may this model run" — every model pi-ai carries runs, with + * CUA's own CDP browser tools. + * + * Anthropic is absent deliberately: its native surfaces are version-gated in + * `providers/anthropic/capabilities.ts`, which {@link cuaNativeSurfaces} reads. + */ +export const CUA_NATIVE_SURFACES: readonly { + readonly provider: CuaProvider; readonly match: CuaModelMatch; - /** URL of the provider documentation establishing computer-use support. */ + readonly surfaces: readonly CuaNativeSurface[]; readonly source: string; - /** Optional tool-catalog capabilities that describe which CUA schemas and state mutations the model supports. */ - readonly capabilities?: CuaModelCapabilities; -} +}[] = [ + { provider: "openai", match: { kind: "exact", id: "gpt-5.6-sol" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.6-sol" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.4" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.4" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.4-mini" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.4-mini" }, + { provider: "openai", match: { kind: "family", family: "gpt-5.5" }, surfaces: ["computer"], source: "https://developers.openai.com/api/docs/models/gpt-5.5" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.6-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, + { provider: "google", match: { kind: "exact", id: "gemini-3.5-flash-lite" }, surfaces: ["browser"], source: "https://ai.google.dev/gemini-api/docs/computer-use" }, +]; -// Muse Spark accepts the full CUA schema set; OpenRouter's provider-level -// defaults are conservative because the proxy fronts many model families. -const MUSE_SPARK_CAPABILITIES: CuaModelCapabilities = Object.freeze({ - acceptsComplexSchemas: true, - acceptsLargeSchemas: true, - serializesStateMutations: true, -}); +/** + * Known request-shape limits. Anything absent from this table gets the + * permissive default and is allowed to try; a provider-side error is the + * feedback. Every entry below is a limit we have documentation for or have + * observed against the live API. + */ +export const CUA_MODEL_QUIRKS: readonly CuaModelQuirk[] = [ + { + provider: "google", + capabilities: { acceptsComplexSchemas: false, acceptsLargeSchemas: false }, + reason: "The Gemini API accepts a subset of JSON Schema for function declarations and rejects browser_wait_for's shape.", + }, + { + provider: "moonshotai", + match: { kind: "exact", id: "kimi-k3" }, + capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, + reason: "Kimi K3 rejects the request outright once browser_act's schema is attached.", + }, + { + provider: "openrouter", + match: { kind: "exact", id: "moonshotai/kimi-k3" }, + capabilities: { acceptsLargeSchemas: false, serializesStateMutations: true }, + reason: "Same Kimi K3 limit, reached through OpenRouter.", + }, + { + provider: "openrouter", + match: { kind: "exact", id: "meta/muse-spark-1.1" }, + capabilities: { serializesStateMutations: true }, + reason: "Muse Spark's computer-use cookbook disables parallel tool calls.", + }, + { + provider: "xai", + capabilities: { serializesStateMutations: true }, + reason: "Grok's computer-use guidance disables parallel tool calls for state-mutating catalogs.", + }, +]; -const KIMI_K3_CAPABILITIES: CuaModelCapabilities = Object.freeze({ +const PERMISSIVE_CAPABILITIES: CuaModelCapabilities = Object.freeze({ acceptsComplexSchemas: true, - acceptsLargeSchemas: false, - serializesStateMutations: true, + acceptsLargeSchemas: true, + serializesStateMutations: false, }); -/** - * Per-provider computer-use support annotations. - * - * pi-ai's model registry is generated from models.dev (see - * node_modules/@earendil-works/pi-ai/scripts/generate-models.ts) and lists every - * model a provider offers. Only some of those models support computer-use, so - * this table layers per-provider CUA-support annotations on top of the - * registry. Each entry cites the official source documenting CUA support. - * - * To verify support and add new entries, follow the `update-models` skill at - * .agents/skills/update-models/SKILL.md. - */ -export const CUA_MODEL_ANNOTATIONS: Record = { - openai: [ - { match: { kind: "exact", id: "gpt-5.6-sol" }, source: "https://developers.openai.com/api/docs/models/gpt-5.6-sol" }, - { match: { kind: "family", family: "gpt-5.4" }, source: "https://developers.openai.com/api/docs/models/gpt-5.4" }, - { match: { kind: "family", family: "gpt-5.4-mini" }, source: "https://developers.openai.com/api/docs/models/gpt-5.4-mini" }, - { match: { kind: "family", family: "gpt-5.5" }, source: "https://developers.openai.com/api/docs/models/gpt-5.5" }, - ], - anthropic: [ - { match: { kind: "family", family: "claude-3-7-sonnet" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-opus-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-opus-5" }, source: "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool" }, - { match: { kind: "family", family: "claude-sonnet-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-sonnet-5" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-haiku-4" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - { match: { kind: "family", family: "claude-fable-5" }, source: "https://docs.anthropic.com/en/docs/build-with-claude/computer-use" }, - ], - google: [ - { match: { kind: "exact", id: "gemini-3.6-flash" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - { match: { kind: "exact", id: "gemini-3.5-flash-lite" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - { match: { kind: "exact", id: "gemini-3.5-flash" }, source: "https://ai.google.dev/gemini-api/docs/computer-use" }, - ], - xai: [ - { match: { kind: "exact", id: "grok-4.5" }, source: "https://docs.x.ai/developers/grok-4-5" }, - ], - // Kimi computer use is custom-function-tool support over Moonshot's - // OpenAI-compatible API, not a provider-native computer tool. K3 ships - // native vision plus screenshot-grounded agentic tool use. - moonshotai: [ - { match: { kind: "exact", id: "kimi-k3" }, source: "https://www.kimi.com/blog/kimi-k3", capabilities: KIMI_K3_CAPABILITIES }, - ], - openrouter: [ - { match: { kind: "exact", id: "moonshotai/kimi-k3" }, source: "https://openrouter.ai/moonshotai/kimi-k3", capabilities: KIMI_K3_CAPABILITIES }, - { match: { kind: "exact", id: "meta/muse-spark-1.1" }, source: "https://openrouter.ai/meta/muse-spark-1.1", capabilities: MUSE_SPARK_CAPABILITIES }, - ], -}; +/** Provider prefixes accepted as aliases for a pi-ai provider id. */ +const PROVIDER_ALIASES: Readonly> = { gemini: "google", moonshot: "moonshotai" }; /** * Split a provider-qualified ref like `"openai:gpt-5.6-sol"` into its parts. * - * `"gemini:"` is accepted as an alias for the canonical `"google:"` prefix - * and normalizes to provider `"google"`; `"moonshot:"` likewise normalizes - * to `"moonshotai"`. Throws when the ref is unqualified or names an - * unsupported provider. + * `"gemini:"` is accepted as an alias for the canonical `"google:"` prefix and + * `"moonshot:"` for `"moonshotai"`. Throws when the ref is unqualified or names + * a provider pi-ai does not carry. */ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: string } { const idx = ref.indexOf(":"); @@ -126,10 +143,10 @@ export function parseCuaModelRef(ref: string): { provider: CuaProvider; model: s throw new Error(`CUA model ref must be provider-qualified as ":"; got "${ref}"`); } const prefix = ref.slice(0, idx); - const provider = prefix === "gemini" ? "google" : prefix === "moonshot" ? "moonshotai" : prefix; + const provider = PROVIDER_ALIASES[prefix] ?? prefix; const model = ref.slice(idx + 1); - if (!isCuaProvider(provider)) { - throw new Error(`unsupported CUA provider "${prefix}" (expected one of: ${CUA_PROVIDERS.join(", ")})`); + if (!cuaProviders().includes(provider)) { + throw new Error(`unknown provider "${prefix}" (pi-ai carries: ${cuaProviders().join(", ")})`); } return { provider, model }; } @@ -139,18 +156,21 @@ export function formatCuaModelRef(provider: CuaProvider, model: string): CuaMode return `${provider}:${model}` as CuaModelRef; } +/** Every provider id pi-ai carries. */ +export function cuaProviders(): readonly CuaProvider[] { + return getBuiltinProviders(); +} + /** - * List the computer-use-capable models this package curates, optionally - * filtered to one provider. Merges pi-ai's registry with local overrides and - * keeps only models annotated in {@link CUA_MODEL_ANNOTATIONS}. + * List the models pi-ai carries, optionally filtered to one provider, each + * annotated with the provider-native surfaces CUA can offer for it. */ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { - const providers = provider ? [provider] : [...CUA_PROVIDERS]; + const providers = provider ? [PROVIDER_ALIASES[provider] ?? provider] : [...cuaProviders()]; const byRef = new Map(); for (const p of providers) { for (const model of getBuiltinModels(p as never) as Model[]) { - if (!supportsCuaProvider(p, model.id)) continue; const ref = formatCuaModelRef(p, model.id); if (byRef.has(ref)) continue; byRef.set(ref, { @@ -158,6 +178,8 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { provider: p, model: model.id, name: model.name, + nativeSurfaces: cuaNativeSurfaces(model), + vision: model.input.includes("image"), }); } } @@ -168,68 +190,101 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { /** * Resolve a {@link CuaModelRef} to a concrete pi-ai model. * - * Throws when the ref is unqualified, names an unsupported provider, or names - * a model without a CUA-support annotation. `"gemini:"` refs are accepted as - * an alias for `"google:"` (see {@link parseCuaModelRef}). + * A ref pi-ai's registry does not carry is synthesized from the provider's + * other models, so a model id works the day the provider ships it rather than + * when models.dev catches up. Throws only for an unqualified ref or a provider + * pi-ai does not carry. */ export function getCuaModel(ref: CuaModelRef): Model { const { provider, model: modelId } = parseCuaModelRef(ref); - if (!supportsCuaProvider(provider, modelId)) { - throw new Error(`unsupported CUA model "${ref}"`); - } const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; if (fromRegistry) return fromRegistry; - throw new Error(`CUA model "${ref}" is supported but not carried by pi-ai's registry`); + return synthesizeCuaModel(provider, modelId); } +/** + * Build a model entry for an id pi-ai's registry does not carry, using another + * model from the same provider for the transport, base URL, and compatibility + * fields it cannot know from the id alone. + * + * The template is the sibling sharing the longest id prefix, and the latest + * such sibling when several tie. Providers migrate transports mid-generation — + * xAI carries grok-4.3 on chat completions and grok-4.5 on Responses — so a new + * id should follow its nearest, newest relative rather than whichever model + * happens to come first. + */ +function synthesizeCuaModel(provider: CuaProvider, modelId: string): Model { + const siblings = getBuiltinModels(provider as never) as Model[]; + if (siblings.length === 0) { + throw new Error(`provider "${provider}" carries no models to infer "${modelId}" from`); + } + let template = siblings[0]!; + let bestPrefix = -1; + siblings.forEach((sibling, index) => { + const prefix = sharedPrefixLength(sibling.id.toLowerCase(), modelId.toLowerCase()); + if (prefix >= bestPrefix) { + bestPrefix = prefix; + template = siblings[index]!; + } + }); + return { ...template, id: modelId, name: modelId }; +} + +function sharedPrefixLength(a: string, b: string): number { + let length = 0; + while (length < a.length && length < b.length && a[length] === b[length]) length += 1; + return length; +} -/** Return the {@link CuaProvider} for a concrete model, or throw when it is not a CUA provider. */ +/** Return the provider id for a concrete model. */ export function providerForModel(model: Model): CuaProvider { - if (!isCuaProvider(model.provider)) { - throw new Error(`unsupported CUA model provider "${model.provider}" (expected one of: ${CUA_PROVIDERS.join(", ")})`); - } return model.provider; } -/** Narrow an arbitrary string to {@link CuaProvider}. */ -export function isCuaProvider(value: string): value is CuaProvider { - return (CUA_PROVIDERS as readonly string[]).includes(value); +/** Provider-native tool surfaces available for a model, if any. */ +export function cuaNativeSurfaces(model: Model): readonly CuaNativeSurface[] { + if (model.provider === "anthropic") { + const surfaces: CuaNativeSurface[] = []; + if (supportsAnthropicNativeComputer(model.id)) surfaces.push("computer"); + if (supportsAnthropicNativeBrowser(model.id)) surfaces.push("browser"); + return surfaces; + } + for (const entry of CUA_NATIVE_SURFACES) { + if (entry.provider === model.provider && matchesModelId(model.id, entry.match)) return entry.surfaces; + } + return []; +} + +/** + * Tool-catalog capabilities for a model: permissive unless a quirk says + * otherwise. Provider-wide quirks apply first, then model-specific ones. + */ +export function cuaModelCapabilities(model: Model): CuaModelCapabilities { + let capabilities = PERMISSIVE_CAPABILITIES; + for (const quirk of CUA_MODEL_QUIRKS) { + if (quirk.provider !== model.provider) continue; + if (quirk.match && !matchesModelId(model.id, quirk.match)) continue; + capabilities = { ...capabilities, ...quirk.capabilities }; + } + return capabilities; } -function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean { - return findCuaAnnotation(provider, modelId) !== undefined; +/** Find the quirks that apply to a model, for diagnostics and menu hints. */ +export function cuaModelQuirks(model: Model): readonly CuaModelQuirk[] { + return CUA_MODEL_QUIRKS.filter( + (quirk) => quirk.provider === model.provider && (!quirk.match || matchesModelId(model.id, quirk.match)), + ); } -/** Return tool-catalog capabilities for a model, using annotation or provider defaults. */ -export function cuaModelCapabilities(model: Model): CuaModelCapabilities { - const annotation = isCuaProvider(model.provider) ? findCuaAnnotation(model.provider, model.id) : undefined; - if (annotation?.capabilities) return annotation.capabilities; - const acceptsComplexSchemas = ["openai", "anthropic", "xai", "moonshotai"].includes(model.provider); - return { - acceptsComplexSchemas, - acceptsLargeSchemas: acceptsComplexSchemas && model.provider !== "moonshotai", - serializesStateMutations: ["xai", "moonshotai"].includes(model.provider), - }; -} - -/** Find the CUA-support annotation covering a model id, if any. */ -export function findCuaAnnotation(provider: CuaProvider, modelId: string): CuaModelAnnotation | undefined { +function matchesModelId(modelId: string, match: CuaModelMatch): boolean { const id = modelId.toLowerCase(); - for (const annotation of CUA_MODEL_ANNOTATIONS[provider]) { - if (annotation.match.kind === "exact") { - if (id === annotation.match.id.toLowerCase()) return annotation; - } else if (isCuaFamilyMatch(id, annotation.match.family.toLowerCase())) { - return annotation; - } - } - return undefined; + return match.kind === "exact" ? id === match.id.toLowerCase() : isCuaFamilyMatch(id, match.family.toLowerCase()); } -// A family annotation covers its root id plus suffixes made of -// hyphen-separated numeric segments: revisions like "claude-opus-4-7" and -// dated snapshots like "gpt-5.5-2026-04-23" or "claude-3-7-sonnet-20250219". -// Named sibling variants ("gpt-5.4-mini") may not support computer use and -// must be annotated explicitly. +// A family entry covers its root id plus suffixes made of hyphen-separated +// numeric segments: revisions like "claude-opus-4-7" and dated snapshots like +// "gpt-5.5-2026-04-23". Named sibling variants ("gpt-5.4-mini") are distinct +// models and need their own entry. function isCuaFamilyMatch(id: string, family: string): boolean { if (id === family) return true; if (!id.startsWith(`${family}-`)) return false; @@ -240,6 +295,6 @@ function isCuaFamilyMatch(id: string, family: string): boolean { } function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { - if (a.provider !== b.provider) return CUA_PROVIDERS.indexOf(a.provider) - CUA_PROVIDERS.indexOf(b.provider); + if (a.provider !== b.provider) return a.provider.localeCompare(b.provider); return a.model.localeCompare(b.model); } diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index 206dea2..9a1bd47 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -1,7 +1,7 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; import type { CuaAction } from "./actions/index"; import type { CuaModelRef } from "./models"; -import { cuaModelCapabilities, getCuaModel, providerForModel } from "./models"; +import { cuaModelCapabilities, getCuaModel } from "./models"; import { anthropicAdaptiveThinkingOnPayload } from "./providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, @@ -417,7 +417,6 @@ function validateToolsetCompatibility(model: Model, entries: readonly CuaCa if (nativeProviderKinds.size > 1) { throw new Error(`selected tools contribute incompatible native provider transports: ${[...nativeProviderKinds].join(", ")}`); } - providerForModel(model); const requiresApis = new Set(entries.flatMap((entry) => bindingRequiresApi(entry.providerBinding))); if (requiresApis.size > 1) { diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index 41a55a9..7529fa2 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from "vitest"; import { - CUA_MODEL_ANNOTATIONS, - CUA_PROVIDERS, + CUA_MODEL_QUIRKS, + CUA_NATIVE_SURFACES, type CuaModelRef, - findCuaAnnotation, + cuaModelCapabilities, + cuaNativeSurfaces, + cuaProviders, formatCuaModelRef, getCuaModel, listCuaModels, @@ -16,16 +18,16 @@ describe("CUA model refs", () => { expect(formatCuaModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); }); - it("rejects unqualified and unsupported refs", () => { + it("rejects unqualified refs and unknown providers, but not unknown models", () => { expect(() => getCuaModel("gpt-5.5" as never)).toThrow(/provider-qualified/); - expect(() => getCuaModel("bogus:model" as never)).toThrow(/unsupported CUA provider/); - expect(() => getCuaModel("openai:gpt-3.5" as never)).toThrow(/unsupported CUA model/); + expect(() => getCuaModel("bogus:model" as never)).toThrow(/unknown provider/); + // A model id pi-ai's registry has not caught up with still resolves: the + // provider decides whether it exists, not a table in this package. + expect(getCuaModel("openai:gpt-3.5" as never).id).toBe("gpt-3.5"); }); - it("names the valid providers in the unsupported-provider error", () => { - expect(() => parseCuaModelRef("bogus:model")).toThrow( - 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, xai, moonshotai, openrouter)', - ); + it("names pi-ai's providers in the unknown-provider error", () => { + expect(() => parseCuaModelRef("bogus:model")).toThrow(/unknown provider "bogus" \(pi-ai carries: /); }); it("accepts gemini: as an alias for google:", () => { @@ -120,85 +122,118 @@ describe("CUA model refs", () => { expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); }); - it("rejects supported model IDs that pi-ai does not carry", () => { - // Dated snapshots match the family annotation but pi-ai's registry - // (generated from models.dev) only carries family roots. - expect(() => getCuaModel("openai:gpt-5.5-2026-04-23")).toThrow( - /not carried by pi-ai's registry/, - ); + it("synthesizes models pi-ai's registry does not carry", () => { + // pi-ai's registry (generated from models.dev) carries family roots, not + // dated snapshots, and lags a provider's newest ids. Both still resolve, + // inheriting the transport and base URL from the provider's other models. + const snapshot = getCuaModel("openai:gpt-5.5-2026-04-23"); + expect(snapshot.id).toBe("gpt-5.5-2026-04-23"); + expect(snapshot.provider).toBe("openai"); + expect(snapshot.api).toBe(getCuaModel("openai:gpt-5.5").api); + expect(snapshot.baseUrl).toBe(getCuaModel("openai:gpt-5.5").baseUrl); + + // The motivating case: a model the provider has shipped and models.dev + // has not picked up yet. + expect(getCuaModel("xai:grok-4.6").id).toBe("grok-4.6"); }); -}); -describe("CUA support annotations", () => { - it("covers every provider", () => { - for (const provider of CUA_PROVIDERS) { - expect(CUA_MODEL_ANNOTATIONS[provider].length).toBeGreaterThan(0); - } + it("synthesizes from the nearest, newest sibling", () => { + // xAI carries grok-4.3 on chat completions and grok-4.5 on Responses, so + // picking the wrong sibling would send a new Grok to the wrong transport. + expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); + expect(getCuaModel("xai:grok-4.6").api).toBe("openai-responses"); + expect(getCuaModel("xai:grok-4.6").baseUrl).toBe(getCuaModel("xai:grok-4.5").baseUrl); + expect(getCuaModel("anthropic:claude-opus-6").api).toBe("anthropic-messages"); }); +}); - it("cites an official source for every annotation", () => { - for (const provider of CUA_PROVIDERS) { - for (const annotation of CUA_MODEL_ANNOTATIONS[provider]) { - expect(annotation.source).toMatch(/^https?:\/\//); - } +describe("native surfaces", () => { + it("cites first-party documentation for every entry", () => { + for (const entry of CUA_NATIVE_SURFACES) { + expect(entry.source).toMatch(/^https?:\/\//); + expect(entry.surfaces.length).toBeGreaterThan(0); } }); it("matches family roots, dated snapshots, and numeric revisions", () => { - expect(findCuaAnnotation("openai", "gpt-5.5")?.match).toEqual({ kind: "family", family: "gpt-5.5" }); - expect(findCuaAnnotation("openai", "gpt-5.5-2026-04-23")?.match).toEqual({ kind: "family", family: "gpt-5.5" }); - expect(findCuaAnnotation("openai", "gpt-5.4-mini")?.match).toEqual({ kind: "family", family: "gpt-5.4-mini" }); - expect(findCuaAnnotation("openai", "gpt-5.4-mini-2026-03-17")?.match).toEqual({ kind: "family", family: "gpt-5.4-mini" }); - expect(findCuaAnnotation("anthropic", "claude-opus-4-7")).toBeDefined(); - expect(findCuaAnnotation("anthropic", "claude-opus-5")?.match).toEqual({ kind: "family", family: "claude-opus-5" }); - expect(findCuaAnnotation("anthropic", "claude-opus-5-20260724")?.match).toEqual({ kind: "family", family: "claude-opus-5" }); - expect(findCuaAnnotation("anthropic", "claude-3-7-sonnet-20250219")).toBeDefined(); - }); - - it("does not match adjacent families", () => { - expect(findCuaAnnotation("openai", "gpt-5.55-foo")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.6")).toBeUndefined(); - expect(findCuaAnnotation("anthropic", "claude-3-5-sonnet")).toBeUndefined(); - }); - - it("does not match named sibling variants of a family", () => { - expect(findCuaAnnotation("openai", "gpt-5.4-nano")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.4-pro")).toBeUndefined(); - expect(findCuaAnnotation("openai", "gpt-5.5-pro")).toBeUndefined(); - const openaiModels = listCuaModels("openai").map((model) => model.model); - expect(openaiModels).not.toContain("gpt-5.4-nano"); - expect(openaiModels).not.toContain("gpt-5.4-pro"); - expect(openaiModels).toContain("gpt-5.5"); - }); - - it("matches exact-id annotations", () => { - expect(findCuaAnnotation("openai", "gpt-5.6-sol")?.match).toEqual({ kind: "exact", id: "gpt-5.6-sol" }); - expect(findCuaAnnotation("openai", "gpt-5.6-sol-20260728")).toBeUndefined(); - expect(findCuaAnnotation("google", "gemini-3.6-flash")).toBeDefined(); - expect(findCuaAnnotation("openrouter", "meta/muse-spark-1.1")).toBeDefined(); - expect(findCuaAnnotation("xai", "grok-4.5")).toBeDefined(); - expect(findCuaAnnotation("xai", "grok-4.5-latest")).toBeUndefined(); - expect(findCuaAnnotation("xai", "grok-4.3")).toBeUndefined(); - expect(findCuaAnnotation("moonshotai", "kimi-k3")).toBeDefined(); - expect(findCuaAnnotation("moonshotai", "kimi-k2.5")).toBeUndefined(); - expect(findCuaAnnotation("moonshotai", "kimi-latest")).toBeUndefined(); - expect(findCuaAnnotation("google", "gemini-3.5-flash-lite")).toBeDefined(); - }); - - it("advertises only Google's current documented computer-use models", () => { - expect(listCuaModels("google").map((model) => model.model)).toEqual([ - "gemini-3.5-flash", - "gemini-3.5-flash-lite", - "gemini-3.6-flash", - ]); - for (const retired of [ - "gemini-2.5-computer-use-preview-10-2025", - "gemini-3-flash-preview", - "gemini-3.1-flash-lite", - "gemini-3-pro-preview", - ]) { - expect(findCuaAnnotation("google", retired)).toBeUndefined(); - expect(() => getCuaModel(`google:${retired}` as CuaModelRef)).toThrow(/unsupported CUA model/); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.5-2026-04-23"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-mini"))).toEqual(["computer"]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5"))).toEqual(["computer", "browser"]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-opus-5-20260724"))).toEqual(["computer", "browser"]); + }); + + it("does not match adjacent families or named sibling variants", () => { + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-nano"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("openai:gpt-5.4-pro"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("anthropic:claude-3-5-sonnet"))).toEqual([]); + }); + + it("reports no native surface for models that have none, without refusing them", () => { + expect(cuaNativeSurfaces(getCuaModel("moonshotai:kimi-k3"))).toEqual([]); + expect(cuaNativeSurfaces(getCuaModel("xai:grok-4.5"))).toEqual([]); + // A model with no native surface still resolves and runs on CUA's own tools. + expect(getCuaModel("xai:grok-4.5").provider).toBe("xai"); + }); + + it("surfaces the flag on catalog listings", () => { + const google = listCuaModels("google"); + const flash = google.find((model) => model.model === "gemini-3.6-flash"); + expect(flash?.nativeSurfaces).toEqual(["browser"]); + expect(flash?.vision).toBe(true); + expect(google.some((model) => model.nativeSurfaces.length === 0)).toBe(true); + }); +}); + +describe("model quirks", () => { + it("explains why every quirk exists", () => { + for (const quirk of CUA_MODEL_QUIRKS) { + expect(quirk.reason.length).toBeGreaterThan(20); + expect(Object.keys(quirk.capabilities).length).toBeGreaterThan(0); } }); + + it("defaults to permissive for a model with no quirk", () => { + expect(cuaModelCapabilities(getCuaModel("openai:gpt-5.6-sol"))).toEqual({ + acceptsComplexSchemas: true, + acceptsLargeSchemas: true, + serializesStateMutations: false, + }); + // Including a model pi-ai's registry does not carry. + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).acceptsComplexSchemas).toBe(true); + }); + + it("keeps the limits we have evidence for", () => { + // Observed live: the Gemini API rejects browser_wait_for's schema shape. + expect(cuaModelCapabilities(getCuaModel("google:gemini-3.6-flash")).acceptsComplexSchemas).toBe(false); + // Observed live: Kimi K3 rejects the request once browser_act is attached. + expect(cuaModelCapabilities(getCuaModel("moonshotai:kimi-k3")).acceptsLargeSchemas).toBe(false); + expect(cuaModelCapabilities(getCuaModel("openrouter:moonshotai/kimi-k3")).acceptsLargeSchemas).toBe(false); + // Muse Spark accepts the large schema but serializes state mutations. + const muse = cuaModelCapabilities(getCuaModel("openrouter:meta/muse-spark-1.1")); + expect(muse.acceptsLargeSchemas).toBe(true); + expect(muse.serializesStateMutations).toBe(true); + }); + + it("applies a provider-wide quirk to every model from that provider", () => { + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.5")).serializesStateMutations).toBe(true); + expect(cuaModelCapabilities(getCuaModel("xai:grok-4.6")).serializesStateMutations).toBe(true); + }); +}); + +describe("catalog passthrough", () => { + it("exposes every provider pi-ai carries", () => { + expect(cuaProviders().length).toBeGreaterThan(20); + expect(cuaProviders()).toContain("openai"); + expect(cuaProviders()).toContain("groq"); + expect(cuaProviders()).toContain("zai"); + }); + + it("lists models no CUA table mentions", () => { + const all = listCuaModels(); + expect(all.length).toBeGreaterThan(100); + // Previously refused for want of a table entry. + expect(all.some((model) => model.ref === "xai:grok-4.3")).toBe(true); + expect(all.some((model) => model.provider === "groq")).toBe(true); + }); }); diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 68ba217..cb4559d 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## 0.13.0 - 2026-08-14 + +- `cua models` lists every model pi-ai carries, not a curated subset, and `-p` + accepts any provider it carries. +- `-m` accepts any model id. A bare id that several providers carry now resolves + to the first-party provider rather than erroring, since gateways resell the + same ids; pass a qualified `provider:model` ref to reach a specific one. +- The API-key preflight now runs only for providers CUA documents variable + names for. Any other pi-ai provider is still selectable; pi resolves its + credential when it streams, and failing up front would refuse a model that + works. +- The default interaction toolset is chosen from the model rather than its + provider: a model with a native browser surface gets it, and everything else + gets CUA's CDP tools, with `browser_act` included only where the model accepts + its schema. + ## 0.12.0 - 2026-08-13 - `-m meta:muse-spark-1.1` is removed; use `-m openrouter:meta/muse-spark-1.1`. diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index 5193c0a..0f99c03 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -7,6 +7,7 @@ import { type Skill, } from "@onkernel/cua-agent"; import { + cuaApiKeyEnvVarsForProvider, type CuaModelRef, parseCuaModelRef, requireCuaEnvApiKey, @@ -48,7 +49,7 @@ import { import { type ContextFile, discoverCuaSkills } from "./harness-skills"; import { runPrint } from "./print"; -const MODELS_HELP = `cua models — list supported -m/--model values +const MODELS_HELP = `cua models — list selectable -m/--model values Usage: cua models @@ -57,7 +58,7 @@ Usage: cua models --json Options: - -p, --provider Filter by provider: openai | anthropic | google | gemini | xai | moonshotai | openrouter + -p, --provider Filter by provider id (any pi-ai provider; gemini/moonshot are aliases) --json Output JSON -h, --help Show this help `; @@ -124,14 +125,16 @@ function formatModelsTable(models: ReturnType): stri provider: entry.provider, model: entry.model, default: entry.ref === DEFAULT_CUA_MODEL_REF ? "yes" : "", + native: entry.nativeSurfaces.join(","), name: entry.name, })); - const headers = { ref: "REF", provider: "PROVIDER", model: "MODEL", default: "DEFAULT", name: "NAME" }; + const headers = { ref: "REF", provider: "PROVIDER", model: "MODEL", default: "DEFAULT", native: "NATIVE", name: "NAME" }; const widths = { ref: columnWidth(headers.ref, rows.map((r) => r.ref)), provider: columnWidth(headers.provider, rows.map((r) => r.provider)), model: columnWidth(headers.model, rows.map((r) => r.model)), default: columnWidth(headers.default, rows.map((r) => r.default)), + native: columnWidth(headers.native, rows.map((r) => r.native)), name: columnWidth(headers.name, rows.map((r) => r.name)), }; const lines = [ @@ -140,6 +143,7 @@ function formatModelsTable(models: ReturnType): stri headers.provider.padEnd(widths.provider), headers.model.padEnd(widths.model), headers.default.padEnd(widths.default), + headers.native.padEnd(widths.native), headers.name, ].join(" "), [ @@ -147,6 +151,7 @@ function formatModelsTable(models: ReturnType): stri "-".repeat(widths.provider), "-".repeat(widths.model), "-".repeat(widths.default), + "-".repeat(widths.native), "-".repeat(widths.name), ].join(" "), ]; @@ -157,6 +162,7 @@ function formatModelsTable(models: ReturnType): stri row.provider.padEnd(widths.provider), row.model.padEnd(widths.model), row.default.padEnd(widths.default), + row.native.padEnd(widths.native), row.name, ].join(" "), ); @@ -214,8 +220,10 @@ function resolveAuth(flags: HarnessCliFlags): ResolvedAuth { const { apiKey, baseUrl } = requireKernelApiKey(); const modelRef = resolveCuaModelRef(flags.model); const { provider } = parseCuaModelRef(modelRef); - // Throws naming the env vars the user must set (`requireCuaEnvApiKey`). - requireCuaEnvApiKey(provider); + // Preflight only where CUA documents the variable names; for any other + // pi-ai provider the credential is pi's to resolve when it streams, and + // failing here would refuse a model that works. + if (cuaApiKeyEnvVarsForProvider(provider).length > 0) requireCuaEnvApiKey(provider); return { kernelApiKey: apiKey, kernelBaseUrl: baseUrl, modelRef }; } diff --git a/packages/cli/src/harness-models.ts b/packages/cli/src/harness-models.ts index b691ca6..28ecec2 100644 --- a/packages/cli/src/harness-models.ts +++ b/packages/cli/src/harness-models.ts @@ -4,7 +4,7 @@ import { type CuaProvider, formatCuaModelRef, getCuaModel, - isCuaProvider, + cuaProviders, listCuaModels, parseCuaModelRef, } from "@onkernel/cua-ai"; @@ -13,9 +13,21 @@ import { export const DEFAULT_CUA_MODEL_REF: CuaModelRef = "openai:gpt-5.6-sol"; /** - * Resolve a model ref from CLI input. Accepts either a provider-qualified - * `provider:model` ref or a bare model id when it matches exactly one - * catalog entry. Throws when bare ids are ambiguous or unknown. + * Providers preferred when a bare model id is carried by several of them, in + * order. Gateways and aggregators resell the same ids as the provider that + * trained the model, so `-m gpt-5.5` should mean OpenAI's. + * + * This is a disambiguation preference for bare ids only. It never decides + * whether a model may run, and any provider is still reachable by passing a + * qualified `provider:model` ref. + */ +const BARE_ID_PROVIDER_PREFERENCE: readonly string[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; + +/** + * Resolve a model ref from CLI input. Accepts a provider-qualified + * `provider:model` ref, or a bare model id when exactly one provider carries it + * or one of the preferred providers does. Throws when a bare id is unknown, or + * ambiguous among providers none of which is preferred. */ export function resolveCuaModelRef(input: string | undefined): CuaModelRef { if (!input || !input.trim()) return DEFAULT_CUA_MODEL_REF; @@ -32,22 +44,24 @@ export function resolveCuaModelRef(input: string | undefined): CuaModelRef { throw new Error(`unknown model "${value}" (run \`cua models\` to list supported -m/--model values)`); } if (matches.length > 1) { + const preferred = BARE_ID_PROVIDER_PREFERENCE.map((provider) => matches.find((m) => m.provider === provider)).find(Boolean); + if (preferred) return preferred.ref; const refs = matches.map((m) => m.ref).join(", "); - throw new Error(`ambiguous model "${value}" (matches: ${refs}); pass a provider-qualified ref like "openai:${value}"`); + throw new Error(`ambiguous model "${value}" (matches: ${refs}); pass a provider-qualified ref`); } return matches[0]!.ref; } /** - * List supported models, optionally filtered to a provider. Accepts the - * canonical `"google"`/`"moonshotai"` ids or the CLI-friendly `"gemini"`/ - * `"moonshot"` aliases. + * List selectable models, optionally filtered to a provider. Accepts any + * provider pi-ai carries, plus the CLI-friendly `"gemini"`/`"moonshot"` + * aliases. */ export function listSupportedModels(provider?: string): CuaModelInfo[] { if (!provider) return listCuaModels(); const normalized = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; - if (!isCuaProvider(normalized)) { - throw new Error(`unknown provider "${provider}"`); + if (!cuaProviders().includes(normalized)) { + throw new Error(`unknown provider "${provider}" (pi-ai carries: ${cuaProviders().join(", ")})`); } - return listCuaModels(normalized as CuaProvider); + return listCuaModels(normalized); } diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index f9f71dd..a21a5de 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -12,6 +12,7 @@ import { type Api, cua, cuaModelCapabilities, + cuaNativeSurfaces, type CuaModelRef, getCuaModel, type Model, @@ -93,29 +94,27 @@ function structuredBrowserTools(): CuaCliTool[] { return [...cua.toolsets.browser(), cua.tools.browser.act()]; } -/** CLI policy is explicit application composition, not a CuaAgent default. */ +/** + * CLI interaction policy, asked of the model rather than switched on its + * provider: a model with a provider-native browser surface gets that surface, + * and everything else gets CUA's CDP browser tools, with `browser_act` included + * only where the model accepts its schema. + * + * OpenAI's native computer tool is deliberately not a default: it is a distinct + * interaction style callers opt into through `--tools` or the `/tools` picker. + */ export function defaultInteractionTools(model: CuaModelRef): CuaCliTool[] { - const { provider, model: modelId } = parseCuaModelRef(model); - switch (provider) { - case "openai": - return structuredBrowserTools(); - case "anthropic": - return cua.providers.anthropic.supports.browser(modelId) - ? [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })] - : structuredBrowserTools(); - case "google": - return cua.providers.google.toolsets.browser(); - case "xai": - return structuredBrowserTools(); - case "moonshotai": - case "openrouter": - // Kimi's API rejects the request outright once `browser_act`'s schema - // is attached. OpenRouter fronts several model families, so this is a - // per-model capability question rather than a per-provider one. - return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas - ? structuredBrowserTools() - : cua.toolsets.browser(); + const { provider } = parseCuaModelRef(model); + const resolved = getCuaModel(model); + if (cuaNativeSurfaces(resolved).includes("browser")) { + if (provider === "anthropic") { + return [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; + } + if (provider === "google") return cua.providers.google.toolsets.browser(); } + return cuaModelCapabilities(resolved).acceptsLargeSchemas + ? structuredBrowserTools() + : cua.toolsets.browser(); } function composeSystemPrompt(skills: Skill[], contextFiles: ContextFile[]): string { diff --git a/packages/cli/test/harness-models.test.ts b/packages/cli/test/harness-models.test.ts index 2f87f79..b27c4d5 100644 --- a/packages/cli/test/harness-models.test.ts +++ b/packages/cli/test/harness-models.test.ts @@ -30,14 +30,22 @@ describe("resolveCuaModelRef", () => { expect(() => resolveCuaModelRef("does-not-exist")).toThrow(/unknown model/); }); - it("filters custom provider catalogs", () => { - expect(listSupportedModels("xai").map((model) => model.ref)).toEqual(["xai:grok-4.5"]); - expect(listSupportedModels("moonshotai").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("moonshot").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:meta/muse-spark-1.1", "openrouter:moonshotai/kimi-k3"]); + it("filters to a provider's whole catalog", () => { + // No allowlist: every model the provider carries is listed, including the + // ones no CUA table mentions. + const xai = listSupportedModels("xai").map((model) => model.ref); + expect(xai).toContain("xai:grok-4.5"); + expect(xai).toContain("xai:grok-4.3"); + expect(listSupportedModels("moonshotai").map((model) => model.ref)).toContain("moonshotai:kimi-k3"); + expect(listSupportedModels("moonshot").map((model) => model.ref)).toContain("moonshotai:kimi-k3"); + expect(listSupportedModels("openrouter").map((model) => model.ref)).toContain("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("openrouter:moonshotai/kimi-k3")).toBe("openrouter:moonshotai/kimi-k3"); }); + it("rejects a provider pi-ai does not carry", () => { + expect(() => listSupportedModels("bogus")).toThrow(/unknown provider "bogus"/); + }); + it("treats 'gemini' as an alias for google when filtering", () => { const fromGemini = listSupportedModels("gemini"); const fromGoogle = listSupportedModels("google"); diff --git a/packages/cli/test/model-picker.test.ts b/packages/cli/test/model-picker.test.ts index 621a212..664c309 100644 --- a/packages/cli/test/model-picker.test.ts +++ b/packages/cli/test/model-picker.test.ts @@ -39,8 +39,10 @@ describe("filterModelsForPicker", () => { const google = catalog.filter((m) => m.provider === "google"); expect(google.length).toBeGreaterThan(0); const filtered = filterModelsForPicker(catalog, "google"); - expect(filtered.length).toBeGreaterThan(0); - expect(filtered.every((m) => m.provider === "google")).toBe(true); + // The full pi-ai catalog contains related providers (google-vertex), so + // the query narrows rather than isolating one provider. + expect(filtered.some((m) => m.provider === "google")).toBe(true); + expect(filtered.some((m) => m.provider === "anthropic")).toBe(false); }); it("returns an empty list when nothing matches", () => {