diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md index f09e343b..1cad9402 100644 --- a/.agents/skills/update-models/SKILL.md +++ b/.agents/skills/update-models/SKILL.md @@ -1,6 +1,6 @@ --- name: update-models -description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Moonshot, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. +description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. --- # Update Models @@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer- ## Quick Start -1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`. +1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. 2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them: ```bash eval "$(python3 - <<'PY' import pathlib, re, shlex text = pathlib.Path('~/AGENTS.md').expanduser().read_text() -for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']: +for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text) if m: print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}') @@ -61,7 +61,7 @@ Treat example repos as strongest when they are provider-owned or linked from off There are two enumeration layers: -- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, xAI's OpenAI-compatible `models.list()`, Tzafon's `Lightcone().models.list()` with known-model fallback, and Yutori OpenAPI/docs model enums) to discover what the current API key can access. +- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access. - CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use. When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("")`. @@ -70,7 +70,6 @@ When live discovery finds a new model with passing smoke tests, update `packages Meta: -- Discover with the OpenAI SDK against `https://api.meta.ai/v1` using `META_API_KEY`. - Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. - Pass condition: response output contains a `function_call` for one of the supplied browser actions. - Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`. @@ -124,35 +123,16 @@ Moonshot: - Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn. - Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes. -Tzafon: - -- Discover with `new Lightcone({ apiKey }).models.list()` from `@tzafon/lightcone` when available. -- If model listing is unavailable or returns an undocumented shape, record the error/shape and fall back to known smoke-test candidates such as `tzafon.northstar-cua-fast`. -- Smoke-test `responses.create` with explicit function tools matching the Tzafon template: `click`, `double_click`, `point_and_type`, `key`, `scroll`, `drag`, and `done`. -- Pass condition: response output contains `type: "function_call"` with one of those tool names, or a documented `computer_call` action if Lightcone switches to native computer-use output. -- Track coordinate convention separately from Gemini/Yutori: Tzafon uses a 0-999 grid. - -Yutori: - -- Discover model IDs from `https://docs.yutori.com/openapi.json` plus the Navigator docs. Current expected IDs include `n1.5-latest`, `n1.5-20260428`, `n1-latest`, and `n1-20260203`. -- Smoke-test the OpenAI-compatible `chat.completions` endpoint with `baseURL: "https://api.yutori.com/v1"` and `YUTORI_API_KEY`. -- Pass condition: response `choices[0].message.tool_calls[]` contains browser action function names such as `left_click`, `goto_url`, `type`, `scroll`, or `wait`. -- Track action-space differences between n1 and n1.5. n1 uses the legacy fixed tool set; n1.5 supports `tool_set`, `disable_tools`, expanded actions, and structured JSON output. -- Do not send duplicate browser action schemas when testing local CUA behavior. The local adapter registers matching AgentTools for execution but filters Yutori's built-in browser tool definitions out of the outbound API payload. - ## Native Action Discovery Run action probes when updating adapters or when docs/examples show drift: ```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider meta --models muse-spark-1.1 npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest ``` The probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare: @@ -192,14 +172,11 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W - Update the snapshot in `packages/ai/docs/supported-models.md` to match. - New provider-native action, response field, or tool version: - - Meta: update `packages/ai/src/providers/meta/index.ts` and `provider.ts`, including Responses threading and reasoning compatibility. - OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes. - Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package. - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed. - xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility. - Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`. - - Tzafon: update `packages/ai/src/providers/tzafon/index.ts` and `provider.ts`, including coordinate/action handling. - - Yutori: update `packages/ai/src/providers/yutori/actions.ts`, `index.ts`, and `provider.ts`, including payload filtering and coordinate/action handling. - Shared canonical action semantics go in `packages/ai/src/providers/common.ts`. - New provider or routing rule: diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md index d2e6c099..2c0bca42 100644 --- a/.agents/skills/update-models/reference/README.md +++ b/.agents/skills/update-models/reference/README.md @@ -11,11 +11,8 @@ These scripts support the `update-models` skill. Run them from the repository ro - `OPENAI_API_KEY` - `ANTHROPIC_API_KEY` - `GOOGLE_API_KEY` or `GEMINI_API_KEY` - - `META_API_KEY` - `XAI_API_KEY` - `MOONSHOT_API_KEY` - - `TZAFON_API_KEY` - - `YUTORI_API_KEY` The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools. @@ -32,8 +29,6 @@ Probe native action vocabularies for a specific provider/model: ```bash npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --out /tmp/tzafon-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --out /tmp/yutori-actions.json ``` Clone/update official examples and extract tool-handling evidence: diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts index 1005b43a..330b53bc 100644 --- a/.agents/skills/update-models/reference/audit-official-examples.ts +++ b/.agents/skills/update-models/reference/audit-official-examples.ts @@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path"; import { spawnSync } from "node:child_process"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface ExampleRepo { provider: Provider; @@ -51,14 +51,6 @@ const EXAMPLES: ExampleRepo[] = [ confidence: "provider-owned", patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"], }, - { - provider: "meta", - name: "meta-model-cookbook", - repo: "https://github.com/meta-models/meta-model-cookbook.git", - confidence: "provider-owned", - pathHint: "03_use_cases/13_macos_cua", - patterns: ["muse-spark-1.1", "function_call", "function_call_output", "previous_response_id", "reasoning.encrypted_content", "parallel_tool_calls"], - }, { provider: "xai", name: "xai-sdk-python", @@ -67,24 +59,14 @@ const EXAMPLES: ExampleRepo[] = [ pathHint: "examples", patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"], }, - { - provider: "yutori", - name: "kernel-cli-yutori-template", - repo: "https://github.com/kernel/cli.git", - confidence: "kernel-template", - pathHint: "pkg/templates/typescript/yutori", - patterns: ["YUTORI_API_KEY", "tool_calls", "left_click", "goto_url", "n1-latest", "api.yutori.com"], - }, ]; const ACTION_REGEXES: Record = { openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g], anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g], gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g], - meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g], xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], - yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g], }; function parseArgs(argv: string[]): Args { diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts index f243e6cd..84114001 100644 --- a/.agents/skills/update-models/reference/discover-models.ts +++ b/.agents/skills/update-models/reference/discover-models.ts @@ -1,12 +1,10 @@ #!/usr/bin/env tsx import { existsSync } from "node:fs"; import { readFile, writeFile } from "node:fs/promises"; -import { homedir } from "node:os"; import { join } from "node:path"; import process from "node:process"; -import { parse as parseToml } from "smol-toml"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "tzafon" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface Args { provider: Provider | "all"; @@ -39,16 +37,7 @@ interface ModelResult { cua?: Record; } -const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot", "tzafon", "yutori"]; -const TZAFON_KNOWN_MODELS = [ - "tzafon.northstar-cua-fast", -]; -const YUTORI_DOC_MODELS = [ - "n1.5-latest", - "n1.5-20260428", - "n1-latest", - "n1-20260203", -]; +const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "xai", "moonshot"]; const GEMINI_DOC_COMPUTER_USE_MODELS = [ "gemini-3.5-flash", "gemini-3-flash-preview", @@ -110,7 +99,7 @@ function usage(): never { npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4 Options: - --provider + --provider --models Smoke-test explicit models instead of inferred candidates. --candidate-limit Max inferred candidates per provider. Default: 20. --no-smoke Only list metadata. @@ -138,11 +127,8 @@ async function runProvider(provider: Provider, args: Args): Promise> { return { provider: "openai", metadata_source: "client.models.list()", models, candidates }; } -async function discoverMeta(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const apiKey = process.env.META_API_KEY; - const client = new OpenAI({ apiKey, baseURL: "https://api.meta.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: true, - })); - const candidates = explicitOrCandidates(args, models.map((model) => model.id)); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeMeta(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("meta", models); - return { provider: "meta", metadata_source: "Meta Model API models.list()", models, candidates }; -} - async function smokeMeta(client: any, model: string): Promise { try { const screenshot = await readFile(fixtureScreenshotPath()); @@ -633,304 +595,6 @@ async function discoverGemini(args: Args): Promise> { return { provider: "gemini", metadata_source: "client.models.list()", models, candidates }; } -async function discoverYutori(args: Args): Promise> { - const openapi = await fetchYutoriOpenApi(); - const ids = unique([ - ...extractYutoriModelIds(openapi.raw), - ...YUTORI_DOC_MODELS, - ]); - const models: ModelResult[] = ids.map((id) => ({ - id, - display_name: yutoriDisplayName(id), - created_at: yutoriCreatedAt(id), - raw: { source: "docs.yutori.com/openapi.json" }, - supports_generation: true, - model_docs: { - navigator_docs: "https://docs.yutori.com/reference/navigator", - n1_docs: "https://docs.yutori.com/reference/n1", - n15_docs: "https://docs.yutori.com/reference/n1-5", - openapi_ok: openapi.ok, - tool_set: id.startsWith("n1.5") ? "browser_tools_core-20260403" : "legacy_fixed", - disable_tools: id.startsWith("n1.5") ? "supported" : "not_supported", - coordinate_space: "1000x1000", - }, - })); - const candidates = explicitOrCandidates(args, ids); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { - id, - display_name: yutoriDisplayName(id), - supports_generation: true, - }; - model.computer_use = await smokeYutori(id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("yutori", models); - return { - provider: "yutori", - metadata_source: "https://docs.yutori.com/openapi.json + Navigator docs", - openapi: openapi.summary, - models, - candidates, - }; -} - -async function fetchYutoriOpenApi(): Promise<{ ok: boolean; raw: unknown; summary: Record }> { - const url = "https://docs.yutori.com/openapi.json"; - try { - const response = await fetch(url); - const raw = await response.json(); - return { - ok: response.ok, - raw, - summary: { - url, - ok: response.ok, - model_ids: extractYutoriModelIds(raw), - }, - }; - } catch (err) { - return { - ok: false, - raw: undefined, - summary: { url, ok: false, error: publicError(err) }, - }; - } -} - -function extractYutoriModelIds(raw: unknown): string[] { - const found: string[] = []; - const visit = (value: unknown): void => { - if (!value || typeof value !== "object") return; - if (Array.isArray(value)) { - for (const item of value) visit(item); - return; - } - const obj = value as Record; - if (Array.isArray(obj.enum)) { - for (const item of obj.enum) { - if (typeof item === "string" && /^n1(?:\.5)?-/.test(item)) found.push(item); - } - } - for (const item of Object.values(obj)) visit(item); - }; - visit(raw); - return unique(found); -} - -function yutoriDisplayName(id: string): string { - if (id.startsWith("n1.5")) return `Yutori Navigator ${id.replace("n1.5", "n1.5")}`; - return `Yutori Navigator ${id.replace("n1", "n1")}`; -} - -function yutoriCreatedAt(id: string): string | null { - const match = id.match(/-(\d{4})(\d{2})(\d{2})$/); - if (!match) return null; - return `${match[1]}-${match[2]}-${match[3]}T00:00:00.000Z`; -} - -async function yutoriApiKey(): Promise { - const env = process.env.YUTORI_API_KEY; - if (env && env.trim()) return env; - const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml"); - try { - const raw = parseToml(await readFile(cfgPath, "utf8")) as any; - const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; - const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; - if (typeof key === "string" && key.trim()) return key; - } catch { - // Fall through to a clear credential error. - } - throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)"); -} - -async function smokeYutori(model: string): Promise { - try { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ - apiKey: await yutoriApiKey(), - baseURL: "https://api.yutori.com/v1", - }); - const response = await client.chat.completions.create({ - model, - messages: [{ - role: "user", - content: [ - { type: "text", text: "Use the browser action tool to inspect or interact with this page. Do not answer in text." }, - { type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, - ], - }], - max_completion_tokens: 128, - temperature: 0.3, - }); - const choice = response.choices?.[0]; - const toolCalls: any[] = choice?.message?.tool_calls ?? []; - return { - status: toolCalls.length > 0 ? "pass" : "inconclusive", - tool_name: "browser_actions", - tool_version: model.startsWith("n1.5") ? "n1.5" : "n1", - beta_header: null, - observed_actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), - response_item_types: unique([ - choice?.message?.content ? "text" : undefined, - toolCalls.length ? "tool_calls" : undefined, - ].filter(Boolean) as string[]), - finish_reason: choice?.finish_reason ?? null, - accepts_image_tool_results: "assumed-from-docs", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "browser_actions" }); - } -} - -async function discoverTzafon(args: Args): Promise> { - const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone"); - const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY }); - const modelList = await fetchTzafonModels(client); - const ids = unique([ - ...modelList.ids, - ...TZAFON_KNOWN_MODELS, - ]); - const models: ModelResult[] = ids.map((id) => ({ - id, - display_name: tzafonDisplayName(id), - raw: modelList.rawById[id] ?? { source: "known-model-fallback" }, - supports_generation: true, - model_docs: { - docs: "https://docs.lightcone.ai", - responses_endpoint: "supported", - function_calling: "supported", - computer_use: "supported", - coordinate_space: "0-999", - model_list_endpoint: modelList.source, - }, - })); - const candidates = explicitOrCandidates(args, ids); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { - id, - display_name: tzafonDisplayName(id), - supports_generation: true, - }; - model.computer_use = await smokeTzafon(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("tzafon", models); - return { - provider: "tzafon", - metadata_source: modelList.source, - model_list_error: modelList.error ?? null, - models, - candidates, - }; -} - -async function fetchTzafonModels(client: any): Promise<{ - source: string; - ids: string[]; - rawById: Record; - raw?: unknown; - error?: string; -}> { - try { - const raw = await client.models.list(); - const entries = extractTzafonModelEntries(raw); - const rawById: Record = {}; - for (const entry of entries) rawById[entry.id] = entry.raw; - return { - source: "@tzafon/lightcone models.list()", - ids: entries.map((entry) => entry.id), - rawById, - raw, - }; - } catch (err) { - return { - source: "known-model-fallback (models.list unavailable)", - ids: [], - rawById: {}, - error: publicError(err), - }; - } -} - -function extractTzafonModelEntries(raw: unknown): Array<{ id: string; raw: unknown }> { - const entries: Array<{ id: string; raw: unknown }> = []; - const visit = (value: unknown): void => { - if (!value || typeof value !== "object") return; - if (Array.isArray(value)) { - for (const item of value) visit(item); - return; - } - const obj = value as Record; - const id = obj.id ?? obj.name ?? obj.model; - if (typeof id === "string" && id.trim()) { - entries.push({ id: id.trim(), raw: obj }); - } - for (const child of Object.values(obj)) { - if (child && typeof child === "object") visit(child); - } - }; - visit(raw); - return entries; -} - -function tzafonDisplayName(id: string): string { - if (id === "tzafon.northstar-cua-fast") return "Tzafon Northstar CUA Fast"; - return id; -} - -async function smokeTzafon(client: any, model: string): Promise { - try { - const response = await client.responses.create({ - model, - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Use the computer function tools to inspect this page. Do not answer in text." }, - { type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, - ], - }], - tools: TZAFON_FUNCTION_TOOLS, - instructions: "The screen's coordinate space is a 0-999 grid. Call a function tool instead of answering in text.", - temperature: 0, - max_output_tokens: 128, - }); - const output: any[] = response.output ?? []; - const functionCalls = output.filter((item) => item?.type === "function_call"); - const computerCalls = output.filter((item) => item?.type === "computer_call"); - const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", - tool_name: functionCalls.length ? "function_tools" : "computer_use", - tool_version: null, - beta_header: null, - observed_actions: unique([ - ...functionCalls.map((call) => call?.name).filter(Boolean), - ...computerActions.map((action) => action?.type).filter(Boolean), - ]), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -const TZAFON_FUNCTION_TOOLS = [ - { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } }, - { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } }, - { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } }, - { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } }, - { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } }, - { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } }, - { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } }, -]; - async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise { const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined); diff --git a/.agents/skills/update-models/reference/native-action-probe.ts b/.agents/skills/update-models/reference/native-action-probe.ts index 4c712551..c3b94343 100644 --- a/.agents/skills/update-models/reference/native-action-probe.ts +++ b/.agents/skills/update-models/reference/native-action-probe.ts @@ -1,12 +1,10 @@ #!/usr/bin/env tsx import { existsSync } from "node:fs"; import { readFile, writeFile } from "node:fs/promises"; -import { homedir } from "node:os"; import { join } from "node:path"; import process from "node:process"; -import { parse as parseToml } from "smol-toml"; -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "tzafon" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "xai"; interface ProbePrompt { id: string; @@ -64,8 +62,8 @@ function parseArgs(argv: string[]): Args { throw new Error(`unknown argument: ${arg}`); } } - if (!["openai", "anthropic", "gemini", "xai", "tzafon", "yutori"].includes(out.provider)) { - throw new Error("--provider is required: openai | anthropic | gemini | xai | tzafon | yutori"); + if (!["openai", "anthropic", "gemini", "xai"].includes(out.provider)) { + throw new Error("--provider is required: openai | anthropic | gemini | xai"); } if (!out.model) throw new Error("--model is required"); return out; @@ -76,8 +74,6 @@ function usage(): never { npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --limit 3 `); process.exit(0); } @@ -105,8 +101,6 @@ async function runProbe(provider: Provider, model: string, prompt: ProbePrompt): if (provider === "anthropic") return await probeAnthropic(model, prompt); if (provider === "gemini") return await probeGemini(model, prompt); if (provider === "xai") return await probeXai(model, prompt); - if (provider === "tzafon") return await probeTzafon(model, prompt); - if (provider === "yutori") return await probeYutori(model, prompt); throw new Error(`unknown provider ${provider satisfies never}`); } catch (err) { return { @@ -258,87 +252,6 @@ async function readFixtureScreenshot(): Promise { return readFile(path); } -async function probeYutori(model: string, prompt: ProbePrompt): Promise { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ - apiKey: await yutoriApiKey(), - baseURL: "https://api.yutori.com/v1", - }); - const response = await client.chat.completions.create({ - model, - messages: [{ - role: "user", - content: [ - { type: "text", text: prompt.text }, - { type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, - ], - }], - max_completion_tokens: 128, - temperature: 0.3, - }); - const choice = response.choices?.[0]; - const toolCalls: any[] = choice?.message?.tool_calls ?? []; - return { - id: prompt.id, - status: toolCalls.length ? "pass" : "inconclusive", - actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), - item_types: unique([ - choice?.message?.content ? "text" : undefined, - toolCalls.length ? "tool_calls" : undefined, - ].filter(Boolean) as string[]), - finish_reason: choice?.finish_reason ?? null, - raw_tool_calls: toolCalls.map(redactLargeFields), - }; -} - -async function yutoriApiKey(): Promise { - const env = process.env.YUTORI_API_KEY; - if (env && env.trim()) return env; - const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml"); - try { - const raw = parseToml(await readFile(cfgPath, "utf8")) as any; - const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; - const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; - if (typeof key === "string" && key.trim()) return key; - } catch { - // Fall through to a clear credential error. - } - throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)"); -} - -async function probeTzafon(model: string, prompt: ProbePrompt): Promise { - const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone"); - const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY }); - const response = await client.responses.create({ - model, - input: [{ - role: "user", - content: [ - { type: "input_text", text: `${prompt.text}\nCall one of the available function tools instead of answering in text.` }, - { type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, - ], - }], - tools: TZAFON_FUNCTION_TOOLS, - instructions: "The screen's coordinate space is a 0-999 grid.", - temperature: 0, - max_output_tokens: 128, - }); - const output: any[] = response.output ?? []; - const functionCalls = output.filter((item) => item?.type === "function_call"); - const computerCalls = output.filter((item) => item?.type === "computer_call"); - const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - id: prompt.id, - status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", - actions: unique([ - ...functionCalls.map((call) => call?.name).filter(Boolean), - ...computerActions.map((action) => action?.type).filter(Boolean), - ]), - item_types: unique(output.map((item) => item?.type).filter(Boolean)), - raw_tool_calls: [...functionCalls, ...computerCalls].map(redactLargeFields), - }; -} - const XAI_FUNCTION_TOOLS = [ { type: "function", name: "screenshot", description: "Capture the current browser screenshot.", parameters: { type: "object", properties: {}, additionalProperties: false } }, { type: "function", name: "goto", description: "Navigate to a URL.", parameters: { type: "object", properties: { url: { type: "string" } }, required: ["url"], additionalProperties: false } }, @@ -352,16 +265,6 @@ const XAI_FUNCTION_TOOLS = [ { type: "function", name: "back", description: "Go back in browser history.", parameters: { type: "object", properties: {}, additionalProperties: false } }, ]; -const TZAFON_FUNCTION_TOOLS = [ - { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } }, - { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } }, - { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } }, - { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } }, - { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } }, - { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } }, - { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } }, -]; - async function importDefault(pkg: string, named: string): Promise { try { const mod = await import(pkg); diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts index 95ea4be4..1dfef989 100644 --- a/.agents/skills/update-models/reference/provider-doc-drift.ts +++ b/.agents/skills/update-models/reference/provider-doc-drift.ts @@ -2,7 +2,7 @@ import { readFile, writeFile } from "node:fs/promises"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface Args { examples: string; @@ -30,11 +30,6 @@ const DOCS: Record = { "https://ai.google.dev/gemini-api/docs/computer-use", "https://ai.google.dev/api/models", ], - meta: [ - "https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos", - "https://dev.meta.ai/docs/features/responses", - "https://dev.meta.ai/docs/features/tool-calling", - ], xai: [ "https://docs.x.ai/developers/grok-4-5", "https://docs.x.ai/developers/tools/function-calling", @@ -45,32 +40,22 @@ const DOCS: Record = { "https://platform.kimi.ai/docs/api/tool-use", "https://platform.kimi.ai/docs/guide/use-kimi-vision-model", ], - yutori: [ - "https://docs.yutori.com/reference/navigator", - "https://docs.yutori.com/reference/n1", - "https://docs.yutori.com/reference/n1-5", - "https://docs.yutori.com/openapi.json", - ], }; const LOCAL_FILES: Record = { openai: "packages/ai/src/providers/openai/index.ts", anthropic: "packages/ai/src/providers/anthropic/actions.ts", gemini: "packages/ai/src/providers/gemini/index.ts", - meta: "packages/ai/src/providers/meta/index.ts", xai: "packages/ai/src/providers/xai/index.ts", moonshot: "packages/ai/src/providers/moonshot/index.ts", - yutori: "packages/ai/src/providers/yutori/actions.ts", }; const ACTION_REGEXES: Record = { openai: /\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g, anthropic: /\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g, gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g, - meta: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position|left_click|right_click|middle_click|triple_click|left_click_drag|mouse_move|key|hold_key|left_mouse_down|left_mouse_up)\b/g, xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, - yutori: /\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|go_forward|goto_url|mouse_move|middle_click|mouse_down|mouse_up|hold_key|extract_elements|find|set_element_value|execute_js)\b/g, }; function parseArgs(argv: string[]): Args { @@ -144,8 +129,8 @@ async function checkProvider(provider: Provider, examples: any): Promise.native.computer" and "provider..native.browser" require conflicting payload transforms for "tools.computer_use" ``` ```text @@ -604,7 +604,7 @@ Every provider tool surface must expose the first-party source it mirrors. CUA-a 4. **Batch overlap:** batches are mechanical; `browser_act` remains semantic; browser batches share ref state without a workflow DSL. 5. **Dynamic loading:** `setTools()` uses pi 0.83.0 additive markers only for final, cache-preserving in-tool additions; other changes are eager. 6. **Shared resources:** one resource pool survives tool/model changes and owns the translator and lazy CDP executor. -7. **Provider exports:** the native OpenAI, Anthropic, Google, Tzafon, and Yutori surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use CUA-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. +7. **Provider exports:** the native OpenAI, Anthropic, and Google surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use CUA-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. ## Decisions recorded diff --git a/docs/architecture.md b/docs/architecture.md index f42bf74b..be9ec2cd 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -154,10 +154,8 @@ catalog: - Anthropic native browser/computer declarations replace only their own placeholders and merge required beta headers with caller headers. - OpenAI streams through pi's builtin Responses transport and its automatic - prompt caching; a CUA-owned adapter handles OpenAI's native computer tool and - tool-search namespace round-trips. -- Tzafon replaces only the selected computer identity and fills declaration - dimensions from the actual viewport. + prompt caching by default; a CUA-owned adapter handles OpenAI's native + computer tool and tool-search namespace round-trips. - Anthropic's native browser tool falls back to an equivalent function-tool declaration when the active credential cannot access `browser_20260701`; the selected tool identity, name, schema, and executor remain unchanged. @@ -165,11 +163,33 @@ catalog: declaration plus exact exclusions through the CUA-owned Interactions API adapter. Excluded calls fail with a named catalog error instead of reaching generic tool dispatch. -- Yutori emits its native `tool_set`/`disable_tools` fields while preserving - ordinary function tools. - Meta, xAI, and Moonshot disable parallel tool calls when the selected catalog can mutate browser state. +### Transport derivation + +The transport a model streams through is a function of **(model, selected +tools)**, derived at catalog compilation — never stamped on the model ahead of +time and never branched on a provider name. A `CuaProviderBinding` may declare +`requiresApi`: the api id its provider-native tool needs. `compileCuaToolCatalog` +reads `requiresApi` off the selected bindings after normalizing the requested +catalog and returns a `catalog.model` carrying that api; selecting tools whose +bindings require different transports fails to compile with a named catalog +error. A model resolved with no such tool selected keeps its ordinary registry +api. + +This is why an OpenAI model selected with only CUA browser tools streams +through pi's builtin `openai-responses` transport, but the same model selected +with `cua.providers.openai.tools.computer()` compiles to the CUA-owned +`openai-cua-computer` api — and symmetrically for Google's +`google-cua-interactions` Interactions API versus pi's builtin Google +transport. + +`CuaAgent` and `CuaAgentHarness` push the compiled `catalog.model` into pi on +every construction and on every `setTools()`/`setModel()`, so the derived +transport applies uniformly regardless of which mutation path selected the +tools. + Generated payload processing has fixed order: model preparation, tool serialization, provider fields, then the caller's `onPayload` hook. @@ -185,8 +205,6 @@ serialization, provider fields, then the caller's `onPayload` hook. `browser_act`'s larger schema; - Anthropic's native browser tool when the model supports it; - Google's native browser action set; - - Tzafon's native computer tool configured for a browser; - - Yutori's native N1 or N1.5 browser set plus an explicit screenshot tool; 3. creates and retains its own application-level coding-tool list; 4. passes the complete list to `CuaAgentHarness`; 5. builds a caller-owned prompt from loaded skills and context files; @@ -207,8 +225,7 @@ to it so `ctrl+c` cancels the selector instead of quitting. `filterModelsForPicker`, `moveSelection`, `visibleWindow`) that make its behavior unit-testable without a terminal. - `tui/tool-selection.ts` — pure `/tools` state machine: identity keys matching - `normalizeTool`'s scheme, group badges, atomic provider groups, and - toggle/bulk operations. + `normalizeTool`'s scheme, group badges, and toggle/bulk operations. - `tui/tools-picker.ts` — the `/tools` component. Staged edits applied through `harness.setTools()` with a subset of the application-composed baseline, in baseline order. diff --git a/package-lock.json b/package-lock.json index 0f50d152..1f719a8e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -15,7 +15,6 @@ ], "devDependencies": { "@types/node": "22.18.4", - "smol-toml": "^1.7.0", "tsx": "^4.23.1", "typescript": "5.9.3" }, @@ -4420,12 +4419,6 @@ "integrity": "sha512-wWKOClTTiizcZhXnPY4wikVAwmdYHp8q6DmC+EJUzAMsycb7HB32Kh9RN4+0gExjmPmZSAQjgURXIGATPegAvA==", "license": "MIT" }, - "node_modules/@tzafon/lightcone": { - "version": "0.7.2", - "resolved": "https://registry.npmjs.org/@tzafon/lightcone/-/lightcone-0.7.2.tgz", - "integrity": "sha512-jzXTAOeE77FuuzP8J2dtxXxlBnN3Jb1o/iF7taVLGsT4ch8EemztVKjHH6KOJtu+3KvQzSioYoJCIX6pJ35jTA==", - "license": "Apache-2.0" - }, "node_modules/@vitest/expect": { "version": "3.2.7", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz", @@ -5625,19 +5618,6 @@ "dev": true, "license": "ISC" }, - "node_modules/smol-toml": { - "version": "1.7.0", - "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.7.0.tgz", - "integrity": "sha512-aqVvWoyO21L23mb+drl4RmMXbf6N7FdHjAhTRA9ZBL7apWBgfWC16KjrASI+1p9GAroljyMHj6fK67i0UiTNvQ==", - "dev": true, - "license": "BSD-3-Clause", - "engines": { - "node": ">= 18" - }, - "funding": { - "url": "https://github.com/sponsors/cyyynthia" - } - }, "node_modules/source-map-js": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", @@ -6176,7 +6156,6 @@ "license": "MIT", "dependencies": { "@earendil-works/pi-ai": "0.83.0", - "@tzafon/lightcone": "^0.7.0", "openai": "^6.26.0" }, "devDependencies": { diff --git a/package.json b/package.json index d6f57a8e..4d67b006 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,6 @@ }, "devDependencies": { "@types/node": "22.18.4", - "smol-toml": "^1.7.0", "tsx": "^4.23.1", "typescript": "5.9.3" } diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 54f3fda5..0672ec1e 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,42 @@ # Changelog +## 0.13.0 - 2026-08-13 + +Breaking: Tzafon and Yutori support is removed. + +- Update `@onkernel/cua-ai` to 0.13.0. Constructing a `CuaAgent` or + `CuaAgentHarness` with a Tzafon or Yutori model ref now fails to resolve the + model, and `cua.providers.tzafon` / `cua.providers.yutori` no longer exist. +- The tool-result image replay limit now exempts only OpenAI's native computer + tool, whose protocol requires every `computer_call_output` to carry a + screenshot. Tzafon's native computer results were exempt for the same reason + and are gone with the provider. +- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog + viewport option; the same value is still on `resources.browser.viewport`. + +## 0.12.0 - 2026-08-13 + +- Add `CuaAgentHarness.setModelAndTools()`. A model switch that also swaps + interaction tools has to compile as one pair now that the selected tools + decide the transport: staging the two in sequence produces an intermediate + catalog whose derived transport differs from both the old and the new one, + and records a model change for a transport nothing ever streamed with. + +- Update `@onkernel/cua-ai` to 0.12.0. The model streamed for a Google model + now depends on which tools `CuaAgent`/`CuaAgentHarness` were constructed or + mutated with: selecting Google's native browser toolset still compiles to + the CUA-owned Interactions API, but a Google model selected with only CDP + browser tools now streams through pi's builtin Google transport instead of + always carrying the CUA-owned api. This applies uniformly across + construction, `setTools()`, and `setModel()`, since all three feed the same + compiled `catalog.model` into pi. +- Fix `setTools()` recompiling from the previously *compiled* model instead of + the caller's model selection: dropping a native toolset that had derived a + tool-selection-dependent api (e.g. Google's Interactions API) left + subsequent tools-only recompiles stuck on that api even though the new + selection no longer required it. `CuaToolManager` now recompiles tools-only + changes from the model input the caller last selected. + ## 0.11.0 - 2026-08-13 - `responseThreading` (`CuaAgentOptions`/`CuaAgentHarnessOptions`) no longer diff --git a/packages/agent/README.md b/packages/agent/README.md index 93b9205d..a020bfc7 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -139,15 +139,14 @@ const tools = [ ]; ``` -Other provider groups include OpenAI native computer, Tzafon native computer, -Google's current predefined browser toolset, and Yutori native toolsets. Every -provider surface exposes linked first-party documentation. Meta and xAI use CUA -browser primitives plus `cua.tools.browser.act()` in the provider-matrix -examples. Moonshot uses browser primitives alone because its API rejects -`browser_act`'s larger schema. Compilation rejects incompatible tool/model -combinations before a request. Anthropic's native browser tool -uses an equivalent function-tool transport when the active credential cannot -access `browser_20260701`. +Other provider groups include OpenAI native computer and Google's current +predefined browser toolset. Every provider surface exposes linked first-party +documentation. xAI uses CUA browser primitives plus +`cua.tools.browser.act()` in the provider-matrix examples. Moonshot uses browser +primitives alone because its API rejects `browser_act`'s larger schema. +Compilation rejects incompatible tool/model combinations before a request. +Anthropic's native browser tool uses an equivalent function-tool transport when +the active credential cannot access `browser_20260701`. ## Dynamic catalogs @@ -202,9 +201,9 @@ Tools return only requested feedback: textual markers. `toolResultImageReplayLimit` controls how many recent tool-result images remain -in model context (`4` by default, or `false` to disable projection). Tzafon -native screenshot results are exempt because its continuation protocol requires -the image. +in model context (`4` by default, or `false` to disable projection). OpenAI +native computer results are exempt because its protocol requires each +`computer_call_output` to carry a screenshot. ## Custom tools diff --git a/packages/agent/examples/shared/tools.ts b/packages/agent/examples/shared/tools.ts index 7bf13770..e436193c 100644 --- a/packages/agent/examples/shared/tools.ts +++ b/packages/agent/examples/shared/tools.ts @@ -1,5 +1,7 @@ import { cua, + cuaModelCapabilities, + getCuaModel, parseCuaModelRef, type CuaModelRef, } from "@onkernel/cua-ai"; @@ -29,26 +31,17 @@ export function toolsForModel(model: CuaModelRef): CuaAgentTool[] { case "google": // Current Gemini computer-use models expect Google's predefined browser actions. return cua.providers.google.toolsets.browser(); - case "meta": case "xai": // No first-party native browser surface exists, so use CUA browser primitives // plus verified dependent plans. return structuredBrowserTools(); case "moonshotai": case "openrouter": - // Same as meta/xai, minus browser_act: Kimi's API rejects that tool's - // oversized schema, so Kimi gets the browser primitives only. - return cua.toolsets.browser(); - case "tzafon": - // Northstar's documented native computer schema is its supported interaction contract. - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - // Match Yutori's documented model generation and add explicit visual access. - return [ - ...(modelId.startsWith("n1.5") - ? cua.providers.yutori.toolsets.n15Core() - : cua.providers.yutori.toolsets.n1()), - cua.tools.computer.screenshot(), - ]; + // Same as xai, minus browser_act where the model rejects that tool's + // oversized schema. OpenRouter fronts several model families, so ask + // the model rather than the provider. + return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas + ? structuredBrowserTools() + : cua.toolsets.browser(); } } diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 89886cea..6dd045e9 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -92,7 +92,7 @@ export type CuaAgentOptions = Omit & streamFn?: AgentOptions["streamFn"]; emptyResponseRecovery?: CuaEmptyResponseRecoveryOptions; toolResultImageReplayLimit?: ToolResultImageReplayLimit; - /** Governs Google and Tzafon's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ + /** Governs Google's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ responseThreading?: boolean; retry?: CuaRetryOptions; }; @@ -113,7 +113,7 @@ type CuaAgentHarnessOptionsBase< onPayload?: SimpleStreamOptions["onPayload"]; emptyResponseRecovery?: CuaEmptyResponseRecoveryOptions; toolResultImageReplayLimit?: ToolResultImageReplayLimit; - /** Governs Google and Tzafon's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ + /** Governs Google's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ responseThreading?: boolean; retry?: CuaRetryOptions; }; @@ -258,6 +258,7 @@ export class CuaAgent { setTools(tools: readonly CuaAgentTool[]): void { const prepared = this.tools.prepareTools(tools); + this.coreAgent.state.model = prepared.catalog.model; this.coreAgent.state.tools = this.tools.agentTools(prepared); this.tools.commit(prepared); this.runtimeDirty = true; @@ -402,12 +403,16 @@ export class CuaAgentHarness< getTools(): readonly CuaHarnessTool[] { return this.tools.getTools(); } async setTools(tools: readonly CuaHarnessTool[]): Promise { + const previousModel = this.tools.catalog.model; const previousTools = this.tools.harnessTools(); const prepared = this.tools.prepareTools(tools); const materialized = this.tools.harnessTools(prepared); + const transportChanged = modelTransportChanged(previousModel, prepared.catalog.model); try { + if (transportChanged) await this.coreHarness.setModel(prepared.catalog.model); await this.coreHarness.setTools(materialized, materialized.map((tool) => tool.name)); } catch (error) { + if (transportChanged) await this.coreHarness.setModel(previousModel); await this.coreHarness.setTools(previousTools, previousTools.map((tool) => tool.name)); throw error; } @@ -432,6 +437,28 @@ export class CuaAgentHarness< this.tools.commit(prepared); } + /** + * Select a model and its tool list in one compile. A model switch that also + * swaps interaction tools would otherwise stage through an intermediate + * catalog whose derived transport differs from both the old and the new one, + * recording a `model_change` for a transport nothing ever streamed with. + */ + async setModelAndTools(model: CuaModelInput, tools: readonly CuaHarnessTool[]): Promise { + const previousModel = this.tools.catalog.model; + const previousTools = this.tools.harnessTools(); + const prepared = this.tools.prepareModelAndTools(model, tools); + const materialized = this.tools.harnessTools(prepared); + try { + await this.coreHarness.setModel(prepared.catalog.model); + await this.coreHarness.setTools(materialized, materialized.map((tool) => tool.name)); + } catch (error) { + await this.coreHarness.setModel(previousModel); + await this.coreHarness.setTools(previousTools, previousTools.map((tool) => tool.name)); + throw error; + } + this.tools.commit(prepared); + } + prompt(text: string, options?: { images?: ImageContent[] }) { return this.coreHarness.prompt(text, options); } skill(name: string, additionalInstructions?: string) { return this.coreHarness.skill(name, additionalInstructions); } promptFromTemplate(name: string, args?: string[]) { return this.coreHarness.promptFromTemplate(name, args); } @@ -476,6 +503,11 @@ function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model, next: Model): boolean { + return previous.provider !== next.provider || previous.id !== next.id || previous.api !== next.api; +} + function withCatalogModels( models: Models, manager: CuaToolManager, @@ -529,7 +561,7 @@ function resolveToolResultImageReplayLimit(limit: ToolResultImageReplayLimit | u /** Native computer tool names whose screenshot history the provider protocol requires in full, regardless of the image replay limit. */ function requiredImageToolNames(incoming: CuaIncomingToolPlan): ReadonlySet { - return new Set([incoming.tzafonComputerName, incoming.openaiComputerName].filter((name): name is string => !!name)); + return new Set(incoming.openaiComputerName ? [incoming.openaiComputerName] : []); } function projectToolResultImages( diff --git a/packages/agent/src/resources.ts b/packages/agent/src/resources.ts index e6ce8911..390a2bbc 100644 --- a/packages/agent/src/resources.ts +++ b/packages/agent/src/resources.ts @@ -34,7 +34,6 @@ type ToolContent = Array; export class CuaExecutionResources { readonly browser: KernelBrowser; readonly client: Kernel; - readonly viewport: { readonly width: number; readonly height: number }; private readonly translator: InternalComputerTranslator; /** Each spec is materialized exactly once per resource pool. */ private readonly materialized = new WeakMap(); @@ -47,7 +46,6 @@ export class CuaExecutionResources { }) { this.browser = options.browser; this.client = options.client; - this.viewport = options.browser.viewport ?? { width: 1920, height: 1080 }; this.translator = new InternalComputerTranslator(options); } diff --git a/packages/agent/src/tool-manager.ts b/packages/agent/src/tool-manager.ts index d19f0635..ec81e03f 100644 --- a/packages/agent/src/tool-manager.ts +++ b/packages/agent/src/tool-manager.ts @@ -40,6 +40,14 @@ export type CuaHarnessTool = CuaToo */ export interface PreparedCuaTools = CuaAgentTool> { readonly requested: readonly TRequested[]; + /** + * The model input this state was compiled from, before catalog compilation + * derived a tool-selection-dependent transport onto it. A tools-only + * recompile reuses this (not `catalog.model`, which may carry a transport + * the new tool selection no longer requires) so the derivation re-runs from + * a clean model every time. + */ + readonly modelSelection: CuaModelRef | Model; readonly catalog: CuaToolCatalog; readonly tools: readonly AgentTool[]; readonly harnessTools: readonly AgentHarnessTool[]; @@ -112,7 +120,7 @@ export class CuaToolManager = CuaAgentToo prepareTools(tools: readonly TRequested[]): PreparedCuaTools { this.assertMutationScope("setTools"); - return this.prepare(this.current.catalog.model, tools); + return this.prepare(this.current.modelSelection, tools); } prepareModel(model: CuaModelRef | Model): PreparedCuaTools { @@ -120,6 +128,11 @@ export class CuaToolManager = CuaAgentToo return this.prepare(model, this.current.requested); } + prepareModelAndTools(model: CuaModelRef | Model, tools: readonly TRequested[]): PreparedCuaTools { + this.assertMutationScope("setModelAndTools"); + return this.prepare(model, tools); + } + commit(prepared: PreparedCuaTools): void { this.current = prepared; } @@ -154,7 +167,6 @@ export class CuaToolManager = CuaAgentToo const catalog = compileCuaToolCatalog({ model: typeof model === "string" ? this.resolveModel(model) : model, requestedTools: inputs, - viewport: this.resources.viewport, }); const fingerprints: string[] = []; @@ -174,6 +186,7 @@ export class CuaToolManager = CuaAgentToo return Object.freeze({ requested, + modelSelection: model, catalog, tools: Object.freeze(joined.map((tool) => this.wrapAgentExecutable(tool as AgentTool))), harnessTools: Object.freeze(joined.map((tool) => this.wrapHarnessExecutable(tool))), @@ -215,7 +228,7 @@ export class CuaToolManager = CuaAgentToo }); } - private assertMutationScope(api: "setTools" | "setModel"): void { + private assertMutationScope(api: "setTools" | "setModel" | "setModelAndTools"): void { const scope = this.execution.getStore(); if (scope && scope.executionMode !== "sequential") { throw new Error(`tool "${scope.toolName}" must declare executionMode: "sequential" before calling ${api}() during execution`); diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index ddd61f61..ed459a46 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -4,6 +4,7 @@ import { createCuaModels, getCuaModel, cua, + GOOGLE_CUA_INTERACTIONS_API, isCuaToolSpec, type AssistantMessage, type Context, @@ -104,34 +105,6 @@ describe("CuaAgent explicit tools", () => { expect("getMode" in agent).toBe(false); }); - it("retains protocol-required Tzafon screenshot results outside the image replay limit", async () => { - const contexts: Context[] = []; - const model = getCuaModel("tzafon:tzafon.northstar-cua-fast"); - const nativeComputer = cua.providers.tzafon.tools.computer(); - const ordinaryTool = callerTool("ordinary"); - const image = { type: "image" as const, data: "c2NyZWVuc2hvdA==", mimeType: "image/png" }; - const messages: AgentMessage[] = [ - assistant(model, [{ type: "toolCall", id: "native-shot", name: "computer", arguments: { action: { type: "screenshot" } } }], "toolUse"), - { role: "toolResult", toolCallId: "native-shot", toolName: "computer", content: [image], isError: false, timestamp: 1 }, - assistant(model, [{ type: "toolCall", id: "ordinary-shot", name: "ordinary", arguments: {} }], "toolUse"), - { role: "toolResult", toolCallId: "ordinary-shot", toolName: "ordinary", content: [image], isError: false, timestamp: 2 }, - ]; - const agent = new CuaAgent({ - browser, - client, - tools: [nativeComputer, ordinaryTool], - streamFn: scriptedStream([(selectedModel) => assistant(selectedModel)], contexts), - toolResultImageReplayLimit: 0, - initialState: { model, messages }, - }); - - await agent.prompt("continue"); - - const results = contexts[0]!.messages.filter((message) => message.role === "toolResult"); - expect(results[0]!.content).toEqual([image]); - expect(results[1]!.content).toEqual([{ type: "text", text: "[stale tool-result images omitted]" }]); - }); - it("retains protocol-required OpenAI native computer screenshot results outside the image replay limit", async () => { const contexts: Context[] = []; const model = getCuaModel("openai:gpt-5.5"); @@ -454,6 +427,27 @@ describe("CuaAgent explicit tools", () => { expect(agent.getTools()).toEqual([switcher]); expect(contexts[1]?.messages.find((message) => message.role === "toolResult")).not.toHaveProperty("addedToolNames"); }); + + it("streams with the transport setTools derives, not just the catalog it compiles", async () => { + const streamedApis: string[] = []; + const agent = new CuaAgent({ + browser, + client, + tools: [cua.tools.browser.snapshot()], + streamFn: (model, context, options) => { + streamedApis.push(model.api); + return scriptedStream([(selectedModel) => assistant(selectedModel)])(model, context, options); + }, + initialState: { model: "google:gemini-3.6-flash" }, + }); + expect(agent.getModel().api).toBe("google-generative-ai"); + + agent.setTools(cua.providers.google.toolsets.browser()); + expect(agent.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await agent.prompt("go"); + expect(streamedApis).toEqual([GOOGLE_CUA_INTERACTIONS_API]); + }); }); describe("CuaAgentHarness explicit tools", () => { @@ -578,4 +572,59 @@ describe("CuaAgentHarness explicit tools", () => { expect(successfulCalls).toBe(1); }); + + it("streams with the transport setTools derives, not just the catalog it compiles", async () => { + const streamedApis: string[] = []; + const script = scriptedStream([(selectedModel) => assistant(selectedModel)]); + const harness = new CuaAgentHarness({ + ...(await harnessServices()), + browser, + client, + model: "google:gemini-3.6-flash", + models: modelsFromStream((model, context, options) => { + streamedApis.push(model.api); + return script(model, context, options); + }, "google"), + tools: [cua.tools.browser.snapshot()], + }); + expect(harness.getModel().api).toBe("google-generative-ai"); + + await harness.setTools(cua.providers.google.toolsets.browser()); + expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await harness.prompt("go"); + expect(streamedApis).toEqual([GOOGLE_CUA_INTERACTIONS_API]); + }); + + it("does not record a model change for a setTools() call that leaves the derived transport unchanged", async () => { + const services = await harnessServices(); + const harness = new CuaAgentHarness({ + ...services, + browser, + client, + model: "openai:gpt-5.5", + tools: [callerTool("first")], + }); + await harness.setTools([callerTool("second")]); + const modelChanges = (await services.session.getBranch()).filter((entry) => entry.type === "model_change"); + expect(modelChanges).toEqual([]); + }); + + it("records one model change for a switch that changes both the model and its derived transport", async () => { + const services = await harnessServices(); + const harness = new CuaAgentHarness({ + ...services, + browser, + client, + model: "google:gemini-3.6-flash", + tools: cua.providers.google.toolsets.browser(), + }); + expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await harness.setModelAndTools("openai:gpt-5.5", [cua.tools.browser.snapshot()]); + expect(harness.getModel().api).toBe("openai-responses"); + + const modelChanges = (await services.session.getBranch()).filter((entry) => entry.type === "model_change"); + expect(modelChanges).toHaveLength(1); + }); }); diff --git a/packages/agent/test/e2e.live.test.ts b/packages/agent/test/e2e.live.test.ts index a9d347e0..80b16384 100644 --- a/packages/agent/test/e2e.live.test.ts +++ b/packages/agent/test/e2e.live.test.ts @@ -14,20 +14,16 @@ const LIVE = process.env.CUA_E2E_LIVE === "1"; const KERNEL_API_KEY = process.env.KERNEL_API_KEY; type ProviderCase = { - name: "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshotai" | "tzafon" | "yutori"; + name: "openai" | "anthropic" | "gemini" | "xai" | "moonshotai"; apiKeyEnvVar: string; modelRef: | "openai:gpt-5.6-sol" | "anthropic:claude-opus-5" | "google:gemini-3.6-flash" - | "meta:muse-spark-1.1" | "xai:grok-4.5" - | "moonshotai:kimi-k3" - | "tzafon:tzafon.northstar-cua-fast" - | "yutori:n1.5-latest"; + | "moonshotai:kimi-k3"; prompt: string; expectToolCalls: boolean; - expectReadArtifact?: boolean; timeoutMs: number; ciOptInEnvVar?: string; }; @@ -80,19 +76,6 @@ const cases: ProviderCase[] = [ timeoutMs: 300_000, ciOptInEnvVar: "CUA_E2E_GEMINI", }, - { - name: "meta", - apiKeyEnvVar: "META_API_KEY", - modelRef: "meta:muse-spark-1.1", - prompt: [ - "Use the tool named `browser_screenshot` exactly once to inspect the browser.", - "Pass empty arguments (`{}`).", - "Do not call any other tools.", - "After the tool result, provide a one-sentence summary.", - ].join("\n"), - expectToolCalls: true, - timeoutMs: 180_000, - }, { name: "xai", apiKeyEnvVar: "XAI_API_KEY", @@ -119,32 +102,6 @@ const cases: ProviderCase[] = [ expectToolCalls: true, timeoutMs: 180_000, }, - { - name: "tzafon", - apiKeyEnvVar: "TZAFON_API_KEY", - modelRef: "tzafon:tzafon.northstar-cua-fast", - prompt: [ - "Use the native `computer` tool's `screenshot` action exactly once.", - "Do not call any other tools.", - "After the tool result, provide a one-sentence summary.", - ].join("\n"), - expectToolCalls: true, - timeoutMs: 120_000, - ciOptInEnvVar: "CUA_E2E_TZAFON", - }, - { - name: "yutori", - apiKeyEnvVar: "YUTORI_API_KEY", - modelRef: "yutori:n1.5-latest", - prompt: [ - "Use the function tool named `computer_screenshot` exactly once.", - "Pass empty arguments (`{}`).", - "Do not call any other tools.", - ].join("\n"), - expectToolCalls: true, - expectReadArtifact: false, - timeoutMs: 180_000, - }, ]; const switchCases: ModelSwitchCase[] = [ @@ -187,7 +144,6 @@ function structuredBrowserTools() { function toolsForCase(c: ProviderCase) { switch (c.name) { case "openai": - case "meta": case "xai": return structuredBrowserTools(); case "moonshotai": @@ -197,10 +153,6 @@ function toolsForCase(c: ProviderCase) { return [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; case "gemini": return cua.providers.google.toolsets.browser(); - case "tzafon": - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - return [...cua.providers.yutori.toolsets.n15Core(), cua.tools.computer.screenshot()]; } } @@ -253,7 +205,7 @@ function assertStats(stats: RunStats, c: ProviderCase, runtimeName: "agent" | "h if (c.expectToolCalls) { expect(stats.toolCalls).toBeGreaterThan(0); expect(stats.toolResults).toBeGreaterThan(0); - if (c.expectReadArtifact !== false) expect(stats.hasReadArtifact).toBe(true); + expect(stats.hasReadArtifact).toBe(true); } expect(stats.finalAssistant).toBeDefined(); if (stats.finalAssistant?.role === "assistant") { diff --git a/packages/agent/test/example-provider-matrix.test.ts b/packages/agent/test/example-provider-matrix.test.ts index 1fc91b6f..53cdb2c6 100644 --- a/packages/agent/test/example-provider-matrix.test.ts +++ b/packages/agent/test/example-provider-matrix.test.ts @@ -5,8 +5,6 @@ import { import { describe, expect, it } from "vitest"; import { toolsForModel } from "../examples/shared/tools"; -const viewport = { width: 1440, height: 900 }; - /** * The example matrices are plain scripts: they are excluded from `tsc -b` and are * never executed in CI, so a provider policy that no longer compiles used to be @@ -20,19 +18,17 @@ const models: readonly CuaModelRef[] = [ "anthropic:claude-opus-5", "anthropic:claude-sonnet-5", "google:gemini-3.6-flash", - "meta:muse-spark-1.1", + "openrouter:meta/muse-spark-1.1", "xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3", - "tzafon:tzafon.northstar-cua-fast", - "yutori:n1.5-latest", ]; describe("example provider matrix tool policy", () => { it("compiles a valid catalog for every model the matrices advertise", () => { for (const model of models) { expect( - () => compileCuaToolCatalog({ model, requestedTools: toolsForModel(model), viewport }), + () => compileCuaToolCatalog({ model, requestedTools: toolsForModel(model) }), model, ).not.toThrow(); } @@ -49,7 +45,7 @@ describe("example provider matrix tool policy", () => { ); it("still advertises browser_act where the provider accepts it", () => { - for (const model of ["openai:gpt-5.6-sol", "meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["openai:gpt-5.6-sol", "xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { expect(toolsForModel(model).map((tool) => tool.name), model).toContain("browser_act"); } }); diff --git a/packages/agent/test/resources.test.ts b/packages/agent/test/resources.test.ts index 0ec84b48..f8f36c32 100644 --- a/packages/agent/test/resources.test.ts +++ b/packages/agent/test/resources.test.ts @@ -203,10 +203,10 @@ describe("CuaExecutionResources results and batch boundaries", () => { expect(result.details).not.toHaveProperty("isError"); }); - it("returns status text for Yutori writes without capturing a screenshot", async () => { + it("returns status text for provider-native writes without capturing a screenshot", async () => { const { resources, captureScreenshot } = setup(); - const spec = cua.providers.yutori.toolsets.n15Core().find((tool) => tool.name === "left_click")!; - const result = await resources.materialize(spec).execute("click", { coordinates: [100, 200] }); + const spec = cua.providers.google.toolsets.browser().find((tool) => tool.name === "click")!; + const result = await resources.materialize(spec).execute("click", { x: 100, y: 200 }); expect(result.content).toEqual([{ type: "text", text: "Actions executed successfully." }]); expect(captureScreenshot).not.toHaveBeenCalled(); }); diff --git a/packages/agent/test/tool-manager.test.ts b/packages/agent/test/tool-manager.test.ts index db0b0f75..c9d93d9d 100644 --- a/packages/agent/test/tool-manager.test.ts +++ b/packages/agent/test/tool-manager.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, vi } from "vitest"; -import { callerToolIdentity, cua } from "@onkernel/cua-ai"; +import { callerToolIdentity, cua, GOOGLE_CUA_INTERACTIONS_API } from "@onkernel/cua-ai"; import type Kernel from "@onkernel/sdk"; import { CuaExecutionResources, @@ -92,6 +92,22 @@ describe("CuaToolManager identity join", () => { }); }); +describe("CuaToolManager transport derivation", () => { + it("derives the compiled model's api from selected tools on construction and on both mutation paths", () => { + const manager = new CuaToolManager(setup(), "google:gemini-3.6-flash", [cua.tools.browser.snapshot()]); + expect(manager.catalog.model.api).toBe("google-generative-ai"); + + manager.commit(manager.prepareTools(cua.providers.google.toolsets.browser())); + expect(manager.catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + manager.commit(manager.prepareModel("google:gemini-3.6-flash")); + expect(manager.catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + manager.commit(manager.prepareTools([cua.tools.browser.snapshot()])); + expect(manager.catalog.model.api).toBe("google-generative-ai"); + }); +}); + describe("CuaToolManager implementation identity", () => { it("materializes each spec exactly once across model and tool recompilation", () => { const resources = setup(); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index aa0487c5..f1e3d6a0 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,88 @@ # Changelog +## 0.13.0 - 2026-08-13 + +- Remove the Meta provider. pi-ai ships no `meta` provider, so cua hand-wrote a + model entry and pointed pi's own `openai-responses` transport at + `api.meta.ai`. Meta was the last user of the model-override mechanism, so + `CUA_MODEL_OVERRIDES`, `cuaOverrideModels()`, and `META_API_KEY` go with it, + and `getCuaModel` no longer has a "supported but not registered" fallback. + Muse Spark remains available through pi's OpenRouter catalog as + `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities + because OpenRouter's provider-level defaults are conservative. + +- Remove `routeCuaApi`. With Tzafon and Yutori gone it routed no transport at + all, and its last branch only patched grok-4.5's thinking-level map, price + tiers, and compat flags onto pi-ai's registry entry. Live runs on grok-4.5 + at the default, `off`, and `xhigh` thinking levels behave identically + without it, so model resolution now returns pi-ai's data unmodified. The + only lost detail is a >200k-token price tier that pi's registry does not + carry, which affects `usage.cost` reporting for long requests and nothing + else. + +Breaking: Tzafon and Yutori support is removed. + +- Remove the `tzafon` and `yutori` providers: their `CuaProvider` members, + model annotations and overrides, `cua.providers.tzafon`, + `cua.providers.yutori`, `TZAFON_API_KEY`/`YUTORI_API_KEY`, and the exported + `TZAFON_RESPONSES_API`, `YUTORI_CHAT_COMPLETIONS_API`, + `streamTzafonResponses`, `streamSimpleTzafonResponses`, `streamYutori`, and + `streamSimpleYutori` stream functions. `createCuaModels()` no longer + registers either provider, and refs like `tzafon:tzafon.northstar-cua-fast` + or `yutori:n1.5-latest` now fail to resolve. +- `CuaProviderBinding` loses its `tzafon-native` and `yutori-native` variants, + and `CuaIncomingToolPlan` loses `tzafonComputerName` and `yutoriNames`. The + Yutori-only rule rejecting a partial n1 native action set is gone with them; + every surviving native toolset can be selected in part. +- `CompileCuaToolCatalogOptions.viewport` is removed. It existed only to fill + Tzafon's `display_width`/`display_height` declaration defaults, and no + surviving declaration reads it. +- `routeCuaApi` no longer routes any transport. Every remaining provider gets + its transport either from pi-ai's registry or from the selected tools' + `requiresApi`; what is left is grok-4.5's cost/compat/thinking-level + overrides. +- Drop the `@tzafon/lightcone` dependency. + +## 0.12.0 - 2026-08-13 + +- Route OpenAI's native computer adapter through its own stream function, + `streamOpenAICuaComputer`, selected by `model.api`. `streamOpenAIResponses` + no longer inspects the incoming tool plan to decide which adapter runs: the + compiled api id is the only dispatch key, and the provider wrapper is the + only place that reads it. +- Remove the unreferenced Yutori n1.5 expanded action declarations + (`YUTORI_N15_EXPANDED_ACTION_TYPES`, `YUTORI_N15_EXPANDED_TOOL_SET`, + `YUTORI_N15_ACTION_TYPES`) and the canonical-action type aliases nothing + consumed. None were exported from the package root. The expanded set was + scaffolding for a ref/DOM execution path that does not exist. + +Breaking: Google's api id is now derived from selected tools, not stamped on +every Google model. + +- `compileCuaToolCatalog` derives the compiled model's `api` from the selected + tools' provider bindings: a `CuaProviderBinding` may declare `requiresApi`, + and the returned `catalog.model` carries that transport. Selecting tools + whose bindings require different transports fails to compile with a named + catalog error. This makes transport a function of `(model, selected tools)` + instead of `(model)` alone. +- `getCuaModel("google:...")` no longer forces `google-cua-interactions`. A + Google model resolved without Google's native browser toolset selected now + keeps pi-ai's builtin `google-generative-ai` transport; selecting + `cua.providers.google.toolsets.browser()` still compiles to + `google-cua-interactions` as before. `routeCuaApi` no longer touches Google + at all — it's now scoped to genuinely model-shaped routing (Tzafon and + Yutori, which pi ships no transport for at all, and grok-4.5's cost/compat + overrides). +- Add `OPENAI_CUA_COMPUTER_API` (`"openai-cua-computer"`). A model compiled + with `cua.providers.openai.tools.computer()` selected now carries this api; + the OpenAI provider wrapper dispatches to the CUA adapter on `model.api` + alone for that case. The one remaining request-shape check, + `requiresCuaOpenAINamespaceAdapter` (renamed from `requiresCuaOpenAIAdapter`, + which also tested for the native computer tool), covers only the case that + cannot be derived from the model: a transcript carrying a deferred + tool-search addition or a replayed function-call namespace, which pi-ai's + builtin transport does not round-trip. + ## 0.11.0 - 2026-08-13 Breaking: OpenAI models no longer carry a CUA-owned api id. diff --git a/packages/ai/README.md b/packages/ai/README.md index aef7db5e..d7e7b206 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -157,10 +157,6 @@ cua.providers.google.toolsets.browser({ exclude: ["right_click"] }); // Meta, xAI, and Moonshot use the ordinary CUA browser tools. cua.toolsets.browser(); - -cua.providers.tzafon.tools.computer(); -cua.providers.yutori.toolsets.n1(); -cua.providers.yutori.toolsets.n15Core(); ``` The Google browser set exposes the current predefined action names and uses @@ -240,21 +236,27 @@ additions through pi's active-tool change entries. ## Provider behavior -- **OpenAI**: streams through pi's builtin Responses transport and its - automatic prompt caching by default. A CUA-owned adapter is used only for - OpenAI's native computer tool and for tool-search namespace round-trips. +Transport is derived, not stamped on the model ahead of time: a selected +tool's provider binding may declare `requiresApi`, and `compileCuaToolCatalog` +returns a `catalog.model` carrying that api. Selecting tools whose bindings +require different transports fails to compile. + +- **OpenAI**: a model selected with only ordinary/CUA browser tools streams + through pi's builtin Responses transport and its automatic prompt caching. + Selecting `cua.providers.openai.tools.computer()` derives the CUA-owned + `openai-cua-computer` api instead, which a CUA adapter handles; that same + adapter also covers tool-search namespace round-trips regardless of api, + since pi's builtin transport does not replay them. - **Anthropic**: exact native declarations, beta-header composition, and - adaptive model preparation. -- **Google**: a CUA-owned Interactions API adapter plus the current predefined - browser set with explicit exclusions. + adaptive model preparation. No api fork — every Anthropic model streams + through pi's builtin transport. +- **Google**: a model selected without Google's native browser toolset streams + through pi's builtin transport. Selecting + `cua.providers.google.toolsets.browser()` derives the CUA-owned + `google-cua-interactions` api, which serializes one `computer_use` + declaration plus explicit exclusions through the Interactions API adapter. - **Meta/xAI/Moonshot**: ordinary function tools with serial tool calls when the selected catalog mutates browser state. -- **Tzafon**: identity-scoped native declaration replacement with actual viewport - dimensions. Explicit screenshot and terminal answer actions are supported; - non-screenshot native action loops fail before browser execution because - Tzafon's continuation protocol requires implicit post-action screenshots. -- **Yutori**: identity-scoped native `tool_set`/`disable_tools` fields while - preserving ordinary function tools such as an explicitly selected screenshot. ## API keys @@ -267,8 +269,8 @@ import { ``` Conventional variables are `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, -`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, -`MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`. +`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `XAI_API_KEY`, and +`MOONSHOT_API_KEY`. ## Development diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 612e3c12..6c802201 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -72,19 +72,6 @@ model/tool surfaces are intentionally not exposed. Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use). -## `meta` - -CLI default interaction: CUA browser primitives plus explicit `browser_act`. - -Exact IDs: - -- `muse-spark-1.1` - -Muse Spark uses Meta's OpenAI-compatible Responses API with ordinary function -tools. CUA continues tool loops through `previous_response_id`. - -Source: [Meta computer-use cookbook](https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos). - ## `xai` CLI default interaction: CUA browser primitives plus explicit `browser_act`. @@ -123,39 +110,21 @@ Source: [Kimi K3 announcement](https://www.kimi.com/blog/kimi-k3), [tool use](ht ## `openrouter` -CLI default interaction: CUA browser primitives only. OpenRouter's Kimi K3 -route accepts complex function schemas but rejects the larger `browser_act` -schema, and state-mutating calls are serialized. +CLI default interaction: per model, not per provider. OpenRouter fronts several +model families, so the CLI asks each model whether it accepts `browser_act`'s +schema rather than assuming one answer for the whole provider. Model refs use the `openrouter:` prefix: -- `moonshotai/kimi-k3` - -Kimi K3 uses OpenRouter's OpenAI-compatible chat completions API with ordinary -CUA browser function tools. OpenRouter does not expose the provider-native -computer tools declared by other CUA providers. - -Source: [OpenRouter model page](https://openrouter.ai/moonshotai/kimi-k3). - -## `tzafon` - -Coordinates: normalized 0–999 - -Exact IDs: - -- `tzafon.northstar-cua-fast` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) -- `tzafon.northstar-cua-fast-1.6` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) -- `tzafon.northstar-cua-fast-1.7-experiment` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) - -## `yutori` - -Coordinates: normalized 0–1000 - -Exact IDs: +- `moonshotai/kimi-k3` — browser primitives only: accepts complex function + schemas but rejects the larger `browser_act` schema. State mutations are + serialized. +- `meta/muse-spark-1.1` — browser primitives plus explicit `browser_act`. State + mutations are serialized. -- `n1-latest` -- `n1-20260203` -- `n1.5-latest` -- `n1.5-20260428` +Both use OpenRouter's OpenAI-compatible chat completions API with ordinary CUA +browser function tools. OpenRouter does not expose the provider-native computer +tools declared by other CUA providers. -Source: [Yutori Navigator reference](https://docs.yutori.com/reference/navigator). +Sources: [Kimi K3](https://openrouter.ai/moonshotai/kimi-k3), +[Muse Spark 1.1](https://openrouter.ai/meta/muse-spark-1.1). diff --git a/packages/ai/examples/quickstart.ts b/packages/ai/examples/quickstart.ts index b53dd581..2c991aa8 100644 --- a/packages/ai/examples/quickstart.ts +++ b/packages/ai/examples/quickstart.ts @@ -19,7 +19,6 @@ const screenshot = await readFile(new URL("./screenshot.png", import.meta.url)); const catalog = compileCuaToolCatalog({ model: modelRef, requestedTools: [cua.tools.computer.click()], - viewport: { width: 1440, height: 900 }, }); const response = await cuaModels().complete( diff --git a/packages/ai/package.json b/packages/ai/package.json index 31d39c19..c43e91b6 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -45,7 +45,6 @@ }, "dependencies": { "@earendil-works/pi-ai": "0.83.0", - "@tzafon/lightcone": "^0.7.0", "openai": "^6.26.0" }, "devDependencies": { diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts index 58bef140..70f91387 100644 --- a/packages/ai/src/api-keys.ts +++ b/packages/ai/src/api-keys.ts @@ -12,12 +12,9 @@ const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { openai: ["OPENAI_API_KEY"], anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], - meta: ["META_API_KEY"], xai: ["XAI_API_KEY"], moonshotai: ["MOONSHOT_API_KEY"], openrouter: ["OPENROUTER_API_KEY"], - tzafon: ["TZAFON_API_KEY"], - yutori: ["YUTORI_API_KEY"], }; /** diff --git a/packages/ai/src/cua.ts b/packages/ai/src/cua.ts index a745e75a..091a57e2 100644 --- a/packages/ai/src/cua.ts +++ b/packages/ai/src/cua.ts @@ -8,13 +8,8 @@ import { } from "./actions/index"; import { supportsAnthropicNativeBrowser } from "./providers/anthropic/capabilities"; import { mapNativeBrowserInput, mapNativeComputerInput } from "./providers/anthropic/native"; -import { toCanonicalActions as toTzafonActions } from "./providers/tzafon/provider"; -import { - toCanonicalActions as toYutoriActions, - YUTORI_N1_ACTION_TYPES, - YUTORI_N15_CORE_ACTION_TYPES, - YUTORI_N15_CORE_TOOL_SET, -} from "./providers/yutori/actions"; +import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; +import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; import { CUA_TOOL_SPEC_KIND, type CuaCoordinateContract, @@ -78,9 +73,6 @@ const providerSources = Object.freeze({ openai: "https://developers.openai.com/api/docs/guides/tools-computer-use", anthropic: "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool", google: "https://ai.google.dev/gemini-api/docs/computer-use", - tzafon: "https://huggingface.co/Tzafon/Northstar-CUA-Fast", - yutoriN1: "https://docs.yutori.com/reference/n1", - yutoriN15: "https://docs.yutori.com/reference/n1-5", }); function normalized(range: readonly [number, number]): CuaCoordinateContract { @@ -435,58 +427,12 @@ function openaiNativeComputer(): CuaToolSpec { name: "computer", source: providerSources.openai, declaration, - binding: { kind: "openai-native", declaration }, + binding: { kind: "openai-native", declaration, requiresApi: OPENAI_CUA_COMPUTER_API }, toActions: mapOpenAIComputerInput, coordinates: pixels, }); } -function tzafonNativeComputer(options: { displayWidth?: number; displayHeight?: number } = {}): CuaToolSpec { - const declaration = { - type: "computer_use", - display_width: options.displayWidth, - display_height: options.displayHeight, - environment: "browser", - }; - return providerNativeSpec({ - identity: "provider.tzafon.native.computer.v1", - name: "computer", - source: providerSources.tzafon, - declaration, - binding: { kind: "tzafon-native", declaration }, - toActions(input) { - const action = asInput(input).action; - return toTzafonActions(action).filter((value): value is CuaAction => value.type !== "answer"); - }, - coordinates: normalized([0, 999]), - }); -} - -function yutoriToolset(generation: "n1" | "n15"): CuaToolSpec[] { - const names = generation === "n1" ? YUTORI_N1_ACTION_TYPES : YUTORI_N15_CORE_ACTION_TYPES; - return names.map((nativeName) => { - const identityName = nativeName.replaceAll("_", "-"); - const binding: CuaProviderBinding = { - kind: "yutori-native", - generation, - nativeName, - ...(generation === "n15" ? { toolSet: YUTORI_N15_CORE_TOOL_SET } : {}), - allNativeNames: names, - }; - return providerNativeSpec({ - identity: `provider.yutori.native.${generation}.${identityName}.${generation === "n15" ? "20260403" : "v1"}`, - name: nativeName, - source: generation === "n1" ? providerSources.yutoriN1 : providerSources.yutoriN15, - declaration: { type: "function", name: nativeName }, - binding, - toActions(input) { - return toYutoriActions(nativeName, asInput(input)) ?? []; - }, - coordinates: normalized([0, 1000]), - }); - }); -} - const GOOGLE_BROWSER_ACTIONS = [ "click", "double_click", "triple_click", "middle_click", "right_click", "mouse_down", "mouse_up", "move", "type", "drag_and_drop", "wait", "press_key", "key_down", "key_up", "hotkey", "take_screenshot", @@ -507,7 +453,7 @@ function googleBrowserToolset(options: GoogleBrowserToolsetOptions = {}): CuaToo name: nativeName, source: providerSources.google, declaration: { computerUse: { environment: "ENVIRONMENT_BROWSER" } }, - binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS }, + binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS, requiresApi: GOOGLE_CUA_INTERACTIONS_API }, toActions: (input) => mapGoogleAction(nativeName, asInput(input)), coordinates: normalized([0, 999]), })); @@ -900,11 +846,6 @@ const providers = Object.freeze({ source: providerSources.google, toolsets: Object.freeze({ browser: googleBrowserToolset }), }), - tzafon: Object.freeze({ source: providerSources.tzafon, tools: Object.freeze({ computer: tzafonNativeComputer }) }), - yutori: Object.freeze({ - sources: Object.freeze({ n1: providerSources.yutoriN1, n15Core: providerSources.yutoriN15 }), - toolsets: Object.freeze({ n1: () => yutoriToolset("n1"), n15Core: () => yutoriToolset("n15") }), - }), }); /** Frozen, discoverable tool namespace shared by @onkernel/cua-ai and @onkernel/cua-agent. */ diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index fca2e3c2..1dd48d67 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -4,16 +4,11 @@ export { createCuaModels, cuaModels, GOOGLE_CUA_INTERACTIONS_API, + OPENAI_CUA_COMPUTER_API, streamGoogleInteractions, streamOpenAIResponses, streamSimpleGoogleInteractions, streamSimpleOpenAIResponses, - streamSimpleTzafonResponses, - streamSimpleYutori, - streamTzafonResponses, - streamYutori, - TZAFON_RESPONSES_API, - YUTORI_CHAT_COMPLETIONS_API, } from "./providers"; export * from "./models"; export * from "./api-keys"; diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index cf167abd..e0230852 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -1,9 +1,8 @@ import type { Api, Model } from "@earendil-works/pi-ai"; import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; -import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; /** Providers with curated computer-use model support. */ -export type CuaProvider = "openai" | "anthropic" | "google" | "meta" | "xai" | "moonshotai" | "openrouter" | "tzafon" | "yutori"; +export type CuaProvider = "openai" | "anthropic" | "google" | "xai" | "moonshotai" | "openrouter"; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ export type CuaModelRef = `${CuaProvider}:${string}`; @@ -20,7 +19,7 @@ export interface CuaModelInfo { } /** All providers this package curates computer-use models for. */ -export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter", "tzafon", "yutori"]; +export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; /** * How a {@link CuaModelAnnotation} matches model ids. @@ -51,6 +50,14 @@ export interface CuaModelAnnotation { readonly capabilities?: CuaModelCapabilities; } +// Muse Spark accepts the full CUA schema set; OpenRouter's provider-level +// defaults are conservative because the proxy fronts many model families. +const MUSE_SPARK_CAPABILITIES: CuaModelCapabilities = Object.freeze({ + acceptsComplexSchemas: true, + acceptsLargeSchemas: true, + serializesStateMutations: true, +}); + const KIMI_K3_CAPABILITIES: CuaModelCapabilities = Object.freeze({ acceptsComplexSchemas: true, acceptsLargeSchemas: false, @@ -90,9 +97,6 @@ export const CUA_MODEL_ANNOTATIONS: Record so getCuaModel() can return it directly -// without synthesizing fields at call time. Add an entry here when a provider -// ships a new model before pi-ai picks it up — and add a matching annotation -// in CUA_MODEL_ANNOTATIONS above so the support filter recognizes it. -const CUA_MODEL_OVERRIDES: Record[]> = { - openai: [], - anthropic: [], - google: [], - // pi-ai still lacks Meta's models.dev catalog entry. - meta: [cuaModel("meta", "muse-spark-1.1", "Muse Spark 1.1")], - xai: [], - moonshotai: [], - openrouter: [], - tzafon: [ - cuaModel("tzafon", "tzafon.northstar-cua-fast", "Tzafon Northstar CUA Fast"), - cuaModel("tzafon", "tzafon.northstar-cua-fast-1.6", "Tzafon Northstar CUA Fast 1.6"), - cuaModel("tzafon", "tzafon.northstar-cua-fast-1.7-experiment", "Tzafon Northstar CUA Fast 1.7 (experiment)"), - ], - yutori: [ - cuaModel("yutori", "n1.5-latest", "Yutori Navigator n1.5"), - cuaModel("yutori", "n1.5-20260428", "Yutori Navigator n1.5 (2026-04-28)"), - cuaModel("yutori", "n1-latest", "Yutori Navigator n1"), - cuaModel("yutori", "n1-20260203", "Yutori Navigator n1 (2026-02-03)"), + { match: { kind: "exact", id: "meta/muse-spark-1.1" }, source: "https://openrouter.ai/meta/muse-spark-1.1", capabilities: MUSE_SPARK_CAPABILITIES }, ], }; -/** Models CUA supports that pi-ai's registry does not carry for a provider. */ -export function cuaOverrideModels(provider: CuaProvider): readonly Model[] { - return CUA_MODEL_OVERRIDES[provider]; -} - /** * Split a provider-qualified ref like `"openai:gpt-5.6-sol"` into its parts. * @@ -187,10 +149,6 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { const byRef = new Map(); for (const p of providers) { - for (const model of CUA_MODEL_OVERRIDES[p]) { - const ref = formatCuaModelRef(p, model.id); - byRef.set(ref, { ref, provider: p, model: model.id, name: model.name }); - } for (const model of getBuiltinModels(p as never) as Model[]) { if (!supportsCuaProvider(p, model.id)) continue; const ref = formatCuaModelRef(p, model.id); @@ -220,34 +178,10 @@ export function getCuaModel(ref: CuaModelRef): Model { throw new Error(`unsupported CUA model "${ref}"`); } const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; - if (fromRegistry) return routeCuaApi(fromRegistry); - const override = CUA_MODEL_OVERRIDES[provider].find((m) => m.id === modelId); - if (override) return routeCuaApi(override); - throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`); + if (fromRegistry) return fromRegistry; + throw new Error(`CUA model "${ref}" is supported but not carried by pi-ai's registry`); } -// Route CUA models to provider-specific transports. OpenAI keeps pi-ai's -// builtin "openai-responses" api and streams through its automatic prompt -// caching; the CUA adapter dispatches on request shape (see -// requiresCuaOpenAIAdapter), not on a rerouted api id. Other registry-resolved -// models otherwise carry pi-ai's builtin API ids too. -export function routeCuaApi(model: Model): Model { - if (model.provider === "google" && model.api !== GOOGLE_CUA_INTERACTIONS_API) { - return { ...model, api: GOOGLE_CUA_INTERACTIONS_API }; - } - if (model.provider === "xai" && model.id === "grok-4.5") { - return { - ...model, - thinkingLevelMap: { off: "low", minimal: "low", xhigh: "high" }, - cost: { - ...model.cost, - tiers: [{ inputTokensAbove: 200_000, input: 4, output: 12, cacheRead: 1, cacheWrite: 0 }], - }, - compat: { supportsDeveloperRole: false, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: false }, - }; - } - return model; -} /** Return the {@link CuaProvider} for a concrete model, or throw when it is not a CUA provider. */ export function providerForModel(model: Model): CuaProvider { @@ -270,11 +204,11 @@ function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean { export function cuaModelCapabilities(model: Model): CuaModelCapabilities { const annotation = isCuaProvider(model.provider) ? findCuaAnnotation(model.provider, model.id) : undefined; if (annotation?.capabilities) return annotation.capabilities; - const acceptsComplexSchemas = ["openai", "anthropic", "meta", "xai", "moonshotai"].includes(model.provider); + const acceptsComplexSchemas = ["openai", "anthropic", "xai", "moonshotai"].includes(model.provider); return { acceptsComplexSchemas, acceptsLargeSchemas: acceptsComplexSchemas && model.provider !== "moonshotai", - serializesStateMutations: ["meta", "xai", "moonshotai"].includes(model.provider), + serializesStateMutations: ["xai", "moonshotai"].includes(model.provider), }; } @@ -305,37 +239,6 @@ function isCuaFamilyMatch(id: string, family: string): boolean { .every((segment) => /^\d+$/.test(segment)); } -function cuaModel(provider: "meta" | "tzafon" | "yutori", id: string, name: string): Model { - const base = { - id, - name, - provider, - reasoning: provider === "meta", - input: ["text", "image"], - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - } satisfies Partial>; - - switch (provider) { - case "meta": - // Meta documents the 1,048,576-token context window, and its - // computer-use cookbook configures 128,000 maximum output tokens. - return { - ...base, - api: "openai-responses", - baseUrl: "https://api.meta.ai/v1", - thinkingLevelMap: { off: null, xhigh: "xhigh" }, - cost: { input: 1.25, output: 4.25, cacheRead: 0.15, cacheWrite: 0 }, - contextWindow: 1_048_576, - maxTokens: 128_000, - compat: { supportsDeveloperRole: true, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: true }, - } as Model; - case "tzafon": - return { ...base, api: "tzafon-responses", baseUrl: "https://api.tzafon.ai", contextWindow: 128_000, maxTokens: 4_096 } as Model; - case "yutori": - return { ...base, api: "yutori-chat-completions", baseUrl: "https://api.yutori.com/v1", contextWindow: 128_000, maxTokens: 4_096 } as Model; - } -} - function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { if (a.provider !== b.provider) return CUA_PROVIDERS.indexOf(a.provider) - CUA_PROVIDERS.indexOf(b.provider); return a.model.localeCompare(b.model); diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index fd6de719..3b19e653 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -10,18 +10,11 @@ import { type SimpleStreamOptions, type StreamOptions, } from "@earendil-works/pi-ai"; -import { - stream as piStreamOpenAIResponses, - streamSimple as piStreamSimpleOpenAIResponses, -} from "@earendil-works/pi-ai/api/openai-responses"; import { builtinModels } from "@earendil-works/pi-ai/providers/all"; import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; -import { cuaOverrideModels } from "./models"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; -import { requiresCuaOpenAIAdapter, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; -import { streamSimpleTzafonResponses, streamTzafonResponses, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; -import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; +import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAICuaComputer, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; /** * Build the pi `Models` collection CUA streams through: pi's builtin @@ -29,20 +22,24 @@ import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from ". * * - `anthropic` retries an inaccessible native browser beta through the * selected tool's equivalent function declaration. - * - `openai` streams through pi's builtin `openai-responses` transport and - * its automatic prompt caching by default. The CUA adapter only intercepts - * requests that need it: OpenAI's native computer tool, or a transcript - * carrying a deferred tool-search addition or a replayed function-call - * namespace (see {@link requiresCuaOpenAIAdapter}). - * - `google` intercepts `google-cua-interactions` for current native computer - * use and resolves API keys from `GOOGLE_API_KEY` or `GEMINI_API_KEY`. + * - `openai` streams through pi's builtin `openai-responses` transport and its + * automatic prompt caching by default; a model compiled with OpenAI's native + * computer tool carries `openai-cua-computer` instead, which this wrapper + * routes to the CUA adapter. The one dispatch that cannot be derived from + * `model.api` is a transcript carrying a deferred tool-search addition or a + * replayed function-call namespace (see {@link requiresCuaOpenAINamespaceAdapter}). + * - `google` intercepts `google-cua-interactions` — carried only by a model + * compiled with Google's native computer-use toolset — for current native + * computer use, and resolves API keys from `GOOGLE_API_KEY` or `GEMINI_API_KEY`. + * A Google model compiled without that toolset streams through pi's builtin + * Google transport instead. * - `xai` is pi's builtin provider untouched: Grok streams through pi's * Responses transport, and the catalog supplies its serial-tool-call field. * - `moonshotai` is pi's builtin provider untouched: Kimi streams through the * plain OpenAI-compatible chat completions transport with `MOONSHOT_API_KEY`. - * - `meta`, `tzafon`, and `yutori` are CUA-only providers pi does not ship. - * `meta` speaks the OpenAI Responses wire protocol, so it registers pi's - * builtin transport against Meta's base URL and credentials. + * - `meta` is a CUA-only provider pi does not ship. It speaks the OpenAI + * Responses wire protocol, so it registers pi's builtin transport against + * Meta's base URL and credentials. * * Each call returns an independent collection; register additional providers * or credentials on it freely. Use {@link cuaModels} for the shared default. @@ -52,12 +49,9 @@ export function createCuaModels(options?: CreateModelsOptions): MutableModels { const anthropic = models.getProvider("anthropic"); if (anthropic) models.setProvider(withAnthropicBrowserFallback(anthropic)); const openai = models.getProvider("openai"); - if (openai) models.setProvider(withOpenAICuaComputerAdapter(openai)); + if (openai) models.setProvider(withOpenAICuaAdapter(openai)); const google = models.getProvider("google"); if (google) models.setProvider(withGoogleCuaInteractions(google)); - models.setProvider(metaProvider()); - models.setProvider(tzafonProvider()); - models.setProvider(yutoriProvider()); return models; } @@ -75,20 +69,25 @@ export function cuaModels(): MutableModels { return (defaultCuaModels ??= createCuaModels()); } -// OpenAI models keep pi-ai's builtin "openai-responses" api id. Only requests -// that need cua-ai's adapter (see requiresCuaOpenAIAdapter) are intercepted; -// everything else falls through to pi's builtin provider. -function withOpenAICuaComputerAdapter(base: Provider): Provider { +// The compiled catalog's model.api decides dispatch: OPENAI_CUA_COMPUTER_API +// routes to the CUA adapter, everything else falls through to pi's builtin +// "openai-responses" provider. requiresCuaOpenAINamespaceAdapter is the one +// exception that cannot be derived from the model — see its doc comment. +function withOpenAICuaAdapter(base: Provider): Provider { return { ...base, stream: (model: Model, context: Context, options?: StreamOptions) => - model.api === "openai-responses" && requiresCuaOpenAIAdapter(context, options) - ? streamOpenAIResponses(model as never, context, options) - : base.stream(model, context, options), + model.api === OPENAI_CUA_COMPUTER_API + ? streamOpenAICuaComputer(model as never, context, options) + : requiresCuaOpenAINamespaceAdapter(context) + ? streamOpenAIResponses(model as never, context, options) + : base.stream(model, context, options), streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => - model.api === "openai-responses" && requiresCuaOpenAIAdapter(context, options) - ? streamSimpleOpenAIResponses(model as never, context, options) - : base.streamSimple(model, context, options), + model.api === OPENAI_CUA_COMPUTER_API + ? streamOpenAICuaComputer(model as never, context, options) + : requiresCuaOpenAINamespaceAdapter(context) + ? streamSimpleOpenAIResponses(model as never, context, options) + : base.streamSimple(model, context, options), }; } @@ -108,40 +107,6 @@ function withGoogleCuaInteractions(base: Provider): Provider { } -function metaProvider(): Provider { - return createProvider({ - id: "meta", - name: "Meta", - baseUrl: "https://api.meta.ai/v1", - auth: { apiKey: envApiKeyAuth("Meta Model API key", cuaApiKeyEnvVarsForProvider("meta")) }, - models: cuaOverrideModels("meta"), - api: { stream: piStreamOpenAIResponses, streamSimple: piStreamSimpleOpenAIResponses }, - }); -} - -function tzafonProvider(): Provider { - return createProvider({ - id: "tzafon", - name: "Tzafon", - baseUrl: "https://api.tzafon.ai", - auth: { apiKey: envApiKeyAuth("Tzafon API key", cuaApiKeyEnvVarsForProvider("tzafon")) }, - models: cuaOverrideModels("tzafon"), - api: { stream: streamTzafonResponses, streamSimple: streamSimpleTzafonResponses }, - }); -} - -function yutoriProvider(): Provider { - return createProvider({ - id: "yutori", - name: "Yutori", - baseUrl: "https://api.yutori.com/v1", - auth: { apiKey: envApiKeyAuth("Yutori API key", cuaApiKeyEnvVarsForProvider("yutori")) }, - models: cuaOverrideModels("yutori"), - api: { stream: streamYutori, streamSimple: streamSimpleYutori }, - }); -} export { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; -export { streamOpenAIResponses, streamSimpleOpenAIResponses }; -export { TZAFON_RESPONSES_API, streamSimpleTzafonResponses, streamTzafonResponses }; -export { YUTORI_CHAT_COMPLETIONS_API, streamSimpleYutori, streamYutori }; +export { OPENAI_CUA_COMPUTER_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts index 4dc8b7f1..4facab45 100644 --- a/packages/ai/src/providers/common.ts +++ b/packages/ai/src/providers/common.ts @@ -7,20 +7,8 @@ import type { SimpleStreamOptions, StreamOptions, } from "@earendil-works/pi-ai"; -import type { CuaAction } from "../actions/index"; import type { CuaIncomingToolPlan } from "../tool-catalog"; -/** Return the canonical function-tool name for an action. */ -export function canonicalToolCallName(action: CuaAction): CuaAction["type"] { - return action.type; -} - -/** Convert a canonical action to function-tool arguments. */ -export function canonicalToolCallArguments(action: CuaAction): Record { - const { type: _type, ...args } = action as CuaAction & Record; - return args; -} - /** Prefix a bare hostname/path before browser navigation. */ export function normalizeGotoUrl(value: unknown): string | undefined { if (typeof value !== "string") return undefined; diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/ai/src/providers/openai/provider.ts index 78c6ec6e..96a2ab0c 100644 --- a/packages/ai/src/providers/openai/provider.ts +++ b/packages/ai/src/providers/openai/provider.ts @@ -25,6 +25,9 @@ import { buildBaseOptions } from "@earendil-works/pi-ai/api/simple-options"; import type { CuaIncomingToolPlan } from "../../tool-catalog"; import type { CuaSimpleStreamOptions } from "../common"; +/** CUA-owned api id for OpenAI's native computer tool, derived onto the model by compileCuaToolCatalog when that tool is selected. */ +export const OPENAI_CUA_COMPUTER_API = "openai-cua-computer"; + export interface OpenAIResponsesOptions extends PiOpenAIResponsesOptions { /** @internal Identity-addressed native dispatch compiled from selected tools. */ cuaIncomingToolPlan?: CuaIncomingToolPlan; @@ -34,16 +37,14 @@ export interface OpenAIResponsesOptions extends PiOpenAIResponsesOptions { type OpenAIRequestOptions = Pick; /** - * Whether a request needs cua-ai's OpenAI Responses adapter instead of pi-ai's - * builtin `openai-responses` transport: OpenAI's native computer tool is - * selected, or the transcript carries state (a deferred tool-search addition, - * or a replayed function-call namespace) that only this adapter round-trips. + * The one dispatch cua-ai's OpenAI provider wrapper cannot derive from the + * model's `api`: whether the transcript carries state (a deferred tool-search + * addition, or a replayed function-call namespace) that only this adapter + * round-trips, because pi-ai's builtin `openai-responses` transport does not + * parse or replay `function_call`'s `namespace` field. Every other dispatch — + * including OpenAI's native computer tool — is decided by `model.api` instead. */ -export function requiresCuaOpenAIAdapter(context: Context, options?: unknown): boolean { - // pi's Provider.stream signature hides cua's added option, so the plan is - // read structurally here rather than at both call sites. - const cuaIncomingToolPlan = (options as { cuaIncomingToolPlan?: CuaIncomingToolPlan } | undefined)?.cuaIncomingToolPlan; - if (cuaIncomingToolPlan?.openaiComputerName) return true; +export function requiresCuaOpenAINamespaceAdapter(context: Context): boolean { for (const message of context.messages) { if (message.role === "toolResult" && (message.addedToolNames?.length ?? 0) > 0) return true; if (message.role === "assistant" && message.content.some((part) => part.type === "toolCall" && toolCallNamespace(part))) return true; @@ -51,13 +52,10 @@ export function requiresCuaOpenAIAdapter(context: Context, options?: unknown): b return false; } -export const streamOpenAIResponses: StreamFunction<"openai-responses", OpenAIResponsesOptions> = (model, context, options) => { - if (options?.cuaIncomingToolPlan?.openaiComputerName) return streamOpenAINativeComputer(model, context, options); - return streamOpenAIFunctionTools(model, context, options); -}; +export const streamOpenAIResponses: StreamFunction<"openai-responses", OpenAIResponsesOptions> = (model, context, options) => + streamOpenAIFunctionTools(model, context, options); export const streamSimpleOpenAIResponses: StreamFunction<"openai-responses", CuaSimpleStreamOptions> = (model, context, options) => { - if (options?.cuaIncomingToolPlan?.openaiComputerName) return streamOpenAINativeComputer(model, context, options); const base = buildBaseOptions(model, context, options, options?.apiKey); const clampedReasoning = options?.reasoning ? clampThinkingLevel(model, options.reasoning) : undefined; return streamOpenAIFunctionTools(model, context, { @@ -282,7 +280,10 @@ function applyServiceTierPricing( usage.cost.total = usage.cost.input + usage.cost.output + usage.cost.cacheRead + usage.cost.cacheWrite; } -/** Responses adapter used only when the selected catalog contains OpenAI's native computer tool. */ +/** Responses adapter for models the catalog compiled onto {@link OPENAI_CUA_COMPUTER_API}, i.e. those whose selected tools include OpenAI's native computer. */ +export const streamOpenAICuaComputer: StreamFunction = (model, context, options) => + streamOpenAINativeComputer(model as unknown as Model<"openai-responses">, context, options); + function streamOpenAINativeComputer( model: Model<"openai-responses">, context: Context, diff --git a/packages/ai/src/providers/tzafon/provider.ts b/packages/ai/src/providers/tzafon/provider.ts deleted file mode 100644 index 823c70e9..00000000 --- a/packages/ai/src/providers/tzafon/provider.ts +++ /dev/null @@ -1,487 +0,0 @@ -import { - createAssistantMessageEventStream, - type Api, - type AssistantMessage, - type Context, - type ImageContent, - type Message, - type Model, - type StreamFunction, - type StreamOptions, - type TextContent, - type Tool, - type ToolCall, -} from "@earendil-works/pi-ai"; -import Lightcone from "@tzafon/lightcone"; -import type { CuaAction } from "../../actions/index"; -import { - canonicalToolCallArguments, - canonicalToolCallName, - type CuaSimpleStreamOptions, - responseThreadingDelta, - responseThreadingEnabled, - type ResponseThreadingOptions, -} from "../common"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; - -export const TZAFON_RESPONSES_API = "tzafon-responses"; - -/** Stream options accepted by {@link streamTzafonResponses}. */ -export interface TzafonResponsesOptions extends StreamOptions, ResponseThreadingOptions { - /** @internal Identity-addressed native dispatch compiled from selected tools. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -/** Inputs {@link buildTzafonRequestInput} reads to shape the Responses API request body. */ -export interface TzafonRequestOptions extends ResponseThreadingOptions { - temperature?: number; - maxTokens?: number; - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -/** Responses API request body for {@link Lightcone.responses.create}, including optional threading fields. */ -export interface TzafonRequestBody { - model: string; - input: Array>; - tools: Array>; - instructions?: string; - temperature: number; - max_output_tokens?: number; - previous_response_id?: string; - store?: boolean; -} - -/** - * Build the Tzafon Responses API request body from a context. - * - * Pure and network-free. When response threading is enabled and a prior - * assistant `responseId` exists, the body chains via `previous_response_id` - * with `store: true` and sends only the delta messages; otherwise it replays - * the full message history. - */ -export function buildTzafonRequestInput(model: Model, context: Context, options?: TzafonRequestOptions): TzafonRequestBody { - const body: TzafonRequestBody = { - model: model.id, - input: convertMessages(context.messages, options?.cuaIncomingToolPlan?.tzafonComputerName), - tools: convertTools(context.tools ?? []), - instructions: context.systemPrompt, - temperature: options?.temperature ?? 0, - max_output_tokens: options?.maxTokens ?? model.maxTokens, - }; - if (!responseThreadingEnabled(options)) return body; - const { previousResponseId, deltaMessages } = responseThreadingDelta(context.messages, TZAFON_RESPONSES_API); - if (!previousResponseId) return body; - return { ...body, input: convertMessages(deltaMessages, options?.cuaIncomingToolPlan?.tzafonComputerName), previous_response_id: previousResponseId, store: true }; -} - -export const streamSimpleTzafonResponses: StreamFunction = (model, context, options) => { - return streamTzafonResponses(model, context, options); -}; - -export const streamTzafonResponses: StreamFunction = (model, context, options) => { - const stream = createAssistantMessageEventStream(); - const output = initialAssistantMessage(model); - - void (async () => { - try { - const apiKey = options?.apiKey || process.env.TZAFON_API_KEY; - if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`); - const client = new Lightcone({ - apiKey, - baseURL: model.baseUrl, - defaultHeaders: { ...model.headers, ...options?.headers }, - }); - const payload = buildTzafonRequestInput(model as Model, context, options); - const nextPayload = await options?.onPayload?.(payload, model as Model); - if (options?.signal?.aborted) throw new Error("Request was aborted"); - const response = await client.responses.create((nextPayload ?? payload) as never, { - signal: options?.signal, - }); - if (options?.signal?.aborted) throw new Error("Request was aborted"); - - stream.push({ type: "start", partial: output }); - output.responseId = getString(response, "id") || undefined; - output.usage = usageFromTzafon(getValue(response, "usage")); - for (const item of getArray(response, "output")) { - const type = getString(item, "type"); - if (type === "message") { - const text = extractMessageText(item); - if (text) emitText(stream, output, text); - continue; - } - if (type === "function_call") { - emitToolCall(stream, output, { - type: "toolCall", - id: getString(item, "call_id"), - name: getString(item, "name"), - arguments: parseArguments(getValue(item, "arguments")), - }); - continue; - } - if (type === "computer_call") { - const callId = getString(item, "call_id") || getString(item, "id") || `computer_call_${output.content.length}`; - const rawAction = getValue(item, "action"); - const canonical = toCanonicalActions(rawAction); - for (const action of canonical) if (action.type === "answer") emitText(stream, output, action.text); - const executable = canonical.filter((action): action is CuaAction => action.type !== "answer"); - const nativeName = options?.cuaIncomingToolPlan?.tzafonComputerName; - const textOnlyAction = executable.find((action) => action.type !== "screenshot"); - if (nativeName && textOnlyAction) { - throw new Error( - `Tzafon native computer action "${textOnlyAction.type}" is unsupported: its action loop requires an automatic post-action screenshot, and CUA returns screenshots only when explicitly requested.`, - ); - } - if (nativeName && executable.length > 0) { - emitToolCall(stream, output, { type: "toolCall", id: callId, name: nativeName, arguments: { action: rawAction } }); - continue; - } - for (let actionIndex = 0; actionIndex < executable.length; actionIndex += 1) { - const action = executable[actionIndex]!; - emitToolCall(stream, output, { - type: "toolCall", - id: tzafonToolCallId(callId, actionIndex), - name: canonicalToolCallName(action), - arguments: canonicalToolCallArguments(action), - }); - } - } - } - - output.stopReason = output.content.some((part) => part.type === "toolCall") ? "toolUse" : "stop"; - stream.push({ type: "done", reason: output.stopReason, message: output }); - stream.end(); - } catch (err) { - output.stopReason = options?.signal?.aborted ? "aborted" : "error"; - output.errorMessage = err instanceof Error ? err.message : String(err); - stream.push({ type: "error", reason: output.stopReason, error: output }); - stream.end(); - } - })(); - - return stream; -}; - -/** Derive a unique canonical tool-call id for a Tzafon computer action. */ -export function tzafonToolCallId(callId: string, actionIndex: number): string { - return actionIndex === 0 ? callId : `${callId}:${actionIndex}`; -} - -function initialAssistantMessage(model: Model): AssistantMessage { - return { - role: "assistant", - content: [], - api: model.api, - provider: model.provider, - model: model.id, - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: Date.now(), - }; -} - -function emitText(stream: ReturnType, output: AssistantMessage, text: string): void { - const contentIndex = output.content.length; - const content: TextContent = { type: "text", text }; - output.content.push(content); - stream.push({ type: "text_start", contentIndex, partial: output }); - stream.push({ type: "text_delta", contentIndex, delta: text, partial: output }); - stream.push({ type: "text_end", contentIndex, content: text, partial: output }); -} - -function emitToolCall( - stream: ReturnType, - output: AssistantMessage, - toolCall: ToolCall, -): void { - const contentIndex = output.content.length; - output.content.push(toolCall); - stream.push({ type: "toolcall_start", contentIndex, partial: output }); - stream.push({ type: "toolcall_end", contentIndex, toolCall, partial: output }); -} - -/** A canonical CUA action, or the terminal `answer` text Tzafon emits when it is done. */ -export type TzafonCanonicalAction = CuaAction | { type: "answer"; text: string }; - -/** Normalize one Tzafon `computer_call.action` payload into canonical CUA actions. */ -export function toCanonicalActions(action: unknown): TzafonCanonicalAction[] { - if (!action || typeof action !== "object") return []; - const current = action as Record; - const type = getString(current, "type"); - const x = readOptionalNumber(current, "x"); - const y = readOptionalNumber(current, "y"); - switch (type) { - case "click": - case "left_click": - return x !== undefined && y !== undefined ? [{ type: "click", x, y }] : []; - case "right_click": - return x !== undefined && y !== undefined ? [{ type: "click", x, y, button: "right" }] : []; - case "double_click": - return x !== undefined && y !== undefined ? [{ type: "double_click", x, y }] : []; - case "triple_click": - return x !== undefined && y !== undefined ? [{ type: "double_click", x, y }, { type: "click", x, y }] : []; - case "move": - case "hover": - return x !== undefined && y !== undefined ? [{ type: "move", x, y }] : []; - case "drag": - return toDragAction(current); - case "type": - return [{ type: "type", text: getString(current, "text") }]; - case "keypress": - case "key": - return toKeypressAction(current); - case "scroll": - return [toScrollAction(current)]; - case "hscroll": - return [{ type: "scroll", scroll_x: readOptionalNumber(current, "scroll_x") ?? readOptionalNumber(current, "amount") ?? 0 }]; - case "navigate": - return [{ type: "goto", url: getString(current, "url") }]; - case "wait": - return [{ type: "wait", ms: readOptionalNumber(current, "ms") ?? secondsToMs(readOptionalNumber(current, "seconds")) }]; - case "screenshot": - return [{ type: "screenshot" }]; - case "answer": - case "done": - case "terminate": - return [{ type: "answer", text: getString(current, "result") || getString(current, "text") || getString(current, "status") }]; - default: - return []; - } -} - -function toDragAction(action: Record): CuaAction[] { - const path = getArray(action, "path") - .map((point) => { - if (!point || typeof point !== "object") return undefined; - const x = readOptionalNumber(point, "x"); - const y = readOptionalNumber(point, "y"); - return x !== undefined && y !== undefined ? { x, y } : undefined; - }) - .filter((point): point is { x: number; y: number } => Boolean(point)); - if (path.length >= 2) return [{ type: "drag", path }]; - - const x = readOptionalNumber(action, "x"); - const y = readOptionalNumber(action, "y"); - const endX = readOptionalNumber(action, "end_x") ?? readOptionalNumber(action, "x2"); - const endY = readOptionalNumber(action, "end_y") ?? readOptionalNumber(action, "y2"); - if (x === undefined || y === undefined || endX === undefined || endY === undefined) return []; - return [{ type: "drag", path: [{ x, y }, { x: endX, y: endY }] }]; -} - -function toKeypressAction(action: Record): CuaAction[] { - const keys = getArray(action, "keys") - .map((key) => (typeof key === "string" ? key : undefined)) - .filter((key): key is string => Boolean(key)); - const key = getString(action, "key"); - const text = getString(action, "text"); - const value = keys.length > 0 ? keys : key ? [key] : text ? [text] : []; - return value.length > 0 ? [{ type: "keypress", keys: value }] : []; -} - -function toScrollAction(action: Record): CuaAction { - return { - type: "scroll", - x: readOptionalNumber(action, "x"), - y: readOptionalNumber(action, "y"), - scroll_x: readOptionalNumber(action, "scroll_x"), - scroll_y: readOptionalNumber(action, "scroll_y") ?? readOptionalNumber(action, "amount"), - }; -} - -function secondsToMs(seconds: number | undefined): number | undefined { - return seconds === undefined ? undefined : seconds * 1000; -} - -function convertTools(tools: Tool[]): Array> { - return tools.map((tool) => ({ - type: "function", - name: tool.name, - description: tool.description, - parameters: tool.parameters, - })); -} - -function convertMessages(messages: readonly Message[], nativeComputerName?: string): Array> { - const items: Array> = []; - for (const message of messages) { - if (message.role === "user") { - items.push({ role: "user", content: convertUserContent(message.content) }); - continue; - } - if (message.role === "assistant") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join("\n") - .trim(); - if (text) items.push({ role: "assistant", content: text }); - for (const part of message.content) { - if (part.type !== "toolCall") continue; - if (nativeComputerName && part.name === nativeComputerName) { - items.push({ type: "computer_call", call_id: part.id, action: getValue(part.arguments, "action") }); - } else { - items.push({ - type: "function_call", - call_id: part.id, - name: part.name, - arguments: JSON.stringify(part.arguments ?? {}), - }); - } - } - continue; - } - if (message.role === "toolResult") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join("\n") - .trim(); - const image = [...message.content].reverse().find((part): part is ImageContent => part.type === "image"); - if (nativeComputerName && message.toolName === nativeComputerName) { - if (!image) { - throw new Error( - "Tzafon native computer action loops require image tool results; text-only results are unsupported because CUA does not capture post-action screenshots automatically.", - ); - } - items.push({ - type: "computer_call_output", - call_id: message.toolCallId, - output: { type: "computer_screenshot", image_url: `data:${image.mimeType};base64,${image.data}` }, - }); - continue; - } - items.push({ - type: "function_call_output", - call_id: message.toolCallId, - output: message.isError ? `Error: ${text || "tool execution failed"}` : text || "ok", - }); - if (image) { - items.push({ - role: "user", - content: [ - { type: "input_text", text: "screenshot" }, - { type: "input_image", image_url: `data:${image.mimeType};base64,${image.data}`, detail: "auto" }, - ], - }); - } - } - } - return items; -} - -function convertUserContent(content: string | (TextContent | ImageContent)[]): unknown { - if (typeof content === "string") return [{ type: "input_text", text: content }]; - return content.map((part) => { - if (part.type === "text") return { type: "input_text", text: part.text }; - return { type: "input_image", image_url: `data:${part.mimeType};base64,${part.data}`, detail: "auto" }; - }); -} - -function extractMessageText(item: unknown): string { - return getArray(item, "content") - .map((block) => getString(block, "text")) - .filter(Boolean) - .join("\n") - .trim(); -} - -function parseArguments(value: unknown): Record { - const top = - typeof value === "string" && value.trim() - ? safeJsonParse(value) - : value && typeof value === "object" - ? (value as Record) - : {}; - if (!top || typeof top !== "object") return {}; - // Tzafon sometimes nests JSON-encoded arrays/objects inside the top-level argument object - // (observed: { "actions": "[{...}]" }). Unwrap one level so consumers get real values. - const out: Record = {}; - for (const [key, val] of Object.entries(top)) { - out[key] = normalizeArgumentValue(key, val); - } - return out; -} - -const NUMERIC_ARGUMENT_KEYS = new Set(["x", "y", "scroll_x", "scroll_y", "ms", "duration"]); - -function normalizeArgumentValue(key: string, value: unknown): unknown { - const parsed = typeof value === "string" && looksLikeJson(value) ? safeJsonParse(value) ?? value : value; - if (typeof parsed === "string" && NUMERIC_ARGUMENT_KEYS.has(key)) { - const number = Number.parseFloat(parsed); - return Number.isFinite(number) ? number : parsed; - } - if (Array.isArray(parsed)) { - return parsed.map((item) => normalizeArgumentValue(key, item)); - } - if (parsed && typeof parsed === "object") { - return Object.fromEntries( - Object.entries(parsed).map(([childKey, childValue]) => [childKey, normalizeArgumentValue(childKey, childValue)]), - ); - } - return parsed; -} - -function safeJsonParse(value: string): Record | unknown[] | null { - try { - const parsed = JSON.parse(value); - return parsed && typeof parsed === "object" ? parsed : null; - } catch { - return null; - } -} - -function looksLikeJson(value: string): boolean { - const trimmed = value.trim(); - return trimmed.startsWith("[") || trimmed.startsWith("{"); -} - -function usageFromTzafon(usage: unknown): AssistantMessage["usage"] { - const input = readUsageNumber(usage, "input_tokens"); - const output = readUsageNumber(usage, "output_tokens"); - const cacheRead = readUsageNumber(getValue(usage, "input_tokens_details"), "cached_tokens"); - const totalTokens = readUsageNumber(usage, "total_tokens") || input + output; - return { - input, - output, - cacheRead, - cacheWrite: 0, - totalTokens, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }; -} - -function readUsageNumber(obj: unknown, key: string): number { - return readOptionalNumber(obj, key) ?? 0; -} - -function readOptionalNumber(obj: unknown, key: string): number | undefined { - if (!obj || typeof obj !== "object") return undefined; - const value = (obj as Record)[key]; - if (typeof value === "number" && Number.isFinite(value)) return value; - if (typeof value === "string" && value.trim()) { - const number = Number(value); - return Number.isFinite(number) ? number : undefined; - } - return undefined; -} - -function getArray(obj: unknown, key: string): unknown[] { - const value = getValue(obj, key); - return Array.isArray(value) ? value : []; -} - -function getString(obj: unknown, key: string): string { - const value = getValue(obj, key); - return typeof value === "string" ? value : ""; -} - -function getValue(obj: unknown, key: string): unknown { - if (!obj || typeof obj !== "object") return undefined; - return (obj as Record)[key]; -} diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts deleted file mode 100644 index 9c40ae58..00000000 --- a/packages/ai/src/providers/yutori/actions.ts +++ /dev/null @@ -1,237 +0,0 @@ -import type { CuaAction, CuaActionType } from "../../actions/index"; -import { normalizeGotoUrl } from "../common"; - -/** - * Native Yutori Navigator n1.5 tool-set ids. - * - * Source of truth: - * - https://docs.yutori.com/reference/n1-5 - * - https://docs.yutori.com/llm-quickstart.md - */ -export const YUTORI_N15_CORE_TOOL_SET = "browser_tools_core-20260403"; -export const YUTORI_N15_EXPANDED_TOOL_SET = "browser_tools_expanded-20260403"; - -/** - * DOM/ref-backed Navigator n1.5 actions. We intentionally disable these until - * CuaAgent has the ref/DOM execution path that Yutori documents for the - * expanded tool set. - */ -export const YUTORI_N15_EXPANDED_ACTION_TYPES = [ - "extract_elements", - "find", - "set_element_value", - "execute_js", -] as const; - -/** - * Navigator n1's fixed legacy browser action space. - * - * Source of truth: https://docs.yutori.com/reference/n1 - */ -export const YUTORI_N1_ACTION_TYPES = [ - "left_click", - "double_click", - "right_click", - "triple_click", - "type", - "key_press", - "scroll", - "hover", - "drag", - "goto_url", - "go_back", - "refresh", - "wait", -] as const; - -/** - * Navigator n1.5 core visual action space. These are the actions available - * when `tool_set` is `browser_tools_core-20260403`, which keeps CuaAgent in the - * pure screenshot/coordinate path and avoids DOM refs. - * - * Source of truth: https://docs.yutori.com/reference/n1-5 - */ -export const YUTORI_N15_CORE_ACTION_TYPES = [ - "left_click", - "double_click", - "triple_click", - "middle_click", - "right_click", - "mouse_move", - "mouse_down", - "mouse_up", - "drag", - "scroll", - "type", - "key_press", - "hold_key", - "goto_url", - "go_back", - "go_forward", - "refresh", - "wait", -] as const; - -export const YUTORI_N15_ACTION_TYPES = [ - ...YUTORI_N15_CORE_ACTION_TYPES, - ...YUTORI_N15_EXPANDED_ACTION_TYPES, -] as const; - -/** - * Canonical CUA action types Yutori's native actions normalize into. These are - * the tool-call names {@link streamYutori} emits and the local executors - * CuaAgent installs for Yutori models. - */ -export const YUTORI_CUA_ACTION_TYPES = [ - "click", - "double_click", - "mouse_down", - "mouse_up", - "type", - "keypress", - "scroll", - "move", - "drag", - "wait", - "goto", - "back", - "forward", -] as const satisfies readonly CuaActionType[]; - -type YutoriCanonicalActionType = (typeof YUTORI_CUA_ACTION_TYPES)[number]; - -/** Canonical CUA action shape emitted for Yutori models. */ -export type YutoriAction = Extract; - -export type YutoriN1ActionType = (typeof YUTORI_N1_ACTION_TYPES)[number]; -export type YutoriN15CoreActionType = (typeof YUTORI_N15_CORE_ACTION_TYPES)[number]; -export type YutoriN15ExpandedActionType = (typeof YUTORI_N15_EXPANDED_ACTION_TYPES)[number]; -export type YutoriNativeActionType = YutoriN1ActionType | YutoriN15CoreActionType | YutoriN15ExpandedActionType; - -const DEFAULT_SCROLL_AMOUNT = 3; -const SCROLL_AMOUNT_PER_NOTCH = 120; -const DEFAULT_WAIT_MS = 2000; -const NAVIGATION_WAIT_MS = 1500; -const GOTO_WAIT_MS = 2000; - -export function toCanonicalActions(name: string, args: Record): CuaAction[] | undefined { - const coords = readPoint(args.coordinates); - switch (name) { - case "left_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "right_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, button: "right", ...holdKeys(args.modifier) }] : undefined; - case "middle_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, button: "middle", ...holdKeys(args.modifier) }] : undefined; - case "double_click": - return coords ? [{ type: "double_click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "triple_click": - return coords - ? [ - { type: "double_click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }, - { type: "click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }, - ] - : undefined; - case "mouse_move": - case "hover": - return coords ? [{ type: "move", x: coords.x, y: coords.y }] : undefined; - case "mouse_down": - return coords ? [{ type: "mouse_down", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "mouse_up": - return coords ? [{ type: "mouse_up", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "drag": { - const start = readPoint(args.start_coordinates); - return start && coords ? [{ type: "drag", path: [start, coords], button: "left" }] : undefined; - } - case "scroll": - return toScrollAction(args, coords); - case "type": - return toTypeActions(args); - case "key_press": - return toKeypressAction(args); - case "hold_key": - return toHoldKeyAction(args); - case "goto_url": { - const url = normalizeGotoUrl(args.url); - return url ? [{ type: "goto", url }, { type: "wait", ms: GOTO_WAIT_MS }] : undefined; - } - case "go_back": - return [{ type: "back" }, { type: "wait", ms: NAVIGATION_WAIT_MS }]; - case "go_forward": - return [{ type: "forward" }, { type: "wait", ms: NAVIGATION_WAIT_MS }]; - case "refresh": - return [{ type: "keypress", keys: ["f5"] }, { type: "wait", ms: DEFAULT_WAIT_MS }]; - case "wait": - return [{ type: "wait", ms: secondsToMs(args.duration, DEFAULT_WAIT_MS) }]; - default: - return undefined; - } -} - -function readPoint(value: unknown): { x: number; y: number } | undefined { - if (!Array.isArray(value) || value.length < 2) return undefined; - const x = Number(value[0]); - const y = Number(value[1]); - if (!Number.isFinite(x) || !Number.isFinite(y)) return undefined; - return { x, y }; -} - -function toScrollAction(args: Record, coords: { x: number; y: number } | undefined): CuaAction[] | undefined { - if (!coords) return undefined; - const direction = typeof args.direction === "string" ? args.direction : "down"; - const amount = typeof args.amount === "number" ? args.amount : DEFAULT_SCROLL_AMOUNT; - const ticks = Math.max(1, Math.trunc(amount)) * SCROLL_AMOUNT_PER_NOTCH; - const scroll_x = direction === "left" ? -ticks : direction === "right" ? ticks : 0; - const scroll_y = direction === "up" ? -ticks : direction === "down" ? ticks : 0; - return [{ type: "scroll", x: coords.x, y: coords.y, scroll_x, scroll_y, ...holdKeys(args.modifier) }]; -} - -function toTypeActions(args: Record): CuaAction[] | undefined { - const text = typeof args.text === "string" ? args.text : undefined; - if (text === undefined) return undefined; - const actions: CuaAction[] = []; - if (args.clear_before_typing === true) { - actions.push({ type: "keypress", keys: ["ctrl", "a"] }, { type: "keypress", keys: ["backspace"] }); - } - actions.push({ type: "type", text }); - if (args.press_enter_after === true) actions.push({ type: "keypress", keys: ["enter"] }); - return actions; -} - -function toKeypressAction(args: Record): CuaAction[] | undefined { - const sequence = readKeySequence(args.key_comb ?? args.key); - return sequence.length > 0 ? sequence.map((keys) => ({ type: "keypress", keys })) : undefined; -} - -function toHoldKeyAction(args: Record): CuaAction[] | undefined { - const keys = readKeyCombo(args.key_comb ?? args.key); - return keys.length > 0 ? [{ type: "keypress", keys, duration: secondsToMs(args.duration, 1000) }] : undefined; -} - -function readKeyCombo(value: unknown): string[] { - if (typeof value !== "string") return []; - return value - .split("+") - .map((part) => part.trim()) - .filter(Boolean); -} - -function readKeySequence(value: unknown): string[][] { - if (typeof value !== "string") return []; - return value - .trim() - .split(/\s+/) - .map((part) => readKeyCombo(part)) - .filter((combo) => combo.length > 0); -} - -function holdKeys(value: unknown): { hold_keys?: string[] } { - if (typeof value !== "string") return {}; - const key = value.trim(); - return key ? { hold_keys: [key] } : {}; -} - -function secondsToMs(value: unknown, fallback: number): number { - if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) return fallback; - return Math.round(value * 1000); -} diff --git a/packages/ai/src/providers/yutori/provider.ts b/packages/ai/src/providers/yutori/provider.ts deleted file mode 100644 index f2503020..00000000 --- a/packages/ai/src/providers/yutori/provider.ts +++ /dev/null @@ -1,229 +0,0 @@ -import OpenAI from "openai"; -import type { ChatCompletion, ChatCompletionMessageParam } from "openai/resources/chat/completions"; -import { - createAssistantMessageEventStream, - type Api, - type AssistantMessage, - type Context, - type ImageContent, - type Model, - type SimpleStreamOptions, - type StreamFunction, - type StreamOptions, - type TextContent, - type ToolCall, -} from "@earendil-works/pi-ai"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; - -export const YUTORI_CHAT_COMPLETIONS_API = "yutori-chat-completions"; - -/** Stream options accepted by {@link streamYutori}. */ -export interface YutoriOptions extends StreamOptions { - /** @internal Identity-addressed native dispatch compiled from selected tools. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -export const streamYutori: StreamFunction = (model, context, options) => { - const stream = createAssistantMessageEventStream(); - void runYutoriStream(stream, model, context, options); - return stream; -}; - -export const streamSimpleYutori: StreamFunction = ( - model, - context, - options, -) => streamYutori(model, context, options); - -async function runYutoriStream( - stream: ReturnType, - model: Model, - context: Context, - options: YutoriOptions | undefined, -): Promise { - const output = initialAssistantMessage(model); - try { - const apiKey = options?.apiKey || process.env.YUTORI_API_KEY; - if (!apiKey) throw new Error("missing Yutori API key"); - const client = new OpenAI({ - apiKey, - baseURL: model.baseUrl || "https://api.yutori.com/v1", - defaultHeaders: { ...model.headers, ...options?.headers }, - }); - let payload: Record = { - model: model.id, - messages: convertMessages(context), - max_completion_tokens: options?.maxTokens ?? model.maxTokens, - temperature: options?.temperature ?? 0.3, - }; - const tools = convertTools(context); - if (tools.length > 0) payload.tools = tools; - const nextPayload = await options?.onPayload?.(payload, model); - if (nextPayload !== undefined) payload = nextPayload as Record; - - const { data: response, response: rawResponse } = await client.chat.completions - .create(payload as unknown as Parameters[0], { signal: options?.signal }) - .withResponse(); - const completion = response as ChatCompletion; - await options?.onResponse?.({ status: rawResponse.status, headers: headersToRecord(rawResponse.headers) }, model); - - stream.push({ type: "start", partial: output }); - const choice = completion.choices?.[0]; - const message = choice?.message; - output.responseId = completion.id; - output.usage = usageFromYutori(completion.usage); - if (choice?.finish_reason === "tool_calls") output.stopReason = "toolUse"; - else if (choice?.finish_reason === "length") output.stopReason = "length"; - - const text = typeof message?.content === "string" ? message.content : ""; - if (text) emitText(stream, output, text); - - for (const call of message?.tool_calls ?? []) { - if (call.type !== "function") continue; - const args = parseArguments(call.function.arguments); - const selectedName = options?.cuaIncomingToolPlan?.yutoriNames[call.function.name]; - const contentIndex = output.content.length; - const toolCall: ToolCall = { - type: "toolCall", - id: call.id, - name: selectedName ?? call.function.name, - arguments: args, - }; - output.content.push(toolCall); - output.stopReason = "toolUse"; - stream.push({ type: "toolcall_start", contentIndex, partial: output }); - stream.push({ type: "toolcall_delta", contentIndex, delta: call.function.arguments ?? "", partial: output }); - stream.push({ type: "toolcall_end", contentIndex, toolCall, partial: output }); - } - - stream.push({ type: "done", reason: output.stopReason as "stop" | "length" | "toolUse", message: output }); - stream.end(); - } catch (err) { - output.stopReason = options?.signal?.aborted ? "aborted" : "error"; - output.errorMessage = err instanceof Error ? err.message : String(err); - stream.push({ type: "error", reason: output.stopReason, error: output }); - stream.end(); - } -} - -function initialAssistantMessage(model: Model): AssistantMessage { - return { - role: "assistant", - content: [], - api: model.api, - provider: model.provider, - model: model.id, - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: Date.now(), - }; -} - -function convertMessages(context: Context): ChatCompletionMessageParam[] { - const messages: ChatCompletionMessageParam[] = []; - if (context.systemPrompt) messages.push({ role: "system", content: context.systemPrompt }); - for (const message of context.messages) { - if (message.role === "user") { - messages.push({ - role: "user", - content: typeof message.content === "string" ? message.content : message.content.map(toOpenAIContentPart), - } as ChatCompletionMessageParam); - } else if (message.role === "assistant") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join(""); - const toolCalls = message.content - .filter((part): part is ToolCall => part.type === "toolCall") - .map((part) => ({ - id: part.id, - type: "function" as const, - function: { name: part.name, arguments: JSON.stringify(part.arguments ?? {}) }, - })); - messages.push({ - role: "assistant", - content: text || null, - ...(toolCalls.length > 0 ? { tool_calls: toolCalls } : {}), - }); - } else if (message.role === "toolResult") { - messages.push({ - role: "tool", - tool_call_id: message.toolCallId, - content: message.content.map(toOpenAIContentPart) as unknown as string, - }); - } - } - return messages; -} - -function convertTools(context: Context): Array> { - return (context.tools ?? []).map((tool) => ({ - type: "function", - function: { - name: tool.name, - description: tool.description, - parameters: tool.parameters, - }, - })); -} - -function emitText(stream: ReturnType, output: AssistantMessage, text: string): void { - const contentIndex = output.content.length; - const content: TextContent = { type: "text", text }; - output.content.push(content); - stream.push({ type: "text_start", contentIndex, partial: output }); - stream.push({ type: "text_delta", contentIndex, delta: text, partial: output }); - stream.push({ type: "text_end", contentIndex, content: text, partial: output }); -} - -function toOpenAIContentPart(part: TextContent | ImageContent): Record { - if (part.type === "text") return { type: "text", text: part.text }; - return { type: "image_url", image_url: { url: `data:${part.mimeType};base64,${part.data}` } }; -} - -// Degrade malformed tool-call arguments to {} so one bad call cannot turn the -// whole response into a stopReason "error" turn (mirrors the Tzafon provider). -function parseArguments(value: string | undefined): Record { - if (!value?.trim()) return {}; - try { - const parsed = JSON.parse(value) as unknown; - return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? (parsed as Record) : {}; - } catch { - return {}; - } -} - -function usageFromYutori(usage: unknown): AssistantMessage["usage"] { - const input = readNumber(usage, "prompt_tokens"); - const output = readNumber(usage, "completion_tokens"); - const totalTokens = readNumber(usage, "total_tokens") || input + output; - return { - input, - output, - cacheRead: 0, - cacheWrite: 0, - totalTokens, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }; -} - -function readNumber(value: unknown, key: string): number { - if (!value || typeof value !== "object") return 0; - const n = (value as Record)[key]; - return typeof n === "number" && Number.isFinite(n) ? n : 0; -} - -function headersToRecord(headers: Headers): Record { - const out: Record = {}; - headers.forEach((value, key) => { - out[key] = value; - }); - return out; -} diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index 9b8f3f8b..206dea21 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -1,12 +1,14 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; import type { CuaAction } from "./actions/index"; import type { CuaModelRef } from "./models"; -import { cuaModelCapabilities, getCuaModel, providerForModel, routeCuaApi } from "./models"; +import { cuaModelCapabilities, getCuaModel, providerForModel } from "./models"; import { anthropicAdaptiveThinkingOnPayload } from "./providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer, } from "./providers/anthropic/capabilities"; +import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; +import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; export const CUA_TOOL_SPEC_KIND = "@onkernel/cua-tool-spec/v1" as const; @@ -36,22 +38,18 @@ export type CuaProviderBinding = readonly beta: string; readonly accessFallback?: CuaAnthropicBrowserFallback; } - | { readonly kind: "openai-native"; readonly declaration: Record } | { - readonly kind: "tzafon-native"; + readonly kind: "openai-native"; readonly declaration: Record; - } - | { - readonly kind: "yutori-native"; - readonly generation: "n1" | "n15"; - readonly nativeName: string; - readonly toolSet?: string; - readonly allNativeNames: readonly string[]; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; } | { readonly kind: "google-native"; readonly nativeName: string; readonly allNativeNames: readonly string[]; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; }; /** Declarative CUA tool. Identity is immutable and independent from its model-facing alias. */ @@ -149,8 +147,6 @@ export interface CuaAnthropicBrowserFallback { export interface CuaIncomingToolPlan { readonly anthropicBrowserFallback?: CuaAnthropicBrowserFallback; readonly openaiComputerName?: string; - readonly tzafonComputerName?: string; - readonly yutoriNames: Readonly>; readonly googleNames: Readonly>; /** Google predefined functions disabled by the exact selected native subset. */ readonly googleExcludedNames: readonly string[]; @@ -179,8 +175,6 @@ export interface CuaToolCatalog { export interface CompileCuaToolCatalogOptions { model: CuaModelRef | Model; requestedTools: readonly CuaCatalogToolInput[]; - /** Catalog-planning context; feeds declaration defaulting (e.g. Tzafon display size). */ - viewport: { readonly width: number; readonly height: number }; } /** @@ -199,20 +193,28 @@ const SAFE_TOOL_NAME = /^[A-Za-z][A-Za-z0-9_-]{0,63}$/; /** * Compile exactly one identity-keyed catalog for a model and caller-owned - * requested list. Pure and declaration-only: identical declaration, model, - * and viewport inputs produce identical catalogs, and compilation never - * constructs executable tools or retains the requested input objects. + * requested list. Pure and declaration-only: identical declaration and model + * inputs produce identical catalogs, and compilation never constructs + * executable tools or retains the requested input objects. + * + * The compiled model's `api` is derived here, not stamped on the model ahead + * of time: a selected tool's provider binding may declare `requiresApi`, and + * the returned `catalog.model` carries that transport. A model resolved with + * no such tool selected keeps its ordinary registry `api`. Selecting tools + * whose bindings require different transports fails to compile. */ export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): CuaToolCatalog { - const model = typeof options.model === "string" ? getCuaModel(options.model) : routeCuaApi(options.model); - const viewport = options.viewport; - const normalizedEntries = [...options.requestedTools].map((tool) => normalizeTool(tool, viewport)); - validateCatalog(model, normalizedEntries); + const baseModel = typeof options.model === "string" + ? getCuaModel(options.model) + : resetCatalogDerivedApi(options.model); + const normalizedEntries = [...options.requestedTools].map(normalizeTool); + const requiresApi = validateCatalog(baseModel, normalizedEntries); + const model = requiresApi ? { ...baseModel, api: requiresApi } : baseModel; const drafts = resolveProviderFacingDeclarations(normalizedEntries); const names = new Map(drafts.map((entry) => [entry.identity, entry.name])); const requirements = compileHeaderRequirements(drafts); - const transforms = compilePayloadTransforms(model, drafts, viewport); + const transforms = compilePayloadTransforms(model, drafts); validateTransformClaims(transforms); const incoming = compileIncomingPlan(drafts); const fingerprint = stableStringify({ @@ -262,10 +264,7 @@ export function modelSupportsDeferredTools(model: Model): boolean { return major > 4 || (major === 4 && minor >= 5); } -function normalizeTool( - tool: CuaCatalogToolInput, - viewport: { readonly width: number; readonly height: number }, -): CuaCatalogEntryDraft { +function normalizeTool(tool: CuaCatalogToolInput): CuaCatalogEntryDraft { if (isCuaToolSpec(tool)) { const schemaFingerprint = stableStringify(tool.declaration.parameters); const fingerprint = stableStringify({ @@ -274,15 +273,9 @@ function normalizeTool( schema: schemaFingerprint, coordinates: tool.execution.kind === "actions" ? tool.execution.coordinates : undefined, }); - const inspectedDeclaration = tool.providerBinding?.kind === "tzafon-native" - ? { - ...tool.providerBinding.declaration, - display_width: tool.providerBinding.declaration.display_width ?? viewport.width, - display_height: tool.providerBinding.declaration.display_height ?? viewport.height, - } - : tool.providerBinding && "declaration" in tool.providerBinding - ? tool.providerBinding.declaration - : tool.declaration; + const inspectedDeclaration = tool.providerBinding && "declaration" in tool.providerBinding + ? tool.providerBinding.declaration + : tool.declaration; return Object.freeze({ identity: tool.identity, name: tool.name, @@ -337,30 +330,14 @@ function resolveProviderFacingDeclarations(entries: readonly CuaCatalogEntryDraf }; })() : undefined; - const yutori = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native"); - const yutoriDeclaration = yutori.length > 0 ? (() => { - const binding = yutori[0]!.providerBinding; - if (binding?.kind !== "yutori-native") return undefined; - const selected = new Set(yutori.map((entry) => { - const selectedBinding = entry.providerBinding; - return selectedBinding?.kind === "yutori-native" ? selectedBinding.nativeName : ""; - })); - return { - ...(binding.toolSet ? { tool_set: binding.toolSet } : {}), - disable_tools: binding.allNativeNames.filter((name) => !selected.has(name)), - }; - })() : undefined; - return entries.map((entry) => { - const binding = entry.providerBinding; - const declaration = binding?.kind === "google-native" - ? googleDeclaration - : binding?.kind === "yutori-native" ? yutoriDeclaration : undefined; + const declaration = entry.providerBinding?.kind === "google-native" ? googleDeclaration : undefined; return declaration ? Object.freeze({ ...entry, declaration: Object.freeze(declaration) }) : entry; }); } -function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDraft[]): void { +/** Validate the requested catalog against the model and return the transport its selected tools require, if any. */ +function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { const identities = new Map(); const exactNames = new Map(); const normalizedNames = new Map(); @@ -387,7 +364,7 @@ function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDra validateToolCompatibility(model, entry); } - validateToolsetCompatibility(model, entries); + return validateToolsetCompatibility(model, entries); } function nameCollision( @@ -432,15 +409,8 @@ function validateAnthropicNativeModel(model: Model, identity: string): void } } -function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): void { - const yutoriN1 = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native" && entry.providerBinding.generation === "n1"); - if (yutoriN1.length > 0) { - const all = yutoriN1[0]!.providerBinding; - if (all?.kind === "yutori-native" && yutoriN1.length !== all.allNativeNames.length) { - throw new Error(`Yutori n1 cannot suppress a partial native action set; select the complete cua.providers.yutori.toolsets.n1() toolset`); - } - } - +/** Validate the selected native tools agree on a provider and a transport, and return the transport they require, if any. */ +function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { const nativeProviderKinds = new Set( entries.flatMap((entry) => entry.providerBinding ? [entry.providerBinding.kind.split("-")[0]] : []), ); @@ -448,6 +418,36 @@ function validateToolsetCompatibility(model: Model, entries: readonly CuaCa throw new Error(`selected tools contribute incompatible native provider transports: ${[...nativeProviderKinds].join(", ")}`); } providerForModel(model); + + const requiresApis = new Set(entries.flatMap((entry) => bindingRequiresApi(entry.providerBinding))); + if (requiresApis.size > 1) { + throw new Error(`selected tools require incompatible provider transports: ${[...requiresApis].join(", ")}`); + } + return requiresApis.values().next().value; +} + +/** The transport a provider binding requires, if it declares one. Anthropic never forks transports and declares none. */ +function bindingRequiresApi(binding: CuaProviderBinding | undefined): readonly [Api] | readonly [] { + return binding && binding.kind !== "anthropic-native" && binding.requiresApi ? [binding.requiresApi] : []; +} + +/** + * Model-shaped default transport for each `requiresApi` this module can + * derive, keyed by the derived api itself. A `Model` a caller passes to + * {@link compileCuaToolCatalog} may already carry one of these — e.g. a prior + * catalog's `catalog.model`, fed back in with a different tool selection — so + * derivation resets it here before re-validating, keeping compilation pure + * with respect to the currently requested tools rather than pinning whatever + * transport an earlier selection required. + */ +const CATALOG_DERIVED_API_DEFAULTS: Readonly> = { + [OPENAI_CUA_COMPUTER_API]: "openai-responses", + [GOOGLE_CUA_INTERACTIONS_API]: "google-generative-ai", +}; + +function resetCatalogDerivedApi(model: Model): Model { + const defaultApi = CATALOG_DERIVED_API_DEFAULTS[model.api]; + return defaultApi ? { ...model, api: defaultApi } : model; } function compileHeaderRequirements(entries: readonly CuaCatalogEntryDraft[]): CuaHeaderRequirement[] { @@ -493,11 +493,7 @@ function commaTokens(value: string | undefined): string[] { return value?.split(",").map((token) => token.trim()).filter(Boolean) ?? []; } -function compilePayloadTransforms( - model: Model, - entries: readonly CuaCatalogEntryDraft[], - viewport: { readonly width: number; readonly height: number }, -): CuaPayloadTransform[] { +function compilePayloadTransforms(model: Model, entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform[] { const transforms: CuaPayloadTransform[] = []; if (model.provider === "anthropic") { transforms.push({ @@ -513,14 +509,8 @@ function compilePayloadTransforms( for (const entry of entries) { const binding = entry.providerBinding; if (!binding) continue; - if (binding.kind === "anthropic-native" || binding.kind === "openai-native" || binding.kind === "tzafon-native") { - const declaration = binding.kind === "tzafon-native" - ? { - ...binding.declaration, - display_width: binding.declaration.display_width ?? viewport.width, - display_height: binding.declaration.display_height ?? viewport.height, - } - : binding.declaration; + if (binding.kind === "anthropic-native" || binding.kind === "openai-native") { + const { declaration } = binding; transforms.push({ identity: entry.identity, consumesToolIdentities: [entry.identity], @@ -533,8 +523,6 @@ function compilePayloadTransforms( } } - const yutori = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native"); - if (yutori.length > 0) transforms.push(createYutoriTransform(yutori)); const google = entries.filter((entry) => entry.providerBinding?.kind === "google-native"); if (google.length > 0) transforms.push(createGoogleTransform(google)); @@ -551,35 +539,6 @@ function compilePayloadTransforms( return transforms; } -function createYutoriTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform { - const firstBinding = entries[0]!.providerBinding; - if (firstBinding?.kind !== "yutori-native") throw new Error("invalid Yutori catalog entry"); - const generations = new Set(entries.map((entry) => { - const binding = entry.providerBinding; - return binding?.kind === "yutori-native" ? binding.generation : ""; - })); - if (generations.size !== 1) throw new Error("Yutori n1 and n1.5 native toolsets cannot be combined"); - const selectedNativeNames = entries.map((entry) => (entry.providerBinding as Extract).nativeName); - const selectedSet = new Set(selectedNativeNames); - const disabled = firstBinding.allNativeNames.filter((name) => !selectedSet.has(name)); - const identity = `provider.yutori.native.${firstBinding.generation}`; - return { - identity, - consumesToolIdentities: entries.map((entry) => entry.identity), - writes: ["tools", "tool_set", "disable_tools"], - phase: "tool-declarations", - apply(payload, _model, names) { - const stripped = removeSerializedTools(payload, entries.map((entry) => names.get(entry.identity)!)); - if (!isRecord(stripped)) return stripped; - return { - ...stripped, - ...(firstBinding.toolSet ? { tool_set: firstBinding.toolSet } : {}), - disable_tools: disabled, - }; - }, - }; -} - function createGoogleTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform { const firstBinding = entries[0]!.providerBinding; if (firstBinding?.kind !== "google-native") throw new Error("invalid Google catalog entry"); @@ -641,9 +600,7 @@ function createPayloadPlan( function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncomingToolPlan { let anthropicBrowserFallback: CuaAnthropicBrowserFallback | undefined; let openaiComputerName: string | undefined; - let tzafonComputerName: string | undefined; let googleAllNativeNames: readonly string[] = []; - const yutoriNames: Record = {}; const googleNames: Record = {}; const nativeToolNames: string[] = []; for (const entry of entries) { @@ -652,8 +609,6 @@ function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncom nativeToolNames.push(entry.name); if (binding.kind === "anthropic-native" && binding.accessFallback) anthropicBrowserFallback = binding.accessFallback; else if (binding.kind === "openai-native") openaiComputerName = entry.name; - else if (binding.kind === "tzafon-native") tzafonComputerName = entry.name; - else if (binding.kind === "yutori-native") yutoriNames[binding.nativeName] = entry.name; else if (binding.kind === "google-native") { googleNames[binding.nativeName] = entry.name; googleAllNativeNames = binding.allNativeNames; @@ -664,8 +619,6 @@ function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncom return Object.freeze({ ...(anthropicBrowserFallback ? { anthropicBrowserFallback: Object.freeze(anthropicBrowserFallback) } : {}), ...(openaiComputerName ? { openaiComputerName } : {}), - ...(tzafonComputerName ? { tzafonComputerName } : {}), - yutoriNames: Object.freeze(yutoriNames), googleNames: Object.freeze(googleNames), googleExcludedNames: Object.freeze(googleExcludedNames), nativeToolNames: Object.freeze(nativeToolNames), diff --git a/packages/ai/test/anthropic-browser-fallback.test.ts b/packages/ai/test/anthropic-browser-fallback.test.ts index c490f805..98aa5f6c 100644 --- a/packages/ai/test/anthropic-browser-fallback.test.ts +++ b/packages/ai/test/anthropic-browser-fallback.test.ts @@ -16,8 +16,6 @@ import { } from "../src/index"; import { withAnthropicBrowserFallback } from "../src/providers/anthropic/browser-fallback"; -const viewport = { width: 1440, height: 900 }; - const context: Context = { systemPrompt: "", messages: [{ role: "user", content: [{ type: "text", text: "Use the browser." }], timestamp: 1 }], @@ -30,7 +28,6 @@ describe("Anthropic native browser access fallback", () => { const catalog = compileCuaToolCatalog({ model, requestedTools: [cua.providers.anthropic.tools.browser()], - viewport, }); const payloads: Array<{ tools: unknown[]; headers: SimpleStreamOptions["headers"] }> = []; let calls = 0; @@ -69,7 +66,6 @@ describe("Anthropic native browser access fallback", () => { const catalog = compileCuaToolCatalog({ model, requestedTools: [cua.providers.anthropic.tools.browser()], - viewport, }); let calls = 0; const provider = withAnthropicBrowserFallback(fakeProvider(async (selectedModel) => { diff --git a/packages/ai/test/anthropic-native.integration.test.ts b/packages/ai/test/anthropic-native.integration.test.ts index d187a0ec..3ad553f1 100644 --- a/packages/ai/test/anthropic-native.integration.test.ts +++ b/packages/ai/test/anthropic-native.integration.test.ts @@ -8,8 +8,6 @@ import { const apiKey = process.env.ANTHROPIC_API_KEY; const liveIt = apiKey ? it : it.skip; -const viewport = { width: 1440, height: 900 }; - const cases = [ { name: "computer", @@ -31,7 +29,6 @@ describe("Anthropic early-access native tools", () => { const catalog = compileCuaToolCatalog({ model: "anthropic:claude-opus-5", requestedTools: [current.tool], - viewport, }); const response = await createCuaModels().complete( catalog.model, diff --git a/packages/ai/test/api-keys.test.ts b/packages/ai/test/api-keys.test.ts index 205be36a..770ca168 100644 --- a/packages/ai/test/api-keys.test.ts +++ b/packages/ai/test/api-keys.test.ts @@ -16,8 +16,6 @@ const ENV_KEYS = [ "XAI_API_KEY", "MOONSHOT_API_KEY", "OPENROUTER_API_KEY", - "TZAFON_API_KEY", - "YUTORI_API_KEY", ] as const; const ORIGINAL_ENV = new Map(ENV_KEYS.map((key) => [key, process.env[key]])); @@ -35,7 +33,6 @@ describe("cua api key helpers", () => { expect(cuaApiKeyEnvVarsForProvider("openai")).toEqual(["OPENAI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("google")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("gemini")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("meta")).toEqual(["META_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("xai")).toEqual(["XAI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("moonshotai")).toEqual(["MOONSHOT_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("moonshot")).toEqual(["MOONSHOT_API_KEY"]); @@ -54,8 +51,6 @@ describe("cua api key helpers", () => { it("resolves keys from model refs", () => { process.env.OPENAI_API_KEY = "openai"; expect(getCuaEnvApiKeyForModel("openai:gpt-5.5")).toBe("openai"); - process.env.META_API_KEY = "meta"; - expect(getCuaEnvApiKeyForModel("meta:muse-spark-1.1")).toBe("meta"); process.env.XAI_API_KEY = "xai"; expect(getCuaEnvApiKeyForModel("xai:grok-4.5")).toBe("xai"); process.env.MOONSHOT_API_KEY = "moonshot"; @@ -65,7 +60,6 @@ describe("cua api key helpers", () => { }); it("throws readable errors when missing", () => { - delete process.env.TZAFON_API_KEY; - expect(() => requireCuaEnvApiKey("tzafon")).toThrow("TZAFON_API_KEY"); + delete process.env.META_API_KEY; }); }); diff --git a/packages/ai/test/google-provider.test.ts b/packages/ai/test/google-provider.test.ts index ae84f95f..b907c7f8 100644 --- a/packages/ai/test/google-provider.test.ts +++ b/packages/ai/test/google-provider.test.ts @@ -3,17 +3,17 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import { getCuaModel } from "../src/index"; import * as google from "../src/providers/google/provider"; -const model = getCuaModel("google:gemini-3.6-flash") as Model; +// This adapter is exercised directly (not through compileCuaToolCatalog's +// derivation), so the model must carry the Interactions api itself. +const model: Model = { ...getCuaModel("google:gemini-3.6-flash"), api: google.GOOGLE_CUA_INTERACTIONS_API }; const incoming = { googleNames: { click: "click" }, googleExcludedNames: ["take_screenshot"], - yutoriNames: {}, nativeToolNames: ["click"], }; const screenshotIncoming = { googleNames: { take_screenshot: "take_screenshot" }, googleExcludedNames: ["click"], - yutoriNames: {}, nativeToolNames: ["take_screenshot"], }; const clickTool = { name: "click", description: "Click the page", parameters: { type: "object" } as never }; diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index 6f5628f0..41a55a91 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -3,11 +3,9 @@ import { CUA_MODEL_ANNOTATIONS, CUA_PROVIDERS, type CuaModelRef, - cuaOverrideModels, findCuaAnnotation, formatCuaModelRef, getCuaModel, - GOOGLE_CUA_INTERACTIONS_API, listCuaModels, parseCuaModelRef, } from "../src/index"; @@ -15,7 +13,7 @@ import { describe("CUA model refs", () => { it("parses and formats provider-qualified refs", () => { expect(parseCuaModelRef("openai:gpt-5.5")).toEqual({ provider: "openai", model: "gpt-5.5" }); - expect(formatCuaModelRef("yutori", "n1.5-latest")).toBe("yutori:n1.5-latest"); + expect(formatCuaModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); }); it("rejects unqualified and unsupported refs", () => { @@ -26,7 +24,7 @@ describe("CUA model refs", () => { it("names the valid providers in the unsupported-provider error", () => { expect(() => parseCuaModelRef("bogus:model")).toThrow( - 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, meta, xai, moonshotai, openrouter, tzafon, yutori)', + 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, xai, moonshotai, openrouter)', ); }); @@ -50,13 +48,8 @@ describe("CUA model refs", () => { expect(models.some((model) => "origin" in model)).toBe(false); }); - it("returns override models for refs missing from pi-ai", () => { - const model = getCuaModel("yutori:n1.5-latest"); - expect(model.provider).toBe("yutori"); - expect(model.api).toBe("yutori-chat-completions"); - + it("returns pi-ai registry entries verbatim", () => { const opus = getCuaModel("anthropic:claude-opus-5"); - expect(cuaOverrideModels("anthropic")).toEqual([]); expect(opus).toMatchObject({ provider: "anthropic", api: "anthropic-messages", @@ -67,34 +60,24 @@ describe("CUA model refs", () => { }); expect(opus.compat).toMatchObject({ forceAdaptiveThinking: true, supportsTemperature: false }); - expect(cuaOverrideModels("google")).toEqual([]); expect(getCuaModel("google:gemini-3.6-flash")).toMatchObject({ provider: "google", - api: GOOGLE_CUA_INTERACTIONS_API, + api: "google-generative-ai", contextWindow: 1_048_576, }); - const muse = getCuaModel("meta:muse-spark-1.1"); - expect(muse.provider).toBe("meta"); - expect(muse.api).toBe("openai-responses"); - expect(muse.baseUrl).toBe("https://api.meta.ai/v1"); - expect(muse.contextWindow).toBe(1_048_576); - expect(muse.maxTokens).toBe(128_000); - expect(muse.thinkingLevelMap?.off).toBeNull(); + const muse = getCuaModel("openrouter:meta/muse-spark-1.1"); + expect(muse.provider).toBe("openrouter"); + expect(muse.baseUrl).toBe("https://openrouter.ai/api/v1"); }); - it("uses pi-ai's Grok catalog entry with CUA routing overrides", () => { - expect(cuaOverrideModels("xai")).toEqual([]); + it("returns pi-ai's Grok catalog entry", () => { const grok = getCuaModel("xai:grok-4.5"); expect(grok.provider).toBe("xai"); expect(grok.api).toBe("openai-responses"); expect(grok.baseUrl).toBe("https://api.x.ai/v1"); expect(grok.contextWindow).toBe(500_000); expect(grok.maxTokens).toBe(500_000); - expect(grok.thinkingLevelMap).toEqual({ off: "low", minimal: "low", xhigh: "high" }); - expect(grok.cost.tiers).toEqual([ - { inputTokensAbove: 200_000, input: 4, output: 12, cacheRead: 1, cacheWrite: 0 }, - ]); }); it("uses pi-ai's Kimi catalog entries for both transports", () => { @@ -105,8 +88,7 @@ describe("CUA model refs", () => { expect(listCuaModels("openrouter").map((model) => model.ref)).toContain("openrouter:moonshotai/kimi-k3"); }); - it("uses pi-ai's Kimi catalog entry without CUA routing overrides", () => { - expect(cuaOverrideModels("moonshotai")).toEqual([]); + it("uses pi-ai's Kimi catalog entry", () => { const kimi = getCuaModel("moonshotai:kimi-k3"); expect(kimi.provider).toBe("moonshotai"); expect(kimi.api).toBe("openai-completions"); @@ -126,28 +108,23 @@ describe("CUA model refs", () => { expect(getCuaModel("moonshot:kimi-k3" as CuaModelRef).id).toBe("kimi-k3"); }); - it("loads supported custom provider models without explicit registration", () => { - expect(getCuaModel("tzafon:tzafon.northstar-cua-fast").api).toBe("tzafon-responses"); - expect(getCuaModel("yutori:n1.5-latest").api).toBe("yutori-chat-completions"); - }); - - it("keeps every Responses model on pi's builtin transport except Google's Interactions API", () => { - // A CUA-owned api id now exists only where pi ships no equivalent - // transport. OpenAI, Meta, and xAI all speak the Responses protocol pi - // already implements, including its automatic prompt caching. + it("resolves every model to its ordinary registry transport, independent of tool selection", () => { + // getCuaModel() never derives a tool-driven transport: OPENAI_CUA_COMPUTER_API + // and GOOGLE_CUA_INTERACTIONS_API are only ever carried by a model that + // compileCuaToolCatalog compiled with the matching native tool selected + // (see tool-catalog.test.ts's transport derivation coverage). expect(getCuaModel("openai:gpt-5.6-sol").api).toBe("openai-responses"); expect(getCuaModel("openai:gpt-5.5").api).toBe("openai-responses"); expect(getCuaModel("openai:gpt-5.4-mini").api).toBe("openai-responses"); - expect(getCuaModel("google:gemini-3.6-flash").api).toBe(GOOGLE_CUA_INTERACTIONS_API); - expect(getCuaModel("meta:muse-spark-1.1").api).toBe("openai-responses"); + expect(getCuaModel("google:gemini-3.6-flash").api).toBe("google-generative-ai"); expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); }); - it("rejects supported model IDs that are not in pi-ai or overrides", () => { + it("rejects supported model IDs that pi-ai does not carry", () => { // Dated snapshots match the family annotation but pi-ai's registry // (generated from models.dev) only carries family roots. expect(() => getCuaModel("openai:gpt-5.5-2026-04-23")).toThrow( - /not registered/, + /not carried by pi-ai's registry/, ); }); }); @@ -198,7 +175,7 @@ describe("CUA support annotations", () => { expect(findCuaAnnotation("openai", "gpt-5.6-sol")?.match).toEqual({ kind: "exact", id: "gpt-5.6-sol" }); expect(findCuaAnnotation("openai", "gpt-5.6-sol-20260728")).toBeUndefined(); expect(findCuaAnnotation("google", "gemini-3.6-flash")).toBeDefined(); - expect(findCuaAnnotation("meta", "muse-spark-1.1")).toBeDefined(); + expect(findCuaAnnotation("openrouter", "meta/muse-spark-1.1")).toBeDefined(); expect(findCuaAnnotation("xai", "grok-4.5")).toBeDefined(); expect(findCuaAnnotation("xai", "grok-4.5-latest")).toBeUndefined(); expect(findCuaAnnotation("xai", "grok-4.3")).toBeUndefined(); @@ -206,9 +183,6 @@ describe("CUA support annotations", () => { expect(findCuaAnnotation("moonshotai", "kimi-k2.5")).toBeUndefined(); expect(findCuaAnnotation("moonshotai", "kimi-latest")).toBeUndefined(); expect(findCuaAnnotation("google", "gemini-3.5-flash-lite")).toBeDefined(); - expect(findCuaAnnotation("yutori", "n1.5-latest")).toBeDefined(); - expect(findCuaAnnotation("tzafon", "tzafon.northstar-cua-fast")).toBeDefined(); - expect(findCuaAnnotation("tzafon", "tzafon.northstar-cua-fast-1.6")).toBeDefined(); }); it("advertises only Google's current documented computer-use models", () => { diff --git a/packages/ai/test/openai-adapter-routing.test.ts b/packages/ai/test/openai-adapter-routing.test.ts index 74dcc56e..d258922c 100644 --- a/packages/ai/test/openai-adapter-routing.test.ts +++ b/packages/ai/test/openai-adapter-routing.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from "vitest"; import type { ToolCall } from "@earendil-works/pi-ai"; -import { createCuaModels, getCuaModel } from "../src/index"; +import { createCuaModels, getCuaModel, OPENAI_CUA_COMPUTER_API } from "../src/index"; const { responsesCreate } = vi.hoisted(() => ({ responsesCreate: vi.fn() })); @@ -68,7 +68,7 @@ describe("OpenAI adapter routing", () => { expect(payload.previous_response_id).toBeUndefined(); }); - it("reaches the CUA adapter when the incoming plan selects OpenAI's native computer tool", async () => { + it("reaches the CUA adapter when the model carries OPENAI_CUA_COMPUTER_API", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", output: [{ @@ -77,12 +77,16 @@ describe("OpenAI adapter routing", () => { action: { type: "click", x: 10, y: 20 }, }], }); - const message = await createCuaModels().streamSimple(model, { + // compileCuaToolCatalog derives this api onto the model whenever OpenAI's + // native computer tool is selected; the provider wrapper dispatches on it + // alone, with no request-shape sniffing. + const computerModel = { ...model, api: OPENAI_CUA_COMPUTER_API }; + const message = await createCuaModels().streamSimple(computerModel, { messages: [{ role: "user", content: "click it", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", - cuaIncomingToolPlan: { openaiComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, + cuaIncomingToolPlan: { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, } as never).result(); // Only the CUA native-computer adapter understands computer_call items; @@ -127,6 +131,39 @@ describe("OpenAI adapter routing", () => { expect(call?.namespace).toBe("deferred_tools"); }); + it("keeps cache-relevant payload fields identical across a mid-conversation escalation", async () => { + // A turn can move from pi's builtin transport to the CUA adapter mid-session + // (a deferred tool gets added). If that switch changed `store` or the cache + // key, it would silently invalidate the matched prompt-cache prefix. + responsesCreate.mockReturnValue({ id: "resp_parity", status: "completed", usage: {}, output: [] }); + await createCuaModels().streamSimple(model, { + messages: [{ role: "user", content: "look it up", timestamp: 1 }], + tools, + }, { apiKey: "test", sessionId: "session_parity" }).result(); + const builtinPayload = responsesCreate.mock.calls.at(-1)?.[0] as Record; + + await createCuaModels().streamSimple(model, { + messages: [ + { role: "user", content: "load and look it up", timestamp: 1 }, + { + role: "toolResult", + toolCallId: "call_loader", + toolName: "loader", + content: [{ type: "text", text: "loaded" }], + isError: false, + addedToolNames: ["lookup"], + timestamp: 2, + }, + ], + tools, + }, { apiKey: "test", sessionId: "session_parity" } as never).result(); + const escalatedPayload = responsesCreate.mock.calls.at(-1)?.[0] as Record; + + for (const field of ["store", "prompt_cache_key", "prompt_cache_retention", "prompt_cache_options"]) { + expect(escalatedPayload[field], field).toEqual(builtinPayload[field]); + } + }); + it("pairs replayed namespaces by call id, not ordinal, across an aborted assistant turn", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_4", usage: {}, output: [] }); await createCuaModels().streamSimple(model, { diff --git a/packages/ai/test/openai-native-provider.test.ts b/packages/ai/test/openai-native-provider.test.ts index d4cd523e..9a61d37c 100644 --- a/packages/ai/test/openai-native-provider.test.ts +++ b/packages/ai/test/openai-native-provider.test.ts @@ -16,7 +16,10 @@ vi.mock("openai", () => ({ })); const model = getCuaModel("openai:gpt-5.5") as Model<"openai-responses">; -const incoming = { openaiComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; +// The catalog derives this api when OpenAI's native computer tool is selected; +// the provider wrapper routes it to the adapter under test. +const nativeModel = { ...model, api: openai.OPENAI_CUA_COMPUTER_API } as unknown as Model<"openai-responses">; +const incoming = { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; describe("OpenAI native computer Responses adapter", () => { it("emits one identity-selected local call for actions[] and safety checks", async () => { @@ -30,7 +33,7 @@ describe("OpenAI native computer Responses adapter", () => { pending_safety_checks: [{ id: "check_1", code: "malicious_instructions" }], }], }); - const message = await openai.streamOpenAIResponses(model, { + const message = await openai.streamOpenAICuaComputer(nativeModel, { messages: [], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { @@ -55,7 +58,7 @@ describe("OpenAI native computer Responses adapter", () => { it("sends the same prompt-cache fields as the function-tool path", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_cache", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [{ role: "user", content: "go", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", sessionId: "session_native", cuaIncomingToolPlan: incoming }).result(); @@ -78,7 +81,7 @@ describe("OpenAI native computer Responses adapter", () => { arguments: '{"query":"status"}', }], }); - const first = await openai.streamOpenAIResponses(model, { + const first = await openai.streamOpenAICuaComputer(nativeModel, { messages: [{ role: "user", content: "look it up", timestamp: 1 }], tools: [ { name: "computer", description: "placeholder", parameters: { type: "object" } as never }, @@ -89,7 +92,7 @@ describe("OpenAI native computer Responses adapter", () => { expect(call.namespace).toBe("deferred_tools"); responsesCreate.mockReturnValueOnce({ id: "resp_done", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [ { role: "user", content: "look it up", timestamp: 1 }, first, @@ -122,7 +125,7 @@ describe("OpenAI native computer Responses adapter", () => { it("serializes native results as computer_call_output and ordinary results as function output", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [ { role: "assistant", diff --git a/packages/ai/test/providers.test.ts b/packages/ai/test/providers.test.ts index ea651aba..c38ca2d9 100644 --- a/packages/ai/test/providers.test.ts +++ b/packages/ai/test/providers.test.ts @@ -1,15 +1,10 @@ import { describe, expect, it } from "vitest"; -import { - createCuaModels, - cuaModels, - TZAFON_RESPONSES_API, - YUTORI_CHAT_COMPLETIONS_API, -} from "../src/index"; +import { createCuaModels, cuaModels } from "../src/index"; describe("createCuaModels", () => { - it("registers the CUA-only providers alongside pi's builtins", () => { + it("registers a streamable provider for every CUA provider", () => { const models = createCuaModels(); - for (const id of ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter", "tzafon", "yutori"]) { + for (const id of ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]) { const provider = models.getProvider(id); expect(provider, id).toBeDefined(); expect(provider?.stream).toBeTypeOf("function"); @@ -19,21 +14,11 @@ describe("createCuaModels", () => { it("lists CUA provider catalogs", () => { const models = createCuaModels(); - expect(models.getModel("meta", "muse-spark-1.1")?.api).toBe("openai-responses"); expect(models.getModel("xai", "grok-4.5")?.api).toBe("openai-responses"); expect(models.getModel("moonshotai", "kimi-k3")?.api).toBe("openai-completions"); expect(models.getModel("openrouter", "moonshotai/kimi-k3")?.api).toBe("openai-completions"); + expect(models.getModel("openrouter", "meta/muse-spark-1.1")?.api).toBe("openai-completions"); expect(models.getProvider("openrouter")?.baseUrl).toBe("https://openrouter.ai/api/v1"); - const tzafonIds = models.getModels("tzafon").map((m) => m.id); - expect(tzafonIds).toContain("tzafon.northstar-cua-fast"); - const yutoriIds = models.getModels("yutori").map((m) => m.id); - expect(yutoriIds).toContain("n1.5-latest"); - expect(models.getProvider("tzafon")?.baseUrl).toBe("https://api.tzafon.ai"); - expect(models.getModel("tzafon", "tzafon.northstar-cua-fast")).toMatchObject({ - api: TZAFON_RESPONSES_API, - baseUrl: "https://api.tzafon.ai", - }); - expect(models.getModel("yutori", "n1.5-latest")?.api).toBe(YUTORI_CHAT_COMPLETIONS_API); }); it("keeps builtin catalogs on wrapped providers", () => { @@ -41,8 +26,9 @@ describe("createCuaModels", () => { const openaiIds = models.getModels("openai").map((m) => m.id); expect(openaiIds).toContain("gpt-5.4"); // OpenAI models keep pi's builtin "openai-responses" api id on both the - // collection and getCuaModel(); the wrapped provider only intercepts - // requests that need the CUA adapter (see requiresCuaOpenAIAdapter). + // collection and getCuaModel(); the wrapped provider only intercepts a + // model carrying OPENAI_CUA_COMPUTER_API or a namespace round-trip (see + // requiresCuaOpenAINamespaceAdapter). expect(models.getModel("openai", "gpt-5.4")?.api).toBe("openai-responses"); const xaiIds = models.getModels("xai").map((m) => m.id); @@ -53,15 +39,15 @@ describe("createCuaModels", () => { it("returns independent collections", () => { const a = createCuaModels(); const b = createCuaModels(); - a.deleteProvider("tzafon"); - expect(a.getProvider("tzafon")).toBeUndefined(); - expect(b.getProvider("tzafon")).toBeDefined(); + a.deleteProvider("google"); + expect(a.getProvider("google")).toBeUndefined(); + expect(b.getProvider("google")).toBeDefined(); }); }); describe("cuaModels", () => { it("memoizes the default collection", () => { expect(cuaModels()).toBe(cuaModels()); - expect(cuaModels().getProvider("yutori")).toBeDefined(); + expect(cuaModels().getProvider("openai")).toBeDefined(); }); }); diff --git a/packages/ai/test/tool-catalog.test.ts b/packages/ai/test/tool-catalog.test.ts index 2e70fb27..10d04403 100644 --- a/packages/ai/test/tool-catalog.test.ts +++ b/packages/ai/test/tool-catalog.test.ts @@ -4,13 +4,13 @@ import { callerToolIdentity, compileCuaToolCatalog, cua, + GOOGLE_CUA_INTERACTIONS_API, + OPENAI_CUA_COMPUTER_API, type CuaToolSpec, } from "../src/index"; -const viewport = { width: 1440, height: 900 }; - function compile(model: Parameters[0]["model"], requestedTools: Parameters[0]["requestedTools"]) { - return compileCuaToolCatalog({ model, requestedTools, viewport }); + return compileCuaToolCatalog({ model, requestedTools }); } /** Sanitized caller declaration: cua-ai never receives executable members. */ @@ -78,9 +78,6 @@ describe("cua tool namespace", () => { cua.providers.anthropic.tools.computer(), ]], [cua.providers.google.source, cua.providers.google.toolsets.browser()], - [cua.providers.tzafon.source, [cua.providers.tzafon.tools.computer()]], - [cua.providers.yutori.sources.n1, cua.providers.yutori.toolsets.n1()], - [cua.providers.yutori.sources.n15Core, cua.providers.yutori.toolsets.n15Core()], ]; for (const [source, tools] of surfaces) { expect(source).toMatch(/^https:\/\//); @@ -90,7 +87,7 @@ describe("cua tool namespace", () => { }); it("uses the same CUA-authored browser toolset with custom-function providers", () => { - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5", "moonshotai:kimi-k3"] as const) { + for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { const catalog = compile(model, cua.toolsets.browser()); expect(catalog.entries[0]).toMatchObject({ identity: "cua.browser.snapshot.v1", @@ -187,7 +184,7 @@ describe("compileCuaToolCatalog", () => { }); it("still accepts browser_act on providers that take its schema size", () => { - for (const model of ["openai:gpt-5.5", "anthropic:claude-opus-5", "meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["openai:gpt-5.5", "anthropic:claude-opus-5", "xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { expect(() => compile(model, [cua.tools.browser.act()]), model).not.toThrow(); } }); @@ -197,9 +194,9 @@ describe("compileCuaToolCatalog", () => { expect(() => compile("openai:gpt-5.5", [cua.providers.anthropic.tools.computer()])).toThrow(/requires a anthropic model/); }); - it("replaces only the selected Tzafon identity placeholder", async () => { - const catalog = compile("tzafon:tzafon.northstar-cua-fast", [ - cua.providers.tzafon.tools.computer(), + it("replaces only the selected OpenAI identity placeholder", async () => { + const catalog = compile("openai:gpt-5.5", [ + cua.providers.openai.tools.computer(), callerTool("click"), cua.tools.browser.click(), ]); @@ -212,11 +209,11 @@ describe("compileCuaToolCatalog", () => { }; const next = await catalog.payload.apply(payload, catalog.model) as { tools: Array> }; expect(next.tools).toEqual([ - { type: "computer_use", display_width: 1440, display_height: 900, environment: "browser" }, + { type: "computer" }, { type: "function", name: "click" }, { type: "function", name: "browser_click" }, ]); - expect(catalog.incoming.tzafonComputerName).toBe("computer"); + expect(catalog.incoming.openaiComputerName).toBe("computer"); }); it("composes Anthropic native browser declarations, access fallback, and ordinary functions", async () => { @@ -241,15 +238,21 @@ describe("compileCuaToolCatalog", () => { expect(catalog.entries[0]?.dynamicLoading).toBe("eager-only"); }); - it("serializes Google's current native declaration", async () => { + it("serializes Google's current native declaration and keeps custom functions", async () => { const selected = cua.providers.google.toolsets.browser({ exclude: ["right_click", "triple_click"] }); - const catalog = compile("google:gemini-3.6-flash", selected); - const next = await catalog.payload.apply({ tools: selected.map((tool) => ({ type: "function", name: tool.name })) }, catalog.model) as { tools: unknown[] }; - expect(next.tools).toEqual([{ - type: "computer_use", - environment: "browser", - excluded_predefined_functions: ["triple_click", "right_click"], - }]); + const catalog = compile("google:gemini-3.6-flash", [...selected, callerTool("custom")]); + const next = await catalog.payload.apply({ tools: [ + ...selected.map((tool) => ({ type: "function", name: tool.name })), + { type: "function", name: "custom" }, + ] }, catalog.model) as { tools: unknown[] }; + expect(next.tools).toEqual([ + { + type: "computer_use", + environment: "browser", + excluded_predefined_functions: ["triple_click", "right_click"], + }, + { type: "function", name: "custom" }, + ]); expect(catalog.entries[0]?.declaration).toEqual(next.tools[0]); expect(catalog.entries[0]?.coordinates).toEqual({ type: "normalized", range: [0, 999] }); const click = selected.find((tool) => tool.name === "click")!; @@ -289,41 +292,18 @@ describe("compileCuaToolCatalog", () => { } }); - it("serializes state-mutating Meta/xAI/Moonshot catalogs with serial tool calls", async () => { - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3"] as const) { + it("serializes state-mutating catalogs with serial tool calls", async () => { + for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { const catalog = compile(model, cua.toolsets.browser()); await expect(catalog.payload.apply({ parallel_tool_calls: true }, catalog.model)).resolves.toMatchObject({ parallel_tool_calls: false }); } }); - it("uses selected Yutori identities for disable_tools and keeps custom functions", async () => { - const selected = cua.providers.yutori.toolsets.n15Core().slice(0, 2); - const catalog = compile("yutori:n1.5-latest", [...selected, callerTool("custom")]); - const payload = { messages: [{ role: "user", content: "go" }], tools: [ - ...selected.map((tool) => ({ type: "function", function: { name: tool.name } })), - { type: "function", function: { name: "custom" } }, - ] }; - const next = await catalog.payload.apply(payload, catalog.model) as { - tool_set: string; - disable_tools: string[]; - tools: Array<{ function: { name: string } }>; - messages: Array<{ content: unknown }>; - }; - expect(next.tool_set).toBe("browser_tools_core-20260403"); - expect(next.disable_tools).not.toContain(selected[0]?.name); - expect(next.disable_tools).toContain("right_click"); - expect(next.tools.map((tool) => tool.function.name)).toEqual(["custom"]); - expect(next.messages).toEqual(payload.messages); - }); - - it("rejects partial n1 selection and incompatible model changes", () => { - expect(() => compile("yutori:n1-latest", cua.providers.yutori.toolsets.n1().slice(0, 1))).toThrow(/complete .*n1\(\)/); + it("rejects incompatible model changes", () => { const nativeTools: Array<[CuaToolSpec[], string]> = [ [[cua.providers.anthropic.tools.browser()], "anthropic"], [[cua.providers.openai.tools.computer()], "openai"], [[cua.providers.google.toolsets.browser()[0]!], "google"], - [[cua.providers.tzafon.tools.computer()], "tzafon"], - [cua.providers.yutori.toolsets.n1(), "yutori"], ]; for (const [tools, provider] of nativeTools) { expect(() => compile("openrouter:moonshotai/kimi-k3", tools)).toThrow(new RegExp(`requires a ${provider} model`)); @@ -339,7 +319,7 @@ describe("compileCuaToolCatalog", () => { expect(pixels.entries[0]?.fingerprint).not.toBe(normalized.entries[0]?.fingerprint); }); - it("produces deterministic fingerprints for identical declaration, model, and viewport inputs", () => { + it("produces deterministic fingerprints for identical declaration and model inputs", () => { const compileInputs = () => [cua.tools.browser.snapshot(), cua.tools.computer.click(), callerTool("custom")]; const first = compile("openai:gpt-5.5", compileInputs()); const second = compile("openai:gpt-5.5", compileInputs()); @@ -348,3 +328,54 @@ describe("compileCuaToolCatalog", () => { expect(second.toolDeclarations.map((tool) => tool.name)).toEqual(first.toolDeclarations.map((tool) => tool.name)); }); }); + +describe("transport derivation", () => { + it("keeps an OpenAI model on its registry api when only CUA browser tools are selected", () => { + const catalog = compile("openai:gpt-5.5", cua.toolsets.browser()); + expect(catalog.model.api).toBe("openai-responses"); + }); + + it("derives OPENAI_CUA_COMPUTER_API when OpenAI's native computer tool is selected", () => { + const catalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); + expect(catalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + }); + + it("keeps a Google model on pi's builtin transport when only CDP browser tools are selected", () => { + const catalog = compile("google:gemini-3.6-flash", [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + expect(catalog.model.api).toBe("google-generative-ai"); + }); + + it("derives GOOGLE_CUA_INTERACTIONS_API when Google's native browser toolset is selected", () => { + const catalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); + expect(catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + }); + + it("rejects a catalog whose selected tools require conflicting transports", () => { + const [click, scroll] = cua.providers.google.toolsets.browser(); + const conflicting: CuaToolSpec = { + ...scroll!, + identity: "test.conflicting-transport.v1", + providerBinding: { kind: "google-native", nativeName: "conflict", allNativeNames: ["conflict"], requiresApi: OPENAI_CUA_COMPUTER_API }, + }; + expect(() => compile("google:gemini-3.6-flash", [click!, conflicting])).toThrow(/incompatible provider transports/); + }); + + it("re-derives from a model object that already carries a stale derived api, instead of pinning it", () => { + const nativeCatalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); + expect(nativeCatalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + const recompiled = compile(nativeCatalog.model, [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + expect(recompiled.model.api).toBe("google-generative-ai"); + + const reselected = compile(recompiled.model, cua.providers.google.toolsets.browser()); + expect(reselected.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + }); + + it("re-derives an OpenAI model object that already carries a stale derived api, instead of pinning it", () => { + const nativeCatalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); + expect(nativeCatalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + + const recompiled = compile(nativeCatalog.model, cua.toolsets.browser()); + expect(recompiled.model.api).toBe("openai-responses"); + }); +}); diff --git a/packages/ai/test/tzafon-actions.test.ts b/packages/ai/test/tzafon-actions.test.ts deleted file mode 100644 index b45820c6..00000000 --- a/packages/ai/test/tzafon-actions.test.ts +++ /dev/null @@ -1,80 +0,0 @@ -import { describe, expect, it } from "vitest"; -import * as tzafon from "../src/providers/tzafon/provider"; - -describe("Tzafon native action normalization", () => { - it("normalizes click variants", () => { - expect(tzafon.toCanonicalActions({ type: "click", x: 10, y: 20 })).toEqual([{ type: "click", x: 10, y: 20 }]); - expect(tzafon.toCanonicalActions({ type: "left_click", x: 10, y: 20 })).toEqual([{ type: "click", x: 10, y: 20 }]); - expect(tzafon.toCanonicalActions({ type: "right_click", x: 10, y: 20 })).toEqual([ - { type: "click", x: 10, y: 20, button: "right" }, - ]); - expect(tzafon.toCanonicalActions({ type: "double_click", x: 10, y: 20 })).toEqual([ - { type: "double_click", x: 10, y: 20 }, - ]); - expect(tzafon.toCanonicalActions({ type: "triple_click", x: 10, y: 20 })).toEqual([ - { type: "double_click", x: 10, y: 20 }, - { type: "click", x: 10, y: 20 }, - ]); - }); - - it("coerces string coordinates to numbers", () => { - expect(tzafon.toCanonicalActions({ type: "click", x: "10", y: "20" })).toEqual([{ type: "click", x: 10, y: 20 }]); - }); - - it("drops pointer actions without usable coordinates", () => { - expect(tzafon.toCanonicalActions({ type: "click" })).toEqual([]); - expect(tzafon.toCanonicalActions({ type: "hover", x: 5 })).toEqual([]); - expect(tzafon.toCanonicalActions(undefined)).toEqual([]); - expect(tzafon.toCanonicalActions({ type: "unknown_action" })).toEqual([]); - }); - - it("normalizes move, hover, and drag", () => { - expect(tzafon.toCanonicalActions({ type: "move", x: 1, y: 2 })).toEqual([{ type: "move", x: 1, y: 2 }]); - expect(tzafon.toCanonicalActions({ type: "hover", x: 1, y: 2 })).toEqual([{ type: "move", x: 1, y: 2 }]); - expect(tzafon.toCanonicalActions({ type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - expect(tzafon.toCanonicalActions({ type: "drag", x: 1, y: 2, end_x: 3, end_y: 4 })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - expect(tzafon.toCanonicalActions({ type: "drag", x: 1, y: 2, x2: 3, y2: 4 })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - }); - - it("normalizes typing and keyboard actions", () => { - expect(tzafon.toCanonicalActions({ type: "type", text: "hello" })).toEqual([{ type: "type", text: "hello" }]); - expect(tzafon.toCanonicalActions({ type: "key", key: "enter" })).toEqual([{ type: "keypress", keys: ["enter"] }]); - expect(tzafon.toCanonicalActions({ type: "key", text: "esc" })).toEqual([{ type: "keypress", keys: ["esc"] }]); - expect(tzafon.toCanonicalActions({ type: "keypress", keys: ["ctrl", "a"] })).toEqual([ - { type: "keypress", keys: ["ctrl", "a"] }, - ]); - expect(tzafon.toCanonicalActions({ type: "keypress" })).toEqual([]); - }); - - it("normalizes scroll variants", () => { - expect(tzafon.toCanonicalActions({ type: "scroll", x: 5, y: 6, scroll_y: 120 })).toEqual([ - { type: "scroll", x: 5, y: 6, scroll_y: 120 }, - ]); - expect(tzafon.toCanonicalActions({ type: "scroll", amount: 240 })).toEqual([{ type: "scroll", scroll_y: 240 }]); - expect(tzafon.toCanonicalActions({ type: "hscroll", amount: 120 })).toEqual([{ type: "scroll", scroll_x: 120 }]); - expect(tzafon.toCanonicalActions({ type: "hscroll" })).toEqual([{ type: "scroll", scroll_x: 0 }]); - }); - - it("normalizes navigation, waits, and screenshots", () => { - expect(tzafon.toCanonicalActions({ type: "navigate", url: "https://example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - ]); - expect(tzafon.toCanonicalActions({ type: "wait", ms: 500 })).toEqual([{ type: "wait", ms: 500 }]); - expect(tzafon.toCanonicalActions({ type: "wait", seconds: 2 })).toEqual([{ type: "wait", ms: 2000 }]); - expect(tzafon.toCanonicalActions({ type: "screenshot" })).toEqual([{ type: "screenshot" }]); - }); - - it("maps terminal actions to answer text", () => { - expect(tzafon.toCanonicalActions({ type: "answer", text: "done!" })).toEqual([{ type: "answer", text: "done!" }]); - expect(tzafon.toCanonicalActions({ type: "done", result: "ok" })).toEqual([{ type: "answer", text: "ok" }]); - expect(tzafon.toCanonicalActions({ type: "terminate", status: "success" })).toEqual([ - { type: "answer", text: "success" }, - ]); - }); -}); diff --git a/packages/ai/test/tzafon-provider.test.ts b/packages/ai/test/tzafon-provider.test.ts deleted file mode 100644 index 8208527a..00000000 --- a/packages/ai/test/tzafon-provider.test.ts +++ /dev/null @@ -1,149 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; -import type { Model, ToolCall } from "@earendil-works/pi-ai"; -import { getCuaModel } from "../src/index"; -import * as tzafon from "../src/providers/tzafon/provider"; - -const { lightconeConstructor, responsesCreate } = vi.hoisted(() => ({ - lightconeConstructor: vi.fn(), - responsesCreate: vi.fn(), -})); - -vi.mock("@tzafon/lightcone", () => ({ - default: class { - responses = { create: responsesCreate }; - constructor(options: unknown) { - lightconeConstructor(options); - } - }, -})); - -const model = getCuaModel("tzafon:tzafon.northstar-cua-fast") as Model; - -function toolCalls(content: Array<{ type: string }>): ToolCall[] { - return content.filter((part): part is ToolCall => part.type === "toolCall"); -} - -describe("streamTzafonResponses", () => { - it("configures the resolvable Tzafon API endpoint", async () => { - responsesCreate.mockResolvedValueOnce({ id: "resp_endpoint", usage: {}, output: [] }); - - await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - - expect(model.baseUrl).toBe("https://api.tzafon.ai"); - expect(lightconeConstructor).toHaveBeenLastCalledWith(expect.objectContaining({ - apiKey: "test", - baseURL: "https://api.tzafon.ai", - })); - }); - - it("derives unique ids when one computer_call expands to multiple actions", () => { - expect(tzafon.tzafonToolCallId("call_1", 0)).toBe("call_1"); - expect(tzafon.tzafonToolCallId("call_1", 1)).toBe("call_1:1"); - expect(tzafon.tzafonToolCallId("call_1", 2)).toBe("call_1:2"); - }); - - it("unwraps stringified nested arguments and coerces numeric strings on function calls", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_1", - usage: { input_tokens: 1, output_tokens: 2 }, - output: [ - { - type: "function_call", - call_id: "call_1", - name: "computer_batch", - // Observed Tzafon shape: the actions array arrives JSON-encoded - // inside the argument object, with stringified coordinates. - arguments: JSON.stringify({ actions: JSON.stringify([{ type: "click", x: "10", y: "20" }]) }), - }, - ], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!.name).toBe("computer_batch"); - expect(calls[0]!.arguments).toEqual({ actions: [{ type: "click", x: 10, y: 20 }] }); - }); - - it("normalizes non-native computer_call actions with string coordinates", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_2", - usage: {}, - output: [{ type: "computer_call", call_id: "call_2", action: { type: "left_click", x: "500", y: "250" } }], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!.name).toBe("click"); - expect(calls[0]!.arguments).toEqual({ x: 500, y: 250 }); - }); - - it("rejects native computer actions that require automatic post-action screenshots", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_native_click", - usage: {}, - output: [{ type: "computer_call", call_id: "call_click", action: { type: "click", x: 500, y: 500 } }], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { - apiKey: "test", - cuaIncomingToolPlan: { tzafonComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, - }).result(); - - expect(message.stopReason).toBe("error"); - expect(message.errorMessage).toContain('Tzafon native computer action "click" is unsupported'); - expect(toolCalls(message.content)).toEqual([]); - }); - - it("rejects text-only native computer results before sending a request", () => { - expect(() => tzafon.buildTzafonRequestInput(model, { - messages: [ - { - role: "assistant", - content: [{ type: "toolCall", id: "call_click", name: "computer", arguments: { action: { type: "click", x: 500, y: 500 } } }], - api: model.api, - provider: model.provider, - model: model.id, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 1, - }, - { - role: "toolResult", - toolCallId: "call_click", - toolName: "computer", - content: [{ type: "text", text: "Actions executed successfully." }], - isError: false, - timestamp: 2, - }, - ], - tools: [], - }, { - disableResponseThreading: true, - cuaIncomingToolPlan: { tzafonComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, - })).toThrow("text-only results are unsupported"); - }); - - it("degrades malformed function-call arguments to empty args instead of failing the turn", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_3", - usage: {}, - output: [ - { type: "function_call", call_id: "call_bad", name: "custom_tool", arguments: "{not json" }, - { type: "computer_call", call_id: "call_good", action: { type: "left_click", x: 1, y: 2 } }, - ], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(2); - expect(calls[0]!).toMatchObject({ name: "custom_tool", arguments: {} }); - expect(calls[1]!).toMatchObject({ name: "click", arguments: { x: 1, y: 2 } }); - }); -}); diff --git a/packages/ai/test/tzafon-threading.test.ts b/packages/ai/test/tzafon-threading.test.ts deleted file mode 100644 index 36ac2d44..00000000 --- a/packages/ai/test/tzafon-threading.test.ts +++ /dev/null @@ -1,159 +0,0 @@ -import { describe, expect, it } from "vitest"; -import type { Context, Message, Model } from "@earendil-works/pi-ai"; -import * as tzafon from "../src/providers/tzafon/provider"; - -const model = { id: "tzafon.northstar-cua-fast", maxTokens: 4096 } as Model; - -const TURNS = 6; - -/** Build a multi-turn context where each assistant turn carries a distinct responseId followed by a screenshot tool result. */ -function multiTurnContext(): Context { - const messages: Message[] = [{ role: "user", content: "book a flight", timestamp: 0 }]; - for (let turn = 0; turn < TURNS; turn += 1) { - messages.push({ - role: "assistant", - content: [{ type: "toolCall", id: `call_${turn}`, name: "click", arguments: { x: turn, y: turn } }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - responseId: `resp_${turn}`, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 0, - }); - messages.push({ - role: "toolResult", - toolCallId: `call_${turn}`, - toolName: "click", - content: [{ type: "image", mimeType: "image/png", data: `screenshot-${turn}` }], - isError: false, - timestamp: 0, - }); - } - return { messages, tools: [], systemPrompt: "control the browser" }; -} - -function screenshotImageUrls(input: Array>): string[] { - const urls: string[] = []; - for (const item of input) { - const content = item.content; - if (!Array.isArray(content)) continue; - for (const part of content) { - if (part && typeof part === "object" && (part as { type?: string }).type === "input_image") { - urls.push((part as { image_url: string }).image_url); - } - } - } - return urls; -} - -describe("buildTzafonRequestInput response threading", () => { - // Threading ON is the fix: chain via previous_response_id and send only the - // latest screenshot. OFF sends every screenshot — the per-turn growth that - // overflows Tzafon's real 64K window after a few turns. - it("threads the latest delta when enabled (default)", () => { - const body = tzafon.buildTzafonRequestInput(model, multiTurnContext()); - const screenshots = screenshotImageUrls(body.input); - - expect(screenshots).toHaveLength(1); - expect(screenshots[0]).toBe(`data:image/png;base64,screenshot-${TURNS - 1}`); - expect(body.previous_response_id).toBe(`resp_${TURNS - 1}`); - expect(body.store).toBe(true); - }); - - it("sends the full screenshot history when threading is disabled by option (locks the failure mode)", () => { - const body = tzafon.buildTzafonRequestInput(model, multiTurnContext(), { disableResponseThreading: true }); - const screenshots = screenshotImageUrls(body.input); - - expect(screenshots).toHaveLength(TURNS); - expect(screenshots).toEqual(Array.from({ length: TURNS }, (_, turn) => `data:image/png;base64,screenshot-${turn}`)); - expect(body.previous_response_id).toBeUndefined(); - expect(body.store).toBeUndefined(); - }); - - it("falls back to full history when no prior turn carries a responseId", () => { - const context = multiTurnContext(); - for (const message of context.messages) { - if (message.role === "assistant") delete message.responseId; - } - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - expect(body.previous_response_id).toBeUndefined(); - }); - - it("replays full history when the latest assistant turn has no responseId (failed request)", () => { - const context = multiTurnContext(); - // A newer assistant turn whose request errored carries no responseId; threading must - // anchor on it and replay, not chain to the older id and re-send the items past it. - context.messages.push({ - role: "assistant", - content: [{ type: "text", text: "request failed" }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "error", - timestamp: 0, - }); - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(body.previous_response_id).toBeUndefined(); - expect(body.store).toBeUndefined(); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - }); - - it("replays full history when the latest assistant turn is from a different api", () => { - const context = multiTurnContext(); - context.messages.push({ - role: "assistant", - content: [{ type: "text", text: "done" }], - api: "anthropic-messages", - provider: "anthropic", - model: "claude-opus-4-8", - responseId: "msg_anthropic", - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "stop", - timestamp: 0, - }); - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(body.previous_response_id).toBeUndefined(); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - }); - - // Off-path screenshot count scales with turn count; on-path stays constant at one. - it("grows the payload per turn when off but stays flat when on", () => { - const counts = (turns: number, disable: boolean) => { - const messages: Message[] = [{ role: "user", content: "task", timestamp: 0 }]; - for (let turn = 0; turn < turns; turn += 1) { - messages.push({ - role: "assistant", - content: [{ type: "toolCall", id: `c_${turn}`, name: "click", arguments: {} }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - responseId: `r_${turn}`, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 0, - }); - messages.push({ - role: "toolResult", - toolCallId: `c_${turn}`, - toolName: "click", - content: [{ type: "image", mimeType: "image/png", data: `s-${turn}` }], - isError: false, - timestamp: 0, - }); - } - const body = tzafon.buildTzafonRequestInput(model, { messages, tools: [] }, { disableResponseThreading: disable }); - return screenshotImageUrls(body.input).length; - }; - - expect(counts(3, true)).toBe(3); - expect(counts(8, true)).toBe(8); - expect(counts(3, false)).toBe(1); - expect(counts(8, false)).toBe(1); - }); -}); diff --git a/packages/ai/test/yutori-actions.test.ts b/packages/ai/test/yutori-actions.test.ts deleted file mode 100644 index c901302e..00000000 --- a/packages/ai/test/yutori-actions.test.ts +++ /dev/null @@ -1,97 +0,0 @@ -import { describe, expect, it } from "vitest"; -import type { CuaAction } from "../src/index"; -import * as yutori from "../src/providers/yutori/actions"; - -const n15CoreActionArgs = { - left_click: { coordinates: [500, 250] }, - double_click: { coordinates: [500, 250] }, - triple_click: { coordinates: [500, 250] }, - middle_click: { coordinates: [500, 250] }, - right_click: { coordinates: [500, 250] }, - mouse_move: { coordinates: [100, 200] }, - mouse_down: { coordinates: [100, 200] }, - mouse_up: { coordinates: [100, 200] }, - drag: { start_coordinates: [100, 200], coordinates: [300, 400] }, - scroll: { coordinates: [500, 500], direction: "down", amount: 3 }, - type: { text: "hello" }, - key_press: { key: "ctrl+c" }, - hold_key: { key: "shift", duration: 1.5 }, - goto_url: { url: "https://example.com" }, - go_back: {}, - go_forward: {}, - refresh: {}, - wait: { duration: 1 }, -} satisfies Record<(typeof yutori.YUTORI_N15_CORE_ACTION_TYPES)[number], Record>; - -describe("Yutori native action normalization", () => { - it("has canonical mappings for every n1.5 core action", () => { - for (const action of yutori.YUTORI_N15_CORE_ACTION_TYPES) { - const canonical = yutori.toCanonicalActions(action, n15CoreActionArgs[action]); - expect(canonical, `${action} did not map to canonical CUA actions`).toBeDefined(); - expect(canonical!.length, `${action} mapped to an empty action list`).toBeGreaterThan(0); - for (const item of canonical as CuaAction[]) { - expect(typeof item.type).toBe("string"); - } - } - }); - - it("normalizes n1/n1.5 click actions to canonical individual actions", () => { - expect(yutori.toCanonicalActions("left_click", { coordinates: [500, 250] })).toEqual([ - { type: "click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("double_click", { coordinates: [500, 250] })).toEqual([ - { type: "double_click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("triple_click", { coordinates: [500, 250] })).toEqual([ - { type: "double_click", x: 500, y: 250 }, - { type: "click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("middle_click", { coordinates: [500, 250] })).toEqual([ - { type: "click", x: 500, y: 250, button: "middle" }, - ]); - }); - - it("normalizes mouse, drag, type, and keyboard actions", () => { - expect(yutori.toCanonicalActions("mouse_move", { coordinates: [100, 200] })).toEqual([ - { type: "move", x: 100, y: 200 }, - ]); - expect(yutori.toCanonicalActions("drag", { start_coordinates: [100, 200], coordinates: [300, 400] })).toEqual([ - { type: "drag", path: [{ x: 100, y: 200 }, { x: 300, y: 400 }], button: "left" }, - ]); - expect(yutori.toCanonicalActions("type", { text: "hello", clear_before_typing: true, press_enter_after: true })).toEqual([ - { type: "keypress", keys: ["ctrl", "a"] }, - { type: "keypress", keys: ["backspace"] }, - { type: "type", text: "hello" }, - { type: "keypress", keys: ["enter"] }, - ]); - expect(yutori.toCanonicalActions("hold_key", { key: "shift", duration: 1.5 })).toEqual([ - { type: "keypress", keys: ["shift"], duration: 1500 }, - ]); - expect(yutori.toCanonicalActions("key_press", { key: "ctrl+c" })).toEqual([ - { type: "keypress", keys: ["ctrl", "c"] }, - ]); - expect(yutori.toCanonicalActions("key_press", { key: "down down enter" })).toEqual([ - { type: "keypress", keys: ["down"] }, - { type: "keypress", keys: ["down"] }, - { type: "keypress", keys: ["enter"] }, - ]); - }); - - it("normalizes scroll and navigation actions", () => { - expect(yutori.toCanonicalActions("scroll", { coordinates: [500, 500], direction: "down", amount: 3 })).toEqual([ - { type: "scroll", x: 500, y: 500, scroll_x: 0, scroll_y: 360 }, - ]); - expect(yutori.toCanonicalActions("goto_url", { url: "https://example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - { type: "wait", ms: 2000 }, - ]); - expect(yutori.toCanonicalActions("goto_url", { url: "example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - { type: "wait", ms: 2000 }, - ]); - expect(yutori.toCanonicalActions("refresh", {})).toEqual([ - { type: "keypress", keys: ["f5"] }, - { type: "wait", ms: 2000 }, - ]); - }); -}); diff --git a/packages/ai/test/yutori-provider.test.ts b/packages/ai/test/yutori-provider.test.ts deleted file mode 100644 index 533c4f7c..00000000 --- a/packages/ai/test/yutori-provider.test.ts +++ /dev/null @@ -1,72 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; -import type { Model, ToolCall } from "@earendil-works/pi-ai"; -import { getCuaModel } from "../src/index"; -import * as yutori from "../src/providers/yutori/provider"; - -const { completionsCreate } = vi.hoisted(() => ({ completionsCreate: vi.fn() })); - -vi.mock("openai", () => ({ - default: class { - chat = { - completions: { - create: (...args: unknown[]) => ({ - withResponse: async () => ({ data: completionsCreate(...args), response: { status: 200, headers: new Headers() } }), - }), - }, - }; - }, -})); - -const model = getCuaModel("yutori:n1.5-latest") as Model; -const incoming = { yutoriNames: { left_click: "left_click" }, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["left_click"] }; - -function toolCalls(content: Array<{ type: string }>): ToolCall[] { - return content.filter((part): part is ToolCall => part.type === "toolCall"); -} - -describe("streamYutori", () => { - it("dispatches selected native calls through fixed catalog names", async () => { - completionsCreate.mockReturnValueOnce({ - id: "chatcmpl_1", - usage: { prompt_tokens: 1, completion_tokens: 2, total_tokens: 3 }, - choices: [{ - finish_reason: "tool_calls", - message: { - content: "", - tool_calls: [{ type: "function", id: "call_1", function: { name: "left_click", arguments: JSON.stringify({ coordinates: [100, 200] }) } }], - }, - }], - }); - - const message = await yutori.streamYutori(model, { messages: [] }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); - expect(message.stopReason).toBe("toolUse"); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!).toMatchObject({ id: "call_1", name: "left_click", arguments: { coordinates: [100, 200] } }); - }); - - it("degrades malformed selected calls to empty args without reclassifying by name", async () => { - completionsCreate.mockReturnValueOnce({ - id: "chatcmpl_2", - usage: { prompt_tokens: 1, completion_tokens: 2, total_tokens: 3 }, - choices: [{ - finish_reason: "tool_calls", - message: { - content: "", - tool_calls: [ - { type: "function", id: "call_bad", function: { name: "left_click", arguments: "{not json" } }, - { type: "function", id: "call_good", function: { name: "left_click", arguments: JSON.stringify({ coordinates: [100, 200] }) } }, - ], - }, - }], - }); - - const message = await yutori.streamYutori(model, { messages: [] }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(2); - expect(calls[0]!).toMatchObject({ id: "call_bad", name: "left_click", arguments: {} }); - expect(calls[1]!).toMatchObject({ id: "call_good", name: "left_click", arguments: { coordinates: [100, 200] } }); - }); -}); diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 873ed864..68ba217f 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,35 @@ # Changelog +## 0.12.0 - 2026-08-13 + +- `-m meta:muse-spark-1.1` is removed; use `-m openrouter:meta/muse-spark-1.1`. + `META_API_KEY` is no longer read. +- The default interaction toolset for OpenRouter models is now chosen per model + rather than per provider. OpenRouter fronts several model families, and Kimi + K3 rejects `browser_act`'s schema while Muse Spark accepts it, so the CLI + asks the model's capabilities instead of assuming one answer per provider. + +Breaking: Tzafon and Yutori support is removed. + +- Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.13.0. Refs like + `-m tzafon:…` and `-m yutori:…` are no longer accepted, `cua models` no + longer lists either provider, and `TZAFON_API_KEY`/`YUTORI_API_KEY` are no + longer read. +- The `/tools` picker no longer has atomic tool groups. They existed only for + Yutori's n1 action set, which the catalog compiler refused to accept as a + partial selection; every remaining tool toggles on its own. + +## 0.11.0 - 2026-08-13 + +- Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.12.0. The default + Google interaction catalog is unchanged (`defaultInteractionTools` still + selects Google's native browser toolset), so the default `cua` model and + `--print -o jsonl`'s `assistant_usage.api` field for Google are unaffected + as long as that native toolset stays selected, including across an in-session + `/model` switch. Only a `/tools` selection that drops Google's native toolset + now reports pi's builtin `google-generative-ai` api instead of the CUA-owned + one. + ## 0.10.0 - 2026-08-13 - `--print -o jsonl` schema bumps to version 2: every assistant message now diff --git a/packages/cli/README.md b/packages/cli/README.md index e08e49ad..f3d2e861 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -47,10 +47,9 @@ cua models -p openai cua --print --model openai:gpt-5.6-sol "..." cua --print --model anthropic:claude-opus-5 "..." cua --print --model google:gemini-3.6-flash "..." -cua --print --model meta:muse-spark-1.1 "..." +cua --print --model openrouter:meta/muse-spark-1.1 "..." cua --print --model xai:grok-4.5 "..." cua --print --model moonshotai:kimi-k3 "..." -cua --print --model yutori:n1.5-latest "..." # Named sessions (browser stays alive across calls): cua session start login # provisions Kernel browser @@ -116,11 +115,7 @@ live tool list untouched. Applying calls the harness's `setTools()`, which compiles and validates the whole catalog before mutating anything — so a rejected selection reports the error and leaves the session unchanged. -Two constraints show up in the picker: - -- Provider-native action sets that cannot be partially suppressed (currently - Yutori n1) toggle as one group. -- Disabling every tool is allowed and yields a text-only agent. +Disabling every tool is allowed and yields a text-only agent. Selections are session-only and never persisted. `/model` rebuilds the tool list from the new model's defaults and reports `tool selection reset to the new @@ -138,7 +133,7 @@ Run `cua models` to list every supported `-m` / `--model` value and the provider it routes to. Filter by provider with `cua models -p openai`, `cua models -p anthropic`, `cua models -p google` (alias: `gemini`), `cua models -p meta`, `cua models -p xai`, `cua models -p moonshotai` -(alias: `moonshot`), `cua models -p openrouter`, or `cua models -p yutori`. +(alias: `moonshot`), or `cua models -p openrouter`. `-m` / `--model` accepts a provider-qualified `provider:model` ref (e.g. `openai:gpt-5.6-sol`) or a bare model id when it matches exactly one catalog @@ -155,12 +150,9 @@ Configuration is by environment variable. There is no config file. | `ANTHROPIC_API_KEY` | Anthropic API key (required when `-m anthropic:…`) | | `GOOGLE_API_KEY` | Google API key (required when `-m google:…`) | | `GEMINI_API_KEY` | alias of `GOOGLE_API_KEY` | -| `META_API_KEY` | Meta Model API key (required when `-m meta:…`) | | `XAI_API_KEY` | xAI API key (required when `-m xai:…`) | | `MOONSHOT_API_KEY` | Moonshot AI API key (required when `-m moonshotai:…`) | | `OPENROUTER_API_KEY` | OpenRouter API key (required when `-m openrouter:…`) | -| `TZAFON_API_KEY` | Tzafon API key (required when `-m tzafon:…`) | -| `YUTORI_API_KEY` | Yutori API key (required when `-m yutori:…`) | | `KERNEL_BASE_URL` | override Kernel base URL | | `OPENAI_BASE_URL` | override OpenAI base URL | | `ANTHROPIC_BASE_URL` | override Anthropic base URL | @@ -168,8 +160,6 @@ Configuration is by environment variable. There is no config file. | `META_BASE_URL` | override Meta Model API base URL | | `XAI_BASE_URL` | override xAI API base URL | | `MOONSHOTAI_BASE_URL` | override Moonshot AI base URL | -| `TZAFON_BASE_URL` | override Tzafon base URL | -| `YUTORI_BASE_URL` | override Yutori base URL | | `XDG_DATA_HOME` | sessions dir base (defaults to `~/.local/share`) | | `CUA_IMAGE_PROTOCOL` | force inline image protocol (`kitty`/`iterm2`/`none`/`auto`) | @@ -180,13 +170,9 @@ The CLI chooses one explicit interaction catalog and appends pi's coding tools: CUA browser primitives plus the verified `browser_act` plan tool for OpenAI, Meta, xAI, and older Anthropic models; browser primitives alone for Moonshot, whose API rejects `browser_act`'s larger schema; Anthropic's native browser tool -when supported; Google's native browser action set; Tzafon's native computer -tool in a browser environment; and Yutori's native N1/N1.5 browser set. If the -active Anthropic credential cannot access `browser_20260701`, the same selected -browser tool uses its equivalent function transport. Tzafon's native tool allows -explicit screenshots and terminal answers only; non-screenshot actions fail -before browser execution because CUA does not synthesize the post-action images -its continuation protocol requires. Library callers can select any catalog +when supported; and Google's native browser action set. If the active Anthropic +credential cannot access `browser_20260701`, the same selected browser tool uses +its equivalent function transport. Library callers can select any catalog directly; see [`@onkernel/cua-agent`](../agent). ## Output formats diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index b600e205..5193c0a9 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -57,7 +57,7 @@ Usage: cua models --json Options: - -p, --provider Filter by provider: openai | anthropic | google | gemini | meta | xai | moonshotai | openrouter | tzafon | yutori + -p, --provider Filter by provider: openai | anthropic | google | gemini | xai | moonshotai | openrouter --json Output JSON -h, --help Show this help `; diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 9dbb6b48..5fe06fde 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -55,11 +55,8 @@ Options: openai: openai:gpt-5.6-sol anthropic: anthropic:claude-opus-5 google: google:gemini-3.6-flash - meta: meta:muse-spark-1.1 xai: xai:grok-4.5 moonshot: moonshotai:kimi-k3 - tzafon: tzafon:tzafon.northstar-cua-fast - yutori: yutori:n1.5-latest --thinking Thinking level: off | minimal | low | medium | high | xhigh | max (default: low; applies to providers that support it) --profile Kernel browser profile to load @@ -97,12 +94,9 @@ Environment: ANTHROPIC_API_KEY Anthropic API key (required when -m anthropic:…) GOOGLE_API_KEY Google API key (required when -m google:…) GEMINI_API_KEY Alias for GOOGLE_API_KEY - META_API_KEY Meta Model API key (required when -m meta:…) XAI_API_KEY xAI API key (required when -m xai:…) MOONSHOT_API_KEY Moonshot AI API key (required when -m moonshotai:…) OPENROUTER_API_KEY OpenRouter API key (required when -m openrouter:…) - TZAFON_API_KEY Tzafon API key (required when -m tzafon:…) - YUTORI_API_KEY Yutori API key (required when -m yutori:…) KERNEL_BASE_URL Override Kernel base URL OPENAI_BASE_URL Override OpenAI base URL ANTHROPIC_BASE_URL Override Anthropic base URL @@ -110,8 +104,6 @@ Environment: META_BASE_URL Override Meta Model API base URL XAI_BASE_URL Override xAI API base URL MOONSHOTAI_BASE_URL Override Moonshot AI base URL - TZAFON_BASE_URL Override Tzafon base URL - YUTORI_BASE_URL Override Yutori base URL XDG_DATA_HOME Sessions are stored under \$XDG_DATA_HOME/cua/sessions (defaults to ~/.local/share/cua/sessions) CUA_IMAGE_PROTOCOL Force inline image protocol (\`kitty\`|\`iterm2\`|\`none\`|\`auto\`) diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index 488c6593..f9f71dde 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -11,6 +11,7 @@ import { import { type Api, cua, + cuaModelCapabilities, type CuaModelRef, getCuaModel, type Model, @@ -104,23 +105,16 @@ export function defaultInteractionTools(model: CuaModelRef): CuaCliTool[] { : structuredBrowserTools(); case "google": return cua.providers.google.toolsets.browser(); - case "tzafon": - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - return [ - ...(modelId.startsWith("n1.5") - ? cua.providers.yutori.toolsets.n15Core() - : cua.providers.yutori.toolsets.n1()), - cua.tools.computer.screenshot(), - ]; - case "meta": case "xai": return structuredBrowserTools(); case "moonshotai": case "openrouter": - // Kimi's API rejects the request outright once `browser_act`'s - // schema is attached, so Kimi keeps the browser primitives only. - return cua.toolsets.browser(); + // Kimi's API rejects the request outright once `browser_act`'s schema + // is attached. OpenRouter fronts several model families, so this is a + // per-model capability question rather than a per-provider one. + return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas + ? structuredBrowserTools() + : cua.toolsets.browser(); } } diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts index 9e1d52c8..ecf95eac 100644 --- a/packages/cli/src/tui/main.ts +++ b/packages/cli/src/tui/main.ts @@ -364,9 +364,9 @@ export async function runInteractive(opts: InteractiveOptions): Promise let exitRequested = false; /** - * Apply a model switch. Extracted verbatim from the previous - * `applyModelCommand` so the picker and `/model ` share one path, - * including the three-step tool transition and its rollback. + * Apply a model switch. The picker and `/model ` share this one path. + * A failed switch needs no rollback here: the harness compiles before it + * mutates and restores its own state if the mutation fails. */ const applySwitchModel = async (resolved: CuaModelRef): Promise => { // The exact list installed by this switch, kept so it can become the new @@ -374,22 +374,11 @@ export async function runInteractive(opts: InteractiveOptions): Promise // policy, in which case the switch never touches the tool list at all. let installedTools: readonly CuaCliTool[] | undefined; if (opts.interactionToolsForModel) { - const previousModel = opts.harness.getModel(); - const previousTools = opts.harness.getTools(); installedTools = [...opts.interactionToolsForModel(resolved), ...opts.applicationTools]; - try { - // Native catalogs can be incompatible across providers. Transition - // through the CLI-owned application tools before selecting the new - // model's explicit interaction catalog. - await opts.harness.setTools(opts.applicationTools); - await opts.harness.setModel(resolved); - await opts.harness.setTools(installedTools); - } catch (error) { - await opts.harness.setTools(opts.applicationTools); - await opts.harness.setModel(previousModel); - await opts.harness.setTools(previousTools); - throw error; - } + // Native catalogs are incompatible across providers, and the selected + // tools decide the transport, so the new model and its interaction + // catalog have to compile as one pair rather than in sequence. + await opts.harness.setModelAndTools(resolved, installedTools); } else { await opts.harness.setModel(resolved); } @@ -422,9 +411,9 @@ export async function runInteractive(opts: InteractiveOptions): Promise /** * Serialized entry point for a model switch. Queued behind any in-flight - * `/tools` apply so the apply's `setTools` cannot land between this switch's - * `setModel` and its final `setTools`. Rejects with the underlying failure; - * the harness has already rolled back by then. + * `/tools` apply so the apply's `setTools` cannot land mid-switch and compile + * its tool subset against the other model. Rejects with the underlying + * failure; the harness has already rolled back by then. */ const switchModel = (resolved: CuaModelRef): Promise => catalogQueue.run(() => applySwitchModel(resolved)); diff --git a/packages/cli/src/tui/tool-selection.ts b/packages/cli/src/tui/tool-selection.ts index 73a187db..76f34adc 100644 --- a/packages/cli/src/tui/tool-selection.ts +++ b/packages/cli/src/tui/tool-selection.ts @@ -16,12 +16,6 @@ export interface ToolSelectionItem { label: string; group: ToolGroup; description?: string; - /** - * Tools that the catalog compiler refuses to accept as a partial set, and - * which therefore toggle as one unit. Currently only Yutori's n1 native - * action set (`validateToolsetCompatibility` rejects partial n1 subsets). - */ - atomicGroup?: string; } /** Identity key for a caller-owned tool, using cua-ai's canonical identity helper. */ @@ -34,15 +28,6 @@ function toolGroup(tool: CuaCliTool): ToolGroup { return tool.origin === "provider-native" ? "native" : "cua"; } -function atomicGroupOf(tool: CuaCliTool): string | undefined { - if (!isCuaToolSpec(tool)) return undefined; - const binding = tool.providerBinding; - if (binding?.kind === "yutori-native" && binding.generation === "n1") { - return "provider.yutori.native.n1"; - } - return undefined; -} - function toolDescription(tool: CuaCliTool): string | undefined { const raw = isCuaToolSpec(tool) ? tool.declaration.description : tool.description; if (typeof raw !== "string") return undefined; @@ -59,13 +44,11 @@ function toolDescription(tool: CuaCliTool): string | undefined { export function describeTools(tools: readonly CuaCliTool[]): ToolSelectionItem[] { return tools.map((tool) => { const description = toolDescription(tool); - const atomicGroup = atomicGroupOf(tool); return { key: toolKey(tool), label: tool.name, group: toolGroup(tool), ...(description ? { description } : {}), - ...(atomicGroup ? { atomicGroup } : {}), }; }); } @@ -75,51 +58,25 @@ export function toolSearchText(item: ToolSelectionItem): string { return `${item.label} ${item.group} ${item.key}${item.description ? ` ${item.description}` : ""}`; } -/** Keys that must move together with `key` (itself included). */ -function linkedKeys(items: readonly ToolSelectionItem[], key: string): string[] { - const item = items.find((candidate) => candidate.key === key); - if (!item?.atomicGroup) return [key]; - return items.filter((candidate) => candidate.atomicGroup === item.atomicGroup).map((candidate) => candidate.key); -} - -/** - * Flip one row. Atomic groups move as a unit so a Yutori n1 selection can - * never be staged into a state the catalog compiler would reject. - */ -export function toggleTool( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - key: string, -): Set { +/** Flip one row. */ +export function toggleTool(enabled: ReadonlySet, key: string): Set { const next = new Set(enabled); - const keys = linkedKeys(items, key); - const turnOn = !enabled.has(key); - for (const linked of keys) { - if (turnOn) next.add(linked); - else next.delete(linked); - } + if (enabled.has(key)) next.delete(key); + else next.add(key); return next; } -/** Enable `keys` (expanding atomic groups). */ -export function enableTools( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - keys: readonly string[], -): Set { +/** Enable `keys`. */ +export function enableTools(enabled: ReadonlySet, keys: readonly string[]): Set { const next = new Set(enabled); - for (const key of keys) for (const linked of linkedKeys(items, key)) next.add(linked); + for (const key of keys) next.add(key); return next; } -/** Disable `keys` (expanding atomic groups). */ -export function disableTools( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - keys: readonly string[], -): Set { +/** Disable `keys`. */ +export function disableTools(enabled: ReadonlySet, keys: readonly string[]): Set { const next = new Set(enabled); - for (const key of keys) for (const linked of linkedKeys(items, key)) next.delete(linked); + for (const key of keys) next.delete(key); return next; } diff --git a/packages/cli/src/tui/tools-picker.ts b/packages/cli/src/tui/tools-picker.ts index ed11395a..3f5edab7 100644 --- a/packages/cli/src/tui/tools-picker.ts +++ b/packages/cli/src/tui/tools-picker.ts @@ -177,11 +177,6 @@ export class ToolsPickerComponent extends Container implements Focusable { if (selected.description) { this.listContainer.addChild(new Text(colors.muted(` ${selected.description}`), 0, 0)); } - if (selected.atomicGroup) { - this.listContainer.addChild( - new Text(colors.warning(" toggles as a group: the provider rejects partial native action sets"), 0, 0), - ); - } } } @@ -213,20 +208,20 @@ export class ToolsPickerComponent extends Container implements Focusable { if (kb.matches(data, "tui.select.confirm") || (data === " " && !this.searchInput.getValue())) { const item = this.filtered[this.selectedIndex]; if (item) { - this.staged = toggleTool(this.staged, this.items, item.key); + this.staged = toggleTool(this.staged, item.key); this.refresh(); this.tui.requestRender(); } return; } if (kb.matches(data, "cua.tools.enableAll")) { - this.staged = enableTools(this.staged, this.items, this.bulkTargets()); + this.staged = enableTools(this.staged, this.bulkTargets()); this.refresh(); this.tui.requestRender(); return; } if (kb.matches(data, "cua.tools.clearAll")) { - this.staged = disableTools(this.staged, this.items, this.bulkTargets()); + this.staged = disableTools(this.staged, this.bulkTargets()); this.refresh(); this.tui.requestRender(); return; diff --git a/packages/cli/test/cli-executor.test.ts b/packages/cli/test/cli-executor.test.ts index 543c9697..039860c8 100644 --- a/packages/cli/test/cli-executor.test.ts +++ b/packages/cli/test/cli-executor.test.ts @@ -31,8 +31,6 @@ const PROVIDER_ENV_KEYS = [ "META_API_KEY", "XAI_API_KEY", "MOONSHOT_API_KEY", - "TZAFON_API_KEY", - "YUTORI_API_KEY", ]; function baseFlags(overrides: Partial = {}): HarnessCliFlags { diff --git a/packages/cli/test/harness-assembly.test.ts b/packages/cli/test/harness-assembly.test.ts index d6e92507..0a7fd13d 100644 --- a/packages/cli/test/harness-assembly.test.ts +++ b/packages/cli/test/harness-assembly.test.ts @@ -4,7 +4,6 @@ import { InMemorySessionRepo, type Skill, } from "@onkernel/cua-agent"; -import { cua } from "@onkernel/cua-ai"; import { tmpdir } from "node:os"; import { mkdtempSync } from "node:fs"; import { join } from "node:path"; @@ -24,7 +23,7 @@ describe("buildCuaHarness", () => { const googleNames = defaultInteractionTools("google:gemini-3.6-flash").map((tool) => tool.name); expect(googleNames).toContain("take_screenshot"); expect(googleNames).not.toContain("browser_act"); - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { const tools = defaultInteractionTools(model); expect(tools[0]).toMatchObject({ name: "browser_snapshot", origin: "cua" }); expect(tools.at(-1)?.name).toBe("browser_act"); @@ -36,11 +35,6 @@ describe("buildCuaHarness", () => { expect(kimiNames).not.toContain("browser_act"); expect(kimiNames).toContain("browser_wait_for"); } - expect(defaultInteractionTools("tzafon:tzafon.northstar-cua-fast")[0]?.name).toBe("computer"); - expect(defaultInteractionTools("yutori:n1.5-latest").map((tool) => tool.name)).toEqual([ - ...cua.providers.yutori.toolsets.n15Core().map((tool) => tool.name), - "computer_screenshot", - ]); }); it("installs interaction and coding tools in one explicit default list", async () => { diff --git a/packages/cli/test/harness-models.test.ts b/packages/cli/test/harness-models.test.ts index 4e1377c8..2f87f794 100644 --- a/packages/cli/test/harness-models.test.ts +++ b/packages/cli/test/harness-models.test.ts @@ -11,7 +11,7 @@ describe("resolveCuaModelRef", () => { expect(resolveCuaModelRef("openai:gpt-5.6-sol")).toBe("openai:gpt-5.6-sol"); expect(resolveCuaModelRef("openai:gpt-5.5")).toBe("openai:gpt-5.5"); expect(resolveCuaModelRef("anthropic:claude-opus-5")).toBe("anthropic:claude-opus-5"); - expect(resolveCuaModelRef("meta:muse-spark-1.1")).toBe("meta:muse-spark-1.1"); + expect(resolveCuaModelRef("openrouter:meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("xai:grok-4.5")).toBe("xai:grok-4.5"); expect(resolveCuaModelRef("moonshotai:kimi-k3")).toBe("moonshotai:kimi-k3"); expect(resolveCuaModelRef("moonshot:kimi-k3")).toBe("moonshotai:kimi-k3"); @@ -21,7 +21,7 @@ describe("resolveCuaModelRef", () => { expect(resolveCuaModelRef("gpt-5.6-sol")).toBe("openai:gpt-5.6-sol"); expect(resolveCuaModelRef("gpt-5.5")).toBe("openai:gpt-5.5"); expect(resolveCuaModelRef("claude-opus-5")).toBe("anthropic:claude-opus-5"); - expect(resolveCuaModelRef("muse-spark-1.1")).toBe("meta:muse-spark-1.1"); + expect(resolveCuaModelRef("meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("grok-4.5")).toBe("xai:grok-4.5"); expect(resolveCuaModelRef("kimi-k3")).toBe("moonshotai:kimi-k3"); }); @@ -31,11 +31,10 @@ describe("resolveCuaModelRef", () => { }); it("filters custom provider catalogs", () => { - expect(listSupportedModels("meta").map((model) => model.ref)).toEqual(["meta:muse-spark-1.1"]); expect(listSupportedModels("xai").map((model) => model.ref)).toEqual(["xai:grok-4.5"]); expect(listSupportedModels("moonshotai").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); expect(listSupportedModels("moonshot").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:moonshotai/kimi-k3"]); + expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:meta/muse-spark-1.1", "openrouter:moonshotai/kimi-k3"]); expect(resolveCuaModelRef("openrouter:moonshotai/kimi-k3")).toBe("openrouter:moonshotai/kimi-k3"); }); diff --git a/packages/cli/test/tool-revalidation.test.ts b/packages/cli/test/tool-revalidation.test.ts index d2c5dac7..de67962a 100644 --- a/packages/cli/test/tool-revalidation.test.ts +++ b/packages/cli/test/tool-revalidation.test.ts @@ -24,29 +24,25 @@ describe("/tools selection revalidation", () => { expect(fixture.harness.getTools().map(toolKey)).not.toContain(dropped.key); }); - it("rejects a partial Yutori n1 native subset and leaves the catalog unchanged", async () => { - const modelRef = "yutori:n1-latest"; + it("rejects a duplicated tool name and leaves the catalog unchanged", async () => { + const modelRef = "google:gemini-3.6-flash"; const baseline = [...defaultInteractionTools(modelRef), ...defaultApplicationTools()]; const fixture = await buildTestHarness({ turns: [], modelRef, tools: baseline }); const before = fixture.harness.getTools().map(toolKey); - const items = describeTools(baseline); - const oneNative = items.find((item) => item.atomicGroup)!; - const partial = baseline.filter((tool) => toolKey(tool) !== oneNative.key); - - await expect(fixture.harness.setTools(partial)).rejects.toThrow(/partial native action set/); + const [first] = baseline; + await expect(fixture.harness.setTools([...baseline, first!])).rejects.toThrow(/requested more than once/); // Atomicity: the failed compile must not have mutated live state. expect(fixture.harness.getTools().map(toolKey)).toEqual(before); }); - it("accepts dropping the whole Yutori n1 native group", async () => { - const modelRef = "yutori:n1-latest"; + it("accepts dropping the whole Google native group", async () => { + const modelRef = "google:gemini-3.6-flash"; const baseline = [...defaultInteractionTools(modelRef), ...defaultApplicationTools()]; const fixture = await buildTestHarness({ turns: [], modelRef, tools: baseline }); - const items = describeTools(baseline); - const atomicKeys = new Set(items.filter((item) => item.atomicGroup).map((item) => item.key)); - const next = baseline.filter((tool) => !atomicKeys.has(toolKey(tool))); + const nativeKeys = new Set(describeTools(baseline).filter((item) => item.group === "native").map((item) => item.key)); + const next = baseline.filter((tool) => !nativeKeys.has(toolKey(tool))); await fixture.harness.setTools(next); expect(fixture.harness.getTools().map(toolKey)).toEqual(next.map(toolKey)); @@ -72,10 +68,9 @@ describe("/tools selection revalidation", () => { tools: [...defaultInteractionTools(from), ...application], }); - // Mirrors the TUI's three-step transition in switchModel(). - await fixture.harness.setTools(application); - await fixture.harness.setModel(to); - await fixture.harness.setTools([...defaultInteractionTools(to), ...application]); + // Mirrors switchModel(): the new model and its interaction catalog compile + // as one pair, because the selected tools decide the transport. + await fixture.harness.setModelAndTools(to, [...defaultInteractionTools(to), ...application]); const expected = [...defaultInteractionTools(to), ...application].map(toolKey); expect(fixture.harness.getTools().map(toolKey)).toEqual(expected); diff --git a/packages/cli/test/tool-selection.test.ts b/packages/cli/test/tool-selection.test.ts index 7ada4e18..f3e7b087 100644 --- a/packages/cli/test/tool-selection.test.ts +++ b/packages/cli/test/tool-selection.test.ts @@ -46,20 +46,6 @@ describe("describeTools", () => { const cuaItems = describeTools(defaultInteractionTools("openai:gpt-5.6-sol")); expect(cuaItems.every((item) => item.group === "cua")).toBe(true); }); - - it("marks the Yutori n1 native action set as one atomic group", () => { - const items = describeTools(defaultInteractionTools("yutori:n1-latest")); - const atomic = items.filter((item) => item.atomicGroup); - expect(atomic.length).toBeGreaterThan(1); - expect(new Set(atomic.map((item) => item.atomicGroup))).toEqual(new Set(["provider.yutori.native.n1"])); - // The screenshot helper the CLI appends is not part of the native set. - expect(items.some((item) => !item.atomicGroup)).toBe(true); - }); - - it("leaves Yutori n1.5 individually toggleable", () => { - const items = describeTools(defaultInteractionTools("yutori:n1.5-latest")); - expect(items.every((item) => item.atomicGroup === undefined)).toBe(true); - }); }); describe("toolSearchText", () => { @@ -79,45 +65,24 @@ describe("selection state machine", () => { it("toggles a single tool off and back on", () => { const target = allKeys[0]!; - const off = toggleTool(new Set(allKeys), items, target); + const off = toggleTool(new Set(allKeys), target); expect(off.has(target)).toBe(false); expect(off.size).toBe(allKeys.length - 1); - const on = toggleTool(off, items, target); + const on = toggleTool(off, target); expect(sameSelection(on, new Set(allKeys))).toBe(true); }); it("enables and clears in bulk", () => { - expect(disableTools(new Set(allKeys), items, allKeys).size).toBe(0); - expect(sameSelection(enableTools(new Set(), items, allKeys), new Set(allKeys))).toBe(true); + expect(disableTools(new Set(allKeys), allKeys).size).toBe(0); + expect(sameSelection(enableTools(new Set(), allKeys), new Set(allKeys))).toBe(true); }); it("restricts bulk actions to the keys it is given", () => { const subset = allKeys.slice(0, 2); - const cleared = disableTools(new Set(allKeys), items, subset); + const cleared = disableTools(new Set(allKeys), subset); expect(cleared.size).toBe(allKeys.length - 2); for (const key of subset) expect(cleared.has(key)).toBe(false); }); - - it("moves an atomic group as one unit", () => { - const yutori = describeTools(defaultInteractionTools("yutori:n1-latest")); - const yutoriKeys = yutori.map((item) => item.key); - const nativeKeys = yutori.filter((item) => item.atomicGroup).map((item) => item.key); - const standalone = yutori.filter((item) => !item.atomicGroup).map((item) => item.key); - - const off = toggleTool(new Set(yutoriKeys), yutori, nativeKeys[0]!); - for (const key of nativeKeys) expect(off.has(key)).toBe(false); - for (const key of standalone) expect(off.has(key)).toBe(true); - - const on = toggleTool(off, yutori, nativeKeys[1]!); - for (const key of nativeKeys) expect(on.has(key)).toBe(true); - }); - - it("expands atomic groups for bulk disables too", () => { - const yutori = describeTools(defaultInteractionTools("yutori:n1-latest")); - const first = yutori.find((item) => item.atomicGroup)!; - const cleared = disableTools(new Set(yutori.map((i) => i.key)), yutori, [first.key]); - expect(yutori.filter((i) => i.atomicGroup).every((i) => !cleared.has(i.key))).toBe(true); - }); }); describe("sameSelection", () => { diff --git a/skills/cua-cli/SKILL.md b/skills/cua-cli/SKILL.md index 589ad650..26d720ff 100644 --- a/skills/cua-cli/SKILL.md +++ b/skills/cua-cli/SKILL.md @@ -145,11 +145,10 @@ Useful flags: - `-m ` — pick the LLM for model-mediated subcommands (default `gpt-5.6-sol`). Recommended refs are `openai:gpt-5.6-sol`, `anthropic:claude-opus-5`, - `google:gemini-3.6-flash`, `meta:muse-spark-1.1`, `xai:grok-4.5`, - `moonshotai:kimi-k3`, `tzafon:tzafon.northstar-cua-fast`, and - `yutori:n1.5-latest`. + `google:gemini-3.6-flash`, `xai:grok-4.5`, + and `moonshotai:kimi-k3`. - `cua models` — list supported `-m` values and their providers; filter with - `cua models -p openai|anthropic|google|meta|xai|moonshotai|tzafon|yutori`. + `cua models -p openai|anthropic|google|meta|xai|moonshotai|openrouter`. `gemini` aliases `google`, and `moonshot` aliases `moonshotai`. Model refs print as `provider:model`; `-m` accepts either the full ref or a bare model id that matches exactly one entry. @@ -172,10 +171,9 @@ Useful flags: The CLI selects its interaction tools from the model: structured CUA browser primitives plus `browser_act` verified plans for OpenAI, Meta, xAI, and older Anthropic models; browser primitives alone for Moonshot, whose API rejects -`browser_act`'s schema; native browser tools for current Anthropic and -Google models; Tzafon's native computer tool; and Yutori's documented native -set plus an explicit screenshot tool. It also appends workspace coding tools in -`--print`, TUI, and model-mediated action runs. +`browser_act`'s schema; and native browser tools for current Anthropic and +Google models. It also appends workspace coding tools in `--print`, TUI, and +model-mediated action runs. There is no `--mode`, `--native-tool`, or `--playwright` flag. Those catalogs remain explicit SDK choices rather than CLI defaults. The CLI also does not @@ -304,8 +302,8 @@ testing. It can only remove from that list, never add unsupported tools. Edits are staged — nothing applies until `ctrl+s`, and cancel leaves live state untouched. A selection rejected by catalog validation reports the error and changes nothing. Selections are session-only and are reset to the new model's -defaults by `/model`. Yutori n1's native set toggles as one group; disabling -everything is allowed and yields a text-only agent. +defaults by `/model`. Disabling everything is allowed and yields a text-only +agent. ## Don't forget