From 542690703e22477db6e05ea414f70abcee6bf859 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 21:11:37 +0000 Subject: [PATCH 1/7] Derive the streamed transport from the selected tools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The transport a model streams through is a property of two things: what the endpoint can carry, and which tools the caller selected. Stamping it on the model ahead of time gets that wrong in both directions — every OpenAI model paid for a CUA-owned transport whether or not its native tool was selected, and selecting that tool later could not change the transport. A provider binding now declares the api id its native tool needs (`requiresApi`), and `compileCuaToolCatalog` reads it off the selected bindings and returns a `catalog.model` carrying that api. Selecting tools whose bindings require different transports fails to compile with a named error. Compilation is idempotent with respect to a model that already carries a derived api, so feeding `catalog.model` back in with a different tool selection re-derives from the new selection instead of pinning the old transport. `routeCuaApi` shrinks to what is genuinely model-shaped: grok-4.5's cost and compat overrides, and the api ids of Tzafon and Yutori, for which pi ships no transport at all. Google's api id moves to the derivation, so a Gemini model selected with only CDP browser tools now streams through pi's builtin Google transport. Provider dispatch is api-keyed: the OpenAI wrapper routes `openai-cua-computer` to the native adapter with no request inspection, and the providers cua constructs use pi's `api` map form. One request-shaped predicate remains, for the case that cannot be derived from a tool list: a transcript carrying deferred-tool state needs the CUA adapter because pi does not round-trip OpenAI's `namespace` field on `function_call` items, which the API rejects with a 400. Both mutation paths push the compiled model into pi. `setTools()` previously recompiled the catalog without propagating the model, so a tool selection that changed the derived transport was compiled but never streamed. --- docs/architecture.md | 31 ++++++++- packages/agent/CHANGELOG.md | 17 +++++ packages/agent/src/agent.ts | 10 +++ packages/agent/src/tool-manager.ts | 11 +++- packages/agent/test/agent.test.ts | 59 +++++++++++++++++ packages/agent/test/tool-manager.test.ts | 18 ++++- packages/ai/CHANGELOG.md | 29 ++++++++ packages/ai/README.md | 29 ++++++-- packages/ai/src/cua.ts | 12 ++-- packages/ai/src/index.ts | 1 + packages/ai/src/models.ts | 24 ++++--- packages/ai/src/providers.ts | 43 ++++++------ packages/ai/src/providers/openai/provider.ts | 19 +++--- packages/ai/src/tool-catalog.ts | 66 +++++++++++++++++-- packages/ai/test/google-provider.test.ts | 4 +- packages/ai/test/models.test.ts | 14 ++-- .../ai/test/openai-adapter-routing.test.ts | 43 +++++++++++- packages/ai/test/providers.test.ts | 5 +- packages/ai/test/tool-catalog.test.ts | 53 +++++++++++++++ packages/cli/CHANGELOG.md | 11 ++++ 20 files changed, 429 insertions(+), 70 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index f42bf74..81eba63 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -154,8 +154,8 @@ catalog: - Anthropic native browser/computer declarations replace only their own placeholders and merge required beta headers with caller headers. - OpenAI streams through pi's builtin Responses transport and its automatic - prompt caching; a CUA-owned adapter handles OpenAI's native computer tool and - tool-search namespace round-trips. + prompt caching by default; a CUA-owned adapter handles OpenAI's native + computer tool and tool-search namespace round-trips. - Tzafon replaces only the selected computer identity and fills declaration dimensions from the actual viewport. - Anthropic's native browser tool falls back to an equivalent function-tool @@ -170,6 +170,33 @@ catalog: - Meta, xAI, and Moonshot disable parallel tool calls when the selected catalog can mutate browser state. +### Transport derivation + +The transport a model streams through is a function of **(model, selected +tools)**, derived at catalog compilation — never stamped on the model ahead of +time and never branched on a provider name. A `CuaProviderBinding` may declare +`requiresApi`: the api id its provider-native tool needs. `compileCuaToolCatalog` +reads `requiresApi` off the selected bindings after normalizing the requested +catalog and returns a `catalog.model` carrying that api; selecting tools whose +bindings require different transports fails to compile with a named catalog +error. A model resolved with no such tool selected keeps its ordinary registry +api. + +This is why an OpenAI model selected with only CUA browser tools streams +through pi's builtin `openai-responses` transport, but the same model selected +with `cua.providers.openai.tools.computer()` compiles to the CUA-owned +`openai-cua-computer` api — and symmetrically for Google's +`google-cua-interactions` Interactions API versus pi's builtin Google +transport. Tzafon and Yutori declare `requiresApi` too, but their models always +carry that api regardless of tool selection: pi ships no transport for either +provider at all, so `routeCuaApi` (model-shaped, not tool-shaped) forces it +unconditionally. + +`CuaAgent` and `CuaAgentHarness` push the compiled `catalog.model` into pi on +every construction and on every `setTools()`/`setModel()`, so the derived +transport applies uniformly regardless of which mutation path selected the +tools. + Generated payload processing has fixed order: model preparation, tool serialization, provider fields, then the caller's `onPayload` hook. diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 54f3fda..5ce0ab9 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,22 @@ # Changelog +## 0.12.0 - 2026-08-13 + +- Update `@onkernel/cua-ai` to 0.12.0. The model streamed for a Google model + now depends on which tools `CuaAgent`/`CuaAgentHarness` were constructed or + mutated with: selecting Google's native browser toolset still compiles to + the CUA-owned Interactions API, but a Google model selected with only CDP + browser tools now streams through pi's builtin Google transport instead of + always carrying the CUA-owned api. This applies uniformly across + construction, `setTools()`, and `setModel()`, since all three feed the same + compiled `catalog.model` into pi. +- Fix `setTools()` recompiling from the previously *compiled* model instead of + the caller's model selection: dropping a native toolset that had derived a + tool-selection-dependent api (e.g. Google's Interactions API) left + subsequent tools-only recompiles stuck on that api even though the new + selection no longer required it. `CuaToolManager` now recompiles tools-only + changes from the model input the caller last selected. + ## 0.11.0 - 2026-08-13 - `responseThreading` (`CuaAgentOptions`/`CuaAgentHarnessOptions`) no longer diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index 89886ce..b837e5c 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -258,6 +258,7 @@ export class CuaAgent { setTools(tools: readonly CuaAgentTool[]): void { const prepared = this.tools.prepareTools(tools); + this.coreAgent.state.model = prepared.catalog.model; this.coreAgent.state.tools = this.tools.agentTools(prepared); this.tools.commit(prepared); this.runtimeDirty = true; @@ -402,12 +403,16 @@ export class CuaAgentHarness< getTools(): readonly CuaHarnessTool[] { return this.tools.getTools(); } async setTools(tools: readonly CuaHarnessTool[]): Promise { + const previousModel = this.tools.catalog.model; const previousTools = this.tools.harnessTools(); const prepared = this.tools.prepareTools(tools); const materialized = this.tools.harnessTools(prepared); + const transportChanged = modelTransportChanged(previousModel, prepared.catalog.model); try { + if (transportChanged) await this.coreHarness.setModel(prepared.catalog.model); await this.coreHarness.setTools(materialized, materialized.map((tool) => tool.name)); } catch (error) { + if (transportChanged) await this.coreHarness.setModel(previousModel); await this.coreHarness.setTools(previousTools, previousTools.map((tool) => tool.name)); throw error; } @@ -476,6 +481,11 @@ function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model, next: Model): boolean { + return previous.provider !== next.provider || previous.id !== next.id || previous.api !== next.api; +} + function withCatalogModels( models: Models, manager: CuaToolManager, diff --git a/packages/agent/src/tool-manager.ts b/packages/agent/src/tool-manager.ts index d19f063..6e527e9 100644 --- a/packages/agent/src/tool-manager.ts +++ b/packages/agent/src/tool-manager.ts @@ -40,6 +40,14 @@ export type CuaHarnessTool = CuaToo */ export interface PreparedCuaTools = CuaAgentTool> { readonly requested: readonly TRequested[]; + /** + * The model input this state was compiled from, before catalog compilation + * derived a tool-selection-dependent transport onto it. A tools-only + * recompile reuses this (not `catalog.model`, which may carry a transport + * the new tool selection no longer requires) so the derivation re-runs from + * a clean model every time. + */ + readonly modelSelection: CuaModelRef | Model; readonly catalog: CuaToolCatalog; readonly tools: readonly AgentTool[]; readonly harnessTools: readonly AgentHarnessTool[]; @@ -112,7 +120,7 @@ export class CuaToolManager = CuaAgentToo prepareTools(tools: readonly TRequested[]): PreparedCuaTools { this.assertMutationScope("setTools"); - return this.prepare(this.current.catalog.model, tools); + return this.prepare(this.current.modelSelection, tools); } prepareModel(model: CuaModelRef | Model): PreparedCuaTools { @@ -174,6 +182,7 @@ export class CuaToolManager = CuaAgentToo return Object.freeze({ requested, + modelSelection: model, catalog, tools: Object.freeze(joined.map((tool) => this.wrapAgentExecutable(tool as AgentTool))), harnessTools: Object.freeze(joined.map((tool) => this.wrapHarnessExecutable(tool))), diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index ddd61f6..e5c5ab9 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -4,6 +4,7 @@ import { createCuaModels, getCuaModel, cua, + GOOGLE_CUA_INTERACTIONS_API, isCuaToolSpec, type AssistantMessage, type Context, @@ -454,6 +455,27 @@ describe("CuaAgent explicit tools", () => { expect(agent.getTools()).toEqual([switcher]); expect(contexts[1]?.messages.find((message) => message.role === "toolResult")).not.toHaveProperty("addedToolNames"); }); + + it("streams with the transport setTools derives, not just the catalog it compiles", async () => { + const streamedApis: string[] = []; + const agent = new CuaAgent({ + browser, + client, + tools: [cua.tools.browser.snapshot()], + streamFn: (model, context, options) => { + streamedApis.push(model.api); + return scriptedStream([(selectedModel) => assistant(selectedModel)])(model, context, options); + }, + initialState: { model: "google:gemini-3.6-flash" }, + }); + expect(agent.getModel().api).toBe("google-generative-ai"); + + agent.setTools(cua.providers.google.toolsets.browser()); + expect(agent.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await agent.prompt("go"); + expect(streamedApis).toEqual([GOOGLE_CUA_INTERACTIONS_API]); + }); }); describe("CuaAgentHarness explicit tools", () => { @@ -578,4 +600,41 @@ describe("CuaAgentHarness explicit tools", () => { expect(successfulCalls).toBe(1); }); + + it("streams with the transport setTools derives, not just the catalog it compiles", async () => { + const streamedApis: string[] = []; + const script = scriptedStream([(selectedModel) => assistant(selectedModel)]); + const harness = new CuaAgentHarness({ + ...(await harnessServices()), + browser, + client, + model: "google:gemini-3.6-flash", + models: modelsFromStream((model, context, options) => { + streamedApis.push(model.api); + return script(model, context, options); + }, "google"), + tools: [cua.tools.browser.snapshot()], + }); + expect(harness.getModel().api).toBe("google-generative-ai"); + + await harness.setTools(cua.providers.google.toolsets.browser()); + expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await harness.prompt("go"); + expect(streamedApis).toEqual([GOOGLE_CUA_INTERACTIONS_API]); + }); + + it("does not record a model change for a setTools() call that leaves the derived transport unchanged", async () => { + const services = await harnessServices(); + const harness = new CuaAgentHarness({ + ...services, + browser, + client, + model: "openai:gpt-5.5", + tools: [callerTool("first")], + }); + await harness.setTools([callerTool("second")]); + const modelChanges = (await services.session.getBranch()).filter((entry) => entry.type === "model_change"); + expect(modelChanges).toEqual([]); + }); }); diff --git a/packages/agent/test/tool-manager.test.ts b/packages/agent/test/tool-manager.test.ts index db0b0f7..c9d93d9 100644 --- a/packages/agent/test/tool-manager.test.ts +++ b/packages/agent/test/tool-manager.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, vi } from "vitest"; -import { callerToolIdentity, cua } from "@onkernel/cua-ai"; +import { callerToolIdentity, cua, GOOGLE_CUA_INTERACTIONS_API } from "@onkernel/cua-ai"; import type Kernel from "@onkernel/sdk"; import { CuaExecutionResources, @@ -92,6 +92,22 @@ describe("CuaToolManager identity join", () => { }); }); +describe("CuaToolManager transport derivation", () => { + it("derives the compiled model's api from selected tools on construction and on both mutation paths", () => { + const manager = new CuaToolManager(setup(), "google:gemini-3.6-flash", [cua.tools.browser.snapshot()]); + expect(manager.catalog.model.api).toBe("google-generative-ai"); + + manager.commit(manager.prepareTools(cua.providers.google.toolsets.browser())); + expect(manager.catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + manager.commit(manager.prepareModel("google:gemini-3.6-flash")); + expect(manager.catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + manager.commit(manager.prepareTools([cua.tools.browser.snapshot()])); + expect(manager.catalog.model.api).toBe("google-generative-ai"); + }); +}); + describe("CuaToolManager implementation identity", () => { it("materializes each spec exactly once across model and tool recompilation", () => { const resources = setup(); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index aa0487c..9e7418e 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## 0.12.0 - 2026-08-13 + +Breaking: Google's api id is now derived from selected tools, not stamped on +every Google model. + +- `compileCuaToolCatalog` derives the compiled model's `api` from the selected + tools' provider bindings: a `CuaProviderBinding` may declare `requiresApi`, + and the returned `catalog.model` carries that transport. Selecting tools + whose bindings require different transports fails to compile with a named + catalog error. This makes transport a function of `(model, selected tools)` + instead of `(model)` alone. +- `getCuaModel("google:...")` no longer forces `google-cua-interactions`. A + Google model resolved without Google's native browser toolset selected now + keeps pi-ai's builtin `google-generative-ai` transport; selecting + `cua.providers.google.toolsets.browser()` still compiles to + `google-cua-interactions` as before. `routeCuaApi` no longer touches Google + at all — it's now scoped to genuinely model-shaped routing (Tzafon and + Yutori, which pi ships no transport for at all, and grok-4.5's cost/compat + overrides). +- Add `OPENAI_CUA_COMPUTER_API` (`"openai-cua-computer"`). A model compiled + with `cua.providers.openai.tools.computer()` selected now carries this api; + the OpenAI provider wrapper dispatches to the CUA adapter on `model.api` + alone for that case. The one remaining request-shape check, + `requiresCuaOpenAINamespaceAdapter` (renamed from `requiresCuaOpenAIAdapter`, + which also tested for the native computer tool), covers only the case that + cannot be derived from the model: a transcript carrying a deferred + tool-search addition or a replayed function-call namespace, which pi-ai's + builtin transport does not round-trip. + ## 0.11.0 - 2026-08-13 Breaking: OpenAI models no longer carry a CUA-owned api id. diff --git a/packages/ai/README.md b/packages/ai/README.md index aef7db5..76892cf 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -240,21 +240,36 @@ additions through pi's active-tool change entries. ## Provider behavior -- **OpenAI**: streams through pi's builtin Responses transport and its - automatic prompt caching by default. A CUA-owned adapter is used only for - OpenAI's native computer tool and for tool-search namespace round-trips. +Transport is derived, not stamped on the model ahead of time: a selected +tool's provider binding may declare `requiresApi`, and `compileCuaToolCatalog` +returns a `catalog.model` carrying that api. Selecting tools whose bindings +require different transports fails to compile. + +- **OpenAI**: a model selected with only ordinary/CUA browser tools streams + through pi's builtin Responses transport and its automatic prompt caching. + Selecting `cua.providers.openai.tools.computer()` derives the CUA-owned + `openai-cua-computer` api instead, which a CUA adapter handles; that same + adapter also covers tool-search namespace round-trips regardless of api, + since pi's builtin transport does not replay them. - **Anthropic**: exact native declarations, beta-header composition, and - adaptive model preparation. -- **Google**: a CUA-owned Interactions API adapter plus the current predefined - browser set with explicit exclusions. + adaptive model preparation. No api fork — every Anthropic model streams + through pi's builtin transport. +- **Google**: a model selected without Google's native browser toolset streams + through pi's builtin transport. Selecting + `cua.providers.google.toolsets.browser()` derives the CUA-owned + `google-cua-interactions` api, which serializes one `computer_use` + declaration plus explicit exclusions through the Interactions API adapter. - **Meta/xAI/Moonshot**: ordinary function tools with serial tool calls when the selected catalog mutates browser state. - **Tzafon**: identity-scoped native declaration replacement with actual viewport dimensions. Explicit screenshot and terminal answer actions are supported; non-screenshot native action loops fail before browser execution because - Tzafon's continuation protocol requires implicit post-action screenshots. + Tzafon's continuation protocol requires implicit post-action screenshots. Pi + ships no Tzafon transport, so every Tzafon model carries CUA's own api + regardless of tool selection. - **Yutori**: identity-scoped native `tool_set`/`disable_tools` fields while preserving ordinary function tools such as an explicitly selected screenshot. + Pi ships no Yutori transport either, so the same unconditional api applies. ## API keys diff --git a/packages/ai/src/cua.ts b/packages/ai/src/cua.ts index a745e75..10293c7 100644 --- a/packages/ai/src/cua.ts +++ b/packages/ai/src/cua.ts @@ -8,13 +8,16 @@ import { } from "./actions/index"; import { supportsAnthropicNativeBrowser } from "./providers/anthropic/capabilities"; import { mapNativeBrowserInput, mapNativeComputerInput } from "./providers/anthropic/native"; -import { toCanonicalActions as toTzafonActions } from "./providers/tzafon/provider"; +import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; +import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; +import { toCanonicalActions as toTzafonActions, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; import { toCanonicalActions as toYutoriActions, YUTORI_N1_ACTION_TYPES, YUTORI_N15_CORE_ACTION_TYPES, YUTORI_N15_CORE_TOOL_SET, } from "./providers/yutori/actions"; +import { YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; import { CUA_TOOL_SPEC_KIND, type CuaCoordinateContract, @@ -435,7 +438,7 @@ function openaiNativeComputer(): CuaToolSpec { name: "computer", source: providerSources.openai, declaration, - binding: { kind: "openai-native", declaration }, + binding: { kind: "openai-native", declaration, requiresApi: OPENAI_CUA_COMPUTER_API }, toActions: mapOpenAIComputerInput, coordinates: pixels, }); @@ -453,7 +456,7 @@ function tzafonNativeComputer(options: { displayWidth?: number; displayHeight?: name: "computer", source: providerSources.tzafon, declaration, - binding: { kind: "tzafon-native", declaration }, + binding: { kind: "tzafon-native", declaration, requiresApi: TZAFON_RESPONSES_API }, toActions(input) { const action = asInput(input).action; return toTzafonActions(action).filter((value): value is CuaAction => value.type !== "answer"); @@ -472,6 +475,7 @@ function yutoriToolset(generation: "n1" | "n15"): CuaToolSpec[] { nativeName, ...(generation === "n15" ? { toolSet: YUTORI_N15_CORE_TOOL_SET } : {}), allNativeNames: names, + requiresApi: YUTORI_CHAT_COMPLETIONS_API, }; return providerNativeSpec({ identity: `provider.yutori.native.${generation}.${identityName}.${generation === "n15" ? "20260403" : "v1"}`, @@ -507,7 +511,7 @@ function googleBrowserToolset(options: GoogleBrowserToolsetOptions = {}): CuaToo name: nativeName, source: providerSources.google, declaration: { computerUse: { environment: "ENVIRONMENT_BROWSER" } }, - binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS }, + binding: { kind: "google-native", nativeName, allNativeNames: GOOGLE_BROWSER_ACTIONS, requiresApi: GOOGLE_CUA_INTERACTIONS_API }, toActions: (input) => mapGoogleAction(nativeName, asInput(input)), coordinates: normalized([0, 999]), })); diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index fca2e3c..b4e90a0 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -4,6 +4,7 @@ export { createCuaModels, cuaModels, GOOGLE_CUA_INTERACTIONS_API, + OPENAI_CUA_COMPUTER_API, streamGoogleInteractions, streamOpenAIResponses, streamSimpleGoogleInteractions, diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index cf167ab..b50b5c3 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -1,6 +1,7 @@ import type { Api, Model } from "@earendil-works/pi-ai"; import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; -import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; +import { TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; +import { YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; /** Providers with curated computer-use model support. */ export type CuaProvider = "openai" | "anthropic" | "google" | "meta" | "xai" | "moonshotai" | "openrouter" | "tzafon" | "yutori"; @@ -226,14 +227,21 @@ export function getCuaModel(ref: CuaModelRef): Model { throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`); } -// Route CUA models to provider-specific transports. OpenAI keeps pi-ai's -// builtin "openai-responses" api and streams through its automatic prompt -// caching; the CUA adapter dispatches on request shape (see -// requiresCuaOpenAIAdapter), not on a rerouted api id. Other registry-resolved -// models otherwise carry pi-ai's builtin API ids too. +// Route CUA models to provider-specific transports that are properties of the +// model itself, not of which tools a caller selects. Tool-driven transport +// selection (OpenAI's native computer tool, Google's Interactions API) is +// derived by compileCuaToolCatalog from the selected tools' provider bindings +// instead; see CuaProviderBinding.requiresApi. What remains here is model-only: +// Tzafon and Yutori get no transport from pi-ai at all, so every model on +// those providers always carries CUA's own api id regardless of tool +// selection, and grok-4.5 carries cost/compat/thinking-level overrides pi-ai's +// registry does not have yet. export function routeCuaApi(model: Model): Model { - if (model.provider === "google" && model.api !== GOOGLE_CUA_INTERACTIONS_API) { - return { ...model, api: GOOGLE_CUA_INTERACTIONS_API }; + if (model.provider === "tzafon" && model.api !== TZAFON_RESPONSES_API) { + return { ...model, api: TZAFON_RESPONSES_API }; + } + if (model.provider === "yutori" && model.api !== YUTORI_CHAT_COMPLETIONS_API) { + return { ...model, api: YUTORI_CHAT_COMPLETIONS_API }; } if (model.provider === "xai" && model.id === "grok-4.5") { return { diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index fd6de71..c3d42be 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -19,7 +19,7 @@ import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; import { cuaOverrideModels } from "./models"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; -import { requiresCuaOpenAIAdapter, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; +import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; import { streamSimpleTzafonResponses, streamTzafonResponses, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; @@ -29,13 +29,17 @@ import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from ". * * - `anthropic` retries an inaccessible native browser beta through the * selected tool's equivalent function declaration. - * - `openai` streams through pi's builtin `openai-responses` transport and - * its automatic prompt caching by default. The CUA adapter only intercepts - * requests that need it: OpenAI's native computer tool, or a transcript - * carrying a deferred tool-search addition or a replayed function-call - * namespace (see {@link requiresCuaOpenAIAdapter}). - * - `google` intercepts `google-cua-interactions` for current native computer - * use and resolves API keys from `GOOGLE_API_KEY` or `GEMINI_API_KEY`. + * - `openai` streams through pi's builtin `openai-responses` transport and its + * automatic prompt caching by default; a model compiled with OpenAI's native + * computer tool carries `openai-cua-computer` instead, which this wrapper + * routes to the CUA adapter. The one dispatch that cannot be derived from + * `model.api` is a transcript carrying a deferred tool-search addition or a + * replayed function-call namespace (see {@link requiresCuaOpenAINamespaceAdapter}). + * - `google` intercepts `google-cua-interactions` — carried only by a model + * compiled with Google's native computer-use toolset — for current native + * computer use, and resolves API keys from `GOOGLE_API_KEY` or `GEMINI_API_KEY`. + * A Google model compiled without that toolset streams through pi's builtin + * Google transport instead. * - `xai` is pi's builtin provider untouched: Grok streams through pi's * Responses transport, and the catalog supplies its serial-tool-call field. * - `moonshotai` is pi's builtin provider untouched: Kimi streams through the @@ -52,7 +56,7 @@ export function createCuaModels(options?: CreateModelsOptions): MutableModels { const anthropic = models.getProvider("anthropic"); if (anthropic) models.setProvider(withAnthropicBrowserFallback(anthropic)); const openai = models.getProvider("openai"); - if (openai) models.setProvider(withOpenAICuaComputerAdapter(openai)); + if (openai) models.setProvider(withOpenAICuaAdapter(openai)); const google = models.getProvider("google"); if (google) models.setProvider(withGoogleCuaInteractions(google)); models.setProvider(metaProvider()); @@ -75,18 +79,19 @@ export function cuaModels(): MutableModels { return (defaultCuaModels ??= createCuaModels()); } -// OpenAI models keep pi-ai's builtin "openai-responses" api id. Only requests -// that need cua-ai's adapter (see requiresCuaOpenAIAdapter) are intercepted; -// everything else falls through to pi's builtin provider. -function withOpenAICuaComputerAdapter(base: Provider): Provider { +// The compiled catalog's model.api decides dispatch: OPENAI_CUA_COMPUTER_API +// routes to the CUA adapter, everything else falls through to pi's builtin +// "openai-responses" provider. requiresCuaOpenAINamespaceAdapter is the one +// exception that cannot be derived from the model — see its doc comment. +function withOpenAICuaAdapter(base: Provider): Provider { return { ...base, stream: (model: Model, context: Context, options?: StreamOptions) => - model.api === "openai-responses" && requiresCuaOpenAIAdapter(context, options) + model.api === OPENAI_CUA_COMPUTER_API || requiresCuaOpenAINamespaceAdapter(context) ? streamOpenAIResponses(model as never, context, options) : base.stream(model, context, options), streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => - model.api === "openai-responses" && requiresCuaOpenAIAdapter(context, options) + model.api === OPENAI_CUA_COMPUTER_API || requiresCuaOpenAINamespaceAdapter(context) ? streamSimpleOpenAIResponses(model as never, context, options) : base.streamSimple(model, context, options), }; @@ -115,7 +120,7 @@ function metaProvider(): Provider { baseUrl: "https://api.meta.ai/v1", auth: { apiKey: envApiKeyAuth("Meta Model API key", cuaApiKeyEnvVarsForProvider("meta")) }, models: cuaOverrideModels("meta"), - api: { stream: piStreamOpenAIResponses, streamSimple: piStreamSimpleOpenAIResponses }, + api: { "openai-responses": { stream: piStreamOpenAIResponses, streamSimple: piStreamSimpleOpenAIResponses } }, }); } @@ -126,7 +131,7 @@ function tzafonProvider(): Provider { baseUrl: "https://api.tzafon.ai", auth: { apiKey: envApiKeyAuth("Tzafon API key", cuaApiKeyEnvVarsForProvider("tzafon")) }, models: cuaOverrideModels("tzafon"), - api: { stream: streamTzafonResponses, streamSimple: streamSimpleTzafonResponses }, + api: { [TZAFON_RESPONSES_API]: { stream: streamTzafonResponses, streamSimple: streamSimpleTzafonResponses } }, }); } @@ -137,11 +142,11 @@ function yutoriProvider(): Provider { baseUrl: "https://api.yutori.com/v1", auth: { apiKey: envApiKeyAuth("Yutori API key", cuaApiKeyEnvVarsForProvider("yutori")) }, models: cuaOverrideModels("yutori"), - api: { stream: streamYutori, streamSimple: streamSimpleYutori }, + api: { [YUTORI_CHAT_COMPLETIONS_API]: { stream: streamYutori, streamSimple: streamSimpleYutori } }, }); } export { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; -export { streamOpenAIResponses, streamSimpleOpenAIResponses }; +export { OPENAI_CUA_COMPUTER_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; export { TZAFON_RESPONSES_API, streamSimpleTzafonResponses, streamTzafonResponses }; export { YUTORI_CHAT_COMPLETIONS_API, streamSimpleYutori, streamYutori }; diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/ai/src/providers/openai/provider.ts index 78c6ec6..f2e4b52 100644 --- a/packages/ai/src/providers/openai/provider.ts +++ b/packages/ai/src/providers/openai/provider.ts @@ -25,6 +25,9 @@ import { buildBaseOptions } from "@earendil-works/pi-ai/api/simple-options"; import type { CuaIncomingToolPlan } from "../../tool-catalog"; import type { CuaSimpleStreamOptions } from "../common"; +/** CUA-owned api id for OpenAI's native computer tool, derived onto the model by compileCuaToolCatalog when that tool is selected. */ +export const OPENAI_CUA_COMPUTER_API = "openai-cua-computer"; + export interface OpenAIResponsesOptions extends PiOpenAIResponsesOptions { /** @internal Identity-addressed native dispatch compiled from selected tools. */ cuaIncomingToolPlan?: CuaIncomingToolPlan; @@ -34,16 +37,14 @@ export interface OpenAIResponsesOptions extends PiOpenAIResponsesOptions { type OpenAIRequestOptions = Pick; /** - * Whether a request needs cua-ai's OpenAI Responses adapter instead of pi-ai's - * builtin `openai-responses` transport: OpenAI's native computer tool is - * selected, or the transcript carries state (a deferred tool-search addition, - * or a replayed function-call namespace) that only this adapter round-trips. + * The one dispatch cua-ai's OpenAI provider wrapper cannot derive from the + * model's `api`: whether the transcript carries state (a deferred tool-search + * addition, or a replayed function-call namespace) that only this adapter + * round-trips, because pi-ai's builtin `openai-responses` transport does not + * parse or replay `function_call`'s `namespace` field. Every other dispatch — + * including OpenAI's native computer tool — is decided by `model.api` instead. */ -export function requiresCuaOpenAIAdapter(context: Context, options?: unknown): boolean { - // pi's Provider.stream signature hides cua's added option, so the plan is - // read structurally here rather than at both call sites. - const cuaIncomingToolPlan = (options as { cuaIncomingToolPlan?: CuaIncomingToolPlan } | undefined)?.cuaIncomingToolPlan; - if (cuaIncomingToolPlan?.openaiComputerName) return true; +export function requiresCuaOpenAINamespaceAdapter(context: Context): boolean { for (const message of context.messages) { if (message.role === "toolResult" && (message.addedToolNames?.length ?? 0) > 0) return true; if (message.role === "assistant" && message.content.some((part) => part.type === "toolCall" && toolCallNamespace(part))) return true; diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index 9b8f3f8..13182eb 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -7,6 +7,8 @@ import { supportsAnthropicNativeBrowser, supportsAnthropicNativeComputer, } from "./providers/anthropic/capabilities"; +import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; +import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; export const CUA_TOOL_SPEC_KIND = "@onkernel/cua-tool-spec/v1" as const; @@ -36,10 +38,17 @@ export type CuaProviderBinding = readonly beta: string; readonly accessFallback?: CuaAnthropicBrowserFallback; } - | { readonly kind: "openai-native"; readonly declaration: Record } + | { + readonly kind: "openai-native"; + readonly declaration: Record; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; + } | { readonly kind: "tzafon-native"; readonly declaration: Record; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; } | { readonly kind: "yutori-native"; @@ -47,11 +56,15 @@ export type CuaProviderBinding = readonly nativeName: string; readonly toolSet?: string; readonly allNativeNames: readonly string[]; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; } | { readonly kind: "google-native"; readonly nativeName: string; readonly allNativeNames: readonly string[]; + /** Transport this binding requires the compiled catalog's model to carry. */ + readonly requiresApi?: Api; }; /** Declarative CUA tool. Identity is immutable and independent from its model-facing alias. */ @@ -202,12 +215,21 @@ const SAFE_TOOL_NAME = /^[A-Za-z][A-Za-z0-9_-]{0,63}$/; * requested list. Pure and declaration-only: identical declaration, model, * and viewport inputs produce identical catalogs, and compilation never * constructs executable tools or retains the requested input objects. + * + * The compiled model's `api` is derived here, not stamped on the model ahead + * of time: a selected tool's provider binding may declare `requiresApi`, and + * the returned `catalog.model` carries that transport. A model resolved with + * no such tool selected keeps its ordinary registry `api`. Selecting tools + * whose bindings require different transports fails to compile. */ export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): CuaToolCatalog { - const model = typeof options.model === "string" ? getCuaModel(options.model) : routeCuaApi(options.model); + const baseModel = typeof options.model === "string" + ? getCuaModel(options.model) + : routeCuaApi(resetCatalogDerivedApi(options.model)); const viewport = options.viewport; const normalizedEntries = [...options.requestedTools].map((tool) => normalizeTool(tool, viewport)); - validateCatalog(model, normalizedEntries); + const requiresApi = validateCatalog(baseModel, normalizedEntries); + const model = requiresApi ? { ...baseModel, api: requiresApi } : baseModel; const drafts = resolveProviderFacingDeclarations(normalizedEntries); const names = new Map(drafts.map((entry) => [entry.identity, entry.name])); @@ -360,7 +382,8 @@ function resolveProviderFacingDeclarations(entries: readonly CuaCatalogEntryDraf }); } -function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDraft[]): void { +/** Validate the requested catalog against the model and return the transport its selected tools require, if any. */ +function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { const identities = new Map(); const exactNames = new Map(); const normalizedNames = new Map(); @@ -387,7 +410,7 @@ function validateCatalog(model: Model, entries: readonly CuaCatalogEntryDra validateToolCompatibility(model, entry); } - validateToolsetCompatibility(model, entries); + return validateToolsetCompatibility(model, entries); } function nameCollision( @@ -432,7 +455,8 @@ function validateAnthropicNativeModel(model: Model, identity: string): void } } -function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): void { +/** Validate the selected native tools agree on a provider and a transport, and return the transport they require, if any. */ +function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { const yutoriN1 = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native" && entry.providerBinding.generation === "n1"); if (yutoriN1.length > 0) { const all = yutoriN1[0]!.providerBinding; @@ -448,6 +472,36 @@ function validateToolsetCompatibility(model: Model, entries: readonly CuaCa throw new Error(`selected tools contribute incompatible native provider transports: ${[...nativeProviderKinds].join(", ")}`); } providerForModel(model); + + const requiresApis = new Set(entries.flatMap((entry) => bindingRequiresApi(entry.providerBinding))); + if (requiresApis.size > 1) { + throw new Error(`selected tools require incompatible provider transports: ${[...requiresApis].join(", ")}`); + } + return requiresApis.values().next().value; +} + +/** The transport a provider binding requires, if it declares one. Anthropic never forks transports and declares none. */ +function bindingRequiresApi(binding: CuaProviderBinding | undefined): readonly [Api] | readonly [] { + return binding && binding.kind !== "anthropic-native" && binding.requiresApi ? [binding.requiresApi] : []; +} + +/** + * Model-shaped default transport for each `requiresApi` this module can + * derive, keyed by the derived api itself. A `Model` a caller passes to + * {@link compileCuaToolCatalog} may already carry one of these — e.g. a prior + * catalog's `catalog.model`, fed back in with a different tool selection — so + * derivation resets it here before re-validating, keeping compilation pure + * with respect to the currently requested tools rather than pinning whatever + * transport an earlier selection required. + */ +const CATALOG_DERIVED_API_DEFAULTS: Readonly> = { + [OPENAI_CUA_COMPUTER_API]: "openai-responses", + [GOOGLE_CUA_INTERACTIONS_API]: "google-generative-ai", +}; + +function resetCatalogDerivedApi(model: Model): Model { + const defaultApi = CATALOG_DERIVED_API_DEFAULTS[model.api]; + return defaultApi ? { ...model, api: defaultApi } : model; } function compileHeaderRequirements(entries: readonly CuaCatalogEntryDraft[]): CuaHeaderRequirement[] { diff --git a/packages/ai/test/google-provider.test.ts b/packages/ai/test/google-provider.test.ts index ae84f95..c2c6b9a 100644 --- a/packages/ai/test/google-provider.test.ts +++ b/packages/ai/test/google-provider.test.ts @@ -3,7 +3,9 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import { getCuaModel } from "../src/index"; import * as google from "../src/providers/google/provider"; -const model = getCuaModel("google:gemini-3.6-flash") as Model; +// This adapter is exercised directly (not through compileCuaToolCatalog's +// derivation), so the model must carry the Interactions api itself. +const model: Model = { ...getCuaModel("google:gemini-3.6-flash"), api: google.GOOGLE_CUA_INTERACTIONS_API }; const incoming = { googleNames: { click: "click" }, googleExcludedNames: ["take_screenshot"], diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index 6f5628f..b9f5650 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -7,7 +7,6 @@ import { findCuaAnnotation, formatCuaModelRef, getCuaModel, - GOOGLE_CUA_INTERACTIONS_API, listCuaModels, parseCuaModelRef, } from "../src/index"; @@ -70,7 +69,7 @@ describe("CUA model refs", () => { expect(cuaOverrideModels("google")).toEqual([]); expect(getCuaModel("google:gemini-3.6-flash")).toMatchObject({ provider: "google", - api: GOOGLE_CUA_INTERACTIONS_API, + api: "google-generative-ai", contextWindow: 1_048_576, }); @@ -131,14 +130,15 @@ describe("CUA model refs", () => { expect(getCuaModel("yutori:n1.5-latest").api).toBe("yutori-chat-completions"); }); - it("keeps every Responses model on pi's builtin transport except Google's Interactions API", () => { - // A CUA-owned api id now exists only where pi ships no equivalent - // transport. OpenAI, Meta, and xAI all speak the Responses protocol pi - // already implements, including its automatic prompt caching. + it("resolves every model to its ordinary registry transport, independent of tool selection", () => { + // getCuaModel() never derives a tool-driven transport: OPENAI_CUA_COMPUTER_API + // and GOOGLE_CUA_INTERACTIONS_API are only ever carried by a model that + // compileCuaToolCatalog compiled with the matching native tool selected + // (see tool-catalog.test.ts's transport derivation coverage). expect(getCuaModel("openai:gpt-5.6-sol").api).toBe("openai-responses"); expect(getCuaModel("openai:gpt-5.5").api).toBe("openai-responses"); expect(getCuaModel("openai:gpt-5.4-mini").api).toBe("openai-responses"); - expect(getCuaModel("google:gemini-3.6-flash").api).toBe(GOOGLE_CUA_INTERACTIONS_API); + expect(getCuaModel("google:gemini-3.6-flash").api).toBe("google-generative-ai"); expect(getCuaModel("meta:muse-spark-1.1").api).toBe("openai-responses"); expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); }); diff --git a/packages/ai/test/openai-adapter-routing.test.ts b/packages/ai/test/openai-adapter-routing.test.ts index 74dcc56..92c2a23 100644 --- a/packages/ai/test/openai-adapter-routing.test.ts +++ b/packages/ai/test/openai-adapter-routing.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from "vitest"; import type { ToolCall } from "@earendil-works/pi-ai"; -import { createCuaModels, getCuaModel } from "../src/index"; +import { createCuaModels, getCuaModel, OPENAI_CUA_COMPUTER_API } from "../src/index"; const { responsesCreate } = vi.hoisted(() => ({ responsesCreate: vi.fn() })); @@ -68,7 +68,7 @@ describe("OpenAI adapter routing", () => { expect(payload.previous_response_id).toBeUndefined(); }); - it("reaches the CUA adapter when the incoming plan selects OpenAI's native computer tool", async () => { + it("reaches the CUA adapter when the model carries OPENAI_CUA_COMPUTER_API", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", output: [{ @@ -77,7 +77,11 @@ describe("OpenAI adapter routing", () => { action: { type: "click", x: 10, y: 20 }, }], }); - const message = await createCuaModels().streamSimple(model, { + // compileCuaToolCatalog derives this api onto the model whenever OpenAI's + // native computer tool is selected; the provider wrapper dispatches on it + // alone, with no request-shape sniffing. + const computerModel = { ...model, api: OPENAI_CUA_COMPUTER_API }; + const message = await createCuaModels().streamSimple(computerModel, { messages: [{ role: "user", content: "click it", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { @@ -127,6 +131,39 @@ describe("OpenAI adapter routing", () => { expect(call?.namespace).toBe("deferred_tools"); }); + it("keeps cache-relevant payload fields identical across a mid-conversation escalation", async () => { + // A turn can move from pi's builtin transport to the CUA adapter mid-session + // (a deferred tool gets added). If that switch changed `store` or the cache + // key, it would silently invalidate the matched prompt-cache prefix. + responsesCreate.mockReturnValue({ id: "resp_parity", status: "completed", usage: {}, output: [] }); + await createCuaModels().streamSimple(model, { + messages: [{ role: "user", content: "look it up", timestamp: 1 }], + tools, + }, { apiKey: "test", sessionId: "session_parity" }).result(); + const builtinPayload = responsesCreate.mock.calls.at(-1)?.[0] as Record; + + await createCuaModels().streamSimple(model, { + messages: [ + { role: "user", content: "load and look it up", timestamp: 1 }, + { + role: "toolResult", + toolCallId: "call_loader", + toolName: "loader", + content: [{ type: "text", text: "loaded" }], + isError: false, + addedToolNames: ["lookup"], + timestamp: 2, + }, + ], + tools, + }, { apiKey: "test", sessionId: "session_parity" } as never).result(); + const escalatedPayload = responsesCreate.mock.calls.at(-1)?.[0] as Record; + + for (const field of ["store", "prompt_cache_key", "prompt_cache_retention", "prompt_cache_options"]) { + expect(escalatedPayload[field], field).toEqual(builtinPayload[field]); + } + }); + it("pairs replayed namespaces by call id, not ordinal, across an aborted assistant turn", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_4", usage: {}, output: [] }); await createCuaModels().streamSimple(model, { diff --git a/packages/ai/test/providers.test.ts b/packages/ai/test/providers.test.ts index ea651ab..f7408fe 100644 --- a/packages/ai/test/providers.test.ts +++ b/packages/ai/test/providers.test.ts @@ -41,8 +41,9 @@ describe("createCuaModels", () => { const openaiIds = models.getModels("openai").map((m) => m.id); expect(openaiIds).toContain("gpt-5.4"); // OpenAI models keep pi's builtin "openai-responses" api id on both the - // collection and getCuaModel(); the wrapped provider only intercepts - // requests that need the CUA adapter (see requiresCuaOpenAIAdapter). + // collection and getCuaModel(); the wrapped provider only intercepts a + // model carrying OPENAI_CUA_COMPUTER_API or a namespace round-trip (see + // requiresCuaOpenAINamespaceAdapter). expect(models.getModel("openai", "gpt-5.4")?.api).toBe("openai-responses"); const xaiIds = models.getModels("xai").map((m) => m.id); diff --git a/packages/ai/test/tool-catalog.test.ts b/packages/ai/test/tool-catalog.test.ts index 2e70fb2..d193735 100644 --- a/packages/ai/test/tool-catalog.test.ts +++ b/packages/ai/test/tool-catalog.test.ts @@ -4,6 +4,8 @@ import { callerToolIdentity, compileCuaToolCatalog, cua, + GOOGLE_CUA_INTERACTIONS_API, + OPENAI_CUA_COMPUTER_API, type CuaToolSpec, } from "../src/index"; @@ -348,3 +350,54 @@ describe("compileCuaToolCatalog", () => { expect(second.toolDeclarations.map((tool) => tool.name)).toEqual(first.toolDeclarations.map((tool) => tool.name)); }); }); + +describe("transport derivation", () => { + it("keeps an OpenAI model on its registry api when only CUA browser tools are selected", () => { + const catalog = compile("openai:gpt-5.5", cua.toolsets.browser()); + expect(catalog.model.api).toBe("openai-responses"); + }); + + it("derives OPENAI_CUA_COMPUTER_API when OpenAI's native computer tool is selected", () => { + const catalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); + expect(catalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + }); + + it("keeps a Google model on pi's builtin transport when only CDP browser tools are selected", () => { + const catalog = compile("google:gemini-3.6-flash", [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + expect(catalog.model.api).toBe("google-generative-ai"); + }); + + it("derives GOOGLE_CUA_INTERACTIONS_API when Google's native browser toolset is selected", () => { + const catalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); + expect(catalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + }); + + it("rejects a catalog whose selected tools require conflicting transports", () => { + const [click, scroll] = cua.providers.google.toolsets.browser(); + const conflicting: CuaToolSpec = { + ...scroll!, + identity: "test.conflicting-transport.v1", + providerBinding: { kind: "google-native", nativeName: "conflict", allNativeNames: ["conflict"], requiresApi: OPENAI_CUA_COMPUTER_API }, + }; + expect(() => compile("google:gemini-3.6-flash", [click!, conflicting])).toThrow(/incompatible provider transports/); + }); + + it("re-derives from a model object that already carries a stale derived api, instead of pinning it", () => { + const nativeCatalog = compile("google:gemini-3.6-flash", cua.providers.google.toolsets.browser()); + expect(nativeCatalog.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + const recompiled = compile(nativeCatalog.model, [cua.tools.browser.snapshot(), cua.tools.browser.click()]); + expect(recompiled.model.api).toBe("google-generative-ai"); + + const reselected = compile(recompiled.model, cua.providers.google.toolsets.browser()); + expect(reselected.model.api).toBe(GOOGLE_CUA_INTERACTIONS_API); + }); + + it("re-derives an OpenAI model object that already carries a stale derived api, instead of pinning it", () => { + const nativeCatalog = compile("openai:gpt-5.5", [cua.providers.openai.tools.computer()]); + expect(nativeCatalog.model.api).toBe(OPENAI_CUA_COMPUTER_API); + + const recompiled = compile(nativeCatalog.model, cua.toolsets.browser()); + expect(recompiled.model.api).toBe("openai-responses"); + }); +}); diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 873ed86..1573a28 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## 0.11.0 - 2026-08-13 + +- Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.12.0. The default + Google interaction catalog is unchanged (`defaultInteractionTools` still + selects Google's native browser toolset), so the default `cua` model and + `--print -o jsonl`'s `assistant_usage.api` field for Google are unaffected + as long as that native toolset stays selected, including across an in-session + `/model` switch. Only a `/tools` selection that drops Google's native toolset + now reports pi's builtin `google-generative-ai` api instead of the CUA-owned + one. + ## 0.10.0 - 2026-08-13 - `--print -o jsonl` schema bumps to version 2: every assistant message now From 918e155d07587fca930b0f6e893d1f3d2877f85c Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 21:25:45 +0000 Subject: [PATCH 2/7] Compile a model switch and its tools as one pair The CLI staged a model switch as setTools(application tools), setModel(next), setTools(next interaction tools). Now that the selected tools decide the transport, the first of those compiles an intermediate catalog whose derived transport belongs to neither the old nor the new selection, and pushing it records a model change for a transport nothing ever streamed with. Add `CuaAgentHarness.setModelAndTools()`, which compiles the pair once and pushes it once, and switch the TUI's `/model` flow and its rollback to it. --- packages/agent/CHANGELOG.md | 6 ++++++ packages/agent/src/agent.ts | 22 ++++++++++++++++++++++ packages/agent/src/tool-manager.ts | 7 ++++++- packages/agent/test/agent.test.ts | 18 ++++++++++++++++++ packages/cli/src/tui/main.ts | 14 +++++--------- 5 files changed, 57 insertions(+), 10 deletions(-) diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 5ce0ab9..3bb135b 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -2,6 +2,12 @@ ## 0.12.0 - 2026-08-13 +- Add `CuaAgentHarness.setModelAndTools()`. A model switch that also swaps + interaction tools has to compile as one pair now that the selected tools + decide the transport: staging the two in sequence produces an intermediate + catalog whose derived transport differs from both the old and the new one, + and records a model change for a transport nothing ever streamed with. + - Update `@onkernel/cua-ai` to 0.12.0. The model streamed for a Google model now depends on which tools `CuaAgent`/`CuaAgentHarness` were constructed or mutated with: selecting Google's native browser toolset still compiles to diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index b837e5c..ff86d64 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -437,6 +437,28 @@ export class CuaAgentHarness< this.tools.commit(prepared); } + /** + * Select a model and its tool list in one compile. A model switch that also + * swaps interaction tools would otherwise stage through an intermediate + * catalog whose derived transport differs from both the old and the new one, + * recording a `model_change` for a transport nothing ever streamed with. + */ + async setModelAndTools(model: CuaModelInput, tools: readonly CuaHarnessTool[]): Promise { + const previousModel = this.tools.catalog.model; + const previousTools = this.tools.harnessTools(); + const prepared = this.tools.prepareModelAndTools(model, tools); + const materialized = this.tools.harnessTools(prepared); + try { + await this.coreHarness.setModel(prepared.catalog.model); + await this.coreHarness.setTools(materialized, materialized.map((tool) => tool.name)); + } catch (error) { + await this.coreHarness.setModel(previousModel); + await this.coreHarness.setTools(previousTools, previousTools.map((tool) => tool.name)); + throw error; + } + this.tools.commit(prepared); + } + prompt(text: string, options?: { images?: ImageContent[] }) { return this.coreHarness.prompt(text, options); } skill(name: string, additionalInstructions?: string) { return this.coreHarness.skill(name, additionalInstructions); } promptFromTemplate(name: string, args?: string[]) { return this.coreHarness.promptFromTemplate(name, args); } diff --git a/packages/agent/src/tool-manager.ts b/packages/agent/src/tool-manager.ts index 6e527e9..aa216cb 100644 --- a/packages/agent/src/tool-manager.ts +++ b/packages/agent/src/tool-manager.ts @@ -128,6 +128,11 @@ export class CuaToolManager = CuaAgentToo return this.prepare(model, this.current.requested); } + prepareModelAndTools(model: CuaModelRef | Model, tools: readonly TRequested[]): PreparedCuaTools { + this.assertMutationScope("setModelAndTools"); + return this.prepare(model, tools); + } + commit(prepared: PreparedCuaTools): void { this.current = prepared; } @@ -224,7 +229,7 @@ export class CuaToolManager = CuaAgentToo }); } - private assertMutationScope(api: "setTools" | "setModel"): void { + private assertMutationScope(api: "setTools" | "setModel" | "setModelAndTools"): void { const scope = this.execution.getStore(); if (scope && scope.executionMode !== "sequential") { throw new Error(`tool "${scope.toolName}" must declare executionMode: "sequential" before calling ${api}() during execution`); diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index e5c5ab9..0a34fa6 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -637,4 +637,22 @@ describe("CuaAgentHarness explicit tools", () => { const modelChanges = (await services.session.getBranch()).filter((entry) => entry.type === "model_change"); expect(modelChanges).toEqual([]); }); + + it("records one model change for a switch that changes both the model and its derived transport", async () => { + const services = await harnessServices(); + const harness = new CuaAgentHarness({ + ...services, + browser, + client, + model: "google:gemini-3.6-flash", + tools: cua.providers.google.toolsets.browser(), + }); + expect(harness.getModel().api).toBe(GOOGLE_CUA_INTERACTIONS_API); + + await harness.setModelAndTools("openai:gpt-5.5", [cua.tools.browser.snapshot()]); + expect(harness.getModel().api).toBe("openai-responses"); + + const modelChanges = (await services.session.getBranch()).filter((entry) => entry.type === "model_change"); + expect(modelChanges).toHaveLength(1); + }); }); diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts index 9e1d52c..0e41623 100644 --- a/packages/cli/src/tui/main.ts +++ b/packages/cli/src/tui/main.ts @@ -378,16 +378,12 @@ export async function runInteractive(opts: InteractiveOptions): Promise const previousTools = opts.harness.getTools(); installedTools = [...opts.interactionToolsForModel(resolved), ...opts.applicationTools]; try { - // Native catalogs can be incompatible across providers. Transition - // through the CLI-owned application tools before selecting the new - // model's explicit interaction catalog. - await opts.harness.setTools(opts.applicationTools); - await opts.harness.setModel(resolved); - await opts.harness.setTools(installedTools); + // Native catalogs are incompatible across providers, and the selected + // tools decide the transport, so the new model and its interaction + // catalog have to compile as one pair rather than in sequence. + await opts.harness.setModelAndTools(resolved, installedTools); } catch (error) { - await opts.harness.setTools(opts.applicationTools); - await opts.harness.setModel(previousModel); - await opts.harness.setTools(previousTools); + await opts.harness.setModelAndTools(previousModel, previousTools); throw error; } } else { From e87ad3665dddef318ab40e8fc02e1bf263ab1bfb Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 21:35:53 +0000 Subject: [PATCH 3/7] Drop the redundant model-switch rollback in the TUI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `setModelAndTools` compiles before it mutates and restores its own state if the mutation fails, so the caller's catch was re-entering setModel/setTools for a switch that never landed — journaling model changes for a transport that was never streamed with. Update the neighboring comments and the CLI test that mirrored the old three-step transition. --- packages/cli/src/tui/main.ts | 27 ++++++++------------- packages/cli/test/tool-revalidation.test.ts | 7 +++--- 2 files changed, 13 insertions(+), 21 deletions(-) diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts index 0e41623..ecf95ea 100644 --- a/packages/cli/src/tui/main.ts +++ b/packages/cli/src/tui/main.ts @@ -364,9 +364,9 @@ export async function runInteractive(opts: InteractiveOptions): Promise let exitRequested = false; /** - * Apply a model switch. Extracted verbatim from the previous - * `applyModelCommand` so the picker and `/model ` share one path, - * including the three-step tool transition and its rollback. + * Apply a model switch. The picker and `/model ` share this one path. + * A failed switch needs no rollback here: the harness compiles before it + * mutates and restores its own state if the mutation fails. */ const applySwitchModel = async (resolved: CuaModelRef): Promise => { // The exact list installed by this switch, kept so it can become the new @@ -374,18 +374,11 @@ export async function runInteractive(opts: InteractiveOptions): Promise // policy, in which case the switch never touches the tool list at all. let installedTools: readonly CuaCliTool[] | undefined; if (opts.interactionToolsForModel) { - const previousModel = opts.harness.getModel(); - const previousTools = opts.harness.getTools(); installedTools = [...opts.interactionToolsForModel(resolved), ...opts.applicationTools]; - try { - // Native catalogs are incompatible across providers, and the selected - // tools decide the transport, so the new model and its interaction - // catalog have to compile as one pair rather than in sequence. - await opts.harness.setModelAndTools(resolved, installedTools); - } catch (error) { - await opts.harness.setModelAndTools(previousModel, previousTools); - throw error; - } + // Native catalogs are incompatible across providers, and the selected + // tools decide the transport, so the new model and its interaction + // catalog have to compile as one pair rather than in sequence. + await opts.harness.setModelAndTools(resolved, installedTools); } else { await opts.harness.setModel(resolved); } @@ -418,9 +411,9 @@ export async function runInteractive(opts: InteractiveOptions): Promise /** * Serialized entry point for a model switch. Queued behind any in-flight - * `/tools` apply so the apply's `setTools` cannot land between this switch's - * `setModel` and its final `setTools`. Rejects with the underlying failure; - * the harness has already rolled back by then. + * `/tools` apply so the apply's `setTools` cannot land mid-switch and compile + * its tool subset against the other model. Rejects with the underlying + * failure; the harness has already rolled back by then. */ const switchModel = (resolved: CuaModelRef): Promise => catalogQueue.run(() => applySwitchModel(resolved)); diff --git a/packages/cli/test/tool-revalidation.test.ts b/packages/cli/test/tool-revalidation.test.ts index d2c5dac..ad3f5cf 100644 --- a/packages/cli/test/tool-revalidation.test.ts +++ b/packages/cli/test/tool-revalidation.test.ts @@ -72,10 +72,9 @@ describe("/tools selection revalidation", () => { tools: [...defaultInteractionTools(from), ...application], }); - // Mirrors the TUI's three-step transition in switchModel(). - await fixture.harness.setTools(application); - await fixture.harness.setModel(to); - await fixture.harness.setTools([...defaultInteractionTools(to), ...application]); + // Mirrors switchModel(): the new model and its interaction catalog compile + // as one pair, because the selected tools decide the transport. + await fixture.harness.setModelAndTools(to, [...defaultInteractionTools(to), ...application]); const expected = [...defaultInteractionTools(to), ...application].map(toolKey); expect(fixture.harness.getTools().map(toolKey)).toEqual(expected); From c10cc14a672780c10d4938573b52b3544274ae6d Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 22:39:45 +0000 Subject: [PATCH 4/7] Make the compiled api the only OpenAI dispatch key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The provider wrapper routed on model.api and the stream function then re-decided on the incoming tool plan — two mechanisms for one decision. Give the native computer adapter its own stream function, selected by api in the wrapper, so the compiled api id is the only dispatch key. Also delete the Yutori n1.5 expanded action declarations and the canonical action type aliases nothing referenced. The expanded set was scaffolding for a ref/DOM execution path that does not exist, and none of it was exported from the package root. --- packages/ai/CHANGELOG.md | 11 ++++ packages/ai/src/providers.ts | 18 ++++--- packages/ai/src/providers/openai/provider.ts | 12 ++--- packages/ai/src/providers/yutori/actions.ts | 51 +------------------ .../ai/test/openai-native-provider.test.ts | 13 +++-- 5 files changed, 37 insertions(+), 68 deletions(-) diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 9e7418e..939ccfe 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -2,6 +2,17 @@ ## 0.12.0 - 2026-08-13 +- Route OpenAI's native computer adapter through its own stream function, + `streamOpenAICuaComputer`, selected by `model.api`. `streamOpenAIResponses` + no longer inspects the incoming tool plan to decide which adapter runs: the + compiled api id is the only dispatch key, and the provider wrapper is the + only place that reads it. +- Remove the unreferenced Yutori n1.5 expanded action declarations + (`YUTORI_N15_EXPANDED_ACTION_TYPES`, `YUTORI_N15_EXPANDED_TOOL_SET`, + `YUTORI_N15_ACTION_TYPES`) and the canonical-action type aliases nothing + consumed. None were exported from the package root. The expanded set was + scaffolding for a ref/DOM execution path that does not exist. + Breaking: Google's api id is now derived from selected tools, not stamped on every Google model. diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index c3d42be..d87eba5 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -19,7 +19,7 @@ import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; import { cuaOverrideModels } from "./models"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; -import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; +import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAICuaComputer, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; import { streamSimpleTzafonResponses, streamTzafonResponses, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; @@ -87,13 +87,17 @@ function withOpenAICuaAdapter(base: Provider): Provider { return { ...base, stream: (model: Model, context: Context, options?: StreamOptions) => - model.api === OPENAI_CUA_COMPUTER_API || requiresCuaOpenAINamespaceAdapter(context) - ? streamOpenAIResponses(model as never, context, options) - : base.stream(model, context, options), + model.api === OPENAI_CUA_COMPUTER_API + ? streamOpenAICuaComputer(model as never, context, options) + : requiresCuaOpenAINamespaceAdapter(context) + ? streamOpenAIResponses(model as never, context, options) + : base.stream(model, context, options), streamSimple: (model: Model, context: Context, options?: SimpleStreamOptions) => - model.api === OPENAI_CUA_COMPUTER_API || requiresCuaOpenAINamespaceAdapter(context) - ? streamSimpleOpenAIResponses(model as never, context, options) - : base.streamSimple(model, context, options), + model.api === OPENAI_CUA_COMPUTER_API + ? streamOpenAICuaComputer(model as never, context, options) + : requiresCuaOpenAINamespaceAdapter(context) + ? streamSimpleOpenAIResponses(model as never, context, options) + : base.streamSimple(model, context, options), }; } diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/ai/src/providers/openai/provider.ts index f2e4b52..96a2ab0 100644 --- a/packages/ai/src/providers/openai/provider.ts +++ b/packages/ai/src/providers/openai/provider.ts @@ -52,13 +52,10 @@ export function requiresCuaOpenAINamespaceAdapter(context: Context): boolean { return false; } -export const streamOpenAIResponses: StreamFunction<"openai-responses", OpenAIResponsesOptions> = (model, context, options) => { - if (options?.cuaIncomingToolPlan?.openaiComputerName) return streamOpenAINativeComputer(model, context, options); - return streamOpenAIFunctionTools(model, context, options); -}; +export const streamOpenAIResponses: StreamFunction<"openai-responses", OpenAIResponsesOptions> = (model, context, options) => + streamOpenAIFunctionTools(model, context, options); export const streamSimpleOpenAIResponses: StreamFunction<"openai-responses", CuaSimpleStreamOptions> = (model, context, options) => { - if (options?.cuaIncomingToolPlan?.openaiComputerName) return streamOpenAINativeComputer(model, context, options); const base = buildBaseOptions(model, context, options, options?.apiKey); const clampedReasoning = options?.reasoning ? clampThinkingLevel(model, options.reasoning) : undefined; return streamOpenAIFunctionTools(model, context, { @@ -283,7 +280,10 @@ function applyServiceTierPricing( usage.cost.total = usage.cost.input + usage.cost.output + usage.cost.cacheRead + usage.cost.cacheWrite; } -/** Responses adapter used only when the selected catalog contains OpenAI's native computer tool. */ +/** Responses adapter for models the catalog compiled onto {@link OPENAI_CUA_COMPUTER_API}, i.e. those whose selected tools include OpenAI's native computer. */ +export const streamOpenAICuaComputer: StreamFunction = (model, context, options) => + streamOpenAINativeComputer(model as unknown as Model<"openai-responses">, context, options); + function streamOpenAINativeComputer( model: Model<"openai-responses">, context: Context, diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts index 9c40ae5..3cd2c53 100644 --- a/packages/ai/src/providers/yutori/actions.ts +++ b/packages/ai/src/providers/yutori/actions.ts @@ -1,4 +1,4 @@ -import type { CuaAction, CuaActionType } from "../../actions/index"; +import type { CuaAction } from "../../actions/index"; import { normalizeGotoUrl } from "../common"; /** @@ -9,19 +9,6 @@ import { normalizeGotoUrl } from "../common"; * - https://docs.yutori.com/llm-quickstart.md */ export const YUTORI_N15_CORE_TOOL_SET = "browser_tools_core-20260403"; -export const YUTORI_N15_EXPANDED_TOOL_SET = "browser_tools_expanded-20260403"; - -/** - * DOM/ref-backed Navigator n1.5 actions. We intentionally disable these until - * CuaAgent has the ref/DOM execution path that Yutori documents for the - * expanded tool set. - */ -export const YUTORI_N15_EXPANDED_ACTION_TYPES = [ - "extract_elements", - "find", - "set_element_value", - "execute_js", -] as const; /** * Navigator n1's fixed legacy browser action space. @@ -72,42 +59,6 @@ export const YUTORI_N15_CORE_ACTION_TYPES = [ "wait", ] as const; -export const YUTORI_N15_ACTION_TYPES = [ - ...YUTORI_N15_CORE_ACTION_TYPES, - ...YUTORI_N15_EXPANDED_ACTION_TYPES, -] as const; - -/** - * Canonical CUA action types Yutori's native actions normalize into. These are - * the tool-call names {@link streamYutori} emits and the local executors - * CuaAgent installs for Yutori models. - */ -export const YUTORI_CUA_ACTION_TYPES = [ - "click", - "double_click", - "mouse_down", - "mouse_up", - "type", - "keypress", - "scroll", - "move", - "drag", - "wait", - "goto", - "back", - "forward", -] as const satisfies readonly CuaActionType[]; - -type YutoriCanonicalActionType = (typeof YUTORI_CUA_ACTION_TYPES)[number]; - -/** Canonical CUA action shape emitted for Yutori models. */ -export type YutoriAction = Extract; - -export type YutoriN1ActionType = (typeof YUTORI_N1_ACTION_TYPES)[number]; -export type YutoriN15CoreActionType = (typeof YUTORI_N15_CORE_ACTION_TYPES)[number]; -export type YutoriN15ExpandedActionType = (typeof YUTORI_N15_EXPANDED_ACTION_TYPES)[number]; -export type YutoriNativeActionType = YutoriN1ActionType | YutoriN15CoreActionType | YutoriN15ExpandedActionType; - const DEFAULT_SCROLL_AMOUNT = 3; const SCROLL_AMOUNT_PER_NOTCH = 120; const DEFAULT_WAIT_MS = 2000; diff --git a/packages/ai/test/openai-native-provider.test.ts b/packages/ai/test/openai-native-provider.test.ts index d4cd523..d4a584a 100644 --- a/packages/ai/test/openai-native-provider.test.ts +++ b/packages/ai/test/openai-native-provider.test.ts @@ -16,6 +16,9 @@ vi.mock("openai", () => ({ })); const model = getCuaModel("openai:gpt-5.5") as Model<"openai-responses">; +// The catalog derives this api when OpenAI's native computer tool is selected; +// the provider wrapper routes it to the adapter under test. +const nativeModel = { ...model, api: openai.OPENAI_CUA_COMPUTER_API } as unknown as Model<"openai-responses">; const incoming = { openaiComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; describe("OpenAI native computer Responses adapter", () => { @@ -30,7 +33,7 @@ describe("OpenAI native computer Responses adapter", () => { pending_safety_checks: [{ id: "check_1", code: "malicious_instructions" }], }], }); - const message = await openai.streamOpenAIResponses(model, { + const message = await openai.streamOpenAICuaComputer(nativeModel, { messages: [], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { @@ -55,7 +58,7 @@ describe("OpenAI native computer Responses adapter", () => { it("sends the same prompt-cache fields as the function-tool path", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_cache", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [{ role: "user", content: "go", timestamp: 1 }], tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", sessionId: "session_native", cuaIncomingToolPlan: incoming }).result(); @@ -78,7 +81,7 @@ describe("OpenAI native computer Responses adapter", () => { arguments: '{"query":"status"}', }], }); - const first = await openai.streamOpenAIResponses(model, { + const first = await openai.streamOpenAICuaComputer(nativeModel, { messages: [{ role: "user", content: "look it up", timestamp: 1 }], tools: [ { name: "computer", description: "placeholder", parameters: { type: "object" } as never }, @@ -89,7 +92,7 @@ describe("OpenAI native computer Responses adapter", () => { expect(call.namespace).toBe("deferred_tools"); responsesCreate.mockReturnValueOnce({ id: "resp_done", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [ { role: "user", content: "look it up", timestamp: 1 }, first, @@ -122,7 +125,7 @@ describe("OpenAI native computer Responses adapter", () => { it("serializes native results as computer_call_output and ordinary results as function output", async () => { responsesCreate.mockReturnValueOnce({ id: "resp_2", usage: {}, output: [] }); - await openai.streamOpenAIResponses(model, { + await openai.streamOpenAICuaComputer(nativeModel, { messages: [ { role: "assistant", From d74a7ba306af94d3ae4f4ca0280ef27a73afc71b Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 23:24:55 +0000 Subject: [PATCH 5/7] Remove Tzafon and Yutori provider support Neither provider is used, and both were the last models whose transport was model-shaped rather than derived from the selected tools. Removing them deletes two provider adapters, five test files, the `@tzafon/lightcone` dependency, and the concepts that existed only to serve them. What got simpler beyond the deletion: - `CuaProviderBinding` drops from five variants to three, and the per-entry provider-facing declaration collapses to a single branch. - The atomic-group concept disappears end to end. Yutori's n1 toolset was the only native set that rejected a partial selection, so the compiler rule, the CLI `/tools` picker's linked-key handling, and its warning line all go. - `compileCuaToolCatalog` loses its `viewport` option: Tzafon's declaration sizing was the only reader, so the tool manager no longer threads the browser viewport into compilation. - `CuaIncomingToolPlan` sheds two fields, and `routeCuaApi` loses both of its remaining transport branches. The OpenAI native-computer exemption from the tool-result image replay limit stays, along with its regression test. --- .agents/skills/update-models/SKILL.md | 28 +- .../skills/update-models/reference/README.md | 4 - .../reference/audit-official-examples.ts | 11 +- .../reference/discover-models.ts | 317 +----------- .../reference/native-action-probe.ts | 103 +--- .../reference/provider-doc-drift.ts | 14 +- .../update-models/reference/report-schema.md | 11 +- .github/workflows/ci.yml | 4 - README.md | 4 - docs/agent-tool-configuration-spec.md | 8 +- docs/architecture.md | 14 +- package-lock.json | 21 - package.json | 1 - packages/agent/CHANGELOG.md | 14 + packages/agent/README.md | 23 +- packages/agent/examples/shared/tools.ts | 11 - packages/agent/src/agent.ts | 6 +- packages/agent/src/resources.ts | 2 - packages/agent/src/tool-manager.ts | 1 - packages/agent/test/agent.test.ts | 28 - packages/agent/test/e2e.live.test.ts | 39 +- .../test/example-provider-matrix.test.ts | 6 +- packages/agent/test/resources.test.ts | 6 +- packages/ai/CHANGELOG.md | 25 + packages/ai/README.md | 17 +- packages/ai/docs/supported-models.md | 23 - packages/ai/examples/quickstart.ts | 1 - packages/ai/package.json | 1 - packages/ai/src/api-keys.ts | 2 - packages/ai/src/cua.ts | 63 --- packages/ai/src/index.ts | 6 - packages/ai/src/models.ts | 88 +--- packages/ai/src/providers.ts | 34 +- packages/ai/src/providers/common.ts | 12 - packages/ai/src/providers/tzafon/provider.ts | 487 ------------------ packages/ai/src/providers/yutori/actions.ts | 188 ------- packages/ai/src/providers/yutori/provider.ts | 229 -------- packages/ai/src/tool-catalog.ts | 127 +---- .../test/anthropic-browser-fallback.test.ts | 4 - .../test/anthropic-native.integration.test.ts | 3 - packages/ai/test/api-keys.test.ts | 6 +- packages/ai/test/google-provider.test.ts | 2 - packages/ai/test/models.test.ts | 16 +- .../ai/test/openai-adapter-routing.test.ts | 2 +- .../ai/test/openai-native-provider.test.ts | 2 +- packages/ai/test/providers.test.ts | 29 +- packages/ai/test/tool-catalog.test.ts | 66 +-- packages/ai/test/tzafon-actions.test.ts | 80 --- packages/ai/test/tzafon-provider.test.ts | 149 ------ packages/ai/test/tzafon-threading.test.ts | 159 ------ packages/ai/test/yutori-actions.test.ts | 97 ---- packages/ai/test/yutori-provider.test.ts | 72 --- packages/cli/CHANGELOG.md | 12 + packages/cli/README.md | 23 +- packages/cli/src/cli-harness.ts | 2 +- packages/cli/src/cli.ts | 6 - packages/cli/src/harness.ts | 9 - packages/cli/src/tui/tool-selection.ts | 63 +-- packages/cli/src/tui/tools-picker.ts | 11 +- packages/cli/test/cli-executor.test.ts | 2 - packages/cli/test/harness-assembly.test.ts | 6 - packages/cli/test/tool-revalidation.test.ts | 20 +- packages/cli/test/tool-selection.test.ts | 45 +- skills/cua-cli/SKILL.md | 16 +- 64 files changed, 207 insertions(+), 2674 deletions(-) delete mode 100644 packages/ai/src/providers/tzafon/provider.ts delete mode 100644 packages/ai/src/providers/yutori/actions.ts delete mode 100644 packages/ai/src/providers/yutori/provider.ts delete mode 100644 packages/ai/test/tzafon-actions.test.ts delete mode 100644 packages/ai/test/tzafon-provider.test.ts delete mode 100644 packages/ai/test/tzafon-threading.test.ts delete mode 100644 packages/ai/test/yutori-actions.test.ts delete mode 100644 packages/ai/test/yutori-provider.test.ts diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md index f09e343..5be03cd 100644 --- a/.agents/skills/update-models/SKILL.md +++ b/.agents/skills/update-models/SKILL.md @@ -1,6 +1,6 @@ --- name: update-models -description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Moonshot, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. +description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results. --- # Update Models @@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer- ## Quick Start -1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`. +1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. 2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them: ```bash eval "$(python3 - <<'PY' import pathlib, re, shlex text = pathlib.Path('~/AGENTS.md').expanduser().read_text() -for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']: +for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text) if m: print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}') @@ -61,7 +61,7 @@ Treat example repos as strongest when they are provider-owned or linked from off There are two enumeration layers: -- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, xAI's OpenAI-compatible `models.list()`, Tzafon's `Lightcone().models.list()` with known-model fallback, and Yutori OpenAPI/docs model enums) to discover what the current API key can access. +- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access. - CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use. When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("")`. @@ -124,22 +124,6 @@ Moonshot: - Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn. - Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes. -Tzafon: - -- Discover with `new Lightcone({ apiKey }).models.list()` from `@tzafon/lightcone` when available. -- If model listing is unavailable or returns an undocumented shape, record the error/shape and fall back to known smoke-test candidates such as `tzafon.northstar-cua-fast`. -- Smoke-test `responses.create` with explicit function tools matching the Tzafon template: `click`, `double_click`, `point_and_type`, `key`, `scroll`, `drag`, and `done`. -- Pass condition: response output contains `type: "function_call"` with one of those tool names, or a documented `computer_call` action if Lightcone switches to native computer-use output. -- Track coordinate convention separately from Gemini/Yutori: Tzafon uses a 0-999 grid. - -Yutori: - -- Discover model IDs from `https://docs.yutori.com/openapi.json` plus the Navigator docs. Current expected IDs include `n1.5-latest`, `n1.5-20260428`, `n1-latest`, and `n1-20260203`. -- Smoke-test the OpenAI-compatible `chat.completions` endpoint with `baseURL: "https://api.yutori.com/v1"` and `YUTORI_API_KEY`. -- Pass condition: response `choices[0].message.tool_calls[]` contains browser action function names such as `left_click`, `goto_url`, `type`, `scroll`, or `wait`. -- Track action-space differences between n1 and n1.5. n1 uses the legacy fixed tool set; n1.5 supports `tool_set`, `disable_tools`, expanded actions, and structured JSON output. -- Do not send duplicate browser action schemas when testing local CUA behavior. The local adapter registers matching AgentTools for execution but filters Yutori's built-in browser tool definitions out of the outbound API payload. - ## Native Action Discovery Run action probes when updating adapters or when docs/examples show drift: @@ -151,8 +135,6 @@ npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest ``` The probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare: @@ -198,8 +180,6 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed. - xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility. - Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`. - - Tzafon: update `packages/ai/src/providers/tzafon/index.ts` and `provider.ts`, including coordinate/action handling. - - Yutori: update `packages/ai/src/providers/yutori/actions.ts`, `index.ts`, and `provider.ts`, including payload filtering and coordinate/action handling. - Shared canonical action semantics go in `packages/ai/src/providers/common.ts`. - New provider or routing rule: diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md index d2e6c09..d6ca42b 100644 --- a/.agents/skills/update-models/reference/README.md +++ b/.agents/skills/update-models/reference/README.md @@ -14,8 +14,6 @@ These scripts support the `update-models` skill. Run them from the repository ro - `META_API_KEY` - `XAI_API_KEY` - `MOONSHOT_API_KEY` - - `TZAFON_API_KEY` - - `YUTORI_API_KEY` The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools. @@ -32,8 +30,6 @@ Probe native action vocabularies for a specific provider/model: ```bash npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --out /tmp/tzafon-actions.json -npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --out /tmp/yutori-actions.json ``` Clone/update official examples and extract tool-handling evidence: diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts index 1005b43..524c60a 100644 --- a/.agents/skills/update-models/reference/audit-official-examples.ts +++ b/.agents/skills/update-models/reference/audit-official-examples.ts @@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path"; import { spawnSync } from "node:child_process"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; interface ExampleRepo { provider: Provider; @@ -67,14 +67,6 @@ const EXAMPLES: ExampleRepo[] = [ pathHint: "examples", patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"], }, - { - provider: "yutori", - name: "kernel-cli-yutori-template", - repo: "https://github.com/kernel/cli.git", - confidence: "kernel-template", - pathHint: "pkg/templates/typescript/yutori", - patterns: ["YUTORI_API_KEY", "tool_calls", "left_click", "goto_url", "n1-latest", "api.yutori.com"], - }, ]; const ACTION_REGEXES: Record = { @@ -84,7 +76,6 @@ const ACTION_REGEXES: Record = { meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g], xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], - yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g], }; function parseArgs(argv: string[]): Args { diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts index f243e6c..0fa9952 100644 --- a/.agents/skills/update-models/reference/discover-models.ts +++ b/.agents/skills/update-models/reference/discover-models.ts @@ -1,12 +1,10 @@ #!/usr/bin/env tsx import { existsSync } from "node:fs"; import { readFile, writeFile } from "node:fs/promises"; -import { homedir } from "node:os"; import { join } from "node:path"; import process from "node:process"; -import { parse as parseToml } from "smol-toml"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "tzafon" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; interface Args { provider: Provider | "all"; @@ -39,16 +37,7 @@ interface ModelResult { cua?: Record; } -const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot", "tzafon", "yutori"]; -const TZAFON_KNOWN_MODELS = [ - "tzafon.northstar-cua-fast", -]; -const YUTORI_DOC_MODELS = [ - "n1.5-latest", - "n1.5-20260428", - "n1-latest", - "n1-20260203", -]; +const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot"]; const GEMINI_DOC_COMPUTER_USE_MODELS = [ "gemini-3.5-flash", "gemini-3-flash-preview", @@ -110,7 +99,7 @@ function usage(): never { npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4 Options: - --provider + --provider --models Smoke-test explicit models instead of inferred candidates. --candidate-limit Max inferred candidates per provider. Default: 20. --no-smoke Only list metadata. @@ -141,8 +130,6 @@ async function runProvider(provider: Provider, args: Args): Promise> { return { provider: "gemini", metadata_source: "client.models.list()", models, candidates }; } -async function discoverYutori(args: Args): Promise> { - const openapi = await fetchYutoriOpenApi(); - const ids = unique([ - ...extractYutoriModelIds(openapi.raw), - ...YUTORI_DOC_MODELS, - ]); - const models: ModelResult[] = ids.map((id) => ({ - id, - display_name: yutoriDisplayName(id), - created_at: yutoriCreatedAt(id), - raw: { source: "docs.yutori.com/openapi.json" }, - supports_generation: true, - model_docs: { - navigator_docs: "https://docs.yutori.com/reference/navigator", - n1_docs: "https://docs.yutori.com/reference/n1", - n15_docs: "https://docs.yutori.com/reference/n1-5", - openapi_ok: openapi.ok, - tool_set: id.startsWith("n1.5") ? "browser_tools_core-20260403" : "legacy_fixed", - disable_tools: id.startsWith("n1.5") ? "supported" : "not_supported", - coordinate_space: "1000x1000", - }, - })); - const candidates = explicitOrCandidates(args, ids); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { - id, - display_name: yutoriDisplayName(id), - supports_generation: true, - }; - model.computer_use = await smokeYutori(id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("yutori", models); - return { - provider: "yutori", - metadata_source: "https://docs.yutori.com/openapi.json + Navigator docs", - openapi: openapi.summary, - models, - candidates, - }; -} - -async function fetchYutoriOpenApi(): Promise<{ ok: boolean; raw: unknown; summary: Record }> { - const url = "https://docs.yutori.com/openapi.json"; - try { - const response = await fetch(url); - const raw = await response.json(); - return { - ok: response.ok, - raw, - summary: { - url, - ok: response.ok, - model_ids: extractYutoriModelIds(raw), - }, - }; - } catch (err) { - return { - ok: false, - raw: undefined, - summary: { url, ok: false, error: publicError(err) }, - }; - } -} - -function extractYutoriModelIds(raw: unknown): string[] { - const found: string[] = []; - const visit = (value: unknown): void => { - if (!value || typeof value !== "object") return; - if (Array.isArray(value)) { - for (const item of value) visit(item); - return; - } - const obj = value as Record; - if (Array.isArray(obj.enum)) { - for (const item of obj.enum) { - if (typeof item === "string" && /^n1(?:\.5)?-/.test(item)) found.push(item); - } - } - for (const item of Object.values(obj)) visit(item); - }; - visit(raw); - return unique(found); -} - -function yutoriDisplayName(id: string): string { - if (id.startsWith("n1.5")) return `Yutori Navigator ${id.replace("n1.5", "n1.5")}`; - return `Yutori Navigator ${id.replace("n1", "n1")}`; -} - -function yutoriCreatedAt(id: string): string | null { - const match = id.match(/-(\d{4})(\d{2})(\d{2})$/); - if (!match) return null; - return `${match[1]}-${match[2]}-${match[3]}T00:00:00.000Z`; -} - -async function yutoriApiKey(): Promise { - const env = process.env.YUTORI_API_KEY; - if (env && env.trim()) return env; - const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml"); - try { - const raw = parseToml(await readFile(cfgPath, "utf8")) as any; - const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; - const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; - if (typeof key === "string" && key.trim()) return key; - } catch { - // Fall through to a clear credential error. - } - throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)"); -} - -async function smokeYutori(model: string): Promise { - try { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ - apiKey: await yutoriApiKey(), - baseURL: "https://api.yutori.com/v1", - }); - const response = await client.chat.completions.create({ - model, - messages: [{ - role: "user", - content: [ - { type: "text", text: "Use the browser action tool to inspect or interact with this page. Do not answer in text." }, - { type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, - ], - }], - max_completion_tokens: 128, - temperature: 0.3, - }); - const choice = response.choices?.[0]; - const toolCalls: any[] = choice?.message?.tool_calls ?? []; - return { - status: toolCalls.length > 0 ? "pass" : "inconclusive", - tool_name: "browser_actions", - tool_version: model.startsWith("n1.5") ? "n1.5" : "n1", - beta_header: null, - observed_actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), - response_item_types: unique([ - choice?.message?.content ? "text" : undefined, - toolCalls.length ? "tool_calls" : undefined, - ].filter(Boolean) as string[]), - finish_reason: choice?.finish_reason ?? null, - accepts_image_tool_results: "assumed-from-docs", - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "browser_actions" }); - } -} - -async function discoverTzafon(args: Args): Promise> { - const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone"); - const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY }); - const modelList = await fetchTzafonModels(client); - const ids = unique([ - ...modelList.ids, - ...TZAFON_KNOWN_MODELS, - ]); - const models: ModelResult[] = ids.map((id) => ({ - id, - display_name: tzafonDisplayName(id), - raw: modelList.rawById[id] ?? { source: "known-model-fallback" }, - supports_generation: true, - model_docs: { - docs: "https://docs.lightcone.ai", - responses_endpoint: "supported", - function_calling: "supported", - computer_use: "supported", - coordinate_space: "0-999", - model_list_endpoint: modelList.source, - }, - })); - const candidates = explicitOrCandidates(args, ids); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((m) => m.id === id) ?? { - id, - display_name: tzafonDisplayName(id), - supports_generation: true, - }; - model.computer_use = await smokeTzafon(client, id); - if (!models.find((m) => m.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("tzafon", models); - return { - provider: "tzafon", - metadata_source: modelList.source, - model_list_error: modelList.error ?? null, - models, - candidates, - }; -} - -async function fetchTzafonModels(client: any): Promise<{ - source: string; - ids: string[]; - rawById: Record; - raw?: unknown; - error?: string; -}> { - try { - const raw = await client.models.list(); - const entries = extractTzafonModelEntries(raw); - const rawById: Record = {}; - for (const entry of entries) rawById[entry.id] = entry.raw; - return { - source: "@tzafon/lightcone models.list()", - ids: entries.map((entry) => entry.id), - rawById, - raw, - }; - } catch (err) { - return { - source: "known-model-fallback (models.list unavailable)", - ids: [], - rawById: {}, - error: publicError(err), - }; - } -} - -function extractTzafonModelEntries(raw: unknown): Array<{ id: string; raw: unknown }> { - const entries: Array<{ id: string; raw: unknown }> = []; - const visit = (value: unknown): void => { - if (!value || typeof value !== "object") return; - if (Array.isArray(value)) { - for (const item of value) visit(item); - return; - } - const obj = value as Record; - const id = obj.id ?? obj.name ?? obj.model; - if (typeof id === "string" && id.trim()) { - entries.push({ id: id.trim(), raw: obj }); - } - for (const child of Object.values(obj)) { - if (child && typeof child === "object") visit(child); - } - }; - visit(raw); - return entries; -} - -function tzafonDisplayName(id: string): string { - if (id === "tzafon.northstar-cua-fast") return "Tzafon Northstar CUA Fast"; - return id; -} - -async function smokeTzafon(client: any, model: string): Promise { - try { - const response = await client.responses.create({ - model, - input: [{ - role: "user", - content: [ - { type: "input_text", text: "Use the computer function tools to inspect this page. Do not answer in text." }, - { type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, - ], - }], - tools: TZAFON_FUNCTION_TOOLS, - instructions: "The screen's coordinate space is a 0-999 grid. Call a function tool instead of answering in text.", - temperature: 0, - max_output_tokens: 128, - }); - const output: any[] = response.output ?? []; - const functionCalls = output.filter((item) => item?.type === "function_call"); - const computerCalls = output.filter((item) => item?.type === "computer_call"); - const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", - tool_name: functionCalls.length ? "function_tools" : "computer_use", - tool_version: null, - beta_header: null, - observed_actions: unique([ - ...functionCalls.map((call) => call?.name).filter(Boolean), - ...computerActions.map((action) => action?.type).filter(Boolean), - ]), - response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), - error: null, - }; - } catch (err) { - return smokeError(err, { tool_name: "function_tools" }); - } -} - -const TZAFON_FUNCTION_TOOLS = [ - { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } }, - { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } }, - { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } }, - { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } }, - { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } }, - { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } }, - { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } }, -]; - async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise { const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider; const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined); diff --git a/.agents/skills/update-models/reference/native-action-probe.ts b/.agents/skills/update-models/reference/native-action-probe.ts index 4c71255..c3b9434 100644 --- a/.agents/skills/update-models/reference/native-action-probe.ts +++ b/.agents/skills/update-models/reference/native-action-probe.ts @@ -1,12 +1,10 @@ #!/usr/bin/env tsx import { existsSync } from "node:fs"; import { readFile, writeFile } from "node:fs/promises"; -import { homedir } from "node:os"; import { join } from "node:path"; import process from "node:process"; -import { parse as parseToml } from "smol-toml"; -type Provider = "openai" | "anthropic" | "gemini" | "xai" | "tzafon" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "xai"; interface ProbePrompt { id: string; @@ -64,8 +62,8 @@ function parseArgs(argv: string[]): Args { throw new Error(`unknown argument: ${arg}`); } } - if (!["openai", "anthropic", "gemini", "xai", "tzafon", "yutori"].includes(out.provider)) { - throw new Error("--provider is required: openai | anthropic | gemini | xai | tzafon | yutori"); + if (!["openai", "anthropic", "gemini", "xai"].includes(out.provider)) { + throw new Error("--provider is required: openai | anthropic | gemini | xai"); } if (!out.model) throw new Error("--model is required"); return out; @@ -76,8 +74,6 @@ function usage(): never { npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --limit 3 - npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --limit 3 `); process.exit(0); } @@ -105,8 +101,6 @@ async function runProbe(provider: Provider, model: string, prompt: ProbePrompt): if (provider === "anthropic") return await probeAnthropic(model, prompt); if (provider === "gemini") return await probeGemini(model, prompt); if (provider === "xai") return await probeXai(model, prompt); - if (provider === "tzafon") return await probeTzafon(model, prompt); - if (provider === "yutori") return await probeYutori(model, prompt); throw new Error(`unknown provider ${provider satisfies never}`); } catch (err) { return { @@ -258,87 +252,6 @@ async function readFixtureScreenshot(): Promise { return readFile(path); } -async function probeYutori(model: string, prompt: ProbePrompt): Promise { - const OpenAI = await importDefault("openai", "OpenAI"); - const client = new OpenAI({ - apiKey: await yutoriApiKey(), - baseURL: "https://api.yutori.com/v1", - }); - const response = await client.chat.completions.create({ - model, - messages: [{ - role: "user", - content: [ - { type: "text", text: prompt.text }, - { type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, - ], - }], - max_completion_tokens: 128, - temperature: 0.3, - }); - const choice = response.choices?.[0]; - const toolCalls: any[] = choice?.message?.tool_calls ?? []; - return { - id: prompt.id, - status: toolCalls.length ? "pass" : "inconclusive", - actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), - item_types: unique([ - choice?.message?.content ? "text" : undefined, - toolCalls.length ? "tool_calls" : undefined, - ].filter(Boolean) as string[]), - finish_reason: choice?.finish_reason ?? null, - raw_tool_calls: toolCalls.map(redactLargeFields), - }; -} - -async function yutoriApiKey(): Promise { - const env = process.env.YUTORI_API_KEY; - if (env && env.trim()) return env; - const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml"); - try { - const raw = parseToml(await readFile(cfgPath, "utf8")) as any; - const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; - const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; - if (typeof key === "string" && key.trim()) return key; - } catch { - // Fall through to a clear credential error. - } - throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)"); -} - -async function probeTzafon(model: string, prompt: ProbePrompt): Promise { - const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone"); - const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY }); - const response = await client.responses.create({ - model, - input: [{ - role: "user", - content: [ - { type: "input_text", text: `${prompt.text}\nCall one of the available function tools instead of answering in text.` }, - { type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, - ], - }], - tools: TZAFON_FUNCTION_TOOLS, - instructions: "The screen's coordinate space is a 0-999 grid.", - temperature: 0, - max_output_tokens: 128, - }); - const output: any[] = response.output ?? []; - const functionCalls = output.filter((item) => item?.type === "function_call"); - const computerCalls = output.filter((item) => item?.type === "computer_call"); - const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); - return { - id: prompt.id, - status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", - actions: unique([ - ...functionCalls.map((call) => call?.name).filter(Boolean), - ...computerActions.map((action) => action?.type).filter(Boolean), - ]), - item_types: unique(output.map((item) => item?.type).filter(Boolean)), - raw_tool_calls: [...functionCalls, ...computerCalls].map(redactLargeFields), - }; -} - const XAI_FUNCTION_TOOLS = [ { type: "function", name: "screenshot", description: "Capture the current browser screenshot.", parameters: { type: "object", properties: {}, additionalProperties: false } }, { type: "function", name: "goto", description: "Navigate to a URL.", parameters: { type: "object", properties: { url: { type: "string" } }, required: ["url"], additionalProperties: false } }, @@ -352,16 +265,6 @@ const XAI_FUNCTION_TOOLS = [ { type: "function", name: "back", description: "Go back in browser history.", parameters: { type: "object", properties: {}, additionalProperties: false } }, ]; -const TZAFON_FUNCTION_TOOLS = [ - { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } }, - { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } }, - { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } }, - { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } }, - { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } }, - { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } }, - { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } }, -]; - async function importDefault(pkg: string, named: string): Promise { try { const mod = await import(pkg); diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts index 95ea4be..070dd51 100644 --- a/.agents/skills/update-models/reference/provider-doc-drift.ts +++ b/.agents/skills/update-models/reference/provider-doc-drift.ts @@ -2,7 +2,7 @@ import { readFile, writeFile } from "node:fs/promises"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori"; +type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; interface Args { examples: string; @@ -45,12 +45,6 @@ const DOCS: Record = { "https://platform.kimi.ai/docs/api/tool-use", "https://platform.kimi.ai/docs/guide/use-kimi-vision-model", ], - yutori: [ - "https://docs.yutori.com/reference/navigator", - "https://docs.yutori.com/reference/n1", - "https://docs.yutori.com/reference/n1-5", - "https://docs.yutori.com/openapi.json", - ], }; const LOCAL_FILES: Record = { @@ -60,7 +54,6 @@ const LOCAL_FILES: Record = { meta: "packages/ai/src/providers/meta/index.ts", xai: "packages/ai/src/providers/xai/index.ts", moonshot: "packages/ai/src/providers/moonshot/index.ts", - yutori: "packages/ai/src/providers/yutori/actions.ts", }; const ACTION_REGEXES: Record = { @@ -70,7 +63,6 @@ const ACTION_REGEXES: Record = { meta: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position|left_click|right_click|middle_click|triple_click|left_click_drag|mouse_move|key|hold_key|left_mouse_down|left_mouse_up)\b/g, xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, - yutori: /\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|go_forward|goto_url|mouse_move|middle_click|mouse_down|mouse_up|hold_key|extract_elements|find|set_element_value|execute_js)\b/g, }; function parseArgs(argv: string[]): Args { @@ -184,10 +176,6 @@ function notesFor(provider: Provider, documentedToolVersions: string[], exampleT if (provider === "moonshot") { notes.push("Moonshot uses developer-defined function tools over OpenAI-compatible chat completions and does not document a coordinate protocol; Kimi grounding emits 0-1 width/height fractions, so compare against CUA's fractional coordinate contract."); } - if (provider === "yutori") { - notes.push("Yutori Navigator emits OpenAI-compatible tool_calls for built-in browser actions; local AgentTools should execute those names but outbound payloads should not duplicate the built-in browser schemas."); - notes.push("Track n1 vs n1.5 separately because n1.5 can add tool_set/disable_tools and expanded browser actions."); - } return notes; } diff --git a/.agents/skills/update-models/reference/report-schema.md b/.agents/skills/update-models/reference/report-schema.md index 6dcc5d4..9bd2bc5 100644 --- a/.agents/skills/update-models/reference/report-schema.md +++ b/.agents/skills/update-models/reference/report-schema.md @@ -14,9 +14,7 @@ Use this shape for JSON reports and the same fields when writing a Markdown summ "gemini": {}, "meta": {}, "xai": {}, - "moonshot": {}, - "tzafon": {}, - "yutori": {} + "moonshot": {} }, "example_evidence": {}, "drift": {}, @@ -42,12 +40,7 @@ Use this shape for JSON reports and the same fields when writing a Markdown summ "streaming": "supported", "function_calling": "supported", "computer_use": "supported", - "responses_endpoint": "supported", - "navigator_docs": "https://docs.yutori.com/reference/navigator", - "tool_set": "browser_tools_core-20260403", - "disable_tools": "supported", - "coordinate_space": "1000x1000", - "model_list_endpoint": "@tzafon/lightcone models.list()" + "responses_endpoint": "supported" }, "computer_use": { "status": "pass", diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8a76da2..ed49e50 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -135,8 +135,6 @@ jobs: META_API_KEY: ${{ secrets.META_API_KEY }} XAI_API_KEY: ${{ secrets.XAI_API_KEY }} MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} - TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }} - YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }} run: npm run test:integration --workspace @onkernel/cua-ai agent-e2e: @@ -164,6 +162,4 @@ jobs: META_API_KEY: ${{ secrets.META_API_KEY }} XAI_API_KEY: ${{ secrets.XAI_API_KEY }} MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} - TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }} - YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }} run: npm test --workspace @onkernel/cua-agent -- test/e2e.live.test.ts diff --git a/README.md b/README.md index 6d903f1..28c54f0 100644 --- a/README.md +++ b/README.md @@ -97,7 +97,6 @@ export GOOGLE_API_KEY=... # for gemini-3.6-flash export META_API_KEY=... # for muse-spark-1.1 export XAI_API_KEY=xai-... # for grok-4.5 export MOONSHOT_API_KEY=sk-... # for kimi-k3 -export YUTORI_API_KEY=yt_... # for n1.5-latest export KERNEL_API_KEY=sk_... # always required # single-shot @@ -121,9 +120,6 @@ cua -p --model xai:grok-4.5 "Same prompt" # Moonshot Kimi K3 cua -p --model moonshotai:kimi-k3 "Same prompt" -# Yutori Navigator -cua -p --model n1.5-latest "Same prompt" - # interactive TUI (default mode) cua cua "summarize https://news.ycombinator.com" diff --git a/docs/agent-tool-configuration-spec.md b/docs/agent-tool-configuration-spec.md index 93ca071..4125edb 100644 --- a/docs/agent-tool-configuration-spec.md +++ b/docs/agent-tool-configuration-spec.md @@ -419,7 +419,7 @@ Composition must: 6. Establish explicit resource sharing without adding tools. 7. Install exactly the requested tools. -Payload transforms must operate on explicit tool identities, not infer ownership from names such as `click`. This is required for providers like Tzafon and Yutori, whose current native-tool adapters classify or replace tools by name. +Payload transforms must operate on explicit tool identities, not infer ownership from names such as `click`. This is required for native-tool adapters that classify or replace tools by name. ## Mid-conversation tool changes and provider caches @@ -512,7 +512,7 @@ A provider capability description may include: Coordinate uncertainty in one computer tool must not disable coordinate-free browser tools such as snapshots, refs, semantic waits, or action plans. -Tzafon and Yutori adapters compose by selected identity: Tzafon replaces only its native computer placeholder, while Yutori removes only selected native placeholders and preserves unrelated function tools. +Native adapters compose by selected identity: OpenAI replaces only its native computer placeholder, while Google removes only selected native placeholders and preserves unrelated function tools. ## Removal of `computer_use_extra` @@ -542,7 +542,7 @@ anthropic browser_20260701 cannot be used with model openai:gpt-5.6-sol ``` ```text -tools "tzafon_computer" and "browser_click" require conflicting payload transforms +tools "provider..native.computer" and "provider..native.browser" require conflicting payload transforms for "tools.computer_use" ``` ```text @@ -604,7 +604,7 @@ Every provider tool surface must expose the first-party source it mirrors. CUA-a 4. **Batch overlap:** batches are mechanical; `browser_act` remains semantic; browser batches share ref state without a workflow DSL. 5. **Dynamic loading:** `setTools()` uses pi 0.83.0 additive markers only for final, cache-preserving in-tool additions; other changes are eager. 6. **Shared resources:** one resource pool survives tool/model changes and owns the translator and lazy CDP executor. -7. **Provider exports:** the native OpenAI, Anthropic, Google, Tzafon, and Yutori surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use CUA-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. +7. **Provider exports:** the native OpenAI, Anthropic, and Google surfaces are namespaced, cite first-party sources, and are tested against their declared contracts. Meta, xAI, and Moonshot use CUA-authored browser tools; the CLI explicitly appends `browser_act` to the Meta and xAI catalogs. Moonshot is excluded: its API accepts the complex `browser_wait_for` schema but rejects a request carrying `browser_act`'s much larger one, so the catalog gates oversized schemas separately from merely-complex ones. ## Decisions recorded diff --git a/docs/architecture.md b/docs/architecture.md index 81eba63..be9ec2c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -156,8 +156,6 @@ catalog: - OpenAI streams through pi's builtin Responses transport and its automatic prompt caching by default; a CUA-owned adapter handles OpenAI's native computer tool and tool-search namespace round-trips. -- Tzafon replaces only the selected computer identity and fills declaration - dimensions from the actual viewport. - Anthropic's native browser tool falls back to an equivalent function-tool declaration when the active credential cannot access `browser_20260701`; the selected tool identity, name, schema, and executor remain unchanged. @@ -165,8 +163,6 @@ catalog: declaration plus exact exclusions through the CUA-owned Interactions API adapter. Excluded calls fail with a named catalog error instead of reaching generic tool dispatch. -- Yutori emits its native `tool_set`/`disable_tools` fields while preserving - ordinary function tools. - Meta, xAI, and Moonshot disable parallel tool calls when the selected catalog can mutate browser state. @@ -187,10 +183,7 @@ through pi's builtin `openai-responses` transport, but the same model selected with `cua.providers.openai.tools.computer()` compiles to the CUA-owned `openai-cua-computer` api — and symmetrically for Google's `google-cua-interactions` Interactions API versus pi's builtin Google -transport. Tzafon and Yutori declare `requiresApi` too, but their models always -carry that api regardless of tool selection: pi ships no transport for either -provider at all, so `routeCuaApi` (model-shaped, not tool-shaped) forces it -unconditionally. +transport. `CuaAgent` and `CuaAgentHarness` push the compiled `catalog.model` into pi on every construction and on every `setTools()`/`setModel()`, so the derived @@ -212,8 +205,6 @@ serialization, provider fields, then the caller's `onPayload` hook. `browser_act`'s larger schema; - Anthropic's native browser tool when the model supports it; - Google's native browser action set; - - Tzafon's native computer tool configured for a browser; - - Yutori's native N1 or N1.5 browser set plus an explicit screenshot tool; 3. creates and retains its own application-level coding-tool list; 4. passes the complete list to `CuaAgentHarness`; 5. builds a caller-owned prompt from loaded skills and context files; @@ -234,8 +225,7 @@ to it so `ctrl+c` cancels the selector instead of quitting. `filterModelsForPicker`, `moveSelection`, `visibleWindow`) that make its behavior unit-testable without a terminal. - `tui/tool-selection.ts` — pure `/tools` state machine: identity keys matching - `normalizeTool`'s scheme, group badges, atomic provider groups, and - toggle/bulk operations. + `normalizeTool`'s scheme, group badges, and toggle/bulk operations. - `tui/tools-picker.ts` — the `/tools` component. Staged edits applied through `harness.setTools()` with a subset of the application-composed baseline, in baseline order. diff --git a/package-lock.json b/package-lock.json index 0f50d15..1f719a8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -15,7 +15,6 @@ ], "devDependencies": { "@types/node": "22.18.4", - "smol-toml": "^1.7.0", "tsx": "^4.23.1", "typescript": "5.9.3" }, @@ -4420,12 +4419,6 @@ "integrity": "sha512-wWKOClTTiizcZhXnPY4wikVAwmdYHp8q6DmC+EJUzAMsycb7HB32Kh9RN4+0gExjmPmZSAQjgURXIGATPegAvA==", "license": "MIT" }, - "node_modules/@tzafon/lightcone": { - "version": "0.7.2", - "resolved": "https://registry.npmjs.org/@tzafon/lightcone/-/lightcone-0.7.2.tgz", - "integrity": "sha512-jzXTAOeE77FuuzP8J2dtxXxlBnN3Jb1o/iF7taVLGsT4ch8EemztVKjHH6KOJtu+3KvQzSioYoJCIX6pJ35jTA==", - "license": "Apache-2.0" - }, "node_modules/@vitest/expect": { "version": "3.2.7", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz", @@ -5625,19 +5618,6 @@ "dev": true, "license": "ISC" }, - "node_modules/smol-toml": { - "version": "1.7.0", - "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.7.0.tgz", - "integrity": "sha512-aqVvWoyO21L23mb+drl4RmMXbf6N7FdHjAhTRA9ZBL7apWBgfWC16KjrASI+1p9GAroljyMHj6fK67i0UiTNvQ==", - "dev": true, - "license": "BSD-3-Clause", - "engines": { - "node": ">= 18" - }, - "funding": { - "url": "https://github.com/sponsors/cyyynthia" - } - }, "node_modules/source-map-js": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", @@ -6176,7 +6156,6 @@ "license": "MIT", "dependencies": { "@earendil-works/pi-ai": "0.83.0", - "@tzafon/lightcone": "^0.7.0", "openai": "^6.26.0" }, "devDependencies": { diff --git a/package.json b/package.json index d6f57a8..4d67b00 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,6 @@ }, "devDependencies": { "@types/node": "22.18.4", - "smol-toml": "^1.7.0", "tsx": "^4.23.1", "typescript": "5.9.3" } diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 3bb135b..0672ec1 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## 0.13.0 - 2026-08-13 + +Breaking: Tzafon and Yutori support is removed. + +- Update `@onkernel/cua-ai` to 0.13.0. Constructing a `CuaAgent` or + `CuaAgentHarness` with a Tzafon or Yutori model ref now fails to resolve the + model, and `cua.providers.tzafon` / `cua.providers.yutori` no longer exist. +- The tool-result image replay limit now exempts only OpenAI's native computer + tool, whose protocol requires every `computer_call_output` to carry a + screenshot. Tzafon's native computer results were exempt for the same reason + and are gone with the provider. +- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog + viewport option; the same value is still on `resources.browser.viewport`. + ## 0.12.0 - 2026-08-13 - Add `CuaAgentHarness.setModelAndTools()`. A model switch that also swaps diff --git a/packages/agent/README.md b/packages/agent/README.md index 93b9205..7da52af 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -139,15 +139,14 @@ const tools = [ ]; ``` -Other provider groups include OpenAI native computer, Tzafon native computer, -Google's current predefined browser toolset, and Yutori native toolsets. Every -provider surface exposes linked first-party documentation. Meta and xAI use CUA -browser primitives plus `cua.tools.browser.act()` in the provider-matrix -examples. Moonshot uses browser primitives alone because its API rejects -`browser_act`'s larger schema. Compilation rejects incompatible tool/model -combinations before a request. Anthropic's native browser tool -uses an equivalent function-tool transport when the active credential cannot -access `browser_20260701`. +Other provider groups include OpenAI native computer and Google's current +predefined browser toolset. Every provider surface exposes linked first-party +documentation. Meta and xAI use CUA browser primitives plus +`cua.tools.browser.act()` in the provider-matrix examples. Moonshot uses browser +primitives alone because its API rejects `browser_act`'s larger schema. +Compilation rejects incompatible tool/model combinations before a request. +Anthropic's native browser tool uses an equivalent function-tool transport when +the active credential cannot access `browser_20260701`. ## Dynamic catalogs @@ -202,9 +201,9 @@ Tools return only requested feedback: textual markers. `toolResultImageReplayLimit` controls how many recent tool-result images remain -in model context (`4` by default, or `false` to disable projection). Tzafon -native screenshot results are exempt because its continuation protocol requires -the image. +in model context (`4` by default, or `false` to disable projection). OpenAI +native computer results are exempt because its protocol requires each +`computer_call_output` to carry a screenshot. ## Custom tools diff --git a/packages/agent/examples/shared/tools.ts b/packages/agent/examples/shared/tools.ts index 7bf1377..8b1514d 100644 --- a/packages/agent/examples/shared/tools.ts +++ b/packages/agent/examples/shared/tools.ts @@ -39,16 +39,5 @@ export function toolsForModel(model: CuaModelRef): CuaAgentTool[] { // Same as meta/xai, minus browser_act: Kimi's API rejects that tool's // oversized schema, so Kimi gets the browser primitives only. return cua.toolsets.browser(); - case "tzafon": - // Northstar's documented native computer schema is its supported interaction contract. - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - // Match Yutori's documented model generation and add explicit visual access. - return [ - ...(modelId.startsWith("n1.5") - ? cua.providers.yutori.toolsets.n15Core() - : cua.providers.yutori.toolsets.n1()), - cua.tools.computer.screenshot(), - ]; } } diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts index ff86d64..6dd045e 100644 --- a/packages/agent/src/agent.ts +++ b/packages/agent/src/agent.ts @@ -92,7 +92,7 @@ export type CuaAgentOptions = Omit & streamFn?: AgentOptions["streamFn"]; emptyResponseRecovery?: CuaEmptyResponseRecoveryOptions; toolResultImageReplayLimit?: ToolResultImageReplayLimit; - /** Governs Google and Tzafon's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ + /** Governs Google's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ responseThreading?: boolean; retry?: CuaRetryOptions; }; @@ -113,7 +113,7 @@ type CuaAgentHarnessOptionsBase< onPayload?: SimpleStreamOptions["onPayload"]; emptyResponseRecovery?: CuaEmptyResponseRecoveryOptions; toolResultImageReplayLimit?: ToolResultImageReplayLimit; - /** Governs Google and Tzafon's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ + /** Governs Google's `previous_response_id`-style continuation. Every other provider streams through pi's transports and their automatic prompt caching regardless of this flag. Defaults to `true`. */ responseThreading?: boolean; retry?: CuaRetryOptions; }; @@ -561,7 +561,7 @@ function resolveToolResultImageReplayLimit(limit: ToolResultImageReplayLimit | u /** Native computer tool names whose screenshot history the provider protocol requires in full, regardless of the image replay limit. */ function requiredImageToolNames(incoming: CuaIncomingToolPlan): ReadonlySet { - return new Set([incoming.tzafonComputerName, incoming.openaiComputerName].filter((name): name is string => !!name)); + return new Set(incoming.openaiComputerName ? [incoming.openaiComputerName] : []); } function projectToolResultImages( diff --git a/packages/agent/src/resources.ts b/packages/agent/src/resources.ts index e6ce891..390a2bb 100644 --- a/packages/agent/src/resources.ts +++ b/packages/agent/src/resources.ts @@ -34,7 +34,6 @@ type ToolContent = Array; export class CuaExecutionResources { readonly browser: KernelBrowser; readonly client: Kernel; - readonly viewport: { readonly width: number; readonly height: number }; private readonly translator: InternalComputerTranslator; /** Each spec is materialized exactly once per resource pool. */ private readonly materialized = new WeakMap(); @@ -47,7 +46,6 @@ export class CuaExecutionResources { }) { this.browser = options.browser; this.client = options.client; - this.viewport = options.browser.viewport ?? { width: 1920, height: 1080 }; this.translator = new InternalComputerTranslator(options); } diff --git a/packages/agent/src/tool-manager.ts b/packages/agent/src/tool-manager.ts index aa216cb..ec81e03 100644 --- a/packages/agent/src/tool-manager.ts +++ b/packages/agent/src/tool-manager.ts @@ -167,7 +167,6 @@ export class CuaToolManager = CuaAgentToo const catalog = compileCuaToolCatalog({ model: typeof model === "string" ? this.resolveModel(model) : model, requestedTools: inputs, - viewport: this.resources.viewport, }); const fingerprints: string[] = []; diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index 0a34fa6..ed459a4 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -105,34 +105,6 @@ describe("CuaAgent explicit tools", () => { expect("getMode" in agent).toBe(false); }); - it("retains protocol-required Tzafon screenshot results outside the image replay limit", async () => { - const contexts: Context[] = []; - const model = getCuaModel("tzafon:tzafon.northstar-cua-fast"); - const nativeComputer = cua.providers.tzafon.tools.computer(); - const ordinaryTool = callerTool("ordinary"); - const image = { type: "image" as const, data: "c2NyZWVuc2hvdA==", mimeType: "image/png" }; - const messages: AgentMessage[] = [ - assistant(model, [{ type: "toolCall", id: "native-shot", name: "computer", arguments: { action: { type: "screenshot" } } }], "toolUse"), - { role: "toolResult", toolCallId: "native-shot", toolName: "computer", content: [image], isError: false, timestamp: 1 }, - assistant(model, [{ type: "toolCall", id: "ordinary-shot", name: "ordinary", arguments: {} }], "toolUse"), - { role: "toolResult", toolCallId: "ordinary-shot", toolName: "ordinary", content: [image], isError: false, timestamp: 2 }, - ]; - const agent = new CuaAgent({ - browser, - client, - tools: [nativeComputer, ordinaryTool], - streamFn: scriptedStream([(selectedModel) => assistant(selectedModel)], contexts), - toolResultImageReplayLimit: 0, - initialState: { model, messages }, - }); - - await agent.prompt("continue"); - - const results = contexts[0]!.messages.filter((message) => message.role === "toolResult"); - expect(results[0]!.content).toEqual([image]); - expect(results[1]!.content).toEqual([{ type: "text", text: "[stale tool-result images omitted]" }]); - }); - it("retains protocol-required OpenAI native computer screenshot results outside the image replay limit", async () => { const contexts: Context[] = []; const model = getCuaModel("openai:gpt-5.5"); diff --git a/packages/agent/test/e2e.live.test.ts b/packages/agent/test/e2e.live.test.ts index a9d347e..f305429 100644 --- a/packages/agent/test/e2e.live.test.ts +++ b/packages/agent/test/e2e.live.test.ts @@ -14,7 +14,7 @@ const LIVE = process.env.CUA_E2E_LIVE === "1"; const KERNEL_API_KEY = process.env.KERNEL_API_KEY; type ProviderCase = { - name: "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshotai" | "tzafon" | "yutori"; + name: "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshotai"; apiKeyEnvVar: string; modelRef: | "openai:gpt-5.6-sol" @@ -22,12 +22,9 @@ type ProviderCase = { | "google:gemini-3.6-flash" | "meta:muse-spark-1.1" | "xai:grok-4.5" - | "moonshotai:kimi-k3" - | "tzafon:tzafon.northstar-cua-fast" - | "yutori:n1.5-latest"; + | "moonshotai:kimi-k3"; prompt: string; expectToolCalls: boolean; - expectReadArtifact?: boolean; timeoutMs: number; ciOptInEnvVar?: string; }; @@ -119,32 +116,6 @@ const cases: ProviderCase[] = [ expectToolCalls: true, timeoutMs: 180_000, }, - { - name: "tzafon", - apiKeyEnvVar: "TZAFON_API_KEY", - modelRef: "tzafon:tzafon.northstar-cua-fast", - prompt: [ - "Use the native `computer` tool's `screenshot` action exactly once.", - "Do not call any other tools.", - "After the tool result, provide a one-sentence summary.", - ].join("\n"), - expectToolCalls: true, - timeoutMs: 120_000, - ciOptInEnvVar: "CUA_E2E_TZAFON", - }, - { - name: "yutori", - apiKeyEnvVar: "YUTORI_API_KEY", - modelRef: "yutori:n1.5-latest", - prompt: [ - "Use the function tool named `computer_screenshot` exactly once.", - "Pass empty arguments (`{}`).", - "Do not call any other tools.", - ].join("\n"), - expectToolCalls: true, - expectReadArtifact: false, - timeoutMs: 180_000, - }, ]; const switchCases: ModelSwitchCase[] = [ @@ -197,10 +168,6 @@ function toolsForCase(c: ProviderCase) { return [cua.providers.anthropic.tools.browser({ version: "20260701", javascript: true })]; case "gemini": return cua.providers.google.toolsets.browser(); - case "tzafon": - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - return [...cua.providers.yutori.toolsets.n15Core(), cua.tools.computer.screenshot()]; } } @@ -253,7 +220,7 @@ function assertStats(stats: RunStats, c: ProviderCase, runtimeName: "agent" | "h if (c.expectToolCalls) { expect(stats.toolCalls).toBeGreaterThan(0); expect(stats.toolResults).toBeGreaterThan(0); - if (c.expectReadArtifact !== false) expect(stats.hasReadArtifact).toBe(true); + expect(stats.hasReadArtifact).toBe(true); } expect(stats.finalAssistant).toBeDefined(); if (stats.finalAssistant?.role === "assistant") { diff --git a/packages/agent/test/example-provider-matrix.test.ts b/packages/agent/test/example-provider-matrix.test.ts index 1fc91b6..7add76a 100644 --- a/packages/agent/test/example-provider-matrix.test.ts +++ b/packages/agent/test/example-provider-matrix.test.ts @@ -5,8 +5,6 @@ import { import { describe, expect, it } from "vitest"; import { toolsForModel } from "../examples/shared/tools"; -const viewport = { width: 1440, height: 900 }; - /** * The example matrices are plain scripts: they are excluded from `tsc -b` and are * never executed in CI, so a provider policy that no longer compiles used to be @@ -24,15 +22,13 @@ const models: readonly CuaModelRef[] = [ "xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3", - "tzafon:tzafon.northstar-cua-fast", - "yutori:n1.5-latest", ]; describe("example provider matrix tool policy", () => { it("compiles a valid catalog for every model the matrices advertise", () => { for (const model of models) { expect( - () => compileCuaToolCatalog({ model, requestedTools: toolsForModel(model), viewport }), + () => compileCuaToolCatalog({ model, requestedTools: toolsForModel(model) }), model, ).not.toThrow(); } diff --git a/packages/agent/test/resources.test.ts b/packages/agent/test/resources.test.ts index 0ec84b4..f8f36c3 100644 --- a/packages/agent/test/resources.test.ts +++ b/packages/agent/test/resources.test.ts @@ -203,10 +203,10 @@ describe("CuaExecutionResources results and batch boundaries", () => { expect(result.details).not.toHaveProperty("isError"); }); - it("returns status text for Yutori writes without capturing a screenshot", async () => { + it("returns status text for provider-native writes without capturing a screenshot", async () => { const { resources, captureScreenshot } = setup(); - const spec = cua.providers.yutori.toolsets.n15Core().find((tool) => tool.name === "left_click")!; - const result = await resources.materialize(spec).execute("click", { coordinates: [100, 200] }); + const spec = cua.providers.google.toolsets.browser().find((tool) => tool.name === "click")!; + const result = await resources.materialize(spec).execute("click", { x: 100, y: 200 }); expect(result.content).toEqual([{ type: "text", text: "Actions executed successfully." }]); expect(captureScreenshot).not.toHaveBeenCalled(); }); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 939ccfe..7b50ef0 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## 0.13.0 - 2026-08-13 + +Breaking: Tzafon and Yutori support is removed. + +- Remove the `tzafon` and `yutori` providers: their `CuaProvider` members, + model annotations and overrides, `cua.providers.tzafon`, + `cua.providers.yutori`, `TZAFON_API_KEY`/`YUTORI_API_KEY`, and the exported + `TZAFON_RESPONSES_API`, `YUTORI_CHAT_COMPLETIONS_API`, + `streamTzafonResponses`, `streamSimpleTzafonResponses`, `streamYutori`, and + `streamSimpleYutori` stream functions. `createCuaModels()` no longer + registers either provider, and refs like `tzafon:tzafon.northstar-cua-fast` + or `yutori:n1.5-latest` now fail to resolve. +- `CuaProviderBinding` loses its `tzafon-native` and `yutori-native` variants, + and `CuaIncomingToolPlan` loses `tzafonComputerName` and `yutoriNames`. The + Yutori-only rule rejecting a partial n1 native action set is gone with them; + every surviving native toolset can be selected in part. +- `CompileCuaToolCatalogOptions.viewport` is removed. It existed only to fill + Tzafon's `display_width`/`display_height` declaration defaults, and no + surviving declaration reads it. +- `routeCuaApi` no longer routes any transport. Every remaining provider gets + its transport either from pi-ai's registry or from the selected tools' + `requiresApi`; what is left is grok-4.5's cost/compat/thinking-level + overrides. +- Drop the `@tzafon/lightcone` dependency. + ## 0.12.0 - 2026-08-13 - Route OpenAI's native computer adapter through its own stream function, diff --git a/packages/ai/README.md b/packages/ai/README.md index 76892cf..3f615cb 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -157,10 +157,6 @@ cua.providers.google.toolsets.browser({ exclude: ["right_click"] }); // Meta, xAI, and Moonshot use the ordinary CUA browser tools. cua.toolsets.browser(); - -cua.providers.tzafon.tools.computer(); -cua.providers.yutori.toolsets.n1(); -cua.providers.yutori.toolsets.n15Core(); ``` The Google browser set exposes the current predefined action names and uses @@ -261,15 +257,6 @@ require different transports fails to compile. declaration plus explicit exclusions through the Interactions API adapter. - **Meta/xAI/Moonshot**: ordinary function tools with serial tool calls when the selected catalog mutates browser state. -- **Tzafon**: identity-scoped native declaration replacement with actual viewport - dimensions. Explicit screenshot and terminal answer actions are supported; - non-screenshot native action loops fail before browser execution because - Tzafon's continuation protocol requires implicit post-action screenshots. Pi - ships no Tzafon transport, so every Tzafon model carries CUA's own api - regardless of tool selection. -- **Yutori**: identity-scoped native `tool_set`/`disable_tools` fields while - preserving ordinary function tools such as an explicitly selected screenshot. - Pi ships no Yutori transport either, so the same unconditional api applies. ## API keys @@ -282,8 +269,8 @@ import { ``` Conventional variables are `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, -`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, -`MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`. +`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, and +`MOONSHOT_API_KEY`. ## Development diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 612e3c1..5b5f20e 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -136,26 +136,3 @@ CUA browser function tools. OpenRouter does not expose the provider-native computer tools declared by other CUA providers. Source: [OpenRouter model page](https://openrouter.ai/moonshotai/kimi-k3). - -## `tzafon` - -Coordinates: normalized 0–999 - -Exact IDs: - -- `tzafon.northstar-cua-fast` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) -- `tzafon.northstar-cua-fast-1.6` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) -- `tzafon.northstar-cua-fast-1.7-experiment` ([model card](https://huggingface.co/Tzafon/Northstar-CUA-Fast)) - -## `yutori` - -Coordinates: normalized 0–1000 - -Exact IDs: - -- `n1-latest` -- `n1-20260203` -- `n1.5-latest` -- `n1.5-20260428` - -Source: [Yutori Navigator reference](https://docs.yutori.com/reference/navigator). diff --git a/packages/ai/examples/quickstart.ts b/packages/ai/examples/quickstart.ts index b53dd58..2c991aa 100644 --- a/packages/ai/examples/quickstart.ts +++ b/packages/ai/examples/quickstart.ts @@ -19,7 +19,6 @@ const screenshot = await readFile(new URL("./screenshot.png", import.meta.url)); const catalog = compileCuaToolCatalog({ model: modelRef, requestedTools: [cua.tools.computer.click()], - viewport: { width: 1440, height: 900 }, }); const response = await cuaModels().complete( diff --git a/packages/ai/package.json b/packages/ai/package.json index 31d39c1..c43e91b 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -45,7 +45,6 @@ }, "dependencies": { "@earendil-works/pi-ai": "0.83.0", - "@tzafon/lightcone": "^0.7.0", "openai": "^6.26.0" }, "devDependencies": { diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts index 58bef14..241ea57 100644 --- a/packages/ai/src/api-keys.ts +++ b/packages/ai/src/api-keys.ts @@ -16,8 +16,6 @@ const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { xai: ["XAI_API_KEY"], moonshotai: ["MOONSHOT_API_KEY"], openrouter: ["OPENROUTER_API_KEY"], - tzafon: ["TZAFON_API_KEY"], - yutori: ["YUTORI_API_KEY"], }; /** diff --git a/packages/ai/src/cua.ts b/packages/ai/src/cua.ts index 10293c7..091a57e 100644 --- a/packages/ai/src/cua.ts +++ b/packages/ai/src/cua.ts @@ -10,14 +10,6 @@ import { supportsAnthropicNativeBrowser } from "./providers/anthropic/capabiliti import { mapNativeBrowserInput, mapNativeComputerInput } from "./providers/anthropic/native"; import { GOOGLE_CUA_INTERACTIONS_API } from "./providers/google/provider"; import { OPENAI_CUA_COMPUTER_API } from "./providers/openai/provider"; -import { toCanonicalActions as toTzafonActions, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; -import { - toCanonicalActions as toYutoriActions, - YUTORI_N1_ACTION_TYPES, - YUTORI_N15_CORE_ACTION_TYPES, - YUTORI_N15_CORE_TOOL_SET, -} from "./providers/yutori/actions"; -import { YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; import { CUA_TOOL_SPEC_KIND, type CuaCoordinateContract, @@ -81,9 +73,6 @@ const providerSources = Object.freeze({ openai: "https://developers.openai.com/api/docs/guides/tools-computer-use", anthropic: "https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool", google: "https://ai.google.dev/gemini-api/docs/computer-use", - tzafon: "https://huggingface.co/Tzafon/Northstar-CUA-Fast", - yutoriN1: "https://docs.yutori.com/reference/n1", - yutoriN15: "https://docs.yutori.com/reference/n1-5", }); function normalized(range: readonly [number, number]): CuaCoordinateContract { @@ -444,53 +433,6 @@ function openaiNativeComputer(): CuaToolSpec { }); } -function tzafonNativeComputer(options: { displayWidth?: number; displayHeight?: number } = {}): CuaToolSpec { - const declaration = { - type: "computer_use", - display_width: options.displayWidth, - display_height: options.displayHeight, - environment: "browser", - }; - return providerNativeSpec({ - identity: "provider.tzafon.native.computer.v1", - name: "computer", - source: providerSources.tzafon, - declaration, - binding: { kind: "tzafon-native", declaration, requiresApi: TZAFON_RESPONSES_API }, - toActions(input) { - const action = asInput(input).action; - return toTzafonActions(action).filter((value): value is CuaAction => value.type !== "answer"); - }, - coordinates: normalized([0, 999]), - }); -} - -function yutoriToolset(generation: "n1" | "n15"): CuaToolSpec[] { - const names = generation === "n1" ? YUTORI_N1_ACTION_TYPES : YUTORI_N15_CORE_ACTION_TYPES; - return names.map((nativeName) => { - const identityName = nativeName.replaceAll("_", "-"); - const binding: CuaProviderBinding = { - kind: "yutori-native", - generation, - nativeName, - ...(generation === "n15" ? { toolSet: YUTORI_N15_CORE_TOOL_SET } : {}), - allNativeNames: names, - requiresApi: YUTORI_CHAT_COMPLETIONS_API, - }; - return providerNativeSpec({ - identity: `provider.yutori.native.${generation}.${identityName}.${generation === "n15" ? "20260403" : "v1"}`, - name: nativeName, - source: generation === "n1" ? providerSources.yutoriN1 : providerSources.yutoriN15, - declaration: { type: "function", name: nativeName }, - binding, - toActions(input) { - return toYutoriActions(nativeName, asInput(input)) ?? []; - }, - coordinates: normalized([0, 1000]), - }); - }); -} - const GOOGLE_BROWSER_ACTIONS = [ "click", "double_click", "triple_click", "middle_click", "right_click", "mouse_down", "mouse_up", "move", "type", "drag_and_drop", "wait", "press_key", "key_down", "key_up", "hotkey", "take_screenshot", @@ -904,11 +846,6 @@ const providers = Object.freeze({ source: providerSources.google, toolsets: Object.freeze({ browser: googleBrowserToolset }), }), - tzafon: Object.freeze({ source: providerSources.tzafon, tools: Object.freeze({ computer: tzafonNativeComputer }) }), - yutori: Object.freeze({ - sources: Object.freeze({ n1: providerSources.yutoriN1, n15Core: providerSources.yutoriN15 }), - toolsets: Object.freeze({ n1: () => yutoriToolset("n1"), n15Core: () => yutoriToolset("n15") }), - }), }); /** Frozen, discoverable tool namespace shared by @onkernel/cua-ai and @onkernel/cua-agent. */ diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index b4e90a0..1dd48d6 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -9,12 +9,6 @@ export { streamOpenAIResponses, streamSimpleGoogleInteractions, streamSimpleOpenAIResponses, - streamSimpleTzafonResponses, - streamSimpleYutori, - streamTzafonResponses, - streamYutori, - TZAFON_RESPONSES_API, - YUTORI_CHAT_COMPLETIONS_API, } from "./providers"; export * from "./models"; export * from "./api-keys"; diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index b50b5c3..7b44cd2 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -1,10 +1,8 @@ import type { Api, Model } from "@earendil-works/pi-ai"; import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; -import { TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; -import { YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; /** Providers with curated computer-use model support. */ -export type CuaProvider = "openai" | "anthropic" | "google" | "meta" | "xai" | "moonshotai" | "openrouter" | "tzafon" | "yutori"; +export type CuaProvider = "openai" | "anthropic" | "google" | "meta" | "xai" | "moonshotai" | "openrouter"; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ export type CuaModelRef = `${CuaProvider}:${string}`; @@ -21,7 +19,7 @@ export interface CuaModelInfo { } /** All providers this package curates computer-use models for. */ -export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter", "tzafon", "yutori"]; +export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter"]; /** * How a {@link CuaModelAnnotation} matches model ids. @@ -106,17 +104,6 @@ export const CUA_MODEL_ANNOTATIONS: Record[]> = { xai: [], moonshotai: [], openrouter: [], - tzafon: [ - cuaModel("tzafon", "tzafon.northstar-cua-fast", "Tzafon Northstar CUA Fast"), - cuaModel("tzafon", "tzafon.northstar-cua-fast-1.6", "Tzafon Northstar CUA Fast 1.6"), - cuaModel("tzafon", "tzafon.northstar-cua-fast-1.7-experiment", "Tzafon Northstar CUA Fast 1.7 (experiment)"), - ], - yutori: [ - cuaModel("yutori", "n1.5-latest", "Yutori Navigator n1.5"), - cuaModel("yutori", "n1.5-20260428", "Yutori Navigator n1.5 (2026-04-28)"), - cuaModel("yutori", "n1-latest", "Yutori Navigator n1"), - cuaModel("yutori", "n1-20260203", "Yutori Navigator n1 (2026-02-03)"), - ], }; /** Models CUA supports that pi-ai's registry does not carry for a provider. */ @@ -227,22 +203,14 @@ export function getCuaModel(ref: CuaModelRef): Model { throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`); } -// Route CUA models to provider-specific transports that are properties of the -// model itself, not of which tools a caller selects. Tool-driven transport -// selection (OpenAI's native computer tool, Google's Interactions API) is -// derived by compileCuaToolCatalog from the selected tools' provider bindings -// instead; see CuaProviderBinding.requiresApi. What remains here is model-only: -// Tzafon and Yutori get no transport from pi-ai at all, so every model on -// those providers always carries CUA's own api id regardless of tool -// selection, and grok-4.5 carries cost/compat/thinking-level overrides pi-ai's -// registry does not have yet. +// Apply the model overrides that are properties of the model itself, not of +// which tools a caller selects. Tool-driven transport selection (OpenAI's +// native computer tool, Google's Interactions API) is derived by +// compileCuaToolCatalog from the selected tools' provider bindings instead; +// see CuaProviderBinding.requiresApi. What remains here is model-only: +// grok-4.5 carries cost/compat/thinking-level overrides pi-ai's registry does +// not have yet. export function routeCuaApi(model: Model): Model { - if (model.provider === "tzafon" && model.api !== TZAFON_RESPONSES_API) { - return { ...model, api: TZAFON_RESPONSES_API }; - } - if (model.provider === "yutori" && model.api !== YUTORI_CHAT_COMPLETIONS_API) { - return { ...model, api: YUTORI_CHAT_COMPLETIONS_API }; - } if (model.provider === "xai" && model.id === "grok-4.5") { return { ...model, @@ -313,35 +281,23 @@ function isCuaFamilyMatch(id: string, family: string): boolean { .every((segment) => /^\d+$/.test(segment)); } -function cuaModel(provider: "meta" | "tzafon" | "yutori", id: string, name: string): Model { - const base = { +// Meta documents the 1,048,576-token context window, and its computer-use +// cookbook configures 128,000 maximum output tokens. +function cuaModel(provider: "meta", id: string, name: string): Model { + return { id, name, provider, - reasoning: provider === "meta", + reasoning: true, input: ["text", "image"], - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, - } satisfies Partial>; - - switch (provider) { - case "meta": - // Meta documents the 1,048,576-token context window, and its - // computer-use cookbook configures 128,000 maximum output tokens. - return { - ...base, - api: "openai-responses", - baseUrl: "https://api.meta.ai/v1", - thinkingLevelMap: { off: null, xhigh: "xhigh" }, - cost: { input: 1.25, output: 4.25, cacheRead: 0.15, cacheWrite: 0 }, - contextWindow: 1_048_576, - maxTokens: 128_000, - compat: { supportsDeveloperRole: true, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: true }, - } as Model; - case "tzafon": - return { ...base, api: "tzafon-responses", baseUrl: "https://api.tzafon.ai", contextWindow: 128_000, maxTokens: 4_096 } as Model; - case "yutori": - return { ...base, api: "yutori-chat-completions", baseUrl: "https://api.yutori.com/v1", contextWindow: 128_000, maxTokens: 4_096 } as Model; - } + api: "openai-responses", + baseUrl: "https://api.meta.ai/v1", + thinkingLevelMap: { off: null, xhigh: "xhigh" }, + cost: { input: 1.25, output: 4.25, cacheRead: 0.15, cacheWrite: 0 }, + contextWindow: 1_048_576, + maxTokens: 128_000, + compat: { supportsDeveloperRole: true, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: true }, + } as Model; } function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index d87eba5..4deaf72 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -20,8 +20,6 @@ import { cuaOverrideModels } from "./models"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAICuaComputer, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; -import { streamSimpleTzafonResponses, streamTzafonResponses, TZAFON_RESPONSES_API } from "./providers/tzafon/provider"; -import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from "./providers/yutori/provider"; /** * Build the pi `Models` collection CUA streams through: pi's builtin @@ -44,9 +42,9 @@ import { streamSimpleYutori, streamYutori, YUTORI_CHAT_COMPLETIONS_API } from ". * Responses transport, and the catalog supplies its serial-tool-call field. * - `moonshotai` is pi's builtin provider untouched: Kimi streams through the * plain OpenAI-compatible chat completions transport with `MOONSHOT_API_KEY`. - * - `meta`, `tzafon`, and `yutori` are CUA-only providers pi does not ship. - * `meta` speaks the OpenAI Responses wire protocol, so it registers pi's - * builtin transport against Meta's base URL and credentials. + * - `meta` is a CUA-only provider pi does not ship. It speaks the OpenAI + * Responses wire protocol, so it registers pi's builtin transport against + * Meta's base URL and credentials. * * Each call returns an independent collection; register additional providers * or credentials on it freely. Use {@link cuaModels} for the shared default. @@ -60,8 +58,6 @@ export function createCuaModels(options?: CreateModelsOptions): MutableModels { const google = models.getProvider("google"); if (google) models.setProvider(withGoogleCuaInteractions(google)); models.setProvider(metaProvider()); - models.setProvider(tzafonProvider()); - models.setProvider(yutoriProvider()); return models; } @@ -128,29 +124,5 @@ function metaProvider(): Provider { }); } -function tzafonProvider(): Provider { - return createProvider({ - id: "tzafon", - name: "Tzafon", - baseUrl: "https://api.tzafon.ai", - auth: { apiKey: envApiKeyAuth("Tzafon API key", cuaApiKeyEnvVarsForProvider("tzafon")) }, - models: cuaOverrideModels("tzafon"), - api: { [TZAFON_RESPONSES_API]: { stream: streamTzafonResponses, streamSimple: streamSimpleTzafonResponses } }, - }); -} - -function yutoriProvider(): Provider { - return createProvider({ - id: "yutori", - name: "Yutori", - baseUrl: "https://api.yutori.com/v1", - auth: { apiKey: envApiKeyAuth("Yutori API key", cuaApiKeyEnvVarsForProvider("yutori")) }, - models: cuaOverrideModels("yutori"), - api: { [YUTORI_CHAT_COMPLETIONS_API]: { stream: streamYutori, streamSimple: streamSimpleYutori } }, - }); -} - export { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; export { OPENAI_CUA_COMPUTER_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; -export { TZAFON_RESPONSES_API, streamSimpleTzafonResponses, streamTzafonResponses }; -export { YUTORI_CHAT_COMPLETIONS_API, streamSimpleYutori, streamYutori }; diff --git a/packages/ai/src/providers/common.ts b/packages/ai/src/providers/common.ts index 4dc8b7f..4facab4 100644 --- a/packages/ai/src/providers/common.ts +++ b/packages/ai/src/providers/common.ts @@ -7,20 +7,8 @@ import type { SimpleStreamOptions, StreamOptions, } from "@earendil-works/pi-ai"; -import type { CuaAction } from "../actions/index"; import type { CuaIncomingToolPlan } from "../tool-catalog"; -/** Return the canonical function-tool name for an action. */ -export function canonicalToolCallName(action: CuaAction): CuaAction["type"] { - return action.type; -} - -/** Convert a canonical action to function-tool arguments. */ -export function canonicalToolCallArguments(action: CuaAction): Record { - const { type: _type, ...args } = action as CuaAction & Record; - return args; -} - /** Prefix a bare hostname/path before browser navigation. */ export function normalizeGotoUrl(value: unknown): string | undefined { if (typeof value !== "string") return undefined; diff --git a/packages/ai/src/providers/tzafon/provider.ts b/packages/ai/src/providers/tzafon/provider.ts deleted file mode 100644 index 823c70e..0000000 --- a/packages/ai/src/providers/tzafon/provider.ts +++ /dev/null @@ -1,487 +0,0 @@ -import { - createAssistantMessageEventStream, - type Api, - type AssistantMessage, - type Context, - type ImageContent, - type Message, - type Model, - type StreamFunction, - type StreamOptions, - type TextContent, - type Tool, - type ToolCall, -} from "@earendil-works/pi-ai"; -import Lightcone from "@tzafon/lightcone"; -import type { CuaAction } from "../../actions/index"; -import { - canonicalToolCallArguments, - canonicalToolCallName, - type CuaSimpleStreamOptions, - responseThreadingDelta, - responseThreadingEnabled, - type ResponseThreadingOptions, -} from "../common"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; - -export const TZAFON_RESPONSES_API = "tzafon-responses"; - -/** Stream options accepted by {@link streamTzafonResponses}. */ -export interface TzafonResponsesOptions extends StreamOptions, ResponseThreadingOptions { - /** @internal Identity-addressed native dispatch compiled from selected tools. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -/** Inputs {@link buildTzafonRequestInput} reads to shape the Responses API request body. */ -export interface TzafonRequestOptions extends ResponseThreadingOptions { - temperature?: number; - maxTokens?: number; - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -/** Responses API request body for {@link Lightcone.responses.create}, including optional threading fields. */ -export interface TzafonRequestBody { - model: string; - input: Array>; - tools: Array>; - instructions?: string; - temperature: number; - max_output_tokens?: number; - previous_response_id?: string; - store?: boolean; -} - -/** - * Build the Tzafon Responses API request body from a context. - * - * Pure and network-free. When response threading is enabled and a prior - * assistant `responseId` exists, the body chains via `previous_response_id` - * with `store: true` and sends only the delta messages; otherwise it replays - * the full message history. - */ -export function buildTzafonRequestInput(model: Model, context: Context, options?: TzafonRequestOptions): TzafonRequestBody { - const body: TzafonRequestBody = { - model: model.id, - input: convertMessages(context.messages, options?.cuaIncomingToolPlan?.tzafonComputerName), - tools: convertTools(context.tools ?? []), - instructions: context.systemPrompt, - temperature: options?.temperature ?? 0, - max_output_tokens: options?.maxTokens ?? model.maxTokens, - }; - if (!responseThreadingEnabled(options)) return body; - const { previousResponseId, deltaMessages } = responseThreadingDelta(context.messages, TZAFON_RESPONSES_API); - if (!previousResponseId) return body; - return { ...body, input: convertMessages(deltaMessages, options?.cuaIncomingToolPlan?.tzafonComputerName), previous_response_id: previousResponseId, store: true }; -} - -export const streamSimpleTzafonResponses: StreamFunction = (model, context, options) => { - return streamTzafonResponses(model, context, options); -}; - -export const streamTzafonResponses: StreamFunction = (model, context, options) => { - const stream = createAssistantMessageEventStream(); - const output = initialAssistantMessage(model); - - void (async () => { - try { - const apiKey = options?.apiKey || process.env.TZAFON_API_KEY; - if (!apiKey) throw new Error(`No API key for provider: ${model.provider}`); - const client = new Lightcone({ - apiKey, - baseURL: model.baseUrl, - defaultHeaders: { ...model.headers, ...options?.headers }, - }); - const payload = buildTzafonRequestInput(model as Model, context, options); - const nextPayload = await options?.onPayload?.(payload, model as Model); - if (options?.signal?.aborted) throw new Error("Request was aborted"); - const response = await client.responses.create((nextPayload ?? payload) as never, { - signal: options?.signal, - }); - if (options?.signal?.aborted) throw new Error("Request was aborted"); - - stream.push({ type: "start", partial: output }); - output.responseId = getString(response, "id") || undefined; - output.usage = usageFromTzafon(getValue(response, "usage")); - for (const item of getArray(response, "output")) { - const type = getString(item, "type"); - if (type === "message") { - const text = extractMessageText(item); - if (text) emitText(stream, output, text); - continue; - } - if (type === "function_call") { - emitToolCall(stream, output, { - type: "toolCall", - id: getString(item, "call_id"), - name: getString(item, "name"), - arguments: parseArguments(getValue(item, "arguments")), - }); - continue; - } - if (type === "computer_call") { - const callId = getString(item, "call_id") || getString(item, "id") || `computer_call_${output.content.length}`; - const rawAction = getValue(item, "action"); - const canonical = toCanonicalActions(rawAction); - for (const action of canonical) if (action.type === "answer") emitText(stream, output, action.text); - const executable = canonical.filter((action): action is CuaAction => action.type !== "answer"); - const nativeName = options?.cuaIncomingToolPlan?.tzafonComputerName; - const textOnlyAction = executable.find((action) => action.type !== "screenshot"); - if (nativeName && textOnlyAction) { - throw new Error( - `Tzafon native computer action "${textOnlyAction.type}" is unsupported: its action loop requires an automatic post-action screenshot, and CUA returns screenshots only when explicitly requested.`, - ); - } - if (nativeName && executable.length > 0) { - emitToolCall(stream, output, { type: "toolCall", id: callId, name: nativeName, arguments: { action: rawAction } }); - continue; - } - for (let actionIndex = 0; actionIndex < executable.length; actionIndex += 1) { - const action = executable[actionIndex]!; - emitToolCall(stream, output, { - type: "toolCall", - id: tzafonToolCallId(callId, actionIndex), - name: canonicalToolCallName(action), - arguments: canonicalToolCallArguments(action), - }); - } - } - } - - output.stopReason = output.content.some((part) => part.type === "toolCall") ? "toolUse" : "stop"; - stream.push({ type: "done", reason: output.stopReason, message: output }); - stream.end(); - } catch (err) { - output.stopReason = options?.signal?.aborted ? "aborted" : "error"; - output.errorMessage = err instanceof Error ? err.message : String(err); - stream.push({ type: "error", reason: output.stopReason, error: output }); - stream.end(); - } - })(); - - return stream; -}; - -/** Derive a unique canonical tool-call id for a Tzafon computer action. */ -export function tzafonToolCallId(callId: string, actionIndex: number): string { - return actionIndex === 0 ? callId : `${callId}:${actionIndex}`; -} - -function initialAssistantMessage(model: Model): AssistantMessage { - return { - role: "assistant", - content: [], - api: model.api, - provider: model.provider, - model: model.id, - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: Date.now(), - }; -} - -function emitText(stream: ReturnType, output: AssistantMessage, text: string): void { - const contentIndex = output.content.length; - const content: TextContent = { type: "text", text }; - output.content.push(content); - stream.push({ type: "text_start", contentIndex, partial: output }); - stream.push({ type: "text_delta", contentIndex, delta: text, partial: output }); - stream.push({ type: "text_end", contentIndex, content: text, partial: output }); -} - -function emitToolCall( - stream: ReturnType, - output: AssistantMessage, - toolCall: ToolCall, -): void { - const contentIndex = output.content.length; - output.content.push(toolCall); - stream.push({ type: "toolcall_start", contentIndex, partial: output }); - stream.push({ type: "toolcall_end", contentIndex, toolCall, partial: output }); -} - -/** A canonical CUA action, or the terminal `answer` text Tzafon emits when it is done. */ -export type TzafonCanonicalAction = CuaAction | { type: "answer"; text: string }; - -/** Normalize one Tzafon `computer_call.action` payload into canonical CUA actions. */ -export function toCanonicalActions(action: unknown): TzafonCanonicalAction[] { - if (!action || typeof action !== "object") return []; - const current = action as Record; - const type = getString(current, "type"); - const x = readOptionalNumber(current, "x"); - const y = readOptionalNumber(current, "y"); - switch (type) { - case "click": - case "left_click": - return x !== undefined && y !== undefined ? [{ type: "click", x, y }] : []; - case "right_click": - return x !== undefined && y !== undefined ? [{ type: "click", x, y, button: "right" }] : []; - case "double_click": - return x !== undefined && y !== undefined ? [{ type: "double_click", x, y }] : []; - case "triple_click": - return x !== undefined && y !== undefined ? [{ type: "double_click", x, y }, { type: "click", x, y }] : []; - case "move": - case "hover": - return x !== undefined && y !== undefined ? [{ type: "move", x, y }] : []; - case "drag": - return toDragAction(current); - case "type": - return [{ type: "type", text: getString(current, "text") }]; - case "keypress": - case "key": - return toKeypressAction(current); - case "scroll": - return [toScrollAction(current)]; - case "hscroll": - return [{ type: "scroll", scroll_x: readOptionalNumber(current, "scroll_x") ?? readOptionalNumber(current, "amount") ?? 0 }]; - case "navigate": - return [{ type: "goto", url: getString(current, "url") }]; - case "wait": - return [{ type: "wait", ms: readOptionalNumber(current, "ms") ?? secondsToMs(readOptionalNumber(current, "seconds")) }]; - case "screenshot": - return [{ type: "screenshot" }]; - case "answer": - case "done": - case "terminate": - return [{ type: "answer", text: getString(current, "result") || getString(current, "text") || getString(current, "status") }]; - default: - return []; - } -} - -function toDragAction(action: Record): CuaAction[] { - const path = getArray(action, "path") - .map((point) => { - if (!point || typeof point !== "object") return undefined; - const x = readOptionalNumber(point, "x"); - const y = readOptionalNumber(point, "y"); - return x !== undefined && y !== undefined ? { x, y } : undefined; - }) - .filter((point): point is { x: number; y: number } => Boolean(point)); - if (path.length >= 2) return [{ type: "drag", path }]; - - const x = readOptionalNumber(action, "x"); - const y = readOptionalNumber(action, "y"); - const endX = readOptionalNumber(action, "end_x") ?? readOptionalNumber(action, "x2"); - const endY = readOptionalNumber(action, "end_y") ?? readOptionalNumber(action, "y2"); - if (x === undefined || y === undefined || endX === undefined || endY === undefined) return []; - return [{ type: "drag", path: [{ x, y }, { x: endX, y: endY }] }]; -} - -function toKeypressAction(action: Record): CuaAction[] { - const keys = getArray(action, "keys") - .map((key) => (typeof key === "string" ? key : undefined)) - .filter((key): key is string => Boolean(key)); - const key = getString(action, "key"); - const text = getString(action, "text"); - const value = keys.length > 0 ? keys : key ? [key] : text ? [text] : []; - return value.length > 0 ? [{ type: "keypress", keys: value }] : []; -} - -function toScrollAction(action: Record): CuaAction { - return { - type: "scroll", - x: readOptionalNumber(action, "x"), - y: readOptionalNumber(action, "y"), - scroll_x: readOptionalNumber(action, "scroll_x"), - scroll_y: readOptionalNumber(action, "scroll_y") ?? readOptionalNumber(action, "amount"), - }; -} - -function secondsToMs(seconds: number | undefined): number | undefined { - return seconds === undefined ? undefined : seconds * 1000; -} - -function convertTools(tools: Tool[]): Array> { - return tools.map((tool) => ({ - type: "function", - name: tool.name, - description: tool.description, - parameters: tool.parameters, - })); -} - -function convertMessages(messages: readonly Message[], nativeComputerName?: string): Array> { - const items: Array> = []; - for (const message of messages) { - if (message.role === "user") { - items.push({ role: "user", content: convertUserContent(message.content) }); - continue; - } - if (message.role === "assistant") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join("\n") - .trim(); - if (text) items.push({ role: "assistant", content: text }); - for (const part of message.content) { - if (part.type !== "toolCall") continue; - if (nativeComputerName && part.name === nativeComputerName) { - items.push({ type: "computer_call", call_id: part.id, action: getValue(part.arguments, "action") }); - } else { - items.push({ - type: "function_call", - call_id: part.id, - name: part.name, - arguments: JSON.stringify(part.arguments ?? {}), - }); - } - } - continue; - } - if (message.role === "toolResult") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join("\n") - .trim(); - const image = [...message.content].reverse().find((part): part is ImageContent => part.type === "image"); - if (nativeComputerName && message.toolName === nativeComputerName) { - if (!image) { - throw new Error( - "Tzafon native computer action loops require image tool results; text-only results are unsupported because CUA does not capture post-action screenshots automatically.", - ); - } - items.push({ - type: "computer_call_output", - call_id: message.toolCallId, - output: { type: "computer_screenshot", image_url: `data:${image.mimeType};base64,${image.data}` }, - }); - continue; - } - items.push({ - type: "function_call_output", - call_id: message.toolCallId, - output: message.isError ? `Error: ${text || "tool execution failed"}` : text || "ok", - }); - if (image) { - items.push({ - role: "user", - content: [ - { type: "input_text", text: "screenshot" }, - { type: "input_image", image_url: `data:${image.mimeType};base64,${image.data}`, detail: "auto" }, - ], - }); - } - } - } - return items; -} - -function convertUserContent(content: string | (TextContent | ImageContent)[]): unknown { - if (typeof content === "string") return [{ type: "input_text", text: content }]; - return content.map((part) => { - if (part.type === "text") return { type: "input_text", text: part.text }; - return { type: "input_image", image_url: `data:${part.mimeType};base64,${part.data}`, detail: "auto" }; - }); -} - -function extractMessageText(item: unknown): string { - return getArray(item, "content") - .map((block) => getString(block, "text")) - .filter(Boolean) - .join("\n") - .trim(); -} - -function parseArguments(value: unknown): Record { - const top = - typeof value === "string" && value.trim() - ? safeJsonParse(value) - : value && typeof value === "object" - ? (value as Record) - : {}; - if (!top || typeof top !== "object") return {}; - // Tzafon sometimes nests JSON-encoded arrays/objects inside the top-level argument object - // (observed: { "actions": "[{...}]" }). Unwrap one level so consumers get real values. - const out: Record = {}; - for (const [key, val] of Object.entries(top)) { - out[key] = normalizeArgumentValue(key, val); - } - return out; -} - -const NUMERIC_ARGUMENT_KEYS = new Set(["x", "y", "scroll_x", "scroll_y", "ms", "duration"]); - -function normalizeArgumentValue(key: string, value: unknown): unknown { - const parsed = typeof value === "string" && looksLikeJson(value) ? safeJsonParse(value) ?? value : value; - if (typeof parsed === "string" && NUMERIC_ARGUMENT_KEYS.has(key)) { - const number = Number.parseFloat(parsed); - return Number.isFinite(number) ? number : parsed; - } - if (Array.isArray(parsed)) { - return parsed.map((item) => normalizeArgumentValue(key, item)); - } - if (parsed && typeof parsed === "object") { - return Object.fromEntries( - Object.entries(parsed).map(([childKey, childValue]) => [childKey, normalizeArgumentValue(childKey, childValue)]), - ); - } - return parsed; -} - -function safeJsonParse(value: string): Record | unknown[] | null { - try { - const parsed = JSON.parse(value); - return parsed && typeof parsed === "object" ? parsed : null; - } catch { - return null; - } -} - -function looksLikeJson(value: string): boolean { - const trimmed = value.trim(); - return trimmed.startsWith("[") || trimmed.startsWith("{"); -} - -function usageFromTzafon(usage: unknown): AssistantMessage["usage"] { - const input = readUsageNumber(usage, "input_tokens"); - const output = readUsageNumber(usage, "output_tokens"); - const cacheRead = readUsageNumber(getValue(usage, "input_tokens_details"), "cached_tokens"); - const totalTokens = readUsageNumber(usage, "total_tokens") || input + output; - return { - input, - output, - cacheRead, - cacheWrite: 0, - totalTokens, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }; -} - -function readUsageNumber(obj: unknown, key: string): number { - return readOptionalNumber(obj, key) ?? 0; -} - -function readOptionalNumber(obj: unknown, key: string): number | undefined { - if (!obj || typeof obj !== "object") return undefined; - const value = (obj as Record)[key]; - if (typeof value === "number" && Number.isFinite(value)) return value; - if (typeof value === "string" && value.trim()) { - const number = Number(value); - return Number.isFinite(number) ? number : undefined; - } - return undefined; -} - -function getArray(obj: unknown, key: string): unknown[] { - const value = getValue(obj, key); - return Array.isArray(value) ? value : []; -} - -function getString(obj: unknown, key: string): string { - const value = getValue(obj, key); - return typeof value === "string" ? value : ""; -} - -function getValue(obj: unknown, key: string): unknown { - if (!obj || typeof obj !== "object") return undefined; - return (obj as Record)[key]; -} diff --git a/packages/ai/src/providers/yutori/actions.ts b/packages/ai/src/providers/yutori/actions.ts deleted file mode 100644 index 3cd2c53..0000000 --- a/packages/ai/src/providers/yutori/actions.ts +++ /dev/null @@ -1,188 +0,0 @@ -import type { CuaAction } from "../../actions/index"; -import { normalizeGotoUrl } from "../common"; - -/** - * Native Yutori Navigator n1.5 tool-set ids. - * - * Source of truth: - * - https://docs.yutori.com/reference/n1-5 - * - https://docs.yutori.com/llm-quickstart.md - */ -export const YUTORI_N15_CORE_TOOL_SET = "browser_tools_core-20260403"; - -/** - * Navigator n1's fixed legacy browser action space. - * - * Source of truth: https://docs.yutori.com/reference/n1 - */ -export const YUTORI_N1_ACTION_TYPES = [ - "left_click", - "double_click", - "right_click", - "triple_click", - "type", - "key_press", - "scroll", - "hover", - "drag", - "goto_url", - "go_back", - "refresh", - "wait", -] as const; - -/** - * Navigator n1.5 core visual action space. These are the actions available - * when `tool_set` is `browser_tools_core-20260403`, which keeps CuaAgent in the - * pure screenshot/coordinate path and avoids DOM refs. - * - * Source of truth: https://docs.yutori.com/reference/n1-5 - */ -export const YUTORI_N15_CORE_ACTION_TYPES = [ - "left_click", - "double_click", - "triple_click", - "middle_click", - "right_click", - "mouse_move", - "mouse_down", - "mouse_up", - "drag", - "scroll", - "type", - "key_press", - "hold_key", - "goto_url", - "go_back", - "go_forward", - "refresh", - "wait", -] as const; - -const DEFAULT_SCROLL_AMOUNT = 3; -const SCROLL_AMOUNT_PER_NOTCH = 120; -const DEFAULT_WAIT_MS = 2000; -const NAVIGATION_WAIT_MS = 1500; -const GOTO_WAIT_MS = 2000; - -export function toCanonicalActions(name: string, args: Record): CuaAction[] | undefined { - const coords = readPoint(args.coordinates); - switch (name) { - case "left_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "right_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, button: "right", ...holdKeys(args.modifier) }] : undefined; - case "middle_click": - return coords ? [{ type: "click", x: coords.x, y: coords.y, button: "middle", ...holdKeys(args.modifier) }] : undefined; - case "double_click": - return coords ? [{ type: "double_click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "triple_click": - return coords - ? [ - { type: "double_click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }, - { type: "click", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }, - ] - : undefined; - case "mouse_move": - case "hover": - return coords ? [{ type: "move", x: coords.x, y: coords.y }] : undefined; - case "mouse_down": - return coords ? [{ type: "mouse_down", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "mouse_up": - return coords ? [{ type: "mouse_up", x: coords.x, y: coords.y, ...holdKeys(args.modifier) }] : undefined; - case "drag": { - const start = readPoint(args.start_coordinates); - return start && coords ? [{ type: "drag", path: [start, coords], button: "left" }] : undefined; - } - case "scroll": - return toScrollAction(args, coords); - case "type": - return toTypeActions(args); - case "key_press": - return toKeypressAction(args); - case "hold_key": - return toHoldKeyAction(args); - case "goto_url": { - const url = normalizeGotoUrl(args.url); - return url ? [{ type: "goto", url }, { type: "wait", ms: GOTO_WAIT_MS }] : undefined; - } - case "go_back": - return [{ type: "back" }, { type: "wait", ms: NAVIGATION_WAIT_MS }]; - case "go_forward": - return [{ type: "forward" }, { type: "wait", ms: NAVIGATION_WAIT_MS }]; - case "refresh": - return [{ type: "keypress", keys: ["f5"] }, { type: "wait", ms: DEFAULT_WAIT_MS }]; - case "wait": - return [{ type: "wait", ms: secondsToMs(args.duration, DEFAULT_WAIT_MS) }]; - default: - return undefined; - } -} - -function readPoint(value: unknown): { x: number; y: number } | undefined { - if (!Array.isArray(value) || value.length < 2) return undefined; - const x = Number(value[0]); - const y = Number(value[1]); - if (!Number.isFinite(x) || !Number.isFinite(y)) return undefined; - return { x, y }; -} - -function toScrollAction(args: Record, coords: { x: number; y: number } | undefined): CuaAction[] | undefined { - if (!coords) return undefined; - const direction = typeof args.direction === "string" ? args.direction : "down"; - const amount = typeof args.amount === "number" ? args.amount : DEFAULT_SCROLL_AMOUNT; - const ticks = Math.max(1, Math.trunc(amount)) * SCROLL_AMOUNT_PER_NOTCH; - const scroll_x = direction === "left" ? -ticks : direction === "right" ? ticks : 0; - const scroll_y = direction === "up" ? -ticks : direction === "down" ? ticks : 0; - return [{ type: "scroll", x: coords.x, y: coords.y, scroll_x, scroll_y, ...holdKeys(args.modifier) }]; -} - -function toTypeActions(args: Record): CuaAction[] | undefined { - const text = typeof args.text === "string" ? args.text : undefined; - if (text === undefined) return undefined; - const actions: CuaAction[] = []; - if (args.clear_before_typing === true) { - actions.push({ type: "keypress", keys: ["ctrl", "a"] }, { type: "keypress", keys: ["backspace"] }); - } - actions.push({ type: "type", text }); - if (args.press_enter_after === true) actions.push({ type: "keypress", keys: ["enter"] }); - return actions; -} - -function toKeypressAction(args: Record): CuaAction[] | undefined { - const sequence = readKeySequence(args.key_comb ?? args.key); - return sequence.length > 0 ? sequence.map((keys) => ({ type: "keypress", keys })) : undefined; -} - -function toHoldKeyAction(args: Record): CuaAction[] | undefined { - const keys = readKeyCombo(args.key_comb ?? args.key); - return keys.length > 0 ? [{ type: "keypress", keys, duration: secondsToMs(args.duration, 1000) }] : undefined; -} - -function readKeyCombo(value: unknown): string[] { - if (typeof value !== "string") return []; - return value - .split("+") - .map((part) => part.trim()) - .filter(Boolean); -} - -function readKeySequence(value: unknown): string[][] { - if (typeof value !== "string") return []; - return value - .trim() - .split(/\s+/) - .map((part) => readKeyCombo(part)) - .filter((combo) => combo.length > 0); -} - -function holdKeys(value: unknown): { hold_keys?: string[] } { - if (typeof value !== "string") return {}; - const key = value.trim(); - return key ? { hold_keys: [key] } : {}; -} - -function secondsToMs(value: unknown, fallback: number): number { - if (typeof value !== "number" || !Number.isFinite(value) || value <= 0) return fallback; - return Math.round(value * 1000); -} diff --git a/packages/ai/src/providers/yutori/provider.ts b/packages/ai/src/providers/yutori/provider.ts deleted file mode 100644 index f250302..0000000 --- a/packages/ai/src/providers/yutori/provider.ts +++ /dev/null @@ -1,229 +0,0 @@ -import OpenAI from "openai"; -import type { ChatCompletion, ChatCompletionMessageParam } from "openai/resources/chat/completions"; -import { - createAssistantMessageEventStream, - type Api, - type AssistantMessage, - type Context, - type ImageContent, - type Model, - type SimpleStreamOptions, - type StreamFunction, - type StreamOptions, - type TextContent, - type ToolCall, -} from "@earendil-works/pi-ai"; -import type { CuaIncomingToolPlan } from "../../tool-catalog"; - -export const YUTORI_CHAT_COMPLETIONS_API = "yutori-chat-completions"; - -/** Stream options accepted by {@link streamYutori}. */ -export interface YutoriOptions extends StreamOptions { - /** @internal Identity-addressed native dispatch compiled from selected tools. */ - cuaIncomingToolPlan?: CuaIncomingToolPlan; -} - -export const streamYutori: StreamFunction = (model, context, options) => { - const stream = createAssistantMessageEventStream(); - void runYutoriStream(stream, model, context, options); - return stream; -}; - -export const streamSimpleYutori: StreamFunction = ( - model, - context, - options, -) => streamYutori(model, context, options); - -async function runYutoriStream( - stream: ReturnType, - model: Model, - context: Context, - options: YutoriOptions | undefined, -): Promise { - const output = initialAssistantMessage(model); - try { - const apiKey = options?.apiKey || process.env.YUTORI_API_KEY; - if (!apiKey) throw new Error("missing Yutori API key"); - const client = new OpenAI({ - apiKey, - baseURL: model.baseUrl || "https://api.yutori.com/v1", - defaultHeaders: { ...model.headers, ...options?.headers }, - }); - let payload: Record = { - model: model.id, - messages: convertMessages(context), - max_completion_tokens: options?.maxTokens ?? model.maxTokens, - temperature: options?.temperature ?? 0.3, - }; - const tools = convertTools(context); - if (tools.length > 0) payload.tools = tools; - const nextPayload = await options?.onPayload?.(payload, model); - if (nextPayload !== undefined) payload = nextPayload as Record; - - const { data: response, response: rawResponse } = await client.chat.completions - .create(payload as unknown as Parameters[0], { signal: options?.signal }) - .withResponse(); - const completion = response as ChatCompletion; - await options?.onResponse?.({ status: rawResponse.status, headers: headersToRecord(rawResponse.headers) }, model); - - stream.push({ type: "start", partial: output }); - const choice = completion.choices?.[0]; - const message = choice?.message; - output.responseId = completion.id; - output.usage = usageFromYutori(completion.usage); - if (choice?.finish_reason === "tool_calls") output.stopReason = "toolUse"; - else if (choice?.finish_reason === "length") output.stopReason = "length"; - - const text = typeof message?.content === "string" ? message.content : ""; - if (text) emitText(stream, output, text); - - for (const call of message?.tool_calls ?? []) { - if (call.type !== "function") continue; - const args = parseArguments(call.function.arguments); - const selectedName = options?.cuaIncomingToolPlan?.yutoriNames[call.function.name]; - const contentIndex = output.content.length; - const toolCall: ToolCall = { - type: "toolCall", - id: call.id, - name: selectedName ?? call.function.name, - arguments: args, - }; - output.content.push(toolCall); - output.stopReason = "toolUse"; - stream.push({ type: "toolcall_start", contentIndex, partial: output }); - stream.push({ type: "toolcall_delta", contentIndex, delta: call.function.arguments ?? "", partial: output }); - stream.push({ type: "toolcall_end", contentIndex, toolCall, partial: output }); - } - - stream.push({ type: "done", reason: output.stopReason as "stop" | "length" | "toolUse", message: output }); - stream.end(); - } catch (err) { - output.stopReason = options?.signal?.aborted ? "aborted" : "error"; - output.errorMessage = err instanceof Error ? err.message : String(err); - stream.push({ type: "error", reason: output.stopReason, error: output }); - stream.end(); - } -} - -function initialAssistantMessage(model: Model): AssistantMessage { - return { - role: "assistant", - content: [], - api: model.api, - provider: model.provider, - model: model.id, - usage: { - input: 0, - output: 0, - cacheRead: 0, - cacheWrite: 0, - totalTokens: 0, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }, - stopReason: "stop", - timestamp: Date.now(), - }; -} - -function convertMessages(context: Context): ChatCompletionMessageParam[] { - const messages: ChatCompletionMessageParam[] = []; - if (context.systemPrompt) messages.push({ role: "system", content: context.systemPrompt }); - for (const message of context.messages) { - if (message.role === "user") { - messages.push({ - role: "user", - content: typeof message.content === "string" ? message.content : message.content.map(toOpenAIContentPart), - } as ChatCompletionMessageParam); - } else if (message.role === "assistant") { - const text = message.content - .filter((part): part is TextContent => part.type === "text") - .map((part) => part.text) - .join(""); - const toolCalls = message.content - .filter((part): part is ToolCall => part.type === "toolCall") - .map((part) => ({ - id: part.id, - type: "function" as const, - function: { name: part.name, arguments: JSON.stringify(part.arguments ?? {}) }, - })); - messages.push({ - role: "assistant", - content: text || null, - ...(toolCalls.length > 0 ? { tool_calls: toolCalls } : {}), - }); - } else if (message.role === "toolResult") { - messages.push({ - role: "tool", - tool_call_id: message.toolCallId, - content: message.content.map(toOpenAIContentPart) as unknown as string, - }); - } - } - return messages; -} - -function convertTools(context: Context): Array> { - return (context.tools ?? []).map((tool) => ({ - type: "function", - function: { - name: tool.name, - description: tool.description, - parameters: tool.parameters, - }, - })); -} - -function emitText(stream: ReturnType, output: AssistantMessage, text: string): void { - const contentIndex = output.content.length; - const content: TextContent = { type: "text", text }; - output.content.push(content); - stream.push({ type: "text_start", contentIndex, partial: output }); - stream.push({ type: "text_delta", contentIndex, delta: text, partial: output }); - stream.push({ type: "text_end", contentIndex, content: text, partial: output }); -} - -function toOpenAIContentPart(part: TextContent | ImageContent): Record { - if (part.type === "text") return { type: "text", text: part.text }; - return { type: "image_url", image_url: { url: `data:${part.mimeType};base64,${part.data}` } }; -} - -// Degrade malformed tool-call arguments to {} so one bad call cannot turn the -// whole response into a stopReason "error" turn (mirrors the Tzafon provider). -function parseArguments(value: string | undefined): Record { - if (!value?.trim()) return {}; - try { - const parsed = JSON.parse(value) as unknown; - return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? (parsed as Record) : {}; - } catch { - return {}; - } -} - -function usageFromYutori(usage: unknown): AssistantMessage["usage"] { - const input = readNumber(usage, "prompt_tokens"); - const output = readNumber(usage, "completion_tokens"); - const totalTokens = readNumber(usage, "total_tokens") || input + output; - return { - input, - output, - cacheRead: 0, - cacheWrite: 0, - totalTokens, - cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, - }; -} - -function readNumber(value: unknown, key: string): number { - if (!value || typeof value !== "object") return 0; - const n = (value as Record)[key]; - return typeof n === "number" && Number.isFinite(n) ? n : 0; -} - -function headersToRecord(headers: Headers): Record { - const out: Record = {}; - headers.forEach((value, key) => { - out[key] = value; - }); - return out; -} diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index 13182eb..c608041 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -44,21 +44,6 @@ export type CuaProviderBinding = /** Transport this binding requires the compiled catalog's model to carry. */ readonly requiresApi?: Api; } - | { - readonly kind: "tzafon-native"; - readonly declaration: Record; - /** Transport this binding requires the compiled catalog's model to carry. */ - readonly requiresApi?: Api; - } - | { - readonly kind: "yutori-native"; - readonly generation: "n1" | "n15"; - readonly nativeName: string; - readonly toolSet?: string; - readonly allNativeNames: readonly string[]; - /** Transport this binding requires the compiled catalog's model to carry. */ - readonly requiresApi?: Api; - } | { readonly kind: "google-native"; readonly nativeName: string; @@ -162,8 +147,6 @@ export interface CuaAnthropicBrowserFallback { export interface CuaIncomingToolPlan { readonly anthropicBrowserFallback?: CuaAnthropicBrowserFallback; readonly openaiComputerName?: string; - readonly tzafonComputerName?: string; - readonly yutoriNames: Readonly>; readonly googleNames: Readonly>; /** Google predefined functions disabled by the exact selected native subset. */ readonly googleExcludedNames: readonly string[]; @@ -192,8 +175,6 @@ export interface CuaToolCatalog { export interface CompileCuaToolCatalogOptions { model: CuaModelRef | Model; requestedTools: readonly CuaCatalogToolInput[]; - /** Catalog-planning context; feeds declaration defaulting (e.g. Tzafon display size). */ - viewport: { readonly width: number; readonly height: number }; } /** @@ -212,9 +193,9 @@ const SAFE_TOOL_NAME = /^[A-Za-z][A-Za-z0-9_-]{0,63}$/; /** * Compile exactly one identity-keyed catalog for a model and caller-owned - * requested list. Pure and declaration-only: identical declaration, model, - * and viewport inputs produce identical catalogs, and compilation never - * constructs executable tools or retains the requested input objects. + * requested list. Pure and declaration-only: identical declaration and model + * inputs produce identical catalogs, and compilation never constructs + * executable tools or retains the requested input objects. * * The compiled model's `api` is derived here, not stamped on the model ahead * of time: a selected tool's provider binding may declare `requiresApi`, and @@ -226,15 +207,14 @@ export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): Cu const baseModel = typeof options.model === "string" ? getCuaModel(options.model) : routeCuaApi(resetCatalogDerivedApi(options.model)); - const viewport = options.viewport; - const normalizedEntries = [...options.requestedTools].map((tool) => normalizeTool(tool, viewport)); + const normalizedEntries = [...options.requestedTools].map(normalizeTool); const requiresApi = validateCatalog(baseModel, normalizedEntries); const model = requiresApi ? { ...baseModel, api: requiresApi } : baseModel; const drafts = resolveProviderFacingDeclarations(normalizedEntries); const names = new Map(drafts.map((entry) => [entry.identity, entry.name])); const requirements = compileHeaderRequirements(drafts); - const transforms = compilePayloadTransforms(model, drafts, viewport); + const transforms = compilePayloadTransforms(model, drafts); validateTransformClaims(transforms); const incoming = compileIncomingPlan(drafts); const fingerprint = stableStringify({ @@ -284,10 +264,7 @@ export function modelSupportsDeferredTools(model: Model): boolean { return major > 4 || (major === 4 && minor >= 5); } -function normalizeTool( - tool: CuaCatalogToolInput, - viewport: { readonly width: number; readonly height: number }, -): CuaCatalogEntryDraft { +function normalizeTool(tool: CuaCatalogToolInput): CuaCatalogEntryDraft { if (isCuaToolSpec(tool)) { const schemaFingerprint = stableStringify(tool.declaration.parameters); const fingerprint = stableStringify({ @@ -296,15 +273,9 @@ function normalizeTool( schema: schemaFingerprint, coordinates: tool.execution.kind === "actions" ? tool.execution.coordinates : undefined, }); - const inspectedDeclaration = tool.providerBinding?.kind === "tzafon-native" - ? { - ...tool.providerBinding.declaration, - display_width: tool.providerBinding.declaration.display_width ?? viewport.width, - display_height: tool.providerBinding.declaration.display_height ?? viewport.height, - } - : tool.providerBinding && "declaration" in tool.providerBinding - ? tool.providerBinding.declaration - : tool.declaration; + const inspectedDeclaration = tool.providerBinding && "declaration" in tool.providerBinding + ? tool.providerBinding.declaration + : tool.declaration; return Object.freeze({ identity: tool.identity, name: tool.name, @@ -359,25 +330,8 @@ function resolveProviderFacingDeclarations(entries: readonly CuaCatalogEntryDraf }; })() : undefined; - const yutori = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native"); - const yutoriDeclaration = yutori.length > 0 ? (() => { - const binding = yutori[0]!.providerBinding; - if (binding?.kind !== "yutori-native") return undefined; - const selected = new Set(yutori.map((entry) => { - const selectedBinding = entry.providerBinding; - return selectedBinding?.kind === "yutori-native" ? selectedBinding.nativeName : ""; - })); - return { - ...(binding.toolSet ? { tool_set: binding.toolSet } : {}), - disable_tools: binding.allNativeNames.filter((name) => !selected.has(name)), - }; - })() : undefined; - return entries.map((entry) => { - const binding = entry.providerBinding; - const declaration = binding?.kind === "google-native" - ? googleDeclaration - : binding?.kind === "yutori-native" ? yutoriDeclaration : undefined; + const declaration = entry.providerBinding?.kind === "google-native" ? googleDeclaration : undefined; return declaration ? Object.freeze({ ...entry, declaration: Object.freeze(declaration) }) : entry; }); } @@ -457,14 +411,6 @@ function validateAnthropicNativeModel(model: Model, identity: string): void /** Validate the selected native tools agree on a provider and a transport, and return the transport they require, if any. */ function validateToolsetCompatibility(model: Model, entries: readonly CuaCatalogEntryDraft[]): Api | undefined { - const yutoriN1 = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native" && entry.providerBinding.generation === "n1"); - if (yutoriN1.length > 0) { - const all = yutoriN1[0]!.providerBinding; - if (all?.kind === "yutori-native" && yutoriN1.length !== all.allNativeNames.length) { - throw new Error(`Yutori n1 cannot suppress a partial native action set; select the complete cua.providers.yutori.toolsets.n1() toolset`); - } - } - const nativeProviderKinds = new Set( entries.flatMap((entry) => entry.providerBinding ? [entry.providerBinding.kind.split("-")[0]] : []), ); @@ -547,11 +493,7 @@ function commaTokens(value: string | undefined): string[] { return value?.split(",").map((token) => token.trim()).filter(Boolean) ?? []; } -function compilePayloadTransforms( - model: Model, - entries: readonly CuaCatalogEntryDraft[], - viewport: { readonly width: number; readonly height: number }, -): CuaPayloadTransform[] { +function compilePayloadTransforms(model: Model, entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform[] { const transforms: CuaPayloadTransform[] = []; if (model.provider === "anthropic") { transforms.push({ @@ -567,14 +509,8 @@ function compilePayloadTransforms( for (const entry of entries) { const binding = entry.providerBinding; if (!binding) continue; - if (binding.kind === "anthropic-native" || binding.kind === "openai-native" || binding.kind === "tzafon-native") { - const declaration = binding.kind === "tzafon-native" - ? { - ...binding.declaration, - display_width: binding.declaration.display_width ?? viewport.width, - display_height: binding.declaration.display_height ?? viewport.height, - } - : binding.declaration; + if (binding.kind === "anthropic-native" || binding.kind === "openai-native") { + const { declaration } = binding; transforms.push({ identity: entry.identity, consumesToolIdentities: [entry.identity], @@ -587,8 +523,6 @@ function compilePayloadTransforms( } } - const yutori = entries.filter((entry) => entry.providerBinding?.kind === "yutori-native"); - if (yutori.length > 0) transforms.push(createYutoriTransform(yutori)); const google = entries.filter((entry) => entry.providerBinding?.kind === "google-native"); if (google.length > 0) transforms.push(createGoogleTransform(google)); @@ -605,35 +539,6 @@ function compilePayloadTransforms( return transforms; } -function createYutoriTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform { - const firstBinding = entries[0]!.providerBinding; - if (firstBinding?.kind !== "yutori-native") throw new Error("invalid Yutori catalog entry"); - const generations = new Set(entries.map((entry) => { - const binding = entry.providerBinding; - return binding?.kind === "yutori-native" ? binding.generation : ""; - })); - if (generations.size !== 1) throw new Error("Yutori n1 and n1.5 native toolsets cannot be combined"); - const selectedNativeNames = entries.map((entry) => (entry.providerBinding as Extract).nativeName); - const selectedSet = new Set(selectedNativeNames); - const disabled = firstBinding.allNativeNames.filter((name) => !selectedSet.has(name)); - const identity = `provider.yutori.native.${firstBinding.generation}`; - return { - identity, - consumesToolIdentities: entries.map((entry) => entry.identity), - writes: ["tools", "tool_set", "disable_tools"], - phase: "tool-declarations", - apply(payload, _model, names) { - const stripped = removeSerializedTools(payload, entries.map((entry) => names.get(entry.identity)!)); - if (!isRecord(stripped)) return stripped; - return { - ...stripped, - ...(firstBinding.toolSet ? { tool_set: firstBinding.toolSet } : {}), - disable_tools: disabled, - }; - }, - }; -} - function createGoogleTransform(entries: readonly CuaCatalogEntryDraft[]): CuaPayloadTransform { const firstBinding = entries[0]!.providerBinding; if (firstBinding?.kind !== "google-native") throw new Error("invalid Google catalog entry"); @@ -695,9 +600,7 @@ function createPayloadPlan( function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncomingToolPlan { let anthropicBrowserFallback: CuaAnthropicBrowserFallback | undefined; let openaiComputerName: string | undefined; - let tzafonComputerName: string | undefined; let googleAllNativeNames: readonly string[] = []; - const yutoriNames: Record = {}; const googleNames: Record = {}; const nativeToolNames: string[] = []; for (const entry of entries) { @@ -706,8 +609,6 @@ function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncom nativeToolNames.push(entry.name); if (binding.kind === "anthropic-native" && binding.accessFallback) anthropicBrowserFallback = binding.accessFallback; else if (binding.kind === "openai-native") openaiComputerName = entry.name; - else if (binding.kind === "tzafon-native") tzafonComputerName = entry.name; - else if (binding.kind === "yutori-native") yutoriNames[binding.nativeName] = entry.name; else if (binding.kind === "google-native") { googleNames[binding.nativeName] = entry.name; googleAllNativeNames = binding.allNativeNames; @@ -718,8 +619,6 @@ function compileIncomingPlan(entries: readonly CuaCatalogEntryDraft[]): CuaIncom return Object.freeze({ ...(anthropicBrowserFallback ? { anthropicBrowserFallback: Object.freeze(anthropicBrowserFallback) } : {}), ...(openaiComputerName ? { openaiComputerName } : {}), - ...(tzafonComputerName ? { tzafonComputerName } : {}), - yutoriNames: Object.freeze(yutoriNames), googleNames: Object.freeze(googleNames), googleExcludedNames: Object.freeze(googleExcludedNames), nativeToolNames: Object.freeze(nativeToolNames), diff --git a/packages/ai/test/anthropic-browser-fallback.test.ts b/packages/ai/test/anthropic-browser-fallback.test.ts index c490f80..98aa5f6 100644 --- a/packages/ai/test/anthropic-browser-fallback.test.ts +++ b/packages/ai/test/anthropic-browser-fallback.test.ts @@ -16,8 +16,6 @@ import { } from "../src/index"; import { withAnthropicBrowserFallback } from "../src/providers/anthropic/browser-fallback"; -const viewport = { width: 1440, height: 900 }; - const context: Context = { systemPrompt: "", messages: [{ role: "user", content: [{ type: "text", text: "Use the browser." }], timestamp: 1 }], @@ -30,7 +28,6 @@ describe("Anthropic native browser access fallback", () => { const catalog = compileCuaToolCatalog({ model, requestedTools: [cua.providers.anthropic.tools.browser()], - viewport, }); const payloads: Array<{ tools: unknown[]; headers: SimpleStreamOptions["headers"] }> = []; let calls = 0; @@ -69,7 +66,6 @@ describe("Anthropic native browser access fallback", () => { const catalog = compileCuaToolCatalog({ model, requestedTools: [cua.providers.anthropic.tools.browser()], - viewport, }); let calls = 0; const provider = withAnthropicBrowserFallback(fakeProvider(async (selectedModel) => { diff --git a/packages/ai/test/anthropic-native.integration.test.ts b/packages/ai/test/anthropic-native.integration.test.ts index d187a0e..3ad553f 100644 --- a/packages/ai/test/anthropic-native.integration.test.ts +++ b/packages/ai/test/anthropic-native.integration.test.ts @@ -8,8 +8,6 @@ import { const apiKey = process.env.ANTHROPIC_API_KEY; const liveIt = apiKey ? it : it.skip; -const viewport = { width: 1440, height: 900 }; - const cases = [ { name: "computer", @@ -31,7 +29,6 @@ describe("Anthropic early-access native tools", () => { const catalog = compileCuaToolCatalog({ model: "anthropic:claude-opus-5", requestedTools: [current.tool], - viewport, }); const response = await createCuaModels().complete( catalog.model, diff --git a/packages/ai/test/api-keys.test.ts b/packages/ai/test/api-keys.test.ts index 205be36..a484791 100644 --- a/packages/ai/test/api-keys.test.ts +++ b/packages/ai/test/api-keys.test.ts @@ -16,8 +16,6 @@ const ENV_KEYS = [ "XAI_API_KEY", "MOONSHOT_API_KEY", "OPENROUTER_API_KEY", - "TZAFON_API_KEY", - "YUTORI_API_KEY", ] as const; const ORIGINAL_ENV = new Map(ENV_KEYS.map((key) => [key, process.env[key]])); @@ -65,7 +63,7 @@ describe("cua api key helpers", () => { }); it("throws readable errors when missing", () => { - delete process.env.TZAFON_API_KEY; - expect(() => requireCuaEnvApiKey("tzafon")).toThrow("TZAFON_API_KEY"); + delete process.env.META_API_KEY; + expect(() => requireCuaEnvApiKey("meta")).toThrow("META_API_KEY"); }); }); diff --git a/packages/ai/test/google-provider.test.ts b/packages/ai/test/google-provider.test.ts index c2c6b9a..b907c7f 100644 --- a/packages/ai/test/google-provider.test.ts +++ b/packages/ai/test/google-provider.test.ts @@ -9,13 +9,11 @@ const model: Model = { ...getCuaModel const incoming = { googleNames: { click: "click" }, googleExcludedNames: ["take_screenshot"], - yutoriNames: {}, nativeToolNames: ["click"], }; const screenshotIncoming = { googleNames: { take_screenshot: "take_screenshot" }, googleExcludedNames: ["click"], - yutoriNames: {}, nativeToolNames: ["take_screenshot"], }; const clickTool = { name: "click", description: "Click the page", parameters: { type: "object" } as never }; diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index b9f5650..f8041e2 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -14,7 +14,7 @@ import { describe("CUA model refs", () => { it("parses and formats provider-qualified refs", () => { expect(parseCuaModelRef("openai:gpt-5.5")).toEqual({ provider: "openai", model: "gpt-5.5" }); - expect(formatCuaModelRef("yutori", "n1.5-latest")).toBe("yutori:n1.5-latest"); + expect(formatCuaModelRef("meta", "muse-spark-1.1")).toBe("meta:muse-spark-1.1"); }); it("rejects unqualified and unsupported refs", () => { @@ -25,7 +25,7 @@ describe("CUA model refs", () => { it("names the valid providers in the unsupported-provider error", () => { expect(() => parseCuaModelRef("bogus:model")).toThrow( - 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, meta, xai, moonshotai, openrouter, tzafon, yutori)', + 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, meta, xai, moonshotai, openrouter)', ); }); @@ -50,10 +50,6 @@ describe("CUA model refs", () => { }); it("returns override models for refs missing from pi-ai", () => { - const model = getCuaModel("yutori:n1.5-latest"); - expect(model.provider).toBe("yutori"); - expect(model.api).toBe("yutori-chat-completions"); - const opus = getCuaModel("anthropic:claude-opus-5"); expect(cuaOverrideModels("anthropic")).toEqual([]); expect(opus).toMatchObject({ @@ -125,11 +121,6 @@ describe("CUA model refs", () => { expect(getCuaModel("moonshot:kimi-k3" as CuaModelRef).id).toBe("kimi-k3"); }); - it("loads supported custom provider models without explicit registration", () => { - expect(getCuaModel("tzafon:tzafon.northstar-cua-fast").api).toBe("tzafon-responses"); - expect(getCuaModel("yutori:n1.5-latest").api).toBe("yutori-chat-completions"); - }); - it("resolves every model to its ordinary registry transport, independent of tool selection", () => { // getCuaModel() never derives a tool-driven transport: OPENAI_CUA_COMPUTER_API // and GOOGLE_CUA_INTERACTIONS_API are only ever carried by a model that @@ -206,9 +197,6 @@ describe("CUA support annotations", () => { expect(findCuaAnnotation("moonshotai", "kimi-k2.5")).toBeUndefined(); expect(findCuaAnnotation("moonshotai", "kimi-latest")).toBeUndefined(); expect(findCuaAnnotation("google", "gemini-3.5-flash-lite")).toBeDefined(); - expect(findCuaAnnotation("yutori", "n1.5-latest")).toBeDefined(); - expect(findCuaAnnotation("tzafon", "tzafon.northstar-cua-fast")).toBeDefined(); - expect(findCuaAnnotation("tzafon", "tzafon.northstar-cua-fast-1.6")).toBeDefined(); }); it("advertises only Google's current documented computer-use models", () => { diff --git a/packages/ai/test/openai-adapter-routing.test.ts b/packages/ai/test/openai-adapter-routing.test.ts index 92c2a23..d258922 100644 --- a/packages/ai/test/openai-adapter-routing.test.ts +++ b/packages/ai/test/openai-adapter-routing.test.ts @@ -86,7 +86,7 @@ describe("OpenAI adapter routing", () => { tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }], }, { apiKey: "test", - cuaIncomingToolPlan: { openaiComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, + cuaIncomingToolPlan: { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, } as never).result(); // Only the CUA native-computer adapter understands computer_call items; diff --git a/packages/ai/test/openai-native-provider.test.ts b/packages/ai/test/openai-native-provider.test.ts index d4a584a..9a61d37 100644 --- a/packages/ai/test/openai-native-provider.test.ts +++ b/packages/ai/test/openai-native-provider.test.ts @@ -19,7 +19,7 @@ const model = getCuaModel("openai:gpt-5.5") as Model<"openai-responses">; // The catalog derives this api when OpenAI's native computer tool is selected; // the provider wrapper routes it to the adapter under test. const nativeModel = { ...model, api: openai.OPENAI_CUA_COMPUTER_API } as unknown as Model<"openai-responses">; -const incoming = { openaiComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; +const incoming = { openaiComputerName: "computer", googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }; describe("OpenAI native computer Responses adapter", () => { it("emits one identity-selected local call for actions[] and safety checks", async () => { diff --git a/packages/ai/test/providers.test.ts b/packages/ai/test/providers.test.ts index f7408fe..429f34d 100644 --- a/packages/ai/test/providers.test.ts +++ b/packages/ai/test/providers.test.ts @@ -1,15 +1,10 @@ import { describe, expect, it } from "vitest"; -import { - createCuaModels, - cuaModels, - TZAFON_RESPONSES_API, - YUTORI_CHAT_COMPLETIONS_API, -} from "../src/index"; +import { createCuaModels, cuaModels } from "../src/index"; describe("createCuaModels", () => { it("registers the CUA-only providers alongside pi's builtins", () => { const models = createCuaModels(); - for (const id of ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter", "tzafon", "yutori"]) { + for (const id of ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter"]) { const provider = models.getProvider(id); expect(provider, id).toBeDefined(); expect(provider?.stream).toBeTypeOf("function"); @@ -24,16 +19,8 @@ describe("createCuaModels", () => { expect(models.getModel("moonshotai", "kimi-k3")?.api).toBe("openai-completions"); expect(models.getModel("openrouter", "moonshotai/kimi-k3")?.api).toBe("openai-completions"); expect(models.getProvider("openrouter")?.baseUrl).toBe("https://openrouter.ai/api/v1"); - const tzafonIds = models.getModels("tzafon").map((m) => m.id); - expect(tzafonIds).toContain("tzafon.northstar-cua-fast"); - const yutoriIds = models.getModels("yutori").map((m) => m.id); - expect(yutoriIds).toContain("n1.5-latest"); - expect(models.getProvider("tzafon")?.baseUrl).toBe("https://api.tzafon.ai"); - expect(models.getModel("tzafon", "tzafon.northstar-cua-fast")).toMatchObject({ - api: TZAFON_RESPONSES_API, - baseUrl: "https://api.tzafon.ai", - }); - expect(models.getModel("yutori", "n1.5-latest")?.api).toBe(YUTORI_CHAT_COMPLETIONS_API); + expect(models.getModels("meta").map((m) => m.id)).toContain("muse-spark-1.1"); + expect(models.getProvider("meta")?.baseUrl).toBe("https://api.meta.ai/v1"); }); it("keeps builtin catalogs on wrapped providers", () => { @@ -54,15 +41,15 @@ describe("createCuaModels", () => { it("returns independent collections", () => { const a = createCuaModels(); const b = createCuaModels(); - a.deleteProvider("tzafon"); - expect(a.getProvider("tzafon")).toBeUndefined(); - expect(b.getProvider("tzafon")).toBeDefined(); + a.deleteProvider("meta"); + expect(a.getProvider("meta")).toBeUndefined(); + expect(b.getProvider("meta")).toBeDefined(); }); }); describe("cuaModels", () => { it("memoizes the default collection", () => { expect(cuaModels()).toBe(cuaModels()); - expect(cuaModels().getProvider("yutori")).toBeDefined(); + expect(cuaModels().getProvider("meta")).toBeDefined(); }); }); diff --git a/packages/ai/test/tool-catalog.test.ts b/packages/ai/test/tool-catalog.test.ts index d193735..ccf380e 100644 --- a/packages/ai/test/tool-catalog.test.ts +++ b/packages/ai/test/tool-catalog.test.ts @@ -9,10 +9,8 @@ import { type CuaToolSpec, } from "../src/index"; -const viewport = { width: 1440, height: 900 }; - function compile(model: Parameters[0]["model"], requestedTools: Parameters[0]["requestedTools"]) { - return compileCuaToolCatalog({ model, requestedTools, viewport }); + return compileCuaToolCatalog({ model, requestedTools }); } /** Sanitized caller declaration: cua-ai never receives executable members. */ @@ -80,9 +78,6 @@ describe("cua tool namespace", () => { cua.providers.anthropic.tools.computer(), ]], [cua.providers.google.source, cua.providers.google.toolsets.browser()], - [cua.providers.tzafon.source, [cua.providers.tzafon.tools.computer()]], - [cua.providers.yutori.sources.n1, cua.providers.yutori.toolsets.n1()], - [cua.providers.yutori.sources.n15Core, cua.providers.yutori.toolsets.n15Core()], ]; for (const [source, tools] of surfaces) { expect(source).toMatch(/^https:\/\//); @@ -199,9 +194,9 @@ describe("compileCuaToolCatalog", () => { expect(() => compile("openai:gpt-5.5", [cua.providers.anthropic.tools.computer()])).toThrow(/requires a anthropic model/); }); - it("replaces only the selected Tzafon identity placeholder", async () => { - const catalog = compile("tzafon:tzafon.northstar-cua-fast", [ - cua.providers.tzafon.tools.computer(), + it("replaces only the selected OpenAI identity placeholder", async () => { + const catalog = compile("openai:gpt-5.5", [ + cua.providers.openai.tools.computer(), callerTool("click"), cua.tools.browser.click(), ]); @@ -214,11 +209,11 @@ describe("compileCuaToolCatalog", () => { }; const next = await catalog.payload.apply(payload, catalog.model) as { tools: Array> }; expect(next.tools).toEqual([ - { type: "computer_use", display_width: 1440, display_height: 900, environment: "browser" }, + { type: "computer" }, { type: "function", name: "click" }, { type: "function", name: "browser_click" }, ]); - expect(catalog.incoming.tzafonComputerName).toBe("computer"); + expect(catalog.incoming.openaiComputerName).toBe("computer"); }); it("composes Anthropic native browser declarations, access fallback, and ordinary functions", async () => { @@ -243,15 +238,21 @@ describe("compileCuaToolCatalog", () => { expect(catalog.entries[0]?.dynamicLoading).toBe("eager-only"); }); - it("serializes Google's current native declaration", async () => { + it("serializes Google's current native declaration and keeps custom functions", async () => { const selected = cua.providers.google.toolsets.browser({ exclude: ["right_click", "triple_click"] }); - const catalog = compile("google:gemini-3.6-flash", selected); - const next = await catalog.payload.apply({ tools: selected.map((tool) => ({ type: "function", name: tool.name })) }, catalog.model) as { tools: unknown[] }; - expect(next.tools).toEqual([{ - type: "computer_use", - environment: "browser", - excluded_predefined_functions: ["triple_click", "right_click"], - }]); + const catalog = compile("google:gemini-3.6-flash", [...selected, callerTool("custom")]); + const next = await catalog.payload.apply({ tools: [ + ...selected.map((tool) => ({ type: "function", name: tool.name })), + { type: "function", name: "custom" }, + ] }, catalog.model) as { tools: unknown[] }; + expect(next.tools).toEqual([ + { + type: "computer_use", + environment: "browser", + excluded_predefined_functions: ["triple_click", "right_click"], + }, + { type: "function", name: "custom" }, + ]); expect(catalog.entries[0]?.declaration).toEqual(next.tools[0]); expect(catalog.entries[0]?.coordinates).toEqual({ type: "normalized", range: [0, 999] }); const click = selected.find((tool) => tool.name === "click")!; @@ -298,34 +299,11 @@ describe("compileCuaToolCatalog", () => { } }); - it("uses selected Yutori identities for disable_tools and keeps custom functions", async () => { - const selected = cua.providers.yutori.toolsets.n15Core().slice(0, 2); - const catalog = compile("yutori:n1.5-latest", [...selected, callerTool("custom")]); - const payload = { messages: [{ role: "user", content: "go" }], tools: [ - ...selected.map((tool) => ({ type: "function", function: { name: tool.name } })), - { type: "function", function: { name: "custom" } }, - ] }; - const next = await catalog.payload.apply(payload, catalog.model) as { - tool_set: string; - disable_tools: string[]; - tools: Array<{ function: { name: string } }>; - messages: Array<{ content: unknown }>; - }; - expect(next.tool_set).toBe("browser_tools_core-20260403"); - expect(next.disable_tools).not.toContain(selected[0]?.name); - expect(next.disable_tools).toContain("right_click"); - expect(next.tools.map((tool) => tool.function.name)).toEqual(["custom"]); - expect(next.messages).toEqual(payload.messages); - }); - - it("rejects partial n1 selection and incompatible model changes", () => { - expect(() => compile("yutori:n1-latest", cua.providers.yutori.toolsets.n1().slice(0, 1))).toThrow(/complete .*n1\(\)/); + it("rejects incompatible model changes", () => { const nativeTools: Array<[CuaToolSpec[], string]> = [ [[cua.providers.anthropic.tools.browser()], "anthropic"], [[cua.providers.openai.tools.computer()], "openai"], [[cua.providers.google.toolsets.browser()[0]!], "google"], - [[cua.providers.tzafon.tools.computer()], "tzafon"], - [cua.providers.yutori.toolsets.n1(), "yutori"], ]; for (const [tools, provider] of nativeTools) { expect(() => compile("openrouter:moonshotai/kimi-k3", tools)).toThrow(new RegExp(`requires a ${provider} model`)); @@ -341,7 +319,7 @@ describe("compileCuaToolCatalog", () => { expect(pixels.entries[0]?.fingerprint).not.toBe(normalized.entries[0]?.fingerprint); }); - it("produces deterministic fingerprints for identical declaration, model, and viewport inputs", () => { + it("produces deterministic fingerprints for identical declaration and model inputs", () => { const compileInputs = () => [cua.tools.browser.snapshot(), cua.tools.computer.click(), callerTool("custom")]; const first = compile("openai:gpt-5.5", compileInputs()); const second = compile("openai:gpt-5.5", compileInputs()); diff --git a/packages/ai/test/tzafon-actions.test.ts b/packages/ai/test/tzafon-actions.test.ts deleted file mode 100644 index b45820c..0000000 --- a/packages/ai/test/tzafon-actions.test.ts +++ /dev/null @@ -1,80 +0,0 @@ -import { describe, expect, it } from "vitest"; -import * as tzafon from "../src/providers/tzafon/provider"; - -describe("Tzafon native action normalization", () => { - it("normalizes click variants", () => { - expect(tzafon.toCanonicalActions({ type: "click", x: 10, y: 20 })).toEqual([{ type: "click", x: 10, y: 20 }]); - expect(tzafon.toCanonicalActions({ type: "left_click", x: 10, y: 20 })).toEqual([{ type: "click", x: 10, y: 20 }]); - expect(tzafon.toCanonicalActions({ type: "right_click", x: 10, y: 20 })).toEqual([ - { type: "click", x: 10, y: 20, button: "right" }, - ]); - expect(tzafon.toCanonicalActions({ type: "double_click", x: 10, y: 20 })).toEqual([ - { type: "double_click", x: 10, y: 20 }, - ]); - expect(tzafon.toCanonicalActions({ type: "triple_click", x: 10, y: 20 })).toEqual([ - { type: "double_click", x: 10, y: 20 }, - { type: "click", x: 10, y: 20 }, - ]); - }); - - it("coerces string coordinates to numbers", () => { - expect(tzafon.toCanonicalActions({ type: "click", x: "10", y: "20" })).toEqual([{ type: "click", x: 10, y: 20 }]); - }); - - it("drops pointer actions without usable coordinates", () => { - expect(tzafon.toCanonicalActions({ type: "click" })).toEqual([]); - expect(tzafon.toCanonicalActions({ type: "hover", x: 5 })).toEqual([]); - expect(tzafon.toCanonicalActions(undefined)).toEqual([]); - expect(tzafon.toCanonicalActions({ type: "unknown_action" })).toEqual([]); - }); - - it("normalizes move, hover, and drag", () => { - expect(tzafon.toCanonicalActions({ type: "move", x: 1, y: 2 })).toEqual([{ type: "move", x: 1, y: 2 }]); - expect(tzafon.toCanonicalActions({ type: "hover", x: 1, y: 2 })).toEqual([{ type: "move", x: 1, y: 2 }]); - expect(tzafon.toCanonicalActions({ type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - expect(tzafon.toCanonicalActions({ type: "drag", x: 1, y: 2, end_x: 3, end_y: 4 })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - expect(tzafon.toCanonicalActions({ type: "drag", x: 1, y: 2, x2: 3, y2: 4 })).toEqual([ - { type: "drag", path: [{ x: 1, y: 2 }, { x: 3, y: 4 }] }, - ]); - }); - - it("normalizes typing and keyboard actions", () => { - expect(tzafon.toCanonicalActions({ type: "type", text: "hello" })).toEqual([{ type: "type", text: "hello" }]); - expect(tzafon.toCanonicalActions({ type: "key", key: "enter" })).toEqual([{ type: "keypress", keys: ["enter"] }]); - expect(tzafon.toCanonicalActions({ type: "key", text: "esc" })).toEqual([{ type: "keypress", keys: ["esc"] }]); - expect(tzafon.toCanonicalActions({ type: "keypress", keys: ["ctrl", "a"] })).toEqual([ - { type: "keypress", keys: ["ctrl", "a"] }, - ]); - expect(tzafon.toCanonicalActions({ type: "keypress" })).toEqual([]); - }); - - it("normalizes scroll variants", () => { - expect(tzafon.toCanonicalActions({ type: "scroll", x: 5, y: 6, scroll_y: 120 })).toEqual([ - { type: "scroll", x: 5, y: 6, scroll_y: 120 }, - ]); - expect(tzafon.toCanonicalActions({ type: "scroll", amount: 240 })).toEqual([{ type: "scroll", scroll_y: 240 }]); - expect(tzafon.toCanonicalActions({ type: "hscroll", amount: 120 })).toEqual([{ type: "scroll", scroll_x: 120 }]); - expect(tzafon.toCanonicalActions({ type: "hscroll" })).toEqual([{ type: "scroll", scroll_x: 0 }]); - }); - - it("normalizes navigation, waits, and screenshots", () => { - expect(tzafon.toCanonicalActions({ type: "navigate", url: "https://example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - ]); - expect(tzafon.toCanonicalActions({ type: "wait", ms: 500 })).toEqual([{ type: "wait", ms: 500 }]); - expect(tzafon.toCanonicalActions({ type: "wait", seconds: 2 })).toEqual([{ type: "wait", ms: 2000 }]); - expect(tzafon.toCanonicalActions({ type: "screenshot" })).toEqual([{ type: "screenshot" }]); - }); - - it("maps terminal actions to answer text", () => { - expect(tzafon.toCanonicalActions({ type: "answer", text: "done!" })).toEqual([{ type: "answer", text: "done!" }]); - expect(tzafon.toCanonicalActions({ type: "done", result: "ok" })).toEqual([{ type: "answer", text: "ok" }]); - expect(tzafon.toCanonicalActions({ type: "terminate", status: "success" })).toEqual([ - { type: "answer", text: "success" }, - ]); - }); -}); diff --git a/packages/ai/test/tzafon-provider.test.ts b/packages/ai/test/tzafon-provider.test.ts deleted file mode 100644 index 8208527..0000000 --- a/packages/ai/test/tzafon-provider.test.ts +++ /dev/null @@ -1,149 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; -import type { Model, ToolCall } from "@earendil-works/pi-ai"; -import { getCuaModel } from "../src/index"; -import * as tzafon from "../src/providers/tzafon/provider"; - -const { lightconeConstructor, responsesCreate } = vi.hoisted(() => ({ - lightconeConstructor: vi.fn(), - responsesCreate: vi.fn(), -})); - -vi.mock("@tzafon/lightcone", () => ({ - default: class { - responses = { create: responsesCreate }; - constructor(options: unknown) { - lightconeConstructor(options); - } - }, -})); - -const model = getCuaModel("tzafon:tzafon.northstar-cua-fast") as Model; - -function toolCalls(content: Array<{ type: string }>): ToolCall[] { - return content.filter((part): part is ToolCall => part.type === "toolCall"); -} - -describe("streamTzafonResponses", () => { - it("configures the resolvable Tzafon API endpoint", async () => { - responsesCreate.mockResolvedValueOnce({ id: "resp_endpoint", usage: {}, output: [] }); - - await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - - expect(model.baseUrl).toBe("https://api.tzafon.ai"); - expect(lightconeConstructor).toHaveBeenLastCalledWith(expect.objectContaining({ - apiKey: "test", - baseURL: "https://api.tzafon.ai", - })); - }); - - it("derives unique ids when one computer_call expands to multiple actions", () => { - expect(tzafon.tzafonToolCallId("call_1", 0)).toBe("call_1"); - expect(tzafon.tzafonToolCallId("call_1", 1)).toBe("call_1:1"); - expect(tzafon.tzafonToolCallId("call_1", 2)).toBe("call_1:2"); - }); - - it("unwraps stringified nested arguments and coerces numeric strings on function calls", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_1", - usage: { input_tokens: 1, output_tokens: 2 }, - output: [ - { - type: "function_call", - call_id: "call_1", - name: "computer_batch", - // Observed Tzafon shape: the actions array arrives JSON-encoded - // inside the argument object, with stringified coordinates. - arguments: JSON.stringify({ actions: JSON.stringify([{ type: "click", x: "10", y: "20" }]) }), - }, - ], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!.name).toBe("computer_batch"); - expect(calls[0]!.arguments).toEqual({ actions: [{ type: "click", x: 10, y: 20 }] }); - }); - - it("normalizes non-native computer_call actions with string coordinates", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_2", - usage: {}, - output: [{ type: "computer_call", call_id: "call_2", action: { type: "left_click", x: "500", y: "250" } }], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!.name).toBe("click"); - expect(calls[0]!.arguments).toEqual({ x: 500, y: 250 }); - }); - - it("rejects native computer actions that require automatic post-action screenshots", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_native_click", - usage: {}, - output: [{ type: "computer_call", call_id: "call_click", action: { type: "click", x: 500, y: 500 } }], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { - apiKey: "test", - cuaIncomingToolPlan: { tzafonComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, - }).result(); - - expect(message.stopReason).toBe("error"); - expect(message.errorMessage).toContain('Tzafon native computer action "click" is unsupported'); - expect(toolCalls(message.content)).toEqual([]); - }); - - it("rejects text-only native computer results before sending a request", () => { - expect(() => tzafon.buildTzafonRequestInput(model, { - messages: [ - { - role: "assistant", - content: [{ type: "toolCall", id: "call_click", name: "computer", arguments: { action: { type: "click", x: 500, y: 500 } } }], - api: model.api, - provider: model.provider, - model: model.id, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 1, - }, - { - role: "toolResult", - toolCallId: "call_click", - toolName: "computer", - content: [{ type: "text", text: "Actions executed successfully." }], - isError: false, - timestamp: 2, - }, - ], - tools: [], - }, { - disableResponseThreading: true, - cuaIncomingToolPlan: { tzafonComputerName: "computer", yutoriNames: {}, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["computer"] }, - })).toThrow("text-only results are unsupported"); - }); - - it("degrades malformed function-call arguments to empty args instead of failing the turn", async () => { - responsesCreate.mockResolvedValueOnce({ - id: "resp_3", - usage: {}, - output: [ - { type: "function_call", call_id: "call_bad", name: "custom_tool", arguments: "{not json" }, - { type: "computer_call", call_id: "call_good", action: { type: "left_click", x: 1, y: 2 } }, - ], - }); - - const message = await tzafon.streamTzafonResponses(model, { messages: [] }, { apiKey: "test" }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(2); - expect(calls[0]!).toMatchObject({ name: "custom_tool", arguments: {} }); - expect(calls[1]!).toMatchObject({ name: "click", arguments: { x: 1, y: 2 } }); - }); -}); diff --git a/packages/ai/test/tzafon-threading.test.ts b/packages/ai/test/tzafon-threading.test.ts deleted file mode 100644 index 36ac2d4..0000000 --- a/packages/ai/test/tzafon-threading.test.ts +++ /dev/null @@ -1,159 +0,0 @@ -import { describe, expect, it } from "vitest"; -import type { Context, Message, Model } from "@earendil-works/pi-ai"; -import * as tzafon from "../src/providers/tzafon/provider"; - -const model = { id: "tzafon.northstar-cua-fast", maxTokens: 4096 } as Model; - -const TURNS = 6; - -/** Build a multi-turn context where each assistant turn carries a distinct responseId followed by a screenshot tool result. */ -function multiTurnContext(): Context { - const messages: Message[] = [{ role: "user", content: "book a flight", timestamp: 0 }]; - for (let turn = 0; turn < TURNS; turn += 1) { - messages.push({ - role: "assistant", - content: [{ type: "toolCall", id: `call_${turn}`, name: "click", arguments: { x: turn, y: turn } }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - responseId: `resp_${turn}`, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 0, - }); - messages.push({ - role: "toolResult", - toolCallId: `call_${turn}`, - toolName: "click", - content: [{ type: "image", mimeType: "image/png", data: `screenshot-${turn}` }], - isError: false, - timestamp: 0, - }); - } - return { messages, tools: [], systemPrompt: "control the browser" }; -} - -function screenshotImageUrls(input: Array>): string[] { - const urls: string[] = []; - for (const item of input) { - const content = item.content; - if (!Array.isArray(content)) continue; - for (const part of content) { - if (part && typeof part === "object" && (part as { type?: string }).type === "input_image") { - urls.push((part as { image_url: string }).image_url); - } - } - } - return urls; -} - -describe("buildTzafonRequestInput response threading", () => { - // Threading ON is the fix: chain via previous_response_id and send only the - // latest screenshot. OFF sends every screenshot — the per-turn growth that - // overflows Tzafon's real 64K window after a few turns. - it("threads the latest delta when enabled (default)", () => { - const body = tzafon.buildTzafonRequestInput(model, multiTurnContext()); - const screenshots = screenshotImageUrls(body.input); - - expect(screenshots).toHaveLength(1); - expect(screenshots[0]).toBe(`data:image/png;base64,screenshot-${TURNS - 1}`); - expect(body.previous_response_id).toBe(`resp_${TURNS - 1}`); - expect(body.store).toBe(true); - }); - - it("sends the full screenshot history when threading is disabled by option (locks the failure mode)", () => { - const body = tzafon.buildTzafonRequestInput(model, multiTurnContext(), { disableResponseThreading: true }); - const screenshots = screenshotImageUrls(body.input); - - expect(screenshots).toHaveLength(TURNS); - expect(screenshots).toEqual(Array.from({ length: TURNS }, (_, turn) => `data:image/png;base64,screenshot-${turn}`)); - expect(body.previous_response_id).toBeUndefined(); - expect(body.store).toBeUndefined(); - }); - - it("falls back to full history when no prior turn carries a responseId", () => { - const context = multiTurnContext(); - for (const message of context.messages) { - if (message.role === "assistant") delete message.responseId; - } - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - expect(body.previous_response_id).toBeUndefined(); - }); - - it("replays full history when the latest assistant turn has no responseId (failed request)", () => { - const context = multiTurnContext(); - // A newer assistant turn whose request errored carries no responseId; threading must - // anchor on it and replay, not chain to the older id and re-send the items past it. - context.messages.push({ - role: "assistant", - content: [{ type: "text", text: "request failed" }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "error", - timestamp: 0, - }); - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(body.previous_response_id).toBeUndefined(); - expect(body.store).toBeUndefined(); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - }); - - it("replays full history when the latest assistant turn is from a different api", () => { - const context = multiTurnContext(); - context.messages.push({ - role: "assistant", - content: [{ type: "text", text: "done" }], - api: "anthropic-messages", - provider: "anthropic", - model: "claude-opus-4-8", - responseId: "msg_anthropic", - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "stop", - timestamp: 0, - }); - - const body = tzafon.buildTzafonRequestInput(model, context); - expect(body.previous_response_id).toBeUndefined(); - expect(screenshotImageUrls(body.input)).toHaveLength(TURNS); - }); - - // Off-path screenshot count scales with turn count; on-path stays constant at one. - it("grows the payload per turn when off but stays flat when on", () => { - const counts = (turns: number, disable: boolean) => { - const messages: Message[] = [{ role: "user", content: "task", timestamp: 0 }]; - for (let turn = 0; turn < turns; turn += 1) { - messages.push({ - role: "assistant", - content: [{ type: "toolCall", id: `c_${turn}`, name: "click", arguments: {} }], - api: tzafon.TZAFON_RESPONSES_API, - provider: "tzafon", - model: model.id, - responseId: `r_${turn}`, - usage: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 } }, - stopReason: "toolUse", - timestamp: 0, - }); - messages.push({ - role: "toolResult", - toolCallId: `c_${turn}`, - toolName: "click", - content: [{ type: "image", mimeType: "image/png", data: `s-${turn}` }], - isError: false, - timestamp: 0, - }); - } - const body = tzafon.buildTzafonRequestInput(model, { messages, tools: [] }, { disableResponseThreading: disable }); - return screenshotImageUrls(body.input).length; - }; - - expect(counts(3, true)).toBe(3); - expect(counts(8, true)).toBe(8); - expect(counts(3, false)).toBe(1); - expect(counts(8, false)).toBe(1); - }); -}); diff --git a/packages/ai/test/yutori-actions.test.ts b/packages/ai/test/yutori-actions.test.ts deleted file mode 100644 index c901302..0000000 --- a/packages/ai/test/yutori-actions.test.ts +++ /dev/null @@ -1,97 +0,0 @@ -import { describe, expect, it } from "vitest"; -import type { CuaAction } from "../src/index"; -import * as yutori from "../src/providers/yutori/actions"; - -const n15CoreActionArgs = { - left_click: { coordinates: [500, 250] }, - double_click: { coordinates: [500, 250] }, - triple_click: { coordinates: [500, 250] }, - middle_click: { coordinates: [500, 250] }, - right_click: { coordinates: [500, 250] }, - mouse_move: { coordinates: [100, 200] }, - mouse_down: { coordinates: [100, 200] }, - mouse_up: { coordinates: [100, 200] }, - drag: { start_coordinates: [100, 200], coordinates: [300, 400] }, - scroll: { coordinates: [500, 500], direction: "down", amount: 3 }, - type: { text: "hello" }, - key_press: { key: "ctrl+c" }, - hold_key: { key: "shift", duration: 1.5 }, - goto_url: { url: "https://example.com" }, - go_back: {}, - go_forward: {}, - refresh: {}, - wait: { duration: 1 }, -} satisfies Record<(typeof yutori.YUTORI_N15_CORE_ACTION_TYPES)[number], Record>; - -describe("Yutori native action normalization", () => { - it("has canonical mappings for every n1.5 core action", () => { - for (const action of yutori.YUTORI_N15_CORE_ACTION_TYPES) { - const canonical = yutori.toCanonicalActions(action, n15CoreActionArgs[action]); - expect(canonical, `${action} did not map to canonical CUA actions`).toBeDefined(); - expect(canonical!.length, `${action} mapped to an empty action list`).toBeGreaterThan(0); - for (const item of canonical as CuaAction[]) { - expect(typeof item.type).toBe("string"); - } - } - }); - - it("normalizes n1/n1.5 click actions to canonical individual actions", () => { - expect(yutori.toCanonicalActions("left_click", { coordinates: [500, 250] })).toEqual([ - { type: "click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("double_click", { coordinates: [500, 250] })).toEqual([ - { type: "double_click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("triple_click", { coordinates: [500, 250] })).toEqual([ - { type: "double_click", x: 500, y: 250 }, - { type: "click", x: 500, y: 250 }, - ]); - expect(yutori.toCanonicalActions("middle_click", { coordinates: [500, 250] })).toEqual([ - { type: "click", x: 500, y: 250, button: "middle" }, - ]); - }); - - it("normalizes mouse, drag, type, and keyboard actions", () => { - expect(yutori.toCanonicalActions("mouse_move", { coordinates: [100, 200] })).toEqual([ - { type: "move", x: 100, y: 200 }, - ]); - expect(yutori.toCanonicalActions("drag", { start_coordinates: [100, 200], coordinates: [300, 400] })).toEqual([ - { type: "drag", path: [{ x: 100, y: 200 }, { x: 300, y: 400 }], button: "left" }, - ]); - expect(yutori.toCanonicalActions("type", { text: "hello", clear_before_typing: true, press_enter_after: true })).toEqual([ - { type: "keypress", keys: ["ctrl", "a"] }, - { type: "keypress", keys: ["backspace"] }, - { type: "type", text: "hello" }, - { type: "keypress", keys: ["enter"] }, - ]); - expect(yutori.toCanonicalActions("hold_key", { key: "shift", duration: 1.5 })).toEqual([ - { type: "keypress", keys: ["shift"], duration: 1500 }, - ]); - expect(yutori.toCanonicalActions("key_press", { key: "ctrl+c" })).toEqual([ - { type: "keypress", keys: ["ctrl", "c"] }, - ]); - expect(yutori.toCanonicalActions("key_press", { key: "down down enter" })).toEqual([ - { type: "keypress", keys: ["down"] }, - { type: "keypress", keys: ["down"] }, - { type: "keypress", keys: ["enter"] }, - ]); - }); - - it("normalizes scroll and navigation actions", () => { - expect(yutori.toCanonicalActions("scroll", { coordinates: [500, 500], direction: "down", amount: 3 })).toEqual([ - { type: "scroll", x: 500, y: 500, scroll_x: 0, scroll_y: 360 }, - ]); - expect(yutori.toCanonicalActions("goto_url", { url: "https://example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - { type: "wait", ms: 2000 }, - ]); - expect(yutori.toCanonicalActions("goto_url", { url: "example.com" })).toEqual([ - { type: "goto", url: "https://example.com" }, - { type: "wait", ms: 2000 }, - ]); - expect(yutori.toCanonicalActions("refresh", {})).toEqual([ - { type: "keypress", keys: ["f5"] }, - { type: "wait", ms: 2000 }, - ]); - }); -}); diff --git a/packages/ai/test/yutori-provider.test.ts b/packages/ai/test/yutori-provider.test.ts deleted file mode 100644 index 533c4f7..0000000 --- a/packages/ai/test/yutori-provider.test.ts +++ /dev/null @@ -1,72 +0,0 @@ -import { describe, expect, it, vi } from "vitest"; -import type { Model, ToolCall } from "@earendil-works/pi-ai"; -import { getCuaModel } from "../src/index"; -import * as yutori from "../src/providers/yutori/provider"; - -const { completionsCreate } = vi.hoisted(() => ({ completionsCreate: vi.fn() })); - -vi.mock("openai", () => ({ - default: class { - chat = { - completions: { - create: (...args: unknown[]) => ({ - withResponse: async () => ({ data: completionsCreate(...args), response: { status: 200, headers: new Headers() } }), - }), - }, - }; - }, -})); - -const model = getCuaModel("yutori:n1.5-latest") as Model; -const incoming = { yutoriNames: { left_click: "left_click" }, googleNames: {}, googleExcludedNames: [], nativeToolNames: ["left_click"] }; - -function toolCalls(content: Array<{ type: string }>): ToolCall[] { - return content.filter((part): part is ToolCall => part.type === "toolCall"); -} - -describe("streamYutori", () => { - it("dispatches selected native calls through fixed catalog names", async () => { - completionsCreate.mockReturnValueOnce({ - id: "chatcmpl_1", - usage: { prompt_tokens: 1, completion_tokens: 2, total_tokens: 3 }, - choices: [{ - finish_reason: "tool_calls", - message: { - content: "", - tool_calls: [{ type: "function", id: "call_1", function: { name: "left_click", arguments: JSON.stringify({ coordinates: [100, 200] }) } }], - }, - }], - }); - - const message = await yutori.streamYutori(model, { messages: [] }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); - expect(message.stopReason).toBe("toolUse"); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(1); - expect(calls[0]!).toMatchObject({ id: "call_1", name: "left_click", arguments: { coordinates: [100, 200] } }); - }); - - it("degrades malformed selected calls to empty args without reclassifying by name", async () => { - completionsCreate.mockReturnValueOnce({ - id: "chatcmpl_2", - usage: { prompt_tokens: 1, completion_tokens: 2, total_tokens: 3 }, - choices: [{ - finish_reason: "tool_calls", - message: { - content: "", - tool_calls: [ - { type: "function", id: "call_bad", function: { name: "left_click", arguments: "{not json" } }, - { type: "function", id: "call_good", function: { name: "left_click", arguments: JSON.stringify({ coordinates: [100, 200] }) } }, - ], - }, - }], - }); - - const message = await yutori.streamYutori(model, { messages: [] }, { apiKey: "test", cuaIncomingToolPlan: incoming }).result(); - expect(message.stopReason).toBe("toolUse"); - expect(message.errorMessage).toBeUndefined(); - const calls = toolCalls(message.content); - expect(calls).toHaveLength(2); - expect(calls[0]!).toMatchObject({ id: "call_bad", name: "left_click", arguments: {} }); - expect(calls[1]!).toMatchObject({ id: "call_good", name: "left_click", arguments: { coordinates: [100, 200] } }); - }); -}); diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 1573a28..7025256 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## 0.12.0 - 2026-08-13 + +Breaking: Tzafon and Yutori support is removed. + +- Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.13.0. Refs like + `-m tzafon:…` and `-m yutori:…` are no longer accepted, `cua models` no + longer lists either provider, and `TZAFON_API_KEY`/`YUTORI_API_KEY` are no + longer read. +- The `/tools` picker no longer has atomic tool groups. They existed only for + Yutori's n1 action set, which the catalog compiler refused to accept as a + partial selection; every remaining tool toggles on its own. + ## 0.11.0 - 2026-08-13 - Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.12.0. The default diff --git a/packages/cli/README.md b/packages/cli/README.md index e08e49a..2a4d72d 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -50,7 +50,6 @@ cua --print --model google:gemini-3.6-flash "..." cua --print --model meta:muse-spark-1.1 "..." cua --print --model xai:grok-4.5 "..." cua --print --model moonshotai:kimi-k3 "..." -cua --print --model yutori:n1.5-latest "..." # Named sessions (browser stays alive across calls): cua session start login # provisions Kernel browser @@ -116,11 +115,7 @@ live tool list untouched. Applying calls the harness's `setTools()`, which compiles and validates the whole catalog before mutating anything — so a rejected selection reports the error and leaves the session unchanged. -Two constraints show up in the picker: - -- Provider-native action sets that cannot be partially suppressed (currently - Yutori n1) toggle as one group. -- Disabling every tool is allowed and yields a text-only agent. +Disabling every tool is allowed and yields a text-only agent. Selections are session-only and never persisted. `/model` rebuilds the tool list from the new model's defaults and reports `tool selection reset to the new @@ -138,7 +133,7 @@ Run `cua models` to list every supported `-m` / `--model` value and the provider it routes to. Filter by provider with `cua models -p openai`, `cua models -p anthropic`, `cua models -p google` (alias: `gemini`), `cua models -p meta`, `cua models -p xai`, `cua models -p moonshotai` -(alias: `moonshot`), `cua models -p openrouter`, or `cua models -p yutori`. +(alias: `moonshot`), or `cua models -p openrouter`. `-m` / `--model` accepts a provider-qualified `provider:model` ref (e.g. `openai:gpt-5.6-sol`) or a bare model id when it matches exactly one catalog @@ -159,8 +154,6 @@ Configuration is by environment variable. There is no config file. | `XAI_API_KEY` | xAI API key (required when `-m xai:…`) | | `MOONSHOT_API_KEY` | Moonshot AI API key (required when `-m moonshotai:…`) | | `OPENROUTER_API_KEY` | OpenRouter API key (required when `-m openrouter:…`) | -| `TZAFON_API_KEY` | Tzafon API key (required when `-m tzafon:…`) | -| `YUTORI_API_KEY` | Yutori API key (required when `-m yutori:…`) | | `KERNEL_BASE_URL` | override Kernel base URL | | `OPENAI_BASE_URL` | override OpenAI base URL | | `ANTHROPIC_BASE_URL` | override Anthropic base URL | @@ -168,8 +161,6 @@ Configuration is by environment variable. There is no config file. | `META_BASE_URL` | override Meta Model API base URL | | `XAI_BASE_URL` | override xAI API base URL | | `MOONSHOTAI_BASE_URL` | override Moonshot AI base URL | -| `TZAFON_BASE_URL` | override Tzafon base URL | -| `YUTORI_BASE_URL` | override Yutori base URL | | `XDG_DATA_HOME` | sessions dir base (defaults to `~/.local/share`) | | `CUA_IMAGE_PROTOCOL` | force inline image protocol (`kitty`/`iterm2`/`none`/`auto`) | @@ -180,13 +171,9 @@ The CLI chooses one explicit interaction catalog and appends pi's coding tools: CUA browser primitives plus the verified `browser_act` plan tool for OpenAI, Meta, xAI, and older Anthropic models; browser primitives alone for Moonshot, whose API rejects `browser_act`'s larger schema; Anthropic's native browser tool -when supported; Google's native browser action set; Tzafon's native computer -tool in a browser environment; and Yutori's native N1/N1.5 browser set. If the -active Anthropic credential cannot access `browser_20260701`, the same selected -browser tool uses its equivalent function transport. Tzafon's native tool allows -explicit screenshots and terminal answers only; non-screenshot actions fail -before browser execution because CUA does not synthesize the post-action images -its continuation protocol requires. Library callers can select any catalog +when supported; and Google's native browser action set. If the active Anthropic +credential cannot access `browser_20260701`, the same selected browser tool uses +its equivalent function transport. Library callers can select any catalog directly; see [`@onkernel/cua-agent`](../agent). ## Output formats diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index b600e20..d5e267e 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -57,7 +57,7 @@ Usage: cua models --json Options: - -p, --provider Filter by provider: openai | anthropic | google | gemini | meta | xai | moonshotai | openrouter | tzafon | yutori + -p, --provider Filter by provider: openai | anthropic | google | gemini | meta | xai | moonshotai | openrouter --json Output JSON -h, --help Show this help `; diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 9dbb6b4..d34e005 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -58,8 +58,6 @@ Options: meta: meta:muse-spark-1.1 xai: xai:grok-4.5 moonshot: moonshotai:kimi-k3 - tzafon: tzafon:tzafon.northstar-cua-fast - yutori: yutori:n1.5-latest --thinking Thinking level: off | minimal | low | medium | high | xhigh | max (default: low; applies to providers that support it) --profile Kernel browser profile to load @@ -101,8 +99,6 @@ Environment: XAI_API_KEY xAI API key (required when -m xai:…) MOONSHOT_API_KEY Moonshot AI API key (required when -m moonshotai:…) OPENROUTER_API_KEY OpenRouter API key (required when -m openrouter:…) - TZAFON_API_KEY Tzafon API key (required when -m tzafon:…) - YUTORI_API_KEY Yutori API key (required when -m yutori:…) KERNEL_BASE_URL Override Kernel base URL OPENAI_BASE_URL Override OpenAI base URL ANTHROPIC_BASE_URL Override Anthropic base URL @@ -110,8 +106,6 @@ Environment: META_BASE_URL Override Meta Model API base URL XAI_BASE_URL Override xAI API base URL MOONSHOTAI_BASE_URL Override Moonshot AI base URL - TZAFON_BASE_URL Override Tzafon base URL - YUTORI_BASE_URL Override Yutori base URL XDG_DATA_HOME Sessions are stored under \$XDG_DATA_HOME/cua/sessions (defaults to ~/.local/share/cua/sessions) CUA_IMAGE_PROTOCOL Force inline image protocol (\`kitty\`|\`iterm2\`|\`none\`|\`auto\`) diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index 488c659..3099587 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -104,15 +104,6 @@ export function defaultInteractionTools(model: CuaModelRef): CuaCliTool[] { : structuredBrowserTools(); case "google": return cua.providers.google.toolsets.browser(); - case "tzafon": - return [cua.providers.tzafon.tools.computer()]; - case "yutori": - return [ - ...(modelId.startsWith("n1.5") - ? cua.providers.yutori.toolsets.n15Core() - : cua.providers.yutori.toolsets.n1()), - cua.tools.computer.screenshot(), - ]; case "meta": case "xai": return structuredBrowserTools(); diff --git a/packages/cli/src/tui/tool-selection.ts b/packages/cli/src/tui/tool-selection.ts index 73a187d..76f34ad 100644 --- a/packages/cli/src/tui/tool-selection.ts +++ b/packages/cli/src/tui/tool-selection.ts @@ -16,12 +16,6 @@ export interface ToolSelectionItem { label: string; group: ToolGroup; description?: string; - /** - * Tools that the catalog compiler refuses to accept as a partial set, and - * which therefore toggle as one unit. Currently only Yutori's n1 native - * action set (`validateToolsetCompatibility` rejects partial n1 subsets). - */ - atomicGroup?: string; } /** Identity key for a caller-owned tool, using cua-ai's canonical identity helper. */ @@ -34,15 +28,6 @@ function toolGroup(tool: CuaCliTool): ToolGroup { return tool.origin === "provider-native" ? "native" : "cua"; } -function atomicGroupOf(tool: CuaCliTool): string | undefined { - if (!isCuaToolSpec(tool)) return undefined; - const binding = tool.providerBinding; - if (binding?.kind === "yutori-native" && binding.generation === "n1") { - return "provider.yutori.native.n1"; - } - return undefined; -} - function toolDescription(tool: CuaCliTool): string | undefined { const raw = isCuaToolSpec(tool) ? tool.declaration.description : tool.description; if (typeof raw !== "string") return undefined; @@ -59,13 +44,11 @@ function toolDescription(tool: CuaCliTool): string | undefined { export function describeTools(tools: readonly CuaCliTool[]): ToolSelectionItem[] { return tools.map((tool) => { const description = toolDescription(tool); - const atomicGroup = atomicGroupOf(tool); return { key: toolKey(tool), label: tool.name, group: toolGroup(tool), ...(description ? { description } : {}), - ...(atomicGroup ? { atomicGroup } : {}), }; }); } @@ -75,51 +58,25 @@ export function toolSearchText(item: ToolSelectionItem): string { return `${item.label} ${item.group} ${item.key}${item.description ? ` ${item.description}` : ""}`; } -/** Keys that must move together with `key` (itself included). */ -function linkedKeys(items: readonly ToolSelectionItem[], key: string): string[] { - const item = items.find((candidate) => candidate.key === key); - if (!item?.atomicGroup) return [key]; - return items.filter((candidate) => candidate.atomicGroup === item.atomicGroup).map((candidate) => candidate.key); -} - -/** - * Flip one row. Atomic groups move as a unit so a Yutori n1 selection can - * never be staged into a state the catalog compiler would reject. - */ -export function toggleTool( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - key: string, -): Set { +/** Flip one row. */ +export function toggleTool(enabled: ReadonlySet, key: string): Set { const next = new Set(enabled); - const keys = linkedKeys(items, key); - const turnOn = !enabled.has(key); - for (const linked of keys) { - if (turnOn) next.add(linked); - else next.delete(linked); - } + if (enabled.has(key)) next.delete(key); + else next.add(key); return next; } -/** Enable `keys` (expanding atomic groups). */ -export function enableTools( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - keys: readonly string[], -): Set { +/** Enable `keys`. */ +export function enableTools(enabled: ReadonlySet, keys: readonly string[]): Set { const next = new Set(enabled); - for (const key of keys) for (const linked of linkedKeys(items, key)) next.add(linked); + for (const key of keys) next.add(key); return next; } -/** Disable `keys` (expanding atomic groups). */ -export function disableTools( - enabled: ReadonlySet, - items: readonly ToolSelectionItem[], - keys: readonly string[], -): Set { +/** Disable `keys`. */ +export function disableTools(enabled: ReadonlySet, keys: readonly string[]): Set { const next = new Set(enabled); - for (const key of keys) for (const linked of linkedKeys(items, key)) next.delete(linked); + for (const key of keys) next.delete(key); return next; } diff --git a/packages/cli/src/tui/tools-picker.ts b/packages/cli/src/tui/tools-picker.ts index ed11395..3f5edab 100644 --- a/packages/cli/src/tui/tools-picker.ts +++ b/packages/cli/src/tui/tools-picker.ts @@ -177,11 +177,6 @@ export class ToolsPickerComponent extends Container implements Focusable { if (selected.description) { this.listContainer.addChild(new Text(colors.muted(` ${selected.description}`), 0, 0)); } - if (selected.atomicGroup) { - this.listContainer.addChild( - new Text(colors.warning(" toggles as a group: the provider rejects partial native action sets"), 0, 0), - ); - } } } @@ -213,20 +208,20 @@ export class ToolsPickerComponent extends Container implements Focusable { if (kb.matches(data, "tui.select.confirm") || (data === " " && !this.searchInput.getValue())) { const item = this.filtered[this.selectedIndex]; if (item) { - this.staged = toggleTool(this.staged, this.items, item.key); + this.staged = toggleTool(this.staged, item.key); this.refresh(); this.tui.requestRender(); } return; } if (kb.matches(data, "cua.tools.enableAll")) { - this.staged = enableTools(this.staged, this.items, this.bulkTargets()); + this.staged = enableTools(this.staged, this.bulkTargets()); this.refresh(); this.tui.requestRender(); return; } if (kb.matches(data, "cua.tools.clearAll")) { - this.staged = disableTools(this.staged, this.items, this.bulkTargets()); + this.staged = disableTools(this.staged, this.bulkTargets()); this.refresh(); this.tui.requestRender(); return; diff --git a/packages/cli/test/cli-executor.test.ts b/packages/cli/test/cli-executor.test.ts index 543c969..039860c 100644 --- a/packages/cli/test/cli-executor.test.ts +++ b/packages/cli/test/cli-executor.test.ts @@ -31,8 +31,6 @@ const PROVIDER_ENV_KEYS = [ "META_API_KEY", "XAI_API_KEY", "MOONSHOT_API_KEY", - "TZAFON_API_KEY", - "YUTORI_API_KEY", ]; function baseFlags(overrides: Partial = {}): HarnessCliFlags { diff --git a/packages/cli/test/harness-assembly.test.ts b/packages/cli/test/harness-assembly.test.ts index d6e9250..4f945d0 100644 --- a/packages/cli/test/harness-assembly.test.ts +++ b/packages/cli/test/harness-assembly.test.ts @@ -4,7 +4,6 @@ import { InMemorySessionRepo, type Skill, } from "@onkernel/cua-agent"; -import { cua } from "@onkernel/cua-ai"; import { tmpdir } from "node:os"; import { mkdtempSync } from "node:fs"; import { join } from "node:path"; @@ -36,11 +35,6 @@ describe("buildCuaHarness", () => { expect(kimiNames).not.toContain("browser_act"); expect(kimiNames).toContain("browser_wait_for"); } - expect(defaultInteractionTools("tzafon:tzafon.northstar-cua-fast")[0]?.name).toBe("computer"); - expect(defaultInteractionTools("yutori:n1.5-latest").map((tool) => tool.name)).toEqual([ - ...cua.providers.yutori.toolsets.n15Core().map((tool) => tool.name), - "computer_screenshot", - ]); }); it("installs interaction and coding tools in one explicit default list", async () => { diff --git a/packages/cli/test/tool-revalidation.test.ts b/packages/cli/test/tool-revalidation.test.ts index ad3f5cf..de67962 100644 --- a/packages/cli/test/tool-revalidation.test.ts +++ b/packages/cli/test/tool-revalidation.test.ts @@ -24,29 +24,25 @@ describe("/tools selection revalidation", () => { expect(fixture.harness.getTools().map(toolKey)).not.toContain(dropped.key); }); - it("rejects a partial Yutori n1 native subset and leaves the catalog unchanged", async () => { - const modelRef = "yutori:n1-latest"; + it("rejects a duplicated tool name and leaves the catalog unchanged", async () => { + const modelRef = "google:gemini-3.6-flash"; const baseline = [...defaultInteractionTools(modelRef), ...defaultApplicationTools()]; const fixture = await buildTestHarness({ turns: [], modelRef, tools: baseline }); const before = fixture.harness.getTools().map(toolKey); - const items = describeTools(baseline); - const oneNative = items.find((item) => item.atomicGroup)!; - const partial = baseline.filter((tool) => toolKey(tool) !== oneNative.key); - - await expect(fixture.harness.setTools(partial)).rejects.toThrow(/partial native action set/); + const [first] = baseline; + await expect(fixture.harness.setTools([...baseline, first!])).rejects.toThrow(/requested more than once/); // Atomicity: the failed compile must not have mutated live state. expect(fixture.harness.getTools().map(toolKey)).toEqual(before); }); - it("accepts dropping the whole Yutori n1 native group", async () => { - const modelRef = "yutori:n1-latest"; + it("accepts dropping the whole Google native group", async () => { + const modelRef = "google:gemini-3.6-flash"; const baseline = [...defaultInteractionTools(modelRef), ...defaultApplicationTools()]; const fixture = await buildTestHarness({ turns: [], modelRef, tools: baseline }); - const items = describeTools(baseline); - const atomicKeys = new Set(items.filter((item) => item.atomicGroup).map((item) => item.key)); - const next = baseline.filter((tool) => !atomicKeys.has(toolKey(tool))); + const nativeKeys = new Set(describeTools(baseline).filter((item) => item.group === "native").map((item) => item.key)); + const next = baseline.filter((tool) => !nativeKeys.has(toolKey(tool))); await fixture.harness.setTools(next); expect(fixture.harness.getTools().map(toolKey)).toEqual(next.map(toolKey)); diff --git a/packages/cli/test/tool-selection.test.ts b/packages/cli/test/tool-selection.test.ts index 7ada4e1..f3e7b08 100644 --- a/packages/cli/test/tool-selection.test.ts +++ b/packages/cli/test/tool-selection.test.ts @@ -46,20 +46,6 @@ describe("describeTools", () => { const cuaItems = describeTools(defaultInteractionTools("openai:gpt-5.6-sol")); expect(cuaItems.every((item) => item.group === "cua")).toBe(true); }); - - it("marks the Yutori n1 native action set as one atomic group", () => { - const items = describeTools(defaultInteractionTools("yutori:n1-latest")); - const atomic = items.filter((item) => item.atomicGroup); - expect(atomic.length).toBeGreaterThan(1); - expect(new Set(atomic.map((item) => item.atomicGroup))).toEqual(new Set(["provider.yutori.native.n1"])); - // The screenshot helper the CLI appends is not part of the native set. - expect(items.some((item) => !item.atomicGroup)).toBe(true); - }); - - it("leaves Yutori n1.5 individually toggleable", () => { - const items = describeTools(defaultInteractionTools("yutori:n1.5-latest")); - expect(items.every((item) => item.atomicGroup === undefined)).toBe(true); - }); }); describe("toolSearchText", () => { @@ -79,45 +65,24 @@ describe("selection state machine", () => { it("toggles a single tool off and back on", () => { const target = allKeys[0]!; - const off = toggleTool(new Set(allKeys), items, target); + const off = toggleTool(new Set(allKeys), target); expect(off.has(target)).toBe(false); expect(off.size).toBe(allKeys.length - 1); - const on = toggleTool(off, items, target); + const on = toggleTool(off, target); expect(sameSelection(on, new Set(allKeys))).toBe(true); }); it("enables and clears in bulk", () => { - expect(disableTools(new Set(allKeys), items, allKeys).size).toBe(0); - expect(sameSelection(enableTools(new Set(), items, allKeys), new Set(allKeys))).toBe(true); + expect(disableTools(new Set(allKeys), allKeys).size).toBe(0); + expect(sameSelection(enableTools(new Set(), allKeys), new Set(allKeys))).toBe(true); }); it("restricts bulk actions to the keys it is given", () => { const subset = allKeys.slice(0, 2); - const cleared = disableTools(new Set(allKeys), items, subset); + const cleared = disableTools(new Set(allKeys), subset); expect(cleared.size).toBe(allKeys.length - 2); for (const key of subset) expect(cleared.has(key)).toBe(false); }); - - it("moves an atomic group as one unit", () => { - const yutori = describeTools(defaultInteractionTools("yutori:n1-latest")); - const yutoriKeys = yutori.map((item) => item.key); - const nativeKeys = yutori.filter((item) => item.atomicGroup).map((item) => item.key); - const standalone = yutori.filter((item) => !item.atomicGroup).map((item) => item.key); - - const off = toggleTool(new Set(yutoriKeys), yutori, nativeKeys[0]!); - for (const key of nativeKeys) expect(off.has(key)).toBe(false); - for (const key of standalone) expect(off.has(key)).toBe(true); - - const on = toggleTool(off, yutori, nativeKeys[1]!); - for (const key of nativeKeys) expect(on.has(key)).toBe(true); - }); - - it("expands atomic groups for bulk disables too", () => { - const yutori = describeTools(defaultInteractionTools("yutori:n1-latest")); - const first = yutori.find((item) => item.atomicGroup)!; - const cleared = disableTools(new Set(yutori.map((i) => i.key)), yutori, [first.key]); - expect(yutori.filter((i) => i.atomicGroup).every((i) => !cleared.has(i.key))).toBe(true); - }); }); describe("sameSelection", () => { diff --git a/skills/cua-cli/SKILL.md b/skills/cua-cli/SKILL.md index 589ad65..bd761c1 100644 --- a/skills/cua-cli/SKILL.md +++ b/skills/cua-cli/SKILL.md @@ -146,10 +146,9 @@ Useful flags: - `-m ` — pick the LLM for model-mediated subcommands (default `gpt-5.6-sol`). Recommended refs are `openai:gpt-5.6-sol`, `anthropic:claude-opus-5`, `google:gemini-3.6-flash`, `meta:muse-spark-1.1`, `xai:grok-4.5`, - `moonshotai:kimi-k3`, `tzafon:tzafon.northstar-cua-fast`, and - `yutori:n1.5-latest`. + and `moonshotai:kimi-k3`. - `cua models` — list supported `-m` values and their providers; filter with - `cua models -p openai|anthropic|google|meta|xai|moonshotai|tzafon|yutori`. + `cua models -p openai|anthropic|google|meta|xai|moonshotai|openrouter`. `gemini` aliases `google`, and `moonshot` aliases `moonshotai`. Model refs print as `provider:model`; `-m` accepts either the full ref or a bare model id that matches exactly one entry. @@ -172,10 +171,9 @@ Useful flags: The CLI selects its interaction tools from the model: structured CUA browser primitives plus `browser_act` verified plans for OpenAI, Meta, xAI, and older Anthropic models; browser primitives alone for Moonshot, whose API rejects -`browser_act`'s schema; native browser tools for current Anthropic and -Google models; Tzafon's native computer tool; and Yutori's documented native -set plus an explicit screenshot tool. It also appends workspace coding tools in -`--print`, TUI, and model-mediated action runs. +`browser_act`'s schema; and native browser tools for current Anthropic and +Google models. It also appends workspace coding tools in `--print`, TUI, and +model-mediated action runs. There is no `--mode`, `--native-tool`, or `--playwright` flag. Those catalogs remain explicit SDK choices rather than CLI defaults. The CLI also does not @@ -304,8 +302,8 @@ testing. It can only remove from that list, never add unsupported tools. Edits are staged — nothing applies until `ctrl+s`, and cancel leaves live state untouched. A selection rejected by catalog validation reports the error and changes nothing. Selections are session-only and are reset to the new model's -defaults by `/model`. Yutori n1's native set toggles as one group; disabling -everything is allowed and yields a text-only agent. +defaults by `/model`. Disabling everything is allowed and yields a text-only +agent. ## Don't forget From 13e312b1a51a207fb782289f8a33517f00d0d837 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Thu, 13 Aug 2026 23:46:09 +0000 Subject: [PATCH 6/7] Remove routeCuaApi With Tzafon and Yutori gone it routed no transport at all: the transport comes from the selected tools, and its last branch only patched grok-4.5's thinking-level map, price tiers, and compat flags onto pi-ai's registry entry. Live runs on grok-4.5 at the default, off, and xhigh thinking levels behave identically without those patches, so model resolution now returns pi-ai's data unmodified. The one detail that goes with it is a >200k-token price tier pi's registry does not carry, which affects usage.cost reporting on long requests and nothing else. --- packages/ai/CHANGELOG.md | 9 +++++++++ packages/ai/src/models.ts | 25 ++----------------------- packages/ai/src/tool-catalog.ts | 4 ++-- packages/ai/test/models.test.ts | 11 ++++++----- 4 files changed, 19 insertions(+), 30 deletions(-) diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 7b50ef0..37b8222 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -2,6 +2,15 @@ ## 0.13.0 - 2026-08-13 +- Remove `routeCuaApi`. With Tzafon and Yutori gone it routed no transport at + all, and its last branch only patched grok-4.5's thinking-level map, price + tiers, and compat flags onto pi-ai's registry entry. Live runs on grok-4.5 + at the default, `off`, and `xhigh` thinking levels behave identically + without it, so model resolution now returns pi-ai's data unmodified. The + only lost detail is a >200k-token price tier that pi's registry does not + carry, which affects `usage.cost` reporting for long requests and nothing + else. + Breaking: Tzafon and Yutori support is removed. - Remove the `tzafon` and `yutori` providers: their `CuaProvider` members, diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index 7b44cd2..f3187ca 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -197,33 +197,12 @@ export function getCuaModel(ref: CuaModelRef): Model { throw new Error(`unsupported CUA model "${ref}"`); } const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; - if (fromRegistry) return routeCuaApi(fromRegistry); + if (fromRegistry) return fromRegistry; const override = CUA_MODEL_OVERRIDES[provider].find((m) => m.id === modelId); - if (override) return routeCuaApi(override); + if (override) return override; throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`); } -// Apply the model overrides that are properties of the model itself, not of -// which tools a caller selects. Tool-driven transport selection (OpenAI's -// native computer tool, Google's Interactions API) is derived by -// compileCuaToolCatalog from the selected tools' provider bindings instead; -// see CuaProviderBinding.requiresApi. What remains here is model-only: -// grok-4.5 carries cost/compat/thinking-level overrides pi-ai's registry does -// not have yet. -export function routeCuaApi(model: Model): Model { - if (model.provider === "xai" && model.id === "grok-4.5") { - return { - ...model, - thinkingLevelMap: { off: "low", minimal: "low", xhigh: "high" }, - cost: { - ...model.cost, - tiers: [{ inputTokensAbove: 200_000, input: 4, output: 12, cacheRead: 1, cacheWrite: 0 }], - }, - compat: { supportsDeveloperRole: false, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: false }, - }; - } - return model; -} /** Return the {@link CuaProvider} for a concrete model, or throw when it is not a CUA provider. */ export function providerForModel(model: Model): CuaProvider { diff --git a/packages/ai/src/tool-catalog.ts b/packages/ai/src/tool-catalog.ts index c608041..206dea2 100644 --- a/packages/ai/src/tool-catalog.ts +++ b/packages/ai/src/tool-catalog.ts @@ -1,7 +1,7 @@ import type { Api, Model, Tool } from "@earendil-works/pi-ai"; import type { CuaAction } from "./actions/index"; import type { CuaModelRef } from "./models"; -import { cuaModelCapabilities, getCuaModel, providerForModel, routeCuaApi } from "./models"; +import { cuaModelCapabilities, getCuaModel, providerForModel } from "./models"; import { anthropicAdaptiveThinkingOnPayload } from "./providers/anthropic/adaptive-thinking"; import { supportsAnthropicNativeBrowser, @@ -206,7 +206,7 @@ const SAFE_TOOL_NAME = /^[A-Za-z][A-Za-z0-9_-]{0,63}$/; export function compileCuaToolCatalog(options: CompileCuaToolCatalogOptions): CuaToolCatalog { const baseModel = typeof options.model === "string" ? getCuaModel(options.model) - : routeCuaApi(resetCatalogDerivedApi(options.model)); + : resetCatalogDerivedApi(options.model); const normalizedEntries = [...options.requestedTools].map(normalizeTool); const requiresApi = validateCatalog(baseModel, normalizedEntries); const model = requiresApi ? { ...baseModel, api: requiresApi } : baseModel; diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index f8041e2..e608b6f 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { getBuiltinModel } from "@earendil-works/pi-ai/providers/all"; import { CUA_MODEL_ANNOTATIONS, CUA_PROVIDERS, @@ -78,7 +79,7 @@ describe("CUA model refs", () => { expect(muse.thinkingLevelMap?.off).toBeNull(); }); - it("uses pi-ai's Grok catalog entry with CUA routing overrides", () => { + it("returns pi-ai's Grok catalog entry unmodified", () => { expect(cuaOverrideModels("xai")).toEqual([]); const grok = getCuaModel("xai:grok-4.5"); expect(grok.provider).toBe("xai"); @@ -86,10 +87,10 @@ describe("CUA model refs", () => { expect(grok.baseUrl).toBe("https://api.x.ai/v1"); expect(grok.contextWindow).toBe(500_000); expect(grok.maxTokens).toBe(500_000); - expect(grok.thinkingLevelMap).toEqual({ off: "low", minimal: "low", xhigh: "high" }); - expect(grok.cost.tiers).toEqual([ - { inputTokensAbove: 200_000, input: 4, output: 12, cacheRead: 1, cacheWrite: 0 }, - ]); + // pi's registry data is used as-is: no CUA-owned thinking-level, cost, or + // compat patching survives model resolution. + expect(grok.thinkingLevelMap).toEqual(getBuiltinModel("xai", "grok-4.5").thinkingLevelMap); + expect(grok.cost.tiers).toBeUndefined(); }); it("uses pi-ai's Kimi catalog entries for both transports", () => { From 13fa4e633c6eecaa8abd53f081a8e58aaab439d2 Mon Sep 17 00:00:00 2001 From: rgarcia <72655+rgarcia@users.noreply.github.com> Date: Fri, 14 Aug 2026 00:12:17 +0000 Subject: [PATCH 7/7] Remove the Meta provider pi-ai ships no `meta` provider, so cua hand-wrote a model entry and pointed pi's own `openai-responses` transport at `api.meta.ai`. Nothing about that was Meta-specific except a base URL, an env var, and the model entry itself. Meta was the last user of the model-override mechanism, so `CUA_MODEL_OVERRIDES`, `cuaOverrideModels()`, and the private `cuaModel()` helper go with it, and `getCuaModel` no longer needs a "supported but not registered" fallback. Muse Spark stays reachable: it is in pi-ai's OpenRouter catalog as `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities because OpenRouter's provider-level defaults are conservative. That exposed a real bug the provider switch was hiding. OpenRouter fronts several model families, and the CLI assumed one toolset per provider: Kimi K3 rejects `browser_act`'s schema while Muse Spark accepts it. The default interaction toolset for OpenRouter now asks the model's capabilities instead. --- .agents/skills/update-models/SKILL.md | 7 +- .../skills/update-models/reference/README.md | 1 - .../reference/audit-official-examples.ts | 11 +-- .../reference/discover-models.ts | 31 +-------- .../reference/provider-doc-drift.ts | 16 +---- .../update-models/reference/report-schema.md | 1 - README.md | 5 +- packages/agent/README.md | 2 +- packages/agent/examples/shared/tools.ts | 12 ++-- packages/agent/test/e2e.live.test.ts | 17 +---- .../test/example-provider-matrix.test.ts | 4 +- packages/ai/CHANGELOG.md | 9 +++ packages/ai/README.md | 2 +- packages/ai/docs/supported-models.md | 34 ++++------ packages/ai/src/api-keys.ts | 1 - packages/ai/src/models.ts | 68 ++++--------------- packages/ai/src/providers.ts | 16 ----- packages/ai/test/api-keys.test.ts | 4 -- packages/ai/test/models.test.ts | 37 +++------- packages/ai/test/providers.test.ts | 16 ++--- packages/ai/test/tool-catalog.test.ts | 8 +-- packages/cli/CHANGELOG.md | 7 ++ packages/cli/README.md | 3 +- packages/cli/src/cli-harness.ts | 2 +- packages/cli/src/cli.ts | 2 - packages/cli/src/harness.ts | 11 +-- packages/cli/test/harness-assembly.test.ts | 2 +- packages/cli/test/harness-models.test.ts | 7 +- skills/cua-cli/SKILL.md | 2 +- 29 files changed, 103 insertions(+), 235 deletions(-) diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md index 5be03cd..1cad940 100644 --- a/.agents/skills/update-models/SKILL.md +++ b/.agents/skills/update-models/SKILL.md @@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer- ## Quick Start -1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. +1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. 2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them: ```bash eval "$(python3 - <<'PY' import pathlib, re, shlex text = pathlib.Path('~/AGENTS.md').expanduser().read_text() -for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: +for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']: m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text) if m: print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}') @@ -70,7 +70,6 @@ When live discovery finds a new model with passing smoke tests, update `packages Meta: -- Discover with the OpenAI SDK against `https://api.meta.ai/v1` using `META_API_KEY`. - Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions. - Pass condition: response output contains a `function_call` for one of the supplied browser actions. - Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`. @@ -129,7 +128,6 @@ Moonshot: Run action probes when updating adapters or when docs/examples show drift: ```bash -npx tsx .agents/skills/update-models/reference/discover-models.ts --provider meta --models muse-spark-1.1 npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 @@ -174,7 +172,6 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W - Update the snapshot in `packages/ai/docs/supported-models.md` to match. - New provider-native action, response field, or tool version: - - Meta: update `packages/ai/src/providers/meta/index.ts` and `provider.ts`, including Responses threading and reasoning compatibility. - OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes. - Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package. - Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed. diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md index d6ca42b..2c0bca4 100644 --- a/.agents/skills/update-models/reference/README.md +++ b/.agents/skills/update-models/reference/README.md @@ -11,7 +11,6 @@ These scripts support the `update-models` skill. Run them from the repository ro - `OPENAI_API_KEY` - `ANTHROPIC_API_KEY` - `GOOGLE_API_KEY` or `GEMINI_API_KEY` - - `META_API_KEY` - `XAI_API_KEY` - `MOONSHOT_API_KEY` diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts index 524c60a..330b53b 100644 --- a/.agents/skills/update-models/reference/audit-official-examples.ts +++ b/.agents/skills/update-models/reference/audit-official-examples.ts @@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path"; import { spawnSync } from "node:child_process"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface ExampleRepo { provider: Provider; @@ -51,14 +51,6 @@ const EXAMPLES: ExampleRepo[] = [ confidence: "provider-owned", patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"], }, - { - provider: "meta", - name: "meta-model-cookbook", - repo: "https://github.com/meta-models/meta-model-cookbook.git", - confidence: "provider-owned", - pathHint: "03_use_cases/13_macos_cua", - patterns: ["muse-spark-1.1", "function_call", "function_call_output", "previous_response_id", "reasoning.encrypted_content", "parallel_tool_calls"], - }, { provider: "xai", name: "xai-sdk-python", @@ -73,7 +65,6 @@ const ACTION_REGEXES: Record = { openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g], anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g], gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g], - meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g], xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], }; diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts index 0fa9952..8411400 100644 --- a/.agents/skills/update-models/reference/discover-models.ts +++ b/.agents/skills/update-models/reference/discover-models.ts @@ -4,7 +4,7 @@ import { readFile, writeFile } from "node:fs/promises"; import { join } from "node:path"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface Args { provider: Provider | "all"; @@ -37,7 +37,7 @@ interface ModelResult { cua?: Record; } -const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot"]; +const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "xai", "moonshot"]; const GEMINI_DOC_COMPUTER_USE_MODELS = [ "gemini-3.5-flash", "gemini-3-flash-preview", @@ -99,7 +99,7 @@ function usage(): never { npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4 Options: - --provider + --provider --models Smoke-test explicit models instead of inferred candidates. --candidate-limit Max inferred candidates per provider. Default: 20. --no-smoke Only list metadata. @@ -127,7 +127,6 @@ async function runProvider(provider: Provider, args: Args): Promise> { return { provider: "openai", metadata_source: "client.models.list()", models, candidates }; } -async function discoverMeta(args: Args): Promise> { - const OpenAI = await importDefault("openai", "OpenAI"); - const apiKey = process.env.META_API_KEY; - const client = new OpenAI({ apiKey, baseURL: "https://api.meta.ai/v1" }); - const rawModels = await collectAsync(client.models.list()); - const models: ModelResult[] = rawModels.map((m) => ({ - id: String(m.id), - display_name: String(m.id), - created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, - raw: m, - supports_generation: true, - })); - const candidates = explicitOrCandidates(args, models.map((model) => model.id)); - if (args.smoke) { - await Promise.all(candidates.map(async (id) => { - const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; - model.computer_use = await smokeMeta(client, id); - if (!models.find((candidate) => candidate.id === id)) models.unshift(model); - })); - } - await annotateCuaSupport("meta", models); - return { provider: "meta", metadata_source: "Meta Model API models.list()", models, candidates }; -} - async function smokeMeta(client: any, model: string): Promise { try { const screenshot = await readFile(fixtureScreenshotPath()); diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts index 070dd51..1dfef98 100644 --- a/.agents/skills/update-models/reference/provider-doc-drift.ts +++ b/.agents/skills/update-models/reference/provider-doc-drift.ts @@ -2,7 +2,7 @@ import { readFile, writeFile } from "node:fs/promises"; import process from "node:process"; -type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot"; +type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot"; interface Args { examples: string; @@ -30,11 +30,6 @@ const DOCS: Record = { "https://ai.google.dev/gemini-api/docs/computer-use", "https://ai.google.dev/api/models", ], - meta: [ - "https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos", - "https://dev.meta.ai/docs/features/responses", - "https://dev.meta.ai/docs/features/tool-calling", - ], xai: [ "https://docs.x.ai/developers/grok-4-5", "https://docs.x.ai/developers/tools/function-calling", @@ -51,7 +46,6 @@ const LOCAL_FILES: Record = { openai: "packages/ai/src/providers/openai/index.ts", anthropic: "packages/ai/src/providers/anthropic/actions.ts", gemini: "packages/ai/src/providers/gemini/index.ts", - meta: "packages/ai/src/providers/meta/index.ts", xai: "packages/ai/src/providers/xai/index.ts", moonshot: "packages/ai/src/providers/moonshot/index.ts", }; @@ -60,7 +54,6 @@ const ACTION_REGEXES: Record = { openai: /\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g, anthropic: /\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g, gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g, - meta: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position|left_click|right_click|middle_click|triple_click|left_click_drag|mouse_move|key|hold_key|left_mouse_down|left_mouse_up)\b/g, xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g, }; @@ -136,8 +129,8 @@ async function checkProvider(provider: Provider, examples: any): Promise { ); it("still advertises browser_act where the provider accepts it", () => { - for (const model of ["openai:gpt-5.6-sol", "meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["openai:gpt-5.6-sol", "xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { expect(toolsForModel(model).map((tool) => tool.name), model).toContain("browser_act"); } }); diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 37b8222..f1e3d6a 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -2,6 +2,15 @@ ## 0.13.0 - 2026-08-13 +- Remove the Meta provider. pi-ai ships no `meta` provider, so cua hand-wrote a + model entry and pointed pi's own `openai-responses` transport at + `api.meta.ai`. Meta was the last user of the model-override mechanism, so + `CUA_MODEL_OVERRIDES`, `cuaOverrideModels()`, and `META_API_KEY` go with it, + and `getCuaModel` no longer has a "supported but not registered" fallback. + Muse Spark remains available through pi's OpenRouter catalog as + `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities + because OpenRouter's provider-level defaults are conservative. + - Remove `routeCuaApi`. With Tzafon and Yutori gone it routed no transport at all, and its last branch only patched grok-4.5's thinking-level map, price tiers, and compat flags onto pi-ai's registry entry. Live runs on grok-4.5 diff --git a/packages/ai/README.md b/packages/ai/README.md index 3f615cb..d7e7b20 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -269,7 +269,7 @@ import { ``` Conventional variables are `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, -`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, and +`GOOGLE_API_KEY`/`GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`. ## Development diff --git a/packages/ai/docs/supported-models.md b/packages/ai/docs/supported-models.md index 5b5f20e..6c80220 100644 --- a/packages/ai/docs/supported-models.md +++ b/packages/ai/docs/supported-models.md @@ -72,19 +72,6 @@ model/tool surfaces are intentionally not exposed. Source: [Gemini computer use docs](https://ai.google.dev/gemini-api/docs/computer-use). -## `meta` - -CLI default interaction: CUA browser primitives plus explicit `browser_act`. - -Exact IDs: - -- `muse-spark-1.1` - -Muse Spark uses Meta's OpenAI-compatible Responses API with ordinary function -tools. CUA continues tool loops through `previous_response_id`. - -Source: [Meta computer-use cookbook](https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos). - ## `xai` CLI default interaction: CUA browser primitives plus explicit `browser_act`. @@ -123,16 +110,21 @@ Source: [Kimi K3 announcement](https://www.kimi.com/blog/kimi-k3), [tool use](ht ## `openrouter` -CLI default interaction: CUA browser primitives only. OpenRouter's Kimi K3 -route accepts complex function schemas but rejects the larger `browser_act` -schema, and state-mutating calls are serialized. +CLI default interaction: per model, not per provider. OpenRouter fronts several +model families, so the CLI asks each model whether it accepts `browser_act`'s +schema rather than assuming one answer for the whole provider. Model refs use the `openrouter:` prefix: -- `moonshotai/kimi-k3` +- `moonshotai/kimi-k3` — browser primitives only: accepts complex function + schemas but rejects the larger `browser_act` schema. State mutations are + serialized. +- `meta/muse-spark-1.1` — browser primitives plus explicit `browser_act`. State + mutations are serialized. -Kimi K3 uses OpenRouter's OpenAI-compatible chat completions API with ordinary -CUA browser function tools. OpenRouter does not expose the provider-native -computer tools declared by other CUA providers. +Both use OpenRouter's OpenAI-compatible chat completions API with ordinary CUA +browser function tools. OpenRouter does not expose the provider-native computer +tools declared by other CUA providers. -Source: [OpenRouter model page](https://openrouter.ai/moonshotai/kimi-k3). +Sources: [Kimi K3](https://openrouter.ai/moonshotai/kimi-k3), +[Muse Spark 1.1](https://openrouter.ai/meta/muse-spark-1.1). diff --git a/packages/ai/src/api-keys.ts b/packages/ai/src/api-keys.ts index 241ea57..70f9138 100644 --- a/packages/ai/src/api-keys.ts +++ b/packages/ai/src/api-keys.ts @@ -12,7 +12,6 @@ const CUA_PROVIDER_API_KEY_ENV_VARS: Record = { openai: ["OPENAI_API_KEY"], anthropic: ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"], google: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], - meta: ["META_API_KEY"], xai: ["XAI_API_KEY"], moonshotai: ["MOONSHOT_API_KEY"], openrouter: ["OPENROUTER_API_KEY"], diff --git a/packages/ai/src/models.ts b/packages/ai/src/models.ts index f3187ca..e023085 100644 --- a/packages/ai/src/models.ts +++ b/packages/ai/src/models.ts @@ -2,7 +2,7 @@ import type { Api, Model } from "@earendil-works/pi-ai"; import { getBuiltinModel, getBuiltinModels } from "@earendil-works/pi-ai/providers/all"; /** Providers with curated computer-use model support. */ -export type CuaProvider = "openai" | "anthropic" | "google" | "meta" | "xai" | "moonshotai" | "openrouter"; +export type CuaProvider = "openai" | "anthropic" | "google" | "xai" | "moonshotai" | "openrouter"; /** Provider-qualified model reference, e.g. `"openai:gpt-5.6-sol"` or `"google:gemini-3.6-flash"`. */ export type CuaModelRef = `${CuaProvider}:${string}`; @@ -19,7 +19,7 @@ export interface CuaModelInfo { } /** All providers this package curates computer-use models for. */ -export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter"]; +export const CUA_PROVIDERS: readonly CuaProvider[] = ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]; /** * How a {@link CuaModelAnnotation} matches model ids. @@ -50,6 +50,14 @@ export interface CuaModelAnnotation { readonly capabilities?: CuaModelCapabilities; } +// Muse Spark accepts the full CUA schema set; OpenRouter's provider-level +// defaults are conservative because the proxy fronts many model families. +const MUSE_SPARK_CAPABILITIES: CuaModelCapabilities = Object.freeze({ + acceptsComplexSchemas: true, + acceptsLargeSchemas: true, + serializesStateMutations: true, +}); + const KIMI_K3_CAPABILITIES: CuaModelCapabilities = Object.freeze({ acceptsComplexSchemas: true, acceptsLargeSchemas: false, @@ -89,9 +97,6 @@ export const CUA_MODEL_ANNOTATIONS: Record so getCuaModel() can return it directly -// without synthesizing fields at call time. Add an entry here when a provider -// ships a new model before pi-ai picks it up — and add a matching annotation -// in CUA_MODEL_ANNOTATIONS above so the support filter recognizes it. -const CUA_MODEL_OVERRIDES: Record[]> = { - openai: [], - anthropic: [], - google: [], - // pi-ai still lacks Meta's models.dev catalog entry. - meta: [cuaModel("meta", "muse-spark-1.1", "Muse Spark 1.1")], - xai: [], - moonshotai: [], - openrouter: [], -}; - -/** Models CUA supports that pi-ai's registry does not carry for a provider. */ -export function cuaOverrideModels(provider: CuaProvider): readonly Model[] { - return CUA_MODEL_OVERRIDES[provider]; -} - /** * Split a provider-qualified ref like `"openai:gpt-5.6-sol"` into its parts. * @@ -164,10 +149,6 @@ export function listCuaModels(provider?: CuaProvider): CuaModelInfo[] { const byRef = new Map(); for (const p of providers) { - for (const model of CUA_MODEL_OVERRIDES[p]) { - const ref = formatCuaModelRef(p, model.id); - byRef.set(ref, { ref, provider: p, model: model.id, name: model.name }); - } for (const model of getBuiltinModels(p as never) as Model[]) { if (!supportsCuaProvider(p, model.id)) continue; const ref = formatCuaModelRef(p, model.id); @@ -198,9 +179,7 @@ export function getCuaModel(ref: CuaModelRef): Model { } const fromRegistry = getBuiltinModel(provider as never, modelId as never) as Model | undefined; if (fromRegistry) return fromRegistry; - const override = CUA_MODEL_OVERRIDES[provider].find((m) => m.id === modelId); - if (override) return override; - throw new Error(`CUA model "${ref}" is supported but not registered. Add it to pi-ai (models.dev) or CUA_MODEL_OVERRIDES.`); + throw new Error(`CUA model "${ref}" is supported but not carried by pi-ai's registry`); } @@ -225,11 +204,11 @@ function supportsCuaProvider(provider: CuaProvider, modelId: string): boolean { export function cuaModelCapabilities(model: Model): CuaModelCapabilities { const annotation = isCuaProvider(model.provider) ? findCuaAnnotation(model.provider, model.id) : undefined; if (annotation?.capabilities) return annotation.capabilities; - const acceptsComplexSchemas = ["openai", "anthropic", "meta", "xai", "moonshotai"].includes(model.provider); + const acceptsComplexSchemas = ["openai", "anthropic", "xai", "moonshotai"].includes(model.provider); return { acceptsComplexSchemas, acceptsLargeSchemas: acceptsComplexSchemas && model.provider !== "moonshotai", - serializesStateMutations: ["meta", "xai", "moonshotai"].includes(model.provider), + serializesStateMutations: ["xai", "moonshotai"].includes(model.provider), }; } @@ -260,25 +239,6 @@ function isCuaFamilyMatch(id: string, family: string): boolean { .every((segment) => /^\d+$/.test(segment)); } -// Meta documents the 1,048,576-token context window, and its computer-use -// cookbook configures 128,000 maximum output tokens. -function cuaModel(provider: "meta", id: string, name: string): Model { - return { - id, - name, - provider, - reasoning: true, - input: ["text", "image"], - api: "openai-responses", - baseUrl: "https://api.meta.ai/v1", - thinkingLevelMap: { off: null, xhigh: "xhigh" }, - cost: { input: 1.25, output: 4.25, cacheRead: 0.15, cacheWrite: 0 }, - contextWindow: 1_048_576, - maxTokens: 128_000, - compat: { supportsDeveloperRole: true, sessionAffinityFormat: "openai-nosession", supportsLongCacheRetention: true }, - } as Model; -} - function compareCuaModels(a: CuaModelInfo, b: CuaModelInfo): number { if (a.provider !== b.provider) return CUA_PROVIDERS.indexOf(a.provider) - CUA_PROVIDERS.indexOf(b.provider); return a.model.localeCompare(b.model); diff --git a/packages/ai/src/providers.ts b/packages/ai/src/providers.ts index 4deaf72..3b19e65 100644 --- a/packages/ai/src/providers.ts +++ b/packages/ai/src/providers.ts @@ -10,13 +10,8 @@ import { type SimpleStreamOptions, type StreamOptions, } from "@earendil-works/pi-ai"; -import { - stream as piStreamOpenAIResponses, - streamSimple as piStreamSimpleOpenAIResponses, -} from "@earendil-works/pi-ai/api/openai-responses"; import { builtinModels } from "@earendil-works/pi-ai/providers/all"; import { cuaApiKeyEnvVarsForProvider } from "./api-keys"; -import { cuaOverrideModels } from "./models"; import { withAnthropicBrowserFallback } from "./providers/anthropic/browser-fallback"; import { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions } from "./providers/google/provider"; import { OPENAI_CUA_COMPUTER_API, requiresCuaOpenAINamespaceAdapter, streamOpenAICuaComputer, streamOpenAIResponses, streamSimpleOpenAIResponses } from "./providers/openai/provider"; @@ -57,7 +52,6 @@ export function createCuaModels(options?: CreateModelsOptions): MutableModels { if (openai) models.setProvider(withOpenAICuaAdapter(openai)); const google = models.getProvider("google"); if (google) models.setProvider(withGoogleCuaInteractions(google)); - models.setProvider(metaProvider()); return models; } @@ -113,16 +107,6 @@ function withGoogleCuaInteractions(base: Provider): Provider { } -function metaProvider(): Provider { - return createProvider({ - id: "meta", - name: "Meta", - baseUrl: "https://api.meta.ai/v1", - auth: { apiKey: envApiKeyAuth("Meta Model API key", cuaApiKeyEnvVarsForProvider("meta")) }, - models: cuaOverrideModels("meta"), - api: { "openai-responses": { stream: piStreamOpenAIResponses, streamSimple: piStreamSimpleOpenAIResponses } }, - }); -} export { GOOGLE_CUA_INTERACTIONS_API, streamGoogleInteractions, streamSimpleGoogleInteractions }; export { OPENAI_CUA_COMPUTER_API, streamOpenAIResponses, streamSimpleOpenAIResponses }; diff --git a/packages/ai/test/api-keys.test.ts b/packages/ai/test/api-keys.test.ts index a484791..770ca16 100644 --- a/packages/ai/test/api-keys.test.ts +++ b/packages/ai/test/api-keys.test.ts @@ -33,7 +33,6 @@ describe("cua api key helpers", () => { expect(cuaApiKeyEnvVarsForProvider("openai")).toEqual(["OPENAI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("google")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("gemini")).toEqual(["GOOGLE_API_KEY", "GEMINI_API_KEY"]); - expect(cuaApiKeyEnvVarsForProvider("meta")).toEqual(["META_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("xai")).toEqual(["XAI_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("moonshotai")).toEqual(["MOONSHOT_API_KEY"]); expect(cuaApiKeyEnvVarsForProvider("moonshot")).toEqual(["MOONSHOT_API_KEY"]); @@ -52,8 +51,6 @@ describe("cua api key helpers", () => { it("resolves keys from model refs", () => { process.env.OPENAI_API_KEY = "openai"; expect(getCuaEnvApiKeyForModel("openai:gpt-5.5")).toBe("openai"); - process.env.META_API_KEY = "meta"; - expect(getCuaEnvApiKeyForModel("meta:muse-spark-1.1")).toBe("meta"); process.env.XAI_API_KEY = "xai"; expect(getCuaEnvApiKeyForModel("xai:grok-4.5")).toBe("xai"); process.env.MOONSHOT_API_KEY = "moonshot"; @@ -64,6 +61,5 @@ describe("cua api key helpers", () => { it("throws readable errors when missing", () => { delete process.env.META_API_KEY; - expect(() => requireCuaEnvApiKey("meta")).toThrow("META_API_KEY"); }); }); diff --git a/packages/ai/test/models.test.ts b/packages/ai/test/models.test.ts index e608b6f..41a55a9 100644 --- a/packages/ai/test/models.test.ts +++ b/packages/ai/test/models.test.ts @@ -1,10 +1,8 @@ import { describe, expect, it } from "vitest"; -import { getBuiltinModel } from "@earendil-works/pi-ai/providers/all"; import { CUA_MODEL_ANNOTATIONS, CUA_PROVIDERS, type CuaModelRef, - cuaOverrideModels, findCuaAnnotation, formatCuaModelRef, getCuaModel, @@ -15,7 +13,7 @@ import { describe("CUA model refs", () => { it("parses and formats provider-qualified refs", () => { expect(parseCuaModelRef("openai:gpt-5.5")).toEqual({ provider: "openai", model: "gpt-5.5" }); - expect(formatCuaModelRef("meta", "muse-spark-1.1")).toBe("meta:muse-spark-1.1"); + expect(formatCuaModelRef("openrouter", "meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); }); it("rejects unqualified and unsupported refs", () => { @@ -26,7 +24,7 @@ describe("CUA model refs", () => { it("names the valid providers in the unsupported-provider error", () => { expect(() => parseCuaModelRef("bogus:model")).toThrow( - 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, meta, xai, moonshotai, openrouter)', + 'unsupported CUA provider "bogus" (expected one of: openai, anthropic, google, xai, moonshotai, openrouter)', ); }); @@ -50,9 +48,8 @@ describe("CUA model refs", () => { expect(models.some((model) => "origin" in model)).toBe(false); }); - it("returns override models for refs missing from pi-ai", () => { + it("returns pi-ai registry entries verbatim", () => { const opus = getCuaModel("anthropic:claude-opus-5"); - expect(cuaOverrideModels("anthropic")).toEqual([]); expect(opus).toMatchObject({ provider: "anthropic", api: "anthropic-messages", @@ -63,34 +60,24 @@ describe("CUA model refs", () => { }); expect(opus.compat).toMatchObject({ forceAdaptiveThinking: true, supportsTemperature: false }); - expect(cuaOverrideModels("google")).toEqual([]); expect(getCuaModel("google:gemini-3.6-flash")).toMatchObject({ provider: "google", api: "google-generative-ai", contextWindow: 1_048_576, }); - const muse = getCuaModel("meta:muse-spark-1.1"); - expect(muse.provider).toBe("meta"); - expect(muse.api).toBe("openai-responses"); - expect(muse.baseUrl).toBe("https://api.meta.ai/v1"); - expect(muse.contextWindow).toBe(1_048_576); - expect(muse.maxTokens).toBe(128_000); - expect(muse.thinkingLevelMap?.off).toBeNull(); + const muse = getCuaModel("openrouter:meta/muse-spark-1.1"); + expect(muse.provider).toBe("openrouter"); + expect(muse.baseUrl).toBe("https://openrouter.ai/api/v1"); }); - it("returns pi-ai's Grok catalog entry unmodified", () => { - expect(cuaOverrideModels("xai")).toEqual([]); + it("returns pi-ai's Grok catalog entry", () => { const grok = getCuaModel("xai:grok-4.5"); expect(grok.provider).toBe("xai"); expect(grok.api).toBe("openai-responses"); expect(grok.baseUrl).toBe("https://api.x.ai/v1"); expect(grok.contextWindow).toBe(500_000); expect(grok.maxTokens).toBe(500_000); - // pi's registry data is used as-is: no CUA-owned thinking-level, cost, or - // compat patching survives model resolution. - expect(grok.thinkingLevelMap).toEqual(getBuiltinModel("xai", "grok-4.5").thinkingLevelMap); - expect(grok.cost.tiers).toBeUndefined(); }); it("uses pi-ai's Kimi catalog entries for both transports", () => { @@ -101,8 +88,7 @@ describe("CUA model refs", () => { expect(listCuaModels("openrouter").map((model) => model.ref)).toContain("openrouter:moonshotai/kimi-k3"); }); - it("uses pi-ai's Kimi catalog entry without CUA routing overrides", () => { - expect(cuaOverrideModels("moonshotai")).toEqual([]); + it("uses pi-ai's Kimi catalog entry", () => { const kimi = getCuaModel("moonshotai:kimi-k3"); expect(kimi.provider).toBe("moonshotai"); expect(kimi.api).toBe("openai-completions"); @@ -131,15 +117,14 @@ describe("CUA model refs", () => { expect(getCuaModel("openai:gpt-5.5").api).toBe("openai-responses"); expect(getCuaModel("openai:gpt-5.4-mini").api).toBe("openai-responses"); expect(getCuaModel("google:gemini-3.6-flash").api).toBe("google-generative-ai"); - expect(getCuaModel("meta:muse-spark-1.1").api).toBe("openai-responses"); expect(getCuaModel("xai:grok-4.5").api).toBe("openai-responses"); }); - it("rejects supported model IDs that are not in pi-ai or overrides", () => { + it("rejects supported model IDs that pi-ai does not carry", () => { // Dated snapshots match the family annotation but pi-ai's registry // (generated from models.dev) only carries family roots. expect(() => getCuaModel("openai:gpt-5.5-2026-04-23")).toThrow( - /not registered/, + /not carried by pi-ai's registry/, ); }); }); @@ -190,7 +175,7 @@ describe("CUA support annotations", () => { expect(findCuaAnnotation("openai", "gpt-5.6-sol")?.match).toEqual({ kind: "exact", id: "gpt-5.6-sol" }); expect(findCuaAnnotation("openai", "gpt-5.6-sol-20260728")).toBeUndefined(); expect(findCuaAnnotation("google", "gemini-3.6-flash")).toBeDefined(); - expect(findCuaAnnotation("meta", "muse-spark-1.1")).toBeDefined(); + expect(findCuaAnnotation("openrouter", "meta/muse-spark-1.1")).toBeDefined(); expect(findCuaAnnotation("xai", "grok-4.5")).toBeDefined(); expect(findCuaAnnotation("xai", "grok-4.5-latest")).toBeUndefined(); expect(findCuaAnnotation("xai", "grok-4.3")).toBeUndefined(); diff --git a/packages/ai/test/providers.test.ts b/packages/ai/test/providers.test.ts index 429f34d..c38ca2d 100644 --- a/packages/ai/test/providers.test.ts +++ b/packages/ai/test/providers.test.ts @@ -2,9 +2,9 @@ import { describe, expect, it } from "vitest"; import { createCuaModels, cuaModels } from "../src/index"; describe("createCuaModels", () => { - it("registers the CUA-only providers alongside pi's builtins", () => { + it("registers a streamable provider for every CUA provider", () => { const models = createCuaModels(); - for (const id of ["openai", "anthropic", "google", "meta", "xai", "moonshotai", "openrouter"]) { + for (const id of ["openai", "anthropic", "google", "xai", "moonshotai", "openrouter"]) { const provider = models.getProvider(id); expect(provider, id).toBeDefined(); expect(provider?.stream).toBeTypeOf("function"); @@ -14,13 +14,11 @@ describe("createCuaModels", () => { it("lists CUA provider catalogs", () => { const models = createCuaModels(); - expect(models.getModel("meta", "muse-spark-1.1")?.api).toBe("openai-responses"); expect(models.getModel("xai", "grok-4.5")?.api).toBe("openai-responses"); expect(models.getModel("moonshotai", "kimi-k3")?.api).toBe("openai-completions"); expect(models.getModel("openrouter", "moonshotai/kimi-k3")?.api).toBe("openai-completions"); + expect(models.getModel("openrouter", "meta/muse-spark-1.1")?.api).toBe("openai-completions"); expect(models.getProvider("openrouter")?.baseUrl).toBe("https://openrouter.ai/api/v1"); - expect(models.getModels("meta").map((m) => m.id)).toContain("muse-spark-1.1"); - expect(models.getProvider("meta")?.baseUrl).toBe("https://api.meta.ai/v1"); }); it("keeps builtin catalogs on wrapped providers", () => { @@ -41,15 +39,15 @@ describe("createCuaModels", () => { it("returns independent collections", () => { const a = createCuaModels(); const b = createCuaModels(); - a.deleteProvider("meta"); - expect(a.getProvider("meta")).toBeUndefined(); - expect(b.getProvider("meta")).toBeDefined(); + a.deleteProvider("google"); + expect(a.getProvider("google")).toBeUndefined(); + expect(b.getProvider("google")).toBeDefined(); }); }); describe("cuaModels", () => { it("memoizes the default collection", () => { expect(cuaModels()).toBe(cuaModels()); - expect(cuaModels().getProvider("meta")).toBeDefined(); + expect(cuaModels().getProvider("openai")).toBeDefined(); }); }); diff --git a/packages/ai/test/tool-catalog.test.ts b/packages/ai/test/tool-catalog.test.ts index ccf380e..10d0440 100644 --- a/packages/ai/test/tool-catalog.test.ts +++ b/packages/ai/test/tool-catalog.test.ts @@ -87,7 +87,7 @@ describe("cua tool namespace", () => { }); it("uses the same CUA-authored browser toolset with custom-function providers", () => { - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5", "moonshotai:kimi-k3"] as const) { + for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { const catalog = compile(model, cua.toolsets.browser()); expect(catalog.entries[0]).toMatchObject({ identity: "cua.browser.snapshot.v1", @@ -184,7 +184,7 @@ describe("compileCuaToolCatalog", () => { }); it("still accepts browser_act on providers that take its schema size", () => { - for (const model of ["openai:gpt-5.5", "anthropic:claude-opus-5", "meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["openai:gpt-5.5", "anthropic:claude-opus-5", "xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { expect(() => compile(model, [cua.tools.browser.act()]), model).not.toThrow(); } }); @@ -292,8 +292,8 @@ describe("compileCuaToolCatalog", () => { } }); - it("serializes state-mutating Meta/xAI/Moonshot catalogs with serial tool calls", async () => { - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3"] as const) { + it("serializes state-mutating catalogs with serial tool calls", async () => { + for (const model of ["xai:grok-4.5", "moonshotai:kimi-k3", "openrouter:moonshotai/kimi-k3", "openrouter:meta/muse-spark-1.1"] as const) { const catalog = compile(model, cua.toolsets.browser()); await expect(catalog.payload.apply({ parallel_tool_calls: true }, catalog.model)).resolves.toMatchObject({ parallel_tool_calls: false }); } diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md index 7025256..68ba217 100644 --- a/packages/cli/CHANGELOG.md +++ b/packages/cli/CHANGELOG.md @@ -2,6 +2,13 @@ ## 0.12.0 - 2026-08-13 +- `-m meta:muse-spark-1.1` is removed; use `-m openrouter:meta/muse-spark-1.1`. + `META_API_KEY` is no longer read. +- The default interaction toolset for OpenRouter models is now chosen per model + rather than per provider. OpenRouter fronts several model families, and Kimi + K3 rejects `browser_act`'s schema while Muse Spark accepts it, so the CLI + asks the model's capabilities instead of assuming one answer per provider. + Breaking: Tzafon and Yutori support is removed. - Update `@onkernel/cua-ai` and `@onkernel/cua-agent` to 0.13.0. Refs like diff --git a/packages/cli/README.md b/packages/cli/README.md index 2a4d72d..f3d2e86 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -47,7 +47,7 @@ cua models -p openai cua --print --model openai:gpt-5.6-sol "..." cua --print --model anthropic:claude-opus-5 "..." cua --print --model google:gemini-3.6-flash "..." -cua --print --model meta:muse-spark-1.1 "..." +cua --print --model openrouter:meta/muse-spark-1.1 "..." cua --print --model xai:grok-4.5 "..." cua --print --model moonshotai:kimi-k3 "..." @@ -150,7 +150,6 @@ Configuration is by environment variable. There is no config file. | `ANTHROPIC_API_KEY` | Anthropic API key (required when `-m anthropic:…`) | | `GOOGLE_API_KEY` | Google API key (required when `-m google:…`) | | `GEMINI_API_KEY` | alias of `GOOGLE_API_KEY` | -| `META_API_KEY` | Meta Model API key (required when `-m meta:…`) | | `XAI_API_KEY` | xAI API key (required when `-m xai:…`) | | `MOONSHOT_API_KEY` | Moonshot AI API key (required when `-m moonshotai:…`) | | `OPENROUTER_API_KEY` | OpenRouter API key (required when `-m openrouter:…`) | diff --git a/packages/cli/src/cli-harness.ts b/packages/cli/src/cli-harness.ts index d5e267e..5193c0a 100644 --- a/packages/cli/src/cli-harness.ts +++ b/packages/cli/src/cli-harness.ts @@ -57,7 +57,7 @@ Usage: cua models --json Options: - -p, --provider Filter by provider: openai | anthropic | google | gemini | meta | xai | moonshotai | openrouter + -p, --provider Filter by provider: openai | anthropic | google | gemini | xai | moonshotai | openrouter --json Output JSON -h, --help Show this help `; diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index d34e005..5fe06fd 100644 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -55,7 +55,6 @@ Options: openai: openai:gpt-5.6-sol anthropic: anthropic:claude-opus-5 google: google:gemini-3.6-flash - meta: meta:muse-spark-1.1 xai: xai:grok-4.5 moonshot: moonshotai:kimi-k3 --thinking Thinking level: off | minimal | low | medium | high | xhigh | max @@ -95,7 +94,6 @@ Environment: ANTHROPIC_API_KEY Anthropic API key (required when -m anthropic:…) GOOGLE_API_KEY Google API key (required when -m google:…) GEMINI_API_KEY Alias for GOOGLE_API_KEY - META_API_KEY Meta Model API key (required when -m meta:…) XAI_API_KEY xAI API key (required when -m xai:…) MOONSHOT_API_KEY Moonshot AI API key (required when -m moonshotai:…) OPENROUTER_API_KEY OpenRouter API key (required when -m openrouter:…) diff --git a/packages/cli/src/harness.ts b/packages/cli/src/harness.ts index 3099587..f9f71dd 100644 --- a/packages/cli/src/harness.ts +++ b/packages/cli/src/harness.ts @@ -11,6 +11,7 @@ import { import { type Api, cua, + cuaModelCapabilities, type CuaModelRef, getCuaModel, type Model, @@ -104,14 +105,16 @@ export function defaultInteractionTools(model: CuaModelRef): CuaCliTool[] { : structuredBrowserTools(); case "google": return cua.providers.google.toolsets.browser(); - case "meta": case "xai": return structuredBrowserTools(); case "moonshotai": case "openrouter": - // Kimi's API rejects the request outright once `browser_act`'s - // schema is attached, so Kimi keeps the browser primitives only. - return cua.toolsets.browser(); + // Kimi's API rejects the request outright once `browser_act`'s schema + // is attached. OpenRouter fronts several model families, so this is a + // per-model capability question rather than a per-provider one. + return cuaModelCapabilities(getCuaModel(model)).acceptsLargeSchemas + ? structuredBrowserTools() + : cua.toolsets.browser(); } } diff --git a/packages/cli/test/harness-assembly.test.ts b/packages/cli/test/harness-assembly.test.ts index 4f945d0..0a7fd13 100644 --- a/packages/cli/test/harness-assembly.test.ts +++ b/packages/cli/test/harness-assembly.test.ts @@ -23,7 +23,7 @@ describe("buildCuaHarness", () => { const googleNames = defaultInteractionTools("google:gemini-3.6-flash").map((tool) => tool.name); expect(googleNames).toContain("take_screenshot"); expect(googleNames).not.toContain("browser_act"); - for (const model of ["meta:muse-spark-1.1", "xai:grok-4.5"] as const) { + for (const model of ["xai:grok-4.5", "openrouter:meta/muse-spark-1.1"] as const) { const tools = defaultInteractionTools(model); expect(tools[0]).toMatchObject({ name: "browser_snapshot", origin: "cua" }); expect(tools.at(-1)?.name).toBe("browser_act"); diff --git a/packages/cli/test/harness-models.test.ts b/packages/cli/test/harness-models.test.ts index 4e1377c..2f87f79 100644 --- a/packages/cli/test/harness-models.test.ts +++ b/packages/cli/test/harness-models.test.ts @@ -11,7 +11,7 @@ describe("resolveCuaModelRef", () => { expect(resolveCuaModelRef("openai:gpt-5.6-sol")).toBe("openai:gpt-5.6-sol"); expect(resolveCuaModelRef("openai:gpt-5.5")).toBe("openai:gpt-5.5"); expect(resolveCuaModelRef("anthropic:claude-opus-5")).toBe("anthropic:claude-opus-5"); - expect(resolveCuaModelRef("meta:muse-spark-1.1")).toBe("meta:muse-spark-1.1"); + expect(resolveCuaModelRef("openrouter:meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("xai:grok-4.5")).toBe("xai:grok-4.5"); expect(resolveCuaModelRef("moonshotai:kimi-k3")).toBe("moonshotai:kimi-k3"); expect(resolveCuaModelRef("moonshot:kimi-k3")).toBe("moonshotai:kimi-k3"); @@ -21,7 +21,7 @@ describe("resolveCuaModelRef", () => { expect(resolveCuaModelRef("gpt-5.6-sol")).toBe("openai:gpt-5.6-sol"); expect(resolveCuaModelRef("gpt-5.5")).toBe("openai:gpt-5.5"); expect(resolveCuaModelRef("claude-opus-5")).toBe("anthropic:claude-opus-5"); - expect(resolveCuaModelRef("muse-spark-1.1")).toBe("meta:muse-spark-1.1"); + expect(resolveCuaModelRef("meta/muse-spark-1.1")).toBe("openrouter:meta/muse-spark-1.1"); expect(resolveCuaModelRef("grok-4.5")).toBe("xai:grok-4.5"); expect(resolveCuaModelRef("kimi-k3")).toBe("moonshotai:kimi-k3"); }); @@ -31,11 +31,10 @@ describe("resolveCuaModelRef", () => { }); it("filters custom provider catalogs", () => { - expect(listSupportedModels("meta").map((model) => model.ref)).toEqual(["meta:muse-spark-1.1"]); expect(listSupportedModels("xai").map((model) => model.ref)).toEqual(["xai:grok-4.5"]); expect(listSupportedModels("moonshotai").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); expect(listSupportedModels("moonshot").map((model) => model.ref)).toEqual(["moonshotai:kimi-k3"]); - expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:moonshotai/kimi-k3"]); + expect(listSupportedModels("openrouter").map((model) => model.ref)).toEqual(["openrouter:meta/muse-spark-1.1", "openrouter:moonshotai/kimi-k3"]); expect(resolveCuaModelRef("openrouter:moonshotai/kimi-k3")).toBe("openrouter:moonshotai/kimi-k3"); }); diff --git a/skills/cua-cli/SKILL.md b/skills/cua-cli/SKILL.md index bd761c1..26d720f 100644 --- a/skills/cua-cli/SKILL.md +++ b/skills/cua-cli/SKILL.md @@ -145,7 +145,7 @@ Useful flags: - `-m ` — pick the LLM for model-mediated subcommands (default `gpt-5.6-sol`). Recommended refs are `openai:gpt-5.6-sol`, `anthropic:claude-opus-5`, - `google:gemini-3.6-flash`, `meta:muse-spark-1.1`, `xai:grok-4.5`, + `google:gemini-3.6-flash`, `xai:grok-4.5`, and `moonshotai:kimi-k3`. - `cua models` — list supported `-m` values and their providers; filter with `cua models -p openai|anthropic|google|meta|xai|moonshotai|openrouter`.