Derive the streamed transport from the selected tools - #76
Conversation
The transport a model streams through is a property of two things: what the endpoint can carry, and which tools the caller selected. Stamping it on the model ahead of time gets that wrong in both directions — every OpenAI model paid for a CUA-owned transport whether or not its native tool was selected, and selecting that tool later could not change the transport. A provider binding now declares the api id its native tool needs (`requiresApi`), and `compileCuaToolCatalog` reads it off the selected bindings and returns a `catalog.model` carrying that api. Selecting tools whose bindings require different transports fails to compile with a named error. Compilation is idempotent with respect to a model that already carries a derived api, so feeding `catalog.model` back in with a different tool selection re-derives from the new selection instead of pinning the old transport. `routeCuaApi` shrinks to what is genuinely model-shaped: grok-4.5's cost and compat overrides, and the api ids of Tzafon and Yutori, for which pi ships no transport at all. Google's api id moves to the derivation, so a Gemini model selected with only CDP browser tools now streams through pi's builtin Google transport. Provider dispatch is api-keyed: the OpenAI wrapper routes `openai-cua-computer` to the native adapter with no request inspection, and the providers cua constructs use pi's `api` map form. One request-shaped predicate remains, for the case that cannot be derived from a tool list: a transcript carrying deferred-tool state needs the CUA adapter because pi does not round-trip OpenAI's `namespace` field on `function_call` items, which the API rejects with a 400. Both mutation paths push the compiled model into pi. `setTools()` previously recompiled the catalog without propagating the model, so a tool selection that changed the derived transport was compiled but never streamed.
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Spurious model changes on tool transitions
- I added an opt-in
setToolsmodel-sync suppression path and used it for the CLI’s transitional/modeltool step so only the actual model switch is journaled, with a regression test confirming a singlemodel_change.
- I added an opt-in
Or push these changes by commenting:
@cursor push c0df1c33bd
Preview (c0df1c33bd)
diff --git a/packages/agent/src/agent.ts b/packages/agent/src/agent.ts
--- a/packages/agent/src/agent.ts
+++ b/packages/agent/src/agent.ts
@@ -137,6 +137,11 @@
toolContext: AgentHarnessToolContextSource<TContext>;
});
+interface CuaHarnessSetToolsOptions {
+ /** Skip syncing a transport-only model drift while another call immediately sets the final model. */
+ syncModel?: boolean;
+}
+
/** Pi Agent behavior with an explicit, identity-keyed CUA tool catalog. */
export class CuaAgent {
private readonly coreAgent: Agent;
@@ -402,17 +407,18 @@
getTools(): readonly CuaHarnessTool<TContext>[] { return this.tools.getTools(); }
- async setTools(tools: readonly CuaHarnessTool<TContext>[]): Promise<void> {
+ async setTools(tools: readonly CuaHarnessTool<TContext>[], options: CuaHarnessSetToolsOptions = {}): Promise<void> {
+ const syncModel = options.syncModel ?? true;
const previousModel = this.tools.catalog.model;
const previousTools = this.tools.harnessTools();
const prepared = this.tools.prepareTools(tools);
const materialized = this.tools.harnessTools(prepared);
const transportChanged = modelTransportChanged(previousModel, prepared.catalog.model);
try {
- if (transportChanged) await this.coreHarness.setModel(prepared.catalog.model);
+ if (transportChanged && syncModel) await this.coreHarness.setModel(prepared.catalog.model);
await this.coreHarness.setTools(materialized, materialized.map((tool) => tool.name));
} catch (error) {
- if (transportChanged) await this.coreHarness.setModel(previousModel);
+ if (transportChanged && syncModel) await this.coreHarness.setModel(previousModel);
await this.coreHarness.setTools(previousTools, previousTools.map((tool) => tool.name));
throw error;
}
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -381,11 +381,11 @@
// Native catalogs can be incompatible across providers. Transition
// through the CLI-owned application tools before selecting the new
// model's explicit interaction catalog.
- await opts.harness.setTools(opts.applicationTools);
+ await opts.harness.setTools(opts.applicationTools, { syncModel: false });
await opts.harness.setModel(resolved);
await opts.harness.setTools(installedTools);
} catch (error) {
- await opts.harness.setTools(opts.applicationTools);
+ await opts.harness.setTools(opts.applicationTools, { syncModel: false });
await opts.harness.setModel(previousModel);
await opts.harness.setTools(previousTools);
throw error;
diff --git a/packages/cli/test/tool-revalidation.test.ts b/packages/cli/test/tool-revalidation.test.ts
--- a/packages/cli/test/tool-revalidation.test.ts
+++ b/packages/cli/test/tool-revalidation.test.ts
@@ -81,4 +81,22 @@
expect(fixture.harness.getTools().map(toolKey)).toEqual(expected);
expect(fixture.harness.getModel().provider).toBe("anthropic");
});
+
+ it("does not journal the temporary transport hop inside a /model transition", async () => {
+ const from = "google:gemini-3.6-flash";
+ const to = "anthropic:claude-opus-5";
+ const application = defaultApplicationTools();
+ const fixture = await buildTestHarness({
+ turns: [],
+ modelRef: from,
+ tools: [...defaultInteractionTools(from), ...application],
+ });
+
+ await fixture.harness.setTools(application, { syncModel: false });
+ await fixture.harness.setModel(to);
+ await fixture.harness.setTools([...defaultInteractionTools(to), ...application]);
+
+ const modelChanges = (await fixture.session.getBranch()).filter((entry) => entry.type === "model_change");
+ expect(modelChanges).toHaveLength(1);
+ });
});You can send follow-ups to the cloud agent here.
The CLI staged a model switch as setTools(application tools), setModel(next), setTools(next interaction tools). Now that the selected tools decide the transport, the first of those compiles an intermediate catalog whose derived transport belongs to neither the old nor the new selection, and pushing it records a model change for a transport nothing ever streamed with. Add `CuaAgentHarness.setModelAndTools()`, which compiles the pair once and pushes it once, and switch the TUI's `/model` flow and its rollback to it.
|
Good catch, fixed in the latest commit — and it pointed at something structural rather than cosmetic. The Added New test asserts exactly one Separately, I validated both Google paths live against a real Kernel browser, since CI has no Gemini key:
The second is new capability: before this change every Google model was force-routed to the CUA Interactions adapter, so driving Gemini through pi's stock transport wasn't expressible. bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Redundant model-switch rollback
- Removed the extra CLI-level rollback in applySwitchModel so failed setModelAndTools switches now rely solely on the harness’s built-in atomic rollback path.
Or push these changes by commenting:
@cursor push b86b73dc66
Preview (b86b73dc66)
diff --git a/packages/cli/src/tui/main.ts b/packages/cli/src/tui/main.ts
--- a/packages/cli/src/tui/main.ts
+++ b/packages/cli/src/tui/main.ts
@@ -374,18 +374,11 @@
// policy, in which case the switch never touches the tool list at all.
let installedTools: readonly CuaCliTool[] | undefined;
if (opts.interactionToolsForModel) {
- const previousModel = opts.harness.getModel();
- const previousTools = opts.harness.getTools();
installedTools = [...opts.interactionToolsForModel(resolved), ...opts.applicationTools];
- try {
- // Native catalogs are incompatible across providers, and the selected
- // tools decide the transport, so the new model and its interaction
- // catalog have to compile as one pair rather than in sequence.
- await opts.harness.setModelAndTools(resolved, installedTools);
- } catch (error) {
- await opts.harness.setModelAndTools(previousModel, previousTools);
- throw error;
- }
+ // Native catalogs are incompatible across providers, and the selected
+ // tools decide the transport, so the new model and its interaction
+ // catalog have to compile as one pair rather than in sequence.
+ await opts.harness.setModelAndTools(resolved, installedTools);
} else {
await opts.harness.setModel(resolved);
}You can send follow-ups to the cloud agent here.
`setModelAndTools` compiles before it mutates and restores its own state if the mutation fails, so the caller's catch was re-entering setModel/setTools for a switch that never landed — journaling model changes for a transport that was never streamed with. Update the neighboring comments and the CLI test that mirrored the old three-step transition.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit e87ad36. Configure here.
The provider wrapper routed on model.api and the stream function then re-decided on the incoming tool plan — two mechanisms for one decision. Give the native computer adapter its own stream function, selected by api in the wrapper, so the compiled api id is the only dispatch key. Also delete the Yutori n1.5 expanded action declarations and the canonical action type aliases nothing referenced. The expanded set was scaffolding for a ref/DOM execution path that does not exist, and none of it was exported from the package root.
|
Swept Killed: double dispatch on the OpenAI path. The wrapper routed on Killed: dead Yutori declarations (−51 lines). Ruled out: Google's threading. Worth checking, since it is the same So the delta-only input is Google's protocol, not an optimization cua added. The shared threading helpers in Separate finding, not fixed here: Google's usage came back all zeros across a real 6-turn run, so the The rest of bugbot run |
Neither provider is used, and both were the last models whose transport was model-shaped rather than derived from the selected tools. Removing them deletes two provider adapters, five test files, the `@tzafon/lightcone` dependency, and the concepts that existed only to serve them. What got simpler beyond the deletion: - `CuaProviderBinding` drops from five variants to three, and the per-entry provider-facing declaration collapses to a single branch. - The atomic-group concept disappears end to end. Yutori's n1 toolset was the only native set that rejected a partial selection, so the compiler rule, the CLI `/tools` picker's linked-key handling, and its warning line all go. - `compileCuaToolCatalog` loses its `viewport` option: Tzafon's declaration sizing was the only reader, so the tool manager no longer threads the browser viewport into compilation. - `CuaIncomingToolPlan` sheds two fields, and `routeCuaApi` loses both of its remaining transport branches. The OpenAI native-computer exemption from the tool-result image replay limit stays, along with its regression test.
With Tzafon and Yutori gone it routed no transport at all: the transport comes from the selected tools, and its last branch only patched grok-4.5's thinking-level map, price tiers, and compat flags onto pi-ai's registry entry. Live runs on grok-4.5 at the default, off, and xhigh thinking levels behave identically without those patches, so model resolution now returns pi-ai's data unmodified. The one detail that goes with it is a >200k-token price tier pi's registry does not carry, which affects usage.cost reporting on long requests and nothing else.
pi-ai ships no `meta` provider, so cua hand-wrote a model entry and pointed pi's own `openai-responses` transport at `api.meta.ai`. Nothing about that was Meta-specific except a base URL, an env var, and the model entry itself. Meta was the last user of the model-override mechanism, so `CUA_MODEL_OVERRIDES`, `cuaOverrideModels()`, and the private `cuaModel()` helper go with it, and `getCuaModel` no longer needs a "supported but not registered" fallback. Muse Spark stays reachable: it is in pi-ai's OpenRouter catalog as `openrouter:meta/muse-spark-1.1`, annotated with explicit capabilities because OpenRouter's provider-level defaults are conservative. That exposed a real bug the provider switch was hiding. OpenRouter fronts several model families, and the CLI assumed one toolset per provider: Kimi K3 rejects `browser_act`'s schema while Muse Spark accepts it. The default interaction toolset for OpenRouter now asks the model's capabilities instead.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Empty missing-key API test
- The missing-key test now deletes OPENROUTER_API_KEY and asserts requireCuaEnvApiKey("openrouter") throws the expected readable error message.
- ✅ Fixed: Orphaned Meta smoke helper
- The unused smokeMeta function was removed from discover-models.ts because the Meta provider path no longer exists.
Or push these changes by commenting:
@cursor push 94e3808d4c
Preview (94e3808d4c)
diff --git a/.agents/skills/update-models/SKILL.md b/.agents/skills/update-models/SKILL.md
--- a/.agents/skills/update-models/SKILL.md
+++ b/.agents/skills/update-models/SKILL.md
@@ -1,6 +1,6 @@
---
name: update-models
-description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Moonshot, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
+description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
---
# Update Models
@@ -9,14 +9,14 @@
## Quick Start
-1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
+1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`.
2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them:
```bash
eval "$(python3 - <<'PY'
import pathlib, re, shlex
text = pathlib.Path('~/AGENTS.md').expanduser().read_text()
-for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
+for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']:
m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text)
if m:
print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}')
@@ -61,7 +61,7 @@
There are two enumeration layers:
-- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, xAI's OpenAI-compatible `models.list()`, Tzafon's `Lightcone().models.list()` with known-model fallback, and Yutori OpenAPI/docs model enums) to discover what the current API key can access.
+- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access.
- CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use.
When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("<provider>")`.
@@ -70,7 +70,6 @@
Meta:
-- Discover with the OpenAI SDK against `https://api.meta.ai/v1` using `META_API_KEY`.
- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions.
- Pass condition: response output contains a `function_call` for one of the supplied browser actions.
- Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`.
@@ -124,35 +123,16 @@
- Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn.
- Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes.
-Tzafon:
-
-- Discover with `new Lightcone({ apiKey }).models.list()` from `@tzafon/lightcone` when available.
-- If model listing is unavailable or returns an undocumented shape, record the error/shape and fall back to known smoke-test candidates such as `tzafon.northstar-cua-fast`.
-- Smoke-test `responses.create` with explicit function tools matching the Tzafon template: `click`, `double_click`, `point_and_type`, `key`, `scroll`, `drag`, and `done`.
-- Pass condition: response output contains `type: "function_call"` with one of those tool names, or a documented `computer_call` action if Lightcone switches to native computer-use output.
-- Track coordinate convention separately from Gemini/Yutori: Tzafon uses a 0-999 grid.
-
-Yutori:
-
-- Discover model IDs from `https://docs.yutori.com/openapi.json` plus the Navigator docs. Current expected IDs include `n1.5-latest`, `n1.5-20260428`, `n1-latest`, and `n1-20260203`.
-- Smoke-test the OpenAI-compatible `chat.completions` endpoint with `baseURL: "https://api.yutori.com/v1"` and `YUTORI_API_KEY`.
-- Pass condition: response `choices[0].message.tool_calls[]` contains browser action function names such as `left_click`, `goto_url`, `type`, `scroll`, or `wait`.
-- Track action-space differences between n1 and n1.5. n1 uses the legacy fixed tool set; n1.5 supports `tool_set`, `disable_tools`, expanded actions, and structured JSON output.
-- Do not send duplicate browser action schemas when testing local CUA behavior. The local adapter registers matching AgentTools for execution but filters Yutori's built-in browser tool definitions out of the outbound API payload.
-
## Native Action Discovery
Run action probes when updating adapters or when docs/examples show drift:
```bash
-npx tsx .agents/skills/update-models/reference/discover-models.ts --provider meta --models muse-spark-1.1
npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5
-npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast
-npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latestThe probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare:
@@ -192,14 +172,11 @@
-
Update the snapshot in
packages/ai/docs/supported-models.mdto match. -
New provider-native action, response field, or tool version:
-
- Meta: update
packages/ai/src/providers/meta/index.tsandprovider.ts, including Responses threading and reasoning compatibility. - OpenAI: update
packages/ai/src/providers/openai/index.tsand its action vocabulary, plus the shared canonical types inpackages/ai/src/providers/common.tsif the action set changes. - Anthropic: update the
ANTHROPIC_CUA_ACTION_TYPESset inpackages/ai/src/providers/anthropic/actions.tsandindex.ts. The computer tool version andcomputer-use-*beta header are selected bypi-aiper model, so a new dated tool version usually means bumping@earendil-works/pi-ai, not editing this package. - Gemini: update
packages/ai/src/providers/gemini/index.ts, including coordinate handling if needed. - xAI: update
packages/ai/src/providers/xai/index.tsandprovider.ts, including normalized coordinate instructions, Responses threading, and reasoning compatibility. - Moonshot: update
packages/ai/src/providers/moonshot/index.ts, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtinopenai-completionstransport, so wire-format changes usually mean bumping@earendil-works/pi-ai.
- Meta: update
-
- Tzafon: update
packages/ai/src/providers/tzafon/index.tsandprovider.ts, including coordinate/action handling.
- Tzafon: update
-
- Yutori: update
packages/ai/src/providers/yutori/actions.ts,index.ts, andprovider.ts, including payload filtering and coordinate/action handling. - Shared canonical action semantics go in
packages/ai/src/providers/common.ts.
- Yutori: update
-
New provider or routing rule:
diff --git a/.agents/skills/update-models/reference/README.md b/.agents/skills/update-models/reference/README.md
--- a/.agents/skills/update-models/reference/README.md
+++ b/.agents/skills/update-models/reference/README.md
@@ -11,11 +11,8 @@
OPENAI_API_KEYANTHROPIC_API_KEYGOOGLE_API_KEYorGEMINI_API_KEY-
META_API_KEYXAI_API_KEYMOONSHOT_API_KEY
-
TZAFON_API_KEY
-
YUTORI_API_KEY
The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools.
@@ -32,8 +29,6 @@
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json
-npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --out /tmp/tzafon-actions.json
-npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --out /tmp/yutori-actions.jsonClone/update official examples and extract tool-handling evidence:
diff --git a/.agents/skills/update-models/reference/audit-official-examples.ts b/.agents/skills/update-models/reference/audit-official-examples.ts
--- a/.agents/skills/update-models/reference/audit-official-examples.ts
+++ b/.agents/skills/update-models/reference/audit-official-examples.ts
@@ -5,7 +5,7 @@
import { spawnSync } from "node:child_process";
import process from "node:process";
-type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot";
interface ExampleRepo {
provider: Provider;
@@ -52,14 +52,6 @@
patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"],
},
{
-
provider: "meta", -
name: "meta-model-cookbook", -
repo: "https://github.com/meta-models/meta-model-cookbook.git", -
confidence: "provider-owned", -
pathHint: "03_use_cases/13_macos_cua", -
patterns: ["muse-spark-1.1", "function_call", "function_call_output", "previous_response_id", "reasoning.encrypted_content", "parallel_tool_calls"], - },
- {
provider: "xai",
name: "xai-sdk-python",
repo: "https://github.com/xai-org/xai-sdk-python.git",
@@ -67,24 +59,14 @@
pathHint: "examples",
patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"],
}, - {
-
provider: "yutori", -
name: "kernel-cli-yutori-template", -
repo: "https://github.com/kernel/cli.git", -
confidence: "kernel-template", -
pathHint: "pkg/templates/typescript/yutori", -
patterns: ["YUTORI_API_KEY", "tool_calls", "left_click", "goto_url", "n1-latest", "api.yutori.com"], - },
];
const ACTION_REGEXES: Record<Provider, RegExp[]> = {
openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g],
anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g],
gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g],
- meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g],
xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g],
moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g], - yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g],
};
function parseArgs(argv: string[]): Args {
diff --git a/.agents/skills/update-models/reference/discover-models.ts b/.agents/skills/update-models/reference/discover-models.ts
--- a/.agents/skills/update-models/reference/discover-models.ts
+++ b/.agents/skills/update-models/reference/discover-models.ts
@@ -1,12 +1,10 @@
#!/usr/bin/env tsx
import { existsSync } from "node:fs";
import { readFile, writeFile } from "node:fs/promises";
-import { homedir } from "node:os";
import { join } from "node:path";
import process from "node:process";
-import { parse as parseToml } from "smol-toml";
-type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "tzafon" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot";
interface Args {
provider: Provider | "all";
@@ -39,16 +37,7 @@
cua?: Record<string, unknown>;
}
-const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot", "tzafon", "yutori"];
-const TZAFON_KNOWN_MODELS = [
- "tzafon.northstar-cua-fast",
-];
-const YUTORI_DOC_MODELS = [ - "n1.5-latest",
- "n1.5-20260428",
- "n1-latest",
- "n1-20260203",
-];
+const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "xai", "moonshot"];
const GEMINI_DOC_COMPUTER_USE_MODELS = [
"gemini-3.5-flash",
"gemini-3-flash-preview",
@@ -110,7 +99,7 @@
npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4
Options:
- --provider <all|openai|anthropic|gemini|meta|xai|moonshot|tzafon|yutori>
- --provider <all|openai|anthropic|gemini|xai|moonshot>
--models Smoke-test explicit models instead of inferred candidates.
--candidate-limit Max inferred candidates per provider. Default: 20.
--no-smoke Only list metadata.
@@ -138,11 +127,8 @@
if (provider === "openai") return await discoverOpenAI(args);
if (provider === "anthropic") return await discoverAnthropic(args);
if (provider === "gemini") return await discoverGemini(args);
-
if (provider === "meta") return await discoverMeta(args); if (provider === "xai") return await discoverXai(args); if (provider === "moonshot") return await discoverMoonshot(args); -
if (provider === "tzafon") return await discoverTzafon(args); -
} catch (err) {
if (provider === "yutori") return await discoverYutori(args); throw new Error(`unknown provider ${provider satisfies never}`);
return {
@@ -179,74 +165,6 @@
return { provider: "openai", metadata_source: "client.models.list()", models, candidates };
}
-async function discoverMeta(args: Args): Promise<Record<string, unknown>> {
- const OpenAI = await importDefault("openai", "OpenAI");
- const apiKey = process.env.META_API_KEY;
- const client = new OpenAI({ apiKey, baseURL: "https://api.meta.ai/v1" });
- const rawModels = await collectAsync(client.models.list());
- const models: ModelResult[] = rawModels.map((m) => ({
-
id: String(m.id), -
display_name: String(m.id), -
created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, -
raw: m, -
supports_generation: true, - }));
- const candidates = explicitOrCandidates(args, models.map((model) => model.id));
- if (args.smoke) {
-
await Promise.all(candidates.map(async (id) => { -
const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true }; -
model.computer_use = await smokeMeta(client, id); -
if (!models.find((candidate) => candidate.id === id)) models.unshift(model); -
})); - }
- await annotateCuaSupport("meta", models);
- return { provider: "meta", metadata_source: "Meta Model API models.list()", models, candidates };
-}
-async function smokeMeta(client: any, model: string): Promise {
- try {
-
const screenshot = await readFile(fixtureScreenshotPath()); -
const response = await client.responses.create({ -
model, -
store: false, -
parallel_tool_calls: false, -
max_output_tokens: 512, -
reasoning: { effort: "low" }, -
input: [{ -
role: "user", -
content: [ -
{ type: "input_text", text: "Call the click tool for the sign in link. Do not answer only in text." }, -
{ type: "input_image", image_url: `data:image/png;base64,${screenshot.toString("base64")}` }, -
], -
}], -
tools: [{ -
type: "function", -
name: "click", -
description: "Click at normalized 0-1000 screen coordinates.", -
parameters: { -
type: "object", -
properties: { x: { type: "number" }, y: { type: "number" } }, -
required: ["x", "y"], -
additionalProperties: false, -
}, -
}], -
}); -
const output: any[] = response.output ?? []; -
const calls = output.filter((item) => item?.type === "function_call"); -
return { -
status: calls.length > 0 ? "pass" : "inconclusive", -
tool_name: "function_tools", -
tool_version: null, -
beta_header: null, -
observed_actions: unique(calls.map((call) => call?.name).filter(Boolean)), -
response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), -
error: null, -
}; - } catch (err) {
-
return smokeError(err, { tool_name: "function_tools" }); - }
-}
async function discoverXai(args: Args): Promise<Record<string, unknown>> {
const OpenAI = await importDefault("openai", "OpenAI");
const client = new OpenAI({ apiKey: process.env.XAI_API_KEY, baseURL: "https://api.x.ai/v1" });
@@ -633,304 +551,6 @@
return { provider: "gemini", metadata_source: "client.models.list()", models, candidates };
}
-async function discoverYutori(args: Args): Promise<Record<string, unknown>> {
- const openapi = await fetchYutoriOpenApi();
- const ids = unique([
-
...extractYutoriModelIds(openapi.raw), -
...YUTORI_DOC_MODELS, - ]);
- const models: ModelResult[] = ids.map((id) => ({
-
id, -
display_name: yutoriDisplayName(id), -
created_at: yutoriCreatedAt(id), -
raw: { source: "docs.yutori.com/openapi.json" }, -
supports_generation: true, -
model_docs: { -
navigator_docs: "https://docs.yutori.com/reference/navigator", -
n1_docs: "https://docs.yutori.com/reference/n1", -
n15_docs: "https://docs.yutori.com/reference/n1-5", -
openapi_ok: openapi.ok, -
tool_set: id.startsWith("n1.5") ? "browser_tools_core-20260403" : "legacy_fixed", -
disable_tools: id.startsWith("n1.5") ? "supported" : "not_supported", -
coordinate_space: "1000x1000", -
}, - }));
- const candidates = explicitOrCandidates(args, ids);
- if (args.smoke) {
-
await Promise.all(candidates.map(async (id) => { -
const model = models.find((m) => m.id === id) ?? { -
id, -
display_name: yutoriDisplayName(id), -
supports_generation: true, -
}; -
model.computer_use = await smokeYutori(id); -
if (!models.find((m) => m.id === id)) models.unshift(model); -
})); - }
- await annotateCuaSupport("yutori", models);
- return {
-
provider: "yutori", -
metadata_source: "https://docs.yutori.com/openapi.json + Navigator docs", -
openapi: openapi.summary, -
models, -
candidates, - };
-}
-async function fetchYutoriOpenApi(): Promise<{ ok: boolean; raw: unknown; summary: Record<string, unknown> }> {
- const url = "https://docs.yutori.com/openapi.json";
- try {
-
const response = await fetch(url); -
const raw = await response.json(); -
return { -
ok: response.ok, -
raw, -
summary: { -
url, -
ok: response.ok, -
model_ids: extractYutoriModelIds(raw), -
}, -
}; - } catch (err) {
-
return { -
ok: false, -
raw: undefined, -
summary: { url, ok: false, error: publicError(err) }, -
}; - }
-}
-function extractYutoriModelIds(raw: unknown): string[] {
- const found: string[] = [];
- const visit = (value: unknown): void => {
-
if (!value || typeof value !== "object") return; -
if (Array.isArray(value)) { -
for (const item of value) visit(item); -
return; -
} -
const obj = value as Record<string, unknown>; -
if (Array.isArray(obj.enum)) { -
for (const item of obj.enum) { -
if (typeof item === "string" && /^n1(?:\.5)?-/.test(item)) found.push(item); -
} -
} -
for (const item of Object.values(obj)) visit(item); - };
- visit(raw);
- return unique(found);
-}
-function yutoriDisplayName(id: string): string {
- if (id.startsWith("n1.5")) return
Yutori Navigator ${id.replace("n1.5", "n1.5")}; - return
Yutori Navigator ${id.replace("n1", "n1")};
-}
-function yutoriCreatedAt(id: string): string | null {
- const match = id.match(/-(\d{4})(\d{2})(\d{2})$/);
- if (!match) return null;
- return
${match[1]}-${match[2]}-${match[3]}T00:00:00.000Z;
-}
-async function yutoriApiKey(): Promise {
- const env = process.env.YUTORI_API_KEY;
- if (env && env.trim()) return env;
- const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml");
- try {
-
const raw = parseToml(await readFile(cfgPath, "utf8")) as any; -
const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; -
const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; -
if (typeof key === "string" && key.trim()) return key; - } catch {
-
// Fall through to a clear credential error. - }
- throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)");
-}
-async function smokeYutori(model: string): Promise {
- try {
-
const OpenAI = await importDefault("openai", "OpenAI"); -
const client = new OpenAI({ -
apiKey: await yutoriApiKey(), -
baseURL: "https://api.yutori.com/v1", -
}); -
const response = await client.chat.completions.create({ -
model, -
messages: [{ -
role: "user", -
content: [ -
{ type: "text", text: "Use the browser action tool to inspect or interact with this page. Do not answer in text." }, -
{ type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, -
], -
}], -
max_completion_tokens: 128, -
temperature: 0.3, -
}); -
const choice = response.choices?.[0]; -
const toolCalls: any[] = choice?.message?.tool_calls ?? []; -
return { -
status: toolCalls.length > 0 ? "pass" : "inconclusive", -
tool_name: "browser_actions", -
tool_version: model.startsWith("n1.5") ? "n1.5" : "n1", -
beta_header: null, -
observed_actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), -
response_item_types: unique([ -
choice?.message?.content ? "text" : undefined, -
toolCalls.length ? "tool_calls" : undefined, -
].filter(Boolean) as string[]), -
finish_reason: choice?.finish_reason ?? null, -
accepts_image_tool_results: "assumed-from-docs", -
error: null, -
}; - } catch (err) {
-
return smokeError(err, { tool_name: "browser_actions" }); - }
-}
-async function discoverTzafon(args: Args): Promise<Record<string, unknown>> {
- const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone");
- const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY });
- const modelList = await fetchTzafonModels(client);
- const ids = unique([
-
...modelList.ids, -
...TZAFON_KNOWN_MODELS, - ]);
- const models: ModelResult[] = ids.map((id) => ({
-
id, -
display_name: tzafonDisplayName(id), -
raw: modelList.rawById[id] ?? { source: "known-model-fallback" }, -
supports_generation: true, -
model_docs: { -
docs: "https://docs.lightcone.ai", -
responses_endpoint: "supported", -
function_calling: "supported", -
computer_use: "supported", -
coordinate_space: "0-999", -
model_list_endpoint: modelList.source, -
}, - }));
- const candidates = explicitOrCandidates(args, ids);
- if (args.smoke) {
-
await Promise.all(candidates.map(async (id) => { -
const model = models.find((m) => m.id === id) ?? { -
id, -
display_name: tzafonDisplayName(id), -
supports_generation: true, -
}; -
model.computer_use = await smokeTzafon(client, id); -
if (!models.find((m) => m.id === id)) models.unshift(model); -
})); - }
- await annotateCuaSupport("tzafon", models);
- return {
-
provider: "tzafon", -
metadata_source: modelList.source, -
model_list_error: modelList.error ?? null, -
models, -
candidates, - };
-}
-async function fetchTzafonModels(client: any): Promise<{
- source: string;
- ids: string[];
- rawById: Record<string, unknown>;
- raw?: unknown;
- error?: string;
-}> { - try {
-
const raw = await client.models.list(); -
const entries = extractTzafonModelEntries(raw); -
const rawById: Record<string, unknown> = {}; -
for (const entry of entries) rawById[entry.id] = entry.raw; -
return { -
source: "@tzafon/lightcone models.list()", -
ids: entries.map((entry) => entry.id), -
rawById, -
raw, -
}; - } catch (err) {
-
return { -
source: "known-model-fallback (models.list unavailable)", -
ids: [], -
rawById: {}, -
error: publicError(err), -
}; - }
-}
-function extractTzafonModelEntries(raw: unknown): Array<{ id: string; raw: unknown }> {
- const entries: Array<{ id: string; raw: unknown }> = [];
- const visit = (value: unknown): void => {
-
if (!value || typeof value !== "object") return; -
if (Array.isArray(value)) { -
for (const item of value) visit(item); -
return; -
} -
const obj = value as Record<string, unknown>; -
const id = obj.id ?? obj.name ?? obj.model; -
if (typeof id === "string" && id.trim()) { -
entries.push({ id: id.trim(), raw: obj }); -
} -
for (const child of Object.values(obj)) { -
if (child && typeof child === "object") visit(child); -
} - };
- visit(raw);
- return entries;
-}
-function tzafonDisplayName(id: string): string {
- if (id === "tzafon.northstar-cua-fast") return "Tzafon Northstar CUA Fast";
- return id;
-}
-async function smokeTzafon(client: any, model: string): Promise {
- try {
-
const response = await client.responses.create({ -
model, -
input: [{ -
role: "user", -
content: [ -
{ type: "input_text", text: "Use the computer function tools to inspect this page. Do not answer in text." }, -
{ type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, -
], -
}], -
tools: TZAFON_FUNCTION_TOOLS, -
instructions: "The screen's coordinate space is a 0-999 grid. Call a function tool instead of answering in text.", -
temperature: 0, -
max_output_tokens: 128, -
}); -
const output: any[] = response.output ?? []; -
const functionCalls = output.filter((item) => item?.type === "function_call"); -
const computerCalls = output.filter((item) => item?.type === "computer_call"); -
const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []); -
return { -
status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", -
tool_name: functionCalls.length ? "function_tools" : "computer_use", -
tool_version: null, -
beta_header: null, -
observed_actions: unique([ -
...functionCalls.map((call) => call?.name).filter(Boolean), -
...computerActions.map((action) => action?.type).filter(Boolean), -
]), -
response_item_types: unique(output.map((item) => item?.type).filter(Boolean)), -
error: null, -
}; - } catch (err) {
-
return smokeError(err, { tool_name: "function_tools" }); - }
-}
-const TZAFON_FUNCTION_TOOLS = [
- { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } },
- { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } },
- { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } },
- { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } },
- { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } },
- { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } },
- { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } },
-];
async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise {
const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider;
const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined);
diff --git a/.agents/skills/update-models/reference/native-action-probe.ts b/.agents/skills/update-models/reference/native-action-probe.ts
--- a/.agents/skills/update-models/reference/native-action-probe.ts
+++ b/.agents/skills/update-models/reference/native-action-probe.ts
@@ -1,12 +1,10 @@
#!/usr/bin/env tsx
import { existsSync } from "node:fs";
import { readFile, writeFile } from "node:fs/promises";
-import { homedir } from "node:os";
import { join } from "node:path";
import process from "node:process";
-import { parse as parseToml } from "smol-toml";
-type Provider = "openai" | "anthropic" | "gemini" | "xai" | "tzafon" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "xai";
interface ProbePrompt {
id: string;
@@ -64,8 +62,8 @@
throw new Error(unknown argument: ${arg});
}
}
- if (!["openai", "anthropic", "gemini", "xai", "tzafon", "yutori"].includes(out.provider)) {
-
throw new Error("--provider is required: openai | anthropic | gemini | xai | tzafon | yutori");
- if (!["openai", "anthropic", "gemini", "xai"].includes(out.provider)) {
-
}
throw new Error("--provider is required: openai | anthropic | gemini | xai");
if (!out.model) throw new Error("--model is required");
return out;
@@ -76,8 +74,6 @@
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/actions.json
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7 --limit 3
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --limit 3
- npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --limit 3
- npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --limit 3
`);
process.exit(0);
}
@@ -105,8 +101,6 @@
if (provider === "anthropic") return await probeAnthropic(model, prompt);
if (provider === "gemini") return await probeGemini(model, prompt);
if (provider === "xai") return await probeXai(model, prompt); -
if (provider === "tzafon") return await probeTzafon(model, prompt); -
} catch (err) {
if (provider === "yutori") return await probeYutori(model, prompt); throw new Error(`unknown provider ${provider satisfies never}`);
return {
@@ -258,87 +252,6 @@
return readFile(path);
}
-async function probeYutori(model: string, prompt: ProbePrompt): Promise {
- const OpenAI = await importDefault("openai", "OpenAI");
- const client = new OpenAI({
-
apiKey: await yutoriApiKey(), -
baseURL: "https://api.yutori.com/v1", - });
- const response = await client.chat.completions.create({
-
model, -
messages: [{ -
role: "user", -
content: [ -
{ type: "text", text: prompt.text }, -
{ type: "image_url", image_url: { url: "https://docs.yutori.com/assets/google_homepage_2024.jpg" } }, -
], -
}], -
max_completion_tokens: 128, -
temperature: 0.3, - });
- const choice = response.choices?.[0];
- const toolCalls: any[] = choice?.message?.tool_calls ?? [];
- return {
-
id: prompt.id, -
status: toolCalls.length ? "pass" : "inconclusive", -
actions: unique(toolCalls.map((call) => call?.function?.name).filter(Boolean)), -
item_types: unique([ -
choice?.message?.content ? "text" : undefined, -
toolCalls.length ? "tool_calls" : undefined, -
].filter(Boolean) as string[]), -
finish_reason: choice?.finish_reason ?? null, -
raw_tool_calls: toolCalls.map(redactLargeFields), - };
-}
-async function yutoriApiKey(): Promise {
- const env = process.env.YUTORI_API_KEY;
- if (env && env.trim()) return env;
- const cfgPath = join(process.env.XDG_CONFIG_HOME ?? join(homedir(), ".config"), "cua", "config.toml");
- try {
-
const raw = parseToml(await readFile(cfgPath, "utf8")) as any; -
const profile = typeof raw?.default_profile === "string" ? raw.default_profile : undefined; -
const key = profile ? raw?.profiles?.[profile]?.yutori_api_key : undefined; -
if (typeof key === "string" && key.trim()) return key; - } catch {
-
// Fall through to a clear credential error. - }
- throw new Error("missing Yutori API key (set YUTORI_API_KEY or yutori_api_key in the default cua config profile)");
-}
-async function probeTzafon(model: string, prompt: ProbePrompt): Promise {
- const Lightcone = await importDefault("@tzafon/lightcone", "Lightcone");
- const client = new Lightcone({ apiKey: process.env.TZAFON_API_KEY });
- const response = await client.responses.create({
-
model, -
input: [{ -
role: "user", -
content: [ -
{ type: "input_text", text: `${prompt.text}\nCall one of the available function tools instead of answering in text.` }, -
{ type: "input_image", image_url: "https://docs.yutori.com/assets/google_homepage_2024.jpg", detail: "auto" }, -
], -
}], -
tools: TZAFON_FUNCTION_TOOLS, -
instructions: "The screen's coordinate space is a 0-999 grid.", -
temperature: 0, -
max_output_tokens: 128, - });
- const output: any[] = response.output ?? [];
- const functionCalls = output.filter((item) => item?.type === "function_call");
- const computerCalls = output.filter((item) => item?.type === "computer_call");
- const computerActions = computerCalls.flatMap((call) => Array.isArray(call.actions) ? call.actions : call.action ? [call.action] : []);
- return {
-
id: prompt.id, -
status: functionCalls.length || computerCalls.length ? "pass" : "inconclusive", -
actions: unique([ -
...functionCalls.map((call) => call?.name).filter(Boolean), -
...computerActions.map((action) => action?.type).filter(Boolean), -
]), -
item_types: unique(output.map((item) => item?.type).filter(Boolean)), -
raw_tool_calls: [...functionCalls, ...computerCalls].map(redactLargeFields), - };
-}
const XAI_FUNCTION_TOOLS = [
{ type: "function", name: "screenshot", description: "Capture the current browser screenshot.", parameters: { type: "object", properties: {}, additionalProperties: false } },
{ type: "function", name: "goto", description: "Navigate to a URL.", parameters: { type: "object", properties: { url: { type: "string" } }, required: ["url"], additionalProperties: false } },
@@ -352,16 +265,6 @@
{ type: "function", name: "back", description: "Go back in browser history.", parameters: { type: "object", properties: {}, additionalProperties: false } },
];
-const TZAFON_FUNCTION_TOOLS = [
- { type: "function", name: "click", description: "Single click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, button: { type: "string", enum: ["left", "right"] } }, required: ["x", "y"] } },
- { type: "function", name: "double_click", description: "Double click at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" } }, required: ["x", "y"] } },
- { type: "function", name: "point_and_type", description: "Click at position then type text.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, text: { type: "string" }, press_enter: { type: "boolean" } }, required: ["x", "y", "text"] } },
- { type: "function", name: "key", description: "Press key combo.", parameters: { type: "object", properties: { keys: { type: "string" } }, required: ["keys"] } },
- { type: "function", name: "scroll", description: "Scroll at (x, y) in 0-999 grid.", parameters: { type: "object", properties: { x: { type: "integer" }, y: { type: "integer" }, dy: { type: "integer" } }, required: ["x", "y", "dy"] } },
- { type: "function", name: "drag", description: "Drag from (x1, y1) to (x2, y2) in 0-999 grid.", parameters: { type: "object", properties: { x1: { type: "integer" }, y1: { type: "integer" }, x2: { type: "integer" }, y2: { type: "integer" } }, required: ["x1", "y1", "x2", "y2"] } },
- { type: "function", name: "done", description: "Task complete. Report findings.", parameters: { type: "object", properties: { result: { type: "string" } }, required: ["result"] } },
-];
async function importDefault(pkg: string, named: string): Promise {
try {
const mod = await import(pkg);
diff --git a/.agents/skills/update-models/reference/provider-doc-drift.ts b/.agents/skills/update-models/reference/provider-doc-drift.ts
--- a/.agents/skills/update-models/reference/provider-doc-drift.ts
+++ b/.agents/skills/update-models/reference/provider-doc-drift.ts
@@ -2,7 +2,7 @@
import { readFile, writeFile } from "node:fs/promises";
import process from "node:process";
-type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori";
+type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot";
interface Args {
examples: string;
@@ -30,11 +30,6 @@
"https://ai.google.dev/gemini-api/docs/computer-use",
"https://ai.google.dev/api/models",
],
- meta: [
-
"https://dev.meta.ai/docs/getting-started/cookbook/computer-use-macos", -
"https://dev.meta.ai/docs/features/responses", -
"https://dev.meta.ai/docs/features/tool-calling", - ],
xai: [
"https://docs.x.ai/developers/grok-4-5",
"https://docs.x.ai/developers/tools/function-calling",
@@ -45,32 +40,22 @@
"https://platform.kimi.ai/docs/api/tool-use",
"https://platform.kimi.ai/docs/guide/use-kimi-vision-model",
], - yutori: [
-
"https://docs.yutori.com/reference/navigator", -
"https://docs.yutori.com/reference/n1", -
"https://docs.yutori.com/reference/n1-5", -
"https://docs.yutori.com/openapi.json", - ],
};
... diff truncated: showing 800 of 5908 lines
</details>
<sub>You can send follow-ups to the cloud agent <a href="https://cursor.com/agents/bc-05eb2aa4-7847-43b5-9543-1c9c94decf16">here</a>.</sub>
<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END -->
<sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 13fa4e633c6eecaa8abd53f081a8e58aaab439d2. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
| @@ -65,7 +60,6 @@ describe("cua api key helpers", () => { | |||
| }); | |||
|
|
|||
| it("throws readable errors when missing", () => { | |||
There was a problem hiding this comment.
Empty missing-key API test
Low Severity
The "throws readable errors when missing" test only deletes META_API_KEY and asserts nothing. Meta auth was removed in this change, so the case no longer covers requireCuaEnvApiKey and always passes without checking error text for a live provider such as openrouter.
Reviewed by Cursor Bugbot for commit 13fa4e6. Configure here.
| display_name: String(m.id), | ||
| created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null, | ||
| raw: m, | ||
| supports_generation: true, |
There was a problem hiding this comment.
Orphaned Meta smoke helper
Low Severity
smokeMeta remains after discoverMeta and the meta provider were removed from this script. Nothing calls it, so Meta’s smoke path is dead code that still suggests a first-party Meta discovery path that no longer exists.
Reviewed by Cursor Bugbot for commit 13fa4e6. Configure here.
1 participant
Summary
Phase 1 of the cua evolution plan. The transport a model streams through depends on two things — what the endpoint can carry, and which tools the caller selected — so it belongs where both are known: catalog compilation.
Before #75 the api id was stamped on the model, which got it wrong in both directions: every OpenAI model paid for a CUA-owned transport whether or not its native tool was selected, and selecting that tool later couldn't change the transport. #75 fixed the symptom by having the provider wrapper sniff the request shape. This makes the rule explicit instead.
The rule
A
CuaProviderBindingdeclaresrequiresApi— the api id its provider-native tool needs.compileCuaToolCatalogreads it off the selected bindings and returns acatalog.modelcarrying that api. Tools whose bindings require different transports fail to compile with a named error.openai-responses(pi builtin)cua.providers.openai.tools.computer()openai-cua-computer(CUA adapter)google-generative-ai(pi builtin)cua.providers.google.toolsets.browser()google-cua-interactions(CUA adapter)Compilation is idempotent with respect to a model that already carries a derived api, so feeding
catalog.modelback in with a different tool selection re-derives rather than pinning the old transport. That path is real: the TUI's failed-model-switch rollback passesgetModel()straight back intosetModel().What moved
routeCuaApishrinks to what is genuinely model-shaped: grok-4.5's cost/compat/thinking overrides, and the api ids of Tzafon and Yutori. Google's moves to the derivation — so a Gemini model selected with only CDP browser tools now streams through pi's builtin Google transport. The CLI still selects Google's native toolset by default, so shipped behavior is unchanged.openai-cua-computerto the native adapter with no request inspection; the providers cua constructs use pi's documentedapimap form.requiresCuaOpenAINamespaceAdapter, for the case that cannot be derived from a tool list: a transcript carrying deferred-tool state needs the CUA adapter, because pi does not round-trip OpenAI'snamespacefield onfunction_callitems and the API rejects it with a 400.packages/agent/test/openai-deferred-tools.test.tsremains the gate.Two bugs found while building it
setTools()never propagated the compiled model into pi. A tool selection that changed the derived transport was compiled and then not streamed — the derivation would have silently not applied on the tools path. BothCuaAgentandCuaAgentHarnessnow pushcatalog.model; the harness only does so when the transport actually moved, to avoid spuriousmodel_changesession entries on ordinary tool swaps.New tests assert the model pi actually streams with after
setTools(), not justcatalog.model.api— the gap that let the first bug hide.Testing
npm run typecheck— clean.Known maintenance point
CATALOG_DERIVED_API_DEFAULTSintool-catalog.tsmaps each derived api back to its ordinary registry default for the idempotence reset. A futurerequiresApion a registry-backed provider needs an entry there too. Behavioral tests cover both current providers (compile with native toolset, recompile without, assert it returns to the registry api), so a forgotten entry fails loudly for anything following the same pattern.Note
High Risk
Removes three provider integrations and changes which API transport runs for Google/OpenAI based on tool selection—easy to break live agents, CI secrets, and existing model refs.
Overview
Transport is chosen at catalog compile time. Native tool bindings can declare
requiresApi;compileCuaToolCatalogputs that api oncatalog.model(e.g. OpenAI native computer →openai-cua-computer, Google browser toolset →google-cua-interactions). Google models with only CUA browser tools keep pi’s builtin Google api instead of always using Interactions. OpenAI dispatch keys offmodel.api, withrequiresCuaOpenAINamespaceAdapteronly for deferred tool-search / namespace replay. Tools-only recompiles reset from storedmodelSelectionso dropping native tools does not leave a stale derived api.Agent/runtime fixes:
setTools()/setModel()push the compiled model into pi when the transport changes;CuaAgentHarness.setModelAndTools()avoids a bogus intermediate transport. Tool-result image replay exempts only OpenAI native computer (Tzafon exemption removed).compileCuaToolCatalogno longer takesviewport;CuaExecutionResources.viewportis removed.Breaking provider cleanup: Drops Meta as a first-class provider (
CUA_MODEL_OVERRIDES,routeCuaApi, dedicated stream registration). Tzafon and Yutori are fully removed (packages, keys, bindings, streams, tests). Muse Spark is documented asopenrouter:meta/muse-spark-1.1. CI, README, agent examples, and update-models skill/scripts are trimmed to match.Reviewed by Cursor Bugbot for commit 13fa4e6. Bugbot is set up for automated code reviews on this repo. Configure here.