Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 4 additions & 27 deletions .agents/skills/update-models/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: update-models
description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Moonshot, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, and Moonshot models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
---

# Update Models
Expand All @@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer-

## Quick Start

1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `XAI_API_KEY`, and `MOONSHOT_API_KEY`.
2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them:

```bash
eval "$(python3 - <<'PY'
import pathlib, re, shlex
text = pathlib.Path('~/AGENTS.md').expanduser().read_text()
for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY']:
m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text)
if m:
print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}')
Expand Down Expand Up @@ -61,7 +61,7 @@ Treat example repos as strongest when they are provider-owned or linked from off

There are two enumeration layers:

- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, xAI's OpenAI-compatible `models.list()`, Tzafon's `Lightcone().models.list()` with known-model fallback, and Yutori OpenAPI/docs model enums) to discover what the current API key can access.
- Live provider availability: `reference/discover-models.ts` uses provider APIs and docs (`OpenAI().models.list()`, `Anthropic().models.list({ limit: 1000 })`, `GoogleGenAI().models.list()` / documented Gemini computer-use IDs, and xAI's OpenAI-compatible `models.list()`) to discover what the current API key can access.
- CUA-supported refs: `listCuaModels(provider?)` from `@onkernel/cua-ai` reads `packages/ai/src/models.ts` and returns the provider-qualified refs CUA accepts (e.g. `anthropic:claude-opus-4-7`). The `CUA_MODEL_ANNOTATIONS` table there is also what `getCuaModel()` and runtime provider routing use.

When live discovery finds a new model with passing smoke tests, update `packages/ai/src/models.ts`; then verify it appears in `listCuaModels("<provider>")`.
Expand All @@ -70,7 +70,6 @@ When live discovery finds a new model with passing smoke tests, update `packages

Meta:

- Discover with the OpenAI SDK against `https://api.meta.ai/v1` using `META_API_KEY`.
- Smoke-test the Responses API with screenshot input and explicit function tools matching CUA's canonical actions.
- Pass condition: response output contains a `function_call` for one of the supplied browser actions.
- Use `store: true` plus `previous_response_id` for CUA tool loops. Meta rejects `include: ["reasoning.encrypted_content"]` on requests that set `previous_response_id`.
Expand Down Expand Up @@ -124,35 +123,16 @@ Moonshot:
- Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn.
- Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes.

Tzafon:

- Discover with `new Lightcone({ apiKey }).models.list()` from `@tzafon/lightcone` when available.
- If model listing is unavailable or returns an undocumented shape, record the error/shape and fall back to known smoke-test candidates such as `tzafon.northstar-cua-fast`.
- Smoke-test `responses.create` with explicit function tools matching the Tzafon template: `click`, `double_click`, `point_and_type`, `key`, `scroll`, `drag`, and `done`.
- Pass condition: response output contains `type: "function_call"` with one of those tool names, or a documented `computer_call` action if Lightcone switches to native computer-use output.
- Track coordinate convention separately from Gemini/Yutori: Tzafon uses a 0-999 grid.

Yutori:

- Discover model IDs from `https://docs.yutori.com/openapi.json` plus the Navigator docs. Current expected IDs include `n1.5-latest`, `n1.5-20260428`, `n1-latest`, and `n1-20260203`.
- Smoke-test the OpenAI-compatible `chat.completions` endpoint with `baseURL: "https://api.yutori.com/v1"` and `YUTORI_API_KEY`.
- Pass condition: response `choices[0].message.tool_calls[]` contains browser action function names such as `left_click`, `goto_url`, `type`, `scroll`, or `wait`.
- Track action-space differences between n1 and n1.5. n1 uses the legacy fixed tool set; n1.5 supports `tool_set`, `disable_tools`, expanded actions, and structured JSON output.
- Do not send duplicate browser action schemas when testing local CUA behavior. The local adapter registers matching AgentTools for execution but filters Yutori's built-in browser tool definitions out of the outbound API payload.

## Native Action Discovery

Run action probes when updating adapters or when docs/examples show drift:

```bash
npx tsx .agents/skills/update-models/reference/discover-models.ts --provider meta --models muse-spark-1.1
npx tsx .agents/skills/update-models/reference/discover-models.ts --provider xai --models grok-4.5
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider anthropic --model claude-opus-4-7
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider gemini --model gemini-3-flash-preview
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest
```

The probe does not execute browser actions. It elicits tool calls for screenshot, click, type, keypress, scroll, drag, hover/move, wait, back/forward, and navigation. Compare:
Expand Down Expand Up @@ -192,14 +172,11 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W
- Update the snapshot in `packages/ai/docs/supported-models.md` to match.

- New provider-native action, response field, or tool version:
- Meta: update `packages/ai/src/providers/meta/index.ts` and `provider.ts`, including Responses threading and reasoning compatibility.
- OpenAI: update `packages/ai/src/providers/openai/index.ts` and its action vocabulary, plus the shared canonical types in `packages/ai/src/providers/common.ts` if the action set changes.
- Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package.
- Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed.
- xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility.
- Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`.
- Tzafon: update `packages/ai/src/providers/tzafon/index.ts` and `provider.ts`, including coordinate/action handling.
- Yutori: update `packages/ai/src/providers/yutori/actions.ts`, `index.ts`, and `provider.ts`, including payload filtering and coordinate/action handling.
- Shared canonical action semantics go in `packages/ai/src/providers/common.ts`.

- New provider or routing rule:
Expand Down
5 changes: 0 additions & 5 deletions .agents/skills/update-models/reference/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,8 @@ These scripts support the `update-models` skill. Run them from the repository ro
- `OPENAI_API_KEY`
- `ANTHROPIC_API_KEY`
- `GOOGLE_API_KEY` or `GEMINI_API_KEY`
- `META_API_KEY`
- `XAI_API_KEY`
- `MOONSHOT_API_KEY`
- `TZAFON_API_KEY`
- `YUTORI_API_KEY`

The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools.

Expand All @@ -32,8 +29,6 @@ Probe native action vocabularies for a specific provider/model:
```bash
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider openai --model gpt-5.5 --out /tmp/openai-actions.json
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider xai --model grok-4.5 --out /tmp/xai-actions.json
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider tzafon --model tzafon.northstar-cua-fast --out /tmp/tzafon-actions.json
npx tsx .agents/skills/update-models/reference/native-action-probe.ts --provider yutori --model n1.5-latest --out /tmp/yutori-actions.json
```

Clone/update official examples and extract tool-handling evidence:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path";
import { spawnSync } from "node:child_process";
import process from "node:process";

type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori";
type Provider = "openai" | "anthropic" | "gemini" | "xai" | "moonshot";

interface ExampleRepo {
provider: Provider;
Expand Down Expand Up @@ -51,14 +51,6 @@ const EXAMPLES: ExampleRepo[] = [
confidence: "provider-owned",
patterns: ["computer_use", "ComputerUse", "function_call", "functionCall", "FunctionResponse", "safety_decision"],
},
{
provider: "meta",
name: "meta-model-cookbook",
repo: "https://github.com/meta-models/meta-model-cookbook.git",
confidence: "provider-owned",
pathHint: "03_use_cases/13_macos_cua",
patterns: ["muse-spark-1.1", "function_call", "function_call_output", "previous_response_id", "reasoning.encrypted_content", "parallel_tool_calls"],
},
{
provider: "xai",
name: "xai-sdk-python",
Expand All @@ -67,24 +59,14 @@ const EXAMPLES: ExampleRepo[] = [
pathHint: "examples",
patterns: ["grok-4.5", "function_call", "tool_call", "previous_response_id", "reasoning_effort", "parallel_tool_calls"],
},
{
provider: "yutori",
name: "kernel-cli-yutori-template",
repo: "https://github.com/kernel/cli.git",
confidence: "kernel-template",
pathHint: "pkg/templates/typescript/yutori",
patterns: ["YUTORI_API_KEY", "tool_calls", "left_click", "goto_url", "n1-latest", "api.yutori.com"],
},
];

const ACTION_REGEXES: Record<Provider, RegExp[]> = {
openai: [/\b(click|double_click|scroll|type|wait|keypress|drag|move|screenshot)\b/g],
anthropic: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up|cursor_position|zoom)\b/g],
gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g],
meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g],
xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g],
moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g],
yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g],
};

function parseArgs(argv: string[]): Args {
Expand Down
Loading
Loading