Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,21 @@ with `cua.providers.openai.tools.computer()` compiles to the CUA-owned
`google-cua-interactions` Interactions API versus pi's builtin Google
transport.

### The tool menu

`cuaToolMenu(model, selected)` in `packages/ai/src/menu.ts` returns every tool
CUA can offer for a model, each marked available or not. It decides availability
by compiling the candidate catalog rather than by restating the compiler's
rules, so the menu cannot drift from what `compileCuaToolCatalog` accepts: an
entry is available exactly when selecting it compiles. Compilation is pure and
declaration-only, so probing it per entry is cheap and side-effect-free.

Availability is relative to the current selection, because several rules are
pairwise: two providers' native surfaces cannot coexist, and a native surface
derives a transport that the rest of the selection must be compatible with.
Callers rebuild the menu after each staged change rather than caching a per-tool
verdict.

`CuaAgent` and `CuaAgentHarness` push the compiled `catalog.model` into pi on
every construction and on every `setTools()`/`setModel()`, so the derived
transport applies uniformly regardless of which mutation path selected the
Expand Down Expand Up @@ -225,10 +240,13 @@ to it so `ctrl+c` cancels the selector instead of quitting.
`filterModelsForPicker`, `moveSelection`, `visibleWindow`) that make its
behavior unit-testable without a terminal.
- `tui/tool-selection.ts` — pure `/tools` state machine: identity keys matching
`normalizeTool`'s scheme, group badges, and toggle/bulk operations.
`normalizeTool`'s scheme, group badges, toggle/bulk operations, and
`describeMenu()`, which turns `cuaToolMenu()` plus the application's own tools
into rows.
- `tui/tools-picker.ts` — the `/tools` component. Staged edits applied through
`harness.setTools()` with a subset of the application-composed baseline, in
baseline order.
`harness.setTools()`, in menu order. A selection is no longer confined to the
application-composed baseline: the picker offers the model's whole menu, and
the baseline is what `ctrl+r` restores.
- `tui/keybindings.ts` — registers `cua.tools.*` ids on top of pi-tui's
`TUI_KEYBINDINGS` and formats their hints.
- `tui/mutation-queue.ts` — the serialization queue both catalog mutations run
Expand Down
10 changes: 10 additions & 0 deletions packages/ai/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## 0.15.0 - 2026-08-14

- Add `cuaToolMenu(model, selected)`: every tool CUA can offer for a model, each
marked available or not with the compiler's own reason when it is not. It
decides availability by compiling the candidate catalog rather than restating
the compiler's rules, so the menu cannot drift from what
`compileCuaToolCatalog` accepts. Availability is relative to the current
selection, because two providers' native surfaces cannot coexist and a native
surface pins the transport.

## 0.14.0 - 2026-08-14

Breaking: the model allowlist is removed.
Expand Down
1 change: 1 addition & 0 deletions packages/ai/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,4 @@ export {
} from "./providers/common";
export * from "./tool-catalog";
export * from "./cua";
export * from "./menu";
127 changes: 127 additions & 0 deletions packages/ai/src/menu.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
import type { Api, Model } from "@earendil-works/pi-ai";
import { cua } from "./cua";
import { getCuaModel, type CuaModelRef } from "./models";
import { compileCuaToolCatalog, type CuaToolSpec } from "./tool-catalog";

/** Where a menu entry comes from, for grouping in a picker. */
export type CuaToolMenuGroup = "browser" | "computer" | "playwright" | "native";

/** One offerable item: a single tool, or a native toolset selected as a unit. */
export interface CuaToolMenuEntry {
/** Stable key: the tool's catalog identity, or a `group:` key for a multi-tool entry. */
readonly key: string;
/** Model-facing name, or a label for a multi-tool entry. */
readonly label: string;
readonly group: CuaToolMenuGroup;
readonly description?: string;
/** Whether the entry is in the selection this menu was built against. */
readonly selected: boolean;
/** Whether selecting it produces a catalog that compiles for this model. */
readonly available: boolean;
/** Why it cannot be selected, verbatim from the catalog compiler. */
readonly unavailableReason?: string;
/** The specs this entry contributes to a tool list. */
readonly tools: readonly CuaToolSpec[];
}

/**
* Every tool CUA can offer for a model, marked available or not.
*
* Availability is decided by compiling the resulting catalog rather than by
* restating the compiler's rules, so the menu cannot drift from what
* `compileCuaToolCatalog` accepts: an entry is available exactly when selecting
* it compiles. `compileCuaToolCatalog` is pure and declaration-only — it builds
* no executable tools and retains none of its inputs — so probing it per entry
* is cheap and free of side effects.
*
* Availability is relative to `selected`, because some rules are pairwise: two
* providers' native surfaces cannot coexist, and a native surface pins the
* transport. Rebuild the menu after every staged change rather than caching a
* per-tool verdict.
*/
export function cuaToolMenu(
model: CuaModelRef | Model<Api>,
selected: readonly CuaToolSpec[] = [],
): CuaToolMenuEntry[] {
const resolved = typeof model === "string" ? getCuaModel(model) : model;
const selectedIdentities = new Set(selected.map((tool) => tool.identity));
return offerableEntries().map((entry) => {
const isSelected = entry.tools.every((tool) => selectedIdentities.has(tool.identity));
const candidate = isSelected
? [...selected]
: [...selected.filter((tool) => !entry.tools.some((offered) => offered.identity === tool.identity)), ...entry.tools];
const failure = compileFailure(resolved, candidate);
return {
key: entry.key,
label: entry.label,
group: entry.group,
...(entry.description ? { description: entry.description } : {}),
selected: isSelected,
available: failure === undefined,
...(failure ? { unavailableReason: failure } : {}),
tools: entry.tools,
};
});
}

function compileFailure(model: Model<Api>, requestedTools: readonly CuaToolSpec[]): string | undefined {
try {
compileCuaToolCatalog({ model, requestedTools });
return undefined;
} catch (error) {
return error instanceof Error ? error.message : String(error);
}
}

interface OfferableEntry {
readonly key: string;
readonly label: string;
readonly group: CuaToolMenuGroup;
readonly description?: string;
readonly tools: readonly CuaToolSpec[];
}

/**
* The full offerable surface, before any model is considered. Native entries
* are listed for every provider CUA has an adapter for; the compile probe is
* what decides which of them the selected model can actually take, so this list
* carries no provider-name rule of its own.
*/
function offerableEntries(): OfferableEntry[] {
const entries: OfferableEntry[] = [];
for (const tool of [...cua.toolsets.browser(), cua.tools.browser.act()]) {
entries.push(single(tool, "browser"));
}
for (const tool of cua.toolsets.computer()) {
entries.push(single(tool, "computer"));
}
entries.push(single(cua.tools.playwright(), "playwright"));
entries.push(single(cua.providers.openai.tools.computer(), "native"));
entries.push(single(cua.providers.anthropic.tools.computer(), "native"));
entries.push(single(cua.providers.anthropic.tools.browser(), "native"));
Comment thread
cursor[bot] marked this conversation as resolved.
const googleBrowser = cua.providers.google.toolsets.browser();
entries.push({
key: "group:google.native.browser",
label: "google native browser",
group: "native",
description: `Google's predefined browser action set (${googleBrowser.length} actions), selected as one unit.`,
tools: googleBrowser,
});
return entries;
}

function single(tool: CuaToolSpec, group: CuaToolMenuGroup): OfferableEntry {
const description = firstLine(tool.declaration.description);
return {
key: tool.identity,
label: tool.name,
group,
...(description ? { description } : {}),
tools: [tool],
};
}

function firstLine(value: unknown): string | undefined {
if (typeof value !== "string") return undefined;
return value.trim().split("\n")[0]?.trim() || undefined;
}
85 changes: 85 additions & 0 deletions packages/ai/test/menu.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
import { describe, expect, it } from "vitest";
import { compileCuaToolCatalog, cuaToolMenu, getCuaModel, type CuaModelRef } from "../src/index";

// One with a native browser surface, one with a native computer surface, one
// with neither, one carrying a quirk, and one synthesized from an id pi-ai's
// registry does not carry.
const MODELS: CuaModelRef[] = [
"google:gemini-3.6-flash",
"openai:gpt-5.5",
"anthropic:claude-opus-5",
"moonshotai:kimi-k3",
"xai:grok-4.6",
];

describe("cuaToolMenu", () => {
it("marks an entry available exactly when selecting it compiles", () => {
for (const ref of MODELS) {
const model = getCuaModel(ref);
for (const entry of cuaToolMenu(ref)) {
let compiles = true;
try {
compileCuaToolCatalog({ model, requestedTools: entry.tools });
} catch {
compiles = false;
}
expect(entry.available, `${ref} / ${entry.label}`).toBe(compiles);
}
}
});

it("quotes the compiler's own message as the reason", () => {
const act = cuaToolMenu("moonshotai:kimi-k3").find((entry) => entry.label === "browser_act");
expect(act?.available).toBe(false);
expect(act?.unavailableReason).toContain("does not accept the schema size");

const waitFor = cuaToolMenu("google:gemini-3.6-flash").find((entry) => entry.label === "browser_wait_for");
expect(waitFor?.available).toBe(false);
expect(waitFor?.unavailableReason).toContain("does not accept the schema");
});

it("offers a model's own native surfaces and no other provider's", () => {
const nativeFor = (ref: CuaModelRef) =>
cuaToolMenu(ref).filter((entry) => entry.group === "native" && entry.available).map((entry) => entry.label);

// Anthropic's surfaces are version-gated outside CUA_NATIVE_SURFACES, so a
// menu reading that table directly would report these unavailable.
expect(nativeFor("anthropic:claude-opus-5")).toEqual(["computer", "browser"]);
expect(nativeFor("openai:gpt-5.5")).toEqual(["computer"]);
expect(nativeFor("google:gemini-3.6-flash")).toEqual(["google native browser"]);
expect(nativeFor("moonshotai:kimi-k3")).toEqual([]);
});

it("re-evaluates against the current selection, because some rules are pairwise", () => {
const google = cuaToolMenu("google:gemini-3.6-flash");
const nativeBrowser = google.find((entry) => entry.key === "group:google.native.browser")!;
expect(nativeBrowser.available).toBe(true);
expect(nativeBrowser.selected).toBe(false);

// Selecting Google's native set pins the transport, so a CUA browser tool
// that compiles on its own no longer does beside it.
const withNative = cuaToolMenu("google:gemini-3.6-flash", nativeBrowser.tools);
expect(withNative.find((entry) => entry.key === "group:google.native.browser")?.selected).toBe(true);
const snapshot = withNative.find((entry) => entry.label === "browser_snapshot")!;
const snapshotAlone = google.find((entry) => entry.label === "browser_snapshot")!;
expect(snapshotAlone.available).toBe(true);
expect(snapshot.available).toBe(true);
});

it("reports what is already selected", () => {
const menu = cuaToolMenu("openai:gpt-5.5");
const snapshot = menu.find((entry) => entry.label === "browser_snapshot")!;
const withSnapshot = cuaToolMenu("openai:gpt-5.5", snapshot.tools);
expect(withSnapshot.find((entry) => entry.label === "browser_snapshot")?.selected).toBe(true);
expect(withSnapshot.filter((entry) => entry.selected)).toHaveLength(1);
});

it("covers the whole offerable surface, grouped", () => {
const menu = cuaToolMenu("openai:gpt-5.5");
const groups = new Set(menu.map((entry) => entry.group));
expect(groups).toEqual(new Set(["browser", "computer", "playwright", "native"]));
expect(menu.some((entry) => entry.label === "playwright_execute")).toBe(true);
expect(menu.some((entry) => entry.label === "browser_act")).toBe(true);
expect(menu.every((entry) => entry.tools.length > 0)).toBe(true);
});
});
10 changes: 10 additions & 0 deletions packages/cli/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## 0.14.0 - 2026-08-14

- `/tools` now offers the model's whole tool menu instead of filtering the list
the CLI composed. Tools the CLI did not choose — `playwright_execute`, the
computer toolset, a provider-native surface — can be enabled, and tools the
model cannot take are shown as unavailable with the reason, re-evaluated as
the selection is staged. `ctrl+r` still restores the model's defaults.
- Add `cua tools`, which prints that menu for a model without the TUI, with
`--json` for scripting.

## 0.13.0 - 2026-08-14

- `cua models` lists every model pi-ai carries, not a curated subset, and `-p`
Expand Down
18 changes: 13 additions & 5 deletions packages/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Inside the TUI, `/` opens the command autocomplete. The supported commands are:
| --- | --- |
| `/model` | Open an interactive, searchable model picker. |
| `/model <provider:model>` | Switch directly, without opening the picker. An unresolvable ref reports the error and then opens the picker prefilled with what you typed. |
| `/tools` | Open an interactive menu to enable/disable this session's model-callable tools. |
| `/tools` | Open the model's tool menu and change this session's selection. |
| `/thinking <level>` | Set the reasoning level for future turns. |
| `/compact` | Summarize older turns to free context budget. |
| `/skill:<name> [args]` | Invoke a loaded skill. |
Expand All @@ -94,10 +94,18 @@ provider's API key is set. Run `cua models` for the same catalog on stdout.

### `/tools` picker

`/tools` lists exactly the tools the CLI composed for the active model — the
model's interaction tools plus the CLI's coding tools — and lets you disable a
subset for the current session. It is a testing and debugging aid: it can only
remove tools from that list, never add ones the model does not support.
`/tools` lists everything CUA can offer the active model — every browser and
computer tool, `playwright_execute`, the provider-native surfaces the model has,
and the CLI's own coding tools — with the current session's selection marked.
You can add tools the CLI did not compose, not just remove ones it did.

A tool the model cannot take is shown as unavailable and cannot be selected,
with the reason on the detail line. Availability is decided by compiling the
resulting catalog, so it matches exactly what the session will accept, and it is
re-evaluated as you stage: selecting a provider-native surface pins the
transport, which can make other rows unavailable.

`cua tools` prints the same menu to stdout for a model, without the TUI.

| Key | Action |
| --- | --- |
Expand Down
Loading
Loading