Skip to content

Add attach(), the handle the agent classes will retire into #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rgarcia merged 3 commits into from
Aug 14, 2026
Merged

Add attach(), the handle the agent classes will retire into #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
327f36a
Add attach(), the handle the agent classes will retire into
rgarcia Aug 14, 2026
c025d40
Collapse per-merge changelog headings into Unreleased
rgarcia Aug 14, 2026
a7de6f8
Spend an empty-response retry only once the follow-up is queued
rgarcia Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions .agents/skills/release/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,25 +93,38 @@ and every version returned by `npm view <package> versions --json`.

## Changelog

Update only the changelog for packages being released:
Changes land under a `## Unreleased` heading as they merge, so every changelog
has at most one unreleased section:

- `packages/ai/CHANGELOG.md`
- `packages/agent/CHANGELOG.md`
- `packages/cli/CHANGELOG.md`

Add a new top entry:
Releasing renames that heading in place — do not add a second top entry:

```markdown
## <version> - YYYY-MM-DD

- ...
```

Then read the section as a whole before tagging. It accumulated over several
merges, so it can carry entries that contradict each other or describe a state
that never shipped: an API added and then removed, or a note that a provider
"keeps" a behavior when a later entry deletes that provider. Consumers upgrade
from the previous release, not through the intermediate steps, so collapse
those into the net change and drop what nobody can observe. Cross-package
"update `@onkernel/cua-ai` to X" notes belong here too — the version is not
known until this step.

Write customer-facing changes. Do not dump commit subjects, internal issue
names, Slack context, or vague entries like "misc improvements." Group details
only when it improves readability. If the release is only metadata or docs,
say that plainly.

Merges between releases add to `## Unreleased`, creating it directly under
`# Changelog` when it is absent. Never invent a version heading for a merge:
package versions are chosen at release time from the accumulated changes, and a
per-merge heading claims a release that never happened.

## Edit Release Metadata

Set versions explicitly:
Expand Down
90 changes: 43 additions & 47 deletions packages/agent/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,55 @@
# Changelog

## 0.14.0 - 2026-08-14

- `CuaAgentHarness` no longer refuses a model ref that is absent from its
supplied `Models` collection: it falls back to the registry, and an id the
registry lacks is synthesized. Update `@onkernel/cua-ai` to 0.14.0.

## 0.13.0 - 2026-08-13

Breaking: Tzafon and Yutori support is removed.

- Update `@onkernel/cua-ai` to 0.13.0. Constructing a `CuaAgent` or
`CuaAgentHarness` with a Tzafon or Yutori model ref now fails to resolve the
model, and `cua.providers.tzafon` / `cua.providers.yutori` no longer exist.
- The tool-result image replay limit now exempts only OpenAI's native computer
tool, whose protocol requires every `computer_call_output` to carry a
screenshot. Tzafon's native computer results were exempt for the same reason
and are gone with the provider.
- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog
viewport option; the same value is still on `resources.browser.viewport`.

## 0.12.0 - 2026-08-13

## Unreleased

- Add `attach({ browser, client })`, returning a handle that compiles
(model, tools) pairs into plain pi objects: the model carrying the transport
its tools derive, executables materialized against the handle's browser pool,
a `Models` collection adding provider retry, required headers, the catalog's
payload transforms and the tool-result image bound, and an `install(harness)`
for the behaviors that are pi event handlers rather than constructor options.
The handle owns what actually persists — the Kernel client and browser, the
translator, the raw-CDP executor, ref and frame state — so a spec materializes
once across repeat compiles.
- `CuaAgent` and `CuaAgentHarness` are unchanged and now share their internals
with `attach()` rather than owning private copies. They are slated to retire
in favor of the handle.
- Add `CuaAgentHarness.setModelAndTools()`. A model switch that also swaps
interaction tools has to compile as one pair now that the selected tools
decide the transport: staging the two in sequence produces an intermediate
catalog whose derived transport differs from both the old and the new one,
and records a model change for a transport nothing ever streamed with.

- Update `@onkernel/cua-ai` to 0.12.0. The model streamed for a Google model
now depends on which tools `CuaAgent`/`CuaAgentHarness` were constructed or
mutated with: selecting Google's native browser toolset still compiles to
the CUA-owned Interactions API, but a Google model selected with only CDP
browser tools now streams through pi's builtin Google transport instead of
always carrying the CUA-owned api. This applies uniformly across
construction, `setTools()`, and `setModel()`, since all three feed the same
compiled `catalog.model` into pi.
catalog whose derived transport differs from both the old and the new one, and
records a model change for a transport nothing ever streamed with.
- `CuaAgentHarness` no longer refuses a model ref that is absent from its
supplied `Models` collection: it falls back to the registry, and an id the
registry lacks is synthesized.
- The model streamed for a Google model now depends on which tools `CuaAgent` /
`CuaAgentHarness` were constructed or mutated with: selecting Google's native
browser toolset still compiles to the CUA-owned Interactions API, but a Google
model selected with only CDP browser tools now streams through pi's builtin
Google transport instead of always carrying the CUA-owned api. This applies
uniformly across construction, `setTools()`, and `setModel()`, since all three
feed the same compiled `catalog.model` into pi.
- Fix `setTools()` recompiling from the previously *compiled* model instead of
the caller's model selection: dropping a native toolset that had derived a
tool-selection-dependent api (e.g. Google's Interactions API) left
subsequent tools-only recompiles stuck on that api even though the new
selection no longer required it. `CuaToolManager` now recompiles tools-only
changes from the model input the caller last selected.

## 0.11.0 - 2026-08-13

tool-selection-dependent api (e.g. Google's Interactions API) left subsequent
tools-only recompiles stuck on that api even though the new selection no
longer required it. `CuaToolManager` now recompiles tools-only changes from
the model input the caller last selected.
- `responseThreading` (`CuaAgentOptions`/`CuaAgentHarnessOptions`) no longer
affects OpenAI models: OpenAI now streams through pi-ai's builtin Responses
transport and its automatic prompt caching regardless of this flag. The
option still governs Google and Tzafon's `previous_response_id`-style
continuation.
- Exempt OpenAI's native computer tool from the tool-result image replay
limit, alongside Tzafon: its `computer_call_output` items must each carry a
screenshot, and stateless replay no longer leaves them in provider-stored
state.
transport and its automatic prompt caching regardless of this flag. The option
still governs Google's `previous_response_id`-style continuation.
- Exempt OpenAI's native computer tool from the tool-result image replay limit.
Its `computer_call_output` items must each carry a screenshot, and stateless
replay no longer leaves them in provider-stored state.

Breaking: Tzafon and Yutori support is removed.

- Constructing a `CuaAgent` or `CuaAgentHarness` with a Tzafon or Yutori model
ref now fails to resolve the model, and `cua.providers.tzafon` /
`cua.providers.yutori` no longer exist.
- Remove `CuaExecutionResources.viewport`. It only fed the removed catalog
viewport option; the same value is still on `resources.browser.viewport`.

## 0.10.0 - 2026-08-04

Expand Down
174 changes: 17 additions & 157 deletions packages/agent/src/agent.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,25 +45,25 @@ import {
} from "./provider-retry";
import { CuaExecutionResources, type CuaExecutionDetails } from "./resources";
import { CuaToolManager, type CuaAgentTool, type CuaHarnessTool } from "./tool-manager";
import {
type CuaEmptyResponseRecoveryOptions,
type CuaModelInput,
defaultCuaStream,
hasExecutionError,
isEmptyAssistantResponse,
modelTransportChanged,
projectToolResultImages,
requiredImageToolNames,
resolveEmptyResponseRecovery,
resolveModelFromCollection,
resolveResponseThreading,
resolveToolResultImageReplayLimit,
type ToolResultImageReplayLimit,
turnFailureStopMessage,
withCatalogModels,
} from "./attach";
import type { KernelBrowser } from "./translator/translator";

/** A registered CUA model reference or an already resolved pi model. */
export type CuaModelInput = CuaModelRef | Model<Api>;

const DEFAULT_TOOL_RESULT_IMAGE_REPLAY_LIMIT = 4;
const OMITTED_TOOL_RESULT_IMAGES = "[stale tool-result images omitted]";

/** Maximum recent tool-result images retained in model context, or `false` to retain all images. Provider-required native tool images are always retained. */
export type ToolResultImageReplayLimit = number | false;

/** Optional follow-up policy for otherwise empty successful assistant responses. */
export interface CuaEmptyResponseRecoveryOptions {
/** User message queued to ask the model to continue. */
followUp: string;
/** Maximum automatic follow-ups per prompt. */
maxAttempts: number;
}

/** Mutable conversation state exposed by {@link CuaAgent}. */
export interface CuaAgentState {
systemPrompt: string;
Expand Down Expand Up @@ -493,143 +493,3 @@ export class CuaAgentHarness<
await this.tools.resources.dispose();
}
}

const defaultCuaStream: StreamFn = (model, context, options) => cuaModels().streamSimple(model, context, options);

function resolveModelFromCollection(ref: CuaModelRef, models: Models): Model<Api> {
const { provider, model: id } = parseCuaModelRef(ref);
return models.getModel(provider, id) ?? getCuaModel(ref);
}

/** Whether a tools-only recompile actually changed the model pi streams with, so `setTools()` only pushes `setModel()` (and its session/event side effects) when the derived transport moved. */
function modelTransportChanged(previous: Model<Api>, next: Model<Api>): boolean {
return previous.provider !== next.provider || previous.id !== next.id || previous.api !== next.api;
}

function withCatalogModels(
models: Models,
manager: CuaToolManager<any>,
imageReplayLimit: ToolResultImageReplayLimit,
responseThreading: boolean,
): Models {
const contextFor = (context: Context) => projectModelContext(
context,
imageReplayLimit,
requiredImageToolNames(manager.catalog.incoming),
);
const optionsFor = <T extends SimpleStreamOptions | undefined>(options: T): T => {
const catalog = manager.catalog;
const callerOnPayload = options?.onPayload;
return {
...options,
headers: catalog.headers.merge(options?.headers),
disableResponseThreading: responseThreading ? undefined : true,
cuaIncomingToolPlan: catalog.incoming,
onPayload: async (payload: unknown, model: Model<Api>) => {
const generated = await catalog.payload.apply(payload, model);
return callerOnPayload ? (await callerOnPayload(generated, model)) ?? generated : generated;
},
} as T;
};
return {
getProviders: () => models.getProviders(),
getProvider: (id) => models.getProvider(id),
getModels: (provider) => models.getModels(provider),
getModel: (provider, id) => models.getModel(provider, id),
refresh: (provider) => models.refresh(provider),
getAuth: (input, overrides) => models.getAuth(input as never, overrides),
checkAuth: (providerId) => models.checkAuth(providerId),
getAvailable: (providerId) => models.getAvailable(providerId),
login: (providerId, type, interaction) => models.login(providerId, type, interaction),
logout: (providerId) => models.logout(providerId),
stream: (model, context, options) => models.stream(model, contextFor(context), optionsFor(options)),
complete: (model, context, options) => models.complete(model, contextFor(context), optionsFor(options)),
streamSimple: (model, context, options) => models.streamSimple(model, contextFor(context), optionsFor(options)),
completeSimple: (model, context, options) => models.completeSimple(model, contextFor(context), optionsFor(options)),
};
}

function resolveToolResultImageReplayLimit(limit: ToolResultImageReplayLimit | undefined): ToolResultImageReplayLimit {
if (limit === undefined) return DEFAULT_TOOL_RESULT_IMAGE_REPLAY_LIMIT;
if (limit !== false && (!Number.isFinite(limit) || !Number.isInteger(limit) || limit < 0)) {
throw new TypeError("toolResultImageReplayLimit must be a finite non-negative integer or false");
}
return limit;
}

/** Native computer tool names whose screenshot history the provider protocol requires in full, regardless of the image replay limit. */
function requiredImageToolNames(incoming: CuaIncomingToolPlan): ReadonlySet<string> {
return new Set(incoming.openaiComputerName ? [incoming.openaiComputerName] : []);
}

function projectToolResultImages<TMessage extends AgentMessage>(
messages: TMessage[],
limit: ToolResultImageReplayLimit,
requiredToolNames: ReadonlySet<string> = new Set(),
): TMessage[] {
if (limit === false) return messages;
let imageCount = 0;
for (const message of messages) {
if (message.role === "toolResult" && !requiredToolNames.has(message.toolName)) {
imageCount += message.content.filter((block) => block.type === "image").length;
}
}
if (imageCount <= limit) return messages;
const firstRetainedImage = Math.max(0, imageCount - limit);
let imageOrdinal = 0;
return messages.map((message) => {
if (message.role !== "toolResult" || requiredToolNames.has(message.toolName)) return message;
let changed = false;
let markerInserted = false;
const content = [] as typeof message.content;
for (const block of message.content) {
if (block.type !== "image" || imageOrdinal++ >= firstRetainedImage) {
content.push(block);
continue;
}
changed = true;
if (!markerInserted) {
content.push({ type: "text", text: OMITTED_TOOL_RESULT_IMAGES });
markerInserted = true;
}
}
return changed ? { ...message, content } as TMessage : message;
});
}

function projectModelContext(
context: Context,
imageReplayLimit: ToolResultImageReplayLimit,
requiredToolNames: ReadonlySet<string>,
): Context {
const messages = projectToolResultImages(context.messages, imageReplayLimit, requiredToolNames);
return messages === context.messages ? context : { ...context, messages };
}

function resolveEmptyResponseRecovery(options: CuaEmptyResponseRecoveryOptions | undefined): CuaEmptyResponseRecoveryOptions | undefined {
if (!options) return undefined;
if (options.followUp.trim().length === 0) throw new Error("emptyResponseRecovery.followUp must not be blank");
if (!Number.isInteger(options.maxAttempts) || options.maxAttempts < 0) throw new Error("emptyResponseRecovery.maxAttempts must be a non-negative finite integer");
return { followUp: options.followUp, maxAttempts: options.maxAttempts };
}

function resolveResponseThreading(value: boolean | undefined): boolean {
if (value !== undefined && typeof value !== "boolean") throw new TypeError("responseThreading must be a boolean");
return value ?? true;
}

function isEmptyAssistantResponse(message: AgentMessage): boolean {
return message.role === "assistant" && message.stopReason === "stop" && message.content.length === 0;
}

function hasExecutionError(details: unknown): boolean {
return Boolean(details && typeof details === "object" && (details as CuaExecutionDetails).isError === true);
}

function turnFailureStopMessage(manager: CuaToolManager<any>): string | undefined {
for (const entry of manager.catalog.entries) {
const execution = manager.specFor(entry.identity)?.execution;
if (execution?.kind === "actions" && execution.stopTurnOnFailureMessage) return execution.stopTurnOnFailureMessage;
}
return undefined;
}
Loading
Loading