Stream OpenAI through pi-ai's builtin Responses transport

[rgarcia]

Summary

OpenAI models were routed to a CUA-owned openai-cua-responses api that threaded previous_response_id with store: true and sent only the messages after the last stored response. The premise was token savings. That does not hold: a threaded request still bills the reconstructed prompt, with the matched prefix charged at the cache-read rate — cua's own usageFromResponse had to subtract cached_tokens out of input_tokens to account for exactly that.

pi-ai's builtin Responses transport gets the same caching from prompt_cache_key + prompt_cache_retention with store: false, and replays reasoning through include: ["reasoning.encrypted_content"]. It also adds prompt_cache_options, strict-mode schema conversion, grammar-tool conversion, and cache-write usage that the fork lacked.

So OpenAI models now keep pi-ai's builtin openai-responses api id and stream through its transport by default.

What changed

• routeCuaApi no longer reroutes OpenAI; OPENAI_CUA_RESPONSES_API is removed.
• The CUA OpenAI adapter is retained but dispatches on request shape rather than a rerouted api id (requiresCuaOpenAIAdapter): OpenAI's native computer tool, or a transcript carrying a deferred tool-search addition or a replayed function-call namespace. pi-ai 0.83.0 still does not round-trip function-call namespaces, so that path is load-bearing — packages/agent/test/openai-deferred-tools.test.ts is the regression gate.
• Escalated requests now send prompt_cache_options and the same strict-mode/grammar tool options as pi's builtin, so the cached prefix survives a mid-conversation switch between the two paths.
• Google, Meta, xAI, and Tzafon threading is untouched. Each either owns its whole transport or threads a provider-specific continuation field (previous_interaction_id, Lightcone's required screenshot history) with no builtin equivalent. responseThreading still governs those four; its JSDoc and the changelogs now say so.

Two consequences of stateless replay, both handled here:

• OpenAI native computer results are no longer held in provider-stored state, so computer joins Tzafon in the exemption from the tool-result image replay limit. Without it, computer_call_output items past the 4-image default would serialize with no screenshot.
• Transcript namespaces are paired by call_id instead of by position. Converting the full transcript exposed an ordinal bug: a dropped aborted assistant message shifted every later namespace.

Measurement

--print -o jsonl gains an assistant_usage event (schema version 2) with input, output, cache_read, cache_write, reasoning, total_tokens, and a derived cache_hit_ratio. Nothing surfaced usage before this, so there was no way to compare.

Billed prompt tokens are input + cache_read + cache_write (the provider already subtracts cached and cache-write tokens out of input), and the ratio is cache_read over that total.

Measured, 10 live runs on openai:gpt-5.5 against a fixed Wikipedia browsing task (3 runs/arm) and a screenshot-heavy variant (2 runs/arm), same prompt and model both arms, each run's transport verified from the api field:

	billed prompt tokens/turn	full-price prompt tokens/turn	cache hit ratio	cost/turn (list)
snapshot task, threaded	162,379	29,828	0.816	$0.2171
snapshot task, stateless	67,194	3,649	0.946	$0.0517
screenshot task, threaded	219,270	39,935	0.817	$0.2903
screenshot task, stateless	57,988	6,395	0.890	$0.0600

Billed prompt tokens per turn fell 59% and 74%; cost per turn fell 76% and 79%; the cache hit ratio went up in both. Run totals: $1.95 -> $0.47 and $3.19 -> $0.96.

The mechanism is the one the image bound hinted at, and it is broader than images. Under previous_response_id the provider reconstructs the entire stored conversation, so nothing the client does — toolResultImageReplayLimit, context projection — can shrink what gets billed. Stateless replay sends a bounded projected transcript instead. Threaded per-turn cost also climbed run over run (0.197 -> 0.236 across three identical runs) while stateless stayed flat (0.0513 -> 0.0520).

The cost figures understate the gap: 11 of 47 threaded requests exceeded gpt-5.5's 272k-token threshold into its 2x price tier, and the table prices everything at the base rate. Zero of 57 stateless requests came close (max 79,786).

One methodology note for anyone reproducing this: run the CLI with NODE_OPTIONS=--conditions=source. Without it, @onkernel/cua-ai resolves to dist/, and a stale build silently makes both arms run the same transport.

Testing

• npm run typecheck — clean.
• npm test per workspace: cua-ai 112 passed, cua-agent 292 passed / 19 skipped, cua-cli 142 passed / 14 skipped.
• Deleted openai-threading.test.ts; added openai-adapter-routing.test.ts covering all three dispatch cases plus the default request shape (store: false, prompt_cache_key present, no previous_response_id), and an agent-level regression test for the native-computer image exemption.
• No live API validation. This ran without OpenAI or Kernel credentials, so the A/B above is unmeasured and per-model smoke runs (gpt-5.6-sol, gpt-5.5, gpt-5.4, gpt-5.4-mini) have not happened. pi's strict-mode converter can rewrite tool schemas in ways mocked-SDK tests will not catch.

Notes for review

• getCuaModel("openai:*").api changes from "openai-cua-responses" to "openai-responses". Persisted session transcripts carry api per assistant message, so resumed sessions will contain mixed ids. Nothing reads that field for behavior once threading is gone.
• Losing store: true also means OpenAI no longer retains these conversations server-side, so they are no longer retrievable by response id for debugging.
• Changelog entries are stamped 0.11.0 / 0.10.0 dated today to match the existing heading convention; the release flow re-dates them at publish.

---

Note

High Risk
This changes core LLM transport, billing/context behavior for OpenAI and Meta/xAI, and multi-turn tool/computer protocols; regressions could affect cost, deferred tools, and native computer loops without live API validation noted in the PR.

Overview
OpenAI no longer uses a CUA-owned openai-cua-responses api or previous_response_id / store: true threading. Models stay on pi-ai’s builtin openai-responses transport with stateless replay and automatic prompt caching. The CUA adapter runs only when requiresCuaOpenAIAdapter says so: native computer tool, deferred tool-search additions, or transcripts that need function-call namespace round-trips pi’s transport drops.

The adapter’s function-tool path drops threading, aligns prompt cache / strict / grammar options with pi, and pairs replayed namespaces by call_id instead of transcript position (fixes shifts when aborted assistant turns are omitted). Native computer requests now send the same cache fields with store: false.

Meta and xAI lose custom Responses forks; Meta registers pi’s builtin transport on its base URL, Grok uses pi’s xAI provider. responseThreading on the agent applies only to Google and Tzafon; OpenAI’s native computer tool joins Tzafon in skipping the tool-result image replay cap.

CLI --print -o jsonl bumps to schema v2 with per-turn assistant_usage (tokens, cache fields, cache_hit_ratio). Docs/changelogs and tests reflect routing, adapter dispatch, and removed threading suites.

^{Reviewed by Cursor Bugbot for commit 3f8238c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Known follow-up

The native computer path no longer has multi-turn reasoning continuity. Under threading, OpenAI held reasoning items server-side; stateless replay drops them, and this adapter's own convertMessages has no reasoning-item replay to compensate. Adding include: ["reasoning.encrypted_content"] alone would not fix it — the encrypted content would come back and never be resent. Implementing replay requires emitting each reasoning item in the exact position its following call demands (a 400 if wrong), which needs a live OpenAI key to validate, so it is deliberately out of this PR.

What else got cut, and what provably cannot

The first pass left the OpenAI adapter in place and only routed around it, which came out net +54 source lines. Second pass:

Deleted (−355 lines): the xAI and Meta Responses forks. Both existed only to thread previous_response_id and to set parallel_tool_calls: false — and the catalog already emits that field for every provider its capabilities mark as serializing state mutations, xAI and Meta included. pi's registry gives grok-4.5 openai-responses plus xAI's base URL outright, so xAI is now pi's builtin provider untouched; Meta keeps only its provider registration. Verified live: both complete a multi-turn browser task with no errors and 93–99% cache hits on later turns.

Kept, with proof: the OpenAI function-tool adapter. Replaying a tool-search-loaded call through pi's builtin transport returns a hard 400:

Missing namespace for function_call 'lookup'. It does not exist in the default
namespace. Round-trip the model's function_call item with its namespace field
included.  (param: input[6].namespace)

pi never emits or parses that field. So the adapter is load-bearing, and the escalation predicate is what keeps any session that loads a deferred tool from 400ing.

Deliberately not touched: OpenAI's native computer adapter (~210 lines). Deleting it drops a documented product surface, not dead code — that belongs to the "do native tools stay first-class?" decision, not this PR.

Net effect on the branch: source −90, tests −32, docs/changelogs +62.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Native path loses prompt caching
  • The native OpenAI computer path now applies the same prompt-cache fields, session headers, and reasoning encrypted-content include behavior as the function-tool path while keeping store disabled.

Or push these changes by commenting:

@cursor push 3343dcdff2

Preview (3343dcdff2)

diff --git a/packages/ai/src/providers/openai/provider.ts b/packages/ai/src/providers/openai/provider.ts
--- a/packages/ai/src/providers/openai/provider.ts
+++ b/packages/ai/src/providers/openai/provider.ts
@@ -281,26 +281,41 @@
 	const output = initialAssistantMessage(model);
 	void (async () => {
 		try {
-			const apiKey = options?.apiKey || process.env.OPENAI_API_KEY;
-			if (!apiKey) throw new Error("No API key for provider: openai");
+			const openAIOptions = options as OpenAIResponsesOptions | undefined;
+			const apiKey = openAIApiKey(openAIOptions);
 			const nativeName = options?.cuaIncomingToolPlan?.openaiComputerName;
 			if (!nativeName) throw new Error("OpenAI native computer incoming plan is missing");
 			const placement = splitDeferredTools(context);
+			const retention = cacheRetention(openAIOptions);
 			let payload: Record<string, unknown> = {
 				model: model.id,
 				instructions: context.systemPrompt,
 				input: convertMessages(context.messages, nativeName, placement.deferred),
 				tools: convertTools(placement.immediate),
 				max_output_tokens: options?.maxTokens ?? model.maxTokens,
+				prompt_cache_key: retention === "none" ? undefined : clampOpenAIPromptCacheKey(openAIOptions?.sessionId),
+				prompt_cache_retention: retention === "long" && model.compat?.supportsLongCacheRetention !== false ? "24h" : undefined,
+				prompt_cache_options: retention === "none" && model.compat?.supportsExplicitPromptCacheMode ? { mode: "explicit" } : undefined,
 				store: false,
 			};
+			if (model.reasoning) {
+				if (openAIOptions?.reasoningEffort || openAIOptions?.reasoningSummary) {
+					const effort = openAIOptions.reasoningEffort
+						? (model.thinkingLevelMap?.[openAIOptions.reasoningEffort] ?? openAIOptions.reasoningEffort)
+						: "medium";
+					payload.reasoning = { effort, summary: openAIOptions.reasoningSummary || "auto" };
+					payload.include = ["reasoning.encrypted_content"];
+				} else if (model.thinkingLevelMap?.off !== null) {
+					payload.reasoning = { effort: model.thinkingLevelMap?.off ?? "none" };
+				}
+			}
 			payload = ((await options?.onPayload?.(payload, model)) ?? payload) as Record<string, unknown>;
-			const client = new OpenAI({
-				apiKey,
-				baseURL: model.baseUrl || "https://api.openai.com/v1",
-				defaultHeaders: { ...model.headers, ...options?.headers },
+			const client = createOpenAIClient(model, openAIOptions, apiKey);
+			const request = client.responses.create(payload as never, {
+				...(options?.signal ? { signal: options.signal } : {}),
+				...(options?.timeoutMs !== undefined ? { timeout: options.timeoutMs } : {}),
+				maxRetries: options?.maxRetries ?? 0,
 			});
-			const request = client.responses.create(payload as never, { signal: options?.signal });
 			const { data: response, response: rawResponse } = await request.withResponse();
 			await options?.onResponse?.({ status: rawResponse.status, headers: headersToRecord(rawResponse.headers) }, model);
 			if (options?.signal?.aborted) throw new Error("Request was aborted");

diff --git a/packages/ai/test/openai-native-provider.test.ts b/packages/ai/test/openai-native-provider.test.ts
--- a/packages/ai/test/openai-native-provider.test.ts
+++ b/packages/ai/test/openai-native-provider.test.ts
@@ -3,10 +3,17 @@
 import { getCuaModel } from "../src/index";
 import * as openai from "../src/providers/openai/provider";
 
-const { responsesCreate } = vi.hoisted(() => ({ responsesCreate: vi.fn() }));
+const { openaiClientOptions, responsesCreate } = vi.hoisted(() => ({
+	openaiClientOptions: vi.fn(),
+	responsesCreate: vi.fn(),
+}));
 
 vi.mock("openai", () => ({
 	default: class {
+		constructor(options: unknown) {
+			openaiClientOptions(options);
+		}
+
 		responses = {
 			create: (...args: unknown[]) => ({
 				withResponse: async () => ({ data: responsesCreate(...args), response: { status: 200, headers: new Headers() } }),
@@ -35,6 +42,8 @@
 			tools: [{ name: "computer", description: "placeholder", parameters: { type: "object" } as never }],
 		}, {
 			apiKey: "test",
+			sessionId: "native_session",
+			reasoningEffort: "medium",
 			cuaIncomingToolPlan: incoming,
 			onPayload: (payload) => ({ ...(payload as Record<string, unknown>), tools: [{ type: "computer" }] }),
 		}).result();
@@ -50,7 +59,15 @@
 
 		const payload = responsesCreate.mock.calls.at(-1)?.[0] as Record<string, unknown>;
 		expect(payload.store).toBe(false);
+		expect(payload.prompt_cache_key).toBe("native_session");
+		expect(payload.include).toEqual(["reasoning.encrypted_content"]);
 		expect(payload.previous_response_id).toBeUndefined();
+		expect(openaiClientOptions).toHaveBeenCalledWith(expect.objectContaining({
+			defaultHeaders: expect.objectContaining({
+				session_id: "native_session",
+				"x-client-request-id": "native_session",
+			}),
+		}));
 	});
 
 	it("round-trips function-call namespaces beside the native computer adapter", async () => {

_{You can send follow-ups to the cloud agent here.}

[rgarcia]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 85ed1f2. Configure here.}

[rgarcia]

Ran the A/B live on openai:gpt-5.5, 10 runs total, both arms verified by the api field on each assistant message (openai-cua-responses vs openai-responses).

Billed prompt tokens per turn: −59% on a snapshot-driven browsing task, −74% on a screenshot-heavy one. Cost per turn: −76% and −79%. Cache hit ratio went up, not down (0.816 → 0.946 and 0.817 → 0.890).

Threading was not saving tokens, it was multiplying them. With store: true + previous_response_id the provider reconstructs the whole stored conversation each turn, so no client-side bound — including toolResultImageReplayLimit — can shrink what gets billed. Threaded cost per turn also grew run over run ($0.197 → $0.236 across three identical runs) while the stateless arm stayed flat ($0.0513 → $0.0520).

The cost numbers are conservative: 11 of 47 threaded requests crossed gpt-5.5's 272k-token threshold into its 2x tier, priced here at the base rate anyway. Zero of 57 stateless requests got near it (max 79,786 billed prompt tokens).

Details and the reproduction procedure are in the PR description.

[rgarcia]

Cut more. Source is now net −90 lines (was +54); the whole branch is net −60.

The xAI and Meta Responses forks are gone (−355). Both were only threading plus parallel_tool_calls: false, and the catalog already emits that field for those providers, so with threading removed neither fork did anything pi's builtin transport doesn't. Live-checked both models on a multi-turn browser task: no errors, 93–99% cache hits on later turns.

The OpenAI function-tool adapter stays, and now there is a reason on the record rather than an assumption. Replaying a tool-search-loaded call through pi's builtin transport 400s:

Missing namespace for function_call 'lookup'. It does not exist in the default namespace. Round-trip the model's function_call item with its namespace field included.

pi never emits or parses namespace, so that adapter — and the escalation predicate that routes to it — is what keeps deferred-tool sessions working.

Left alone on purpose: OpenAI's native computer adapter (~210 lines). That is a product surface with docs and templates behind it, so it belongs to the native-tools decision, not this cleanup.

bugbot run