diff --git a/docs-site/src/content/docs/ja/reference/adapters.md b/docs-site/src/content/docs/ja/reference/adapters.md index ea97ddbcff..0718edc94e 100644 --- a/docs-site/src/content/docs/ja/reference/adapters.md +++ b/docs-site/src/content/docs/ja/reference/adapters.md @@ -27,6 +27,9 @@ interface ProviderAdapter { - 内部メッセージを OpenAI role に変換し、ツールは `{type:"function", function:{…}}` と `tool_choice`(`auto`/`none`/`required` または指定関数)にマッピングします。 +- **ツール結果内の画像**は、`role:"tool"` がテキスト専用のため、ツールラウンドが閉じた後に後続の + user vision メッセージ(`image_url` パート)として送られます。ツールメッセージ側には `[image]` + マーカーがアンカーとして残ります。 - **Codex の GPT-5 アイデンティティプロンプトを書き直し**、モデル中立な紹介に変えます。そのためルーティングされたモデルが自分を OpenAI だと主張しません。 - 正確な段階がないときは **`reasoning_effort` をモデルが公表したサブセットに合わせて調整**します。 プロバイダーが明示的に alias を設定しない限り、`xhigh` と `max` は異なるラベルのまま保ちます。`provider.noReasoningModels` に含まれる id には値を **一切送りません**。 diff --git a/docs-site/src/content/docs/ko/reference/adapters.md b/docs-site/src/content/docs/ko/reference/adapters.md index ddfe868649..59bd47eac5 100644 --- a/docs-site/src/content/docs/ko/reference/adapters.md +++ b/docs-site/src/content/docs/ko/reference/adapters.md @@ -31,6 +31,9 @@ interface ProviderAdapter { - 내부 메시지를 OpenAI role로 변환하고, 툴은 `{type:"function", function:{…}}`과 `tool_choice`(`auto`/`none`/`required` 또는 지정 함수)로 매핑합니다. +- **툴 결과에 든 이미지**는 `role:"tool"`이 텍스트 전용이므로, 툴 라운드가 닫힌 뒤 후속 + user vision 메시지(`image_url` 파트)로 전달됩니다. 툴 메시지에는 `[image]` 마커가 앵커로 + 남습니다. - **Codex의 GPT-5 정체성 프롬프트를 다시 작성**해 모델 중립적인 소개로 바꿉니다. 따라서 라우팅된 모델이 자신을 OpenAI라고 주장하지 않습니다. - 정확한 단계가 없으면 **`reasoning_effort`를 모델이 알린 하위 집합에 맞춰 조정**합니다. diff --git a/docs-site/src/content/docs/reference/adapters.md b/docs-site/src/content/docs/reference/adapters.md index 25cd3b962e..19676001ac 100644 --- a/docs-site/src/content/docs/reference/adapters.md +++ b/docs-site/src/content/docs/reference/adapters.md @@ -31,6 +31,9 @@ provider — xAI, Kimi, DeepSeek, GLM, Groq, OpenRouter, Ollama (local & cloud), - Converts internal messages to OpenAI roles; maps tools to `{type:"function", function:{…}}` and `tool_choice` (`auto`/`none`/`required` or a named function). +- **Tool-result images** ride in a follow-up user vision message (`image_url` parts) released once + the tool round closes, since `role:"tool"` content is text-only; the `[image]` marker stays in the + tool message as the anchor. - **Rewrites Codex's GPT-5 identity prompt** to a model-agnostic intro so routed models don't claim to be OpenAI. - **Clamps `reasoning_effort`** to the model's advertised subset when an exact tier is unavailable; diff --git a/docs-site/src/content/docs/ru/reference/adapters.md b/docs-site/src/content/docs/ru/reference/adapters.md index 80c7ae6477..0565c280df 100644 --- a/docs-site/src/content/docs/ru/reference/adapters.md +++ b/docs-site/src/content/docs/ru/reference/adapters.md @@ -34,6 +34,9 @@ interface ProviderAdapter { - Преобразует внутренние сообщения в роли OpenAI; инструменты отображаются в `{type:"function", function:{…}}` и `tool_choice` (`auto`/`none`/`required` или именованная функция). +- **Изображения из результатов инструментов** отправляются отдельным последующим user-сообщением + (части `image_url`) после закрытия раунда инструментов, так как содержимое `role:"tool"` может + быть только текстом; маркер `[image]` остаётся в сообщении инструмента как якорь. - **Переписывает идентификационный промпт Codex про GPT-5** в модельно-нейтральное вступление, чтобы маршрутизируемые модели не заявляли, что они от OpenAI. - **Прижимает `reasoning_effort`** к объявленному моделью подмножеству, когда точный уровень diff --git a/docs-site/src/content/docs/zh-cn/reference/adapters.md b/docs-site/src/content/docs/zh-cn/reference/adapters.md index 6a3d89a98d..dd8eeb54cc 100644 --- a/docs-site/src/content/docs/zh-cn/reference/adapters.md +++ b/docs-site/src/content/docs/zh-cn/reference/adapters.md @@ -30,6 +30,8 @@ interface ProviderAdapter { - 把内部消息转换成 OpenAI role;工具映射为 `{type:"function", function:{…}}` 和 `tool_choice`(`auto`/`none`/`required` 或具名函数)。 +- **工具结果中的图片**会在工具轮次结束后,作为后续 user vision 消息(`image_url` 部分)发送, + 因为 `role:"tool"` 的内容只能是文本;`[image]` 标记仍保留在工具消息中作为锚点。 - **重写 Codex 的 GPT-5 身份提示词**,改成与模型无关的介绍,避免路由模型自称 OpenAI。 - 精确层级不可用时,**把 `reasoning_effort` 限制到模型公布的子集**。除非 provider 显式配置 alias,`xhigh` 与 `max` 保持为不同标签。对于 `provider.noReasoningModels` 中的 id,则**完全 diff --git a/src/adapters/openai-chat.ts b/src/adapters/openai-chat.ts index 1abd9743a9..0aae451977 100644 --- a/src/adapters/openai-chat.ts +++ b/src/adapters/openai-chat.ts @@ -77,6 +77,24 @@ function developerSystemText(message: OcxMessage): string | undefined { return message.content.map(part => (part as OcxTextContent).text).join(""); } +/** + * Chat-completions image_url parts for images carried inside a tool result (issue #888). role:"tool" + * content is text-only on every chat provider, so these ride in a follow-up user message instead of + * being flattened to the "[image]" marker the model can't actually see. Data URLs and remote https + * URLs are both valid in image_url.url, unlike Gemini inline_data which needs base64. + */ +function toolResultImageChatParts(content: string | OcxContentPart[]): unknown[] { + if (typeof content === "string") return []; + const parts: unknown[] = []; + for (const p of content) { + // Skip parts without a usable URL (the tool-output parser accepts the empty file_id shape): + // a {"url":""} part would fail the whole request where the "[image]" marker degrades safely. + if (p.type !== "image" || !p.imageUrl) continue; + parts.push({ type: "image_url", image_url: { url: p.imageUrl, ...(p.detail ? { detail: p.detail } : {}) } }); + } + return parts; +} + function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderConfig): unknown[] { const out: unknown[] = []; const { context, options } = parsed; @@ -91,6 +109,7 @@ function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderCon interface PendingToolCall { id: string; name: string } let pendingToolCalls: PendingToolCall[] = []; let deferredBarrierMessages: unknown[] = []; + let pendingToolResultImageParts: unknown[] = []; let mintedIdSeq = 0; const seenWireCallIds = new Set(); @@ -109,6 +128,22 @@ function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderCon deferredBarrierMessages = []; }; + // Tool-result images collected during the open round land in ONE user vision message once the + // round closes — never inside it, where strict providers (Kimi/Moonshot) 400 on interleaved + // user messages. Released before deferred barriers so the images stay adjacent to the results + // they came from (mirrors google.ts sibling inline_data parts and the Kiro carrier images). + const flushToolResultImages = (): void => { + if (pendingToolResultImageParts.length === 0) return; + out.push({ + role: "user", + content: [ + { type: "text", text: "[ocx] image output from the preceding tool result(s):" }, + ...pendingToolResultImageParts, + ], + }); + pendingToolResultImageParts = []; + }; + // Close an unresolved tool round with explicit unavailable-result messages. The wording // must not claim interruption, success, failure, or user intent: execution status is // UNKNOWN, and for user-input tools this must not read as an answer. @@ -122,6 +157,7 @@ function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderCon }); } pendingToolCalls = []; + flushToolResultImages(); releaseDeferredBarriers(); }; @@ -232,8 +268,12 @@ function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderCon tool_call_id: toolCallId, content: contentPartsToText(msg.content), }); + pendingToolResultImageParts.push(...toolResultImageChatParts(msg.content)); pendingToolCalls.splice(matchIdx, 1); - if (pendingToolCalls.length === 0) releaseDeferredBarriers(); + if (pendingToolCalls.length === 0) { + flushToolResultImages(); + releaseDeferredBarriers(); + } } else { if (!toolCallId) toolCallId = `call_orphan_${out.length}`; // No matching call in the open round. Close any unresolved round first so the @@ -257,6 +297,8 @@ function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProviderCon tool_call_id: toolCallId, content: contentPartsToText(msg.content), }); + pendingToolResultImageParts.push(...toolResultImageChatParts(msg.content)); + flushToolResultImages(); } break; } diff --git a/tests/openai-chat-tool-result-images.test.ts b/tests/openai-chat-tool-result-images.test.ts new file mode 100644 index 0000000000..b2c389e1fc --- /dev/null +++ b/tests/openai-chat-tool-result-images.test.ts @@ -0,0 +1,150 @@ +import { expect, test } from "bun:test"; +import { createOpenAIChatAdapter } from "../src/adapters/openai-chat"; +import type { OcxContentPart, OcxMessage, OcxParsedRequest, OcxProviderConfig } from "../src/types"; + +// Issue #888: role:"tool" content is text-only on chat-completions providers, so images inside a +// tool result were flattened to an "[image]" marker and vision-capable routed models hallucinated +// what they never saw. Tool-result images now ride in a follow-up user vision message released when +// the tool round closes, without splitting the round (strict providers reject interleaved users). + +const provider: OcxProviderConfig = { + adapter: "openai-chat", + baseUrl: "https://example.test/v1", + apiKey: "sk-test", + authMode: "key", +}; + +const IMAGE_URL = "data:image/png;base64,aGVsbG8taW1hZ2UtYnl0ZXM="; + +interface ChatPart { + type: string; + text?: string; + image_url?: { url: string; detail?: string }; +} + +interface ChatMsg { + role: string; + content?: string | ChatPart[]; + tool_calls?: { id: string; function: { name: string; arguments: string } }[]; + tool_call_id?: string; +} + +function wire(messages: OcxMessage[]): ChatMsg[] { + const parsed: OcxParsedRequest = { + modelId: "test-model", + context: { messages }, + stream: false, + options: {}, + }; + const req = createOpenAIChatAdapter(provider).buildRequest(parsed) as { body: string }; + return (JSON.parse(req.body) as { messages: ChatMsg[] }).messages; +} + +function user(text: string): OcxMessage { + return { role: "user", content: text, timestamp: 0 }; +} + +function assistantWithCalls(calls: { id: string; name: string }[]): OcxMessage { + return { + role: "assistant", + content: calls.map(c => ({ type: "toolCall" as const, id: c.id, name: c.name, arguments: {} })), + timestamp: 0, + }; +} + +function toolResult(callId: string, name: string, content: string | OcxContentPart[]): OcxMessage { + return { role: "toolResult", toolCallId: callId, toolName: name, content, isError: false, timestamp: 0 }; +} + +/** The carrier is a user message whose parts start with an "[ocx]" text label followed by image_url parts. */ +function isImageCarrier(msg: ChatMsg): boolean { + if (msg.role !== "user" || !Array.isArray(msg.content)) return false; + const [head, ...rest] = msg.content; + return head?.type === "text" && typeof head.text === "string" && head.text.startsWith("[ocx]") + && rest.length > 0 && rest.every(p => p.type === "image_url"); +} + +/** Every role:"tool" message must sit in an unbroken block right after its assistant tool_calls message. */ +function assertRoundsUnbroken(messages: ChatMsg[]): void { + for (let i = 0; i < messages.length; i++) { + const m = messages[i]; + if (m.role !== "tool") continue; + let j = i - 1; + while (j >= 0 && messages[j].role === "tool") j--; + expect(j).toBeGreaterThanOrEqual(0); + expect(messages[j].role).toBe("assistant"); + expect((messages[j].tool_calls ?? []).map(tc => tc.id)).toContain(m.tool_call_id); + } +} + +test("tool-result images ride a follow-up user message; text, detail, and https URLs survive", () => { + const messages = wire([ + user("read the screenshot"), + assistantWithCalls([{ id: "call_1", name: "Read" }]), + toolResult("call_1", "Read", [ + { type: "text", text: "1 match found" }, + { type: "image", imageUrl: IMAGE_URL, detail: "high" }, + { type: "image", imageUrl: "https://example.test/shot.png" }, + { type: "image", imageUrl: "" }, // empty file_id shape: keeps its marker, never reaches the carrier + ]), + ]); + assertRoundsUnbroken(messages); + const tool = messages.find(m => m.role === "tool")!; + expect(tool.content).toBe("1 match found[image][image][image]"); + const carrier = messages.find(isImageCarrier)!; + expect(carrier).toBeDefined(); + expect(messages.indexOf(carrier)).toBe(messages.indexOf(tool) + 1); + const parts = (carrier.content as ChatPart[]).filter(p => p.type === "image_url"); + expect(parts.map(p => p.image_url)).toEqual([ + { url: IMAGE_URL, detail: "high" }, + { url: "https://example.test/shot.png" }, + ]); +}); + +test("images from a multi-call round flush once, only after the whole round closes", () => { + const messages = wire([ + assistantWithCalls([{ id: "call_1", name: "shot" }, { id: "call_2", name: "list" }]), + toolResult("call_1", "shot", [{ type: "image", imageUrl: IMAGE_URL }]), + toolResult("call_2", "list", "file1.txt"), + ]); + assertRoundsUnbroken(messages); + const toolIdx = messages.map((m, i) => (m.role === "tool" ? i : -1)).filter(i => i >= 0); + expect(toolIdx).toEqual([toolIdx[0], toolIdx[0] + 1]); // nothing interleaves the round + const carriers = messages.filter(isImageCarrier); + expect(carriers.length).toBe(1); + expect(messages.indexOf(carriers[0])).toBe(toolIdx[1] + 1); +}); + +test("orphan tool result with an image still emits the carrier after its synthesized pair", () => { + const messages = wire([ + user("hi"), + toolResult("call_orphan", "shot", [{ type: "image", imageUrl: IMAGE_URL }]), + ]); + assertRoundsUnbroken(messages); + const tool = messages.find(m => m.role === "tool")!; + expect(tool.content).toBe("[image]"); + const carrier = messages.find(isImageCarrier)!; + expect(messages.indexOf(carrier)).toBe(messages.indexOf(tool) + 1); +}); + +test("interrupted round: the synthetic closure still flushes collected images", () => { + const messages = wire([ + assistantWithCalls([{ id: "call_1", name: "shot" }, { id: "call_2", name: "list" }]), + toolResult("call_1", "shot", [{ type: "image", imageUrl: IMAGE_URL }]), + ]); + assertRoundsUnbroken(messages); + const carrier = messages.find(isImageCarrier)!; + expect(carrier).toBeDefined(); + expect(messages.indexOf(carrier)).toBe(messages.length - 1); +}); + +test("image-free tool results emit no carrier and an unchanged wire", () => { + const messages = wire([ + user("hi"), + assistantWithCalls([{ id: "call_1", name: "list" }]), + toolResult("call_1", "list", "file1.txt"), + user("thanks"), + ]); + expect(messages.some(isImageCarrier)).toBe(false); + expect(messages.map(m => m.role)).toEqual(["user", "assistant", "tool", "user"]); +});