fix(translation): preserve multimodal tool results

[ting-hong-shieh]

What changed

• Preserve text, image, and document blocks nested in Anthropic tool_result.content when decoding into the neutral IR.
• Keep textual output in the required OpenAI Chat tool messages and lower non-text output into a following multimodal user message.
• Emit all parallel tool-result messages before the lowered multimodal content so their ordering remains valid.
• Report the role lowering through the existing lossy-conversion policy; strict policy rejects it.

Why

The Anthropic decoder previously flattened every non-text block inside tool_result.content into JSON text. By the time the OpenAI Chat encoder ran, the image or document structure had already been lost, so the downstream model received base64 or metadata as ordinary text.

OpenAI Chat tool messages accept text content but not image or file parts. User messages accept those multimodal parts, so the translation now splits the content at that protocol boundary instead of silently stringifying it.

Closes #380.

Validation

• cargo fmt --all --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace

Regression coverage includes base64 images, base64 documents, parallel tool-result ordering, diagnostics, and strict rejection policy.

Summary by CodeRabbit

• New Features

  • Improved handling of document content, including file IDs, base64 data, and optional filenames.
  • Preserved text, images, documents, and other content types in tool results instead of flattening them into text.
  • Improved conversion of multimodal tool results to OpenAI Chat messages while preserving content order.

• Bug Fixes

  • Added diagnostics for conversions that move non-text content or may lose information.
  • Added an option to reject lossy conversions with a clear error.

[coderabbitai]

Walkthrough

Anthropic document and tool-result blocks now retain normalized multimodal content. OpenAI Chat translation keeps text in tool messages and moves non-text content to a subsequent user message. Tests cover ordering, diagnostics, and strict rejection.

Changes

Multimodal tool-result translation

Layer / File(s)	Summary
Anthropic content normalization crates/switchyard-translation/src/codecs/anthropic/buffered.rs	Document sources decode into FileSource values. Tool-result arrays preserve text, image, document, and unknown blocks.
OpenAI Chat tool-result lowering crates/switchyard-translation/src/codecs/openai_chat/buffered.rs	Text and refusal content remain in tool messages. Non-text content moves to a later user message, with a lossy-conversion diagnostic.
Translation behavior validation crates/switchyard-translation/tests/request_translation.rs	Tests verify data URLs, parallel-result ordering, document and image handling, diagnostics, and strict-policy errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit with a multimodal ear,
Images and documents now appear clear.
Tool text stays tucked where tool words belong,
User messages carry the pixels along.
With diagnostics hopping beside,
Strict policies guard the ride.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: preserving multimodal tool results during translation.
Linked Issues check	✅ Passed	The changes preserve text, image, and document tool-result content, lower non-text blocks validly, preserve ordering, and report lossy conversion as required by issue #380.
Out of Scope Changes check	✅ Passed	The implementation and regression tests are directly related to preserving multimodal tool results and validating the required OpenAI Chat lowering behavior.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-translation/src/codecs/anthropic/buffered.rs`:
- Around line 625-648: Update decode_anthropic_file_source so its final
unsupported-source fallback stores Value::Object(block.clone()) rather than only
source.clone(), preserving the document wrapper for encode_one_anthropic_block.
Add a regression test covering an unsupported document source and verifying the
re-encoded block retains the complete original document structure.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1f81d3d1-0110-497d-b9c4-ca5e5eb91786

📥 Commits

Reviewing files that changed from the base of the PR and between 48b3b71 and c5b6488.

📒 Files selected for processing (3)

• crates/switchyard-translation/src/codecs/anthropic/buffered.rs
• crates/switchyard-translation/src/codecs/openai_chat/buffered.rs
• crates/switchyard-translation/tests/request_translation.rs

[ayushag-nv]

@ting-hong-shieh Thanks for putting this PR. Can you please attach a snapshot of the output or trace before this fix and after the fix. Will be easy to understand and review with that. Thanks

[ting-hong-shieh]

Thanks, @ayushag-nv. I ran the same buffered Anthropic → OpenAI Chat translation at the PR base (48b3b71d) and the current head (bd4aea4d). No provider call was involved.

Input tool_result.content:

[
  {"type": "text", "text": "here it is:"},
  {
    "type": "image",
    "source": {
      "type": "base64",
      "media_type": "image/png",
      "data": "iVBORw0KGgo="
    }
  }
]

Before (48b3b71d):

{
  "messages": [
    {
      "role": "tool",
      "tool_call_id": "toolu_1",
      "content": "here it is: {\"source\":{\"data\":\"iVBORw0KGgo=\",\"media_type\":\"image/png\",\"type\":\"base64\"},\"type\":\"image\"}"
    }
  ],
  "diagnostics": []
}

The image block is serialized into ordinary tool-message text.

After (bd4aea4d):

{
  "messages": [
    {
      "role": "tool",
      "tool_call_id": "toolu_1",
      "content": "here it is:"
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgo="
          }
        }
      ]
    }
  ],
  "diagnostics": [
    {
      "code": "lossy_conversion",
      "severity": "Warning",
      "message": "OpenAI Chat tool messages only support text; non-text tool-result content was moved to a user message",
      "source": "anthropic_messages",
      "target": "openai_chat"
    }
  ]
}

Text now remains in the tool message, while the image is preserved as multimodal content in the following user message. The role lowering is also reported through the existing lossy-conversion diagnostic.

[ayushag-nv]

This normalization loses source.media_type and moves the document’s top-level title into source.filename. With preservation disabled, a valid base64 Anthropic document therefore re-encodes without the required media_type and with title in the wrong place, which can make the upstream request invalid. Could we retain the Anthropic fields here (or preserve the raw block) and add a round-trip test for a known base64 document?

[ting-hong-shieh]

Addressed in d7fcd6d. Anthropic image and document blocks now keep their complete provider wrappers in the IR, so preservation-disabled round trips retain source.media_type, top-level title, and the remaining document fields. OpenAI Chat conversion maps only portable base64 data plus title; Anthropic-managed file_id values use the existing lossy/reject path instead of being forwarded as OpenAI IDs.

Added round-trip and cross-provider regression coverage. cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace pass locally.