evals: enable the MCP tool surfaces on the claude_code and codex harnesses

[miguelg719]

Summary

Stacked on #2649. --tool playwright_mcp and --tool chrome_devtools_mcp now execute on --harness claude_code and --harness codex — the first wiring of the MCP surfaces into the agent harnesses (previously core-tier only).

Design: the generic agentMount { via: "mcp", mcpServers } contract variant, produced by the tools and delivered by the adapters — no surface-specific adapter code. The agent spawns its own MCP server instance against the runner-provided CDP browser; the harness keeps its own session on the same browser for evidence capture.

• Tools: export the mount (via new buildPlaywrightMcpLaunchSpec / buildChromeDevtoolsMcpLaunchSpec helpers — named for the facade branch to rebase onto) + captureEvidence. Mount requires a providedEndpoint, so a tool-launched isolated browser can never split agent and observer.
• claude_code: mcpServers passthrough to the SDK, mcp__<server> tool allowlisting, per-step observation triggered from the runner's tool_result stream keyed by tool_use id.
• codex: mcp_servers via SDK config overrides; observation on mcp_tool_call completion events; MCP calls count against the evals: merge-readiness fixes for the external-harness stack #2649 tool-step budget.
• Trajectory adapters: observedToolName matcher generalizes observation attachment to MCP steps; the strict === count guard is kept (the facade's >= relaxation is deliberately not adopted — it reintroduces positional misattribution; the trailing-capture undercount wants an attempted-run count instead, follow-up).

Deliberately out (punted with the facade): stagehand_playwright_code, callback batching, the facade's codex-only prepareCodexPlaywrightMcpAdapter (superseded by the generic path).

Verification

• typecheck / oxfmt clean, 420/420 unit tests (4 new: surface resolution both harnesses, MCP observation attachment both adapters)
• Live smoke (claude_code × playwright_mcp, WebVoyager, LOCAL): agent drove mcp__playwright__* tools for 50 steps, per-step url/aria probe evidence attached to MCP steps only, verifier graded a 5-criterion rubric (0 unverifiable)
• codex × chrome_devtools_mcp: config injection + server spawn validated on Browserbase (run failed on an unrelated model-pairing error; mcp_servers reached Codex and servers started)

Known rough edge: probe screenshots through the harness-side playwright_mcp session are flaky under the 10s observation timeout (url/aria reliable; ~1/40 screenshots in the smoke). Follow-up: bump timeout for MCP surfaces or capture via raw CDP.

---

Summary by cubic

Enable MCP browser tool surfaces on the claude_code and codex harnesses. --tool playwright_mcp and --tool chrome_devtools_mcp now run end-to-end with per-step evidence attached to MCP steps; plus reliability fixes for capture and tab syncing.

• New Features

  • Tools export an MCP agent mount (via: "mcp") with buildPlaywrightMcpLaunchSpec / buildChromeDevtoolsMcpLaunchSpec, plus captureEvidence.
  • Claude Code: passes mcpServers to the SDK, allows mcp__<server> tools, and attaches observations from the tool_result stream (tracks tool_use id → name).
  • Codex: injects mcp_servers via SDK config overrides; records observations on mcp_tool_call completion; MCP calls count toward the tool-step budget.
  • Trajectory adapters: add an observedToolName matcher so observations attach to MCP steps; strict count guard remains to avoid misattribution.
  • Startup: MCP surfaces use runner-provided CDP endpoints (local or Browserbase) so agent and harness share the same browser.

• Bug Fixes

  • Evidence capture re-points the harness session to the browser’s visible tab via CDP before probing (best-effort fallback if unavailable).
  • Observation recorder settles in-flight captures before drain; drainStepObservations is async on both harnesses to avoid missing last-step evidence.
  • Codex captureEvidence is bounded by EVAL_CAPTURE_EVIDENCE_TIMEOUT_MS (default 15s) to prevent stalls.

^{Written for commit 544b54a. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 544b54a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 10 files

Architecture diagram

sequenceDiagram
    participant Runner as Eval Runner
    participant Tool as MCP Surface Tool (playwright_mcp / chrome_devtools_mcp)
    participant Browser as CDP Browser (shared)
    participant MCP as Agent MCP Server (harness-spawned)
    participant CC as Claude Code Harness
    participant CX as Codex Harness
    participant Obs as Observation Recorder
    participant Traj as Trajectory Adapter

    Note over Runner,Traj: MCP agent-mount surfaces on claude_code / codex harnesses

    Runner->>Tool: start(providedEndpoint)
    Tool->>Browser: NEW: harness-side session attaches to CDP browser (evidence capture)
    Tool-->>Runner: NEW: agentMount { via: "mcp", mcpServers, promptInstructions } + captureEvidence

    alt providedEndpoint present (local or Browserbase runner-provided CDP)
        Note over Tool,MCP: Mount requires providedEndpoint so agent's MCP server and harness session share one browser
        Runner->>CC: NEW: prepare adapter - mcpServers passthrough, mcp__<server> allowlist, onToolResult
        Runner->>CX: NEW: prepare adapter - codexConfig.mcp_servers, recordObservation, observedToolMatcher
    else no providedEndpoint (tool_launch_local)
        Note over Runner,Tool: No agentMount - agent has no browser surface
    end

    par Claude Code path
        CC->>MCP: NEW: spawn server via SDK mcpServers
        MCP->>Browser: attach (same CDP endpoint as harness session)
        CC->>MCP: drive mcp__playwright__* / mcp__chrome-devtools__* tools
        MCP-->>CC: tool_result (carries only tool_use id)
        CC->>CC: NEW: map tool_use id -> tool name from assistant messages
        CC->>Obs: NEW: onToolResult(name) -> record()
    and Codex path
        CX->>MCP: NEW: spawn server via config mcp_servers override
        MCP->>Browser: attach (same CDP endpoint as harness session)
        CX->>MCP: invoke <server>.<tool> mcp_tool_call
        MCP-->>CX: item.completed (mcp_tool_call)
        CX->>Obs: NEW: recordObservation() (counts against tool-step budget)
    end

    loop per agent step (both harnesses)
        Obs->>Tool: captureEvidence()
        Tool->>Browser: probe url / aria tree / screenshot (best-effort)
        Tool-->>Obs: ProbeEvidence
    end
    Obs-->>Runner: drain stepObservations (runIndex order)

    Runner->>Traj: fromHarnessResult(messages/events, stepObservations, observedToolName)
    Traj->>Traj: CHANGED: observedToolName matcher selects MCP steps consuming observation indexes
    alt observed call count == recorded run count (strict === guard kept)
        Traj->>Traj: attach probeEvidence per ordinal (no misattribution)
    else mismatch
        Note over Traj: Refuse attachment - evidence_insufficient path
    end
    Traj-->>Runner: NormalizedTrajectory (probeEvidence on MCP steps)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>