evals: codex harness across all code tool surfaces

[shriyatheunicorn]

Summary

Codex joins claude_code as an external harness across all three code tool surfaces (stagehand_code, playwright_code, cdp_code) — recreating #2397 on the LLMExposure architecture.

codex-sdk has no in-process MCP mounting (unlike claude-agent-sdk), and an external MCP process could not share this process's live surface handles. So the codex mount for a code_handles exposure is a loopback HTTP bridge: this process executes snippets against the in-memory handles; the codex workspace gets a tiny browser_run.mjs client that posts a snippet file to the bridge and prints the result. Scope semantics are identical to the claude_code run tool.

• codexCodeBridge.ts (new): the bridge server + generated workspace client
• codexToolAdapter.ts: code-surface mounting (exposure selection, bridge lifecycle, tmpdir cwd, prompt rewrite from run-tool to browser_run.mjs invocation)
• codexRunner.ts / harnesses/codexAdapter.ts: harness-observed terminal artifact anchors the verifier's final observation, mirroring the claude_code runner
• Tests: bridge round-trip + widened surface resolution

Part of the nondeterministic-evals stack; stacks on the exposure-adapter PR.

Follow-up on top of the stack: #2650 adds the MCP tool surfaces (playwright_mcp / chrome_devtools_mcp) to both external harnesses; MCP tool calls there count against the codex tool-step budget introduced in #2649.

---

Summary by cubic

Enable Codex on stagehand_code, playwright_code, and cdp_code via a loopback HTTP bridge that runs snippets against in-memory handles, with terminal artifact capture in the runner for grounded grading. browse_cli remains supported with sane startup defaults, and code surfaces now have clear browser_run.mjs guidance.

• New Features

  • Bridge server/client in codexCodeBridge.ts: browser_run.mjs posts snippet code to /run; executes with handles, startUrl, task, and console; timeout via EVAL_CODEX_RUN_TOOL_TIMEOUT_MS; non-zero exit on failure.
  • Tool adapter and runner: codexToolAdapter.ts mounts browse_cli, stagehand_code, playwright_code, cdp_code; defaults to tool-owned browsers for browse_cli/stagehand_code and runner-provided CDP for playwright_code/cdp_code; writes browser_run.mjs and rewrites AGENT_RUN_TOOL_NAME; captures evidence via captureEvidence and passes it to codexAdapter.ts; robust cleanup.

• Bug Fixes

  • Final observation is only anchored when a screenshot is present; URL-only artifacts no longer qualify.
  • Redacts credential fragments (e.g., signingKey, sk-*) from snippet error messages before crossing the bridge or hitting logs.

^{Written for commit a710ed6. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: a710ed6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

Review completed against the latest diff

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files

Architecture diagram

sequenceDiagram
    participant CodexAgent as Codex Agent
    participant CodexHarness as Eval Harness
    participant ToolAdapter as Codex Tool Adapter
    participant CodeBridge as Codex Code Bridge
    participant Surface as Tool Surface
    participant Trajectory as Trajectory Adapter

    Note over CodexAgent,Trajectory: NEW: Codex Code Surface Flow (stagehand_code/playwright_code/cdp_code)

    CodexHarness->>ToolAdapter: prepareCodexToolAdapter(surface, plan)
    ToolAdapter->>ToolAdapter: Determine code surface variant
    
    alt browse_cli
        ToolAdapter->>ToolAdapter: Use existing browse CLI adapter
    else code surface (stagehand_code, playwright_code, cdp_code)
        ToolAdapter->>Surface: prepareLLMExposure(plan, environment)
        Surface-->>ToolAdapter: exposure with code_handles
        
        ToolAdapter->>CodeBridge: startCodeBridge(exposure, plan)
        CodeBridge->>CodeBridge: Create HTTP server on random port
        CodeBridge-->>ToolAdapter: bridge instance with port
        
        ToolAdapter->>ToolAdapter: Create temp directory
        ToolAdapter->>ToolAdapter: Write browser_run.mjs client script
        ToolAdapter-->>CodexHarness: adapter with cwd, env, promptInstructions, cleanup
    end

    Note over CodexAgent,CodexHarness: Agent Tool Invocation

    CodexHarness->>ToolAdapter: Get prompt instructions
    ToolAdapter-->>CodexHarness: Rewritten instructions (run-tool → browser_run.mjs)
    
    CodexAgent->>CodexAgent: Write snippet.js file
    CodexAgent->>CodexAgent: Execute: node browser_run.mjs snippet.js
    
    CodexAgent->>CodeBridge: POST /run with snippet code
    CodeBridge->>CodeBridge: Execute snippet with handles, startUrl, task, console
    
    alt Execution succeeds
        CodeBridge-->>CodexAgent: HTTP 200 { ok: true, result: "..." }
    else Execution fails
        CodeBridge-->>CodexAgent: HTTP 200 { ok: false, error: "..." }
    else Timeout (EVAL_CODEX_RUN_TOOL_TIMEOUT_MS)
        CodeBridge-->>CodexAgent: HTTP 200 { ok: false, error: "timed out" }
    end

    Note over CodexHarness,Trajectory: Final Artifact Capture

    CodexHarness->>ToolAdapter: captureFinalState()
    ToolAdapter->>Surface: captureFinalState()
    Surface-->>ToolAdapter: TerminalArtifact { screenshot, url }
    ToolAdapter-->>CodexHarness: terminal state

    CodexHarness->>Trajectory: gradeExternalTrajectory(with terminalArtifact)
    Trajectory->>Trajectory: Build finalObservation from terminal state
    Trajectory-->>CodexHarness: Graded trajectory

    Note over CodexHarness,ToolAdapter: Cleanup

    CodexHarness->>ToolAdapter: cleanup()
    ToolAdapter->>CodeBridge: close bridge server
    ToolAdapter->>Surface: cleanup exposure
    ToolAdapter->>ToolAdapter: Remove temp directory

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>