evals: run agent suites exclusively through the external harnesses

[shriyatheunicorn]

Summary

Agent suites (webvoyager, onlineMind2Web, webtailbench, odysseysbench) run exclusively through the external harnesses (claude_code / codex); the v3 agent path is removed from the bench framework. Net −5,028 lines.

Why now

The agent tier's history on this lineage:

Date	Change	Event
2026-05-01	#2011 Evals v2	agent bench tasks created
2026-07-23	v4-spike v3 purge	deleted
2026-07-27	#2401 Restore v3 evals	restored against the published stagehand-v3 npm package

#2401 restored the v3 agent tier as a stopgap because v4 had no agent story. The nondeterministic-evals stack under this PR — the LLMExposure contract, the external harnesses, per-step evidence collection — is that story, so the stopgap retires. Suites are now planner-level constructs (virtual registry entries over the dataset builders in suites/), not task files.

What changes

• stagehand bench harness is act/extract/observe-only on the Stagehand SDK; both initV3 branches, the --api path, and the v3 context fields are gone
• An explicit suite target on the stagehand harness errors with harness guidance; broad targets omit suites; dry-runs surface planner errors in the payload instead of crashing
• Deleted as dead once the path is gone: the 47 v3-context agent tasks, runWithVerifier, TrajectoryRecorder (its external-harness successors — gradeExternalTrajectory, observationRecorder — live on the stack below), the agent-mode/CUA matrix machinery, and the --agent-mode/--agent-modes flags
• Plus three run fixes found during live benchmarking: extract-schema steering in the stagehand_code prompt, @browserbasehq/sdk/ws resolved from the evals package (the old resolver pointed at packages/core, which does not exist on this lineage), and a CLI stderr filter for the stale-frame CDP log flood

Scope boundaries

• Deterministic evals untouched: zero changes to defineTask.ts / types.ts / context.ts / framework barrel (the evals: port the act/extract/observe bench suite to the v4 SDK #2494 / evals: make the bench framework v4-only for act/extract/observe #2587 stack); a/e/o task-file v4 typing remains that stack's deliverable
• The legacy runner (--legacy) keeps its own v3 wiring and is scheduled for a separate removal — done: evals: merge-readiness fixes for the external-harness stack #2649 (stacked on this PR) removes the legacy runner outright, along with GAIA
• Zero changes under packages/server / packages/extension

Stacked on top

• evals: merge-readiness fixes for the external-harness stack #2649 — merge-readiness fixes: gate counts verifier-failed rows as ungraded and fails gated batches; GAIA removed entirely; legacy runner removed; codex tool-step budget; per-dataset prompt constraints restored (Mind2Web stay-on-site); agent-mode tag hygiene; stagehand_code screenshot Buffer fix
• evals: enable the MCP tool surfaces on the claude_code and codex harnesses #2650 — playwright_mcp / chrome_devtools_mcp enabled on both external harnesses

---

Summary by cubic

Routes all agent benchmark suites (webvoyager, onlineMind2Web, webtailbench, odysseysbench) through external harnesses (claude_code, codex) and removes the v3 agent path. The stagehand harness is now act/extract/observe-only; suites are discovered as virtual entries; and the stagehand_code prompt now requires single-word, all-required extract schema keys.

• Bug Fixes

  • Filtered Stagehand stale-frame CDP ERROR lines from CLI stderr; other errors still print.
  • Fixed Browserbase target prep by resolving @browserbasehq/sdk and ws from packages/evals dependencies.

• Migration

  • Run suites with --harness claude_code or --harness codex; the stagehand harness no longer runs agent suites and will error on explicit suite targets (dry-run payloads include planner errors).
  • Remove usage of --agent-mode, --agent-modes, and the agentModes config key. The legacy runner (--legacy) keeps its own v3 wiring.

^{Written for commit 600adf1. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 600adf1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 67 files

Confidence score: 2/5

• In packages/evals/framework/benchHarness.ts, provider-filtered deterministic runs can fail before browser startup because --provider openai resolves to gpt-4o-mini while initStagehand expects an openai/-prefixed model for API key lookup, which can block this workflow entirely—normalize provider-selected model names (or key resolution) so OpenAI models are consistently prefixed.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/evals/framework/benchHarness.ts">

<violation number="1" location="packages/evals/framework/benchHarness.ts:121">
P1: Provider-filtered deterministic runs now fail before browser startup: `--provider openai` produces `gpt-4o-mini`, but `initStagehand` cannot resolve its API key without an `openai/` prefix. Normalize provider-selected model IDs before this call (and preserve mappings for every supported provider).</violation>
</file>

Architecture diagram

sequenceDiagram
    participant CLI as CLI (run.ts)
    participant Disc as Discovery
    participant Plan as BenchPlanner
    participant Run as BenchRunner
    participant SH as Stagehand Harness
    participant EH as External Harness (claude_code/codex)
    participant Ver as VerifierAdapter
    participant V3E as V3Evaluator

    Note over CLI,V3E: Bench suite dispatch - agent suites run exclusively via external harnesses

    CLI->>Disc: discoverTasks()
    Disc->>Disc: NEW: register virtual agent suite entries (webvoyager, onlineMind2Web, webtailbench, odysseysbench)
    Disc-->>CLI: TaskRegistry (suites as virtual entries)

    CLI->>Plan: generateBenchTestcases(tasks, {harness, datasetFilter})

    alt harness = stagehand
        Plan->>Plan: CHANGED: filter out suite-kind tasks
        alt explicit datasetFilter targeting a suite
            Plan-->>CLI: throw "Agent benchmark suites require an external harness"
            CLI->>CLI: NEW: surface planner error in dry-run payload (instead of crash)
        else broad target
            Plan-->>CLI: stagehand matrix (a/e/o tasks only)
        end
    else harness = claude_code | codex
        Plan->>Plan: keep virtual suite entries
        Plan-->>CLI: matrix with suite tasks
    end

    CLI->>Run: executeBenchTask(task, options)
    Run->>Run: CHANGED: buildBenchMatrixRow without agentMode/isCUA

    alt suite task (external_agent_benchmarks)
        Run->>EH: start external harness
        EH->>EH: run agent (claude_code/codex binary + observe)
        EH-->>Run: trajectory + per-step evidence
        Run->>Ver: gradeExternalTrajectory()
        Ver->>V3E: NEW: createVerifierEvaluator (EVAL_VERIFIER_MODEL override)
        V3E-->>Ver: EvaluationResult
        Ver-->>Run: TaskResult
    else act/extract/observe task
        Run->>SH: start stagehand harness
        SH->>SH: CHANGED: reject agent v3 paths (agentMode, isCUA, useApi)
        alt unsupported (agentMode / isCUA / useApi / non-a-e-o category)
            SH-->>Run: error with "--harness claude_code or --harness codex" guidance
        else valid a/e/o
            SH->>SH: initStagehand (v4 SDK only)
            SH-->>Run: v4 ctx {stagehand, page}
            Run->>Run: run task definition fn(ctx)
        end
    end

    Run-->>CLI: TaskResult

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Provider-filtered deterministic runs now fail before browser startup: --provider openai produces gpt-4o-mini, but initStagehand cannot resolve its API key without an openai/ prefix. Normalize provider-selected model IDs before this call (and preserve mappings for every supported provider).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/evals/framework/benchHarness.ts, line 121:

<comment>Provider-filtered deterministic runs now fail before browser startup: `--provider openai` produces `gpt-4o-mini`, but `initStagehand` cannot resolve its API key without an `openai/` prefix. Normalize provider-selected model IDs before this call (and preserve mappings for every supported provider).</comment>

<file context>
@@ -126,141 +89,49 @@ function buildExternalHarnessTaskSpec(
+    const { initStagehand } = await import("../initStagehand.js");
+    const v4Result = await initStagehand({
+      logger,
+      modelName: input.modelName,
+      environment: config.environment,
+    });
</file context>