evals: merge-readiness fixes for the external-harness stack

[miguelg719]

Summary

The remaining merge blockers for the #2591→#2611 stack, implemented per the review conclusions (verifier fallback accepted on empirical evidence; GAIA removal decided; Odysseys rubric item dropped as irrelevant).

• Gate insurance for the verifier fallback: verifier-failed rows (verifierError set, _success self-reported) now count as ungraded runs on their arm. The per-arm verifiability line reports them, and when EVAL_MAX_UNVERIFIABLE_CRITERIA is configured, any ungraded run fails the batch. This changes no grading behavior — it converts "the fallback hasn't fired in hundreds of runs" into "if it ever fires, the batch fails loudly instead of publishing self-graded rows."
• GAIA fully removed (dataset, suite builder, index.eval.ts fan-out, category overrides, legacy-only sets, help text, README). b:gaia → "Unknown benchmark", instead of a half-present suite pointing at a broken --legacy path.
• Planner: a stagehand-harness selection that reduces to nothing runnable (suite name, agent category, dataset shorthand) errors with harness guidance instead of planning zero cases; broad targets still omit suites and plan the rest.
• Codex step budget: tool steps (command executions + MCP tool calls) capped via stream abort — EVAL_CODEX_MAX_STEPS, falling back to the pre-migration AGENT_EVAL_MAX_STEPS, default 50. Runs get a citable, reproducible action budget.
• Per-dataset prompt constraints restored from the retired suite modules: OnlineMind2Web's stay-on-site rule (validity: tasks were passable by googling the answer) and the navigate guidance for WebTailBench/Odysseys.
• Tag hygiene: dom/hybrid/cua stripped from external-harness testcases so Braintrust tag filters don't pool them with historical Stagehand agent rows.
• stagehand_code screenshot fix (ported from the facade worktree): v4 SDK returns Uint8Array; Buffer.isBuffer consumers silently dropped screenshots. Now normalized to Buffer.

Deliberately not here

• Pre-navigation to startUrl: pending an explicit protocol decision (self-navigation as the documented norm vs. pre-goto for cross-era comparability).
• The facade's >= relaxation of the codex observation guard: kept strict === — the relaxation reintroduces positional misattribution; the underlying trailing-capture undercount needs an attempted-run count in the recorder (follow-up with the facade work).
• MCP tool surfaces and the Playwright facade: follow-up stack.

Verification

typecheck clean · 416/416 unit tests (3 GAIA-era tests updated to the new behavior, 1 new gate test) · oxfmt clean

---

Summary by cubic

Hardens external-harness evals for merge: enforces strict verifier gating, removes GAIA and the legacy runner/flag, adds a Codex step budget, restores dataset prompt guidance, fixes screenshot handling, and bounds setup-failure cleanup timeouts. Also cleans up docs/payloads, fixes a Codex abort-listener leak in long batches, tightens a dry-run test assertion, and removes the --provider flag (provider now derived from the model id).

• New Features

  • Verifier gate: verifier-failed rows now count as ungraded; any ungraded run fails a gated batch. Per-arm report shows ungraded counts. Respects EVAL_MAX_UNVERIFIABLE_CRITERIA.
  • Codex step budget: cap tool steps via EVAL_CODEX_MAX_STEPS (fallback AGENT_EVAL_MAX_STEPS, default 50) with stream abort and clear stop reason.
  • Restored dataset guidance in prompts: OnlineMind2Web stay-on-site rule; navigation hints for WebTailBench and Odysseys.
  • Planner: selecting stagehand harness that leaves nothing runnable now errors with guidance instead of planning zero cases.

• Refactors

  • GAIA fully removed; b:gaia now reports “Unknown benchmark.” b:osworld also reports “Unknown benchmark.”
  • Removed legacy runner and --legacy flag; deleted legacy files and commands. Dry-run payload drops skippedTasks; preview/help text, ARCHITECTURE.mmd, and README updated.
  • Removed --provider and provider-based model filtering; provider metadata is inferred from provider/model. Dropped EVAL_PROVIDER env, config/help surfaces, and related tests.
  • Tag hygiene: strip dom, hybrid, cua from external-harness testcases to keep Braintrust filters separated from legacy Stagehand runs.
  • stagehand_code screenshot bytes normalized to Buffer to fix dropped screenshots with the v4 SDK.
  • Codex runner: detach the abort forwarder after each run to avoid accumulating listeners in long batches.
  • External-harness adapters: apply the normal teardown timeout (EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS) to setup-failure cleanup to prevent wedging rows.
  • Docs: dry-run diagram lists the actual payload fields (target, normalizedTarget, tasks, envOverrides, runOptions, matrix, error?).
  • Tests: broad dry-run now asserts the unique planned task set to avoid model-multiplicity noise.

^{Written for commit 66bc69d. Summary will update on new commits.}

---

Update (second commit): the legacy runner is removed outright. --legacy could no longer run anything on this branch (a/e/o need the v4 harness context it never builds; the agent modules and suite targets it resolves are gone), so the escape hatch was routing users to a guaranteed failure. Deleted: cli-legacy.ts, index.eval.ts, args.ts, tui/commands/legacy.ts, the evals:old script, the --legacy flag, LEGACY_ONLY_BENCHMARKS (b:osworld now reports "Unknown benchmark"), and the legacy-skip plumbing in the run command, dry-run payload, and preview. Net −1,885 lines. Gates re-run green (typecheck, 416/416, oxfmt).

[changeset-bot]

⚠️ No Changeset found

Latest commit: 66bc69d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 19 files

Architecture diagram

sequenceDiagram
    participant TUI as TUI / CLI (run)
    participant Plan as Bench Planner
    participant Suite as Suite Builders
    participant Prom as Prompt Builder
    participant Codex as Codex Runner
    participant SDK as Codex SDK / Thread
    participant Shot as stagehand_code
    participant Verif as Verifier Adapter
    participant Gate as Verifier Gate (run.ts)

    Note over TUI,Gate: External-harness eval run (webvoyager / onlineMind2Web / webtailbench / odysseys)
    Note over TUI: GAIA removed - b:gaia shorthand now reports "Unknown benchmark"

    TUI->>Plan: generateBenchTestcases(harness=stagehand)
    alt Selection reduces to nothing runnable (suite-only)
        Plan-->>TUI: CHANGED: EvalsError - re-run with --harness claude_code or codex
    else Broad target
        Plan->>Plan: omit agent suites from stagehand plan, keep rest
    end

    TUI->>Suite: build<Dataset>Testcases(models)
    Suite->>Suite: CHANGED: strip dom/hybrid/cua tags from testcases
    Suite-->>TUI: rows tagged harness/<harness>

    TUI->>Codex: runCodexAgent(plan)
    Codex->>Codex: NEW: read EVAL_CODEX_MAX_STEPS fallback AGENT_EVAL_MAX_STEPS, default 50
    Codex->>SDK: runStreamed(prompt, signal: budgetController)
    loop Agent tool loop
        SDK-->>Codex: event item.completed
        alt command_execution or mcp_tool_call
            Codex->>Codex: toolStepCount += 1
            opt toolStepCount >= maxToolSteps
                Codex->>SDK: abort stream (stop reason: tool step budget exhausted)
            end
        end
    end

    Note over Prom,Codex: NEW: dataset guidance added to prompt (codex + claude_code runners)
    Codex->>Prom: datasetPromptGuidance(dataset)
    Prom-->>Codex: stay-on-site (onlineMind2Web) / navigation hint (webtailbench, odysseys)

    Note over Shot,Verif: CHANGED: v4 SDK screenshot bytes normalized to Buffer
    Codex->>Shot: page.screenshot() / captureStagehandEvidence()
    Shot-->>Codex: Buffer.from(bytes)

    Codex->>Verif: gradeExternalTrajectory(trajectory)
    alt Verifier fails (verifierError set)
        Verif-->>Codex: self-reported _success only (ungraded)
    else Verifier grades
        Verif-->>Codex: criterionCount + evidenceInsufficient
    end
    Codex-->>TUI: TaskResult

    TUI->>Gate: summarizeArmVerifiability(results)
    Gate-->>TUI: per-arm gradedRuns + ungradedRuns
    Note over TUI,Gate: Verifiability line reports ungraded (self-reported) count
    alt EVAL_MAX_UNVERIFIABLE_CRITERIA set and any arm has ungraded runs
        TUI->>TUI: NEW: fail batch (exit 1) - self-reported rows never published
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 11 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 14 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>