evals: wire the verifier into third-party harnesses for evidence collection

[shriyatheunicorn]

Summary

STG-2752: external-harness runs now carry per-step observations and a final observation grounded in harness-captured evidence, and unverifiable-criteria counts are reported per arm and can gate the batch.

• ToolStartResult.captureEvidence (optional hook): each code surface captures current page state (url + screenshot + aria tree) through its own handles — one shared capture per surface, resolved through session.activePage() at capture time
• observationRecorder.ts (new): run-indexed probe buffer; a failed capture leaves a gap instead of shifting later observations onto the wrong step; EVAL_HARNESS_OBSERVATIONS=none excludes a run from evidence collection
• Probes record after each run-tool execution in claude_code and each bridge run in codex; the trajectory adapters attach them to code steps only, and refuse attachment when counts mismatch (misattribution is worse than a gap)
• tui run prints per-arm verifiability and enforces the EVAL_MAX_UNVERIFIABLE_CRITERIA gate

Verified live: in webvoyager runs the verifier now rejects agent claims that the collected step evidence does not support (e.g. asserted ingredients absent from the extracted list).

Stacked follow-ups: #2649 extends the gate to count verifier-failed (ungraded, self-reported) rows per arm and fail gated batches containing any; #2650 extends per-step observation to the MCP tool surfaces via the runner event streams.

---

Summary by cubic

Wires verifier-backed evidence capture into claude_code and codex so runs include per-step URL/screenshot probes and a harness-observed final artifact. Adds per-arm verifiability reporting and an optional gate for STG-2752.

• New Features

  • Added ObservationRecorder for per-step probes; opt out with EVAL_HARNESS_OBSERVATIONS=none, timeout via EVAL_OBSERVATION_TIMEOUT_MS.
  • Probes record after each run_tool in claude_code and each bridge run in codex; adapters attach only to code steps (LLM_RUN_TOOL_NAME, browser_run.mjs) as probeEvidence. Final observation prefers the harness-captured artifact, else the last agent screenshot.
  • tui run prints per-arm verifiability and enforces the EVAL_MAX_UNVERIFIABLE_CRITERIA gate.

• Bug Fixes

  • Bridge probe callbacks are non-blocking; failed captures leave indexed gaps instead of shifting evidence.
  • codex only attaches observations when recorded bridge runs match filtered steps; claude_code captures a terminal artifact before cleanup; final observation requires a screenshot.
  • Strict integer parsing for EVAL_MAX_UNVERIFIABLE_CRITERIA; the gate fails loudly when set but no arms were graded.

^{Written for commit 88d32fd. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 88d32fd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 16 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 16 files

Architecture diagram

sequenceDiagram
    participant CLI as TUI Run
    participant Runner as External Harness Runner
    participant Adapter as Tool Adapter
    participant Recorder as ObservationRecorder
    participant Exposure as LLMExposure
    participant Browser as Browser/CDP
    participant Harness as Harness Adapter
    participant Gate as Verifier Gate

    Note over CLI,Gate: Evidence Collection Flow for External Harnesses

    CLI->>Runner: runCommand(options)
    Runner->>Adapter: prepareToolAdapter(exposure)
    alt Observations enabled (EVAL_HARNESS_OBSERVATIONS != none)
        Adapter->>Recorder: new ObservationRecorder(exposure.captureObservation)
        Note over Recorder: Buffers probes by run index
    end

    Note over Runner,Harness: Per-Step Observation Collection

    loop Each agent tool execution
        Runner->>Exposure: runTool(code)
        Exposure->>Browser: Execute code
        Exposure-->>Runner: Return result
        alt Recorder exists
            Runner->>Recorder: record()
            Recorder->>Exposure: captureObservation()
            Exposure->>Browser: Capture screenshot + URL
            alt Capture success
                Browser-->>Exposure: TerminalArtifact
                Exposure-->>Recorder: Artifact data
                Recorder->>Recorder: Buffer at current runIndex
            else Capture failure/timeout
                Exposure-->>Recorder: Error/empty
                Note over Recorder: Leave gap (no shift)
            end
        end
    end

    Note over Runner,Harness: Final State & Trajectory Assembly

    Runner->>Exposure: captureFinalState()
    Exposure->>Browser: Capture final screenshot + URL
    Browser-->>Exposure: TerminalArtifact
    Exposure-->>Runner: terminalArtifact
    Runner->>Adapter: drainStepObservations()
    Adapter->>Recorder: drain()
    Recorder-->>Adapter: StepObservation[]
    Adapter-->>Runner: stepObservations

    Runner->>Harness: fromHarnessResult({terminalArtifact, stepObservations})
    alt claude_code adapter
        Harness->>Harness: Map observations to LLM_RUN_TOOL_NAME steps
        Note over Harness: Attach probeEvidence per step
        alt Has screenshot artifact
            Harness->>Harness: finalObservation = terminalArtifact
        else Fallback
            Harness->>Harness: finalObservation = last agent screenshot
        end
    else codex adapter
        Harness->>Harness: Filter bridge runs (browser_run.mjs)
        alt Bridge runs >= recorded observations
            Harness->>Harness: Map by ordinal, attach probeEvidence
        else Mismatch detected
            Note over Harness: Skip attachment (misattribution risk)
        end
    end

    Note over Harness,Gate: Verifiability Aggregation & Gating

    CLI->>CLI: Collect all results
    CLI->>Gate: summarizeArmVerifiability(results, harness)
    Gate->>Gate: Group by (harness × surface × model)
    Gate-->>CLI: ArmVerifiability[]
    
    CLI->>CLI: Print per-arm summary
    alt EVAL_MAX_UNVERIFIABLE_CRITERIA set
        CLI->>Gate: resolveUnverifiableCriteriaLimit()
        CLI->>Gate: armsOverLimit(arms, limit)
        Gate-->>CLI: Arms exceeding limit
        alt Arms over limit
            CLI->>CLI: Log violations, set exitCode=1
        else No graded arms with gate set
            CLI->>CLI: Log warning, set exitCode=1
        end
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>