feat(evals): add agent evaluation suite

apps/sim/evals/README.md

@@ -0,0 +1,228 @@
+# Agent harness evaluations
+
+Measurement for the agent harness — the code that turns a model's tool calls
+into executed tools, feeds the results back, and keeps the turn alive when a
+tool fails. Unit and integration tests prove the harness handles the cases we
+already know about; evals measure whether it still behaves across a suite of
+scenarios when the harness changes.
+
+## What runs
+
+The first suite lives in [`agent-tool-use/`](./agent-tool-use) and drives the
+real OpenAI-compatible streaming tool loop
+(`apps/sim/providers/openai-compat/streaming-tool-loop.ts`) — the loop that
+serves OpenAI, DeepSeek, Groq, Cerebras, and the other OpenAI-compatible
+providers. The model is **scripted**: each scenario supplies the assistant turns
+(tool calls or a final answer) and the result of each tool call. That keeps the
+suite deterministic and runnable in CI with no provider key, while the thing
+being measured — tool dispatch, result feedback, error recovery — is real
+production code.
+
+The suites cover four behaviors:
+
+| Category | What it measures |
+| --- | --- |
+| `tool-selection` | The loop dispatches the tool the model asked for, including from a set of distractors. |
+| `planning` | Multi-turn, dependent and parallel tool calls execute in the right order and all results reach the next turn. |
+| `retrieval` | Values returned by a tool survive into the final answer instead of being dropped or invented. |
+| `recovery` | Tool errors, unknown tool names, and malformed argument JSON are fed back to the model rather than thrown out of the loop. |
+
+## Run it
+
+From `apps/sim`:
+
+```sh
+bun run test:evals
+```
+
+The command writes a JSON report and a Markdown summary to
+`test-results/evals/agent-tool-use.{json,md}` (gitignored) and fails the process
+if any scenario fails. To point the report somewhere else, run Vitest directly:
+
+```sh
+EVAL_REPORT_PATH=/tmp/agent-tool-use.json bunx vitest run evals/agent-tool-use
+```
+
+The suite is also picked up by the normal `bun run test` run, so a regression
+fails CI even without the dedicated command.
+
+## Run against a real model (live)
+
+The same scenarios can be replayed against a live model. This is opt-in and
+never runs in CI. DeepSeek is wired first; any OpenAI-compatible provider works
+through `createOpenAICompatLiveCompletion` in `live.ts`.
+
+```sh
+cd apps/sim
+DEEPSEEK_API_KEY=... bun run test:evals:live
+```
+
+Useful knobs:
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `EVAL_TRIALS` | `3` | Runs per scenario. Models are nondeterministic, so results are pass rates. |
+| `EVAL_MIN_PASS_RATE` | `0` | When > 0, fail a scenario below this pass rate (0–1). |
+| `EVAL_MODEL` | `deepseek-chat` | Model id sent to the provider. |
+| `EVAL_TIMEOUT_MS` | `180000` | Per-request timeout. |
+| `EVAL_REPORT_PATH` | `test-results/evals/agent-tool-use-live.json` | Report location. |
+| `EVAL_RECORD` | `0` | Set to `1` to also write the first trial's transcript to `fixtures/`. |
+
+Live runs relax exact assertions: `toolCallSequence` becomes an ordered
+subsequence, `successfulToolCalls` becomes a minimum, and scripted-only cases
+(malformed JSON, unknown tool) are skipped. A scenario-level `liveExpect`
+overrides the scripted expectation where a real model cannot reproduce it (for
+example, an exact retry count). The report is at
+`test-results/evals/agent-tool-use-live.{json,md}` with pass rates, average
+iterations, latency, and the failed check names.
+
+### Record and replay
+
+A live run is nondeterministic and needs a key; a fixture is neither. Record one
+trial, then replay it forever through the real loop with no network:
+
+```sh
+cd apps/sim
+EVAL_RECORD=1 DEEPSEEK_API_KEY=... bun run test:evals:live   # writes fixtures/*.json
+bun run test:evals                                           # replays them, no key
+```
+
+`fixtures/<scenario>.json` holds the raw streamed chunks per model call, so a
+diff shows a behavior change exactly as the model produced it. Fixtures are
+committed and reviewed like snapshots. `agent-tool-use.replay.test.ts` replays
+each one through `createOpenAICompatStreamingToolLoopStream` and scores it with
+the same checks; the suite skips until at least one fixture exists. Re-record a
+fixture when the scenario, prompt, or model intentionally changes.
+
+### Compare models
+
+Run the same scenarios across several models and get a scenario × model matrix:
+
+```sh
+cd apps/sim
+EVAL_MODELS=deepseek:deepseek-chat,deepseek:deepseek-reasoner \
+  DEEPSEEK_API_KEY=... bun run test:evals:compare
+```
+
+A spec is `provider:model`; a bare model id defaults to DeepSeek. Providers are
+DeepSeek, OpenAI, Groq, and OpenRouter, each reading its key from
+`<PROVIDER>_API_KEY`. The report is
+`test-results/evals/agent-tool-use-compare.{json,md}`: per-model pass rate,
+iterations, latency, and tokens, plus a per-scenario pass-rate matrix.
+`EVAL_MIN_PASS_RATE` fails a model below a floor.
+
+### LLM-as-judge
+
+Substring and regex checks measure phrasing, not correctness. `judge.ts` scores
+an answer against a rubric with a judge model and returns structured numbers:
+
+```sh
+cd apps/sim
+DEEPSEEK_API_KEY=... bun run test:evals:judge
+```
+
+`judgeAnswer` takes the judge transport, the user request, the answer, optional
+tool evidence, and a rubric of weighted criteria, and returns a verdict. Pass a
+`judge` option to `runScenario` to add a `judge` check alongside the
+deterministic ones. The judge transport is an ordinary OpenAI-compatible
+completion, so a recorded transcript can replay it deterministically in CI.
+
+## Add a case
+
+1. Open [`agent-tool-use/scenarios.ts`](./agent-tool-use/scenarios.ts) and add
+   an entry to `AGENT_TOOL_USE_SCENARIOS`.
+2. Declare the `tools` the model may call and the `script` it produces. A
+   `tools` turn lists the calls the model emits; an `answer` turn ends the run.
+   Attach each call's stub `result` (or leave it to default to a successful
+   empty output).
+3. Add the assertions you care about under `expect`: the ordered
+   `toolCallSequence`, `requiredTools`/`forbiddenTools`, `finalContent`,
+   `maxIterations`, and tool call counts. Every assertion becomes a named check
+   in the report.
+4. Run `bun run test:evals`.
+
+A scenario is data, not code — there is no harness change needed for a new case.
+
+### Simulating a failure
+
+- **Tool error:** give the call `result: { success: false, error: '...' }`.
+- **Unknown tool:** call a `name` that is not in `tools`; the loop returns a
+  tool-not-found error to the model.
+- **Malformed arguments:** set `argumentsJson` to an invalid or non-object JSON
+  string. The loop must not execute the call and must return the parse error to
+  the model.
+
+## Executor-level scenarios
+
+[`agent-tool-use/executor-harness.ts`](./agent-tool-use/executor-harness.ts)
+runs a case through a real `DAGExecutor`: a Start block → Agent block workflow,
+with only the provider boundary (`executeProviderRequest`) mocked. This covers
+what the loop harness cannot — agent-block input wiring, variable resolution
+from Start outputs, and the executor's run/error handling. Tool dispatch stays
+covered by the loop suite.
+
+Add a case to `EXECUTOR_SCENARIOS` in `executor-harness.ts`:
+
+- `workflowInput` is exposed on the Start block; reference an output with
+  `<start.field>` from the Agent prompt.
+- `agent` is the Agent block config (`model`, `systemPrompt`, `userPrompt`).
+- `providerResponse` is what the mocked provider returns (`content`,
+  `toolCalls`, `tokens`).
+- `expect` uses the loop's checks plus `resolvedInput` (a substring that must
+  reach the provider messages), `succeeds` (expected `ExecutionResult.success`),
+  and `providerCalls` (exact provider call count).
+- Set `agent.retry` to exercise the executor's per-block retry policy, or
+  `agent.fallbackModels` to exercise model fallback. Make the first
+  `providerResponse` a `reject` and the next one succeeds; assert
+  `providerCalls` and `lastRequestModel` to prove which path recovered.
+
+Both suites write one report, so executor rows appear alongside loop rows.
+
+## Context evals
+
+[`agent-context/`](./agent-context/) drives the Agent block through the executor
+with conversation memory on. The memory read is stubbed per conversation id, so
+the provider request shows exactly what the handler assembled: prior history,
+then the new user prompt, with the system prompt preserved, and the conversation
+id must match. Windowing inside the memory service (`sliding_window`, token
+budgets) is covered by its unit tests; this suite covers the assembly the model
+sees.
+
+```sh
+cd apps/sim
+bun run test:evals:context   # writes test-results/evals/agent-context.{json,md}
+```
+
+Scenarios live in [`agent-context/scenarios.ts`](./agent-context/scenarios.ts)
+and reuse the executor harness, so a case is the same shape as an executor case
+plus `agent.memory`.
+
+## Report shape
+
+`report.json` is machine-readable for dashboards and trend tracking; `report.md`
+is the same data as a table. Each result carries the scenario id, pass/fail,
+every named check with a failure detail, the final content, the executed tool
+invocations, and metrics: iterations, tool call counts (success/error), latency,
+model/tool time, first-response time, and token usage.
+
+## Orchestration evals
+
+[`agent-orchestration/`](./agent-orchestration/) runs a parent workflow that
+invokes a child through the Workflow block, driven by the real `DAGExecutor`.
+The child definition loader and the child `Executor` are mocked, so spawn →
+wait → aggregate → recover runs without a database; the parent executor, the
+workflow handler, the serializer, and the output mapping are real. Two cases:
+the child's output reaches the parent, and a child failure fails the parent.
+
+```sh
+cd apps/sim
+bun run test:evals:orchestration   # writes test-results/evals/agent-orchestration.{json,md}
+```
+
+## Scope and next steps
+
+Harnesses share one result shape and report: the tool loop, the `DAGExecutor`,
+and the orchestration parent/child path. The executor suite covers both recovery
+paths — block retry (`executor-retries-failed-block`) and model fallback
+(`executor-falls-back-to-secondary-model`). Further expansion (real child
+execution, model routing) is tracked as follow-up work.

apps/sim/evals/agent-context/agent-context.eval.test.ts

@@ -0,0 +1,67 @@
+import {
+  permissionCheckMock,
+  permissionCheckMockFns,
+} from '@sim/testing/mocks/permission-check.mock'
+import { providersMock } from '@sim/testing/mocks/providers.mock'
+import { providersConversationHistoryMock } from '@sim/testing/mocks/providers-conversation-history.mock'
+import { providersUtilsMock, providersUtilsMockFns } from '@sim/testing/mocks/providers-utils.mock'
+import { toolsMock } from '@sim/testing/mocks/tools.mock'
+import { workspaceFileSecretProvenanceMock } from '@sim/testing/mocks/workspace-file-secret-provenance.mock'
+import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
+import { AGENT_CONTEXT_SCENARIOS } from '@/evals/agent-context/scenarios'
+import { runExecutorScenario } from '@/evals/agent-tool-use/executor-harness'
+import { writeEvalReport } from '@/evals/agent-tool-use/report'
+import type { AgentToolUseResult } from '@/evals/agent-tool-use/types'
+
+vi.mock('@/providers/conversation-history', () => providersConversationHistoryMock)
+vi.mock('@/tools', () => toolsMock)
+vi.mock('@/providers/utils', () => providersUtilsMock)
+vi.mock('@/providers', () => providersMock)
+vi.mock('@/ee/access-control/utils/permission-check', () => permissionCheckMock)
+vi.mock(
+  '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance',
+  () => workspaceFileSecretProvenanceMock
+)
+vi.mock('@/lib/memory/agent-turn-session', () => ({
+  openAgentTurnSession: vi.fn(async () => undefined),
+}))
+vi.mock('@/lib/internal/mcp/discover-tools', () => ({
+  discoverMcpServerToolsAsExecutor: vi.fn(async () => []),
+}))
+vi.mock('@/lib/internal/custom-tools/read-available-by-id-or-title', () => ({
+  readAvailableCustomToolByIdOrTitleAsExecutor: vi.fn(async () => undefined),
+}))
+vi.mock('@/executor/utils/http', () => ({
+  buildAuthHeaders: vi.fn(async () => ({ 'Content-Type': 'application/json' })),
+  buildAPIUrl: vi.fn((path: string) => path),
+  extractAPIErrorMessage: vi.fn(async () => 'request failed'),
+}))
+vi.mock('@/lib/execution/cancellation', () => ({
+  subscribeToExecutionCancellation: vi.fn(async () => () => {}),
+  isExecutionCancelled: vi.fn(async () => false),
+}))
+
+const results: AgentToolUseResult[] = []
+
+beforeEach(() => {
+  permissionCheckMockFns.mockValidateModelProvider.mockResolvedValue(undefined)
+  providersUtilsMockFns.mockGetProviderFromModel.mockReturnValue('mock-provider')
+})
+
+afterAll(() => {
+  const reportPath = process.env.EVAL_CONTEXT_REPORT_PATH
+  if (reportPath) writeEvalReport(results, reportPath)
+})
+
+describe('agent context eval suite', () => {
+  it.each(AGENT_CONTEXT_SCENARIOS)('$id: $name', async (scenario) => {
+    const result = await runExecutorScenario(scenario)
+    results.push(result)
+
+    const failed = result.checks.filter((entry) => !entry.passed)
+    expect(
+      failed,
+      failed.map((entry) => `${entry.name}: ${entry.detail}`).join('; ') || undefined
+    ).toEqual([])
+  })
+})

apps/sim/evals/agent-context/scenarios.ts

@@ -0,0 +1,71 @@
+import type { ExecutorScenario } from '@/evals/agent-tool-use/executor-harness'
+
+/**
+ * Agent context evals.
+ *
+ * These drive the real Agent block inside the `DAGExecutor` with conversation
+ * memory on. The memory read is stubbed per conversation id, so the provider
+ * request shows exactly what the handler assembled: prior history, then the new
+ * user prompt, with the system prompt preserved. A handler that passed the
+ * wrong conversation id, dropped history, or reordered the prompt fails.
+ *
+ * Windowing inside the memory service (`sliding_window`, token budgets) is
+ * covered by its own unit tests; this suite covers the assembly the model sees.
+ */
+export const AGENT_CONTEXT_SCENARIOS: ExecutorScenario[] = [
+  {
+    id: 'context-remembers-prior-turns',
+    name: 'includes prior conversation memory before the new user prompt',
+    category: 'retrieval',
+    description:
+      'Memory holds two prior turns. The provider request must contain both, in order, followed by the new user prompt, with the system prompt present.',
+    workflowInput: {},
+    agent: {
+      model: 'gpt-4o',
+      systemPrompt: 'You are a helpful assistant.',
+      userPrompt: 'What is my name?',
+      memory: {
+        conversationId: 'conv-ada',
+        history: [
+          { role: 'user', content: 'My name is Ada.' },
+          { role: 'assistant', content: 'Nice to meet you, Ada.' },
+        ],
+      },
+    },
+    providerResponse: {
+      content: 'Your name is Ada.',
+      tokens: { input: 30, output: 5, total: 35 },
+    },
+    expect: {
+      succeeds: true,
+      finalContent: 'Ada',
+    },
+  },
+  {
+    id: 'context-isolates-conversations',
+    name: 'reads the conversation named by the block, not another',
+    category: 'retrieval',
+    description:
+      'The memory stub only returns history for the block conversation id; any other id yields a placeholder. A handler that passed the wrong id would surface the placeholder and fail.',
+    workflowInput: {},
+    agent: {
+      model: 'gpt-4o',
+      userPrompt: 'What did we decide?',
+      memory: {
+        conversationId: 'conv-b',
+        history: [
+          { role: 'user', content: 'We decided to ship on Friday.' },
+          { role: 'assistant', content: 'Shipping Friday.' },
+        ],
+      },
+    },
+    providerResponse: {
+      content: 'You decided to ship on Friday.',
+      tokens: { input: 25, output: 6, total: 31 },
+    },
+    expect: {
+      succeeds: true,
+      finalContent: 'Friday',
+    },
+  },
+]