Skip to content

evals.md

Latest commit

 

History

History
487 lines (412 loc) · 22.7 KB

evals.md

File metadata and controls

487 lines (412 loc) · 22.7 KB
Raw

Evals

Evals are regression tests for agent behaviour: which tools the agent called, in what order, with which arguments, how many steps it took, and what it said. They are written with defineEval(), live in *.eval.ts files, run under vitest, and are best run in CI with lousho eval, which prints a summary and writes JUnit and JSON reports.

vitest is required. lousho eval does not run eval files itself: it runs your *.eval.ts files through vitest, resolved from your project's own node_modules (Lousho does not bundle it). Install it as a dev dependency first; without it, lousho eval exits 1 with vitest was not found from <dir> and the install command.

npm install --save-dev vitest
npx lousho eval

Writing a trajectory eval

Give defineEval() an agent (from createAgent()) and a test function. Send messages with t.send(), then assert on the run. Using mockModel as the agent's provider makes the eval deterministic: no network, no API key, no flakiness. This is the default way to write an eval; use a real model only for the judge evals below.

// refund.eval.ts
import { z } from 'zod';
import { createAgent, defineEval, defineTool, includes } from '@lousho/build-ai-agent';
import { mockModel } from '@lousho/build-ai-agent/testing';

const lookupOrder = defineTool({
  name: 'lookup_order',
  description: 'Look up an order by id',
  input: z.object({ orderId: z.string() }),
  execute: ({ orderId }) => ({ orderId, status: 'delivered' }),
});

defineEval({
  name: 'refund flow',
  tags: ['smoke'],
  // A factory builds a fresh agent (and a fresh script) for every case.
  agent: () =>
    createAgent({
      instructions: 'You handle refund requests.',
      tools: [lookupOrder],
      provider: mockModel([
        { toolCalls: [{ name: 'lookup_order', args: { orderId: '42' } }] },
        { text: 'Order 42 can be refunded within 30 days.' },
      ]),
    }),
  async test(t) {
    await t.send('Refund order 42');
    t.completed();
    t.calledTool('lookup_order', { args: { orderId: '42' } });
    t.notCalledTool('issue_refund');
    t.check('mentions policy', t.reply, includes('30 days'));
    t.maxSteps(4);
  },
});

defineEval() also still accepts the original form ({ name, agent, input, provider, score, threshold }): it keeps working unchanged, and its result shows up in lousho eval as one score assertion.

Assertions

Every assertion is recorded; the case fails at the end and lists every failed gate, so one run shows all that is wrong.

Call Kind Passes when
t.completed() gate the latest run ended with a normal stop (not an error, abort, pending approval, or maxSteps cut-off)
t.calledTool(name, { args?, times?, subagent? }) gate the tool was called; args is a partial deep match (the call's arguments must contain these keys with equal values); times is an exact count of matching calls; subagent counts only that sub-agent's calls
t.notCalledTool(name, { subagent? }) gate the tool was never called, by the lead or any sub-agent
t.toolOrder([a, b], { subagent? }) gate the tools were called in this order (other calls may sit in between)
t.maxSteps(n) gate the agent took at most n model steps
t.maxTokens(n) gate the agent used at most n tokens
t.maxCostUsd(n, { unknownCost? }) gate the run's reported cost is at most n USD. When the SDK reports no cost (the model has no known price; see registerModel) the gate fails, since the budget cannot be checked; with unknownCost: 'skip' it is skipped instead (counts as passed, marked skipped, with a console.warn)
t.check(name, value, check) gate check passes for value
t.soft(name, value, check) soft never fails the run (unless --strict); the score is recorded and reported
await t.judge(rubric) n/a returns a 0 to 1 score from an LLM judge (see Judge evals)

Checks, for t.check() and t.soft(): includes(text), matches(regex), equals(value) (yes/no, score 1 or 0), and atLeast(n), atMost(n) (the number is the score).

Failure messages say what was actually seen:

calledTool('lookup_order') failed: tools called were [search_docs, issue_refund]
calledTool('lookup_order', {"args":{"orderId":"41"}}) failed: 'lookup_order' was called 1 time(s) but no call matched the expected args; closest call differs in orderId: expected "41", got "42"
completed() failed: finishReason was 'max-steps' (the run hit maxSteps while still calling tools)

Other context members: t.reply (latest reply text), t.result (latest ExecutionResult), t.toolCalls (every call so far, with parsed arguments). You can send() more than once; steps, tokens and tool calls add up.

Sub-agents

When the agent delegates with subagents, the calls a sub-agent makes are part of t.toolCalls. Each follows the task call that started it and carries a subagentPath (['researcher'], or ['researcher', 'fetcher'] when it delegated again); the lead's own calls have none. The assertions search the whole tree by default, so t.notCalledTool('delete_repo') fails if a sub-agent called it. Pass { subagent: 'researcher' } to look at one sub-agent (and the sub-agents under it) only:

import { createAgent, defineEval } from '@lousho/build-ai-agent';

const researcher = createAgent({ model: 'openai/gpt-4o-mini', instructions: 'You research.', description: 'Finds sources' });
const agent = createAgent({ model: 'openai/gpt-4o-mini', subagents: { researcher } });

defineEval({
  name: 'research delegates the search',
  agent,
  async test(t) {
    await t.send('Research electric bikes');
    t.calledTool('web_search', { subagent: 'researcher' });
    t.notCalledTool('delete_repo');
  },
});

The calls come from the run's tool.start events, which t.send() collects through send(message, { onEvent }). A remote target (--url) streams no sub-agent events, so it only shows the lead's calls. The classic form ({ agent, input, provider, score }) forwards every AgentExecutor.execute() option, subagents and skills included, and its EvalResult.toolCalls includes the sub-agent calls too.

Repeating a case (pass@k)

A model is not deterministic, so one run says little. repeat: n runs every case n times; the case passes when all n runs pass, or when at least passAt: k of them do:

import { createAgent, defineEval } from '@lousho/build-ai-agent';

const agent = createAgent({ model: 'openai/gpt-4o-mini', instructions: 'You handle refunds.' });

defineEval({
  name: 'refund flow',
  agent,
  repeat: 5,
  passAt: 4, // pass@4 of 5
  async test(t) {
    await t.send('Refund order 42');
    t.calledTool('issue_refund');
  },
});

Each run is its own row in the report (refund flow [refund flow #1], ...) with its own cassette. timeoutMs covers all runs of a case, so raise it with repeat.

Datasets

Pass cases and test runs once per case, each reported separately. A case is any object; its label (or name, or its input) names it.

import { z } from 'zod';
import { createAgent, defineEval, defineTool } from '@lousho/build-ai-agent';
import { mockModel } from '@lousho/build-ai-agent/testing';

const lookupOrder = defineTool({
  name: 'lookup_order',
  description: 'Look up an order by id',
  input: z.object({ orderId: z.string() }),
  execute: ({ orderId }) => ({ orderId }),
});

defineEval({
  name: 'order lookups',
  agent: () =>
    createAgent({
      tools: [lookupOrder],
      provider: mockModel([
        { toolCalls: [{ name: 'lookup_order', args: { orderId: '7' } }] },
        { text: 'Found it.' },
      ]),
    }),
  cases: [
    { input: 'Where is order 7?', orderId: '7' },
    { input: 'Status of #7', orderId: '7', label: 'hash syntax' },
  ],
  async test(t, c) {
    await t.send(c.input);
    t.calledTool('lookup_order', { args: { orderId: c.orderId } });
  },
});

Pass a factory (agent: () => createAgent(...)) so each case gets its own agent and its own mockModel script. With a single shared agent, a scripted model runs out of turns on the second case.

Judge evals

t.judge(rubric) grades the latest reply with an LLM and returns a score from 0 to 1. It needs a judge provider, and lousho never calls a real LLM unless you configured one: without judge, t.judge() throws an error saying how to fix it.

// tone.judge.eval.ts: run with `lousho eval --judge`
import { createAgent, defineEval, atLeast } from '@lousho/build-ai-agent';
import { mockModel } from '@lousho/build-ai-agent/testing';

defineEval({
  name: 'tone',
  agent: createAgent({ provider: mockModel(['Happy to help with that refund!']) }),
  // A real provider in practice; mockModel keeps this example offline.
  judge: { provider: mockModel(['0.9']), model: 'judge-model' },
  async test(t) {
    await t.send('I want my money back');
    t.soft('polite', await t.judge('Is the reply polite?'), atLeast(0.7));
  },
});

judge.model is optional: without it the judge runs on its provider's own model. The judge sees the rubric, the user's request and the agent's reply, runs at temperature 0 unless you set temperature, and is asked to end with a SCORE: <0..1> line. Its reply is read by the last score-labelled number (SCORE: 0.9, **Score:** 8/10, {"score": 0.8}), so a number in its reasoning ("1 of 5 criteria") is ignored. Without a label the whole reply must be one number (0.9, **0.9**). A number is scaled into 0 to 1 when it is written as 8/10, 8 out of 10, 85% or a bare whole number from 2 to 100 (8 reads as 8/10, 85 as 85/100). A reply with no labelled or bare number scores 0. parseJudgeScore(text) exposes the same parsing.

Files named *.judge.eval.ts are never picked up by a normal run (and npm test); only lousho eval --judge (or npm run test:evals:judge in this repository) runs them.

lousho eval

lousho eval [globs...] [--tag t] [--junit path] [--json path] [--strict] [--judge] [--record | --replay | --drift [--drift-usage]] [--url <base> [--token <bearer>]] [--config path] [--timeout ms]
Option Meaning
globs... eval files to run (default: every **/*.eval.ts, or **/*.judge.eval.ts with --judge)
--tag t only run evals whose tags include t (repeat or comma-separate for several)
--junit path write a JUnit XML report
--json path write the summary and every structured result as JSON
--strict soft failures fail the run
--judge run *.judge.eval.ts files instead of the normal ones
--url base run every case against the deployed agent at base instead of in-process; --token (or LOUSHO_EVAL_TOKEN) is the bearer token. See Run evals against a deployment
--config path use your own vitest config instead of the generated one
--timeout ms time limit per case in milliseconds (0 = none); see Time limits
--record run against the real provider and write one cassette per case (below)
--replay run every case from its cassette, with no network; a missing cassette fails the case
--drift re-record into a temp directory and report how each case's trajectory changed (--drift-usage also compares tokens)

It spawns the vitest installed in your project (vitest is your dependency; if it is missing the command prints npm install --save-dev vitest and exits 1), then prints one row per case with its scores and duration, totals, and the gate failures and soft failures listed separately. The "with soft failures" total (softFailed in the JSON summary) counts every case with a failed soft assertion, including cases that also failed a gate.

Exit code: 1 on any gate failure (or soft failure with --strict, or when vitest itself fails, for example a file that does not load, or is not installed, or a --junit/--json report cannot be written, or when no case ran at all: no eval file matched, or a --tag matched no case), 2 for a usage error (bad arguments), otherwise 0. A case that vitest fails after the case itself finished (it ran past its time limit) is reported as that case failing.

How results are collected: each case appends one JSON line to a file named by the LOUSHO_EVAL_RESULTS environment variable, which lousho eval sets and reads back. This is more robust than a custom vitest reporter: it works across vitest versions and worker pools, and needs no module loaded from your project.

Time limits

An agent run on a real model takes far longer than vitest's 5 s default test timeout. The config lousho eval generates gives every case 10 minutes, except under --replay, which keeps vitest's default because nothing waits on a model. --timeout ms sets the limit for every case (0 turns it off). With --config, your config's testTimeout applies unless you pass --timeout. One eval can set its own limit, which wins over both:

import { createAgent, defineEval } from '@lousho/build-ai-agent';

defineEval({
  name: 'research report',
  agent: createAgent({ model: 'openai/gpt-4o-mini' }),
  timeoutMs: 15 * 60_000,
  async test(t) {
    await t.send('Summarize the latest release notes.');
    t.completed();
  },
});

A case that runs out of time is reported as an error naming the limit, in the summary and in the JUnit and JSON reports.

JUnit

The report is standard JUnit: one testsuite per eval, one testcase per case. A failed gate is a <failure message="..."> carrying the diagnostic message; a case that threw or timed out is an <error>; soft failures are <system-out> notes (or failures under --strict), also on a case that failed a gate. When vitest fails something no eval case reported - an eval file that does not load, a plain test() that fails - the report gets an <error> testcase for it under a vitest suite, one per file (or one for the whole run when --config is used and vitest names no file), so a broken run never writes an empty, green-looking report.

In CI

# .github/workflows/evals.yml
name: evals
on: [pull_request]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npx lousho eval --junit reports/evals.xml --json reports/evals.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-reports
          path: reports/
      - uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: reports/evals.xml

Run --tag smoke on every pull request and the full set nightly; run --judge only where you accept real LLM calls and their cost.

Environment variables

lousho eval passes its flags to the vitest process that runs the eval files through environment variables. You do not set them yourself; they are listed so you recognise them in a CI log or a custom runner:

Variable Set from Meaning
LOUSHO_EVAL_CASSETTES --record, --replay, CI Cassette mode for the run: record, replay or auto (replay a case whose cassette exists, run the rest live). Under replay and auto a model: spec resolves without its API key.
LOUSHO_EVAL_DRIFT_DIR --drift With record: where the drift run writes its cassettes instead of next to the eval file.
LOUSHO_EVAL_CONFIG --config The config path, so re-record hints can repeat it.
LOUSHO_EVAL_TAGS --tag Comma-separated tags; only cases with one of them run.
LOUSHO_EVAL_RESULTS always The file each case appends its result line to, which lousho eval reads back (see lousho eval).
LOUSHO_EVAL_URL --url Base URL of the deployed agent to run the cases against.
LOUSHO_EVAL_TOKEN --token Bearer token for --url.
LOUSHO_ALLOW_LLM_JUDGE --judge Set to 1 inside the judge runner. Without it, llmJudge() / llmCritique() called outside that runner throw LOUSHO_EVALS_INVALID, so a misplaced judge eval never makes a paid model call in a normal test run.

Record, replay and drift

An eval against a real model is slow, costs money and needs a key; the same eval on mockModel only tests the script you wrote. lousho eval sits in between: it records each case once against the real provider through recordReplay and replays the recording in CI. Your eval files do not change.

npx lousho eval --record        # real provider: writes the cassettes, commit them
npx lousho eval --replay        # every case from its cassette, no network, no key
npx lousho eval --drift         # re-record and diff each case's trajectory
  • --record runs every case with its agent's real provider and writes one cassette per case next to the eval file: __cassettes__/<eval-name>/<case>-<hash>.json. The name is the slugged case label plus a short hash of the full label, for example refund-flow/polite-1f3a9c2e.json, so two cases whose labels slug or truncate alike still get their own file; an eval without cases writes default.json. Give each case a unique label so the names stay stable: two cases that would still share a cassette (the same label twice) fail at record time instead of overwriting each other. A case whose agent uses more than one provider (a sub-agent on another model) gets one named after the sub-agent, <case>-<hash>.<agent>.json (older .2.json files are still replayed). Replay matches each call by its request, not its position, so parallel sub-agent tasks replay whatever order the live model answered in; a mismatch stops the run with LOUSHO_CASSETTE_INVALID even when it happens inside a sub-agent. The files use the normal cassette format, with API keys redacted; review and commit them. Re-recording a case whose exchanges did not change leaves its file untouched (recordedAt is kept), so only real changes show in a diff. Cassettes recorded by an older SDK under the unhashed <case>.json name are still replayed; the next --record writes the new name, after which the old file can be deleted.
  • --replay never calls the model. A case with no cassette fails with no cassette for "<eval> [<case>]" at ... and the --record command to run (with your --config, if you gave one); when the agent's requests changed, the replayed model call fails with a CassetteMismatchError naming the first difference and the same lousho eval --record command. Plain lousho eval with CI set replays every case that has a cassette and runs the rest live; without CI it runs live as before.
  • --drift re-records every case into a temp directory (the committed cassettes are not touched) and compares each with its committed cassette: the ordered tool names, each call's arguments (JSON-normalized, so key order does not count), the step count and the finish reason. Token usage changes on every real run, so it is compared only with --drift-usage. The summary gets a Drift: table (eval, case, field, committed, current); each drifted case gets a soft drift assertion, so it shows as PASS (soft fail) and as a <system-out> note in the JUnit report. With --strict, drift is a gate failure: the case fails, JUnit has a <failure>, and the run exits 1.
Drift:
EVAL         CASE    FIELD  COMMITTED                     CURRENT
refund flow  polite  args   lookup_order {"orderId":"42"}  lookup_order {"orderId":"43"}

Run --drift nightly or before a model upgrade, and --replay (or plain lousho eval in CI) on every pull request. Judges are not recorded: grade with a real judge only in *.judge.eval.ts files.

How it works: the run loop sends every model call (generate or stream, from a plain run, a streamed run, a sub-agent or an approval resume) through one provider-interception seam, setProviderInterceptor() (exported from @lousho/build-ai-agent/testing; src/providers/interception.ts in the source). Nothing is installed by default, so the seam is free. While a case runs under one of these modes, lousho eval answers it with the provider wrapped by recordReplay() for that case's cassette. You can use the seam yourself to put any wrapper at the model boundary: the interceptor gets the run's provider and returns the one to call (it must return the same wrapper for the same provider, and leave a provider it already wrapped alone). A cassette failure it raises - a replay mismatch or a missing cassette - reaches send() typed (CassetteMismatchError, LOUSHO_CASSETTE_INVALID), not as a compacted provider error.

Run evals against a deployment

The same eval file that gates CI in-process can smoke-test a deployed agent (the node server or Cloudflare Worker). Point lousho eval at its base URL and every case runs against the deployment instead of the in-process agent:

npx lousho eval --url https://agent.example.com --token "$DEPLOY_TOKEN"   # or LOUSHO_EVAL_TOKEN

Each case gets its own remote session (POST /chat { sessionId, input }, one session per case, shared by its t.send() calls) and the SSE stream is read to run.done. The events become the same result an in-process run produces: tool calls, final reply, finish reason, steps and usage. So t.calledTool(), t.completed(), scorers, t.judge(), the summary and --junit work unchanged. The agent in the file is not used; give the deployment's own model and tools the behaviour you want to check.

  • A check that needs data the stream does not carry fails and says so (for example maxTokens when run.done has no usage); maxCostUsd fails when there is no cost, unless you pass { unknownCost: 'skip' }.
  • --url cannot be combined with --record, --replay or --drift (LOUSHO_CONFIG_CONFLICTING_OPTIONS): cassettes record a provider in-process. Score/threshold evals (score: form) also need an in-process provider and fail with the same code; use trajectory evals.
  • An unreachable deployment, a non-2xx answer or a truncated stream fails that case with LOUSHO_REMOTE_REQUEST_FAILED, and 401 with LOUSHO_REMOTE_UNAUTHORIZED; the run goes on. The token is never printed or written to a report.

In code, pass a target (this overrides agent; --url overrides both):

import { defineEval, remoteTarget } from '@lousho/build-ai-agent';

defineEval({
  name: 'deployed refund flow',
  target: remoteTarget({ url: 'https://agent.example.com', auth: process.env.LOUSHO_EVAL_TOKEN }),
  async test(t) {
    await t.send('Refund order 42');
    t.completed();
    t.calledTool('lookup_order');
  },
});

remoteTarget({ url, auth?, fetch? }) takes an injectable fetch, which is how the SDK's own tests run the real routes in-process without a socket.