diff --git a/.changeset/ai-harness-goal.md b/.changeset/ai-harness-goal.md new file mode 100644 index 0000000000..52b3d593a8 --- /dev/null +++ b/.changeset/ai-harness-goal.md @@ -0,0 +1,5 @@ +--- +'@tanstack/ai-harness': minor +--- + +Add the `goal({ judge, maxRounds })` plugin to `@tanstack/ai-harness/plugins`. `/goal ` sets a goal and starts a turn. After each turn, the judge model checks the goal, and the plugin starts the next turn until the goal is met, the round limit is reached, a turn waits for approval or fails, or the user sends a message. `/goal`, `/goal stop`, and `/goal resume` show and control it. The `GoalMet` event fires when the goal is met, and `selectGoal(session.snapshot())` reads the goal for a UI. diff --git a/docs/config.json b/docs/config.json index e8b8d260a4..e22d1ee12e 100644 --- a/docs/config.json +++ b/docs/config.json @@ -1042,6 +1042,11 @@ "to": "harness/code-mode", "addedAt": "2026-10-13" }, + { + "label": "Work until a goal is met", + "to": "harness/goal", + "addedAt": "2026-10-13" + }, { "label": "Run agents from a harness", "to": "harness/subagents", diff --git a/docs/harness/goal.md b/docs/harness/goal.md new file mode 100644 index 0000000000..045d37c0c6 --- /dev/null +++ b/docs/harness/goal.md @@ -0,0 +1,113 @@ +--- +title: Work until a goal is met +id: harness-goal +order: 12 +description: "Give the harness a goal, for example all tests pass. After each turn, a judge model checks the goal, and the harness keeps working until the goal is met." +keywords: + - tanstack ai + - harness + - goal + - judge + - agent loop +--- + +You ask your agent to make all tests pass. It fixes one test, then it stops and waits for you. You type "keep going" again and again. + +The `goal()` plugin keeps the harness working. You give the goal one time. After each turn, a judge model reads the goal and the end of the conversation. If the goal is not met, the plugin starts the next turn. + +## 1. Add the plugin + +```ts group=harness-goal +import { createHarnessHost, defineHarness } from '@tanstack/ai-harness' +import { goal } from '@tanstack/ai-harness/plugins' +import { openaiText } from '@tanstack/ai-openai' +import { memoryPersistence } from '@tanstack/ai-persistence' + +export const coder = defineHarness({ + name: 'acme/coder', + adapter: openaiText('gpt-6.1-sol'), + plugins: () => [goal({ judge: openaiText('gpt-6-luna') })], +}) + +const host = createHarnessHost({ persistence: memoryPersistence() }) +const session = await host.open(coder, { threadId: 'fix-tests' }) +``` + +`judge` is the model that decides if the goal is met. It can be a smaller model than the main model, or the same model. + +## 2. Give it a goal + +```ts group=harness-goal +await session.command('goal', 'all tests pass') +``` + +The first turn starts. Each new turn tells the model the reason of the last check: + +```text +Keep working on the goal: all tests pass. Last check: The date parser test still fails. +``` + +Run `session.command('goal')` to see the result: + +```text +Goal: all tests pass +Status: met, round 3 of 20. +Last check: All 12 tests pass. +``` + +## How it stops + +The plugin starts no new turn in these cases: + +- The judge says that the goal is met. The status is `met`. +- The plugin started `maxRounds` turns (20 by default). The status is `stopped`, and the last reason stays. +- A turn waits for an approval, or a turn fails. The status is `paused`. +- You send your own message. The status is `paused`, and your message runs as a normal turn. + +To change the limit, set `maxRounds`, for example `goal({ judge, maxRounds: 50 })`. + +## Commands + +| Command | What it does | +| --- | --- | +| `/goal ` | Sets the goal and starts the first turn. | +| `/goal` | Shows the goal, the status, the round, and the last reason. | +| `/goal stop` | Ends the goal. The running turn finishes, and no new turn starts. | +| `/goal resume` | Continues a paused or stopped goal with a new round count. It also starts an active goal again when no turn runs or waits. The goal is plugin state, so this also works after a restart. | + +## Show the goal in your UI + +The goal is plugin state. `selectGoal` reads it from `session.snapshot()`: + +```ts group=harness-goal +import { selectGoal } from '@tanstack/ai-harness/plugins' + +const current = selectGoal(session.snapshot()) +if (current) console.log(`${current.text}: ${current.status}, round ${current.round}`) +``` + +It returns `null` when the session has no goal. + +## Do something when the goal is met + +The plugin sends the `GoalMet` event with the goal and the last reason. Listen for it in your own plugin: + +```ts group=harness-goal +import { definePlugin } from '@tanstack/ai-harness' +import { GoalMet } from '@tanstack/ai-harness/plugins' + +export const notify = definePlugin({ + name: 'acme/notify', + setup: (ctx) => { + ctx.on(GoalMet, (met) => console.log(`Done: ${met.goal}. ${met.reason}`)) + }, +}) +``` + +Clients get the same event as a `harness.plugin.event` with the name `tanstack/goal:met`. + +## What you have now + +- A harness that keeps working until the judge says that the goal is met. +- A round limit, so the loop cannot run without end. +- `/goal`, `/goal stop`, and `/goal resume` to see and control the goal. diff --git a/packages/ai-harness/src/first-party/goal.ts b/packages/ai-harness/src/first-party/goal.ts new file mode 100644 index 0000000000..096cbd1f7a --- /dev/null +++ b/packages/ai-harness/src/first-party/goal.ts @@ -0,0 +1,346 @@ +import { EventType, chat } from '@tanstack/ai' +import { defineCommand } from '../commands' +import { createPluginEvent } from '../extensions' +import { definePlugin } from '../plugins' +import { textOf, transcriptText } from './session-tools' +import type { AnyTextAdapter, KeyedAdapter } from '@tanstack/ai' +import type { ChatTurnResult, Operation } from '../types' + +/** The goal of a session. It is plugin state, so it survives restarts. */ +export interface Goal { + text: string + status: 'active' | 'paused' | 'met' | 'stopped' + /** The turns the plugin started for this goal. */ + round: number + /** The reason of the last check. */ + reason: string + /** Why the goal paused or stopped. Empty while it is active or met. */ + note: string + /** The message of the turn that the plugin queued, until that turn starts. */ + queued: string +} + +/** Sent when the judge says that the goal is met. */ +export const GoalMet = createPluginEvent<{ goal: string; reason: string }>( + 'tanstack/goal:met', +) + +const JUDGE_PROMPT = + 'You check the work of an agent. Read the goal and the last part of the conversation. Set met to true only when the conversation shows that the goal is done. Give a short reason.' +const USER_MESSAGE = + 'You sent a message, so the goal paused. Run /goal resume to continue.' +const TURN_FAILED = 'The last turn failed. Run /goal resume to try again.' +const WAITS_FOR_APPROVAL = + 'The last turn waits for approval. Answer it, then run /goal resume.' + +function isVerdict(value: unknown): value is { met: boolean; reason: string } { + return ( + typeof value === 'object' && + value !== null && + 'met' in value && + typeof value.met === 'boolean' && + 'reason' in value && + typeof value.reason === 'string' + ) +} + +function isGoal(value: unknown): value is Goal { + return ( + typeof value === 'object' && + value !== null && + 'text' in value && + typeof value.text === 'string' && + 'status' in value && + typeof value.status === 'string' && + 'round' in value && + typeof value.round === 'number' && + 'reason' in value && + typeof value.reason === 'string' && + 'note' in value && + typeof value.note === 'string' && + 'queued' in value && + typeof value.queued === 'string' + ) +} + +/** + * The goal in a state with plugin state, for example `session.snapshot()`, + * or `null` when there is none. + */ +export function selectGoal(state: { plugins: Record }) { + const saved = state.plugins['tanstack/goal'] + return isGoal(saved) ? saved : null +} + +function keepWorking(text: string, reason: string) { + return `Keep working on the goal: ${text}. Last check: ${reason}` +} + +function describe(saved: Goal | null, maxRounds: number) { + if (!saved) return 'No goal. Start one with /goal .' + return [ + `Goal: ${saved.text}`, + `Status: ${saved.status}, round ${saved.round} of ${maxRounds}.`, + saved.note, + saved.reason === '' ? '' : `Last check: ${saved.reason}`, + ] + .filter((line) => line !== '') + .join('\n') +} + +/** + * Keep the harness working until a goal is met. `/goal ` starts the + * goal. After each turn, `judge` reads the goal and the end of the transcript + * and decides if the goal is met. If it is not, the plugin starts the next + * turn, up to `maxRounds` turns (default 20). + * + * The loop stops when the goal is met, at the round limit, when a turn waits + * for approval or fails, and when the user sends a message. `/goal` shows the + * goal, `/goal stop` ends it, and `/goal resume` continues it. A + * `keyedAdapter(...)` judge is built with the user's key for each check. + * + * @example + * ```ts + * plugins: () => [goal({ judge: openaiText('gpt-6-luna') })] + * ``` + */ +export function goal(options: { + judge: AnyTextAdapter | KeyedAdapter + maxRounds?: number +}) { + const maxRounds = options.maxRounds ?? 20 + return definePlugin({ + name: 'tanstack/goal', + setup: async (ctx) => { + const state = ctx.state(null) + // The prompt runs for each turn, so keep a copy of the goal at hand. + let current = await state.get() + // The run id of the plugin's own turn while it runs. + let ownRun: string | undefined + // The turn the plugin queued, until it starts. + let queuedTurn: Operation | undefined + + const queue = (message: string) => { + queuedTurn = ctx.session.prompt(message) + } + + /** Cancel the queued turn if it has not started. */ + const dropQueued = async () => { + const turn = queuedTurn + queuedTurn = undefined + if (turn?.status() === 'accepted') await turn.cancel() + } + + /** Change the goal, but only while `text` is still the active goal. */ + const change = async (text: string, next: Partial) => { + let applied = false + current = await state.update((saved) => { + const isActive = + saved !== null && saved.status === 'active' && saved.text === text + applied = isActive + return isActive ? { ...saved, ...next } : saved + }) + return applied + } + + /** Make `text` the active goal with a new round count, and start a turn. */ + const begin = async (text: string, reason: string) => { + const message = + reason === '' + ? `Work toward this goal: ${text}` + : keepWorking(text, reason) + // A turn that is still running or queued belongs to the old goal. + ownRun = undefined + await dropQueued() + current = await state.update(() => ({ + text, + status: 'active', + round: 0, + reason, + note: '', + queued: message, + })) + queue(message) + } + + /** Pause the goal when the plugin's own turn ends early. */ + const pauseOwnTurn = async (runId: string, note: string) => { + if (runId !== ownRun || !current) return + ownRun = undefined + await change(current.text, { status: 'paused', note }) + } + + const judge = async (text: string) => { + // ponytail: the last 10 messages, clipped. Summarize first if a judge needs more. + const transcript = transcriptText( + (await ctx.session.transcript()).slice(-10), + ).slice(-8000) + const verdict = await chat({ + adapter: await ctx.keys.adapter(options.judge), + messages: [ + { + role: 'user', + content: `${JUDGE_PROMPT}\n\nGoal: ${text}\n\nThe last part of the conversation:\n${transcript}`, + }, + ], + outputSchema: { + type: 'object', + properties: { + met: { type: 'boolean' }, + reason: { type: 'string' }, + }, + required: ['met', 'reason'], + }, + }) + if (!isVerdict(verdict)) + throw new Error('The judge did not answer with met and reason.') + return verdict + } + + return { + prompts: [ + { + id: 'tanstack/goal:active', + text: () => + current?.status === 'active' + ? `You work toward this goal: ${current.text}. Keep going until it is done.` + : '', + }, + ], + commands: { + goal: defineCommand({ + description: + 'Work until a goal is met: /goal . Also /goal, /goal stop, /goal resume', + run: async (input: unknown) => { + const arg = typeof input === 'string' ? input.trim() : '' + switch (arg) { + case '': + return describe(current, maxRounds) + case 'stop': { + const isRunning = + current?.status === 'active' || current?.status === 'paused' + if (!isRunning) return 'No goal is running.' + await dropQueued() + current = await state.update( + (saved) => + saved && { + ...saved, + status: 'stopped', + note: 'You stopped the goal.', + }, + ) + return describe(current, maxRounds) + } + case 'resume': { + const { status, queuedTurns } = ctx.session.snapshot() + // An active goal with no turn running or queued is stalled: + // its queued turn was cancelled, or the host stopped. + const isStalled = + current?.status === 'active' && + status === 'idle' && + queuedTurns === 0 + const canResume = + isStalled || + current?.status === 'paused' || + current?.status === 'stopped' + if (!current || !canResume) + return describe(current, maxRounds) + await begin(current.text, current.reason) + return describe(current, maxRounds) + } + default: + await begin(arg, '') + return describe(current, maxRounds) + } + }, + }), + }, + middleware: [ + { + name: 'tanstack/goal', + onConfig: async (run, config) => { + const saved = current + const last = config.messages.at(-1) + // A turn with a new user message. A resume has none. + const isNewMessage = + run.phase === 'init' && + last !== undefined && + last.role === 'user' + if (!isNewMessage || saved?.status !== 'active') return + if (textOf(last) !== saved.queued) { + // A late steer runs before the goal's queued turn. Drop that turn. + await dropQueued() + await change(saved.text, { + status: 'paused', + note: USER_MESSAGE, + }) + return + } + queuedTurn = undefined + ownRun = run.runId + await change(saved.text, { round: saved.round + 1, queued: '' }) + }, + onChunk: async (run, chunk) => { + const isApproval = + chunk.type === EventType.RUN_FINISHED && + chunk.outcome?.type === 'interrupt' + if (isApproval) await pauseOwnTurn(run.runId, WAITS_FOR_APPROVAL) + if (chunk.type === EventType.RUN_ERROR) + await pauseOwnTurn(run.runId, TURN_FAILED) + }, + onError: (run) => pauseOwnTurn(run.runId, TURN_FAILED), + onAbort: (run) => + pauseOwnTurn( + run.runId, + 'The last turn was cancelled. Run /goal resume to continue.', + ), + // chat() waits for onFinish, so the turn is still running here and + // the next turn can only queue. It starts after this turn ends. + onFinish: async (run) => { + const saved = current + if (run.runId !== ownRun || saved?.status !== 'active') return + ownRun = undefined + const { text, round } = saved + const verdict = await judge(text).catch((error: unknown) => + error instanceof Error ? error.message : String(error), + ) + if (typeof verdict === 'string') { + await change(text, { + status: 'paused', + note: `The judge failed: ${verdict}`, + }) + return + } + const { reason } = verdict + if (verdict.met) { + const met = await change(text, { status: 'met', reason }) + if (met) ctx.emit(GoalMet, { goal: text, reason }) + return + } + if (round >= maxRounds) { + await change(text, { + status: 'stopped', + reason, + note: `The round limit (${maxRounds}) is reached. Run /goal resume for more rounds.`, + }) + return + } + // The user queued a message while this turn ran. + if (ctx.session.snapshot().queuedTurns > 0) { + await change(text, { + status: 'paused', + reason, + note: USER_MESSAGE, + }) + return + } + const message = keepWorking(text, reason) + if (await change(text, { reason, queued: message })) + queue(message) + }, + }, + ], + } + }, + }) +} diff --git a/packages/ai-harness/src/first-party/index.ts b/packages/ai-harness/src/first-party/index.ts index 374deb8d06..e0a09d0ada 100644 --- a/packages/ai-harness/src/first-party/index.ts +++ b/packages/ai-harness/src/first-party/index.ts @@ -24,6 +24,8 @@ export { formatTodos, todos } from './todos' export type { Todo } from './todos' export { fileCommands, projectInstructions } from './files' export { compact, usage } from './session-tools' +export { GoalMet, goal, selectGoal } from './goal' +export type { Goal } from './goal' export { agents, builtInAgents } from './agents' export type { AgentProfile } from './agents' export { question } from './question' diff --git a/packages/ai-harness/tests/goal.test.ts b/packages/ai-harness/tests/goal.test.ts new file mode 100644 index 0000000000..317177bfa3 --- /dev/null +++ b/packages/ai-harness/tests/goal.test.ts @@ -0,0 +1,436 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { toolDefinition } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHost, defineHarness, definePlugin } from '../src' +import { GoalMet, goal, selectGoal } from '../src/first-party' +import { + after, + gate, + messageTexts, + mockAdapter, + text, + toolCall, + untilAborted, +} from './helpers' +import type { AnyTextAdapter, AnyTool } from '@tanstack/ai' +import type { HarnessSession } from '../src' +import type { Goal } from '../src/first-party' + +interface Verdict { + met: boolean + reason: string +} + +/** A judge that answers with `verdicts` in order. Keeps what it read. */ +function scriptedJudge(verdicts: Array) { + const { adapter } = mockAdapter([]) + const read: Array = [] + const judge: AnyTextAdapter = { + ...adapter, + structuredOutput: async (options) => { + read.push(JSON.stringify(options.chatOptions.messages)) + const data = verdicts[read.length - 1] ?? { + met: false, + reason: 'No verdict left.', + } + return { data, rawText: JSON.stringify(data) } + }, + } + return { judge, read } +} + +async function start(options: { + replies: Parameters[0] + verdicts?: Array + judge?: AnyTextAdapter + maxRounds?: number + tools?: Array + persistence?: ReturnType + busy?: 'queue' | 'reject' +}) { + const main = mockAdapter(options.replies) + const scripted = scriptedJudge(options.verdicts ?? []) + const met: Array<{ goal: string; reason: string }> = [] + const listener = definePlugin({ + name: 'test/goal-listener', + setup: (ctx) => { + ctx.on(GoalMet, (value) => met.push(value)) + }, + }) + const persistence = options.persistence ?? memoryPersistence() + const host = createHarnessHost({ persistence }) + const session = await host.open( + defineHarness({ + name: 'test/goal', + adapter: main.adapter, + tools: options.tools ?? [], + busy: options.busy, + plugins: () => [ + goal({ + judge: options.judge ?? scripted.judge, + maxRounds: options.maxRounds, + }), + listener, + ], + }), + { threadId: 't' }, + ) + return { + host, + session, + persistence, + calls: main.calls, + read: scripted.read, + met, + } +} + +/** What `/goal` shows. */ +async function status(session: HarnessSession) { + return session.command('goal') +} + +/** Wait until `/goal` shows `expected`. */ +async function waitForStatus(session: HarnessSession, expected: string) { + await vi.waitFor(async () => expect(await status(session)).toBe(expected)) +} + +/** The last message the main model got on `call`. */ +function lastMessage(call: unknown) { + return messageTexts(call).at(-1) +} + +describe('goal', () => { + it('runs turns until the judge says the goal is met', async () => { + const { host, session, calls, read, met } = await start({ + replies: [ + () => text('wrote the test'), + () => text('fixed the bug'), + () => text('all green'), + ], + verdicts: [ + { met: false, reason: 'The test fails.' }, + { met: false, reason: 'One test still fails.' }, + { met: true, reason: 'All tests pass.' }, + ], + }) + + expect(await session.command('goal', 'all tests pass')).toBe( + 'Goal: all tests pass\nStatus: active, round 0 of 20.', + ) + await waitForStatus( + session, + 'Goal: all tests pass\nStatus: met, round 3 of 20.\nLast check: All tests pass.', + ) + + expect(calls).toHaveLength(3) + expect(lastMessage(calls[0])).toBe('Work toward this goal: all tests pass') + expect(lastMessage(calls[2])).toBe( + 'Keep working on the goal: all tests pass. Last check: One test still fails.', + ) + expect(JSON.stringify(calls[0].systemPrompts)).toContain( + 'You work toward this goal: all tests pass.', + ) + expect(read).toHaveLength(3) + expect(read[0]).toContain('Goal: all tests pass') + expect(read[0]).toContain('assistant: wrote the test') + expect(met).toEqual([{ goal: 'all tests pass', reason: 'All tests pass.' }]) + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + expect(session.snapshot().queuedTurns).toBe(0) + await host.close() + }) + + it('keeps going on a harness that rejects prompts while busy', async () => { + const { host, session, calls } = await start({ + replies: [() => text('try one'), () => text('done')], + verdicts: [ + { met: false, reason: 'Not yet.' }, + { met: true, reason: 'Done.' }, + ], + busy: 'reject', + }) + + await session.command('goal', 'finish') + await waitForStatus( + session, + 'Goal: finish\nStatus: met, round 2 of 20.\nLast check: Done.', + ) + expect(calls).toHaveLength(2) + await host.close() + }) + + it('stops at the round limit and keeps the last reason', async () => { + const { host, session, calls } = await start({ + replies: [() => text('try one'), () => text('try two')], + verdicts: [ + { met: false, reason: 'Two tests fail.' }, + { met: false, reason: 'One test fails.' }, + ], + maxRounds: 2, + }) + + await session.command('goal', 'all tests pass') + await waitForStatus( + session, + 'Goal: all tests pass\nStatus: stopped, round 2 of 2.\nThe round limit (2) is reached. Run /goal resume for more rounds.\nLast check: One test fails.', + ) + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + expect(calls).toHaveLength(2) + await host.close() + }) + + it('shows the goal, stops it, and resumes it', async () => { + const hold = gate() + const { host, session, calls, read } = await start({ + replies: [after(hold.opened, 'half done'), () => text('done')], + verdicts: [{ met: true, reason: 'The file exists.' }], + }) + + expect(await status(session)).toBe( + 'No goal. Start one with /goal .', + ) + expect(await session.command('goal', 'stop')).toBe('No goal is running.') + + await session.command('goal', 'create hello.txt') + await vi.waitFor(() => expect(calls).toHaveLength(1)) + expect(await session.command('goal', 'stop')).toBe( + 'Goal: create hello.txt\nStatus: stopped, round 1 of 20.\nYou stopped the goal.', + ) + hold.open() + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + expect(read).toHaveLength(0) + + expect(await session.command('goal', 'resume')).toBe( + 'Goal: create hello.txt\nStatus: active, round 0 of 20.', + ) + await waitForStatus( + session, + 'Goal: create hello.txt\nStatus: met, round 1 of 20.\nLast check: The file exists.', + ) + expect(calls).toHaveLength(2) + expect(lastMessage(calls[1])).toBe( + 'Work toward this goal: create hello.txt', + ) + await host.close() + }) + + it('resumes an active goal whose queued turn was cancelled', async () => { + const hold = gate() + const { host, session, calls } = await start({ + replies: [after(hold.opened, 'It is noon.'), () => text('done')], + verdicts: [{ met: true, reason: 'Done.' }], + }) + + const question = session.prompt('what time is it?') + await vi.waitFor(() => expect(calls).toHaveLength(1)) + await session.command('goal', 'finish') + await vi.waitFor(() => expect(session.inputs()).toHaveLength(1)) + const [waiting] = session.inputs() + if (waiting) await session.cancelInput(waiting.inputId) + hold.open() + await question + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + expect(await status(session)).toBe( + 'Goal: finish\nStatus: active, round 0 of 20.', + ) + + await session.command('goal', 'resume') + await waitForStatus( + session, + 'Goal: finish\nStatus: met, round 1 of 20.\nLast check: Done.', + ) + expect(calls).toHaveLength(2) + await host.close() + }) + + it('pauses when the user sends a message while a goal turn runs', async () => { + const hold = gate() + const { host, session, calls } = await start({ + replies: [ + after(hold.opened, 'step one'), + () => text('It is noon.'), + () => text('step two'), + ], + verdicts: [ + { met: false, reason: 'Not yet.' }, + { met: true, reason: 'Done.' }, + ], + }) + + await session.command('goal', 'all tests pass') + await vi.waitFor(() => expect(calls).toHaveLength(1)) + const question = session.prompt('what time is it?') + hold.open() + await question + + expect(await status(session)).toBe( + 'Goal: all tests pass\nStatus: paused, round 1 of 20.\nYou sent a message, so the goal paused. Run /goal resume to continue.\nLast check: Not yet.', + ) + expect(calls).toHaveLength(2) + expect(lastMessage(calls[1])).toBe('what time is it?') + + await session.command('goal', 'resume') + await waitForStatus( + session, + 'Goal: all tests pass\nStatus: met, round 1 of 20.\nLast check: Done.', + ) + // No extra goal turn ran after the user's message. + expect(calls).toHaveLength(3) + expect(lastMessage(calls[2])).toBe( + 'Keep working on the goal: all tests pass. Last check: Not yet.', + ) + await host.close() + }) + + it('pauses, and queues no goal turn, when a late steer joins its turn', async () => { + const hold = gate() + const { host, session, calls } = await start({ + replies: [ + after(hold.opened, 'step one'), + () => text('It is noon.'), + () => text('an extra goal turn'), + ], + verdicts: [{ met: false, reason: 'Not yet.' }], + }) + + await session.command('goal', 'all tests pass') + await vi.waitFor(() => expect(calls).toHaveLength(1)) + // The last model call of the turn is in flight, so the steer comes too late. + await session.steer('what time is it?') + hold.open() + + await waitForStatus( + session, + 'Goal: all tests pass\nStatus: paused, round 1 of 20.\nYou sent a message, so the goal paused. Run /goal resume to continue.\nLast check: Not yet.', + ) + await vi.waitFor(() => { + expect(session.snapshot().status).toBe('idle') + expect(session.snapshot().queuedTurns).toBe(0) + }) + expect(calls).toHaveLength(2) + expect(lastMessage(calls[1])).toBe('what time is it?') + await host.close() + }) + + it('pauses when a user message starts a turn while the goal is active', async () => { + // A host that stopped during a goal turn leaves the goal active. + const persistence = memoryPersistence() + const active: Goal = { + text: 'all tests pass', + status: 'active', + round: 1, + reason: '', + note: '', + queued: '', + } + await persistence.stores.metadata.set('plugin:tanstack/goal', 't', active) + const { host, session, calls, read } = await start({ + persistence, + replies: [() => text('Hello.')], + }) + + await session.prompt('hello') + expect(await status(session)).toBe( + 'Goal: all tests pass\nStatus: paused, round 1 of 20.\nYou sent a message, so the goal paused. Run /goal resume to continue.', + ) + expect(calls).toHaveLength(1) + expect(read).toHaveLength(0) + await host.close() + }) + + it('does not start a turn when a turn waits for approval', async () => { + const remove = toolDefinition({ + name: 'remove', + description: 'Remove a file', + needsApproval: true, + inputSchema: z.object({ path: z.string() }), + }).server(async () => ({ removed: true })) + const { host, session, calls, read } = await start({ + replies: [() => toolCall('remove', { path: 'old.txt' })], + tools: [remove], + }) + + await session.command('goal', 'clean the folder') + await vi.waitFor(() => + expect(session.snapshot().status).toBe('requires_action'), + ) + expect(await status(session)).toBe( + 'Goal: clean the folder\nStatus: paused, round 1 of 20.\nThe last turn waits for approval. Answer it, then run /goal resume.', + ) + expect(calls).toHaveLength(1) + expect(read).toHaveLength(0) + expect(session.snapshot().queuedTurns).toBe(0) + await host.close() + }) + + it('pauses when the judge does not answer with a verdict', async () => { + const { adapter: judge } = mockAdapter([]) + const { host, session, calls } = await start({ + replies: [() => text('did something')], + judge, + }) + + await session.command('goal', 'all tests pass') + await waitForStatus( + session, + 'Goal: all tests pass\nStatus: paused, round 1 of 20.\nThe judge failed: The judge did not answer with met and reason.', + ) + expect(calls).toHaveLength(1) + await host.close() + }) + + it('keeps the goal and its state across a restart', async () => { + const persistence = memoryPersistence() + const first = await start({ + persistence, + replies: [() => text('step one'), untilAborted()], + verdicts: [{ met: false, reason: 'Half done.' }], + }) + await first.session.command('goal', 'all tests pass') + await vi.waitFor(() => expect(first.calls).toHaveLength(2)) + await first.host.close() + + const second = await start({ + persistence, + replies: [() => text('all green')], + verdicts: [{ met: true, reason: 'All tests pass.' }], + }) + expect(await status(second.session)).toBe( + 'Goal: all tests pass\nStatus: paused, round 2 of 20.\nThe last turn was cancelled. Run /goal resume to continue.\nLast check: Half done.', + ) + await second.session.command('goal', 'resume') + await waitForStatus( + second.session, + 'Goal: all tests pass\nStatus: met, round 1 of 20.\nLast check: All tests pass.', + ) + expect(lastMessage(second.calls[0])).toBe( + 'Keep working on the goal: all tests pass. Last check: Half done.', + ) + await second.host.close() + }) + + it('gives a UI the goal through the session snapshot', async () => { + const { host, session } = await start({ + replies: [() => text('done')], + verdicts: [{ met: true, reason: 'Done.' }], + }) + expect(selectGoal(session.snapshot())).toBeNull() + + await session.command('goal', 'finish') + + await vi.waitFor(() => + expect(selectGoal(session.snapshot())).toMatchObject({ + text: 'finish', + status: 'met', + }), + ) + await host.close() + }) + + it('gives no goal for goal plugin state that is not a goal', () => { + const state = { plugins: { 'tanstack/goal': { text: 'finish' } } } + + expect(selectGoal(state)).toBeNull() + }) +}) diff --git a/testing/e2e/fixtures/harness-first-party/basic.json b/testing/e2e/fixtures/harness-first-party/basic.json index 22bb740ed8..1c1b3e99ce 100644 --- a/testing/e2e/fixtures/harness-first-party/basic.json +++ b/testing/e2e/fixtures/harness-first-party/basic.json @@ -103,6 +103,24 @@ "content": "Here is a plan for Rome." } }, + { + "match": { + "userMessage": "Work toward this goal: [first-party-goal] finish the report", + "model": "gpt-5.4-nano" + }, + "response": { + "content": "{\"met\":true,\"reason\":\"The report is done.\"}" + } + }, + { + "match": { + "userMessage": "Work toward this goal: [first-party-goal] finish the report", + "model": "gpt-5.5" + }, + "response": { + "content": "The report is done." + } + }, { "match": { "userMessage": "[first-party-code-mode] which tools" diff --git a/testing/e2e/src/routes/api.harness-first-party.ts b/testing/e2e/src/routes/api.harness-first-party.ts index 814b622241..be06b724de 100644 --- a/testing/e2e/src/routes/api.harness-first-party.ts +++ b/testing/e2e/src/routes/api.harness-first-party.ts @@ -14,8 +14,10 @@ import { import { TitleFailed, agents, + goal, permissions, projectInstructions, + selectGoal, title, } from '@tanstack/ai-harness/plugins' import { memoryPersistence } from '@tanstack/ai-persistence' @@ -195,6 +197,37 @@ async function sessionTitle( return { text: turn.text, title: entry?.title, failures } } +/** + * `goal()`: `/goal` starts a turn, the judge says that the goal is met, and + * no next turn starts. + */ +async function goalLoop( + host: HarnessHost, + model: () => AnyTextAdapter, + judgeModel: AnyTextAdapter, +) { + const session = await host.open( + defineHarness({ + name: 'e2e/harness-goal', + adapter: model(), + plugins: () => [goal({ judge: judgeModel })], + }), + { threadId: 'e2e-goal' }, + ) + await session.command('goal', '[first-party-goal] finish the report') + const deadline = Date.now() + 10_000 + while ( + selectGoal(session.snapshot())?.status !== 'met' && + Date.now() < deadline + ) { + await new Promise((resolve) => setTimeout(resolve, 50)) + } + return { + goal: selectGoal(session.snapshot()), + shown: await session.command('goal'), + } +} + /** * `codeMode()` from `@tanstack/ai-code-mode/harness`: the safe `list_issues` * tool leaves the tool list and becomes `external_list_issues` in the code @@ -229,7 +262,7 @@ async function codeModeTools(host: HarnessHost, model: () => AnyTextAdapter) { /** * First-party harness plugins, one scenario per request: - * `?scenario=agents|plan-subagent|env|title|code-mode&testId=...`. The models are + * `?scenario=agents|plan-subagent|env|title|goal|code-mode&testId=...`. The models are * OpenAI adapters against aimock. */ export const Route = createFileRoute('/api/harness-first-party')({ @@ -261,6 +294,15 @@ export const Route = createFileRoute('/api/harness-first-party')({ ).adapter return Response.json(await sessionTitle(host, model, titleModel)) } + if (scenario === 'goal') { + const judgeModel = createTextAdapter( + 'openai', + 'gpt-5.4-nano', + undefined, + testId, + ).adapter + return Response.json(await goalLoop(host, model, judgeModel)) + } if (scenario === 'code-mode') { return Response.json(await codeModeTools(host, model)) } diff --git a/testing/e2e/tests/harness-first-party.spec.ts b/testing/e2e/tests/harness-first-party.spec.ts index 4c2b950613..4a8382130e 100644 --- a/testing/e2e/tests/harness-first-party.spec.ts +++ b/testing/e2e/tests/harness-first-party.spec.ts @@ -3,7 +3,7 @@ import type { APIRequestContext } from '@playwright/test' /** * First-party harness plugins in a session: `agents()`, `permissions()`, - * `projectInstructions()`, `title()`, and `codeMode()`. Each test runs one scenario of + * `projectInstructions()`, `title()`, `goal()`, and `codeMode()`. Each test runs one scenario of * `/api/harness-first-party`. The model is the OpenAI adapter against * aimock, and aimock's journal shows what each model call sent. */ @@ -124,6 +124,32 @@ test.describe('harness first-party plugins', () => { }) }) + test('goal(): the judge says that the goal is met after one turn', async ({ + request, + testId, + aimockPort, + }) => { + expect(await runScenario(request, 'goal', testId)).toEqual({ + goal: { + text: '[first-party-goal] finish the report', + status: 'met', + round: 1, + reason: 'The report is done.', + note: '', + queued: '', + }, + shown: [ + 'Goal: [first-party-goal] finish the report', + 'Status: met, round 1 of 20.', + 'Last check: The report is done.', + ].join('\n'), + }) + + // One turn of the main model and one check of the judge. + const calls = await modelCalls(request, aimockPort, testId) + expect(calls).toHaveLength(2) + }) + test('codeMode(): a safe tool moves behind execute_typescript', async ({ request, testId,