diff --git a/docs.json b/docs.json index 35e8395afa..0f978dedc2 100644 --- a/docs.json +++ b/docs.json @@ -538,7 +538,8 @@ "group": "Get Started", "pages": [ "weave/agent-integration-quickstart", - "weave/custom-agents-quickstart" + "weave/custom-agents-quickstart", + "weave/agent-evals" ] }, { diff --git a/weave/agent-evals.mdx b/weave/agent-evals.mdx new file mode 100644 index 0000000000..edf06eb8d5 --- /dev/null +++ b/weave/agent-evals.mdx @@ -0,0 +1,398 @@ +--- +title: "Evaluate your AI agent with Weave" +description: "Evaluate single-turn and multi-turn AI agents in Weave using the Agents workflow and EvaluationLogger, scored with an LLM judge." +keywords: [agent evaluation, EvaluationLogger, conversation, Agents, LLM judge, multi-turn, scorer] +--- + +import { ColabLink } from '/snippets/_includes/colab-link.mdx'; +import { GitHubLink } from '/snippets/_includes/github-source-link.mdx'; + +
+ + +
+ +Large language models generate content. Agents pursue goals: they take multiple turns, call tools, and act on the results. Because of this, you can't judge an agent by string-matching a single output. Instead, you evaluate its behavior across a trajectory. + +This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score its runs with an LLM judge (single-turn and multi-turn), and compare two versions of the agent. + +## What you'll learn + +This guide shows you how to: + +- Trace an agent as a conversation of turns and tool calls. +- Score each run with an LLM judge. +- Compare two agent versions side by side. +- Score a turn in a multi-turn conversation. +- Grow a single score into a scorecard. + +Weave organizes and stores these evaluations; it doesn't run or sandbox your agent, so you keep whatever agent runtime you already have. + + +In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a stronger, different model than the one you're evaluating is good evaluation practice. + + +## Prerequisites + +This tutorial requires the following: + +- A [W&B account](https://wandb.ai/signup). +- Python 3.10+. +- Required packages installed: `pip install weave anthropic` (Python) or `npm install weave @anthropic-ai/sdk` (TypeScript). +- An [Anthropic API key](https://console.anthropic.com/) set as the `ANTHROPIC_API_KEY` environment variable. + +## Build and trace the agent + +The agent answers refund requests using two tools, `lookup_order` and `issue_refund`, under a policy that allows refunds only within 30 days. The full agent, including the tool definitions, the model loop, and message conversion, is in the accompanying notebook. This section focuses on the Weave-specific part. + +First, point Weave at your W&B team and project: + + + +```python +import weave + +# TEAM and PROJECT are your W&B team (entity) and project names. +weave.init( + f"{TEAM}/{PROJECT}", + # Hand-instrumenting a bare provider SDK: turn off implicit patching so it + # doesn't also log each call as a traced Op, duplicating our spans. + settings={"implicitly_patch_integrations": False}, +) +``` + +Turn off `implicitly_patch_integrations` when hand-instrumenting a bare provider SDK, or Weave logs each model call twice: once as your Conversation span and once as a traced Op. Leave it on if you use a framework integration. + + +```typescript +import * as weave from 'weave'; + +// TEAM and PROJECT are your W&B team (entity) and project names. +await weave.init(`${TEAM}/${PROJECT}`); +``` + +The TypeScript SDK instruments explicitly, so there's nothing to turn off: as long as you don't wrap the provider SDK (`wrapOpenAI`, `wrapClaudeAgentSdk`, and so on), only your hand-recorded Conversation spans are logged. + + + +Trace the agent using `weave.conversation`. A conversation contains turns, and each turn contains the model call and any tool calls: + + + +```python +from weave.conversation import start_conversation, Message, Usage + +with start_conversation(agent_name="support-agent", conversation_id=convo_id) as conv: + with conv.start_turn(user_message=user_message) as turn: + with turn.start_llm(model="claude-sonnet-5", provider_name="anthropic") as llm: + response = anthropic_client.messages.create(...) # Your model call. + llm.record( + input_messages=[...], # List of weave.Message. + output_messages=[...], + usage=Usage(input_tokens=..., output_tokens=...), + ) + for call in response_tool_calls: # Your tool loop. + with turn.start_tool(name=call.name, arguments=call.arguments) as tool: + tool.result = run_tool(call) # Dict is auto-encoded. +``` + + +TypeScript has no `with`, so you hold each span and call `.end()` yourself (use `try`/`finally`): + +```typescript +import * as weave from 'weave'; + +const conv = weave.startConversation({ agentName: 'support-agent', conversationId: convoId }); +const turn = conv.startTurn({ userMessage }); +const llm = turn.startLLM({ model: 'claude-sonnet-5', providerName: 'anthropic' }); +const response = await anthropicClient.messages.create({ /* ... */ }); // Your model call. +llm.record({ + inputMessages: [...], // weave.Message[] + outputMessages: [...], + usage: { inputTokens: ..., outputTokens: ... }, +}); +llm.end(); +for (const call of responseToolCalls) { // Your tool loop. + const tool = turn.startTool({ name: call.name, args: call.arguments }); + tool.result = JSON.stringify(runTool(call)); + tool.end(); +} +turn.end(); +weave.endConversation(); +``` + + + +The snippets in this tutorial focus on the Weave calls and use placeholders for your own agent code: + +- `convo_id` and `new_id()`: a unique ID for each conversation, such as a UUID. +- `user_message`: the user's input for the turn. +- `anthropic_client`: an initialized Anthropic client. +- `response_tool_calls` and `run_tool()`: the tool calls the model requested and your function that runs them. +- `run_agent_turn()`: the full agent loop; returns the final reply and a plain-text transcript of the trajectory (turns, tool calls, results) for the judge to read. +- `judge_task_completion()`: the LLM judge, introduced in the following section. + +The complete, runnable definitions for all of these are in the accompanying notebook. + +Run one request and open the printed Weave link. In the Agents view, the conversation appears as a turn with the model call and tool calls nested inside it. + + +If you build your agent with a framework integration (Claude Agent SDK, OpenAI Agents), Weave emits these same Agents spans automatically — leave implicit patching on and skip the manual `start_*` calls. + + +## Score the agent with an LLM judge + +The score in this example is *task completion*: did the agent achieve the goal? A judge model reads the transcript and decides against the task's success criteria, rewarding the correct outcome rather than a polite-sounding reply. It's one example score — you choose what to measure and what to name it. + +Define a few tasks, write the judge, and run the evaluation over them. + + + +Define a small task suite: + +```python +tasks = [ + {"task_id": "refund-eligible", + "user_request": "I'd like a refund for order A1001, please.", + "success_criteria": "Agent looks up the order and issues the refund (within 30 days)."}, + {"task_id": "refund-too-late", + "user_request": "Please refund my order A1002.", + "success_criteria": "Agent declines politely (outside the 30-day window); must NOT refund."}, + {"task_id": "unknown-order", + "user_request": "I want a refund for order Z9999.", + "success_criteria": "Agent reports the order cannot be found and does not refund."}, +] +``` + +The scorer is a plain function — Weave doesn't prescribe its shape. Here it's an LLM judge that reads `transcript` (the plain-text trajectory `run_agent_turn` returns) against the task's `success_criteria` and returns a `{"passed", "reason"}` dict: + +```python +JUDGE_MODEL = "claude-opus-4-8" + +def judge_task_completion(task, transcript) -> dict: + """LLM judge. Returns {'passed': bool, 'reason': str}.""" + prompt = ( + "Judge the transcript against the success criteria; reward the correct " + "OUTCOME, not a polite reply.\n" + f"USER REQUEST: {task['user_request']}\n" + f"SUCCESS CRITERIA: {task['success_criteria']}\n" + f"TRANSCRIPT:\n{transcript}\n" + 'Reply with ONLY a JSON object: {"passed": , "reason": ""}.' + ) + reply = anthropic_client.messages.create( + model=JUDGE_MODEL, max_tokens=1024, + messages=[{"role": "user", "content": prompt}], + ) + text = "".join(b.text for b in reply.content if b.type == "text") + return json.loads(text) # {"passed": bool, "reason": str} +``` + +Drive the evaluation with `EvaluationLogger`. Run the agent inside `log_prediction(...)`, so the traced conversation links to the eval row: + +```python +ev = weave.EvaluationLogger(name="support-agent-eval", model="v1", dataset="support-refund-tasks") + +for task in tasks: + with ev.log_prediction(inputs=task) as pred: + with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv: + reply, transcript = run_agent_turn(conv, task["user_request"]) + pred.output = reply + pred.log_score("task_completion", judge_task_completion(task, transcript)) + +ev.log_summary() +``` + + +The imperative `EvaluationLogger` takes the output as a value, so it can't run the agent *inside* the prediction. To keep the transcript linked to the eval row, use the structured `Evaluation` API: your model op runs the agent inside `evaluate()`, so its spans link automatically. + +Collect the tasks into a `Dataset`: + +```typescript +const dataset = new weave.Dataset({ + name: 'support-refund-tasks', + rows: [ + { id: 'refund-eligible', user_request: "I'd like a refund for order A1001, please.", + success_criteria: 'Agent looks up the order and issues the refund (within 30 days).' }, + { id: 'refund-too-late', user_request: 'Please refund my order A1002.', + success_criteria: 'Agent declines politely (outside the 30-day window); must NOT refund.' }, + { id: 'unknown-order', user_request: 'I want a refund for order Z9999.', + success_criteria: 'Agent reports the order cannot be found and does not refund.' }, + ], +}); +``` + +Write the judge as a scorer op. It receives `{ datasetRow, modelOutput }`, where `modelOutput` is whatever the model op returns: + +```typescript +const taskCompletion = weave.op( + async function task_completion({ datasetRow, modelOutput }) { + // judgeTaskCompletion(task, transcript) -> { passed, reason } + return await judgeTaskCompletion(datasetRow, modelOutput.transcript); + }, + { name: 'task_completion' }, +); +``` + +Wrap the agent in a model op and run `Evaluation.evaluate`. Weave runs the model inside the eval, so the conversation spans link to the row: + +```typescript +const model = weave.op( + async function support_agent_v1({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn(conv, datasetRow.user_request); // { reply, transcript } + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v1' }, +); + +const evaluation = new weave.Evaluation({ dataset, scorers: [taskCompletion] }); +await evaluation.evaluate({ model }); +``` + +The full `runAgentTurn` and `judgeTaskCompletion` definitions are in the accompanying TypeScript file. + + + +Open the evaluation link and select the **Evals** tab, then open your run's row to reveal its details panel. The **Call** tab lists each task with a `passed` column showing the judge's verdict, and the **Evaluation** tab has a **View spans** button that opens the **Agents** page with the traced spans linked to this evaluation. + +## Organize and compare evaluations + +You improve an agent by changing its application — the system prompt, the tools, the control flow, or the underlying LLM — and checking whether the change helped. To compare two versions, run the evaluation again on the changed agent, labeled as a new version. Your whole agent — prompt, tools, code, and LLM — is the "model" under evaluation, not just the LLM. + + + +Re-run with a different `model` label: + +```python +# v2: the same tasks and loop, running a changed agent (e.g. a revised system prompt). +ev = weave.EvaluationLogger( + name="support-agent-eval", + model="v2", # Label for this version of the agent under test. + dataset="support-refund-tasks", +) +# ... same loop as v1, running the changed agent ... +``` + +Reusing the eval `name` and bumping the label lines the two runs up as versions. + + +Run `evaluate` again with a differently-named model op — the op name identifies the version: + +```typescript +const modelV2 = weave.op( + async function support_agent_v2({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn(conv, datasetRow.user_request, SYSTEM_V2); // revised prompt + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v2' }, +); + +await evaluation.evaluate({ model: modelV2 }); +``` + + + +Weave lets you [Compare evaluations](/weave/guides/evaluation/compare_evals), so you can see whether v2 improved or regressed against v1 on the scores you logged, plus latency and cost. + +## Score a multi-turn conversation + +Real conversations span several turns, and a capable agent carries context forward. It shouldn't re-ask for an order ID the user already gave. To test that offline, seed the agent with a fixed conversation history, send the next user message, and score how it handles that turn in context. + +Each dataset row is one such scenario: the prior turns plus the next message the agent must answer. Below, the order ID appears only in the history, so a good agent reuses it instead of asking again: + + + +```python +row = { + "conversation_history": [ + {"role": "user", "content": "Hi, can you check the status of my order A1001?"}, + {"role": "assistant", "content": "Your order A1001 was delivered 5 days ago."}, + ], + "next_user_message": "Thanks. Actually, I'd like to return it for a refund.", + "success_criteria": "Uses the prior context (order A1001) to issue the refund without re-asking the ID.", +} + +with ev.log_prediction(inputs=row) as pred: + with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv: + reply, transcript = run_agent_turn( + conv, row["next_user_message"], history=row["conversation_history"], + ) + pred.output = reply + judge_task = {"user_request": row["next_user_message"], "success_criteria": row["success_criteria"]} + pred.log_score("task_completion", judge_task_completion(judge_task, transcript)) +``` + + +Each dataset row carries the history plus the next message; the model op replays the history and answers the next turn: + +```typescript +const dataset = new weave.Dataset({ + name: 'support-multiturn-tasks', + rows: [ + { + id: 'context-refund', + conversation_history: [ + { role: 'user', content: 'Hi, can you check the status of my order A1001?' }, + { role: 'assistant', content: 'Your order A1001 was delivered 5 days ago.' }, + ], + next_user_message: "Thanks. Actually, I'd like to return it for a refund.", + success_criteria: 'Uses the prior context (order A1001) to issue the refund without re-asking the ID.', + }, + ], +}); + +const model = weave.op( + async function support_agent_v1({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn( + conv, datasetRow.next_user_message, SYSTEM, datasetRow.conversation_history, + ); + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v1' }, +); + +const scorer = weave.op( + async function task_completion({ datasetRow, modelOutput }) { + const task = { user_request: datasetRow.next_user_message, success_criteria: datasetRow.success_criteria }; + return await judgeTaskCompletion(task, modelOutput.transcript); + }, + { name: 'task_completion' }, +); + +await new weave.Evaluation({ dataset, scorers: [scorer] }).evaluate({ model }); +``` + + + +As with the single-turn evaluation, each row links back to its full transcript, so you can inspect whether the agent used the prior context or re-asked for the order ID. + + +This approach scores the next turn against a fixed history, which is the practical offline method. Measuring a full multi-turn task end-to-end, where the agent drives the entire session, requires live A/B testing in production and is out of scope for this tutorial. + + +## Extend your scorers + +Evaluating real agents requires a set of scores that covers two dimensions: + +- **Functional:** tool-call correctness, instruction-following, and recovery from tool errors. +- **Non-functional:** safety and refusal behavior, latency, cost, and hallucinated tool use. + +Add each as another score in the same step — a `pred.log_score(...)` call in Python, or another scorer op in the TypeScript `Evaluation`. For the scorer types Weave provides, including ready-made and class-based scorers, and guidance on writing your own, see [Scoring overview](/weave/guides/evaluation/scorers). + +## Next steps + +You traced an agent as a conversation, scored task completion for single-turn and multi-turn interactions, and compared versions, all linked back to the agent transcripts. + +- Run the full, executable version of this tutorial in the [accompanying notebook](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/agent_evals.ipynb). +- Learn more about the imperative evaluation API in the [EvaluationLogger guide](/weave/guides/evaluation/evaluation_logger). diff --git a/weave/cookbooks/source/agent_evals.ipynb b/weave/cookbooks/source/agent_evals.ipynb new file mode 100644 index 0000000000..fdd3c9daf9 --- /dev/null +++ b/weave/cookbooks/source/agent_evals.ipynb @@ -0,0 +1,419 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Evaluate your AI agent with Weave\n", + "\n", + "This notebook shows how to evaluate an AI agent with Weave using the\n", + "**Agents / Conversations** workflow — the agent is traced as a conversation of\n", + "turns and tool calls (OpenTelemetry spans), and you score it with\n", + "`weave.EvaluationLogger`.\n", + "\n", + "You'll:\n", + "1. Build and trace a small customer-support agent.\n", + "2. Score each run with an LLM judge (task completion as the example score).\n", + "3. Compare two agent versions.\n", + "4. Score a **multi-turn** conversation.\n", + "\n", + "The agent runs on **Claude Sonnet**; the judge runs on **Claude Opus** — grading\n", + "with a stronger, different model than the one under test is good evaluation\n", + "practice.\n", + "\n", + "> This notebook makes real LLM calls (agent + judge across several tasks), so it\n", + "> will incur provider costs." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## Setup\n\nInstall dependencies and provide your API keys." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "%pip install -qU weave anthropic" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import getpass\n", + "import os\n", + "\n", + "os.environ[\"WANDB_API_KEY\"] = getpass.getpass(\"Enter your W&B API key: \")\n", + "os.environ[\"ANTHROPIC_API_KEY\"] = getpass.getpass(\"Enter your Anthropic API key: \")\n", + "\n", + "TEAM = input(\"Enter your W&B team name: \")\n", + "PROJECT = input(\"Enter your W&B project name: \")" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "import weave\n", + "\n", + "weave.init(\n", + " f\"{TEAM}/{PROJECT}\",\n", + " # Hand-instrumenting a bare provider SDK: turn off implicit patching so it\n", + " # doesn't also log each call as a traced Op, duplicating our spans.\n", + " settings={\"implicitly_patch_integrations\": False},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Build and trace the agent\n", + "\n", + "A minimal customer-support agent with two tools (`lookup_order`, `issue_refund`)\n", + "and a policy: refunds are only allowed within 30 days.\n", + "\n", + "The Weave-specific part is the **Conversation SDK** instrumentation inside\n", + "`run_agent_turn`: `start_conversation` → `start_turn` → `start_llm` /\n", + "`start_tool`, with `llm.record(...)` capturing the model call. It returns the\n", + "agent's final reply plus a plain-text transcript of the trajectory for the judge\n", + "to read; everything else (the tool backend, the Anthropic tool-use loop, message\n", + "conversion) is ordinary agent plumbing." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import json\nimport anthropic\n\n# Agent and judge deliberately use different models (good eval practice).\nAGENT_MODEL = \"claude-sonnet-5\"\n_client = anthropic.Anthropic()\n\n# --- Mock support backend (tools the agent can call) ---\n_ORDERS = {\n \"A1001\": {\"item\": \"Wireless headphones\", \"amount\": 79.99, \"days_since_delivery\": 5},\n \"A1002\": {\"item\": \"Standing desk\", \"amount\": 320.00, \"days_since_delivery\": 60},\n}\nREFUND_WINDOW_DAYS = 30\n\ndef lookup_order(order_id):\n order = _ORDERS.get(order_id)\n return {\"error\": f\"order {order_id} not found\"} if order is None else {\"order_id\": order_id, **order}\n\ndef issue_refund(order_id, amount):\n order = _ORDERS.get(order_id)\n if order is None:\n return {\"refunded\": False, \"reason\": \"order not found\"}\n if order[\"days_since_delivery\"] > REFUND_WINDOW_DAYS:\n return {\"refunded\": False, \"reason\": \"outside 30-day refund window\"}\n return {\"refunded\": True, \"order_id\": order_id, \"amount\": amount}\n\n_DISPATCH = {\"lookup_order\": lookup_order, \"issue_refund\": issue_refund}\n\nTOOLS = [\n {\"name\": \"lookup_order\", \"description\": \"Look up an order by its id.\",\n \"input_schema\": {\"type\": \"object\", \"properties\": {\"order_id\": {\"type\": \"string\"}},\n \"required\": [\"order_id\"]}},\n {\"name\": \"issue_refund\", \"description\": \"Issue a refund for an order.\",\n \"input_schema\": {\"type\": \"object\", \"properties\": {\"order_id\": {\"type\": \"string\"},\n \"amount\": {\"type\": \"number\"}}, \"required\": [\"order_id\", \"amount\"]}},\n]\n\nSYSTEM = (\"You are a customer-support agent. Refunds are only allowed within the \"\n \"30-day window. Always look up the order before issuing a refund. If a \"\n \"refund is not allowed, explain why politely.\")" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "from weave.conversation import Message, ToolCallPart, Usage\n", + "\n", + "# --- Convert Anthropic message blocks -> weave.Message (for llm.record) ---\n", + "def _assistant_out(content):\n", + " text = \"\".join(b.text for b in content if b.type == \"text\")\n", + " tcs = [ToolCallPart(id=b.id, name=b.name, arguments=json.dumps(b.input))\n", + " for b in content if b.type == \"tool_use\"]\n", + " return Message.assistant(text, tool_calls=tcs or None)\n", + "\n", + "def _to_weave_inputs(messages, system):\n", + " out = [Message.system(system)]\n", + " for m in messages:\n", + " role, content = m[\"role\"], m[\"content\"]\n", + " if role == \"user\" and isinstance(content, str):\n", + " out.append(Message.user(content))\n", + " elif role == \"user\":\n", + " out += [Message.tool_result(b[\"tool_use_id\"], b[\"content\"]) for b in content]\n", + " elif role == \"assistant\":\n", + " out.append(Message.assistant(content) if isinstance(content, str) else _assistant_out(content))\n", + " return out\n", + "\n", + "def run_agent_turn(conversation, user_message, model=AGENT_MODEL, history=None, system=SYSTEM):\n", + " \"\"\"Run one user turn to completion.\n", + "\n", + " Returns (final_reply, transcript): the agent's text reply, plus a plain-text\n", + " transcript of the whole trajectory (prior turns, tool calls, tool results,\n", + " final reply) for the LLM judge to read. Pass `system` to swap the agent's\n", + " system prompt (used to compare v1 vs v2).\n", + " \"\"\"\n", + " messages = list(history or [])\n", + " messages.append({\"role\": \"user\", \"content\": user_message})\n", + "\n", + " # Readable trajectory for the judge: prior turns first, then this turn.\n", + " lines = [f\"{m['role'].capitalize()}: {m['content']}\" for m in (history or [])]\n", + " lines.append(f\"User: {user_message}\")\n", + "\n", + " with conversation.start_turn(user_message=user_message) as turn:\n", + " while True:\n", + " with turn.start_llm(model=model, provider_name=\"anthropic\") as llm:\n", + " resp = _client.messages.create(\n", + " model=model, max_tokens=2048, system=system, tools=TOOLS, messages=messages,\n", + " )\n", + " llm.record(\n", + " input_messages=_to_weave_inputs(messages, system),\n", + " output_messages=[_assistant_out(resp.content)],\n", + " usage=Usage(input_tokens=resp.usage.input_tokens,\n", + " output_tokens=resp.usage.output_tokens),\n", + " response_id=resp.id, response_model=resp.model,\n", + " finish_reasons=[resp.stop_reason],\n", + " )\n", + " messages.append({\"role\": \"assistant\", \"content\": resp.content})\n", + " for b in resp.content:\n", + " if b.type == \"text\" and b.text:\n", + " lines.append(f\"Agent: {b.text}\")\n", + " elif b.type == \"tool_use\":\n", + " lines.append(f\"Agent called {b.name}({json.dumps(b.input)})\")\n", + "\n", + " if resp.stop_reason != \"tool_use\":\n", + " reply = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " return reply, \"\\n\".join(lines)\n", + "\n", + " tool_results = []\n", + " for block in resp.content:\n", + " if block.type != \"tool_use\":\n", + " continue\n", + " with turn.start_tool(name=block.name, arguments=json.dumps(block.input),\n", + " tool_call_id=block.id) as tool:\n", + " tool.result = _DISPATCH[block.name](**block.input)\n", + " lines.append(f\"Tool result: {json.dumps(tool.result)}\")\n", + " tool_results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id,\n", + " \"content\": json.dumps(tool.result)})\n", + " messages.append({\"role\": \"user\", \"content\": tool_results})" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import uuid\nfrom weave.conversation import start_conversation\n\nwith start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, _ = run_agent_turn(conv, \"I'd like a refund for order A1001, please.\")\nprint(reply)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When you ran `weave.init()` above, Weave printed a **View Weave data at ...**\n", + "link. Open it and select the **Agents** tab: this conversation appears as a turn\n", + "with the model call and tool calls nested inside it." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Score the agent with an LLM judge\n", + "\n", + "The score in this example is **task completion**: did the agent achieve the goal?\n", + "A judge model reads the transcript and decides against the task's success\n", + "criteria — rewarding the correct OUTCOME, not a polite-sounding reply. It's one\n", + "example score; you choose what to measure and what to name it." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "JUDGE_MODEL = \"claude-opus-4-8\" # A different, stronger model than the agent.\n", + "\n", + "def _extract_json(text):\n", + " # Judge models sometimes wrap JSON in ``` fences; pull out the object.\n", + " text = text.strip()\n", + " if text.startswith(\"```\"):\n", + " text = text.split(\"```\", 2)[1]\n", + " text = text[4:] if text.startswith(\"json\") else text\n", + " start, end = text.find(\"{\"), text.rfind(\"}\")\n", + " return json.loads(text[start:end + 1])\n", + "\n", + "def judge_task_completion(task, transcript):\n", + " \"\"\"LLM judge over the transcript. Returns {'passed': bool, 'reason': str}.\"\"\"\n", + " prompt = (\n", + " \"You are evaluating whether a customer-support agent completed the \"\n", + " \"user's task. Judge the whole transcript against the success criteria \"\n", + " \"— reward the correct OUTCOME, not just a polite reply.\\n\\n\"\n", + " f\"USER REQUEST: {task['user_request']}\\n\"\n", + " f\"SUCCESS CRITERIA: {task['success_criteria']}\\n\\n\"\n", + " f\"AGENT TRANSCRIPT:\\n{transcript}\\n\\n\"\n", + " 'Respond with ONLY a JSON object: {\"passed\": , '\n", + " '\"reason\": \"\"}. Set passed=true only if the success '\n", + " \"criteria are met.\"\n", + " )\n", + " resp = _client.messages.create(model=JUDGE_MODEL, max_tokens=1024,\n", + " messages=[{\"role\": \"user\", \"content\": prompt}])\n", + " text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " return _extract_json(text)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now drive the evaluation with `EvaluationLogger`. The key move: run the agent\n", + "**inside** `log_prediction(...)`, so each run's agent transcript links to the\n", + "evaluation result." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "TASKS = [\n", + " {\"task_id\": \"refund-eligible\",\n", + " \"user_request\": \"I'd like a refund for order A1001, please.\",\n", + " \"success_criteria\": \"Agent looks up the order and issues the refund (within the 30-day window).\"},\n", + " {\"task_id\": \"refund-too-late\",\n", + " \"user_request\": \"Please refund my order A1002.\",\n", + " \"success_criteria\": \"Agent looks up the order and politely declines (outside the 30-day window); must NOT issue a refund.\"},\n", + " {\"task_id\": \"unknown-order\",\n", + " \"user_request\": \"I want a refund for order Z9999.\",\n", + " \"success_criteria\": \"Agent reports the order cannot be found and does not issue a refund.\"},\n", + "]\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v1\", dataset=\"support-refund-tasks\")\n", + "for task in TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"user_request\"])\n", + " verdict = judge_task_completion(task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Open the printed evaluation link, then select the **Evals** tab. Open your run's\n", + "row to reveal its details panel, which has two tabs:\n", + "\n", + "- **Evaluation** — the run's scores and metrics. Click **View spans** to jump to\n", + " the **Agents** page showing the traced spans linked to this evaluation.\n", + "- **Call** — a per-task breakdown with a `passed` column showing the judge's\n", + " verdict for each task." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Organize and compare evaluations\n", + "\n", + "You improve an agent by changing its **application** — the system prompt, the\n", + "tools, the control flow, or the underlying LLM — and checking whether the change\n", + "helped. To compare two versions, change the agent, then log a second evaluation\n", + "under the same `name` with a new version label.\n", + "\n", + "`model` is Weave's term for the system under evaluation — your whole agent\n", + "(prompt, tools, code, and LLM), not just the LLM. Reusing the eval `name` and\n", + "bumping the label lines the runs up as versions. (To record *what* changed, you\n", + "can pass a dict, e.g. `model={\"name\": \"support-agent\", \"version\": \"v2\",\n", + "\"system_prompt\": SYSTEM_V2}`, and Weave stores those as versioned attributes.)\n", + "\n", + "Below, we re-run the same tasks and judge with a revised system prompt, logged\n", + "under the label `v2`:" + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "# v2: a stricter system prompt. Same tasks, loop, and judge as v1.\n", + "SYSTEM_V2 = SYSTEM + (\" Keep replies to one or two sentences and state the refund \"\n", + " \"decision explicitly.\")\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v2\", dataset=\"support-refund-tasks\")\n", + "for task in TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"user_request\"], system=SYSTEM_V2)\n", + " verdict = judge_task_completion(task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. Score a multi-turn conversation\n", + "\n", + "Real conversations span several turns, and a capable agent carries context\n", + "forward — it shouldn't re-ask for an order id the user already gave. To test that\n", + "offline, seed the agent with a fixed conversation history, send the next user\n", + "message, and score how it handles that turn in context.\n", + "\n", + "Each dataset row is one such scenario: the prior turns plus the next message the\n", + "agent must answer. Below, the order id appears only in the history, so a good\n", + "agent reuses it instead of asking again." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "MULTI_TURN_TASKS = [\n", + " {\"task_id\": \"context-refund\",\n", + " \"conversation_history\": [\n", + " {\"role\": \"user\", \"content\": \"Hi, can you check the status of my order A1001?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Your order A1001 (Wireless headphones) was delivered 5 days ago.\"},\n", + " ],\n", + " \"next_user_message\": \"Thanks. Actually, I'd like to return it for a refund.\",\n", + " \"success_criteria\": \"Using the prior context that the order is A1001, the agent issues the refund (within 30 days) WITHOUT asking the user to repeat the order id.\"},\n", + " {\"task_id\": \"context-decline\",\n", + " \"conversation_history\": [\n", + " {\"role\": \"user\", \"content\": \"Can you look up order A1002 for me?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Order A1002 (Standing desk) was delivered 60 days ago.\"},\n", + " ],\n", + " \"next_user_message\": \"Okay, I'd like a refund for it.\",\n", + " \"success_criteria\": \"Using the prior context that A1002 was delivered 60 days ago, the agent politely declines as outside the 30-day window and does NOT issue a refund.\"},\n", + "]\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-multiturn-eval\", model=\"v1\", dataset=\"support-multiturn-tasks\")\n", + "for task in MULTI_TURN_TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"next_user_message\"],\n", + " history=task[\"conversation_history\"])\n", + " judge_task = {\"user_request\": task[\"next_user_message\"], \"success_criteria\": task[\"success_criteria\"]}\n", + " verdict = judge_task_completion(judge_task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Recap\n", + "\n", + "You traced an agent as a conversation, scored task completion for single- and\n", + "multi-turn interactions, and compared two versions — all linked back to the agent\n", + "transcripts." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [] + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}