From 5a51afacc23d8b0d7e62a578ad464dd0cc1822ac Mon Sep 17 00:00:00 2001 From: anastasiaguspan Date: Tue, 21 Jul 2026 11:00:11 -0400 Subject: [PATCH 1/3] docs(weave): add "Evaluate your AI agent with Weave" tutorial Add a new Get Started tutorial covering agent evaluation with the Weave Agents/Conversations workflow: tracing an agent as a conversation of turns and tool calls, scoring single-turn and multi-turn task completion with an LLM judge via EvaluationLogger, and comparing agent versions. Includes a companion Colab notebook with the full runnable source. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs.json | 3 +- weave/agent-evals.mdx | 207 +++++++++++++++++++++++ weave/cookbooks/source/agent_evals.ipynb | 136 +++++++++++++++ 3 files changed, 345 insertions(+), 1 deletion(-) create mode 100644 weave/agent-evals.mdx create mode 100644 weave/cookbooks/source/agent_evals.ipynb diff --git a/docs.json b/docs.json index 35e8395afa..0f978dedc2 100644 --- a/docs.json +++ b/docs.json @@ -538,7 +538,8 @@ "group": "Get Started", "pages": [ "weave/agent-integration-quickstart", - "weave/custom-agents-quickstart" + "weave/custom-agents-quickstart", + "weave/agent-evals" ] }, { diff --git a/weave/agent-evals.mdx b/weave/agent-evals.mdx new file mode 100644 index 0000000000..81890a1884 --- /dev/null +++ b/weave/agent-evals.mdx @@ -0,0 +1,207 @@ +--- +title: "Evaluate your AI agent with Weave" +description: "Evaluate single-turn and multi-turn AI agents in Weave using the Agents workflow and EvaluationLogger, scored with an LLM judge." +keywords: [agent evaluation, task completion, LLM judge, EvaluationLogger, conversation, multi-turn, scorer] +--- + +import { ColabLink } from '/snippets/_includes/colab-link.mdx'; +import { GitHubLink } from '/snippets/_includes/github-source-link.mdx'; + +
+ + +
+ +Large language models generate content. Agents pursue goals: they take multiple turns, call tools, and act on the results. Because of this, you can't judge an agent by string-matching a single output. Instead, you evaluate its behavior across a trajectory. + +This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score *task completion* for single-turn and multi-turn interactions, and compare two versions of the agent. + +This guide is for developers building agents who want to measure and improve agent behavior systematically. It complements [Evaluate RAG applications](/weave/tutorial-rag), which evaluates a retrieval pipeline rather than an agent. + +## What you'll learn + +This guide shows you how to: + +- Build and trace an agent as a conversation of turns and tool calls. +- Score single-turn task completion with an LLM judge. +- Organize and compare agent evaluations in the Weave UI. +- Score a multi-turn conversation. +- Extend your scorers beyond task completion. + +A few terms are used throughout: a *task* is one row of your dataset, a *trial* is one run of the agent on a task, the *transcript* is the traced conversation, and a *scorer* (or grader) assigns a score to a trial. Weave is the harness that organizes these. Weave doesn't run or sandbox your agent, so you keep whatever agent runtime you already have. + + +In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a different model than the one you're evaluating is good evaluation practice. + + +## Prerequisites + +This tutorial requires the following: + +- A [W&B account](https://wandb.ai/signup). +- Python 3.10+. +- Required packages installed: `pip install weave anthropic`. +- An [Anthropic API key](https://console.anthropic.com/) set as the `ANTHROPIC_API_KEY` environment variable. + +## Build and trace the agent + +The agent answers refund requests using two tools, `lookup_order` and `issue_refund`, under a policy that allows refunds only within 30 days. The full agent, including the tool definitions, the model loop, and message conversion, is in the accompanying notebook. This section focuses on the Weave-specific part. + +First, initialize Weave: + +```python +import weave + +weave.init( + "agent-eval-tutorial", + # Turn off implicit patching of manual Weave Ops tracing (Traces tab) — the + # bare Anthropic SDK integration would log each LLM call as a legacy Op/Call + # in parallel with our hand-rolled Conversation SDK spans, duplicating it. + settings={"implicitly_patch_integrations": False}, +) +``` + +This tutorial instruments the agent by hand, so it turns implicit patching off. Otherwise Weave's built-in Anthropic integration would also log each model call as a legacy Op, duplicating the spans you record manually. If you build your agent with an agent-framework integration instead, keep implicit patching on, as described in the tip at the end of this section. + +Trace the agent with the Conversation SDK. A conversation contains turns, and each turn contains the model call and any tool calls: + +```python +from weave.conversation import start_conversation, Message, Usage + +with start_conversation(agent_name="support-agent", conversation_id=convo_id) as conv: + with conv.start_turn(user_message=user_message) as turn: + with turn.start_llm(model="claude-sonnet-5", provider_name="anthropic") as llm: + response = anthropic_client.messages.create(...) # your model call + llm.record( + input_messages=[...], # weave.Message list + output_messages=[...], + usage=Usage(input_tokens=..., output_tokens=...), + ) + for call in response_tool_calls: # your tool loop + with turn.start_tool(name=call.name, arguments=call.arguments) as tool: + tool.result = run_tool(call) # dict is auto-encoded +``` + +The snippets in this tutorial focus on the Weave calls and use placeholders for your own agent code: + +- `convo_id` and `new_id()`: a unique ID for each conversation, such as a UUID. +- `user_message`: the user's input for the turn. +- `anthropic_client`: an initialized Anthropic client. +- `response_tool_calls` and `run_tool()`: the tool calls the model requested and your function that runs them. +- `run_agent_turn()`: the full agent loop that ties the preceding pieces together. +- `task_completion()` and `judge_task`: the LLM judge and the task it scores, introduced in the following section. + +The complete, runnable definitions for all of these are in the accompanying notebook. + +Run one request and open the printed Weave link. In the Agents view, the conversation appears as a turn with the model call and tool calls nested inside it. + + +This tutorial instruments the agent by hand. Alternatively, if you build your agent with an agent-framework integration such as the Claude Agent SDK or OpenAI Agents, Weave emits these same Agents spans automatically: keep implicit patching on (the default) and skip the manual `start_*` calls. Auto-patching the bare provider SDK, by contrast, produces legacy Ops and Calls, not Agents spans, which is why the hand-instrumented path turns it off. + + +## Score task completion + +Task completion asks a single question: did the agent achieve the goal? A judge model reads the transcript and decides against the task's success criteria. It rewards the correct outcome, not a polite-sounding reply. + +Define a small task suite: + +```python +tasks = [ + {"task_id": "refund-eligible", + "user_request": "I'd like a refund for order A1001, please.", + "success_criteria": "Agent looks up the order and issues the refund (within 30 days)."}, + {"task_id": "refund-too-late", + "user_request": "Please refund my order A1002.", + "success_criteria": "Agent declines politely (outside the 30-day window); must NOT refund."}, + {"task_id": "unknown-order", + "user_request": "I want a refund for order Z9999.", + "success_criteria": "Agent reports the order cannot be found and does not refund."}, +] +``` + +Write the scorer as an LLM judge that returns a pass or fail with a reason. The judge prompts Claude for a JSON verdict and parses it, which works across SDK versions. The full judge is in the accompanying notebook: + +```python +def task_completion(task, transcript) -> dict: + """LLM judge over the transcript. Returns {'passed': bool, 'reason': str}.""" + ... # a Claude Opus call that reads task + transcript +``` + +Now drive the evaluation with `EvaluationLogger`. The key step is to run the agent inside `log_prediction(...)`, so that the traced agent conversation links to the evaluation result: + +```python +ev = weave.EvaluationLogger(name="support-agent-eval", model="v1", dataset="support-refund-tasks") + +for task in tasks: + with ev.log_prediction(inputs=task) as pred: + with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv: + reply, transcript = run_agent_turn(conv, task["user_request"]) + pred.output = reply + pred.log_score("task_completion", task_completion(task, transcript)) + +ev.log_summary() +``` + +Open the evaluation link. Each row is a task with its task-completion score, output, latency, and cost, and a link to the full agent transcript for that trial. When a task fails, that link takes you straight to the conversation that produced it. + + +Task completion is one signal, not the whole score. Add more signals with extra `pred.log_score(...)` calls in the same block, for example `tool_call_correct` or `instruction_following`. + + +## Organize and compare evaluations + +To compare two versions of your agent, run the same evaluation again with a different `model` label: + +```python +ev = weave.EvaluationLogger(name="support-agent-eval", model="v2", dataset="support-refund-tasks") +# ... same loop, with the agent's prompt or model changed ... +``` + +Weave lays the runs side by side in the comparison view, so you can read the task-completion rate, tool-call correctness, latency, and cost for v1 against v2. This answers the baseline-relative question of whether a change did as well as the baseline. Every row still links to its transcript, so a regression is one click from the failing conversation. + +## Score a multi-turn conversation + +To evaluate a turn in context, load a fixed conversation history, append one new user turn, and score how the agent handles it. In other words, given the conversation state so far, does the agent handle the next turn well? + +Each dataset row carries the prior turns plus the next message. In the following example, the order ID appears only in the history, so a good agent uses that context instead of asking again: + +```python +row = { + "conversation_history": [ + {"role": "user", "content": "Hi, can you check the status of my order A1001?"}, + {"role": "assistant", "content": "Your order A1001 was delivered 5 days ago."}, + ], + "next_user_message": "Thanks. Actually, I'd like to return it for a refund.", + "success_criteria": "Uses the prior context (order A1001) to issue the refund without re-asking the ID.", +} + +with ev.log_prediction(inputs=row) as pred: + with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv: + reply, transcript = run_agent_turn( + conv, row["next_user_message"], history=row["conversation_history"], + ) + pred.output = reply + pred.log_score("task_completion", task_completion(judge_task, transcript)) +``` + +As with the single-turn evaluation, each row links back to its full transcript, so you can inspect whether the agent used the prior context or re-asked for the order ID. + + +This approach scores the next turn against a fixed history, which is the practical offline method. Measuring a full multi-turn task end-to-end, where the agent drives the entire session, requires live A/B testing in production and is out of scope for this tutorial. + + +## Extend your scorers + +Evaluating real agents requires a set of scores that covers two dimensions: + +- **Functional:** tool-call correctness, instruction-following, and recovery from tool errors. +- **Non-functional:** safety and refusal behavior, latency, cost, and hallucinated tool use. + +Add each as another `pred.log_score(name, value)` call inside the prediction block. + +## Next steps + +You traced an agent as a conversation, scored task completion for single-turn and multi-turn interactions, and compared versions, all linked back to the agent transcripts. + +- Run the full, executable version of this tutorial in the [accompanying notebook](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/agent_evals.ipynb). +- Learn more about the imperative evaluation API in the [EvaluationLogger guide](/weave/guides/evaluation/evaluation_logger). diff --git a/weave/cookbooks/source/agent_evals.ipynb b/weave/cookbooks/source/agent_evals.ipynb new file mode 100644 index 0000000000..687c62ea38 --- /dev/null +++ b/weave/cookbooks/source/agent_evals.ipynb @@ -0,0 +1,136 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": "# Evaluate your AI agent with Weave\n\nThis notebook shows how to evaluate an AI agent with Weave using the new\n**Agents / Conversations** workflow \u2014 the agent is traced as a conversation of\nturns and tool calls (OpenTelemetry spans), and you score it with\n`weave.EvaluationLogger`.\n\nYou'll:\n1. Build and trace a small customer-support agent.\n2. Score single-turn **task completion** with an LLM judge.\n3. Compare two agent versions.\n4. Score a **multi-turn** conversation.\n\nThe agent runs on **Claude Sonnet**; the judge runs on **Claude Opus** \u2014 using a\ndifferent model to grade than to act is good evaluation practice.\n\n> This notebook makes real LLM calls (agent + judge across several tasks), so it\n> will incur provider costs." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## Setup\n\nInstall dependencies and provide your API keys." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "!pip install -qU weave anthropic" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import os, getpass\n\n# In Colab, prefer the Secrets panel (key icon) named WANDB_API_KEY / ANTHROPIC_API_KEY.\ntry:\n from google.colab import userdata\n for k in [\"WANDB_API_KEY\", \"ANTHROPIC_API_KEY\"]:\n os.environ.setdefault(k, userdata.get(k))\nexcept Exception:\n for k in [\"WANDB_API_KEY\", \"ANTHROPIC_API_KEY\"]:\n if not os.environ.get(k):\n os.environ[k] = getpass.getpass(f\"{k}: \")" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import weave\n\nweave.init(\n \"agent-eval-tutorial\", # your Weave project\n # Turn off implicit patching of manual Weave Ops tracing (Traces tab) \u2014 the\n # bare Anthropic SDK integration would log each LLM call as a legacy Op/Call\n # in parallel with our hand-rolled Conversation SDK spans, duplicating it.\n settings={\"implicitly_patch_integrations\": False},\n)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 1. Build and trace the agent\n\nA minimal customer-support agent with two tools (`lookup_order`, `issue_refund`)\nand a policy: refunds are only allowed within 30 days.\n\nThe Weave-specific part is the **Conversation SDK** instrumentation inside\n`run_agent_turn`: `start_conversation` \u2192 `start_turn` \u2192 `start_llm` /\n`start_tool`, with `llm.record(...)` capturing the model call. Everything else\n(the tool backend, the Anthropic tool-use loop, message conversion) is ordinary\nagent plumbing." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import json\nimport anthropic\n\n# Agent and judge deliberately use different models (good eval practice).\nAGENT_MODEL = \"claude-sonnet-5\"\n_client = anthropic.Anthropic()\n\n# --- Mock support backend (tools the agent can call) ---\n_ORDERS = {\n \"A1001\": {\"item\": \"Wireless headphones\", \"amount\": 79.99, \"days_since_delivery\": 5},\n \"A1002\": {\"item\": \"Standing desk\", \"amount\": 320.00, \"days_since_delivery\": 60},\n}\nREFUND_WINDOW_DAYS = 30\n\ndef lookup_order(order_id):\n order = _ORDERS.get(order_id)\n return {\"error\": f\"order {order_id} not found\"} if order is None else {\"order_id\": order_id, **order}\n\ndef issue_refund(order_id, amount):\n order = _ORDERS.get(order_id)\n if order is None:\n return {\"refunded\": False, \"reason\": \"order not found\"}\n if order[\"days_since_delivery\"] > REFUND_WINDOW_DAYS:\n return {\"refunded\": False, \"reason\": \"outside 30-day refund window\"}\n return {\"refunded\": True, \"order_id\": order_id, \"amount\": amount}\n\n_DISPATCH = {\"lookup_order\": lookup_order, \"issue_refund\": issue_refund}\n\nTOOLS = [\n {\"name\": \"lookup_order\", \"description\": \"Look up an order by its id.\",\n \"input_schema\": {\"type\": \"object\", \"properties\": {\"order_id\": {\"type\": \"string\"}},\n \"required\": [\"order_id\"]}},\n {\"name\": \"issue_refund\", \"description\": \"Issue a refund for an order.\",\n \"input_schema\": {\"type\": \"object\", \"properties\": {\"order_id\": {\"type\": \"string\"},\n \"amount\": {\"type\": \"number\"}}, \"required\": [\"order_id\", \"amount\"]}},\n]\n\nSYSTEM = (\"You are a customer-support agent. Refunds are only allowed within the \"\n \"30-day window. Always look up the order before issuing a refund. If a \"\n \"refund is not allowed, explain why politely.\")" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "from weave.conversation import Message, ToolCallPart, Usage\n\n# --- Convert Anthropic message blocks <-> weave.Message (agent plumbing) ---\ndef _assistant_out(content):\n text = \"\".join(b.text for b in content if b.type == \"text\")\n tcs = [ToolCallPart(id=b.id, name=b.name, arguments=json.dumps(b.input))\n for b in content if b.type == \"tool_use\"]\n return Message.assistant(text, tool_calls=tcs or None)\n\ndef _to_weave_inputs(messages):\n out = [Message.system(SYSTEM)]\n for m in messages:\n role, content = m[\"role\"], m[\"content\"]\n if role == \"user\" and isinstance(content, str):\n out.append(Message.user(content))\n elif role == \"user\":\n out += [Message.tool_result(b[\"tool_use_id\"], b[\"content\"]) for b in content]\n elif role == \"assistant\":\n out.append(Message.assistant(content) if isinstance(content, str) else _assistant_out(content))\n return out\n\ndef run_agent_turn(conversation, user_message, model=AGENT_MODEL, history=None):\n # Run one user turn to completion; return (final_reply, full_message_log).\n messages = list(history or [])\n messages.append({\"role\": \"user\", \"content\": user_message})\n\n with conversation.start_turn(user_message=user_message) as turn:\n while True:\n with turn.start_llm(model=model, provider_name=\"anthropic\") as llm:\n resp = _client.messages.create(\n model=model, max_tokens=2048, system=SYSTEM, tools=TOOLS, messages=messages,\n )\n llm.record(\n input_messages=_to_weave_inputs(messages),\n output_messages=[_assistant_out(resp.content)],\n usage=Usage(input_tokens=resp.usage.input_tokens,\n output_tokens=resp.usage.output_tokens),\n response_id=resp.id, response_model=resp.model,\n finish_reasons=[resp.stop_reason],\n )\n messages.append({\"role\": \"assistant\", \"content\": resp.content})\n\n if resp.stop_reason != \"tool_use\":\n return \"\".join(b.text for b in resp.content if b.type == \"text\"), messages\n\n tool_results = []\n for block in resp.content:\n if block.type != \"tool_use\":\n continue\n with turn.start_tool(name=block.name, arguments=json.dumps(block.input),\n tool_call_id=block.id) as tool:\n tool.result = _DISPATCH[block.name](**block.input)\n tool_results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id,\n \"content\": json.dumps(tool.result)})\n messages.append({\"role\": \"user\", \"content\": tool_results})" + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "import uuid\nfrom weave.conversation import start_conversation\n\nwith start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, _ = run_agent_turn(conv, \"I'd like a refund for order A1001, please.\")\nprint(reply)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "Open the printed Weave link and find the **Agents** view: the conversation\nappears as a turn with the model call and tool calls nested inside it.\n\n> **Prefer auto-instrumentation?** If you build your agent with an\n> agent-framework integration (e.g. the Claude Agent SDK or OpenAI Agents),\n> Weave emits these same Agents spans automatically \u2014 leave implicit patching on\n> and skip the manual `start_*` calls. Auto-patching the *bare* provider SDK,\n> by contrast, produces legacy Ops/Calls, not Agents spans." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 2. Score task completion (single turn)\n\n**Task completion** = an LLM judge decides whether the agent achieved the goal,\njudged against the task's success criteria from the whole transcript \u2014 not a\ngolden-string match." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "JUDGE_MODEL = \"claude-opus-4-8\" # a different, stronger model than the agent\n\ndef _extract_json(text):\n text = text.strip()\n if text.startswith(\"```\"):\n text = text.split(\"```\", 2)[1]\n text = text[4:] if text.startswith(\"json\") else text\n start, end = text.find(\"{\"), text.rfind(\"}\")\n return json.loads(text[start:end + 1])\n\ndef judge_task_completion(task, transcript):\n prompt = (\n \"You are evaluating whether a customer-support agent completed the \"\n \"user's task. Judge the whole transcript against the success criteria \"\n \"\u2014 reward the correct OUTCOME, not just a polite reply.\\n\\n\"\n f\"USER REQUEST: {task['user_request']}\\n\"\n f\"SUCCESS CRITERIA: {task['success_criteria']}\\n\\n\"\n f\"AGENT TRANSCRIPT:\\n{transcript}\\n\\n\"\n 'Respond with ONLY a JSON object: {\"passed\": , '\n '\"reason\": \"\"}. Set passed=true only if the success '\n \"criteria are met.\"\n )\n resp = _client.messages.create(model=JUDGE_MODEL, max_tokens=1024,\n messages=[{\"role\": \"user\", \"content\": prompt}])\n text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n return _extract_json(text)\n\ndef render_transcript(messages, final_reply):\n lines = []\n for m in messages:\n role, content = m[\"role\"], m[\"content\"]\n if isinstance(content, str):\n lines.append(f\"{role.capitalize()}: {content}\"); continue\n for b in content:\n if isinstance(b, dict):\n lines.append(f\"Tool result: {b['content']}\")\n elif b.type == \"text\" and b.text:\n lines.append(f\"Agent: {b.text}\")\n elif b.type == \"tool_use\":\n lines.append(f\"Agent called {b.name}({json.dumps(b.input)})\")\n lines.append(f\"Final reply: {final_reply}\")\n return \"\\n\".join(lines)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "Now drive the evaluation with `EvaluationLogger`. The key move: run the agent\n**inside** `log_prediction(...)`, so each trial's agent transcript links to the\nevaluation result." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "TASKS = [\n {\"task_id\": \"refund-eligible\",\n \"user_request\": \"I'd like a refund for order A1001, please.\",\n \"success_criteria\": \"Agent looks up the order and issues the refund (within the 30-day window).\"},\n {\"task_id\": \"refund-too-late\",\n \"user_request\": \"Please refund my order A1002.\",\n \"success_criteria\": \"Agent looks up the order and politely declines (outside the 30-day window); must NOT issue a refund.\"},\n {\"task_id\": \"unknown-order\",\n \"user_request\": \"I want a refund for order Z9999.\",\n \"success_criteria\": \"Agent reports the order cannot be found and does not issue a refund.\"},\n]\n\nev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v1\", dataset=\"support-refund-tasks\")\nfor task in TASKS:\n with ev.log_prediction(inputs=task) as pred:\n with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, messages = run_agent_turn(conv, task[\"user_request\"])\n verdict = judge_task_completion(task, render_transcript(messages, reply))\n pred.output = reply\n pred.log_score(\"task_completion\", verdict)\n print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\nev.log_summary()\nprint(\"Evaluation:\", ev.ui_url)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "Open the evaluation link: each row is a task, with its task-completion score,\noutput, latency, and cost \u2014 and a link to the full agent trace for that trial.\n\n**A scorecard, not one number.** Task completion is one signal. Add more with\nextra `pred.log_score(...)` calls in the same block \u2014 e.g. `tool_call_correct`\nor `instruction_following`." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 3. Organize and compare evaluations\n\nRun the same evaluation for a second agent version by changing the `model` label\n(e.g. `EvaluationLogger(model=\"v2\", ...)` after editing the agent's prompt or\nswitching its model). Weave lays the two runs **side by side** in the comparison\nview, so you can read task-completion rate, tool-call correctness, latency, and\ncost for v1 vs v2 \u2014 the baseline-relative question (\"did the candidate do as well\nas the baseline?\"). Every row still links to its agent transcript, so a\nregression is one click from the failing conversation." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 4. Score a multi-turn conversation\n\nTo evaluate a turn *in context*, load a fixed conversation history, append one\nnew user turn, and score how the agent handles it \u2014 i.e. *\"given the\nconversation state so far, does the agent handle the next turn well?\"*\n\nBelow, the order id is only mentioned in the prior turns; a good agent should use\nthat context instead of asking again." + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": "MULTI_TURN_TASKS = [\n {\"task_id\": \"context-refund\",\n \"conversation_history\": [\n {\"role\": \"user\", \"content\": \"Hi, can you check the status of my order A1001?\"},\n {\"role\": \"assistant\", \"content\": \"Your order A1001 (Wireless headphones) was delivered 5 days ago.\"},\n ],\n \"next_user_message\": \"Thanks. Actually, I'd like to return it for a refund.\",\n \"success_criteria\": \"Using the prior context that the order is A1001, the agent issues the refund (within 30 days) WITHOUT asking the user to repeat the order id.\"},\n {\"task_id\": \"context-decline\",\n \"conversation_history\": [\n {\"role\": \"user\", \"content\": \"Can you look up order A1002 for me?\"},\n {\"role\": \"assistant\", \"content\": \"Order A1002 (Standing desk) was delivered 60 days ago.\"},\n ],\n \"next_user_message\": \"Okay, I'd like a refund for it.\",\n \"success_criteria\": \"Using the prior context that A1002 was delivered 60 days ago, the agent politely declines as outside the 30-day window and does NOT issue a refund.\"},\n]\n\nev = weave.EvaluationLogger(name=\"support-agent-multiturn-eval\", model=\"v1\", dataset=\"support-multiturn-tasks\")\nfor task in MULTI_TURN_TASKS:\n with ev.log_prediction(inputs=task) as pred:\n with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, messages = run_agent_turn(conv, task[\"next_user_message\"],\n history=task[\"conversation_history\"])\n judge_task = {\"user_request\": task[\"next_user_message\"], \"success_criteria\": task[\"success_criteria\"]}\n verdict = judge_task_completion(judge_task, render_transcript(messages, reply))\n pred.output = reply\n pred.log_score(\"task_completion\", verdict)\n print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\nev.log_summary()\nprint(\"Evaluation:\", ev.ui_url)" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "> **Scope.** This scores the *next* turn against a fixed history \u2014 the practical\n> offline approach. Measuring a full multi-turn task end-to-end (where the agent\n> drives the whole session) requires live A/B testing in production and is out of\n> scope here." + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": "## 5. Extend your scorers\n\nReal agents need a scorecard covering both:\n- **Functional:** tool-call correctness, instruction-following, recovery from tool errors.\n- **Non-functional:** safety / refusal behavior, latency, cost, hallucinated-tool use.\n\nAdd each as another `pred.log_score(name, value)` inside the prediction block.\n\n## Recap\nYou traced an agent as a conversation, scored task completion for single and\nmulti-turn interactions, and compared versions \u2014 all linked back to the agent\ntranscripts. For **online** evaluation (automatically scoring every agent trace\nin production), see Weave's production monitoring docs." + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [] + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file From 75157bef19c617a331606cf1e303286219c80e58 Mon Sep 17 00:00:00 2001 From: Anastasia Guspan Date: Wed, 22 Jul 2026 15:07:59 -0400 Subject: [PATCH 2/3] Apply suggestion from @anastasiaguspan --- weave/agent-evals.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/weave/agent-evals.mdx b/weave/agent-evals.mdx index 81890a1884..54debe7088 100644 --- a/weave/agent-evals.mdx +++ b/weave/agent-evals.mdx @@ -16,7 +16,6 @@ Large language models generate content. Agents pursue goals: they take multiple This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score *task completion* for single-turn and multi-turn interactions, and compare two versions of the agent. -This guide is for developers building agents who want to measure and improve agent behavior systematically. It complements [Evaluate RAG applications](/weave/tutorial-rag), which evaluates a retrieval pipeline rather than an agent. ## What you'll learn From 0a12a7afd0aa7e8400c604f9de578efc50bd3e2b Mon Sep 17 00:00:00 2001 From: anastasiaguspan Date: Mon, 27 Jul 2026 09:16:33 -0400 Subject: [PATCH 3/3] docs(weave): add Python/TypeScript tabs to agent evals tutorial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold per-language prose and code into on the init, conversation-trace, Score, Compare, and Multi-turn sections, keeping shared prose outside the tabs. Demote task_completion to an example LLM judge rather than a Weave concept, trim the scope note and Extend section, and fix the compare_evals and scorers links. Expand the accompanying Python notebook to a runnable 20 cells: %pip install, getpass key entry, run_agent_turn returning (reply, transcript), judge_task_completion, and Evals-tab navigation clues. The Python path is verified end to end. The TypeScript tabs still reference an accompanying TypeScript file that is not yet in the repo, and their claim that conversation spans link to eval rows does not hold on weave JS 0.16.3 — two SDK bugs (complete-mode project writes, EvalLinkSpanProcessor registration) are with engineering. Resolve both before this page ships. Co-Authored-By: Claude Opus 5 (1M context) --- weave/agent-evals.mdx | 282 +++++++++++++++---- weave/cookbooks/source/agent_evals.ipynb | 327 +++++++++++++++++++++-- 2 files changed, 542 insertions(+), 67 deletions(-) diff --git a/weave/agent-evals.mdx b/weave/agent-evals.mdx index 54debe7088..edf06eb8d5 100644 --- a/weave/agent-evals.mdx +++ b/weave/agent-evals.mdx @@ -1,7 +1,7 @@ --- title: "Evaluate your AI agent with Weave" description: "Evaluate single-turn and multi-turn AI agents in Weave using the Agents workflow and EvaluationLogger, scored with an LLM judge." -keywords: [agent evaluation, task completion, LLM judge, EvaluationLogger, conversation, multi-turn, scorer] +keywords: [agent evaluation, EvaluationLogger, conversation, Agents, LLM judge, multi-turn, scorer] --- import { ColabLink } from '/snippets/_includes/colab-link.mdx'; @@ -14,23 +14,22 @@ import { GitHubLink } from '/snippets/_includes/github-source-link.mdx'; Large language models generate content. Agents pursue goals: they take multiple turns, call tools, and act on the results. Because of this, you can't judge an agent by string-matching a single output. Instead, you evaluate its behavior across a trajectory. -This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score *task completion* for single-turn and multi-turn interactions, and compare two versions of the agent. - +This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score its runs with an LLM judge (single-turn and multi-turn), and compare two versions of the agent. ## What you'll learn This guide shows you how to: -- Build and trace an agent as a conversation of turns and tool calls. -- Score single-turn task completion with an LLM judge. -- Organize and compare agent evaluations in the Weave UI. -- Score a multi-turn conversation. -- Extend your scorers beyond task completion. +- Trace an agent as a conversation of turns and tool calls. +- Score each run with an LLM judge. +- Compare two agent versions side by side. +- Score a turn in a multi-turn conversation. +- Grow a single score into a scorecard. -A few terms are used throughout: a *task* is one row of your dataset, a *trial* is one run of the agent on a task, the *transcript* is the traced conversation, and a *scorer* (or grader) assigns a score to a trial. Weave is the harness that organizes these. Weave doesn't run or sandbox your agent, so you keep whatever agent runtime you already have. +Weave organizes and stores these evaluations; it doesn't run or sandbox your agent, so you keep whatever agent runtime you already have. -In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a different model than the one you're evaluating is good evaluation practice. +In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a stronger, different model than the one you're evaluating is good evaluation practice. ## Prerequisites @@ -39,47 +38,90 @@ This tutorial requires the following: - A [W&B account](https://wandb.ai/signup). - Python 3.10+. -- Required packages installed: `pip install weave anthropic`. +- Required packages installed: `pip install weave anthropic` (Python) or `npm install weave @anthropic-ai/sdk` (TypeScript). - An [Anthropic API key](https://console.anthropic.com/) set as the `ANTHROPIC_API_KEY` environment variable. ## Build and trace the agent The agent answers refund requests using two tools, `lookup_order` and `issue_refund`, under a policy that allows refunds only within 30 days. The full agent, including the tool definitions, the model loop, and message conversion, is in the accompanying notebook. This section focuses on the Weave-specific part. -First, initialize Weave: +First, point Weave at your W&B team and project: + + ```python import weave +# TEAM and PROJECT are your W&B team (entity) and project names. weave.init( - "agent-eval-tutorial", - # Turn off implicit patching of manual Weave Ops tracing (Traces tab) — the - # bare Anthropic SDK integration would log each LLM call as a legacy Op/Call - # in parallel with our hand-rolled Conversation SDK spans, duplicating it. + f"{TEAM}/{PROJECT}", + # Hand-instrumenting a bare provider SDK: turn off implicit patching so it + # doesn't also log each call as a traced Op, duplicating our spans. settings={"implicitly_patch_integrations": False}, ) ``` -This tutorial instruments the agent by hand, so it turns implicit patching off. Otherwise Weave's built-in Anthropic integration would also log each model call as a legacy Op, duplicating the spans you record manually. If you build your agent with an agent-framework integration instead, keep implicit patching on, as described in the tip at the end of this section. +Turn off `implicitly_patch_integrations` when hand-instrumenting a bare provider SDK, or Weave logs each model call twice: once as your Conversation span and once as a traced Op. Leave it on if you use a framework integration. + + +```typescript +import * as weave from 'weave'; + +// TEAM and PROJECT are your W&B team (entity) and project names. +await weave.init(`${TEAM}/${PROJECT}`); +``` -Trace the agent with the Conversation SDK. A conversation contains turns, and each turn contains the model call and any tool calls: +The TypeScript SDK instruments explicitly, so there's nothing to turn off: as long as you don't wrap the provider SDK (`wrapOpenAI`, `wrapClaudeAgentSdk`, and so on), only your hand-recorded Conversation spans are logged. + + +Trace the agent using `weave.conversation`. A conversation contains turns, and each turn contains the model call and any tool calls: + + + ```python from weave.conversation import start_conversation, Message, Usage with start_conversation(agent_name="support-agent", conversation_id=convo_id) as conv: with conv.start_turn(user_message=user_message) as turn: with turn.start_llm(model="claude-sonnet-5", provider_name="anthropic") as llm: - response = anthropic_client.messages.create(...) # your model call + response = anthropic_client.messages.create(...) # Your model call. llm.record( - input_messages=[...], # weave.Message list + input_messages=[...], # List of weave.Message. output_messages=[...], usage=Usage(input_tokens=..., output_tokens=...), ) - for call in response_tool_calls: # your tool loop + for call in response_tool_calls: # Your tool loop. with turn.start_tool(name=call.name, arguments=call.arguments) as tool: - tool.result = run_tool(call) # dict is auto-encoded + tool.result = run_tool(call) # Dict is auto-encoded. +``` + + +TypeScript has no `with`, so you hold each span and call `.end()` yourself (use `try`/`finally`): + +```typescript +import * as weave from 'weave'; + +const conv = weave.startConversation({ agentName: 'support-agent', conversationId: convoId }); +const turn = conv.startTurn({ userMessage }); +const llm = turn.startLLM({ model: 'claude-sonnet-5', providerName: 'anthropic' }); +const response = await anthropicClient.messages.create({ /* ... */ }); // Your model call. +llm.record({ + inputMessages: [...], // weave.Message[] + outputMessages: [...], + usage: { inputTokens: ..., outputTokens: ... }, +}); +llm.end(); +for (const call of responseToolCalls) { // Your tool loop. + const tool = turn.startTool({ name: call.name, args: call.arguments }); + tool.result = JSON.stringify(runTool(call)); + tool.end(); +} +turn.end(); +weave.endConversation(); ``` + + The snippets in this tutorial focus on the Weave calls and use placeholders for your own agent code: @@ -87,21 +129,25 @@ The snippets in this tutorial focus on the Weave calls and use placeholders for - `user_message`: the user's input for the turn. - `anthropic_client`: an initialized Anthropic client. - `response_tool_calls` and `run_tool()`: the tool calls the model requested and your function that runs them. -- `run_agent_turn()`: the full agent loop that ties the preceding pieces together. -- `task_completion()` and `judge_task`: the LLM judge and the task it scores, introduced in the following section. +- `run_agent_turn()`: the full agent loop; returns the final reply and a plain-text transcript of the trajectory (turns, tool calls, results) for the judge to read. +- `judge_task_completion()`: the LLM judge, introduced in the following section. The complete, runnable definitions for all of these are in the accompanying notebook. Run one request and open the printed Weave link. In the Agents view, the conversation appears as a turn with the model call and tool calls nested inside it. -This tutorial instruments the agent by hand. Alternatively, if you build your agent with an agent-framework integration such as the Claude Agent SDK or OpenAI Agents, Weave emits these same Agents spans automatically: keep implicit patching on (the default) and skip the manual `start_*` calls. Auto-patching the bare provider SDK, by contrast, produces legacy Ops and Calls, not Agents spans, which is why the hand-instrumented path turns it off. +If you build your agent with a framework integration (Claude Agent SDK, OpenAI Agents), Weave emits these same Agents spans automatically — leave implicit patching on and skip the manual `start_*` calls. -## Score task completion +## Score the agent with an LLM judge + +The score in this example is *task completion*: did the agent achieve the goal? A judge model reads the transcript and decides against the task's success criteria, rewarding the correct outcome rather than a polite-sounding reply. It's one example score — you choose what to measure and what to name it. -Task completion asks a single question: did the agent achieve the goal? A judge model reads the transcript and decides against the task's success criteria. It rewards the correct outcome, not a polite-sounding reply. +Define a few tasks, write the judge, and run the evaluation over them. + + Define a small task suite: ```python @@ -118,15 +164,30 @@ tasks = [ ] ``` -Write the scorer as an LLM judge that returns a pass or fail with a reason. The judge prompts Claude for a JSON verdict and parses it, which works across SDK versions. The full judge is in the accompanying notebook: +The scorer is a plain function — Weave doesn't prescribe its shape. Here it's an LLM judge that reads `transcript` (the plain-text trajectory `run_agent_turn` returns) against the task's `success_criteria` and returns a `{"passed", "reason"}` dict: ```python -def task_completion(task, transcript) -> dict: - """LLM judge over the transcript. Returns {'passed': bool, 'reason': str}.""" - ... # a Claude Opus call that reads task + transcript +JUDGE_MODEL = "claude-opus-4-8" + +def judge_task_completion(task, transcript) -> dict: + """LLM judge. Returns {'passed': bool, 'reason': str}.""" + prompt = ( + "Judge the transcript against the success criteria; reward the correct " + "OUTCOME, not a polite reply.\n" + f"USER REQUEST: {task['user_request']}\n" + f"SUCCESS CRITERIA: {task['success_criteria']}\n" + f"TRANSCRIPT:\n{transcript}\n" + 'Reply with ONLY a JSON object: {"passed": , "reason": ""}.' + ) + reply = anthropic_client.messages.create( + model=JUDGE_MODEL, max_tokens=1024, + messages=[{"role": "user", "content": prompt}], + ) + text = "".join(b.text for b in reply.content if b.type == "text") + return json.loads(text) # {"passed": bool, "reason": str} ``` -Now drive the evaluation with `EvaluationLogger`. The key step is to run the agent inside `log_prediction(...)`, so that the traced agent conversation links to the evaluation result: +Drive the evaluation with `EvaluationLogger`. Run the agent inside `log_prediction(...)`, so the traced conversation links to the eval row: ```python ev = weave.EvaluationLogger(name="support-agent-eval", model="v1", dataset="support-refund-tasks") @@ -136,34 +197,118 @@ for task in tasks: with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv: reply, transcript = run_agent_turn(conv, task["user_request"]) pred.output = reply - pred.log_score("task_completion", task_completion(task, transcript)) + pred.log_score("task_completion", judge_task_completion(task, transcript)) ev.log_summary() ``` + + +The imperative `EvaluationLogger` takes the output as a value, so it can't run the agent *inside* the prediction. To keep the transcript linked to the eval row, use the structured `Evaluation` API: your model op runs the agent inside `evaluate()`, so its spans link automatically. + +Collect the tasks into a `Dataset`: + +```typescript +const dataset = new weave.Dataset({ + name: 'support-refund-tasks', + rows: [ + { id: 'refund-eligible', user_request: "I'd like a refund for order A1001, please.", + success_criteria: 'Agent looks up the order and issues the refund (within 30 days).' }, + { id: 'refund-too-late', user_request: 'Please refund my order A1002.', + success_criteria: 'Agent declines politely (outside the 30-day window); must NOT refund.' }, + { id: 'unknown-order', user_request: 'I want a refund for order Z9999.', + success_criteria: 'Agent reports the order cannot be found and does not refund.' }, + ], +}); +``` -Open the evaluation link. Each row is a task with its task-completion score, output, latency, and cost, and a link to the full agent transcript for that trial. When a task fails, that link takes you straight to the conversation that produced it. +Write the judge as a scorer op. It receives `{ datasetRow, modelOutput }`, where `modelOutput` is whatever the model op returns: - -Task completion is one signal, not the whole score. Add more signals with extra `pred.log_score(...)` calls in the same block, for example `tool_call_correct` or `instruction_following`. - +```typescript +const taskCompletion = weave.op( + async function task_completion({ datasetRow, modelOutput }) { + // judgeTaskCompletion(task, transcript) -> { passed, reason } + return await judgeTaskCompletion(datasetRow, modelOutput.transcript); + }, + { name: 'task_completion' }, +); +``` + +Wrap the agent in a model op and run `Evaluation.evaluate`. Weave runs the model inside the eval, so the conversation spans link to the row: + +```typescript +const model = weave.op( + async function support_agent_v1({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn(conv, datasetRow.user_request); // { reply, transcript } + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v1' }, +); + +const evaluation = new weave.Evaluation({ dataset, scorers: [taskCompletion] }); +await evaluation.evaluate({ model }); +``` + +The full `runAgentTurn` and `judgeTaskCompletion` definitions are in the accompanying TypeScript file. + + + +Open the evaluation link and select the **Evals** tab, then open your run's row to reveal its details panel. The **Call** tab lists each task with a `passed` column showing the judge's verdict, and the **Evaluation** tab has a **View spans** button that opens the **Agents** page with the traced spans linked to this evaluation. ## Organize and compare evaluations -To compare two versions of your agent, run the same evaluation again with a different `model` label: +You improve an agent by changing its application — the system prompt, the tools, the control flow, or the underlying LLM — and checking whether the change helped. To compare two versions, run the evaluation again on the changed agent, labeled as a new version. Your whole agent — prompt, tools, code, and LLM — is the "model" under evaluation, not just the LLM. + + + +Re-run with a different `model` label: ```python -ev = weave.EvaluationLogger(name="support-agent-eval", model="v2", dataset="support-refund-tasks") -# ... same loop, with the agent's prompt or model changed ... +# v2: the same tasks and loop, running a changed agent (e.g. a revised system prompt). +ev = weave.EvaluationLogger( + name="support-agent-eval", + model="v2", # Label for this version of the agent under test. + dataset="support-refund-tasks", +) +# ... same loop as v1, running the changed agent ... ``` -Weave lays the runs side by side in the comparison view, so you can read the task-completion rate, tool-call correctness, latency, and cost for v1 against v2. This answers the baseline-relative question of whether a change did as well as the baseline. Every row still links to its transcript, so a regression is one click from the failing conversation. +Reusing the eval `name` and bumping the label lines the two runs up as versions. + + +Run `evaluate` again with a differently-named model op — the op name identifies the version: + +```typescript +const modelV2 = weave.op( + async function support_agent_v2({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn(conv, datasetRow.user_request, SYSTEM_V2); // revised prompt + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v2' }, +); + +await evaluation.evaluate({ model: modelV2 }); +``` + + + +Weave lets you [Compare evaluations](/weave/guides/evaluation/compare_evals), so you can see whether v2 improved or regressed against v1 on the scores you logged, plus latency and cost. ## Score a multi-turn conversation -To evaluate a turn in context, load a fixed conversation history, append one new user turn, and score how the agent handles it. In other words, given the conversation state so far, does the agent handle the next turn well? +Real conversations span several turns, and a capable agent carries context forward. It shouldn't re-ask for an order ID the user already gave. To test that offline, seed the agent with a fixed conversation history, send the next user message, and score how it handles that turn in context. -Each dataset row carries the prior turns plus the next message. In the following example, the order ID appears only in the history, so a good agent uses that context instead of asking again: +Each dataset row is one such scenario: the prior turns plus the next message the agent must answer. Below, the order ID appears only in the history, so a good agent reuses it instead of asking again: + + ```python row = { "conversation_history": [ @@ -180,8 +325,55 @@ with ev.log_prediction(inputs=row) as pred: conv, row["next_user_message"], history=row["conversation_history"], ) pred.output = reply - pred.log_score("task_completion", task_completion(judge_task, transcript)) + judge_task = {"user_request": row["next_user_message"], "success_criteria": row["success_criteria"]} + pred.log_score("task_completion", judge_task_completion(judge_task, transcript)) +``` + + +Each dataset row carries the history plus the next message; the model op replays the history and answers the next turn: + +```typescript +const dataset = new weave.Dataset({ + name: 'support-multiturn-tasks', + rows: [ + { + id: 'context-refund', + conversation_history: [ + { role: 'user', content: 'Hi, can you check the status of my order A1001?' }, + { role: 'assistant', content: 'Your order A1001 was delivered 5 days ago.' }, + ], + next_user_message: "Thanks. Actually, I'd like to return it for a refund.", + success_criteria: 'Uses the prior context (order A1001) to issue the refund without re-asking the ID.', + }, + ], +}); + +const model = weave.op( + async function support_agent_v1({ datasetRow }) { + const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() }); + try { + return await runAgentTurn( + conv, datasetRow.next_user_message, SYSTEM, datasetRow.conversation_history, + ); + } finally { + weave.endConversation(); + } + }, + { name: 'support-agent-v1' }, +); + +const scorer = weave.op( + async function task_completion({ datasetRow, modelOutput }) { + const task = { user_request: datasetRow.next_user_message, success_criteria: datasetRow.success_criteria }; + return await judgeTaskCompletion(task, modelOutput.transcript); + }, + { name: 'task_completion' }, +); + +await new weave.Evaluation({ dataset, scorers: [scorer] }).evaluate({ model }); ``` + + As with the single-turn evaluation, each row links back to its full transcript, so you can inspect whether the agent used the prior context or re-asked for the order ID. @@ -196,7 +388,7 @@ Evaluating real agents requires a set of scores that covers two dimensions: - **Functional:** tool-call correctness, instruction-following, and recovery from tool errors. - **Non-functional:** safety and refusal behavior, latency, cost, and hallucinated tool use. -Add each as another `pred.log_score(name, value)` call inside the prediction block. +Add each as another score in the same step — a `pred.log_score(...)` call in Python, or another scorer op in the TypeScript `Evaluation`. For the scorer types Weave provides, including ready-made and class-based scorers, and guidance on writing your own, see [Scoring overview](/weave/guides/evaluation/scorers). ## Next steps diff --git a/weave/cookbooks/source/agent_evals.ipynb b/weave/cookbooks/source/agent_evals.ipynb index 687c62ea38..fdd3c9daf9 100644 --- a/weave/cookbooks/source/agent_evals.ipynb +++ b/weave/cookbooks/source/agent_evals.ipynb @@ -3,7 +3,27 @@ { "cell_type": "markdown", "metadata": {}, - "source": "# Evaluate your AI agent with Weave\n\nThis notebook shows how to evaluate an AI agent with Weave using the new\n**Agents / Conversations** workflow \u2014 the agent is traced as a conversation of\nturns and tool calls (OpenTelemetry spans), and you score it with\n`weave.EvaluationLogger`.\n\nYou'll:\n1. Build and trace a small customer-support agent.\n2. Score single-turn **task completion** with an LLM judge.\n3. Compare two agent versions.\n4. Score a **multi-turn** conversation.\n\nThe agent runs on **Claude Sonnet**; the judge runs on **Claude Opus** \u2014 using a\ndifferent model to grade than to act is good evaluation practice.\n\n> This notebook makes real LLM calls (agent + judge across several tasks), so it\n> will incur provider costs." + "source": [ + "# Evaluate your AI agent with Weave\n", + "\n", + "This notebook shows how to evaluate an AI agent with Weave using the\n", + "**Agents / Conversations** workflow — the agent is traced as a conversation of\n", + "turns and tool calls (OpenTelemetry spans), and you score it with\n", + "`weave.EvaluationLogger`.\n", + "\n", + "You'll:\n", + "1. Build and trace a small customer-support agent.\n", + "2. Score each run with an LLM judge (task completion as the example score).\n", + "3. Compare two agent versions.\n", + "4. Score a **multi-turn** conversation.\n", + "\n", + "The agent runs on **Claude Sonnet**; the judge runs on **Claude Opus** — grading\n", + "with a stronger, different model than the one under test is good evaluation\n", + "practice.\n", + "\n", + "> This notebook makes real LLM calls (agent + judge across several tasks), so it\n", + "> will incur provider costs." + ] }, { "cell_type": "markdown", @@ -15,26 +35,58 @@ "metadata": {}, "execution_count": null, "outputs": [], - "source": "!pip install -qU weave anthropic" + "source": [ + "%pip install -qU weave anthropic" + ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], - "source": "import os, getpass\n\n# In Colab, prefer the Secrets panel (key icon) named WANDB_API_KEY / ANTHROPIC_API_KEY.\ntry:\n from google.colab import userdata\n for k in [\"WANDB_API_KEY\", \"ANTHROPIC_API_KEY\"]:\n os.environ.setdefault(k, userdata.get(k))\nexcept Exception:\n for k in [\"WANDB_API_KEY\", \"ANTHROPIC_API_KEY\"]:\n if not os.environ.get(k):\n os.environ[k] = getpass.getpass(f\"{k}: \")" + "source": [ + "import getpass\n", + "import os\n", + "\n", + "os.environ[\"WANDB_API_KEY\"] = getpass.getpass(\"Enter your W&B API key: \")\n", + "os.environ[\"ANTHROPIC_API_KEY\"] = getpass.getpass(\"Enter your Anthropic API key: \")\n", + "\n", + "TEAM = input(\"Enter your W&B team name: \")\n", + "PROJECT = input(\"Enter your W&B project name: \")" + ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], - "source": "import weave\n\nweave.init(\n \"agent-eval-tutorial\", # your Weave project\n # Turn off implicit patching of manual Weave Ops tracing (Traces tab) \u2014 the\n # bare Anthropic SDK integration would log each LLM call as a legacy Op/Call\n # in parallel with our hand-rolled Conversation SDK spans, duplicating it.\n settings={\"implicitly_patch_integrations\": False},\n)" + "source": [ + "import weave\n", + "\n", + "weave.init(\n", + " f\"{TEAM}/{PROJECT}\",\n", + " # Hand-instrumenting a bare provider SDK: turn off implicit patching so it\n", + " # doesn't also log each call as a traced Op, duplicating our spans.\n", + " settings={\"implicitly_patch_integrations\": False},\n", + ")" + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "## 1. Build and trace the agent\n\nA minimal customer-support agent with two tools (`lookup_order`, `issue_refund`)\nand a policy: refunds are only allowed within 30 days.\n\nThe Weave-specific part is the **Conversation SDK** instrumentation inside\n`run_agent_turn`: `start_conversation` \u2192 `start_turn` \u2192 `start_llm` /\n`start_tool`, with `llm.record(...)` capturing the model call. Everything else\n(the tool backend, the Anthropic tool-use loop, message conversion) is ordinary\nagent plumbing." + "source": [ + "## 1. Build and trace the agent\n", + "\n", + "A minimal customer-support agent with two tools (`lookup_order`, `issue_refund`)\n", + "and a policy: refunds are only allowed within 30 days.\n", + "\n", + "The Weave-specific part is the **Conversation SDK** instrumentation inside\n", + "`run_agent_turn`: `start_conversation` → `start_turn` → `start_llm` /\n", + "`start_tool`, with `llm.record(...)` capturing the model call. It returns the\n", + "agent's final reply plus a plain-text transcript of the trajectory for the judge\n", + "to read; everything else (the tool backend, the Anthropic tool-use loop, message\n", + "conversion) is ordinary agent plumbing." + ] }, { "cell_type": "code", @@ -48,7 +100,80 @@ "metadata": {}, "execution_count": null, "outputs": [], - "source": "from weave.conversation import Message, ToolCallPart, Usage\n\n# --- Convert Anthropic message blocks <-> weave.Message (agent plumbing) ---\ndef _assistant_out(content):\n text = \"\".join(b.text for b in content if b.type == \"text\")\n tcs = [ToolCallPart(id=b.id, name=b.name, arguments=json.dumps(b.input))\n for b in content if b.type == \"tool_use\"]\n return Message.assistant(text, tool_calls=tcs or None)\n\ndef _to_weave_inputs(messages):\n out = [Message.system(SYSTEM)]\n for m in messages:\n role, content = m[\"role\"], m[\"content\"]\n if role == \"user\" and isinstance(content, str):\n out.append(Message.user(content))\n elif role == \"user\":\n out += [Message.tool_result(b[\"tool_use_id\"], b[\"content\"]) for b in content]\n elif role == \"assistant\":\n out.append(Message.assistant(content) if isinstance(content, str) else _assistant_out(content))\n return out\n\ndef run_agent_turn(conversation, user_message, model=AGENT_MODEL, history=None):\n # Run one user turn to completion; return (final_reply, full_message_log).\n messages = list(history or [])\n messages.append({\"role\": \"user\", \"content\": user_message})\n\n with conversation.start_turn(user_message=user_message) as turn:\n while True:\n with turn.start_llm(model=model, provider_name=\"anthropic\") as llm:\n resp = _client.messages.create(\n model=model, max_tokens=2048, system=SYSTEM, tools=TOOLS, messages=messages,\n )\n llm.record(\n input_messages=_to_weave_inputs(messages),\n output_messages=[_assistant_out(resp.content)],\n usage=Usage(input_tokens=resp.usage.input_tokens,\n output_tokens=resp.usage.output_tokens),\n response_id=resp.id, response_model=resp.model,\n finish_reasons=[resp.stop_reason],\n )\n messages.append({\"role\": \"assistant\", \"content\": resp.content})\n\n if resp.stop_reason != \"tool_use\":\n return \"\".join(b.text for b in resp.content if b.type == \"text\"), messages\n\n tool_results = []\n for block in resp.content:\n if block.type != \"tool_use\":\n continue\n with turn.start_tool(name=block.name, arguments=json.dumps(block.input),\n tool_call_id=block.id) as tool:\n tool.result = _DISPATCH[block.name](**block.input)\n tool_results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id,\n \"content\": json.dumps(tool.result)})\n messages.append({\"role\": \"user\", \"content\": tool_results})" + "source": [ + "from weave.conversation import Message, ToolCallPart, Usage\n", + "\n", + "# --- Convert Anthropic message blocks -> weave.Message (for llm.record) ---\n", + "def _assistant_out(content):\n", + " text = \"\".join(b.text for b in content if b.type == \"text\")\n", + " tcs = [ToolCallPart(id=b.id, name=b.name, arguments=json.dumps(b.input))\n", + " for b in content if b.type == \"tool_use\"]\n", + " return Message.assistant(text, tool_calls=tcs or None)\n", + "\n", + "def _to_weave_inputs(messages, system):\n", + " out = [Message.system(system)]\n", + " for m in messages:\n", + " role, content = m[\"role\"], m[\"content\"]\n", + " if role == \"user\" and isinstance(content, str):\n", + " out.append(Message.user(content))\n", + " elif role == \"user\":\n", + " out += [Message.tool_result(b[\"tool_use_id\"], b[\"content\"]) for b in content]\n", + " elif role == \"assistant\":\n", + " out.append(Message.assistant(content) if isinstance(content, str) else _assistant_out(content))\n", + " return out\n", + "\n", + "def run_agent_turn(conversation, user_message, model=AGENT_MODEL, history=None, system=SYSTEM):\n", + " \"\"\"Run one user turn to completion.\n", + "\n", + " Returns (final_reply, transcript): the agent's text reply, plus a plain-text\n", + " transcript of the whole trajectory (prior turns, tool calls, tool results,\n", + " final reply) for the LLM judge to read. Pass `system` to swap the agent's\n", + " system prompt (used to compare v1 vs v2).\n", + " \"\"\"\n", + " messages = list(history or [])\n", + " messages.append({\"role\": \"user\", \"content\": user_message})\n", + "\n", + " # Readable trajectory for the judge: prior turns first, then this turn.\n", + " lines = [f\"{m['role'].capitalize()}: {m['content']}\" for m in (history or [])]\n", + " lines.append(f\"User: {user_message}\")\n", + "\n", + " with conversation.start_turn(user_message=user_message) as turn:\n", + " while True:\n", + " with turn.start_llm(model=model, provider_name=\"anthropic\") as llm:\n", + " resp = _client.messages.create(\n", + " model=model, max_tokens=2048, system=system, tools=TOOLS, messages=messages,\n", + " )\n", + " llm.record(\n", + " input_messages=_to_weave_inputs(messages, system),\n", + " output_messages=[_assistant_out(resp.content)],\n", + " usage=Usage(input_tokens=resp.usage.input_tokens,\n", + " output_tokens=resp.usage.output_tokens),\n", + " response_id=resp.id, response_model=resp.model,\n", + " finish_reasons=[resp.stop_reason],\n", + " )\n", + " messages.append({\"role\": \"assistant\", \"content\": resp.content})\n", + " for b in resp.content:\n", + " if b.type == \"text\" and b.text:\n", + " lines.append(f\"Agent: {b.text}\")\n", + " elif b.type == \"tool_use\":\n", + " lines.append(f\"Agent called {b.name}({json.dumps(b.input)})\")\n", + "\n", + " if resp.stop_reason != \"tool_use\":\n", + " reply = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " return reply, \"\\n\".join(lines)\n", + "\n", + " tool_results = []\n", + " for block in resp.content:\n", + " if block.type != \"tool_use\":\n", + " continue\n", + " with turn.start_tool(name=block.name, arguments=json.dumps(block.input),\n", + " tool_call_id=block.id) as tool:\n", + " tool.result = _DISPATCH[block.name](**block.input)\n", + " lines.append(f\"Tool result: {json.dumps(tool.result)}\")\n", + " tool_results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id,\n", + " \"content\": json.dumps(tool.result)})\n", + " messages.append({\"role\": \"user\", \"content\": tool_results})" + ] }, { "cell_type": "code", @@ -60,63 +185,221 @@ { "cell_type": "markdown", "metadata": {}, - "source": "Open the printed Weave link and find the **Agents** view: the conversation\nappears as a turn with the model call and tool calls nested inside it.\n\n> **Prefer auto-instrumentation?** If you build your agent with an\n> agent-framework integration (e.g. the Claude Agent SDK or OpenAI Agents),\n> Weave emits these same Agents spans automatically \u2014 leave implicit patching on\n> and skip the manual `start_*` calls. Auto-patching the *bare* provider SDK,\n> by contrast, produces legacy Ops/Calls, not Agents spans." + "source": [ + "When you ran `weave.init()` above, Weave printed a **View Weave data at ...**\n", + "link. Open it and select the **Agents** tab: this conversation appears as a turn\n", + "with the model call and tool calls nested inside it." + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "## 2. Score task completion (single turn)\n\n**Task completion** = an LLM judge decides whether the agent achieved the goal,\njudged against the task's success criteria from the whole transcript \u2014 not a\ngolden-string match." + "source": [ + "## 2. Score the agent with an LLM judge\n", + "\n", + "The score in this example is **task completion**: did the agent achieve the goal?\n", + "A judge model reads the transcript and decides against the task's success\n", + "criteria — rewarding the correct OUTCOME, not a polite-sounding reply. It's one\n", + "example score; you choose what to measure and what to name it." + ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], - "source": "JUDGE_MODEL = \"claude-opus-4-8\" # a different, stronger model than the agent\n\ndef _extract_json(text):\n text = text.strip()\n if text.startswith(\"```\"):\n text = text.split(\"```\", 2)[1]\n text = text[4:] if text.startswith(\"json\") else text\n start, end = text.find(\"{\"), text.rfind(\"}\")\n return json.loads(text[start:end + 1])\n\ndef judge_task_completion(task, transcript):\n prompt = (\n \"You are evaluating whether a customer-support agent completed the \"\n \"user's task. Judge the whole transcript against the success criteria \"\n \"\u2014 reward the correct OUTCOME, not just a polite reply.\\n\\n\"\n f\"USER REQUEST: {task['user_request']}\\n\"\n f\"SUCCESS CRITERIA: {task['success_criteria']}\\n\\n\"\n f\"AGENT TRANSCRIPT:\\n{transcript}\\n\\n\"\n 'Respond with ONLY a JSON object: {\"passed\": , '\n '\"reason\": \"\"}. Set passed=true only if the success '\n \"criteria are met.\"\n )\n resp = _client.messages.create(model=JUDGE_MODEL, max_tokens=1024,\n messages=[{\"role\": \"user\", \"content\": prompt}])\n text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n return _extract_json(text)\n\ndef render_transcript(messages, final_reply):\n lines = []\n for m in messages:\n role, content = m[\"role\"], m[\"content\"]\n if isinstance(content, str):\n lines.append(f\"{role.capitalize()}: {content}\"); continue\n for b in content:\n if isinstance(b, dict):\n lines.append(f\"Tool result: {b['content']}\")\n elif b.type == \"text\" and b.text:\n lines.append(f\"Agent: {b.text}\")\n elif b.type == \"tool_use\":\n lines.append(f\"Agent called {b.name}({json.dumps(b.input)})\")\n lines.append(f\"Final reply: {final_reply}\")\n return \"\\n\".join(lines)" + "source": [ + "JUDGE_MODEL = \"claude-opus-4-8\" # A different, stronger model than the agent.\n", + "\n", + "def _extract_json(text):\n", + " # Judge models sometimes wrap JSON in ``` fences; pull out the object.\n", + " text = text.strip()\n", + " if text.startswith(\"```\"):\n", + " text = text.split(\"```\", 2)[1]\n", + " text = text[4:] if text.startswith(\"json\") else text\n", + " start, end = text.find(\"{\"), text.rfind(\"}\")\n", + " return json.loads(text[start:end + 1])\n", + "\n", + "def judge_task_completion(task, transcript):\n", + " \"\"\"LLM judge over the transcript. Returns {'passed': bool, 'reason': str}.\"\"\"\n", + " prompt = (\n", + " \"You are evaluating whether a customer-support agent completed the \"\n", + " \"user's task. Judge the whole transcript against the success criteria \"\n", + " \"— reward the correct OUTCOME, not just a polite reply.\\n\\n\"\n", + " f\"USER REQUEST: {task['user_request']}\\n\"\n", + " f\"SUCCESS CRITERIA: {task['success_criteria']}\\n\\n\"\n", + " f\"AGENT TRANSCRIPT:\\n{transcript}\\n\\n\"\n", + " 'Respond with ONLY a JSON object: {\"passed\": , '\n", + " '\"reason\": \"\"}. Set passed=true only if the success '\n", + " \"criteria are met.\"\n", + " )\n", + " resp = _client.messages.create(model=JUDGE_MODEL, max_tokens=1024,\n", + " messages=[{\"role\": \"user\", \"content\": prompt}])\n", + " text = \"\".join(b.text for b in resp.content if b.type == \"text\")\n", + " return _extract_json(text)" + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "Now drive the evaluation with `EvaluationLogger`. The key move: run the agent\n**inside** `log_prediction(...)`, so each trial's agent transcript links to the\nevaluation result." + "source": [ + "Now drive the evaluation with `EvaluationLogger`. The key move: run the agent\n", + "**inside** `log_prediction(...)`, so each run's agent transcript links to the\n", + "evaluation result." + ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], - "source": "TASKS = [\n {\"task_id\": \"refund-eligible\",\n \"user_request\": \"I'd like a refund for order A1001, please.\",\n \"success_criteria\": \"Agent looks up the order and issues the refund (within the 30-day window).\"},\n {\"task_id\": \"refund-too-late\",\n \"user_request\": \"Please refund my order A1002.\",\n \"success_criteria\": \"Agent looks up the order and politely declines (outside the 30-day window); must NOT issue a refund.\"},\n {\"task_id\": \"unknown-order\",\n \"user_request\": \"I want a refund for order Z9999.\",\n \"success_criteria\": \"Agent reports the order cannot be found and does not issue a refund.\"},\n]\n\nev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v1\", dataset=\"support-refund-tasks\")\nfor task in TASKS:\n with ev.log_prediction(inputs=task) as pred:\n with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, messages = run_agent_turn(conv, task[\"user_request\"])\n verdict = judge_task_completion(task, render_transcript(messages, reply))\n pred.output = reply\n pred.log_score(\"task_completion\", verdict)\n print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\nev.log_summary()\nprint(\"Evaluation:\", ev.ui_url)" + "source": [ + "TASKS = [\n", + " {\"task_id\": \"refund-eligible\",\n", + " \"user_request\": \"I'd like a refund for order A1001, please.\",\n", + " \"success_criteria\": \"Agent looks up the order and issues the refund (within the 30-day window).\"},\n", + " {\"task_id\": \"refund-too-late\",\n", + " \"user_request\": \"Please refund my order A1002.\",\n", + " \"success_criteria\": \"Agent looks up the order and politely declines (outside the 30-day window); must NOT issue a refund.\"},\n", + " {\"task_id\": \"unknown-order\",\n", + " \"user_request\": \"I want a refund for order Z9999.\",\n", + " \"success_criteria\": \"Agent reports the order cannot be found and does not issue a refund.\"},\n", + "]\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v1\", dataset=\"support-refund-tasks\")\n", + "for task in TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"user_request\"])\n", + " verdict = judge_task_completion(task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "Open the evaluation link: each row is a task, with its task-completion score,\noutput, latency, and cost \u2014 and a link to the full agent trace for that trial.\n\n**A scorecard, not one number.** Task completion is one signal. Add more with\nextra `pred.log_score(...)` calls in the same block \u2014 e.g. `tool_call_correct`\nor `instruction_following`." + "source": [ + "Open the printed evaluation link, then select the **Evals** tab. Open your run's\n", + "row to reveal its details panel, which has two tabs:\n", + "\n", + "- **Evaluation** — the run's scores and metrics. Click **View spans** to jump to\n", + " the **Agents** page showing the traced spans linked to this evaluation.\n", + "- **Call** — a per-task breakdown with a `passed` column showing the judge's\n", + " verdict for each task." + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "## 3. Organize and compare evaluations\n\nRun the same evaluation for a second agent version by changing the `model` label\n(e.g. `EvaluationLogger(model=\"v2\", ...)` after editing the agent's prompt or\nswitching its model). Weave lays the two runs **side by side** in the comparison\nview, so you can read task-completion rate, tool-call correctness, latency, and\ncost for v1 vs v2 \u2014 the baseline-relative question (\"did the candidate do as well\nas the baseline?\"). Every row still links to its agent transcript, so a\nregression is one click from the failing conversation." - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": "## 4. Score a multi-turn conversation\n\nTo evaluate a turn *in context*, load a fixed conversation history, append one\nnew user turn, and score how the agent handles it \u2014 i.e. *\"given the\nconversation state so far, does the agent handle the next turn well?\"*\n\nBelow, the order id is only mentioned in the prior turns; a good agent should use\nthat context instead of asking again." + "source": [ + "## 3. Organize and compare evaluations\n", + "\n", + "You improve an agent by changing its **application** — the system prompt, the\n", + "tools, the control flow, or the underlying LLM — and checking whether the change\n", + "helped. To compare two versions, change the agent, then log a second evaluation\n", + "under the same `name` with a new version label.\n", + "\n", + "`model` is Weave's term for the system under evaluation — your whole agent\n", + "(prompt, tools, code, and LLM), not just the LLM. Reusing the eval `name` and\n", + "bumping the label lines the runs up as versions. (To record *what* changed, you\n", + "can pass a dict, e.g. `model={\"name\": \"support-agent\", \"version\": \"v2\",\n", + "\"system_prompt\": SYSTEM_V2}`, and Weave stores those as versioned attributes.)\n", + "\n", + "Below, we re-run the same tasks and judge with a revised system prompt, logged\n", + "under the label `v2`:" + ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], - "source": "MULTI_TURN_TASKS = [\n {\"task_id\": \"context-refund\",\n \"conversation_history\": [\n {\"role\": \"user\", \"content\": \"Hi, can you check the status of my order A1001?\"},\n {\"role\": \"assistant\", \"content\": \"Your order A1001 (Wireless headphones) was delivered 5 days ago.\"},\n ],\n \"next_user_message\": \"Thanks. Actually, I'd like to return it for a refund.\",\n \"success_criteria\": \"Using the prior context that the order is A1001, the agent issues the refund (within 30 days) WITHOUT asking the user to repeat the order id.\"},\n {\"task_id\": \"context-decline\",\n \"conversation_history\": [\n {\"role\": \"user\", \"content\": \"Can you look up order A1002 for me?\"},\n {\"role\": \"assistant\", \"content\": \"Order A1002 (Standing desk) was delivered 60 days ago.\"},\n ],\n \"next_user_message\": \"Okay, I'd like a refund for it.\",\n \"success_criteria\": \"Using the prior context that A1002 was delivered 60 days ago, the agent politely declines as outside the 30-day window and does NOT issue a refund.\"},\n]\n\nev = weave.EvaluationLogger(name=\"support-agent-multiturn-eval\", model=\"v1\", dataset=\"support-multiturn-tasks\")\nfor task in MULTI_TURN_TASKS:\n with ev.log_prediction(inputs=task) as pred:\n with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n reply, messages = run_agent_turn(conv, task[\"next_user_message\"],\n history=task[\"conversation_history\"])\n judge_task = {\"user_request\": task[\"next_user_message\"], \"success_criteria\": task[\"success_criteria\"]}\n verdict = judge_task_completion(judge_task, render_transcript(messages, reply))\n pred.output = reply\n pred.log_score(\"task_completion\", verdict)\n print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\nev.log_summary()\nprint(\"Evaluation:\", ev.ui_url)" + "source": [ + "# v2: a stricter system prompt. Same tasks, loop, and judge as v1.\n", + "SYSTEM_V2 = SYSTEM + (\" Keep replies to one or two sentences and state the refund \"\n", + " \"decision explicitly.\")\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-eval\", model=\"v2\", dataset=\"support-refund-tasks\")\n", + "for task in TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"user_request\"], system=SYSTEM_V2)\n", + " verdict = judge_task_completion(task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "> **Scope.** This scores the *next* turn against a fixed history \u2014 the practical\n> offline approach. Measuring a full multi-turn task end-to-end (where the agent\n> drives the whole session) requires live A/B testing in production and is out of\n> scope here." + "source": [ + "## 4. Score a multi-turn conversation\n", + "\n", + "Real conversations span several turns, and a capable agent carries context\n", + "forward — it shouldn't re-ask for an order id the user already gave. To test that\n", + "offline, seed the agent with a fixed conversation history, send the next user\n", + "message, and score how it handles that turn in context.\n", + "\n", + "Each dataset row is one such scenario: the prior turns plus the next message the\n", + "agent must answer. Below, the order id appears only in the history, so a good\n", + "agent reuses it instead of asking again." + ] + }, + { + "cell_type": "code", + "metadata": {}, + "execution_count": null, + "outputs": [], + "source": [ + "MULTI_TURN_TASKS = [\n", + " {\"task_id\": \"context-refund\",\n", + " \"conversation_history\": [\n", + " {\"role\": \"user\", \"content\": \"Hi, can you check the status of my order A1001?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Your order A1001 (Wireless headphones) was delivered 5 days ago.\"},\n", + " ],\n", + " \"next_user_message\": \"Thanks. Actually, I'd like to return it for a refund.\",\n", + " \"success_criteria\": \"Using the prior context that the order is A1001, the agent issues the refund (within 30 days) WITHOUT asking the user to repeat the order id.\"},\n", + " {\"task_id\": \"context-decline\",\n", + " \"conversation_history\": [\n", + " {\"role\": \"user\", \"content\": \"Can you look up order A1002 for me?\"},\n", + " {\"role\": \"assistant\", \"content\": \"Order A1002 (Standing desk) was delivered 60 days ago.\"},\n", + " ],\n", + " \"next_user_message\": \"Okay, I'd like a refund for it.\",\n", + " \"success_criteria\": \"Using the prior context that A1002 was delivered 60 days ago, the agent politely declines as outside the 30-day window and does NOT issue a refund.\"},\n", + "]\n", + "\n", + "ev = weave.EvaluationLogger(name=\"support-agent-multiturn-eval\", model=\"v1\", dataset=\"support-multiturn-tasks\")\n", + "for task in MULTI_TURN_TASKS:\n", + " with ev.log_prediction(inputs=task) as pred:\n", + " with start_conversation(agent_name=\"support-agent\", conversation_id=uuid.uuid4().hex) as conv:\n", + " reply, transcript = run_agent_turn(conv, task[\"next_user_message\"],\n", + " history=task[\"conversation_history\"])\n", + " judge_task = {\"user_request\": task[\"next_user_message\"], \"success_criteria\": task[\"success_criteria\"]}\n", + " verdict = judge_task_completion(judge_task, transcript)\n", + " pred.output = reply\n", + " pred.log_score(\"task_completion\", verdict)\n", + " print(f\"[{task['task_id']}] passed={verdict['passed']}: {verdict['reason']}\")\n", + "ev.log_summary()\n", + "print(\"Evaluation:\", ev.ui_url)" + ] }, { "cell_type": "markdown", "metadata": {}, - "source": "## 5. Extend your scorers\n\nReal agents need a scorecard covering both:\n- **Functional:** tool-call correctness, instruction-following, recovery from tool errors.\n- **Non-functional:** safety / refusal behavior, latency, cost, hallucinated-tool use.\n\nAdd each as another `pred.log_score(name, value)` inside the prediction block.\n\n## Recap\nYou traced an agent as a conversation, scored task completion for single and\nmulti-turn interactions, and compared versions \u2014 all linked back to the agent\ntranscripts. For **online** evaluation (automatically scoring every agent trace\nin production), see Weave's production monitoring docs." + "source": [ + "## Recap\n", + "\n", + "You traced an agent as a conversation, scored task completion for single- and\n", + "multi-turn interactions, and compared two versions — all linked back to the agent\n", + "transcripts." + ] } ], "metadata": { @@ -133,4 +416,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +}