A scenario run is a real agent interaction — the runner opens a conversation, sends the prompt through the same path a user's message takes, and the model answers with tool calls. The dashboard reports the outcome of that interaction (score, fault, tool names, token counts) but not the interaction itself, so the question a reviewer actually asks — what did the agent see, what did the tool return, why did it answer that way — cannot be answered without leaving the page.
The Interactions view already renders exactly this for an agent run: the message stream, each tool call with its arguments and result, timings. A scenario result should expand into the same view.
What the row shows today
Expanding a scenario in the matrix gives the final answer, the tool names with a ✗ when one errored, the per-criterion scores, and the fault text. From a real run:
visibility_1 expects find_records or record_visibility_status
gpt-4o-mini failed · 0.67 7.16s · 958 tokens
TOOLS find_records record_visibility_status ✗ ×4
[!] tool error — Tool record_visibility_status returned an error…
I encountered an internal error while trying to check the visibility status…
expected tools 1.00 tools succeeded 0.00 response present 1.00
Useful, but it cannot say what find_records was called with, what it returned, what the agent did with that result, or why it retried record_visibility_status four times. Those four retries are the interesting part and they are invisible.
Proposal
Render the run's transcript inside the expanded scenario row, reusing the Interactions components rather than a second renderer:
- the user turn (the scenario prompt as the agent received it, including composed instructions)
- each assistant turn
- each tool call: name, arguments, result or error, duration — the repeats shown as repeats, not collapsed to a count
- the runtime characteristics already summarized in the header (latency, tokens, cost), attributed per turn
Data
Mostly persisted already. EvaluationScenarioResult carries tool_calls (name, arguments, error, detail), output, duration_ms, input_tokens, output_tokens, cost and agent_run_id.
The gap worth checking first: whether the full message stream survives the run. A host adapter may destroy its conversation afterwards (Sparkle's runner destroys the replay chat unless keep_chats is set), so the transcript may need to be captured onto the result — or the conversation retained — rather than read back from the agent's context.
Why it matters
Today a failing scenario tells a reviewer that a tool errored. The transcript tells them whether the agent called it with the wrong arguments, whether it recovered, and whether the answer was reasonable given what came back — which is the difference between "fix the tool" and "fix the prompt". That judgement is the point of the evaluation loop, and it currently happens outside the product.
Related
A scenario run is a real agent interaction — the runner opens a conversation, sends the prompt through the same path a user's message takes, and the model answers with tool calls. The dashboard reports the outcome of that interaction (score, fault, tool names, token counts) but not the interaction itself, so the question a reviewer actually asks — what did the agent see, what did the tool return, why did it answer that way — cannot be answered without leaving the page.
The Interactions view already renders exactly this for an agent run: the message stream, each tool call with its arguments and result, timings. A scenario result should expand into the same view.
What the row shows today
Expanding a scenario in the matrix gives the final answer, the tool names with a ✗ when one errored, the per-criterion scores, and the fault text. From a real run:
Useful, but it cannot say what
find_recordswas called with, what it returned, what the agent did with that result, or why it retriedrecord_visibility_statusfour times. Those four retries are the interesting part and they are invisible.Proposal
Render the run's transcript inside the expanded scenario row, reusing the Interactions components rather than a second renderer:
Data
Mostly persisted already.
EvaluationScenarioResultcarriestool_calls(name, arguments, error, detail),output,duration_ms,input_tokens,output_tokens,costandagent_run_id.The gap worth checking first: whether the full message stream survives the run. A host adapter may destroy its conversation afterwards (Sparkle's runner destroys the replay chat unless
keep_chatsis set), so the transcript may need to be captured onto the result — or the conversation retained — rather than read back from the agent's context.Why it matters
Today a failing scenario tells a reviewer that a tool errored. The transcript tells them whether the agent called it with the wrong arguments, whether it recovered, and whether the answer was reasonable given what came back — which is the difference between "fix the tool" and "fix the prompt". That judgement is the point of the evaluation loop, and it currently happens outside the product.
Related