fix(evals): include agent output in AgentRunHistory [AE-2147]

[rakesh-uipath]

Fixes AE-2147 — reported by James Dickson in #help-agent-builder, filed by Maria Vimer.

The bug

{{AgentRunHistory}} reaches the trajectory judge as a tool-call transcript with the agent's own output missing, so the judge grades an incomplete trace.

It is real, and it is one predicate:

https://github.com/UiPath/uipath-python/blob/30f6bec2/packages/uipath/src/uipath/eval/_helpers/evaluators_helpers.py#L653

for span in workload_trace:
    if span.attributes and (tool_name := span.attributes.get(TOOL_NAME_ATTR)):

A span without tool.name is skipped, and the docstring said so out loud: "Only processes tool spans". The workload's own answer lives on WorkloadExecution.workload_output (populated in runtime.py:1143) and never entered the string.

There is no second channel for it. On BaseLLMTrajectoryEvaluator the actual-output placeholder is the history:

actual_output_placeholder: str = "{{AgentRunHistory}}"   # llm_judge_trajectory_evaluator.py:63
...
return trace_to_str(workload_execution.workload_trace)   # line 83 — output not passed

So the judge never saw the output at all — while its own system prompt promises it "a trace/history of the agent's actions and outputs".

Reproduced against James's run (one Web_Search tool span, agent answered correctly). Before:

[2026-08-26 18:46:40] LLM Response:
  Agent Selected 1 Tool(s):

  Tool: Web_Search
  Arguments: {}

[2026-08-26 18:46:47] Tool Call Response - Web_Search:
{}

Byte-for-byte what his screenshot shows. The judge scored it 10% with the justification "It never produced a user-facing answer identifying Argentina as the most recent World Cup winner" — while the Actual Output panel on the same screen read {"search_results_answer": "Argentina won the most recent FIFA World Cup (Qatar 2022)..."}. The agent was right and got marked down for the serialiser's omission.

The fix

trace_to_str takes the workload output and appends it as a final block; both trajectory evaluators pass it.

[2026-08-26 18:46:47] Tool Call Response - Web_Search:
{}

Agent Output:
{"search_results_answer": "Argentina won the most recent FIFA World Cup (Qatar 2022), ..."}

Fixed in LLMJudgeTrajectoryEvaluator / LLMJudgeTrajectorySimulationEvaluator and in LegacyTrajectoryEvaluator, which had the same defect through the same helper.

Deliberately not changed

Intermediate assistant messages are still not rendered. LLM spans carry the full system prompt, and test_legacy_trajectory_prompt_uses_compact_tool_history exists precisely to keep a 10k-char prompt out of the judge's context. Surfacing intermediate reasoning needs a bounded projection of the LLM spans, not a filter removal — worth doing, separate change. James's other asks in that thread (documenting what each {{...}} binds to in the evaluator UI, eval-driven judge authoring) are product work, not this.

workload_output=None keeps the old behaviour, so no existing caller changes shape.

Tests

packages/uipath/tests/evaluators/test_trajectory_agent_output.py — reproduces the AE-2147 run through both evaluators and asserts the answer reaches the prompt. Both fail on main:

E  AssertionError: assert 'Argentina won the most recent FIFA World Cup (Qatar 2022).' in
   '...Tool Call Response - Web_Search:\n{}\n'

Plus five cases on trace_to_str itself in test_evaluator_helpers.py (ordering, str vs dict output, omitted output, empty output).

Verification

Ran locally, all from packages/uipath:

• uv run pytest — green, exit 0
• uv run ruff check . — All checks passed
• uv run ruff format --check . — 335 files already formatted
• uv run mypy --config-file pyproject.toml . — no issues in 335 source files
• uv run python scripts/lint_httpx_client.py — clean

Version bumped 2.14.14 → 2.14.15; 2.14.14 is already on PyPI so check-version-availability would have failed otherwise.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The legacy evaluator still omits supplied output for runs with no trace spans.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes {{AgentRunHistory}} so trajectory evaluators include the agent’s final output.

Changes:

• Appends workload output to serialized traces.
• Updates modern and legacy trajectory evaluators.
• Adds regression tests.
• Bumps version to 2.14.15.

File summaries

File	Summary
packages/uipath/uv.lock	Updates locked package metadata.
packages/uipath/tests/evaluators/test_trajectory_agent_output.py	Adds evaluator regression coverage.
packages/uipath/tests/evaluators/test_evaluator_helpers.py	Tests trace output serialization.
packages/uipath/src/uipath/eval/evaluators/llm_judge_trajectory_evaluator.py	Passes agent output into judge history.
packages/uipath/src/uipath/eval/evaluators/legacy_trajectory_evaluator.py	Passes output into legacy history, but empty-trace runs still need correction.
packages/uipath/src/uipath/eval/_helpers/evaluators_helpers.py	Serializes and appends agent output.
packages/uipath/pyproject.toml	Bumps the package version.

Review details

• Files reviewed: 6/7 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud