Skip to content

feat(guardrails): send attached file references with llm-as-judge evaluation - #1082

Draft
apetraru-uipath wants to merge 2 commits into
mainfrom
feat/guardrail-judge-file-support
Draft

feat(guardrails): send attached file references with llm-as-judge evaluation#1082
apetraru-uipath wants to merge 2 commits into
mainfrom
feat/guardrail-judge-file-support

Conversation

@apetraru-uipath

Copy link
Copy Markdown
Contributor

Draft until uipath-platform 0.2.29 publishes. This branch imports GuardrailAttachment from it. Before marking ready: raise the uipath-platform floor in pyproject.toml to >=0.2.29, run uv sync, and commit uv.lock. Everything else is complete and green.

What

Agent-scope and LLM-scope LLM-as-Judge guardrails on low-code agents now forward the run's job attachments to the guardrails backend, so the judge can evaluate what is in a file rather than the serialized metadata that currently lands in the prompt.

No new state plumbing was needed: state.inner_state.job_attachments is already populated at every agent- and LLM-scope guardrail node, and the init subgraph wires START → guarded-init → guardrail, so the registry is committed before the Agent-scope PRE guardrail runs.

Notes for reviewers

attachment_refs.py never raises. The low-code guardrail node re-raises everything it sees (guardrail_nodes.py), which terminates the agent run — so letting a transient Orchestrator failure escape would kill a production run over an unscanned file. Every failure path returns [] and the guardrail evaluates the text payload alone. Covered by tests including a partial-failure case.

Gated on validator_type == "llm_as_judge" and UIPATH_FEATURE_GuardrailAttachmentsEnabled, because resolving a SAS url costs an Orchestrator round-trip and no other validator can use the result yet.

Two pre-existing bugs fixed along the way, both now covered by regression tests:

  1. The payload generator ran twice per evaluation — once for observability metadata, once inside _evaluate_builtin_guardrail. Harmless when it was pure string formatting; with attachment resolution it would have doubled every Orchestrator call.
  2. evaluate_guardrail is synchronous and was called directly from an async node, blocking the event loop for the whole round-trip. Now offloaded with asyncio.to_thread, which matters more once the backend fetches files inside that call.

Only attachment identity (id, file name, mime type) reaches observability metadata — the resolved url is a SAS credential, asserted by a test.

Six existing test mocks of evaluate_guardrail needed the new keyword-only argument.

Out of scope by design: the coded-agent middleware and decorator flavors. This is the low-code (agent.json) path only.

Test plan

tests/agent/guardrails/302 passed (baseline 279). tests/cli/ 55 passed, tests/guardrails/ 253 passed, full suite exit 0. ruff, ruff format and mypy clean.

Depends on: UiPath/uipath-python#1895 · pairs with UiPath/Agents#6256

🤖 Generated with Claude Code

apetraru-uipath and others added 2 commits September 13, 2026 01:31
…luation

Agent-scope and LLM-scope LLM-as-Judge guardrails on low-code agents now forward
the run's job attachments to the guardrails backend, so the judge can evaluate
what is *in* a file rather than the serialized metadata that currently lands in
the prompt.

No new state plumbing was needed: state.inner_state.job_attachments is already
populated at every agent- and LLM-scope guardrail node, and the init subgraph
wires START -> guarded-init -> guardrail, so the registry is committed before the
Agent-scope PRE guardrail runs.

New agent/guardrails/attachment_refs.py projects that registry into resolved
references. It never raises: the low-code guardrail node re-raises everything it
sees, so letting a transient Orchestrator failure escape would kill a production
run over an unscanned file. Gated on validator_type == llm_as_judge and the
UIPATH_FEATURE_GuardrailAttachmentsEnabled flag, because resolving a SAS url
costs an Orchestrator round-trip.

Two pre-existing issues in guardrail_nodes.py fixed along the way:

- The payload generator ran twice per evaluation — once for observability
  metadata, once inside _evaluate_builtin_guardrail. It now runs once and the
  text is passed down. A regression test asserts the single invocation.
- evaluate_guardrail is synchronous and was called directly from an async node,
  blocking the event loop for the whole round-trip. Now offloaded with
  asyncio.to_thread, which matters more once the backend fetches files inside
  that call.

Only attachment identity (id, file name, mime type) reaches observability
metadata — the resolved url is a SAS credential, asserted by a test.

Six test mocks of evaluate_guardrail needed the new keyword-only argument.

Full suite green; tests/agent/guardrails 299 passed (was 279).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Widens the supported mime set to the binary types the backend can send to a
vision-capable judge model as content parts. The runtime still only forwards
references — this set exists so an Orchestrator round-trip isn't spent resolving
a file the backend would skip anyway.

tests/agent/guardrails: 302 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 90%)

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant