feat(guardrails): send attached file references with llm-as-judge evaluation - #1082
Draft
apetraru-uipath wants to merge 2 commits into
Draft
feat(guardrails): send attached file references with llm-as-judge evaluation#1082apetraru-uipath wants to merge 2 commits into
apetraru-uipath wants to merge 2 commits into
Conversation
…luation Agent-scope and LLM-scope LLM-as-Judge guardrails on low-code agents now forward the run's job attachments to the guardrails backend, so the judge can evaluate what is *in* a file rather than the serialized metadata that currently lands in the prompt. No new state plumbing was needed: state.inner_state.job_attachments is already populated at every agent- and LLM-scope guardrail node, and the init subgraph wires START -> guarded-init -> guardrail, so the registry is committed before the Agent-scope PRE guardrail runs. New agent/guardrails/attachment_refs.py projects that registry into resolved references. It never raises: the low-code guardrail node re-raises everything it sees, so letting a transient Orchestrator failure escape would kill a production run over an unscanned file. Gated on validator_type == llm_as_judge and the UIPATH_FEATURE_GuardrailAttachmentsEnabled flag, because resolving a SAS url costs an Orchestrator round-trip. Two pre-existing issues in guardrail_nodes.py fixed along the way: - The payload generator ran twice per evaluation — once for observability metadata, once inside _evaluate_builtin_guardrail. It now runs once and the text is passed down. A regression test asserts the single invocation. - evaluate_guardrail is synchronous and was called directly from an async node, blocking the event loop for the whole round-trip. Now offloaded with asyncio.to_thread, which matters more once the backend fetches files inside that call. Only attachment identity (id, file name, mime type) reaches observability metadata — the resolved url is a SAS credential, asserted by a test. Six test mocks of evaluate_guardrail needed the new keyword-only argument. Full suite green; tests/agent/guardrails 299 passed (was 279). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Widens the supported mime set to the binary types the backend can send to a vision-capable judge model as content parts. The runtime still only forwards references — this set exists so an Orchestrator round-trip isn't spent resolving a file the backend would skip anyway. tests/agent/guardrails: 302 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


What
Agent-scope and LLM-scope LLM-as-Judge guardrails on low-code agents now forward the run's job attachments to the guardrails backend, so the judge can evaluate what is in a file rather than the serialized metadata that currently lands in the prompt.
No new state plumbing was needed:
state.inner_state.job_attachmentsis already populated at every agent- and LLM-scope guardrail node, and the init subgraph wiresSTART → guarded-init → guardrail, so the registry is committed before the Agent-scope PRE guardrail runs.Notes for reviewers
attachment_refs.pynever raises. The low-code guardrail node re-raises everything it sees (guardrail_nodes.py), which terminates the agent run — so letting a transient Orchestrator failure escape would kill a production run over an unscanned file. Every failure path returns[]and the guardrail evaluates the text payload alone. Covered by tests including a partial-failure case.Gated on
validator_type == "llm_as_judge"andUIPATH_FEATURE_GuardrailAttachmentsEnabled, because resolving a SAS url costs an Orchestrator round-trip and no other validator can use the result yet.Two pre-existing bugs fixed along the way, both now covered by regression tests:
_evaluate_builtin_guardrail. Harmless when it was pure string formatting; with attachment resolution it would have doubled every Orchestrator call.evaluate_guardrailis synchronous and was called directly from anasyncnode, blocking the event loop for the whole round-trip. Now offloaded withasyncio.to_thread, which matters more once the backend fetches files inside that call.Only attachment identity (id, file name, mime type) reaches observability metadata — the resolved url is a SAS credential, asserted by a test.
Six existing test mocks of
evaluate_guardrailneeded the new keyword-only argument.Out of scope by design: the coded-agent middleware and decorator flavors. This is the low-code (
agent.json) path only.Test plan
tests/agent/guardrails/— 302 passed (baseline 279).tests/cli/55 passed,tests/guardrails/253 passed, full suite exit 0. ruff, ruff format and mypy clean.Depends on: UiPath/uipath-python#1895 · pairs with UiPath/Agents#6256
🤖 Generated with Claude Code