feat(trg): let the llm judge target transcripts, files, and created-files lists by yordis · Pull Request #154 · TrogonStack/rusty-monorepo

yordis · 2026-09-13T17:18:59Z

The judge could only be pointed at one thing, so an assertion about what the skill wrote, or about which files it created, had no way to be graded and had to be left to a human reading the bundle by hand.
Widening what the judge can be shown also widens what can be sent to it, so the payload budget is hardened in the same change rather than left to be discovered by a run that blows past its declared slots.

Stacked on #152.

cursor · 2026-09-13T17:19:04Z

PR Summary

Medium Risk
Changes grading and judge API behavior (payload shape, truncation, mechanical shortcut rules), which can shift pass/fail outcomes for existing suites that use LLM graders or large outputs.

Overview
LLM graders and prose assertions can now aim the judge at specific run artifacts, not only the legacy final_text + top-level outputs bundle.

llm graders gain an optional target (final_text, transcript, any_output, a file path, or new created_files). created_files is fed from the run’s existing output artifact index—no second directory walk. TargetDeclaration tracks whether target was omitted vs written explicitly; an explicit target (even final_text) disables the mechanical shortcut that would otherwise grade criterion text without calling the judge.

Judge payloads are rebuilt around what was targeted: legacy structured JSON for default/any_output (with nested outputs only for any_output), self-describing {target, content} for transcript/file/created_files, missing_reason when a file is absent, and vision attachments for common image extensions via extended JudgeRequest / provider content parts. Mechanical graders (regex, contains, JSON/schema) fail clearly on image targets instead of misreading bytes.

Payload sizing moves from per-artifact 8KB excerpts to one shared 8,000-byte budget with fair shares, rollover, truncation markers, artifacts_omitted, and transcript-aware tail truncation (whole NDJSON lines, optional first+last with middle omission). Docs and evals.json schema updated to match.

^{Reviewed by Cursor Bugbot for commit d4fe22a. Bugbot is set up for automated code reviews on this repo. Configure here.}

coderabbitai · 2026-09-13T17:19:05Z

Warning

Review limit reached

Next included review available in 48 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 31d604f6-1f6a-4e3c-962f-805e97f37e53

📥 Commits

Reviewing files that changed from the base of the PR and between b7bfcce and d4fe22a.

📒 Files selected for processing (5)

crates/trg/docs/reference/ai-skills-eval.md
crates/trg/schemas/evals.json.schema.json
crates/trg/src/agentskills/graders.rs
crates/trg/src/agentskills/grading.rs
crates/trg/src/agentskills/judge.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

…iles lists The judge could only ever see the implicit final-text-plus-outputs blob, so an assertion about the transcript or a specific file had no way to tell the judge what to look at. A per-artifact excerpt cap with no total ceiling also meant payload size grew unbounded with artifact count and could silently drop content without saying so. created_files reuses the artifact index already persisted on the run record instead of walking the output directory a second time. Transcript truncation keeps the tail rather than the head, since the run's outcome and its most recent tool calls sit at the end of the transcript. The budget guards against being asked for more placements than the slot count it was given, rather than panicking on a division by zero. Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

An explicit target on an llm grader stated what to look at; the mechanical shortcut and any_output's file walk both ignored that statement, so a declared target could be silently misgraded and any_output could miss nested files a mechanical grader would have found. created_files also stripped a raw string prefix from artifact paths, which a separator mismatch could leave unstripped. The judge could not be pointed at a picture the run produced, even though both providers it speaks to accept image content parts; a file target resolving to one always failed as unreadable text instead. A flat byte cap on the transcript payload could cut through the exact message a criterion was about and gave no sign that anything was missing. Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

The judge is shown the run's opening message so it can tell how the run began, and the byte it was charged for a newline that is never written cost it that message on transcripts that were within budget all along. Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

…ires Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit d4fe22a. Configure here.}

cursor · 2026-09-13T22:26:42Z

+        GraderOutcome::Deferred { criterion, target } => {
            let mut result = match options.grader {
-                GraderMode::Llm | GraderMode::Auto => grade_with_llm(&criterion, ctx, session)?,
+                GraderMode::Llm | GraderMode::Auto => grade_with_llm(&criterion, &target, declarative, ctx, session)?,


LLM results omit declared target

Medium Severity

describe now prefixes a non-default target onto an llm grader, but the deferred path publishes grade_with_llm's result whose assertion is only the raw criterion. Two graders that share a criterion and differ only by target therefore land in results under the same key, and the report no longer shows which target the judge actually saw.

Additional Locations (1)

crates/trg/src/agentskills/graders.rs#L290-L294

^{Reviewed by Cursor Bugbot for commit d4fe22a. Configure here.}

cursor Bot reviewed Sep 13, 2026

View reviewed changes

Comment thread crates/trg/src/agentskills/grading.rs

Comment thread crates/trg/src/agentskills/grading.rs

Comment thread crates/trg/src/agentskills/grading.rs

yordis force-pushed the yordis/fix-eval-grader-names branch from e498943 to b2b7e89 Compare September 13, 2026 17:38

yordis force-pushed the yordis/feat-eval-judge-targets branch from a70defc to 1c4da1c Compare September 13, 2026 17:43

yordis mentioned this pull request Sep 13, 2026

chore(ci): gate every pull request, not only the ones aimed at main #156

Merged

yordis force-pushed the yordis/fix-eval-grader-names branch 2 times, most recently from ed1749b to 84170e2 Compare September 13, 2026 18:45

yordis force-pushed the yordis/feat-eval-judge-targets branch from b25dc79 to 9c16481 Compare September 13, 2026 18:46

cursor Bot reviewed Sep 13, 2026

View reviewed changes

Comment thread crates/trg/src/agentskills/grading.rs

yordis force-pushed the yordis/fix-eval-grader-names branch from 84170e2 to bd2c2a4 Compare September 13, 2026 22:20

Base automatically changed from yordis/fix-eval-grader-names to main September 13, 2026 22:20

yordis added 3 commits September 13, 2026 18:21

yordis force-pushed the yordis/feat-eval-judge-targets branch from 16cd6dc to 27a2e9e Compare September 13, 2026 22:21

fix(trg): a test helper must construct the record the schema now requ…

d4fe22a

…ires Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

yordis merged commit cf968fc into main Sep 13, 2026
11 checks passed

yordis deleted the yordis/feat-eval-judge-targets branch September 13, 2026 22:22

sht-bot mentioned this pull request Sep 13, 2026

chore(main): release trg 0.10.0 #112

Merged

cursor Bot reviewed Sep 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(trg): let the llm judge target transcripts, files, and created-files lists - #154

feat(trg): let the llm judge target transcripts, files, and created-files lists#154
yordis merged 4 commits into
mainfrom
yordis/feat-eval-judge-targets

yordis commented Sep 13, 2026

Uh oh!

cursor Bot commented Sep 13, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 13, 2026 •

edited

Loading

Review limit reached

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Sep 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

yordis commented Sep 13, 2026

Uh oh!

cursor Bot commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

coderabbitai Bot commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Sep 13, 2026

Choose a reason for hiding this comment

LLM results omit declared target

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cursor Bot commented Sep 13, 2026 •

edited

Loading

coderabbitai Bot commented Sep 13, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading