Skip to content

fix(trg): carry a named grader's fields through the published report - #152

Merged
yordis merged 1 commit into
mainfrom
yordis/fix-eval-grader-names
Sep 13, 2026
Merged

fix(trg): carry a named grader's fields through the published report#152
yordis merged 1 commit into
mainfrom
yordis/fix-eval-grader-names

Conversation

@yordis

@yordis yordis commented Sep 13, 2026

Copy link
Copy Markdown
Member
  • A grader given a name in the suite lost that name by the time it reached report.json, so a reader could not tell which of several graders on the same assertion produced a result.
  • The report was assembled by serializing a result and then adding keys to the map, which silently dropped everything down the path where the value was not an object. Expressing the published shape as a type removes that path rather than guarding it.

Stacked on #150.

@cursor

cursor Bot commented Sep 13, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Report assembly change only; behavior is more complete serialization with regression tests, no auth or runtime path changes.

Overview
Fixes lossy merge of per-run grading into report.json. After eval grade, assertion_results used to be rebuilt field-by-field from each run’s grading.json, so stable grader metadata—name, excluded, rationale, and votes—never reached the published report even when present on disk.

The merge now serializes a PublishedAssertionResult type: run_id and eval_case_id plus a flattened AssertionGradeResult, so new result fields stay in sync with serde rules (optional keys like name still omitted when unset). Reference docs for report.json assertion_results are updated to match.

Tests assert unnamed prose assertions do not gain a name in report.json, and that named/excluded/multi-vote results survive the flatten end-to-end.

Reviewed by Cursor Bugbot for commit bd2c2a4. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Sep 13, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 0211b2f9-4aec-40ef-ae1a-bc871fdbcec7

📥 Commits

Reviewing files that changed from the base of the PR and between 4050107 and bd2c2a4.

📒 Files selected for processing (2)
  • crates/trg/docs/reference/ai-skills-eval.md
  • crates/trg/src/agentskills/grading.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@yordis
yordis force-pushed the yordis/fix-eval-grader-names branch from e498943 to b2b7e89 Compare September 13, 2026 17:38
@yordis
yordis force-pushed the yordis/fix-eval-grader-names branch 2 times, most recently from ed1749b to 84170e2 Compare September 13, 2026 18:45
Base automatically changed from yordis/feat-eval-ungradeable-assertions to main September 13, 2026 22:02
a suite that names a grader deserves a report that says so, not one that silently treats a named check the same as an anonymous one

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis
yordis force-pushed the yordis/fix-eval-grader-names branch from 84170e2 to bd2c2a4 Compare September 13, 2026 22:20
@yordis
yordis merged commit b7bfcce into main Sep 13, 2026
11 checks passed
@yordis
yordis deleted the yordis/fix-eval-grader-names branch September 13, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant