Feat/pairwise judging - #31
Merged
Merged
Conversation
client.evaluations.compare_pairwise(a, b) asks a judge which of two runs answered each question better, instead of comparing two absolute averages that drift with the rubric and cluster in the 7-8 band. get_pairwise / list_pairwise read past comparisons back. agentx.testing.assert_pairwise is the pytest side. Its checks are the ones that actually catch regressions rather than flattering them: must_win treats a tie as a failure (a change that cannot win its own comparison has not earned a green test), max_losses catches a change that lifts the average by improving easy cases while breaking hard ones, and max_flip_rate fails a comparison whose verdicts reversed with the presentation order, because that measured position bias and an inconclusive comparison must not read as a pass. With no both_orders pass there is no flip rate, and the check is skipped rather than passing on a fabricated zero. Verified live against a local engine: the stronger run passed must_win + max_losses=0 + max_flip_rate=0.25, and the same pair with the runs swapped failed with "run A did not win (0 vs 3)". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
assert_evaluation shipped without a mention in any SDK markdown, and assert_pairwise would have followed it. Both are now documented where a CI reader looks, with the distinction that matters between them: one asks whether a run clears the bar, the other whether it beats what it replaces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.