feat(xtest): compare benchmark candidates with each other

[dmihalcik-virtru]

Stage 4 of the six-stage plan. Stacked on #658.

What it does

When a run has more than one candidate, every pair of candidates is compared directly. These comparisons never fail the build.

• Verdicts: FASTER / SLOWER / TIED / INCONCLUSIVE, corrected in their own BH family so they cost the regression gate no power.
• Tails at α/2: neither candidate is privileged, so the question is two-sided. feat(xtest): compare up to four benchmark arms in one run #621 tested each tail at α, which doubles the chance of ranking two equal builds.
• TIED requires the interval to lie inside the ±threshold band, which is TOST at 2.5%.
• Order doesn't matter: swapping a pair turns FASTER into SLOWER, so the order candidates are listed in can't change a conclusion.
• Winner rules: perf/conclusions.py decides winners once, and the JSON (conclusions), terminal and markdown all read that result. A winner must be FASTER than every other candidate, and a tripped A/A control suppresses the winner however narrow the interval. Ties and unresolved competitors say which pair blocked the winner.
• Control: the control now yields every copy-to-copy pair, so the noise floor covers the widest spacing between any two candidates.

This fixes the remaining two #621 review findings: a winner named from the top two point estimates, and a tripped control not suppressing the winner.

Compatibility

• Additive to the stage-3 schema: candidate pairs appear under contrasts, and there's a new top-level conclusions.
• Two-arm runs are unchanged: there's nothing to compare, and conclusions is empty.

Validation

• 278 passed. ruff, ruff format and pyright are clean.
• New tests:
  • Reversing a comparison preserves the conclusion.
  • Candidate contrasts don't change the gate's adjusted p-values.
  • A clear two-candidate difference names a winner without failing the build.
  • A tie names no winner.
  • At three candidates, a resolved first pair plus an inconclusive third competitor names no winner.
  • A tripped control suppresses the winner.
  • With no control, everything is unresolved.
  • Each tail is tested at α/2.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

X-Test Results

✅ java@v0.19.1-main
✅ java@v0.19.1-v0.29.0
✅ js@v0.4.0-v0.29.0
✅ js@main-v0.29.0
✅ go@v0.38.1-main
✅ java@main-main
✅ go@v0.38.1-v0.29.0
✅ go@main-v0.29.0
✅ js@v0.4.0-main
✅ js@main-main
✅ java@main-v0.29.0
✅ go@main-main