Evaluation report: model comparison with confidence intervals, P95 latency and normalised cost

[TonsOfFun]

When an evaluation runs several models, the report should answer the question the run was created to ask: which model should we ship, and is the difference real? Today it shows per-model columns with pass counts, average latency and cost, but nothing that says whether a gap between two models is meaningful or noise.

Reference

The shape worth adopting (from a comparison report shared by @jusbowen — screenshot to be attached) has two parts:

1. A ranked interval plot — one row per model, success rate with a 95% confidence interval, sorted best-first, with a caption naming the method and sample count:

Gemma 4 31B IT       ├────●────┤   85.2%
Qwen3.5-27B        ├─────●─────┤   82.3%
Gemma 4 26B A4B IT ├──────●──────┤ 79.4%
DeepSeek V4 Flash  ├──────●──────┤ 78.4%
                 60%   70%   80%   90%   100%
     Success rate, 95% CI from 5,000 resamples of whole scenarios

Plus the honest read directly under it:

All six pairwise difference intervals include zero — the ranking is not decisive evidence of superiority.

That sentence is the most valuable part. A bare leaderboard invites "31B wins"; the interval note is what stops a team shipping a model change on 42 scenarios of noise.

2. A metrics table, one row per model:

Model	Success	Evidence	No answer	Median / P95	Cost / 1,000 conv.
DeepSeek V4 Flash 0731	78.4%	86.7%	31/384	7.27 / 42.73 s	$0.54
Qwen3.5-27B	82.3%	93.2%	5/384	7.76 / 24.28 s	$3.52
Gemma 4 31B IT	85.2%	95.3%	1/384	4.02 / 22.20 s	$3.63

Note this makes the cost/quality trade-off legible in a way an average cannot: DeepSeek is ~7× cheaper for ~7 points of success rate, and its P95 latency is nearly 2× the others even though its median is competitive.

What exists today

EvaluationRunPanels.jsx already renders per-model columns, per-model pass counts, avg_duration_ms and cost, and the runner already supports multi-model runs via compare_models / selection[:models]. EvaluationScenarioResult carries model, provider, status, score, duration_ms, input_tokens, output_tokens and cost per scenario × model.

So most of the input data is present. What is missing:

• Confidence intervals — bootstrap over whole scenarios (resampling scenarios, not individual results, since scenarios are the independent unit).
• Pairwise difference intervals and the plain-language verdict about whether the ranking is decisive.
• Percentile latency — P50/P95 rather than only the mean; the mean hides exactly the tail that matters.
• Cost normalised to a common denominator (per 1,000 conversations) so models are comparable regardless of how many scenarios ran.
• A ranked visual, rather than reading numbers across columns.

Notes

• Sample size should be surfaced honestly — with 42 scenarios the intervals will be wide, and the view should make that obvious rather than implying more precision than the run supports.
• "Evidence" and "No answer" in the reference map roughly to existing fault categories; worth deciding whether to reuse the current taxonomy (low_quality, missing_content, expected_tool_not_called, …) instead of inventing new columns.
• The pairwise intervals are worth persisting alongside the run (the reference keeps them in a summary.json) so the report does not recompute them on every render.