Problem
Two unchanged GLM runs (docs/performance/2026-09-09-recipe-smoke.md) differed 23–35% in achieved throughput; compare reported the change as eligible because outputs matched. With only two trials per cell there is no spread to judge against.
Evidence
A/A: single 33.8 → 25.9 tok/s, two-lane 70.0 → 45.7 tok/s, no deployment change. A later A/B/A for a scheduler flag needed the A1↔A2 drift (131 s vs 108 s) to interpret B (15 s).
Proposal
compare A B A2: treat A↔A2 as the noise floor; report B's change and mark it within_noise when |change| ≤ the A/A2 delta. Per cell, emit min/median/max and, with ≥3 trials, a bootstrap 95% CI on the median; withhold "change" when intervals overlap.
Problem
Two unchanged GLM runs (
docs/performance/2026-09-09-recipe-smoke.md) differed 23–35% in achieved throughput;comparereported the change as eligible because outputs matched. With only two trials per cell there is no spread to judge against.Evidence
A/A: single 33.8 → 25.9 tok/s, two-lane 70.0 → 45.7 tok/s, no deployment change. A later A/B/A for a scheduler flag needed the A1↔A2 drift (131 s vs 108 s) to interpret B (15 s).
Proposal
compare A B A2: treat A↔A2 as the noise floor; report B's change and mark itwithin_noisewhen |change| ≤ the A/A2 delta. Per cell, emit min/median/max and, with ≥3 trials, a bootstrap 95% CI on the median; withhold "change" when intervals overlap.