Add benchmark budget enforcement with CI reporting - #9150
Conversation
…dget The benchmarking guide asks that each benchmark iteration complete in under 1ms, but nothing enforced it. CodSpeed runs in simulation mode, which estimates cycles from an instruction trace and never reports wall-clock time, so the existing job cannot answer the question. The `divan` dependency is really `codspeed-divan-compat`. Built without `--cfg codspeed` it re-exports CodSpeed's patched divan, which dumps per-iteration statistics as JSON when `CODSPEED_ENV` is set. The new `bench-budget` job rebuilds the sharded benchmarks in walltime mode, runs them once outside the CodSpeed runner, and compares the fastest observed iteration against the budget. Using the minimum rather than the median keeps a loaded shared runner from making the check flaky. Only benchmarks CodSpeed actually measures are checked. That set comes from replaying the analysis-mode binaries, which print one line per benchmark they run; because that is the same build CodSpeed uses, benchmarks gated with `#[cfg(not(codspeed))]` are excluded automatically rather than through an allowlist that would drift. Both modes generate benchmark URIs with identical code, so the two sets join exactly. Results are posted as a sticky PR comment rather than failing the build, and the job only runs on pull requests that touch a `benches/` directory. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…nchmark-ci-check-d50h60
Adds a benchmark that deliberately breaks the 1ms per-iteration rule, so the bench-budget job has something to catch on this pull request. Counting set bits across a 32 MiB bit buffer is memory-bandwidth bound and measures 3.28ms locally, against 71 existing vortex-mask benchmarks that all pass. The work is real rather than a sleep: CodSpeed's simulation instrument excludes system calls, so a sleeping benchmark would appear free there while still consuming CI wall-clock time. This commit must be reverted before merge. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
⏱️ Benchmark iteration budget
CodSpeed's simulation instrument runs each benchmark exactly once, so a slow iteration costs CI time without buying any extra signal. Shrink the input size, or gate the benchmark with
How this is measuredBenchmarks are rebuilt in CodSpeed's walltime mode and run once outside the CodSpeed runner. The reported number is Only benchmarks that CodSpeed actually measures are checked. That set comes from the analysis-mode binaries built by |
Merging this PR will degrade performance by 14.22%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | cuda/bitpacked_u8/unpack/3bw[100M] |
299.9 µs | 349.6 µs | -14.22% |
| 🆕 | Simulation | over_budget_prefix_count |
N/A | 16.2 ms | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/codspeed-benchmark-ci-check-d50h60 (40c3940) with develop (aa7cd40)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
2 participants
Rationale for this change
The benchmarking guide requires that each benchmark iteration complete in less than 1ms to keep benchmarks snappy locally and on CI. However, CodSpeed's simulation instrument reports estimated cycles from instruction traces rather than wall-clock time, so it cannot enforce this rule.
This change adds automated budget checking that:
#[cfg(not(codspeed))]by only checking benchmarks CodSpeed actually measuresThis closes the gap between the documented requirement and CI enforcement.
What changes are included in this PR?
New script:
scripts/check-bench-budget.pychecksubcommand: Reads raw walltime results from one shard, filters to benchmarks CodSpeed measures, and emits a JSON verdictreportsubcommand: Merges per-shard verdicts into a Markdown comment bodyNew tests:
scripts/tests/test_check_bench_budget.pyCI workflow updates:
.github/workflows/codspeed.ymlrun-budget-checkoutput to thechangesjob to detect when benchmarks are modifiedbench-budgetjob that runs on 8 shards (same sharding as the main CodSpeed job):CODSPEED_ENV=1to trigger JSON output from the divan harnessbench-budget-commentjob that merges verdicts and posts a sticky PR commentDocumentation:
docs/developer-guide/benchmarking.md#[cfg(not(codspeed))]benchmarks are automatically exemptCI test:
.github/workflows/ci.ymlWhat APIs are changed? Are there any user-facing changes?
No public APIs are changed. This is purely a CI/tooling addition.
User-facing changes:
https://claude.ai/code/session_014WMg4dzviJZfD3tZgmcNty