Recently, a few performance degradations merged into main from #941 without anybody talking about them. Most notable was a ~14x slowdown to import uxarray (which has since been solved by #1708).
After some internal conversation, there seemed to be agreement that the benchmarking bot comment does not do a good enough job flagging any performance degradation. This is probably because it currently hides them at the bottom of the message.
The current formatting is something like this:
ASV Benchmarking
----------------
<details><summary>Benchmark Comparison Results</summary>
Benchmarks that have improved:
(table of benchmarks)
Benchmarks that have stayed the same:
(table of benchmarks)
Benchmarks that have gotten worse:
(table of benchmarks)
</details>
It should be updated to especially emphasize any benchmarks that have gotten worse. I would recommend the following formatting instead:
ASV Benchmarking
----------------
Benchmarks that have gotten worse:
(table of benchmarks)
Benchmarks that have improved:
(table of benchmarks)
<details><summary>Benchmarks that have stayed the same</summary>
(table of benchmarks)
</details>
Posting this as an issue so that we don't lose track of it.
cc: @cmdupuis3
Recently, a few performance degradations merged into main from #941 without anybody talking about them. Most notable was a ~14x slowdown to
import uxarray(which has since been solved by #1708).After some internal conversation, there seemed to be agreement that the benchmarking bot comment does not do a good enough job flagging any performance degradation. This is probably because it currently hides them at the bottom of the message.
The current formatting is something like this:
It should be updated to especially emphasize any benchmarks that have gotten worse. I would recommend the following formatting instead:
Posting this as an issue so that we don't lose track of it.
cc: @cmdupuis3