Skip to content

Improve visibility of performance degradation in ASV benchmarking bot comment #1739

Description

@Sevans711

Recently, a few performance degradations merged into main from #941 without anybody talking about them. Most notable was a ~14x slowdown to import uxarray (which has since been solved by #1708).

After some internal conversation, there seemed to be agreement that the benchmarking bot comment does not do a good enough job flagging any performance degradation. This is probably because it currently hides them at the bottom of the message.

The current formatting is something like this:

ASV Benchmarking
----------------
<details><summary>Benchmark Comparison Results</summary>

Benchmarks that have improved:
(table of benchmarks)

Benchmarks that have stayed the same:
(table of benchmarks)

Benchmarks that have gotten worse:
(table of benchmarks)

</details>

It should be updated to especially emphasize any benchmarks that have gotten worse. I would recommend the following formatting instead:

ASV Benchmarking
----------------

Benchmarks that have gotten worse:
(table of benchmarks)

Benchmarks that have improved:
(table of benchmarks)

<details><summary>Benchmarks that have stayed the same</summary>
(table of benchmarks)
</details>

Posting this as an issue so that we don't lose track of it.

cc: @cmdupuis3

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    benchmarkingRelated to benchmarks, memory usage, and/or time profilingdeveloper experienceMakes the codebase easier to read, debug, maintain, or extend.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions