Skip to content

Stop timing NumPy in the ASV benchmarks - #3095

Merged
antonwolfy merged 5 commits into
masterfrom
benchmarks-drop-numpy-executor
Oct 14, 2026
Merged

antonwolfy merged 5 commits into
masterfrom
benchmarks-drop-numpy-executor

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR removes the numpy executor from the ASV benchmark suites, so bench_elementwise.py, bench_linalg.py and bench_random.py now time only dpnp.

The suites track dpnp performance from commit to commit, and nothing compares the dpnp and NumPy timings with each other. A dpnp change cannot move a NumPy timing, so those points carry no signal. They did have two costs:

  • They made up half of the timed points in these three suites (470 of 940).
  • Machine noise on them was reported as a dpnp regression. A master commit that only bumped pre-commit hooks was flagged solely because of MatMul.time_matmul('numpy', 16, 'float64') at 1.22x. On a recent PR, two of the three reported regressions were NumPy points, including MatMul.time_dot('numpy', 64, 'float32') at 1.6x, with the same NumPy and MKL builds on both sides.

The change keeps the recorded results comparable:

  • executor stays as a single-value axis ('dpnp'), so result names such as time_dot('dpnp', 64, 'float32') are unchanged.
  • The setup and time_* methods are not modified, including the numpy branches some setup methods no longer use. ASV versions a benchmark by the source of these methods, so editing them would make every result incomparable with the recorded baseline. All 29 such methods are byte-identical to master.

Against a baseline recorded before this change, the NumPy points exist only on the baseline side. asv compare treats them as skipped, so --only-changed hides them and they are not counted as regressions.

benchmarks/README.md is updated to match and gets a new "NumPy is not timed" section explaining the above.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

A dpnp change cannot move a NumPy timing, so the numpy executor points
only doubled the run time of the elementwise, linalg and random suites and
let machine noise on them be reported as a dpnp regression.

Keep `executor` as a single-value axis and leave the `setup`/`time_*`
bodies untouched, so result names and ASV benchmark versions stay
comparable with the recorded history.
@antonwolfy antonwolfy added this to the 0.21.0 release milestone Oct 12, 2026
@antonwolfy antonwolfy self-assigned this Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/index.html

@coveralls

coveralls commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 78.669% (+0.002%) from 78.667% — benchmarks-drop-numpy-executor into master

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Note, the 65536-size unary entries still keep making about half the runs UNSTABLE, but will be addressed in the follow-up.

@antonwolfy
antonwolfy marked this pull request as ready for review October 12, 2026 14:12
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Note, the 65536-size unary entries still keep making about half the runs UNSTABLE, but will be addressed in the follow-up.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.21.0dev14=np2py314h8d9cdd5_6 ran successfully.
Passed: 1376
Failed: 0
Skipped: 6

Comment thread benchmarks/benchmarks/_utils.py Outdated
antonwolfy and others added 2 commits October 13, 2026 14:32
With numpy gone from _EXECUTORS the helper is only called with "dpnp".
It is not part of ASV's benchmark version hash, so benchmark versions
stay unchanged.
@vlad-perevezentsev
vlad-perevezentsev self-requested a review October 14, 2026 10:19

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thank you @antonwolfy !

@antonwolfy
antonwolfy merged commit 0809e2d into master Oct 14, 2026
100 of 107 checks passed
@antonwolfy
antonwolfy deleted the benchmarks-drop-numpy-executor branch October 14, 2026 12:41
github-actions Bot added a commit that referenced this pull request Oct 14, 2026
This PR removes the `numpy` executor from the ASV benchmark suites, so
`bench_elementwise.py`, `bench_linalg.py` and `bench_random.py` now time
only dpnp.

The suites track dpnp performance from commit to commit, and nothing
compares the dpnp and NumPy timings with each other. A dpnp change
cannot move a NumPy timing, so those points carry no signal. They did
have two costs:
- They made up half of the timed points in these three suites (470 of
940).
- Machine noise on them was reported as a dpnp regression. A `master`
commit that only bumped pre-commit hooks was flagged solely because of
`MatMul.time_matmul('numpy', 16, 'float64')` at 1.22x. On a recent PR,
two of the three reported regressions were NumPy points, including
`MatMul.time_dot('numpy', 64, 'float32')` at 1.6x, with the same NumPy
and MKL builds on both sides.

The change keeps the recorded results comparable:
- `executor` stays as a single-value axis (`'dpnp'`), so result names
such as `time_dot('dpnp', 64, 'float32')` are unchanged.
- The `setup` and `time_*` methods are not modified, including the
`numpy` branches some `setup` methods no longer use. ASV versions a
benchmark by the source of these methods, so editing them would make
every result incomparable with the recorded baseline. All 29 such
methods are byte-identical to `master`.

Against a baseline recorded before this change, the NumPy points exist
only on the baseline side. `asv compare` treats them as skipped, so
`--only-changed` hides them and they are not counted as regressions.

`benchmarks/README.md` is updated to match and gets a new "NumPy is not
timed" section explaining the above. 0809e2d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants