Repository navigation
Run dpnp.tensor tests in coverage workflow - #3094
antonwolfy wants to merge 5 commits into
Conversation
Run dpnp.tensor tests in a separate pytest session that appends to the coverage data of the main dpnp session, and merge the LLVM profiles of both sessions. Add --skip-tensor-tests option to keep the previous behavior, and --tensor-cpp-coverage option to build dpnp.tensor pybind11 extensions with coverage instrumentation and include them into C++ coverage. Raise coverage workflow timeouts to account for the extra session.
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3094/index.html |
|
Array API standard conformance tests for dpnp=0.21.0dev13=np2py314h8d9cdd5_10 ran successfully. |
|
No base build to compare — coverage-include-tensor-tests into master |
| run( | ||
| pytest_cmd | ||
| # report once, on the combined data of the last session | ||
| + (["--cov-report="] if args.run_tensor_tests else cov_report) |
There was a problem hiding this comment.
If the tensor tests fail or hit OOM we lose coverage reports from both sessions.
Since these tests may hit OOM (#2860), I suggest keeping cov_report in the first session too.
The second session will overwrite coverage-python.lcov anyway and printing term-missing twice will show coverage from each session
| + (["--cov-report="] if args.run_tensor_tests else cov_report) | |
| # the tensor session below re-reports on the combined data | |
| + cov_report |
There was a problem hiding this comment.
Thanks, good point. Instead of reporting in both sessions, the coverage job is now split into separate steps: build, dpnp tests, dpnp.tensor tests and coverage report generation.
- The dpnp.tensor tests step has its own timeout and retries (also on timeout), so an OOM/abort or a hang of the tensor tests re-runs only that step, without rebuilding dpnp and re-running dpnp tests.
- The pytest sessions don't generate reports anymore. A separate step produces them from the combined
.coveragedata, with the same lcov output as pytest-cov.
Note that reporting in the first session wouldn't keep the coverage on its own: if the tensor session is killed, the job fails and the Coveralls upload is skipped, while the dpnp tests data stays in .coverage anyway.
Add --stage option to gen_coverage.py to run build, dpnp tests, dpnp.tensor tests and report generation separately. Coverage reports are now generated by the report stage from the combined data of all test sessions. Run each stage as a separate workflow step, so dpnp.tensor tests have their own timeout and are retried (also on timeout) without rebuilding dpnp and re-running dpnp tests.
The oneAPI compiler is installed with apt and is not in PATH unless setvars.sh is sourced, which the build and report stages require.
This branch has not been deployed
3 participants
This PR extends the coverage workflow to also run
dpnp.tensortests, which are currently excluded bySKIP_TENSOR_TESTS=1.Because of that,
dpnp/tensorcode is measured only through what dpnp tests happen to reach, and some lines end up covered by chance.For example, line 809 of
dpnp/tensor/_elementwise_common.pyis reached deterministically only bydpnp/tests/tensor/elementwise/test_add.py::test_add_order. In the coverage run it is reached only byTestDotFor0Dimin the third-party tests, and only when the random dtype-pair sampling offor_all_dtypes_combinationhappens to include afloat32xint32/int64pair. The line is missed in about 1/3 of runs (251 of 744 Coveralls builds sincedpnp.tensorwas introduced in #2856), which shows up as ±0.003% coverage noise on PRs without any source changes.Note, enabling
--tensor-cpp-coveragein CI is left for a follow-up. It may also need-ignore-filename-regexrules forlibtensorkernel headers, similar to the existing ones forbackend/kernels.