Skip to content

README.md

Latest commit

 

History

History
105 lines (82 loc) · 4.39 KB

README.md

File metadata and controls

105 lines (82 loc) · 4.39 KB
Raw

Binding Performance Benchmarks

This suite compares wrappers generated by prik and NumPy's f2py on the same machine. It measures both runtime call and NumPy-array overhead and clean, end-to-end build time.

The default prik wrapper and the f2py wrappers measured here keep the GIL held, so the suite reports one like-for-like comparison of their normal generated interfaces.

Build timings treat each tool as a black box. A timed sample starts with an empty output directory and includes source processing, wrapper generation, native and generated-source compilation, and linking. Import verification runs after the timer stops. Both tools use their normal dependency-aware compiler concurrency. The workloads are:

  • the one-source, five-procedure module used by the runtime suite; and
  • the repository's authoritative 155-source Reference BLAS project under examples/blas/native/, with all 155 routines required in each generated extension.

One untimed warm-up precedes six measured clean builds of each workload by default. Tool order alternates between rounds. Set PRIK_BUILD_BENCHMARK_RUNS or PRIK_BUILD_BENCHMARK_WARMUPS to change those counts for local investigation.

Every workload is measured with two compiler profiles:

  • development: -O0; and
  • optimized: -O3 -march=native -mtune=native.

Runtime-call measurements continue to use only the optimized profile.

PRIK_BENCHMARK_FIRST=prik is the default measurement order. Set it to f2py to reverse the order. The publication workflow alternates this setting between runs so one tool is not systematically measured first.

Run the complete correctness check and rigorous benchmark with:

bash run.sh

The script rebuilds both runtime extensions, runs both clean-build profiles, and applies each profile consistently to the native Fortran source, generated Fortran wrapper, and generated C binding. Runtime extensions use the optimized profile. Runtime cases use latency, medium, and bulk sampling budgets: the short and noisier cases use more pyperf worker processes and values, while expensive matrix cases use fewer. Contiguous case groups preserve the table's call, vector, matrix-sum, and matrix-update order as their results are appended to the same per-tool JSON suite, so comparison and publication commands remain unchanged. The script retains f2py's generated sources under build/f2py for local inspection.

To compare existing results without rebuilding:

python3 -m pyperf compare_to \
    results/f2py.json \
    results/prik.json \
    --table

Compare the clean build timings with:

python3 -m pyperf compare_to \
    results/f2py-build.json \
    results/prik-build.json \
    --table

Results are machine-specific. Compare files produced in the same run; CPU, compiler, Python, and NumPy differences can otherwise dominate small timings. The generated build directories, extensions, and result files are local artifacts rather than repository sources.

Publish a Documentation Snapshot

After a completed run, refresh the generated sections of the public Performance page and its chart with:

python3 tools/generate_performance_docs.py

Run this command from the repository root. It reads the runtime and build-time pyperf pairs, checks that each pair contains the same benchmarks and compatible platform metadata, records the host operating-system distribution and compiler, and updates only the marked result sections in docs/user/performance.md plus the runtime and clean-build comparison SVGs in docs/user/assets/. Explanatory prose and the reproduction instructions remain hand-maintained.

The Documentation workflow performs the same generation after successful correctness checks and rigorous measurements on pushes to main. The benchmark job uses GitHub's ubuntu-24.04-arm runner, whose standard hosted pool is based on Microsoft Cobalt 100 processors, and verifies that the allocated machine is ARM64 with the expected Neoverse N2/Cobalt 100 CPU part before measuring. The documentation build and deployment remain separate x86-64 jobs. The workflow keeps the raw pyperf files as an artifact and overlays the generated snapshot only in the website build; it does not create a result commit.

The publication environment pins Python 3.12, NumPy/f2py 2.5.1, pyperf 2.10.0, Meson 1.11.2, Ninja 1.13.0, GNU Fortran 13, and the ARM64 runner label. Update those inputs through a reviewed change so published runs remain comparable.