From 823e52689cb382c8a51c0b391b5087644695fa16 Mon Sep 17 00:00:00 2001 From: "Harlow, Jordan" Date: Wed, 7 Oct 2026 12:46:19 -0600 Subject: [PATCH 1/2] task: agent ready prep --- .github/AGENTS.md | 14 +- .github/ISSUE_TEMPLATE/bug_report.yml | 64 +++++++++ .github/ISSUE_TEMPLATE/config.yml | 8 ++ .github/ISSUE_TEMPLATE/feature_request.yml | 37 +++++ .github/copilot-instructions.md | 3 +- .github/pull_request_template.md | 25 ++++ .gitignore | 12 ++ AGENTS.md | 155 +++++++-------------- CONTRIBUTING.md | 126 +++++++++++++++++ README.md | 7 + _vendored/AGENTS.md | 28 ++-- benchmarks/AGENTS.md | 24 ++++ conda-recipe-cf/AGENTS.md | 30 ++-- conda-recipe/AGENTS.md | 38 ++--- mkl_umath/AGENTS.md | 51 +++---- mkl_umath/src/AGENTS.md | 56 +++----- mkl_umath/tests/AGENTS.md | 40 +++--- 17 files changed, 458 insertions(+), 260 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml create mode 100644 .github/pull_request_template.md create mode 100644 CONTRIBUTING.md create mode 100644 benchmarks/AGENTS.md diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 47576d02..6751dc71 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -3,13 +3,15 @@ CI/CD workflows and repo automation. ## Workflows (source of truth) -- `conda-package.yml` — Intel channel conda build/test pipeline -- `conda-package-cf.yml` — conda-forge-oriented build/test pipeline -- `build-with-clang.yml` — Intel clang compatibility checks -- `build-with-standard-clang.yml` — standard clang compatibility checks -- `build_pip.yml` — pip build pipeline with pre-release NumPy +- `conda-package.yml` — Intel-channel conda build and test +- `conda-package-cf.yml` — conda-forge conda build and test +- `build_pip.yml` — editable pip build with `icx`, including pre-release NumPy +- `build-with-clang.yml` — build with `icx` from the oneAPI apt repository +- `build-with-standard-clang.yml` — build with upstream clang - `pre-commit.yml` — lint/format checks -- `openssf-scorecard.yml` — security scanning +- `coverity.yml` — Coverity static analysis (see `coverity/README.md`) +- `openssf-scorecard.yml` — OpenSSF Scorecard +- `zizmor.yml` — GitHub Actions security lint ## Policy - Treat workflow YAML as canonical for platform/Python matrices. diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 00000000..e436b17b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,64 @@ +name: Bug report +description: Report incorrect results, a crash, or a build failure +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + For security vulnerabilities, do not open an issue — follow + [SECURITY.md](https://github.com/IntelPython/mkl_umath/blob/main/SECURITY.md). + + - type: textarea + id: description + attributes: + label: Description + description: What happened, and what did you expect instead? + validations: + required: true + + - type: textarea + id: reproducer + attributes: + label: Reproducer + description: A minimal, self-contained snippet. Include the ufunc, input values, and dtype. + render: python + validations: + required: true + + - type: dropdown + id: install-source + attributes: + label: How was `mkl_umath` installed? + options: + - Intel conda channel (software.repos.intel.com) + - conda-forge + - pip, Intel index (software.repos.intel.com) + - pip / PyPI + - Built from source + validations: + required: true + + - type: textarea + id: versions + attributes: + label: Versions + description: | + Output of: + ``` + python -c "import mkl_umath, numpy; print(mkl_umath.__version__); print(numpy.__version__)" + ``` + Add your OS and Python version too. + render: shell + validations: + required: true + + - type: textarea + id: notes + attributes: + label: Anything else + description: | + Optional. Whether NumPy patching was active (`mkl_umath.is_patched()`), + whether the result differs from stock NumPy, or a non-default + `MKL_NUM_THREADS`. + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..f348f1ae --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: true +contact_links: + - name: Security vulnerability + url: https://www.intel.com/content/www/us/en/security-center/vulnerability-handling-guidelines.html + about: Report security vulnerabilities through Intel's process, not a public issue. + - name: Question about usage + url: https://github.com/IntelPython/mkl_umath/blob/main/README.md + about: Check the README first, including the patching section. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 00000000..a1d08282 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,37 @@ +name: Feature request +description: Propose a new ufunc loop, dtype, or capability +labels: ["enhancement"] +body: + - type: textarea + id: problem + attributes: + label: What problem does this solve? + description: The use case, not the implementation. + validations: + required: true + + - type: textarea + id: proposal + attributes: + label: Proposal + description: | + What you would like `mkl_umath` to do. If it covers a NumPy ufunc, name + it — NumPy's semantics are the contract for patched loops. + validations: + required: true + + - type: input + id: upstream + attributes: + label: Upstream equivalent + description: Link to the NumPy docs for the ufunc, if there is one. + validations: + required: false + + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + description: Optional. Workarounds you are using today. + validations: + required: false diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d1544886..dd9d0aac 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -19,7 +19,8 @@ Higher-precedence rules override lower-precedence context. ## Contribution expectations - Keep changes atomic and single-purpose. -- Preserve runtime patching API (`use_in_numpy()`, `restore()`, `is_patched()`) unless explicitly requested. +- Preserve the runtime patching API (`patch_numpy_umath()`, `restore_numpy_umath()`, + `is_patched()`, and the `mkl_umath()` context manager) unless explicitly requested. - For behavior changes, update tests in `mkl_umath/tests/` in the same step. - For bugs, include a regression test. - Do not modify generated artifacts directly when template/source files are the intended edit points. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..051112d9 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,25 @@ +# Description + + + +## Verification + + + +- Tests: +- Lint: + +## Not verified + + + +## Checklist + +- [ ] Results match stock NumPy, or the difference is intentional and called out above. +- [ ] Behavior changes have tests in `mkl_umath/tests/`; bug fixes have a regression test. +- [ ] `CHANGELOG.md` updated under `## [dev]` with a `[gh-NNN]` link, or the change isn't user-visible. + + diff --git a/.gitignore b/.gitignore index 3ab62969..92a6e677 100644 --- a/.gitignore +++ b/.gitignore @@ -54,6 +54,12 @@ var/ pip-log.txt pip-delete-this-directory.txt +# Virtual environments, test caches, local env files +.venv/ +venv/ +.pytest_cache/ +.env + # Unit test / coverage reports htmlcov/ .tox/ @@ -87,3 +93,9 @@ mkl_umath/src/__umath_generated.c mkl_umath/src/mkl_umath_loops.c mkl_umath/src/mkl_umath_loops.h mkl_umath/src/_patch.c + +# ASV benchmark artifacts +.asv/ + +# Developer-local coding agent settings +.claude/settings.local.json diff --git a/AGENTS.md b/AGENTS.md index 01525618..7119dd7c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,109 +1,60 @@ -# AGENTS.md +# AGENTS.md — mkl_umath Entry point for agent context in this repo. -## What this repository is -`mkl_umath` exposes Intel® OneMKL-powered universal function loops for NumPy, originally part of Intel® Distribution for Python* and factored out per NEP-36 (Fair Play). - -It provides: -- `mkl_umath._ufuncs` — OneMKL-backed NumPy ufunc loops -- `mkl_umath._patch_numpy` — runtime patching interface (`patch_numpy_umath` `restore_numpy_umath`, `is_patched()`) -- Performance-optimized math operations (sin, cos, exp, log, etc.) using Intel MKL VM +## What this project is +`mkl_umath` provides NumPy ufunc loops backed by Intel® oneMKL Vector Math, and +can patch them into NumPy at runtime. It was factored out of Intel® Distribution +for Python* per NEP-36 (Fair Play). ## Key components -- **Python interface:** `mkl_umath/__init__.py`, `_init_helper.py` -- **Core C implementation:** `mkl_umath/src/` (ufuncsmodule.c, mkl_umath_loops.c.src) -- **Cython patch layer:** `mkl_umath/src/_patch_numpy.pyx` -- **Code generation:** `generate_umath.py`, `generate_umath_doc.py` -- **Build system:** meson-python + Cython - -## Build dependencies -**Required:** -- Compiler toolchain: Intel `icx` or `clang` (with Intel-only flags gated when using clang) -- Intel® oneMKL (`mkl-devel`) -- meson-python, CMake, Ninja, Cython>=3.1.0, NumPy - -**Build against an existing `mkl` installation:** - -Install the build dependencies via Conda: -```bash -conda install -c https://software.repos.intel.com/python/conda \ - mkl-devel dpcpp_linux-64 "cython>=3.1.0" meson-python cmake ninja numpy -``` -or via pip: -```bash -pip install mkl-devel "cython>=3.1.0" meson-python cmake ninja numpy -``` -then build: -```bash -CC=icx pip install --no-deps --no-build-isolation . # clang is also supported in CI -``` - -## CI/CD -- **Platforms:** Linux, Windows -- **Python versions:** 3.10, 3.11, 3.12, 3.13, 3.14 -- **Workflows:** `.github/workflows/` - - `conda-package.yml` — main conda build/test pipeline - - `conda-package-cf.yml` — conda-forge-oriented build/test pipeline - - `build_pip.yml` — validates pip build with pre-release NumPy - - `build-with-clang.yml` — Intel clang compatibility check - - `build-with-standard-clang.yml` — standard clang compatibility check - - `openssf-scorecard.yml` — security scorecard - -## Distribution -- **Conda:** `https://software.repos.intel.com/python/conda` -- **PyPI:** `https://software.repos.intel.com/python/pypi` -- Requires Intel-optimized NumPy from Intel channels - -## Usage -```python -import mkl_umath -mkl_umath.patch_numpy_umath() # Patch NumPy to use MKL loops -# ... perform NumPy operations (now accelerated) ... -mkl_umath.restore_numpy_umath() # Restore original NumPy loops -``` - -## How to work in this repo -- **Performance:** Changes should maintain or improve MKL VM utilization -- **Compatibility:** Must work with upstream NumPy APIs (NEP-36 compliance) -- **Testing:** Add tests to `mkl_umath/tests/test_basic.py` -- **Build hygiene:** `meson.build` is the source of truth for build config — verify Linux + Windows -- **Docs:** Update docstrings via `ufunc_docstrings_numpy{1,2}.py` - -## Code structure -- **Generated code:** `*.src` files are templates (conv_template.py processes them) -- **Precision flags:** fp:precise, fimf-precision=high, fprotect-parens (non-negotiable) -- **Security:** Stack protection, FORTIFY_SOURCE, NX/DEP enforced in `meson.build` -- **Build options:** `opt_report` and `mkl_threading` are exposed via `meson.options` - - -## Common pitfalls -- **NumPy source:** Requires Intel-optimized NumPy from Intel channel (`software.repos.intel.com/python/conda`). PyPI NumPy may cause runtime failures or incorrect results. -- **Precision flags:** `fp:precise`, `fimf-precision=high` enforce IEEE 754 compliance. Removing them breaks numerical correctness in scientific computing. -- **Patching order:** If using multiple Intel patches (e.g., `mkl_random` + `mkl_umath`), apply `mkl_umath` last. Verify with `is_patched()` after each. -- **Compiler/toolchain:** `icx` and `clang` are both supported; when using clang, keep Intel-only flags behind compiler guards. -- **Build validation:** - - After setup: `which ${CC:-icx}` → should resolve to the intended compiler toolchain - - Check: `python -c "import numpy; print(numpy.__version__)"` → confirm NumPy is available - -## Notes -- `_vendored/` contains vendored NumPy code generation utilities -- Version in `mkl_umath/_version.py` (read dynamically by `meson.build`) -- Patching is runtime-only; no NumPy source modification +- **Package and public API:** `mkl_umath/`, `mkl_umath/__init__.py` +- **Ufunc extension:** `mkl_umath/src/ufuncsmodule.c`, plus `__umath_generated.c` + from `mkl_umath/generate_umath.py` +- **Loop templates:** `mkl_umath/src/mkl_umath_loops.{c,h}.src` +- **Patching:** `mkl_umath/src/_patch_numpy.pyx`; persistent and one-shot + patching in `patch.py`, `with_patch.py`, `_patch_startup.py`, and the + `__main__.py` CLI +- **Tests:** `mkl_umath/tests/` +- **Vendored helpers:** `_vendored/` +- **Packaging:** `conda-recipe/`, `conda-recipe-cf/` +- **Benchmarks:** `benchmarks/` + +## Build/runtime basics +- Build system: `pyproject.toml` + `meson.build`, with options in `meson.options` +- Build deps: `mkl-devel`, `numpy`, `meson-python`, `cmake`, `ninja`, `cython`, + and a C compiler (CI uses `icx` and `clang`) +- Runtime deps: `numpy`; the conda recipes add the MKL and compiler runtimes +- Setup, checks, and style: `CONTRIBUTING.md` +- Single test: `pytest mkl_umath/tests/::` +- Single-file lint: `pre-commit run --files ` + +## Development guardrails +- Preserve NumPy ufunc behavior; patched loops stand in for NumPy's own. +- Edit the `*.src` templates and `generate_umath.py`, not generated C. +- Keep the floating-point precision flags in `meson.build`; Intel-only flags stay + behind its compiler checks. +- Keep patching reversible, with `is_patched()` reporting the truth. +- Keep both extensions free-threading compatible. +- Pair behavior changes with tests and keep diffs minimal. +- Avoid hardcoding mutable versions/matrices/channels in docs. + +## Where truth lives +- Build/config: `pyproject.toml`, `meson.build`, `meson.options` +- Dependencies: `pyproject.toml`, `conda-recipe*/meta.yaml` +- CI/workflows: `.github/workflows/*.yml` +- Public API: `mkl_umath/__init__.py`, `mkl_umath/src/_patch_numpy.pyx` +- Tests: `mkl_umath/tests/` + +For behavior policy, see `.github/copilot-instructions.md`. ## Directory map -Below directories have local `AGENTS.md` for deeper context: -- `.github/AGENTS.md` — CI/CD workflows and automation -- `mkl_umath/AGENTS.md` — Python API and code generation -- `mkl_umath/src/AGENTS.md` — C/Cython implementation layer -- `mkl_umath/tests/AGENTS.md` — unit tests and validation -- `conda-recipe/AGENTS.md` — Intel channel conda packaging -- `conda-recipe-cf/AGENTS.md` — conda-forge compatible recipe -- `_vendored/AGENTS.md` — vendored NumPy utilities - ---- - -For broader IntelPython ecosystem context, see: -- `dpnp` (Data Parallel NumPy) -- `mkl_random` (MKL-based random number generation) -- `numba-dpex` (Numba + SYCL) +Use nearest local `AGENTS.md` when present: +- `.github/AGENTS.md` — CI workflows and automation policy +- `mkl_umath/AGENTS.md` — package modules, API, and code generation +- `mkl_umath/src/AGENTS.md` — loop templates and the two extensions +- `mkl_umath/tests/AGENTS.md` — test scope and conventions +- `conda-recipe/AGENTS.md` — Intel-channel conda packaging +- `conda-recipe-cf/AGENTS.md` — conda-forge recipe +- `_vendored/AGENTS.md` — vendored NumPy template tooling +- `benchmarks/AGENTS.md` — ASV performance suite diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..42b9d299 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,126 @@ +# Contributing to `mkl_umath` + +This document covers the development workflow: how to get a working build, how +to run the checks, and what to include in a pull request. + +For end-user installation and usage, see [README.md](README.md). For a map of +the source tree, see [`AGENTS.md`](AGENTS.md), which links to the local +`AGENTS.md` files in directories that have their own rules. Security +vulnerabilities go through the process in [SECURITY.md](SECURITY.md). + +--- + +## Development setup + +Building requires a C compiler, oneMKL headers and libraries (`mkl-devel`), and +NumPy. CI builds with the Intel `icx` compiler and with upstream `clang`. On +Linux, a conda-forge environment provides all of it, including `icx`: + +```sh +# add python=X.Y to target a specific interpreter +conda create -n mkl_umath-dev -c conda-forge --override-channels python pip \ + mkl-devel dpcpp_linux-64 numpy meson-python ninja cmake "cython>=3.1.0" pytest +conda activate mkl_umath-dev +``` + +Then build in place, which reuses the environment's MKL and NumPy: + +```sh +CC=icx pip install -e . --no-build-isolation --no-deps --verbose \ + -Csetup-args=-Dmkl_threading=gnu_thread +``` + +`gnu_thread` matches the conda-forge recipe. The default threading layer, +`intel_thread`, needs Intel's OpenMP runtime; against conda-forge's MKL, +`import mkl_umath` then fails with `undefined symbol: __atomic_compare_exchange`. +`meson.options` lists the other layers. + +`pyproject.toml` defines the supported Python range, and `.github/workflows/` is +canonical for the versions CI covers. `README.md` documents the non-editable +install paths, including the isolated build that resolves its own `mkl-devel` +and `numpy`. + +### Rebuilding + +`meson-python` rebuilds the extensions on import for editable installs, so +editing `.pyx`, `.c.src`, `generate_umath.py`, or `meson.build` and rerunning +`pytest` is usually enough. Generated sources and the compiled extensions live +under `build//` rather than in the source tree. If a build gets into a bad +state, `rm -rf build` and reinstall. + +## Running the checks + +```sh +pytest mkl_umath/tests # test suite +pre-commit run --all-files # lint and format hooks +``` + +To run a single test, use `pytest mkl_umath/tests/::`; to lint one +file, `pre-commit run --files `. + +Install the hooks once with `pre-commit install` and they run on each commit. +`.pre-commit-config.yaml` is the source of truth for the tooling. + +Opening a pull request also runs CI, which builds and tests the package across +platforms and Python versions and runs various lint and static-analysis checks. + +## Code style + +Style is loose, and the pre-commit hooks enforce most of it: + +- Python is formatted with `black` and `isort`, with a line length of 80. +- Cython is not touched by `black`. `isort` sorts its imports, `cython-lint` + checks it against the same 80-column limit, and string literals use double + quotes. +- C sources follow the repository's `.clang-format`. +- Otherwise, match the surrounding code. + +## Dos and don'ts + +**Do** + +- Keep changes atomic and single-purpose. +- Preserve NumPy behavior. Patching swaps these loops in for NumPy's own, so a + result that differs from stock NumPy, including NaN and signed-zero handling, + is a bug. Call out an intentional difference in the PR. +- Add tests in `mkl_umath/tests/` alongside behavior changes, and a regression + test with every bug fix. +- Keep tests deterministic. +- Edit the `*.src` templates in `mkl_umath/src/` and `generate_umath.py` for + loop changes. The C they produce is regenerated on every build. +- Keep patching reversible and observable: anything installed can be + uninstalled, and `is_patched()` reports the truth. +- Keep both extensions free-threading compatible. +- Cite the source-of-truth file for mutable details: `pyproject.toml`, + `meson.build`, `meson.options`, `conda-recipe*/meta.yaml`, + `.github/workflows/`. +- Give benchmark numbers reproducible context — hardware, versions, and the + command you ran. + +**Don't** + +- Commit generated artifacts, or hand-edit generated C. +- Remove or weaken the floating-point precision flags in `meson.build`. +- Hardcode versions, build flags, CI matrices, or channel URLs in documentation. +- Assert on timing or throughput in the test suite. +- Refactor `_vendored/` opportunistically. Keep local diffs minimal and send + fixes upstream where you can. +- Introduce ISA-specific assumptions outside explicit build configuration. + +## Submitting a change + +Work on a branch: the `no-commit-to-branch` hook blocks direct commits to +`main` and `maintenance/*`. + +If the change is user-visible — behavior, API, packaging, or build output — add +a `CHANGELOG.md` entry under `## [dev]` in the matching section, with a +`[gh-NNN](https://github.com/IntelPython/mkl_umath/pull/NNN)` link. The format +follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project +follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Docs, +tooling, and CI-only changes are usually left out. + +Then open the PR and fill in the template, including what you verified locally +and what you left to CI. + +By contributing you agree that your contributions are licensed under the +BSD-3-Clause terms in [LICENSE.txt](LICENSE.txt). diff --git a/README.md b/README.md index 9e6cafad..432da96a 100644 --- a/README.md +++ b/README.md @@ -121,3 +121,10 @@ then build against the existing installation with: ```sh CC=icx pip install --no-build-isolation --no-deps . ``` + +--- +# Contributing + +See [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow: setting up a +build environment, running the tests and lint hooks, code style, and what to +include in a pull request. diff --git a/_vendored/AGENTS.md b/_vendored/AGENTS.md index 0ba2553a..475188d3 100644 --- a/_vendored/AGENTS.md +++ b/_vendored/AGENTS.md @@ -1,21 +1,17 @@ # AGENTS.md — _vendored/ -Vendored dependencies from upstream projects (NumPy). +Build-time template tooling copied from NumPy's `numpy/_build_utils`. ## Files -- **conv_template.py** — NumPy's template processor (from `numpy.distutils`) -- **__init__.py** — Python package marker +- `conv_template.py` — expands `/**begin repeat ... end repeat**/` blocks in + `.src` files +- `process_src_template.py` — command-line wrapper that `meson.build` runs on + the `.src` files; loads `conv_template.py` +- `README.md` — provenance -## Why vendored? -- `numpy.distutils` removed in NumPy 2.0+ / Python 3.12+ -- Needed for `.src` template processing at build time -- Vendored to maintain build compatibility across NumPy versions - -## Maintenance -- Source: NumPy's `numpy/distutils/conv_template.py` -- Update if template syntax changes upstream (rare) -- Do not modify vendored code (keep attribution intact) - -## Usage -- Imported by `generate_umath.py` for `.src` → `.c` conversion -- Processes `/**begin repeat ... end repeat**/` blocks +## Guardrails +- Prefer updating upstream source when feasible; keep local vendored diffs + minimal. +- Do not refactor vendored code opportunistically in unrelated PRs. +- `black` and `isort` are configured to skip the vendored files + (`pyproject.toml`). diff --git a/benchmarks/AGENTS.md b/benchmarks/AGENTS.md new file mode 100644 index 00000000..4d2a6a13 --- /dev/null +++ b/benchmarks/AGENTS.md @@ -0,0 +1,24 @@ +# AGENTS.md — benchmarks/ + +ASV performance suite for `mkl_umath`. + +## Scope +- `asv.conf.json` — ASV configuration, channels, and regression thresholds +- `benchmarks/micro/` — per-ufunc micro-benchmarks +- `benchmarks/npbench/` — end-to-end kernels adapted from npbench +- `benchmarks/_patch_setup.py` — patches NumPy at import +- `README.md` — coverage table, threading default, and run commands + +## Guardrails +- Treat `asv.conf.json` as canonical for ASV settings; treat `README.md` as + canonical for what each module covers. +- Benchmarks run on patched NumPy only: `_patch_setup.py` raises if patching + fails, so results never silently come from stock NumPy. +- Comparability across machines depends on the thread default in + `benchmarks/__init__.py` and the warmup call in each benchmark's `setup`. + Changing either invalidates comparison against existing results — call it + out explicitly. +- Keep inputs deterministic; benchmarks seed their own RNG. +- Report performance numbers with reproducible context: hardware, thread count, + versions, and the command used. +- Results under `benchmarks/.asv/` are local artifacts. diff --git a/conda-recipe-cf/AGENTS.md b/conda-recipe-cf/AGENTS.md index cea38d84..cce41b51 100644 --- a/conda-recipe-cf/AGENTS.md +++ b/conda-recipe-cf/AGENTS.md @@ -1,22 +1,18 @@ # AGENTS.md — conda-recipe-cf/ -Conda-forge compatible build recipe (alternative to Intel channel recipe). +conda-forge variant of the conda recipe. -## Difference from conda-recipe/ -- No `conda_build_config.yaml` (uses conda-forge defaults) -- May use different compiler toolchain -- For conda-forge feedstock integration (if upstreamed) +## Differences from `conda-recipe/` +- Resolves dependencies from conda-forge only +- Installs with `pip install` directly; no wheel build or retag +- Links MKL's GNU threading layer (`-Dmkl_threading=gnu_thread` in `build.sh`) + and uses `llvm-openmp` instead of `intel-openmp` +- Sets its version by hand in `meta.yaml` instead of reading git tags -## Files -- **meta.yaml** — conda-forge compatible metadata -- **build.sh** / **bld.bat** — platform build scripts -- **run_tests.{sh,bat}** — test invocation +Both recipes build with `icx` and use the same `conda_build_config.yaml`. -## Status -- Not currently used in main CI workflows -- Maintained for potential conda-forge submission -- Use `conda-recipe/` for Intel channel builds - -## Notes -- If upstreaming to conda-forge, this recipe should be preferred -- Compiler requirements may differ (Clang/GCC vs Intel icx) +## Guardrails +- Keep conda-forge recipe semantics separate from the Intel-channel recipe. +- Keep the `meta.yaml` version equal to `mkl_umath/_version.py`. +- Keep changes in step with `.github/workflows/conda-package-cf.yml`, which + builds this recipe and runs the full test suite against the result. diff --git a/conda-recipe/AGENTS.md b/conda-recipe/AGENTS.md index 08182026..776941ff 100644 --- a/conda-recipe/AGENTS.md +++ b/conda-recipe/AGENTS.md @@ -1,31 +1,17 @@ # AGENTS.md — conda-recipe/ -Conda package build recipe for Intel channel distribution. +Intel-channel conda packaging. ## Files -- **meta.yaml** — package metadata, dependencies, build requirements -- **build.sh** — Linux build script -- **bld.bat** — Windows build script -- **conda_build_config.yaml** — build matrix (Python versions, numpy pins) -- **run_tests.{sh,bat}** — post-build test invocation +- `meta.yaml` — package metadata, dependencies, and the package test +- `build.sh` / `bld.bat` — build a wheel with `python -m build` using `icx`, + then install it; `build.sh` also retags the wheel's platform +- `conda_build_config.yaml` — NumPy and compiler pins +- `run_tests.sh` / `run_tests.bat` — not used; conda-build runs the test + commands in `meta.yaml` -## Build configuration -- **Channels:** `https://software.repos.intel.com/python/conda`, `conda-forge` -- **Python versions:** 3.10, 3.11, 3.12, 3.13, 3.14 -- **Build system:** meson-python (via `python -m build`) -- **Compilers:** Intel C compiler (icx) -- **Dependencies:** mkl-devel, intel-openmp, dpcpp_{linux,win}-64, numpy - -## Build outputs -- Conda package: `mkl_umath--.conda` -- Platform-specific: `linux-64/`, `win-64/` - -## CI usage -- Built in `.github/workflows/conda-package.yml` -- Artifacts uploaded per Python version -- Test stage uses built artifacts from channel - -## Maintenance -- Keep `conda_build_config.yaml` in sync with CI matrix -- NumPy pin: must match Intel channel NumPy versions -- MKL version: track oneAPI releases +## Guardrails +- Treat recipe files as canonical for packaging intent and dependency pins. +- Keep recipe changes in step with `.github/workflows/conda-package.yml`, which + builds this recipe and runs the full test suite against the result. +- The package test in `meta.yaml` runs only `test_basic.py`. diff --git a/mkl_umath/AGENTS.md b/mkl_umath/AGENTS.md index d5bc8b1c..52ad710e 100644 --- a/mkl_umath/AGENTS.md +++ b/mkl_umath/AGENTS.md @@ -1,36 +1,23 @@ # AGENTS.md — mkl_umath/ -Core MKL-backed ufunc implementation: Python interface, Cython patching, and C/MKL integration. +Package sources: public API, patching entry points, and ufunc code generation. -## Structure -- `__init__.py` — public API surface (`_ufuncs`, `_patch_numpy`, version) -- `_init_helper.py` — module initialization helpers -- `_version.py` — version string (read dynamically by `meson.build`) -- `src/` — C implementation and Cython patch layer -- `tests/` — basic functionality and patching tests -- `generate_umath.py` — code generation for ufunc loops -- `generate_umath_doc.py` — docstring generation -- `ufunc_docstrings_numpy{1,2}.py` — NumPy version-specific docstrings +## Key files +- `__init__.py` — public API: the ufuncs from `_ufuncs` and the patching + functions from `_patch_numpy` +- `generate_umath.py` — generates `__umath_generated.c` in the build directory +- `patch.py`, `_patch_startup.py`, `with_patch.py`, `__main__.py` — persistent + (`.pth`) and one-shot patching behind `python -m mkl_umath` +- `_version.py` — the version; `meson.build` reads it +- `generate_umath_doc.py`, `ufunc_docstrings_numpy{1,2}.py` — docstring sources + adapted from NumPy; the build does not run them +- `src/` — loop templates and the two extensions +- `tests/` — test suite -## Patching API -```python -mkl_umath.patch_numpy_umath() # Replace NumPy loops with MKL -mkl_umath.restore_numpy_umath() # Restore original NumPy loops -mkl_umath.is_patched() # Check patch status -``` - -## Development guardrails -- **API stability:** Patching must be runtime-only, no NumPy source modification -- **Precision:** fp:precise, fimf-precision=high, fprotect-parens are non-negotiable -- **Compatibility:** Must work with upstream NumPy (NEP-36 compliance) -- **Testing:** Add tests to `tests/test_basic.py` for new ufuncs or patch behavior - -## Code generation -- `*.src` files are templates processed by `_vendored/conv_template.py` -- Generated files: `src/__umath_generated.c`, loop implementations -- Docstrings: dual NumPy 1.x/2.x support via separate docstring modules - -## Notes -- `_patch_numpy.pyx` is Cython; changes require Cython rebuild -- MKL VM loops in `src/mkl_umath_loops.c.src` -- `src/ufuncsmodule.c` — NumPy ufunc registration and dispatch +## Guardrails +- Use `patch_numpy_umath()` / `restore_numpy_umath()` in new code and docs; + `use_in_numpy()` and `restore()` are deprecated aliases. +- Keep patching reversible: anything installed can be uninstalled, and + `is_patched()` reports the truth. +- New modules must be listed in `py.install_sources` in `meson.build`, or they + are not installed. diff --git a/mkl_umath/src/AGENTS.md b/mkl_umath/src/AGENTS.md index c2b81013..50f0d975 100644 --- a/mkl_umath/src/AGENTS.md +++ b/mkl_umath/src/AGENTS.md @@ -1,40 +1,24 @@ # AGENTS.md — mkl_umath/src/ -C/Cython implementation layer: MKL VM integration, ufunc loops, and NumPy patching. +C and Cython sources for the ufunc loops and the patching extension. -## Core files -- **ufuncsmodule.c** — NumPy ufunc registration and module init -- **ufuncsmodule.h** — ufunc module public headers -- **mkl_umath_loops.c.src** — MKL VM loop implementations (template, ~60k LOC) -- **mkl_umath_loops.h.src** — loop function declarations (template) -- **_patch_numpy.pyx** — Cython patching layer (runtime NumPy loop replacement) -- **fast_loop_macros.h** — loop generation macros -- **blocking_utils.h** — blocking/chunking utilities for large arrays +## Key files +- `mkl_umath_loops.c.src`, `mkl_umath_loops.h.src` — loop templates, expanded + at build time by `_vendored/process_src_template.py` and compiled into the + `libmkl_umath_loops` shared library +- `ufuncsmodule.c` — the `_ufuncs` extension, built with the generated + `__umath_generated.c` +- `_patch_numpy.pyx` — the `_patch_numpy` extension; swaps loops into NumPy's + ufuncs with `PyUFunc_ReplaceLoopBySignature` and keeps the originals for + restore +- `fast_loop_macros.h`, `blocking_utils.h` — loop helpers -## Template system -- `.src` files are processed by `_vendored/conv_template.py` at build time -- Generates type-specialized loops for float32, float64, complex64, complex128 -- Pattern: `/**begin repeat ... end repeat**/` blocks - -## MKL VM integration -- Calls `vdSin`, `vsExp`, etc. from Intel MKL Vector Math (VM) -- Blocking strategy: chunk large arrays for cache efficiency -- Error handling: MKL VM status → NumPy error state - -## Patching mechanism (_patch_numpy.pyx) -- Cython extension exposing `patch_numpy_umath()`, `restore_numpy_umath()`, - `is_patched()` -- Replaces function pointers in NumPy's ufunc loop tables -- Thread-safe: guards against concurrent patching -- Reversible: stores original pointers for restoration - -## Build output -- `mkl_umath_loops.c` → shared library (libmkl_umath_loops.so/.dll) -- `_patch_numpy.pyx` → Python extension (_patch.*.so) -- `ufuncsmodule.c` + `__umath_generated.c` → `_ufuncs` extension - -## Development notes -- **Precision flags:** fp:precise, fimf-precision=high enforced in `meson.build` -- **Security:** Stack protections, FORTIFY_SOURCE enabled -- **Vectorization:** `-fveclib=SVML -fvectorize` for SIMD (Intel compiler only) -- **Optimization reports:** `-Dopt_report=true` meson option for `-qopt-report=3` +## Guardrails +- Edit the `.src` templates, not the generated `.c`/`.h`. +- Keep results consistent with NumPy for every dtype a loop handles, including + NaN and signed-zero handling. +- Keep the patch lock and the saved original loops so patching stays + thread-safe and reversible. +- Keep both extensions free-threading compatible: `freethreading_compatible=True` + in `_patch_numpy.pyx` and `Py_MOD_GIL_NOT_USED` in `ufuncsmodule.c`. +- Build flags, including precision and hardening flags, live in `meson.build`. diff --git a/mkl_umath/tests/AGENTS.md b/mkl_umath/tests/AGENTS.md index abc3d058..7ebe5716 100644 --- a/mkl_umath/tests/AGENTS.md +++ b/mkl_umath/tests/AGENTS.md @@ -1,29 +1,21 @@ # AGENTS.md — mkl_umath/tests/ -Unit tests for MKL-backed ufuncs and NumPy patching. +Test suite for the loops and patching. `meson.build` installs it with the +package. -## Test files -- **test_basic.py** — core functionality, numerical correctness -- **test_patching.py** — patching API and state transitions -- **test_cli.py** — `python -m mkl_umath` CLI patch install/uninstall/status +## Files +- `test_basic.py` — loop results compared against NumPy +- `test_patching.py` — patch and restore state, and the `mkl_umath()` context + manager +- `test_cli.py` — persistent patch install, uninstall, and status +- `test_freethreading.py` — concurrent ufunc use and patching; the GIL check + runs only on a free-threaded build -## Test coverage -- Ufunc correctness: compare MKL loops vs NumPy reference -- Patching: `patch_numpy_umath()`, `restore_numpy_umath()`, `is_patched()` state transitions -- Edge cases: NaN, Inf, empty arrays, large arrays -- Dtype coverage: float32, float64, complex64, complex128 +## Expectations +- Behavior changes include test updates in the same PR; bug fixes include a + regression test. +- Keep tests deterministic and free of timing assertions. -## Running tests -```bash -pytest mkl_umath/tests/ -``` - -## CI integration -- Tests run in conda-package.yml workflow -- Separate test jobs per Python version (3.10-3.14) -- Linux + Windows platforms - -## Adding tests -- New ufuncs → add to `test_basic.py` with NumPy reference comparison -- Patching behavior → test state transitions and thread safety -- Use `numpy.testing.assert_allclose` for floating-point comparisons +## Entry points +- `pytest mkl_umath/tests` from a checkout +- `pytest --pyargs mkl_umath` against an installed package From 9cefba6f04fa325875b9551dd44698aad6a4546f Mon Sep 17 00:00:00 2001 From: "Harlow, Jordan" Date: Fri, 9 Oct 2026 07:21:50 -0600 Subject: [PATCH 2/2] task: clean up CONTRIBUTING.md --- CONTRIBUTING.md | 130 +++++++++--------------------------------------- 1 file changed, 24 insertions(+), 106 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 42b9d299..aa803756 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,126 +1,44 @@ # Contributing to `mkl_umath` -This document covers the development workflow: how to get a working build, how -to run the checks, and what to include in a pull request. +See [README.md](README.md) for usage, [AGENTS.md](AGENTS.md) for a map of the +source tree, and [SECURITY.md](SECURITY.md) to report a vulnerability. -For end-user installation and usage, see [README.md](README.md). For a map of -the source tree, see [`AGENTS.md`](AGENTS.md), which links to the local -`AGENTS.md` files in directories that have their own rules. Security -vulnerabilities go through the process in [SECURITY.md](SECURITY.md). +## Setup ---- - -## Development setup - -Building requires a C compiler, oneMKL headers and libraries (`mkl-devel`), and -NumPy. CI builds with the Intel `icx` compiler and with upstream `clang`. On -Linux, a conda-forge environment provides all of it, including `icx`: +On Linux: ```sh -# add python=X.Y to target a specific interpreter -conda create -n mkl_umath-dev -c conda-forge --override-channels python pip \ - mkl-devel dpcpp_linux-64 numpy meson-python ninja cmake "cython>=3.1.0" pytest +conda create -n mkl_umath-dev -c conda-forge python pip mkl-devel \ + dpcpp_linux-64 numpy meson-python ninja cmake "cython>=3.1.0" pytest conda activate mkl_umath-dev -``` - -Then build in place, which reuses the environment's MKL and NumPy: - -```sh -CC=icx pip install -e . --no-build-isolation --no-deps --verbose \ +CC=icx pip install -e . --no-build-isolation --no-deps \ -Csetup-args=-Dmkl_threading=gnu_thread ``` -`gnu_thread` matches the conda-forge recipe. The default threading layer, -`intel_thread`, needs Intel's OpenMP runtime; against conda-forge's MKL, -`import mkl_umath` then fails with `undefined symbol: __atomic_compare_exchange`. -`meson.options` lists the other layers. - -`pyproject.toml` defines the supported Python range, and `.github/workflows/` is -canonical for the versions CI covers. `README.md` documents the non-editable -install paths, including the isolated build that resolves its own `mkl-devel` -and `numpy`. - -### Rebuilding +`gnu_thread` is required with conda-forge's MKL; the default threading layer +fails at import. -`meson-python` rebuilds the extensions on import for editable installs, so -editing `.pyx`, `.c.src`, `generate_umath.py`, or `meson.build` and rerunning -`pytest` is usually enough. Generated sources and the compiled extensions live -under `build//` rather than in the source tree. If a build gets into a bad -state, `rm -rf build` and reinstall. - -## Running the checks +## Checks ```sh -pytest mkl_umath/tests # test suite -pre-commit run --all-files # lint and format hooks +pytest mkl_umath/tests +pre-commit run --all-files ``` -To run a single test, use `pytest mkl_umath/tests/::`; to lint one -file, `pre-commit run --files `. - -Install the hooks once with `pre-commit install` and they run on each commit. -`.pre-commit-config.yaml` is the source of truth for the tooling. - -Opening a pull request also runs CI, which builds and tests the package across -platforms and Python versions and runs various lint and static-analysis checks. - -## Code style - -Style is loose, and the pre-commit hooks enforce most of it: - -- Python is formatted with `black` and `isort`, with a line length of 80. -- Cython is not touched by `black`. `isort` sorts its imports, `cython-lint` - checks it against the same 80-column limit, and string literals use double - quotes. -- C sources follow the repository's `.clang-format`. -- Otherwise, match the surrounding code. - -## Dos and don'ts - -**Do** - -- Keep changes atomic and single-purpose. -- Preserve NumPy behavior. Patching swaps these loops in for NumPy's own, so a - result that differs from stock NumPy, including NaN and signed-zero handling, - is a bug. Call out an intentional difference in the PR. -- Add tests in `mkl_umath/tests/` alongside behavior changes, and a regression - test with every bug fix. -- Keep tests deterministic. -- Edit the `*.src` templates in `mkl_umath/src/` and `generate_umath.py` for - loop changes. The C they produce is regenerated on every build. -- Keep patching reversible and observable: anything installed can be - uninstalled, and `is_patched()` reports the truth. -- Keep both extensions free-threading compatible. -- Cite the source-of-truth file for mutable details: `pyproject.toml`, - `meson.build`, `meson.options`, `conda-recipe*/meta.yaml`, - `.github/workflows/`. -- Give benchmark numbers reproducible context — hardware, versions, and the - command you ran. - -**Don't** - -- Commit generated artifacts, or hand-edit generated C. -- Remove or weaken the floating-point precision flags in `meson.build`. -- Hardcode versions, build flags, CI matrices, or channel URLs in documentation. -- Assert on timing or throughput in the test suite. -- Refactor `_vendored/` opportunistically. Keep local diffs minimal and send - fixes upstream where you can. -- Introduce ISA-specific assumptions outside explicit build configuration. +The pre-commit hooks enforce formatting; otherwise, match the surrounding code. -## Submitting a change +## Guidelines -Work on a branch: the `no-commit-to-branch` hook blocks direct commits to -`main` and `maintenance/*`. +- Keep changes small and focused. +- Match stock NumPy results, and call out any intentional difference. +- Add tests with behavior changes, and a regression test with bug fixes. +- Edit the `*.src` templates and `generate_umath.py`, not generated C. +- Keep the floating-point precision flags in `meson.build`. +- Keep patching reversible. -If the change is user-visible — behavior, API, packaging, or build output — add -a `CHANGELOG.md` entry under `## [dev]` in the matching section, with a -`[gh-NNN](https://github.com/IntelPython/mkl_umath/pull/NNN)` link. The format -follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project -follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Docs, -tooling, and CI-only changes are usually left out. +## Pull requests -Then open the PR and fill in the template, including what you verified locally -and what you left to CI. +Work on a branch and fill in the PR template. For user-visible changes, add a +`CHANGELOG.md` entry under `[dev]` with a `gh-NNN` link. -By contributing you agree that your contributions are licensed under the -BSD-3-Clause terms in [LICENSE.txt](LICENSE.txt). +Contributions are licensed under the terms in [LICENSE.txt](LICENSE.txt).