A permutation test that is subtly wrong does not fail - it gives you a confidently wrong answer. So the test suite is not decoration, and it is written to establish things rather than assert them.
pip install -e ".[dev]"
pytest -q # 53 checks| file | what it establishes |
|---|---|
test_exactness.py |
The max-tree really is the exact TFCE integral. |
test_matlab_parity.py |
The Python binding is bit-identical to the MATLAB mex. |
test_tails.py |
The Gamma and Pareto fits beat the counting they replace. |
test_glm.py |
The accelerated GLM returns what an honest refit returns. |
test_core.py |
Shapes, adjacency, and the things that must be refused. |
test_nilearn_compat.py |
The drop-in honours nilearn's contract. |
test_docs.py |
The worked examples in the docs actually run. |
Anyone can say their transform is exact. The check here compares it against an independent
implementation - the naive one, which steps the height and labels components with
scipy.ndimage.label - and requires that reference to converge onto the max-tree at first order
as the step size shrinks.
That is the whole argument. A Riemann sum has an error of order dh, so if the max-tree were merely
another approximation, the two would converge onto each other at some other rate, or onto nothing.
The criterion is the slope of log(error) against log(n_steps), which must be about −1.
Testing the ratio of successive errors instead would be far too noisy, because the error is a maximum over elements.
This is not a reimplementation - it is the same C with a different caller - so anything short of bit-identical output means a binding bug: a transposed volume, a face list read 0-based, a float silently promoted. The fixture was generated by running the MATLAB mex-files over volumes, surfaces and batches, and stores what they returned.
Regenerating it needs MATLAB. Running the test does not - which is the point: CI holds the Python to the MATLAB's answer without MATLAB being anywhere near it.
The honest question about a tail approximation is never "is it exact" - extrapolating a tail from a hundred points cannot be - but "does it beat counting". So the checks build a real permutation distribution from 20,000 permutations, show the fit only the first 1000, and score both the fit and plain counting against the truth.
The fit has to be unbiased, has to land within a factor of two of the truth at least as often as
counting does, and must never return zero - which counting does constantly once the true p drops
below 1/n_perm.
test_out_of_range_vertex_is_refused
test_zero_based_faces_are_refusedThese are not politeness. An out-of-range vertex index used to be written straight past the end of the adjacency arrays - a heap corruption that took the whole process down somewhere else entirely, long after the fact. GIFTI faces are 1-based; handing 0-based faces over is a mistake, not a mode.
Documentation that is not run rots, and a worked example that does not work is worse than none at all
- it is a trap for the one reader who trusts it. So the pipelines in
permutation.md are run in
test_docs.py, on data with a planted effect, and are required to find it while controlling the false-positive rate.
Worth being explicit about, because the gaps are where the risk is.
- Real data. Everything here is synthetic phantoms with known ground truth.
- Exchangeability. This package hands you the GLM and the transform and trusts you to permute correctly. It does not check your design, and it cannot. The MATLAB toolbox does check itself at runtime; this one does not. See the warning in permutation.md.
- Comparison against PALM and FSL
randomise. - The design layer - there isn't one yet. See the status note in the README.
The MATLAB toolbox ships a larger suite (106 checks), which covers the things that only exist there: the three nuisance methods, exchangeability blocks, the half-permutation shortcut, voxel-wise covariates and sequential stopping. See matlab/validation/.