unstable tolerance in test_mesh.py - test_curve_mesh[10-2-float32] in MPI

[drew-parsons]

Debian CI testing at https://ci.debian.net/packages/i/io4dolfinx/testing/amd64/
is showing consistent failure of test_curve_mesh[10-2-float32] in test_mesh.py when running with complex numbers (PETSC_DIR=/usr/lib/petscdir/petsc-complex) and MPI (mpirun -n 2 pytest-3) on amd64 (x86_64)

e.g. https://ci.debian.net/packages/i/io4dolfinx/testing/amd64/69206750/

11373s =================================== FAILURES ===================================
11373s ________________________ test_curve_mesh[10-2-float32] _________________________
11373s 
11373s degree = 2, dtype = <class 'numpy.float32'>, R = 10
11373s 
11373s     @pytest.mark.parametrize("dtype", [np.float32, np.float64])
11373s     @pytest.mark.parametrize("degree", [2, 3])
11373s     @pytest.mark.parametrize("R", [0.1, 1, 10])
11373s     def test_curve_mesh(degree, dtype, R):
11373s         N = 8
11373s         mesh = dolfinx.mesh.create_rectangle(
11373s             MPI.COMM_WORLD,
11373s             [[-1, -1], [1, 1]],
11373s             [N, N],
11373s             diagonal=dolfinx.mesh.DiagonalType.crossed,
11373s             dtype=dtype,
11373s         )
11373s         org_area = dolfinx.fem.form(1 * ufl.dx(domain=mesh), dtype=dtype)
11373s     
11373s         curved_mesh = reconstruct_mesh(mesh, degree)
11373s     
11373s         def transform(x):
11373s             u = R * x[0] * np.sqrt(1 - (x[1] ** 2 / (2)))
11373s             v = R * x[1] * np.sqrt(1 - (x[0] ** 2 / (2)))
11373s             return np.asarray([u, v])
11373s     
11373s         curved_mesh.geometry.x[:, : curved_mesh.geometry.dim] = transform(curved_mesh.geometry.x.T).T
11373s     
11373s         area = dolfinx.fem.form(1 * ufl.dx(domain=curved_mesh), dtype=dtype)
11373s         circumference = dolfinx.fem.form(1 * ufl.ds(domain=curved_mesh), dtype=dtype)
11373s     
11373s         computed_area = curved_mesh.comm.allreduce(dolfinx.fem.assemble_scalar(area), op=MPI.SUM)
11373s         computed_circumference = curved_mesh.comm.allreduce(
11373s             dolfinx.fem.assemble_scalar(circumference), op=MPI.SUM
11373s         )
11373s     
11373s         tol = 10 * np.finfo(dtype).eps
11373s >       assert np.isclose(computed_area, np.pi * R**2, atol=tol)
11373s E       assert np.False_
11373s E        +  where np.False_ = <function isclose at 0x7f16ce5b3af0>(314.15594482421875, (3.141592653589793 * (10 ** 2)), atol=np.float32(1.1920929e-06))
11373s E        +    where <function isclose at 0x7f16ce5b3af0> = np.isclose
11373s E        +    and   3.141592653589793 = np.pi
11373s 
11373s tests/test_mesh.py:43: AssertionError
11373s =========================== short test summary info ============================
11373s FAILED tests/test_mesh.py::test_curve_mesh[10-2-float32] - assert np.False_
11373s ==== 1 failed, 1985 passed, 372 skipped, 1 deselected in 1615.57s (0:26:55) ====

I can't reproduce the failure locally, so it must be hardware dependent. Looks like the test would be happy if the tolerance was relaxed a little.

For the record it's passing on arm64,
https://ci.debian.net/packages/i/io4dolfinx/testing/arm64/69166374/

ppc64el and riscv64 passed once but failed different tests later.

[drew-parsons]

Apparently it's not a complex number issue. Seems to be some intermittency in this error.
It's also failing in MPI with real numbers
(in this test MPI with complex numbers is already skipping test_curve_mesh[10-2-float32])
https://ci.debian.net/data/autopkgtest/testing/amd64/i/io4dolfinx/69538459/log.gz