[RedTimesZero]

Description

This PR is from the NTHU team ( Team NTHU SCC ) participating in the SC26 Student Cluster Competition.
Replace the two blocking MPI_SENDRECV calls per direction in the halo exchange with one MPI_Isend/MPI_Irecv + MPI_Waitall, so a rank no longer waits for the first neighbour before starting the second side.

Fixes #2089

Changes

• s_mpi_sendrecv_variables_buffers is split into s_mpi_pack_halo_buffer, s_mpi_exchange_halo_buffers and s_mpi_unpack_halo_buffer, around a small halo_side descriptor (peer, tags, pack/unpack offsets, buffer offset).
• buff_send/buff_recv are split into two halves (side 2 starts at halo_size + 1). halo_size now uses integer arithmetic so the halves cannot overlap, and an @:ASSERT checks buffer_count <= halo_size + 1.
• Tags, packing/unpacking order and GPU_UPDATE staging are unchanged (now per half).
• The per-half GPU_UPDATE is done in two small helpers (s_mpi_halo_send_to_host, s_mpi_halo_recv_to_device) that take the offset and count as arguments. With the bounds as local variables of s_mpi_exchange_halo_buffers, NVHPC 26.3 computed a zero-length section under OpenMP offload (the runtime reported size=0, nothing moved) and the 2-rank tests failed with --gpu mp.
• m_boundary_common.fpp: comment only, documenting the call order (-1 first, then +1, same arguments).

Documentation: the "How to Extend MPI Halo Exchange" section of docs/documentation/contributing.md still describes the MPI_SENDRECV structure. I left it untouched in this PR; if the approach is accepted I am happy to update it, with whatever wording you prefer.

Testing

Check	Result
./mfc.sh test (full, --gpu acc, 1 H200, 2 ranks)	725 passed, 0 failed, 39 skipped
./mfc.sh test --gpu mp (NVHPC 26.3), 2 MPI Ranks	6 passed, 0 failed
./mfc.sh test --gpu mp, MPI Consistency	3 passed, 0 failed
Restart file after last step vs master, 2D shock droplet, 1/2/3/4 ranks, --gpu acc and --gpu mp	byte-identical
Same, 3D shock droplet, 3 and 4 ranks, --gpu acc and --gpu mp	byte-identical
./mfc.sh format	clean
gfortran 13.2 + Open MPI 5.0, CPU, simulation target	compiles; not run
Cray ftn, Intel ifx, AMD flang	not tested
Full test suite with --gpu mp	not run (only the multi-rank cases above)

Note: the test suite only exercises 2 ranks, so the pairwise path (two distinct neighbours in one direction) is covered by the shock-droplet comparisons above, not by ./mfc.sh test.

Performance

Grind time (ns/gp/eq/rhs), master -> this PR. Unmodified examples, one rank per H200 (one rank per node), --case-optimization, --mpi, no other options.

Backend	Case	Ranks	master	this PR	Change
OpenACC	2D shock droplet	1	0.5846	0.5841	-0.1%
OpenACC	2D shock droplet	2	0.3971	0.3986	+0.4%
OpenACC	2D shock droplet	3	0.3986	0.3706	-7.0%
OpenACC	2D shock droplet	4	0.3688	0.3449	-6.5%
OpenACC	3D shock droplet	3	0.2524	0.2328	-7.8%
OpenACC	3D shock droplet	4	0.2190	0.1984	-9.4%
OpenMP	2D shock droplet	1	0.5859	0.5859	+0.0%
OpenMP	2D shock droplet	2	0.4159	0.4165	+0.1%
OpenMP	2D shock droplet	3	0.3885	0.3738	-3.8%
OpenMP	2D shock droplet	4	1.0467	1.0341	-1.2%
OpenMP	3D shock droplet	3	0.2436	0.2179	-10.5%
OpenMP	3D shock droplet	4	0.2145	0.1890	-11.9%

Notes:

• How measured: master and the proposed version run alternately inside the same Slurm allocation (same nodes), 5 runs each for 2D and 3 runs each for 3D; values are means. Absolute grind times differ between node batches, so only the within-job change is meaningful.
• Why 3D: the 2D case only exercises the x and y directions. The 3D case also exercises the z direction and the 3D branch of the halo_size computation.
• When it helps: the gain needs at least 3 ranks in a direction (or a periodic boundary). With 1-2 ranks both sides go to the same neighbour, so there is nothing to overlap; the measured change there is within -0.1% to +0.4%.
• OpenMP offload: the per-side GPU_UPDATE section bounds are passed to small helper routines as arguments; with local variables NVHPC 26.3 computed a zero-length section under --gpu mp. Before this fix the multi-rank OpenMP runs produced NaN.
• Output: final restart files are byte-identical to master in every configuration above.
• Absolute values: the examples are used unmodified, so the grids are small. Compare the relative change, not the absolute values or values across backends.

Contribution Policy

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

[github-actions]

Claude Code Review

Head SHA: fb5c8d9

Files changed:

• 2
• src/common/m_boundary_common.fpp
• src/common/m_mpi_common.fpp

Findings:
No issues found. Checked in step 4: src/common/m_boundary_common.fpp (context file) and the repo copy of src/common/m_mpi_common.fpp (no prepared context file, so I read it in the working tree). In the paired path, the send and receive tags and the source and destination ranks match across both sides. Each side uses its own half of buff_send and buff_recv, and the allocation of 2*halo_size+2 covers both halves. halo_size is integer(8), so the integer rounding matches the old nint() behavior. Non-RDMA host and device updates cover the right sub-ranges.

[codecov]

Codecov Report

❌ Patch coverage is 60.86957% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.92%. Comparing base (47568a0) to head (fb5c8d9).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/common/m_mpi_common.fpp	60.86%	25 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2090      +/-   ##
==========================================
+ Coverage   61.90%   61.92%   +0.02%     
==========================================
  Files          86       86              
  Lines       22885    22913      +28     
  Branches     3369     3372       +3     
==========================================
+ Hits        14166    14190      +24     
- Misses       6220     6222       +2     
- Partials     2499     2501       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wilfonba]

@RedTimesZero A maintainer will mark your pull request as ready for review when it is suitable to do so

[RedTimesZero]

Thanks for the note, and sorry for marking it ready myself. I'll leave it as a draft and won't change its state in future PRs either.

The Phoenix (NVHPC) gpu-omp job shows MPI-related test failures in this PR. I'm looking into it and will push a fix before asking for review.

[github-actions]

Lines of Code

File	Lines	Diff
src/common/m_mpi_common.fpp	1575	+81

Directory	Lines	Diff
common	10607	+81
total	47666	+81