Debug ibm stability

[danieljvickers]

There has been a growing amount of technical debt on the immersed boundary code for multi-rank cases since the introduction of the IB neighborhoods. This has led to a host of new potential issues that threaten the stability of simulations being run. As I recently began scaling a relatively-difficult case, in terms of opportunities for instability, I made multiple bugfixes that were latent and untested. Some were relatively innocuous and others were extremely problematic, but all obvious bugs. And explanation of the changes are as follows

1. Added a mask to the interior of immersed boundaries for CFL assessment. Since the values interior to the IBs are meant to be non-physical, using them to check CFL causes violations that otherwise are not problematic. Particularly, the interior can produce odd pressures, which affect the speed of sound computed at the grid cell in question. This has been resolved with a mask during CFL checking.
2. Fixed race condition in IB marker write. The multi-IB parallelism that was introduced also created a race condition for actively-colliding particles. Two IBs could try to claim the same grid cell, but nothing deconflicted this race. In the other IB parallelism, we have left the behavior that the IB with the highest global index claims the grid cell, which I mirrored in the many-ib parallelism state
3. Fixed race condition in ghost-point correction. When two spheres are close together, a ghost point on sphere A may sample an image point from the interior of sphere B, which is also a ghost point. Because the ghost points are written in parallel, this generated a race condition. To deconflict this, I have separated the subroutine into two kernel calls. One kernel interpolates the image points and the following updates the ghost point values. This prevents one ghost point being updated before another image point may be interpolated. The initial pass I had Claude write is somewhat clunky, and I will have to touch it up by hand before the merge. But it works and resolves the race.
4. Fixed non-updated num_ibs. The num_ibs is computed in the ib ownership handoff subroutine, but was never updated to the GPU. This means that if new particles entered the neighborhood, then several subroutines would not iterate over them, including the state integration. This caused cross-rank drift and several issues with integrating the forces themselves. I originally thought that this was an error in the updating of the global IB index lookup, which caused me to make changes there that I should revert before merge.
5. Fixed Latent out-of-bounds read in mibm central-difference IB drag (s_compute_viscous_stress_tensor) for boundary-adjacent bodies #1633 out of bounds memory read when integrating the drag coefficeint of IBs near processor boundaries. Details in that issue. The fix I opted for was to extend the integration rather than use one-sided integrals. This keeps results consistent in multi-rank cases.
6. Fix non-updated num_gps, for similar reasons to num_ibs. If the number of ghost points on a rank goes down, then the update loop will update some grid cell that should no longer be interior to an IB, causing potential instabilities in fast-moving IBs where large discontinuities can occur as grid cells are ejected.

The most impactful bug fixes were the update of num_ibs and num_gps, and the checking of interior GP points with a mask during CFL. Current tests show total stability on 8 ranks with 600 IBs that are very light compared to the ambient fluid in a mach 10 shock. Assuming these results hold, then we should have much higher stability, even in non-physically significant regimes. I am currently working on extending this result to higher numbers of IBs and ranks.

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

PR template credit: junegunn

[github-actions]

Claude Code Review

Head SHA: 88877c1

Files changed:

• 10
• src/common/m_constants.fpp
• src/common/m_derived_types.fpp
• src/simulation/m_data_output.fpp
• src/simulation/m_global_parameters.fpp
• src/simulation/m_ib_patches.fpp
• src/simulation/m_ibm.fpp
• src/simulation/m_mpi_proxy.fpp
• src/simulation/m_particle_cloud.fpp
• src/simulation/m_start_up.fpp
• src/simulation/m_time_steppers.fpp

Findings:

• src/simulation/m_ibm.fpp: The PR wires in "TEMPORARY DEBUG INSTRUMENTATION" that is not gated behind any build/debug flag and runs in every production build. s_debug_log_ib_divergence is called unconditionally from m_time_steppers.fpp whenever moving_immersed_boundary_flag is set, and once it detects 2 distinct timesteps where cross-rank IB state disagrees it calls call s_mpi_abort(...) — any real moving-IB, multi-rank run that hits this condition (which the comments imply is a currently-unresolved, reproducible bug) will hard-abort. This must not ship in a merged PR.
• src/simulation/m_ibm.fpp (s_communicate_ib_forces, both ACCUM and BACKPROP directions): the GPU offload directives ($:GPU_PARALLEL_LOOP / $:END_GPU_PARALLEL_LOOP / $:GPU_UPDATE) around the force/torque pack and unpack loops have been commented out "to test whether it's the source of the ... corruption bug", replaced by per-send/per-recv file open/write/close calls gated on trace_active. This disables GPU acceleration for IB force communication and adds blocking file I/O inside the MPI exchange hot path — a correctness-neutral but severe performance regression, and inconsistent with this same PR raising num_local_ibs_max/num_ib_patches_max_namelist 4x (the "these loops are tiny, no perf concern" rationale no longer holds at that scale).
• src/simulation/m_ibm.fpp: new module-level integer, save :: dbg_t_step, dbg_divergence_count, dbg_last_divergent_t_step variables are global mutable state shared across calls/ranks, which CLAUDE.md's Critical Rules explicitly forbid ("NEVER ... global save variables"). They also make s_compute_ib_forces/s_communicate_ib_forces implicitly stateful across timesteps.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_data_output.fpp	1432	+115
src/simulation/m_ibm.fpp	1307	+58
src/simulation/m_start_up.fpp	1277	+33
src/simulation/m_ib_patches.fpp	559	+19
src/simulation/m_particle_cloud.fpp	430	+19
src/common/m_derived_types.fpp	440	+1
src/simulation/m_global_parameters.fpp	748	+1
src/simulation/m_mpi_proxy.fpp	524	+1

Directory	Lines	Diff
common	9779	+1
simulation	28264	+246
total	45876	+247