Merge stochastic physics (SPPT only) into GSL's develop branch

[gsketefian]

This PR enables use of the SPPT scheme of stochastic physics into MPAS-Model. This is addressed in Issue #204.

Main changes:

1. Bring in the stochastic_physics code as a submodule.
2. Make changes to module files, Makefiles, registry xml files, and fortran files, and GitHub workflow files to allow the SPPT stochastic scheme to be used with MPAS.
3. Adds the stochastic_physics code as a submodule. There is a companion PR into the authoritative stochastic_physics repo here.

This PR adds a new GitHub Actions CI workflow (run_mpas_stoch.yml) that runs two tests of MPAS-Model on a winter case using GFS ICs/LBCs and with SPPT enabled. The first test is the "convection_permitting" physics suite, and the second is with the "hrrrv5" suite. These two tests use the same grid, static file, ICs, LBCs, etc as the existing CI test(s) for the winter case with GFS ICs/LBCs, etc. These tests require the setup (namelist, streams, etc) files located in the directories

MPAS-Model/testing_and_setup/ufs-community/cases/ufscommunity.convection_permitting.gfs.winter.stoch
MPAS-Model/testing_and_setup/ufs-community/cases/ufscommunity.hrrrv5.gfs.winter.stoch

Mandatory Questions

• Does this PR include any additions or changes to external inputs (e.g., microphysics lookup tables, static data for gravity-wave drag -- things like that)?
  • No. External data files are not added, but the new CI tests do require the namelist, stream, etc files at MPAS-Model/testing_and_setup/ufs-community/cases/ufscommunity.[convection_permitting|hrrrv5].gfs.winter.stoch, which are under version control.
• Does this PR require updating one or more baselines for the CI tests? If so, what?
  • No.

Priority Reviewers

• Please list the developers/collaborators you'd like to prioritize for review
  • @NingWang325
  • @willmayfield
  • @JeffBeck-NOAA
  • @dustinswales

[guoqing-noaa]

why we need to add ${CMAKE_CURRENT_BINARY_DIR}?

[JeffBeck-NOAA]

@gsketefian or @NingWang325, is this one you can answer?

[gsketefian]

I'm working on remembering the details of why I needed this.

[gsketefian]

@guoqing-noaa Please see my reply here.

[Copilot]

Pull request overview

This PR integrates the NOAA-PSL stochastic_physics package (SPPT scheme only) into MPAS-Model as a git submodule, plumbing the perturbation pattern generation and application into the atmosphere core's init/timestep/RK3 paths and the registry. It also adds a dedicated CI workflow that runs convection_permitting and hrrrv5 winter GFS cases with SPPT enabled, plus build-system support (MKL/LAPACK, NetCDF-Fortran) on several HPC modulefiles and CI images.

Changes:

• Add stochastic_physics as a submodule with corresponding Make/CMake/Registry wiring and SPPT call sites in atm_core_init, atm_do_timestep, and atm_srk3.
• New CI workflow run_mpas_stoch.yml and new test case directories ufscommunity.{convection_permitting,hrrrv5}.gfs.winter.stoch (namelists, streams, stream_lists).
• Build environment updates: LAPACK/BLAS/MKL on modulefiles and CI, NetCDF-Fortran setup, MPAS_USE_MPI_F08 always defined, registry-processed-file paths made absolute in CMake.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
testing_and_setup/ufs-community/cases/ufscommunity.hrrrv5.gfs.winter.stoch/*	New SPPT test case files (namelist, streams, stream lists) for the hrrrv5 suite.
testing_and_setup/ufs-community/cases/ufscommunity.convection_permitting.gfs.winter.stoch/*	New SPPT test case files for the convection_permitting suite.
src/Makefile	Adds $(LAPACK_LIBS) to the link line; introduces an esmf_time_f91 typo in the -I path.
src/core_atmosphere/Registry.xml	Adds Gaussian-grid dims (hard-coded 504/248) and new stoch_pattern_* variables; includes stochastic_physics registry.
src/core_atmosphere/mpas_atm_core.F	Wires stochastic_physics_pattern_init and _adv into init and timestep paths.
src/core_atmosphere/Makefile	Builds and links the stochastic_physics library; reworks post_build to back up existing default_inputs.
src/core_atmosphere/dynamics/mpas_atm_time_integration.F	Calls stochastic_physics_pattern_apply for 'phys' and 'prog' tendencies in atm_srk3.
src/core_atmosphere/dynamics/Makefile	Adds -I../stochastic_physics include path.
src/core_atmosphere/CMakeLists.txt	Defines a stochastic_physics CMake target with BLAS/NetCDF deps and links it into core_atmosphere.
modulefiles/mpas/{derecho,gaeac6,hera,orion,ursa}.intel.lua	Loads Intel MKL and exports NetCDF C/Fortran root env vars.
Makefile	Adds LAPACK_LIBS to relevant build targets; adds new intel-mpi-gaeac6 build target.
CMakeLists.txt	Always defines MPAS_USE_MPI_F08; adds explicit dependency to serialize init_atmosphere on core_atmosphere builds.
cmake/Functions/MPAS_Functions.cmake	Uses absolute ${CMAKE_CURRENT_BINARY_DIR} paths for Registry_processed.xml dependencies.
.gitmodules	Adds src/core_atmosphere/stochastic_physics submodule pointing at dtcenter fork.
.github/workflows/run_mpas.yml, run_mpas_hrrr.yml, build_mpas.yml, build_mpas_intel.yml	Add LAPACK/BLAS, NetCDF setup, and MKL support; minor commented-out trigger lines.
.github/workflows/run_mpas_stoch.yml	New CI workflow that runs baseline vs feature SPPT-enabled MPAS and compares outputs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gsketefian]

I noticed this as well but thought it might be intentional by @NingWang325. Kind of surprising that it works! @NingWang325 can you say if it's intentional or a typo?

[gsketefian]

Turns out the -I flag is not needed at all in this command, so I removed it. See my comment here.

[joeolson42]

Without full comprehension of what these variables are used for, I think Copilot has some valid points here.

[joeolson42]

My browser quit on me so I don't know if this will be a duplicate comments or not... Without fully understanding what these variables are for, it seems Copilot has some good points here.

[JeffBeck-NOAA]

From @NingWang325, these arrays are only used for debugging. These fields aren't being used otherwise. They can be removed.

[JeffBeck-NOAA]

From @NingWang325: this code follows procedure from NOAA-PSL/stochastic_physics to ensure initialization is successful. For now, a message is passed to the log saying that stochastic physics could not be initialized.

We need to stop the run in NOAA-PSL/stochastic_physics if initialization of any expect stochastic physics method fails. Check the "ierr" value and issue a stop command.

[clark-evans]

@gsketefian @NingWang325 @JeffBeck-NOAA I turned Copilot loose on a review of this PR. It generated five comments, the first few of which (at minimum) look like things that need to be addressed before merge.

[joeolson42]

just curious, why "nam_stochy"?

[gsketefian]

@joeolson42 I think this is what the stochastic physics namelist in the FV3 is called, so we just kept it. My guess as to its meaning: "nam" = namelist, "stochy" = "stochastic". Not sure about the "y" in "stochy". I can't find any explanation in the inline comments in the stochastic_physics code. It might have French origins, since I think there were some French developers working on the code at some point. I'm curious too, so I'll ask @pjpegion in the companion PR #97 into the NOAA-PSL/stochastic_physics repo.

Here's the link to my question.

[gsketefian]

@joeolson42 Sorry, turns out @pjpegion doesn't know the answer regarding the "y" in "stochy".

I’d personally prefer a more straightforward name like stoch_phys or stoch_physics for the namelist, though it isn't a strong preference.

[joeolson42]

I'd prefer something like "stoch_physics" too, but since it's just a preference, it can't be considered a mandatory change. Maybe let others weigh in.

[gsketefian]

@joeolson42 I'm still curious about the "y" in "stochy", especially since it's used not just in "nam_stochy" but also all throughout the stochastic_physics code. I had claude look at that code to see if it can come up with an explanation for the "y". It said:

"stochy" reads as an adjectival nickname — like "patchy" or "scratchy" — a natural informal shorthand for "stochastic" that is also more distinctive as an identifier prefix than the bare stem "stoch".

I'm content with that.

[dustinswales]

@gsketefian Should this die and report error if initialization fails?

[gsketefian]

@dustinswales I would think it should at least write out a warning in the log file(s) if not die altogether, but I think @NingWang325 is best suited to answer this one.

[guoqing-noaa]

Lines 128-134: Is this a new requirement due to the stochastic physics?
I don't think we need this dependency before.

[joeolson42]

Conditionally approved, pending any changes needed to address comments.

[dustinswales]

@gsketefian Can you sync this branch with the noaa/develop branch?

[guoqing-noaa]

Lines 217-234:
Did we get any troubles without these newly added ${CMAKE_CURRENT_BINARY_DIR} etc?

If not, I would suggest removing all the newly added ${CMAKE_CURRENT_BINARY_DIR} etc to keep sync'ed with the upstream NCAR's MPAS-Dev/MPAS-Model as much as possible.

[gsketefian]

@guoqing-noaa It took some time to remember why I made these changes. The short answer to your question is that, yes, we did run into trouble without these explicit path additions.

The error does not show up if you're only doing cmake builds. It shows up when you first do a build with the make system that NCAR has (i.e. using the Makefiles that are part of the repo) and THEN do a build with the cmake system. Although this is not what most people would be doing (they'd use either make or cmake), I for one would prefer having make and cmake builds that don't interfere with each other.

I'll describe what happens when building core_init_atmosphere, but the same applies to core_atmosphere.

Assume you first run a successful make build. This will generate the following file (among many others):

.../MPAS-Model/src/core_init_atmosphere/Registry_processed.xml

The way MPAS_Functions.cmake is currently written (i.e. with relative paths) causes the DEPENDS section of the "Parse Registry" build rule, which is given by

    add_custom_command(
            OUTPUT ${ARG_INCLUDES}
            COMMAND mpas_parse_${ARG_CORE} Registry_processed.xml ${CPP_EXTRA_FLAGS}
            COMMENT "CORE ${ARG_CORE}: Parse Registry"
            DEPENDS mpas_parse_${ARG_CORE} Registry_processed.xml)

to erroneously detect the Registry_processed.xml file in .../MPAS-Model/src/core_init_atmosphere created by the make build. That's because in the DEPENDS, only a relative path is given (just Registry_processed.xml), and apparently cmake's behavior for DEPENDS is to first try to resolve this with respect to the source directory CMAKE_CURRENT_SOURCE_DIR, given by

CMAKE_CURRENT_SOURCE_DIR = .../MPAS-Model/src/core_init_atmosphere/

and only if that fails to then try to resolve it with respect to the current cmake build (binary) directory CMAKE_CURRENT_BINARY_DIR, given by

CMAKE_CURRENT_BINARY_DIR = .../MPAS-Model/build/src/core_init_atmosphere/

Since the initial successful make build creates the file ${CMAKE_CURRENT_SOURCE_DIR}/Registry_processed.xml, when one then runs the cmake build the DEPENDS in the "Parse Registry" rule is satisfied, which means cmake does not establish a dependency of the "Parse Registry" rule on the "Pre-Process Registry" rule that is given by

    add_custom_command(
            OUTPUT Registry_processed.xml
            COMMAND ${CPP_EXECUTABLE} -E -P ${CPP_EXTRA_FLAGS} ${CMAKE_CURRENT_SOURCE_DIR}/Registry.xml > Registry_processed.xml
            COMMENT "CORE ${ARG_CORE}: Pre-Process Registry"
            DEPENDS Registry.xml)

Thus, "Parse Registry" runs immediately without waiting for "Pre-Process Registry" to complete. Since "Pre-Process Registry" hasn't yet completed, its output file .../MPAS-Model/build/src/core_init_atmosphere/Registry_processed.xml doesn't yet exist [note that cmake behavior for the COMMAND (and OUTPUT) sections of add_custom_command() is to resolve relative paths using CMAKE_CURRENT_BINARY_DIR, so the "Pre-Process Registry" rule above will place the Registry_processed.xml file it generates in CMAKE_CURRENT_BINARY_DIR, which in this case is .../MPAS-Model/build/src/core_init_atmosphere/]. Since that's the file that "Parse Registry" is looking for as input and it is missing, "Parse Registry" fails with a segfault.

The fix here is to make sure that the DEPENDS in "Parse Registry" checks for Registry_processed.xml ONLY in CMAKE_CURRENT_BINARY_DIR (and never in CMAKE_CURRENT_SOURCE_DIR). The straightforward way to do that is to specify the absolute path in the DEPENDS by adding CMAKE_CURRENT_BINARY_DIR in front of Registry_processed.xml. That's one of the changes made in this PR.

More generally, to get a more robust build configuration that avoids relying on cmake's internal rules for resolving relative paths (which can be complex and obscure), it is best to just use absolute paths in all parts of these add_custom_command() commands. That's what the changes here are doing.

[guoqing-noaa]

@gsketefian Thanks a lot for finding out the original intention and providing the details of this change!
I did get the same errors if I did the "make" build first and then did the "cmake" build without cleaning.

However, if we run make clean CORE=atmosphere and/or make clean CORE=init_atmosphere before doing "cmake", we are good.

Now it goes to different preferences by different views. Usually, we will let cmake find the correct path/file without explicitly specifying ${CMAKE_CURRENT_BINARY_DIR}. This is consistent with the upstream NCAR repo.

If users does "make" first and then gets errors at doing "cmake", we can just suggest these users do clean first, just like we usually do before lauching a new round of make.

[gsketefian]

@gsketefian Can you sync this branch with the noaa/develop branch?

I merged noaa/develop into the PR branch. But I still have a couple of issues to work through.