Support multiple EESSI versions by using ReFrame environments by casparvl · Pull Request #326 · EESSI/test-suite

casparvl · 2026-03-12T15:47:07Z

Add initial functionality to schedule test for all EESSI environments in a single run, by configuring ReFrame environments and using ReFrame's own find_modules function.

For now, these work:

reframe -c test-suite/eessi/testsuite/tests/apps/gromacs.py --list -t CI -t 1_node
...
[List of matched checks]
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /ec5fea98
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /0149252b
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /7f86cc73
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /b44c4f26
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2025.06', 'GROMACS/2025.2-foss-2025a') /10fe4b84
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /8575f1af
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /fa197803
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /848b3575
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /b4a91b63
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'GROMACS/2025.2-foss-2025a') /42464906
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=gpu %scale=1_node %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b-CUDA-12.4.0') /91ff3a13
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=gpu %scale=1_node %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b-CUDA-12.4.0') /0aadbdf1
Found 12 check(s)

reframe -c test-suite/eessi/testsuite/tests/apps/numpy/numpy.py --list -t 1_8_node
...
[List of matched checks]
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.02-gfbf-2022b') /d008c385
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.07-gfbf-2023a') /0b69130e
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.11-gfbf-2023b') /794e1d9e
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:rome', 'EESSI-2025.06', 'SciPy-bundle/2024.05-gfbf-2024a') /a29dae40
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:rome', 'EESSI-2025.06', 'SciPy-bundle/2025.06-gfbf-2025a') /e2755723
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.02-gfbf-2022b') /8d434ed0
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.07-gfbf-2023a') /ec49d0f6
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.11-gfbf-2023b') /e2523a64
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'SciPy-bundle/2024.05-gfbf-2024a') /1b4c0e79
- EESSI_NumPy %scale=1_8_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'SciPy-bundle/2025.06-gfbf-2025a') /5dccf227
Found 10 check(s)

Still need to work on the other tests.

… in a single run, by configuring ReFrame environments and using ReFrame's own find_modules function

casparvl · 2026-03-12T15:56:12Z

Ok, I now simply wrap the find_modules function from ReFrame in our own utils. The only change now required in the tests is that the return should define module_info instead of module_name.

…ct instead

casparvl · 2026-03-12T16:03:35Z

Oh, this broke name_only of course:

WARNING: skipping test file 'test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py': type error: find_modules() got an unexpected keyword argument 'name_only':
| /gpfs/home4/casparl/EESSI/test-suite/eessi/testsuite/tests/apps/openfoam/openfoam.py:90
|     module_info = parameter(find_modules('OpenFOAM/v', name_only=False))

that's something we might want to fix if we do the wrapped thing (if it's easy...)

casparvl · 2026-03-12T16:04:30Z

Metallwalls should also be adapted, since I think it doesn't use EESSI_mixin (?):

WARNING: skipping test 'EESSI_MetalWalls_MW': attribute error: 'EESSI_MetalWalls_MW' object has no attribute 'module_name':
| /gpfs/home4/casparl/EESSI/test-suite/eessi/testsuite/hooks.py:651
|     if not test.module_name:
 (rerun with '-v' for a backtrace)

casparvl · 2026-03-12T16:07:21Z

For me:

reframe -c test-suite/eessi/testsuite/tests/apps/ -t CI -t 1_node --list
...
- EESSI_QuantumESPRESSO_PW %ecut=50 %nbnd=10 %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'QuantumESPRESSO/7.2-foss-2022b') %device_type=cpu /1a10c479
- EESSI_QuantumESPRESSO_PW %ecut=50 %nbnd=10 %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'QuantumESPRESSO/7.3.1-foss-2023a') %device_type=cpu /41f20edb
- EESSI_QuantumESPRESSO_PW %ecut=50 %nbnd=10 %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'QuantumESPRESSO/7.2-foss-2022b') %device_type=cpu /6943f12d
- EESSI_QuantumESPRESSO_PW %ecut=50 %nbnd=10 %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'QuantumESPRESSO/7.3.1-foss-2023a') %device_type=cpu /751de4da
- EESSI_CP2K %scale=1_node %benchmark_info=QS/H2O-32 %module_info=('snellius:rome', 'EESSI-2023.06', 'CP2K/2023.1-foss-2023a') /cb3afdc7
- EESSI_CP2K %scale=1_node %benchmark_info=QS/H2O-32 %module_info=('snellius:rome', 'EESSI-2023.06', 'Libint/2.7.2-GCC-12.3.0-lmax-6-cp2k') /eb966eda
- EESSI_CP2K %scale=1_node %benchmark_info=QS/H2O-32 %module_info=('snellius:genoa', 'EESSI-2023.06', 'CP2K/2023.1-foss-2023a') /db55acb8
- EESSI_CP2K %scale=1_node %benchmark_info=QS/H2O-32 %module_info=('snellius:genoa', 'EESSI-2023.06', 'Libint/2.7.2-GCC-12.3.0-lmax-6-cp2k') /fd4dc36f
- EESSI_LPC3D %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'LPC3D/0.1.2-foss-2023a') /cca8ba7c
- EESSI_LPC3D %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'LPC3D/0.1.2-foss-2023a') /1f8c34a8
- EESSI_NumPy %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.02-gfbf-2022b') /5f66d2a4
- EESSI_NumPy %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.07-gfbf-2023a') /74180bd1
- EESSI_NumPy %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'SciPy-bundle/2023.11-gfbf-2023b') /b13eb6e7
- EESSI_NumPy %scale=1_node %module_info=('snellius:rome', 'EESSI-2025.06', 'SciPy-bundle/2024.05-gfbf-2024a') /4198937c
- EESSI_NumPy %scale=1_node %module_info=('snellius:rome', 'EESSI-2025.06', 'SciPy-bundle/2025.06-gfbf-2025a') /001c65e5
- EESSI_NumPy %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.02-gfbf-2022b') /2406ceda
- EESSI_NumPy %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.07-gfbf-2023a') /848c2beb
- EESSI_NumPy %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'SciPy-bundle/2023.11-gfbf-2023b') /87bf79d8
- EESSI_NumPy %scale=1_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'SciPy-bundle/2024.05-gfbf-2024a') /4d49b0c2
- EESSI_NumPy %scale=1_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'SciPy-bundle/2025.06-gfbf-2025a') /6c960caf
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node /28114225
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node /2985e32f
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:rome', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node /66e93c7a
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node /d3089aa5
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node /84bb5554
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:genoa', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node /087941ec
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node /f273f6d3
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node /aacbc5b0
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:rome', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node /ad4c59aa
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node /6d239a22
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node /943cd266
- EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:genoa', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node /20052531
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node /072871b6
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node /afc1e049
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_A100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node /1d758e9d
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node /8544f946
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node /bcde6240
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_bw %module_info=('snellius:gpu_H100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node /35f1f357
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node /4c2620bf
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node /d12354ac
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_A100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node /f54a219f
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node /ebd15104
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node /e11d33eb
- EESSI_OSU_pt2pt_GPU %benchmark_info=mpi.pt2pt.osu_latency %module_info=('snellius:gpu_H100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node /d6c59637
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node %device_type=cpu /0302bc4a
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node %device_type=cpu /b5209e9a
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:rome', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node %device_type=cpu /63912bd7
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node %device_type=cpu /8b6b78cf
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node %device_type=cpu /a8297e2c
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:genoa', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node %device_type=cpu /28747c55
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node %device_type=gpu /f27f47b5
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node %device_type=gpu /dc6c60d9
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_A100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node %device_type=gpu /ca3b2c97
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node %device_type=gpu /0dc81150
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node %device_type=gpu /07afffdd
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_info=('snellius:gpu_H100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node %device_type=gpu /641920a7
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node %device_type=cpu /7725a99a
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:rome', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node %device_type=cpu /806d6d5f
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:rome', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node %device_type=cpu /ea33fdf4
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.1-1-gompi-2023a') %scale=1_node %device_type=cpu /e84aa818
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:genoa', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023b') %scale=1_node %device_type=cpu /77b25adf
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:genoa', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2025a') %scale=1_node %device_type=cpu /baa4f778
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node %device_type=gpu /7bd9e7db
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node %device_type=gpu /10a3d22c
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_A100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node %device_type=gpu /30435cb0
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.2-gompi-2023a-CUDA-12.1.1') %scale=1_node %device_type=gpu /01d518df
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'OSU-Micro-Benchmarks/7.5-gompi-2023b-CUDA-12.4.0') %scale=1_node %device_type=gpu /4184c316
- EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_info=('snellius:gpu_H100', 'EESSI-2025.06', 'OSU-Micro-Benchmarks/7.5-gompi-2024a-CUDA-12.6.0') %scale=1_node %device_type=gpu /253bc1ba
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /ec5fea98
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /0149252b
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /7f86cc73
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /b44c4f26
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:rome', 'EESSI-2025.06', 'GROMACS/2025.2-foss-2025a') /10fe4b84
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /8575f1af
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /fa197803
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /848b3575
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /b4a91b63
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=1_node %module_info=('snellius:genoa', 'EESSI-2025.06', 'GROMACS/2025.2-foss-2025a') /42464906
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=gpu %scale=1_node %module_info=('snellius:gpu_A100', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b-CUDA-12.4.0') /91ff3a13
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=gpu %scale=1_node %module_info=('snellius:gpu_H100', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b-CUDA-12.4.0') /0aadbdf1
- EESSI_TensorFlow %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'tensorflow-probability/0.20.0-foss-2023a') %device_type=cpu /dc8fa7bb
- EESSI_TensorFlow %scale=1_node %module_info=('snellius:rome', 'EESSI-2023.06', 'TensorFlow/2.13.0-foss-2023a') %device_type=cpu /272d9e0b
- EESSI_TensorFlow %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'tensorflow-probability/0.20.0-foss-2023a') %device_type=cpu /54df3bda
- EESSI_TensorFlow %scale=1_node %module_info=('snellius:genoa', 'EESSI-2023.06', 'TensorFlow/2.13.0-foss-2023a') %device_type=cpu /214a955a

At first look, that looks quite good: all CUDA-enabled modules are only scheduled on our GPU partitions, all non-CUDA modules only on the CPU partitions. Modules are discovered accross both EESSI 2023.06 and EESSI 2025.06. The execution takes quite long - possibly also because we don't cache (yet). Anyway, I'm not disappointed with the current state: it needs some work to fix the last tests, but not a bad starting point.

…ssi_environments' into support_multiple_eessi_environments

…n of sys:part +feat notation directly

smoors · 2026-04-04T12:10:38Z

i must be doing something wrong here (with my suggested fix above).
if i don't load an EESSI module before running reframe, no GROMACS module is found.
if i load EESSI/2023.06, i only get the GROMACS modules of that EESSI version, even for the EESSI-2025.06 environment:

reframe -c test-suite/eessi/testsuite/tests/apps/gromacs.py -t 4_cores -t CI --system hydra:zen4 -v --list
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /571a2288
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /2c579518
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /a9773f33
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /a50f456d
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2025.06', 'GROMACS/2024.1-foss-2023b') /95a816c8
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2025.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /65ca6b4f
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2025.06', 'GROMACS/2024.3-foss-2023b') /6f8d873c
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:zen4', 'EESSI-2025.06', 'GROMACS/2024.4-foss-2023b') /aed0121e

smoors · 2026-04-04T14:03:43Z

ok, solved it by removing 'resolve_module_conflicts': False in my config file (don't remember exactly why that was needed).

smoors · 2026-04-05T08:24:16Z

+    environments = [
+        {'name': 'EESSI-2023.06', 'modules': ['EESSI/2023.06']},
+        {'name': 'EESSI-2025.06', 'modules': ['EESSI/2025.06']},
+    ]
+    environs = ['EESSI-2023.06', 'EESSI-2025.06']


as we are overwriting the environments, this change means that we can no longer use the default environment, which is needed to run local modules.

a simple solution would be to let it depend on an environment variable, e.g. USE_EESSI_MODULES:

if os.getenv('USE_EESSI_MODULES', True): environments = [ {'name': 'EESSI-2023.06', 'modules': ['EESSI/2023.06']}, {'name': 'EESSI-2025.06', 'modules': ['EESSI/2025.06']}, ] environs = ['EESSI-2023.06', 'EESSI-2025.06'] else: environments = [{'name': 'default'}] environs = ['default']

Or should we append this to local config instead of overwrite?

i prefer not to append to the local config, and only allow the environments that we support.

but thinking more about it, we can actually just do this:

environments = [ {'name': 'EESSI-2023.06', 'modules': ['EESSI/2023.06']}, {'name': 'EESSI-2025.06', 'modules': ['EESSI/2025.06']}, {'name': 'default'}, ] environs = ['EESSI-2023.06', 'EESSI-2025.06', 'default']

if the EESSI modules are in MODULEPATH, it will use them, and in that case you should not have any local modules in the MODULEPATH, so there will be no local modules found.

if the EESSI modules are not in MODULEPATH, the user should add -p default to avoid failure when trying to load an EESSI module, which i think isn't too bad.

Hmmm, I was thinking if you do have local environments that need to be loaded (e.g. we have local 2024, 2025 modules that load our local software stacks), it's annoying if everything is overwritten. And you don't really want to keep a separate config for local testing, because that means duplicating a lot of information. But what I then realized: you can just update the environments after calling set_common_required_config. That way, you can still have a single config file AND have local environs appended as well.

ok, i didn't think of that use case. then i agree to just append them.

Co-authored-by: Sam Moors <smoors@users.noreply.github.com>

casparvl · 2026-04-07T15:16:12Z

ok, solved it by removing 'resolve_module_conflicts': False in my config file (don't remember exactly why that was needed).

I have no clue why resolve_module_conflicts (https://reframe-hpc.readthedocs.io/en/stable/config_reference.html#config.general.resolve_module_conflicts) should interfere with any of this. But, do I understand that you now have the expected behavior? I.e.

Your current MODULEPATH contains the EESSI modules (but no application modules such as GROMACS etc directly)
When you list, reframe correctly finds the modules that become available by loading the respective EESSI versions?

smoors · 2026-04-07T18:03:43Z

yes, i get the expected behavior now:

- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:ampere', 'EESSI-2023.06', 'GROMACS/2024.1-foss-2023b') /cc033d46                                
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:ampere', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b-PLUMED-2.9.2') /e5b52802
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:ampere', 'EESSI-2023.06', 'GROMACS/2024.3-foss-2023b') /42c23ed7
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:ampere', 'EESSI-2023.06', 'GROMACS/2024.4-foss-2023b') /cb7345ad
- EESSI_GROMACS %benchmark_info=HECBioSim/Crambin %nb_impl=cpu %scale=4_cores %module_info=('hydra:ampere', 'EESSI-2025.06', 'GROMACS/2025.2-foss-2025a') /4a414393
Found 5 check(s)

checking the ReFrame source code, the way it works as i understand: when it tries to load a module (here EESSI/2025.06) from an environment and it fails, if resolve_module_conflicts is True, then it will unload the conflicting module(s) (EESSI/2023.06) and try again.

smoors · 2026-04-15T20:23:56Z

i looked a bit into implementing a cache for the modules, but it's not as trivial as i thought: we need to create a separate cache for each environment.

also, the find_modules function in reframe looks quite inefficient: it runs ms.available_modules(substr) for each substr, environment and system-partition, while it should ideally only run for each environment and return a list all modules, which can be cached, and which we can do multiple substring searches on (like we did before).

so, i'm now thinking to keep using our own find_modules function, updated to work with multiple environments. what do you think?

…an still load environments for testing local modules

casparvl · 2026-04-29T14:33:31Z

Discussed in the test suite sync, we can actually cache the triplets that find_modules(substr) return, but the problem is: there is no point. We will never call find_modules with the same substr again, so we wouldn't benefit from caching that way.

That means @smoors is right and the only way to properly cache is to implement the caching functionality inside the find_modules function. Somewhere around here

Something like

    ms = rt.runtime().modules_system
    current_system = rt.runtime().system
    snap0 = rt.snapshot()
    global cache
    cache = {}
    for p in current_system.partitions:
        if p not in cache:
            cache[p] = {}
        for e in p.environs:
            rt.loadenv(p.local_env, e)
            if e is not in cache[p]:
                 cache[p][e] = ms.available_modules()
            # Then something like a substr match on cache[p][e] 
            # modules = OrderedSet(ms.available_modules(substr))
            snap0.restore()
            for m in cache[p][e]:
                if _is_valid_for_env(m, e.name):
                    yield (p.fullname, e.name, m)

Ok, we need to figure out the last part (what does _is_valid_for_env() actually does).

casparvl · 2026-04-29T14:39:47Z

CI is failing because the eessi mixin class and the metallwalls test set the default as valid programming environment. We should just replace that by *.

…ult', as we now use ReFrame environments to test over multiple versions of EESSI

casparvl · 2026-04-29T15:23:09Z

Ah, the issue with the CI is a bit more subtle: the GH actions site_configuration does define a site_configuration['systems']['partitions']['environs'] = 'default', but it does not set a site_configurations['environments'], because that is not needed for the default. As a result, our common_config now site_configurations['environments'] with a list that does NOT contain default, and this then causes ReFrame to crash since the default environment isn't specfied there.

One thing that's a bit weird is that ReFrame's standard environment is called builtin, not default. So I'm still not entirely sure how this ever worked...

Anyway, I'll just remove it. That way, the common_config should add the EESSI environments as environs.

casparvl · 2026-04-30T14:23:31Z

    compute_device = parameter([DEVICE_TYPES.CPU])

    @run_after('init')
    def run_after_init(self):


We'll have to duplicate the new run-after-init stuff we do in eessi-mixin here (or port this test to EESSI_mixin, finally?)

Add initial functionality to schedule test for all EESSI environments…

30d4393

… in a single run, by configuring ReFrame environments and using ReFrame's own find_modules function

Make all tests aware that find_modules now returns a module_info obje…

ab0e7ce

…ct instead

satishskamath reviewed Mar 26, 2026

View reviewed changes

Comment thread eessi/testsuite/utils.py Outdated

casparvl commented Mar 26, 2026

View reviewed changes

Comment thread eessi/testsuite/utils.py Outdated

casparvl commented Mar 26, 2026

View reviewed changes

Comment thread eessi/testsuite/utils.py Outdated

casparvl and others added 6 commits March 26, 2026 14:38

Apply suggestion from @casparvl

8f7dad6

Apply suggestion from @casparvl

d6d4c5f

Fix linter errors

0e3a48a

Merge remote-tracking branch 'refs/remotes/origin/support_multiple_ee…

c63fbe0

…ssi_environments' into support_multiple_eessi_environments

Make sure that if reframe is new enough, we simply use the combinatio…

faf21ae

…n of sys:part +feat notation directly

Fix linting issues

8303df6

smoors reviewed Apr 4, 2026

View reviewed changes

Comment thread eessi/testsuite/eessi_mixin.py Outdated

smoors reviewed Apr 5, 2026

View reviewed changes

Comment thread eessi/testsuite/eessi_mixin.py Outdated

casparvl and others added 2 commits April 7, 2026 17:08

add missing 'if'

5d58833

Co-authored-by: Sam Moors <smoors@users.noreply.github.com>

Use at least three letters for variable assignments

63c0eab

Make appending to the existing environment the default, so that one c…

0112ebe

…an still load environments for testing local modules

No longer require that the reframe environment is always called 'defa…

3f4edfa

…ult', as we now use ReFrame environments to test over multiple versions of EESSI

Rely on the common config to set the environs

2a6ddbb

casparvl commented Apr 30, 2026

View reviewed changes

Conversation

casparvl commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl commented Mar 12, 2026

Uh oh!

casparvl commented Mar 12, 2026

Uh oh!

casparvl commented Mar 12, 2026

Uh oh!

casparvl commented Mar 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smoors commented Apr 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smoors commented Apr 4, 2026

Uh oh!

smoors Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

casparvl Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

smoors Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

casparvl Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

smoors Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

casparvl commented Apr 7, 2026

Uh oh!

smoors commented Apr 7, 2026

Uh oh!

smoors commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casparvl commented Apr 29, 2026

Uh oh!

casparvl commented Apr 29, 2026

Uh oh!

casparvl commented Apr 29, 2026

Uh oh!

casparvl Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

casparvl commented Mar 12, 2026 •

edited

Loading

smoors commented Apr 4, 2026 •

edited

Loading

smoors Apr 5, 2026 •

edited

Loading

smoors commented Apr 15, 2026 •

edited

Loading