Skip to content

Cut the simulation timeout from 8 minutes to 4 #343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sjoelund merged 1 commit into from
Aug 30, 2026
Merged

Cut the simulation timeout from 8 minutes to 4 #343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,10 @@ different OpenModelica versions, according to the conditions of the
"defaultTolerance": 1e-6, // tolerance for tests if not specified by the model, defaults to 1e-6
"defaultNumberOfIntervals": 2500, // number of intervals for tests if not specified by the model, defaults to 2500
"ulimitOmc":800, // specify a max timeout for a model build
"ulimitExe":300, // specify a max timeout for a model simulation
"ulimitExe":300, // specify a max timeout for a model simulation, defaults to 240
"ulimitExeModels":{ // the models of this library that are allowed longer, see Simulation timeouts
"MyModelicaLibrary.Examples.SomethingBig":600
},
"ulimitMemory":62000000, // specify a max for the virtual memory of the running process when building a model
"procOMC":0, // [if procOMC = 0 use max procs, use procOMC = 1 if not defined, else use the given value] how many CPU cores should be used to run omc (load Modelica libraries in parallel and generate the C code in parallel)
"procCCompile":0, // [if procCCompile = 0 use max procs, use procCCompile = 1 if not defined, else use the given value] how many CPU cores should be used to compile the generated code
Expand Down Expand Up @@ -201,6 +204,38 @@ Options:
`procCCompile` above for more insight into individual test
parallelization.

### Simulation timeouts

A simulation is killed after `ulimitExe` seconds, 240 by default - 99.5% of the
models that simulate at all are done inside 105 on master, which runs on the
slower test machines. One number per library would have to cover its slowest
model, giving every other model of the library the same licence to hang, so a
library keeps the short timeout and names the forty-odd models that earned more:

```json
"ulimitExeModels":{
"Buildings.DHC.Examples.Combined.SeriesVariableFlow":810
}
```

[update-ulimit-exe.py](./update-ulimit-exe.py) writes those lists from the
results, so they stay measurements rather than guesses. The first line says what
would change, the second changes it:

```bash
./update-ulimit-exe.py --db postgresql://om@openmodelica.org/omdb configs/conf.json
./update-ulimit-exe.py --db postgresql://om@openmodelica.org/omdb --write configs/*.json
```

Both lines only read the database - `--write` writes the configuration files -
so a read-only user is enough for either.

Every model is allowed `--factor` times the longest it has taken over `--runs`
runs of `--branch`. A model that only ever ran into the timeout, and a library
the database has never heard of, are reported rather than guessed at. Run it
after a machine is replaced, after a change that moves the timings, or when the
reports start showing models killed by the timeout.

### Testing FMI with several simulators

Building an FMU costs far more than simulating it: on the twelve models of
Expand Down
89 changes: 77 additions & 12 deletions configs/conf.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
{
"library":"Modelica",
"libraryVersion":"trunk",
"ulimitExeModels":{
"Modelica.Electrical.PowerConverters.Examples.ACAC.SoftStarter":510
},
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"/",
"referenceFileNameExtraName":"$ClassName",
Expand All @@ -12,6 +15,9 @@
"library":"Modelica",
"libraryVersion":"4.1.0",
"libraryVersionExactMatch":true,
"ulimitExeModels":{
"Modelica.Electrical.PowerConverters.Examples.ACAC.SoftStarter":510
},
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"/",
"referenceFileNameExtraName":"$ClassName",
Expand All @@ -22,6 +28,9 @@
"library":"Modelica",
"libraryVersion":"4.0.0",
"libraryVersionExactMatch":true,
"ulimitExeModels":{
"Modelica.Electrical.PowerConverters.Examples.ACAC.SoftStarter":510
},
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"/",
"referenceFileNameExtraName":"$ClassName",
Expand Down Expand Up @@ -174,7 +183,12 @@
"libraryVersion":"maint.11.x",
"libraryVersionNameForTests":"11",
"ulimitOmc":300,
"ulimitExe":400,
"ulimitExeModels":{
"Buildings.Examples.DualFanDualDuct.ClosedLoop":360,
"Buildings.ThermalZones.EnergyPlus_9_6_0.Examples.SmallOffice.Guideline36Spring":420,
"Buildings.ThermalZones.EnergyPlus_9_6_0.Examples.SmallOffice.Guideline36Summer":420,
"Buildings.ThermalZones.EnergyPlus_9_6_0.Examples.SmallOffice.Guideline36Winter":510
},
"runOnceBeforeTesting":[["$resourceLocation/src/ThermalZones/install.py", "--binaries-for-os-only"]],
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"_",
Expand All @@ -187,7 +201,12 @@
"libraryVersion":"maint.12.x",
"libraryVersionNameForTests":"12",
"ulimitOmc":300,
"ulimitExe":400,
"ulimitExeModels":{
"Buildings.Examples.DualFanDualDuct.ClosedLoop":390,
"Buildings.Fluid.Geothermal.ZonedBorefields.Examples.SeriesConnectedZones":390,
"Buildings.ThermalZones.EnergyPlus_24_2_0.Examples.SmallOffice.Guideline36Spring":450,
"Buildings.ThermalZones.EnergyPlus_24_2_0.Examples.SmallOffice.Guideline36Summer":420
},
"runOnceBeforeTesting":[["$resourceLocation/src/ThermalZones/install.py", "--binaries-for-os-only"]],
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"_",
Expand All @@ -200,7 +219,16 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"latest",
"ulimitOmc":300,
"ulimitExe":800,
"ulimitExeModels":{
"Buildings.DHC.Examples.Combined.SeriesConstantFlow":780,
"Buildings.DHC.Examples.Combined.SeriesVariableFlow":810,
"Buildings.Examples.DualFanDualDuct.ClosedLoop":390,
"Buildings.Examples.VAVReheat.Guideline36":330,
"Buildings.Fluid.Geothermal.ZonedBorefields.Examples.SeriesConnectedZones":360,
"Buildings.ThermalZones.EnergyPlus_24_2_0.Examples.SmallOffice.Guideline36Spring":450,
"Buildings.ThermalZones.EnergyPlus_24_2_0.Examples.SmallOffice.Guideline36Summer":420,
"Buildings.ThermalZones.EnergyPlus_24_2_0.Examples.SmallOffice.Guideline36Winter":600
},
"runOnceBeforeTesting":[["$resourceLocation/src/ThermalZones/install.py", "--binaries-for-os-only"]],
"referenceFileExtension":"csv",
"referenceFileNameDelimiter":"_",
Expand All @@ -209,10 +237,18 @@
{
"library":"BuildingSystems",
"libraryVersion":"master",
"libraryVersionNameForTests":""
"libraryVersionNameForTests":"",
"ulimitExeModels":{
"BuildingSystems.Applications.AirConditioningSystems.PhotovoltaicCoolingSystem":450,
"BuildingSystems.Applications.HeatingSystems.SolarHeatingSystem":600,
"BuildingSystems.Buildings.Geometries.Viewfactors.Examples.RectangularCuboidSurfaces":510
}
},
{
"library":"BuildSysPro",
"ulimitExeModels":{
"BuildSysPro.Systems.HVAC.Emission.Examples.MozartJoulePIControlled":330
},
"_comment":"IncludeDirectory says modelica://BuildSysPro/Resources/C-Sources, which does not exist; the IBPSA C sources are in IBPSA/Resources/C-Sources (Resources/IBPSA/C-Sources in 3.5.0 and older). Some of them call str*/malloc/ModelicaError without including the header",
"extraCustomCommands":["setCFlags(getCFlags() + \" -include string.h -include stdlib.h -include ModelicaUtilities.h \\\"-I$libraryLocation/IBPSA/Resources/C-Sources\\\" \\\"-I$libraryLocation/Resources/IBPSA/C-Sources\\\"\")"]
},
Expand All @@ -238,7 +274,6 @@
{
"library":"ClaRa",
"ulimitOmc":800,
"ulimitExe":800,
"referenceFileExtension":"mat",
"referenceFileNameDelimiter":".",
"referenceFiles":{
Expand All @@ -251,7 +286,6 @@
{
"library":"ClaRa",
"ulimitOmc":800,
"ulimitExe":800,
"libraryVersion":"main",
"libraryVersionNameForTests":"dev",
"referenceFileExtension":"mat",
Expand All @@ -275,6 +309,9 @@
},
{
"library":"Dynawo",
"ulimitExeModels":{
"Dynawo.Examples.RVS.TestCases.TestB.TestBNoSvcNoLoadReset":330
},
"extraCustomCommands":["setCommandLineOptions(\"--allowNonStandardModelica=implicitParameterStartAttribute,illegalConditionalContext\");"]
},
{
Expand Down Expand Up @@ -425,7 +462,11 @@
"library":"PhotoVoltaics"
},
{
"library":"PhotoVoltaics_TGM"
"library":"PhotoVoltaics_TGM",
"ulimitExeModels":{
"PhotoVoltaics_TGM.TGM_Comax_Analytical_2016":480,
"PhotoVoltaics_TGM.TGM_Trina_Analytical_2016":510
}
},
{
"library":"PNlib",
Expand Down Expand Up @@ -468,7 +509,9 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"",
"ulimitOmc":300,
"ulimitExe":300,
"ulimitExeModels":{
"ScalableTestSuite.Thermal.DistrictHeating.ScaledExperiments.HeatingSystem_N_80":390
},
"ulimitMemory":12582912,
"optlevel":"-Os -march=native",
"referenceFileExtension":"mat",
Expand All @@ -484,7 +527,9 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"noopt",
"ulimitOmc":300,
"ulimitExe":300,
"ulimitExeModels":{
"ScalableTestSuite.Thermal.Advection.ScaledExperiments.SteamPipe_N_1280":330
},
"ulimitMemory":12582912,
"optlevel":"-O0",
"referenceFileExtension":"mat",
Expand Down Expand Up @@ -528,7 +573,11 @@
"library":"ThermofluidStream",
"libraryVersion":"main",
"libraryVersionNameForTests":"dev",
"ulimitExe":1200,
"ulimitExeModels":{
"ThermofluidStream.Boundaries.Tests.VolumesDirectCoupling":1350,
"ThermofluidStream.Examples.ReverseHeatPump":390,
"ThermofluidStream.Undirected.Boundaries.Tests.VolumesDirectCoupling":420
},
"referenceFileExtension":"mat",
"referenceFileNameDelimiter":".",
"referenceFinalDot":"_ref.",
Expand All @@ -537,7 +586,11 @@
},
{
"library":"ThermofluidStream",
"ulimitExe":1200,
"ulimitExeModels":{
"ThermofluidStream.Boundaries.Tests.VolumesDirectCoupling":1320,
"ThermofluidStream.Examples.ReverseHeatPump":690,
"ThermofluidStream.Undirected.Boundaries.Tests.VolumesDirectCoupling":420
},
"referenceFileExtension":"mat",
"referenceFileNameDelimiter":".",
"referenceFinalDot":"_ref.",
Expand Down Expand Up @@ -567,17 +620,29 @@
},
{
"library":"ThermoPower",
"ulimitExeModels":{
"ThermoPower.Test.DistributedParameterComponents.TestFlow1D2phChen_hf":390,
"ThermoPower.Test.DistributedParameterComponents.TestWaterFlow1DFEM_F":390
},
"optlevel":"-Os -march=native"
},
{
"library":"ThermoPower",
"libraryVersion":"4.0.0-dev",
"ulimitExeModels":{
"ThermoPower.Test.DistributedParameterComponents.TestWaterFlow1DFEM_F":390,
"ThermoPower.Test.DistributedParameterComponents.TestWaterFlow1DFEMnm_F":570
},
"optlevel":"-Os -march=native"
},
{
"library":"ThermoSysPro",
"libraryVersion":"master",
"libraryVersionNameForTests":""
"libraryVersionNameForTests":"",
"ulimitExeModels":{
"ThermoSysPro.Examples.SimpleExamples.TestCentrifugalPump7":570,
"ThermoSysPro.Fluid.Examples.SimpleExamples.TestDynamicWaterHeating":540
}
},
{
"library":"TransiEnt",
Expand Down
39 changes: 35 additions & 4 deletions configs/heavy_tests.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"OB",
"ulimitOmc":400,
"ulimitExe":300,
"ulimitExeModels":{
"ScalableTestSuite.Electrical.DistributionSystemAC.ScaledExperiments.DistributionSystemLinearIndividual_N_56_M_56":720,
"ScalableTestSuite.Electrical.DistributionSystemAC.ScaledExperiments.DistributionSystemLinear_N_56_M_56":390,
"ScalableTestSuite.Mechanical.FlexibleBeam.ScaledExperiments.FlexibleBeamModelica_N_64":480,
"ScalableTestSuite.Thermal.DistrictHeating.ScaledExperiments.HeatingSystem_N_80":480
},
"ulimitMemory":62914560,
"procOMC":0,
"procCCompile":0,
Expand All @@ -22,7 +27,15 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"NB",
"ulimitOmc":400,
"ulimitExe":300,
"ulimitExeModels":{
"ScalableTestSuite.Elementary.ParameterArrays.ScaledExperiments.Table_N_200_M_200":360,
"ScalableTestSuite.Mechanical.HarmonicOscillator.ScaledExperiments.HarmonicOscillatorNetwork_N_320":390,
"ScalableTestSuite.Power.ConceptualPowerSystem.ScaledExperiments.PowerSystemStepLoad_N_64_M_16":510,
"ScalableTestSuite.Power.ConceptualPowerSystem.ScaledExperiments.PowerSystemStepLoad_N_64_M_4":390,
"ScalableTestSuite.Power.ConceptualPowerSystem.ScaledExperiments.PowerSystemStepLoad_N_64_M_8":450,
"ScalableTestSuite.Thermal.Advection.ScaledExperiments.SimpleAdvection_N_3200":390,
"ScalableTestSuite.Thermal.HeatExchanger.ScaledExperiments.CounterCurrentHeatExchangerEquations_N_1280":330
},
"ulimitMemory":62914560,
"procOMC":0,
"procCCompile":0,
Expand All @@ -41,7 +54,26 @@
"libraryVersion":"master",
"libraryVersionNameForTests":"NB",
"ulimitOmc":400,
"ulimitExe":400,
"ulimitExeModels":{
"LargeTestSuite.Electrical.TransmissionLine.TransmissionLineModelica_N_10240":840,
"LargeTestSuite.Electrical.TransmissionLine.TransmissionLineModelica_N_2560":480,
"LargeTestSuite.Elementary.SimpleODE.CascadedFirstOrder_N_102400":900,
"LargeTestSuite.Elementary.SimpleODE.CascadedFirstOrder_N_204800":690,
"LargeTestSuite.Elementary.SimpleODE.CascadedFirstOrder_N_409600":990,
"LargeTestSuite.Elementary.SimpleODE.CascadedFirstOrder_N_51200":330,
"LargeTestSuite.Mechanical.HarmonicOscillator.HarmonicOscillator_N_1638400":570,
"LargeTestSuite.Mechanical.HarmonicOscillator.HarmonicOscillator_N_819200":510,
"LargeTestSuite.Thermal.Advection.SimpleAdvection_N_12800":900,
"LargeTestSuite.Thermal.HeatConduction.OneDHeatTransferTI_FD_N_1310720":360,
"LargeTestSuite.Thermal.HeatConduction.OneDHeatTransferTT_FD_N_1310720":360,
"LargeTestSuite.Thermal.HeatExchanger.CocurrentHeatExchangerEquations_N_1280":420,
"LargeTestSuite.Thermal.HeatExchanger.CocurrentHeatExchangerEquations_N_2560":420,
"LargeTestSuite.Thermal.HeatExchanger.CocurrentHeatExchangerEquations_N_5120":390,
"LargeTestSuite.Thermal.HeatExchanger.CocurrentHeatExchangerEquations_N_81920":660,
"LargeTestSuite.Thermal.HeatExchanger.CounterCurrentHeatExchangerEquations_N_1280":330,
"LargeTestSuite.Thermal.HeatExchanger.CounterCurrentHeatExchangerEquations_N_2560":450,
"LargeTestSuite.Thermal.HeatExchanger.CounterCurrentHeatExchangerEquations_N_81920":660
},
"ulimitMemory":62914560,
"procOMC":0,
"procCCompile":0,
Expand All @@ -54,7 +86,6 @@
"libraryVersionNameForTests":"NB_SymbolicJacobian",
"ignoreModelPrefix":"ScalableTestGrids.Models.Type1Large",
"ulimitOmc":800,
"ulimitExe":300,
"ulimitMemory":62000000,
"procOMC":0,
"procCCompile":0,
Expand Down
2 changes: 1 addition & 1 deletion library.html.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ OpenModelicaLibraryTesting Changes<br>
#metadata#</pre></p>
<p>
BuildModel time limit: #ulimitOmc#s<br>
Simulation time limit: #ulimitExe#s<br>
Simulation time limit: #ulimitExe#s<br>#ulimitExeModels#
Default tolerance: #defaultTolerance#<br>
Default number of intervals: #defaultNumberOfIntervals#<br>
Optimization level: #optlevel#</p>
Expand Down
18 changes: 15 additions & 3 deletions resultsdb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,9 +365,7 @@ def __init__(self, url):
self.host = hostname()
self.claims = []
self.heartbeatThread = None
self.execute(JOB_CLAIM)
self._migrateJobClaim()
self.commit()
self.jobClaimReady = False

def _connect(self):
"""Open the connection, asking the kernel to keep it alive.
Expand Down Expand Up @@ -419,6 +417,19 @@ def commit(self):
self.conn.commit()
self.pending = []

def _prepareJobClaim(self):
"""Create and migrate job_claim, the first time a job is claimed.

Only a test run claims anything, so doing it on connect would make the
report scripts ask for write rights they never use.
"""
if self.jobClaimReady:
return
self.execute(JOB_CLAIM)
self._migrateJobClaim()
self.commit()
self.jobClaimReady = True

def _migrateJobClaim(self):
"""Narrow an older job_claim to (branch, libname).

Expand Down Expand Up @@ -539,6 +550,7 @@ def claim(self, branch, libname, libversion, omcversion, confighash):
A machine that dies stops sending its heartbeat, and after
STALE_CLAIM_MINUTES its jobs are up for grabs again.
"""
self._prepareJobClaim()
key = (branch, libname)
got = self.execute("""INSERT INTO job_claim
(branch, libname, libversion, omcversion, confighash, host, state)
Expand Down
Loading
Loading