Cut the simulation timeout from 8 minutes to 4 - #343
Merged
Conversation
sjoelund
enabled auto-merge (squash)
August 30, 2026 07:47
A simulation was killed after ulimitExe seconds, 8 minutes by default. On master, which runs on the slower test machines, 99.5% of the models that simulate at all are done inside 105 seconds, so the default mostly bought the hanging models time to hang in: about 49 models per master run spent more than two minutes each and failed anyway. One number per library cannot be tightened much on its own, because it has to cover the slowest model of the library and so hands every other model the same licence - Buildings needs 640s for 18 of its 1786 models. A model that has earned longer therefore names itself in ulimitExeModels, and its library keeps the short default. Sizing the default from a held-out month, replaying limits derived from 100 master runs against the next 100: | default | named models | spurious timeouts/run | simulation saved | | ------- | ------------ | --------------------- | ---------------- | | 120s | 98 | 1.65 | 4.6 h/run | | 180s | 55 | 0.99 | 3.7 h/run | | 240s | 40 | 0.48 | 2.9 h/run | | 300s | 25 | 0.13 | 2.0 h/run | Consecutive master runs already differ in some 50 model statuses, 13 of them simulation regressions, so 240s adds about 4% to that where 120s would add 13%. The margin on top of a model's observed maximum matters far less: 1.25 and 3.0 differ by 0.01 h/run and not at all in the breach count, because what overruns is models nothing named. What is committed here saves 2.85 h of simulation per master run, all of it on runs that fail anyway, and drops what a run is licensed to spend simulating from 2822 h to 1324 h. Sizing from master costs the other jobs almost nothing - no newly failing models on v1.26, v1.27 or master-fmi and 4 to 12 on cvode, gbode, cpp and fmi-fmpy - bar newInst-newBackend at 52, where the new backend really is that much slower on OpenIPSL and IBPSA. update-ulimit-exe.py derives the numbers from the results database and writes them back, editing the files as text so that setting one does not reformat the other ninety-five entries, and checking that the result parses to what it was meant to say before writing it. job_claim is now created on the first claim rather than on connect, so the scripts that only read - the reports and this one - no longer need a database user that may write. The watchdogs meant to cut a hung model short cannot kill anything on Windows, which is why they never did: signal.SIGKILL and os.killpg do not exist there, and the AttributeError that raises is not the OSError the kill loops catch. The escape path threw while the worker thread sat in a ZMQ receive that never returns, and the interpreter then waited for that non-daemon thread forever, so a hung model was ended by test.py's outer timeout, 2*ulimitOmc + ulimitExe + 25, rather than by its own - eleven minutes later and with nothing written down. The GitHub sanity check has been hitting this on windows-latest about every fifth run since 7 August, spending 1900 seconds where a passing run spends 120. The kill loops are now one helper per script, the worker threads are daemons, and Windows signals the whole process tree, since the command runs under a shell of its own there and killing the shell leaves the command running. A model that hangs still fails; this only makes it fail at its own timeout, and leave the log that says where it hung. Assisted-by: Claude Opus 5
sjoelund
force-pushed
the
shorter-simulation-timeouts
branch
from
August 30, 2026 08:39
9016b4e to
3a81b90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A simulation was killed after ulimitExe seconds, 8 minutes by default. On master, which runs on the slower test machines, 99.5% of the models that simulate at all are done inside 105 seconds, so the default mostly bought the hanging models time to hang in: about 49 models per master run spent more than two minutes each and failed anyway.
One number per library cannot be tightened much on its own, because it has to cover the slowest model of the library and so hands every other model the same licence - Buildings needs 640s for 18 of its 1786 models. A model that has earned longer therefore names itself in ulimitExeModels, and its library keeps the short default.
Sizing the default from a held-out month, replaying limits derived from 100 master runs against the next 100:
| default | named models | spurious timeouts/run | simulation saved | | ------- | ------------ | --------------------- | ---------------- |
| 120s | 98 | 1.65 | 4.6 h/run |
| 180s | 55 | 0.99 | 3.7 h/run |
| 240s | 40 | 0.48 | 2.9 h/run |
| 300s | 25 | 0.13 | 2.0 h/run |
Consecutive master runs already differ in some 50 model statuses, 13 of them simulation regressions, so 240s adds about 4% to that where 120s would add 13%. The margin on top of a model's observed maximum matters far less: 1.25 and 3.0 differ by 0.01 h/run and not at all in the breach count, because what overruns is models nothing named.
What is committed here saves 2.85 h of simulation per master run, all of it on runs that fail anyway, and drops what a run is licensed to spend simulating from 2822 h to 1324 h. Sizing from master costs the other jobs almost nothing - no newly failing models on v1.26, v1.27 or master-fmi and 4 to 12 on cvode, gbode, cpp and fmi-fmpy - bar newInst-newBackend at 52, where the new backend really is that much slower on OpenIPSL and IBPSA.
update-ulimit-exe.py derives the numbers from the results database and writes them back, editing the files as text so that setting one does not reformat the other ninety-five entries, and checking that the result parses to what it was meant to say before writing it.
job_claim is now created on the first claim rather than on connect, so the scripts that only read - the reports and this one - no longer need a database user that may write.
Assisted-by: Claude Opus 5