Skip to content

Record the phase times a timed-out model reached - #345

Merged
sjoelund merged 1 commit into
masterfrom
alarm-grace-and-phase-times
Aug 31, 2026
Merged

Record the phase times a timed-out model reached#345
sjoelund merged 1 commit into
masterfrom
alarm-grace-and-phase-times

Conversation

@sjoelund

Copy link
Copy Markdown
Member

omc's alarm now asks the running command to unwind before it kills the process, so a command that runs out of time still reports how far it got. Both watchdogs have to outwait that grace, or they kill omc in the middle of producing the answer: sendExpressionTimeout's own deadline and test.py's outer per-model budget. shared.alarmGrace mirrors SystemImpl__alarm's formula.

The simulation phase is also recorded on every way out, not just the ones that reach a handler. sendExpressionTimeout ends the process itself when omc dies, so the caller's except TimeoutError never runs; runningPhase covers that, and now takes the results dict so a later runner's own simulation time survives it too. simElapsed falls back to the wall clock when omc reports no time of its own, which is what an aborted run does.

In the run of 2026-08-30 this was 90 models and 23936 s of wall clock recorded against no phase at all, 34 of them reporting finalphase=0 with every clock at zero after 300-800 s. The master lane, whose simulations are separate processes this script kills itself, had none.

The ladder that turns omc's nested stopwatches into per-phase times also had two guards swapped: the backend's own time was computed under templates != -1 and SimCode's under simcode != -1, each testing the clock of the wrong phase. A translation that died in SimCode was recorded as having died in the backend, and the phase before the failure got no time. It is now one ladder over the clock that reads -1, which is both what says how far the translation got and what says which differences can be taken.

Assisted-by: Claude Opus 5

omc's alarm now asks the running command to unwind before it kills the
process, so a command that runs out of time still reports how far it got.
Both watchdogs have to outwait that grace, or they kill omc in the middle
of producing the answer: `sendExpressionTimeout`'s own deadline and
test.py's outer per-model budget. `shared.alarmGrace` mirrors
`SystemImpl__alarm`'s formula.

The simulation phase is also recorded on every way out, not just the ones
that reach a handler. `sendExpressionTimeout` ends the process itself when
omc dies, so the caller's `except TimeoutError` never runs; `runningPhase`
covers that, and now takes the results dict so a later runner's own
simulation time survives it too. `simElapsed` falls back to the wall clock
when omc reports no time of its own, which is what an aborted run does.

In the run of 2026-08-30 this was 90 models and 23936 s of wall clock
recorded against no phase at all, 34 of them reporting `finalphase=0`
with every clock at zero after 300-800 s. The `master` lane, whose
simulations are separate processes this script kills itself, had none.

The ladder that turns omc's nested stopwatches into per-phase times also
had two guards swapped: the backend's own time was computed under
`templates != -1` and SimCode's under `simcode != -1`, each testing the
clock of the wrong phase. A translation that died in SimCode was recorded
as having died in the backend, and the phase before the failure got no
time. It is now one ladder over the clock that reads -1, which is both
what says how far the translation got and what says which differences can
be taken.

Assisted-by: Claude Opus 5
@sjoelund
sjoelund enabled auto-merge (squash) August 31, 2026 08:58
@sjoelund
sjoelund disabled auto-merge August 31, 2026 09:26
@sjoelund
sjoelund merged commit 1e8481c into master Aug 31, 2026
6 of 7 checks passed
@sjoelund
sjoelund deleted the alarm-grace-and-phase-times branch August 31, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant