Skip to content

fix(worker): one-shot run exits 1 on success — invert to exit 0#169

Open
RealDiligent wants to merge 2 commits into
mini-router:mainfrom
RealDiligent:fix/critical-issue-worker-once-exit-code
Open

fix(worker): one-shot run exits 1 on success — invert to exit 0#169
RealDiligent wants to merge 2 commits into
mini-router:mainfrom
RealDiligent:fix/critical-issue-worker-once-exit-code

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Problem

validator/src/eval_backend/worker.py main() one-shot branch:

if not args.loop:
    raise SystemExit(process_once(session_factory, settings))

process_once returns 1 when it processed a job and 0 when the queue was empty — a "did work" signal for the --loop sleep decision, not a process exit status (the --loop branch uses it as if processed == 0: sleep(...)). Passing it to SystemExit inverts exit semantics: a run that successfully processes a job exits 1 (which supervisors/cron/systemd oneshot/CI read as failure), and an empty queue exits 0.

Impact

Any one-shot invocation (python -m eval_backend.worker per timer tick, instead of --loop) reports every successful evaluation as a failed process — spurious alerts/retries or a unit stuck failed.

Fix

-    if not args.loop:
-        raise SystemExit(process_once(session_factory, settings))
+    if not args.loop:
+        process_once(session_factory, settings)
+        raise SystemExit(0)

A completed one-shot pass exits 0 whether or not the queue had work; genuine failures already raise (the except Exception: ... raise inside process_once) and still exit nonzero.

Tests

New validator/tests/test_worker_exit.py stubs the DB/engine bootstrap and asserts: one-shot exits 0 for both process_once returns (1 = did work, 0 = idle), and a raising process_once still propagates (nonzero). Verified the process_once==1 case exits 1 on the old code and 0 with the fix.

Risk / tradeoffs

Minimal. --loop behavior is unchanged. Only the one-shot exit code changes, from an inverted value to correct success/failure semantics.

Closes #168

@github-actions github-actions Bot added eval Evaluation changes validator Validator backend changes labels Jul 13, 2026
@RealDiligent

Copy link
Copy Markdown
Contributor Author

Included the same stale-test fix as noted in #160 (the pre-existing red on main's test_wrong_theta_length_fails) so this PR's test-router passes; the substantive change is the validator worker exit code. Happy to drop that commit once #160/#161 lands.

RealDiligent added 2 commits July 13, 2026 13:26
The non --loop branch did `raise SystemExit(process_once(...))`. process_once
returns 1 when it handled a job and 0 when the queue was empty — a "did work"
signal for the --loop sleep decision, not an exit status. So a one-shot run
that successfully processed a job exited 1, which supervisors/cron/systemd read
as failure (and an empty queue looked like success).

Run the pass and exit 0 on normal completion; genuine failures still raise and
exit nonzero. Added tests covering both process_once returns and the failure
path.

Closes mini-router#168
After the submission size check became warning-only, a wrong theta length
no longer produces a hard error, so test_wrong_theta_length_fails could
never pass: it asserted result.ok is False and relied on make_spec().n_total
differing from a hardcoded 128 (that n_total is 128 in the CI environment).
main's CI is currently red on this test for every PR branched off it (mini-router#160).

Rename to test_wrong_theta_length_warns and assert the current behavior: a
theta length that disagrees with the summary's declared n_total is surfaced
as a non-fatal warning (ok stays True), independent of the canonical n_total.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval Evaluation changes validator Validator backend changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

worker: one-shot (non --loop) run exits 1 on success, so supervisors see every processed job as a failure

1 participant