Skip to content

Commit 4e8b1c0

Browse files
Ronald Tseronaldtse
authored andcommitted
fix: orchestrator log() crashes on a fresh arm's missing run dir
The first watch line for an arm whose training hasn't started opens chain_log.jsonl inside a directory that only distill_sequence creates — FileNotFoundError killed the orchestrator right after run-005's eval (exactly when it reached arm 4), and every relaunch died the same way before logging anything. mkdir(parents=True, exist_ok=True).
1 parent 577df5c commit 4e8b1c0

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

‎src/gpu/modal_distill.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1710,7 +1710,11 @@ def qwen_next_chain() -> dict:
17101710
ROOT = Path("/checkpoints")
17111711

17121712
def log(run: str, event: str) -> None:
1713-
with (ROOT / run / "chain_log.jsonl").open("a", encoding="utf-8") as fh:
1713+
# mkdir: a fresh arm's run dir does not exist until its training
1714+
# creates it — the first watch line must not crash on that
1715+
run_dir = ROOT / run
1716+
run_dir.mkdir(parents=True, exist_ok=True)
1717+
with (run_dir / "chain_log.jsonl").open("a", encoding="utf-8") as fh:
17141718
fh.write(json.dumps({"t": round(time.time()), "event": event}) + "\n")
17151719
CHECKPOINTS.commit()
17161720

0 commit comments

Comments
 (0)