Repository navigation
Expand file tree
/
Copy pathdispatch_worker.py
More file actions
1893 lines (1773 loc) · 98.7 KB
/
Copy pathdispatch_worker.py
File metadata and controls
1893 lines (1773 loc) · 98.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""dispatch_worker - let the model hand a scoped sub-task to a headless WORKER
agent that runs in the background, then collect its result later.
WHY: some work is a self-contained errand - "read this large module and tell me
where X is handled", "run the test suite and summarise the failures". Doing it
inline burns the main session's context on raw output. A worker does it in its
OWN session (own context, own model choice) and hands back only the answer.
HOW (piggybacks the background-task machinery, adds no new transport):
- the model calls dispatch_worker(task=..., [model=...], [cwd=...]);
- we write a BRIEF file (the task + a GRANT header: leash, model, cwd, limits);
- we bg_launch `lean_coder --agent-run --brief-file B --result-file R` as a
detached task tagged kind="worker", with a lease (self-terminates if this
session dies / stops attending - caps runaway quota);
- the worker runs one brief on THIS box (the driver's provider + auth - no
per-box install/oauth), writes its answer between ===RESULT=== markers to R,
exits. Its file/command tools ride the same executor path this session uses.
- the finished worker is surfaced via /worker (workers are deliberately hidden
from /bg + the bg_status tool, so a session without this tool never sees them).
LEASH: the worker's grant is set by the driver (the main AI) via the `leash` arg
and capped at the driver's OWN live leash - MIN(requested, parent), the leash
principle: a grant is never above the grantor's authority (see _capped_leash).
So read/write/exec workers are ALL supported; the default is 'r' (a scout), and a
write/exec worker inherits approval=auto headless (nobody's there to confirm), so
grant rw/rwe deliberately. This is a driver_only tool: it spawns the worker process
on the driver, so it is never pushed to / run on a remote executor.
WORKER CEILINGS (the cost lever) - each resolves in priority order:
1. the env var (operator LOCKDOWN - still wins, a running session can't move it);
2. else the live cfg value (editable in /settings, saved to config, usable at once);
3. else the built-in default (shown below).
cfg field /settings + config env var default
worker_max_concurrent most workers at once LEANCODER_WORKER_MAX_CONCURRENT 10
worker_idle_timeout lease secs; self-kill LEANCODER_WORKER_IDLE_TIMEOUT 1800
worker_max_iterations agentic-loop cap/worker LEANCODER_WORKER_MAX_ITER 30
model_allowlist (env only) LEANCODER_WORKER_MODELS any
RESUME: with cfg.worker_checkpoint on, a worker dumps its transcript to a
'<brief>.checkpoint' sidecar each iteration; action='resume' (pid=, text=) relaunches a
DEAD/timed-out/incomplete worker from that checkpoint (a NEW pid - resume is a fresh OS
process; lineage kept in meta['resumed_from']). Off by default = no checkpoint file.
"""
import json
import os
import shlex
import sys
import time
from pathlib import Path
# Built-in ceiling defaults (last resort: env var > live cfg > this - see docstring).
_DEFAULTS = {"max_concurrent": 10, "idle_timeout": 1800, "max_iterations": 30,
"max_depth": 1, "max_children": 0}
_ENV = {"max_concurrent": "LEANCODER_WORKER_MAX_CONCURRENT",
"idle_timeout": "LEANCODER_WORKER_IDLE_TIMEOUT",
"max_iterations": "LEANCODER_WORKER_MAX_ITER",
"max_depth": "LEANCODER_WORKER_MAX_DEPTH",
"max_children": "LEANCODER_WORKER_MAX_CHILDREN"}
TOOL = {
"name": "dispatch_worker",
"glyph": "\u2691", # flag: a dispatched sub-agent (distinct from bg's bolt; bigger blast radius - own context/leash, tracked via /worker)
"description": (
"Hand a self-contained sub-task to a background worker agent (own session + "
"context); returns a pid. It runs in the BACKGROUND and you're NOTIFIED "
"AUTOMATICALLY when it finishes (a notice on a later turn) - don't poll it, just "
"carry on or end your turn. For scoped errands whose output would bloat this "
"session, e.g. 'scout this module, report where X is handled'. Default READ-ONLY; "
"leash='rw'/'rwe' to let it edit/run, never above YOUR capability. Manage/steer a "
"worker (and list models, resume a dead one) via `action`."),
"parameters": {
"type": "object",
"properties": {
"action": {"type": "string",
"enum": ["dispatch", "models", "status", "result", "transcript",
"cancel", "pause", "pause_all", "stop_all", "resume",
"resume_all", "inject", "set_plan", "add_note", "promote",
"board_claim", "board_release", "board_list"],
"default": "dispatch",
"description": "Default 'dispatch' (launch from `task`). 'models'=list "
"models you can dispatch on. 'status' [pid]=worker state/"
"runtime/ready. 'result' (pid=, [page=])=FULL result, paged "
"(the finish notice truncates a long one). 'transcript' "
"(pid=, [page=])=read the worker's OWN full reasoning/tool "
"trail, paged (needs worker_checkpoint on). 'cancel' "
"(pid=)=kill+discard it. FLEET lifecycle: 'pause' (pid=, "
"[name=])=save it as a session, then stop it; 'resume' (pid=)"
" relaunches a paused/DIED/capped one here, or dispatch "
"from_session=<name> from any session later; "
"'pause_all'/'stop_all'/'resume_all' ([taskboard=])=do it to "
"every worker (pause=saved, stop=discarded), optionally "
"just one board's team. Steer a RUNNING worker: 'inject' "
"(text=) a correction, 'set_plan' (plan=) replace its plan "
"(omit plan to READ it first), 'add_note' (notes=). "
"'board_claim'/'board_release' (text=<path>)/'board_list'="
"shared-file mutex so peer workers don't clash (no-op for a "
"lone worker)."},
"pid": {"type": "integer",
"description": "Worker to act on (required for 'cancel'; optional for 'status' "
"to show one). From the dispatch return line."},
"task": {"type": "string",
"description": "Full self-contained instruction: what to do and exactly "
"what to report back (the worker has no other context). "
"Required for action='dispatch'."},
"text": {"type": "string",
"description": "For 'inject' (pid=): a mid-task correction/steer the running "
"worker sees on its next reasoning step (does NOT interrupt a "
"running command). Also the fresh steer for 'resume'."},
"model": {"type": "string",
"description": "Optional worker brain - ANY model on an enabled provider "
"(see action='models'); its provider is inferred. Default: "
"inherit your current model."},
"provider": {"type": "string",
"description": "Optional backend for the worker's brain (must be enabled). "
"Only needed to disambiguate when the same model name is on "
"more than one provider; else inferred from `model`."},
"brain_host": {"type": "string",
"description": "Optional ollama endpoint (URL or /machines alias) to run "
"the worker's BRAIN on - distinct from `host` (its TOOLS). "
"Runs the model on a different box (ollama only)."},
"host": {"type": "string",
"description": "Optional box to run the worker's TOOLS on ([user@]host or a "
"/connect name); defaults to this session's target. A password "
"host is prompted ONCE here - a worker can't answer prompts."},
"cwd": {"type": "string",
"description": "Optional working directory (defaults to current)."},
"from_session": {"type": "string",
"description": "Optional: seed this worker from a saved SESSION (name "
"under CONFIG_DIR/sessions, or a path). Its transcript "
"becomes the worker's history and `task` runs as a steer "
"turn on top; checkpointing is forced on so it can be "
"promoted back to a session later."},
"name": {"type": "string",
"description": "For 'promote'/'pause' (pid=): the session name to save the "
"worker under (pause defaults to worker-<pid>)."},
"taskboard": {"type": "string",
"description": "Optional: assign this worker to a named task-board (see "
"the `board` tool). The worker auto-gets the board tool and "
"is told to report its task done there. Usually add+assign "
"the task first, then dispatch with taskboard=<name>."},
"iterations": {"type": "integer",
"description": "Optional tool-call budget, capped at the operator ceiling "
"(default 30; omit = ceiling). On 'dispatch' sets the worker's "
"cap; on 'resume' grants a FRESH budget (use it when the worker "
"died by hitting its cap)."},
"leash": {"type": "string", "enum": ["r", "rw", "rwe"], "default": "r",
"description": "Worker capability: r=read-only (default), rw=edit, rwe=edit+run. "
"Capped at your own leash."},
"tools": {"type": "array", "items": {"type": "string"},
"description": "Optional ALLOWLIST of tool names the worker may use (e.g. "
"[\"read_file\",\"web_fetch\"] for a scout). Omit = your full "
"leash-permitted set. A worker never gets a tool you lack, and "
"the grant is still leash-capped; its plan/note meta tools are "
"always kept."},
"context": {"type": "string",
"description": "Optional CURATED background the worker needs but that isn't "
"the task itself (e.g. 'auth was just refactored; tokens now "
"live in x'). Share the STATE it needs, not your whole "
"transcript (not truncated)."},
"plan": {"type": "string",
"description": "Optional starting plan (GOAL + '- [ ]' TODO) seeding the "
"worker's pinned plan. Also the payload for 'set_plan'."},
"notes": {"type": "string",
"description": "Optional seed notes for the worker's notebook, one per line "
"(tagged as from you). Also the payload for 'add_note'."},
"page": {"type": "integer",
"description": "For 'result'/'transcript' (pid=): which page to show "
"(1-based, default 1). The reply tells you the page count "
"and whether more remain."},
},
"required": [],
},
"driver_only": True, # spawns the worker process on the driver; never pushed remotely
"no_timeout": True, # launches/awaits a background worker agent; long-by-design
# NOT safe: dispatching a worker spends quota + runs commands, so it confirms
# unless approval is auto/armed (same policy as any non-read tool).
}
# Captured from core in setup(). A tool's run() doesn't get lc, so we stash the
# hooks + helpers here at startup (setup runs driver-only, which is where a worker
# is launched - so these are always present when run() fires).
_H = {}
def _ceiling(key):
"""An integer worker ceiling, resolved in priority order:
1. the env var (LEANCODER_WORKER_*) if set + valid - an operator LOCKDOWN that
a running session can't move;
2. else the live cfg value (worker_<key>), editable in /settings + persisted to
config - the user's own knob, usable straight away;
3. else the built-in default.
So users get simple, persistent control via /settings, while an env var still wins
as a hard ceiling when the operator wants one."""
env = os.environ.get(_ENV.get(key, ""), "")
if env.strip():
try:
return int(env)
except ValueError:
pass
cfg = _H.get("cfg")
val = getattr(cfg, f"worker_{key}", None) if cfg else None
if isinstance(val, int):
return val
return _DEFAULTS[key]
def _governor_check(my_depth):
"""Safe-recursion gate for the process trying to spawn a worker. `my_depth` is this
process's depth in the worker tree (driver = 0, its workers = 1, ...). Returns
(ok, error_str, granted_child_budget):
- the DRIVER (depth 0) may always spawn (only max_concurrent bounds it); the child
it makes is depth 1 and gets a child budget of worker_max_children.
- a WORKER (depth >= 1) may spawn ONLY IF depth < worker_max_depth AND it still has
child budget left (granted by its parent, decremented per spawn). The child it
makes is depth my_depth+1 and inherits a SHRINKING budget (its own remaining
minus the one it's spending), so a subtree is bounded by depth x count, never a
fork-bomb. Both gates REFUSE loudly rather than silently succeed."""
cfg = _H.get("cfg")
max_depth = _ceiling("max_depth")
if my_depth <= 0:
# The driver: ungated here (max_concurrent is its bound). Its children are depth 1
# and get the operator's per-worker child allowance.
return True, "", _ceiling("max_children")
if my_depth >= max_depth:
return (False, (f"error: recursion depth limit reached (this worker is at depth "
f"{my_depth}, worker_max_depth is {max_depth}). A worker at the max "
f"depth cannot spawn its own workers. Raise worker_max_depth / "
f"LEANCODER_WORKER_MAX_DEPTH to allow deeper recursion."), 0)
remaining = int(getattr(cfg, "worker_child_budget", 0) or 0)
if remaining <= 0:
return (False, ("error: this worker was granted no child budget (worker_max_children "
"is 0, or its budget is spent), so it cannot spawn workers. This is the "
"safe default - recursion is opt-in. Raise worker_max_children / "
"LEANCODER_WORKER_MAX_CHILDREN to grant one."), 0)
# Spend one from this worker's budget; the child inherits what's left after this spawn.
cfg.worker_child_budget = remaining - 1
return True, "", remaining - 1
def _model_allowlist():
"""The model allowlist (LEANCODER_WORKER_MODELS, comma-separated), or None = any."""
raw = os.environ.get("LEANCODER_WORKER_MODELS", "").strip()
if not raw:
return None
return [m.strip() for m in raw.split(",") if m.strip()]
def _capped_leash(requested):
"""The worker's granted leash = MIN(requested, the parent's LIVE leash). A worker
can NEVER exceed its parent's authority (the leash principle: a grant is <= the
grantor's own). requested defaults to 'r' (read-only). Junk -> 'r'. Reads the
parent's leash off the live cfg captured in setup(), so a runtime /leash change is
respected. Returns (granted_leash, was_downgraded)."""
levels = _H["LEASH_LEVELS"] # ("chat","r","rw","rwe")
req = _H["_norm_leash"](requested) or "r"
if req == "chat": # a worker with no tools is pointless
req = "r"
parent = getattr(_H.get("cfg"), "leash", "rwe") or "rwe"
granted = req if levels.index(req) <= levels.index(parent) else parent
return granted, (granted != req)
def _workers_dir():
"""Where this driver's worker files (brief/result/progress/checkpoint...) live: its
OWN per-process runtime dir, beside its bg task records - so they share one
lifecycle (removed on clean exit, reaped by pid-liveness after a crash; never by
age). Falls back to CONFIG_DIR/workers only on a core that predates run dirs."""
rpd = _H.get("_run_pid_dir")
return Path(rpd()) if rpd else Path(_H["CONFIG_DIR"]) / "workers"
def _compose_brief(task, model, cwd, max_iter, leash="r", provider="", brain_host="",
tools="", context="", plan="", notes="", depth=0, child_budget=0,
checkpoint=False, resume="", board=0, taskboard=""):
"""Build the brief file text: the task wrapped in BRIEF markers + a GRANT header.
`leash` is the already-capped grant (see _capped_leash). `provider` (optional) is
the backend the worker must activate for `model`; absent = inherit the driver's.
`brain_host` (optional) is an ollama inference endpoint for the worker's BRAIN,
distinct from where its tools run; absent = the driver's own host. `tools`
(optional) is a comma-separated allowlist of tool names; absent = full toolset.
`context`/`plan`/`notes` (optional) seed the worker with curated STATE - a
background blob, a starting plan, notebook entries - each emitted as its own marker
block the worker parses in run_agent_brief. Not truncated: the parent decides how
much state the job needs. `depth` is this worker's depth in the tree (driver->worker
= 1); `child_budget` is how many children it may itself spawn (0 = none) - both drive
the safe-recursion governor."""
B, G = _H["BRIEF_MARK"], _H["GRANT_MARK"]
grant = [f"leash: {leash}", f"cwd: {cwd}", f"max_iterations: {max_iter}"]
if model:
grant.append(f"model: {model}")
if provider:
grant.append(f"provider: {provider}")
if brain_host:
grant.append(f"brain_host: {brain_host}")
if tools:
grant.append(f"tools: {tools}")
if depth:
grant.append(f"depth: {depth}")
if child_budget:
grant.append(f"child_budget: {child_budget}")
if checkpoint:
grant.append("checkpoint: 1")
if board:
grant.append(f"board: {board}")
if taskboard:
grant.append(f"taskboard: {taskboard}")
out = [f"{B}\n{task.strip()}\n{B}", f"{G}\n" + "\n".join(grant) + f"\n{G}"]
for text, mark_key in ((context, "SEED_CONTEXT_MARK"), (plan, "SEED_PLAN_MARK"),
(notes, "SEED_NOTES_MARK")):
if text and text.strip():
m = _H[mark_key]
out.append(f"{m}\n{text.strip()}\n{m}")
# A resumed worker: point it at the dead worker's transcript checkpoint. run_agent_brief
# reloads that transcript, then runs `task` as a fresh steer turn on top of it.
if resume:
rm = _H["RESUME_MARK"]
out.append(f"{rm}\n{resume.strip()}\n{rm}")
return "\n".join(out) + "\n"
def _self_argv():
"""The command that launches THIS lean-coder as a headless worker. Uses the same
interpreter + the CORE module file (captured as lc["__file__"] in setup()), so the
worker is the exact build the parent runs (no PATH lookup, no version drift)."""
core = Path(_H["__file__"]).resolve()
return f"{shlex.quote(sys.executable)} {shlex.quote(str(core))}"
def _tb_arg(args):
"""The optional taskboard scope for a team action (pause_all/stop_all/resume_all):
args['taskboard'] stripped, or None = all workers this session (unscoped)."""
tb = (args.get("taskboard") or "").strip()
return tb or None
def run(args, cwd):
if "bg_launch" not in _H:
return ("error: dispatch_worker is not initialised (its setup() did not run; "
"it must be enabled as a driver lean-tool).")
action = (args.get("action") or "dispatch").strip().lower()
if action == "status":
return _worker_status(args.get("pid"))
if action == "models":
return _worker_models()
if action == "result":
return _worker_result(args.get("pid"), args.get("page"))
if action == "cancel":
return _worker_cancel(args.get("pid"))
if action == "pause":
return _worker_pause(args.get("pid"), args.get("name"))
if action == "pause_all":
return _worker_pause_all(_tb_arg(args))
if action == "stop_all":
return _worker_stop_all(_tb_arg(args))
if action == "resume_all":
return _worker_resume_all(cwd, _tb_arg(args))
if action == "inject":
return _worker_inject(args.get("pid"), args.get("text"), source="parent-agent")
if action == "set_plan":
return _worker_set_plan(args.get("pid"), args.get("plan") or args.get("text"))
if action == "add_note":
return _worker_add_note(args.get("pid"), args.get("notes") or args.get("text"))
if action == "resume":
return _worker_resume(args.get("pid"), args.get("text") or args.get("task"), cwd,
iterations=args.get("iterations"))
if action == "transcript":
return _worker_transcript(args.get("pid"), args.get("page"))
if action == "promote":
return _worker_promote(args.get("pid"), args.get("name") or args.get("text"))
if action in ("board_claim", "board_release", "board_list"):
return _worker_board(action, args.get("text") or args.get("task"))
if action != "dispatch":
return ("error: unknown action %r (use dispatch | models | status | result | "
"transcript | cancel | pause | pause_all | stop_all | resume | resume_all | "
"inject | set_plan | add_note | promote | board_claim | board_release | "
"board_list)." % action)
task = (args.get("task") or "").strip()
if not task:
return "error: dispatch_worker action='dispatch' needs a non-empty task."
# Optional: SEED this fresh worker from a saved SESSION (or any checkpoint-shaped
# file) - the unified {meta,messages} envelope means a session file is a valid
# worker checkpoint. Given a bare name, resolve it under CONFIG_DIR/sessions.
# run_agent_brief reloads that transcript, then runs `task` as a steer turn on top.
seed_session = (args.get("from_session") or "").strip()
seed_resume = ""
seed_session_name = "" # the session-file STEM, if seeded from one under sessions/
if seed_session:
p = Path(seed_session)
if not p.is_file():
cand = Path(_H["CONFIG_DIR"]) / "sessions" / (seed_session + ".json")
if cand.is_file():
p = cand
if not p.is_file():
return (f"error: from_session {seed_session!r} not found "
f"(looked for a file and under CONFIG_DIR/sessions/).")
seed_resume = str(p.resolve())
# If it's a real session file (under sessions/, .json), remember its NAME so the
# bg record carries from_session=<name> - lets a REPL that opens the same session
# detect it's being driven by this live worker and offer a take-over (no .lock is
# held: a worker runs headless with autosave off).
try:
sess_dir = (Path(_H["CONFIG_DIR"]) / "sessions").resolve()
if p.resolve().parent == sess_dir and p.suffix == ".json":
seed_session_name = p.stem
except Exception:
seed_session_name = ""
# SAFE-RECURSION governor: is THIS process allowed to spawn a worker at all, and if
# so how many? The driver (worker_depth 0) is governed only by max_concurrent. A
# WORKER (depth >= 1) must pass two gates before it can spawn a child:
# (a) DEPTH: my depth < worker_max_depth (default 1 -> a depth-1 worker is blocked;
# recursion is opt-IN by raising the ceiling).
# (b) CHILD BUDGET: I was granted a child budget by my parent, and I have some left
# (each spawn decrements it). Grandchildren get a SHRINKING budget so a subtree
# is bounded by depth x a decrementing count, not a flat number.
# Both refuse rather than silently succeed - a misfire can't fork-bomb the API.
_cfg = _H.get("cfg")
my_depth = int(getattr(_cfg, "worker_depth", 0) or 0)
ok, why, child_budget = _governor_check(my_depth)
if not ok:
return why
# Operator ceiling: concurrency.
bg_list = _H["bg_list"]
max_conc = _ceiling("max_concurrent")
live = bg_list(kind="worker")
if max_conc and len(live) >= max_conc:
return (f"error: at the worker limit ({max_conc} running). Wait for one to "
f"finish (its result reaches you automatically) or raise "
f"LEANCODER_WORKER_MAX_CONCURRENT.")
# Model + provider. Default: inherit the driver's current model+provider (both
# blank -> the worker keeps whatever it activates). A `model` may be ANY model on
# ANY enabled provider; we infer its provider from the enabled-provider->models
# map. An explicit `provider` disambiguates (same model name on two backends) or
# forces a backend. Validated against the worker allowlist + real availability so
# a bad name fails HERE (loud) rather than silently falling back inside the worker.
model = (args.get("model") or "").strip()
provider = (args.get("provider") or "").strip()
brain_host = (args.get("brain_host") or "").strip()
# brain_host relocates INFERENCE to a different ollama endpoint. The model then
# lives on THAT box, not the driver's, so we can't validate it against the driver's
# enabled-provider models - it's implicitly the ollama provider on the remote
# endpoint. Resolve the alias and default provider to ollama.
if brain_host:
resolve = _H.get("resolve_host") or (lambda h, m=None: h)
machines = getattr(_H.get("cfg"), "machines", {}) or {}
brain_host = _H.get("_norm_host", lambda h: h)(resolve(brain_host, machines))
if not provider:
provider = "ollama"
if provider != "ollama":
return "error: brain_host is ollama-only (inference endpoints are ollama URLs)."
if (model or provider) and not brain_host:
pmodels = _H.get("enabled_provider_models", lambda: {})()
if provider and provider not in pmodels:
return (f"error: provider '{provider}' is not enabled. Enabled: "
f"{', '.join(pmodels) or '(none)'}.")
if model:
allow = _model_allowlist()
if allow and model not in allow:
return f"error: model '{model}' is not in the worker allowlist ({', '.join(allow)})."
# Which enabled provider(s) offer this model?
hosts = [p for p, ms in pmodels.items() if model in (ms or [])]
if provider:
if model not in (pmodels.get(provider) or []):
avail = ', '.join(pmodels.get(provider) or []) or '(unknown)'
return (f"error: model '{model}' is not available on provider "
f"'{provider}'. available there: {avail}")
elif len(hosts) == 1:
provider = hosts[0] # unambiguous -> infer it
elif len(hosts) > 1:
return (f"error: model '{model}' exists on multiple enabled providers "
f"({', '.join(hosts)}); pass provider= to disambiguate.")
else:
allm = ', '.join(sorted({m for ms in pmodels.values() for m in ms})) or '(unknown)'
return (f"error: model '{model}' is not available on any enabled "
f"provider. available: {allm}")
elif brain_host and model:
# Still honour the allowlist (operator lockdown) even for a remote-brain model.
allow = _model_allowlist()
if allow and model not in allow:
return f"error: model '{model}' is not in the worker allowlist ({', '.join(allow)})."
# Where the worker's TOOLS run. Precedence:
# 1. an explicit `host` arg -> run the worker's tools on THAT machine. We ensure a
# live ssh master to it HERE, in the foreground (the one place a password can be
# entered - a detached worker can't answer a prompt). Key auth is silent.
# 2. else the PARENT's live remote -> ride the same executor path the parent uses.
# 3. else local on the driver.
want_host = (args.get("host") or "").strip()
if want_host:
ensure = _H.get("ensure_worker_master")
if ensure is None:
return "error: this build can't target a host for a worker (update lean-coder)."
remote = ensure(want_host)
if "error" in remote:
return (f"error: {remote['error']}. The worker was NOT dispatched - "
f"connect/authenticate to '{want_host}' works from your terminal, "
f"so try again once it's reachable.")
wcwd = (args.get("cwd") or "").strip() or remote["cwd"]
else:
remote = _H.get("active_remote", lambda: None)()
if remote:
# A detached worker can't answer an auth prompt, so the master MUST be live NOW
# (we're in the foreground dispatch call - the one place a password could be
# entered). Verify it; if it's down, tell the operator to reconnect.
alive = _H.get("_ssh_master_alive", lambda *a: False)(remote["host"], remote["ctl"])
if not alive:
return (f"error: connected to {remote['host']} but its ssh master isn't live "
f"- a worker can't authenticate on its own. Run a remote command (or "
f"/connect {remote['host']}) to re-establish it, then retry.")
# cwd is on the REMOTE: default to the parent's remote cwd, don't stat locally.
wcwd = (args.get("cwd") or "").strip() or remote["cwd"]
else:
wcwd = (args.get("cwd") or "").strip() or cwd
if not Path(wcwd).is_dir():
return f"error: cwd is not a directory: {wcwd}"
# Leash: cap the requested grant at the parent's live authority (never exceed it).
leash, downgraded = _capped_leash(args.get("leash") or "r")
cap_note = ""
if downgraded:
cap_note = (f" (capped to '{leash}' - a worker can't exceed your current "
f"'{getattr(_H.get('cfg'), 'leash', '?')}' leash)")
# Optional per-worker tool allowlist. Accept a list or a comma string; validate every
# name against what the PARENT actually exposes so a typo fails loud HERE, not silently
# inside the worker. Meta tools (plan/note/compaction) are always kept, so a caller
# need not list them. Empty after filtering -> treat as no restriction.
tools_arg = args.get("tools")
tools_csv = ""
if tools_arg:
if isinstance(tools_arg, str):
want_tools = [t.strip() for t in tools_arg.split(",") if t.strip()]
else:
want_tools = [str(t).strip() for t in tools_arg if str(t).strip()]
_META = {"update_plan", "note", "request_compact"}
want_tools = [t for t in want_tools if t not in _META]
if want_tools:
parent_tools = _H.get("active_tool_names", lambda: set())()
if parent_tools:
bad = [t for t in want_tools if t not in parent_tools]
if bad:
return (f"error: tool(s) not available to you: {', '.join(bad)}. "
f"You can only grant tools you have. Available: "
f"{', '.join(sorted(parent_tools))}.")
# A named tool the worker's LEASH would filter out must fail here, not be
# reported as granted and then vanish from the worker's surface.
_need = _H.get("tool_min_leash")
if _need:
_lv = _H["LEASH_LEVELS"]
short = [(t, _need(t)) for t in want_tools]
short = [(t, n) for t, n in short if n and _lv.index(n) > _lv.index(leash)]
if short:
return (f"error: at leash '{leash}' the worker can't use "
+ ", ".join(f"{t} (needs '{n}')" for t, n in short)
+ ". Dispatch with a higher leash; tools=[...] still limits it to "
"exactly the tools you list.")
tools_csv = ",".join(want_tools)
# Optional named task-board (Phase 2b): assign this worker to a task DAG board by NAME.
# The worker's run_agent_brief auto-loads the `board` lean-tool when this is set (so it
# can report done/list) and injects the report-done contract into its brief. Distinct
# from the int `board` claim-session above. The driver has usually already added+assigned
# the task on the board; this just tells the worker WHICH board to report to.
taskboard = (args.get("taskboard") or "").strip()
# Optional seed STATE the parent curated: a background blob, a starting plan, and
# notebook lines. NOT size-capped - "share state, not context" is a discipline for
# the parent to exercise, not something to enforce by clipping the payload to a
# useless state (an emergency brief may legitimately need a lot of context; a hard
# cap could silently truncate the one step that matters). Passed through whole.
seed_context = (args.get("context") or "").strip()
seed_plan = (args.get("plan") or "").strip()
seed_notes = (args.get("notes") or "").strip()
# Per-worker iteration budget: the driver MAY request a cap at dispatch (e.g. a tight
# leash for a scout, a longer one for a refactor), but it is capped at the operator
# ceiling (env > cfg > default) - a grant is never above the grantor's authority, and
# an env lockdown stays a hard limit. Junk/non-positive -> silently fall back to the
# ceiling (never crash on a bad arg).
_ceil_iter = _ceiling("max_iterations")
_req_iter = args.get("iterations")
max_iter = _ceil_iter
if _req_iter is not None:
try:
_ri = int(_req_iter)
if _ri > 0:
max_iter = min(_ri, _ceil_iter)
if _ri > _ceil_iter: # tell the driver its request was clamped (silent before)
cap_note += (f" (iterations {_ri} capped to the {_ceil_iter} ceiling - "
f"raise via /set worker_max_iterations)")
except (TypeError, ValueError):
pass
idle_timeout = _ceiling("idle_timeout")
# Progress-staleness watchdog: bark (alert the parent), never bite. The worker bumps
# its .progress file every iteration, so if that mtime goes stale the worker is stuck
# (model looping / hung tool) rather than just quiet - distinct from idle_timeout,
# which fires on the PARENT dying. Default = half the idle timeout, floored at 300s;
# silence != dead, so it only alerts and the operator decides (cancel/inject).
hb_timeout = max(300, idle_timeout // 2) if idle_timeout else 0
# Write the brief + result-file targets in this driver's run dir.
wdir = _workers_dir()
try:
wdir.mkdir(parents=True, exist_ok=True)
except OSError as e:
return f"error: cannot create workers dir: {e}"
# Unique per worker: two workers dispatched in the SAME second must not share a
# brief/result path (os.getpid() is the parent's - identical for all of them), or
# the second's ===RESULT=== overwrites the first's. A per-session monotonic counter
# guarantees uniqueness even within one second.
_H["_seq"] = _H.get("_seq", 0) + 1
stamp = f"{time.strftime('%Y%m%d-%H%M%S')}-{os.getpid()}-{_H['_seq']}"
brief_file = wdir / f"{stamp}.brief"
result_file = wdir / f"{stamp}.result"
# Checkpoint the transcript (opt-in) so a dead/timed-out worker can be action='resume'd.
# A session-seeded worker turns it ON regardless (so it can be promoted back later).
do_checkpoint = bool(getattr(_cfg, "worker_checkpoint", False)) or bool(seed_resume)
# Shared swarm board session: peers coordinate on the driver's board (keyed by the
# driver's pid). A worker dispatching a child passes its own inherited board session
# DOWN so a whole subtree shares one board; the driver seeds it with its own pid.
board_session = int(getattr(_cfg, "worker_board_session", 0) or 0) or os.getpid()
try:
brief_file.write_text(_compose_brief(task, model, wcwd, max_iter, leash=leash,
provider=provider, brain_host=brain_host,
tools=tools_csv, context=seed_context,
plan=seed_plan, notes=seed_notes,
depth=my_depth + 1, child_budget=child_budget,
checkpoint=do_checkpoint, board=board_session,
taskboard=taskboard, resume=seed_resume))
except OSError as e:
return f"error: cannot write brief file: {e}"
# The worker PROCESS always runs on the driver; when attached to a remote its
# driver-local cwd is irrelevant (tools run remotely), so launch it from the parent's
# cwd there. --cwd is the worker's LOCAL cfg.cwd (driver); when remote, run_agent_brief
# leaves cfg.cwd at the driver default and the remote workspace carries wcwd instead.
launch_cwd = cwd if remote else wcwd
cmd = (f"{_self_argv()} --agent-run "
f"--brief-file {shlex.quote(str(brief_file))} "
f"--result-file {shlex.quote(str(result_file))} "
f"--cwd {shlex.quote(launch_cwd)}")
if remote:
cmd += (f" --remote-host {shlex.quote(remote['host'])} "
f"--remote-ctl {shlex.quote(remote['ctl'])}")
if brain_host:
# Point the worker's INFERENCE at a different ollama endpoint than the driver.
# (Distinct from --remote-host, which relocates the worker's TOOLS.)
cmd += f" --host {shlex.quote(brain_host)}"
launched = _H["bg_launch"](cmd, cwd=launch_cwd, kind="worker", idle_timeout=idle_timeout,
heartbeat_timeout=hb_timeout,
heartbeat_file=str(brief_file) + ".progress",
meta=({"from_session": seed_session_name}
if seed_session_name else None))
if "error" in launched:
return f"error: could not launch worker: {launched['error']}"
# Remember the result-file per pid so /worker can harvest it.
where = remote["host"] if remote else "local"
_H["workers"][launched["pid"]] = {"result": str(result_file), "task": task,
"started": time.time(), "model": model or "(default)",
"where": where, "log": launched.get("log"),
"brief": str(brief_file),
"checkpoint": do_checkpoint, "cmd": cmd,
"launch_cwd": launch_cwd, "idle_timeout": idle_timeout,
"hb_timeout": hb_timeout, "taskboard": taskboard}
mstr = model or "current model"
lstr = {"r": "read-only", "rw": "read+write", "rwe": "read+write+exec"}.get(leash, leash)
_H["workers"][launched["pid"]]["leash"] = leash
wnote = f"on {remote['host']}" if remote else "local (driver)"
tnote = f", tools limited to [{tools_csv}]" if tools_csv else ""
seeds = [n for n, v in (("context", seed_context), ("plan", seed_plan),
("notes", seed_notes)) if v]
snote = f", seeded {'+'.join(seeds)}" if seeds else ""
return (f"dispatched worker pid {launched['pid']} ({mstr}, {lstr}, tools {wnote}"
f"{tnote}{snote}, cwd {wcwd}){cap_note}.\n"
f"It runs in the BACKGROUND. You do NOT need to poll for it: when it "
f"finishes, its result is delivered to you automatically on a later turn. "
f"Do NOT call bg_status (workers are hidden from it by design) and do NOT "
f"sleep/cat the result file by hand - just carry on with other work, or if "
f"you have nothing else to do, end your turn and wait for the finish notice. "
f"(The human can run /worker to inspect it.)")
def _read_result(path):
"""The worker's ===RESULT=== block from its result file, or None if not written
yet / unreadable."""
try:
return _H["_extract_marked"](Path(path).read_text(), _H["RESULT_MARK"])
except OSError:
return None
def _read_usage(result_path):
"""A finished worker's token spend from its '<brief>.usage' sidecar (written on exit
by run_agent_brief). Returns (in_tokens, out_tokens); (0, 0) if absent/unreadable.
Observability: summed for /worker status, nothing gates on it."""
try:
txt = Path(_brief_from_result(result_path) + ".usage").read_text()
except OSError:
return 0, 0
vin = vout = 0
for tok in txt.split():
k, _, v = tok.partition("=")
try:
if k == "in":
vin = int(v)
elif k == "out":
vout = int(v)
except ValueError:
pass
return vin, vout
def _accrue_usage(meta):
"""Add a finished worker's token spend to the session's shared totals, ONCE. Guarded
by meta['usage_counted'] so a re-scan can't double-count. Best-effort - a worker with
no .usage sidecar (crashed before writing) simply contributes 0."""
if meta.get("usage_counted"):
return
meta["usage_counted"] = True
vin, vout = _read_usage(meta["result"])
meta["tokens_in"], meta["tokens_out"] = vin, vout
_H["tokens_in"] = _H.get("tokens_in", 0) + vin
_H["tokens_out"] = _H.get("tokens_out", 0) + vout
def _worker_tokens_spent():
"""Cumulative tokens (in+out) spent by all FINISHED workers this session (their .usage
sidecars, accrued once each in _accrue_usage). Still-running workers count 0 until they
exit. Pure observability - surfaced in /worker status; nothing gates on it."""
return _H.get("tokens_in", 0) + _H.get("tokens_out", 0)
def _worker_stderr_tail(result_path, n=15):
"""Last few lines of a FAILED worker's combined stdout+stderr log (bg_launch runs it
with stderr merged into the log), so a crash/early death leaves a visible trace in the
failure notice instead of vanishing. "" if we have no log path or it's unreadable."""
meta = None
for m in _H["workers"].values():
if m.get("result") == result_path:
meta = m
break
log = (meta or {}).get("log")
tail = _H.get("_bg_log_tail")
if not (log and tail):
return ""
try:
return tail(log, n)
except Exception:
return ""
def _read_progress(result_path):
"""The worker's deterministic progress heartbeat (iter/elapsed/last-tool/intent),
written each iteration to <brief>.progress by run_agent_brief. Returns the text
(stripped) or "" if none yet. Facts only - no tokens, no model cooperation."""
try:
return Path(_brief_from_result(result_path) + ".progress").read_text().strip()
except OSError:
return ""
def _read_planview(result_path):
"""The worker's CURRENT pinned plan, mirrored to <brief>.planview each iteration by
run_agent_brief. Lets a parent read the live plan (with the worker's own checkbox
progress) so action='set_plan' can send back an EDITED copy instead of a blind
overwrite. Returns the plan text (stripped) or "" if the worker has no pinned plan."""
try:
return Path(_brief_from_result(result_path) + ".planview").read_text().strip()
except OSError:
return ""
def _inject_count(result_path):
"""How many injects this worker has CONSUMED (lines in <brief>.injects.log), and the
last one's teaser - the delivery ack for action='inject'. (0, "") if none."""
try:
lines = [l for l in Path(_brief_from_result(result_path) + ".injects.log")
.read_text().splitlines() if l.strip()]
except OSError:
return 0, ""
if not lines:
return 0, ""
last = lines[-1].split("consumed:", 1)[-1].strip()
return len(lines), last
def _finished_notice():
"""Scan tracked workers for ones that FINISHED since we last looked, and build the
notice to inject into the model's next turn. A worker finishes one of two ways:
- SUCCESS: its result file got a ===RESULT=== block -> report the (capped) result;
- FAILED: the process is GONE but no result was ever written (crash / early death /
lease-kill before writing) -> report the failure + any stderr tail, so the model
is never left waiting on a worker that will never report. Without this second
branch a dead-without-result worker sits in silent limbo forever (the bug this
fixes).
Reports each newly-finished worker once (dedup via meta['announced']); when that clears
the LAST outstanding worker, appends an 'all N finished' line. Fires a desktop ping per
finish if the notify lean-tool's _ping is present. "" when nothing new finished."""
workers = _H["workers"]
if not workers:
return ""
ping = _H.get("ping")
rows = _worker_rows() # {pid: bg_status row}; absent row => proc gone
lines = []
just_finished = []
for pid, meta in workers.items():
if meta.get("announced"):
continue
res = _read_result(meta["result"])
if res is not None:
meta["announced"] = True
just_finished.append(pid)
_accrue_usage(meta) # meter this worker's token spend into the session
tail = res.strip()
if len(tail) > 1500: # keep a long result from flooding the turn
tail = tail[:1500] + " ...[truncated; /worker %d for the full result]" % pid
lines.append(f"worker {pid} finished ({meta.get('leash','r')}, {meta['model']}) - "
f"task: {meta['task'][:100]}\nresult:\n{tail}")
if ping:
try:
ping("lean-coder worker", f"worker {pid} finished")
except Exception:
pass
continue
# No result. Only a FAILURE if the process is actually gone (a running worker
# just hasn't written yet - leave it). A missing bg_status row => proc gone.
if rows.get(pid) is not None:
continue # still running; wait
meta["announced"] = True
just_finished.append(pid)
errtail = _worker_stderr_tail(meta.get("result"))
msg = (f"worker {pid} FAILED ({meta.get('leash','r')}, {meta['model']}) - "
f"it exited without writing a result "
f"(crash / early death / lease-kill).\ntask: {meta['task'][:100]}")
if errtail:
msg += f"\nstderr tail:\n{errtail}"
lines.append(msg)
if ping:
try:
ping("lean-coder worker", f"worker {pid} FAILED (no result)")
except Exception:
pass
if not just_finished:
return ""
total = len(workers)
outstanding = sum(1 for m in workers.values() if not m.get("announced"))
if outstanding == 0 and total > 1:
lines.append(f"all {total} dispatched workers have finished - collect any "
f"you haven't used with /worker.")
if ping:
try:
ping("lean-coder workers", f"all {total} workers finished")
except Exception:
pass
return "[" + "\n\n".join(lines) + "]"
def _worker_rows():
"""{pid: bg_status row} for this session's workers, keyed by pid. Empty if none."""
return {r["pid"]: r for r in _H["bg_status"](os.getpid(), kind="worker")}
def _worker_status(pid=None):
"""MODEL-facing status of dispatched workers (the tool's action='status'). One
line per worker: state, runtime, result-ready flag, task teaser. `pid` narrows to
one. Note: finished results reach the model automatically - this is for the rare
case it needs to check a slow/suspect one."""
workers = _H["workers"]
if not workers:
return "no workers dispatched this session."
rows = _worker_rows()
if pid is not None:
try:
pid = int(pid)
except (TypeError, ValueError):
return f"error: bad pid {pid!r}."
meta = workers.get(pid)
if not meta:
return f"no worker with pid {pid} this session."
workers = {pid: meta}
out = []
for wp, meta in workers.items():
row = rows.get(wp)
state = row["state"] if row else "gone"
runtime = row["runtime"] if row else "?"
ready = _read_result(meta["result"]) is not None
if ready and not meta.get("usage_counted"):
_accrue_usage(meta) # keep the shared spend current when status is read
line = (f"pid {wp} {state} (ran {runtime}) "
f"result {'ready' if ready else 'pending'} {meta['model']} "
f"task: {meta['task'][:80]}")
if ready and meta.get("tokens_in") is not None:
_t = meta.get("tokens_in", 0) + meta.get("tokens_out", 0)
if _t:
line += f" ({_t:,} tok)"
prog = _read_progress(meta["result"])
if prog and not ready:
line += "\n progress: " + " | ".join(prog.splitlines())
if not ready and pid is not None:
# Only when narrowed to one worker (the plan can be many lines) - show its live
# pinned plan so the parent can steer/edit it via set_plan.
plan = _read_planview(meta["result"])
if plan:
line += "\n plan:\n " + "\n ".join(plan.splitlines())
n_inj, last_inj = _inject_count(meta["result"])
if n_inj:
line += f"\n injects delivered: {n_inj} (last: {last_inj[:80]})"
out.append(line)
spent = _worker_tokens_spent()
if spent:
out.append(f"total worker spend: {spent:,} tokens (finished workers)")
return "\n".join(out)
def _worker_models():
"""List the models available to dispatch a worker on, grouped by enabled provider,
so the driver can pick a cheap leaf model without guessing an exact id. Honours the
operator worker-model allowlist when one is set. Read-only, no side effects."""
pmodels = _H.get("enabled_provider_models", lambda: {})()
if not pmodels:
return ("no enabled providers report models (a remote-brain ollama worker can "
"still be dispatched with brain_host= + model=).")
allow = _model_allowlist()
cur = getattr(_H.get("cfg"), "model", "") or ""
lines = [_H["bold"]("models you can dispatch a worker on:")]
for prov, models in pmodels.items():
models = list(models or [])
if allow:
models = [m for m in models if m in allow]
head = _H["cyan"](prov)
if not models:
lines.append(f" {head}: (none available)")
continue
lines.append(f" {head}:")
for m in models:
tag = _H["dim"](" (current default)") if m == cur else ""
lines.append(f" {m}{tag}")
lines.append(_H["dim"]("dispatch with model=<id> (provider= only if the id is on "
"more than one). Omit model= to use your current default."))
if allow:
lines.append(_H["dim"](f"note: operator allowlist active - only these are grantable."))
return "\n".join(lines)
def _worker_result(pid, page=None):
"""MODEL-facing full result (the tool's action='result'): return a worker's
COMPLETE ===RESULT=== block, paged like 'transcript' (page=) so the core ingestion
cap never silently drops the middle of a long one. The turn-rider finish notice caps
the result at 1500 chars; this is how the model pulls the rest without a human
running /worker. `pid` is required (which worker); with no pid, if exactly one
worker exists, use it, else ask which."""
workers = _H["workers"]
if not workers:
return "no workers dispatched this session."
if pid is None:
if len(workers) == 1:
pid = next(iter(workers))
else:
return ("error: action='result' needs a pid (which worker). "
"Use action='status' to list them.")
try:
pid = int(pid)
except (TypeError, ValueError):
return f"error: bad pid {pid!r}."
meta = workers.get(pid)
if not meta:
return f"no worker with pid {pid} this session."
res = _read_result(meta["result"])
if res is None:
return (f"worker {pid} has no result yet (still running or terminated before "
f"writing one). Use action='status' to check its state.")
# Collecting the result here counts as harvesting it: mark announced + accrue usage so
# a later _finished_notice() scan can't re-report this same worker as FAILED once its
# process exits and its result sidecar is reaped (the false 'exited without writing a
# result' notice - we already HAVE the result in hand right now).
if not meta.get("announced"):
meta["announced"] = True
_accrue_usage(meta)
head = f"worker {pid} result (task: {meta['task'][:100]})"
pages = _paginate_lines(res.strip(), _RESULT_PAGE)
if len(pages) <= 1:
return f"{head}:\n{res.strip()}"
try:
p = min(max(1, int(page or 1)), len(pages))
except (TypeError, ValueError):
p = 1
footer = ("" if p >= len(pages)
else f"\n\n[more: {len(pages) - p} page(s) left - action='result' page={p + 1}]")
return f"{head} - page {p}/{len(pages)}:\n{pages[p - 1]}{footer}"
def _paginate_lines(text, size):
"""Split text into pages of <= size chars, breaking at line boundaries (a single
over-long line is hard-split so no page can exceed size)."""
pages, cur = [], ""
for ln in text.split("\n"):
while len(ln) > size:
if cur:
pages.append(cur)
cur = ""
pages.append(ln[:size])