Skip to content

feat(delegation): stop delegated members with acknowledged, settled receipts - #5308

Open
songoow wants to merge 13 commits into
loopx-project:mainfrom
songoow:codex/delegation-stop
Open

songoow wants to merge 13 commits into
loopx-project:mainfrom
songoow:codex/delegation-stop

Conversation

@songoow

@songoow songoow commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

"## Why\n\nRoadmap R2 requires that stop, cancel and restart retain work and fence old executors. Delegated members could not be stopped at all. The worker held its operation lock for the whole run and rewrote the execution record from memory, so a stop written into that record could neither reach it nor survive it, and a killed run left its Turn and hard lease unaccounted for. Pausing the coordinator or disabling LoopX mode does not stop admitted members either.\n\n## What changed\n\n- Typed state and decision (TS). stopped is a terminal observation reachable from prepared, running and turn_returned. The new collaboration.delegation.stop decision settles a stop only with an acknowledgement from the worker, a free operation lock, a released Turn lane and an exited native Host process group. Free locks without an acknowledgement are unknown, never a fabricated settlement, and elapsed time alone never produces a receipt.\n- Stop receipt beside the record. executions/<h>/<op>.stop.json is written only under the existing .dispatch lock, never into the execution record. Phases are requested, acknowledged, settled, with terminals unknown and noop. It records the requester, the worker (pid, process group, host), the acknowledgement, the lease release and the settlement facts.\n- Delegations.stop. A terminal record returns an identical noop receipt on every call. With no running worker the requester marks the record stopped, releases the hard lease and settles. A same-host worker gets SIGTERM on its process group, and SIGKILL only if it still holds the operation after the grace period. Another host's worker is never signalled and finds the request itself.\n- Settled means the host is gone too. The Turn names the process group its Host supervisor owns in executions/<h>/<op>.host.json, written before the request is sent and updated when the supervisor reports the spawned group; the supervisor reports before the Host gets its input, so no Host runs unattributed. settled additionally requires that group to have exited, read back read-only while the TS supervisor stays the only owner that terminates a Host. A group that cannot be attributed, or one still terminating, keeps the stop acknowledged so a later same-identity read can settle it.\n- Worker and fencing. A SIGTERM handler and checkpoints before running, before each run-once and before completing the Todo raise on a stop. Every record write re-reads the stop under .dispatch and refuses a stop this process did not acknowledge, so a late or other-host worker records no Turn result and completes no Todo. The in-progress Turn journal is kept for inspection.\n- No lock-taking lane probes. Settlement reads the lane's last holder record through the lock-free turn_lane_liveness: released, dead or absent frees the lane; a live same-host holder frees it only outside the recorded worker's process group; another host's or an unattributable holder keeps the stop open, so a stop never refuses a legitimate Turn of the same member. The operation lock has no holder-record reader yet, so once the stop is written settlement probes that kernel lock for an instant; resume, its only single-flight acquirer, refuses a stopped operation before it touches the lock. A regression test races a real lane acquisition against settlement.\n- Surfaces. loopx delegation stop --operation-id ID --execute and the stop_delegation MCP tool return the same receipt. resume refuses stopped work, wait returns on stopped, read shows the stop phase, and inventory and context projections count stopped receipts.\n- Docs. local-delegation.md and goal-chat-continuation.md describe stopping a member and reading its receipt, in English and Chinese.\n\n## Stacking and coordination\n\n- This branch includes #5306's commit feat(turn-driver): read Turn lane liveness without taking the lane unchanged, to reuse lock_holder_host_label() and the lock-free lane read. It drops out once #5306 merges; review from feat(delegation): add stopped observation.\n- #5304 edits collaboration_mcp.py and delegation.ts at adjacent insertion points. The second to merge rebases with a keep-both resolution; #5304 treats any terminal non-accepted status, including stopped, as no wake.\n\n## Checks run on this head\n\n| Check | Result |\n|---|---|\n| pytest tests/test_local_delegation.py tests/test_delegation_cli.py tests/test_delegation_inventory.py tests/test_collaboration_mcp.py tests/test_turn_lane_fence.py plus module budget, import boundary and registry census tests | 79 passed |\n| node --test tests/control_plane_ts/delegation.test.ts | 18 passed, 0 failed |\n| npm run typecheck:control-plane | ok |\n| examples/docs-governance-smoke.py, examples/semantic-vocabulary-drift-smoke.py | ok |\n| loopx canary premerge --from-git-diff | passed: tier=standard, changed_files=18, surfaces=control_plane/docs_project_content/public_boundary/python; selected=16, failures=0 |\n\nNot run: stopping a member on another real host (covered by fixtures only), the packaged App and Lark. The App has no stop control yet; CLI and MCP are the entry points.\n\n## Rollback\n\nRevert the PR. Stop receipts stay as files beside their records and the older code ignores them. Caution: the older resume treats every status other than accepted and rejected as unfinished, so after a revert a resume on a stopped operation would restart it. Check loopx delegation operations for stopped operations before reverting, and do not resume them afterwards.\n\n## Bounded future-facing refactor\n\nApplied: stop settlement reuses the file-lock owner's holder records instead of a second host helper and a lock-taking probe. Deferred: an App stop control and cross-host stop acknowledgement, which need R6 host identity.\n\nThis is a control-plane change; it is left for maintainer review and merge.\n\n\ud83e\udd16 Generated with Claude Code\n"

songoow and others added 8 commits September 29, 2026 12:33
Add `lock_holder_liveness` to the file-lock owner and `turn_lane_liveness`
on top of it. Both classify a lane's last executing Turn from its holder
record alone: `released` on a clean exit, `dead` when the record names
this machine and the pid is gone, `foreign_host` when the pid cannot be
checked here, `unreadable` when a lock file carries no parseable record,
and `live` only when a same-host pid is still alive.

The probe never touches the kernel lock. A probe that acquired it for an
instant would refuse a real `run-once --execute` racing that instant with
`turn_lane_in_flight` for nothing; the new test drives the real fence
wrapper concurrently with a continuous probe and proves the Turn is
admitted exactly once. The holder host label is now single-sourced so the
writer and the readers cannot disagree on what "this machine" is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
Delegated members had no stopped observation: prepared, running and
turn_returned could only end in accepted or rejected. Add "stopped" as a
terminal observation reachable from the three open states and keep the
inventory check driven by the same transition table.

Add the exported collaboration.delegation.stop decision: a stop request
settles only with an acknowledgement from a process that held the operation
lock plus a free operation lock and a free Turn lane lock. Free locks with no
acknowledgement are "unknown", never a fabricated settlement, and a grace
timeout alone changes nothing while a lock is still held. Register the RPC
handler next to the existing delegation observation handlers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
…eceipts

Delegated members could not be stopped: the worker held the operation lock
for the whole run and rewrote the execution record from memory, so nothing
written into that record could reach it or survive it, and a killed run left
its Turn and hard lease unaccounted for.

Add a stop receipt beside the execution record (executions/<h>/<op>.stop.json)
written only under the existing .dispatch lock, never into the record. Its
phases are requested -> acknowledged -> settled with the terminals unknown and
noop, and it records the requester, the lock-holding worker (pid, process
group, host), the acknowledgement (pid, observed status, Turn key), the lease
release outcome and the settlement facts (operation lock, Turn lane lock,
Turn journal status).

Delegations.stop: a terminal record returns an identical noop receipt on every
call. Otherwise the request is written; when no worker holds the operation the
requester takes the lock, marks the record stopped, releases the hard lease
and settles. A same-host holder is SIGTERMed by process group (its run-once
child and host bridge follow) and SIGKILLed only if it still holds the
operation after the grace; another host's holder is never signalled and finds
the request itself. Settlement is the typed collaboration.delegation.stop
decision over lock facts: elapsed time is never a receipt.

Worker: a SIGTERM handler raises DelegationStopRequested once; checkpoints
before running, before each run-once and before completing the Todo raise on
a stop file; the record is written only through a fenced write that re-reads
the stop file under .dispatch and raises DelegationFenced for a stop this
process did not acknowledge, so a late-returning or other-host worker records
no Turn result and completes no Todo. Acknowledgement marks the record
stopped, releases the hard lease and leaves the in_progress Turn journal for
inspection. execute records the worker's pid/pgid/host at entry and
acknowledges from under the lock when a stop already exists.

resume refuses stopped work ("start a new operation id"), wait returns on
stopped, and read exposes the stop phase. file_lock gains local_lock_host and
read_lock_holder so the stop names the holder exactly as lock records do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
Add `loopx delegation stop --operation-id ID --execute` next to
start/resume/adopt (--execute is required for the same reason) and the
`stop_delegation(operation_id)` MCP tool, which runs the blocking stop off
the event loop like wait_delegation. Both return the same receipt and never
resume or rerun work.

The inventory reader skips `<op>.stop.json` sidecars, which sit beside
execution records but are not records, and the delegation context and
subagent context projections count `stopped` receipts instead of dropping
them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
Semantics-first coverage for stopping delegated members:

- stop while a detached worker runs a sleeping fixture host: the worker
  acknowledges SIGTERM under its own lock, the receipt settles only with a
  free operation lock and a lockable Turn lane, the record bytes stay frozen
  afterwards, the Todo stays open, the host and worker processes are gone,
  the Turn journal stays in_progress, and resume is refused without spawning;
- stop after accepted: noop, identical on repeat, no stop file and artifacts
  unchanged; stop without a holder is acknowledged by the requester;
- a worker SIGKILLed before acknowledging settles as unknown, not stopped,
  and resume stays refused;
- a fenced write after another process's acknowledged stop raises and writes
  nothing, while an unacknowledged stop is taken from under the lock at entry;
- CLI stop requires --execute and repeats its receipt; inventory pages past
  stop sidecars and reads a stopped record as stopped;
- TS: stopped is terminal and reachable only from open observations, and the
  stop decision settles only on acknowledgement plus free locks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
Describe `delegation stop --execute` / `stop_delegation` in both reference
documents, in English and Chinese: where the request lives, how a same-host
worker is signalled and acknowledges, why another host's worker is left to
find the request, what settled, unknown and noop prove, and that stopped work
needs a new operation id while its Turn journal and open Todo remain for the
coordinator to inspect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
… lock

The stop settlement probed the member's Turn lane by acquiring its lock for
an instant, which could refuse a legitimate Turn of the same member racing
that instant with turn_lane_in_flight. It also carried its own host label
helper next to the file-lock owner's.

Settlement now reads the lane's last holder record through
turn_lane_liveness: released, dead or absent frees the lane; a live
same-host holder frees it only when it sits outside the recorded worker's
process group; another host's holder, an unattributable holder or an
unreadable record keep the stop open. The operation lock is read the same
way, so a stop never refuses a legitimate resume or status read. The local
host helper is deleted in favour of lock_holder_host_label. A regression
test races a real lane acquisition against settlement and proves the Turn
is admitted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
Regenerated with scripts/generate_project_registry_io_manifest.py after
rebasing onto main. Site ids and classifications are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: song <22676124+songoow@users.noreply.github.com>
@songoow
songoow force-pushed the codex/delegation-stop branch from d5aafc8 to 5e7b7bf Compare September 29, 2026 16:39

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

精确 head:5e7b7bfd8fb425312d25a771911984040ac9ee03。按 LoopX PR review capability 检查整份差异,未把相邻 PR 的发现移植到本 PR。

动机

Roadmap R2 要求 stop/cancel/restart 保留工作并 fence 旧执行者。已有 delegated member 无可用停止入口,杀进程也不能证明结果、Turn 和租约已妥善收尾。本 PR 增加 CLI/MCP 停止与持久 ACK,是合理的有界交付;但用户需要的是“停止已完成”的可信结果,不能在返回 settled 后原执行宿主仍可继续工作。

改动思路

复用既有 binding、GoalRef、operation 单飞锁、dispatch 锁及 Turn lane owner。stop sidecar 表达不可由普通运行状态推导的明确停止意图;worker 在自己的锁下 ACK,fenced writer 禁止较晚的结果/完成写入。TS owner 持有 requested/acknowledged/settled/unknown/noop 决策,Python 适配信号、权威租约释放和日志 IO。CLI delegation stop --execute 与 MCP 返回同一收据,read/wait/inventory 消费持久事实;它没有授予跨 host 信号、跨 peer 权限或 Goal 结算权。

具体改动

整份差异为 17 文件、+1222/−80:停止入口、typed lifecycle、operation worker、lane/holder 只读观察、上下文/清单、派生 registry 清单、中英文档与真实进程测试。检查范围包含随分支引入的 lock/lane seam,而不是只看 stop helper;没有引入相邻 PR 的 execution-facts 变更。sidecar 有明确生产者、requester 身份与锁下更新路径,停止后 resume 拒绝重新运行;未写 stop 意图的外部 SIGTERM 保留原 recoverable 路径。

关键代码讲解

  • Delegations.stop 校验现有 operation/binding,锁下保存停止意图,只向可归属的同 host worker 进程组发信号,终态返回稳定 noop。
  • _acknowledge_stop 在 operation 锁内标记 stopped、记录 ACK,清理 bootstrap 并尝试释放已有硬租约;ACK 本身不是宿主清理完成的证明。
  • _settle_stop 只提供 operation lock free 和 worker lane released 两项事实,再调用 decideDelegationStop;当前缺少实际 native host/后代已 drain 的事实。
  • native host process cleanup 已有独立进程组与异步清理 owner,SIGTERM 后有 300ms grace 再 SIGKILL。worker/lane 释放不能替它宣告清理完成。

对主干的风险

[P1,阻断] settled 可早于实际宿主和后代终止。 在 File、SQLite 两种真实权威后端,我沿用现有 native fixture,令宿主及其同组子进程忽略 SIGTERM,其他路径仍是实际 detached worker → CLI run-once → native Node bridge → host transport。调用 stop 后约 0.26s 返回 phase=settled、operation lock free 和 lane released,但用 macOS ps 独立检查,宿主与子进程此刻都仍存活。native cleanup 稍后会杀掉它们,因此这是“过早结算”,不是声称永久 orphan;这段窗口内旧效果仍可能执行,接续工作也可能与其重叠。

原因是 _signal_worker 的提前返回 与 _settle_stop 把锁释放当成 drain;native bridge/host 位于其他进程组,拥有独立的异步清理生命周期。请复用现有 native host cleanup/readback owner,只有实际 owning host/后代清理完成才能结算;缺少可归属证明时保持 acknowledged/unknown 并允许同身份读回恢复。不要用固定 sleep、扩大超时或跨 host kill 冒充证明。补一个“宿主和同组后代忽略 TERM”的真实 File/SQLite 回归,断言返回 settled 的那一刻两者已退出,并覆盖清理中断/重复读取不产生重复接纳或完成。可放入 tests/test_local_delegation.py,重跑本段所列 pytest/TS 验证。

现有测试只在 stop 返回后再等待退出;其 /proc 检查在 macOS 读不到路径时还会把存活误当退出。请一起改为平台有效的进程检查及即时后置条件。PR 正文也应同步实现:Turn lane 是只读 liveness,operation lock 在 stop 已写后仍有短暂 kernel probe,并非全部“no lock-taking”。

语义与 CI 对齐

亲自运行 66 项 Python、18 项 TS、控制面 typecheck 均通过;独立 native drain 反例在 File/SQLite 各失败一次。这证明现有绿测试没有覆盖上述后置条件。标准 canary 14 个选择检查及 5 个直接检查中,只有 twin-budget 检查失败:原始同一 full-tree budget owner 在不可变基线 902b99698050fc21af2c3a8b0d0d3b8561d0bdd7 和本 head 都报 44/43;扫描 owner、预算和 twin 列表未被此 PR 改动,故归为既有无关失败,不作为本次 request changes 的理由。基线完整 smoke 另有旧 registry 清单行号失配,未掩盖或称其通过。

七种普通 status/Goal Chat 路径在同一夹具下,完整基线/head 观察一致;new stop 是明确请求才触发,安装、帮助发现或普通 SIGTERM 不自动激活停止。远端 CI 未查询,打包 App(无 stop 控件)、Lark、跨真实 host 及真实模型未验证。复用既有 delegation vocabulary 的范围恰当,但 settled 目前违反其停止完成语义,不能由文档描述消除。

我的整体评价

REQUEST_CHANGES。长期推进和用户体验均存在具体回归风险:用户看到结束收据时旧执行仍在 drain。规模与 CLI/MCP 到持久 ACK/读回的 R2 切片大体相称;future-facing 检查建议把 stop 协调收敛到现有 collaboration 边界、复用 native cleanup 事实,避免 Python 再建一个 host 生命周期 owner。较大的模块拆分可在有 characterization 时另作有界整理,不作为本次 LOC 门槛。当前必须先修过早结算并用原生负例证明修复,再复审精确 head;保留 rollback 对 stopped 历史的 caution,不授予自合并权限。

English verdict: REQUEST_CHANGES — exact head 5e7b7bf. A real File/SQLite native-process counterexample returns settled while the owned host and descendant remain alive. The pre-existing 44/43 twin-budget failure is unrelated and is not the blocker.

…up exit

The worker and its Turn lane let go while the Host supervisor is still
terminating the Host, so their release never proved that the old executor
stopped. The Host transport now names the process group its supervisor owns
in a record beside the operation, the drain is read back from that record
without signalling anything, and the typed decision requires an exited group
before a stop may settle. A group that cannot be attributed keeps an
acknowledged stop open for a later same-identity read, and the TS supervisor
stays the only owner that terminates a Host.

Signed-off-by: song <liusongstep@gmail.com>
A real File/SQLite fixture whose host and same-group child ignore SIGTERM
asserts both are gone at the instant settled returns, with a platform-valid
process check in place of the /proc read that treated a live macOS process as
exited. Interrupted cleanup, repeated reads, an unacknowledged holder and an
unrecorded group get their own negative coverage.

Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
@songoow

songoow commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

New exact head: 609530d2fb64e86ae71eed13ce22d4abdfd7150e (3 commits + a signed merge of upstream/main 3ec049e13).

[P1, blocking] settled could precede the owned Host and descendants exiting — fixed

5d67e4795. The Turn now names the process group its Host supervisor owns in a record beside the operation (.host.json), written before the request is sent and updated when the supervisor reports the spawned group; the supervisor also reports before the Host gets its input, so a Host never runs before it is attributable. _settle_stop reads that record (read-only, no signalling) and decideDelegationStop requires a drained group: not_launched or drained plus acknowledgement, free operation lock and a released lane. draining keeps the phase acknowledged with host_process_still_running; an unattributable group keeps it open as host_process_drain_unproven so a later same-identity read can still settle.

Evidence: tests/test_local_delegation.py::test_stop_settles_only_after_the_owned_host_and_its_descendants_exit — real File/SQLite fixture whose host and same-group child ignore SIGTERM; it asserts both are gone at the instant stop returns settled, with settled.host_process == "drained". test_interrupted_host_cleanup_keeps_the_stop_open_until_a_reread_sees_it_drained covers cleanup interruption: the supervisor is stopped mid-cleanup, the stop returns acknowledged/host_process_still_running, three further calls are frozen at acknowledged, resume still refuses, the Todo stays open, one host invocation total; after the orphaned group is gone the next call settles with the same stop_id. No fixed sleep and no cross-host kill: the TS supervisor remains the only owner that terminates a Host, and _await_host_drain only polls release facts.

Attribution races: tests/control_plane/test_host_process.py::test_host_process_drain_reads_live_groups_and_refuses_unattributable_records (live supervisor/Host group → draining; other host, no group, malformed record → unattributable), plus test_host_process_record_names_the_owned_group_and_is_not_inherited (a nested LoopX inside the Host cannot overwrite its parent's record).

Process check: your /proc point is confirmed — the old helper reports a live macOS sleep as gone. process_gone now uses a signal probe plus ps -o stat=; the settled-instant assertion above is what makes it load-bearing.

PR body wording

Your note is right: the lane is read-only via turn_lane_liveness, but the operation lock still gets a brief kernel probe after the stop is written. _operation_lock_free carries that reason in code; I am correcting the "no lock-taking" phrasing in the description.

Not changed

The twin-budget 44/43 failure is pre-existing and unrelated: identical on 902b99698, 3ec049e13 and this head (cli-output-budget-regression-smoke fails the same 15 rows on all three with no head-only row), and the twin set is unchanged. Still untested: cross-host stop acknowledgement, the packaged App (no stop control), Lark, real hosts/models.

Checks on this head

63 Python (delegation CLI/inventory/MCP/local/Turn-lane/host-process) and 33 TS (delegation, host_process, collaboration_inbox_receipts) passed; typecheck:control-plane clean. Mutation: reverting each of the five fix points (readback, settlement fact, drain wait, TS drain requirement, supervisor-alive check) fails its test; restored green. canary premerge --from-git-diff --git-diff-base upstream/main: 14/14 selected, only the three pre-existing failures above.

中文摘要:settled 现在必须等原生 host 及其进程组退出;host 进程组由 TS supervisor 记录、只读回读,无法归属时保持 acknowledged 而不伪造结算。真实 File/SQLite 负例断言返回 settled 的瞬间宿主与同组子进程均已退出,并覆盖清理中断/重复读取、仓库清单旁文件的隔离。既有 44/43 与 CLI 预算失败与本 PR 无关。请复审精确 head。

Please re-review exact head 609530d2fb64e86ae71eed13ce22d4abdfd7150e.

@cocolord cocolord left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

这个 PR 要解决的是一个真实且高价值的问题:已派发的 delegated member 没有可验证的 stop,单纯杀 worker 也不能证明 Turn、原生 Host、后代进程和租约已经安全收尾。新 head 609530d2fb64e86ae71eed13ce22d4abdfd7150e 已修复我上次指出的 native Host 提前结算问题;File/SQLite 的真实进程组测试现在能证明返回 settled 的瞬间 Host 与同组 child 都已退出。

改动思路

实现复用现有 operation 单飞锁、dispatch 锁、Turn lane holder、TS collaboration decision、native Host supervisor 和 task-lease authority。stop.json 保存不可由普通运行状态推导的停止意图,host.json 保存 supervisor 产生的进程归属投影;worker 在 checkpoint 或 fenced write 处确认,decideDelegationStop 再根据 ACK、operation/lane 释放和 Host drain 选择 requested/acknowledged/settled/unknown。CLI 与 MCP 共用同一入口,read/wait/resume/inventory/context 共用持久事实。

具体改动

整份差异为 21 个文件、+1596/-81:12 个 runtime 文件增加 stop CLI/MCP、typed stopped/phase 决策、worker 信号与写 fence、holder liveness、Host spawned 回报/原子 sidecar、租约释放及各投影;7 个测试文件覆盖 TS 转移、真实进程组、File/SQLite、CLI/inventory/lane;2 个文档文件补充中英文使用与回滚语义。

关键代码讲解

  • Delegations.stop 写入一次 stop intent,只向可归属的同机 worker 进程组发信号,并复用同一 stop_id 读回。
  • _settle_stop 汇总 operation lock、Turn lane 与 host_process_drain;这是本轮对旧 blocker 的有效修复。
  • decideDelegationStop 是 typed phase owner,超时本身不制造 terminal receipt。
  • runHostProcess 在 Host 接收输入前报告 owned group;报告失败会取消,TS supervisor 仍是唯一 kill owner。
  • _execute 负责最终 Todo、reply 与 accepted 写入;当前剩余 blocker 都集中在 stop 与这些既有 authority 的交界。

对主干的风险

[P1,阻断] stop 与 Todo 完成/结果发布没有共同线性化边界。 _execute 在 1557 行 只做一次 pre-check,之后 _complete_delegated_todo 与 return_result 都在 dispatch fence 外提交,最后 _observe/_fenced_write 才重新看 stop。我在 File/SQLite 让 worker 停在 Todo effect 内,先通过公开 stop 写入请求,再放行 worker;两种后端都观察到 todo_completed、reply_published,随后却返回 phase=settled,status=stopped。这违反“late/other-host worker 不完成 Todo、不发布结果”的核心承诺,也可能让 successor 与已提交效果重叠。请让 stop 与两个外部 effect 通过同一 authority fence/CAS 决定谁先提交,并分别加入竞态回归。

[P1,阻断] required hard lease 释放失败仍会 terminal settle,且不会重试。 _settle_stop 的 typed 输入没有 lease release;File/SQLite 负例中 release authority 报错后仍得到 settled 和 lease_released=false,第二次 stop 原样返回且不再尝试释放。此时同 operation 已禁止 resume,但 Todo 最长仍会被 45 分钟 hard lease 阻塞。请把 required lease release 纳入 typed settlement,保留同一 stop_id 的可重试/可操作恢复路径,并覆盖失败后恢复。

[P1,阻断] Windows 上已启动 Host 的 stop 没有收敛或 fail-fast 路径。 host_process_drain 在没有 os.killpg 时恒为 unattributable;同机 signal 也依赖 killpg。通过真实 public stop 决策执行该平台分支后,即使 Host record 已是 finished,File/SQLite 都在重复调用中永久保持 acknowledged/host_process_drain_unproven。仓库已有 Windows Host tree-best-effort 支持,文档却没有把 stop 限定为 POSIX。请补 Windows supervisor/tree 完成事实和 CI,或在写入任何 stop/status 前明确 fail fast 并文档化;不能留下不可恢复的 open receipt。

验证方面,旧 blocker 的 8 个 File/SQLite native drain/cleanup-interruption 用例、另外 8 个 stop 用例、29 个 Python host/lane/CLI/inventory 用例、29 个 Node delegation/Host 用例和控制面 typecheck 均通过。远端 shard 1/3/4 的三项失败,我用同一命令在 immutable base 3ec049e138917a8cce4f84197ba196d26445b2b0 与本 head 都复现为相同的 generated-twin / prompt-upgrade 断言,因此是独立 merge-readiness hold,不是上述 request-changes 的依据。

语义与 CI 对齐

stopped 和 settled 是新的公共终态,不是提示性文案;它们会驱动 resume 拒绝、wait 返回、inventory/context 和后续协调决策。当前 typed decision 没有外部 effect 或 required lease 的事实,Windows unattributable 又没有恢复转移,因此实际语义仍宽于实现。最小修复后请重跑 File/SQLite effect race、lease failure/retry、Windows CLI/MCP stop、现有 native process suites 及完整 required CI。

我的整体评价

REQUEST_CHANGES。这个方向和范围有明确收益,stop sidecar、typed owner、只读 lane liveness 以及新的 Host group readback 都放在合理边界;上一轮 blocker 也确实修好了。但 whole-PR 的价值取决于 settled 能否成为后续继续工作的可信边界,目前 exact head 仍允许 late Todo/reply、不可重试的 lease failure,以及 Windows 永不收敛。长期推进和用户体验都因此 not_yet_proven。建议在现有 effect/lease/Host owner 上做有界修复,不再增加平行生命周期;修复后我会按新 exact head 重跑同一组正负路径。

English verdict: REQUEST_CHANGES — exact head 609530d2fb64e86ae71eed13ce22d4abdfd7150e. The prior native-Host early-settlement bug is fixed, but stop can still settle after Todo/reply effects commit, failed required lease release is terminal and never retried, and launched Windows Hosts have no converging or fail-fast stop path. Focused submitted suites pass; the three red repository shards reproduce unchanged on base and head and are unrelated.

@songoow

songoow commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review request — exact head 796f68867

Thanks for the native-process counterexample. It read as premature settlement rather than permanent orphan, and that is what the fix addresses.

[P1] settled could precede the owned Host and its descendants exiting — fixed

Reproduced on the reviewed head in File and SQLite: stopping a Host whose process group ignores SIGTERM returned phase=settled in ~0.26s while the Host and its same-group child were still alive. The cause was the one you named — _settle_stop treated operation-lock release and lane release as the drain, while the native bridge/Host lives in another process group with its own asynchronous cleanup lifecycle owned by host_process.ts (300ms grace, then SIGKILL).

Fix, reusing that owner rather than adding a second host-lifecycle authority:

  • The Turn names the process group its Host supervisor owns in a record beside the operation (.host.json), written before the request is sent and updated when the supervisor reports the spawned group. The supervisor also reports before the Host receives its input, so a Host never runs before it is attributable.
  • _settle_stop reads that record back — read-only, no signalling — and passes host_process to the typed decision. The supervisor stays the only owner that terminates a Host.
  • decideDelegationStop (delegation.ts:348) now requires one of not_launched | drained | draining | unattributable, and only drained / not_launched may settle. draining stays acknowledged with reason host_process_still_running; an unprovable group stays open with host_process_drain_unproven and remains readable by a later same-identity read. No fixed sleep, no widened timeout, no cross-host kill.

The other two points from the review

  • Platform-valid process check. process_gone now uses ps -o stat= -p, which is valid on Linux and macOS, and treats a zombie as exited. An unavailable or failing ps fails the test instead of reading as an exit — the old /proc read returned True on OSError, which is how a live Host could be mistaken for a departed one on macOS.
  • Assert at the instant settled returns, not after a wait. test_settled_stop_means_the_managed_host_and_its_descendants_already_exited makes both the Host and a same-group descendant ignore SIGTERM, so only the managed owner's escalation can end them, and asserts both are gone at the moment the receipt returns. The existing executing-stop test keeps its post-return assertions and now also pins stop["settled"]["host_process"] == "drained".
  • PR body corrected to match the implementation: the Turn lane is read-only liveness, and the operation lock still takes a short kernel probe after the stop is written — it is not "no lock-taking" throughout.

Verification

tests/test_local_delegation.py 32 passed on the merged head (File and SQLite authority backends, real detached worker -> CLI run-once -> native Node bridge -> host transport). Latest origin/main merged with sign-off (--no-ff, no force-push).

The 44/43 twin-budget failure you identified as pre-existing is unchanged and unrelated to this diff.

English verdict request: a stop settles only on evidence that the owning Host group actually drained, an unattributable group stays acknowledged instead of settling, and the negative case is pinned by a real-process test on both backends. Please re-review 796f68867.

@cocolord cocolord left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

这个 PR 要解决的是一个真实且高价值的问题:已经派发的 delegated member 需要一个可验证的 stop 边界,不能只杀 worker、留下 Turn、原生 Host、Todo 或 hard lease 处于含糊状态。当前 exact head 796f688674c3a404daec89cfa6d3b5801683e292 只是把最新主干合入上次已审的 head;目标收益仍然明确,但只有当 settled 能证明后续继续或重派不会与旧执行重叠时,这个收益才成立。

改动思路

实现把不可由运行记录推导的 stop intent 放进独立 stop.json,把 Host supervisor 观察到的进程归属放进 host.json;CLI/MCP 通过 Delegations.stop 发起请求,worker 在 checkpoint 或 fenced write 处 ACK,TypeScript 的 decideDelegationStop 再组合 operation lock、Turn lane 和 Host drain 事实。正常 POSIX 路径是 requested → acknowledged → settled,read/wait/resume/inventory/context 复用同一持久事实;异常路径应当在 effect、lease 或平台清理尚未闭合时保持可恢复、不可误报 terminal。

具体改动

相对 exact base fd5f31bb3ad57448c32df6c7cddde36d07446934,整份差异为 18 个文件、+1322/-53:10 个 runtime 文件增加 stop CLI/MCP、typed stopped/phase 决策、worker 信号与写 fence、Host 进程组 sidecar、lease release 读回及 inventory/context 投影;6 个测试文件覆盖 TS 转移、真实进程组、File/SQLite、CLI 和 inventory;2 个中英文参考文档说明 stop、receipt 与回滚。先前 stacked 的 lane-holder liveness 已进入 base,本 PR 当前复用它。

关键代码讲解

  • Delegations.stop 写入一次 operation-specific intent,只信号可归属的同机 worker,并把重复调用收敛到同一 receipt。
  • _settle_stop 读取 operation lock、Turn lane、Host drain 后调用 typed decision;目前 required lease release 并不是 decision input。
  • decideDelegationStop 是 requested/acknowledged/settled/unknown 的状态 owner,正确地拒绝用超时本身制造 settlement,但没有外部 effect 或 lease-success 事实。
  • host_process_drain 在 POSIX 上只读验证 bridge/process group;没有 os.killpg 时恒为 unattributable。
  • _execute 的最终 effect 段 依次完成 Todo、发布 reply、再通过 fenced _observe 写 accepted;stop 只在 effect 前做瞬时 pre-check。

当前 head 的第一父提交就是上次审过的 609530d…,第二父提交是 exact base;18 个 PR 文件里只有 delegation.ts、delegation_context.py、effect_runtime_handlers.ts 吸收了主干变化。下面三个 blocker 的 Python owner、Host drain 和测试文件均 byte-identical,因此不能把合主干视为修复。

对主干的风险

[P1,阻断] stop 与 Todo completion / result publication 仍没有共同线性化边界。 worker 在 1557 行 做一次 pre-check,之后两个外部 effect 都在 dispatch fence 外提交,最后写 execution record 时才再次看到 stop。exact-head 的 File/SQLite 反例都先持久化 stop,再放行 worker,仍观察到 todo_completed 和 reply_published,最终 receipt 却是 phase=settled,status=stopped。请让 stop 与两个 effect 通过同一 authority fence/CAS 决定先后,并分别提交 race regression。

[P1,阻断] required hard lease 释放失败仍被 terminal settle,且不会重试。 _settle_stop 没把 lease.released 交给 typed decision;两种后端注入一次 authority failure 后都返回 settled, lease_released=false,第二次 stop 原样返回且 release attempt 仍只有一次。此时 resume 已禁止,Todo 却可能被 hard lease 阻塞到 TTL。请把 required release 纳入 settlement obligation,并让同一 stop_id 在失败后可重试或进入明确可操作的恢复态。

[P1,阻断] Windows 上已启动 Host 的 stop 仍没有终结或 fail-fast 路径。 host_process_drain 在缺少 killpg 时恒返回 unattributable;File/SQLite 的公开 stop 路径即使读取到 phase=finished 的 Host record,重复调用仍永久停在 acknowledged/host_process_drain_unproven。通用 windows-powershell 绿灯没有覆盖这个新 stop contract。请提供 Windows supervisor/tree completion 事实并在 Windows CI 覆盖公共路径,或者在写 stop/status 前明确 fail fast 并文档化平台边界。

验证上,官方 exact-head Python focused suite 63 项、Node delegation/Host 29 项、control-plane typecheck、Ruff、diff hygiene、docs-governance 与 semantic-vocabulary smoke 均通过;独立的 6 个 File/SQLite 负例也稳定复现上述三个错误结果。绿色正向覆盖说明主体实现可运行,但没有覆盖 stop 在最后 checkpoint 之后与外部 effect/lease/platform清理交错的语义。

语义与 CI 对齐

stopped 与 settled 是机器消费的公共终态,会驱动 resume 拒绝、wait 返回、inventory/context 和后续重派,不是“guidance”。当前 typed contract 未纳入 Todo/reply commitment 和 required lease release,Windows 的 unattributable 又没有恢复转移,所以 public terminal 名称仍宽于实际保证。最小修复后请重跑 File/SQLite effect race、lease failure→retry、Windows CLI/MCP stop、现有 native Host suites 与完整 required CI。

我的整体评价

REQUEST_CHANGES。需求本身必要,stop sidecar、typed owner、只读 holder/Host facts、CLI/MCP 共用入口的方向也合理;当前 18 文件范围相对这个高风险能力并非单纯“代码太多”。但 exact head 没有修改上轮三个 blocker 的 owner,6 个独立负例全部复现,因此 before/after 的可观察收益仍达不到“安全停止并继续”的承诺,change proportionality 与 terminal authority 都是 not_yet_proven。建议只在现有 Todo/result/lease/Host authority 上做有界修复,不再增加平行生命周期;修复后按新 exact head 复审。

English verdict: REQUEST_CHANGES — exact head 796f688674c3a404daec89cfa6d3b5801683e292. This merge-only head leaves all three blockers unchanged and independently reproducible on File/SQLite: stop can settle after Todo/reply effects commit, failed required lease release is terminal and never retried, and launched Windows Hosts have no converging or fail-fast stop path. Focused Python/Node/static checks pass, but they do not cover these terminal-contract counterexamples.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants