fix(workflow): dispose() releases only the last run's Bots when one lane has two concurrent runs #1472

Written-by: FlyBot Product

What happens

dispose() ends every live run in a lane as failed but releases the run Bots of only the last one. With maxConcurrent = 2 (the default) and two live runs in the same workflow, the first run's two ephemeral Bot ids never reach releaseAll and are then written out of the store: the run itself is gone (active: 0, lane.last is the second run) and nothing names those Bots any more. They stay alive, hidden, and unreachable. This is the exact shape #1387 was merged to prevent: "A record that drops an id before its Bot is gone leaves a hidden Bot that no state names, so nothing can ever find it again."

Introduced by #1465 ("End workflow zombie runs on dispose and reap", merged as 8f1ac121, 2026-10-27T15:13:53Z). The same probe releases all four ids on 53d2c025, the commit before it.

Reachability

This is not a theoretical risk: #1465 (8f1ac121, 2026-10-27T15:13:53Z) is the commit that introduced it, it is current main, and the running instance was lifted from e37f1f8 to 8f1ac12 at about 16:01Z the same day — the leaking code is what users run now.

Trigger surface: the plugin is unloaded or reloaded inside a live host (plugin toggle / config change, dev hot reload) — then the leaked Bots stay alive, hidden and unreachable. On a host exit the Bots die with the process, but their ids were already erased by write(), so nothing can ever name them again either.

Not this path: delete() (:660-686) collects Object.values(run.ephemeral) for every run in lane.active and releaseAlls them before it removes the lane, so deleting a workflow does not leak. The gap is dispose() only.

Root cause (as of 8f1ac121 — locate by symbol, not by number)

• dispose() in plugins/workflow/src/service.js (:1034): for each live running run in lane.active it calls failCutoffRun(...) and then archiveCutoffRun(workflowId, run).
• archiveCutoffRun() (:417-422): removes the run from lane.active, sets lane.last = run, appends one history row. Each archived run overwrites lane.last.
• The collection point (:1046) runs after that whole loop: runs.push(...[...(lane.active ?? []), lane.last].filter(Boolean)). By then the earlier archived runs are in neither lane.active nor lane.last.
• The ids list (:1048) is built from runs only, so just the last archived run's ephemeral values are collected.
• releaseAll(ids, …) releases those, every run in runs loses ephemeral/handed, and the write persists active: 0, lane.last = the last run. The dropped run's ids are in neither the release list nor the store.

Measured

Temporary probe plugins/workflow/test/probe-dispose-multi.test.js (imports ../src/service.js), one two-node workflow, two assign calls: both runs are running because maxConcurrent defaults to 2, then dispose().

const svc = createWorkflowService({
  dir, persist: true, now: () => CLOCK, bots: () => roster,
  spawn: async (spec) => { seq += 1; return `eph-${spec.nodeId}-${seq}` },
  release: async (id) => { released.push(id) },
  dispatch: async () => ({ ok: true }),
})
const made = svc.define({ name: 'Lane', nodes: [
  { id: 'plan', name: 'Planner', role: 'plan', prompt: 'Write the plan.' },
  { id: 'code', name: 'Coder', role: 'code', prompt: 'Implement it.' },
], edges: [{ from: 'plan', to: 'code' }] }).workflows.find(item => item.name === 'Lane')
await svc.assign({ workflowId: made.id, taskId: '#a', title: 'First task' })
await svc.assign({ workflowId: made.id, taskId: '#b', title: 'Second task' })
await svc.dispose()

8f1ac121:

• ephemeral ids: eph-plan-1,eph-code-2,eph-plan-3,eph-code-4 (4)
• released: eph-code-4,eph-plan-3 (2); missing: eph-plan-1,eph-code-2
• store after dispose(): active: 0, lane.last = #b failed, history [["#a","failed"],["#b","failed"]]
• the missing ids no longer appear anywhere in workflows.json

So one dispose() leaks 2 Bots — the two node Bots of the run that was archived first — and write() leaves no trace of them (leftover ids still in file: none), which is why reap() cannot recover them either.

53d2c025 (before #1465), same probe: released all 4, missing none.

Expected

• dispose() ends every live running run in the lane (in-flight nodes failed, run failed, one history row each) — that half is already right.
• It releases all of those runs' Bots, and only then drops ephemeral/handed and writes, keeping #1387's invariant: "A record that drops an id before its Bot is gone leaves a hidden Bot that no state names, so nothing can ever find it again."
• Minimal fix, either shape: take the [...lane.active, lane.last] snapshot before the archive loop, or merge the removed runs back into runs by run id.

Test to add

A runnable regression test in plugins/workflow/test/dispose.test.js: two nodes, two tasks, maxConcurrent = 2, both runs running at dispose() → assert every ephemeral id was released (4), lane.active is empty, lane.history holds the two failed rows. Today node --test $(ls plugins/workflow/test/*.test.js) is 74 pass / 0 fail, and the PASS on #1465 (d37f6a4a, node --test plugins/workflow/test/dispose.test.js plugins/workflow/test/ephemeral.test.js = 14/14) covers one run per lane only, so it missed this.

Scope

plugins/workflow/** only: the snapshot order in dispose() plus the test. The workflow plugin is not our module here — reported, not implemented by this report.

Duplicate check

• workflow dispose releases only one run concurrent maxConcurrent Bots → 1 result: closed #1414, "a run whose ephemeral map is gone leaves its hidden run Bots unreapable". Same symptom family, different root cause: there the map was already gone before reap() could read it; here dispose() itself never releases that run. Not a duplicate.
• workflow unload leaves hidden run Bots concurrent → 1 result: the same #1414.
• 1465 dispose zombie concurrent → 0 results.
• Closed #1387 covers the release-before-forget order for one run; it does not cover the multi-run collection point.