feat(workflow): add failure recovery to return upstream on node failure

changelog.d/1526.added.md

@@ -0,0 +1 @@
+**Workflow failure recovery:** A node can now recover from failure by returning to an upstream node and retrying, instead of stopping the run. Define a failure recovery edge with `onFail: true` pointing from the failing node back to the recovery target, with a configurable `max` attempts bound (1–8, default 3). Example: `{ from: 'review', to: 'draft', onFail: true, max: 3 }` — when Review fails, the workflow resets to Draft and runs through Review again, up to 3 times total. After attempts are exhausted, the run fails cleanly. Does not interfere with existing `failOnce` retry or success-path `loop` edges.

docs/subsystems/workflow.md

@@ -27,7 +27,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher,
 | File | Role |
 | --- | --- |
 | `plugins/workflow/index.js` | Plugin entry (`flybot-workflow`). Endpoints, `run_workflow`, `define_workflow`, the board listener, and the reply middleware that ingests a step result and denies delivery to persistent Bot chats. |
-| `plugins/workflow/src/graph.js` | Nodes, edges, positions, and named inputs and outputs. A node starts when its forward predecessors are settled and a taken edge reaches it. Same-rank nodes run together. A forward cycle is refused. A loop edge is the only cycle. A `when` edge chooses a branch; the other is skipped. A required input with no upstream provider is refused before a run starts. |
+| `plugins/workflow/src/graph.js` | Nodes, edges, positions, and named inputs and outputs. A node starts when its forward predecessors are settled and a taken edge reaches it. Same-rank nodes run together. A forward cycle is refused. A loop edge is the only cycle for success paths. A `when` edge chooses a branch; the other is skipped. A required input with no upstream provider is refused before a run starts. An `onFail` edge provides failure recovery: when a node fails with recovery attempts remaining, the workflow resets to the upstream target and re-runs through the failing node (tracked by `failAttempts` per edge, with a configurable `max` bound, 1–8, default 3). After attempts are exhausted, the node fails permanently. |
 | `plugins/workflow/src/service.js` | The store, lanes (active runs, FIFO queue, terminal history), `maxConcurrent`, `rename`, `delete`, `saveGraph`, `define`, `runNamed`, `ingest`, `stepResult`. `rename` updates a workflow's name and optional description; `delete` removes the workflow, releases ephemeral Bots, and cleans logs/runs/routes. Factory template ids (e.g. `delivery`) are normal saved rows and may be deleted. A spawn hook makes one run Bot per node. Disposing the plugin ends every live run still marked `running` (nodes in flight are failed, the run is archived) and then releases every run Bot and drops those ids from the saved record only once the release has finished, handing the release back as a promise the host awaits, so a shutdown cannot persist a run as `running` with no ephemeral map and cannot leave hidden Bots that no state names (#1387, #1414). A released run Bot also gives its runner name back to the allocator instead of keeping it for the lifetime of the process. A dispatch that does not land is not marked handed. |
 | `plugins/workflow/src/ephemeral.js` | Builds each node's run Bot from its prompt, role, model and tools, and releases the ids the host returns. The `tools` field is **always omitted** from `fb.bot.create` so every node Bot inherits all registered tools (GitHub, Composio, file/shell tools, and team tools limited by `EPHEMERAL_DENIED_TOOLS` at the guard layer). `node.tools` is stored as documentation only in `fields.tools` but does not restrict the grant. All node Bots get `danger-full-access` permission. |
 | `plugins/workflow/src/match.js` | Which open card matches a Workflow or a local Bot. |
@@ -50,7 +50,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher,
 4. Each workflow keeps a FIFO queue and up to `maxConcurrent` live runs (default 2, an integer from 1 to 8 on the record, via `workflow/concurrency`, or in Settings → Workflow graphs). A task that does not fit is enqueued. When a run finishes, every workflow with a free slot is drained, oldest enqueue first. A new assign on a workflow that already has a queue waits behind it. A task does not replace a running run or release that run's Bots. Migrating an old store keeps only runs that were still `running` with ephemeral Bot ids; other legacy scripted rows are dropped from active state (finished tasks may remain in lane history). History records a finished or failed live run. Lowering the cap does not stop a run that has already started. Assigning `{ botId }` claims the card for that local Bot and does not enqueue. A remote Bot is refused. `activeCount` is `runningOf(lane)` — live runs with `live === true` and `status === 'running'`. The stage aggregate chip and the Runs tab (运行记录) must agree: every such run appears as its own running row and progress card, and the chip count does not exceed that list.
 5. A node declares up to eight inputs and eight outputs. A required input is an output of the same name on an ancestor, or `task`, `title`, or `brief` on a root. A start with no provider is refused, and Bots spawned for that attempt are released. On a Bot-backed run a successor stays queued until its required inputs are present. If every forward predecessor is done, failed or skipped and an input is still missing, that node fails and the run stops. The dispatch text carries `Input <name>: <value>`. The Bot reports `OUTPUT <name>: <value>`; a single declared output may be the reply body. Nodes start in edge order.
 6. Each node is a real ephemeral Bot for that run, then it is removed. A permanent sidebar Bot (`botId`) is optional and is not required for the step to run. **Every node Bot gets the same full permissions a normal sidebar Bot gets:** the `tools` field is always omitted from `fb.bot.create`, letting the Bot inherit all registered tools — file/shell tools, GitHub (`mcp__github__*`: pull requests, reviews, comments), Composio, and team tools — limited only by `EPHEMERAL_DENIED_TOOLS` at the guard/approval layer (#1466, 29 team-admin tools). `node.tools` is stored as documentation in `fields.tools` but does not restrict the grant. All node Bots get `danger-full-access` permission. A hidden run Bot cannot show an approval card, so a granted tool is allowed and a connector it did not name is refused. The host stores this on the run Bot as `stepTools` and `permission`.
-7. A loop edge (`loop: true`, `max` from 1 to 8, default 3, optional `until: { output, eq }`) is the only cycle. The body runs again until `until` matches or `max` runs have finished, and the forward edges wait until then. Restarting the body drops the branch choice made inside it and queues a sibling that the previous pass skipped, so the next pass can take the other side. A `when: { output, eq }` edge is taken only when that output equals `eq`. `when: { else: true }` is taken only when no other `when` matched. A node whose forward edges are all closed is `skipped` and does not block a join. A run cannot spin: every loop has a max. Settings → Workflow graphs can add a link, a condition or a loop.
+7. A loop edge (`loop: true`, `max` from 1 to 8, default 3, optional `until: { output, eq }`) is the only cycle for success paths. The body runs again until `until` matches or `max` runs have finished, and the forward edges wait until then. Restarting the body drops the branch choice made inside it and queues a sibling that the previous pass skipped, so the next pass can take the other side. A `when: { output, eq }` edge is taken only when that output equals `eq`. `when: { else: true }` is taken only when no other `when` matched. A node whose forward edges are all closed is `skipped` and does not block a join. A run cannot spin: every loop has a max. An `onFail` edge (`onFail: true`, `max` from 1 to 8, default 3) provides failure recovery: when a node fails, if an `onFail` edge points from that node back to an upstream target and recovery attempts remain (tracked in `failAttempts` per edge), the workflow resets both nodes to their pre-failure states (the target to `running`, the failing node to `queued`) and re-runs from the target through the failing node. After `max` attempts are exhausted, the node fails permanently and the run stops. An `onFail` edge does not interfere with `failOnce` retry (which restarts the same node once) or success-path `loop` edges. Settings → Workflow graphs can add a link, a condition, a loop, or a failure recovery edge.
 8. Auto-assign considers an `open` card only. A saved Bot claim beats a workflow trigger. Otherwise first hit wins: a `workflow:` token, then `workflowId` / `workflow`, then an assignee that names a workflow, then a `bot:` token, then an assignee that names a local Bot, then the longest trigger contained in the title, note or criteria. A miss returns no route. A second ingest of the same task and target is a no-op, and a live run for that task id is not restarted.
 9. The browser draws text only. Names and messages are React text children. The unit renderer asserts a hostile name is not markup.
 10. `enabled: false` registers no endpoint, tool or listener and writes nothing.
@@ -59,4 +59,4 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher,
 
 ## Tests
 
-`plugins/workflow/test/`: graph advance and join, graph edit, assign and persistence, the FIFO queue and concurrency cap, Bot-reply advance, a dispatch that fails without being marked handed, disposing releases the Bots before the record drops their ids and a released runner name is reusable again, assign-to-bot versus assign-to-workflow, inputs and outputs along edges, a loop that breaks and an if/else branch, templates, `define` / `runNamed`, the matcher, ephemeral spawn and cleanup, a stopped run with its settled node map (#1424), the reply middleware, the client canvas, the settings page (including a condition and a loop), rename and delete (including factory-id rows such as `delivery`), and a sidebar click that leaves the Workflows section in place. Run `npm test` in `plugins/workflow`. The host seam that applies `ownerBotId` is `core/test/task-route.test.js`. The host omission of run Bots, and the grant of `stepTools` plus the hidden-approval answer, is `core/test/flybot.test.js`. Which connectors a run Bot may call is `core/test/connector-access.test.js`.
+`plugins/workflow/test/`: graph advance and join, graph edit, assign and persistence, the FIFO queue and concurrency cap, Bot-reply advance, a dispatch that fails without being marked handed, disposing releases the Bots before the record drops their ids and a released runner name is reusable again, assign-to-bot versus assign-to-workflow, inputs and outputs along edges, a loop that breaks and an if/else branch, failure recovery (onFail edges with max attempts, exhaustion behavior, and compatibility with failOnce and loop edges), templates, `define` / `runNamed`, the matcher, ephemeral spawn and cleanup, a stopped run with its settled node map (#1424), the reply middleware, the client canvas, the settings page (including a condition and a loop), rename and delete (including factory-id rows such as `delivery`), and a sidebar click that leaves the Workflows section in place. Run `npm test` in `plugins/workflow`. The host seam that applies `ownerBotId` is `core/test/task-route.test.js`. The host omission of run Bots, and the grant of `stepTools` plus the hidden-approval answer, is `core/test/flybot.test.js`. Which connectors a run Bot may call is `core/test/connector-access.test.js`.

plugins/workflow/README.md

@@ -38,9 +38,13 @@ The delivery template passes `task` → `plan` → `copy` → `patch`, then `rev
 
 ## Branches and loops
 
-An edge may be a condition or a loop. A condition is `when: { output, eq }` on the edge out of the node that produced `output`. The edge is taken only when that output equals `eq`. `when: { else: true }` is the other branch, taken when no `eq` edge matched. A node that no taken edge reaches is skipped. A later join starts once every incoming edge is either taken and done, or skipped.
+An edge may be a condition, a loop, or a failure recovery. A condition is `when: { output, eq }` on the edge out of the node that produced `output`. The edge is taken only when that output equals `eq`. `when: { else: true }` is the other branch, taken when no `eq` edge matched. A node that no taken edge reaches is skipped. A later join starts once every incoming edge is either taken and done, or skipped.
 
-A loop is `loop: true` on the edge that returns to the body, with `max` from 1 to 8 (default 3) and an optional `until: { output, eq }`. The body runs again while `until` does not match and the run count is still under `max`. A branch the previous pass skipped can be chosen the next time. The forward edges wait until the loop stops. A loop is the only cycle the graph allows. Settings → Workflow graphs can add a link, a condition, or a loop. After a live run finishes, the stage keeps showing that run until a new one starts.
+A loop is `loop: true` on the edge that returns to the body, with `max` from 1 to 8 (default 3) and an optional `until: { output, eq }`. The body runs again while `until` does not match and the run count is still under `max`. A branch the previous pass skipped can be chosen the next time. The forward edges wait until the loop stops. A loop is the only cycle the graph allows for success paths.
+
+A failure recovery is `onFail: true` on an edge pointing from the failing node back to an upstream node, with `max` from 1 to 8 (default 3). When the node fails and recovery attempts remain, the workflow resets to the recovery target and runs through the failing node again. After `max` recovery attempts, the node fails permanently and the run stops. Failure recovery edges do not interfere with success-path loops or `failOnce` retry behavior.
+
+Settings → Workflow graphs can add a link, a condition, a loop, or a failure recovery. After a live run finishes, the stage keeps showing that run until a new one starts.
 
 ## When a step does not land
 

plugins/workflow/src/graph.js

@@ -110,6 +110,7 @@
 
 // A loop is the only edge that may cycle. max is how many times the body runs
 // (1–8, default 3). until stops early when that output equals eq.
+// An onFail edge recovers from failure by returning to the from node.
 function cleanEdge(edge) {
   const from = String(edge?.from ?? '')
   const to = String(edge?.to ?? '')
@@ -124,6 +125,15 @@
       ...(until ? { until } : {}),
     }
   }
+  if (edge?.onFail === true) {
+    const max = Number(edge.max)
+    return {
+      from,
+      to,
+      onFail: true,
+      max: Number.isInteger(max) && max >= 1 && max <= 8 ? max : 3,
+    }
+  }
   const when = cleanWhen(edge.when, true)
   return { from, to, ...(when ? { when } : {}) }
 }
@@ -132,7 +142,7 @@
   return `${edge.from}>${edge.to}`
 }
 
-// Longest-path ranks. A forward cycle is refused. A loop edge is not ranked.
+// Longest-path ranks. A forward cycle is refused. Loop and onFail edges are not ranked.
 // Nodes keep their list order inside a rank.
 export function ranksOf(nodes, edges) {
   const ids = nodes.map(node => node.id)
@@ -141,7 +151,7 @@
   const succs = new Map(ids.map(id => [id, []]))
   for (const edge of edges) {
     if (!known.has(edge.from) || !known.has(edge.to)) throw new Error(`Link names an unknown Bot ${edge.from} → ${edge.to}`)
-    if (edge.loop === true) continue
+    if (edge.loop === true || edge.onFail === true) continue
     if (edge.from === edge.to) throw new Error('A Bot cannot connect to itself')
     preds.get(edge.to).push(edge.from)
     succs.get(edge.from).push(edge.to)
@@ -369,7 +379,7 @@
 }
 
 function forwardEdges(graph) {
-  return graph.edges.filter(edge => edge.loop !== true)
+  return graph.edges.filter(edge => edge.loop !== true && edge.onFail !== true)
 }
 
 function predecessors(graph, nodeId) {
@@ -512,6 +522,18 @@
   return null
 }
 
+// Check if a failed node has an onFail edge and attempts remaining.
+function failRecoveryDecision(graph, nodeId, failAttempts) {
+  const edges = graph.edges.filter(edge => edge.onFail === true && edge.from === nodeId)
+  for (const edge of edges) {
+    const ran = (Number(failAttempts?.[edgeKey(edge)]) || 0) + 1
+    const max = Number.isInteger(edge.max) ? edge.max : 3
+    if (ran >= max) continue
+    return { edge, ran }
+  }
+  return null
+}
+
 // Nodes the loop runs again: forward-reachable from the target that can still
 // reach the source. A self-loop is that one node. The exit stays queued.
 function loopBody(graph, source, target) {
@@ -580,8 +602,8 @@
   const index = layerIndex(graph, nodeId)
   const layer = graph.layers[index]
   const outputs = ctx.outputs && typeof ctx.outputs === 'object' ? ctx.outputs : {}
   const loops = { ...(ctx.loops ?? {}) }
   const fired = { ...(ctx.fired ?? {}) }
   const looping = loopDecision(graph, nodeId, outputs, ctx.loops)
   if (looping) {
     loops[edgeKey(looping.edge)] = looping.ran
@@ -626,12 +648,40 @@
   }
 }
 
-export function failNode(graph, status, nodeId) {
+export function failNode(graph, status, nodeId, ctx = {}) {
   if (!nodeById(graph, nodeId)) throw new Error(`Unknown Bot ${nodeId}`)
   if (status[nodeId] !== 'running') throw new Error(`${nodeId} is not running`)
+
+  const failAttempts = { ...(ctx.failAttempts ?? {}) }
+  const recovery = failRecoveryDecision(graph, nodeId, failAttempts)
+
+  if (recovery) {
+    // Failure recovery: reset to the upstream target node
+    failAttempts[edgeKey(recovery.edge)] = recovery.ran
+    const body = loopBody(graph, recovery.edge.to, nodeId)
+    const next = copyStatus(status)
+    const reset = []
+
+    for (const id of body) {
+      next[id] = id === recovery.edge.to ? 'running' : 'queued'
+      reset.push(id)
+    }
+
+    return {
+      status: next,
+      started: [recovery.edge.to],
+      joined: false,
+      finished: false,
+      failAttempts,
+      recovered: true,
+      reset,
+    }
+  }
+
+  // No recovery or attempts exhausted: fail normally
   const next = copyStatus(status)
   next[nodeId] = 'failed'
-  return { status: next, started: [], joined: false, finished: false }
+  return { status: next, started: [], joined: false, finished: false, failAttempts }
 }
 
 export function retryNode(graph, status, nodeId) {

plugins/workflow/src/narrate.js

@@ -42,6 +42,14 @@ function liveLines(workflow, task, event) {
   }
   if (event.type === 'fail') {
     const node = nodeById(workflow, event.nodeId)
+    if (event.recovered === true && event.started.length > 0) {
+      const target = names(workflow, event.started)
+      const text = liveText(event.text, `${node?.name ?? event.nodeId} 失败，已返回 ${target} 重试`)
+      return [
+        { kind: 'bot', botId: event.nodeId, step: stepLabel(workflow, event.nodeId), text },
+        { kind: 'system', tone: 'bad', text: `↺ 失败恢复 → ${target}` },
+      ]
+    }
     const text = liveText(event.text, `${node?.name ?? event.nodeId} 这一步失败了，停在本节点。`)
     return [{ kind: 'bot', botId: event.nodeId, step: stepLabel(workflow, event.nodeId), text }]
   }
@@ -74,6 +82,14 @@ export function linesFor(workflow, task, event) {
   }
   if (event.type === 'fail') {
     const node = nodeById(workflow, event.nodeId)
+    if (event.recovered === true && event.started.length > 0) {
+      const target = names(workflow, event.started)
+      return [
+        { kind: 'bot', botId: event.nodeId, step: stepLabel(workflow, event.nodeId),
+          text: `${node?.name ?? event.nodeId} 失败，已返回 ${target} 重试` },
+        { kind: 'system', tone: 'bad', text: `↺ 失败恢复 → ${target}` },
+      ]
+    }
     const text = workflow.id === DELIVERY_ID && event.nodeId === 'tester'
       ? '快照测试 notice.snap 不一致（旧文案快照未更新）。自动重试 1/2，并已通知 Coder 更新快照。'
       : `${node?.name ?? event.nodeId} 这一步失败了，停在本节点。`