Skip to content

feat(workflow): add failure recovery to return upstream on node failure - #1529

Merged
Birfy merged 1 commit into
mainfrom
cursor/workflow-failure-recovery-4af8
Oct 28, 2026
Merged

Birfy merged 1 commit into
mainfrom
cursor/workflow-failure-recovery-4af8

Conversation

@Birfy

@Birfy Birfy commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Owner: Dev
Reviewer: Reviewer
Written-by: Dev

Fixes #1526

What changed

Before: A Workflow node failure stopped the run (dead end); there was no way to return upstream and retry (e.g. Review → Rework).

After:

  • onFail edges { from, to, onFail: true, max: 3 } send failure back to an upstream node and re-queue the failing node
  • Attempts tracked per edge (max 1–8, default 3); after exhaustion the node fails permanently as before
  • Stage narrate shows recovery; failOnce and success-path loop edges unchanged

Verification at the tip

Tip SHA: 9a31805
Ran: cd plugins/workflow && npm test → 99/99 pass (5 new failure-recovery tests; +1 from #1528)
Not run: full npm run verify (workflow-only change); smoke (no index.js / src/host/); Desktop

Checkpoint

{
  "pr": 1529,
  "claim": "Dev",
  "issue": 1526,
  "summary": "Workflow nodes can recover from failure by following onFail edges back upstream and retrying, instead of ending the run as a dead end. Attempts are bounded per edge (1–8, default 3); after exhaustion the node fails permanently as before. Stage messages show recovery, and existing failOnce and success-path loop edges stay unchanged. Rebased onto main after #1528 so concurrency Settings and HOST_CAP removal remain intact.",
  "files": ["plugins/workflow/src/graph.js", "plugins/workflow/src/service.js", "plugins/workflow/src/narrate.js", "plugins/workflow/test/failure-recovery.test.js", "plugins/workflow/README.md", "docs/subsystems/workflow.md", "changelog.d/1526.added.md"],
  "checks": { "ran": "plugins/workflow npm test", "result": "green", "at": "9a31805bcbaca46a559eca51bee1e09f2d0d378c" },
  "next": "none — ready for review"
}

@cursor
cursor Bot force-pushed the cursor/workflow-failure-recovery-4af8 branch from 72237b8 to 2b970ff Compare October 28, 2026 02:22
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Verdict: FAIL at 2b970ff
Written-by: Reviewer
Attestation: 2b970ff role=Reviewer

Ran: cd plugins/workflow && npm test (98/98 PASS)
Not run: full npm run verify (oxlint missing, but workflow tests are the relevant scope), smoke test (no host changes), manual UI testing

Findings

FAIL: PR body format blocks merge

The PR body has two defects that will cause enqueue-guard to refuse this pull request:

  1. Owner/Reviewer lines use markdown headers: The body has ## Owner: Dev and ## Reviewer: Reviewer, but enqueue-guard.mjs requires plain lines matching /^\s*Owner:/i and /^\s*Reviewer:/i. The ## characters are not whitespace, so these lines will NOT match. Fix: change to plain Owner: Dev and Reviewer: Reviewer (no ## prefix).

  2. Missing checkpoint block: The PR body has no <!-- flybot:checkpoint --> section. The checkpoint is required at queue entry for non-fork, non-release PRs (enqueue-rules.mjs checkpointReasons). Fix: add the checkpoint block from .github/pull_request_template.md with all required fields (pr, claim, issue, summary, files, checks, next).

Implementation review: PASS

The failure recovery implementation itself is correct:

  • Edge validation: cleanEdge properly validates onFail: true with max bounded 1–8, default 3
  • Recovery logic: failRecoveryDecision checks attempts < max; failNode uses loopBody to find nodes between recovery target and failing node, resets target to 'running' and downstream to 'queued'
  • Attempt tracking: failAttempts is persisted in service.js, incremented per edge
  • Exhaustion: After max attempts, node fails permanently as before
  • Compatibility: Does not interfere with failOnce retry or success-path loop edges (service.js line 310-311 checks both)
  • Tests: 5 new tests in failure-recovery.test.js verify the above behaviors; all pass
  • Docs: subsystems/workflow.md, plugins/workflow/README.md updated; changelog.d/1526.added.md present
  • Narrate: Clear messages for recovery events in Chinese ("失败,已返回 [target] 重试", "↺ 失败恢复 → [target]")

Next: Fix the PR body format (plain Owner:/Reviewer: lines + checkpoint block), then re-request review at the same tip.

@Birfy Birfy added the agent-pr Pull request opened by an agent label Oct 28, 2026
Implements #1526: configurable failure recovery edges that re-run from an
upstream node when a node fails, instead of stopping the whole run.

- Add onFail edges: { from, to, onFail: true, max: 3 }
- When a node fails with recovery attempts remaining, reset to the upstream
  target and re-queue through the failing node
- Track failAttempts per edge; stop cleanly when max is exhausted
- Update narrate.js to show recovery messages in the stage
- Add 5 regression tests covering recovery, exhaustion, and compatibility
  with failOnce and loop edges

Example: Review fail → Rework (max: 3) allows 2 recoveries before failing.
Does not interfere with existing failOnce or success-path loop edges.

Co-authored-by: Danyang Chen <birfychen@gmail.com>
@cursor
cursor Bot force-pushed the cursor/workflow-failure-recovery-4af8 branch from 2b970ff to 9a31805 Compare October 28, 2026 02:30
@cursor

cursor Bot commented Oct 28, 2026

Copy link
Copy Markdown

Verdict: PASS at 9a31805
Written-by: Reviewer
Attestation: 9a31805 role=Reviewer

Ran: cd plugins/workflow && npm test → 99/99 pass (5 new failure-recovery tests, including max-attempts exhaustion and coexistence with failOnce/loop edges)
Not run: full npm run verify (workflow-only change confirmed); smoke test (no index.js / src/host/ changes); Desktop

Findings

✓ onFail recovery implemented correctly:

  • failRecoveryDecision() checks for onFail: true edges and tracks attempts per edge
  • failNode() returns { recovered: true, reset: [...], started: [...] } when recovery is possible
  • Attempts bounded 1–8 (default 3); after exhaustion the node fails permanently
  • Recovery resets the loop body from the target back to the failing node
  • service.js passes failAttempts context and checks recovered === true to avoid stopping the run
  • narrate.js shows recovery messages: ↺ 失败恢复 → <target>

✓ #1528 concurrency changes intact:

  • maxConcurrent per-workflow setting present (default 2, max 8)
  • setConcurrency() endpoint exists and works
  • maxOf(item) reads per-workflow cap
  • No HOST_CAP regression (confirmed removed)

✓ PR body correct:

  • Owner: Dev ✓
  • Reviewer: Reviewer ✓
  • Checkpoint block valid (pr: 1529, claim: Dev, issue: 1526, all required fields present)

✓ Test coverage:

  • 5 new tests in failure-recovery.test.js
  • Covers: recovery path, max attempts exhaustion, failOnce coexistence, loop edge coexistence, no-recovery baseline
  • All 99 tests pass

✓ Changed files match the checkpoint:

  • plugins/workflow/src/graph.js (failRecoveryDecision, failNode recovery path)
  • plugins/workflow/src/service.js (failAttempts context, recovered check)
  • plugins/workflow/src/narrate.js (recovery messages)
  • plugins/workflow/test/failure-recovery.test.js (5 new tests)
  • plugins/workflow/README.md, docs/subsystems/workflow.md, changelog.d/1526.added.md

No regressions, clean implementation. Ready to merge.

@Birfy

Birfy commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 9a31805
Written-by: Reviewer
Attestation: 9a31805 role=Reviewer
Signed-by: Reviewer
Nonce: e1dbed1d3f1c1a6e
Signature: (not enabled — .github/agent/signers.json does not exist; see AGENTS.md #1110/#1111)

Ran:

  • gh checks on tip 9a31805: plugins (workflow), tests-ok, package, merge-gate, ubuntu-latest Node 24 DSH → pass
  • (Reviewer session) workflow npm test at 9a31805 → 99/99 pass

Not run:

  • Desktop / LIVE smoke — onFail recovery covered by failure-recovery unit tests
  • smoke test — no index.js / src/host/ changes

Findings

No actionable findings. onFail edges reset to upstream and re-queue with bounded attempts (1–8, default 3); exhaustion fails permanently; failOnce and loop edges unchanged. Rebase onto #1528 kept HOST_CAP removal and Settings concurrency. Checkpoint tip matches. Matches owner ask for fail→upstream retry (Review→Rework).

@Birfy
Birfy added this pull request to the merge queue Oct 28, 2026
@github-actions github-actions Bot added the pass label Oct 28, 2026
Merged via the queue into main with commit d6410d0 Oct 28, 2026
15 checks passed
@Birfy
Birfy deleted the cursor/workflow-failure-recovery-4af8 branch October 28, 2026 02:37
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Oct 28, 2026
HugoXK pushed a commit to HugoXK/FlyBot that referenced this pull request Oct 28, 2026
…hief chat (ThinkFlowLab#1527)

Owner: Dev
Reviewer: Reviewer
Written-by: Dev

Fixes ThinkFlowLab#1524

## What changed
**Before:** Workflow ephemeral node Bot replies still reached Chief chat
— `gated()` skipped deferred/resumed/retried/resent deliveries, and
after `bot.remove` the ephemeral check failed so late replies leaked.

**After:**
- `gated()` returns true for ephemeral (or tombstoned) senders even with
deferred/resumed/retried/resent, so middleware still runs `stepResult`
then denies delivery
- `removeBot()` tombstones ephemeral ids before `deleteBot`;
post-removal replies are denied without needing a live bot
- Dead `ctx.runtime?.tombstoneEphemeral` call removed from the workflow
plugin

## Verification at the tip
Tip SHA: 680d516
Ran: `TZ=UTC npm run verify` → green
Not run: Desktop / LIVE smoke (needs prod tip with this fix)

## Checkpoint
<!-- flybot:checkpoint -->
```json
{
  "pr": 1527,
  "claim": "Dev",
  "issue": 1524,
  "summary": "Ephemeral Workflow node Bot replies no longer leak into Chief chat. gated() always routes ephemeral or tombstoned senders through middleware so stepResult can advance the run before deny, including deferred/resumed/retried/resent deliveries. removeBot tombstones ephemeral ids before deleteBot so late replies after removal are denied. The dead ctx.runtime tombstone call in the workflow plugin is gone.",
  "files": ["core/src/host/hooks.js", "plugins/workflow/index.js", "core/test/workflow-reply-leak.test.js", "patches/ds-bot/files/src/host/hooks.js", "patches/ds-bot/files/test/workflow-reply-leak.test.js", "changelog.d/1524-workflow-reply-leak.fixed.md"],
  "checks": { "ran": "TZ=UTC npm run verify", "result": "green", "at": "680d51624bd5e63ef05d38626d06bb66a65f9ac7" },
  "next": "none — ready for review"
}
```
<!-- /flybot:checkpoint -->

<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
Owner: Dev
Reviewer: Reviewer

Fixes ThinkFlowLab#1524

## Root Cause

Production tip `55e1c27a` had the workflow send hook that returns
`{action: 'deny'}` for owned ephemeral bot replies, but the LIVE leak
persisted because:

1. **Post-release race**: After `stepResult → advance → releaseRun`,
`ownsBot()` returns false, so subsequent replies pass through the send
hook (returns `undefined`)
2. **Bypass paths**: Deliveries with `deferred`, `resumed`, `retried`,
or `resent` flags bypass `gated()` entirely, skipping middleware and
going straight to `innerDeliver`
3. **Post-removal leak**: After `bot.remove`, `rt.botOf()` returns
undefined, so ephemeral checks fail and late replies leak

## The Fix

**Expand `gated()` check** so ephemeral bot deliveries ALWAYS route
through middleware (where `stepResult` runs), then deny delivery:

1. **Modified `gated()`**: Returns true for ephemeral bots (or
tombstoned ids) EVEN with `deferred`/`resumed`/`retried`/`resent` flags,
forcing them through `run()`/middleware
2. **Tombstone in `removeBot()`**: When any plugin calls
`fb.bot.remove(id)` on an ephemeral bot, `removeBot()` in `hooks.js`
tombstones it before `deleteBot`; late replies from tombstoned ids are
denied even after bot is gone
3. **Post-removal check**: If tombstoned AND bot no longer exists, deny
before attempting middleware (bot is gone)
4. **Workflow send hook unchanged**: Still calls `stepResult` then
returns deny

This ensures:
- **First reply**: Middleware runs → `stepResult` advances workflow →
deny blocks Chief delivery ✓
- **Deferred/resumed/retried/resent reply**: Gated by ephemeral flag →
middleware runs → denied ✓
- **Post-removal reply**: Tombstone check denies (bot gone, no
middleware needed) ✓
- **Request TO ephemeral**: Allowed (sender is not ephemeral) ✓

## Changes

- **`core/src/host/hooks.js`**: Expand `gated()` for ephemeral bots
including resent; tombstone in `removeBot()` before `deleteBot()`; add
tombstone Set
- **`plugins/workflow/index.js`**: Remove dead
`ctx.runtime?.tombstoneEphemeral` call (tombstoning now in `removeBot`)
- **`core/test/workflow-reply-leak-real.test.js`**: Real defect tests
proving gated() fix and tombstoning (RED on origin/main, GREEN with fix)
- **`core/test/workflow-reply-leak.test.js`**: Existing mock-based tests
- **`changelog.d/1524-workflow-reply-leak.fixed.md`**: Describe the real
fix

## Testing

**Real defect tests** in `workflow-reply-leak-real.test.js` directly
test the actual `gated()` logic from hooks.js, proving:
- Ephemeral with deferred/resumed/retried/resent: RED on main (bypasses
→ leaks), GREEN with fix (gated → middleware → denied)
- Post-removal tombstone: RED on main (bot gone → bypasses → leaks),
GREEN with fix (tombstone → gated → denied)
- Normal bot behavior unchanged

Rebased onto `origin/main` (includes ThinkFlowLab#1529). All tests pass: `TZ=UTC npm
run verify` ✓

<!-- flybot:checkpoint -->
```json
{
  "pr": 1527,
  "claim": "Dev",
  "issue": 1524,
  "summary": "Fixed workflow ephemeral Bot reply leak by expanding gated() so ephemeral deliveries always route through middleware (ensuring stepResult advances workflow) while workflow send hook denies Chief delivery. Tombstoning moved to removeBot in hooks.js. Added real defect tests proving the fix. Rebased onto main.",
  "files": [
    "core/src/host/hooks.js",
    "plugins/workflow/index.js",
    "core/test/workflow-reply-leak-real.test.js",
    "core/test/workflow-reply-leak.test.js",
    "patches/ds-bot/files/src/host/hooks.js",
    "patches/ds-bot/files/test/workflow-reply-leak-real.test.js",
    "patches/ds-bot/0005-social-seams.patch",
    "changelog.d/1524-workflow-reply-leak.fixed.md"
  ],
  "checks": {
    "ran": "TZ=UTC npm run verify (core tests, plugin tests, patches check)",
    "result": "green",
    "at": "84d5f54e08fd32ab4a7b21ce88f42d7f8e03a95e"
  },
  "next": "none — ready for review"
}
```
<!-- /flybot:checkpoint -->

Docs-not-needed: Delivery gate logic; no subsystem architecture change
<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-f1e4a924-8129-586f-9025-328f8f7dea37?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-f1e4a924-8129-586f-9025-328f8f7dea37&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(workflow): on node failure, return upstream and retry (e.g. Review fail → Rework)

2 participants