Repository navigation
feat(workflow): add failure recovery to return upstream on node failure - #1529
Conversation
72237b8 to
2b970ff
Compare
|
Verdict: FAIL at 2b970ff Ran: cd plugins/workflow && npm test (98/98 PASS) FindingsFAIL: PR body format blocks merge The PR body has two defects that will cause enqueue-guard to refuse this pull request:
Implementation review: PASS The failure recovery implementation itself is correct:
Next: Fix the PR body format (plain Owner:/Reviewer: lines + checkpoint block), then re-request review at the same tip. |
Implements #1526: configurable failure recovery edges that re-run from an upstream node when a node fails, instead of stopping the whole run. - Add onFail edges: { from, to, onFail: true, max: 3 } - When a node fails with recovery attempts remaining, reset to the upstream target and re-queue through the failing node - Track failAttempts per edge; stop cleanly when max is exhausted - Update narrate.js to show recovery messages in the stage - Add 5 regression tests covering recovery, exhaustion, and compatibility with failOnce and loop edges Example: Review fail → Rework (max: 3) allows 2 recoveries before failing. Does not interfere with existing failOnce or success-path loop edges. Co-authored-by: Danyang Chen <birfychen@gmail.com>
2b970ff to
9a31805
Compare
|
Verdict: PASS at 9a31805 Ran: Findings✓ onFail recovery implemented correctly:
✓ #1528 concurrency changes intact:
✓ PR body correct:
✓ Test coverage:
✓ Changed files match the checkpoint:
No regressions, clean implementation. Ready to merge. |
|
Verdict: PASS at 9a31805 Ran:
Not run:
FindingsNo actionable findings. onFail edges reset to upstream and re-queue with bounded attempts (1–8, default 3); exhaustion fails permanently; failOnce and loop edges unchanged. Rebase onto #1528 kept HOST_CAP removal and Settings concurrency. Checkpoint tip matches. Matches owner ask for fail→upstream retry (Review→Rework). |
…hief chat (ThinkFlowLab#1527) Owner: Dev Reviewer: Reviewer Written-by: Dev Fixes ThinkFlowLab#1524 ## What changed **Before:** Workflow ephemeral node Bot replies still reached Chief chat — `gated()` skipped deferred/resumed/retried/resent deliveries, and after `bot.remove` the ephemeral check failed so late replies leaked. **After:** - `gated()` returns true for ephemeral (or tombstoned) senders even with deferred/resumed/retried/resent, so middleware still runs `stepResult` then denies delivery - `removeBot()` tombstones ephemeral ids before `deleteBot`; post-removal replies are denied without needing a live bot - Dead `ctx.runtime?.tombstoneEphemeral` call removed from the workflow plugin ## Verification at the tip Tip SHA: 680d516 Ran: `TZ=UTC npm run verify` → green Not run: Desktop / LIVE smoke (needs prod tip with this fix) ## Checkpoint <!-- flybot:checkpoint --> ```json { "pr": 1527, "claim": "Dev", "issue": 1524, "summary": "Ephemeral Workflow node Bot replies no longer leak into Chief chat. gated() always routes ephemeral or tombstoned senders through middleware so stepResult can advance the run before deny, including deferred/resumed/retried/resent deliveries. removeBot tombstones ephemeral ids before deleteBot so late replies after removal are denied. The dead ctx.runtime tombstone call in the workflow plugin is gone.", "files": ["core/src/host/hooks.js", "plugins/workflow/index.js", "core/test/workflow-reply-leak.test.js", "patches/ds-bot/files/src/host/hooks.js", "patches/ds-bot/files/test/workflow-reply-leak.test.js", "changelog.d/1524-workflow-reply-leak.fixed.md"], "checks": { "ran": "TZ=UTC npm run verify", "result": "green", "at": "680d51624bd5e63ef05d38626d06bb66a65f9ac7" }, "next": "none — ready for review" } ``` <!-- /flybot:checkpoint --> <!-- CURSOR_AGENT_PR_BODY_BEGIN --> Owner: Dev Reviewer: Reviewer Fixes ThinkFlowLab#1524 ## Root Cause Production tip `55e1c27a` had the workflow send hook that returns `{action: 'deny'}` for owned ephemeral bot replies, but the LIVE leak persisted because: 1. **Post-release race**: After `stepResult → advance → releaseRun`, `ownsBot()` returns false, so subsequent replies pass through the send hook (returns `undefined`) 2. **Bypass paths**: Deliveries with `deferred`, `resumed`, `retried`, or `resent` flags bypass `gated()` entirely, skipping middleware and going straight to `innerDeliver` 3. **Post-removal leak**: After `bot.remove`, `rt.botOf()` returns undefined, so ephemeral checks fail and late replies leak ## The Fix **Expand `gated()` check** so ephemeral bot deliveries ALWAYS route through middleware (where `stepResult` runs), then deny delivery: 1. **Modified `gated()`**: Returns true for ephemeral bots (or tombstoned ids) EVEN with `deferred`/`resumed`/`retried`/`resent` flags, forcing them through `run()`/middleware 2. **Tombstone in `removeBot()`**: When any plugin calls `fb.bot.remove(id)` on an ephemeral bot, `removeBot()` in `hooks.js` tombstones it before `deleteBot`; late replies from tombstoned ids are denied even after bot is gone 3. **Post-removal check**: If tombstoned AND bot no longer exists, deny before attempting middleware (bot is gone) 4. **Workflow send hook unchanged**: Still calls `stepResult` then returns deny This ensures: - **First reply**: Middleware runs → `stepResult` advances workflow → deny blocks Chief delivery ✓ - **Deferred/resumed/retried/resent reply**: Gated by ephemeral flag → middleware runs → denied ✓ - **Post-removal reply**: Tombstone check denies (bot gone, no middleware needed) ✓ - **Request TO ephemeral**: Allowed (sender is not ephemeral) ✓ ## Changes - **`core/src/host/hooks.js`**: Expand `gated()` for ephemeral bots including resent; tombstone in `removeBot()` before `deleteBot()`; add tombstone Set - **`plugins/workflow/index.js`**: Remove dead `ctx.runtime?.tombstoneEphemeral` call (tombstoning now in `removeBot`) - **`core/test/workflow-reply-leak-real.test.js`**: Real defect tests proving gated() fix and tombstoning (RED on origin/main, GREEN with fix) - **`core/test/workflow-reply-leak.test.js`**: Existing mock-based tests - **`changelog.d/1524-workflow-reply-leak.fixed.md`**: Describe the real fix ## Testing **Real defect tests** in `workflow-reply-leak-real.test.js` directly test the actual `gated()` logic from hooks.js, proving: - Ephemeral with deferred/resumed/retried/resent: RED on main (bypasses → leaks), GREEN with fix (gated → middleware → denied) - Post-removal tombstone: RED on main (bot gone → bypasses → leaks), GREEN with fix (tombstone → gated → denied) - Normal bot behavior unchanged Rebased onto `origin/main` (includes ThinkFlowLab#1529). All tests pass: `TZ=UTC npm run verify` ✓ <!-- flybot:checkpoint --> ```json { "pr": 1527, "claim": "Dev", "issue": 1524, "summary": "Fixed workflow ephemeral Bot reply leak by expanding gated() so ephemeral deliveries always route through middleware (ensuring stepResult advances workflow) while workflow send hook denies Chief delivery. Tombstoning moved to removeBot in hooks.js. Added real defect tests proving the fix. Rebased onto main.", "files": [ "core/src/host/hooks.js", "plugins/workflow/index.js", "core/test/workflow-reply-leak-real.test.js", "core/test/workflow-reply-leak.test.js", "patches/ds-bot/files/src/host/hooks.js", "patches/ds-bot/files/test/workflow-reply-leak-real.test.js", "patches/ds-bot/0005-social-seams.patch", "changelog.d/1524-workflow-reply-leak.fixed.md" ], "checks": { "ran": "TZ=UTC npm run verify (core tests, plugin tests, patches check)", "result": "green", "at": "84d5f54e08fd32ab4a7b21ce88f42d7f8e03a95e" }, "next": "none — ready for review" } ``` <!-- /flybot:checkpoint --> Docs-not-needed: Delivery gate logic; no subsystem architecture change <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-f1e4a924-8129-586f-9025-328f8f7dea37?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-f1e4a924-8129-586f-9025-328f8f7dea37&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Owner: Dev
Reviewer: Reviewer
Written-by: Dev
Fixes #1526
What changed
Before: A Workflow node failure stopped the run (dead end); there was no way to return upstream and retry (e.g. Review → Rework).
After:
onFailedges{ from, to, onFail: true, max: 3 }send failure back to an upstream node and re-queue the failing nodeVerification at the tip
Tip SHA: 9a31805
Ran:
cd plugins/workflow && npm test→ 99/99 pass (5 new failure-recovery tests; +1 from #1528)Not run: full
npm run verify(workflow-only change); smoke (no index.js / src/host/); DesktopCheckpoint
{ "pr": 1529, "claim": "Dev", "issue": 1526, "summary": "Workflow nodes can recover from failure by following onFail edges back upstream and retrying, instead of ending the run as a dead end. Attempts are bounded per edge (1–8, default 3); after exhaustion the node fails permanently as before. Stage messages show recovery, and existing failOnce and success-path loop edges stay unchanged. Rebased onto main after #1528 so concurrency Settings and HOST_CAP removal remain intact.", "files": ["plugins/workflow/src/graph.js", "plugins/workflow/src/service.js", "plugins/workflow/src/narrate.js", "plugins/workflow/test/failure-recovery.test.js", "plugins/workflow/README.md", "docs/subsystems/workflow.md", "changelog.d/1526.added.md"], "checks": { "ran": "plugins/workflow npm test", "result": "green", "at": "9a31805bcbaca46a559eca51bee1e09f2d0d378c" }, "next": "none — ready for review" }