Repository navigation
Commit 84f41ea
authored
fix(execution): stop in-flight workflow runs when an account is deleted, removed or banned (#9145)
* fix(execution): stop in-flight workflow runs when an account is deleted, removed or banned
No removal path stopped workflow executions. Account deletion cascades the
doomed workspaces' log rows away, after which no cancel path can find the run,
so it kept spending hosted provider keys for an account that no longer existed
until it crashed on foreign keys at finalization. A ban archived the
workspaces and stopped nothing; an admin removal cascaded them the same way.
- Account deletion reads the running executions (and claimed resumes, which
run under their own id) of the doomed workspaces in its transaction, while
the workspace rows are locked, and records one outbox stop per workspace with
the erasure. After commit the stops are delivered a few at a time; a stop
that cannot be delivered stays in the outbox for its worker, and the
committed deletion is never reported as failed.
- A stop sets the cancellation flag each engine watches and sweeps the job
queue for the workspace: a new JobQueueBackend.cancelByWorkspace cancels
queued, delayed and running workflow jobs by the workspaceId tag on
Trigger.dev (pages of 100, ten cancels at a time, 1,000 runs per attempt,
continued without spending an attempt) and by metadata.workspaceId on the
database queue, so queued runs no longer fail preprocessing later.
- A missing Redis completes the stop: the flag is process-local by design and
no retry could reach further. A refused Redis write or a failed sweep retries.
- A ban stops the runs of the workspaces it archived; the admin remove-user
hooks read the runs before the bare user delete and record them after it
commits. Both sit in page module graphs that cannot load the delivery, so
they signal in-flight runs at once and leave the queue sweep to the outbox
worker.
* fix(execution): keep removal stops off the deletion and ban critical paths
- Account deletion no longer awaits the full stop delivery before purging
storage. A slow Trigger.dev listing could hold the response for minutes and
widen the window in which a restart orphans erased objects. It now signals
in-flight runs fire-and-forget right after commit, ahead of every other
post-commit step, under a 10-second deadline; the committed outbox stops
repeat the signal and sweep the job queue in the outbox worker.
- A failure to record a ban's stops is logged instead of thrown. The ban and
archives have committed, so a 500 helped no one and a retry finds no
unarchived workspace to stop.
- Claimed resumes are found from the scope's running and paused log rows and
joined through resume_queue's parent index, not by scanning resume_queue
for claimed rows inside the deletion transaction.
- Tests pin the database queue's job-type filter, so a workspace sweep never
cancels a non-workflow job such as log cleanup.
* test(execution): give each removal-stop fixture workflow a unique name
* fix(execution): close the review's gaps in removal stops
- A ban records and signals each workspace's stops as soon as its archive
commits, through the archive's post-commit stop hook, instead of after every
archive's socket and webhook cleanup. It also stops runs in the owner's
workspaces archived before the ban, which an archive leaves running.
- The outbox handler re-reads a workspace's running executions when it runs.
An archived workspace keeps its rows, so a run admitted before the ban that
logged after its stop was recorded is still signalled.
- The admin remove-user hook handlers move to lib/auth/admin-removal-stops.ts,
and signal the removed runs even when recording their stops fails.
- The database queue's workspace sweep works in batches of 100 up to the
caller's cap and checks its abort signal between them, like the Trigger.dev
sweep, rather than one unbounded update reported as complete.
* refactor(execution): tighten removal stops after final audit
- disableUserResources reads owned workspaces once and stops earlier-archived ones up front, dropping the post-archive read that could fail the ban
- Trigger.dev cancelByWorkspace reuses scanAndCancelTriggerRuns; workspace tag shares one builder with enqueue
- database queue shares abortInlineJob with cancelJob and returns completeness from control flow
- removal stop handler parses its payload with zod and signals while it sweeps
- drop the unused event-id return and redundant bucketing state
* fix(execution): finish the removal sweep's edge cases
- The removal stop handler loads both job queues independently and sweeps
whichever loaded, so a primary queue that fails to load no longer skips the
database queue's sweep. The failure is still rethrown for the outbox to retry.
- A claimed resume is found under a parent log row in the legacy `paused`
status too, one of the statuses a resume claims from.
- The database queue re-checks for a matching active job after a short batch,
since a selected job that finished before the update reached it shrinks the
batch while later jobs still match.
- The ban's completion log no longer reports its runs stopped when recording
their stops failed.
* fix(execution): abort local runs before the removal signal deadline can skip them
The signal deadline bounds only the Redis flag writes. Aborting a run this
process holds needs no Redis, so every execution is aborted first, even after
the deadline has passed. Corrects the ban's note on why a failed stop record is
logged: banning again records the stops again.
* fix(execution): guard ban stops against transfers and refusing runs
- A ban records stops only for workspaces the user still owns, re-read in the
recording transaction and held FOR SHARE until it commits, so a concurrent
ownership transfer waits rather than having its new owner's runs stopped.
Only workspace rows are locked, so no lock order is inverted.
- Trigger.dev's workspace sweep counts only cancelled runs toward its cap, with
a separate bound on failures, so runs that keep refusing cancellation no
longer hold every retry at the front of the listing. cancelByExecution is
unchanged.
* fix(execution): sweep a removed workspace's Trigger.dev runs of any age
The workspace sweep searched only runs created in the last 14 days, so an older
run still queued or waiting survived the removal. It now passes only an upper
time bound, which Trigger.dev treats as no lower bound, and falls back to the
retention window only if Trigger.dev refuses that search as too expensive
before returning a run. cancelByExecution is unchanged.1 parent c3f2b30 commit 84f41ea
18 files changed
Lines changed: 1613 additions & 22 deletions
File tree
- apps/sim/lib
- auth
- core
- async-jobs
- backends
- outbox
- execution
- users
- workflows
- packages/testing/src/mocks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| |||
469 | 470 | | |
470 | 471 | | |
471 | 472 | | |
472 | | - | |
473 | | - | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
474 | 477 | | |
475 | 478 | | |
476 | | - | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
477 | 482 | | |
478 | 483 | | |
479 | 484 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
2 | 8 | | |
3 | 9 | | |
4 | 10 | | |
| |||
503 | 509 | | |
504 | 510 | | |
505 | 511 | | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
21 | 24 | | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| 28 | + | |
25 | 29 | | |
26 | 30 | | |
27 | 31 | | |
| |||
51 | 55 | | |
52 | 56 | | |
53 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
54 | 67 | | |
55 | 68 | | |
56 | 69 | | |
| |||
420 | 433 | | |
421 | 434 | | |
422 | 435 | | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
| 436 | + | |
430 | 437 | | |
431 | 438 | | |
432 | 439 | | |
| |||
517 | 524 | | |
518 | 525 | | |
519 | 526 | | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
520 | 586 | | |
521 | 587 | | |
522 | 588 | | |
| |||
0 commit comments