pool: treat an occupied scheduled key as a satisfied transition - #85
Merged
Conversation
A scheduler transition whose planned key is already running had its dispatch rejected with ErrJobExists, which released the scheduler's ownership record and aborted the rest of the plan. The next transition then proposed a fresh dispatch ID and failed identically, so a scheduled job that outlived one interval — or a key held by a foreign job — produced a permanent re-dispatch loop and starved every job planned after it. An occupied key already satisfies the transition. A job this scheduler dispatched keeps its ownership so later transitions retry the same exact dispatch instead of minting a new identity; a foreign occupant drops the speculative claim and is retried next transition, preserving the rule that a scheduler never owns or stops a job it did not dispatch. Either way the remaining planned jobs still run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found in production: a scheduler whose planned job key was already running looped once per second — dispatch returned
ErrJobExists, which released the scheduler's ownership record and aborted the remaining plan, so the next transition minted a fresh dispatch ID and failed identically. Any scheduled job that outlives one interval (or a key held by a foreign job, e.g. one that was requeued and lost its dispatch identity) never converges, and every job planned after it is starved.An occupied key already satisfies the transition. A job this scheduler dispatched now keeps its ownership record, so later transitions retry the same exact dispatch instead of minting a new identity; a foreign occupant drops the speculative claim and is retried on the next transition, preserving the invariant that a scheduler never owns or stops a job it did not dispatch. Either way the remaining planned jobs still run.
Tests:
TestSchedulerCollisionNeverOwnsForeignJobnow asserts convergence (no error, no ownership, later plan entries still dispatched, foreign job untouched byclearJobs) andTestSchedulerKeepsOwnershipWhenItsOwnJobStillRunspins identity stability across repeated transitions. Full repo-racesuite green against live Redis.