pool: exact dispatch redesign with stale worker and node fencing - #83
Merged
Conversation
raphael
force-pushed
the
pool-exact-dispatch
branch
from
July 25, 2026 05:14
3a753ff to
57f216d
Compare
Dispatch admission, retry resolution, and the start event become one Redis linearization point with real MaxQueuedJobs admission and durable generation-immutable terminal dispatch records. Cleanup runs exclusively under persisted Redis-time leases; scheduler planning and transitions run under distributed ticker ownership with foreign-job protection; settlement is node-owned with retry/join semantics so definitive outcomes are never abandoned. Every mutation a stale process could attempt is fenced at its own Redis linearization point: worker start claims re-verify worker registration, the cleanup fence, and the admitted payload identity; graceful requeue deactivation refuses under a cleanup fence and never recreates a removed registration; and job dispatch plus every scheduler transition and ownership script re-check the node keep-alive registration and node-cleanup field. Cleanup also removes legacy dispatch-map keys so adopted deployments retain no stale metadata.
Graceful shutdown now self-acquires the worker requeue lease - the same Redis-time capability stale-worker cleanup uses - atomically deactivating the worker's registration, and republishes jobs through the shared lease-fenced stable publication records. Exactly one party requeues each job, a duplicate start event can no longer be published by concurrent graceful and takeover requeuers, and an interrupted graceful requeue is finished by takeover after lease expiry. The requeue deadline is WithRequeueTimeout, matching its v1 WithWorkerShutdownTTL meaning. A dispatch payload identity mismatch is now classified as a terminal invariant violation: the event is acknowledged with the error instead of being redelivered forever.
raphael
force-pushed
the
pool-exact-dispatch
branch
from
July 25, 2026 16:25
57f216d to
356c0bc
Compare
Registration now writes membership atomically in Redis, so the writing process's rmap replica converges through the update channel instead of synchronously; the rotation test must poll the replica like the other membership assertions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contract
Stacked on #82 (requires the generational stream lifecycle).
Dispatch admission, retry resolution, and start-event publication become one Redis linearization point with real
MaxQueuedJobsadmission (replacing approximateMAXLENtrimming that silently lost unread jobs) and durable generation-immutable terminal dispatch records; cross-node waiters re-read terminal records on every wake/timeout edge. Cleanup runs exclusively under persisted Redis-time leases; scheduler planning and transitions run under distributed ticker ownership with foreign-job protection; settlement is node-owned with retry/join semantics so a definitive outcome is never abandoned.WorkerTTLis generation-immutable via the manifest (pool format 7); staleness comparisons use RedisTIME.Stale-process fencing happens at each mutation's own linearization point:
DISPATCHIDENTITYMISMATCHis terminal — acked with the error, never redelivered).nodeLivenessFenceLua), closing the window between heartbeat preflight and mutation.Acceptance
go test ./... -raceagainst live Redis: green at this tip.TestStaleWorkerCannotClaimAfterCleanupFence), graceful/takeover lease arbitration incl. expired-lease recovery (TestGracefulRequeueLeaseArbitratesWithCleanup), stale node dispatching/scheduling after node cleanup (TestStaleNodeCannotDispatchOrScheduleAfterCleanup), heartbeat/cleanup linearization, requeue dedup across owners.Known residual, noted not blocking:
DISPATCHLOST/POOLGENERATIONLOSTclaim failures keep their pre-existing requeue classification; revisiting that is backlog.