Skip to content

pool: exact dispatch redesign with stale worker and node fencing - #83

Merged
raphael merged 3 commits into
mainfrom
pool-exact-dispatch
Jul 25, 2026
Merged

pool: exact dispatch redesign with stale worker and node fencing#83
raphael merged 3 commits into
mainfrom
pool-exact-dispatch

Conversation

@raphael

@raphael raphael commented Jul 25, 2026

Copy link
Copy Markdown
Member

Contract

Stacked on #82 (requires the generational stream lifecycle).

Dispatch admission, retry resolution, and start-event publication become one Redis linearization point with real MaxQueuedJobs admission (replacing approximate MAXLEN trimming that silently lost unread jobs) and durable generation-immutable terminal dispatch records; cross-node waiters re-read terminal records on every wake/timeout edge. Cleanup runs exclusively under persisted Redis-time leases; scheduler planning and transitions run under distributed ticker ownership with foreign-job protection; settlement is node-owned with retry/join semantics so a definitive outcome is never abandoned. WorkerTTL is generation-immutable via the manifest (pool format 7); staleness comparisons use Redis TIME.

Stale-process fencing happens at each mutation's own linearization point:

  • A worker start claim re-verifies the worker's registration, the absence of a cleanup fence, and the admitted payload identity (DISPATCHIDENTITYMISMATCH is terminal — acked with the error, never redelivered).
  • Graceful shutdown and stale-worker takeover share one requeue lease: a stopping worker self-acquires the same Redis-time capability cleanup uses (atomically deactivating its registration) and republishes through the shared lease-fenced stable publication records, so exactly one party requeues each job and an interrupted graceful requeue is finished by takeover after lease expiry.
  • Job dispatch and every scheduler transition/ownership script re-check the dispatching node's keep-alive registration and node-cleanup field (nodeLivenessFenceLua), closing the window between heartbeat preflight and mutation.
  • Cleanup deletes legacy dispatch-map keys so adopted deployments retain no stale metadata after generation cleanup.

Acceptance

  • Full repo go test ./... -race against live Redis: green at this tip.
  • Fault injection: worker paused past TTL claiming after takeover (TestStaleWorkerCannotClaimAfterCleanupFence), graceful/takeover lease arbitration incl. expired-lease recovery (TestGracefulRequeueLeaseArbitratesWithCleanup), stale node dispatching/scheduling after node cleanup (TestStaleNodeCannotDispatchOrScheduleAfterCleanup), heartbeat/cleanup linearization, requeue dedup across owners.
  • Diff-scoped review completed; both findings (dual requeue ownership, identity-mismatch classification) fixed in the follow-up commit.

Known residual, noted not blocking: DISPATCHLOST/POOLGENERATIONLOST claim failures keep their pre-existing requeue classification; revisiting that is backlog.

@raphael
raphael force-pushed the pool-exact-dispatch branch from 3a753ff to 57f216d Compare July 25, 2026 05:14
Base automatically changed from streaming-generations to main July 25, 2026 16:24
raphael added 2 commits July 25, 2026 09:25
Dispatch admission, retry resolution, and the start event become one Redis
linearization point with real MaxQueuedJobs admission and durable
generation-immutable terminal dispatch records. Cleanup runs exclusively
under persisted Redis-time leases; scheduler planning and transitions run
under distributed ticker ownership with foreign-job protection; settlement
is node-owned with retry/join semantics so definitive outcomes are never
abandoned.

Every mutation a stale process could attempt is fenced at its own Redis
linearization point: worker start claims re-verify worker registration, the
cleanup fence, and the admitted payload identity; graceful requeue
deactivation refuses under a cleanup fence and never recreates a removed
registration; and job dispatch plus every scheduler transition and ownership
script re-check the node keep-alive registration and node-cleanup field.
Cleanup also removes legacy dispatch-map keys so adopted deployments retain
no stale metadata.
Graceful shutdown now self-acquires the worker requeue lease - the same
Redis-time capability stale-worker cleanup uses - atomically deactivating the
worker's registration, and republishes jobs through the shared lease-fenced
stable publication records. Exactly one party requeues each job, a duplicate
start event can no longer be published by concurrent graceful and takeover
requeuers, and an interrupted graceful requeue is finished by takeover after
lease expiry. The requeue deadline is WithRequeueTimeout, matching its v1
WithWorkerShutdownTTL meaning.

A dispatch payload identity mismatch is now classified as a terminal
invariant violation: the event is acknowledged with the error instead of
being redelivered forever.
@raphael
raphael force-pushed the pool-exact-dispatch branch from 57f216d to 356c0bc Compare July 25, 2026 16:25
Registration now writes membership atomically in Redis, so the writing
process's rmap replica converges through the update channel instead of
synchronously; the rotation test must poll the replica like the other
membership assertions.
@raphael
raphael merged commit ec313e9 into main Jul 25, 2026
5 checks passed
@raphael
raphael deleted the pool-exact-dispatch branch July 25, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant