Abort the rebalance when a grant supersedes a parked revoke - #2839
Draft
delthas wants to merge 1 commit into
Draft
Abort the rebalance when a grant supersedes a parked revoke#2839delthas wants to merge 1 commit into
delthas wants to merge 1 commit into
Conversation
On ERR__REVOKE_PARTITIONS the un-assign is deferred until in-flight work drains. librdkafka however answers a subscribed-topic metadata change with an immediate rejoin, without waiting for that un-assign, so a grant can arrive for a generation that has already moved past the work still draining. Carrying on would mean consuming those partitions again from their last commit while still finishing the previous generation's work: the same entries processed twice, concurrently for consumers that do not order by key. The deferred un-assign then discarded the granted assignment outright, leaving a live group member owning partitions at the broker with no local assignment and nothing left to trigger a rebalance. Treat it like the drain timeout instead. Accept the grant, since the draining entries can only store their offsets while the partitions are held, stop consuming, let the pending drain commit and un-assign as it always does, then leave the group so the partitions are taken over and the liveness probe restarts this consumer. Entries delivered by a consume request that was already outstanding are dropped: the accepted grant reset the fetch position to the last commit, so those are the draining entries coming back around. Issue: BB-835 Claude-Session: https://claude.ai/code/session_01JQoM2qBo8JABC43pXSUiF8
Codecov Report❌ Patch coverage is
Additional details and impacted files
@@ Coverage Diff @@
## improvement/BB-833/bound-consumer-close #2839 +/- ##
===========================================================================
+ Coverage 75.57% 75.58% +0.01%
===========================================================================
Files 200 200
Lines 13979 13998 +19
===========================================================================
+ Hits 10565 10581 +16
- Misses 3404 3407 +3
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #2837 for BB-835, taking the approach francoisferrand asked for in review: fit the buggy event into the existing shutdown automaton and fail like the drain timeout, instead of carrying on with an assignment whose generation has moved on.
Stacked on #2838, which is what makes this viable: leaving the group is now prompt and bounded, so aborting no longer risks the 45s takeover stall that BB-833 describes.
The bug
When a revoke arrives with tasks in flight, the un-assign is deferred until the processing queue and offset ledger drain. librdkafka answers a subscribed-topic metadata change with an immediate rejoin without waiting for that un-assign, so a grant can land mid-drain for a newer generation. The deferred un-assign then discarded that grant, leaving a live group member that owns every partition at the broker with no local assignment, healthy on every liveness signal, consuming nothing, with nothing left to trigger a rebalance.
sequenceDiagram participant BB as BackbeatConsumer participant RK as librdkafka participant B as broker RK->>BB: revoke [p0..p4] Note over BB: tasks in flight,<br/>defer unassign() until drained B-->>RK: metadata: subscribed topic looks changed Note over RK: rejoins now, revoke still unanswered RK->>B: JoinGroup + SyncGroup B-->>RK: new generation RK->>BB: assign [p0..p4] Note over BB: abort: accept the grant, stop consuming Note over BB: pending drain commits, then un-assigns BB->>RK: close, LeaveGroup Note over BB,B: partitions taken over at the committed offset,<br/>this consumer is restarted by the liveness probeWhy abort rather than keep the assignment
Continuing is not just a matter of skipping the stale un-assign. The accepted grant resolves fetch positions from the last commit, so the uncommitted tail comes back around while the previous generation's copies of those same entries are still being processed. For consumers that order by message key the two copies serialize, but
GarbageCollectorandLifecycleBucketProcessorboth setorderByFunc: null, so there the same entry can be processed twice concurrently. Aborting avoids that state entirely rather than relying on every task tolerating it.What it does
_waitingDrainis set means librdkafka rejoined without our answer. One condition, no generation counter needed, and it covers the asynchronous offset-publish window because the flag stays set until the un-assign runs._close()from Leave the consumer group promptly on shutdown #2838, then report unhealthy so the liveness probe restarts us, with the sameCRASH_ON_REBALANCE_TIMEOUTexit S3C uses for the drain timeout.Trade-off against #2837
This is strictly for 2.12 and later. On librdkafka 2.3.0 the trigger fires about once a second under metadata thrash, which would be a restart loop rather than a recovery.
Tests
Issue: BB-835
https://claude.ai/code/session_01JQoM2qBo8JABC43pXSUiF8