Leave the consumer group promptly on shutdown

[delthas]

Problem

close() unsubscribed then waited for the rebalance callback to un-assign before disconnecting:

this._consumer.unsubscribe();
this.once('unassign', () => next());   // unbounded

librdkafka delivers no such callback when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress. So the wait never completed: close() never returned, the SIGTERM handler never reached process.exit, and the pod was SIGKILLed with the member still registered at the broker. The group then held zero partitions until session.timeout.ms (45s) evicted it — the takeover stall this ticket is about.

Mechanism

rd_kafka_consumer_close() (reached via disconnect()) already performs the whole protocol exit from any join state. Its terminate0 calls the internal unsubscribe with leave=true, which bypasses the postponement that blocks the application-level unsubscribe(), and it sends the LeaveGroup on its own. Its one requirement is that any rebalance callback it delivers on the close queue is answered — with unassign() (assign() is refused once the close has started).

sequenceDiagram
    participant App as close()
    participant RK as node-rdkafka
    participant CG as librdkafka cgrp
    App->>RK: unsubscribe()
    Note over CG: postponed while a rebalance is in progress
    App->>RK: disconnect()
    RK->>CG: consumer_close() then terminate0
    Note over CG: internal unsubscribe(leave=true)<br/>bypasses the postponement
    CG-->>App: REVOKE (if partitions still held)
    App->>RK: unassign() — answered at once
    CG->>CG: LeaveGroup sent
    RK-->>App: 'disconnected'

Loading

This corrects the F_LEAVE_ON_UNASSIGN_DONE framing in #2819: the leave comes from terminate0, and the explicit pre-disconnect unsubscribe/unassign are only there to drain in-flight work first, not to arm the leave.

Approach

Rather than gate disconnect() on an un-assign that may never arrive:

• Wait for the drain only when a revoke is actually coming — one already parked and draining, or about to be triggered by the unsubscribe because we still hold partitions — and bound that wait. A revoke genuinely in progress drains under the full budget (its own watchdog emits unassign at maxPollIntervalMs - 1000); a revoke that never starts (an unsubscribe postponed by an in-flight rebalance) is given only a short grace, then we let disconnect() perform the leave. Waiting the full budget there would run past the pod's grace period.
• Answer every rebalance callback raised once shutting down — decline a grant, un-assign a revoke — so disconnect() cannot wedge on an unanswered one.
• Bound the disconnect wait itself.
• Stop fetching once the shutdown has started, so the pipeline the drain is waiting on cannot refill.
• Bound the wait for an in-flight offset publish, rather than rescheduling close() indefinitely if the ZooKeeper publish never calls back.

The first commit migrates close() to async/await (a behavior-preserving refactor, close(cb) kept as a thin shim), per the repo's async/await migration guidance; the second is the fix above.

The un-assign wait uses a guarded _getAssignments() (mirroring _getSubscription()), since assignments() throws ERR__STATE on a closing client (BB-845).

close() flow

flowchart TD
    A["close(cb)"] --> B["_closing = true<br/>stop fetch loop, cron, circuit breaker"]
    B --> C{"consumer connected?"}
    C -->|no| Z["disconnect step"]
    C -->|yes| D["best-effort unsubscribe()"]
    D --> E{"waiting on a drain<br/>or partitions held?"}
    E -->|no| Z
    E -->|yes| W["wait for 'unassign' (bounded)"]
    W -->|"'unassign' emitted<br/>(drain done or watchdog)"| Z
    W -->|"grace elapsed,<br/>revoke in progress: keep waiting"| W
    W -->|"grace elapsed,<br/>no revoke started"| G["give up: unassign()"]
    G --> Z
    Z --> H["_disconnecting = true<br/>disconnect(), bounded 5s"]
    H --> I["cb()"]

Loading

Rebalance callbacks during shutdown

Only the disconnect phase short-circuits the drain; the same single drain path handles a revoke while running and while shutting down.

stateDiagram-v2
    direction LR
    [*] --> Running
    Running --> ShuttingDown : close() called
    ShuttingDown --> Disconnecting : drain done or bound hit
    Disconnecting --> [*] : disconnected

    note right of Running
        ASSIGN -> assign(partitions)
        REVOKE -> drain, then unassign()
    end note
    note right of ShuttingDown
        fetch loop stopped
        ASSIGN -> decline: unassign()
        REVOKE -> drain, then unassign()
    end note
    note right of Disconnecting
        ASSIGN -> decline: unassign()
        REVOKE -> unassign() now, no drain
    end note

Loading

Relationship to the earlier stack

Supersedes the substance of #2819 / #2834 / #2835, rebuilt as one change on development/9.5 (not stacked on #2818). It keeps a single drain path — the existing revoke handler — rather than adding a second one for shutdown. The double-close() coalescing (#2836) is intentionally left out.

Known residual

When the leaving member has its own JoinGroup in flight (it was revoked and is rejoining), its explicit LeaveGroup is transport-blocked behind that request. close() still returns within its bound and the remaining members take over via a normal rebalance; only that member's fast explicit leave is delayed. This is a librdkafka transport-ordering limitation, not specific to this change; lowering session.timeout.ms (BB-844) bounds its cost.

Tests

• Unit tests covering every shutdown path (the un-assign wait and its grace/re-arm, the bounded disconnect, declining a grant and answering a revoke during shutdown, and stopping consumption).
• Two real-broker departure tests: prompt takeover when the group is settled, and close() returning within its bound when called during a rebalance.

Issue: BB-833

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/bypass_source_branch_lineage	Bypass the cross-branch contamination check	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request.
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[codecov]

Codecov Report

❌ Patch coverage is 95.77465% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.57%. Comparing base (56a3c42) to head (2dfe4d5).

Files with missing lines	Patch %	Lines
lib/BackbeatConsumer.js	95.77%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/constants.js	100.00% <ø> (ø)
lib/BackbeatConsumer.js	94.45% <95.77%> (+0.70%)	⬆️

... and 2 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	80.27% <ø> (ø)
Core Library	81.42% <95.77%> (-0.41%)	⬇️
Ingestion	70.09% <ø> (ø)
Lifecycle	80.46% <ø> (ø)
Oplog Populator	85.83% <ø> (ø)
Replication	62.01% <ø> (ø)
Bucket Scanner	85.76% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.5    #2838      +/-   ##
===================================================
- Coverage            75.72%   75.57%   -0.15%     
===================================================
  Files                  200      200              
  Lines                13926    13979      +53     
===================================================
+ Hits                 10545    10565      +20     
- Misses                3371     3404      +33     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.07% <0.00%> (-0.04%)	⬇️
api:routes	8.83% <0.00%> (-0.04%)	⬇️
bucket-scanner	85.76% <ø> (ø)
ft_test:queuepopulator	9.20% <8.45%> (-1.83%)	⬇️
ingestion	12.21% <1.40%> (-0.05%)	⬇️
lib	9.13% <91.54%> (+0.33%)	⬆️
lifecycle	19.45% <64.78%> (+0.14%)	⬆️
notification	1.01% <0.00%> (-0.01%)	⬇️
oplogPopulator	0.13% <0.00%> (-0.01%)	⬇️
replication	18.99% <64.78%> (+0.16%)	⬆️
unit	55.39% <94.36%> (+0.46%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.