Proposal: automatic lame-duck handling in the Active/Passive connection

[scottf]

The problem, concretely

When a broker is taken down gracefully it enters lame-duck mode (LDM): it stops accepting new connections and, after a grace period, evicts its existing clients so they reconnect elsewhere. LDM is the normal path for a rolling restart / planned maintenance.

I measured what actually happens on an LDM today (real 3-node cluster, real nats-server --signal ldm):

• The server sends the client a lame-duck notice almost immediately — the LAME_DUCK event fires at ~0.85 s.
• But nothing acts on it. The base NATS client only raises the event; it does not move. The Active/Passive layer currently ignores it too (the hook is an empty // TODO).
• So the client sits on the lame-duck broker until the server forcibly evicts it — at ~10.8 s with the server's default grace period — and only then reconnects, via the normal server-pool walk.

The whole reason to run Active/Passive is that a warm second connection is already open on another broker, so failover can be near-instant. On an LDM we're throwing that away: instead of a sub-second switch onto the already-connected passive, we wait ten seconds and then do a cold pool reconnect — the exact thing the passive exists to avoid. During those ten seconds the active is pinned to a broker that is on its way out.

What I'd like to add

An option on ApConnection that makes it act on the lame-duck notice instead of waiting for eviction:

• Active connection goes lame-duck → immediately switch over to the passive (promote the already-open passive socket to be the new active), then arm a fresh passive on another broker. This is the existing switchToPassive() failover, just triggered automatically by the LDM notice instead of by hand. It's near-seamless because the passive is already connected and authenticated.
• Passive connection goes lame-duck → proactively re-home the passive onto a healthy broker, so it doesn't ride its own broker down to eviction. The active is untouched.

Net effect: a planned broker shutdown becomes a sub-second, warm switch instead of a ~10-second cold reconnect.

The setting

I'd expose it as an enum on ApOptions so it's explicit and leaves room for more behaviors later, rather than a bare boolean:

ApOptions.builder(options)
    .lameDuckHandling(LameDuckHandling.SWITCH_TO_PASSIVE)   // proposed default
    .build();

Proposed values:

• NONE — today's behavior. No proactive move; the client rides the LDM until the server evicts it (~grace period), then reconnects through the pool. Zero risk of moving off a broker that's still fine, at the cost of the slow failover described above.
• SWITCH_TO_PASSIVE — the automatic handling above: active LDM → immediate warm switch onto the passive; passive LDM → proactively re-home the passive. This is the behavior the failover test scenarios assume.

If the customer wants a middle ground, we can add a third value later — e.g. PASSIVE_ONLY (re-home the passive proactively, but let the active ride its LDM out to eviction) — but I'd only build that if they ask; two values cover the real choice.

Recommended default

SWITCH_TO_PASSIVE. A customer who has chosen Active/Passive has already paid for a warm standby specifically to make failover fast; defaulting to NONE would mean the standby sits idle during exactly the planned-maintenance events it's best suited for. The switch is low-risk — the passive is already a live, healthy connection, and if for some reason there is no usable passive at that moment the connection simply falls back to the normal reconnect (i.e. no worse than NONE).

That said, this is a behavior change from what ships today, so I want the customer's explicit yes. If they'd rather keep the current behavior and opt in per-deployment, we set the default to NONE and they turn it on. This is the one thing I need them to decide: is automatic handling the default, or opt-in?

Edge cases (how the automatic path behaves)

• No usable passive when the active goes LDM — fall back to a normal reconnect (same as NONE). The switch is best-effort; it never makes things worse than doing nothing.
• A reconnect is already in progress on the active — the LDM switch is a no-op; the in-flight reconnect will itself land on / promote the passive. (This mirrors how switchToPassive() already guards itself.)
• Passive is also lame-duck (e.g. two brokers drained at once) — the active still switches onto the passive's socket; the passive re-home then picks a third, healthy broker. If none is healthy yet, normal infinite-reconnect takes over on both sides.
• Test/maintenance timing — for our own integration tests we'll also shorten the server's lame_duck_grace_period so the eviction-based (NONE) paths don't each wait ~10 s; that's a test-cluster config knob, not part of this option.

Implementation

This is an Active/Passive-layer change only — three touch points. It reacts to the LAME_DUCK event the base client already raises and calls failover methods the library already has (switchToPassive(), passiveForceReconnect()).

1. New enum — LameDuckHandling

package io.nats.client.impl;

public enum LameDuckHandling {
    /** Take no proactive action; ride the lame-duck until the server evicts (~grace period), then reconnect normally. Today's behavior. */
    NONE,
    /** Active lame-duck -> switchToPassive() steal onto the warm passive; passive lame-duck -> re-home the passive. */
    SWITCH_TO_PASSIVE
}

(PASSIVE_ONLY can be added later if wanted; two values cover the real choice.)

2. ApOptions — carry the setting

• Add public final LameDuckHandling lameDuckHandling;, assigned in the ApOptions(Builder) constructor.
• Builder: a lameDuckHandling field defaulting to the chosen default (the open question above — recommended SWITCH_TO_PASSIVE, or NONE to ship conservative), a lameDuckHandling(LameDuckHandling) setter, a null-guard in build(), and the same field carried in the copy-constructor Builder(ApOptions ap).

3. ApConnection — react to the event

Today BridgeConnectionListener.connectionEvent(...) forwards every event to the server pools and does nothing else (the activeConnectionEvent hook is an empty // TODO). Add a lame-duck branch on each side:

// in connectionEvent(conn, type, time, uriDetails), after the existing pool-forwarding:
if (type == Events.LAME_DUCK) {
    if (activeListener) onActiveLameDuck();
    else               onPassiveLameDuck();
}

private void onActiveLameDuck() {
    if (apOptions.lameDuckHandling != LameDuckHandling.SWITCH_TO_PASSIVE) return;
    // Run OFF the callback executor: switchToPassive() drives a reconnect (the steal) and must not
    // block the connection's event dispatch, and the status updates it fires re-enter this listener.
    lameDuckExecutor.execute(() -> {
        try {
            switchToPassive();                 // promote the warm passive; reArmPassive() follows
        }
        catch (IllegalStateException noPassive) {
            // no live passive to steal -> ride eviction (no worse than NONE); nothing to do
        }
        catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    });
}

private void onPassiveLameDuck() {
    if (apOptions.lameDuckHandling != LameDuckHandling.SWITCH_TO_PASSIVE) return;
    lameDuckExecutor.execute(() -> {
        try {
            passiveForceReconnect();           // re-home the passive onto a healthy broker; active untouched
        }
        catch (Exception e) {
            processException(e);
        }
    });
}

Threading + lifecycle:

• Field: private final ExecutorService lameDuckExecutor = Executors.newSingleThreadExecutor(r -> { Thread t = new Thread(r, "ap-lameduck"); t.setDaemon(true); return t; }); (create only when lameDuckHandling != NONE to avoid an idle thread otherwise).
• close() — add lameDuckExecutor.shutdownNow(); alongside the existing passive-close / super.close() / shutdownExecutors().

Why async matters: connection-listener callbacks run on the connection's callback executor. switchToPassive() internally does updateStatus(DISCONNECTED); reconnectImpl(), which itself fires DISCONNECTED/RECONNECTED events back through this same listener — running it inline would block dispatch and re-enter on the dispatch thread. Dispatching to lameDuckExecutor keeps the event handler non-blocking; switchToPassive()'s existing tryingToConnect guard already makes it a no-op if a reconnect is in flight, so a LDM racing an active failure is safe.

Tests for the feature

Testable on its own with a small cluster + the lame-duck signal (no producer/consumer harness needed):

• SWITCH_TO_PASSIVE: active broker LDM → active's connected URL becomes the old passive URL well under the ~10 s eviction window; a fresh passive arms on a third broker. Non-vacuous check: with NONE, the active stays on the lame-duck broker until eviction (~10 s).
• Passive broker LDM → passive re-homes to a healthy broker; active URL/status unchanged.
• No-passive edge: active LDM while the passive is absent → falls back to eviction/normal reconnect, no exception escapes.
• NONE: today's behavior is preserved (no proactive move).

Why this is an Active/Passive-layer change

It's entirely in this library — reacting to an event the base NATS client already raises, and calling failover methods this library already has (switchToPassive(), passiveForceReconnect()). Nothing changes in the core NATS client.

What I need back

1. Do they want automatic LDM handling at all? (If no, we leave it as NONE/manual and the LDM failover stays ~10 s.)
2. If yes, is SWITCH_TO_PASSIVE the default, or opt-in with NONE as default?
3. Do they want the PASSIVE_ONLY middle option, or are two values enough?