Api to drain to passive

[scottf]

Perform a planned move of application traffic from the active socket to the passive socket, without a reconnect: new outbound traffic starts flowing on what was the passive socket, while the old active socket is kept alive for a period so that work already in flight on it can still complete, after which the old active is retired (ideally repurposed as the new warm passive rather than closed).

[scottf]

Analysis

Summary

Evaluated on the merits, the feature is viable and addresses a real gap, in a specific bounded form. It is not viable in the "keep both alive indefinitely" form. The decision comes down to one tradeoff the application owner has to make, not to a hard impossibility.

The real gap it addresses

The existing switchToPassive is a steal: it force-closes the active socket and promotes the passive's socket under the same connection identity. That is clean and loss-free for one category of in-flight work — queued outbound publishes are carried across and flushed on the promoted socket. But the steal can lose two categories of in-flight inbound work, because it discards the old socket:

• Messages that the server had already sent toward the old socket but which had not yet been read and delivered — they're sitting in the old socket's buffer when it's force-closed.
• Responses to requests that were already in flight — if the responder published the reply toward the old socket before the promoted socket's inbox interest propagated, that reply lands on a socket that's being discarded.

So the motivation is legitimate. A graceful move that keeps the old socket readable for a window would let that in-flight inbound work land instead of being dropped. This is a genuine improvement over the steal for a planned switch, not a solution in search of a problem.

How it would actually work

The coherent design keeps the single connection identity (the application keeps talking to the same object) and moves only the primary I/O:

1. Make the was-passive socket the connection's primary socket for all new outbound traffic — publishes, new subscriptions, and new requests go out on it. This is the existing steal machinery, which re-registers the connection's subscriptions (including its reply inbox) on the new socket.
2. Keep reading the old socket for a bounded drain window, delivering whatever arrives into the same connection — the same subscription map and the same outstanding-request map. This is the one new mechanism the steal doesn't have today: a second reader feeding the one connection identity.
3. When the window elapses, stop and unsubscribe the old socket. Then either close it or, more efficiently, keep its socket warm and repurpose it as the new passive — no reconnect needed, since it's already a live authenticated connection.

The window has to be bounded by a timeout, not run "until truly done." In-flight inbound messages arrive quickly and are finite, but a request response can legitimately never come (the request may simply time out), so "wait until the old socket has nothing left" has no guaranteed end. A bounded window is the correct and only workable termination — the same shape the connection's existing drain(timeout) uses.

The one tradeoff that decides it

While both sockets are readable during the window, the connection has the same subscription interest live on two sockets at once. That produces two effects, and they are not symmetric:

• Request/reply is essentially self-correcting. Outstanding requests are matched and removed on first response (responsesAwaiting.remove), so a duplicate reply arriving on the other socket is dropped as an orphan. Keeping both readable during the window therefore recovers in-flight responses (good) without duplicating them to the caller (harmless). This is the part I over-stated as a wall in the first pass; on the delivery path it is actually the easy case.
• Plain subscriptions are the real exposure. The message-delivery path does not dedup — it enqueues whatever arrives. So during the window, a plain subscription with interest on both sockets can see the same message twice, and because the two sockets have no ordering relationship to each other, messages arriving across them can be reordered relative to strict per-subject order.

So the honest framing is a loss-versus-duplication tradeoff over a bounded window, not a feasibility question:

• switchToPassive (steal): zero duplication, zero reordering, but a window of possible loss of in-flight inbound messages and responses.
• graceful swap (keep old readable): zero loss of in-flight inbound, responses recovered and auto-deduped, but a bounded window of possible duplicate and reordered plain-subscription messages.

Neither is strictly better. Which one is correct depends entirely on the consumer.

When it's worth building, and when it isn't

• Worth it for consumers that are idempotent or already assume at-least-once, and that don't depend on strict per-subject ordering. For them, a brief, bounded window of duplicates during a planned switch is a good price for not dropping in-flight inbound work. Request/reply-heavy workloads benefit the most, since responses are recovered and deduped for free.
• Not worth it for consumers that require exactly-once-ish behavior at the application layer without their own dedup, or that rely on strict subject ordering. For them the duplicate/reorder window is a correctness regression, and the loss-free steal (switchToPassive) is the better planned-switch primitive.

Risk: a connection outage during the swap

A steal (switchToPassive) is close to atomic — one handover, and at any instant the connection depends on exactly one socket. The graceful swap is not: for the duration of the drain window it deliberately runs with two live sockets feeding one identity, one of them (the new primary) freshly promoted and not yet load-proven. An outage that lands inside that window is therefore materially riskier than an outage during a steal, and the risk differs by which socket drops.

The new primary (was passive) drops. This is the worst case. All new outbound traffic has just been pointed at this socket, and it was a warm standby that had never actually carried load — so its liveness at the moment of promotion is the least proven thing in the system. If it was silently half-open (TCP still thinks it's up, the server is gone), the swap "succeeds," new publishes go into a black hole, and the failure only surfaces when the ping/timeout machinery eventually trips. If by then the old active has already been partially dismantled (subscriptions unsubbed, socket demoted or closed), there is nothing healthy to fall back to: a fresh dead primary and a torn-down former primary at the same time. Recovery drops to a full pool reconnect even though a perfectly good socket existed moments earlier.

The old active (draining) drops. Lower severity, but it silently forfeits the feature's entire benefit for that switch: the in-flight inbound messages and responses the drain window was there to catch are gone the instant that socket dies. No corruption, just "the guarantee you opted into didn't happen this time." There is also an implementation hazard: because the old socket is now a second reader feeding the one connection identity, its error must be routed to "end the drain early," never to "the active died." If it leaks into the reconnect path it would spawn a spurious failover on the healthy new primary — turning a harmless drain-socket loss into a real reconnect of good traffic.

Both drop (client-side partition / full cluster outage). The connection must fall into full reconnect, and the danger is state consistency at that moment. Mid-swap, subscription interest is split across two sockets and the reply inbox may be in transition, and there may be no cleanly-armed passive (the old passive was consumed as the new primary; the old active is half-retired). Convergence depends on the client-side subscriber map being the single source of truth that a reconnect resubscribes from — so the swap must never mutate that map in a way that loses a subscription if it is interrupted. Server-side SUB/UNSUB has to be treated as derived and re-derivable, not as state the swap can strand.

Outstanding requests straddling the outage are the most fragile item of all, because a reply can only be delivered if the inbox interest is registered on a reachable socket at the instant the responder publishes it — and mid-swap that is exactly what is uncertain. A request sent on the old socket whose reply arrives after that socket dies is lost; a request sent on the new primary is exposed to that socket being unproven.

What contains the risk is ordering discipline, and it is the core of a safe implementation:

1. Do not dismantle the old active until the new primary is proven live — a completed ping/pong round trip on it, not merely "the writer started." This collapses the dangerous middle state into "fully on old" or "fully on new," with no interval where a healthy socket has been torn down for an unproven one.
2. Keep the old active fully subscribed until that commit point, so that if the new primary fails first, the swap can simply abort and stay where it was.
3. Bound the "prove the new primary" wait as well as the drain window, so a half-open new socket makes the swap give up and abort rather than hang.
4. Isolate the drain reader from the reconnect machinery: an error on the old socket ends the drain, and only an error on the primary drives failover.

With that discipline the residual risk reduces to an acceptable shape — "the drain benefit is best-effort and is silently skipped if the specific socket dies" — and stays away from the unacceptable ones — a standby-less or subscription-inconsistent connection. Without it, an outage in the window can leave the connection worse off than if the swap had never been attempted, which is the strongest argument for making the operation strictly ordered and abortable rather than a fire-and-forget role flip.

Recommendation

Build it, but as a bounded, opt-in graceful drain-and-switch, not as an indefinite dual-active. The settled name is drainToPassive(Duration drainWait) — a deliberately distinct method rather than a switchToPassive overload, so the duplicate/reorder tradeoff below is opted into by choosing the method, never stumbled into by adding an argument. It (a) moves primary I/O to the passive socket, (b) continues reading the old socket for the caller-supplied drainWait window so in-flight inbound work lands, and (c) then retires the old socket, preferably by demoting it to the warm passive rather than closing and reconnecting. Document plainly that the window can duplicate and reorder plain-subscription messages, so the caller opts in with eyes open; note that request/reply is unaffected because responses dedup on match.

Do not offer an unbounded "keep both active indefinitely" mode. That version never terminates for subscriptions (subscription interest has no natural end) and only defers the same unsub-migration decision forever while paying the duplicate/reorder cost the whole time. The value is entirely in the bounded window; the indefinite version has the costs without a terminating benefit.

Implementation plan for drainToPassive

Signature and end state

public void drainToPassive(Duration drainWait) throws InterruptedException;

Guard exactly like switchToPassive: throw IllegalStateException if there is no connected passive to move to. End state after the call returns: the connection's primary I/O runs on what was the passive socket, a fresh passive is armed on another server (the active/passive invariant is preserved throughout), and the old active socket is being read for up to drainWait before it is retired.

The two-reader mechanism (the crux)

Today a connection has exactly one live reader; the steal adopts the passive's reader and hands the dead one back precisely so there is never two. drainToPassive needs two readers feeding the one connection identity at once:

• Primary reader = the was-passive socket's reader, adopted into this connection (existing Hook 1, reader.setConnection(this)), same as the steal. All new outbound (writer, new subs, new requests) runs on this socket.
• Drain reader = the OLD active socket's reader, which is already pointed at this connection — so instead of stopping it and force-closing its port (what the steal does), we keep it running and hold it in a new drainReader field. It keeps delivering into this connection's subscribers / responsesAwaiting maps for the window.

No new jnats hook is required to run two readers — the reader field is just duplicated into drainReader, and delivery targets are already concurrent maps (verify ConsumerMessageQueue.push is safe under two concurrent producer threads before relying on it).

The one part that needs a decision: drain-reader error isolation

NatsConnectionReader calls connection.handleCommunicationIssue(io) on socket death whenever its running flag is true (NatsConnectionReader.java:247), and that drives a reconnect. For the drain reader that is wrong: an unplanned death of the old socket during the window would spawn a reconnect on the healthy primary. The reader conflates "keep reading" with "own the reconnect" in the single running flag, and there is no read-only mode. Two resolutions:

1. Small jnats change (clean, preferred). Add a read-only mode to NatsConnectionReader: on error, stop silently rather than calling handleCommunicationIssue. Same size as the existing Hook 1 change. Cost is a jnats snapshot + build.gradle bump.
2. Ap-only heuristic (no jnats change, more fragile). Override handleCommunicationIssue in ApConnection; during the window, swallow the error if the primary is still healthy (treat as a drain-socket failure), otherwise reconnect for real. Avoids upstream coordination but is racy when both sockets die together.

The planned window-end path needs neither: drainReader.stop() first (flips running false), then force-close the old port, so the shutdown IOException is skipped — the same ordering the steal already relies on.

Ordered algorithm

1. Prove the new primary is live. Ping the passive socket and wait for the pong, bounded (reuse the connection timeout). If it doesn't return, abort and leave the active untouched — never dismantle a healthy active for an unproven target.
2. Promote, keeping the old reader alive. Adopt the passive's reader/port as primary and restart the primary writer on it (steal machinery). Do NOT force-close the old port and do NOT stop the old reader — retain it as drainReader. Stop the old socket's writer (no more outbound there) and cancel its ping/cleanup tasks.
3. Reset primary liveness state and softPing the new primary — same cleanUpPongQueue / needPing.set(true) / softPing() step the steal uses, applied to the promoted socket.
4. Arm a fresh passive (reArmPassive), so a warm standby exists for the whole window.
5. Schedule window end at now + drainWait on the scheduled executor: drainReader.stop(), then force-close the old port in the background. That retires the old socket.

Retirement: v1 vs v2

• v1 (recommended first): at window end, close the old socket. The fresh passive from step 4 already satisfies the invariant, so nothing else is needed. Simple and correct.
• v2 (optimization, defer): instead of closing the old socket and having armed a brand-new passive, demote the old socket into the passive role so no new passive connect is needed. This is the reverse of a steal (wrap an existing live socket back into a passive NatsConnection) and is materially more complex; not worth it until v1 is proven.

Test plan

• Move happens: after drainToPassive, connected URL is the old passive's server, a fresh passive is armed elsewhere, primary RTT() works.
• In-flight inbound lands: a message/response arriving on the old socket after the promotion is still delivered during the window (the whole point).
• Drain window terminates: after drainWait the old socket is closed and its reader stopped; no leak.
• Guard: no connected passive → IllegalStateException, active untouched.
• Abort: passive proves dead in step 1 → active untouched, no move.
• Error isolation: kill the old socket mid-window → NO reconnect on the primary (this is the test that pins whichever isolation approach is chosen).
• Each new behavior verified non-vacuous by disabling the code and seeing exactly its assertion fail, per the pattern used for switchToPassive.

Build sequence

1. Resolve the error-isolation decision first (spike the jnats read-only mode, or prove the ap-only override) — it gates everything.
2. Implement steps 1–5 with v1 retirement.
3. Tests, including the mid-window kill.
4. Defer v2 demote-to-passive.

Relationship to the first-pass analysis

The first pass (SWAP_ACTIVE_PASSIVE_ANALYSIS.md) concluded "don't build it." That conclusion holds specifically for the indefinite "two live actives with the same subscriptions" reading of the feature, and its two "walls" are real for that reading. What that pass got wrong by treating the walls as absolute: (1) it counted duplicate request responses as a problem when the delivery path already dedups them, and (2) it framed "no drained state" as a dead end when a bounded timeout is a perfectly good termination — the same one drain() uses. Seen without the prior, the feature is a legitimate bounded primitive with a real tradeoff, and the honest answer is "yes, in bounded opt-in form, for consumers that tolerate a brief duplicate/reorder window."

Appendix — grounding in the code

• Steal / identity-preserving promotion and re-subscription on the new socket: ApConnection.reconnectImplConnect; NatsConnection.reconnectImpl drives sendSubscriptionMessage / resendSubscriptions.
• Outbound publishes preserved across the steal: enterWaitingForEndReconnectMode.
• Response dedup (why duplicate replies are harmless): NatsConnection.deliverReply — responsesAwaiting.remove(key), second arrival returns null and is counted as an orphan.
• No dedup on plain messages (why duplicate subscription delivery reaches the app): NatsConnection.deliverMessage — looks up the subscription by SID and enqueues, with no duplicate check.
• Bounded-drain precedent and its terminating-by-timeout shape: NatsConnection.drain(Duration).
• Per-connection reply inbox identity: mainInbox = createInbox() + ".*", createInbox() = inboxPrefix + nuid.next().

[scottf]

switchToPassive vs forceReconnect

The comment was true, switchToPassive() does pretty much the same as forceReconnect() but it includes a guard that fails fast if there is no healthy passive to switch to, rather than quietly reconnecting through your normal server list—plus its name clearly states its purpose. Either one moves you to the passive. So the on-demand control is in place. We can leave it in, giving you a choice of which to use.

The core thing you asked for — being able to move traffic off a server you've decided is underperforming — already works today. On an active/passive connection, promoting the passive to active is what happens on any reconnect: the automatic failover when the active's socket dies, an explicit forceReconnect(), and the dedicated switchToPassive() call all do the same thing under the hood. Each one lets go of the current active, promotes the warm passive socket to be the new active, re-establishes your subscriptions on it, and arms a fresh passive on another server. So triggering the switch yourself is a single call, and it reuses the exact same, already-proven failover path that a real outage would.

Why I'm against building the "graceful drain" version

I've been prototyping the "graceful" version of the active-to-passive switch — the one that keeps the old connection open to drain in-flight messages while new traffic moves to the standby — and I've concluded we should not ship it. The plain switch is fine; what I'm against is specifically the "drain the old one while the new one runs" behavior. The reasons are fundamental, not something a bit more engineering would polish away.

Start with the part that can't be fixed. Draining means both sockets are live and delivering into the same message stream at once, and two connections to two different servers have no defined order between them — their messages interleave however the timing happens to fall. That interleaving is inherent to reading two sockets at the same time; no amount of cleverness removes it. And it's worst in exactly the case that makes you want to switch: you switch because a server is slow, and "slow" means it has a backlog of undelivered messages. Draining that backlog mixes those stale, late messages in with the fresh ones from the fast server, so the feature misbehaves most precisely when it's needed most.

Then there's the part that's merely hard, and it's where the real complexity lives. I have to know when a subscription is "done" on the old connection so I can re-establish it on the new one, and I can't leave the same subscription live on both at once. So I'd have to end each one somehow — on a timeout, or after a message count — and both are guesses, with different failure modes, that the library would be making on your behalf, even with your assistance via parameterization. Doing that correctly across every subscription is a lot of machinery.

Underneath both of those is something the library simply can't know: whether order even matters to you. At this layer a message is just a message. The extension can't tell an order-sensitive subscriber from a request/reply flow (which wouldn't care) from a publisher (which has nothing to drain). So the one case that would actually be corrupted is the one case it can't identify in order to protect it — which means getting this right would need your input for each application anyway, at which point the library isn't really saving you the work.

Finally, the payoff is small even when it works. NATS delivery is at-most-once, so dropping whatever was still queued on the slow server at the moment you switch is already within the delivery contract you're operating under — it isn't data loss in the sense the contract promises. So the drain would take on real correctness risk to hold onto something that's already acceptable to drop.

But you can manually accomplish the objective

You can already accomplish what you're really after, on your own terms, with what NATS gives you today. Subscriptions and dispatchers both have a drain method — subscription.drain(Duration) / dispatcher.drain(Duration) — that unsubscribes and then processes whatever is still in flight for that subscription, in order, up to a timeout you set. So if there's a subscription whose backlog you don't want to lose when you move off a slow server, you can drain it yourself first — on the current connection, where the messages still arrive in order on a single socket — and then do the switch and resubscribe on the new connection. That gives you exactly the "don't drop the in-flight work" behavior you're looking for, except you're the one deciding which subscriptions it applies to and what ordering means for them, which is the part the library can't safely guess. The only cost is a bit of time finishing the drain on the slow server before you switch — but that's the honest tradeoff either way: you either wait for those messages or you let them go. And if you want finer control than drain gives you, the plain unsubscribe calls are there too — subscription.unsubscribe() to stop a subscription immediately, or subscription.unsubscribe(int after) to stop it automatically after a given number of additional messages — so you can shape exactly how much of the backlog you take before moving.

So my recommendation is to ship the immediate switch, which gives you the control you asked for — move off a struggling server on demand — without any of these problems, and to leave the library-side draining behavior out. The immediate switch also carries your outgoing (already-queued) messages over to the new connection, so what you give up automatically is only the unread inbound on the slow server — and if that inbound matters for a given subscription, the manual drain above lets you keep it, correctly and under your control.