Repository navigation
Redis client does not work when disconnected and reconnected on Linux #2894
Description
Redis client does not work when disconnected and reconnected on Linux
Configure config. ReconnectRetroPolicy=new Exponential Retro (5000);
All other configurations are correct. Under Windows, reconnection is not a problem, but under Linux, reconnection does not work. If the time exceeds 20 seconds, the background cannot be pulled up and reconnected.
The server test versions are Windows 11 and Ubuntu 20
Activity
dpvq commentedon Oct 8, 2026
I have been experiencing the same problem on linux. I am using the latest version
3.3.1. It does not receive any published information after the connection is reestablished.I see a
int: Connectingon the status of the connection.The subscriber IsConnected becomes false and is not recovered.
mgravell commentedon Oct 8, 2026
I will investigate, thanks.
mgravell commentedon Oct 9, 2026
problem rather than fixing it.
What I tried. A client with a live subscription connected through a local TCP proxy to a TLS Redis, with the proxy injecting four kinds of fault for 40 seconds (and one for 4 minutes) before recovering:
On 3.3.1 and on current main, on Linux, the client reconnected and pub/sub resumed within 0.5 to 6 seconds of the fault ending, in every case. A 4-minute stall that aborted about 50 connection attempts still recovered in 3 seconds, and the aborted attempts did not leak sockets.
What matched your snapshot. While the "TCP accepted, TLS never completes" fault was still in progress, the client looked exactly like yours:
int: Connecting; sub: n/a; not in use: DidNotRespond, with the socket count climbing by one every few seconds. That state means each reconnect attempt is getting a TCP connection but never completing the handshake, and the client is giving up on each one after the connect timeout and trying again. It is the client behaving correctly against a path that is still broken, not a client that has wedged. I could not find any way to leave it in that state once the path healed.What would help. Two things from your side:
Last Connection Exception:part. In my runs it saysConnectTimeoutwhen TCP itself hangs, andThere was an authentication failure; check that passwords...when TCP connects but TLS never completes. (That second wording is misleading for a stalled handshake; I will look at that separately.) Which one you see tells us where the attempts are dying.*.redis.cache.windows.net:6380that could accept TCP without passing TLS through: a corporate proxy, an egress firewall, a service mesh sidecar, or a VPN. The symptom is consistent with the client's reconnect attempts being accepted by something that is not Redis - presumably the azure redis proxy.Also worth noting: 3.3.1 shares one connection for commands and pub/sub under RESP3. The next release uses a dedicated pub/sub connection by default (#3264), and reports the stage at which each connection attempt fails (#3257), which would make this much easier to diagnose from your side. Upgrading when that ships is a non-breaking change from 3.3.1.
If you can capture the exception text (and ideally the connection log from
ConnectionMultiplexer.Connect(config, log)across a failure), I am happy to dig further.dpvq commentedon Oct 14, 2026
Hello @mgravell. When it does not reconnect anymore there is no exception logged (and I added my logger to the constructor properly
new new LoggerTextWriter(_logger)).I had a special situation where my primary DNS was not responding, but the secondary was good. A connection to Redis is established and I simulate the disconnection by disabling the network adapter, waiting 2 minutes and enabling it. This is the way to simulate my problem.
So can you put an invalid primary DNS server and connect to Redis then disconnect and reconnect the network adapter after 2 minutes to see if you can reproduce the problem? You may need to restart the OS after changing the DNS to be sure it's applied.
Also I am using the new version 3.4.5 of the library.
Before disabling the connection
int: ConnectedEstablished; sub: ConnectedEstablishedAfter the 2 minute disconnection and connection restablished
int: Connecting; sub: Connecting; not in use: DidNotRespondmgravell commentedon Oct 15, 2026
Thanks, the DNS detail was the key. I can now reproduce this exactly, on Linux, on 3.4.5.
The client resolves the host name as part of each connection attempt, under the same
connectTimeout(5 s by default) as the TCP connect. With a primary DNS server that does not answer, glibc waits out its own timeout before trying the secondary, so every lookup takes at least 5 s, every reconnect attempt is cancelled during resolution, and the client never gets as far as opening a socket. That is theint: Connecting; sub: Connecting; not in use: DidNotRespondyou see, with the socket count climbing on each retry. Windows' resolver remembers that the first server is dead and skips it, which is why the same setup recovers there. Your initial connection worked because the OS still had the name cached; toggling the adapter flushed it.There is a second, smaller problem on our side that makes it worse: the reconnect schedule aborts an attempt that is still in progress, and with the default settings that can happen after 2.5 s, before
connectTimeouthas elapsed. I will fix that, and I am looking at having reconnects fall back to the last address we successfully used when resolution fails, so a DNS outage does not take the client down.Until then, a workaround that I have verified against the same reproduction: add
resolveDns=true,sslHost=xxx.redis.cache.windows.netto the configuration (replace with your cache name). The name is then resolved once at startup and reconnects use the address directly. The explicitsslHostis required, otherwise TLS validation will fail against the IP."No exception logged" is expected: the logger passed to
Connectonly covers the initial connection, and repeated connect failures are deliberately not re-raised throughConnectionFailed.ConfigurationOptions.LoggerFactoryis the way to see them.Do not abandon a connection attempt before ConnectTimeout has elapsed (…
mgravell commentedon Oct 23, 2026
3.4.20 is out and includes the first part: an in-flight connection attempt is no longer abandoned before
connectTimeouthas elapsed, so a slow lookup gets its full budget. The second part (falling back to the last address we successfully connected to when resolution fails) is still to come; theresolveDns=true,sslHost=...workaround above remains valid in the meantime.dpvq commentedon Oct 23, 2026
Thank you very much for the quick response and the fix.
Merge main into v4 (19 commits to 2026-10-23), with the v3-only fixes…