Skip to content

Fail connection-dropped commands with RedisConnectionException, never a raw transport exception or a cancellation - #3287

Merged
mgravell merged 1 commit into
mainfrom
marc/issue-3283
Oct 22, 2026
Merged

mgravell merged 1 commit into
mainfrom
marc/issue-3283

Conversation

@mgravell

Copy link
Copy Markdown
Collaborator

Fixes #3283

Causes (all three in the report confirmed)

  1. The drain hands out raw or null exceptions. RecordConnectionFailed builds its detailed RedisConnectionException only for the bridge's current connection and only on the first report. A repeat report (write path and read loop both reporting), or one against a connection the bridge has already replaced, drains _writtenAwaitingResponse with the raw inner exception, or with null, which TaskResultBox turns into TaskCanceledException. Every EOF path in the read loop reports with no exception.
  2. A write failure races the drain. The write-lock catch fails the message with a proper exception, then reports the connection, which drains the same still-queued message again with the raw exception. Last write wins, hence the RESPite writer's InvalidOperationException in the reported stack trace.
  3. Bridge refusals use Cancel(). FailDueToNoConnection, TimedOutBeforeWrite and the fail-fast bin complete the message as cancelled. ExecuteAsyncImpl then tries to fault the task with the real exception, but the box's continuation is queued to the thread pool and sometimes gets there first. A batch has no such translation at all.

Changes

  • RecordConnectionFailed: if the exception chosen for the drain is not already a RedisException, wrap it in a RedisConnectionException carrying the failure type and the original as InnerException.
  • PhysicalBridge: the three refusal paths set the same exception the caller path would have built (ExceptionFactory.NoConnectionAvailable / Timeout) instead of cancelling. Message.Cancel itself is untouched; transactions rely on it for aborted conditions.
  • ConnectionFailureExceptionTests: the drain with null and with an IOException inner, on a bridge-less dummy connection (exactly the non-current path), plus the no-connection refusal observed at the message's own task rather than through the caller's translation.

Verification

Check Result
Reporter's repro verbatim, 90 s, before 884 TaskCanceledException, 415 InvalidOperationException
Same repro, 90 s, after only ok and RedisConnectionException
New tests + write-failure, backlog, batch, transaction tests 469 passed
Full net10.0 suite on the docker topology 6665 passed, 161 skipped, 1 failed (FailoverTests.SubscriptionsSurvivePrimarySwitchAsync, pre-existing flake, same rate on main)
dotnet build Build.csproj -c Release /p:CI=true 0 warnings, 0 errors

Note: TimedOutBeforeWrite now builds the full timeout exception in the bridge while the single-command caller still builds its own from the result, so that path allocates twice. It only runs when the write lock cannot be taken within the timeout; left as is rather than restructure the caller.

Related: #3282 (#3286), the batch hang from the same report series.

… raw or cancelled (#3283)

When a connection failed, most queued commands failed with RedisConnectionException
but a few received the raw transport exception (IOException, or the RESPite
writer's InvalidOperationException) or a TaskCanceledException with no token in
play. Three sources, all confirmed with the reporter's repro:

- RecordConnectionFailed builds its detailed RedisConnectionException only for
  the bridge's current connection and only on the first report; a repeat report
  (the write path and the read loop both reporting), or one against a connection
  the bridge has already replaced, drained the queue with the raw inner
  exception - or with null, which the result box turns into a cancellation
  (every EOF path in the read loop reports with no exception).
- The write-lock catch fails the message properly and then reports the
  connection, which drained the same still-queued message again with the raw
  exception; last write wins.
- FailDueToNoConnection, TimedOutBeforeWrite and the fail-fast bin completed the
  message via Cancel(); ExecuteAsyncImpl then races to fault the task with the
  real exception, and the box's thread-pool continuation sometimes wins. A batch
  has no caller translating the result at all.

Fix: the drain always uses a RedisConnectionException (wrapping whatever it was
given, with the failure type), and the bridge's refusals set the exception the
caller path would have built instead of cancelling. Message.Cancel itself is
unchanged; transactions rely on it for aborted conditions.

ConnectionFailureExceptionTests covers the drain (null and IOException inner,
on a bridge-less connection, i.e. the non-current path) and the no-connection
refusal observed at the message's own task. Reporter's repro, 90 s: 884
TaskCanceledException + 415 InvalidOperationException before; none after.

Fixes #3283
@mgravell
mgravell merged commit 10caa39 into main Oct 22, 2026
7 of 9 checks passed
@mgravell
mgravell deleted the marc/issue-3283 branch October 22, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant