Skip to content

Route as a cluster only with a slot map; take HELLO mode/role from the interactive connection only - #3285

Merged
mgravell merged 2 commits into
mainfrom
marc/issue-3284-hello-cluster-mode
Oct 22, 2026
Merged

mgravell merged 2 commits into
mainfrom
marc/issue-3284-hello-cluster-mode

Conversation

@mgravell

Copy link
Copy Markdown
Collaborator

Addresses #3284 (cross-slot commands refused client-side on a non-clustered Azure Managed Redis after a connection blip on 3.4.5).

Diagnosis

Reproduced with the in-process server (HelloClusterModeUnitTests): a server whose HELLO says mode: cluster while INFO says redis_mode:standalone, with every CLUSTER subcommand refused.

  • The subscription connection's handshake is HELLO alone (no INFO, no CLUSTER), and under RESP3 pub/sub has had its own connection since Use a dedicated pub/sub connection under RESP3 by default #3264. AMR prefers RESP3.
  • When only that leg reconnects (the /Subscription SocketClosed in the report), its HELLO retypes the server as Cluster and nothing on that leg corrects it. On first connect the interactive leg's INFO lands last and wins, which is why it ran for 24h first.
  • The type change triggers a reconfigure; with an already-connected interactive leg that is a PING tracer only, so the server now counts as a cluster and the multiplexer enters cluster mode with no slot map. Cross-slot commands then throw. This is also the path that produced RedisServerException "ERR command is not allowed" after v3.4.0 update #3274 (same reporter, pub/sub via SignalR).

Still worth measuring on a real Enterprise-policy AMR database: HELLO 3 mode versus INFO server redis_mode. The fix holds either way, but it decides whether the refused-CLUSTER test model should say cluster in INFO or only in HELLO.

Changes

  • Cluster routing requires a slot map. Servers that identify as a cluster but give us no map are routed as a single standalone server (new event id 133). The map is never cleared once populated, so an established cluster whose refresh is refused (Tolerate refused CLUSTER SLOTS/NODES during topology updates #3275) keeps routing as one. When a map arrives, stale redundant-primary flags from an earlier election are cleared.
  • AnyServer accepts a cluster-typed server while routing as standalone (ServerType.IsRoutableAs); it previously required exact equality, which dropped every connection in that mode.
  • HELLO mode/role are applied only from the interactive connection. Per-server facts come from the connection that does discovery; the subscription leg must not overwrite them.
  • Toy server: overridable HelloModeValue so HELLO and INFO can disagree. Docs: one sentence in Configuration.md.

Deliberately not done: HELLO version/vendor on the subscription leg are still applied (harmless, both legs agree); could be gated the same way for consistency.

Verification

  • New tests: pass under RESP2 and RESP3 (RESP3 fails before the change with the exact exception from the report).
  • ClusterCommandsRefusedUnitTests extended to assert standalone routing and a working cross-slot MGET.
  • All in-process-server test classes, net10.0: 645 passed, 16 skipped.
  • Full suite against the docker topology, twice: 6661 passed, 161 skipped, 1 failed. The failure is FailoverTests.SubscriptionsSurvivePrimarySwitchAsync (RESP2), which fails at the same rate (2 of 3 class-level runs) on a baseline worktree at the branch point and passes in isolation: pre-existing flake, unrelated.
  • dotnet build Build.csproj -c Release /p:CI=true: 0 warnings, 0 errors.

…e interactive connection (#3284)

A proxied deployment can answer HELLO with "mode: cluster" while INFO says
redis_mode:standalone and every CLUSTER subcommand is refused (Azure Managed
Redis with the Enterprise clustering policy). Under RESP3 pub/sub now has its
own connection (#3264), whose handshake is HELLO alone: when that leg alone
reconnected, its HELLO retyped the server as a cluster with no INFO to follow,
the resulting reconfigure put the multiplexer into cluster mode with no slot
map, and cross-slot commands were refused client-side.

- Cluster *routing* now requires a slot map: servers that identify as a cluster
  but give us no map are routed as a single standalone server (new event id
  133). The map is never cleared once populated, so an established cluster
  whose refresh is refused (#3275) keeps routing as one. When a map does
  arrive, stale redundant-primary flags from an earlier election are cleared.
- AnyServer accepts a cluster-typed server while routing as standalone
  (ServerType.IsRoutableAs), so such a server stays a valid target.
- HELLO "mode" and "role" are applied only from the interactive connection;
  the subscription connection does no discovery and must not overwrite
  server-level facts.

Tests: HelloClusterModeUnitTests reproduces the subscription-only reconnect
under both protocols (fails under RESP3 before this change);
ClusterCommandsRefusedUnitTests asserts standalone routing and a working
cross-slot MGET when CLUSTER is refused. The in-process server gains an
overridable HelloModeValue so HELLO and INFO can disagree.
…range slot ranges

The routing decision in ReconfigureAsync needs only "is there any slot map",
not a count. Add ServerSelectionStrategy.IsSlotMapEmpty, derived from whether
the map has been allocated: the map is allocated only immediately before a
write, writers never store null, and nothing clears an entry, so a non-null
map always covers at least one slot. CountCoveredSlots remains for the log
line in the cluster branch, where it has always been.

That derivation is only sound if nothing can allocate without writing, so
UpdateClusterRange now ignores an inverted or out-of-range range instead of
allocating and then writing nothing (or indexing out of bounds). SlotRange
does not validate ordering and the reply-driven paths build ranges straight
from what the server sent.

SlotMapUnitTests covers the rejected ranges leaving the map unallocated.
@mgravell

Copy link
Copy Markdown
Collaborator Author

Follow-up commit (8d6e3cc): the routing decision now uses ServerSelectionStrategy.IsSlotMapEmpty (derived from whether the map was allocated, O(1)) instead of the scan; CountCoveredSlots stays only for the log line in the cluster branch, as before. UpdateClusterRange ignores inverted or out-of-range ranges so the map is never allocated without an owner being written (previously an out-of-range slot would have thrown IndexOutOfRange there). SlotMapUnitTests covers the rejected ranges. Release build clean; in-process test classes 646 passed.

@mgravell
mgravell marked this pull request as ready for review October 22, 2026 09:59
@mgravell
mgravell merged commit 7870ba9 into main Oct 22, 2026
6 checks passed
@mgravell
mgravell deleted the marc/issue-3284-hello-cluster-mode branch October 22, 2026 10:11
FloatingMilkshake pushed a commit to FloatingMilkshake/MechanicalMilkshake that referenced this pull request Oct 24, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [StackExchange.Redis](https://seredis.dev/) ([source](https://github.com/StackExchange/StackExchange.Redis)) | nuget | patch | `3.4.0` → `3.4.20` |

---

### Release Notes

<details>
<summary>StackExchange/StackExchange.Redis (StackExchange.Redis)</summary>

### [`v3.4.20`](https://github.com/StackExchange/StackExchange.Redis/releases/tag/3.4.20)

[Compare Source](StackExchange/StackExchange.Redis@3.4.5...3.4.20)

#### Stability improvements

- Do not abandon a connection attempt before ConnectTimeout has elapsed by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3277](StackExchange/StackExchange.Redis#3277)
- Read HELLO "server" before trusting "version" as the Redis API version by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3279](StackExchange/StackExchange.Redis#3279)
- Sentinel soft-fail follow-ups: discovery failure as RedisConnectionException, late sentinel init, clear LastException by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3280](StackExchange/StackExchange.Redis#3280)
- Sentinel: re-resolve the primary when it is demoted on a healthy connection by [@&#8203;rebaseandpanic](https://github.com/rebaseandpanic) in [#&#8203;3281](StackExchange/StackExchange.Redis#3281)
- Route as a cluster only with a slot map; take HELLO mode/role from the interactive connection only by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3285](StackExchange/StackExchange.Redis#3285)
- Fail connection-dropped commands with RedisConnectionException, never a raw transport exception or a cancellation by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3287](StackExchange/StackExchange.Redis#3287)
- Connection groups: a member answering LOADING triggers an immediate re-probe and re-select by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3289](StackExchange/StackExchange.Redis#3289)
- IBatch: complete commands refused by a poisoned connection; divert a batch's tail on reconnect by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3286](StackExchange/StackExchange.Redis#3286)
- Give multi-message sub-commands and injected prefix commands a write time by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3293](StackExchange/StackExchange.Redis#3293)
- Send primary-only commands to the primary when the caller prefers a replica by [@&#8203;AsadIqbal-Dev](https://github.com/AsadIqbal-Dev) in [#&#8203;3291](StackExchange/StackExchange.Redis#3291)

#### Maintenance

- Tests: stabilise two recurring Windows CI flakes (hash-field expiry clock skew, backlog counter quiescence) by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3288](StackExchange/StackExchange.Redis#3288)
- Tests: don't race the connect's own reconfigures in AllClusterCommandsRefusedIsQuiet by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3290](StackExchange/StackExchange.Redis#3290)
- Refresh package pins: 10.0.13 servicing family and tooling by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3294](StackExchange/StackExchange.Redis#3294)
- xunit.v3 4.0 on Microsoft.Testing.Platform by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3295](StackExchange/StackExchange.Redis#3295)
- StyleCop.Analyzers 1.3.151-beta, and the tidy-up it asks for by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3296](StackExchange/StackExchange.Redis#3296)

#### New Contributors

- [@&#8203;AsadIqbal-Dev](https://github.com/AsadIqbal-Dev) made their first contribution in [#&#8203;3291](StackExchange/StackExchange.Redis#3291)

**Full Changelog**: <StackExchange/StackExchange.Redis@3.4.5...3.4.20>

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODEuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE4MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://git.milkshake.services/FloatingMilkshake/MechanicalMilkshake/pulls/10
Reviewed-by: Milkshake <me@floatingmilkshake.com>
FloatingMilkshake pushed a commit to FloatingMilkshake/Azusa that referenced this pull request Oct 24, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [StackExchange.Redis](https://seredis.dev/) ([source](https://github.com/StackExchange/StackExchange.Redis)) | nuget | patch | `3.4.0` → `3.4.20` |

---

### Release Notes

<details>
<summary>StackExchange/StackExchange.Redis (StackExchange.Redis)</summary>

### [`v3.4.20`](https://github.com/StackExchange/StackExchange.Redis/releases/tag/3.4.20)

[Compare Source](StackExchange/StackExchange.Redis@3.4.5...3.4.20)

#### Stability improvements

- Do not abandon a connection attempt before ConnectTimeout has elapsed by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3277](StackExchange/StackExchange.Redis#3277)
- Read HELLO "server" before trusting "version" as the Redis API version by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3279](StackExchange/StackExchange.Redis#3279)
- Sentinel soft-fail follow-ups: discovery failure as RedisConnectionException, late sentinel init, clear LastException by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3280](StackExchange/StackExchange.Redis#3280)
- Sentinel: re-resolve the primary when it is demoted on a healthy connection by [@&#8203;rebaseandpanic](https://github.com/rebaseandpanic) in [#&#8203;3281](StackExchange/StackExchange.Redis#3281)
- Route as a cluster only with a slot map; take HELLO mode/role from the interactive connection only by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3285](StackExchange/StackExchange.Redis#3285)
- Fail connection-dropped commands with RedisConnectionException, never a raw transport exception or a cancellation by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3287](StackExchange/StackExchange.Redis#3287)
- Connection groups: a member answering LOADING triggers an immediate re-probe and re-select by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3289](StackExchange/StackExchange.Redis#3289)
- IBatch: complete commands refused by a poisoned connection; divert a batch's tail on reconnect by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3286](StackExchange/StackExchange.Redis#3286)
- Give multi-message sub-commands and injected prefix commands a write time by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3293](StackExchange/StackExchange.Redis#3293)
- Send primary-only commands to the primary when the caller prefers a replica by [@&#8203;AsadIqbal-Dev](https://github.com/AsadIqbal-Dev) in [#&#8203;3291](StackExchange/StackExchange.Redis#3291)

#### Maintenance

- Tests: stabilise two recurring Windows CI flakes (hash-field expiry clock skew, backlog counter quiescence) by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3288](StackExchange/StackExchange.Redis#3288)
- Tests: don't race the connect's own reconfigures in AllClusterCommandsRefusedIsQuiet by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3290](StackExchange/StackExchange.Redis#3290)
- Refresh package pins: 10.0.13 servicing family and tooling by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3294](StackExchange/StackExchange.Redis#3294)
- xunit.v3 4.0 on Microsoft.Testing.Platform by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3295](StackExchange/StackExchange.Redis#3295)
- StyleCop.Analyzers 1.3.151-beta, and the tidy-up it asks for by [@&#8203;mgravell](https://github.com/mgravell) in [#&#8203;3296](StackExchange/StackExchange.Redis#3296)

#### New Contributors

- [@&#8203;AsadIqbal-Dev](https://github.com/AsadIqbal-Dev) made their first contribution in [#&#8203;3291](StackExchange/StackExchange.Redis#3291)

**Full Changelog**: <StackExchange/StackExchange.Redis@3.4.5...3.4.20>

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODEuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE4MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Reviewed-on: https://git.milkshake.services/FloatingMilkshake/Azusa/pulls/10
Reviewed-by: Milkshake <me@floatingmilkshake.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant