Repository navigation
Connection groups: a member answering LOADING triggers an immediate re-probe and re-select - #3289
Merged
Merged
Conversation
…e-probe and re-select, like a breaker trip The group's fast-path only reacted to ConnectionFailureType.CircuitBreaker. A server that is loading fails every reconnect during the handshake, so commands routed to it die in the backlog without a reply, and the breaker only observes replies - it can never be what moves traffic away. This is the shape of the recurring Windows CI failure of WithRetry_FailsOverBetweenGroupsOnLoading (six attempts in twelve days, always 20 x 5000ms backlog timeouts): when the heartbeat rather than the test's command met the first LOADING, nothing ever failed over. LOADING now joins CircuitBreaker in the fast-path; the health check still decides where traffic goes, so a member whose probe passes is kept (WithRetry_WhenFailoverNeverArrives_KeepsRetryingSameMember). New LoadingRerouteTests fails without the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The group's
ConnectionFailedfast-path only reacted toConnectionFailureType.CircuitBreaker. A server that is loading its dataset fails every reconnect during the handshake, so commands routed to it die in the backlog without a reply, and the circuit breaker only observes replies (PhysicalConnection.ObserveMessageResult): it can never be what moves traffic away from a loading member.This is the shape of the most persistent Windows CI flake,
RetryEndToEndTests.WithRetry_FailsOverBetweenGroupsOnLoading(six failed attempts in twelve days, net481 only, always 1m41s = 20 attempts x 5000ms backlog timeout). When the heartbeat rather than the test's GET met the first-LOADING, member A's connection was torn down, its handshake ECHO kept getting LOADING, every attempt queued against A and timed out, and the group never failed over.Change:
LoadingjoinsCircuitBreakerin the fast-path: mark the member unhealthy, re-probe, re-select, debounced as before. The health check still decides where traffic goes, so a member whose probe passes is kept (WithRetry_WhenFailoverNeverArrives_KeepsRetryingSameMemberis unchanged and still passes). Ordinary socket drops are still left to reconnect.Verification
LoadingRerouteTests.LoadingMember_ReroutesAwayPromptlyfails without the library change (both protocols) and passes with it.docs/Failover.mdupdated. Release build ofBuild.csprojclean.Companion test-only stabilisation: see the
marc/ci-flakesPR.