From 754c0947c237bca44e954b2d61ec6b41da392f06 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 14 Oct 2026 14:04:44 +0100 Subject: [PATCH] DOC-7185 Document StackExchange.Redis failback health checks and lag-aware check Adds failback health checks (v3.4.5, StackExchange.Redis#3242) to the .NET geographic failover page, as two H4 subsections under Failback configuration. MultiGroupOptions.FailbackHealthCheck and ConnectionGroupMember.FailbackHealthCheck decide only whether an inactive member is eligible to become active. They never mark a member unhealthy. HealthCheck.LagAware(LagAwareOptions) is the built-in Redis Software Active-Active check against the database availability API. The page adds a LagAwareOptions property table, a FailbackHealthCheck row in the member property table, and one sentence on the existing anti-flap tiebreak. Every name, default and behavior was read from the 3.4.5 source, not the PR description. That covers the PublicAPI.Unshipped.txt entries, the XML docs in LagAwareOptions.cs and HealthCheck.LagAware.cs, and the MultiGroupMultiplexer selection code (IsFailbackEligible, _provisional). The 3.4.5 tag is #3242's own merge commit. Upstream docs/Failover.md compares SE.Redis with "other clients" (a 5000 ms default tolerance, never selecting a lagging member). That comparison was left out: it describes other libraries, unverified, and our per-client pages document their own defaults. SetUserPemCertificate() is net6.0-only (#if NET), so the page qualifies it. db_viewer is named as a management role, matching the Redis Software REST API docs. Learned: SE.Redis LagAware is a failback eligibility check, not a liveness check; it never fails over the active member Constraint: failback checks never mark members unhealthy or remove the active member; keep the page from implying they trigger failover Gaps: C# snippet not compiled (NuGet is sandbox-denied); the ~1 minute lag-blind window is upstream's Redis Software 8.0 measurement, not verified here Gaps: no full Hugo build (PostCSS/npm blocked in the worktree); new links and anchors checked by script Ticket: DOC-7185 Co-Authored-By: Claude Opus 5.5 --- content/develop/clients/dotnet/failover.md | 86 ++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/content/develop/clients/dotnet/failover.md b/content/develop/clients/dotnet/failover.md index 472f5b0c06..5062745878 100644 --- a/content/develop/clients/dotnet/failover.md +++ b/content/develop/clients/dotnet/failover.md @@ -113,6 +113,7 @@ endpoint: | `HealthCheck` | Group default | Per-member [health check](#health-check-configuration) override. | | `CircuitBreaker` | Group default | Per-member [circuit breaker](#circuit-breaker-configuration) override. | | `FailbackDelay` | Group default | Per-member [failback delay](#failback-configuration) override. | +| `FailbackHealthCheck` | Group default | Per-member [failback health check](#failback-health-checks) override. | | `SkipInitialHealthCheck` | `false` | If `true`, skip the health check performed when the member is first added to the group. | Note that you can also adjust weights at runtime, for example in response to changing conditions: @@ -319,6 +320,91 @@ automatic failback is effectively disabled. However, you can use the `ResetIsUnh `TryFailoverTo()` methods to enable a member manually (see [Manual failover](#manual-failover) for more information). +When two members are otherwise equally good candidates, the group keeps the member that is +already active rather than switching between them. + +#### Failback health checks + +A [health check](#health-check-configuration) tells you whether a member is alive, but +that isn't always enough to decide whether to fail back to it. For example, in an +[Active-Active](/content/operate/rs/databases/active-active/_index.md) deployment, a region that +has just recovered can be reachable but still behind the other regions. Failing back to it +too early means your app reads stale data and makes changes based on it. + +From v3.4.5, you can add a *failback health check* that a member must pass before it can +become the active member. Set it for the whole group with +`MultiGroupOptions.FailbackHealthCheck`, or for one member with +`ConnectionGroupMember.FailbackHealthCheck`. The value is a `HealthCheck`, so you can use a +[custom probe](#custom-health-check-probes) or the built-in +[lag-aware check](#lag-aware-failback-check) for Redis Software. No failback health check +is configured by default. + +A failback health check works differently from the main health check: + +- It only decides whether an inactive member is *eligible* to become active. It never + marks a member as unhealthy, and it never moves traffic away from the active member. +- A member is eligible only if the check reports that it is healthy. A result of + unhealthy or inconclusive (for example, when the probe can't reach the service it + queries) leaves the member ineligible. +- If no connected member is eligible, the group falls back to selecting from the members + that are alive, so a precaution against stale data doesn't cause an outage. A member + selected this way is *provisional*: the group switches to an eligible member as soon as + one is available, regardless of weight. +- Calling `TryFailoverTo()` overrides the failback health check, in the same way that it + overrides `IsUnhealthy`. + +#### Lag-aware failback check (Redis Software only) {#lag-aware-failback-check} + +For Redis Software Active-Active databases, `HealthCheck.LagAware()` uses the cluster's +[database availability API](/content/operate/rs/monitoring/db-availability.md) to check +whether a member is available and has caught up with the other regions. The only option +you must supply is a callback that returns the cluster's management credentials: + +```csharp +ConnectionGroupMember member = new("redis-14460.us-west.example.com:14460,password=", name: "US West") +{ + Weight = 10, + FailbackHealthCheck = HealthCheck.LagAware(new LagAwareOptions + { + // Called for every request, so the credentials can rotate. + Credentials = _ => new(new NetworkCredential("", "")), + }), +}; +``` + +These are the cluster's REST API credentials, not the database's. A user with the +`db_viewer` management role is sufficient. If the cluster can't be queried (for +example, if the REST API is unreachable or rejects the credentials), the result is +inconclusive, so the member stays ineligible but nothing fails over. + +The `LagAwareOptions` class provides the following properties: + +| Property | Default | Description | +| :-- | :-- | :-- | +| `Credentials` | (Required) | Callback that returns the cluster management credentials as a `NetworkCredential`. | +| `RestEndpoint` | Derived | URL of the cluster's REST API. By default, this is derived from the member's first endpoint: `https://:9443/` for an endpoint named `redis-.`, or `https://:9443/` otherwise. Set it explicitly if you connect using a hostname that doesn't follow this pattern. Redirects are never followed. | +| `DatabaseId` | Discovered | The database ID on the cluster. By default, the database is found by matching the member's first endpoint against the cluster's databases. | +| `LagCheck` | `LagCheckMode.Auto` | Whether to check lag as well as availability. `Auto` checks lag only for Active-Active databases, `Enabled` always checks it (so a database that isn't Active-Active is always reported unavailable), and `Disabled` checks availability only. | +| `LagTolerance` | Cluster setting | Maximum acceptable lag. By default, no tolerance is sent, so the cluster's own `availability_lag_tolerance_ms` setting applies. | +| `CertificateValidation` | Platform default | Callback to validate the REST API's TLS certificate. | +| `ClientCertificate` | None | Client certificate to present if the REST API requires mutual TLS. | + +The cluster's management certificate is usually self-signed, so it won't validate against +the platform's trusted certificates. Call `TrustIssuer()` on your `LagAwareOptions` instance +with the path to the cluster's certificate (or its issuer) to trust it. This is a separate +setting from the TLS configuration for the database connection. Similarly, use +`SetUserPfxCertificate()` (or `SetUserPemCertificate()`, on .NET 6 and later) to load a +client certificate from a file. + +`HealthCheck.LagAware()` runs a single probe per check. To change the probe count or +timeout, pass `HealthCheckProbe.LagAware(options)` as the `Probe` of a +`HealthCheck.Builder` instead. + +When the link between regions breaks, the database availability API can keep reporting +the database as caught up for around a minute before it detects the problem (as measured +on Redis Software 8.0). A failback during that window isn't protected by the lag check, so +set a [`FailbackDelay`](#failback-configuration) of at least a minute alongside it. + ## Health check configuration Each health check consists of one or more separate *probes*, each of which is a simple