From 12ab6a13cab738fc3f30a43a9387caa0e0ed4e65 Mon Sep 17 00:00:00 2001 From: Harlan Crystal Date: Mon, 27 Jul 2026 17:21:16 -0700 Subject: [PATCH 1/3] Re-watch channels when /sync refuses the reconnect catch-up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On reconnect the SDK catches up by calling /sync with the timestamp of the last event received before the disconnect. The server refuses the request when the gap is too large — code 4 / HTTP 400, "Too many events to sync, please use a more recent last_sync_at parameter" — which the ~1000-event limit reaches long before the 30-day bound the guard above it checks. That cap counts events, not messages, across every cid passed in the one call, so a single message can contribute a message.new plus a message.read per member. A player who leaves the app backgrounded on a busy channel and returns hours later hits it every time. The failure had no handler. FetchAndProcessEventsSinceLastReceivedEvent is called fire-and-forget through LogIfFailed, so the exception reached the logger and nothing else: the watched channels kept the state they had before the disconnect, missing every message since, until something unrelated happened to re-fetch them. And _disconnectionLastEventReceivedAt stayed stale, so the next reconnect failed exactly the same way. In one of our production titles this is 6k+ such warnings across 4.2k users in 30 days; for a live room or an open feed it is a silent correctness gap, not just noise. Now the low-level client drops the stale sync point and rethrows, and RestoreStateLostDuringDisconnect catches the input error and re-watches every watched channel — the same full state fetch the initial watch does, which is the only way to recover once the events are past replay. Each channel is attempted independently: this runs after the stale sync point has been dropped, so it is the only recovery this reconnect gets, and a single failure escaping the loop would leave every remaining channel silently stale for the rest of the session. Failures are expected here, not exotic — a channel torn down while offline returns 403 on every read, and a long watched list can trip a 429 part-way. Known limitation, flagged in a comment: GetOrCreateChannelWithIdAsync is get-OR-create, so re-watching a channel that was hard-deleted while offline recreates it server-side as an empty channel. Fixing that properly means consulting SyncResponse.InaccessibleCids — already returned by /sync and currently ignored — to skip channels the server says are gone, rather than discovering it one 403 at a time. Happy to take that on in this PR if you would rather not merge the get-or-create behavior. --- Assets/Plugins/StreamChat/Changelog.txt | 1 + .../StreamChatLowLevelClient.cs | 26 ++++++-- .../StreamChat/Core/StreamChatClient.cs | 60 ++++++++++++++++++- 3 files changed, 79 insertions(+), 8 deletions(-) diff --git a/Assets/Plugins/StreamChat/Changelog.txt b/Assets/Plugins/StreamChat/Changelog.txt index 4157fdea..c6c015a6 100644 --- a/Assets/Plugins/StreamChat/Changelog.txt +++ b/Assets/Plugins/StreamChat/Changelog.txt @@ -19,6 +19,7 @@ Features: Fixes: +* Fix watched channels silently staying stale after a reconnect whose /sync catch-up the server refuses as too large ("Too many events to sync", HTTP 400 / code 4 - reachable after a long disconnect on a busy channel, since the ~1000-event limit counts events across every cid in the call). The failure previously reached only the fire-and-forget logger, and the stale sync point was kept so every subsequent reconnect failed identically. The SDK now drops the stale sync point and re-watches every watched channel instead, which is the same full state fetch the initial watch performs. Each channel is restored independently so one failure (e.g. a 403 on a channel deleted while offline) does not abandon the rest. * Fix the 30-day staleness guard on the reconnect /sync catch-up computing its age backwards (lastEventReceivedAt - now, which is negative for any past timestamp), so the guard never fired and /sync was called even with a LastSyncAt the server rejects. Also removes a dead local that was almost certainly the intended operand. * TaskUtils.LogIfFailed now logs connectivity/transport failures as warnings instead of errors/exceptions. The SDK fire-and-forgets its connect, reconnect, and state-restore operations through LogIfFailed; when the device is offline these fail with HttpRequestException / WebException / SocketException / IOException / TimeoutException, which the reconnect flow recovers from - so surfacing them at error severity flooded crash/error reporting (Sentry, Bugsnag, etc.) with handled, non-actionable noise. Genuine (non-connectivity) failures still log as exceptions. Complements the connection-attempt-timeout fix from PR #213. diff --git a/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs b/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs index 7000c00b..2ebc00d9 100644 --- a/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs +++ b/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs @@ -14,6 +14,7 @@ using StreamChat.Core.LowLevelClient.API.Internal; using StreamChat.Core.LowLevelClient.Events; using StreamChat.Core.LowLevelClient.Models; +using StreamChat.Core.LowLevelClient.Responses; using StreamChat.Core.Web; using StreamChat.Libs; using StreamChat.Libs.AppInfo; @@ -456,12 +457,27 @@ public async Task FetchAndProcessEventsSinceLastReceivedEvent(IEnumerable 1000 events - var response = await ChannelApi.SyncAsync(new SyncRequest + SyncResponse response; + try { - ChannelCids = channelCids.ToList(), - LastSyncAt = lastEventReceivedAt, - Watch = true, - }); + response = await ChannelApi.SyncAsync(new SyncRequest + { + ChannelCids = channelCids.ToList(), + LastSyncAt = lastEventReceivedAt, + Watch = true, + }); + } + catch (StreamApiException e) when (e.IsInputError()) + { + // The gap is too large for /sync — more than the ~1000 events the server + // will replay (the StreamTodo above), which a busy channel reaches long before the + // 30-day bound checked above. Drop the sync point so the next reconnect starts from + // a fresh one instead of failing the same way forever, and let the caller re-hydrate + // the channels: the missed events are gone either way, and only a full state fetch + // brings the watched channels back up to date. + _disconnectionLastEventReceivedAt = null; + throw; + } if (response.Events.Count == 0) { diff --git a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs index d05928fd..2accdd7c 100644 --- a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs +++ b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs @@ -1141,14 +1141,68 @@ private void OnConnected(HealthCheckEventInternalDTO dto) RestoreStateLostDuringDisconnect().LogIfFailed(); } - private Task RestoreStateLostDuringDisconnect() + private async Task RestoreStateLostDuringDisconnect() { if (!WatchedChannels.Any()) { - return Task.CompletedTask; + return; + } + + try + { + await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent( + WatchedChannels.Select(c => c.Cid)); + } + catch (StreamApiException e) when (e.IsInputError()) + { + // /sync refused the catch-up because too much accumulated while we were + // disconnected (see FetchAndProcessEventsSinceLastReceivedEvent). Without this the + // exception only reached the fire-and-forget logger at the call site: the watched + // channels silently stayed as they were before the disconnect, missing every + // message since, until something else happened to re-fetch them. Re-watch instead — + // it is the same full state fetch the initial watch does. + _logs.Warning("The /sync catch-up was refused as too large; re-watching " + + $"{WatchedChannels.Count} channel(s) to restore their state instead."); + await RewatchChannelsAsync(); + } + } + + // Full state re-fetch of every watched channel, used when /sync cannot bridge the + // disconnect gap. Snapshotted because each re-watch writes the cache the list is built from. + // + // Every channel is attempted independently. This runs AFTER the stale sync point has been + // dropped, so it is the only recovery this reconnect gets and there is no later retry: a + // single failure escaping the loop would leave every remaining channel silently stale for the + // rest of the session. Failures are expected here, not exotic — a channel torn down while we + // were offline returns 403 on every read, and a long watched list can trip a 429 part-way. + // Log each one and keep going so the channels that CAN be restored are. + // + // Known limitation: GetOrCreateChannelWithIdAsync is get-OR-CREATE, so re-watching a channel + // that was hard-deleted while we were offline recreates it server-side as an empty channel. + // Fixing that properly means consulting SyncResponse.InaccessibleCids (already returned by + // /sync and currently ignored) to skip channels the server says are gone, rather than + // discovering it one 403 at a time. + private async Task RewatchChannelsAsync() + { + int failed = 0; + foreach (IStreamChannel channel in WatchedChannels.ToList()) + { + try + { + await GetOrCreateChannelWithIdAsync(channel.Type, channel.Id); + } + catch (Exception e) + { + failed++; + _logs.Warning($"Re-watch failed for channel {channel.Type}:{channel.Id}; " + + $"its local state stays as it was before the disconnect. {e.Message}"); + } } - return LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent(WatchedChannels.Select(c => c.Cid)); + if (failed > 0) + { + _logs.Warning($"Re-watch completed with {failed} channel(s) unrestored."); + } } private void OnDisconnected() => Disconnected?.Invoke(); From e55fee65b21747762a5cea36eec9c4636414253f Mon Sep 17 00:00:00 2001 From: Harlan Crystal Date: Mon, 17 Aug 2026 12:09:37 -0700 Subject: [PATCH 2/3] Signal ChannelsRewatched when a re-watch replaces channel state The re-watch recovery replaces a channel's messages wholesale and raises no per-message events for the window it replaced, so a consumer that rebuilds its UI from message events alone keeps rendering the rows it had before the disconnect - the recovery restores local state but nothing tells the UI to read it. Raise IStreamChatClient.ChannelsRewatched with the re-watched channels so consumers can rebuild from IStreamChannel.Messages. Raised even when some channels failed to restore: the ones that succeeded did have their state replaced, and rebuilding from a failed channel's unchanged list is harmless. --- Assets/Plugins/StreamChat/Changelog.txt | 1 + .../StreamChat/Core/IStreamChatClient.cs | 8 ++++++++ .../Plugins/StreamChat/Core/StreamChatClient.cs | 17 ++++++++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/Assets/Plugins/StreamChat/Changelog.txt b/Assets/Plugins/StreamChat/Changelog.txt index c6c015a6..7e2cd894 100644 --- a/Assets/Plugins/StreamChat/Changelog.txt +++ b/Assets/Plugins/StreamChat/Changelog.txt @@ -16,6 +16,7 @@ Features: * Add a public StreamApiException constructor (statusCode, code, errorMessage, moreInfo, duration, exceptionFields). StreamApiException is a public, catch-and-branch type (via the StreamApiExceptionExtensions.Is* helpers), but until now it could only be constructed inside the SDK from the internal APIErrorInternalDTO, so integrators could not build one to unit-test their own error handling (e.g. simulating a 403 / code 70 "no access to channels" response). The new constructor maps directly to the type's public properties and keeps APIErrorInternalDTO internal. * Add IStreamClientConfig.OptimisticMessageInsert (default true). When true (the existing behavior), a message you send is inserted into the local channel state and raised via IStreamChannel.MessageReceived immediately, before the server's message.new echo arrives. Set it to false to skip the optimistic local insert and wait for the server echo instead, so every participant - including the sender - observes messages in the same server-defined order. Useful when consistent cross-client ordering matters more than instant local feedback (e.g. a shared, broadcast-ordered feed). +* Add IStreamChatClient.ChannelsRewatched, raised with the channels whose local state a reconnect re-watch just replaced. A re-watch raises no per-message events for the window it replaced, so handle this to rebuild your UI from IStreamChannel.Messages. Fixes: diff --git a/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs b/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs index e72e016b..ccedb651 100644 --- a/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs +++ b/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs @@ -70,6 +70,14 @@ public interface IStreamChatClient : IDisposable, IStreamChatClientEventsListene /// event ChannelMemberRemovedHandler RemovedFromChannelAsMember; + /// + /// Channels whose local state was just replaced wholesale by a reconnect re-watch, which + /// raises no per-message events for the window it replaced. A consumer rendering one of + /// these channels must rebuild from rather than wait + /// for message events, otherwise it keeps showing the rows it had before the disconnect. + /// + event ChannelsRewatchedHandler ChannelsRewatched; + /// /// Raised when an becomes available locally. Use this to bind /// per-thread UI and to subscribe to the thread's own events such as diff --git a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs index 2accdd7c..cbd4e9a1 100644 --- a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs +++ b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs @@ -67,6 +67,11 @@ namespace StreamChat.Core /// public delegate void ChannelMemberRemovedHandler(IStreamChannel channel, IStreamChannelMember member); + /// + /// Channels whose local state was replaced wholesale by a reconnect re-watch handler + /// + public delegate void ChannelsRewatchedHandler(IReadOnlyList channels); + /// public sealed class StreamChatClient : IStreamChatClient { @@ -87,6 +92,8 @@ public sealed class StreamChatClient : IStreamChatClient public event ChannelMemberAddedHandler AddedToChannelAsMember; public event ChannelMemberRemovedHandler RemovedFromChannelAsMember; + public event ChannelsRewatchedHandler ChannelsRewatched; + public event StreamThreadChangeHandler ThreadTracked; public event StreamThreadChangeHandler ThreadUntracked; @@ -1184,8 +1191,9 @@ await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent( // discovering it one 403 at a time. private async Task RewatchChannelsAsync() { + List channels = WatchedChannels.ToList(); int failed = 0; - foreach (IStreamChannel channel in WatchedChannels.ToList()) + foreach (IStreamChannel channel in channels) { try { @@ -1203,6 +1211,13 @@ private async Task RewatchChannelsAsync() { _logs.Warning($"Re-watch completed with {failed} channel(s) unrestored."); } + + // A re-watch replaces the channel's messages wholesale, without raising the per-message + // events a consumer would normally rebuild from, so without this signal an open UI keeps + // rendering the rows it had before the disconnect. Raised even when some channels failed: + // the ones that succeeded did have their state replaced, and a consumer rebuilding from a + // failed channel's (unchanged) list is harmless. + ChannelsRewatched?.Invoke(channels); } private void OnDisconnected() => Disconnected?.Invoke(); From 72e6fc74bd59ba4164e6e7b19466b866c6a9ab52 Mon Sep 17 00:00:00 2001 From: Harlan Crystal Date: Mon, 17 Aug 2026 12:21:14 -0700 Subject: [PATCH 3/3] Let channel types opt out of the long-gap /sync replay After a long outage the reconnect catch-up replays every missed event through the consumer's handlers one at a time. For a channel whose consumer only ever displays a bounded latest window, that replay ends at the same visible state a single re-watch request would have produced - so on a busy channel the resume spends its cost on work that is immediately discarded. SetRewatchOnReconnectChannelTypes lets the integrator name the channel types that should be restored by a bounded re-watch instead when the outage exceeds maxSyncReplayGap (default 60s). Types that are not listed keep the precise replay, and short outages always replay, where the backlog is small and the replay keeps consumers seamless. Unset by default, so behavior is unchanged. The outage is the age of the sync point - the last handled event - and not the time since the Disconnected transition. Detection can lag the real outage by its entire length: a mobile OS suspends the process while backgrounded, so the dead socket is only noticed on resume and a transition stamp would measure seconds for an hours-long background, never engaging this path on the platform that needs it. Health check events advance the watermark every ~30s while connected. The re-watch also no longer depends on the replay's outcome: a transient /sync failure is logged and the re-watch still runs, rather than leaving the opted-in channels with no recovery for the rest of the session. --- Assets/Plugins/StreamChat/Changelog.txt | 1 + .../StreamChat/Core/IStreamChatClient.cs | 18 +++ .../StreamChatLowLevelClient.cs | 16 +++ .../StreamChat/Core/StreamChatClient.cs | 107 ++++++++++++++++-- 4 files changed, 132 insertions(+), 10 deletions(-) diff --git a/Assets/Plugins/StreamChat/Changelog.txt b/Assets/Plugins/StreamChat/Changelog.txt index 7e2cd894..f06eb186 100644 --- a/Assets/Plugins/StreamChat/Changelog.txt +++ b/Assets/Plugins/StreamChat/Changelog.txt @@ -17,6 +17,7 @@ Features: * Add a public StreamApiException constructor (statusCode, code, errorMessage, moreInfo, duration, exceptionFields). StreamApiException is a public, catch-and-branch type (via the StreamApiExceptionExtensions.Is* helpers), but until now it could only be constructed inside the SDK from the internal APIErrorInternalDTO, so integrators could not build one to unit-test their own error handling (e.g. simulating a 403 / code 70 "no access to channels" response). The new constructor maps directly to the type's public properties and keeps APIErrorInternalDTO internal. * Add IStreamClientConfig.OptimisticMessageInsert (default true). When true (the existing behavior), a message you send is inserted into the local channel state and raised via IStreamChannel.MessageReceived immediately, before the server's message.new echo arrives. Set it to false to skip the optimistic local insert and wait for the server echo instead, so every participant - including the sender - observes messages in the same server-defined order. Useful when consistent cross-client ordering matters more than instant local feedback (e.g. a shared, broadcast-ordered feed). * Add IStreamChatClient.ChannelsRewatched, raised with the channels whose local state a reconnect re-watch just replaced. A re-watch raises no per-message events for the window it replaced, so handle this to rebuild your UI from IStreamChannel.Messages. +* Add IStreamChatClient.SetRewatchOnReconnectChannelTypes(channelTypes, maxSyncReplayGap). The listed channel types skip the event-by-event /sync replay when the outage exceeded maxSyncReplayGap (default 60s) and are restored by a bounded re-watch instead, reported via ChannelsRewatched. Unset by default, so every channel replays as before. Use it for channel types that only display a latest window, where replaying an hour of backlog event by event ends at the state one re-watch request would have produced. Fixes: diff --git a/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs b/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs index ccedb651..d0aa2d8f 100644 --- a/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs +++ b/Assets/Plugins/StreamChat/Core/IStreamChatClient.cs @@ -78,6 +78,24 @@ public interface IStreamChatClient : IDisposable, IStreamChatClientEventsListene /// event ChannelsRewatchedHandler ChannelsRewatched; + /// + /// Opt channel types out of the event-by-event /sync replay that follows a reconnect when + /// the outage was longer than (default 60 seconds). + /// Channels of these types are restored by a bounded re-watch instead - the same latest-page + /// state fetch the initial watch performs, one request per channel - and reported via + /// so consumers can rebuild any UI showing them. + /// + /// Use this for channel types whose consumers only ever display a bounded latest window + /// (livestream-style or announcement channels): replaying an hour's backlog event by event + /// costs one handler pass per event and ends with the same visible state the single + /// re-watch request would have produced. Leave types whose consumers need every individual + /// event (e.g. anything persisting full history locally) unlisted; they keep the replay. + /// + /// Replaces the previously configured set. Shorter outages always replay. + /// + void SetRewatchOnReconnectChannelTypes(IEnumerable channelTypes, + TimeSpan? maxSyncReplayGap = null); + /// /// Raised when an becomes available locally. Use this to bind /// per-thread UI and to subscribe to the thread's own events such as diff --git a/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs b/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs index 2ebc00d9..47d1132b 100644 --- a/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs +++ b/Assets/Plugins/StreamChat/Core/LowLevelClient/StreamChatLowLevelClient.cs @@ -619,6 +619,22 @@ internal async Task ConnectUserAsync(string apiKey, string u /// private DateTimeOffset? _disconnectionLastEventReceivedAt; + /// + /// Age of the reconnect sync point: how long ago the connection actually went quiet, which + /// is the gap the /sync replay would have to bridge. Health check events advance + /// every ~30s while connected, so this measure is immune + /// to LATE DETECTION of an outage - a mobile OS suspends the process while the app is + /// backgrounded and the dead socket is only noticed on resume, so any stamp taken at the + /// Disconnected transition would measure seconds for an hours-long background. Compares a + /// server-stamped event time against local Now, the same subtraction the 30-day guard in + /// makes; clock skew only nudges + /// borderline gaps between two correct recovery paths. + /// + internal TimeSpan? TimeSinceDisconnectSyncPoint + => _disconnectionLastEventReceivedAt.HasValue + ? _timeService.Now - _disconnectionLastEventReceivedAt.Value + : (TimeSpan?)null; + private async Task RefreshAuthTokenFromProvider() { #if STREAM_DEBUG_ENABLED diff --git a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs index cbd4e9a1..0d73b06b 100644 --- a/Assets/Plugins/StreamChat/Core/StreamChatClient.cs +++ b/Assets/Plugins/StreamChat/Core/StreamChatClient.cs @@ -972,6 +972,21 @@ internal Task RefreshChannelState(string cid) private readonly StreamPollsApi _pollsApi; private readonly List _watchedChannels = new List(); + // Channel types that skip the event-by-event /sync replay after a long disconnect and are + // restored by a bounded re-watch instead. Empty = replay everything, the original behavior. + // See SetRewatchOnReconnectChannelTypes. + private readonly HashSet _rewatchOnReconnectChannelTypes = new HashSet(); + + // Outage length past which the /sync replay stops being worth it for latest-window channel + // types: the replay hands every missed event to the (typically expensive) downstream + // handlers one at a time, while a re-watch fetches just the channel's latest page in one + // request. Short blips stay on the replay path, which is cheaper and seamless for consumers. + // The default comfortably exceeds a transient network blip while sitting far below the + // backlogs that degrade a resume (an app backgrounded on a busy channel for minutes to + // hours). + private static readonly TimeSpan DefaultMaxSyncReplayGap = TimeSpan.FromSeconds(60); + private TimeSpan _maxSyncReplayGap = DefaultMaxSyncReplayGap; + private TaskCompletionSource _connectUserTaskSource; private CancellationToken _connectUserCancellationToken; private CancellationTokenSource _connectUserCancellationTokenSource; @@ -1155,10 +1170,38 @@ private async Task RestoreStateLostDuringDisconnect() return; } + // The outage is the age of the sync point - the last handled event - and not the time + // since the Disconnected transition. Detection can lag the real outage by its entire + // length: a mobile OS suspends the process while the app is backgrounded, so the dead + // socket is only noticed on resume and a stamp taken at the transition would measure + // seconds for an hours-long background, never engaging the long-gap path below on + // exactly the platform that needs it. Health check events advance the watermark every + // ~30s while connected, so it tracks the real quiet period regardless of when the + // client noticed. + TimeSpan? outage = InternalLowLevelClient.TimeSinceDisconnectSyncPoint; + + // Past a long outage the opted-in channel types skip the replay: it hands every missed + // event to the consumer's handlers one at a time, while these channels only ever + // display their latest page - which a re-watch fetches in a single request. Types that + // are not opted in keep the precise replay. Short outages replay everything: the + // backlog is small and the replay keeps consumers seamless. + List channelsToRewatch = null; + IEnumerable channelsToReplay = WatchedChannels; + if (outage > _maxSyncReplayGap && _rewatchOnReconnectChannelTypes.Count > 0) + { + channelsToRewatch = WatchedChannels + .Where(c => _rewatchOnReconnectChannelTypes.Contains(c.Type)).ToList(); + channelsToReplay = WatchedChannels + .Where(c => !_rewatchOnReconnectChannelTypes.Contains(c.Type)); + } + try { - await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent( - WatchedChannels.Select(c => c.Cid)); + List replayCids = channelsToReplay.Select(c => c.Cid).ToList(); + if (replayCids.Count > 0) + { + await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent(replayCids); + } } catch (StreamApiException e) when (e.IsInputError()) { @@ -1167,18 +1210,63 @@ await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent( // exception only reached the fire-and-forget logger at the call site: the watched // channels silently stayed as they were before the disconnect, missing every // message since, until something else happened to re-fetch them. Re-watch instead — - // it is the same full state fetch the initial watch does. + // it is the same full state fetch the initial watch does. The refused replay left + // its channels with nothing, so every watched channel re-watches here, not just the + // opted-in set. _logs.Warning("The /sync catch-up was refused as too large; re-watching " + $"{WatchedChannels.Count} channel(s) to restore their state instead."); - await RewatchChannelsAsync(); + channelsToRewatch = WatchedChannels.ToList(); + } + catch (Exception e) + { + // The replay failed transiently (5xx, or the network dropping again mid-reconnect). + // The re-watch below does not depend on it, so letting this propagate - to nothing + // but the caller's fire-and-forget logger - would skip the re-watch and leave the + // opted-in channels with no recovery at all this reconnect. Log and continue; the + // replay channels keep their sync point, so the next reconnect retries them. + _logs.Warning($"The /sync catch-up failed ({e.Message}); continuing with the " + + "re-watch of the channels that opted out of the replay."); + } + + if (channelsToRewatch != null && channelsToRewatch.Count > 0) + { + await RewatchChannelsAsync(channelsToRewatch); + } + } + + /// + /// Opt channel types out of the event-by-event /sync replay that follows a reconnect when + /// the outage was longer than (default 60 seconds). + /// Channels of these types are restored by a bounded re-watch instead - the same latest-page + /// state fetch the initial watch performs, one request per channel - and reported via + /// so consumers can rebuild any UI showing them. + /// + /// Use this for channel types whose consumers only ever display a bounded latest window + /// (livestream-style or announcement channels): replaying an hour's backlog event by event + /// costs one handler pass per event and ends with the same visible state the single + /// re-watch request would have produced. Leave types whose consumers need every individual + /// event (e.g. anything persisting full history locally) unlisted; they keep the replay. + /// + /// Replaces the previously configured set. Shorter outages always replay. + /// + public void SetRewatchOnReconnectChannelTypes(IEnumerable channelTypes, + TimeSpan? maxSyncReplayGap = null) + { + _maxSyncReplayGap = maxSyncReplayGap ?? DefaultMaxSyncReplayGap; + _rewatchOnReconnectChannelTypes.Clear(); + foreach (ChannelType channelType in channelTypes) + { + _rewatchOnReconnectChannelTypes.Add(channelType); } } - // Full state re-fetch of every watched channel, used when /sync cannot bridge the - // disconnect gap. Snapshotted because each re-watch writes the cache the list is built from. + // Full state re-fetch of the given watched channels, used when /sync cannot - or should not + // - bridge the disconnect gap. Callers pass a snapshot because each re-watch writes the + // cache the WatchedChannels list is built from. // - // Every channel is attempted independently. This runs AFTER the stale sync point has been - // dropped, so it is the only recovery this reconnect gets and there is no later retry: a + // Every channel is attempted independently. On the /sync-refused path this runs AFTER the + // stale sync point has been dropped, so it is the only recovery this reconnect gets and + // there is no later retry: a // single failure escaping the loop would leave every remaining channel silently stale for the // rest of the session. Failures are expected here, not exotic — a channel torn down while we // were offline returns 403 on every read, and a long watched list can trip a 429 part-way. @@ -1189,9 +1277,8 @@ await LowLevelClient.FetchAndProcessEventsSinceLastReceivedEvent( // Fixing that properly means consulting SyncResponse.InaccessibleCids (already returned by // /sync and currently ignored) to skip channels the server says are gone, rather than // discovering it one 403 at a time. - private async Task RewatchChannelsAsync() + private async Task RewatchChannelsAsync(IReadOnlyList channels) { - List channels = WatchedChannels.ToList(); int failed = 0; foreach (IStreamChannel channel in channels) {