Notifier spins without backoff when the Start context ends by deadline #1493

Summary

If the context passed to Client.Start ends by deadline rather than cancellation, the notifier's reconnect loop spins with no backoff and logs Error running listener (will attempt reconnect after backoff) hundreds of thousands of times per second, forever.

Where

internal/notifier/notifier.go (v0.47.0, and unchanged on master at 2f6b27e):

for attempt := 0; ; attempt++ {
    if err := n.listenAndWait(ctx); err != nil {
        if errors.Is(err, context.Canceled) {
            break
        }
        sleepDuration := serviceutil.ExponentialBackoff(attempt, serviceutil.MaxAttemptsBeforeResetDefault)
        n.Logger.ErrorContext(ctx, n.Name+": Error running listener (will attempt reconnect after backoff)", ...)
        ...
        serviceutil.CancellableSleep(ctx, sleepDuration)
    }
}

Once ctx is done with context.DeadlineExceeded, listenAndWait fails immediately, the context.Canceled check doesn't match, and CancellableSleep returns immediately because ctx is already done. The loop never exits and never sleeps.

Reproduction

1. Create a client against a healthy Postgres.
2. ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second), then client.Start(ctx).
3. Wait past the deadline.

We measured 379,487 Error running listener lines in the 2 seconds after the deadline, and 0 when the same context is cancelled instead. Left running, it wrote a ~476 GB log in about 6 hours before we found it.

Suggested fix

Exit the loop when the context is done for any reason, e.g. if ctx.Err() != nil { break } (or check context.DeadlineExceeded too). The other errors.Is(err, context.Canceled) exits in the notifier may deserve the same treatment.

Workaround

Start River on a context that can only end by cancellation, e.g. runCtx, cancel := context.WithCancel(context.WithoutCancel(ctx)); context.AfterFunc(ctx, cancel).