fix(server): back off relay client restarts - #8427
Conversation
When cloudflared exited, the supervisor spawned it again immediately. A bad tunnel token or a child that dies on start was a tight spawn loop. Wait 500ms, then 1s, 2s, up to 10s, before the next start. A new applyConfig still starts right away.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — This focused server-side fix limits repeated Cloudflare relay respawns with a tested exponential backoff, while preserving immediate explicit configuration reconciliation and guarding against stale restarts. Its production impact is confined to connector recovery after process exits, with no schema, deployment, or sensitive-package changes. You can add or adjust custom eligibility rules. Learn more. |
What Changed
The Cloudflare relay supervisor now waits before it respawns
cloudflared.If the child exits and that config is still desired, the next start waits 500ms, then 1s, 2s, up to 10s. The wait does not hold the reconcile lock, so a new
applyConfigstill starts right away and resets the backoff.Why
A bad tunnel token or a child that dies on start was a tight spawn loop. CPU plus process churn, with
TUNNEL_TOKENin the child env the whole time.Native telemetry already backs off the same way. The relay client did not.
Checklist
Note
Medium Risk
Changes connector supervision and timing for cloud tunnel availability; mis-tuned backoff or lock handling could delay recovery or allow brief spawn races, but scope is limited to managed relay restarts.
Overview
Adds exponential backoff when the Cloudflare relay (
cloudflared) supervisor respawns a child that exited while the same tunnel config is still desired. Delays start at 500ms, double per failure, and cap at 10s via exportedrelayClientRestartDelay.The supervisor releases the reconcile lock during the sleep, then re-acquires it to call reconcile only if nothing else is active and the desired config still matches.
applyConfigresets the attempt counter to 0, so an explicit config change can restart immediately instead of waiting through backoff.Tests cover the delay curve and use TestClock so the post-exit restart test asserts no second spawn until ~500ms has passed.
Reviewed by Cursor Bugbot for commit 90f9ea0. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add exponential backoff to relay client restarts in
ManagedEndpointRuntimeRelay client restarts now delay with exponential backoff starting at 500 ms, doubling per attempt, and capped at 10 seconds via
relayClientRestartDelay. A newrestartAttemptReftracks the attempt count;applyConfigresets it to 0 whenever a new config is applied. After the backoff sleep,superviseConnectorre-checks state before restarting so it only restarts if the connector is still desired and not already active. Behavioral Change: restarts are no longer immediate — there is now a delay of up to 10 seconds between process exit and respawn; existing callers expecting instant restarts will see delayed restarts.Macroscope summarized 90f9ea0.