Skip to content

fix(server): back off relay client restarts - #8427

Open
Adolanium wants to merge 1 commit into
pingdotgg:mainfrom
Adolanium:fix/relay-client-restart-backoff
Open

fix(server): back off relay client restarts#8427
Adolanium wants to merge 1 commit into
pingdotgg:mainfrom
Adolanium:fix/relay-client-restart-backoff

Conversation

@Adolanium

@Adolanium Adolanium commented Aug 27, 2026

Copy link
Copy Markdown

What Changed

The Cloudflare relay supervisor now waits before it respawns cloudflared.

If the child exits and that config is still desired, the next start waits 500ms, then 1s, 2s, up to 10s. The wait does not hold the reconcile lock, so a new applyConfig still starts right away and resets the backoff.

Why

A bad tunnel token or a child that dies on start was a tight spawn loop. CPU plus process churn, with TUNNEL_TOKEN in the child env the whole time.

Native telemetry already backs off the same way. The relay client did not.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • I included before/after screenshots for any UI changes (N/A)
  • I included a video for animation/interaction changes (N/A)

Note

Medium Risk
Changes connector supervision and timing for cloud tunnel availability; mis-tuned backoff or lock handling could delay recovery or allow brief spawn races, but scope is limited to managed relay restarts.

Overview
Adds exponential backoff when the Cloudflare relay (cloudflared) supervisor respawns a child that exited while the same tunnel config is still desired. Delays start at 500ms, double per failure, and cap at 10s via exported relayClientRestartDelay.

The supervisor releases the reconcile lock during the sleep, then re-acquires it to call reconcile only if nothing else is active and the desired config still matches. applyConfig resets the attempt counter to 0, so an explicit config change can restart immediately instead of waiting through backoff.

Tests cover the delay curve and use TestClock so the post-exit restart test asserts no second spawn until ~500ms has passed.

Reviewed by Cursor Bugbot for commit 90f9ea0. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add exponential backoff to relay client restarts in ManagedEndpointRuntime

Relay client restarts now delay with exponential backoff starting at 500 ms, doubling per attempt, and capped at 10 seconds via relayClientRestartDelay. A new restartAttemptRef tracks the attempt count; applyConfig resets it to 0 whenever a new config is applied. After the backoff sleep, superviseConnector re-checks state before restarting so it only restarts if the connector is still desired and not already active. Behavioral Change: restarts are no longer immediate — there is now a delay of up to 10 seconds between process exit and respawn; existing callers expecting instant restarts will see delayed restarts.

Macroscope summarized 90f9ea0.

When cloudflared exited, the supervisor spawned it again immediately. A bad tunnel token or a child that dies on start was a tight spawn loop.

Wait 500ms, then 1s, 2s, up to 10s, before the next start. A new applyConfig still starts right away.
@coderabbitai

coderabbitai Bot commented Aug 27, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 11a3d583-7068-4e64-b39e-0603a6379e02

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Aug 27, 2026
@macroscopeapp

macroscopeapp Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 90f9ea0

Macroscope's review found this PR approvable — This focused server-side fix limits repeated Cloudflare relay respawns with a tested exponential backoff, while preserving immediate explicit configuration reconciliation and guarding against stale restarts. Its production impact is confined to connector recovery after process exits, with no schema, deployment, or sensitive-package changes.

You can add or adjust custom eligibility rules. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant