Repository navigation
Conversation
cloudflared is a second process in the container and cotel is the main one, so when cloudflared's edge discovery hit a resolver that was not up yet it logged "Initiating shutdown", exited for good, and `restart: unless-stopped` never fired. The container reported Up (healthy) with RestartCount 0 while the public ingest path was dark and every span exported to it was dropped. supervise-cloudflared.sh now owns the process and restarts it on every exit for as long as the container lives, waiting 1s and doubling to a 60s cap (COTEL_TUNNEL_RESTART_BACKOFF_MAX), resetting after a run that stayed up at least that long; a stop signal reaches the child it is waiting on, so a container stop is not held for a backoff. Making the tunnel's death fatal to the container instead was the alternative and is rejected deliberately: it couples a working data plane to an unreachable edge, and dropping LAN ingest on :4318 over a DNS hiccup is the worse of the two failures. /healthz gains a `tunnel` object - ok, ready_connections, error - read from cloudflared's own /ready route rather than from process liveness, because a cloudflared that is up with no registered edge connection drops traffic exactly like one that is gone. It is an added field: a dead tunnel changes neither `ok` nor the status code, so the container HEALTHCHECK and the deploy gate keep their meaning. The key is absent when no tunnel is configured, so a tokenless local or CI instance says nothing about a tunnel rather than reporting a dead one. Verified end to end against a real tunnel, in a container started with a refusing resolver: cloudflared exited on the unresolvable edge SRV record, was restarted on the 1/2/4/5s backoff, and registered four edge connections on its own once the resolver was fixed. /healthz went ok:false (metrics listener unreachable) to ok:false (ready_connections 0) to ok:true (ready_connections 4), answering HTTP 200 with ok:true throughout, and the token appeared nowhere in the container log. Co-Authored-By: Wayland <wayland@agents.flopbut.local> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 50 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
Closing unmerged: the scope behind this PR was cancelled by the board (the scheduled watcher was removed entirely; any external monitoring will be proposed and reviewed before implementation). Branch deleted; restorable from this PR page if ever needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
cloudflared is a second process in the container and cotel is the main one, so
restart: unless-stoppedwatches the wrong one. When the deploy host rebooted on 2026-10-14 and cloudflared's edge discovery hit a resolver that was not up yet, it loggedInitiating shutdown, exited for good, and the container went on reportingUp (healthy)withRestartCount0 for 11 h 39 min while every agent exporting to the public ingest URL lost its spans./healthzanswered200 ok:truethe whole time, and nothing in its body could have said otherwise.What changed
scripts/supervise-cloudflared.sh(new) owns cloudflared and restarts it on every exit for as long as the container lives: 1s, doubling to aCOTEL_TUNNEL_RESTART_BACKOFF_MAX(60s) cap, reset after a run that stayed up at least that long. A stop signal reaches whatever it is waiting on, so a container stop is not held for the length of a backoff. The entrypoint starts it instead of starting cloudflared directly; the token still never reaches cloudflared's environment.:4318over a DNS hiccup is the worse of the two failures./healthzgains atunnelobject (ok,ready_connections,error), read from cloudflared's own/readyroute rather than from process liveness — a cloudflared that is up with no registered edge connection drops traffic exactly like one that is gone. It is an added field: a dead tunnel changes neitheroknor the status code, so the containerHEALTHCHECK, the deploy gate and~/ops/cotel-healthz.sh's 503/stale/empty classifications are untouched. The key is absent when no tunnel is configured, so a tokenless local or CI instance says nothing about a tunnel rather than reporting a dead one./healthzcontract + two env vars),docs/operations/cloudflare-tunnel-remote.md(a "Keeping the tunnel up" section with the failure and the rejected alternative),docs/operations/health-probe.md(why the LAN probe reports the field but does not yet classify it), CHANGELOG.How it was verified
End to end against a real throwaway Cloudflare tunnel (created in the isolated
rzegaccount for this test and deleted afterwards), in a container built from this branch and started with a refusing resolver:--dns 127.0.0.1ERR Initiating shutdown error="Couldn't resolve SRV record … region1.v2.argotunnel.com … connection refused"— the production shapecloudflared exited with status 1 after 1s; restarting in 1s, then2s,4s,5s(cap)/healthz, tunnel down{"ok":true,…,"tunnel":{"ok":false,"ready_connections":null,"error":"cloudflared metrics listener 127.0.0.1:2000 unreachable: … connection refused"}}/healthz, cloudflared up but unconnected"tunnel":{"ok":false,"ready_connections":0,"error":"cloudflared has no registered edge connection"}INF Registered tunnel connection … location=waw06 protocol=quic×4, with no intervention/healthzafter recovery"tunnel":{"ok":true,"ready_connections":4}— HTTP 200 andok:truethroughout,RestartCount0, containerhealthydocker stopdocker logsAutomated:
go vet+go test ./internal/tunnel/ ./internal/dashboard/green (ingolang:1.24-bookworm, no Go toolchain on this host),bash scripts/supervise-cloudflared_test.sh17/17 against a stub cloudflared (now wired into CI's Build & Test job),npm run buildindocs/green. The smoke job gains an assertion that a tokenless/healthzhas notunnelkey.Deploying this
Merging to
maindeploys and recreates the container, and the tunnel token exists only in the GitHub secret and in the live container's environment — so this must go out through the Deploy workflow (gh workflow run deploy.yml), which substitutes the secret. A hand-rundocker compose up -don the host would blank the token and drop the public path. After the deploy, check the fourRegistered tunnel connectionlines,tunnel.ok: trueon/healthz, and a 401 fromhttps://otlp.aignite.pl/v1/traces.🤖 Generated with Claude Code