diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cc3f7a2..50efddc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,6 +36,12 @@ jobs: - name: go test run: CGO_ENABLED=1 go test ./... + # The container's cloudflared supervisor, against a stub cloudflared: it + # keeps the tunnel alive and is in no Go package, so nothing above covers + # it. + - name: cloudflared supervisor tests + run: bash scripts/supervise-cloudflared_test.sh + smoke: name: Smoke test runs-on: ubuntu-latest @@ -71,6 +77,15 @@ jobs: echo "healthz: $RESP" echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d['spans'] == 3, f'expected 3 spans, got {d[\"spans\"]}'" + # This compose run has no tunnel token, so /healthz must say nothing about + # a tunnel. A "tunnel" key here would mean a tokenless deployment reports + # a dead tunnel — every local and CI instance red for a tunnel nobody + # asked for. + - name: Verify healthz omits the tunnel field without a token + run: | + curl -sf http://localhost:8080/healthz \ + | python3 -c "import sys,json; d=json.load(sys.stdin); assert 'tunnel' not in d, f'unexpected tunnel field: {d}'" + - name: Dashboard health check run: curl -sf http://localhost:8080/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e05865..0713f7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `/healthz` and `GET /api/v1/health` report `last_ingest_at` and `newest_span_age_seconds`. A cotel that is up but receiving nothing used to be indistinguishable from a healthy idle one — the dashboard answered `{"ok":true,"spans":0}` for six days of total ingest loss — and the age was not derivable from any other field, so no external poller could have caught it. The age is measured from the span's `ingested_at`, not its `start_time`: replaying an archive carries the original ingest time, so a restore cannot make a dark instance look alive, and a span arriving late with an old `start_time` still counts as fresh traffic. An empty database reports `null` for both rather than `0`, because "nothing ever arrived" is not "something arrived just now". Staleness deliberately does not change the status code on either endpoint — `/healthz` drives the container `HEALTHCHECK`, and the threshold at which silence becomes an alert belongs to whoever polls, not to the container. Both additions are additive; `ok`, `spans` and `span_count` are unchanged ### Fixed +- A dead Cloudflare tunnel no longer stays dead, and no longer hides behind a healthy container. cloudflared is a second process in the container and cotel is the main one, so `restart: unless-stopped` watches the wrong one: when the deploy host rebooted on 2026-10-14 and cloudflared's edge discovery hit a resolver that was not up yet (`lookup _v2-origintunneld._tcp.argotunnel.com on 127.0.0.11:53: no such host`), it logged `Initiating shutdown`, exited for good, and the container went on reporting `Up (healthy)` with `RestartCount` 0 for 11 h 39 min while every agent exporting to the public ingest URL got a Cloudflare error and lost its spans. The failure is not in cloudflared's own retry loop - that covers connection and protocol errors, not a start that cannot find the edge at all. `scripts/supervise-cloudflared.sh` now owns the process and restarts it on every exit for as long as the container lives, waiting 1s and doubling to a `COTEL_TUNNEL_RESTART_BACKOFF_MAX` (60s) cap, resetting the delay after a cloudflared that stayed up at least that long; a stop signal reaches whatever it is waiting on, so a container stop is not held for the length of a backoff. Making the tunnel's death fatal to the container instead - and letting Docker's restart policy handle it - was rejected deliberately: it would couple a working data plane to an unreachable edge, and dropping LAN ingest on `:4318` over a DNS hiccup is the worse of the two failures. Verified against a real tunnel in a container started with an unreachable resolver: cloudflared failed edge discovery and exited, was restarted on the backoff, and registered four edge connections on its own once the resolver was fixed, with no intervention. +- `/healthz` reports the tunnel it is supposed to be running. The endpoint answered `200 ok:true` throughout that outage, and nothing in its body could have said otherwise, so the only signal was the 6-hour ingest-staleness threshold - which a naturally quiet instance would never have crossed. A deployment with a tunnel now carries a `tunnel` object (`ok`, `ready_connections`, `error`), read from cloudflared's own `/ready` route rather than from process liveness, because a cloudflared that is up with no registered edge connection drops traffic exactly like one that is gone. It is an added field: `ok`, `spans` and the two freshness fields keep their meanings, and a dead tunnel deliberately changes neither `ok` nor the status code - this endpoint drives the container `HEALTHCHECK` and the deploy gate, and failing it would turn a dead public path into a dead container. The key is absent entirely when no tunnel is configured, so a tokenless local or CI instance says nothing about a tunnel rather than reporting a dead one; the smoke job asserts that. The address of cloudflared's metrics listener is `COTEL_TUNNEL_METRICS_ADDR` (`127.0.0.1:2000`), and the entrypoint passes it to cotel only on the paths where it actually started a tunnel - The Cloudflare tunnel token is no longer printed into the container log. `cloudflared` logs at INFO the value of every environment variable whose whole `KEY=VALUE` string merely *contains* `TUNNEL_` - matched against the pair, so a hit can come from the name or from the value - and redacts only the two variable names it declares itself, `TUNNEL_TOKEN` and `TUNNEL_CRED_CONTENTS`. Our own `CLOUDFLARE_TUNNEL_TOKEN` therefore matched the filter, missed the redaction list, and landed in `docker logs` in full on every start. That it leaked at all was luck rather than design: the dump splits `KEY=VALUE` on *every* `=` and drops the pair unless exactly two fields come back, so a token whose base64 carries `=` padding is silently skipped while one whose length happens to be a multiple of four is printed. The entrypoint now copies the token into a shell variable, `unset`s the exported one, and passes the value as `--token`, which cloudflared does redact (`Settings: map[token:*****]`). The reach was wider than the host: `scripts/wait-for-healthy.sh` copies container logs into the CI run log on a failed deploy, and the only thing standing between the token and a public job log was GitHub's literal-match secret filter. Upgrading `cloudflared` would **not** have fixed this - the dump is unchanged between 2024.11.1 and 2026.9.3, both verified by running the binaries against a synthetic padding-free token - so the pinned version stays where it is. `docs/operations/cloudflare-tunnel-remote.md` now tabulates what token mode does and does not hide: the log is clean, while `docker inspect`, `/proc/1/environ` (an environment block is a snapshot taken at `exec`, so a later `unset` cannot clear it) and `ps` inside the container all still carry the value - each of which already requires host or container access - A schema apply that corrupts an ART index now fails the start instead of serving traffic. After `storage.Open`, cotel runs the same `CHECKPOINT` used on shutdown; a hard failure exits 3 and writes `.checkpoint-failed` before the gates open, so the deploy health gate catches it. A timeout stays benign and the process continues. Measured on a 35 000-span file: the extra start checkpoint is <1 ms when schema is skipped, 8 ms immediately after a schema re-apply that rebuilds the four indexes - `/healthz` stops claiming `ok` when its query failed. The span count was read with a discarded error, so an unreadable database answered `{"ok":true,"spans":0}` — the same body as a brand-new empty one, which made `ok` very nearly a constant and no signal at all. A failed query now answers `503` with `{"ok":false}`, matching the pre-storage readiness gate on the same endpoint, so the container probe cannot report a dead database as healthy. Staleness is still not a failure and still answers `200` diff --git a/Dockerfile b/Dockerfile index 9600665..c24e34f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -54,6 +54,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ COPY --from=builder /cotel /usr/local/bin/cotel COPY --from=cloudflared-dl /usr/local/bin/cloudflared /usr/local/bin/cloudflared COPY scripts/entrypoint.sh /usr/local/bin/entrypoint.sh +COPY scripts/supervise-cloudflared.sh /usr/local/bin/supervise-cloudflared.sh # Data directory — mount a named volume here. VOLUME /data diff --git a/README.md b/README.md index 7733c2b..769503b 100644 --- a/README.md +++ b/README.md @@ -304,6 +304,7 @@ docker inspect --format '{{.State.Health.Status}}' cotel # starting → health | `spans` | Rows in `spans` | | `last_ingest_at` | When the newest span was **accepted** (RFC 3339), or `null` if nothing was ever ingested | | `newest_span_age_seconds` | Seconds since `last_ingest_at`, or `null` if nothing was ever ingested | +| `tunnel` | **Absent** unless this deployment runs a Cloudflare tunnel. Then an object: `ok` (is the public path up), `ready_connections` (edge connections cloudflared has registered, `null` if it could not be asked at all) and `error` (why not, when `ok` is false) | The age is measured from `ingested_at`, not `start_time`: importing an archive replays the original ingest time, so a restore cannot make a dark instance look @@ -321,6 +322,23 @@ report a dead database as healthy. The same two fields are on `GET /api/v1/health` — see [docs/operations/api-reference.md](docs/operations/api-reference.md). +The `tunnel` object exists because a dead tunnel looks like nothing from here. +cloudflared is a second process in the container, cotel is the main one, so +cloudflared can exit and leave the container `(healthy)` with `RestartCount` 0 +while every client exporting to the public ingest URL gets a Cloudflare error +and loses its spans. The state is read from cloudflared's own metrics listener +(`/ready`), not from process liveness: a cloudflared that is up but has +registered no edge connection drops traffic exactly like one that is gone. +`ok: true` requires at least one registered connection. + +Like staleness, a dead tunnel **does not change the status code** and does not +make `ok` false. LAN ingest on `:4318` keeps working while the public path is +down, and this endpoint drives the container `HEALTHCHECK` and the deploy gate — +taking the container down over an unreachable edge would turn a partial outage +into a total one. The field is the signal; the threshold belongs to whoever +polls. Nothing is reported when no tunnel is configured, so a local or CI +instance says nothing about a tunnel rather than claiming a dead one. + The retention and snapshot workers' health is deliberately **not** here. This endpoint is what `cotel --healthcheck`, the Docker `HEALTHCHECK` and the deploy's `wait-for-healthy.sh` read, so a failing backup reported through `ok` would mark @@ -544,6 +562,8 @@ from the network at startup instead of bundling it). | `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/snapshots`. Note that `docker volume prune` on a stopped deploy deletes it - the volume counts as in use only while the container exists. | | `COTEL_WAL_AUTOCHECKPOINT` | `4MB` | DuckDB `checkpoint_threshold`: the write-ahead log is folded into the main file once it grows past this size. Lower values bound how much WAL an ungraceful kill leaves to replay on the next open; higher values checkpoint less often during ingest. DuckDB's own default is `16MB`. | | `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel | +| `COTEL_TUNNEL_METRICS_ADDR` | `127.0.0.1:2000` | Where cloudflared's metrics listener binds inside the container (`--metrics`). Its `/ready` route is what the `tunnel` field on `/healthz` reports. The entrypoint passes it on only when it actually started a tunnel, so setting it on a tokenless deployment does not make `/healthz` report a tunnel that was never meant to run | +| `COTEL_TUNNEL_RESTART_BACKOFF_MAX` | `60` | Cap in seconds on the wait between cloudflared restarts. The supervisor restarts it for as long as the container lives, doubling the delay from 1s up to this cap; a cloudflared that stayed up at least this long resets the delay | | `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) | | `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. | | `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). | diff --git a/cmd/cotel/main.go b/cmd/cotel/main.go index 9d89363..af53a13 100644 --- a/cmd/cotel/main.go +++ b/cmd/cotel/main.go @@ -25,6 +25,7 @@ import ( "github.com/Flopsstuff/cotel/internal/importpkg" "github.com/Flopsstuff/cotel/internal/ingest" "github.com/Flopsstuff/cotel/internal/storage" + "github.com/Flopsstuff/cotel/internal/tunnel" ) func main() { @@ -127,7 +128,13 @@ func main() { dashMux.Handle("/api/v1/export", auth.Middleware(db, export.NewHandler(db))) dashMux.Handle("/api/v1/import", auth.Middleware(db, importpkg.NewHandler(db))) dashMux.Handle("/api/v1/", apiHandler) - dashMux.Handle("/", dashboard.New(ro)) + + dashHandler := dashboard.New(ro) + if prober := tunnel.FromEnv(); prober != nil { + dashHandler = dashHandler.SetTunnelProber(prober) + log.Printf("tunnel health on /healthz reads cloudflared metrics at %s", prober.Addr()) + } + dashMux.Handle("/", dashHandler) // Open the gates: from here the ports serve live traffic instead of 503. srv.ingest.set(ingestMux) diff --git a/docs/operations/cloudflare-tunnel-remote.md b/docs/operations/cloudflare-tunnel-remote.md index 16c5bba..3d3caae 100644 --- a/docs/operations/cloudflare-tunnel-remote.md +++ b/docs/operations/cloudflare-tunnel-remote.md @@ -72,7 +72,7 @@ docker run -d \ docker compose up -d ``` -The entrypoint detects `CLOUDFLARE_TUNNEL_TOKEN`, removes it from the environment it passes on, and runs `cloudflared tunnel run --token …` in the background. +The entrypoint detects `CLOUDFLARE_TUNNEL_TOKEN`, removes it from the environment it passes on, and starts `scripts/supervise-cloudflared.sh` in the background, which runs `cloudflared tunnel run --token …` and restarts it for as long as the container lives (see [Keeping the tunnel up](#keeping-the-tunnel-up)). ## Where the token is visible @@ -82,7 +82,7 @@ The token is the authority to serve your public hostnames, so it is worth knowin |---|---| | `docker compose logs cotel` / `docker logs` | No - the entrypoint keeps the variable out of cloudflared's environment, and cloudflared redacts the `--token` flag to `token:*****` | | `docker inspect` / `docker compose config` | **Yes** - it is container configuration; whoever can run these can read it | -| `/proc/1/environ` inside the container | **Yes** - a process's environment block is a snapshot taken at `exec`, so unsetting the variable afterwards does not clear it | +| `/proc/1/environ` inside the container, and the same file for the supervisor | **Yes** - a process's environment block is a snapshot taken at `exec`, so unsetting the variable afterwards does not clear it. The entrypoint hands the token to `supervise-cloudflared.sh` through its environment, so that process has a copy too; cloudflared itself does not, which is the one that matters for the log | | `ps` inside the container | **Yes** - it is an argument to `cloudflared` | The log line matters separately from the rest: the deploy health gate (`scripts/wait-for-healthy.sh`) copies container logs into the CI run log on a failed deploy, which is a far wider audience than "someone with a shell on the host". The other three rows all require host or container access, which already implies access to the token by other means. @@ -105,12 +105,56 @@ A few things are worth knowing when reading the startup log: Newer cloudflared can read the token from a file (`--token-file`) instead of an argument, which would flip the last row of the table above to "No". cotel does not use it, because that row is the only one it changes: whoever can run `ps` inside the container can read `/proc/1/environ` too, so the same person reaches the same value by a path `--token-file` does not touch. It would need a tmpfs mount and a secret on disk to close nothing. Getting the token out of the container altogether means [local-config mode](./cloudflare-tunnel-local.md), where the credentials are a host-mounted file. +## Keeping the tunnel up + +cloudflared does not run as the container's main process — cotel does — so for +a long time a dead tunnel was invisible and permanent. It cost 11 h 39 min of +telemetry on 2026-10-14: the host rebooted, cloudflared started before the +host's resolver was usable, and edge discovery failed on the one DNS query it +cannot retry past: + +``` +ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed + error="lookup _v2-origintunneld._tcp.argotunnel.com on 127.0.0.11:53: no such host" +INF Tunnel server stopped +ERR Initiating shutdown +``` + +`restart: unless-stopped` never fired, because the process that died was not the +one Docker watches. The container stayed `Up (healthy)` with `RestartCount` 0, +the LAN ingest port kept answering, and every agent exporting to the public URL +got Cloudflare 1033 until someone restarted the container by hand. + +Two things close that, both in this repo: + +- **`scripts/supervise-cloudflared.sh`** runs cloudflared and restarts it on + every exit, forever, waiting 1s and doubling up to `60s` + (`COTEL_TUNNEL_RESTART_BACKOFF_MAX`); a cloudflared that stayed up at least + that long resets the delay. The alternative — making the tunnel's death fatal + to the container so Docker's restart policy handles it — was rejected because + it couples a working data plane to an unreachable edge: dropping LAN ingest on + `:4318` over a DNS hiccup is the worse of the two failures. +- **`/healthz` reports the tunnel.** The entrypoint points cloudflared's metrics + listener at `COTEL_TUNNEL_METRICS_ADDR` (`127.0.0.1:2000` by default) and + tells cotel where it is; `/healthz` then carries a `tunnel` object read from + cloudflared's `/ready`, so "the container is healthy and the tunnel is dead" + is a state an operator and a poller can both see. It is an added field and a + deliberate non-failure: see + [What `/healthz` returns](https://github.com/Flopsstuff/cotel/blob/main/README.md#what-healthz-returns). + +The supervisor is exercised by `bash scripts/supervise-cloudflared_test.sh` +against a stub cloudflared — no Docker, no credentials, no network. + ## Verifying the tunnel ```sh -# Container logs — look for "cloudflared started in token mode" +# Container logs — look for "cloudflared started in token mode under supervision", +# then cloudflared's own "Registered tunnel connection" (one per edge connection) docker compose logs cotel +# The tunnel as cotel sees it: ok plus the number of registered edge connections +curl -s http://localhost:8080/healthz | python3 -m json.tool + # Tunnel status in the Cloudflare dashboard: # Zero Trust → Networks → Tunnels → cotel → should show "Healthy" diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index e9fe153..634e9b3 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -72,6 +72,24 @@ If `last_ingest_at` / `newest_span_age_seconds` are absent, the probe degrades to liveness (HTTP 200) so it keeps working before that contract is on production. +### The `tunnel` field is reported, not yet classified + +`/healthz` also carries a `tunnel` object on a deployment that runs one — the +state of the cloudflared process in the container, read from its `/ready` route +([README](https://github.com/Flopsstuff/cotel/blob/main/README.md#what-healthz-returns)). The LAN probe deliberately +does **not** classify it yet: every verdict above routes the on-call to a +different place, and "the application is fine, the public path is not" already +has a half of its own with its own words, its own dedup marker and its own +re-probe instructions. Folding it into the LAN half's exit codes would page the +database wording for a Cloudflare fault. The field is there for an operator +reading `/healthz` directly, and for whoever decides which half should own that +classification. + +Where it helps today is the ambiguity the edge half cannot resolve on its own: a +red public-ingest probe says the chain is broken somewhere between DNS and the +origin, while `tunnel.ok: false` beside a green `ok: true` names cloudflared +itself, from inside the container, before anyone opens a shell on the host. + ### The snapshot claim, asked of a second endpoint A green `/healthz` is followed by a second question, to `/api/v1/health` on the diff --git a/internal/dashboard/handler.go b/internal/dashboard/handler.go index 12aa829..59c4f5f 100644 --- a/internal/dashboard/handler.go +++ b/internal/dashboard/handler.go @@ -2,6 +2,7 @@ package dashboard import ( + "context" "database/sql" "embed" "encoding/json" @@ -11,6 +12,7 @@ import ( "time" "github.com/Flopsstuff/cotel/internal/storage" + "github.com/Flopsstuff/cotel/internal/tunnel" ) //go:embed static @@ -21,14 +23,27 @@ type DB interface { Query(query string, args ...any) (*sql.Rows, error) } +// TunnelProber answers for the cloudflared process running beside cotel. +type TunnelProber interface { + Health(ctx context.Context) tunnel.Health +} + type Handler struct { - db DB + db DB + tunnel TunnelProber } func New(db DB) *Handler { return &Handler{db: db} } +// SetTunnelProber makes /healthz report the tunnel's state. Left unset on a +// deployment that runs no tunnel, and the key is then absent entirely. +func (h *Handler) SetTunnelProber(p TunnelProber) *Handler { + h.tunnel = p + return h +} + func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { path := r.URL.Path @@ -65,13 +80,20 @@ type healthzResponse struct { // and reporting 0 would read as "ingested just now". LastIngestAt *string `json:"last_ingest_at"` NewestSpanAgeSeconds *int64 `json:"newest_span_age_seconds"` + // Absent unless the deployment runs a Cloudflare tunnel. A dead tunnel does + // not make "ok" false: this endpoint drives the container HEALTHCHECK and + // the deploy gate, and LAN ingest keeps working while the public path is + // down — whoever polls decides what an unreachable edge is worth. + Tunnel *tunnel.Health `json:"tunnel,omitempty"` } // serveHealthz reports liveness plus ingest freshness. Staleness never changes // the status code: this endpoint drives the container HEALTHCHECK, and a quiet // weekend is not a broken service — the poller owns the staleness threshold. // A failed query is different: the body says ok:false and the code says -// unavailable, matching the pre-storage readiness gate. +// unavailable, matching the pre-storage readiness gate. A dead tunnel is +// reported in the body for the same reason staleness is, and changes the status +// code just as little. func (h *Handler) serveHealthz(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Type", "application/json") @@ -81,12 +103,17 @@ func (h *Handler) serveHealthz(w http.ResponseWriter, r *http.Request) { writeJSON(w, healthzResponse{OK: false}) return } - writeJSON(w, healthzResponse{ + resp := healthzResponse{ OK: true, Spans: fresh.SpanCount, LastIngestAt: fresh.Timestamp(), NewestSpanAgeSeconds: fresh.AgeSeconds(time.Now()), - }) + } + if h.tunnel != nil { + th := h.tunnel.Health(r.Context()) + resp.Tunnel = &th + } + writeJSON(w, resp) } func writeJSON(w http.ResponseWriter, v any) { diff --git a/internal/dashboard/healthz_test.go b/internal/dashboard/healthz_test.go index 5d0d242..f7fdb22 100644 --- a/internal/dashboard/healthz_test.go +++ b/internal/dashboard/healthz_test.go @@ -1,6 +1,7 @@ package dashboard_test import ( + "context" "database/sql" "encoding/json" "net/http" @@ -10,6 +11,7 @@ import ( "github.com/Flopsstuff/cotel/internal/dashboard" "github.com/Flopsstuff/cotel/internal/storage" + "github.com/Flopsstuff/cotel/internal/tunnel" ) func openTestDBRW(t *testing.T) (*storage.DB, *storage.ReadDB) { @@ -189,3 +191,70 @@ func TestHealthzFailedQueryIsNotOK(t *testing.T) { t.Errorf("a failed query must not answer 200, got %d", code) } } + +// stubProber stands in for cloudflared's metrics listener. +type stubProber struct{ health tunnel.Health } + +func (s stubProber) Health(context.Context) tunnel.Health { return s.health } + +func TestHealthzReportsTunnelWhenOneRuns(t *testing.T) { + _, ro := openTestDBRW(t) + ready := 4 + code, body := getHealthz(t, dashboard.New(ro).SetTunnelProber( + stubProber{health: tunnel.Health{OK: true, ReadyConnections: &ready}})) + + if code != http.StatusOK { + t.Fatalf("want 200, got %d", code) + } + tun, ok := body["tunnel"].(map[string]any) + if !ok { + t.Fatalf("want a tunnel object, got %v", body["tunnel"]) + } + if tun["ok"] != true { + t.Errorf("want tunnel.ok=true, got %v", tun["ok"]) + } + if tun["ready_connections"] != float64(4) { + t.Errorf("want tunnel.ready_connections=4, got %v", tun["ready_connections"]) + } + if _, present := tun["error"]; present { + t.Errorf("a healthy tunnel must not carry an error, got %v", tun["error"]) + } +} + +// The dead tunnel this field exists for: the application and LAN ingest are +// fine, so the status code and ok must not move — only the new field does. +func TestHealthzDeadTunnelIsReportedWithoutFailingTheCheck(t *testing.T) { + _, ro := openTestDBRW(t) + code, body := getHealthz(t, dashboard.New(ro).SetTunnelProber( + stubProber{health: tunnel.Health{Error: "cloudflared metrics listener 127.0.0.1:2000 unreachable: connection refused"}})) + + if code != http.StatusOK { + t.Errorf("a dead tunnel must not change the status code: want 200, got %d", code) + } + if body["ok"] != true { + t.Errorf("a dead tunnel must not change ok, got %v", body["ok"]) + } + tun, ok := body["tunnel"].(map[string]any) + if !ok { + t.Fatalf("want a tunnel object, got %v", body["tunnel"]) + } + if tun["ok"] != false { + t.Errorf("want tunnel.ok=false, got %v", tun["ok"]) + } + if tun["ready_connections"] != nil { + t.Errorf("an unreachable cloudflared has no connection count, got %v", tun["ready_connections"]) + } + if tun["error"] == nil || tun["error"] == "" { + t.Errorf("a dead tunnel must say why, got %v", tun["error"]) + } +} + +// A tokenless deployment (local dev, CI) runs no tunnel, and must not report a +// dead one: the key is absent rather than false. +func TestHealthzOmitsTunnelWhenNoneIsConfigured(t *testing.T) { + _, ro := openTestDBRW(t) + _, body := getHealthz(t, dashboard.New(ro)) + if v, present := body["tunnel"]; present { + t.Errorf("want no tunnel key without a tunnel, got %v", v) + } +} diff --git a/internal/tunnel/tunnel.go b/internal/tunnel/tunnel.go new file mode 100644 index 0000000..e32961e --- /dev/null +++ b/internal/tunnel/tunnel.go @@ -0,0 +1,108 @@ +// Package tunnel reports the state of the cloudflared process the container +// entrypoint runs beside cotel. +// +// A dead tunnel is invisible from inside cotel: it has its own process, its own +// failure modes, and the application keeps answering on the LAN while the +// public ingest URL every client exports to returns a Cloudflare error. The +// state comes from cloudflared's own metrics listener rather than from process +// liveness, because a running cloudflared with no registered edge connection +// drops traffic exactly like a dead one. +package tunnel + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "os" + "strings" + "time" +) + +// EnvMetricsAddr names cloudflared's metrics listener. The entrypoint exports +// it only on the paths where it actually started a tunnel, so unset means "no +// tunnel is expected here" rather than "the tunnel is down" — which is what +// keeps a tokenless local or CI instance out of the tunnel verdict. +const EnvMetricsAddr = "COTEL_TUNNEL_METRICS_ADDR" + +// probeTimeout bounds one /ready request. The listener is in this container, so +// anything slower than this is as good as unreachable — and /healthz drives the +// container HEALTHCHECK, whose own timeout is 5s. +const probeTimeout = 2 * time.Second + +// Health is the tunnel's state, reported under the "tunnel" key of /healthz. +type Health struct { + OK bool `json:"ok"` + // Null when cloudflared could not be asked at all. That is distinct from + // zero, which means it answered and has no edge connection. + ReadyConnections *int `json:"ready_connections"` + Error string `json:"error,omitempty"` +} + +// Prober asks cloudflared's metrics listener for its readiness. +type Prober struct { + addr string + url string + client *http.Client +} + +// New returns a Prober for a cloudflared metrics listener address +// (host:port, as passed to cloudflared's --metrics). +func New(addr string) *Prober { + return &Prober{ + addr: addr, + url: "http://" + addr + "/ready", + client: &http.Client{Timeout: probeTimeout}, + } +} + +// FromEnv returns a Prober for the address in EnvMetricsAddr, or nil when the +// variable is unset or empty — the deployment then runs no tunnel and /healthz +// says nothing about one. +func FromEnv() *Prober { + addr := strings.TrimSpace(os.Getenv(EnvMetricsAddr)) + if addr == "" { + return nil + } + return New(addr) +} + +// Addr is the metrics listener this Prober asks. +func (p *Prober) Addr() string { return p.addr } + +// Health probes cloudflared's /ready route. Every failure is reported as data +// rather than an error: the caller is a health endpoint that must answer. +func (p *Prober) Health(ctx context.Context) Health { + ctx, cancel := context.WithTimeout(ctx, probeTimeout) + defer cancel() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, p.url, nil) + if err != nil { + return Health{Error: fmt.Sprintf("cloudflared metrics address %q is unusable: %v", p.addr, err)} + } + resp, err := p.client.Do(req) + if err != nil { + return Health{Error: fmt.Sprintf("cloudflared metrics listener %s unreachable: %v", p.addr, err)} + } + defer resp.Body.Close() //nolint:errcheck + + var body struct { + ReadyConnections *int `json:"readyConnections"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return Health{Error: fmt.Sprintf("cloudflared /ready returned HTTP %d with an unreadable body: %v", resp.StatusCode, err)} + } + + h := Health{ReadyConnections: body.ReadyConnections} + switch { + case body.ReadyConnections == nil: + h.Error = fmt.Sprintf("cloudflared /ready returned HTTP %d without a connection count", resp.StatusCode) + case *body.ReadyConnections < 1: + h.Error = "cloudflared has no registered edge connection" + case resp.StatusCode != http.StatusOK: + h.Error = fmt.Sprintf("cloudflared /ready returned HTTP %d", resp.StatusCode) + default: + h.OK = true + } + return h +} diff --git a/internal/tunnel/tunnel_test.go b/internal/tunnel/tunnel_test.go new file mode 100644 index 0000000..2242260 --- /dev/null +++ b/internal/tunnel/tunnel_test.go @@ -0,0 +1,100 @@ +package tunnel_test + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/Flopsstuff/cotel/internal/tunnel" +) + +// readyBody is cloudflared 2026.9.3's /ready payload, verified against the +// binary the image pins. +const readyBody = `{"status":200,"readyConnections":4,"connectorId":"eca7539d-e94c-4516-a71e-dac7690f5430"}` + +func proberFor(t *testing.T, status int, body string) *tunnel.Prober { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/ready" { + t.Errorf("want a request to /ready, got %s", r.URL.Path) + } + w.WriteHeader(status) + w.Write([]byte(body)) //nolint:errcheck + })) + t.Cleanup(srv.Close) + return tunnel.New(strings.TrimPrefix(srv.URL, "http://")) +} + +func TestHealthRegisteredConnectionsAreOK(t *testing.T) { + h := proberFor(t, http.StatusOK, readyBody).Health(context.Background()) + if !h.OK { + t.Errorf("want ok, got %+v", h) + } + if h.ReadyConnections == nil || *h.ReadyConnections != 4 { + t.Errorf("want 4 ready connections, got %v", h.ReadyConnections) + } + if h.Error != "" { + t.Errorf("want no error, got %q", h.Error) + } +} + +// cloudflared is running but has registered nothing: traffic is dropped exactly +// as if the process were gone, so process liveness is not the signal. +func TestHealthZeroConnectionsIsNotOK(t *testing.T) { + h := proberFor(t, http.StatusServiceUnavailable, `{"status":503,"readyConnections":0}`).Health(context.Background()) + if h.OK { + t.Errorf("zero connections must not read as ok, got %+v", h) + } + if h.ReadyConnections == nil || *h.ReadyConnections != 0 { + t.Errorf("a cloudflared that answered must report its count, got %v", h.ReadyConnections) + } + if h.Error == "" { + t.Error("want a reason") + } +} + +// The failure mode this whole field exists for: cloudflared exited, so nothing +// answers on the metrics port. +func TestHealthUnreachableListenerIsNotOK(t *testing.T) { + // Port 0 never has a listener, so this fails without waiting for a timeout. + h := tunnel.New("127.0.0.1:0").Health(context.Background()) + if h.OK { + t.Errorf("want not ok, got %+v", h) + } + if h.ReadyConnections != nil { + t.Errorf("an unreachable cloudflared has no count, got %v", *h.ReadyConnections) + } + if !strings.Contains(h.Error, "unreachable") { + t.Errorf("want an unreachable error, got %q", h.Error) + } +} + +func TestHealthUnreadableBodyIsNotOK(t *testing.T) { + h := proberFor(t, http.StatusOK, "not json").Health(context.Background()) + if h.OK || h.Error == "" { + t.Errorf("want not ok with a reason, got %+v", h) + } +} + +// Unset means "this deployment runs no tunnel", which is not the same claim as +// "the tunnel is down" — a tokenless local or CI instance must make neither. +func TestFromEnvIsNilWithoutAnAddress(t *testing.T) { + t.Setenv(tunnel.EnvMetricsAddr, "") + if p := tunnel.FromEnv(); p != nil { + t.Errorf("want nil prober, got one for %q", p.Addr()) + } + t.Setenv(tunnel.EnvMetricsAddr, " ") + if p := tunnel.FromEnv(); p != nil { + t.Errorf("whitespace is not an address, got a prober for %q", p.Addr()) + } + t.Setenv(tunnel.EnvMetricsAddr, "127.0.0.1:2000") + p := tunnel.FromEnv() + if p == nil { + t.Fatal("want a prober when the address is set") + } + if p.Addr() != "127.0.0.1:2000" { + t.Errorf("want the configured address, got %q", p.Addr()) + } +} diff --git a/scripts/entrypoint.sh b/scripts/entrypoint.sh index 75c02ef..dcc6797 100755 --- a/scripts/entrypoint.sh +++ b/scripts/entrypoint.sh @@ -5,20 +5,26 @@ # Local-config mode: /etc/cloudflared/config.yml mounted → cloudflared tunnel --config ... run # Token mode takes precedence if both are present. No tunnel is started if neither is set. # +# cloudflared runs under scripts/supervise-cloudflared.sh, which restarts it for +# as long as the container lives: on its own it exits for good on a failed edge +# discovery, and nothing here would notice, because cotel is the main process. +# # On SIGTERM/SIGINT the signal is forwarded to cotel first and we wait for it to # exit, so cotel can CHECKPOINT the DuckDB WAL before the container stops; # otherwise the next start replays the WAL (a multi-minute cost on a large DB). # cloudflared is stopped afterwards. -CLOUDFLARED_PID="" +CLOUDFLARED_SUPERVISOR_PID="" COTEL_PID="" CLOUDFLARED_CONFIG_DEFAULT=/etc/cloudflared/config.yml +CLOUDFLARED_SUPERVISOR=/usr/local/bin/supervise-cloudflared.sh +TUNNEL_METRICS_ADDR_DEFAULT=127.0.0.1:2000 stop_cloudflared() { - if [ -n "${CLOUDFLARED_PID}" ]; then - echo "entrypoint: stopping cloudflared (PID ${CLOUDFLARED_PID})" - kill "${CLOUDFLARED_PID}" 2>/dev/null || true - wait "${CLOUDFLARED_PID}" 2>/dev/null || true + if [ -n "${CLOUDFLARED_SUPERVISOR_PID}" ]; then + echo "entrypoint: stopping the cloudflared supervisor (PID ${CLOUDFLARED_SUPERVISOR_PID})" + kill "${CLOUDFLARED_SUPERVISOR_PID}" 2>/dev/null || true + wait "${CLOUDFLARED_SUPERVISOR_PID}" 2>/dev/null || true fi } @@ -47,20 +53,32 @@ trap terminate TERM INT cfd_token="${CLOUDFLARE_TUNNEL_TOKEN:-}" unset CLOUDFLARE_TUNNEL_TOKEN +cfd_config="${CLOUDFLARED_CONFIG:-${CLOUDFLARED_CONFIG_DEFAULT}}" + +# cotel reports the tunnel's state on /healthz only when this variable reaches +# it, so it is exported below exclusively on the paths that actually start a +# tunnel. An operator-set value must not survive into a deployment with no +# tunnel: a tokenless instance (local dev, CI) would then report a dead one. +tunnel_metrics_addr="${COTEL_TUNNEL_METRICS_ADDR:-${TUNNEL_METRICS_ADDR_DEFAULT}}" +unset COTEL_TUNNEL_METRICS_ADDR + if [ -n "${cfd_token}" ]; then - # cloudflared 2026.4.0 changed the --edge-ip-version default from 4 to auto, - # which follows whichever family the resolver answers with first and only - # falls back to the other one after a connection has already failed. Keep - # the old default here, where cotel owns the whole invocation; local-config - # mode is left alone because this env var outranks a config.yml setting. - export TUNNEL_EDGE_IP_VERSION="${TUNNEL_EDGE_IP_VERSION:-4}" - cloudflared tunnel run --token "${cfd_token}" & - CLOUDFLARED_PID=$! - echo "entrypoint: cloudflared started in token mode (PID ${CLOUDFLARED_PID})" -elif [ -f "${CLOUDFLARED_CONFIG:-${CLOUDFLARED_CONFIG_DEFAULT}}" ]; then - cloudflared tunnel --config "${CLOUDFLARED_CONFIG:-${CLOUDFLARED_CONFIG_DEFAULT}}" run & - CLOUDFLARED_PID=$! - echo "entrypoint: cloudflared started in local-config mode (PID ${CLOUDFLARED_PID})" + # Passed through the supervisor's environment rather than its arguments, so + # the token does not gain a second `ps` row; the supervisor unsets it before + # cloudflared inherits anything. + CLOUDFLARE_TUNNEL_TOKEN="${cfd_token}" \ + COTEL_TUNNEL_METRICS_ADDR="${tunnel_metrics_addr}" \ + "${CLOUDFLARED_SUPERVISOR}" & + CLOUDFLARED_SUPERVISOR_PID=$! + export COTEL_TUNNEL_METRICS_ADDR="${tunnel_metrics_addr}" + echo "entrypoint: cloudflared started in token mode under supervision (supervisor PID ${CLOUDFLARED_SUPERVISOR_PID}, metrics ${tunnel_metrics_addr})" +elif [ -f "${cfd_config}" ]; then + CLOUDFLARED_CONFIG="${cfd_config}" \ + COTEL_TUNNEL_METRICS_ADDR="${tunnel_metrics_addr}" \ + "${CLOUDFLARED_SUPERVISOR}" & + CLOUDFLARED_SUPERVISOR_PID=$! + export COTEL_TUNNEL_METRICS_ADDR="${tunnel_metrics_addr}" + echo "entrypoint: cloudflared started in local-config mode under supervision (supervisor PID ${CLOUDFLARED_SUPERVISOR_PID}, metrics ${tunnel_metrics_addr})" fi /usr/local/bin/cotel "$@" & diff --git a/scripts/supervise-cloudflared.sh b/scripts/supervise-cloudflared.sh new file mode 100755 index 0000000..dc5e40a --- /dev/null +++ b/scripts/supervise-cloudflared.sh @@ -0,0 +1,122 @@ +#!/bin/sh +# supervise-cloudflared.sh — run cloudflared beside cotel and keep it running. +# +# cloudflared has its own retry loop for connection and protocol errors, but it +# does not cover every way a start can fail. Edge discovery is the one that cost +# us a day of telemetry: a container that comes up before the host's resolver is +# usable gets "no such host" for _v2-origintunneld._tcp.argotunnel.com, logs +# "Initiating shutdown" and exits for good. Nothing noticed, because cotel — not +# cloudflared — is the container's main process: `restart: unless-stopped` never +# fired, the container stayed (healthy) with RestartCount 0, and every agent +# exporting to the public ingest URL lost its spans silently. +# +# So every cloudflared exit is retried here, forever, with a capped backoff, +# rather than being made fatal to the container. Both halves of that choice +# matter: a tunnel that stays dead is the failure this exists to prevent, and +# killing the container over an unreachable edge would also drop LAN ingest on +# :4318, which keeps working throughout. +# +# Environment: +# CLOUDFLARE_TUNNEL_TOKEN token mode (takes precedence). Consumed +# here and removed from the environment +# cloudflared inherits — see the entrypoint +# for why it must not reach it. +# CLOUDFLARED_CONFIG local-config mode: path to a config.yml. +# COTEL_TUNNEL_METRICS_ADDR cloudflared's metrics listener, passed on +# as TUNNEL_METRICS. Its /ready route is +# what cotel's /healthz reports. +# COTEL_TUNNEL_RESTART_BACKOFF_MAX cap on the restart delay, seconds (60). + +set -u + +log() { echo "supervise-cloudflared: $*"; } + +BACKOFF_MAX="${COTEL_TUNNEL_RESTART_BACKOFF_MAX:-60}" +case "${BACKOFF_MAX}" in + ''|*[!0-9]*) + log "warning: invalid COTEL_TUNNEL_RESTART_BACKOFF_MAX='${BACKOFF_MAX}', using 60" + BACKOFF_MAX=60 + ;; +esac +[ "${BACKOFF_MAX}" -ge 1 ] || BACKOFF_MAX=1 + +# cloudflared logs the value of every environment variable whose whole KEY=VALUE +# pair merely contains "TUNNEL_", and redacts only the two names it owns itself. +# The token goes in as a flag, which it does redact, and leaves the environment. +cfd_token="${CLOUDFLARE_TUNNEL_TOKEN:-}" +unset CLOUDFLARE_TUNNEL_TOKEN + +if [ -n "${COTEL_TUNNEL_METRICS_ADDR:-}" ]; then + TUNNEL_METRICS="${COTEL_TUNNEL_METRICS_ADDR}" + export TUNNEL_METRICS +fi + +if [ -n "${cfd_token}" ]; then + mode="token" + # 2026.4.0 changed the --edge-ip-version default from 4 to auto, which + # follows whichever family the resolver answers with first and only falls + # back after a connection has already failed. Keep the old default where + # cotel owns the whole invocation; local-config mode is left alone because + # this env var outranks a config.yml setting. + TUNNEL_EDGE_IP_VERSION="${TUNNEL_EDGE_IP_VERSION:-4}" + export TUNNEL_EDGE_IP_VERSION + set -- tunnel run --token "${cfd_token}" +elif [ -n "${CLOUDFLARED_CONFIG:-}" ]; then + mode="local-config" + set -- tunnel --config "${CLOUDFLARED_CONFIG}" run +else + log "neither CLOUDFLARE_TUNNEL_TOKEN nor CLOUDFLARED_CONFIG is set; nothing to supervise" + exit 2 +fi + +child="" +stopping=0 + +# The entrypoint stops this supervisor by signalling it, so the signal must +# reach the cloudflared (or sleep) it is currently waiting on, and must not be +# followed by another restart. +stop() { + stopping=1 + if [ -n "${child}" ]; then + log "stop requested, signalling supervised child (PID ${child})" + kill "${child}" 2>/dev/null || true + wait "${child}" 2>/dev/null || true + fi + exit 0 +} +trap stop TERM INT + +delay=1 +while :; do + started="$(date +%s)" + cloudflared "$@" & + child=$! + log "cloudflared started in ${mode} mode (PID ${child})" + wait "${child}" + status=$? + child="" + [ "${stopping}" -eq 1 ] && exit 0 + + ran=$(( $(date +%s) - started )) + # A cloudflared that stayed up at least as long as the cap was working, so + # the next failure starts from a short delay again instead of inheriting + # the backoff of an outage that is over. + if [ "${ran}" -ge "${BACKOFF_MAX}" ]; then + delay=1 + fi + log "cloudflared exited with status ${status} after ${ran}s; restarting in ${delay}s" + + # Backgrounded so a stop signal lands during the wait rather than after the + # full delay: a foreground sleep would hold the container's stop grace + # period for up to BACKOFF_MAX seconds. + sleep "${delay}" & + child=$! + wait "${child}" 2>/dev/null || true + child="" + [ "${stopping}" -eq 1 ] && exit 0 + + delay=$(( delay * 2 )) + if [ "${delay}" -gt "${BACKOFF_MAX}" ]; then + delay="${BACKOFF_MAX}" + fi +done diff --git a/scripts/supervise-cloudflared_test.sh b/scripts/supervise-cloudflared_test.sh new file mode 100755 index 0000000..515daa0 --- /dev/null +++ b/scripts/supervise-cloudflared_test.sh @@ -0,0 +1,219 @@ +#!/usr/bin/env bash +# +# supervise-cloudflared_test.sh — drive supervise-cloudflared.sh against a stub +# cloudflared. Covers the restart loop (the dead-tunnel failure this script +# exists for), the capped backoff, a prompt stop while it is backing off, both +# tunnel modes' argv and environment, and the no-mode exit. +# +# No Docker, no credentials and no network: the stub records how it was invoked +# and exits on command. + +set -uo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SUPERVISOR="$ROOT/scripts/supervise-cloudflared.sh" +PASS=0 +FAIL=0 + +fail() { + echo "FAIL: $*" + FAIL=$((FAIL + 1)) +} + +pass() { + echo "PASS: $*" + PASS=$((PASS + 1)) +} + +want_in() { + local name="$1" needle="$2" haystack="$3" + if [[ "$haystack" == *"$needle"* ]]; then + pass "$name" + else + fail "$name — output did not contain '$needle': +$haystack" + fi +} + +want_not_in() { + local name="$1" needle="$2" haystack="$3" + if [[ "$haystack" == *"$needle"* ]]; then + fail "$name — output unexpectedly contained '$needle': +$haystack" + else + pass "$name" + fi +} + +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"; pkill -f "$WORK/bin/cloudflared" 2>/dev/null' EXIT + +mkdir -p "$WORK/bin" +cat >"$WORK/bin/cloudflared" <<'STUB' +#!/bin/sh +# Stub cloudflared. Records each invocation, then either exits (the failure the +# supervisor must absorb) or stays up (a tunnel that registered). +n=$(cat "$STUB_STATE/count" 2>/dev/null || echo 0) +n=$((n + 1)) +echo "$n" >"$STUB_STATE/count" +{ + echo "run ${n} argv: $*" + echo "run ${n} TUNNEL_METRICS=${TUNNEL_METRICS:-}" + echo "run ${n} TUNNEL_EDGE_IP_VERSION=${TUNNEL_EDGE_IP_VERSION:-}" + echo "run ${n} CLOUDFLARE_TUNNEL_TOKEN=${CLOUDFLARE_TUNNEL_TOKEN:-}" +} >>"$STUB_STATE/log" +if [ "${n}" -ge "${STUB_STAY_UP_FROM:-99999}" ]; then + echo "run ${n} staying up" >>"$STUB_STATE/log" + while :; do sleep 0.2; done +fi +exit "${STUB_EXIT:-1}" +STUB +chmod +x "$WORK/bin/cloudflared" + +# wait_for +wait_for() { + local file="$1" needle="$2" deadline=$(( $(date +%s) + $3 )) + while [ "$(date +%s)" -lt "$deadline" ]; do + if [ -f "$file" ] && grep -qF "$needle" "$file"; then + return 0 + fi + sleep 0.2 + done + return 1 +} + +# +# 1. A cloudflared that keeps exiting is restarted until it stays up, with the +# delay doubling up to the cap. This is the boot-time edge-discovery failure: +# on its own cloudflared exits for good and the tunnel never comes back. +# +state="$WORK/state1" +mkdir -p "$state" +out="$WORK/out1" +STUB_STATE="$state" STUB_STAY_UP_FROM=3 \ +PATH="$WORK/bin:$PATH" \ +CLOUDFLARE_TUNNEL_TOKEN="test-token" \ +COTEL_TUNNEL_METRICS_ADDR="127.0.0.1:2999" \ +COTEL_TUNNEL_RESTART_BACKOFF_MAX=2 \ + sh "$SUPERVISOR" >"$out" 2>&1 & +sup=$! + +if wait_for "$state/log" "run 3 staying up" 20; then + pass "restarts cloudflared until it stays up (no manual intervention)" +else + fail "cloudflared was not restarted to a running state within 20s; log: +$(cat "$state/log" 2>/dev/null) +$(cat "$out")" +fi + +log1="$(cat "$state/log" 2>/dev/null)" +sup_out="$(cat "$out")" +want_in "token mode runs 'tunnel run --token'" "argv: tunnel run --token test-token" "$log1" +want_in "the metrics address reaches cloudflared as TUNNEL_METRICS" "TUNNEL_METRICS=127.0.0.1:2999" "$log1" +want_in "token mode pins the edge IP version" "TUNNEL_EDGE_IP_VERSION=4" "$log1" +want_in "the token is kept out of cloudflared's environment" "CLOUDFLARE_TUNNEL_TOKEN=" "$log1" +want_in "the first restart waits 1s" "restarting in 1s" "$sup_out" +want_in "the delay doubles up to the cap" "restarting in 2s" "$sup_out" +want_not_in "the delay does not exceed the cap" "restarting in 4s" "$sup_out" + +# Stopping the supervisor must stop the cloudflared it is running and must not +# start another one: the container's shutdown path signals this process. +kill -TERM "$sup" 2>/dev/null +wait "$sup" +sup_rc=$? +if [ "$sup_rc" -eq 0 ]; then + pass "a stop signal exits the supervisor cleanly" +else + fail "want exit 0 on SIGTERM, got $sup_rc; output: +$(cat "$out")" +fi + +sleep 0.5 +if pgrep -f "$WORK/bin/cloudflared" >/dev/null 2>&1; then + fail "cloudflared survived the supervisor's stop: $(pgrep -af "$WORK/bin/cloudflared")" + pkill -f "$WORK/bin/cloudflared" 2>/dev/null +else + pass "the supervised cloudflared is stopped with the supervisor" +fi + +if [ "$(cat "$state/count")" = "3" ]; then + pass "no restart after the stop signal" +else + fail "want 3 cloudflared runs, got $(cat "$state/count")" +fi + +# +# 2. A stop that lands mid-backoff is not held until the delay is over: the +# container's stop grace period is shorter than the cap. +# +state="$WORK/state2" +mkdir -p "$state" +out="$WORK/out2" +STUB_STATE="$state" \ +PATH="$WORK/bin:$PATH" \ +CLOUDFLARE_TUNNEL_TOKEN="test-token" \ +COTEL_TUNNEL_RESTART_BACKOFF_MAX=30 \ + sh "$SUPERVISOR" >"$out" 2>&1 & +sup=$! + +if wait_for "$out" "restarting in 4s" 20; then + started="$(date +%s)" + kill -TERM "$sup" 2>/dev/null + wait "$sup" + sup_rc=$? + elapsed=$(( $(date +%s) - started )) + if [ "$sup_rc" -eq 0 ] && [ "$elapsed" -le 2 ]; then + pass "a stop during the backoff exits within ${elapsed}s" + else + fail "want exit 0 within 2s during the backoff, got exit $sup_rc after ${elapsed}s" + fi +else + kill -TERM "$sup" 2>/dev/null + wait "$sup" 2>/dev/null + fail "the backoff never reached 4s within 20s; output: +$(cat "$out")" +fi + +# +# 3. Local-config mode: same loop, cloudflared's own invocation. +# +state="$WORK/state3" +mkdir -p "$state" +out="$WORK/out3" +touch "$WORK/config.yml" +STUB_STATE="$state" STUB_STAY_UP_FROM=1 \ +PATH="$WORK/bin:$PATH" \ +CLOUDFLARED_CONFIG="$WORK/config.yml" \ +COTEL_TUNNEL_METRICS_ADDR="127.0.0.1:2998" \ + sh "$SUPERVISOR" >"$out" 2>&1 & +sup=$! +if wait_for "$state/log" "run 1 staying up" 20; then + log3="$(cat "$state/log")" + want_in "local-config mode runs 'tunnel --config … run'" "argv: tunnel --config $WORK/config.yml run" "$log3" + want_in "local-config mode passes the metrics address" "TUNNEL_METRICS=127.0.0.1:2998" "$log3" + want_in "local-config mode leaves the edge IP version to config.yml" "TUNNEL_EDGE_IP_VERSION=" "$log3" +else + fail "local-config mode did not start cloudflared within 20s; output: +$(cat "$out")" +fi +kill -TERM "$sup" 2>/dev/null +wait "$sup" 2>/dev/null + +# +# 4. Neither mode configured: say so and exit, rather than loop on a cloudflared +# that cannot work. The entrypoint only starts this script when one is set. +# +out="$WORK/out4" +PATH="$WORK/bin:$PATH" COTEL_TUNNEL_RESTART_BACKOFF_MAX=notanumber \ + sh "$SUPERVISOR" >"$out" 2>&1 +rc=$? +if [ "$rc" -eq 2 ]; then + pass "no tunnel configured exits 2" +else + fail "want exit 2 with no tunnel configured, got $rc" +fi +want_in "an unusable backoff cap is reported, not obeyed" "invalid COTEL_TUNNEL_RESTART_BACKOFF_MAX" "$(cat "$out")" + +echo +echo "supervise-cloudflared_test: ${PASS} passed, ${FAIL} failed" +[ "$FAIL" -eq 0 ]