fix(tunnel): supervise cloudflared and report its state on /healthz

.github/workflows/ci.yml

@@ -36,6 +36,12 @@ jobs:
       - name: go test
         run: CGO_ENABLED=1 go test ./...
 
+      # The container's cloudflared supervisor, against a stub cloudflared: it
+      # keeps the tunnel alive and is in no Go package, so nothing above covers
+      # it.
+      - name: cloudflared supervisor tests
+        run: bash scripts/supervise-cloudflared_test.sh
+
   smoke:
     name: Smoke test
     runs-on: ubuntu-latest
@@ -71,6 +77,15 @@ jobs:
           echo "healthz: $RESP"
           echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); assert d['spans'] == 3, f'expected 3 spans, got {d[\"spans\"]}'"
 
+      # This compose run has no tunnel token, so /healthz must say nothing about
+      # a tunnel. A "tunnel" key here would mean a tokenless deployment reports
+      # a dead tunnel — every local and CI instance red for a tunnel nobody
+      # asked for.
+      - name: Verify healthz omits the tunnel field without a token
+        run: |
+          curl -sf http://localhost:8080/healthz \
+            | python3 -c "import sys,json; d=json.load(sys.stdin); assert 'tunnel' not in d, f'unexpected tunnel field: {d}'"
+
       - name: Dashboard health check
         run: curl -sf http://localhost:8080/
 

CHANGELOG.md

@@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `/healthz` and `GET /api/v1/health` report `last_ingest_at` and `newest_span_age_seconds`. A cotel that is up but receiving nothing used to be indistinguishable from a healthy idle one — the dashboard answered `{"ok":true,"spans":0}` for six days of total ingest loss — and the age was not derivable from any other field, so no external poller could have caught it. The age is measured from the span's `ingested_at`, not its `start_time`: replaying an archive carries the original ingest time, so a restore cannot make a dark instance look alive, and a span arriving late with an old `start_time` still counts as fresh traffic. An empty database reports `null` for both rather than `0`, because "nothing ever arrived" is not "something arrived just now". Staleness deliberately does not change the status code on either endpoint — `/healthz` drives the container `HEALTHCHECK`, and the threshold at which silence becomes an alert belongs to whoever polls, not to the container. Both additions are additive; `ok`, `spans` and `span_count` are unchanged
 
 ### Fixed
+- A dead Cloudflare tunnel no longer stays dead, and no longer hides behind a healthy container. cloudflared is a second process in the container and cotel is the main one, so `restart: unless-stopped` watches the wrong one: when the deploy host rebooted on 2026-10-14 and cloudflared's edge discovery hit a resolver that was not up yet (`lookup _v2-origintunneld._tcp.argotunnel.com on 127.0.0.11:53: no such host`), it logged `Initiating shutdown`, exited for good, and the container went on reporting `Up (healthy)` with `RestartCount` 0 for 11 h 39 min while every agent exporting to the public ingest URL got a Cloudflare error and lost its spans. The failure is not in cloudflared's own retry loop - that covers connection and protocol errors, not a start that cannot find the edge at all. `scripts/supervise-cloudflared.sh` now owns the process and restarts it on every exit for as long as the container lives, waiting 1s and doubling to a `COTEL_TUNNEL_RESTART_BACKOFF_MAX` (60s) cap, resetting the delay after a cloudflared that stayed up at least that long; a stop signal reaches whatever it is waiting on, so a container stop is not held for the length of a backoff. Making the tunnel's death fatal to the container instead - and letting Docker's restart policy handle it - was rejected deliberately: it would couple a working data plane to an unreachable edge, and dropping LAN ingest on `:4318` over a DNS hiccup is the worse of the two failures. Verified against a real tunnel in a container started with an unreachable resolver: cloudflared failed edge discovery and exited, was restarted on the backoff, and registered four edge connections on its own once the resolver was fixed, with no intervention.
+- `/healthz` reports the tunnel it is supposed to be running. The endpoint answered `200 ok:true` throughout that outage, and nothing in its body could have said otherwise, so the only signal was the 6-hour ingest-staleness threshold - which a naturally quiet instance would never have crossed. A deployment with a tunnel now carries a `tunnel` object (`ok`, `ready_connections`, `error`), read from cloudflared's own `/ready` route rather than from process liveness, because a cloudflared that is up with no registered edge connection drops traffic exactly like one that is gone. It is an added field: `ok`, `spans` and the two freshness fields keep their meanings, and a dead tunnel deliberately changes neither `ok` nor the status code - this endpoint drives the container `HEALTHCHECK` and the deploy gate, and failing it would turn a dead public path into a dead container. The key is absent entirely when no tunnel is configured, so a tokenless local or CI instance says nothing about a tunnel rather than reporting a dead one; the smoke job asserts that. The address of cloudflared's metrics listener is `COTEL_TUNNEL_METRICS_ADDR` (`127.0.0.1:2000`), and the entrypoint passes it to cotel only on the paths where it actually started a tunnel
 - The Cloudflare tunnel token is no longer printed into the container log. `cloudflared` logs at INFO the value of every environment variable whose whole `KEY=VALUE` string merely *contains* `TUNNEL_` - matched against the pair, so a hit can come from the name or from the value - and redacts only the two variable names it declares itself, `TUNNEL_TOKEN` and `TUNNEL_CRED_CONTENTS`. Our own `CLOUDFLARE_TUNNEL_TOKEN` therefore matched the filter, missed the redaction list, and landed in `docker logs` in full on every start. That it leaked at all was luck rather than design: the dump splits `KEY=VALUE` on *every* `=` and drops the pair unless exactly two fields come back, so a token whose base64 carries `=` padding is silently skipped while one whose length happens to be a multiple of four is printed. The entrypoint now copies the token into a shell variable, `unset`s the exported one, and passes the value as `--token`, which cloudflared does redact (`Settings: map[token:*****]`). The reach was wider than the host: `scripts/wait-for-healthy.sh` copies container logs into the CI run log on a failed deploy, and the only thing standing between the token and a public job log was GitHub's literal-match secret filter. Upgrading `cloudflared` would **not** have fixed this - the dump is unchanged between 2024.11.1 and 2026.9.3, both verified by running the binaries against a synthetic padding-free token - so the pinned version stays where it is. `docs/operations/cloudflare-tunnel-remote.md` now tabulates what token mode does and does not hide: the log is clean, while `docker inspect`, `/proc/1/environ` (an environment block is a snapshot taken at `exec`, so a later `unset` cannot clear it) and `ps` inside the container all still carry the value - each of which already requires host or container access
 - A schema apply that corrupts an ART index now fails the start instead of serving traffic. After `storage.Open`, cotel runs the same `CHECKPOINT` used on shutdown; a hard failure exits 3 and writes `<db>.checkpoint-failed` before the gates open, so the deploy health gate catches it. A timeout stays benign and the process continues. Measured on a 35 000-span file: the extra start checkpoint is <1 ms when schema is skipped, 8 ms immediately after a schema re-apply that rebuilds the four indexes
 - `/healthz` stops claiming `ok` when its query failed. The span count was read with a discarded error, so an unreadable database answered `{"ok":true,"spans":0}` — the same body as a brand-new empty one, which made `ok` very nearly a constant and no signal at all. A failed query now answers `503` with `{"ok":false}`, matching the pre-storage readiness gate on the same endpoint, so the container probe cannot report a dead database as healthy. Staleness is still not a failure and still answers `200`

Dockerfile

@@ -54,6 +54,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 COPY --from=builder /cotel /usr/local/bin/cotel
 COPY --from=cloudflared-dl /usr/local/bin/cloudflared /usr/local/bin/cloudflared
 COPY scripts/entrypoint.sh /usr/local/bin/entrypoint.sh
+COPY scripts/supervise-cloudflared.sh /usr/local/bin/supervise-cloudflared.sh
 
 # Data directory — mount a named volume here.
 VOLUME /data

README.md

@@ -304,6 +304,7 @@ docker inspect --format '{{.State.Health.Status}}' cotel   # starting → health
 | `spans` | Rows in `spans` |
 | `last_ingest_at` | When the newest span was **accepted** (RFC 3339), or `null` if nothing was ever ingested |
 | `newest_span_age_seconds` | Seconds since `last_ingest_at`, or `null` if nothing was ever ingested |
+| `tunnel` | **Absent** unless this deployment runs a Cloudflare tunnel. Then an object: `ok` (is the public path up), `ready_connections` (edge connections cloudflared has registered, `null` if it could not be asked at all) and `error` (why not, when `ok` is false) |
 
 The age is measured from `ingested_at`, not `start_time`: importing an archive
 replays the original ingest time, so a restore cannot make a dark instance look
@@ -321,6 +322,23 @@ report a dead database as healthy. The same two fields are on
 `GET /api/v1/health` — see
 [docs/operations/api-reference.md](docs/operations/api-reference.md).
 
+The `tunnel` object exists because a dead tunnel looks like nothing from here.
+cloudflared is a second process in the container, cotel is the main one, so
+cloudflared can exit and leave the container `(healthy)` with `RestartCount` 0
+while every client exporting to the public ingest URL gets a Cloudflare error
+and loses its spans. The state is read from cloudflared's own metrics listener
+(`/ready`), not from process liveness: a cloudflared that is up but has
+registered no edge connection drops traffic exactly like one that is gone.
+`ok: true` requires at least one registered connection.
+
+Like staleness, a dead tunnel **does not change the status code** and does not
+make `ok` false. LAN ingest on `:4318` keeps working while the public path is
+down, and this endpoint drives the container `HEALTHCHECK` and the deploy gate —
+taking the container down over an unreachable edge would turn a partial outage
+into a total one. The field is the signal; the threshold belongs to whoever
+polls. Nothing is reported when no tunnel is configured, so a local or CI
+instance says nothing about a tunnel rather than claiming a dead one.
+
 The retention and snapshot workers' health is deliberately **not** here. This
 endpoint is what `cotel --healthcheck`, the Docker `HEALTHCHECK` and the deploy's
 `wait-for-healthy.sh` read, so a failing backup reported through `ok` would mark
@@ -544,6 +562,8 @@ from the network at startup instead of bundling it).
 | `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/snapshots`. Note that `docker volume prune` on a stopped deploy deletes it - the volume counts as in use only while the container exists. |
 | `COTEL_WAL_AUTOCHECKPOINT` | `4MB` | DuckDB `checkpoint_threshold`: the write-ahead log is folded into the main file once it grows past this size. Lower values bound how much WAL an ungraceful kill leaves to replay on the next open; higher values checkpoint less often during ingest. DuckDB's own default is `16MB`. |
 | `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel |
+| `COTEL_TUNNEL_METRICS_ADDR` | `127.0.0.1:2000` | Where cloudflared's metrics listener binds inside the container (`--metrics`). Its `/ready` route is what the `tunnel` field on `/healthz` reports. The entrypoint passes it on only when it actually started a tunnel, so setting it on a tokenless deployment does not make `/healthz` report a tunnel that was never meant to run |
+| `COTEL_TUNNEL_RESTART_BACKOFF_MAX` | `60` | Cap in seconds on the wait between cloudflared restarts. The supervisor restarts it for as long as the container lives, doubling the delay from 1s up to this cap; a cloudflared that stayed up at least this long resets the delay |
 | `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) |
 | `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. |
 | `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). |

cmd/cotel/main.go

@@ -25,6 +25,7 @@ import (
 	"github.com/Flopsstuff/cotel/internal/importpkg"
 	"github.com/Flopsstuff/cotel/internal/ingest"
 	"github.com/Flopsstuff/cotel/internal/storage"
+	"github.com/Flopsstuff/cotel/internal/tunnel"
 )
 
 func main() {
@@ -127,7 +128,13 @@ func main() {
 	dashMux.Handle("/api/v1/export", auth.Middleware(db, export.NewHandler(db)))
 	dashMux.Handle("/api/v1/import", auth.Middleware(db, importpkg.NewHandler(db)))
 	dashMux.Handle("/api/v1/", apiHandler)
-	dashMux.Handle("/", dashboard.New(ro))
+
+	dashHandler := dashboard.New(ro)
+	if prober := tunnel.FromEnv(); prober != nil {
+		dashHandler = dashHandler.SetTunnelProber(prober)
+		log.Printf("tunnel health on /healthz reads cloudflared metrics at %s", prober.Addr())
+	}
+	dashMux.Handle("/", dashHandler)
 
 	// Open the gates: from here the ports serve live traffic instead of 503.
 	srv.ingest.set(ingestMux)

docs/operations/cloudflare-tunnel-remote.md

@@ -72,7 +72,7 @@ docker run -d \
 docker compose up -d
 ```
 
-The entrypoint detects `CLOUDFLARE_TUNNEL_TOKEN`, removes it from the environment it passes on, and runs `cloudflared tunnel run --token …` in the background.
+The entrypoint detects `CLOUDFLARE_TUNNEL_TOKEN`, removes it from the environment it passes on, and starts `scripts/supervise-cloudflared.sh` in the background, which runs `cloudflared tunnel run --token …` and restarts it for as long as the container lives (see [Keeping the tunnel up](#keeping-the-tunnel-up)).
 
 ## Where the token is visible
 
@@ -82,7 +82,7 @@ The token is the authority to serve your public hostnames, so it is worth knowin
 |---|---|
 | `docker compose logs cotel` / `docker logs` | No - the entrypoint keeps the variable out of cloudflared's environment, and cloudflared redacts the `--token` flag to `token:*****` |
 | `docker inspect` / `docker compose config` | **Yes** - it is container configuration; whoever can run these can read it |
-| `/proc/1/environ` inside the container | **Yes** - a process's environment block is a snapshot taken at `exec`, so unsetting the variable afterwards does not clear it |
+| `/proc/1/environ` inside the container, and the same file for the supervisor | **Yes** - a process's environment block is a snapshot taken at `exec`, so unsetting the variable afterwards does not clear it. The entrypoint hands the token to `supervise-cloudflared.sh` through its environment, so that process has a copy too; cloudflared itself does not, which is the one that matters for the log |
 | `ps` inside the container | **Yes** - it is an argument to `cloudflared` |
 
 The log line matters separately from the rest: the deploy health gate (`scripts/wait-for-healthy.sh`) copies container logs into the CI run log on a failed deploy, which is a far wider audience than "someone with a shell on the host". The other three rows all require host or container access, which already implies access to the token by other means.
@@ -105,12 +105,56 @@ A few things are worth knowing when reading the startup log:
 
 Newer cloudflared can read the token from a file (`--token-file`) instead of an argument, which would flip the last row of the table above to "No". cotel does not use it, because that row is the only one it changes: whoever can run `ps` inside the container can read `/proc/1/environ` too, so the same person reaches the same value by a path `--token-file` does not touch. It would need a tmpfs mount and a secret on disk to close nothing. Getting the token out of the container altogether means [local-config mode](./cloudflare-tunnel-local.md), where the credentials are a host-mounted file.
 
+## Keeping the tunnel up
+
+cloudflared does not run as the container's main process — cotel does — so for
+a long time a dead tunnel was invisible and permanent. It cost 11 h 39 min of
+telemetry on 2026-10-14: the host rebooted, cloudflared started before the
+host's resolver was usable, and edge discovery failed on the one DNS query it
+cannot retry past:
+
+```
+ERR edge discovery: error looking up Cloudflare edge IPs: the DNS query failed
+  error="lookup _v2-origintunneld._tcp.argotunnel.com on 127.0.0.11:53: no such host"
+INF Tunnel server stopped
+ERR Initiating shutdown
+```
+
+`restart: unless-stopped` never fired, because the process that died was not the
+one Docker watches. The container stayed `Up (healthy)` with `RestartCount` 0,
+the LAN ingest port kept answering, and every agent exporting to the public URL
+got Cloudflare 1033 until someone restarted the container by hand.
+
+Two things close that, both in this repo:
+
+- **`scripts/supervise-cloudflared.sh`** runs cloudflared and restarts it on
+  every exit, forever, waiting 1s and doubling up to `60s`
+  (`COTEL_TUNNEL_RESTART_BACKOFF_MAX`); a cloudflared that stayed up at least
+  that long resets the delay. The alternative — making the tunnel's death fatal
+  to the container so Docker's restart policy handles it — was rejected because
+  it couples a working data plane to an unreachable edge: dropping LAN ingest on
+  `:4318` over a DNS hiccup is the worse of the two failures.
+- **`/healthz` reports the tunnel.** The entrypoint points cloudflared's metrics
+  listener at `COTEL_TUNNEL_METRICS_ADDR` (`127.0.0.1:2000` by default) and
+  tells cotel where it is; `/healthz` then carries a `tunnel` object read from
+  cloudflared's `/ready`, so "the container is healthy and the tunnel is dead"
+  is a state an operator and a poller can both see. It is an added field and a
+  deliberate non-failure: see
+  [What `/healthz` returns](https://github.com/Flopsstuff/cotel/blob/main/README.md#what-healthz-returns).
+
+The supervisor is exercised by `bash scripts/supervise-cloudflared_test.sh`
+against a stub cloudflared — no Docker, no credentials, no network.
+
 ## Verifying the tunnel
 
 ```sh
-# Container logs — look for "cloudflared started in token mode"
+# Container logs — look for "cloudflared started in token mode under supervision",
+# then cloudflared's own "Registered tunnel connection" (one per edge connection)
 docker compose logs cotel
 
+# The tunnel as cotel sees it: ok plus the number of registered edge connections
+curl -s http://localhost:8080/healthz | python3 -m json.tool
+
 # Tunnel status in the Cloudflare dashboard:
 # Zero Trust → Networks → Tunnels → cotel → should show "Healthy"