From 20ab24e0e68fbd5bb6c9b1305f66fa45678c6229 Mon Sep 17 00:00:00 2001 From: Soren Date: Wed, 14 Oct 2026 23:26:14 +0200 Subject: [PATCH 1/2] fix(probe): exit 7 when the observer cannot resolve the name DNS failure on the prober used to share exit 1 with a dead prod host, so the Pi timer burned the prod streak and paged the wrong alert while public ingest stayed unwatched. Classify curl rc 6 (and resolve timeouts) as exit 7. Co-Authored-By: Soren Co-Authored-By: Cursor Composer 2.5 --- docs/operations/health-probe.md | 23 +++++++++++++++++----- scripts/probe-edge-ingest.sh | 32 ++++++++++++++++++++++++++----- scripts/probe-edge-ingest_test.sh | 4 ++++ scripts/probe-healthz.sh | 32 +++++++++++++++++++++++++------ scripts/probe-healthz_test.sh | 4 ++++ 5 files changed, 79 insertions(+), 16 deletions(-) diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index e9fe153..97ed77e 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -55,12 +55,13 @@ they send the on-call looking in different places: | Probe verdict | Exit | Typical cause | |---|---|---| -| `unreachable` | 1 | Process not listening, host down, or the tunnel is not forwarding | +| `unreachable` | 1 | Process not listening, host down, or the tunnel is not forwarding (name already resolved) | | `HTTP 503 database unreadable` | 2 | Process is up; DuckDB is not | | `HTTP ` | 2 | Crash loop, proxy error, unexpected handler | | `ingest stale` / `empty database` | 3 | Process is up and the DB reads; spans are not being accepted | | `cloudflare access blocked` | 4 | Probe config, **not** an outage — see below | | `no current database snapshot` | 6 | Everything above is fine; the backup is not happening — see below | +| `observer cannot resolve name` | 7 | The **prober host** could not resolve the name — production was never asked | A 200 with `ok: true` is not enough. Staleness is a body field (`newest_span_age_seconds`); the endpoint keeps 200 on a quiet instance so the @@ -139,12 +140,13 @@ ingest handler. No Access service token is involved, and none is needed. | Probe verdict | Exit | Typical cause | |---|---|---| | `HTTP 401 from the ingest handler` | 0 | the public ingest path is up | -| `public ingest unreachable` | 1 | DNS gone, connection refused, timeout — a blanked tunnel token looks like this | +| `public ingest unreachable` | 1 | connection refused or timeout after resolve — a blanked tunnel token looks like this | | `cloudflare reached, the origin did not answer` | 2 | 502/530 and friends: the tunnel is up, the origin is not | | `/v1/traces is not routed` | 2 | 404 — wrong tunnel target, or the ingest port is not served | | `accepting unauthenticated spans` | 2 | 200 to an invalid token: reachable, but the auth policy is open | | `401 but not from the ingest handler` | 2 | a 401 whose body is not the application's JSON — an interstitial, not cotel | | `cloudflare access now fronts the ingest host` | 4 | Access was enabled on this host, which rejects every agent's spans too | +| `observer cannot resolve name` | 7 | The **prober host** could not resolve the name — public ingest was never asked | Two deliberate narrownesses: @@ -177,6 +179,15 @@ different people doing different things, so they never share an alert: reader to look at the tunnel and DNS rather than restart the container. The pager takes those from `PC_ALERT_SUBJECT` and `PC_ALERT_LEAD`. +Exit **7** is a third class: the observer could not resolve the name, so +production was never sampled. Collapsing it into exit 1 used to burn the prod +streak and page "cotel prod /healthz is red" while a real public-ingest outage +stayed invisible behind the resolver failure. The Pi wrapper therefore pages +exit 7 under its own markers (`[cotel-observer-dns]` / +`[cotel-observer-dns-edge]`), with their own streaks, and does **not** suppress +the other half — a blind LAN half must not hide a red edge half. The alert text +says production state is unknown and asks for a check from another machine. + The LAN half needs the same split **inside** its one marker, because it reddens on two unrelated things: a `/healthz` that does not answer, and a `/healthz` that answers while the backup has stopped. So when the caller passes no @@ -264,8 +275,9 @@ journalctl --user -u cotel-healthz.service -n 50 ``` `--probe-only` exits with the LAN probe's own code when that half is red -(1 unreachable, 2 HTTP, 3 stale/empty), **5** when the LAN half is green and the -public ingest half is not, and 0 when both are green. The sentinel is not the +(1 unreachable, 2 HTTP, 3 stale/empty, 6 snapshot, 7 observer DNS), **5** when +the LAN half is green and the public ingest half is not, and 0 when both are +green. The sentinel is not the edge probe's own code on purpose: 1 and 2 would read as the LAN half's "unreachable" and "HTTP", which send the reader to the host instead of to Cloudflare. @@ -701,7 +713,8 @@ bash scripts/page-cotel-health_test.sh ``` `probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, -`3` stale/empty, `4` Access blocked, `6` no current database snapshot. +`3` stale/empty, `4` Access blocked, `6` no current database snapshot, +`7` observer cannot resolve name. `probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401, `1` unreachable, `2` answered but not with the handler's 401, `4` Access now diff --git a/scripts/probe-edge-ingest.sh b/scripts/probe-edge-ingest.sh index 2f845c1..9264a0e 100755 --- a/scripts/probe-edge-ingest.sh +++ b/scripts/probe-edge-ingest.sh @@ -28,10 +28,14 @@ # # Exit codes: # 0 the public ingest path is up (the application answered 401) -# 1 unreachable (DNS, refused, timeout) — tunnel down or host gone +# 1 unreachable (refused, timeout after resolve) — tunnel down or host gone # 2 something answered, but it was not the ingest handler's 401 # 4 Cloudflare Access is in front of the ingest host — which also rejects # every agent's spans, so it is an outage, not a probe-config problem +# 7 this host could not resolve the name (public ingest was never asked) +# +# 7 is separate from 1 so ~/ops/cotel-healthz.sh can page an observer-resolver +# alert without burning the public-ingest streak. set -euo pipefail @@ -71,15 +75,33 @@ set -e if [ "$CURL_RC" -ne 0 ]; then err="$(tr '\n' ' ' <"$ERR" | sed 's/[[:space:]]*$//')" + exit_code=1 case "$CURL_RC" in - 6) why="dns lookup failed" ;; + 6) + why="dns lookup failed" + exit_code=7 + ;; 7) why="connection refused" ;; - 28) why="timed out after ${CONNECT_TIMEOUT}s" ;; + 28) + case "$err" in + *[Rr]esolv*) + why="dns resolve timed out after ${CONNECT_TIMEOUT}s" + exit_code=7 + ;; + *) + why="timed out after ${CONNECT_TIMEOUT}s" + ;; + esac + ;; 35|60) why="tls handshake failed" ;; *) why="curl exit ${CURL_RC}" ;; esac - echo "probe-edge-ingest: FAILED — public ingest unreachable (${why}${err:+; ${err}}) url=${URL}" - exit 1 + if [ "$exit_code" -eq 7 ]; then + echo "probe-edge-ingest: FAILED — observer cannot resolve name (${why}${err:+; ${err}}) url=${URL}" + else + echo "probe-edge-ingest: FAILED — public ingest unreachable (${why}${err:+; ${err}}) url=${URL}" + fi + exit "$exit_code" fi location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" diff --git a/scripts/probe-edge-ingest_test.sh b/scripts/probe-edge-ingest_test.sh index e74f602..51e14eb 100755 --- a/scripts/probe-edge-ingest_test.sh +++ b/scripts/probe-edge-ingest_test.sh @@ -141,6 +141,10 @@ expect "connection refused is unreachable" 1 "public ingest unreachable" \ expect "timeout is unreachable" 1 "public ingest unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/v1/traces" +# A name that does not exist: curl rc 6. Must not share exit 1 with "ingest down". +expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \ + env CONNECT_TIMEOUT=2 "$PROBE" "http://no-such-host.invalid./v1/traces" + expect "a non-numeric timeout is rejected, not silently defaulted" 2 "CONNECT_TIMEOUT must be a positive integer" \ env CONNECT_TIMEOUT=soon "$PROBE" "$BASE/v1/traces" diff --git a/scripts/probe-healthz.sh b/scripts/probe-healthz.sh index 49b53d1..ad0c117 100755 --- a/scripts/probe-healthz.sh +++ b/scripts/probe-healthz.sh @@ -30,15 +30,17 @@ # # Exit codes: # 0 healthy -# 1 unreachable (timeout, refused, DNS) +# 1 unreachable (timeout, refused — after the name resolved) # 2 HTTP non-200 # 3 200 but ingest is stale or the database has never accepted a span # 4 Cloudflare Access blocked the probe (cannot see /healthz) # 6 /healthz is green but the database has no current snapshot +# 7 this host could not resolve the name (prod was never asked) # # 6 and not 5: ~/ops/cotel-healthz.sh already spends 5 on "the LAN half is # green and the public ingest half is not", and passes a probe's own code -# through otherwise. +# through otherwise. 7 is separate from 1 so the host wrapper can page an +# observer-resolver alert without burning the prod /healthz streak. set -euo pipefail @@ -129,14 +131,32 @@ set -e if [ "$CURL_RC" -ne 0 ]; then err="$(tr '\n' ' ' <"$ERR" | sed 's/[[:space:]]*$//')" + exit_code=1 case "$CURL_RC" in - 6) why="dns lookup failed" ;; + 6) + why="dns lookup failed" + exit_code=7 + ;; 7) why="connection refused" ;; - 28) why="timed out after ${CONNECT_TIMEOUT}s" ;; + 28) + case "$err" in + *[Rr]esolv*) + why="dns resolve timed out after ${CONNECT_TIMEOUT}s" + exit_code=7 + ;; + *) + why="timed out after ${CONNECT_TIMEOUT}s" + ;; + esac + ;; *) why="curl exit ${CURL_RC}" ;; esac - echo "probe-healthz: FAILED — unreachable (${why}${err:+; ${err}}) url=${URL}" - exit 1 + if [ "$exit_code" -eq 7 ]; then + echo "probe-healthz: FAILED — observer cannot resolve name (${why}${err:+; ${err}}) url=${URL}" + else + echo "probe-healthz: FAILED — unreachable (${why}${err:+; ${err}}) url=${URL}" + fi + exit "$exit_code" fi if access_blocked "$HDRS" "$HTTP_CODE"; then diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index 9b55ae6..0a1fd6a 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -239,6 +239,10 @@ expect "connection refused is unreachable" 1 "unreachable" \ expect "timeout is unreachable" 1 "unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/healthz" +# A name that does not exist: curl rc 6. Must not share exit 1 with "prod down". +expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \ + env CONNECT_TIMEOUT=2 "$PROBE" "http://no-such-host.invalid./healthz" + # ---- the snapshot claim, read from /api/v1/health on a green /healthz ---- expect "a fresh snapshot is reported on the green line" 0 "snapshot last run" \ From 77a9efc9f462c59bcc1bff0c5b38828400f79fb7 Mon Sep 17 00:00:00 2001 From: Daedalus Date: Wed, 14 Oct 2026 23:33:59 +0200 Subject: [PATCH 2/2] docs(probe): list edge exit 7, pin no_proxy in the unresolved-name tests The edge exit-code summary omitted 7 while its verdict table listed it, so the two operator references disagreed. The unresolved-name tests now clear the proxy variables: with a proxy configured curl hands the name to it and never returns rc 6, which fails the assertion for a reason unrelated to the classification under test. Co-Authored-By: Daedalus Co-Authored-By: Claude Opus 4.8 --- docs/operations/health-probe.md | 2 +- scripts/probe-edge-ingest_test.sh | 3 ++- scripts/probe-healthz_test.sh | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 97ed77e..02884a5 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -718,4 +718,4 @@ bash scripts/page-cotel-health_test.sh `probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401, `1` unreachable, `2` answered but not with the handler's 401, `4` Access now -fronts the ingest host. +fronts the ingest host, `7` observer cannot resolve name. diff --git a/scripts/probe-edge-ingest_test.sh b/scripts/probe-edge-ingest_test.sh index 51e14eb..ca42a97 100755 --- a/scripts/probe-edge-ingest_test.sh +++ b/scripts/probe-edge-ingest_test.sh @@ -142,8 +142,9 @@ expect "timeout is unreachable" 1 "public ingest unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/v1/traces" # A name that does not exist: curl rc 6. Must not share exit 1 with "ingest down". +# no_proxy so an environment-configured proxy cannot resolve the name for curl. expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \ - env CONNECT_TIMEOUT=2 "$PROBE" "http://no-such-host.invalid./v1/traces" + env CONNECT_TIMEOUT=2 NO_PROXY='*' no_proxy='*' "$PROBE" "http://no-such-host.invalid./v1/traces" expect "a non-numeric timeout is rejected, not silently defaulted" 2 "CONNECT_TIMEOUT must be a positive integer" \ env CONNECT_TIMEOUT=soon "$PROBE" "$BASE/v1/traces" diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index 0a1fd6a..87464e4 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -240,8 +240,9 @@ expect "timeout is unreachable" 1 "unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/healthz" # A name that does not exist: curl rc 6. Must not share exit 1 with "prod down". +# no_proxy so an environment-configured proxy cannot resolve the name for curl. expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \ - env CONNECT_TIMEOUT=2 "$PROBE" "http://no-such-host.invalid./healthz" + env CONNECT_TIMEOUT=2 NO_PROXY='*' no_proxy='*' "$PROBE" "http://no-such-host.invalid./healthz" # ---- the snapshot claim, read from /api/v1/health on a green /healthz ----