fix(probe): exit 7 when observer DNS fails (do not page prod red)

docs/operations/health-probe.md

@@ -55,12 +55,13 @@ they send the on-call looking in different places:
 
 | Probe verdict | Exit | Typical cause |
 |---|---|---|
-| `unreachable` | 1 | Process not listening, host down, or the tunnel is not forwarding |
+| `unreachable` | 1 | Process not listening, host down, or the tunnel is not forwarding (name already resolved) |
 | `HTTP 503 database unreadable` | 2 | Process is up; DuckDB is not |
 | `HTTP <other>` | 2 | Crash loop, proxy error, unexpected handler |
 | `ingest stale` / `empty database` | 3 | Process is up and the DB reads; spans are not being accepted |
 | `cloudflare access blocked` | 4 | Probe config, **not** an outage — see below |
 | `no current database snapshot` | 6 | Everything above is fine; the backup is not happening — see below |
+| `observer cannot resolve name` | 7 | The **prober host** could not resolve the name — production was never asked |
 
 A 200 with `ok: true` is not enough. Staleness is a body field
 (`newest_span_age_seconds`); the endpoint keeps 200 on a quiet instance so the
@@ -139,12 +140,13 @@ ingest handler. No Access service token is involved, and none is needed.
 | Probe verdict | Exit | Typical cause |
 |---|---|---|
 | `HTTP 401 from the ingest handler` | 0 | the public ingest path is up |
-| `public ingest unreachable` | 1 | DNS gone, connection refused, timeout — a blanked tunnel token looks like this |
+| `public ingest unreachable` | 1 | connection refused or timeout after resolve — a blanked tunnel token looks like this |
 | `cloudflare reached, the origin did not answer` | 2 | 502/530 and friends: the tunnel is up, the origin is not |
 | `/v1/traces is not routed` | 2 | 404 — wrong tunnel target, or the ingest port is not served |
 | `accepting unauthenticated spans` | 2 | 200 to an invalid token: reachable, but the auth policy is open |
 | `401 but not from the ingest handler` | 2 | a 401 whose body is not the application's JSON — an interstitial, not cotel |
 | `cloudflare access now fronts the ingest host` | 4 | Access was enabled on this host, which rejects every agent's spans too |
+| `observer cannot resolve name` | 7 | The **prober host** could not resolve the name — public ingest was never asked |
 
 Two deliberate narrownesses:
 
@@ -177,6 +179,15 @@ different people doing different things, so they never share an alert:
   reader to look at the tunnel and DNS rather than restart the container. The
   pager takes those from `PC_ALERT_SUBJECT` and `PC_ALERT_LEAD`.
 
+Exit **7** is a third class: the observer could not resolve the name, so
+production was never sampled. Collapsing it into exit 1 used to burn the prod
+streak and page "cotel prod /healthz is red" while a real public-ingest outage
+stayed invisible behind the resolver failure. The Pi wrapper therefore pages
+exit 7 under its own markers (`[cotel-observer-dns]` /
+`[cotel-observer-dns-edge]`), with their own streaks, and does **not** suppress
+the other half — a blind LAN half must not hide a red edge half. The alert text
+says production state is unknown and asks for a check from another machine.
+
 The LAN half needs the same split **inside** its one marker, because it reddens
 on two unrelated things: a `/healthz` that does not answer, and a `/healthz`
 that answers while the backup has stopped. So when the caller passes no
@@ -264,8 +275,9 @@ journalctl --user -u cotel-healthz.service -n 50
 ```
 
 `--probe-only` exits with the LAN probe's own code when that half is red
-(1 unreachable, 2 HTTP, 3 stale/empty), **5** when the LAN half is green and the
-public ingest half is not, and 0 when both are green. The sentinel is not the
+(1 unreachable, 2 HTTP, 3 stale/empty, 6 snapshot, 7 observer DNS), **5** when
+the LAN half is green and the public ingest half is not, and 0 when both are
+green. The sentinel is not the
 edge probe's own code on purpose: 1 and 2 would read as the LAN half's
 "unreachable" and "HTTP", which send the reader to the host instead of to
 Cloudflare.
@@ -701,8 +713,9 @@ bash scripts/page-cotel-health_test.sh
 ```
 
 `probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200,
-`3` stale/empty, `4` Access blocked, `6` no current database snapshot.
+`3` stale/empty, `4` Access blocked, `6` no current database snapshot,
+`7` observer cannot resolve name.
 
 `probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401,
 `1` unreachable, `2` answered but not with the handler's 401, `4` Access now
-fronts the ingest host.
+fronts the ingest host, `7` observer cannot resolve name.

scripts/probe-edge-ingest.sh

@@ -28,10 +28,14 @@
 #
 # Exit codes:
 #   0  the public ingest path is up (the application answered 401)
-#   1  unreachable (DNS, refused, timeout) — tunnel down or host gone
+#   1  unreachable (refused, timeout after resolve) — tunnel down or host gone
 #   2  something answered, but it was not the ingest handler's 401
 #   4  Cloudflare Access is in front of the ingest host — which also rejects
 #      every agent's spans, so it is an outage, not a probe-config problem
+#   7  this host could not resolve the name (public ingest was never asked)
+#
+# 7 is separate from 1 so ~/ops/cotel-healthz.sh can page an observer-resolver
+# alert without burning the public-ingest streak.
 
 set -euo pipefail
 
@@ -71,15 +75,33 @@ set -e
 
 if [ "$CURL_RC" -ne 0 ]; then
     err="$(tr '\n' ' ' <"$ERR" | sed 's/[[:space:]]*$//')"
+    exit_code=1
     case "$CURL_RC" in
-        6)  why="dns lookup failed" ;;
+        6)
+            why="dns lookup failed"
+            exit_code=7
+            ;;
         7)  why="connection refused" ;;
-        28) why="timed out after ${CONNECT_TIMEOUT}s" ;;
+        28)
+            case "$err" in
+                *[Rr]esolv*)
+                    why="dns resolve timed out after ${CONNECT_TIMEOUT}s"
+                    exit_code=7
+                    ;;
+                *)
+                    why="timed out after ${CONNECT_TIMEOUT}s"
+                    ;;
+            esac
+            ;;
         35|60) why="tls handshake failed" ;;
         *)  why="curl exit ${CURL_RC}" ;;
     esac
-    echo "probe-edge-ingest: FAILED — public ingest unreachable (${why}${err:+; ${err}}) url=${URL}"
-    exit 1
+    if [ "$exit_code" -eq 7 ]; then
+        echo "probe-edge-ingest: FAILED — observer cannot resolve name (${why}${err:+; ${err}}) url=${URL}"
+    else
+        echo "probe-edge-ingest: FAILED — public ingest unreachable (${why}${err:+; ${err}}) url=${URL}"
+    fi
+    exit "$exit_code"
 fi
 
 location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")"

scripts/probe-edge-ingest_test.sh

@@ -141,6 +141,11 @@ expect "connection refused is unreachable" 1 "public ingest unreachable" \
 expect "timeout is unreachable" 1 "public ingest unreachable" \
     env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/v1/traces"
 
+# A name that does not exist: curl rc 6. Must not share exit 1 with "ingest down".
+# no_proxy so an environment-configured proxy cannot resolve the name for curl.
+expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \
+    env CONNECT_TIMEOUT=2 NO_PROXY='*' no_proxy='*' "$PROBE" "http://no-such-host.invalid./v1/traces"
+
 expect "a non-numeric timeout is rejected, not silently defaulted" 2 "CONNECT_TIMEOUT must be a positive integer" \
     env CONNECT_TIMEOUT=soon "$PROBE" "$BASE/v1/traces"
 

scripts/probe-healthz.sh

@@ -30,15 +30,17 @@
 #
 # Exit codes:
 #   0  healthy
-#   1  unreachable (timeout, refused, DNS)
+#   1  unreachable (timeout, refused — after the name resolved)
 #   2  HTTP non-200
 #   3  200 but ingest is stale or the database has never accepted a span
 #   4  Cloudflare Access blocked the probe (cannot see /healthz)
 #   6  /healthz is green but the database has no current snapshot
+#   7  this host could not resolve the name (prod was never asked)
 #
 # 6 and not 5: ~/ops/cotel-healthz.sh already spends 5 on "the LAN half is
 # green and the public ingest half is not", and passes a probe's own code
-# through otherwise.
+# through otherwise. 7 is separate from 1 so the host wrapper can page an
+# observer-resolver alert without burning the prod /healthz streak.
 
 set -euo pipefail
 
@@ -129,14 +131,32 @@ set -e
 
 if [ "$CURL_RC" -ne 0 ]; then
     err="$(tr '\n' ' ' <"$ERR" | sed 's/[[:space:]]*$//')"
+    exit_code=1
     case "$CURL_RC" in
-        6)  why="dns lookup failed" ;;
+        6)
+            why="dns lookup failed"
+            exit_code=7
+            ;;
         7)  why="connection refused" ;;
-        28) why="timed out after ${CONNECT_TIMEOUT}s" ;;
+        28)
+            case "$err" in
+                *[Rr]esolv*)
+                    why="dns resolve timed out after ${CONNECT_TIMEOUT}s"
+                    exit_code=7
+                    ;;
+                *)
+                    why="timed out after ${CONNECT_TIMEOUT}s"
+                    ;;
+            esac
+            ;;
         *)  why="curl exit ${CURL_RC}" ;;
     esac
-    echo "probe-healthz: FAILED — unreachable (${why}${err:+; ${err}}) url=${URL}"
-    exit 1
+    if [ "$exit_code" -eq 7 ]; then
+        echo "probe-healthz: FAILED — observer cannot resolve name (${why}${err:+; ${err}}) url=${URL}"
+    else
+        echo "probe-healthz: FAILED — unreachable (${why}${err:+; ${err}}) url=${URL}"
+    fi
+    exit "$exit_code"
 fi
 
 if access_blocked "$HDRS" "$HTTP_CODE"; then

scripts/probe-healthz_test.sh

@@ -239,6 +239,11 @@ expect "connection refused is unreachable" 1 "unreachable" \
 expect "timeout is unreachable" 1 "unreachable" \
     env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/healthz"
 
+# A name that does not exist: curl rc 6. Must not share exit 1 with "prod down".
+# no_proxy so an environment-configured proxy cannot resolve the name for curl.
+expect "unresolvable name is observer dns, not unreachable" 7 "observer cannot resolve name" \
+    env CONNECT_TIMEOUT=2 NO_PROXY='*' no_proxy='*' "$PROBE" "http://no-such-host.invalid./healthz"
+
 # ---- the snapshot claim, read from /api/v1/health on a green /healthz ----
 
 expect "a fresh snapshot is reported on the green line" 0 "snapshot last run" \