diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index 2c32bbc..84d2b1e 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -32,6 +32,8 @@ on: paths: - "scripts/probe-healthz.sh" - "scripts/probe-healthz_test.sh" + - "scripts/probe-edge-ingest.sh" + - "scripts/probe-edge-ingest_test.sh" - "scripts/page-cotel-health.sh" - "scripts/page-cotel-health_test.sh" - ".github/workflows/health-probe.yml" @@ -48,6 +50,8 @@ jobs: - uses: actions/checkout@v5 - name: Run probe-healthz tests run: bash scripts/probe-healthz_test.sh + - name: Run probe-edge-ingest tests + run: bash scripts/probe-edge-ingest_test.sh - name: Run pager tests run: bash scripts/page-cotel-health_test.sh diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 55448fc..7514be0 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -1,35 +1,38 @@ -# Production /healthz probe +# Production health probes -A **systemd timer on the Pi** asks the **dashboard** `/healthz` on the -production instance every 10 minutes and pages a Paperclip agent once two -consecutive ticks fail. It exists because a dead or silent cotel otherwise has -no one looking: `Deploy` only runs on a push to `main`, the Cloudflare tunnel -stays green while the process is 503 or not listening, and a red Actions run in -this company does not wake anyone on its own. +A **systemd timer on the Pi** asks production two questions every 10 minutes and +pages a Paperclip agent once two consecutive ticks fail either of them: + +- **the LAN half** — the **dashboard** `/healthz` on robmini, over the LAN; +- **the edge half** — the **public OTLP ingest URL**, over the internet. + +They exist because a dead or silent cotel otherwise has no one looking: `Deploy` +only runs on a push to `main`, the Cloudflare tunnel stays green while the +process is 503 or not listening, and a red Actions run in this company does not +wake anyone on its own. The scheduler used to be GitHub Actions `cron`, and that is the part that did not work: see [Why the schedule is not in this repo](#why-the-schedule-is-not-in-this-repo) and [ADR-0022](../decisions/0022-health-probe-scheduler-outside-github.md). The -workflow still exists, dispatch-only, for drills and for the one vantage point -outside the LAN. - -## The three vantage points - -The same endpoint is asked from three places, because each is blind to -something the others see. - -| | `cotel-healthz.timer` (the Pi) | `probe-loopback` (dispatch) | `probe-edge` (dispatch) | -|---|---|---|---| -| Scheduled | **yes, every 10 min** | no | no | -| Runner | the Pi, `~/ops/cotel-healthz.sh` | `[self-hosted, flopsstuff, docker]` — the deploy host | `ubuntu-latest` | -| Target | `http://robmini.local:8080/healthz` (LAN) | `http://127.0.0.1:8080/healthz` | `https://cotel.aignite.pl/healthz` | -| Cloudflare in the path | no | no | yes (tunnel + Access) | -| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, **and the host being off** | all of that except host-off | all of that **plus** tunnel down, DNS, Access misconfigured | -| Blind to | the tunnel, DNS and Access — a LAN-healthy cotel unreachable from the internet reads green | **its own host being down** — with the runner off the job queues, producing no colour at all | nothing in the path, but see the Access caveat below | -| Access service token | not needed | not needed | required | -| Alert dedup marker | `[cotel-health-probe]` | `[cotel-health-probe-drill]` | `[cotel-health-probe-drill-edge]` | - -The Pi timer is the authoritative path, and it is the only one that pages the +workflow still exists, dispatch-only, for drills. + +## The vantage points + +Production is asked from several places, because each is blind to something the +others see. The first two columns are the scheduled halves of one tick. + +| | `cotel-healthz.timer` LAN half | `cotel-healthz.timer` edge half | `probe-loopback` (dispatch) | `probe-edge` (dispatch) | +|---|---|---|---|---| +| Scheduled | **yes, every 10 min** | **yes, every 10 min** | no | no | +| Runner | the Pi, `~/ops/cotel-healthz.sh` | the Pi, same tick | `[self-hosted, flopsstuff, docker]` — the deploy host | `ubuntu-latest` | +| Target | `http://robmini.local:8080/healthz` (LAN) | `https://otlp.aignite.pl/v1/traces` (internet) | `http://127.0.0.1:8080/healthz` | `https://cotel.aignite.pl/healthz` | +| Cloudflare in the path | no | yes (DNS + tunnel) | no | yes (tunnel + Access) | +| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, **and the host being off** | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | all of the LAN half except host-off | the dashboard host through Cloudflare | +| Blind to | the tunnel, DNS — a LAN-healthy cotel unreachable from the internet reads green | the database and ingest freshness: this check stops at the auth boundary | **its own host being down** — with the runner off the job queues, producing no colour at all | nothing in the path, but see the Access caveat below | +| Access service token | not needed | **not needed** | not needed | required | +| Alert dedup marker | `[cotel-health-probe]` | `[cotel-ingest-edge]` | `[cotel-health-probe-drill]` | `[cotel-health-probe-drill-edge]` | + +The Pi timer is the authoritative path, and it is the only one that pages a production marker. Probing over the LAN rather than from the deploy host closes the gap that the self-hosted job cannot: a job on that host cannot report the host being off, because GitHub queues the run rather than failing it, and @@ -39,23 +42,16 @@ trace is a run sitting in `queued`, which is also what a busy runner looks like. From the Pi, an absent host is a refused connection or a timeout, which is a verdict. -What the Pi cannot see is everything Cloudflare adds: tunnel, DNS and Access. -That half is unscheduled today, and it is also **not observing anything** — the -Access service token is not allowed on the application, so the edge probe -reports exit 4 (see the Access caveat below). Allowing that token is what would -make an edge schedule worth having; until then an hourly edge run would watch a -login redirect. - -The three use **different dedup markers** on purpose. On a single marker they -would fight: a green tick on one would ask for the close of the alert another -had just raised. Dispatched runs are separated from the Pi timer for the +Every prober uses a **different dedup marker** on purpose. On a single marker +they would fight: a green tick on one would ask for the close of the alert +another had just raised. Dispatched runs are separated from the Pi timer for the stronger version of the same reason — a drill must not touch a real alert; see [Running a drill](#running-a-drill). -## What the probe checks +## What the LAN half checks -The probe classifies failure modes with distinct text, because they send the -on-call looking in different places: +`scripts/probe-healthz.sh` classifies failure modes with distinct text, because +they send the on-call looking in different places: | Probe verdict | Exit | Typical cause | |---|---|---| @@ -75,13 +71,76 @@ If `last_ingest_at` / `newest_span_age_seconds` are absent, the probe degrades to liveness (HTTP 200) so it keeps working before that contract is on production. -### The Access caveat (why the edge half warns instead of failing) +## What the edge half checks, and why it needs no Access token + +`scripts/probe-edge-ingest.sh` asks `https://otlp.aignite.pl/v1/traces` — the +endpoint every agent exports to — and expects **exactly HTTP 401**. + +The 401 is the whole point. The ingest host is **not** behind Cloudflare +Access, so an unauthenticated request reaches cotel's own auth middleware, and +only a working chain can produce that answer: DNS resolving, the tunnel +forwarding, the process listening, and `/v1/traces` routed to the auth-wrapped +ingest handler. No Access service token is involved, and none is needed. + +| Probe verdict | Exit | Typical cause | +|---|---|---| +| `HTTP 401 from the ingest handler` | 0 | the public ingest path is up | +| `public ingest unreachable` | 1 | DNS gone, connection refused, timeout — a blanked tunnel token looks like this | +| `cloudflare reached, the origin did not answer` | 2 | 502/530 and friends: the tunnel is up, the origin is not | +| `/v1/traces is not routed` | 2 | 404 — wrong tunnel target, or the ingest port is not served | +| `accepting unauthenticated spans` | 2 | 200 to an invalid token: reachable, but the auth policy is open | +| `401 but not from the ingest handler` | 2 | a 401 whose body is not the application's JSON — an interstitial, not cotel | +| `cloudflare access now fronts the ingest host` | 4 | Access was enabled on this host, which rejects every agent's spans too | + +Two deliberate narrownesses: + +- **The expected status is a single code, not "any answer".** Accepting any + response would let a Cloudflare interstitial or a parked-domain page pass as + health, which is the failure this half exists to catch. +- **The request carries a deliberately invalid `cotel_` bearer**, not no bearer + at all. The auth middleware rejects an unknown token before the handler sees + the request, so the expected 401 holds whichever way `allow_anonymous` is set + on the instance; with anonymous ingest allowed, a tokenless request would + reach the handler and answer 405. It is a `GET`, so even a bypassed auth + check could not write anything. + +This half is blind to what the LAN half sees: it stops at the auth boundary and +says nothing about the database or ingest freshness. That is why both run. + +### The two halves are classified apart, and page apart + +"The process is dead" and "the process is fine, the public path is down" need +different people doing different things, so they never share an alert: + +- **separate dedup markers** — `[cotel-health-probe]` and `[cotel-ingest-edge]`. + On one marker the halves would fight, a green tick of one asking for the close + of the alert the other just raised; +- **separate failure streaks** — `reports/cotel-healthz/state` and + `state-edge`, so this host's uplink blinking cannot page for a healthy + application and vice versa; +- **different words** — the edge alert's title, first line and wake reason say + that the application is alive and its public ingest path is not, and tell the + reader to look at the tunnel and DNS rather than restart the container. The + pager takes those from `PC_ALERT_SUBJECT` and `PC_ALERT_LEAD`. + +One asymmetry on purpose: **while the LAN half is red, the edge half does not +page.** A dead process makes the public path unreachable as a consequence, and a +second alert would send its reader hunting Cloudflare for a dead container. The +edge streak keeps counting through the suppression, so the moment the process +comes back and the public path does not, that half pages on the next tick. The +tick log names the suppression explicitly: + +``` +[…] edge RED (streak 3), not paging: the LAN /healthz half is red too, and its alert covers this — probe-edge-ingest: FAILED — … +``` -Cloudflare Access sits in front of the dashboard. The workflow sends the same -Access service-token headers as `paperclip-issue-sync.yml`, but that token is -**not currently allowed on the `cotel.aignite.pl` application**: the request -comes back as a 302 to the Access login, exactly as it does with no token at -all. +### The Access caveat (the dispatched dashboard-edge job) + +Cloudflare Access sits in front of the **dashboard** host, `cotel.aignite.pl`. +The dispatch-only `probe-edge` job sends the same Access service-token headers +as `paperclip-issue-sync.yml`, but that token is **not allowed on that +application**: the request comes back as a 302 to the Access login, exactly as +it does with no token at all. So `probe-edge` reports exit 4 as a **warning, and the job stays green**. That is deliberate. Failing on exit 4 would paint the workflow red on every run until @@ -89,36 +148,56 @@ the token is allowed, and a probe that is always red is one nobody reads — worse than no probe, because it also buries a genuine red. Exit 4 never pages either: it says nothing about whether cotel is up. -The price of that choice: while the token is unauthorized, **the edge half is -not watching anything**, and the only thing saying so is a warning annotation -on an otherwise-green run. To switch it on, allow the service token on the -`cotel.aignite.pl` Access application (or bypass `/healthz` only). No code -change is needed — the half starts observing on the next scheduled hour. +**This caveat is not a gap in the scheduled coverage, and a service token is not +what would close one.** That was the earlier reading of it — that a meaningful +check from outside the LAN had to wait for the token to be allowed on +`cotel.aignite.pl`. It does not: the ingest host answers 401 to anyone, and that +401 proves DNS, tunnel, process and route, which is exactly what the LAN half +cannot see. The scheduled edge half above uses it and asks for no credential. +What an Access-allowed token would add is narrower: the *dashboard* host's view +through Cloudflare, and the database and freshness fields that only `/healthz` +carries. Worth having, not worth blocking on. ## Schedule Every **10 minutes**, from `cotel-healthz.timer` on the Pi -(`OnCalendar=*:0/10`, `Persistent=true`). It pages on the **second consecutive** -failure, so the worst-case detection delay is about 20 minutes and a single -blinked tick wakes nobody. The 6h ingest-age threshold is independent of the -poll interval: a quiet night is not an alert. +(`OnCalendar=*:0/10`, `Persistent=true`). Both halves run in one tick, each +paging on its own **second consecutive** failure, so the worst-case detection +delay is about 20 minutes and a single blinked tick wakes nobody. The 6h +ingest-age threshold is independent of the poll interval: a quiet night is not +an alert. The timer, its unit files and its runbook live in `~/ops` on the Pi (`~/ops/README.md`, section *cotel health probe*) — not in this repo, because a scheduler inside the repo is exactly what did not work. The scripts it runs -**are** this repo's: each tick materializes `scripts/probe-healthz.sh` and -`scripts/page-cotel-health.sh` from `origin/main` with `git show`, so a fix here -reaches the timer with no sync step and the timer does not care which branch the -shared checkout has yanked. +**are** this repo's: each tick materializes `scripts/probe-healthz.sh`, +`scripts/probe-edge-ingest.sh` and `scripts/page-cotel-health.sh` from +`origin/main` with `git show`, so a fix here reaches the timer with no sync step +and the timer does not care which branch the shared checkout has yanked. + +One consequence of materializing by ref: a probe script that is **not at that +ref** is reported as a half that is not deployed, on every tick and in +`--status`, and does not fail the unit. The LAN half must not go dark because +the other half has not landed — and the half starts running by itself on the +tick after the script reaches `origin/main`, with no action on the host. ```sh # on the Pi -~/ops/cotel-healthz.sh --status # streak, credential, next elapse, last 20 ticks -~/ops/cotel-healthz.sh --probe-only # probe now, page nothing, touch no state +~/ops/cotel-healthz.sh --status # both streaks, credential, next elapse, last 20 ticks +~/ops/cotel-healthz.sh --probe-only # probe both halves now, page nothing, touch no state +~/ops/cotel-healthz.sh --half edge --probe-only # the public ingest half alone +~/ops/cotel-healthz.sh --half edge # one real tick of that half alone systemctl --user list-timers cotel-healthz.timer journalctl --user -u cotel-healthz.service -n 50 ``` +`--probe-only` exits with the LAN probe's own code when that half is red +(1 unreachable, 2 HTTP, 3 stale/empty), **5** when the LAN half is green and the +public ingest half is not, and 0 when both are green. The sentinel is not the +edge probe's own code on purpose: 1 and 2 would read as the LAN half's +"unreachable" and "HTTP", which send the reader to the host instead of to +Cloudflare. + A tick exits non-zero **only when the watcher itself is broken** (no credential, the scripts cannot be materialized, the pager cannot reach Paperclip). Production being red is a successful tick — it pages Paperclip. That @@ -157,8 +236,8 @@ the classification and pager tests remain a merge gate. ## Running a drill -Every rehearsal pages a **drill** marker, never the production one. Only the Pi -timer uses `cotel-health-probe`; the workflow's jobs pin +Every rehearsal pages a **drill** marker, never a production one. Only the Pi +timer uses `cotel-health-probe` and `cotel-ingest-edge`; the workflow's jobs pin `cotel-health-probe-drill` and `cotel-health-probe-drill-edge` unconditionally, and a drill from the Pi has to pass the drill marker itself: @@ -171,10 +250,22 @@ PC_ORIGIN_ID=cotel-health-probe-drill COTEL_HEALTHZ_URL=http://127.0.0.1:9/healt PC_ORIGIN_ID=cotel-health-probe-drill ~/ops/cotel-healthz.sh # green: routes the close ``` +The edge half drills the same way, with its own marker variable and `--half +edge` so the rehearsal does not also run a real LAN tick: + +```sh +# on the Pi — a red pair against a closed port, on the ingest drill marker +PC_EDGE_ORIGIN_ID=cotel-ingest-edge-drill COTEL_INGEST_URL=http://127.0.0.1:9/v1/traces \ + ~/ops/cotel-healthz.sh --half edge # first tick: streak 1, pages nobody +PC_EDGE_ORIGIN_ID=cotel-ingest-edge-drill COTEL_INGEST_URL=http://127.0.0.1:9/v1/traces \ + ~/ops/cotel-healthz.sh --half edge # second tick: raises the drill alert +PC_EDGE_ORIGIN_ID=cotel-ingest-edge-drill ~/ops/cotel-healthz.sh --half edge # green: routes the close +``` + Two things to put back afterwards: the failure streak in -`~/ops/reports/cotel-healthz/state` is shared with production (a green tick -resets it to 0, which the third command above does), and the drill alert is a -real issue someone has to close. +`~/ops/reports/cotel-healthz/state` (or `state-edge`) is shared with production +(a green tick resets it to 0, which the third command above does), and the drill +alert is a real issue someone has to close. Dedup is per marker, so the sets never see each other: a red drill cannot attach itself to a standing real alert, and — the half that actually bites — a @@ -248,8 +339,9 @@ users), this company has already watched red Actions sit unnoticed, and `~/ops/ALERT` plus a desktop `notify-send` reach whoever is at the Pi — which is the owner, not an agent. -On red, the prober opens a Paperclip issue titled -`cotel prod /healthz is red []`, assigned to Daedalus. That assignment +On red, the prober opens a Paperclip issue titled `cotel is red +[]` — `prod /healthz` by default, `public ingest at ` for the edge +half — assigned to Daedalus. That assignment is the wake, and it is the only tracker issue the pager ever creates. A later tick searches `q=` and acts on the open issue whose **title** contains it. Search also matches comments and descriptions, so the first hit is not the @@ -342,6 +434,13 @@ rather than reporting an alert nobody was assigned. | `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. A caller with no run id should pass a coarse time bucket; the Pi timer sends `pi-` | | `PC_SOURCE_LINE` | names Actions | First paragraph of a new alert: who opened it. Override it if you are not the workflow | | `PC_REPROBE_HINT` | names an Actions dispatch | How the woken agent re-probes. The Pi timer replaces it with `~/ops/cotel-healthz.sh --probe-only` | +| `PC_ALERT_SUBJECT` | `prod /healthz` | What is red, in the title and in every wake reason. The edge half sets `public ingest at ` | +| `PC_ALERT_LEAD` | names the `/healthz` probe | First line of a new alert's description. The edge half says the application is alive and the public path is not | + +The last two exist because the dedup marker is machine-facing. Without its own +subject, a second watcher mints an alert whose title and wake reason claim +production `/healthz` is red — pointing the reader at the wrong half of the +system. ### The woken agent re-probes, and the alert tells it to @@ -494,7 +593,7 @@ needs no other change. ## Local use ```sh -# on the Pi — the authoritative prober, over the LAN +# on the Pi — the authoritative prober, both halves ~/ops/cotel-healthz.sh --probe-only ~/ops/cotel-healthz.sh --status @@ -504,16 +603,28 @@ HEALTHZ_URL=http://127.0.0.1:8080/healthz scripts/probe-healthz.sh # through Cloudflare (needs an Access service token allowed on the app) scripts/probe-healthz.sh -# a closed local port — the failure demo +# the public ingest path, from anywhere, no credential at all +scripts/probe-edge-ingest.sh + +# a closed local port — the failure demo, either probe scripts/probe-healthz.sh http://127.0.0.1:1/healthz +scripts/probe-edge-ingest.sh http://127.0.0.1:1/v1/traces # the classification tests, including 503 vs stale vs empty vs refused bash scripts/probe-healthz_test.sh +# the ingest classification tests: the app's JSON 401 vs an interstitial 401, +# an Access redirect, a tunnel 5xx, an unrouted 404, anonymous ingest +bash scripts/probe-edge-ingest_test.sh + # the pager against a fake curl — dedup, the staleness window, per-call failure # labels, and the non-GitHub caller's overrides bash scripts/page-cotel-health_test.sh ``` -Exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, `3` stale/empty, -`4` Access blocked. +`probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, +`3` stale/empty, `4` Access blocked. + +`probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401, +`1` unreachable, `2` answered but not with the handler's 401, `4` Access now +fronts the ingest host. diff --git a/scripts/page-cotel-health.sh b/scripts/page-cotel-health.sh index f25a629..67a9e19 100755 --- a/scripts/page-cotel-health.sh +++ b/scripts/page-cotel-health.sh @@ -43,6 +43,11 @@ # "manual"); a caller with no run id should pass a coarse # time bucket so a dropped recovery wake is retried # instead of leaving a resolved alert standing forever. +# +# A caller watching something other than the dashboard's /healthz overrides two +# more, or its alert reads word for word like a dead process: +# PC_ALERT_SUBJECT what is red, in the title and in every wake reason +# PC_ALERT_LEAD the first line of a new alert's description set -euo pipefail @@ -322,6 +327,13 @@ SOURCE_LINE="${PC_SOURCE_LINE:-The health probe in Flopsstuff/cotel opened this DEFAULT_REPROBE="Re-probe by dispatching **Health probe** in Flopsstuff/cotel with **page unchecked** (Actions → Health probe → Run workflow), and read its verdict. Do not just \`curl\` the URL: the probe also classifies 503, stale ingest and an empty database, and \`${PROBED_URL}\` may be the deploy host's own loopback, which only that runner can reach. Never pass a \`loopback_url\` override while checking a real alert — that probes something else." REPROBE_HINT="${PC_REPROBE_HINT:-$DEFAULT_REPROBE}" +# What this alert is about. The dedup marker already keeps two watchers out of +# each other's alerts, but the marker is machine-facing: without its own subject +# a second watcher mints an alert whose title and wake reason claim production +# /healthz is red, which sends the reader to the wrong half of the system. +SUBJECT="${PC_ALERT_SUBJECT:-prod /healthz}" +LEAD="${PC_ALERT_LEAD:-Production cotel /healthz probe is red.}" + # The woken reader must be able to tell an exercise from an outage without # opening Actions: a dispatched run against an overridden URL is a drill, the # hourly schedule against the real endpoint is not. The probe text alone does @@ -343,7 +355,7 @@ if [ -n "$RUN_LINE" ]; then ${RUN_LINE}" fi -TITLE="cotel prod /healthz is red ${MARKER}" +TITLE="cotel ${SUBJECT} is red ${MARKER}" case "$ACTION" in raise) @@ -368,16 +380,16 @@ case "$ACTION" in # gate as the resolve, so a second red hour used to wake nobody. bucket=$(( $(date +%s) / 21600 )) payload="$(wake_payload "cotel_health_still_red" \ - "Production /healthz is still red and this alert is already open. Add the probe output below to the alert; do not open another." \ + "cotel ${SUBJECT} is still red and this alert is already open. Add the probe output below to the alert; do not open another." \ "$reason")" wake_assignee \ "cotel-health-still-red:${EXISTING_ID}:${bucket}" \ - "cotel prod /healthz is still red${RUN_SUFFIX} — alert ${EXISTING_IDENT:-$EXISTING_ID} is already open; add the probe output to it" \ + "cotel ${SUBJECT} is still red${RUN_SUFFIX} — alert ${EXISTING_IDENT:-$EXISTING_ID} is already open; add the probe output to it" \ "$payload" exit 0 fi fi - description="$(printf 'Production cotel /healthz probe is red.\n\n```\n%s\n```\n\n%s\n\n%s\n\n## If you are woken on this issue again, re-probe before acting\n\n**This description always reads red.** The probe wrote it once, when the outage was detected, and cannot edit it afterwards; the wake that brought you here carries no probe output either. So nothing above tells you the state of production right now — check it yourself before you act.\n\n%s\n\n- **Green** — the outage is over. Close this issue as done, citing the probe you checked with.\n- **Still red** — the outage continues. Add that output here as the current evidence.\n' "$reason" "$CONTEXT_BLOCK" "$SOURCE_LINE" "$REPROBE_HINT")" + description="$(printf '%s\n\n```\n%s\n```\n\n%s\n\n%s\n\n## If you are woken on this issue again, re-probe before acting\n\n**This description always reads red.** The probe wrote it once, when the outage was detected, and cannot edit it afterwards; the wake that brought you here carries no probe output either. So nothing above tells you the state of production right now — check it yourself before you act.\n\n%s\n\n- **Green** — the outage is over. Close this issue as done, citing the probe you checked with.\n- **Still red** — the outage continues. Add that output here as the current evidence.\n' "$LEAD" "$reason" "$CONTEXT_BLOCK" "$SOURCE_LINE" "$REPROBE_HINT")" # Same-title creates collapse into a recent open issue unless this # asks not to. The stale alert is that issue, so without the opt-out # the response is the alert we just refused to dedup into. @@ -438,11 +450,11 @@ case "$ACTION" in green="$(cat "$PROBE_OUT")" fi payload="$(wake_payload "cotel_health_recovery" \ - "Production /healthz is green again. Close this alert as done, with the green run URL in the closing comment." \ + "cotel ${SUBJECT} is green again. Close this alert as done, citing the green probe in the closing comment." \ "$green")" wake_assignee \ "cotel-health-recovery:${EXISTING_ID}:${CALL_ID}" \ - "cotel prod /healthz recovered${RUN_SUFFIX} — close alert ${EXISTING_IDENT:-$EXISTING_ID} as done" \ + "cotel ${SUBJECT} recovered${RUN_SUFFIX} — close alert ${EXISTING_IDENT:-$EXISTING_ID} as done" \ "$payload" ;; *) diff --git a/scripts/page-cotel-health_test.sh b/scripts/page-cotel-health_test.sh index ff72cca..e2e47b6 100755 --- a/scripts/page-cotel-health_test.sh +++ b/scripts/page-cotel-health_test.sh @@ -862,6 +862,78 @@ if body.get("idempotencyKey") != "cotel-health-recovery:alert-id:pi-bucket": PY2 pass "PC_CALL_ID wins over GITHUB_RUN_ID" +# 15. A second watcher names its own subject. Without it the alert reads word +# for word like a dead dashboard, which is the wrong half of the system: a +# reachable process whose public ingest path is down needs Cloudflare looked at, +# not the container. +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"edge-id","identifier":"ALT-E","title":"cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]"}' >"$TMP/create.json" +out="$( + export PC_ORIGIN_ID=cotel-ingest-edge + export PC_ALERT_SUBJECT="public ingest at otlp.aignite.pl" + export PC_ALERT_LEAD="cotel answers on the LAN, but its public ingest path is not accepting spans." + run_page 200 raise "$PROBE_FILE" +)" +case "$out" in + "page-cotel-health: opened ALT-E edge-id") pass "subject override creates" ;; + *) fail "subject override creates — output: $out" ;; +esac +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +title = payload.get("title") or "" +if title != "cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]": + bad.append("title=" + title) +description = payload.get("description") or "" +if not description.startswith("cotel answers on the LAN, but its public ingest path is not accepting spans."): + bad.append("description does not lead with PC_ALERT_LEAD: " + description[:120]) +if "/healthz" in title or "Production cotel /healthz probe is red." in description: + bad.append("the edge alert still reads as a dead dashboard") +if bad: + raise SystemExit("; ".join(bad)) +PY +pass "subject and lead replace both places the alert claims /healthz is red" + +# Both wake paths carry the subject too: the reason line is the only text a +# woken run is guaranteed to read. +write_fixture <<'JSON' +[ + { + "id": "edge-alert-id", + "identifier": "ALT-E1", + "title": "cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]", + "status": "todo", + "assigneeAgentId": "agent-on-call" + } +] +JSON +for action in raise resolve; do + reset_log + out="$( + export PC_ORIGIN_ID=cotel-ingest-edge + export PC_ALERT_SUBJECT="public ingest at otlp.aignite.pl" + export PC_CALL_ID="pi-edge-bucket" + run_page 200 "$action" "$PROBE_FILE" 2>"$TMP/err" + )" || { fail "edge $action wake — exit $?"; cat "$TMP/err"; } + python3 - "$(wake_body)" <<'PY' +import json, sys +body = json.loads(sys.argv[1]) +reason = body.get("reason") or "" +instruction = (body.get("payload") or {}).get("instruction") or "" +if "public ingest at otlp.aignite.pl" not in reason: + raise SystemExit("reason does not name the subject: " + reason) +if "public ingest at otlp.aignite.pl" not in instruction: + raise SystemExit("instruction does not name the subject: " + instruction) +if "/healthz" in reason: + raise SystemExit("reason still claims /healthz: " + reason) +PY + pass "edge $action wake names the subject, not /healthz" +done + echo echo "passed=$PASS failed=$FAIL" if [ "$FAIL" -ne 0 ]; then diff --git a/scripts/probe-edge-ingest.sh b/scripts/probe-edge-ingest.sh new file mode 100755 index 0000000..2f845c1 --- /dev/null +++ b/scripts/probe-edge-ingest.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# +# probe-edge-ingest.sh — ask the public OTLP ingest URL and classify the answer. +# +# This is the half no LAN probe can see: DNS, the Cloudflare tunnel, and the +# ingest route on the process behind it. A `docker compose up -d` without the +# environment blanks CLOUDFLARE_TUNNEL_TOKEN and takes the public endpoint down +# while /healthz on the LAN stays green — agents keep exporting spans into a +# hole and say nothing. +# +# The check needs no Cloudflare Access service token, because the ingest host is +# not behind Access: an unauthenticated request reaches cotel's own auth +# middleware and comes back 401 from the application. That 401 is the evidence — +# it can only be produced by DNS resolving, the tunnel forwarding, the process +# listening, and /v1/traces being routed. A blanked tunnel token answers with a +# connection error or a Cloudflare 5xx instead. +# +# The expected status is exactly 401. "Any answer" would pass a Cloudflare +# interstitial, and 200 is not reachable without a real token. +# +# Usage: +# scripts/probe-edge-ingest.sh [URL] +# +# Env: +# INGEST_URL default https://otlp.aignite.pl/v1/traces +# CONNECT_TIMEOUT default 15 +# INGEST_PROBE_TOKEN the deliberately invalid bearer; see below +# +# Exit codes: +# 0 the public ingest path is up (the application answered 401) +# 1 unreachable (DNS, refused, timeout) — tunnel down or host gone +# 2 something answered, but it was not the ingest handler's 401 +# 4 Cloudflare Access is in front of the ingest host — which also rejects +# every agent's spans, so it is an outage, not a probe-config problem + +set -euo pipefail + +URL="${1:-${INGEST_URL:-https://otlp.aignite.pl/v1/traces}}" +CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-15}" +# A bearer that cannot be valid, rather than no bearer at all. cotel's auth +# middleware rejects an unknown `cotel_` token before the ingest handler sees +# the request, so the expected 401 does not depend on how `allow_anonymous` is +# set on the instance — with anonymous ingest allowed, a tokenless request would +# reach the handler and answer 405 to this GET. +PROBE_TOKEN="${INGEST_PROBE_TOKEN:-cotel_edge-probe-not-a-real-token}" + +if ! [ "$CONNECT_TIMEOUT" -gt 0 ] 2>/dev/null; then + echo "probe-edge-ingest: FAILED — CONNECT_TIMEOUT must be a positive integer, got '${CONNECT_TIMEOUT}'" + exit 2 +fi + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +BODY="$TMP/body" +HDRS="$TMP/headers" +ERR="$TMP/curlerr" + +# GET, not the POST a real exporter sends: the auth check runs first either way, +# and a GET cannot write anything should the token check ever be bypassed. +set +e +HTTP_CODE="$(curl -sS \ + --max-time "$CONNECT_TIMEOUT" \ + --connect-timeout "$CONNECT_TIMEOUT" \ + -H "Authorization: Bearer ${PROBE_TOKEN}" \ + -D "$HDRS" \ + -o "$BODY" \ + -w '%{http_code}' \ + "$URL" 2>"$ERR")" +CURL_RC=$? +set -e + +if [ "$CURL_RC" -ne 0 ]; then + err="$(tr '\n' ' ' <"$ERR" | sed 's/[[:space:]]*$//')" + case "$CURL_RC" in + 6) why="dns lookup failed" ;; + 7) why="connection refused" ;; + 28) why="timed out after ${CONNECT_TIMEOUT}s" ;; + 35|60) why="tls handshake failed" ;; + *) why="curl exit ${CURL_RC}" ;; + esac + echo "probe-edge-ingest: FAILED — public ingest unreachable (${why}${err:+; ${err}}) url=${URL}" + exit 1 +fi + +location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" +wwwauth="$(awk 'tolower($0) ~ /^www-authenticate:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" +case "$location $wwwauth" in + *cloudflareaccess.com*|*Cloudflare-Access*) + echo "probe-edge-ingest: FAILED — cloudflare access now fronts the public ingest host (HTTP ${HTTP_CODE}); every agent's spans are rejected the same way url=${URL}" + exit 4 + ;; +esac + +snippet="$(tr '\n' ' ' <"$BODY" | sed 's/[[:space:]]\{1,\}/ /g; s/^ //; s/ $//' | cut -c1-160)" + +if [ "$HTTP_CODE" = "401" ]; then + # Only the application writes a JSON error body here. A Cloudflare + # interstitial or a proxy's own 401 is HTML, and accepting it would make the + # probe green while the origin is unreachable. + case "$snippet" in + '{'*'"error"'*) + echo "probe-edge-ingest: OK — HTTP 401 from the ingest handler (dns, tunnel, process and /v1/traces all answering) url=${URL}" + exit 0 + ;; + esac + echo "probe-edge-ingest: FAILED — HTTP 401 but not from the ingest handler (body is not the application's JSON error) url=${URL}${snippet:+ body=${snippet}}" + exit 2 +fi + +case "$HTTP_CODE" in + 200) + echo "probe-edge-ingest: FAILED — HTTP 200 to an invalid token: the public ingest path is reachable but is accepting unauthenticated spans url=${URL}" + ;; + 404) + echo "probe-edge-ingest: FAILED — HTTP 404: the host answers but /v1/traces is not routed (wrong tunnel target, or the ingest port is not served) url=${URL}" + ;; + 502|503|504|520|521|522|523|524|530) + echo "probe-edge-ingest: FAILED — HTTP ${HTTP_CODE}: cloudflare reached, the origin did not answer (tunnel down or ingest not listening) url=${URL}${snippet:+ body=${snippet}}" + ;; + *) + echo "probe-edge-ingest: FAILED — HTTP ${HTTP_CODE}, wanted 401 from the ingest handler url=${URL}${snippet:+ body=${snippet}}" + ;; +esac +exit 2 diff --git a/scripts/probe-edge-ingest_test.sh b/scripts/probe-edge-ingest_test.sh new file mode 100755 index 0000000..e74f602 --- /dev/null +++ b/scripts/probe-edge-ingest_test.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# +# probe-edge-ingest_test.sh — drive probe-edge-ingest.sh against a local mock. +# Covers the one green answer (the application's JSON 401) and every way a +# reachable-but-wrong answer must stay red: a Cloudflare interstitial that also +# returns 401, an Access login redirect, an origin-unreachable 5xx, an unrouted +# 404, and an unauthenticated 200. + +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +PROBE="$ROOT/scripts/probe-edge-ingest.sh" +PASS=0 +FAIL=0 + +fail() { + echo "FAIL: $*" + FAIL=$((FAIL + 1)) +} + +pass() { + echo "PASS: $*" + PASS=$((PASS + 1)) +} + +expect() { + local name="$1" want_rc="$2" want_needle="$3" + shift 3 + local out rc + set +e + out="$("$@" 2>&1)" + rc=$? + set -e + if [ "$rc" -ne "$want_rc" ]; then + fail "$name — exit $rc, want $want_rc; output: $out" + return + fi + case "$out" in + *"$want_needle"*) pass "$name" ;; + *) fail "$name — output did not contain '$want_needle': $out" ;; + esac +} + +MOCK_PY="$(mktemp)" +cat >"$MOCK_PY" <<'PY' +from http.server import BaseHTTPRequestHandler, HTTPServer +import os + +PORT = int(os.environ["PROBE_MOCK_PORT"]) + + +class H(BaseHTTPRequestHandler): + def do_GET(self): + # The live shape: cotel's auth middleware rejecting an unknown token. + if self.path == "/v1/traces": + self.send_response(401) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b'{"error":"unauthorized"}') + return + # A proxy's own 401 — the status matches, the answer is not cotel's. + if self.path == "/edge-401": + self.send_response(401) + self.send_header("Content-Type", "text/html") + self.end_headers() + self.wfile.write(b"401 Unauthorized") + return + if self.path == "/access": + self.send_response(302) + self.send_header( + "Location", + "https://flopbut.cloudflareaccess.com/cdn-cgi/access/login/example", + ) + self.send_header("WWW-Authenticate", 'Cloudflare-Access realm="cotel"') + self.end_headers() + return + if self.path == "/tunnel-down": + self.send_response(530) + self.send_header("Content-Type", "text/html") + self.end_headers() + self.wfile.write(b"Error 1033 Argo Tunnel error") + return + if self.path == "/anonymous": + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(b"{}") + return + self.send_response(404) + self.end_headers() + self.wfile.write(b"404 page not found") + + def log_message(self, fmt, *args): + return + + +HTTPServer(("127.0.0.1", PORT), H).serve_forever() +PY +trap 'rm -f "$MOCK_PY"; kill "$MOCK_PID" 2>/dev/null || true' EXIT + +PROBE_MOCK_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()')" +export PROBE_MOCK_PORT +python3 "$MOCK_PY" & +MOCK_PID=$! + +for i in 1 2 3 4 5 6 7 8 9 10; do + if curl -s -o /dev/null --max-time 1 "http://127.0.0.1:${PROBE_MOCK_PORT}/v1/traces"; then + break + fi + sleep 0.1 + if [ "$i" -eq 10 ]; then + echo "mock server did not start on 127.0.0.1:${PROBE_MOCK_PORT}" + exit 1 + fi +done + +BASE="http://127.0.0.1:${PROBE_MOCK_PORT}" + +expect "the application's JSON 401 is the green answer" 0 "HTTP 401 from the ingest handler" \ + "$PROBE" "$BASE/v1/traces" +expect "an HTML 401 is not the ingest handler" 2 "not from the ingest handler" \ + "$PROBE" "$BASE/edge-401" +expect "an Access redirect is an ingest outage of its own" 4 "cloudflare access now fronts" \ + "$PROBE" "$BASE/access" +expect "a tunnel error is origin-unreachable, not 401" 2 "cloudflare reached, the origin did not answer" \ + "$PROBE" "$BASE/tunnel-down" +expect "an unrouted path is named as routing, not as down" 2 "/v1/traces is not routed" \ + "$PROBE" "$BASE/nope" +expect "a 200 to an invalid token is red and says why" 2 "accepting unauthenticated spans" \ + "$PROBE" "$BASE/anonymous" + +# The URL can be given by env as well as argv, because the Pi timer sets it. +expect "INGEST_URL is honoured" 0 "HTTP 401 from the ingest handler" \ + env INGEST_URL="$BASE/v1/traces" "$PROBE" + +# 127.0.0.1:1 is unprivileged and closed on this host — connection refused. +expect "connection refused is unreachable" 1 "public ingest unreachable" \ + env CONNECT_TIMEOUT=2 "$PROBE" "http://127.0.0.1:1/v1/traces" + +# TEST-NET-1 is the documentation range; the connect hangs until the timeout. +expect "timeout is unreachable" 1 "public ingest unreachable" \ + env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/v1/traces" + +expect "a non-numeric timeout is rejected, not silently defaulted" 2 "CONNECT_TIMEOUT must be a positive integer" \ + env CONNECT_TIMEOUT=soon "$PROBE" "$BASE/v1/traces" + +echo +echo "passed=$PASS failed=$FAIL" +[ "$FAIL" -eq 0 ]