diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index dab5003..e9fe153 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -104,6 +104,11 @@ where snapshots are expected, and turns every "cannot assert" into red. Exit **6** and not 5 because `~/ops/cotel-healthz.sh` already spends 5 on "the LAN half is green and the public ingest half is not". +This verdict pages under the LAN half's marker, but not in its words: the alert +is titled `cotel prod database snapshot is red` and leads with "the application +is alive; its backup is not". See [the two halves are classified +apart](#the-two-halves-are-classified-apart-and-page-apart). + Two vantage points this must not be turned on for: anything reaching cotel through `cotel.aignite.pl`, where Cloudflare Access answers instead of the application (the default `auto` degrades quietly there, `require` would page on @@ -172,6 +177,24 @@ different people doing different things, so they never share an alert: reader to look at the tunnel and DNS rather than restart the container. The pager takes those from `PC_ALERT_SUBJECT` and `PC_ALERT_LEAD`. +The LAN half needs the same split **inside** its one marker, because it reddens +on two unrelated things: a `/healthz` that does not answer, and a `/healthz` +that answers while the backup has stopped. So when the caller passes no +`PC_ALERT_SUBJECT`, the pager classifies the subject from the probe verdict: + +| Verdict | Title | Lead says | +|---|---|---| +| `no current database snapshot`, or `snapshots are required …` | `cotel prod database snapshot is red [cotel-health-probe]` | the application is alive, the backup is not; read the `snapshot` object in `/api/v1/health` and the worker's logs; do **not** restart cotel | +| anything else | `cotel prod /healthz is red [cotel-health-probe]` | the `/healthz` probe is red | + +An explicit `PC_ALERT_SUBJECT`/`PC_ALERT_LEAD` always wins over the +classification — that is how the edge half keeps its own words. + +**The marker stays out of it.** Both subjects share one alert slot, so two red +ticks of this half — in either order, snapshot then `/healthz` or the reverse — +still dedup into one ticket. Were the subject part of the dedup key, one +watcher's consecutive reds would mint an alert each. + One asymmetry on purpose: **while the LAN half is red, the edge half does not page.** A dead process makes the public path unreachable as a consequence, and a second alert would send its reader hunting Cloudflare for a dead container. The @@ -483,13 +506,16 @@ rather than reporting an alert nobody was assigned. | `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. A caller with no run id should pass a coarse time bucket; the Pi timer sends `pi-` | | `PC_SOURCE_LINE` | names Actions | First paragraph of a new alert: who opened it. Override it if you are not the workflow | | `PC_REPROBE_HINT` | names an Actions dispatch | How the woken agent re-probes. The Pi timer replaces it with `~/ops/cotel-healthz.sh --probe-only` | -| `PC_ALERT_SUBJECT` | `prod /healthz` | What is red, in the title and in every wake reason. The edge half sets `public ingest at ` | -| `PC_ALERT_LEAD` | names the `/healthz` probe | First line of a new alert's description. The edge half says the application is alive and the public path is not | +| `PC_ALERT_SUBJECT` | classified from the verdict: `prod database snapshot` on a snapshot failure, else `prod /healthz` | What is red, in the title and in every wake reason. The edge half sets `public ingest at ` | +| `PC_ALERT_LEAD` | classified the same way | First line of a new alert's description. The edge half says the application is alive and the public path is not | The last two exist because the dedup marker is machine-facing. Without its own subject, a second watcher mints an alert whose title and wake reason claim production `/healthz` is red — pointing the reader at the wrong half of the -system. +system. The same wrong title came out of the LAN half's own snapshot verdict, +which is why those two defaults are derived from the verdict rather than fixed; +see [the two halves are classified +apart](#the-two-halves-are-classified-apart-and-page-apart). ### The woken agent re-probes, and the alert tells it to diff --git a/scripts/page-cotel-health.sh b/scripts/page-cotel-health.sh index 67a9e19..201c630 100755 --- a/scripts/page-cotel-health.sh +++ b/scripts/page-cotel-health.sh @@ -48,6 +48,9 @@ # more, or its alert reads word for word like a dead process: # PC_ALERT_SUBJECT what is red, in the title and in every wake reason # PC_ALERT_LEAD the first line of a new alert's description +# Left unset, both are classified from the probe verdict, because this probe +# reddens on a dead process and on a dead backup through the same marker and +# the second must not be titled as the first. An explicit value always wins. set -euo pipefail @@ -327,12 +330,34 @@ SOURCE_LINE="${PC_SOURCE_LINE:-The health probe in Flopsstuff/cotel opened this DEFAULT_REPROBE="Re-probe by dispatching **Health probe** in Flopsstuff/cotel with **page unchecked** (Actions → Health probe → Run workflow), and read its verdict. Do not just \`curl\` the URL: the probe also classifies 503, stale ingest and an empty database, and \`${PROBED_URL}\` may be the deploy host's own loopback, which only that runner can reach. Never pass a \`loopback_url\` override while checking a real alert — that probes something else." REPROBE_HINT="${PC_REPROBE_HINT:-$DEFAULT_REPROBE}" +# Read here, not in the action below, because the subject is derived from it. +# The missing-file error still belongs to the raise path. +VERDICT="" +if [ -n "$PROBE_OUT" ] && [ -f "$PROBE_OUT" ]; then + VERDICT="$(cat "$PROBE_OUT")" +fi + # What this alert is about. The dedup marker already keeps two watchers out of # each other's alerts, but the marker is machine-facing: without its own subject # a second watcher mints an alert whose title and wake reason claim production # /healthz is red, which sends the reader to the wrong half of the system. -SUBJECT="${PC_ALERT_SUBJECT:-prod /healthz}" -LEAD="${PC_ALERT_LEAD:-Production cotel /healthz probe is red.}" +# +# One watcher has the same problem internally: this probe reddens on two +# unrelated things through one marker, and a snapshot failure is a green +# /healthz with a dead backup — titled as `/healthz`, the alert names the one +# part that is working. So the default subject is classified from the verdict, +# while the marker is deliberately left alone: both subjects share one alert +# slot, so a second red tick still dedups into the first ticket. +DEFAULT_SUBJECT="prod /healthz" +DEFAULT_LEAD="Production cotel /healthz probe is red." +case "$VERDICT" in + *"no current database snapshot"*|*"snapshots are required on this instance"*) + DEFAULT_SUBJECT="prod database snapshot" + DEFAULT_LEAD="**The application is alive; its backup is not.** cotel is answering \`/healthz\`, so the process and the live database are fine — what has stopped is the snapshot worker, and the newest restore point is aging out. Nothing is down for users right now; what is gone is the ability to recover if something does go down. Read the \`snapshot\` object in \`/api/v1/health\` on the host for the worker's own report (\`status\`, \`last_run_at\`, \`last_error\`), then the container logs for the snapshot worker. Do **not** restart cotel on the assumption that production is down — it is not, and a restart neither fixes the worker nor produces a restore point." + ;; +esac +SUBJECT="${PC_ALERT_SUBJECT:-$DEFAULT_SUBJECT}" +LEAD="${PC_ALERT_LEAD:-$DEFAULT_LEAD}" # The woken reader must be able to tell an exercise from an outage without # opening Actions: a dispatched run against an overridden URL is a drill, the @@ -363,7 +388,7 @@ case "$ACTION" in echo "page-cotel-health: FAILED — raise needs a probe output file" exit 1 fi - reason="$(cat "$PROBE_OUT")" + reason="$VERDICT" read_open STALE_ID="" STALE_IDENT="" @@ -445,13 +470,9 @@ case "$ACTION" in echo "page-cotel-health: no open alert" exit 0 fi - green="" - if [ -n "$PROBE_OUT" ] && [ -f "$PROBE_OUT" ]; then - green="$(cat "$PROBE_OUT")" - fi payload="$(wake_payload "cotel_health_recovery" \ "cotel ${SUBJECT} is green again. Close this alert as done, citing the green probe in the closing comment." \ - "$green")" + "$VERDICT")" wake_assignee \ "cotel-health-recovery:${EXISTING_ID}:${CALL_ID}" \ "cotel ${SUBJECT} recovered${RUN_SUFFIX} — close alert ${EXISTING_IDENT:-$EXISTING_ID} as done" \ diff --git a/scripts/page-cotel-health_test.sh b/scripts/page-cotel-health_test.sh index e2e47b6..99215d1 100755 --- a/scripts/page-cotel-health_test.sh +++ b/scripts/page-cotel-health_test.sh @@ -934,6 +934,179 @@ PY pass "edge $action wake names the subject, not /healthz" done +# 16. A snapshot verdict is a green /healthz with a dead backup, so the alert +# names the snapshot. Titled `/healthz`, it names the one part that works and +# sends the reader to restart a healthy container. +SNAP_FILE="$TMP/snapshot.out" +printf '%s\n' "probe-healthz: FAILED — no current database snapshot: red newest snapshot is 90000s old last_run_at=2026-10-04T12:00:00Z threshold=43200s url=http://127.0.0.1:8080/api/v1/health" >"$SNAP_FILE" +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"snap-id","identifier":"ALT-S","title":"cotel prod database snapshot is red [cotel-health-probe]"}' >"$TMP/create.json" +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot raise — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: opened ALT-S snap-id") pass "snapshot verdict creates" ;; + *) fail "snapshot verdict creates — output: $out" ;; +esac +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +title = payload.get("title") or "" +if title != "cotel prod database snapshot is red [cotel-health-probe]": + bad.append("title=" + title) +description = payload.get("description") or "" +if not description.startswith("**The application is alive; its backup is not.**"): + bad.append("lead does not name the backup: " + description[:120]) +if "Production cotel /healthz probe is red." in description: + bad.append("the snapshot alert still leads with the /healthz lead") +for needle in ("/api/v1/health", "Do **not** restart cotel"): + if needle not in description: + bad.append("lead lacks " + needle) +if bad: + raise SystemExit("; ".join(bad)) +PY +pass "snapshot alert names the snapshot in the title and the lead" + +# The same classification reaches the wake reason, which is the one line a +# woken run is guaranteed to read. +write_fixture <<'JSON' +[ + { + "id": "snap-alert-id", + "identifier": "ALT-S1", + "title": "cotel prod database snapshot is red [cotel-health-probe]", + "status": "todo", + "assigneeAgentId": "agent-on-call" + } +] +JSON +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot still-red wake — exit $?"; cat "$TMP/err"; } +python3 - "$(wake_body)" <<'PY' +import json, sys +body = json.loads(sys.argv[1]) +reason = body.get("reason") or "" +if "prod database snapshot" not in reason: + raise SystemExit("reason does not name the snapshot: " + reason) +if "/healthz" in reason: + raise SystemExit("reason still claims /healthz: " + reason) +PY +pass "snapshot still-red wake reason names the snapshot" + +# Two red ticks in a row are still one ticket: the subject changed, the dedup +# marker did not. Were the subject part of the marker, each verdict would mint +# its own alert for the same watcher. +assert_log_lacks "snapshot second tick does not create" "POST http://paperclip.test/api/companies/company-1/issues" +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot third tick — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: woke agent-on-call for ALT-S1 snap-alert-id (run run-7)") + pass "a repeated snapshot red dedups into the open alert" + ;; + *) fail "a repeated snapshot red dedups into the open alert — output: $out" ;; +esac +assert_log_lacks "snapshot third tick does not create either" "POST http://paperclip.test/api/companies/company-1/issues" +assert_log_has "snapshot dedup searches the unchanged marker" "q=cotel-health-probe&" + +# The marker is one alert slot for both verdicts: a snapshot red dedups into an +# alert a /healthz red opened, and the other way round. +write_fixture <<'JSON' +[ + { + "id": "healthz-alert-id", + "identifier": "ALT-H1", + "title": "cotel prod /healthz is red [cotel-health-probe]", + "status": "todo", + "assigneeAgentId": "agent-on-call" + } +] +JSON +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot onto healthz alert — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: woke agent-on-call for ALT-H1 healthz-alert-id (run run-7)") + pass "a snapshot red dedups into an open /healthz alert" + ;; + *) fail "a snapshot red dedups into an open /healthz alert — output: $out" ;; +esac + +# `SNAPSHOT_CHECK=require` on an instance that asserts nothing is the same +# class of failure and gets the same subject. +REQUIRE_FILE="$TMP/require.out" +printf '%s\n' "probe-healthz: FAILED — snapshots are required on this instance and it does not report one: snapshot status unknown — the worker has not run yet, or snapshots are disabled on this instance" >"$REQUIRE_FILE" +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"snap-id","identifier":"ALT-S","title":"cotel prod database snapshot is red [cotel-health-probe]"}' >"$TMP/create.json" +out="$(run_page 200 raise "$REQUIRE_FILE" 2>"$TMP/err")" || { fail "require raise — exit $?"; cat "$TMP/err"; } +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +if (payload.get("title") or "") != "cotel prod database snapshot is red [cotel-health-probe]": + raise SystemExit("title=" + str(payload.get("title"))) +PY +pass "an unasserted snapshot under require also names the snapshot" + +# 17. An ordinary red /healthz is unchanged, to the character: the classifier +# must not rewrite the alert every other watcher's runbook was written against. +for verdict in \ + "probe-healthz: FAILED — unreachable (connection refused)" \ + "probe-healthz: FAILED — HTTP 503 (body: duckdb: IO Error)" \ + "probe-healthz: FAILED — ingest is stale: newest span 30000s old threshold=21600s" +do + printf '%s\n' "$verdict" >"$TMP/plain.out" + reset_log + out="$(run_page 200 raise "$TMP/plain.out" 2>"$TMP/err")" || { fail "plain raise — exit $?"; cat "$TMP/err"; } + python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +if (payload.get("title") or "") != "cotel prod /healthz is red [cotel-health-probe]": + bad.append("title=" + str(payload.get("title"))) +if not (payload.get("description") or "").startswith("Production cotel /healthz probe is red."): + bad.append("lead=" + (payload.get("description") or "")[:80]) +if bad: + raise SystemExit("; ".join(bad)) +PY + pass "a plain red keeps the /healthz title and lead — ${verdict:0:40}" +done + +# 18. An explicit subject still wins over the classifier. The edge half passes +# one, and a snapshot phrase appearing in its probe output must not rename it. +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"edge-id","identifier":"ALT-E","title":"cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]"}' >"$TMP/create.json" +out="$( + export PC_ORIGIN_ID=cotel-ingest-edge + export PC_ALERT_SUBJECT="public ingest at otlp.aignite.pl" + export PC_ALERT_LEAD="cotel answers on the LAN, but its public ingest path is not accepting spans." + run_page 200 raise "$SNAP_FILE" +)" +case "$out" in + "page-cotel-health: opened ALT-E edge-id") pass "explicit subject creates on a snapshot verdict" ;; + *) fail "explicit subject creates on a snapshot verdict — output: $out" ;; +esac +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +if (payload.get("title") or "") != "cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]": + bad.append("title=" + str(payload.get("title"))) +description = payload.get("description") or "" +if not description.startswith("cotel answers on the LAN, but its public ingest path is not accepting spans."): + bad.append("lead=" + description[:120]) +if "database snapshot" in (payload.get("title") or ""): + bad.append("the classifier overrode the explicit subject") +if bad: + raise SystemExit("; ".join(bad)) +PY +pass "an explicit subject and lead beat the verdict classifier" + echo echo "passed=$PASS failed=$FAIL" if [ "$FAIL" -ne 0 ]; then diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index 865f9fb..9b55ae6 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -254,6 +254,16 @@ expect "the overdue window is the caller's" 0 "snapshot last run" \ expect "a stale ingest outranks a broken snapshot" 3 "ingest stale" \ "$PROBE" "$BASE/snap-both-red/healthz" +# The pager classifies the alert's title off these two phrases. Reword either +# one and a dead backup is paged as a dead /healthz again, with every test in +# both suites still green — so the wording is pinned here, on this side too. +expect "a failed snapshot cycle carries the phrase the pager titles on" 6 "no current database snapshot" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "an overdue snapshot carries the same phrase" 6 "no current database snapshot" \ + "$PROBE" "$BASE/snap-stale/healthz" +expect "the require-mode verdict carries the phrase the pager titles on" 6 "snapshots are required on this instance" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-unknown/healthz" + # The ambiguous half: no claim either way. Red only where the caller says # snapshots are expected, because "unknown" is also how every local instance # reports snapshots being off.