From d241e269c0c1b8a4776aefbef19642c4984c6027 Mon Sep 17 00:00:00 2001 From: Wayland Date: Tue, 6 Oct 2026 20:17:49 +0200 Subject: [PATCH 1/2] fix(health): name the snapshot in the alert, not the /healthz that is green MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A snapshot failure pages under the LAN half's marker but got the LAN half's words: "cotel prod /healthz is red", which names the one part of production that is still working. The reader starts at the container. The pager already receives the probe verdict, so the subject and lead are classified from it when the caller passes none: a snapshot verdict titles the alert after the snapshot and leads with "the application is alive; its backup is not", pointing at /api/v1/health and the worker's logs instead of a restart. An explicit PC_ALERT_SUBJECT/PC_ALERT_LEAD — the edge half's — still wins. The dedup marker is untouched: both subjects share one alert slot, so two red ticks of this half still dedup into one ticket in either order. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 5 --- docs/operations/health-probe.md | 32 +++++- scripts/page-cotel-health.sh | 37 +++++-- scripts/page-cotel-health_test.sh | 173 ++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+), 11 deletions(-) diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index dab5003..e9fe153 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -104,6 +104,11 @@ where snapshots are expected, and turns every "cannot assert" into red. Exit **6** and not 5 because `~/ops/cotel-healthz.sh` already spends 5 on "the LAN half is green and the public ingest half is not". +This verdict pages under the LAN half's marker, but not in its words: the alert +is titled `cotel prod database snapshot is red` and leads with "the application +is alive; its backup is not". See [the two halves are classified +apart](#the-two-halves-are-classified-apart-and-page-apart). + Two vantage points this must not be turned on for: anything reaching cotel through `cotel.aignite.pl`, where Cloudflare Access answers instead of the application (the default `auto` degrades quietly there, `require` would page on @@ -172,6 +177,24 @@ different people doing different things, so they never share an alert: reader to look at the tunnel and DNS rather than restart the container. The pager takes those from `PC_ALERT_SUBJECT` and `PC_ALERT_LEAD`. +The LAN half needs the same split **inside** its one marker, because it reddens +on two unrelated things: a `/healthz` that does not answer, and a `/healthz` +that answers while the backup has stopped. So when the caller passes no +`PC_ALERT_SUBJECT`, the pager classifies the subject from the probe verdict: + +| Verdict | Title | Lead says | +|---|---|---| +| `no current database snapshot`, or `snapshots are required …` | `cotel prod database snapshot is red [cotel-health-probe]` | the application is alive, the backup is not; read the `snapshot` object in `/api/v1/health` and the worker's logs; do **not** restart cotel | +| anything else | `cotel prod /healthz is red [cotel-health-probe]` | the `/healthz` probe is red | + +An explicit `PC_ALERT_SUBJECT`/`PC_ALERT_LEAD` always wins over the +classification — that is how the edge half keeps its own words. + +**The marker stays out of it.** Both subjects share one alert slot, so two red +ticks of this half — in either order, snapshot then `/healthz` or the reverse — +still dedup into one ticket. Were the subject part of the dedup key, one +watcher's consecutive reds would mint an alert each. + One asymmetry on purpose: **while the LAN half is red, the edge half does not page.** A dead process makes the public path unreachable as a consequence, and a second alert would send its reader hunting Cloudflare for a dead container. The @@ -483,13 +506,16 @@ rather than reporting an alert nobody was assigned. | `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. A caller with no run id should pass a coarse time bucket; the Pi timer sends `pi-` | | `PC_SOURCE_LINE` | names Actions | First paragraph of a new alert: who opened it. Override it if you are not the workflow | | `PC_REPROBE_HINT` | names an Actions dispatch | How the woken agent re-probes. The Pi timer replaces it with `~/ops/cotel-healthz.sh --probe-only` | -| `PC_ALERT_SUBJECT` | `prod /healthz` | What is red, in the title and in every wake reason. The edge half sets `public ingest at ` | -| `PC_ALERT_LEAD` | names the `/healthz` probe | First line of a new alert's description. The edge half says the application is alive and the public path is not | +| `PC_ALERT_SUBJECT` | classified from the verdict: `prod database snapshot` on a snapshot failure, else `prod /healthz` | What is red, in the title and in every wake reason. The edge half sets `public ingest at ` | +| `PC_ALERT_LEAD` | classified the same way | First line of a new alert's description. The edge half says the application is alive and the public path is not | The last two exist because the dedup marker is machine-facing. Without its own subject, a second watcher mints an alert whose title and wake reason claim production `/healthz` is red — pointing the reader at the wrong half of the -system. +system. The same wrong title came out of the LAN half's own snapshot verdict, +which is why those two defaults are derived from the verdict rather than fixed; +see [the two halves are classified +apart](#the-two-halves-are-classified-apart-and-page-apart). ### The woken agent re-probes, and the alert tells it to diff --git a/scripts/page-cotel-health.sh b/scripts/page-cotel-health.sh index 67a9e19..201c630 100755 --- a/scripts/page-cotel-health.sh +++ b/scripts/page-cotel-health.sh @@ -48,6 +48,9 @@ # more, or its alert reads word for word like a dead process: # PC_ALERT_SUBJECT what is red, in the title and in every wake reason # PC_ALERT_LEAD the first line of a new alert's description +# Left unset, both are classified from the probe verdict, because this probe +# reddens on a dead process and on a dead backup through the same marker and +# the second must not be titled as the first. An explicit value always wins. set -euo pipefail @@ -327,12 +330,34 @@ SOURCE_LINE="${PC_SOURCE_LINE:-The health probe in Flopsstuff/cotel opened this DEFAULT_REPROBE="Re-probe by dispatching **Health probe** in Flopsstuff/cotel with **page unchecked** (Actions → Health probe → Run workflow), and read its verdict. Do not just \`curl\` the URL: the probe also classifies 503, stale ingest and an empty database, and \`${PROBED_URL}\` may be the deploy host's own loopback, which only that runner can reach. Never pass a \`loopback_url\` override while checking a real alert — that probes something else." REPROBE_HINT="${PC_REPROBE_HINT:-$DEFAULT_REPROBE}" +# Read here, not in the action below, because the subject is derived from it. +# The missing-file error still belongs to the raise path. +VERDICT="" +if [ -n "$PROBE_OUT" ] && [ -f "$PROBE_OUT" ]; then + VERDICT="$(cat "$PROBE_OUT")" +fi + # What this alert is about. The dedup marker already keeps two watchers out of # each other's alerts, but the marker is machine-facing: without its own subject # a second watcher mints an alert whose title and wake reason claim production # /healthz is red, which sends the reader to the wrong half of the system. -SUBJECT="${PC_ALERT_SUBJECT:-prod /healthz}" -LEAD="${PC_ALERT_LEAD:-Production cotel /healthz probe is red.}" +# +# One watcher has the same problem internally: this probe reddens on two +# unrelated things through one marker, and a snapshot failure is a green +# /healthz with a dead backup — titled as `/healthz`, the alert names the one +# part that is working. So the default subject is classified from the verdict, +# while the marker is deliberately left alone: both subjects share one alert +# slot, so a second red tick still dedups into the first ticket. +DEFAULT_SUBJECT="prod /healthz" +DEFAULT_LEAD="Production cotel /healthz probe is red." +case "$VERDICT" in + *"no current database snapshot"*|*"snapshots are required on this instance"*) + DEFAULT_SUBJECT="prod database snapshot" + DEFAULT_LEAD="**The application is alive; its backup is not.** cotel is answering \`/healthz\`, so the process and the live database are fine — what has stopped is the snapshot worker, and the newest restore point is aging out. Nothing is down for users right now; what is gone is the ability to recover if something does go down. Read the \`snapshot\` object in \`/api/v1/health\` on the host for the worker's own report (\`status\`, \`last_run_at\`, \`last_error\`), then the container logs for the snapshot worker. Do **not** restart cotel on the assumption that production is down — it is not, and a restart neither fixes the worker nor produces a restore point." + ;; +esac +SUBJECT="${PC_ALERT_SUBJECT:-$DEFAULT_SUBJECT}" +LEAD="${PC_ALERT_LEAD:-$DEFAULT_LEAD}" # The woken reader must be able to tell an exercise from an outage without # opening Actions: a dispatched run against an overridden URL is a drill, the @@ -363,7 +388,7 @@ case "$ACTION" in echo "page-cotel-health: FAILED — raise needs a probe output file" exit 1 fi - reason="$(cat "$PROBE_OUT")" + reason="$VERDICT" read_open STALE_ID="" STALE_IDENT="" @@ -445,13 +470,9 @@ case "$ACTION" in echo "page-cotel-health: no open alert" exit 0 fi - green="" - if [ -n "$PROBE_OUT" ] && [ -f "$PROBE_OUT" ]; then - green="$(cat "$PROBE_OUT")" - fi payload="$(wake_payload "cotel_health_recovery" \ "cotel ${SUBJECT} is green again. Close this alert as done, citing the green probe in the closing comment." \ - "$green")" + "$VERDICT")" wake_assignee \ "cotel-health-recovery:${EXISTING_ID}:${CALL_ID}" \ "cotel ${SUBJECT} recovered${RUN_SUFFIX} — close alert ${EXISTING_IDENT:-$EXISTING_ID} as done" \ diff --git a/scripts/page-cotel-health_test.sh b/scripts/page-cotel-health_test.sh index e2e47b6..99215d1 100755 --- a/scripts/page-cotel-health_test.sh +++ b/scripts/page-cotel-health_test.sh @@ -934,6 +934,179 @@ PY pass "edge $action wake names the subject, not /healthz" done +# 16. A snapshot verdict is a green /healthz with a dead backup, so the alert +# names the snapshot. Titled `/healthz`, it names the one part that works and +# sends the reader to restart a healthy container. +SNAP_FILE="$TMP/snapshot.out" +printf '%s\n' "probe-healthz: FAILED — no current database snapshot: red newest snapshot is 90000s old last_run_at=2026-10-04T12:00:00Z threshold=43200s url=http://127.0.0.1:8080/api/v1/health" >"$SNAP_FILE" +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"snap-id","identifier":"ALT-S","title":"cotel prod database snapshot is red [cotel-health-probe]"}' >"$TMP/create.json" +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot raise — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: opened ALT-S snap-id") pass "snapshot verdict creates" ;; + *) fail "snapshot verdict creates — output: $out" ;; +esac +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +title = payload.get("title") or "" +if title != "cotel prod database snapshot is red [cotel-health-probe]": + bad.append("title=" + title) +description = payload.get("description") or "" +if not description.startswith("**The application is alive; its backup is not.**"): + bad.append("lead does not name the backup: " + description[:120]) +if "Production cotel /healthz probe is red." in description: + bad.append("the snapshot alert still leads with the /healthz lead") +for needle in ("/api/v1/health", "Do **not** restart cotel"): + if needle not in description: + bad.append("lead lacks " + needle) +if bad: + raise SystemExit("; ".join(bad)) +PY +pass "snapshot alert names the snapshot in the title and the lead" + +# The same classification reaches the wake reason, which is the one line a +# woken run is guaranteed to read. +write_fixture <<'JSON' +[ + { + "id": "snap-alert-id", + "identifier": "ALT-S1", + "title": "cotel prod database snapshot is red [cotel-health-probe]", + "status": "todo", + "assigneeAgentId": "agent-on-call" + } +] +JSON +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot still-red wake — exit $?"; cat "$TMP/err"; } +python3 - "$(wake_body)" <<'PY' +import json, sys +body = json.loads(sys.argv[1]) +reason = body.get("reason") or "" +if "prod database snapshot" not in reason: + raise SystemExit("reason does not name the snapshot: " + reason) +if "/healthz" in reason: + raise SystemExit("reason still claims /healthz: " + reason) +PY +pass "snapshot still-red wake reason names the snapshot" + +# Two red ticks in a row are still one ticket: the subject changed, the dedup +# marker did not. Were the subject part of the marker, each verdict would mint +# its own alert for the same watcher. +assert_log_lacks "snapshot second tick does not create" "POST http://paperclip.test/api/companies/company-1/issues" +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot third tick — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: woke agent-on-call for ALT-S1 snap-alert-id (run run-7)") + pass "a repeated snapshot red dedups into the open alert" + ;; + *) fail "a repeated snapshot red dedups into the open alert — output: $out" ;; +esac +assert_log_lacks "snapshot third tick does not create either" "POST http://paperclip.test/api/companies/company-1/issues" +assert_log_has "snapshot dedup searches the unchanged marker" "q=cotel-health-probe&" + +# The marker is one alert slot for both verdicts: a snapshot red dedups into an +# alert a /healthz red opened, and the other way round. +write_fixture <<'JSON' +[ + { + "id": "healthz-alert-id", + "identifier": "ALT-H1", + "title": "cotel prod /healthz is red [cotel-health-probe]", + "status": "todo", + "assigneeAgentId": "agent-on-call" + } +] +JSON +reset_log +out="$(run_page 200 raise "$SNAP_FILE" 2>"$TMP/err")" || { fail "snapshot onto healthz alert — exit $?"; cat "$TMP/err"; } +case "$out" in + "page-cotel-health: woke agent-on-call for ALT-H1 healthz-alert-id (run run-7)") + pass "a snapshot red dedups into an open /healthz alert" + ;; + *) fail "a snapshot red dedups into an open /healthz alert — output: $out" ;; +esac + +# `SNAPSHOT_CHECK=require` on an instance that asserts nothing is the same +# class of failure and gets the same subject. +REQUIRE_FILE="$TMP/require.out" +printf '%s\n' "probe-healthz: FAILED — snapshots are required on this instance and it does not report one: snapshot status unknown — the worker has not run yet, or snapshots are disabled on this instance" >"$REQUIRE_FILE" +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"snap-id","identifier":"ALT-S","title":"cotel prod database snapshot is red [cotel-health-probe]"}' >"$TMP/create.json" +out="$(run_page 200 raise "$REQUIRE_FILE" 2>"$TMP/err")" || { fail "require raise — exit $?"; cat "$TMP/err"; } +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +if (payload.get("title") or "") != "cotel prod database snapshot is red [cotel-health-probe]": + raise SystemExit("title=" + str(payload.get("title"))) +PY +pass "an unasserted snapshot under require also names the snapshot" + +# 17. An ordinary red /healthz is unchanged, to the character: the classifier +# must not rewrite the alert every other watcher's runbook was written against. +for verdict in \ + "probe-healthz: FAILED — unreachable (connection refused)" \ + "probe-healthz: FAILED — HTTP 503 (body: duckdb: IO Error)" \ + "probe-healthz: FAILED — ingest is stale: newest span 30000s old threshold=21600s" +do + printf '%s\n' "$verdict" >"$TMP/plain.out" + reset_log + out="$(run_page 200 raise "$TMP/plain.out" 2>"$TMP/err")" || { fail "plain raise — exit $?"; cat "$TMP/err"; } + python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +if (payload.get("title") or "") != "cotel prod /healthz is red [cotel-health-probe]": + bad.append("title=" + str(payload.get("title"))) +if not (payload.get("description") or "").startswith("Production cotel /healthz probe is red."): + bad.append("lead=" + (payload.get("description") or "")[:80]) +if bad: + raise SystemExit("; ".join(bad)) +PY + pass "a plain red keeps the /healthz title and lead — ${verdict:0:40}" +done + +# 18. An explicit subject still wins over the classifier. The edge half passes +# one, and a snapshot phrase appearing in its probe output must not rename it. +write_fixture <<'JSON' +[] +JSON +reset_log +printf '%s\n' '{"id":"edge-id","identifier":"ALT-E","title":"cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]"}' >"$TMP/create.json" +out="$( + export PC_ORIGIN_ID=cotel-ingest-edge + export PC_ALERT_SUBJECT="public ingest at otlp.aignite.pl" + export PC_ALERT_LEAD="cotel answers on the LAN, but its public ingest path is not accepting spans." + run_page 200 raise "$SNAP_FILE" +)" +case "$out" in + "page-cotel-health: opened ALT-E edge-id") pass "explicit subject creates on a snapshot verdict" ;; + *) fail "explicit subject creates on a snapshot verdict — output: $out" ;; +esac +python3 - "$(create_payload)" <<'PY' +import json, sys +payload = json.loads(sys.argv[1]) +bad = [] +if (payload.get("title") or "") != "cotel public ingest at otlp.aignite.pl is red [cotel-ingest-edge]": + bad.append("title=" + str(payload.get("title"))) +description = payload.get("description") or "" +if not description.startswith("cotel answers on the LAN, but its public ingest path is not accepting spans."): + bad.append("lead=" + description[:120]) +if "database snapshot" in (payload.get("title") or ""): + bad.append("the classifier overrode the explicit subject") +if bad: + raise SystemExit("; ".join(bad)) +PY +pass "an explicit subject and lead beat the verdict classifier" + echo echo "passed=$PASS failed=$FAIL" if [ "$FAIL" -ne 0 ]; then From f59d2da734d64d814098bca3719489ab9ea5b382 Mon Sep 17 00:00:00 2001 From: Daedalus Date: Tue, 6 Oct 2026 20:27:15 +0200 Subject: [PATCH 2/2] test(health): pin the verdict wording the pager titles on The snapshot subject is classified from two substrings of the probe's verdict, and nothing on the probe's side held that wording: rewording the prefix left all 97 pager assertions and all 27 probe assertions green while the alert silently went back to titling a dead backup as a dead /healthz. Three assertions pin both phrases from the probe end, so a reword has to pass through a red test. Co-Authored-By: Daedalus Co-Authored-By: Claude Opus 5 --- scripts/probe-healthz_test.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index 865f9fb..9b55ae6 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -254,6 +254,16 @@ expect "the overdue window is the caller's" 0 "snapshot last run" \ expect "a stale ingest outranks a broken snapshot" 3 "ingest stale" \ "$PROBE" "$BASE/snap-both-red/healthz" +# The pager classifies the alert's title off these two phrases. Reword either +# one and a dead backup is paged as a dead /healthz again, with every test in +# both suites still green — so the wording is pinned here, on this side too. +expect "a failed snapshot cycle carries the phrase the pager titles on" 6 "no current database snapshot" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "an overdue snapshot carries the same phrase" 6 "no current database snapshot" \ + "$PROBE" "$BASE/snap-stale/healthz" +expect "the require-mode verdict carries the phrase the pager titles on" 6 "snapshots are required on this instance" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-unknown/healthz" + # The ambiguous half: no claim either way. Red only where the caller says # snapshots are expected, because "unknown" is also how every local instance # reports snapshots being off.