From 04afcd6a9c2a36471955e7284348a8b6be7318b5 Mon Sep 17 00:00:00 2001 From: Wayland Date: Tue, 6 Oct 2026 20:00:57 +0200 Subject: [PATCH] feat(health): page when the database snapshot stops happening The snapshot worker's report lives only on /api/v1/health, and the scheduled probe reads /healthz, so a backup could die or start failing and nothing would say so until a human opened the endpoint by hand. A green /healthz is now followed by a second question to /api/v1/health, whose address is derived from the /healthz URL the probe is already given - so the host wrapper on the Pi needs no edit and the check reaches the timer with the merge. Red (exit 6, not 5, which the wrapper already spends on "LAN green, public ingest not") on status "error", on a last_run_at older than SNAPSHOT_STALE_AFTER_SECONDS, and on a status "ok" whose timestamp is missing or unparseable. status "unknown" stays silent: it means both "the worker has not run yet" and "snapshots are disabled", and disabled is the shipped default everywhere but production, so reddening on it would lie on every local instance. SNAPSHOT_CHECK=require is the caller asserting snapshots are expected here and turns that - plus a missing snapshot field and an unreadable /api/v1/health - red too; off skips the question. /healthz is unchanged in fields and in the meaning of ok, with a test pinning both against a worker reporting a failed cycle: that endpoint is the container liveness contract, and a failed backup must not mark a working container unhealthy or fail a deploy. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 1 + README.md | 16 +- docs/index.md | 3 + docs/operations/duckdb-snapshots.md | 66 ++++++++ docs/operations/health-probe.md | 56 ++++++- internal/dashboard/healthz_test.go | 41 +++++ scripts/probe-healthz.sh | 241 +++++++++++++++++++++++++--- scripts/probe-healthz_test.sh | 113 ++++++++++++- 8 files changed, 506 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8571b52..00532b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- The scheduled health probe watches the database snapshots, not just the process. `scripts/probe-healthz.sh` follows a green `/healthz` with a second question to `/api/v1/health` — the only place the snapshot worker's report exists — and goes red (exit 6) on `status: "error"`, on a `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 43200 s, two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`), and on a `status: "ok"` whose timestamp is missing or unparseable. The worker could previously die or start failing and nothing would say so until a human opened the endpoint, which is the shape of the failure that already cost six days of blind outage. The claim is deliberately kept off `/healthz`: that endpoint is the container liveness contract read by `cotel --healthcheck`, the Docker `HEALTHCHECK` and `scripts/wait-for-healthy.sh`, and a failed backup must not mark a working container unhealthy or fail the next deploy — the boundary is checked at the edge and the backup claim lives on the alert path instead. `status: "unknown"` stays silent by default, because it means both "the worker has not run yet" and "snapshots are disabled", and disabled is the shipped default everywhere but production: a probe reddening on it would lie on every local instance. `SNAPSHOT_CHECK=require` is the caller asserting snapshots are expected here and turns that — along with a missing `snapshot` field and an unreadable `/api/v1/health` — red too; `SNAPSHOT_CHECK=off` skips the question. The `/api/v1/health` address is derived from the `/healthz` URL the probe is already given, so the host wrapper on the Pi needs no edit and the check reaches the timer with the merge. Exit 6 rather than 5, which `~/ops/cotel-healthz.sh` already spends on "the LAN half is green and the public ingest half is not" ([docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md), [docs/operations/health-probe.md](docs/operations/health-probe.md)) - cotel snapshots its own database. A worker runs `EXPORT DATABASE ... (FORMAT PARQUET, COMPRESSION ZSTD)` at startup and then every `COTEL_SNAPSHOT_INTERVAL`, into one dated directory per snapshot under `COTEL_SNAPSHOT_DIR` (`/snapshots` in compose, its own volume), keeping the `COTEL_SNAPSHOT_KEEP` newest. It had to be taken from inside cotel: DuckDB has one writer and the live process holds the file lock, so no external process can open the database even read-only. Measured on the production host against a probe copy of the live 152.6 MB database, the export holds the single connection for 0.15-0.3 s and writes 4.2 MiB - 1/35th of the file for a full copy of all six tables - and because it runs on that same connection it cannot race the WAL checkpoint: the two are serialised by construction rather than by a lock. The format is the point: Parquet plus a plain-text `schema.sql` is readable by any DuckDB build, and an import *rebuilds* the secondary indexes from the data, so a snapshot of a database with a damaged ART index - the September 2026 failure, a file whose rows were all readable - restores to a healthy one, where a byte copy of the volume reproduces the damage faithfully. `snapshot.json` is written last and by nothing else, so its presence is the only "complete" signal; it carries the row count per table, read back out of the Parquet files, so a restore is checked against what the snapshot claims rather than trusted. A snapshot is only taken when the newest complete one is older than the interval, so a restart - or a crash loop - cannot spend the retained window on snapshots minutes apart. Pruning keeps the newest N complete snapshots, deletes incomplete ones, runs only after a successful export, and leaves directories whose name is not a snapshot instant alone ([ADR-0023](docs/decisions/0023-production-database-snapshots.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md)) - `cotel --db-import ` restores a snapshot into `COTEL_DB_PATH` and verifies every table against the snapshot's manifest, failing with the table and both numbers named if a count does not match. It needs nothing but the image already on the host - no DuckDB CLI with its version matched by hand, which is the step of the recovery procedure most able to destroy a file. It refuses a target that already holds tables (the snapshot's `schema.sql` issues plain `CREATE TABLE`, so importing over data would fail halfway and leave a mixed database) and a snapshot with no `snapshot.json`. Note that the paths in a snapshot's `load.sql` are **absolute**, so a snapshot cannot be moved or renamed and still be imported; it must be visible at the path it was written to - `GET /api/v1/health` reports a `snapshot` object (`status`, `last_run_at`, `last_error`, `last_dir`), and a failed export flips the top-level `status` to `degraded` - the same treatment a failing retention roll-up gets, because a backup that has been failing quietly for a month is worse than a known absent one. `unknown` covers both a worker that has not run yet and snapshots left disabled, which is the default outside the compose file diff --git a/README.md b/README.md index d4ec572..416bbed 100644 --- a/README.md +++ b/README.md @@ -321,6 +321,13 @@ report a dead database as healthy. The same two fields are on `GET /api/v1/health` — see [docs/operations/api-reference.md](docs/operations/api-reference.md). +The retention and snapshot workers' health is deliberately **not** here. This +endpoint is what `cotel --healthcheck`, the Docker `HEALTHCHECK` and the deploy's +`wait-for-healthy.sh` read, so a failing backup reported through `ok` would mark +a working container unhealthy and fail the next deploy. Those claims live on +`GET /api/v1/health`, which the health probe reads separately — see +[Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). + ### The deploy waits for healthy `docker compose up -d` returns as soon as the container has *started*, which is @@ -406,8 +413,10 @@ docker run --rm --entrypoint /usr/local/bin/cotel \ ``` The worker's last outcome is reported on `GET /api/v1/health` under a `snapshot` -object, and a failed export degrades the top-level `status`. Full procedure, -including promoting a restored volume: +object, and a failed export degrades the top-level `status`. The scheduled +health probe reads that object on every tick and pages when the backup stops +producing restore points, so nobody has to open the endpoint by hand. Full +procedure, including promoting a restored volume and what the probe calls red: [Database Snapshots and Restore](docs/operations/duckdb-snapshots.md). ## Retention defaults @@ -528,6 +537,9 @@ from the network at startup instead of bundling it). | `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel | | `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) | | `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. | +| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. | +| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. | | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. | ## Architecture diff --git a/docs/index.md b/docs/index.md index 596e1fd..fe94b6a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -73,6 +73,9 @@ Restart Claude Code. Telemetry starts flowing immediately. | `COTEL_SNAPSHOT_INTERVAL` | `6h` | How often a snapshot is taken (Go duration) | | `COTEL_SNAPSHOT_KEEP` | `56` | How many complete snapshots to keep - 14 days at the default interval | | `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not the binary: the Docker volume mounted at `/snapshots` | +| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not the binary: `auto` pages only on an affirmatively broken snapshot, `require` also when the instance claims none, `off` never asks. See [Production health probes](./operations/health-probe#the-snapshot-claim-asked-of-a-second-endpoint) | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not the binary: how old the newest snapshot may be - two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h` | +| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not the binary: overrides the derived `/api/v1/health` address | ## Data & retention diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index c94718b..164970f 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -76,6 +76,72 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ cat /snapshots/2026-10-06T12-00-00Z/snapshot.json ``` +## Who is watching it + +Nobody has to remember to run the command above: the scheduled health probe +asks the same endpoint. The LAN half of +[the 10-minute tick](./health-probe) reads `/healthz` first and, when that is +green, asks `/api/v1/health` and classifies the `snapshot` object. A red verdict +pages the same way a dead process does — two consecutive failures, the +`[cotel-health-probe]` marker, an issue whose assignment wakes an agent. + +The claim is deliberately **not** on `/healthz`. That endpoint is the container +liveness contract: `cotel --healthcheck`, the Docker `HEALTHCHECK` and +`scripts/wait-for-healthy.sh` all read it, so folding a failed backup into +`ok: false` would mark a working container unhealthy and fail the next deploy. +A missing restore point is an alerting fact, not a liveness fact, so it lives on +the alert path. + +### What is red + +| `snapshot` in `/api/v1/health` | Verdict | Why | +|---|---|---| +| `status: "error"` | **red** (probe exit 6) | The worker ran and failed. `last_error` is quoted into the alert | +| `status: "ok"`, `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` | **red** | The worker is not completing cycles any more; the newest restore point is aging out | +| `status: "ok"`, `last_run_at` absent or unparseable | **red** | The claim contradicts itself | +| `status: "unknown"` | silent by default | Means *both* "has not run yet" and "snapshots are disabled", which is the shipped default outside compose — red here would cry wolf on every local instance | +| no `snapshot` field, or `/api/v1/health` not readable | silent by default | An older binary, or a vantage point that cannot see the endpoint | + +The last two rows flip to red under `SNAPSHOT_CHECK=require`, which is the +caller asserting "snapshots are expected on this instance". Default is `auto`: +page only on affirmative evidence that the backup is broken. `SNAPSHOT_CHECK=off` +stops the probe asking at all. + +### The threshold + +`SNAPSHOT_STALE_AFTER_SECONDS` defaults to **43200 s (12 h)** — two +`COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`. Two and not one because +a cycle is scheduled *from* the last one, so a single interval leaves no room +for the export itself and a tick landing just before the next run would be red +every time. If you change `COTEL_SNAPSHOT_INTERVAL`, change this with it: the +probe reads the server's answer, not the server's configuration, and cannot +know the interval on its own. + +### Checking it by hand + +```bash +# the question the probe asks, on the LAN (the dashboard host is behind Access) +curl -s http://robmini.local:8080/api/v1/health | jq '{status, snapshot}' + +# the probe's own verdict, classification and all +HEALTHZ_URL=http://robmini.local:8080/healthz scripts/probe-healthz.sh + +# the same, demanding that snapshots be configured here +SNAPSHOT_CHECK=require HEALTHZ_URL=http://robmini.local:8080/healthz \ + scripts/probe-healthz.sh + +# on the Pi: both halves of the scheduled tick, paging nothing +~/ops/cotel-healthz.sh --probe-only +``` + +The probe takes one URL — `/healthz` — and derives the `/api/v1/health` address +from it, so there is no second address to keep in step on the host. A green line +names the snapshot it found: + +``` +probe-healthz: OK — HTTP 200 ingest age 2s (threshold 21600s) url=http://robmini.local:8080/healthz | snapshot last run 2026-10-06T17:39:54Z (age 909s, threshold 43200s) dir=/snapshots/2026-10-06T17-39-54Z +``` + ## Restoring from a snapshot `cotel --db-import ` creates the database at `COTEL_DB_PATH` from a diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 7514be0..dab5003 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -27,7 +27,7 @@ others see. The first two columns are the scheduled halves of one tick. | Runner | the Pi, `~/ops/cotel-healthz.sh` | the Pi, same tick | `[self-hosted, flopsstuff, docker]` — the deploy host | `ubuntu-latest` | | Target | `http://robmini.local:8080/healthz` (LAN) | `https://otlp.aignite.pl/v1/traces` (internet) | `http://127.0.0.1:8080/healthz` | `https://cotel.aignite.pl/healthz` | | Cloudflare in the path | no | yes (DNS + tunnel) | no | yes (tunnel + Access) | -| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, **and the host being off** | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | all of the LAN half except host-off | the dashboard host through Cloudflare | +| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, a snapshot worker that has stopped producing restore points, **and the host being off** | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | all of the LAN half except host-off | the dashboard host through Cloudflare | | Blind to | the tunnel, DNS — a LAN-healthy cotel unreachable from the internet reads green | the database and ingest freshness: this check stops at the auth boundary | **its own host being down** — with the runner off the job queues, producing no colour at all | nothing in the path, but see the Access caveat below | | Access service token | not needed | **not needed** | not needed | required | | Alert dedup marker | `[cotel-health-probe]` | `[cotel-ingest-edge]` | `[cotel-health-probe-drill]` | `[cotel-health-probe-drill-edge]` | @@ -60,6 +60,7 @@ they send the on-call looking in different places: | `HTTP ` | 2 | Crash loop, proxy error, unexpected handler | | `ingest stale` / `empty database` | 3 | Process is up and the DB reads; spans are not being accepted | | `cloudflare access blocked` | 4 | Probe config, **not** an outage — see below | +| `no current database snapshot` | 6 | Everything above is fine; the backup is not happening — see below | A 200 with `ok: true` is not enough. Staleness is a body field (`newest_span_age_seconds`); the endpoint keeps 200 on a quiet instance so the @@ -71,6 +72,54 @@ If `last_ingest_at` / `newest_span_age_seconds` are absent, the probe degrades to liveness (HTTP 200) so it keeps working before that contract is on production. +### The snapshot claim, asked of a second endpoint + +A green `/healthz` is followed by a second question, to `/api/v1/health` on the +same host — the only place the snapshot worker's own report exists. Without it +the backup could die and nothing would say so until somebody opened the +endpoint by hand, which is the failure class that already cost six days of +blind outage once. + +The address is **derived** from the `/healthz` URL (same entry point, other +path), so the host wrapper configures one URL and a change here needs no edit +on the Pi. `API_HEALTH_URL` overrides the derivation. + +| `snapshot` object | Verdict | +|---|---| +| `status: "error"` | red, exit 6, with `last_error` quoted | +| `status: "ok"` and `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 12 h) | red, exit 6 | +| `status: "ok"` and `last_run_at` absent or unparseable | red, exit 6 | +| `status: "unknown"` | not asserted; said on the green line | +| no `snapshot` field, or `/api/v1/health` not readable | not asserted; said on the green line | + +The split is the point. `error` and an overdue timestamp are *affirmative* +evidence that the restore point is gone, and can be trusted anywhere. +`unknown` is not: it means both "the worker has not run yet" and "snapshots are +disabled", and disabled is the shipped default everywhere but production — a +probe that reddened on it would lie on every developer's instance. +`SNAPSHOT_CHECK=require` is the caller asserting that this instance is one +where snapshots are expected, and turns every "cannot assert" into red. +`SNAPSHOT_CHECK=off` skips the question entirely. + +Exit **6** and not 5 because `~/ops/cotel-healthz.sh` already spends 5 on "the +LAN half is green and the public ingest half is not". + +Two vantage points this must not be turned on for: anything reaching cotel +through `cotel.aignite.pl`, where Cloudflare Access answers instead of the +application (the default `auto` degrades quietly there, `require` would page on +a probe-configuration problem), and the edge half, which never reads +`/healthz` at all. + +| Variable | Default | Role | +|---|---|---| +| `SNAPSHOT_CHECK` | `auto` | `auto` pages on affirmative failure only; `require` also pages when the instance makes no snapshot claim; `off` never asks | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | How old the newest snapshot may be. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; move it with the interval | +| `API_HEALTH_URL` | derived from the `/healthz` URL | Override when the snapshot report is not at the same entry point | + +What counts as red, why the threshold is two intervals, and how to check it by +hand are in +[Database Snapshots and Restore](./duckdb-snapshots#who-is-watching-it). + ## What the edge half checks, and why it needs no Access token `scripts/probe-edge-ingest.sh` asks `https://otlp.aignite.pl/v1/traces` — the @@ -600,6 +649,9 @@ needs no other change. # the deploy host, from the deploy host — no Access token involved HEALTHZ_URL=http://127.0.0.1:8080/healthz scripts/probe-healthz.sh +# the same, demanding that this instance report a current snapshot +SNAPSHOT_CHECK=require HEALTHZ_URL=http://127.0.0.1:8080/healthz scripts/probe-healthz.sh + # through Cloudflare (needs an Access service token allowed on the app) scripts/probe-healthz.sh @@ -623,7 +675,7 @@ bash scripts/page-cotel-health_test.sh ``` `probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, -`3` stale/empty, `4` Access blocked. +`3` stale/empty, `4` Access blocked, `6` no current database snapshot. `probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401, `1` unreachable, `2` answered but not with the handler's 401, `4` Access now diff --git a/internal/dashboard/healthz_test.go b/internal/dashboard/healthz_test.go index 0d30391..5d0d242 100644 --- a/internal/dashboard/healthz_test.go +++ b/internal/dashboard/healthz_test.go @@ -121,6 +121,47 @@ func TestHealthzImportedSpanStaysStale(t *testing.T) { } } +// The snapshot worker's report is deliberately absent from this contract, and +// a failing backup is deliberately not a /healthz failure: the container +// HEALTHCHECK and the deploy's wait-for-healthy read this endpoint, so a dead +// backup here would mark a working container unhealthy and fail deploys. The +// claim about backups is made on the alert path instead, by the probe reading +// /api/v1/health. +func TestHealthzIgnoresTheSnapshotWorker(t *testing.T) { + db, ro := openTestDBRW(t) + for k, v := range map[string]string{ + "snapshot_last_status": "error", + "snapshot_last_error": "export failed: No space left on device", + "snapshot_last_run_at": "2026-01-01T00:00:00Z", + } { + if err := db.SetSetting(k, v); err != nil { + t.Fatalf("SetSetting(%s): %v", k, err) + } + } + + code, body := getHealthz(t, dashboard.New(ro)) + if code != http.StatusOK { + t.Errorf("a failing snapshot worker must not change the status code: want 200, got %d", code) + } + if body["ok"] != true { + t.Errorf("a failing snapshot worker must not change ok, got %v", body["ok"]) + } + + want := map[string]bool{ + "ok": true, "spans": true, "last_ingest_at": true, "newest_span_age_seconds": true, + } + for key := range body { + if !want[key] { + t.Errorf("unexpected field %q on /healthz: this contract only ever gains fields by decision", key) + } + } + for key := range want { + if _, present := body[key]; !present { + t.Errorf("field %q is missing from /healthz", key) + } + } +} + // brokenDB fails every single-row read, standing in for an unreadable database. type brokenDB struct { inner dashboard.DB diff --git a/scripts/probe-healthz.sh b/scripts/probe-healthz.sh index 3ca28c7..49b53d1 100755 --- a/scripts/probe-healthz.sh +++ b/scripts/probe-healthz.sh @@ -8,6 +8,13 @@ # answers 200. Missing freshness keys degrade to liveness so the probe # stays useful before that contract is deployed. # +# A green /healthz is then followed by a second question to /api/v1/health, +# which is where the snapshot worker's own report lives. /healthz deliberately +# does not carry it: that endpoint is the container liveness contract, and a +# failed backup must not mark a working container unhealthy or fail a deploy. +# The claim "there is a restore point" therefore has to be made here, on the +# alert path, or nothing makes it at all. +# # Usage: # scripts/probe-healthz.sh [URL] # @@ -17,6 +24,9 @@ # CONNECT_TIMEOUT default 15 # CF_ACCESS_CLIENT_ID optional Cloudflare Access service token # CF_ACCESS_CLIENT_SECRET optional; sent only when both are set +# SNAPSHOT_CHECK auto (default) | require | off +# SNAPSHOT_STALE_AFTER_SECONDS default 43200 (2x the shipped 6h interval) +# API_HEALTH_URL override; derived from the /healthz URL otherwise # # Exit codes: # 0 healthy @@ -24,12 +34,19 @@ # 2 HTTP non-200 # 3 200 but ingest is stale or the database has never accepted a span # 4 Cloudflare Access blocked the probe (cannot see /healthz) +# 6 /healthz is green but the database has no current snapshot +# +# 6 and not 5: ~/ops/cotel-healthz.sh already spends 5 on "the LAN half is +# green and the public ingest half is not", and passes a probe's own code +# through otherwise. set -euo pipefail URL="${1:-${HEALTHZ_URL:-https://cotel.aignite.pl/healthz}}" STALE_AFTER_SECONDS="${STALE_AFTER_SECONDS:-21600}" CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-15}" +SNAPSHOT_CHECK="${SNAPSHOT_CHECK:-auto}" +SNAPSHOT_STALE_AFTER_SECONDS="${SNAPSHOT_STALE_AFTER_SECONDS:-43200}" if ! [ "$STALE_AFTER_SECONDS" -gt 0 ] 2>/dev/null; then echo "probe-healthz: FAILED — STALE_AFTER_SECONDS must be a positive integer, got '${STALE_AFTER_SECONDS}'" @@ -39,6 +56,30 @@ if ! [ "$CONNECT_TIMEOUT" -gt 0 ] 2>/dev/null; then echo "probe-healthz: FAILED — CONNECT_TIMEOUT must be a positive integer, got '${CONNECT_TIMEOUT}'" exit 2 fi +if ! [ "$SNAPSHOT_STALE_AFTER_SECONDS" -gt 0 ] 2>/dev/null; then + echo "probe-healthz: FAILED — SNAPSHOT_STALE_AFTER_SECONDS must be a positive integer, got '${SNAPSHOT_STALE_AFTER_SECONDS}'" + exit 2 +fi +case "$SNAPSHOT_CHECK" in + auto|require|off) ;; + *) + echo "probe-healthz: FAILED — SNAPSHOT_CHECK must be auto, require or off, got '${SNAPSHOT_CHECK}'" + exit 2 + ;; +esac + +# The snapshot report is the same entry point with another path, so the caller +# configures one URL. A /healthz served under a prefix keeps that prefix. +derive_api_health_url() { + local u="${1%%\#*}" + u="${u%%\?*}" + u="${u%/}" + case "$u" in + */healthz) printf '%s/api/v1/health' "${u%/healthz}" ;; + *) printf '%s/api/v1/health' "$(printf '%s' "$u" | sed -E 's#^([a-zA-Z][a-zA-Z0-9+.-]*://[^/]+).*#\1#')" ;; + esac +} +API_HEALTH_URL="${API_HEALTH_URL:-$(derive_api_health_url "$URL")}" TMP="$(mktemp -d)" trap 'rm -rf "$TMP"' EXIT @@ -46,23 +87,43 @@ BODY="$TMP/body" HDRS="$TMP/headers" ERR="$TMP/curlerr" -CURL_ARGS=( - -sS - --max-time "$CONNECT_TIMEOUT" - --connect-timeout "$CONNECT_TIMEOUT" - -D "$HDRS" - -o "$BODY" - -w '%{http_code}' -) -if [ -n "${CF_ACCESS_CLIENT_ID:-}" ] && [ -n "${CF_ACCESS_CLIENT_SECRET:-}" ]; then - CURL_ARGS+=( - -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" - -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" +http_get() { + local url="$1" body="$2" hdrs="$3" + local args=( + -sS + --max-time "$CONNECT_TIMEOUT" + --connect-timeout "$CONNECT_TIMEOUT" + -D "$hdrs" + -o "$body" + -w '%{http_code}' ) -fi + if [ -n "${CF_ACCESS_CLIENT_ID:-}" ] && [ -n "${CF_ACCESS_CLIENT_SECRET:-}" ]; then + args+=( + -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" + -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" + ) + fi + curl "${args[@]}" "$url" 2>"$ERR" +} + +# An Access-fronted host answers the probe with a login redirect rather than +# the application, which is a probe-configuration verdict, not an outage. +access_blocked() { + local hdrs="$1" code="$2" location wwwauth + location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$hdrs")" + wwwauth="$(awk 'tolower($0) ~ /^www-authenticate:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$hdrs")" + case "$code" in + 302|401|403) ;; + *) return 1 ;; + esac + case "$location $wwwauth" in + *cloudflareaccess.com*|*Cloudflare-Access*) return 0 ;; + esac + return 1 +} set +e -HTTP_CODE="$(curl "${CURL_ARGS[@]}" "$URL" 2>"$ERR")" +HTTP_CODE="$(http_get "$URL" "$BODY" "$HDRS")" CURL_RC=$? set -e @@ -78,15 +139,9 @@ if [ "$CURL_RC" -ne 0 ]; then exit 1 fi -location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" -wwwauth="$(awk 'tolower($0) ~ /^www-authenticate:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" -if [ "$HTTP_CODE" = "302" ] || [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "403" ]; then - case "$location $wwwauth" in - *cloudflareaccess.com*|*Cloudflare-Access*) - echo "probe-healthz: FAILED — cloudflare access blocked the probe (HTTP ${HTTP_CODE}); cannot verify /healthz url=${URL}" - exit 4 - ;; - esac +if access_blocked "$HDRS" "$HTTP_CODE"; then + echo "probe-healthz: FAILED — cloudflare access blocked the probe (HTTP ${HTTP_CODE}); cannot verify /healthz url=${URL}" + exit 4 fi if [ "$HTTP_CODE" != "200" ]; then @@ -149,18 +204,152 @@ PY eval_rc=$? set -e +# snapshot_verdict asks /api/v1/health and sets SNAP_CLASS / SNAP_TEXT. +# +# SNAP_CLASS is one of: +# ok the worker reported a complete snapshot inside the window +# unverified the instance makes no claim either way — snapshots disabled, +# the worker has not run yet, or the endpoint is not readable +# red the worker reported a failure, or its newest snapshot is older +# than the window +# +# Only "red" is affirmative evidence of a missing restore point, so only it +# pages under SNAPSHOT_CHECK=auto. "unverified" is the ambiguous half that +# would otherwise make every local instance, where snapshots ship disabled, +# cry wolf; SNAPSHOT_CHECK=require is the caller asserting that this instance +# is one where snapshots are expected, which turns it red too. +SNAP_CLASS="" +SNAP_TEXT="" +snapshot_verdict() { + local body="$TMP/snapbody" hdrs="$TMP/snapheaders" code rc out + + set +e + code="$(http_get "$API_HEALTH_URL" "$body" "$hdrs")" + rc=$? + set -e + + if [ "$rc" -ne 0 ]; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health not readable (curl exit ${rc}) url=${API_HEALTH_URL}" + return + fi + if access_blocked "$hdrs" "$code"; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health is behind cloudflare access (HTTP ${code}); the snapshot claim can only be made on the LAN url=${API_HEALTH_URL}" + return + fi + if [ "$code" != "200" ]; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health answered HTTP ${code} url=${API_HEALTH_URL}" + return + fi + + set +e + out="$(python3 - "$body" "$SNAPSHOT_STALE_AFTER_SECONDS" <<'PY' +import datetime, json, sys + +path, stale_after = sys.argv[1], int(sys.argv[2]) + + +def parse(ts): + try: + t = datetime.datetime.fromisoformat(str(ts).strip().replace("Z", "+00:00")) + except ValueError: + return None + return t if t.tzinfo else t.replace(tzinfo=datetime.timezone.utc) + + +try: + body = json.loads(open(path, encoding="utf-8").read()) +except json.JSONDecodeError as e: + print(f"unverified /api/v1/health body is not JSON ({e})") + sys.exit(0) +if not isinstance(body, dict): + print("unverified /api/v1/health body is not an object") + sys.exit(0) + +snap = body.get("snapshot") +if not isinstance(snap, dict): + print("unverified /api/v1/health carries no snapshot field (binary predates the contract?)") + sys.exit(0) + +status = snap.get("status") or "unknown" +last = snap.get("last_run_at") or "" +err = " ".join(str(snap.get("last_error") or "").split())[:160] + +if status == "error": + print(f"red snapshot worker reported error last_run_at={last or 'never'}" + (f" last_error={err}" if err else "")) + sys.exit(0) +if status == "unknown": + print("unverified snapshot status unknown — the worker has not run yet, or snapshots are disabled on this instance") + sys.exit(0) +if status != "ok": + print(f"red snapshot status {status!r} is not a status this contract defines") + sys.exit(0) + +when = parse(last) +if when is None: + print(f"red snapshot status ok but last_run_at is missing or unparseable ({last!r})") + sys.exit(0) +age = int((datetime.datetime.now(datetime.timezone.utc) - when).total_seconds()) +if age > stale_after: + print(f"red newest snapshot is {age}s old last_run_at={last} threshold={stale_after}s") + sys.exit(0) +print(f"ok last run {last} (age {age}s, threshold {stale_after}s)" + (f" dir={snap['last_dir']}" if snap.get("last_dir") else "")) +PY +)" + rc=$? + set -e + if [ "$rc" -ne 0 ]; then + SNAP_CLASS=unverified + SNAP_TEXT="snapshot evaluator exit ${rc} (${out})" + return + fi + SNAP_CLASS="${out%% *}" + SNAP_TEXT="${out#* }" +} + +# Reached only on a green /healthz: a dead process or an unreadable database is +# the louder verdict and keeps the line to itself. +report_green() { + local line="$1" + if [ "$SNAPSHOT_CHECK" = "off" ]; then + echo "$line" + exit 0 + fi + snapshot_verdict + case "$SNAP_CLASS" in + red) + echo "probe-healthz: FAILED — no current database snapshot: ${SNAP_TEXT} url=${API_HEALTH_URL}" + exit 6 + ;; + unverified) + if [ "$SNAPSHOT_CHECK" = "require" ]; then + echo "probe-healthz: FAILED — snapshots are required on this instance and it does not report one: ${SNAP_TEXT}" + exit 6 + fi + echo "${line} | snapshot not asserted (${SNAP_TEXT})" + exit 0 + ;; + *) + echo "${line} | snapshot ${SNAP_TEXT}" + exit 0 + ;; + esac +} + case "$eval_rc" in 0) case "$eval_out" in liveness) - echo "probe-healthz: OK — HTTP 200 liveness (freshness fields absent) url=${URL}" + report_green "probe-healthz: OK — HTTP 200 liveness (freshness fields absent) url=${URL}" ;; fresh*) age="${eval_out#fresh }" - echo "probe-healthz: OK — HTTP 200 ingest age ${age%% *}s (threshold ${STALE_AFTER_SECONDS}s) url=${URL}" + report_green "probe-healthz: OK — HTTP 200 ingest age ${age%% *}s (threshold ${STALE_AFTER_SECONDS}s) url=${URL}" ;; *) - echo "probe-healthz: OK — HTTP 200 url=${URL}" + report_green "probe-healthz: OK — HTTP 200 url=${URL}" ;; esac exit 0 diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index c9be989..865f9fb 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -2,7 +2,12 @@ # # probe-healthz_test.sh — drive probe-healthz.sh against a local mock. # Covers 200 / non-200 / unreachable, plus staleness, empty-DB nulls, -# missing freshness keys, and an Access login redirect. +# missing freshness keys, an Access login redirect, and the snapshot claim +# the probe makes against /api/v1/health on a green /healthz. +# +# The snapshot cases are served under a path prefix (/snap-*/healthz and +# /snap-*/api/v1/health) so that each one also exercises the URL derivation: +# the probe is given only the /healthz URL. set -euo pipefail @@ -42,11 +47,37 @@ expect() { MOCK_PY="$(mktemp)" cat >"$MOCK_PY" <<'PY' from http.server import BaseHTTPRequestHandler, HTTPServer +import datetime import json import os PORT = int(os.environ["PROBE_MOCK_PORT"]) + +def iso(delta): + t = datetime.datetime.now(datetime.timezone.utc) + delta + return t.strftime("%Y-%m-%dT%H:%M:%SZ") + + +NOW = iso(datetime.timedelta()) +LONG_AGO = iso(datetime.timedelta(days=-3)) + +GREEN_HEALTHZ = { + "ok": True, + "spans": 12, + "last_ingest_at": NOW, + "newest_span_age_seconds": 3, +} +STALE_HEALTHZ = dict(GREEN_HEALTHZ, last_ingest_at=LONG_AGO, newest_span_age_seconds=259200) + + +def api_health(snapshot): + body = {"status": "ok", "span_count": 12, "retention": {"status": "ok"}} + if snapshot is not None: + body["snapshot"] = snapshot + return (200, body) + + ROUTES = { "/ok-liveness": (200, {"ok": True, "spans": 3}), "/ok-fresh": ( @@ -96,6 +127,36 @@ ROUTES = { ), "/crash": (500, {"error": "boom"}), "/not-json": (200, b"not json"), + # The instance the non-prefixed cases above are served by: healthy, with a + # snapshot taken moments ago. + "/api/v1/health": api_health( + {"status": "ok", "last_run_at": NOW, "last_dir": "/snapshots/" + NOW} + ), + "/snap-ok/healthz": (200, GREEN_HEALTHZ), + "/snap-ok/api/v1/health": api_health( + {"status": "ok", "last_run_at": NOW, "last_dir": "/snapshots/" + NOW} + ), + "/snap-error/healthz": (200, GREEN_HEALTHZ), + "/snap-error/api/v1/health": api_health( + { + "status": "error", + "last_run_at": LONG_AGO, + "last_error": "export failed: No space left on device", + } + ), + "/snap-stale/healthz": (200, GREEN_HEALTHZ), + "/snap-stale/api/v1/health": api_health({"status": "ok", "last_run_at": LONG_AGO}), + "/snap-unknown/healthz": (200, GREEN_HEALTHZ), + "/snap-unknown/api/v1/health": api_health({"status": "unknown"}), + "/snap-missing/healthz": (200, GREEN_HEALTHZ), + "/snap-missing/api/v1/health": api_health(None), + # No /snap-apidown/api/v1/health route: the endpoint 404s. + "/snap-apidown/healthz": (200, GREEN_HEALTHZ), + # Ingest is stale *and* the snapshot is broken: the louder verdict wins. + "/snap-both-red/healthz": (200, STALE_HEALTHZ), + "/snap-both-red/api/v1/health": api_health( + {"status": "error", "last_run_at": LONG_AGO, "last_error": "export failed"} + ), } @@ -178,6 +239,56 @@ expect "connection refused is unreachable" 1 "unreachable" \ expect "timeout is unreachable" 1 "unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/healthz" +# ---- the snapshot claim, read from /api/v1/health on a green /healthz ---- + +expect "a fresh snapshot is reported on the green line" 0 "snapshot last run" \ + "$PROBE" "$BASE/snap-ok/healthz" +expect "a failed snapshot cycle is red" 6 "snapshot worker reported error" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "a failed snapshot cycle names the worker's error" 6 "No space left on device" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "an overdue snapshot is red" 6 "newest snapshot is" \ + "$PROBE" "$BASE/snap-stale/healthz" +expect "the overdue window is the caller's" 0 "snapshot last run" \ + env SNAPSHOT_STALE_AFTER_SECONDS=999999 "$PROBE" "$BASE/snap-stale/healthz" +expect "a stale ingest outranks a broken snapshot" 3 "ingest stale" \ + "$PROBE" "$BASE/snap-both-red/healthz" + +# The ambiguous half: no claim either way. Red only where the caller says +# snapshots are expected, because "unknown" is also how every local instance +# reports snapshots being off. +expect "status unknown is not red by default" 0 "snapshot not asserted" \ + "$PROBE" "$BASE/snap-unknown/healthz" +expect "status unknown is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-unknown/healthz" +expect "an absent snapshot field degrades, it does not red" 0 "carries no snapshot field" \ + "$PROBE" "$BASE/snap-missing/healthz" +expect "an absent snapshot field is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-missing/healthz" +expect "an unreadable /api/v1/health degrades, it does not red" 0 "answered HTTP 404" \ + "$PROBE" "$BASE/snap-apidown/healthz" +expect "an unreadable /api/v1/health is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-apidown/healthz" + +# Off must not merely ignore the answer — it must not ask the question, which +# is what keeps the probe usable against an Access-fronted host. +off_out="$(SNAPSHOT_CHECK=off "$PROBE" "$BASE/snap-error/healthz" 2>&1)" && off_rc=0 || off_rc=$? +if [ "$off_rc" -eq 0 ] && [ "${off_out#*snapshot}" = "$off_out" ]; then + pass "SNAPSHOT_CHECK=off says nothing about snapshots" +else + fail "SNAPSHOT_CHECK=off — exit $off_rc, output: $off_out" +fi + +expect "an unknown SNAPSHOT_CHECK value is a config failure" 2 "must be auto, require or off" \ + env SNAPSHOT_CHECK=sometimes "$PROBE" "$BASE/snap-ok/healthz" +expect "a non-numeric snapshot window is a config failure" 2 "SNAPSHOT_STALE_AFTER_SECONDS" \ + env SNAPSHOT_STALE_AFTER_SECONDS=soon "$PROBE" "$BASE/snap-ok/healthz" + +# The /healthz URL is the only address the caller gives; a derived address that +# dropped the prefix would hit the healthy root instance and report green. +expect "API_HEALTH_URL overrides the derivation" 6 "snapshot worker reported error" \ + env API_HEALTH_URL="$BASE/snap-error/api/v1/health" "$PROBE" "$BASE/snap-ok/healthz" + echo echo "passed=$PASS failed=$FAIL" [ "$FAIL" -eq 0 ]