diff --git a/CHANGELOG.md b/CHANGELOG.md index 8571b52..00532b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- The scheduled health probe watches the database snapshots, not just the process. `scripts/probe-healthz.sh` follows a green `/healthz` with a second question to `/api/v1/health` — the only place the snapshot worker's report exists — and goes red (exit 6) on `status: "error"`, on a `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 43200 s, two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`), and on a `status: "ok"` whose timestamp is missing or unparseable. The worker could previously die or start failing and nothing would say so until a human opened the endpoint, which is the shape of the failure that already cost six days of blind outage. The claim is deliberately kept off `/healthz`: that endpoint is the container liveness contract read by `cotel --healthcheck`, the Docker `HEALTHCHECK` and `scripts/wait-for-healthy.sh`, and a failed backup must not mark a working container unhealthy or fail the next deploy — the boundary is checked at the edge and the backup claim lives on the alert path instead. `status: "unknown"` stays silent by default, because it means both "the worker has not run yet" and "snapshots are disabled", and disabled is the shipped default everywhere but production: a probe reddening on it would lie on every local instance. `SNAPSHOT_CHECK=require` is the caller asserting snapshots are expected here and turns that — along with a missing `snapshot` field and an unreadable `/api/v1/health` — red too; `SNAPSHOT_CHECK=off` skips the question. The `/api/v1/health` address is derived from the `/healthz` URL the probe is already given, so the host wrapper on the Pi needs no edit and the check reaches the timer with the merge. Exit 6 rather than 5, which `~/ops/cotel-healthz.sh` already spends on "the LAN half is green and the public ingest half is not" ([docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md), [docs/operations/health-probe.md](docs/operations/health-probe.md)) - cotel snapshots its own database. A worker runs `EXPORT DATABASE ... (FORMAT PARQUET, COMPRESSION ZSTD)` at startup and then every `COTEL_SNAPSHOT_INTERVAL`, into one dated directory per snapshot under `COTEL_SNAPSHOT_DIR` (`/snapshots` in compose, its own volume), keeping the `COTEL_SNAPSHOT_KEEP` newest. It had to be taken from inside cotel: DuckDB has one writer and the live process holds the file lock, so no external process can open the database even read-only. Measured on the production host against a probe copy of the live 152.6 MB database, the export holds the single connection for 0.15-0.3 s and writes 4.2 MiB - 1/35th of the file for a full copy of all six tables - and because it runs on that same connection it cannot race the WAL checkpoint: the two are serialised by construction rather than by a lock. The format is the point: Parquet plus a plain-text `schema.sql` is readable by any DuckDB build, and an import *rebuilds* the secondary indexes from the data, so a snapshot of a database with a damaged ART index - the September 2026 failure, a file whose rows were all readable - restores to a healthy one, where a byte copy of the volume reproduces the damage faithfully. `snapshot.json` is written last and by nothing else, so its presence is the only "complete" signal; it carries the row count per table, read back out of the Parquet files, so a restore is checked against what the snapshot claims rather than trusted. A snapshot is only taken when the newest complete one is older than the interval, so a restart - or a crash loop - cannot spend the retained window on snapshots minutes apart. Pruning keeps the newest N complete snapshots, deletes incomplete ones, runs only after a successful export, and leaves directories whose name is not a snapshot instant alone ([ADR-0023](docs/decisions/0023-production-database-snapshots.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md)) - `cotel --db-import ` restores a snapshot into `COTEL_DB_PATH` and verifies every table against the snapshot's manifest, failing with the table and both numbers named if a count does not match. It needs nothing but the image already on the host - no DuckDB CLI with its version matched by hand, which is the step of the recovery procedure most able to destroy a file. It refuses a target that already holds tables (the snapshot's `schema.sql` issues plain `CREATE TABLE`, so importing over data would fail halfway and leave a mixed database) and a snapshot with no `snapshot.json`. Note that the paths in a snapshot's `load.sql` are **absolute**, so a snapshot cannot be moved or renamed and still be imported; it must be visible at the path it was written to - `GET /api/v1/health` reports a `snapshot` object (`status`, `last_run_at`, `last_error`, `last_dir`), and a failed export flips the top-level `status` to `degraded` - the same treatment a failing retention roll-up gets, because a backup that has been failing quietly for a month is worse than a known absent one. `unknown` covers both a worker that has not run yet and snapshots left disabled, which is the default outside the compose file diff --git a/README.md b/README.md index d4ec572..416bbed 100644 --- a/README.md +++ b/README.md @@ -321,6 +321,13 @@ report a dead database as healthy. The same two fields are on `GET /api/v1/health` — see [docs/operations/api-reference.md](docs/operations/api-reference.md). +The retention and snapshot workers' health is deliberately **not** here. This +endpoint is what `cotel --healthcheck`, the Docker `HEALTHCHECK` and the deploy's +`wait-for-healthy.sh` read, so a failing backup reported through `ok` would mark +a working container unhealthy and fail the next deploy. Those claims live on +`GET /api/v1/health`, which the health probe reads separately — see +[Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). + ### The deploy waits for healthy `docker compose up -d` returns as soon as the container has *started*, which is @@ -406,8 +413,10 @@ docker run --rm --entrypoint /usr/local/bin/cotel \ ``` The worker's last outcome is reported on `GET /api/v1/health` under a `snapshot` -object, and a failed export degrades the top-level `status`. Full procedure, -including promoting a restored volume: +object, and a failed export degrades the top-level `status`. The scheduled +health probe reads that object on every tick and pages when the backup stops +producing restore points, so nobody has to open the endpoint by hand. Full +procedure, including promoting a restored volume and what the probe calls red: [Database Snapshots and Restore](docs/operations/duckdb-snapshots.md). ## Retention defaults @@ -528,6 +537,9 @@ from the network at startup instead of bundling it). | `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel | | `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) | | `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. | +| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. | +| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. | | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. | ## Architecture diff --git a/docs/index.md b/docs/index.md index 596e1fd..fe94b6a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -73,6 +73,9 @@ Restart Claude Code. Telemetry starts flowing immediately. | `COTEL_SNAPSHOT_INTERVAL` | `6h` | How often a snapshot is taken (Go duration) | | `COTEL_SNAPSHOT_KEEP` | `56` | How many complete snapshots to keep - 14 days at the default interval | | `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not the binary: the Docker volume mounted at `/snapshots` | +| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not the binary: `auto` pages only on an affirmatively broken snapshot, `require` also when the instance claims none, `off` never asks. See [Production health probes](./operations/health-probe#the-snapshot-claim-asked-of-a-second-endpoint) | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not the binary: how old the newest snapshot may be - two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h` | +| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not the binary: overrides the derived `/api/v1/health` address | ## Data & retention diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index c94718b..164970f 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -76,6 +76,72 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ cat /snapshots/2026-10-06T12-00-00Z/snapshot.json ``` +## Who is watching it + +Nobody has to remember to run the command above: the scheduled health probe +asks the same endpoint. The LAN half of +[the 10-minute tick](./health-probe) reads `/healthz` first and, when that is +green, asks `/api/v1/health` and classifies the `snapshot` object. A red verdict +pages the same way a dead process does — two consecutive failures, the +`[cotel-health-probe]` marker, an issue whose assignment wakes an agent. + +The claim is deliberately **not** on `/healthz`. That endpoint is the container +liveness contract: `cotel --healthcheck`, the Docker `HEALTHCHECK` and +`scripts/wait-for-healthy.sh` all read it, so folding a failed backup into +`ok: false` would mark a working container unhealthy and fail the next deploy. +A missing restore point is an alerting fact, not a liveness fact, so it lives on +the alert path. + +### What is red + +| `snapshot` in `/api/v1/health` | Verdict | Why | +|---|---|---| +| `status: "error"` | **red** (probe exit 6) | The worker ran and failed. `last_error` is quoted into the alert | +| `status: "ok"`, `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` | **red** | The worker is not completing cycles any more; the newest restore point is aging out | +| `status: "ok"`, `last_run_at` absent or unparseable | **red** | The claim contradicts itself | +| `status: "unknown"` | silent by default | Means *both* "has not run yet" and "snapshots are disabled", which is the shipped default outside compose — red here would cry wolf on every local instance | +| no `snapshot` field, or `/api/v1/health` not readable | silent by default | An older binary, or a vantage point that cannot see the endpoint | + +The last two rows flip to red under `SNAPSHOT_CHECK=require`, which is the +caller asserting "snapshots are expected on this instance". Default is `auto`: +page only on affirmative evidence that the backup is broken. `SNAPSHOT_CHECK=off` +stops the probe asking at all. + +### The threshold + +`SNAPSHOT_STALE_AFTER_SECONDS` defaults to **43200 s (12 h)** — two +`COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`. Two and not one because +a cycle is scheduled *from* the last one, so a single interval leaves no room +for the export itself and a tick landing just before the next run would be red +every time. If you change `COTEL_SNAPSHOT_INTERVAL`, change this with it: the +probe reads the server's answer, not the server's configuration, and cannot +know the interval on its own. + +### Checking it by hand + +```bash +# the question the probe asks, on the LAN (the dashboard host is behind Access) +curl -s http://robmini.local:8080/api/v1/health | jq '{status, snapshot}' + +# the probe's own verdict, classification and all +HEALTHZ_URL=http://robmini.local:8080/healthz scripts/probe-healthz.sh + +# the same, demanding that snapshots be configured here +SNAPSHOT_CHECK=require HEALTHZ_URL=http://robmini.local:8080/healthz \ + scripts/probe-healthz.sh + +# on the Pi: both halves of the scheduled tick, paging nothing +~/ops/cotel-healthz.sh --probe-only +``` + +The probe takes one URL — `/healthz` — and derives the `/api/v1/health` address +from it, so there is no second address to keep in step on the host. A green line +names the snapshot it found: + +``` +probe-healthz: OK — HTTP 200 ingest age 2s (threshold 21600s) url=http://robmini.local:8080/healthz | snapshot last run 2026-10-06T17:39:54Z (age 909s, threshold 43200s) dir=/snapshots/2026-10-06T17-39-54Z +``` + ## Restoring from a snapshot `cotel --db-import ` creates the database at `COTEL_DB_PATH` from a diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 7514be0..dab5003 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -27,7 +27,7 @@ others see. The first two columns are the scheduled halves of one tick. | Runner | the Pi, `~/ops/cotel-healthz.sh` | the Pi, same tick | `[self-hosted, flopsstuff, docker]` — the deploy host | `ubuntu-latest` | | Target | `http://robmini.local:8080/healthz` (LAN) | `https://otlp.aignite.pl/v1/traces` (internet) | `http://127.0.0.1:8080/healthz` | `https://cotel.aignite.pl/healthz` | | Cloudflare in the path | no | yes (DNS + tunnel) | no | yes (tunnel + Access) | -| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, **and the host being off** | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | all of the LAN half except host-off | the dashboard host through Cloudflare | +| Catches | process dead, crash loop, 503 DuckDB, stale/empty ingest, a snapshot worker that has stopped producing restore points, **and the host being off** | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | all of the LAN half except host-off | the dashboard host through Cloudflare | | Blind to | the tunnel, DNS — a LAN-healthy cotel unreachable from the internet reads green | the database and ingest freshness: this check stops at the auth boundary | **its own host being down** — with the runner off the job queues, producing no colour at all | nothing in the path, but see the Access caveat below | | Access service token | not needed | **not needed** | not needed | required | | Alert dedup marker | `[cotel-health-probe]` | `[cotel-ingest-edge]` | `[cotel-health-probe-drill]` | `[cotel-health-probe-drill-edge]` | @@ -60,6 +60,7 @@ they send the on-call looking in different places: | `HTTP ` | 2 | Crash loop, proxy error, unexpected handler | | `ingest stale` / `empty database` | 3 | Process is up and the DB reads; spans are not being accepted | | `cloudflare access blocked` | 4 | Probe config, **not** an outage — see below | +| `no current database snapshot` | 6 | Everything above is fine; the backup is not happening — see below | A 200 with `ok: true` is not enough. Staleness is a body field (`newest_span_age_seconds`); the endpoint keeps 200 on a quiet instance so the @@ -71,6 +72,54 @@ If `last_ingest_at` / `newest_span_age_seconds` are absent, the probe degrades to liveness (HTTP 200) so it keeps working before that contract is on production. +### The snapshot claim, asked of a second endpoint + +A green `/healthz` is followed by a second question, to `/api/v1/health` on the +same host — the only place the snapshot worker's own report exists. Without it +the backup could die and nothing would say so until somebody opened the +endpoint by hand, which is the failure class that already cost six days of +blind outage once. + +The address is **derived** from the `/healthz` URL (same entry point, other +path), so the host wrapper configures one URL and a change here needs no edit +on the Pi. `API_HEALTH_URL` overrides the derivation. + +| `snapshot` object | Verdict | +|---|---| +| `status: "error"` | red, exit 6, with `last_error` quoted | +| `status: "ok"` and `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 12 h) | red, exit 6 | +| `status: "ok"` and `last_run_at` absent or unparseable | red, exit 6 | +| `status: "unknown"` | not asserted; said on the green line | +| no `snapshot` field, or `/api/v1/health` not readable | not asserted; said on the green line | + +The split is the point. `error` and an overdue timestamp are *affirmative* +evidence that the restore point is gone, and can be trusted anywhere. +`unknown` is not: it means both "the worker has not run yet" and "snapshots are +disabled", and disabled is the shipped default everywhere but production — a +probe that reddened on it would lie on every developer's instance. +`SNAPSHOT_CHECK=require` is the caller asserting that this instance is one +where snapshots are expected, and turns every "cannot assert" into red. +`SNAPSHOT_CHECK=off` skips the question entirely. + +Exit **6** and not 5 because `~/ops/cotel-healthz.sh` already spends 5 on "the +LAN half is green and the public ingest half is not". + +Two vantage points this must not be turned on for: anything reaching cotel +through `cotel.aignite.pl`, where Cloudflare Access answers instead of the +application (the default `auto` degrades quietly there, `require` would page on +a probe-configuration problem), and the edge half, which never reads +`/healthz` at all. + +| Variable | Default | Role | +|---|---|---| +| `SNAPSHOT_CHECK` | `auto` | `auto` pages on affirmative failure only; `require` also pages when the instance makes no snapshot claim; `off` never asks | +| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | How old the newest snapshot may be. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; move it with the interval | +| `API_HEALTH_URL` | derived from the `/healthz` URL | Override when the snapshot report is not at the same entry point | + +What counts as red, why the threshold is two intervals, and how to check it by +hand are in +[Database Snapshots and Restore](./duckdb-snapshots#who-is-watching-it). + ## What the edge half checks, and why it needs no Access token `scripts/probe-edge-ingest.sh` asks `https://otlp.aignite.pl/v1/traces` — the @@ -600,6 +649,9 @@ needs no other change. # the deploy host, from the deploy host — no Access token involved HEALTHZ_URL=http://127.0.0.1:8080/healthz scripts/probe-healthz.sh +# the same, demanding that this instance report a current snapshot +SNAPSHOT_CHECK=require HEALTHZ_URL=http://127.0.0.1:8080/healthz scripts/probe-healthz.sh + # through Cloudflare (needs an Access service token allowed on the app) scripts/probe-healthz.sh @@ -623,7 +675,7 @@ bash scripts/page-cotel-health_test.sh ``` `probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, -`3` stale/empty, `4` Access blocked. +`3` stale/empty, `4` Access blocked, `6` no current database snapshot. `probe-edge-ingest.sh` exit codes: `0` the ingest handler answered 401, `1` unreachable, `2` answered but not with the handler's 401, `4` Access now diff --git a/internal/dashboard/healthz_test.go b/internal/dashboard/healthz_test.go index 0d30391..5d0d242 100644 --- a/internal/dashboard/healthz_test.go +++ b/internal/dashboard/healthz_test.go @@ -121,6 +121,47 @@ func TestHealthzImportedSpanStaysStale(t *testing.T) { } } +// The snapshot worker's report is deliberately absent from this contract, and +// a failing backup is deliberately not a /healthz failure: the container +// HEALTHCHECK and the deploy's wait-for-healthy read this endpoint, so a dead +// backup here would mark a working container unhealthy and fail deploys. The +// claim about backups is made on the alert path instead, by the probe reading +// /api/v1/health. +func TestHealthzIgnoresTheSnapshotWorker(t *testing.T) { + db, ro := openTestDBRW(t) + for k, v := range map[string]string{ + "snapshot_last_status": "error", + "snapshot_last_error": "export failed: No space left on device", + "snapshot_last_run_at": "2026-01-01T00:00:00Z", + } { + if err := db.SetSetting(k, v); err != nil { + t.Fatalf("SetSetting(%s): %v", k, err) + } + } + + code, body := getHealthz(t, dashboard.New(ro)) + if code != http.StatusOK { + t.Errorf("a failing snapshot worker must not change the status code: want 200, got %d", code) + } + if body["ok"] != true { + t.Errorf("a failing snapshot worker must not change ok, got %v", body["ok"]) + } + + want := map[string]bool{ + "ok": true, "spans": true, "last_ingest_at": true, "newest_span_age_seconds": true, + } + for key := range body { + if !want[key] { + t.Errorf("unexpected field %q on /healthz: this contract only ever gains fields by decision", key) + } + } + for key := range want { + if _, present := body[key]; !present { + t.Errorf("field %q is missing from /healthz", key) + } + } +} + // brokenDB fails every single-row read, standing in for an unreadable database. type brokenDB struct { inner dashboard.DB diff --git a/scripts/probe-healthz.sh b/scripts/probe-healthz.sh index 3ca28c7..49b53d1 100755 --- a/scripts/probe-healthz.sh +++ b/scripts/probe-healthz.sh @@ -8,6 +8,13 @@ # answers 200. Missing freshness keys degrade to liveness so the probe # stays useful before that contract is deployed. # +# A green /healthz is then followed by a second question to /api/v1/health, +# which is where the snapshot worker's own report lives. /healthz deliberately +# does not carry it: that endpoint is the container liveness contract, and a +# failed backup must not mark a working container unhealthy or fail a deploy. +# The claim "there is a restore point" therefore has to be made here, on the +# alert path, or nothing makes it at all. +# # Usage: # scripts/probe-healthz.sh [URL] # @@ -17,6 +24,9 @@ # CONNECT_TIMEOUT default 15 # CF_ACCESS_CLIENT_ID optional Cloudflare Access service token # CF_ACCESS_CLIENT_SECRET optional; sent only when both are set +# SNAPSHOT_CHECK auto (default) | require | off +# SNAPSHOT_STALE_AFTER_SECONDS default 43200 (2x the shipped 6h interval) +# API_HEALTH_URL override; derived from the /healthz URL otherwise # # Exit codes: # 0 healthy @@ -24,12 +34,19 @@ # 2 HTTP non-200 # 3 200 but ingest is stale or the database has never accepted a span # 4 Cloudflare Access blocked the probe (cannot see /healthz) +# 6 /healthz is green but the database has no current snapshot +# +# 6 and not 5: ~/ops/cotel-healthz.sh already spends 5 on "the LAN half is +# green and the public ingest half is not", and passes a probe's own code +# through otherwise. set -euo pipefail URL="${1:-${HEALTHZ_URL:-https://cotel.aignite.pl/healthz}}" STALE_AFTER_SECONDS="${STALE_AFTER_SECONDS:-21600}" CONNECT_TIMEOUT="${CONNECT_TIMEOUT:-15}" +SNAPSHOT_CHECK="${SNAPSHOT_CHECK:-auto}" +SNAPSHOT_STALE_AFTER_SECONDS="${SNAPSHOT_STALE_AFTER_SECONDS:-43200}" if ! [ "$STALE_AFTER_SECONDS" -gt 0 ] 2>/dev/null; then echo "probe-healthz: FAILED — STALE_AFTER_SECONDS must be a positive integer, got '${STALE_AFTER_SECONDS}'" @@ -39,6 +56,30 @@ if ! [ "$CONNECT_TIMEOUT" -gt 0 ] 2>/dev/null; then echo "probe-healthz: FAILED — CONNECT_TIMEOUT must be a positive integer, got '${CONNECT_TIMEOUT}'" exit 2 fi +if ! [ "$SNAPSHOT_STALE_AFTER_SECONDS" -gt 0 ] 2>/dev/null; then + echo "probe-healthz: FAILED — SNAPSHOT_STALE_AFTER_SECONDS must be a positive integer, got '${SNAPSHOT_STALE_AFTER_SECONDS}'" + exit 2 +fi +case "$SNAPSHOT_CHECK" in + auto|require|off) ;; + *) + echo "probe-healthz: FAILED — SNAPSHOT_CHECK must be auto, require or off, got '${SNAPSHOT_CHECK}'" + exit 2 + ;; +esac + +# The snapshot report is the same entry point with another path, so the caller +# configures one URL. A /healthz served under a prefix keeps that prefix. +derive_api_health_url() { + local u="${1%%\#*}" + u="${u%%\?*}" + u="${u%/}" + case "$u" in + */healthz) printf '%s/api/v1/health' "${u%/healthz}" ;; + *) printf '%s/api/v1/health' "$(printf '%s' "$u" | sed -E 's#^([a-zA-Z][a-zA-Z0-9+.-]*://[^/]+).*#\1#')" ;; + esac +} +API_HEALTH_URL="${API_HEALTH_URL:-$(derive_api_health_url "$URL")}" TMP="$(mktemp -d)" trap 'rm -rf "$TMP"' EXIT @@ -46,23 +87,43 @@ BODY="$TMP/body" HDRS="$TMP/headers" ERR="$TMP/curlerr" -CURL_ARGS=( - -sS - --max-time "$CONNECT_TIMEOUT" - --connect-timeout "$CONNECT_TIMEOUT" - -D "$HDRS" - -o "$BODY" - -w '%{http_code}' -) -if [ -n "${CF_ACCESS_CLIENT_ID:-}" ] && [ -n "${CF_ACCESS_CLIENT_SECRET:-}" ]; then - CURL_ARGS+=( - -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" - -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" +http_get() { + local url="$1" body="$2" hdrs="$3" + local args=( + -sS + --max-time "$CONNECT_TIMEOUT" + --connect-timeout "$CONNECT_TIMEOUT" + -D "$hdrs" + -o "$body" + -w '%{http_code}' ) -fi + if [ -n "${CF_ACCESS_CLIENT_ID:-}" ] && [ -n "${CF_ACCESS_CLIENT_SECRET:-}" ]; then + args+=( + -H "CF-Access-Client-Id: ${CF_ACCESS_CLIENT_ID}" + -H "CF-Access-Client-Secret: ${CF_ACCESS_CLIENT_SECRET}" + ) + fi + curl "${args[@]}" "$url" 2>"$ERR" +} + +# An Access-fronted host answers the probe with a login redirect rather than +# the application, which is a probe-configuration verdict, not an outage. +access_blocked() { + local hdrs="$1" code="$2" location wwwauth + location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$hdrs")" + wwwauth="$(awk 'tolower($0) ~ /^www-authenticate:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$hdrs")" + case "$code" in + 302|401|403) ;; + *) return 1 ;; + esac + case "$location $wwwauth" in + *cloudflareaccess.com*|*Cloudflare-Access*) return 0 ;; + esac + return 1 +} set +e -HTTP_CODE="$(curl "${CURL_ARGS[@]}" "$URL" 2>"$ERR")" +HTTP_CODE="$(http_get "$URL" "$BODY" "$HDRS")" CURL_RC=$? set -e @@ -78,15 +139,9 @@ if [ "$CURL_RC" -ne 0 ]; then exit 1 fi -location="$(awk 'tolower($0) ~ /^location:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" -wwwauth="$(awk 'tolower($0) ~ /^www-authenticate:/{sub(/^[^:]+:[[:space:]]*/,""); sub(/\r$/,""); print; exit}' "$HDRS")" -if [ "$HTTP_CODE" = "302" ] || [ "$HTTP_CODE" = "401" ] || [ "$HTTP_CODE" = "403" ]; then - case "$location $wwwauth" in - *cloudflareaccess.com*|*Cloudflare-Access*) - echo "probe-healthz: FAILED — cloudflare access blocked the probe (HTTP ${HTTP_CODE}); cannot verify /healthz url=${URL}" - exit 4 - ;; - esac +if access_blocked "$HDRS" "$HTTP_CODE"; then + echo "probe-healthz: FAILED — cloudflare access blocked the probe (HTTP ${HTTP_CODE}); cannot verify /healthz url=${URL}" + exit 4 fi if [ "$HTTP_CODE" != "200" ]; then @@ -149,18 +204,152 @@ PY eval_rc=$? set -e +# snapshot_verdict asks /api/v1/health and sets SNAP_CLASS / SNAP_TEXT. +# +# SNAP_CLASS is one of: +# ok the worker reported a complete snapshot inside the window +# unverified the instance makes no claim either way — snapshots disabled, +# the worker has not run yet, or the endpoint is not readable +# red the worker reported a failure, or its newest snapshot is older +# than the window +# +# Only "red" is affirmative evidence of a missing restore point, so only it +# pages under SNAPSHOT_CHECK=auto. "unverified" is the ambiguous half that +# would otherwise make every local instance, where snapshots ship disabled, +# cry wolf; SNAPSHOT_CHECK=require is the caller asserting that this instance +# is one where snapshots are expected, which turns it red too. +SNAP_CLASS="" +SNAP_TEXT="" +snapshot_verdict() { + local body="$TMP/snapbody" hdrs="$TMP/snapheaders" code rc out + + set +e + code="$(http_get "$API_HEALTH_URL" "$body" "$hdrs")" + rc=$? + set -e + + if [ "$rc" -ne 0 ]; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health not readable (curl exit ${rc}) url=${API_HEALTH_URL}" + return + fi + if access_blocked "$hdrs" "$code"; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health is behind cloudflare access (HTTP ${code}); the snapshot claim can only be made on the LAN url=${API_HEALTH_URL}" + return + fi + if [ "$code" != "200" ]; then + SNAP_CLASS=unverified + SNAP_TEXT="/api/v1/health answered HTTP ${code} url=${API_HEALTH_URL}" + return + fi + + set +e + out="$(python3 - "$body" "$SNAPSHOT_STALE_AFTER_SECONDS" <<'PY' +import datetime, json, sys + +path, stale_after = sys.argv[1], int(sys.argv[2]) + + +def parse(ts): + try: + t = datetime.datetime.fromisoformat(str(ts).strip().replace("Z", "+00:00")) + except ValueError: + return None + return t if t.tzinfo else t.replace(tzinfo=datetime.timezone.utc) + + +try: + body = json.loads(open(path, encoding="utf-8").read()) +except json.JSONDecodeError as e: + print(f"unverified /api/v1/health body is not JSON ({e})") + sys.exit(0) +if not isinstance(body, dict): + print("unverified /api/v1/health body is not an object") + sys.exit(0) + +snap = body.get("snapshot") +if not isinstance(snap, dict): + print("unverified /api/v1/health carries no snapshot field (binary predates the contract?)") + sys.exit(0) + +status = snap.get("status") or "unknown" +last = snap.get("last_run_at") or "" +err = " ".join(str(snap.get("last_error") or "").split())[:160] + +if status == "error": + print(f"red snapshot worker reported error last_run_at={last or 'never'}" + (f" last_error={err}" if err else "")) + sys.exit(0) +if status == "unknown": + print("unverified snapshot status unknown — the worker has not run yet, or snapshots are disabled on this instance") + sys.exit(0) +if status != "ok": + print(f"red snapshot status {status!r} is not a status this contract defines") + sys.exit(0) + +when = parse(last) +if when is None: + print(f"red snapshot status ok but last_run_at is missing or unparseable ({last!r})") + sys.exit(0) +age = int((datetime.datetime.now(datetime.timezone.utc) - when).total_seconds()) +if age > stale_after: + print(f"red newest snapshot is {age}s old last_run_at={last} threshold={stale_after}s") + sys.exit(0) +print(f"ok last run {last} (age {age}s, threshold {stale_after}s)" + (f" dir={snap['last_dir']}" if snap.get("last_dir") else "")) +PY +)" + rc=$? + set -e + if [ "$rc" -ne 0 ]; then + SNAP_CLASS=unverified + SNAP_TEXT="snapshot evaluator exit ${rc} (${out})" + return + fi + SNAP_CLASS="${out%% *}" + SNAP_TEXT="${out#* }" +} + +# Reached only on a green /healthz: a dead process or an unreadable database is +# the louder verdict and keeps the line to itself. +report_green() { + local line="$1" + if [ "$SNAPSHOT_CHECK" = "off" ]; then + echo "$line" + exit 0 + fi + snapshot_verdict + case "$SNAP_CLASS" in + red) + echo "probe-healthz: FAILED — no current database snapshot: ${SNAP_TEXT} url=${API_HEALTH_URL}" + exit 6 + ;; + unverified) + if [ "$SNAPSHOT_CHECK" = "require" ]; then + echo "probe-healthz: FAILED — snapshots are required on this instance and it does not report one: ${SNAP_TEXT}" + exit 6 + fi + echo "${line} | snapshot not asserted (${SNAP_TEXT})" + exit 0 + ;; + *) + echo "${line} | snapshot ${SNAP_TEXT}" + exit 0 + ;; + esac +} + case "$eval_rc" in 0) case "$eval_out" in liveness) - echo "probe-healthz: OK — HTTP 200 liveness (freshness fields absent) url=${URL}" + report_green "probe-healthz: OK — HTTP 200 liveness (freshness fields absent) url=${URL}" ;; fresh*) age="${eval_out#fresh }" - echo "probe-healthz: OK — HTTP 200 ingest age ${age%% *}s (threshold ${STALE_AFTER_SECONDS}s) url=${URL}" + report_green "probe-healthz: OK — HTTP 200 ingest age ${age%% *}s (threshold ${STALE_AFTER_SECONDS}s) url=${URL}" ;; *) - echo "probe-healthz: OK — HTTP 200 url=${URL}" + report_green "probe-healthz: OK — HTTP 200 url=${URL}" ;; esac exit 0 diff --git a/scripts/probe-healthz_test.sh b/scripts/probe-healthz_test.sh index c9be989..865f9fb 100755 --- a/scripts/probe-healthz_test.sh +++ b/scripts/probe-healthz_test.sh @@ -2,7 +2,12 @@ # # probe-healthz_test.sh — drive probe-healthz.sh against a local mock. # Covers 200 / non-200 / unreachable, plus staleness, empty-DB nulls, -# missing freshness keys, and an Access login redirect. +# missing freshness keys, an Access login redirect, and the snapshot claim +# the probe makes against /api/v1/health on a green /healthz. +# +# The snapshot cases are served under a path prefix (/snap-*/healthz and +# /snap-*/api/v1/health) so that each one also exercises the URL derivation: +# the probe is given only the /healthz URL. set -euo pipefail @@ -42,11 +47,37 @@ expect() { MOCK_PY="$(mktemp)" cat >"$MOCK_PY" <<'PY' from http.server import BaseHTTPRequestHandler, HTTPServer +import datetime import json import os PORT = int(os.environ["PROBE_MOCK_PORT"]) + +def iso(delta): + t = datetime.datetime.now(datetime.timezone.utc) + delta + return t.strftime("%Y-%m-%dT%H:%M:%SZ") + + +NOW = iso(datetime.timedelta()) +LONG_AGO = iso(datetime.timedelta(days=-3)) + +GREEN_HEALTHZ = { + "ok": True, + "spans": 12, + "last_ingest_at": NOW, + "newest_span_age_seconds": 3, +} +STALE_HEALTHZ = dict(GREEN_HEALTHZ, last_ingest_at=LONG_AGO, newest_span_age_seconds=259200) + + +def api_health(snapshot): + body = {"status": "ok", "span_count": 12, "retention": {"status": "ok"}} + if snapshot is not None: + body["snapshot"] = snapshot + return (200, body) + + ROUTES = { "/ok-liveness": (200, {"ok": True, "spans": 3}), "/ok-fresh": ( @@ -96,6 +127,36 @@ ROUTES = { ), "/crash": (500, {"error": "boom"}), "/not-json": (200, b"not json"), + # The instance the non-prefixed cases above are served by: healthy, with a + # snapshot taken moments ago. + "/api/v1/health": api_health( + {"status": "ok", "last_run_at": NOW, "last_dir": "/snapshots/" + NOW} + ), + "/snap-ok/healthz": (200, GREEN_HEALTHZ), + "/snap-ok/api/v1/health": api_health( + {"status": "ok", "last_run_at": NOW, "last_dir": "/snapshots/" + NOW} + ), + "/snap-error/healthz": (200, GREEN_HEALTHZ), + "/snap-error/api/v1/health": api_health( + { + "status": "error", + "last_run_at": LONG_AGO, + "last_error": "export failed: No space left on device", + } + ), + "/snap-stale/healthz": (200, GREEN_HEALTHZ), + "/snap-stale/api/v1/health": api_health({"status": "ok", "last_run_at": LONG_AGO}), + "/snap-unknown/healthz": (200, GREEN_HEALTHZ), + "/snap-unknown/api/v1/health": api_health({"status": "unknown"}), + "/snap-missing/healthz": (200, GREEN_HEALTHZ), + "/snap-missing/api/v1/health": api_health(None), + # No /snap-apidown/api/v1/health route: the endpoint 404s. + "/snap-apidown/healthz": (200, GREEN_HEALTHZ), + # Ingest is stale *and* the snapshot is broken: the louder verdict wins. + "/snap-both-red/healthz": (200, STALE_HEALTHZ), + "/snap-both-red/api/v1/health": api_health( + {"status": "error", "last_run_at": LONG_AGO, "last_error": "export failed"} + ), } @@ -178,6 +239,56 @@ expect "connection refused is unreachable" 1 "unreachable" \ expect "timeout is unreachable" 1 "unreachable" \ env CONNECT_TIMEOUT=1 "$PROBE" "http://192.0.2.1:9/healthz" +# ---- the snapshot claim, read from /api/v1/health on a green /healthz ---- + +expect "a fresh snapshot is reported on the green line" 0 "snapshot last run" \ + "$PROBE" "$BASE/snap-ok/healthz" +expect "a failed snapshot cycle is red" 6 "snapshot worker reported error" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "a failed snapshot cycle names the worker's error" 6 "No space left on device" \ + "$PROBE" "$BASE/snap-error/healthz" +expect "an overdue snapshot is red" 6 "newest snapshot is" \ + "$PROBE" "$BASE/snap-stale/healthz" +expect "the overdue window is the caller's" 0 "snapshot last run" \ + env SNAPSHOT_STALE_AFTER_SECONDS=999999 "$PROBE" "$BASE/snap-stale/healthz" +expect "a stale ingest outranks a broken snapshot" 3 "ingest stale" \ + "$PROBE" "$BASE/snap-both-red/healthz" + +# The ambiguous half: no claim either way. Red only where the caller says +# snapshots are expected, because "unknown" is also how every local instance +# reports snapshots being off. +expect "status unknown is not red by default" 0 "snapshot not asserted" \ + "$PROBE" "$BASE/snap-unknown/healthz" +expect "status unknown is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-unknown/healthz" +expect "an absent snapshot field degrades, it does not red" 0 "carries no snapshot field" \ + "$PROBE" "$BASE/snap-missing/healthz" +expect "an absent snapshot field is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-missing/healthz" +expect "an unreadable /api/v1/health degrades, it does not red" 0 "answered HTTP 404" \ + "$PROBE" "$BASE/snap-apidown/healthz" +expect "an unreadable /api/v1/health is red when snapshots are required" 6 "snapshots are required" \ + env SNAPSHOT_CHECK=require "$PROBE" "$BASE/snap-apidown/healthz" + +# Off must not merely ignore the answer — it must not ask the question, which +# is what keeps the probe usable against an Access-fronted host. +off_out="$(SNAPSHOT_CHECK=off "$PROBE" "$BASE/snap-error/healthz" 2>&1)" && off_rc=0 || off_rc=$? +if [ "$off_rc" -eq 0 ] && [ "${off_out#*snapshot}" = "$off_out" ]; then + pass "SNAPSHOT_CHECK=off says nothing about snapshots" +else + fail "SNAPSHOT_CHECK=off — exit $off_rc, output: $off_out" +fi + +expect "an unknown SNAPSHOT_CHECK value is a config failure" 2 "must be auto, require or off" \ + env SNAPSHOT_CHECK=sometimes "$PROBE" "$BASE/snap-ok/healthz" +expect "a non-numeric snapshot window is a config failure" 2 "SNAPSHOT_STALE_AFTER_SECONDS" \ + env SNAPSHOT_STALE_AFTER_SECONDS=soon "$PROBE" "$BASE/snap-ok/healthz" + +# The /healthz URL is the only address the caller gives; a derived address that +# dropped the prefix would hit the healthy root instance and report green. +expect "API_HEALTH_URL overrides the derivation" 6 "snapshot worker reported error" \ + env API_HEALTH_URL="$BASE/snap-error/api/v1/health" "$PROBE" "$BASE/snap-ok/healthz" + echo echo "passed=$PASS failed=$FAIL" [ "$FAIL" -eq 0 ]