feat(health): page when the database snapshot stops happening

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- The scheduled health probe watches the database snapshots, not just the process. `scripts/probe-healthz.sh` follows a green `/healthz` with a second question to `/api/v1/health` — the only place the snapshot worker's report exists — and goes red (exit 6) on `status: "error"`, on a `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` (default 43200 s, two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`), and on a `status: "ok"` whose timestamp is missing or unparseable. The worker could previously die or start failing and nothing would say so until a human opened the endpoint, which is the shape of the failure that already cost six days of blind outage. The claim is deliberately kept off `/healthz`: that endpoint is the container liveness contract read by `cotel --healthcheck`, the Docker `HEALTHCHECK` and `scripts/wait-for-healthy.sh`, and a failed backup must not mark a working container unhealthy or fail the next deploy — the boundary is checked at the edge and the backup claim lives on the alert path instead. `status: "unknown"` stays silent by default, because it means both "the worker has not run yet" and "snapshots are disabled", and disabled is the shipped default everywhere but production: a probe reddening on it would lie on every local instance. `SNAPSHOT_CHECK=require` is the caller asserting snapshots are expected here and turns that — along with a missing `snapshot` field and an unreadable `/api/v1/health` — red too; `SNAPSHOT_CHECK=off` skips the question. The `/api/v1/health` address is derived from the `/healthz` URL the probe is already given, so the host wrapper on the Pi needs no edit and the check reaches the timer with the merge. Exit 6 rather than 5, which `~/ops/cotel-healthz.sh` already spends on "the LAN half is green and the public ingest half is not" ([docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md), [docs/operations/health-probe.md](docs/operations/health-probe.md))
 - cotel snapshots its own database. A worker runs `EXPORT DATABASE ... (FORMAT PARQUET, COMPRESSION ZSTD)` at startup and then every `COTEL_SNAPSHOT_INTERVAL`, into one dated directory per snapshot under `COTEL_SNAPSHOT_DIR` (`/snapshots` in compose, its own volume), keeping the `COTEL_SNAPSHOT_KEEP` newest. It had to be taken from inside cotel: DuckDB has one writer and the live process holds the file lock, so no external process can open the database even read-only. Measured on the production host against a probe copy of the live 152.6 MB database, the export holds the single connection for 0.15-0.3 s and writes 4.2 MiB - 1/35th of the file for a full copy of all six tables - and because it runs on that same connection it cannot race the WAL checkpoint: the two are serialised by construction rather than by a lock. The format is the point: Parquet plus a plain-text `schema.sql` is readable by any DuckDB build, and an import *rebuilds* the secondary indexes from the data, so a snapshot of a database with a damaged ART index - the September 2026 failure, a file whose rows were all readable - restores to a healthy one, where a byte copy of the volume reproduces the damage faithfully. `snapshot.json` is written last and by nothing else, so its presence is the only "complete" signal; it carries the row count per table, read back out of the Parquet files, so a restore is checked against what the snapshot claims rather than trusted. A snapshot is only taken when the newest complete one is older than the interval, so a restart - or a crash loop - cannot spend the retained window on snapshots minutes apart. Pruning keeps the newest N complete snapshots, deletes incomplete ones, runs only after a successful export, and leaves directories whose name is not a snapshot instant alone ([ADR-0023](docs/decisions/0023-production-database-snapshots.md), [docs/operations/duckdb-snapshots.md](docs/operations/duckdb-snapshots.md))
 - `cotel --db-import <dir>` restores a snapshot into `COTEL_DB_PATH` and verifies every table against the snapshot's manifest, failing with the table and both numbers named if a count does not match. It needs nothing but the image already on the host - no DuckDB CLI with its version matched by hand, which is the step of the recovery procedure most able to destroy a file. It refuses a target that already holds tables (the snapshot's `schema.sql` issues plain `CREATE TABLE`, so importing over data would fail halfway and leave a mixed database) and a snapshot with no `snapshot.json`. Note that the paths in a snapshot's `load.sql` are **absolute**, so a snapshot cannot be moved or renamed and still be imported; it must be visible at the path it was written to
 - `GET /api/v1/health` reports a `snapshot` object (`status`, `last_run_at`, `last_error`, `last_dir`), and a failed export flips the top-level `status` to `degraded` - the same treatment a failing retention roll-up gets, because a backup that has been failing quietly for a month is worse than a known absent one. `unknown` covers both a worker that has not run yet and snapshots left disabled, which is the default outside the compose file

README.md

@@ -321,6 +321,13 @@ report a dead database as healthy. The same two fields are on
 `GET /api/v1/health` — see
 [docs/operations/api-reference.md](docs/operations/api-reference.md).
 
+The retention and snapshot workers' health is deliberately **not** here. This
+endpoint is what `cotel --healthcheck`, the Docker `HEALTHCHECK` and the deploy's
+`wait-for-healthy.sh` read, so a failing backup reported through `ok` would mark
+a working container unhealthy and fail the next deploy. Those claims live on
+`GET /api/v1/health`, which the health probe reads separately — see
+[Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint).
+
 ### The deploy waits for healthy
 
 `docker compose up -d` returns as soon as the container has *started*, which is
@@ -406,8 +413,10 @@ docker run --rm --entrypoint /usr/local/bin/cotel \
 ```
 
 The worker's last outcome is reported on `GET /api/v1/health` under a `snapshot`
-object, and a failed export degrades the top-level `status`. Full procedure,
-including promoting a restored volume:
+object, and a failed export degrades the top-level `status`. The scheduled
+health probe reads that object on every tick and pages when the backup stops
+producing restore points, so nobody has to open the endpoint by hand. Full
+procedure, including promoting a restored volume and what the probe calls red:
 [Database Snapshots and Restore](docs/operations/duckdb-snapshots.md).
 
 ## Retention defaults
@@ -528,6 +537,9 @@ from the network at startup instead of bundling it).
 | `CLOUDFLARE_TUNNEL_TOKEN` | _(unset)_ | When set, starts `cloudflared tunnel run` before cotel; enables public HTTPS access via Cloudflare Tunnel |
 | `TUNNEL_EDGE_IP_VERSION` | `4` in token mode | Read by `cloudflared`, not by cotel: the address family used to reach the Cloudflare edge (`4`, `6` or `auto`). cloudflared's own default became `auto` in 2026.4.0, which tries whichever family the resolver answers with first and falls back only after a connection has failed; token mode pins `4` unless you set this. Not set in local-config mode, where `config.yml` owns the setting. See [token mode](docs/operations/cloudflare-tunnel-remote.md#the-bundled-cloudflared) |
 | `COTEL_PUBLIC_INGEST_URL` | _(unset)_ | Absolute `http`/`https` URL of the public OTLP ingest endpoint (e.g. `https://cotel-ingest.yourdomain.com`). When set, the Setup page substitutes this URL into the copy-paste Claude Code snippets. |
+| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). |
+| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. |
+| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. |
 | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. |
 
 ## Architecture

docs/index.md

@@ -73,6 +73,9 @@ Restart Claude Code. Telemetry starts flowing immediately.
 | `COTEL_SNAPSHOT_INTERVAL` | `6h` | How often a snapshot is taken (Go duration) |
 | `COTEL_SNAPSHOT_KEEP` | `56` | How many complete snapshots to keep - 14 days at the default interval |
 | `COTEL_SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `docker-compose.yml`, not the binary: the Docker volume mounted at `/snapshots` |
+| `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not the binary: `auto` pages only on an affirmatively broken snapshot, `require` also when the instance claims none, `off` never asks. See [Production health probes](./operations/health-probe#the-snapshot-claim-asked-of-a-second-endpoint) |
+| `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not the binary: how old the newest snapshot may be - two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h` |
+| `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not the binary: overrides the derived `/api/v1/health` address |
 
 ## Data & retention
 

docs/operations/duckdb-snapshots.md

@@ -76,6 +76,72 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \
   cat /snapshots/2026-10-06T12-00-00Z/snapshot.json
 ```
 
+## Who is watching it
+
+Nobody has to remember to run the command above: the scheduled health probe
+asks the same endpoint. The LAN half of
+[the 10-minute tick](./health-probe) reads `/healthz` first and, when that is
+green, asks `/api/v1/health` and classifies the `snapshot` object. A red verdict
+pages the same way a dead process does — two consecutive failures, the
+`[cotel-health-probe]` marker, an issue whose assignment wakes an agent.
+
+The claim is deliberately **not** on `/healthz`. That endpoint is the container
+liveness contract: `cotel --healthcheck`, the Docker `HEALTHCHECK` and
+`scripts/wait-for-healthy.sh` all read it, so folding a failed backup into
+`ok: false` would mark a working container unhealthy and fail the next deploy.
+A missing restore point is an alerting fact, not a liveness fact, so it lives on
+the alert path.
+
+### What is red
+
+| `snapshot` in `/api/v1/health` | Verdict | Why |
+|---|---|---|
+| `status: "error"` | **red** (probe exit 6) | The worker ran and failed. `last_error` is quoted into the alert |
+| `status: "ok"`, `last_run_at` older than `SNAPSHOT_STALE_AFTER_SECONDS` | **red** | The worker is not completing cycles any more; the newest restore point is aging out |
+| `status: "ok"`, `last_run_at` absent or unparseable | **red** | The claim contradicts itself |
+| `status: "unknown"` | silent by default | Means *both* "has not run yet" and "snapshots are disabled", which is the shipped default outside compose — red here would cry wolf on every local instance |
+| no `snapshot` field, or `/api/v1/health` not readable | silent by default | An older binary, or a vantage point that cannot see the endpoint |
+
+The last two rows flip to red under `SNAPSHOT_CHECK=require`, which is the
+caller asserting "snapshots are expected on this instance". Default is `auto`:
+page only on affirmative evidence that the backup is broken. `SNAPSHOT_CHECK=off`
+stops the probe asking at all.
+
+### The threshold
+
+`SNAPSHOT_STALE_AFTER_SECONDS` defaults to **43200 s (12 h)** — two
+`COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`. Two and not one because
+a cycle is scheduled *from* the last one, so a single interval leaves no room
+for the export itself and a tick landing just before the next run would be red
+every time. If you change `COTEL_SNAPSHOT_INTERVAL`, change this with it: the
+probe reads the server's answer, not the server's configuration, and cannot
+know the interval on its own.
+
+### Checking it by hand
+
+```bash
+# the question the probe asks, on the LAN (the dashboard host is behind Access)
+curl -s http://robmini.local:8080/api/v1/health | jq '{status, snapshot}'
+
+# the probe's own verdict, classification and all
+HEALTHZ_URL=http://robmini.local:8080/healthz scripts/probe-healthz.sh
+
+# the same, demanding that snapshots be configured here
+SNAPSHOT_CHECK=require HEALTHZ_URL=http://robmini.local:8080/healthz \
+  scripts/probe-healthz.sh
+
+# on the Pi: both halves of the scheduled tick, paging nothing
+~/ops/cotel-healthz.sh --probe-only
+```
+
+The probe takes one URL — `/healthz` — and derives the `/api/v1/health` address
+from it, so there is no second address to keep in step on the host. A green line
+names the snapshot it found:
+
+```
+probe-healthz: OK — HTTP 200 ingest age 2s (threshold 21600s) url=http://robmini.local:8080/healthz | snapshot last run 2026-10-06T17:39:54Z (age 909s, threshold 43200s) dir=/snapshots/2026-10-06T17-39-54Z
+```
+
 ## Restoring from a snapshot
 
 `cotel --db-import <dir>` creates the database at `COTEL_DB_PATH` from a