diff --git a/docs/.vitepress/config.js b/docs/.vitepress/config.js index 890e66a..f54aae5 100644 --- a/docs/.vitepress/config.js +++ b/docs/.vitepress/config.js @@ -64,6 +64,7 @@ export default defineConfig({ { text: "ADR-0021 — Recovery Wakes the Alert's Assignee", link: '/decisions/0021-recovery-wakes-the-alerts-assignee' }, { text: 'ADR-0022 — Health Probe Scheduler Outside This Repo', link: '/decisions/0022-health-probe-scheduler-outside-github' }, { text: 'ADR-0023 - Production Database Snapshots', link: '/decisions/0023-production-database-snapshots' }, + { text: 'ADR-0024 - Off-Host Snapshot Copy is a Pi-Side Pull', link: '/decisions/0024-off-host-snapshot-copy-pi-pull' }, ], }, ], diff --git a/docs/decisions/0024-off-host-snapshot-copy-pi-pull.md b/docs/decisions/0024-off-host-snapshot-copy-pi-pull.md new file mode 100644 index 0000000..ad275c5 --- /dev/null +++ b/docs/decisions/0024-off-host-snapshot-copy-pi-pull.md @@ -0,0 +1,155 @@ +# ADR 0024 - The off-host copy is a Pi-side pull, and it cannot be an rsync of the volume + +**Date:** 2026-10-12 +**Status:** Accepted +**Deciders:** Prospero (CEO, risk model and mechanism), Daedalus (CTO, transport and implementation) + +--- + +## Context + +[ADR-0023](./0023-production-database-snapshots) closes file-level corruption, a bad +migration and accidental deletion, and says so: snapshots land on the same disk as +production, "an off-host copy is a separate decision with its own dependencies (where +to put it, what credential it uses); it is deliberately not in this one." + +Those dependencies were the reason it was deferred, and both of them dissolve if the +copy goes to the Pi: no new account, no new secret, no spend. The one question left +was the risk model, and it is not a technical one. The CEO answered it on 2026-10-12: + +> **"другая машина, то же здание"** is accepted for cotel snapshots. Off-site is not +> required. Revisit when cotel starts holding data whose loss is materially expensive +> (billing-grade usage, records the company leans on) - at which point it is a board +> matter again, because an external credential comes back with it. + +So the Pi is the destination, and this ADR is only about what actually reaches it and +how anyone knows it did. + +### What the volume turned out to be + +The decision was described as "rsync-pull of the snapshots volume over the existing +ssh path". **That shape is not implementable, and the reason is worth recording rather +than working around quietly.** Snapshots live in the Docker named volume +`cotel-snapshots`, and robmini's `DockerRootDir` is `/var/lib/docker` - a path that +**does not exist on that host**, because it is inside Docker Desktop's Linux VM: + +``` +$ ssh robmini "docker info -f '{{.DockerRootDir}}'; ls -ld /var/lib/docker" +/var/lib/docker +ls: /var/lib/docker: No such file or directory +``` + +There is no host-side directory for `rsync` to read, so every option below has to +bring the bytes out through a container. + +### Measured, 2026-10-12 + +| | | +|---|---| +| snapshots on the volume | **25**, 118 MB, one per 6 h | +| one snapshot | 4.8 MB, 9 files | +| at full `COTEL_SNAPSHOT_KEEP=56` | ~270 MB | +| free on the Pi's NVMe | 304 GB | +| robmini host tooling | `rsync`, `tar`, `docker`; macOS 26.6 | +| runtime image contents | `debian:bookworm-slim` + `ca-certificates`: has `tar`, **no** `rsync` | + +Capacity is irrelevant at this size, three orders of magnitude over. The constraint is +the transport. + +--- + +## Options considered + +| Option | Verdict | +|---|---| +| 1. `rsync` a host path | **Impossible.** No host path exists for a named volume on macOS | +| 2. `rsync --rsync-path=` into a throwaway container | Needs `rsync` inside an image (the runtime image has none) plus remote-shell wiring to make the container the rsync peer. Clever, and the thing a future maintainer debugs at 2am | +| 3. **Per-snapshot `tar` stream over the existing ssh path, driven by a list diff** | **Chosen** | +| 4. Push from robmini | Rejected: a schedule and a credential on the deploy host, and a host that is gone cannot push | + +Option 3 is boring because the data makes it boring. A snapshot directory is +**immutable once complete** and says so itself - `snapshot.json` is written last and +nothing else writes it - so the sync rule is a set difference, not a delta algorithm: +pull the directories that are not here yet, skip the ones that are, ignore the ones +with no manifest at the source. There is nothing for an incremental protocol to earn +on a 4.8 MB directory that will never be rewritten. + +Option 4 fails the same test ADR-0022 applied to the health probe: the watcher belongs +on the machine that survives the failure being watched. A push from robmini is a backup +that stops exactly when it is needed. + +### The transport, exercised before this was decided + +``` +ssh robmini "docker run --rm -v cotel-snapshots:/snapshots:ro debian:bookworm-slim \ + tar -cf - -C /snapshots " | tar -xf - -C +``` + +One directory, **1.16 s wall clock**, and all **9 of 9 files sha256-identical** to the +source read back out of the volume. Read-only in two senses: the mount is `:ro`, and +the production container is not touched at all - the stream comes from a throwaway +container of the image cotel's own runtime is built on, which is already on that host. + +--- + +## Decision + +**A Pi-side pull on a systemd timer, hourly, using option 3's transport.** + +- **The script lives in this repo** (`scripts/pull-snapshots.sh`), and the timer in + `~/ops` materializes it from `origin/main` at each tick. That is + [ADR-0022](./0022-health-probe-scheduler-outside-github)'s pattern, unchanged and for + the same reasons: the code path stays reviewed here and reaches the schedule with no + sync step. +- **Hourly, not every 6 h.** A tick that finds nothing new costs one ssh round trip, so + the cadence is a free variable; matching it to the snapshot interval would put the + mirror's worst case at nearly two intervals behind for no saving. +- **The mirror lives at `~/cotel-snapshots/`, mode 700** - outside `~/projects`, + because that checkout is shared by every agent and a copy inside a repo is a copy + that can be committed, and outside `~/ops/reports/`, which is swept on a 31-day + retention rule that has nothing to do with this data. +- **The mirror does not mirror deletions.** It prunes on its own count + (`COTEL_SNAPSHOT_MIRROR_KEEP`, default 56, never below one), and it never deletes a + local snapshot because the source stopped having it. A sync that propagated source + deletions would faithfully reproduce the one accident ADR-0023 named as newly + dangerous - `docker volume prune` on a stopped deploy - into the copy that exists to + survive it. This is the single reason the implementation is not `rsync --delete`, and + it must not be "simplified" later. +- **Verification is per-file sha256 against the source, plus the manifest**, checked on + every pulled directory; a directory that fails either is removed and the tick fails. + +--- + +## Consequences + +- **Premises loss stays uncovered**, by decision and not by omission, with the revisit + condition quoted above. The Pi and robmini are in the same building. +- **The Pi holds a second copy of every agent's ingest token.** `users.token` is + plaintext by design, and a snapshot that omitted it would not restore to a working + instance, so the mirror carries the same secrets as the data volume and is treated + the same way: 700, and never inside a repository. +- **The mirror cannot be restored on the Pi today, and the proof available there is + byte-identity rather than a restore.** `--db-import` needs the cotel binary, and the + published image is **amd64 only** - `docker manifest inspect` lists one platform, and + production's own arm64 image was built locally on robmini by compose. So the Pi can + prove the copy is the copy (sha256 + `snapshot.json` row counts), while the restore + procedure in + [Database Snapshots and Restore](../operations/duckdb-snapshots#restoring-from-a-snapshot) + runs wherever a cotel binary for that platform exists. Restorability of the artifact + itself is established at the source, with the drill in ADR-0023. Publishing an arm64 + manifest would make the mirror independently verifiable on the Pi; that is a + packaging decision with its own ticket, not a prerequisite for having the copy. +- **One more script and timer in `~/ops` that this company maintains.** Named as the + honest cost when the option was chosen, and accepted. It is also a write into the + owner's own maintenance tree rather than a company repo, which is sanctioned per + case - the same gate `cotel-healthz.timer` went through. +- **A failing pull is loud; an absent timer is not.** The unit's non-zero exit reaches + `~/ops/ALERT` and a desktop notification through the `OnFailure=` path the other ops + units already use, so a pull that breaks is announced. A timer that was never + installed, or was disabled, raises nothing - the same residual + [ADR-0022](./0022-health-probe-scheduler-outside-github) accepted for the probe's own + schedule, and the reason the mirror's newest instant is worth reading in the runbook + rather than assumed. +- **The source keeps its own watcher.** The 10-minute tick already pages when snapshots + stop happening on robmini; this ADR adds no claim to `/api/v1/health`, because the + mirror is not something production can observe. diff --git a/docs/decisions/index.md b/docs/decisions/index.md index a90a81f..67ddedc 100644 --- a/docs/decisions/index.md +++ b/docs/decisions/index.md @@ -31,3 +31,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially. | [ADR-0021](./0021-recovery-wakes-the-alerts-assignee) | Recovery wakes the alert's assignee, and dedup is time-bounded | Accepted | | [ADR-0022](./0022-health-probe-scheduler-outside-github) | The health probe's scheduler lives outside this repo | Accepted | | [ADR-0023](./0023-production-database-snapshots) | Snapshots: `EXPORT DATABASE` to Parquet, on a timer, from inside cotel | Accepted | +| [ADR-0024](./0024-off-host-snapshot-copy-pi-pull) | The off-host copy is a Pi-side pull, and it cannot be an rsync of the volume | Accepted | diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 164970f..d4c4efa 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -234,7 +234,10 @@ of the credentials. - **Host loss.** Snapshots land on the same disk as production. They insure against file-level corruption, a bad migration and accidental deletion - the failures cotel has actually had - and not against the disk or the machine - going away. An off-host copy is a separate decision. + going away. The off-host copy is decided - + [ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull): an hourly pull + to the Pi over the existing LAN ssh path, with premises loss accepted as + uncovered. Until that pull is running, this bullet is the whole story. - **`docker volume prune`.** The snapshots volume is only "in use" while the container exists, so a prune on a stopped deploy takes the backups with it. This is the one new way the volume layout can bite; no code can prevent it.