Fl0p · Oct 12, 2026 · Oct 12, 2026
diff --git a/docs/.vitepress/config.js b/docs/.vitepress/config.js
@@ -64,6 +64,7 @@ export default defineConfig({
             { text: "ADR-0021 — Recovery Wakes the Alert's Assignee", link: '/decisions/0021-recovery-wakes-the-alerts-assignee' },
             { text: 'ADR-0022 — Health Probe Scheduler Outside This Repo', link: '/decisions/0022-health-probe-scheduler-outside-github' },
             { text: 'ADR-0023 - Production Database Snapshots', link: '/decisions/0023-production-database-snapshots' },
+            { text: 'ADR-0024 - Off-Host Snapshot Copy is a Pi-Side Pull', link: '/decisions/0024-off-host-snapshot-copy-pi-pull' },
           ],
         },
       ],

diff --git a/docs/decisions/0024-off-host-snapshot-copy-pi-pull.md b/docs/decisions/0024-off-host-snapshot-copy-pi-pull.md
@@ -0,0 +1,155 @@
+# ADR 0024 - The off-host copy is a Pi-side pull, and it cannot be an rsync of the volume
+
+**Date:** 2026-10-12
+**Status:** Accepted
+**Deciders:** Prospero (CEO, risk model and mechanism), Daedalus (CTO, transport and implementation)
+
+---
+
+## Context
+
+[ADR-0023](./0023-production-database-snapshots) closes file-level corruption, a bad
+migration and accidental deletion, and says so: snapshots land on the same disk as
+production, "an off-host copy is a separate decision with its own dependencies (where
+to put it, what credential it uses); it is deliberately not in this one."
+
+Those dependencies were the reason it was deferred, and both of them dissolve if the
+copy goes to the Pi: no new account, no new secret, no spend. The one question left
+was the risk model, and it is not a technical one. The CEO answered it on 2026-10-12:
+
+> **"другая машина, то же здание"** is accepted for cotel snapshots. Off-site is not
+> required. Revisit when cotel starts holding data whose loss is materially expensive
+> (billing-grade usage, records the company leans on) - at which point it is a board
+> matter again, because an external credential comes back with it.
+
+So the Pi is the destination, and this ADR is only about what actually reaches it and
+how anyone knows it did.
+
+### What the volume turned out to be
+
+The decision was described as "rsync-pull of the snapshots volume over the existing
+ssh path". **That shape is not implementable, and the reason is worth recording rather
+than working around quietly.** Snapshots live in the Docker named volume
+`cotel-snapshots`, and robmini's `DockerRootDir` is `/var/lib/docker` - a path that
+**does not exist on that host**, because it is inside Docker Desktop's Linux VM:
+
+```
+$ ssh robmini "docker info -f '{{.DockerRootDir}}'; ls -ld /var/lib/docker"
+/var/lib/docker
+ls: /var/lib/docker: No such file or directory
+```
+
+There is no host-side directory for `rsync` to read, so every option below has to
+bring the bytes out through a container.
+
+### Measured, 2026-10-12
+
+| | |
+|---|---|
+| snapshots on the volume | **25**, 118 MB, one per 6 h |
+| one snapshot | 4.8 MB, 9 files |
+| at full `COTEL_SNAPSHOT_KEEP=56` | ~270 MB |
+| free on the Pi's NVMe | 304 GB |
+| robmini host tooling | `rsync`, `tar`, `docker`; macOS 26.6 |
+| runtime image contents | `debian:bookworm-slim` + `ca-certificates`: has `tar`, **no** `rsync` |
+
+Capacity is irrelevant at this size, three orders of magnitude over. The constraint is
+the transport.
+
+---
+
+## Options considered
+
+| Option | Verdict |
+|---|---|
+| 1. `rsync` a host path | **Impossible.** No host path exists for a named volume on macOS |
+| 2. `rsync --rsync-path=` into a throwaway container | Needs `rsync` inside an image (the runtime image has none) plus remote-shell wiring to make the container the rsync peer. Clever, and the thing a future maintainer debugs at 2am |
+| 3. **Per-snapshot `tar` stream over the existing ssh path, driven by a list diff** | **Chosen** |
+| 4. Push from robmini | Rejected: a schedule and a credential on the deploy host, and a host that is gone cannot push |
+
+Option 3 is boring because the data makes it boring. A snapshot directory is
+**immutable once complete** and says so itself - `snapshot.json` is written last and
+nothing else writes it - so the sync rule is a set difference, not a delta algorithm:
+pull the directories that are not here yet, skip the ones that are, ignore the ones
+with no manifest at the source. There is nothing for an incremental protocol to earn
+on a 4.8 MB directory that will never be rewritten.
+
+Option 4 fails the same test ADR-0022 applied to the health probe: the watcher belongs
+on the machine that survives the failure being watched. A push from robmini is a backup
+that stops exactly when it is needed.
+
+### The transport, exercised before this was decided
+
+```
+ssh robmini "docker run --rm -v cotel-snapshots:/snapshots:ro debian:bookworm-slim \
+  tar -cf - -C /snapshots <instant>" | tar -xf - -C <mirror>
+```
+
+One directory, **1.16 s wall clock**, and all **9 of 9 files sha256-identical** to the
+source read back out of the volume. Read-only in two senses: the mount is `:ro`, and
+the production container is not touched at all - the stream comes from a throwaway
+container of the image cotel's own runtime is built on, which is already on that host.
+
+---
+
+## Decision
+
+**A Pi-side pull on a systemd timer, hourly, using option 3's transport.**
+
+- **The script lives in this repo** (`scripts/pull-snapshots.sh`), and the timer in
+  `~/ops` materializes it from `origin/main` at each tick. That is
+  [ADR-0022](./0022-health-probe-scheduler-outside-github)'s pattern, unchanged and for
+  the same reasons: the code path stays reviewed here and reaches the schedule with no
+  sync step.
+- **Hourly, not every 6 h.** A tick that finds nothing new costs one ssh round trip, so
+  the cadence is a free variable; matching it to the snapshot interval would put the
+  mirror's worst case at nearly two intervals behind for no saving.
+- **The mirror lives at `~/cotel-snapshots/`, mode 700** - outside `~/projects`,
+  because that checkout is shared by every agent and a copy inside a repo is a copy
+  that can be committed, and outside `~/ops/reports/`, which is swept on a 31-day
+  retention rule that has nothing to do with this data.
+- **The mirror does not mirror deletions.** It prunes on its own count
+  (`COTEL_SNAPSHOT_MIRROR_KEEP`, default 56, never below one), and it never deletes a
+  local snapshot because the source stopped having it. A sync that propagated source
+  deletions would faithfully reproduce the one accident ADR-0023 named as newly
+  dangerous - `docker volume prune` on a stopped deploy - into the copy that exists to
+  survive it. This is the single reason the implementation is not `rsync --delete`, and
+  it must not be "simplified" later.
+- **Verification is per-file sha256 against the source, plus the manifest**, checked on
+  every pulled directory; a directory that fails either is removed and the tick fails.
+
+---
+
+## Consequences
+
+- **Premises loss stays uncovered**, by decision and not by omission, with the revisit
+  condition quoted above. The Pi and robmini are in the same building.
+- **The Pi holds a second copy of every agent's ingest token.** `users.token` is
+  plaintext by design, and a snapshot that omitted it would not restore to a working
+  instance, so the mirror carries the same secrets as the data volume and is treated
+  the same way: 700, and never inside a repository.
+- **The mirror cannot be restored on the Pi today, and the proof available there is
+  byte-identity rather than a restore.** `--db-import` needs the cotel binary, and the
+  published image is **amd64 only** - `docker manifest inspect` lists one platform, and
+  production's own arm64 image was built locally on robmini by compose. So the Pi can
+  prove the copy is the copy (sha256 + `snapshot.json` row counts), while the restore
+  procedure in
+  [Database Snapshots and Restore](../operations/duckdb-snapshots#restoring-from-a-snapshot)
+  runs wherever a cotel binary for that platform exists. Restorability of the artifact
+  itself is established at the source, with the drill in ADR-0023. Publishing an arm64
+  manifest would make the mirror independently verifiable on the Pi; that is a
+  packaging decision with its own ticket, not a prerequisite for having the copy.
+- **One more script and timer in `~/ops` that this company maintains.** Named as the
+  honest cost when the option was chosen, and accepted. It is also a write into the
+  owner's own maintenance tree rather than a company repo, which is sanctioned per
+  case - the same gate `cotel-healthz.timer` went through.
+- **A failing pull is loud; an absent timer is not.** The unit's non-zero exit reaches
+  `~/ops/ALERT` and a desktop notification through the `OnFailure=` path the other ops
+  units already use, so a pull that breaks is announced. A timer that was never
+  installed, or was disabled, raises nothing - the same residual
+  [ADR-0022](./0022-health-probe-scheduler-outside-github) accepted for the probe's own
+  schedule, and the reason the mirror's newest instant is worth reading in the runbook
+  rather than assumed.
+- **The source keeps its own watcher.** The 10-minute tick already pages when snapshots
+  stop happening on robmini; this ADR adds no claim to `/api/v1/health`, because the
+  mirror is not something production can observe.
diff --git a/docs/decisions/index.md b/docs/decisions/index.md
@@ -31,3 +31,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially.
 | [ADR-0021](./0021-recovery-wakes-the-alerts-assignee) | Recovery wakes the alert's assignee, and dedup is time-bounded | Accepted |
 | [ADR-0022](./0022-health-probe-scheduler-outside-github) | The health probe's scheduler lives outside this repo | Accepted |
 | [ADR-0023](./0023-production-database-snapshots) | Snapshots: `EXPORT DATABASE` to Parquet, on a timer, from inside cotel | Accepted |
+| [ADR-0024](./0024-off-host-snapshot-copy-pi-pull) | The off-host copy is a Pi-side pull, and it cannot be an rsync of the volume | Accepted |
diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md
@@ -234,7 +234,10 @@ of the credentials.
 - **Host loss.** Snapshots land on the same disk as production. They insure
   against file-level corruption, a bad migration and accidental deletion - the
   failures cotel has actually had - and not against the disk or the machine
-  going away. An off-host copy is a separate decision.
+  going away. The off-host copy is decided -
+  [ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull): an hourly pull
+  to the Pi over the existing LAN ssh path, with premises loss accepted as
+  uncovered. Until that pull is running, this bullet is the whole story.
 - **`docker volume prune`.** The snapshots volume is only "in use" while the
   container exists, so a prune on a stopped deploy takes the backups with it.
   This is the one new way the volume layout can bite; no code can prevent it.