From ea13b427d2188456d712a80bfd57afe61e5efae1 Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 12 Oct 2026 21:20:06 +0200 Subject: [PATCH 1/2] feat(ops): pull the production snapshots off the deploy host A Pi-side pull, because a Docker named volume on macOS has no host path for rsync to read: the bytes come out of a throwaway read-only container over the existing ssh path. A snapshot directory is immutable once complete, so the sync rule is a set difference and not a delta - pull the complete source instants that are not here, skip the ones that are. Every pulled directory is verified before it lands: per-file sha256 against the same bytes read back out of the source volume, plus snapshot.json parsing and agreeing with the instant it was pulled under. Extraction goes into .partial- and is renamed only after that passes, so an interrupted pull leaves nothing a later tick reads as present. The mirror does not mirror deletions. It prunes on its own count, never below one, and never deletes a local snapshot because the source stopped having it - propagating a source deletion would reproduce `docker volume prune` on a stopped deploy into the copy that exists to survive it. A test holds the rule. --status answers "is the mirror current" without trusting that a timer ran, --dry-run writes nothing, and the exit codes split source-unreachable from bad-listing from failed-verification from unusable-mirror so the schedule's alert says which one happened. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 5 --- .github/workflows/health-probe.yml | 4 + README.md | 6 + docs/operations/duckdb-snapshots.md | 116 ++++++- scripts/pull-snapshots.sh | 455 ++++++++++++++++++++++++++++ scripts/pull-snapshots_test.sh | 422 ++++++++++++++++++++++++++ 5 files changed, 1000 insertions(+), 3 deletions(-) create mode 100755 scripts/pull-snapshots.sh create mode 100755 scripts/pull-snapshots_test.sh diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index 84d2b1e..5c68584 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -36,6 +36,8 @@ on: - "scripts/probe-edge-ingest_test.sh" - "scripts/page-cotel-health.sh" - "scripts/page-cotel-health_test.sh" + - "scripts/pull-snapshots.sh" + - "scripts/pull-snapshots_test.sh" - ".github/workflows/health-probe.yml" permissions: @@ -54,6 +56,8 @@ jobs: run: bash scripts/probe-edge-ingest_test.sh - name: Run pager tests run: bash scripts/page-cotel-health_test.sh + - name: Run snapshot mirror tests + run: bash scripts/pull-snapshots_test.sh probe-loopback: name: Probe /healthz on the deploy host diff --git a/README.md b/README.md index 416bbed..211c142 100644 --- a/README.md +++ b/README.md @@ -540,6 +540,12 @@ from the network at startup instead of bundling it). | `SNAPSHOT_CHECK` | `auto` | Read by `scripts/probe-healthz.sh`, not by the binary: whether a green `/healthz` is followed by a snapshot check against `/api/v1/health`. `auto` pages only on affirmative failure (`status: error`, or a snapshot older than the window); `require` also pages when the instance makes no snapshot claim at all, which is what you want wherever snapshots are expected; `off` never asks. See [Production health probes](docs/operations/health-probe.md#the-snapshot-claim-asked-of-a-second-endpoint). | | `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | Read by `scripts/probe-healthz.sh`, not by the binary: how old `snapshot.last_run_at` may get before the probe goes red. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; the probe reads the server's answer, not its configuration, so move this when you move the interval. | | `API_HEALTH_URL` | _(derived from the `/healthz` URL)_ | Read by `scripts/probe-healthz.sh`, not by the binary: overrides the `/api/v1/health` address the probe derives from the `/healthz` URL it was given. | +| `SNAPSHOT_SSH_HOST` | `robmini` | Read by `scripts/pull-snapshots.sh`, not by the binary: the ssh destination the off-host snapshot mirror pulls from. See [the off-host mirror](docs/operations/duckdb-snapshots.md#the-off-host-mirror). | +| `SNAPSHOT_VOLUME` | `cotel-snapshots` | Read by `scripts/pull-snapshots.sh`: the Docker volume on that host the snapshots are read out of, through a read-only throwaway container. | +| `SNAPSHOT_MIRROR_DIR` | `$HOME/cotel-snapshots` | Read by `scripts/pull-snapshots.sh`: where the mirror lands on the pulling host. Created mode 700 if its parent exists; it carries the same plaintext ingest tokens as the data volume, so it belongs outside any repository. | +| `SNAPSHOT_MIRROR_KEEP` | `56` | Read by `scripts/pull-snapshots.sh`: how many newest complete snapshots the mirror keeps. Never below one, and never a deletion made because the source stopped having a snapshot. | +| `SNAPSHOT_HELPER_IMAGE` | `debian:bookworm-slim` | Read by `scripts/pull-snapshots.sh`: the image the throwaway read-only container runs. It needs `tar`, `find` and `sha256sum`; cotel's own runtime base already has them and is already on the deploy host. | +| `SNAPSHOT_SSH_TIMEOUT` | `30` | Read by `scripts/pull-snapshots.sh`: ssh connect timeout and keepalive interval in seconds, so a tick against a dead host fails rather than hangs. | | `COTEL_DATA_VOLUME` | `cotel-data-repaired-20261004` | Read by `docker-compose.yml`, not by the binary: the Docker volume mounted at `/data`. Point it at a restored copy to bring an instance up *without* writing to the volume being restored from — Docker has no `volume rename`, so the only other way to serve repaired data under the expected name is to overwrite the damaged original, which is also the forensic evidence. The default is a restored copy, not the `cotel_cotel-data` name compose derives by itself: that volume holds a database the binary can no longer open and is kept untouched as evidence. A fresh install that has neither volume gets the default created empty, which is correct. | ## Architecture diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index d4c4efa..d82221b 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -229,15 +229,125 @@ of every agent's token. Treat the snapshots volume exactly like the data volume: it is not a file to hand around, and an off-host copy of it is an off-host copy of the credentials. +## The off-host mirror + +Snapshots land on the same disk as production, so a second copy lives on +another machine: the Pi pulls the complete snapshots it does not have yet and +keeps them. The decision, the transport it was chosen on and the risk that is +knowingly *not* covered are +[ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull). + +```bash +scripts/pull-snapshots.sh # one tick: pull what is missing, prune +scripts/pull-snapshots.sh --status # is the mirror current? +scripts/pull-snapshots.sh --dry-run # what a tick would do; writes nothing +``` + +The bytes come out through a throwaway `--rm` container with the volume mounted +`:ro`, because a Docker named volume on macOS has no host-side path for `rsync` +to read - it is inside Docker Desktop's VM. Production is never touched. + +### Where it lives, and why there + +`SNAPSHOT_MIRROR_DIR`, default `$HOME/cotel-snapshots`, mode 700. Deliberately +**not** under `~/projects`: that checkout is shared, and a copy inside a +repository is a copy that can be committed. Deliberately **not** under +`~/ops/reports/`, which is swept on a 31-day retention rule that has nothing to +do with this data. The schedule itself is a user systemd timer in the owner's +`~/ops` tree, which materializes the script from `main` at each tick - the same +arrangement the health probe's schedule uses, and for the same reason. + +The mirror carries the same secrets as the data volume: `users.token` is +plaintext by design, so every snapshot in it holds a copy of every agent's +ingest token. Treat the directory like the database itself - mode 700, never +inside a repository, not a thing to hand around. + +### Reading `--status` + +This is what answers "is the mirror current" without trusting that the timer +ran: + +``` +pull-snapshots: status + mirror /home/flop/cotel-snapshots — 25 complete snapshots, 118M on disk, keep 56 + local newest 2026-10-12T17-39-54Z (age 1h37m) + source robmini:cotel-snapshots — newest 2026-10-12T17-39-54Z (age 1h37m), 25 complete + lag 0s — the mirror holds the newest source snapshot + to pull 0 source snapshot(s) not here yet +``` + +`lag` is the gap between the newest snapshot here and the newest one at the +source. At the shipped `6h` interval and an hourly tick, a lag under an hour is +a mirror that is working; a lag that keeps growing means the tick is no longer +running, and nothing else will tell you that - a timer that was never installed +or was disabled raises no alert. + +### Exit codes + +A tick's non-zero exit is what the `~/ops` unit turns into `~/ops/ALERT` plus a +desktop notification, so the codes are meant to be read: + +| | | +|---|---| +| `0` | Tick completed, including "nothing to do" | +| `1` | Source unreachable - ssh or `docker run` failed | +| `2` | The remote listing was empty, or was not snapshot directories at all | +| `3` | A pulled snapshot failed verification; the partial was deleted and nothing landed | +| `4` | The mirror directory is missing and uncreatable, or not writable | +| `64` | Invalid invocation or configuration - not a tick outcome | + +Every pulled directory is verified before it lands: per-file `sha256` against +the same bytes read back out of the source volume, plus `snapshot.json` parsing +and agreeing with the instant it was pulled under. Extraction goes into +`.partial-` and is renamed to `` only after that passes, so an +interrupted pull cannot leave a directory a later tick reads as present. + +### The mirror does not mirror deletions + +Local pruning is `SNAPSHOT_MIRROR_KEEP` newest complete snapshots, never below +one. A local snapshot is **never** deleted because the source stopped having it. +This is the single reason the implementation is not `rsync --delete`: +propagating a source deletion would faithfully reproduce `docker volume prune` +on a stopped deploy - the accident the next section names as the new one - into +the copy that exists to survive it. A test holds the rule, and it must not be +"simplified" later. + +### Restoring from the mirror + +The mirror's directories *are* the snapshot directories, named after their +instants, so this is [the restore procedure above](#restoring-from-a-snapshot) +with the mirror mounted where the volume would be: + +```bash +docker run --rm --entrypoint /usr/local/bin/cotel \ + -v "$HOME/cotel-snapshots:/snapshots:ro" \ + -v cotel-data-restore-probe:/data \ + ghcr.io/flopsstuff/cotel:latest --db-import /snapshots/2026-10-12T17-39-54Z +``` + +`/snapshots` is not a convention here but a requirement: `load.sql` holds +absolute `/snapshots//...` paths, which is also why each mirrored +directory keeps the source's name exactly. + +**That command does not run on the Pi.** `--db-import` needs a cotel binary for +the host's platform and the published image is amd64 only, so the proof +available on the Pi is byte-identity - `sha256` per file and the manifest, both +checked on every pull - rather than a restore. Restore from the mirror wherever +an amd64 cotel binary exists, by copying the directory there; restorability of +the artifact itself is established at the source, with the drill in +[ADR-0023](../decisions/0023-production-database-snapshots). + ## What snapshots do not cover - **Host loss.** Snapshots land on the same disk as production. They insure against file-level corruption, a bad migration and accidental deletion - the failures cotel has actually had - and not against the disk or the machine going away. The off-host copy is decided - - [ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull): an hourly pull - to the Pi over the existing LAN ssh path, with premises loss accepted as - uncovered. Until that pull is running, this bullet is the whole story. + [ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull) - and the pull + that implements it is [the off-host mirror](#the-off-host-mirror): an hourly + pull to the Pi over the existing LAN ssh path, with premises loss accepted as + uncovered because it is another machine in the same building. A mirror is only + as current as its last tick, so read `--status` instead of assuming. - **`docker volume prune`.** The snapshots volume is only "in use" while the container exists, so a prune on a stopped deploy takes the backups with it. This is the one new way the volume layout can bite; no code can prevent it. diff --git a/scripts/pull-snapshots.sh b/scripts/pull-snapshots.sh new file mode 100755 index 0000000..3ca85dd --- /dev/null +++ b/scripts/pull-snapshots.sh @@ -0,0 +1,455 @@ +#!/usr/bin/env bash +# +# pull-snapshots.sh — mirror the production snapshot volume onto this host. +# +# The snapshots live in a Docker named volume on the deploy host, which is +# macOS: a named volume has no host-side path there (it is inside Docker +# Desktop's VM), so rsync has nothing to read and the bytes have to come out +# through a throwaway read-only container. See ADR-0024. +# +# A snapshot directory is immutable once complete — snapshot.json is written +# last and nothing else writes it — so the sync rule is a set difference and +# not a delta: pull the complete source instants that are not here, skip the +# ones that are. +# +# The mirror does NOT mirror deletions. It prunes on its own count +# (SNAPSHOT_MIRROR_KEEP, never below one) and never deletes a local snapshot +# because the source stopped having it. That is the single reason this is not +# `rsync --delete`: propagating a source deletion would faithfully reproduce +# `docker volume prune` on a stopped deploy into the copy that exists to +# survive it. Do not "simplify" it later. +# +# The landed directory is named exactly after its instant because load.sql +# holds absolute /snapshots//... paths — that is what makes the mirror +# restorable by mounting it at /snapshots. +# +# Usage: +# scripts/pull-snapshots.sh # one tick: pull what is missing, prune +# scripts/pull-snapshots.sh --status # is the mirror current? +# scripts/pull-snapshots.sh --dry-run # what a tick would do; writes nothing +# +# Env: +# SNAPSHOT_SSH_HOST default robmini +# SNAPSHOT_VOLUME default cotel-snapshots +# SNAPSHOT_MIRROR_DIR default $HOME/cotel-snapshots +# SNAPSHOT_MIRROR_KEEP default 56 (newest complete local snapshots; min 1) +# SNAPSHOT_HELPER_IMAGE default debian:bookworm-slim +# SNAPSHOT_SSH_TIMEOUT default 30 (connect timeout and keepalive interval) +# +# Exit codes: +# 0 tick completed, including "nothing to do" +# 1 source unreachable (ssh or docker failed) +# 2 remote listing empty or malformed +# 3 a pulled snapshot failed verification (sha256 or manifest); nothing landed +# 4 mirror directory missing or unwritable +# 64 invalid invocation or configuration (not a tick outcome) +# +# Needs only ssh, tar, sha256sum, jq and the standard text tools (awk, sed, +# grep, find, comm, du, date). Nothing runs on the source except a read-only +# throwaway container; the production container is never touched. + +set -euo pipefail + +SSH_HOST="${SNAPSHOT_SSH_HOST:-robmini}" +VOLUME="${SNAPSHOT_VOLUME:-cotel-snapshots}" +MIRROR_DIR="${SNAPSHOT_MIRROR_DIR:-$HOME/cotel-snapshots}" +KEEP="${SNAPSHOT_MIRROR_KEEP:-56}" +HELPER_IMAGE="${SNAPSHOT_HELPER_IMAGE:-debian:bookworm-slim}" +SSH_TIMEOUT="${SNAPSHOT_SSH_TIMEOUT:-30}" + +INSTANT_RE='^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}-[0-9]{2}-[0-9]{2}Z$' + +usage() { + cat <<'EOF' +pull-snapshots.sh — mirror the production snapshot volume onto this host. + +Usage: + scripts/pull-snapshots.sh one tick: pull what is missing, prune + scripts/pull-snapshots.sh --status is the mirror current? + scripts/pull-snapshots.sh --dry-run what a tick would do; writes nothing + +Env: + SNAPSHOT_SSH_HOST default robmini + SNAPSHOT_VOLUME default cotel-snapshots + SNAPSHOT_MIRROR_DIR default $HOME/cotel-snapshots + SNAPSHOT_MIRROR_KEEP default 56 (newest complete local snapshots; min 1) + SNAPSHOT_HELPER_IMAGE default debian:bookworm-slim + SNAPSHOT_SSH_TIMEOUT default 30 + +Exit codes: + 0 tick completed, including "nothing to do" + 1 source unreachable (ssh or docker failed) + 2 remote listing empty or malformed + 3 a pulled snapshot failed verification; nothing landed + 4 mirror directory missing or unwritable + 64 invalid invocation or configuration +EOF +} + +say() { echo "pull-snapshots: $*"; } + +die() { # $1 message, $2 exit code + echo "pull-snapshots: FAILED — $1" + exit "$2" +} + +MODE=tick +if [ "$#" -gt 1 ]; then + die "too many arguments; one of --status, --dry-run, --help" 64 +fi +case "${1:-}" in + "") MODE=tick ;; + --status) MODE=status ;; + --dry-run) MODE=dry-run ;; + -h|--help) usage; exit 0 ;; + *) die "unknown argument '${1}'; one of --status, --dry-run, --help" 64 ;; +esac + +if ! [ "$SSH_TIMEOUT" -gt 0 ] 2>/dev/null; then + die "SNAPSHOT_SSH_TIMEOUT must be a positive integer, got '${SSH_TIMEOUT}'" 64 +fi +if ! [ "$KEEP" -ge 0 ] 2>/dev/null; then + die "SNAPSHOT_MIRROR_KEEP must be a non-negative integer, got '${KEEP}'" 64 +fi +[ "$KEEP" -lt 1 ] && KEEP=1 +if [ -z "$MIRROR_DIR" ] || [ "$MIRROR_DIR" = "/" ]; then + die "SNAPSHOT_MIRROR_DIR must be a real directory path, got '${MIRROR_DIR}'" 64 +fi +MIRROR_DIR="${MIRROR_DIR%/}" + +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +# --- the source ------------------------------------------------------------- + +ssh_remote() { # $1 remote command string + # -n is load-bearing, not hygiene: without it ssh reads the caller's stdin, + # and the tick's stdin is the list of instants still to pull — one pull + # would swallow the rest of the list and the tick would report success. + ssh -n \ + -o BatchMode=yes \ + -o ConnectTimeout="$SSH_TIMEOUT" \ + -o ServerAliveInterval="$SSH_TIMEOUT" \ + -o ServerAliveCountMax=3 \ + "$SSH_HOST" "$1" +} + +in_helper() { # $1 command for the helper container + printf 'docker run --rm -v %s:/snapshots:ro %s %s' "$VOLUME" "$HELPER_IMAGE" "$1" +} + +# The trailing `exit 0` matters: the loop's status is that of the last `[ -f ]` +# test, so a newest directory without a manifest would make a perfectly good +# listing look like a failed ssh. +remote_list_raw() { + ssh_remote "$(in_helper "sh -c 'for d in /snapshots/*/; do [ -f \"\$d/snapshot.json\" ] && basename \"\$d\"; done; exit 0'")" +} + +# Complete source instants, oldest first. Names that are not snapshot instants +# belong to whatever else shares the volume and are ignored, not rejected. +remote_instants() { # $1 output file + local raw="$TMP/remote-raw" + if ! remote_list_raw >"$raw" 2>"$TMP/remote-err"; then + local err + err="$(tr '\n' ' ' <"$TMP/remote-err" | sed 's/[[:space:]]*$//' | cut -c1-200)" + die "source unreachable — ${SSH_HOST}:${VOLUME} listing failed${err:+ (${err})}" 1 + fi + tr -d '\r' <"$raw" | grep -E "$INSTANT_RE" | LC_ALL=C sort >"$1" || true + if [ ! -s "$1" ]; then + local snippet + snippet="$(tr '\n' ' ' <"$raw" | sed 's/[[:space:]]\{1,\}/ /g' | cut -c1-200)" + die "remote listing empty or malformed on ${SSH_HOST}:${VOLUME}${snippet:+ — got: ${snippet}}" 2 + fi +} + +# --- the mirror ------------------------------------------------------------- + +ensure_mirror() { + if [ ! -d "$MIRROR_DIR" ]; then + local parent + parent="$(dirname "$MIRROR_DIR")" + [ -d "$parent" ] || die "mirror directory ${MIRROR_DIR} is missing and its parent ${parent} does not exist" 4 + mkdir -m 700 "$MIRROR_DIR" 2>/dev/null || die "mirror directory ${MIRROR_DIR} is missing and cannot be created" 4 + say "created mirror directory ${MIRROR_DIR} (mode 700)" + fi + [ -w "$MIRROR_DIR" ] && [ -x "$MIRROR_DIR" ] || die "mirror directory ${MIRROR_DIR} is not writable" 4 +} + +local_instants() { # every instant-named directory, oldest first + local d i + for d in "$MIRROR_DIR"/*/; do + [ -d "$d" ] || continue + i="${d%/}" + i="${i##*/}" + [[ "$i" =~ $INSTANT_RE ]] || continue + echo "$i" + done | LC_ALL=C sort +} + +local_complete() { # instant-named directories that hold a manifest, oldest first + local i + while read -r i; do + if [ -n "$i" ] && [ -f "$MIRROR_DIR/$i/snapshot.json" ]; then + echo "$i" + fi + done < <(local_instants) +} + +# A partial from a killed run, and an instant-named directory with no manifest, +# are both unreadable as "present" and both have to go before a pull lands. +clean_leftovers() { + local d i + for d in "$MIRROR_DIR"/.partial-*; do + if [ -e "$d" ]; then + say "removing leftover partial ${d##*/}" + rm -rf "$d" + fi + done + while read -r i; do + if [ -n "$i" ] && [ ! -f "$MIRROR_DIR/$i/snapshot.json" ]; then + say "removing incomplete local snapshot ${i} (no snapshot.json)" + rm -rf "${MIRROR_DIR:?}/$i" + fi + done < <(local_instants) +} + +sha_tree() { # $1 directory — sha256 of every file, path-sorted, paths relative + ( cd "$1" && find . -type f | LC_ALL=C sort | xargs -r sha256sum ) +} + +compare_sha() { # $1 source sums, $2 local sums — prints differences, 1 if any + awk ' + function path(line) { return substr(line, index(line, " ") + 2) } + FNR == NR { remote[path($0)] = $1; next } + { + p = path($0) + seen[p] = 1 + if (!(p in remote)) { printf "%s: not present at the source, local=%s\n", p, $1; bad = 1; next } + if (remote[p] != $1) { printf "%s: source=%s local=%s\n", p, remote[p], $1; bad = 1 } + } + END { + for (p in remote) if (!(p in seen)) { printf "%s: missing from the pulled copy, source=%s\n", p, remote[p]; bad = 1 } + exit bad ? 1 : 0 + } + ' "$1" "$2" +} + +pull_one() { # $1 instant + local i="$1" partial="$MIRROR_DIR/.partial-$1" rcs out manifest_instant + rm -rf "$partial" + mkdir -m 700 "$partial" + + set +e + ssh_remote "$(in_helper "tar -cf - -C /snapshots '$i'")" \ + | tar -xf - -C "$partial" --strip-components=1 + rcs="${PIPESTATUS[*]}" + set -e + case "$rcs" in + "0 0") ;; + 0\ *) rm -rf "$partial"; die "extracting ${i} failed (tar exit ${rcs#* }); nothing landed" 3 ;; + *) rm -rf "$partial"; die "source unreachable while streaming ${i} (ssh exit ${rcs%% *})" 1 ;; + esac + + if ! ssh_remote "$(in_helper "sh -c 'cd /snapshots/$i && find . -type f | LC_ALL=C sort | xargs -r sha256sum'")" \ + >"$TMP/source-sums" 2>/dev/null; then + rm -rf "$partial" + die "source unreachable while hashing ${i}" 1 + fi + if [ ! -s "$TMP/source-sums" ]; then + rm -rf "$partial" + die "source reported no files for ${i}; nothing landed" 3 + fi + sha_tree "$partial" >"$TMP/local-sums" + if ! out="$(compare_sha "$TMP/source-sums" "$TMP/local-sums")"; then + rm -rf "$partial" + while read -r line; do + if [ -n "$line" ]; then say "verify ${i}: ${line}"; fi + done <<<"$out" + die "sha256 verification failed for ${i}; partial removed, nothing landed" 3 + fi + + if ! manifest_instant="$(jq -er '.instant' "$partial/snapshot.json" 2>/dev/null)"; then + rm -rf "$partial" + die "snapshot.json for ${i} is absent, unparseable or has no instant; nothing landed" 3 + fi + if [ "${manifest_instant//:/-}" != "$i" ]; then + rm -rf "$partial" + die "snapshot.json claims instant ${manifest_instant} but was pulled as ${i}; nothing landed" 3 + fi + + if [ -e "$MIRROR_DIR/$i" ]; then + rm -rf "$partial" + die "${MIRROR_DIR}/${i} appeared while it was being pulled; nothing landed" 3 + fi + mv "$partial" "$MIRROR_DIR/$i" + say "pulled ${i} ($(wc -l <"$TMP/source-sums" | tr -d ' ') files verified)" +} + +prune_local() { + local complete=() i n excess k + while read -r i; do + if [ -n "$i" ]; then complete+=("$i"); fi + done < <(local_complete) + n="${#complete[@]}" + if [ "$n" -le "$KEEP" ]; then return 0; fi + excess=$((n - KEEP)) + for ((k = 0; k < excess; k++)); do + say "pruning ${complete[k]} (keeping the ${KEEP} newest)" + rm -rf "${MIRROR_DIR:?}/${complete[k]}" + done +} + +# --- reporting -------------------------------------------------------------- + +instant_epoch() { # $1 instant directory name + local d="${1%%T*}" t="${1#*T}" + t="${t%Z}" + date -u -d "${d}T${t//-/:}Z" +%s 2>/dev/null || echo 0 +} + +fmt_age() { # $1 seconds + local s="$1" + if [ "$s" -lt 0 ]; then s=0; fi + if [ "$s" -lt 60 ]; then + printf '%ds' "$s" + elif [ "$s" -lt 3600 ]; then + printf '%dm%ds' "$((s / 60))" "$((s % 60))" + elif [ "$s" -lt 86400 ]; then + printf '%dh%dm' "$((s / 3600))" "$(((s % 3600) / 60))" + else + printf '%dd%dh' "$((s / 86400))" "$(((s % 86400) / 3600))" + fi +} + +newest_of() { # $1 file of instants, oldest first + tail -n 1 "$1" +} + +count_of() { grep -c . <"$1" || true; } + +# --- modes ------------------------------------------------------------------ + +do_status() { + local remote="$TMP/remote" local_list="$TMP/local" now bytes + now="$(date -u +%s)" + + if [ ! -d "$MIRROR_DIR" ]; then + say "status" + echo " mirror ${MIRROR_DIR} — ABSENT, nothing has ever been pulled here" + remote_instants "$remote" + local rn re + rn="$(newest_of "$remote")" + re="$(instant_epoch "$rn")" + echo " source ${SSH_HOST}:${VOLUME} — newest ${rn} (age $(fmt_age $((now - re)))), $(count_of "$remote") complete" + echo " lag unknown — there is no mirror" + die "mirror directory ${MIRROR_DIR} is missing" 4 + fi + + local_complete >"$local_list" + remote_instants "$remote" + bytes="$(du -sh "$MIRROR_DIR" 2>/dev/null | cut -f1)" + + local ln le rn re lagline + ln="$(newest_of "$local_list")" + rn="$(newest_of "$remote")" + re="$(instant_epoch "$rn")" + + say "status" + echo " mirror ${MIRROR_DIR} — $(count_of "$local_list") complete snapshots, ${bytes:-?} on disk, keep ${KEEP}" + if [ -n "$ln" ]; then + le="$(instant_epoch "$ln")" + echo " local newest ${ln} (age $(fmt_age $((now - le))))" + else + echo " local newest none — the mirror holds no complete snapshot" + fi + echo " source ${SSH_HOST}:${VOLUME} — newest ${rn} (age $(fmt_age $((now - re)))), $(count_of "$remote") complete" + if [ -z "$ln" ]; then + lagline="unknown — the mirror is empty" + elif [ "$ln" = "$rn" ]; then + lagline="0s — the mirror holds the newest source snapshot" + else + lagline="$(fmt_age $((re - le))) behind the source" + fi + echo " lag ${lagline}" + + local missing + missing="$(LC_ALL=C comm -23 "$remote" "$local_list" | grep -c . || true)" + echo " to pull ${missing} source snapshot(s) not here yet" +} + +do_dry_run() { + local remote="$TMP/remote" local_list="$TMP/local" pull="$TMP/pull" d i n + say "dry run — nothing will be written" + if [ ! -d "$MIRROR_DIR" ]; then + echo " mirror ${MIRROR_DIR} — absent, would be created (mode 700)" + : >"$local_list" + else + local_complete >"$local_list" + echo " mirror ${MIRROR_DIR} — $(count_of "$local_list") complete snapshots, keep ${KEEP}" + for d in "$MIRROR_DIR"/.partial-*; do + if [ -e "$d" ]; then echo " would clean leftover partial ${d##*/}"; fi + done + while read -r i; do + if [ -n "$i" ] && [ ! -f "$MIRROR_DIR/$i/snapshot.json" ]; then + echo " would clean incomplete local snapshot ${i} (no snapshot.json)" + fi + done < <(local_instants) + fi + + remote_instants "$remote" + LC_ALL=C comm -23 "$remote" "$local_list" >"$pull" || true + if [ -s "$pull" ]; then + while read -r i; do + [ -n "$i" ] && echo " would pull ${i}" + done <"$pull" + else + echo " would pull nothing — every complete source snapshot is already here" + fi + + n=$(( $(count_of "$local_list") + $(count_of "$pull") )) + if [ "$n" -gt "$KEEP" ]; then + # Oldest first, and only ever from the local set: a source snapshot is + # pulled before it can be pruned, so the arithmetic here is the same + # one a real tick does after pulling. + local over=$((n - KEEP)) k=0 + while read -r i; do + [ -n "$i" ] || continue + [ "$k" -ge "$over" ] && break + echo " would prune ${i}" + k=$((k + 1)) + done < <(cat "$local_list" "$pull" | LC_ALL=C sort) + else + echo " would prune nothing — ${n} complete snapshots against keep ${KEEP}" + fi + echo " would delete nothing because the source stopped having it (the mirror does not mirror deletions)" +} + +do_tick() { + local remote="$TMP/remote" local_list="$TMP/local" pull="$TMP/pull" i pulled=0 skipped=0 + ensure_mirror + clean_leftovers + remote_instants "$remote" + local_complete >"$local_list" + LC_ALL=C comm -23 "$remote" "$local_list" >"$pull" || true + skipped="$(LC_ALL=C comm -12 "$remote" "$local_list" | grep -c . || true)" + + while read -r -u 3 i; do + if [ -n "$i" ]; then + pull_one "$i" + pulled=$((pulled + 1)) + fi + done 3<"$pull" + + prune_local + + local_complete >"$local_list" + local newest + newest="$(newest_of "$local_list")" + say "OK — pulled ${pulled}, already here ${skipped}; mirror holds $(count_of "$local_list") complete snapshots${newest:+, newest ${newest}} in ${MIRROR_DIR}" +} + +case "$MODE" in + status) do_status ;; + dry-run) do_dry_run ;; + tick) do_tick ;; +esac diff --git a/scripts/pull-snapshots_test.sh b/scripts/pull-snapshots_test.sh new file mode 100755 index 0000000..7fbd21d --- /dev/null +++ b/scripts/pull-snapshots_test.sh @@ -0,0 +1,422 @@ +#!/usr/bin/env bash +# +# pull-snapshots_test.sh — drive pull-snapshots.sh against a fake source. +# +# A stub `ssh` on PATH serves a directory tree as if it were the snapshot +# volume, so nothing here needs network, docker or the deploy host. The stub +# can also stream bytes from a tampered copy while hashing the pristine one, +# which is how the verification path is exercised. + +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +PULL="$ROOT/scripts/pull-snapshots.sh" +PASS=0 +FAIL=0 + +fail() { + echo "FAIL: $*" + FAIL=$((FAIL + 1)) +} + +pass() { + echo "PASS: $*" + PASS=$((PASS + 1)) +} + +WORK="$(mktemp -d)" +BIN="$WORK/bin" +mkdir -p "$BIN" +trap 'rm -rf "$WORK"' EXIT + +cat >"$BIN/ssh" <<'STUB' +#!/usr/bin/env bash +# Fake ssh: the remote command is the last argument. FAKE_REMOTE is the volume, +# FAKE_TAR_SRC (default FAKE_REMOTE) is what the tar stream reads from. +set -uo pipefail +cmd="${!#}" + +# Real ssh consumes its caller's stdin unless -n is given, which is how a +# multi-instant tick can lose the rest of its work list. Emulate that: a +# bounded read so an idle stdin cannot hang the suite. +nflag=0 +for a in "$@"; do + if [ "$a" = "-n" ]; then nflag=1; fi +done +if [ "$nflag" = "0" ]; then read -t 0.2 -r _swallowed || true; fi +if [ "${FAKE_SSH_FAIL:-0}" = "1" ]; then + echo "ssh: connect to host ${FAKE_SSH_HOST:-fake} port 22: Connection refused" >&2 + exit 255 +fi +src="${FAKE_TAR_SRC:-$FAKE_REMOTE}" +case "$cmd" in + *"tar -cf -"*) + instant="$(printf '%s' "$cmd" | sed -n "s|.*-C /snapshots '\([^']*\)'.*|\1|p")" + exec tar -cf - -C "$src" "$instant" + ;; + *sha256sum*) + instant="$(printf '%s' "$cmd" | sed -n "s|.*cd /snapshots/\([^ ]*\) &&.*|\1|p")" + cd "$FAKE_REMOTE/$instant" || exit 1 + find . -type f | LC_ALL=C sort | xargs -r sha256sum + ;; + *snapshot.json*) + if [ "${FAKE_LIST_GARBAGE:-0}" = "1" ]; then + echo "Unable to find image 'debian:bookworm-slim' locally" + exit 0 + fi + for d in "$FAKE_REMOTE"/*/; do + if [ -f "$d/snapshot.json" ]; then basename "$d"; fi + done + exit 0 + ;; + *) + echo "fake ssh: unexpected remote command: $cmd" >&2 + exit 97 + ;; +esac +STUB +chmod +x "$BIN/ssh" +export PATH="$BIN:$PATH" + +instant_rfc() { # 2026-10-06T12-00-00Z -> 2026-10-06T12:00:00Z + local d="${1%%T*}" t="${1#*T}" + t="${t%Z}" + printf '%sT%sZ' "$d" "${t//-/:}" +} + +make_snapshot() { # $1 root, $2 instant, $3 optional "incomplete" + local d="$1/$2" + mkdir -p "$d" + printf 'spans parquet bytes for %s\n' "$2" >"$d/spans.parquet" + printf 'users parquet bytes for %s\n' "$2" >"$d/users.parquet" + printf 'CREATE TABLE spans(x INTEGER);\n' >"$d/schema.sql" + printf "COPY spans FROM '/snapshots/%s/spans.parquet' (FORMAT PARQUET);\n" "$2" >"$d/load.sql" + if [ "${3:-}" = "incomplete" ]; then + return 0 + fi + printf '{"instant":"%s","duration_ms":120,"format":"parquet-zstd","schema_version":10,"tables":{"spans":1,"users":1},"directory":"/snapshots/%s"}\n' \ + "$(instant_rfc "$2")" "$2" >"$d/snapshot.json" +} + +# A case gets its own source tree and its own mirror. +new_case() { # $1 name + CASE="$WORK/$1" + FAKE_REMOTE="$CASE/remote" + MIRROR="$CASE/mirror" + mkdir -p "$FAKE_REMOTE" "$MIRROR" + export FAKE_REMOTE + unset FAKE_TAR_SRC FAKE_SSH_FAIL FAKE_LIST_GARBAGE || true +} + +run_pull() { # remaining args passed to the script + SNAPSHOT_SSH_HOST=fake \ + SNAPSHOT_VOLUME=fake-snapshots \ + SNAPSHOT_MIRROR_DIR="$MIRROR" \ + SNAPSHOT_MIRROR_KEEP="${KEEP:-56}" \ + "$PULL" "$@" 2>&1 +} + +# expect NAME WANT_RC [args...] +expect() { + local name="$1" want_rc="$2" + shift 2 + set +e + OUT="$(run_pull "$@")" + RC=$? + set -e + if [ "$RC" -ne "$want_rc" ]; then + fail "$name — exit $RC, want $want_rc; output: $OUT" + return 1 + fi + pass "$name (exit $RC)" + return 0 +} + +mirror_fingerprint() { # every path under the mirror plus the hash of every file + ( cd "$MIRROR" && find . | LC_ALL=C sort && find . -type f | LC_ALL=C sort | xargs -r sha256sum ) +} + +assert_landed() { # $1 instant — present, complete, byte-identical to the source + local i="$1" + if [ ! -f "$MIRROR/$i/snapshot.json" ]; then + fail "landed $i — no snapshot.json in the mirror" + return + fi + local a b + a="$( cd "$FAKE_REMOTE/$i" && find . -type f | LC_ALL=C sort | xargs -r sha256sum )" + b="$( cd "$MIRROR/$i" && find . -type f | LC_ALL=C sort | xargs -r sha256sum )" + if [ "$a" = "$b" ]; then + pass "landed $i — $(printf '%s' "$b" | grep -c .) files sha256-identical to the source" + else + fail "landed $i — content differs from the source" + fi +} + +assert_absent() { # $1 path relative to the mirror + if [ -e "$MIRROR/$1" ]; then fail "absent $1 — still in the mirror"; else pass "absent $1"; fi +} + +assert_present() { # $1 path relative to the mirror + if [ -e "$MIRROR/$1" ]; then pass "present $1"; else fail "present $1 — gone from the mirror"; fi +} + +assert_no_partials() { + local leftovers + leftovers="$(find "$MIRROR" -maxdepth 1 -name '.partial-*' | wc -l | tr -d ' ')" + if [ "$leftovers" = "0" ]; then pass "no leftover partials"; else fail "no leftover partials — found $leftovers"; fi +} + +assert_contains() { # $1 label, $2 needle (searched in $OUT) + case "$OUT" in + *"$2"*) pass "$1" ;; + *) fail "$1 — output did not contain '$2': $OUT" ;; + esac +} + +echo "== nothing new: a tick over a current mirror writes nothing" +new_case nothing-new +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-10T06-00-00Z +BEFORE="$(mirror_fingerprint)" +expect "nothing new is exit 0" 0 +assert_contains "nothing new says it pulled nothing" "pulled 0, already here 2" +if [ "$BEFORE" = "$(mirror_fingerprint)" ]; then + pass "nothing new wrote nothing" +else + fail "nothing new wrote nothing — the mirror changed" +fi + +echo +echo "== one new instant: pulled, verified, landed under its own name" +new_case one-new +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +expect "one new instant is exit 0" 0 +assert_contains "one new instant is reported" "pulled 2026-10-10T06-00-00Z" +assert_landed 2026-10-10T06-00-00Z +assert_no_partials +if grep -q "/snapshots/2026-10-10T06-00-00Z/spans.parquet" "$MIRROR/2026-10-10T06-00-00Z/load.sql"; then + pass "load.sql keeps its absolute /snapshots path, so the landed name must be the instant" +else + fail "load.sql no longer names the absolute snapshot path" +fi + +echo +echo "== several new instants in one tick: every one of them lands" +new_case many-new +for h in 00 06 12 18; do make_snapshot "$FAKE_REMOTE" "2026-10-10T${h}-00-00Z"; done +expect "four new instants in one tick is exit 0" 0 +assert_contains "the tick counts all four" "pulled 4, already here 0" +for h in 00 06 12 18; do assert_landed "2026-10-10T${h}-00-00Z"; done +assert_no_partials + +echo +echo "== a source directory without snapshot.json is never pulled" +new_case no-manifest +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z incomplete +expect "half-written source snapshot is skipped, exit 0" 0 +assert_landed 2026-10-10T00-00-00Z +assert_absent 2026-10-10T06-00-00Z +assert_no_partials + +echo +echo "== sha mismatch: exit 3, partial removed, nothing landed" +new_case sha-mismatch +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +# The stream comes from a tampered copy; the hashes come from the real volume. +FAKE_TAR_SRC="$CASE/tampered" +mkdir -p "$FAKE_TAR_SRC" +cp -r "$FAKE_REMOTE/2026-10-10T00-00-00Z" "$FAKE_TAR_SRC/" +printf 'corrupted on the wire\n' >"$FAKE_TAR_SRC/2026-10-10T00-00-00Z/spans.parquet" +export FAKE_TAR_SRC +expect "corrupted transfer is exit 3" 3 +assert_contains "the mismatching file is named" "./spans.parquet: source=" +assert_contains "verification failure is explicit" "sha256 verification failed" +assert_absent 2026-10-10T00-00-00Z +assert_no_partials +unset FAKE_TAR_SRC + +echo +echo "== ssh failure: exit 1, mirror untouched" +new_case ssh-down +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-09T18-00-00Z +BEFORE="$(mirror_fingerprint)" +export FAKE_SSH_FAIL=1 +expect "unreachable source is exit 1" 1 +assert_contains "unreachable is named as such" "source unreachable" +unset FAKE_SSH_FAIL +if [ "$BEFORE" = "$(mirror_fingerprint)" ]; then + pass "unreachable source left the mirror untouched" +else + fail "unreachable source left the mirror untouched — it changed" +fi + +echo +echo "== a listing that is not snapshots at all: exit 2" +new_case garbage-list +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +export FAKE_LIST_GARBAGE=1 +expect "malformed remote listing is exit 2" 2 +assert_contains "malformed listing quotes what it got" "remote listing empty or malformed" +unset FAKE_LIST_GARBAGE + +echo +echo "== an empty source volume: exit 2" +new_case empty-list +expect "empty remote listing is exit 2" 2 + +echo +echo "== the source list shrinks: local copies stay (the mirror does not mirror deletions)" +new_case source-shrinks +make_snapshot "$FAKE_REMOTE" 2026-10-10T18-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T12-00-00Z +expect "a source that lost snapshots is still exit 0" 0 +assert_present 2026-10-10T00-00-00Z +assert_present 2026-10-10T06-00-00Z +assert_present 2026-10-10T12-00-00Z +assert_landed 2026-10-10T18-00-00Z + +echo +echo "== pruning keeps the newest KEEP and never drops the last one" +new_case prune +make_snapshot "$FAKE_REMOTE" 2026-10-10T18-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T12-00-00Z +KEEP=2 expect "prune to 2 is exit 0" 0 +assert_absent 2026-10-10T00-00-00Z +assert_absent 2026-10-10T06-00-00Z +assert_present 2026-10-10T12-00-00Z +assert_present 2026-10-10T18-00-00Z + +new_case prune-floor +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +KEEP=0 expect "keep 0 is clamped to 1, not to zero snapshots" 0 +assert_absent 2026-10-10T00-00-00Z +assert_present 2026-10-10T06-00-00Z +if [ "$(find "$MIRROR" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')" = "1" ]; then + pass "keep 0 left exactly one snapshot standing" +else + fail "keep 0 did not leave exactly one snapshot: $(find "$MIRROR" -mindepth 1 -maxdepth 1 -type d)" +fi + +echo +echo "== an interrupted previous run: its partial is cleaned up and never read as present" +new_case interrupted +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +# A partial from a killed run, and an instant-named directory that never got +# its manifest: both have to be removed and both instants re-pulled. +mkdir -p "$MIRROR/.partial-2026-10-10T06-00-00Z" +printf 'half a file\n' >"$MIRROR/.partial-2026-10-10T06-00-00Z/spans.parquet" +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z incomplete +expect "a tick after an interruption is exit 0" 0 +assert_contains "the leftover partial is reported" "removing leftover partial .partial-2026-10-10T06-00-00Z" +assert_contains "the manifest-less local copy is reported" "removing incomplete local snapshot 2026-10-10T00-00-00Z" +assert_no_partials +assert_landed 2026-10-10T00-00-00Z +assert_landed 2026-10-10T06-00-00Z + +echo +echo "== an unwritable mirror: exit 4" +new_case unwritable +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +chmod 500 "$MIRROR" +expect "unwritable mirror is exit 4" 4 +assert_contains "unwritable mirror says so" "is not writable" +chmod 700 "$MIRROR" + +echo +echo "== a mirror whose parent does not exist: exit 4, nothing created" +new_case missing-parent +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +MIRROR="$CASE/no/such/parent/mirror" +expect "mirror under a missing parent is exit 4" 4 +assert_contains "the missing parent is named" "does not exist" +if [ -e "$CASE/no" ]; then fail "nothing created — $CASE/no exists"; else pass "nothing created under the missing parent"; fi + +echo +echo "== a missing mirror is created on the first tick" +new_case first-tick +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +rmdir "$MIRROR" +expect "first tick into a missing mirror is exit 0" 0 +assert_contains "mirror creation is reported" "created mirror directory" +assert_landed 2026-10-10T00-00-00Z +if [ "$(stat -c '%a' "$MIRROR")" = "700" ]; then + pass "the mirror is created mode 700 (it carries the same plaintext tokens as the data volume)" +else + fail "the mirror is not mode 700: $(stat -c '%a' "$MIRROR")" +fi + +echo +echo "== --dry-run touches nothing" +new_case dry-run +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +BEFORE="$(mirror_fingerprint)" +KEEP=1 expect "--dry-run is exit 0" 0 --dry-run +assert_contains "--dry-run names what it would pull" "would pull 2026-10-10T06-00-00Z" +assert_contains "--dry-run names what it would prune" "would prune 2026-10-10T00-00-00Z" +assert_contains "--dry-run restates the deletion rule" "does not mirror deletions" +if [ "$BEFORE" = "$(mirror_fingerprint)" ]; then + pass "--dry-run wrote nothing" +else + fail "--dry-run wrote nothing — the mirror changed" +fi + +echo +echo "== --dry-run on a current mirror has nothing to do" +new_case dry-run-current +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +expect "--dry-run on a current mirror is exit 0" 0 --dry-run +assert_contains "--dry-run says there is nothing to pull" "would pull nothing" +assert_contains "--dry-run says there is nothing to prune" "would prune nothing" + +echo +echo "== --status prints the block a human reads" +new_case status +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$FAKE_REMOTE" 2026-10-10T06-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +expect "--status is exit 0" 0 --status +assert_contains "--status names the newest local instant" "local newest 2026-10-10T00-00-00Z (age " +assert_contains "--status counts the mirror and its bytes" "complete snapshots," +assert_contains "--status names the newest source instant" "newest 2026-10-10T06-00-00Z (age " +assert_contains "--status prints the lag" "lag 6h0m behind the source" +assert_contains "--status says how much is outstanding" "to pull 1 source snapshot" + +new_case status-current +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +make_snapshot "$MIRROR" 2026-10-10T00-00-00Z +expect "--status on a current mirror is exit 0" 0 --status +assert_contains "--status calls a current mirror current" "0s — the mirror holds the newest source snapshot" + +new_case status-no-mirror +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +rmdir "$MIRROR" +expect "--status without a mirror is exit 4" 4 --status +assert_contains "--status without a mirror says so" "ABSENT" + +echo +echo "== invocation errors" +new_case usage +make_snapshot "$FAKE_REMOTE" 2026-10-10T00-00-00Z +expect "an unknown flag is exit 64" 64 --mirror-deletions +expect "--help is exit 0" 0 --help +assert_contains "--help lists the exit codes" "4 mirror directory missing or unwritable" +KEEP=not-a-number expect "a non-numeric keep is exit 64" 64 + +echo +echo "passed=$PASS failed=$FAIL" +[ "$FAIL" -eq 0 ] From af009dad64068b3bcfe37ee6bbb822a33f71a541 Mon Sep 17 00:00:00 2001 From: Daedalus Date: Mon, 12 Oct 2026 21:31:23 +0200 Subject: [PATCH 2/2] docs(ops): state the real invariant behind the pull loop's stdin guards The comment said -n was load-bearing because the tick's stdin is the work list. It is not: the loop reads the list on fd 3, so either guard alone is sufficient. Reverting one keeps the suite green and only reverting both reproduces the silent single-pull success, which is what the comment is there to prevent. Co-Authored-By: Daedalus Co-Authored-By: Claude Opus 5 --- scripts/pull-snapshots.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/scripts/pull-snapshots.sh b/scripts/pull-snapshots.sh index 3ca85dd..c14f75b 100755 --- a/scripts/pull-snapshots.sh +++ b/scripts/pull-snapshots.sh @@ -123,9 +123,10 @@ trap 'rm -rf "$TMP"' EXIT # --- the source ------------------------------------------------------------- ssh_remote() { # $1 remote command string - # -n is load-bearing, not hygiene: without it ssh reads the caller's stdin, - # and the tick's stdin is the list of instants still to pull — one pull - # would swallow the rest of the list and the tick would report success. + # -n is load-bearing, not hygiene: without it ssh reads the caller's stdin. + # The tick's loop reads its work list on fd 3 for the same reason, so either + # guard alone is enough — both are kept because the failure they prevent is + # a tick that pulls one snapshot, swallows the rest of the list and exits 0. ssh -n \ -o BatchMode=yes \ -o ConnectTimeout="$SSH_TIMEOUT" \