Repository navigation
feat(ops): pull the production snapshots off the deploy host - #145
Merged
Merged
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 31 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
A Pi-side pull, because a Docker named volume on macOS has no host path for rsync to read: the bytes come out of a throwaway read-only container over the existing ssh path. A snapshot directory is immutable once complete, so the sync rule is a set difference and not a delta - pull the complete source instants that are not here, skip the ones that are. Every pulled directory is verified before it lands: per-file sha256 against the same bytes read back out of the source volume, plus snapshot.json parsing and agreeing with the instant it was pulled under. Extraction goes into .partial-<instant> and is renamed only after that passes, so an interrupted pull leaves nothing a later tick reads as present. The mirror does not mirror deletions. It prunes on its own count, never below one, and never deletes a local snapshot because the source stopped having it - propagating a source deletion would reproduce `docker volume prune` on a stopped deploy into the copy that exists to survive it. A test holds the rule. --status answers "is the mirror current" without trusting that a timer ran, --dry-run writes nothing, and the exit codes split source-unreachable from bad-listing from failed-verification from unusable-mirror so the schedule's alert says which one happened. Co-Authored-By: Wayland <wayland@agents.flopbut.local> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fl0p
force-pushed
the
flo-1042-pull-snapshots
branch
from
October 12, 2026 19:23
f1687ee to
ea13b42
Compare
The comment said -n was load-bearing because the tick's stdin is the work list. It is not: the loop reads the list on fd 3, so either guard alone is sufficient. Reverting one keeps the suite green and only reverting both reproduces the silent single-pull success, which is what the comment is there to prevent. Co-Authored-By: Daedalus <daedalus@agents.flopbut.local> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The repo half of the off-host snapshot mirror:
scripts/pull-snapshots.sh, itsunit tests, the CI step that runs them, and the runbook section. Per the ticket's
scope boundary, nothing here installs a timer and nothing here writes into
~/ops- the schedule stays with its own ticket.The transport is the one ADR-0024 measured: a throwaway
--rmcontainer withcotel-snapshotsmounted:ro,tar -cf -over the existing ssh path. Asnapshot directory is immutable once complete, so the sync rule is a set
difference, not a delta.
The bug the real run found
The first production run pulled 1 of 25 snapshots and exited 0.
sshreadsits caller's stdin unless told not to, and the tick's stdin was the list of
instants still to pull - the first pull swallowed the rest of the list and the
script reported success. Fixed with
ssh -nplus reading the work list on fd 3,and the test stub now emulates ssh's stdin consumption so the case is guarded:
reverting either fix turns
several new instants in one tickred.Verification
bash scripts/pull-snapshots_test.sh- green, 81 assertionsNo network, no docker, no deploy host: a stub
sshonPATHserves a directorytree as the volume, and can stream from a tampered copy while hashing the
pristine one, which is how exit 3 is exercised.
A real run from the Pi against production
First run, empty mirror - all 25 snapshots, 26.8 s wall clock, 9 files verified
each:
Second run - nothing to do, one ssh round trip, exit 0:
--status:--dry-runon a mirror that is already current:118 MB on disk, mode 700, 303 GB free on the Pi. The mirror is left in place.
Docs
cd docs && npm run build- VitePress build complete, no dead links.Two things to look at
64is an addition to the ticket's five codes, for an unknown flag ora non-numeric
SNAPSHOT_MIRROR_KEEP/SNAPSHOT_SSH_TIMEOUT.0-4keepexactly the meanings the ticket assigned, and a config error is not a tick
outcome, so folding it into
2or4would make the alert lie about whathappened. Say the word and it collapses into one of the five.
Host lossbullet is a conflict resolution. docs(decisions): ADR-0024 - the off-host snapshot copy is a Pi-side pull #143 rewrote that exactbullet while this branch was open; rebased onto it, the bullet now keeps your
ADR-0024 link and adds the pointer to the new section, with a line saying a
mirror is only as current as its last tick.
SNAPSHOT_MIRROR_KEEP), not the ADR's prose(
COTEL_SNAPSHOT_MIRROR_KEEP) - theCOTEL_prefix in this repo means "readby the binary", and none of these are.
Not in scope, by the ticket
No
~/opswrapper, no systemd unit, no timer. Nothing is scheduled: themirror is current as of this run and will go stale until the schedule half
lands.
🤖 Generated with Claude Code