feat(backup): back up Matrix — the timeline and the recordings - #87
Merged
Merged
Conversation
The backup stacklet could only archive directories that never change: photos, scanned documents. That left every database out, and for Matrix it left out the part that makes a recording a message. The media store holds the family's actual voice recordings; the database holds who spoke, in which room, when, and which file belongs to which event. Without it those recordings are anonymous blobs. Postgres cannot be rsynced. Its files change under you and a copy taken mid-write will not restore. So a snapshot dumps it instead and packs the dump with the files a homeserver cannot come back without: the signing key, which is its identity, and homeserver.yaml, whose macaroon secret keeps every existing login valid. One dated tarball per run, never overwriting the last, which turns a mutable database into the same append-only shape the vault already keeps. Synapse is the first instance; Paperless and Immich have the same gap and the same declaration will fit them. Verified against the live rig rather than stubs: a real snapshot is 225 KB, restores into a scratch database with zero errors, and comes back with all 251 events and 36 media rows. A restored media row resolves to its file on disk at the right size and the right format. pg_dump takes its own consistent view, so this runs against a live homeserver with nothing stopped. Three decisions worth knowing: - Recordings stay out of the tarball. They ride the incremental archive path, so they are never re-copied and never subject to snapshot retention. A rotating snapshot can age out; an append-only archive cannot, and that is the right home for something irreplaceable. - Snapshots run before the sync, so a dump is never newer than the media it references. Media without a row is a harmless orphan; a row without its media is a broken message. - A failed dump does not stop the archives, because a database that will not dump must not cost you the photos. It does fail the run, because a backup that quietly captured less than it was asked to is the exact thing backups exist to prevent. Local tarballs are pruned to the last 7. That cannot reach the vault: the engine syncs with --ignore-existing and never --delete.
The Matrix media store holds the family's actual recordings. It was in no backup at all, because adding it would have broken every backup on a new install: Synapse does not create the media directory until somebody sends the first photo, and preflight treated one missing source as fatal for the whole run. Preflight now tells two situations apart that used to look alike. Nothing there yet — a directory missing or empty — is skipped and reported, and the run continues. Nothing is at risk: the engine syncs with --ignore-existing and never --delete, so an empty source copies nothing and the vault keeps everything it already held. Some files but far fewer than declared is the signal the guard exists for, and it still aborts the whole run so a human looks first. Skipped sources are reported in the run record, so an empty source shows up in the summary and the Matrix notification rather than quietly not being there. "skipped" was already a status the engine modelled and the notifier rendered; nothing had ever produced it. Only local_content is archived: the originals. Thumbnails regenerate from them and url_cache is a cache of other people's web pages. famstack ships with message and media retention disabled, so nothing in there is ever purged, which is what makes it append-only. Verified on the live rig, both branches: an uncreated media store skips while documents still sync, and a source holding 3 of an expected 500 files aborts.
arthware-dev
changed the title
min_files was a constant a developer wrote in a manifest while guessing at a household they would never see. It could only ever catch "dropped to almost exactly zero", and only when the author happened to pick a non-zero number. A library of 50,000 photos reduced to 11 sails straight past min_files = 10 — precisely the disaster the check existed for. Every source is now judged against what it held on the previous run, read from the history the engine already writes. That baseline is self-calibrating: it means the same thing whether a household has fifty files or five hundred thousand, it needs no manifest field, and the case above becomes unmissable. The vault could not serve as the baseline even though it is right there. It is a cumulative high-water mark, and a family deletes things over the years, so source and vault drift apart legitimately until the comparison is noise. min_files is gone from every manifest and from the wire format, replaced by a rolling flag the orchestrator sets. A rolling source is one pruned on purpose — a snapshot staging area keeps a fixed window — so its shrinking is normal operation rather than a loss. That distinction belongs to the framework, which knows it, not to a stacklet author, who would have to guess again. The guard is deliberately loose at half of last run's count: it is looking for catastrophe, not policing somebody's tidying. The canary remains the separate, precise tripwire for "encrypted in place", which no count can detect. Verified on the live rig: a first run with no history syncs, 57 files down from a recorded 50,000 aborts, and a rolling source shrinking to its window passes.
The guide still said databases and config were not covered and pointed at Time Machine. It now describes what a snapshot is and why a database cannot be rsynced, lists what a Synapse snapshot contains and why the signing key and homeserver.yaml travel with the dump, and gives the restore sequence in the order that matters: database first, recordings second, because media the database does not know about is harmless while a message whose recording is missing is broken. Also documents the source-shrink guard alongside the canary, and corrects the Paperless downgrade note, which claimed the stacklet cannot do databases at all.
A dump only restores into something compatible with what wrote it, and
the failure is not subtle: a Paperless 3.x database will not boot under
2.x, and there is no downgrade. Nothing reads this yet, and no restore
tooling exists. It is recorded now because it is the one part of a
snapshot that cannot be added afterwards, and every tarball taken without
it stays ambiguous forever.
Each manifest now carries the image, version label and digest of every
container in the stacklet, plus the Postgres server version. Live example:
stack-messages-synapse matrixdotorg/synapse:latest
1.160.0 sha256:1231c84d...
stack-messages-db postgres:16-alpine sha256:cf78e766...
postgres 16.15
The digest is the load-bearing field. The homeserver runs a `latest` tag,
which names a different image every month and nothing identifiable in
five years; the digest still names this exact image whenever somebody
comes back to it.
Containers are found by their compose project label rather than a list in
the manifest, so a stacklet cannot forget to declare a service it later
adds. A container without a version label records image and digest alone,
which is what Postgres does.
Version capture never costs the dump. Docker unreachable, a container
stopped, an image pruned: all recorded as empty and the database still
gets written.
The stacklet README and the framework reference both still described the old contract. Three things were wrong, two of them from this branch. `min_files` appears in every manifest example and has its own row in the field table, including the advice to "keep low enough that fresh installs don't trip it" — which is exactly the guidance that made it useless. The field no longer exists; the engine derives its baseline from the source's own previous run. `[[backup.snapshot]]` was documented as "reserved, not yet implemented, declare an archive section today". It ships. Both docs now carry the full declaration, the ordering guarantee, and what the tarball records. The README also still announced the stacklet as scaffold-only with hooks raising NotImplementedError. The engine has been ported and in use for some time; no NotImplementedError remains anywhere in the tree. Its status section now says what is covered, what is not, and that restore is deliberately still manual. Adds a section on the two source guards, the canary and the shrink check, including the note that the skip-when-empty behaviour depends on the engine syncing with --ignore-existing and never --delete.
Adds non-negotiable 12: comments explain the code, not sell it. Literate but neutral, with a before/after example. The comments written in this branch had drifted into persuasive prose: rhetorical contrasts, dramatic framing, closing lines that existed for rhythm rather than information. This rewrites them across the snapshot, voice and scribe modules and their tests, keeping the explanatory depth and changing only the tone.
The discovery tests write their own manifest, so they verify the parser against an example the test also wrote. Renaming a key in stacklets/messages/stacklet.toml left them green, and only a live snapshot would have shown it. That is how the namespacing change reached a commit without unit coverage of the declaration it changed. Adds two tests reading the real manifest through a temporary instance directory. Confirmed they fail when the `container` key is misspelled.
An instance that has been backing up photos and documents for months upgrades into a release that adds snapshots and a second archive. Two ways that could go wrong, both now covered against the real engine and a real APFS vault image. The shrink check reads each source's previous count from history, and every run recorded before this release predates that field. An absent baseline is treated as no baseline, so the first upgraded sync proceeds and records counts the next one can use. The new sources are added without disturbing what is already on the vault, which is immutable and cannot be rewritten. Also narrows the baseline to sources that actually synced. A failed source recorded a count of zero, which was ignored only because zero is falsy; skipping any status other than "ok" states the rule instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backs up the Matrix homeserver. The timeline goes as dated snapshots, the
uploads as an append-only archive.
What lands on the vault
data/messages-synapse/.tar.gzper run: database dump, signing key,homeserver.yaml,MANIFEST.json. Around 220 KB.data/messages-media/Nothing to configure. The next
stack backup syncadds both alongsideexisting photo and document archives; it does not touch what is already
there.
pg_dumpruns against the live server, so there is no downtimeand nobody is logged out.
New manifest contract
A stacklet declares state that cannot be rsynced:
Snapshots run before the sync, so a dump is never newer than the files it
references. Each tarball records the image versions and digests that
produced it, for a restore to check against.
Paperless and Immich have the same gap and the same declaration fits them.
Also in here
min_filesis removed from the manifest contract. Sources are comparedagainst their own previous run instead, which needs no configuration and
catches losses at any scale.
the whole run.
Docs
docs/admin-guide.md: what a snapshot contains, and the restore steps.docs/stack-reference.md: the[[backup.snapshot]]fields.Restore remains manual.
stack backup restoreandon_restorehooks arestill planned.