feat(backup): back up Matrix — the timeline and the recordings

AGENTS.md

@@ -89,6 +89,7 @@ Apply to every role, every session.
 9. **Announce actions before running them.** No silent long running or integration test runs, scripts, or background commands.
 10. **No em dashes in user-facing prose.** Use hyphens or sentence breaks.
 11. **Put data in through the front door.** Seed and exercise a stacklet the way a family does, never by writing to the service behind it. Documents go through `tools/family-docs/ingest.py` (Matrix room -> archivist -> OCR -> classify -> mirror), never a direct `POST /api/documents/post_document/`. The back door skips the pipeline, so what lands is not what users get: no tags, no correspondent, no document type, no rewritten title, no summary note, no vault entry. Any conclusion drawn from that data is about a system famstack does not ship.
+12. **Comments explain the code, not sell it.** Literate but neutral: state the constraint and the reason, without rhetoric or drama. Write `# Tags are mutable, so only the digest identifies the image this dump came from.` rather than `# Tags lie over time -- the digest is the load-bearing field that still names this exact image in five years.`
 
 ## Deeper docs (load on demand)
 

docs/admin-guide.md

@@ -683,7 +683,7 @@ All commands output JSON when piped. Use `--json` to force it, `--pretty` to for
 
 ## Backups
 
-This is the part everyone skips and regrets. famstack ships an opt-in backup stacklet for the irreplaceable file-level data (photo originals, scanned documents). It does not yet cover stacklet databases or your config files; for those, layer it with Time Machine or a periodic tar.
+This is the part everyone skips and regrets. famstack ships an opt-in backup stacklet that covers the irreplaceable data: photo originals, scanned documents, and everything in your family chat including the voice messages. Immich's and Paperless's databases are not covered yet, so layer it with Time Machine or a periodic tar if you want full coverage today.
 
 ### The backup stacklet
 
@@ -695,15 +695,75 @@ Run `stack up backup`. You need an APFS-formatted external drive plugged in. The
 |---|---|
 | Immich photo originals | `/Volumes/<disk>/data/photos-library/` |
 | Paperless archived PDFs | `/Volumes/<disk>/data/docs-media/` |
+| Matrix uploads: voice messages, photos, files | `/Volumes/<disk>/data/messages-media/` |
+| Matrix timeline, as dated snapshots | `/Volumes/<disk>/data/messages-synapse/` |
 
-Postgres databases for both are not backed up yet. You get your files back but lose albums, tags, custom fields, and saved views. Pg-dump snapshots will ship as `[[backup.snapshot]]` in a later release.
+Immich's and Paperless's Postgres databases are still not covered. You get those files back but lose albums, tags, custom fields and saved views. They use the same snapshot mechanism the chat server already uses, so wiring them up is a small change rather than a new design.
 
 **How the protection works**
 
 Every file written to the archive gets the kernel `uchg` flag. macOS refuses to modify or delete uchg files, even with `sudo`. `rsync --ignore-existing` means files already in the archive are skipped on every run, so the backup is append-only by design and accidental `rm -rf` on your main system cannot propagate.
 
+A second check watches the sources themselves. Before each run the engine compares every source against **the number of files it held on the previous run**, recorded in its own history. A source that is suddenly empty, or that has lost more than half its files, aborts the whole run so you look before anything else happens. There is nothing to configure: the baseline is the source's own past, so it means the same thing whether you have fifty files or five hundred thousand. A source that has never had any data, like the chat media store before anyone sends a photo, is simply skipped and reported.
+
 A **canary file** is a tripwire (named after the canary miners used to take underground to detect bad air). famstack plants a small file with known contents inside `~/famstack-data`; before every sync the engine reads it and refuses to proceed if the contents have changed. If something has been encrypting or modifying files under the data directory, the canary will not match what was planted and the sync aborts before opening the archive, so the corrupted state cannot propagate.
 
+**Database snapshots**
+
+Files that never change are easy: rsync copies them and the kernel locks them. A database is the opposite. Its files change under you continuously, and a copy taken mid-write will not restore. So databases are dumped instead.
+
+Before each sync the backup stacklet runs `pg_dump` against every database that declares itself, and packs the dump into one dated `.tar.gz`. `pg_dump` takes its own consistent view, so nothing stops and nobody gets logged out while it runs. The tarballs then ride to the archive disk as an ordinary append-only source: each run adds one, no run ever touches an older one.
+
+The chat server is the first thing wired up, and it is the one that matters most. The media store holds your family's actual voice recordings; the database is what makes them messages. Without it you would restore a folder of anonymous audio blobs with no sender, no room and no date.
+
+A Synapse snapshot is around 200 KB and contains:
+
+| File | Why |
+|---|---|
+| `synapse.sql` | The whole timeline: rooms, messages, who said what when |
+| `homeserver.yaml` | Config, and the secret that keeps existing logins valid |
+| `*.signing.key` | The server's identity |
+| `MANIFEST.json` | What this file is, what produced it, and how to put it back |
+
+The manifest also records the exact image versions and digests that were running when the snapshot was taken. A dump only restores into something compatible with what wrote it, and the failure is not subtle: a Paperless 3.x database will not boot under 2.x, and there is no downgrade. The digest matters more than the tag, because `latest` names a different image every month and nothing identifiable in five years. Nothing reads this yet. It is recorded now because it is the one part of a snapshot that cannot be added afterwards.
+
+Those config files carry live secrets, including the database password. That is deliberate, because a dump without them restores a server nobody can log into, and it is one more reason the archive disk is a physical object you keep somewhere safe.
+
+The last seven tarballs are kept on the internal disk; the archive disk keeps every one ever made. Pruning locally is safe precisely because the archive never deletes.
+
+**Restoring the chat server**
+
+Recover the snapshot and the media together, and do the database first. Media the database does not know about is harmless; a message whose recording is missing is broken.
+
+```bash
+# 1. Take the snapshot off the archive disk and unpack it
+sudo chflags nouchg /Volumes/<disk>/data/messages-synapse/synapse-<date>.tar.gz
+cp /Volumes/<disk>/data/messages-synapse/synapse-<date>.tar.gz ~/
+tar xzf ~/synapse-<date>.tar.gz -C ~/restore/
+
+# 2. Stop the homeserver so nothing writes while you work
+./stack down messages
+
+# 3. Put the database back
+docker start stack-messages-db
+docker exec stack-messages-db dropdb -U synapse --if-exists synapse
+docker exec stack-messages-db createdb -U synapse synapse
+docker exec -i stack-messages-db psql -U synapse -d synapse < ~/restore/synapse.sql
+
+# 4. Put the config and identity back
+cp ~/restore/homeserver.yaml ~/restore/*.signing.key \
+   ~/famstack-data/messages/synapse/
+
+# 5. Put the recordings back
+sudo chflags -R nouchg /Volumes/<disk>/data/messages-media/
+cp -R /Volumes/<disk>/data/messages-media/* \
+   ~/famstack-data/messages/synapse/media_store/local_content/
+
+./stack up messages
+```
+
+`MANIFEST.json` inside every tarball repeats the essentials, so this works even years from now with no famstack and no documentation to hand.
+
 **Daily operation**
 
 ```bash
@@ -788,7 +848,7 @@ git pull
 ./stack up docs
 ```
 
-The backup stacklet does not cover this. It archives files, not the Postgres database, so a `stack backup sync` alone will not get you back to 2.x.
+The backup stacklet does not cover this yet. It archives Paperless's files but not its Postgres database, so a `stack backup sync` alone will not get you back to 2.x. (Database snapshots exist and are wired up for the chat server; Paperless is next.)
 
 **If Watchtower already moved you to 3.x.** Older famstack releases tracked the `:latest` Paperless tag, and Watchtower's nightly pull rolled some instances from 2.x straight to 3.0 without asking. If that happened to you, your database is already migrated and pinning the image back to 2.20.15 will not start. You need a backup taken before the 3.x boot. Without one, staying on 3.x is the only option, which this release makes the supported path. Paperless is now pinned to an exact tag, so Watchtower can still deliver 3.0.x patches but can no longer jump a major version on its own.
 

docs/stack-reference.md

@@ -375,16 +375,23 @@ job.
 ```toml
 # stacklets/photos/stacklet.toml
 [[backup.archive]]
-name      = "library"
-path      = "{data_dir}/photos/library/library"
-min_files = 10
+name = "library"
+path = "{data_dir}/photos/library/library"
 ```
 
 | Field | Description |
 |---|---|
 | `name` | Short slug for this source. Combined with the stacklet id, this becomes the global source id (`photos/library`). Used in `stack backup status` output and (future) `--source=` selection. |
 | `path` | Filesystem path to sync. Template variables from the rendered environment are available (`{data_dir}`, etc.). |
-| `min_files` | Coarse ransomware smoke test. The engine counts files at `path` before syncing and refuses if the count is below this. The canary file is the precise tripwire; this is the dumb-and-cheap secondary check. Keep low enough that fresh installs don't trip it. |
+
+There is no threshold to declare. The engine judges each source against
+the number of files it held on the previous run, recorded in its own
+history, and refuses to sync one that is suddenly empty or has lost more
+than half its files. This replaced a `min_files` constant, which could
+not work: a number written at authoring time cannot know the scale of the
+household it guards, so a library of 50,000 photos reduced to 11 passed
+`min_files = 10` without complaint. A source that has never held anything
+is skipped rather than failing the run.
 
 **`[[backup.archive]]`** declares an append-only store: files are added,
 never modified, never deleted. The engine commits to kernel-enforced
@@ -393,10 +400,41 @@ genuinely append-only (photo originals, archived PDFs), this is the
 right section. (Storage-industry vocabulary calls this WORM — Write
 Once Read Many.)
 
-**`[[backup.snapshot]]`** is reserved for time-stamped point-in-time
-captures of mutable state (Postgres dumps, Docker volume tarballs). Not
-yet implemented — declare an `archive` section today; a `snapshot`
-section will be added later when DB-restore semantics ship.
+**`[[backup.snapshot]]`** declares mutable state that cannot be rsynced.
+A database changes under you continuously and a copy taken mid-write will
+not restore, so it is dumped instead: one `pg_dump` per run, packed with
+whatever small files must travel beside it, into a dated `.tar.gz`. Each
+run adds a tarball and never touches an older one, which gives mutable
+state the same append-only shape an archive has.
+
+```toml
+# stacklets/messages/stacklet.toml
+[[backup.snapshot]]
+name     = "synapse"
+postgres = { container = "stack-messages-db", database = "synapse", user = "synapse" }
+include  = ["{data_dir}/messages/synapse/homeserver.yaml",
+            "{data_dir}/messages/synapse/*.signing.key"]
+```
+
+| Field | Description |
+|---|---|
+| `name` | Short slug, as for an archive. Becomes the source id (`messages/synapse`). |
+| `postgres` | How to capture the state, namespaced by what captures it. `container` is where the database runs (the dump goes through `docker exec`), `database` and `user` are what to dump and as whom. |
+| `include` | Files that must travel with the dump for a restore to be possible. Globs allowed; a path this install never created is skipped rather than failing the snapshot. |
+
+The capture key is namespaced so a stacklet keeping state somewhere other
+than Postgres can declare a snapshot later without the contract having to
+pretend every database looks like this one. Driving a containerised
+Postgres lives in `stack.postgres`, shared with whatever eventually
+restores one, rather than inside the backup coordinator where only it
+could reach it.
+
+Snapshots run before the sync, so a dump is never newer than the files it
+references. `pg_dump` takes its own consistent view, so nothing stops and
+nobody is logged out while it runs. The tarball records the image
+versions and digests that produced it; nothing reads that yet, but it is
+the one part of a snapshot that cannot be added afterwards, and a restore
+has to be able to refuse an incompatible target.
 
 A stacklet may declare zero, one, or several entries of each kind. Sources
 flow to every configured target whose engine supports the declared

lib/stack/postgres.py

@@ -0,0 +1,96 @@
+"""pg_dump and psql against a Postgres running in a container.
+
+Three stacklets keep their state in Postgres: messages (the Matrix
+timeline), docs (Paperless) and photos (Immich). Each needs the same two
+operations, extracting a consistent dump and loading one back, so the
+invocations live here rather than inside whichever component needed them
+first. The backup coordinator calls `dump` today and a restore path will
+call `restore`.
+
+Command construction is separated from execution so the exact arguments
+can be asserted in tests without a running database.
+"""
+
+from __future__ import annotations
+
+import subprocess
+from typing import List
+
+
+class PostgresError(RuntimeError):
+    """A pg_dump or psql invocation that exited non-zero."""
+
+
+# ── Commands ───────────────────────────────────────────────────────────────
+
+def dump_command(container: str, database: str, user: str) -> List[str]:
+    """Arguments for a plain-SQL dump of `database`.
+
+    Plain SQL rather than pg_dump's custom format, because the output is
+    read back by `psql`, which exists in every Postgres image. A custom
+    dump would additionally require a `pg_restore` of compatible version.
+    """
+    return ["docker", "exec", container,
+            "pg_dump", "-U", user, "-d", database]
+
+
+def restore_command(container: str, database: str, user: str) -> List[str]:
+    """Arguments for loading a plain-SQL dump from stdin.
+
+    `-i` keeps the container's stdin connected. Without it `psql`
+    receives no input and blocks indefinitely.
+    """
+    return ["docker", "exec", "-i", container,
+            "psql", "-U", user, "-d", database]
+
+
+def version_command(container: str, database: str, user: str) -> List[str]:
+    """Arguments for reading the server version.
+
+    `-t` drops the column header and `-A` the alignment padding, leaving
+    the bare value on stdout.
+    """
+    return ["docker", "exec", container,
+            "psql", "-U", user, "-d", database,
+            "-tAc", "show server_version;"]
+
+
+# ── Execution ──────────────────────────────────────────────────────────────
+
+def dump(container: str, database: str, user: str) -> bytes:
+    """Return a consistent SQL dump of `database`.
+
+    `pg_dump` reads from a single MVCC snapshot, so the service keeps
+    running and accepting writes for the duration.
+
+    Raises `PostgresError` carrying the beginning of stderr, which is
+    where Postgres reports an unknown database or a failed authentication.
+    """
+    proc = subprocess.run(
+        dump_command(container, database, user), capture_output=True,
+    )
+    if proc.returncode != 0:
+        detail = proc.stderr.decode(errors="replace").strip()[:400]
+        raise PostgresError(
+            f"pg_dump failed for {database} in {container}: {detail}"
+        )
+    return proc.stdout
+
+
+def server_version(container: str, database: str, user: str) -> str:
+    """Return the server version, or an empty string if it cannot be read.
+
+    This is recorded as metadata beside a dump rather than used for any
+    decision, so an unreachable or stopped container is reported as
+    unknown instead of raising.
+    """
+    try:
+        proc = subprocess.run(
+            version_command(container, database, user),
+            capture_output=True, timeout=30,
+        )
+    except (OSError, subprocess.SubprocessError):
+        return ""
+    if proc.returncode != 0:
+        return ""
+    return proc.stdout.decode(errors="replace").strip()