feat(backup): one vault directory per stacklet

docs/admin-guide.md

@@ -693,13 +693,17 @@ Run `stack up backup`. You need an APFS-formatted external drive plugged in. The
 
 | Source | Path on the archive disk |
 |---|---|
-| Immich photo originals | `/Volumes/<disk>/data/photos-library/` |
-| Paperless archived PDFs | `/Volumes/<disk>/data/docs-media/` |
-| Matrix uploads: voice messages, photos, files | `/Volumes/<disk>/data/messages-media/` |
-| Matrix timeline, as dated snapshots | `/Volumes/<disk>/data/messages-synapse/` |
+| Immich photo originals | `/Volumes/<disk>/data/photos/library/` |
+| Paperless archived PDFs | `/Volumes/<disk>/data/docs/media/` |
+| Matrix uploads: voice messages, photos, files | `/Volumes/<disk>/data/messages/media/` |
+| Matrix timeline, as dated snapshots | `/Volumes/<disk>/data/messages/synapse/` |
 
 Immich's and Paperless's Postgres databases are still not covered. You get those files back but lose albums, tags, custom fields and saved views. They use the same snapshot mechanism the chat server already uses, so wiring them up is a small change rather than a new design.
 
+**Disks from an earlier release**
+
+Backups written by 0.3.0-beta.3 and earlier used one flat directory per source, `data/photos-library/` where this release writes `data/photos/library/`. Both layouts are read, and a sync keeps using whichever directory it finds, so leaving it alone is a valid choice. To move to the current layout, connect the disk and run `stack backup migrate`. It renames the directories in place, so nothing is copied however much is on the disk, and the files keep the immutable flag that makes the archive append-only. Add `--dry-run` to see what it would do first.
+
 **How the protection works**
 
 Every file written to the archive gets the kernel `uchg` flag. macOS refuses to modify or delete uchg files, even with `sudo`. `rsync --ignore-existing` means files already in the archive are skipped on every run, so the backup is append-only by design and accidental `rm -rf` on your main system cannot propagate.
@@ -737,8 +741,8 @@ Recover the snapshot and the media together, and do the database first. Media th
 
 ```bash
 # 1. Take the snapshot off the archive disk and unpack it
-sudo chflags nouchg /Volumes/<disk>/data/messages-synapse/synapse-<date>.tar.gz
-cp /Volumes/<disk>/data/messages-synapse/synapse-<date>.tar.gz ~/
+sudo chflags nouchg /Volumes/<disk>/data/messages/synapse/synapse-<date>.tar.gz
+cp /Volumes/<disk>/data/messages/synapse/synapse-<date>.tar.gz ~/
 tar xzf ~/synapse-<date>.tar.gz -C ~/restore/
 
 # 2. Stop the homeserver so nothing writes while you work
@@ -755,8 +759,8 @@ cp ~/restore/homeserver.yaml ~/restore/*.signing.key \
    ~/famstack-data/messages/synapse/
 
 # 5. Put the recordings back
-sudo chflags -R nouchg /Volumes/<disk>/data/messages-media/
-cp -R /Volumes/<disk>/data/messages-media/* \
+sudo chflags -R nouchg /Volumes/<disk>/data/messages/media/
+cp -R /Volumes/<disk>/data/messages/media/* \
    ~/famstack-data/messages/synapse/media_store/local_content/
 
 ./stack up messages
@@ -780,8 +784,8 @@ The scheduled nightly run leaves the disk mounted between runs (cron cannot trig
 Plug the archive disk into any Mac and browse the files in Finder. To copy locked files out:
 
 ```bash
-sudo chflags -R nouchg /Volumes/<disk>/data/photos-library/<folder>/
-cp -R /Volumes/<disk>/data/photos-library/<folder>/ ~/recovered/
+sudo chflags -R nouchg /Volumes/<disk>/data/photos/library/<folder>/
+cp -R /Volumes/<disk>/data/photos/library/<folder>/ ~/recovered/
 ```
 
 A `stack backup restore` command and `on_restore` hooks for database recovery are planned but not yet shipped.

docs/stack-reference.md

@@ -381,7 +381,7 @@ path = "{data_dir}/photos/library/library"
 
 | Field | Description |
 |---|---|
-| `name` | Short slug for this source. Combined with the stacklet id, this becomes the global source id (`photos/library`). Used in `stack backup status` output and (future) `--source=` selection. |
+| `name` | Short slug for this source. Combined with the stacklet id, this becomes the global source id (`photos/library`), which is also the directory it occupies on a vault (`data/photos/library/`). Used in `stack backup status` output and (future) `--source=` selection. |
 | `path` | Filesystem path to sync. Template variables from the rendered environment are available (`{data_dir}`, etc.). |
 
 There is no threshold to declare. The engine judges each source against

stacklets/backup/README.md

@@ -89,13 +89,30 @@ engine supports append-only semantics. Adding a second target later
 ```
 stack backup sync     [--dry-run] [--no-eject] [--verbose]
 stack backup status   # last run, source counts, cron presence
+stack backup migrate  [--dry-run]  # move a vault to the current layout
 ```
 
 Per-stacklet aliases (`stack photos backup`, `stack docs backup`) and
 restore (`stack backup restore --source=…`) are intentionally not in
 v1 — they'll layer on once the engine port lands and the manifest
 contract has been exercised on at least one production sync.
 
+## Vault layout
+
+Each source owns one directory under `data/`, named after its id:
+`photos/library` lands in `data/photos/library/`. One directory per
+stacklet keeps everything that has to be restored together in one
+place, and it stays unambiguous when a stacklet id or a source name
+contains a hyphen.
+
+Vaults written by 0.3.0-beta.3 and earlier hold the flat form,
+`data/photos-library/`. The engine reads both and keeps writing into
+whichever directory it finds, because adopting the new path would copy
+every file a second time and the old tree could not be removed
+afterwards: its files are locked immutable. `stack backup migrate`
+renames the directories in place, which moves no data and preserves the
+locks.
+
 ## Guarding the sources
 
 Two checks run before anything is written.

stacklets/backup/cli/_orchestrator.py

@@ -66,6 +66,29 @@ class Target:
     schedule: str    # Cron expression; informational at this level
 
 
+# ── Vault layout ───────────────────────────────────────────────────────────
+
+def vault_subdir(source_id: str) -> str:
+    """Where a source's files live on the vault, relative to the mount.
+
+    The source id is already ``{stacklet_id}/{name}``, so the vault
+    mirrors it: ``data/messages/synapse``. One directory per stacklet
+    groups everything that has to be restored together.
+    """
+    return f"data/{source_id}"
+
+
+def legacy_vault_subdir(source_id: str) -> str:
+    """The flat form used by vaults written with 0.3.0-beta.3 or earlier.
+
+    Both halves of a source id may contain hyphens, so ``data/{id with
+    the slash replaced}`` cannot be split back into stacklet and name.
+    It is still read, because those directories exist on disks in the
+    field; ``stack backup migrate`` renames them.
+    """
+    return f"data/{source_id.replace('/', '-')}"
+
+
 # ── Source discovery ───────────────────────────────────────────────────────
 
 def discover_archive_sources(
@@ -81,10 +104,8 @@ def discover_archive_sources(
     (currently just ``{data_dir}``) are rendered into the path field.
 
     The source ``id`` is ``{stacklet_id}/{archive.name}`` so a single
-    stacklet can declare multiple archives without collision. The
-    vault subdirectory is derived as ``data/{stacklet_id}-{name}`` —
-    short, stable, and namespaced so future stacklets can't accidentally
-    clobber existing archive directories.
+    stacklet can declare multiple archives without collision, and the
+    vault subdirectory mirrors it (see :func:`vault_subdir`).
     """
     stacklets_dir = repo_root / "stacklets"
     if not stacklets_dir.is_dir():
@@ -119,11 +140,12 @@ def discover_archive_sources(
                 # will surface the problem with a useful error.
                 rendered_path = raw_path
 
+            source_id = f"{stacklet_id}/{name}"
             sources.append(SourceRecord(
-                id=f"{stacklet_id}/{name}",
+                id=source_id,
                 display=stacklet_display,
                 src_path=Path(rendered_path),
-                vault_subdir=f"data/{stacklet_id}-{name}",
+                vault_subdir=vault_subdir(source_id),
             ))
 
     return sources

stacklets/backup/cli/_snapshot.py

@@ -38,7 +38,7 @@
 
 from stack import postgres
 
-from _orchestrator import SourceRecord
+from _orchestrator import SourceRecord, vault_subdir
 
 
 # Tarballs retained on the internal disk. The vault retains all of them,
@@ -50,7 +50,10 @@
 class SnapshotSpec:
     """One ``[[backup.snapshot]]`` entry, after template rendering."""
 
-    id: str                     # "{stacklet_id}/{name}", e.g. "messages/synapse"
+    # "{stacklet_id}/{name}", e.g. "messages/synapse". Doubles as the
+    # directory holding this snapshot's tarballs, both in the staging
+    # area and under the vault's `data/`.
+    id: str
     display: str                # Human-readable, e.g. "Messages"
     name: str                   # "synapse"
     # Capture parameters, keyed by the mechanism that reads them. The
@@ -75,13 +78,6 @@ def user(self) -> str:
     def stacklet_id(self) -> str:
         return self.id.split("/", 1)[0]
 
-    @property
-    def subdir(self) -> str:
-        """Directory holding this snapshot's tarballs, used both on the
-        internal disk and under the vault's `data/`. Qualified by stacklet
-        so two stacklets choosing the same `name` do not collide."""
-        return f"{self.stacklet_id}-{self.name}"
-
 
 # ── Discovery ──────────────────────────────────────────────────────────────
 
@@ -229,7 +225,7 @@ def take_snapshot(
     # Taken first, so a failure propagates before any file exists.
     sql = dump(spec)
 
-    out_dir = out_root / spec.subdir
+    out_dir = out_root / spec.id
     out_dir.mkdir(parents=True, exist_ok=True)
     # The timestamp has second resolution, so two runs within the same
     # second would otherwise produce the same name. A suffix keeps them
@@ -334,8 +330,8 @@ def snapshot_source(spec: SnapshotSpec, out_root: Path) -> SourceRecord:
     return SourceRecord(
         id=spec.id,
         display=spec.display,
-        src_path=out_root / spec.subdir,
-        vault_subdir=f"data/{spec.subdir}",
+        src_path=out_root / spec.id,
+        vault_subdir=vault_subdir(spec.id),
         rolling=True,
     )
 

stacklets/backup/cli/migrate.py

@@ -0,0 +1,182 @@
+"""stack backup migrate — move a vault to the current directory layout.
+
+Vaults written with 0.3.0-beta.3 or earlier hold one flat directory per
+source: ``data/messages-synapse``. Current releases nest them under the
+stacklet, ``data/messages/synapse``, so everything belonging to one
+stacklet restores from one place and a hyphen inside a stacklet id or a
+source name stays unambiguous.
+
+Both layouts are read. A sync keeps using whichever flat directory it
+finds, so an un-migrated vault goes on working and no file is ever
+copied twice. This command performs the one-time rename.
+
+What moves is the directory, not its contents: the files inside keep the
+immutable flag that makes the vault append-only, and nothing is copied.
+The disk must already be mounted, because unlocking and mounting it is
+``stack backup sync``'s job.
+
+Usage:
+  stack backup migrate            rename legacy directories on every target
+  stack backup migrate --dry-run  list what would be renamed
+"""
+
+HELP = "Move a vault's data directories to the current layout"
+
+import argparse
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Tuple
+
+_here = Path(__file__).parent
+sys.path.insert(0, str(_here))
+from _orchestrator import (  # noqa: E402
+    discover_archive_sources,
+    get_targets,
+    legacy_vault_subdir,
+    vault_subdir,
+)
+from _snapshot import discover_snapshots  # noqa: E402
+
+
+@dataclass
+class Move:
+    """One source's directory on one vault, and what became of it."""
+
+    display: str
+    old: Path
+    new: Path
+    status: str        # "moved" | "occupied" | "failed"
+    reason: str = ""
+
+
+# ── Planning ───────────────────────────────────────────────────────────────
+
+def declared_sources(
+    repo_root: Path, instance_dir: Path, data_dir: Path,
+) -> List[Tuple[str, str]]:
+    """Every source that could own a directory on a vault, as (id, display).
+
+    Archives and snapshots both land in ``data/`` under their source id,
+    so both are candidates. A stacklet that has since been disabled is
+    not discovered and keeps its flat directory, which stays readable.
+    """
+    found = {
+        s.id: s.display
+        for s in discover_archive_sources(repo_root, instance_dir, data_dir)
+    }
+    for spec in discover_snapshots(repo_root, instance_dir, data_dir):
+        found.setdefault(spec.id, spec.display)
+    return sorted(found.items())
+
+
+def migrate_vault(
+    mount_point: Path, sources: List[Tuple[str, str]], *, dry_run: bool,
+) -> List[Move]:
+    """Rename each legacy directory found on this vault, and report.
+
+    A source whose new directory already exists is left alone. That
+    means both layouts hold data, which no rename can reconcile: the
+    files are immutable, so they cannot be merged into one tree without
+    unlocking them, and choosing one would hide the other.
+    """
+    moves: List[Move] = []
+
+    for source_id, display in sources:
+        old = mount_point / legacy_vault_subdir(source_id)
+        new = mount_point / vault_subdir(source_id)
+        if not old.is_dir():
+            continue
+
+        if new.exists():
+            moves.append(Move(display, old, new, "occupied"))
+            continue
+
+        if not dry_run:
+            try:
+                new.parent.mkdir(parents=True, exist_ok=True)
+                old.rename(new)
+            except OSError as e:
+                moves.append(Move(display, old, new, "failed", str(e)))
+                continue
+
+        moves.append(Move(display, old, new, "moved"))
+
+    return moves
+
+
+# ── Output ─────────────────────────────────────────────────────────────────
+
+def _render(target_name: str, moves: List[Move], dry_run: bool) -> None:
+    from stack.prompt import bold, dim, done, error, nl, warn
+
+    nl()
+    bold(f"Target '{target_name}'")
+    if not moves:
+        dim("Already on the current layout.")
+        return
+
+    for m in moves:
+        nested = f"{m.new.parent.name}/{m.new.name}"
+        if m.status == "moved":
+            verb = "would move" if dry_run else "moved"
+            done(f"{m.display}: {verb} {m.old.name} to {nested}")
+        elif m.status == "occupied":
+            warn(f"{m.display}: both {m.old.name} and {nested} hold data. "
+                 "Merge them by hand, then re-run.")
+        else:
+            error(f"{m.display}: could not move {m.old.name} ({m.reason})")
+
+
+# ── Entry point ────────────────────────────────────────────────────────────
+
+def _parse_args(argv: list) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(prog="stack backup migrate",
+                                     description=HELP)
+    parser.add_argument("--dry-run", action="store_true",
+                        help="List what would be renamed (no changes).")
+    return parser.parse_args(argv)
+
+
+def run(args, stacklet, config):
+    """Entry point invoked by the framework via ``stack backup migrate``."""
+    parsed = _parse_args(args or [])
+
+    repo_root = Path(config.get("repo_root", "."))
+    instance_dir = Path(config.get("instance_dir", repo_root))
+    data_dir = Path(config.get("data_dir", "."))
+
+    targets = get_targets(config.get("stack", {}))
+    if not targets:
+        return {
+            "error": "No backup targets configured. Add a [backup.targets.<name>] "
+                     "block to stack.toml (see stack.example.toml)."
+        }
+
+    sources = declared_sources(repo_root, instance_dir, data_dir)
+    unreachable: List[str] = []
+    problems = 0
+    migrated = 0
+
+    for target in targets:
+        mount_point = Path("/Volumes") / target.disk
+        if not mount_point.is_dir():
+            unreachable.append(target.name)
+            continue
+
+        moves = migrate_vault(mount_point, sources, dry_run=parsed.dry_run)
+        _render(target.name, moves, parsed.dry_run)
+        migrated += sum(1 for m in moves if m.status == "moved")
+        problems += sum(1 for m in moves if m.status != "moved")
+
+    if unreachable and len(unreachable) == len(targets):
+        return {
+            "error": "No backup disk is mounted. Connect it and run "
+                     "'stack backup sync' once, which unlocks and mounts it."
+        }
+    for name in unreachable:
+        print(f"  [{name}] disk not mounted, skipped", file=sys.stderr)
+
+    if problems:
+        return {"error": "Some directories could not be moved; see above"}
+    return {"ok": True, "moved": migrated, "dry_run": parsed.dry_run}

stacklets/backup/cli/sync.py

@@ -92,7 +92,7 @@ def _take_snapshots(
             continue
         size_kb = max(1, path.stat().st_size // 1024)
         print(f"    {spec.display}: {path.name} ({size_kb} KB)")
-        prune_snapshots(out_root / spec.subdir)
+        prune_snapshots(out_root / spec.id)
         sources.append(snapshot_source(spec, out_root))
 
     return sources, failed