Etap 1: Time Machine przez SMB do maszyny Lima (obok starego układu)

.github/workflows/release.yml

@@ -47,6 +47,7 @@ on:
       - "mac-app/Sources/**"
       - "mac-app/Resources/**"
       - "launchd/**"
+      - "vm/**"
       - "config/**"
       - "packaging/homebrew/**"
   workflow_dispatch:

.gitignore

@@ -8,6 +8,10 @@ logs/
 # rclone config (contains OAuth tokens) - never commit
 rclone.conf
 *.conf
+# ...except the machine's templates (Samba, FUSE, systemd drop-ins) - no
+# secrets in them; rclone.conf stays ignored there too.
+!vm/**/*.conf
+vm/**/rclone.conf
 
 # Generated launchd files (contain the user's local paths)
 launchd/*.plist

README.md

@@ -7,35 +7,35 @@
 Native Time Machine, backed by Google Drive. No external disk, no NAS, nothing
 plugged in at home.
 
-Time Machine writes to a disk image that macOS mounts like any other volume. The
-image's contents live on Google Drive, behind a local write buffer. Backups land
-in the buffer at SSD speed and drain to the cloud in the background, so a dropped
-connection stalls the upload instead of interrupting the backup.
+Time Machine backs up over SMB to a small Linux machine that runs on this Mac
+([Lima](https://lima-vm.io), Apple Virtualization). The machine keeps the
+backup's small files on its own disk and its bands on Google Drive, behind a
+local upload buffer: backups land in the buffer at disk speed and drain to the
+cloud in the background, so a dropped connection stalls the upload instead of
+interrupting the backup.
 
 ```
-Time Machine
-  -> /Volumes/CloudMachine                  attached sparsebundle; Time Machine sees plain APFS
-     -> ~/.cloudmachine/drive               rclone mount over FUSE-T
-        -> ~/.cloudmachine/cache            100 GB write buffer
-        -> gdrive:CloudMachine/mac-studio   Google Drive
+Time Machine --SMB--> machine cmsmb (Lima)
+                        Samba    /srv/tm-local/<Computer>.sparsebundle   image metadata, local
+                                 bands/ -> punchfs -> rclone mount         120 GB upload buffer
+                                 gdrive:CloudMachine/<folder>-tm           Google Drive
 ```
 
-**No network filesystem sits in the write path.** That is the whole point.
-Time Machine over SMB to a NAS or a cloud share is the common approach and the
-fragile one — a sparsebundle written directly over a network link corrupts when
-the link drops. Here Time Machine talks to a locally attached image and never
-knows the cloud exists.
+Why a machine at all: macOS 27 no longer accepts a disk image as a new Time
+Machine destination (`tmutil setdestination` answers error 45), and refuses SMB
+to the Mac's own address. A share served from a machine on the Mac is what Time
+Machine still accepts. Why each layer inside it is there: [`vm/README.md`](vm/README.md).
 
 ---
 
 ## Requirements
 
-- macOS 14 (Sonoma) or newer, and an administrator password for one step
-  (pointing Time Machine at CloudMachine).
+- macOS 14 (Sonoma) or newer, Apple silicon, and an administrator password for
+  one step (pointing Time Machine at the machine).
 - A Google account with room to spare.
-- [Homebrew](https://brew.sh).
-- Nothing else at runtime. CloudMachine installs its own `rclone` and its own
-  copy of FUSE-T.
+- [Homebrew](https://brew.sh) and Lima from it (`brew install lima`).
+- About 200 GB of free disk: the machine's disk file grows with the upload
+  buffer.
 
 The menu-bar app, CLI output and notifications follow the system language:
 Polish when Polish is the first preferred language, English otherwise.
@@ -47,6 +47,9 @@ so they read the same whoever sends them to you.
 - **Not notarised.** Releases are signed with a self-signed certificate, not
   with an Apple Developer ID. The Homebrew cask clears the quarantine flag; a
   DMG downloaded by hand gets Gatekeeper's "unidentified developer" warning.
+- **Google's daily upload limit and a full Drive are not named in the app.**
+  rclone's log that says so lives in the machine; what the app does show is
+  the backlog growing and the guard pausing Time Machine.
 
 ---
 
@@ -59,33 +62,43 @@ open -a CloudMachine
 
 CloudMachine lives in the menu bar. Its window opens on a **Required Setup
 Steps** card listing what is left to do on this Mac, in order, each with a
-button — or, for the two steps an app cannot do, a command to copy:
+button — or, for the steps an app cannot do, a command to copy:
 
-1. **Install rclone** and **Install FUSE-T.** CloudMachine downloads its own
-   copies; nothing else is installed system-wide.
+1. **Install rclone.** CloudMachine downloads its own copy; it signs in to
+   Google Drive and writes the configuration the machine gets a copy of.
 2. **Connect Google Drive** — a command to run in Terminal. It opens Google's
    sign-in in the browser and waits for your approval. Enter your own OAuth
    credentials first (see [below](#your-own-google-oauth-credentials)).
-3. **Grant Full Disk Access** — opens the right pane of System Settings.
+3. **Install Lima** — `brew install lima`, to copy.
+4. **Set up the machine** — creates and starts `cmsmb` (the first start
+   downloads Ubuntu and installs packages, several minutes), copies rclone's
+   configuration into it and sets its SMB password.
+5. **Grant Full Disk Access** — opens the right pane of System Settings.
    Without it CloudMachine cannot read when Time Machine last *finished* a
    backup, which is the one check that matters.
-4. **Install agents** — the launchd agents that keep the Drive mounted, attach
-   the image and watch the backup.
-5. **Create image**, then **Attach image** — the backup image on Google Drive.
-   The size is a ceiling, not an allocation: the image is sparse, and Drive
-   only holds what has been written.
-6. **Point Time Machine at CloudMachine** — a `sudo tmutil setdestination`
-   command to copy, because only an administrator can change it.
+6. **Install agents** — the launchd agents that start the machine at login,
+   pause Time Machine while the upload buffer is full, and watch the backup.
+7. **Point Time Machine at the machine** — a `sudo tmutil setdestination
+   smb://tm:…@<address>/CloudMachine` command. The card shows the password
+   masked; **Copy** puts the real command on the clipboard.
+8. **Convert the image** — once Time Machine has created its first, empty
+   image, stop that backup (`tmutil stopbackup`) and press the button
+   (`cloudmachine-agent vm-prepare-bundle`): the image gets 32 MB bands, and
+   the bands move to Google Drive.
 
 When the card disappears, setup is done. Turn on automatic backups in System
 Settings → General → Time Machine, or run `sudo tmutil enable`.
 
+The same from Terminal: `install-rclone`, `configure-remote`, `vm-setup
+--install-agents` (prints the `tmutil` command), then `vm-prepare-bundle`;
+`vm-status` shows everything at once.
+
 ### Several Macs, one Google account
 
 Install CloudMachine on each Mac and go through the same steps; they can all
 use the same Google account and the same OAuth credentials. Each Mac backs up
-into its own folder, `gdrive:CloudMachine/<folder>`, holding
-`<folder>.sparsebundle`, so their backups never mix.
+into its own folder, `gdrive:CloudMachine/<folder>-tm`, so their backups never
+mix.
 
 The folder name is chosen once, in the **Connect Google Drive** step: the card
 has a **Folder on Google Drive** field, filled in from the computer name, and
@@ -103,33 +116,38 @@ It works two ways:
 
 - **Time Machine quota.** The card gives a `sudo tmutil setquota` command to
   copy; Time Machine then deletes this Mac's oldest backups to stay under it.
-  The quota is 70% of the limit, because the image on Drive grows larger than
-  the backup inside it (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
+  The quota is 70% of the limit, because the bands on Drive grow larger than
+  the backup inside them (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
 - **An alarm on the real usage.** The watchdog measures this Mac's folder on
   Drive every few hours and warns from 90% of the limit, and again above it.
-  The card and `drive-status` show the usage.
+  The card and `vm-status` show the usage.
 
 ### Upgrading and uninstalling
 
-`brew upgrade --cask renacode/tap/cloudmachine` replaces the app without
-restarting the Google Drive mount. The cask in the tap can be a version
-behind the latest GitHub Release: it is updated by a pull request after each
-release (see `packaging/README.md`).
-When Homebrew reopens the app, it reloads the background agents so they run
-the new version; `cloudmachine-agent drive-status` shows `Agents: OK`. Changes
-to the mount itself wait for its next start. One such change is rclone's
-control interface moving to a private socket: until then the old mount takes
-commands on `127.0.0.1:5572` without a password, and the app window shows a red
-card, the menu bar an alert, and the watchdog a notification (once, then after
-1, 2, 4 and every 7 days). **Restart the mount** on that card - or
-`cloudmachine-agent restart-mount` - detaches the image, waits until everything
-has reached Google Drive, restarts the mount and attaches the image again; it
-refuses while Time Machine is backing up. Restarting the Mac after
-`prepare-shutdown` does the same. If a new version does not show up, run
-`brew update` first - Homebrew refreshes the tap only now and then. Neither `brew uninstall` nor
-`--zap` touches the launchd agents or the upload buffer in `~/.cloudmachine`,
-which may hold backups that have not reached Google Drive yet. Run
-`cloudmachine-agent prepare-shutdown` before uninstalling.
+`brew upgrade --cask renacode/tap/cloudmachine` replaces the app; the machine
+keeps running. The cask in the tap can be a version behind the latest GitHub
+Release: it is updated by a pull request after each release (see
+`packaging/README.md`). If a new version does not show up, run `brew update`
+first - Homebrew refreshes the tap only now and then. When Homebrew reopens
+the app, it reloads the background agents so they run the new version;
+`cloudmachine-agent vm-status` shows `Agents: OK`.
+
+**Coming from the disk-image layout** (an rclone mount on the Mac and an image
+in `/Volumes/CloudMachine`): the app removes its launchd agents
+(`gdrive-buffer`, `gdrive-attach`, `buffer-guard`) on its first launch and
+lists in `~/Library/Logs/CloudMachine/cloudmachine.log` what that layout left
+behind - `~/.cloudmachine/cache`, `drive`, `fuse`, the old image in
+`gdrive:CloudMachine/<folder>` - without deleting any of it. The new layout
+starts a new backup in `<folder>-tm`; delete the old one when you no longer
+need to restore from it.
+
+Before restarting the Mac, **Prepare for restart** in the menu (or
+`cloudmachine-agent prepare-shutdown`) stops the backup, gives the upload up
+to 30 minutes, and shuts the machine down cleanly; it starts again at the next
+login. Neither `brew uninstall` nor `--zap` touches the launchd agents or the
+machine, whose buffer may hold backups that have not reached Google Drive yet:
+run `prepare-shutdown`, then `limactl delete cmsmb` and remove
+`~/Library/LaunchAgents/com.renacode.cloudmachine.*.plist`.
 
 ### Your own Google OAuth credentials
 
@@ -147,13 +165,13 @@ compete with every other rclone user.
 The connection uses scope `drive.file`, which grants access only to files this
 application itself created. Full `drive` scope would hand out read, write and
 **delete** over the entire Google account, which is far more than a folder of
-disk-image bands needs — especially with `--drive-use-trash=false`, where a
+backup bands needs — especially with `--drive-use-trash=false`, where a
 delete has no bin to recover from.
 
 On a Mac that is already connected, keep the connection it has. A new sign-in
 with different credentials sees only what it created itself, so the existing
 backup would vanish from view; `configure-remote` refuses to replace a working
-connection unless told twice, and checks the new token against the image first
+connection unless told twice, and checks the new token against the backup first
 (see `renacode-infra: docs/projekty/cloudmachine/setup-cli.md`, "Replacing the connection").
 
 ---
@@ -165,15 +183,16 @@ repository (`renacode-infra: docs/projekty/cloudmachine/`):
 
 - Running CloudMachine (`operations.md`) — checking from Terminal,
   knowing when backups stop, reading the logs, before rebooting.
-- Restoring (`restoring.md`) — getting files back, and checking the image.
+- Restoring (`restoring.md`) — getting files back.
 - How it works, and what it costs (`design.md`) — measured traffic and
   storage, and why each piece is the way it is.
 - Setting up from Terminal (`setup-cli.md`) — the same setup without the app.
-- Building from source (`building.md`) — local builds, tests, releases,
-  measurement harnesses.
+- Building from source (`building.md`) — local builds, tests, releases.
 
 ---
 
 ## Licence
 
-MIT. FUSE-T and rclone keep their own licences.
+MIT. rclone, Lima and Samba keep their own licences. One file is GPLv2:
+`vm/punchfs/punchfs.c`, based on libfuse's passthrough example (see
+`vm/punchfs/README.md`).

launchd/com.renacode.cloudmachine.app.plist.template

@@ -21,8 +21,8 @@
     would launch it over and over. The user closing the window should stay
     closed, not be an invitation to restart.
 
-    The backups themselves do NOT depend on this app - the gdrive-buffer,
-    gdrive-attach and buffer-guard agents do them. This is only a viewer.
+    The backups themselves do NOT depend on this app - the machine (started
+    by vm-start) and the vm-guard agent do them. This is only a viewer.
   -->
   <key>StandardOutPath</key>
   <string>__CM_LOG_DIR__/launchd-app.out.log</string>

launchd/com.renacode.cloudmachine.backup-health.plist.template

@@ -23,8 +23,9 @@
     Every 30 minutes, with an hourly cycle and a 3 h threshold - often enough
     that a broken cycle does not sit unnoticed for a day, and rarely enough to
     cost nothing. The watchdog reads the local Time Machine preferences file,
-    NOT the Google Drive mount: a supervisor that hangs together with what it
-    supervises goes silent exactly when it should speak.
+    not the backup destination, and every question to the machine has a time
+    limit: a supervisor that hangs together with what it supervises goes
+    silent exactly when it should speak.
   -->
   <key>StartInterval</key>
   <integer>1800</integer>

launchd/com.renacode.cloudmachine.vm-guard.plist.template

@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>Label</key>
+  <string>com.renacode.cloudmachine.vm-guard</string>
+  <!--
+    The machine's buffer guard (`vm-guard`): one check per run, so
+    StartInterval rather than KeepAlive. Installed by install-launchd
+    (also run by `vm-setup --install-agents`).
+  -->
+  <key>ProgramArguments</key>
+  <array>
+    <string>__CM_AGENT_BIN__</string>
+    <string>vm-guard</string>
+  </array>
+  <key>StartInterval</key>
+  <integer>60</integer>
+  <key>RunAtLoad</key>
+  <true/>
+  <key>EnvironmentVariables</key>
+  <dict>
+    <key>PATH</key>
+    <string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
+  </dict>
+  <key>StandardOutPath</key>
+  <string>__CM_LOG_DIR__/launchd-vm-guard.out.log</string>
+  <key>StandardErrorPath</key>
+  <string>__CM_LOG_DIR__/launchd-vm-guard.err.log</string>
+</dict>
+</plist>