feat: retention backup - archive-before-delete + automatic history cleanup

docs/RETENTION_BACKUP_PLAN.md

@@ -0,0 +1,62 @@
+# Retention backup: archive-before-delete + automatic history cleanup
+
+Status: Phase 1 and Phase 2 code-complete on branch `feat/retention-backup` (pending on-device verification + PR).
+
+## Problem
+
+Continuous screen capture writes full-resolution PNGs into `captures/` with no retention:
+~170MB/day (~5GB/month) measured on a live profile (1,379 files / 838MB over ~5 days). Nothing
+prunes automatically - the only lever is the manual "delete older than N days" button in
+Settings > Data & Privacy, and that delete is permanent. There is no way to keep old history
+before clearing it: the existing backup engine (`src/main/backup/`) exports projects +
+conversations only, never the capture/meeting files that retention deletes.
+
+## Phase 1 - "Back up & delete" (manual, this branch)
+
+A fail-closed archive step in front of the existing category delete.
+
+User-visible behavior:
+
+1. Settings > Data & Privacy grows a "Back up & delete" action next to the existing delete and
+   retention buttons for the file-centric categories (captures, meetings, generated images).
+2. Clicking it opens the normal save dialog. The user picks any destination (external SSD, NAS).
+3. The app stages one ZIP - e.g. `offgrid-captures-before-2026-07-26.zip` - containing every file
+   the delete would remove, plus a `manifest.json` (category, cutoff, created-at, file count,
+   total bytes).
+4. Only after the ZIP is confirmed delivered does the real delete run. Cancel or any archive
+   failure = nothing is deleted, ever.
+
+Design:
+
+- **One source of truth for "what a category deletes".** The per-category userData dir list moves
+  out of `clearCategory`'s switch into a pure module (`src/main/data-categories.ts`) that both the
+  delete path and the archive path read. Two lists would drift into "backed up X, deleted Y".
+- **Collector** - `collectCategoryFiles(dirs, olderThanDays?)` returns exactly the files
+  `clearDirs`/`clearDirsOlderThan` would remove (same mtime cutoff).
+- **Stager** - streams the files into a ZIP via JSZip with STORE compression (PNGs do not
+  compress; the corpus is ~1GB, so never buffer it in memory) + writes `manifest.json`.
+- **Orchestrator** - `archiveThenClear(category, olderThanDays)`: collect -> stage -> deliver via
+  the existing `DesktopBackupSink` (save dialog) -> on confirmed delivery only, call the existing
+  `clearCategory`. Zero files to archive skips the dialog and clears directly. The destination is
+  injectable (sink today, fixed folder later) so Phase 2 reuses the same seam.
+- **Untouched contracts** - `clearCategory` itself does not change; pro's
+  `clearRemovedCaptureProjections` keys off missing files and we copy before deleting, so the pro
+  side needs zero changes.
+- **Tests in the same pass** - the dir-map DRY guard, age-cutoff selection, ZIP + manifest
+  contents, and the ordering contract (canceled/failed archive leaves every file in place), run
+  against real temp dirs with a fake sink.
+
+## Phase 2 - automatic history cleanup (next)
+
+One setting plus a nightly job, built on the Phase 1 seam:
+
+- Settings: "Keep screen history for 30 / 60 / 90 days / forever" + optional archive folder.
+- A scheduled daily job runs the same archive-then-delete machinery with a fixed-folder
+  destination instead of a dialog: old frames archived (if a folder is set), then pruned. Disk
+  usage stays flat at roughly one retention window.
+- Ships OFF by default; deleting history silently is an explicit opt-in.
+- This matches the field standard: Microsoft Recall caps storage and deletes oldest-first;
+  Rewind asks once how long to keep history.
+
+Out of scope for both phases (tracked separately in the bloat notes): compressing captures at
+write time (WebP/JPEG instead of PNG) and encoding frames into HEVC segments for Replay.

e2e/retention-backup.spec.ts

@@ -0,0 +1,94 @@
+/**
+ * Archive-before-delete surface (Settings > Data & privacy): the "Back up first"
+ * toggle exists for exactly the file-centric categories, arms visibly, and the
+ * summary reflects seeded capture files. Fresh temp profile; UI-state clicks only -
+ * the actual archive flow opens a native save dialog, which is covered by the
+ * unit/integration tests (retention-archive.test.ts), not driven here.
+ */
+import { test, expect, type ElectronApplication, type Page } from '@playwright/test'
+import { launchOffGrid } from './helpers/launch'
+import os from 'os'
+import path from 'path'
+import fs from 'fs'
+import { openSettingsSection } from './helpers/settings'
+import { completeOnboarding } from './helpers/onboarding'
+
+let app: ElectronApplication
+let page: Page
+let userDataDir: string
+
+test.beforeAll(async () => {
+  userDataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'offgrid-retention-'))
+  // Seed a few old capture files so the captures row has data and enabled buttons.
+  const captures = path.join(userDataDir, 'captures')
+  fs.mkdirSync(captures, { recursive: true })
+  const old = new Date(Date.now() - 10 * 86_400_000)
+  for (const name of ['capture-1.png', 'capture-2.png', 'capture-3.png']) {
+    const p = path.join(captures, name)
+    fs.writeFileSync(p, 'fake-png-bytes')
+    fs.utimesSync(p, old, old)
+  }
+  app = await launchOffGrid({
+    env: {
+      ...process.env,
+      OFFGRID_USER_DATA: userDataDir,
+      OFFGRID_PRO: '0',
+      NODE_ENV: 'production'
+    }
+  })
+  page = await app.firstWindow()
+  await page.emulateMedia({ reducedMotion: 'reduce' })
+  await page.waitForLoadState('domcontentloaded')
+  await completeOnboarding(page)
+})
+
+test.afterAll(async () => {
+  await app?.close()
+  fs.rmSync(userDataDir, { recursive: true, force: true })
+})
+
+test('Back up first is offered for file categories and arms visibly', async () => {
+  await page.getByRole('button', { name: 'Settings', exact: true }).first().click()
+  await openSettingsSection(page, 'Data & privacy')
+  await expect(page.getByText('Your data on this device')).toBeVisible()
+
+  // Exactly the archivable categories offer the toggle; chats does not.
+  await expect(
+    page.getByRole('button', { name: 'Back up Screen captures before deleting' })
+  ).toBeVisible()
+  await expect(
+    page.getByRole('button', { name: 'Back up Meetings before deleting' })
+  ).toBeVisible()
+  await expect(
+    page.getByRole('button', { name: 'Back up Generated images & artifacts before deleting' })
+  ).toBeVisible()
+  await expect(page.getByRole('button', { name: /Back up Chats/ })).toHaveCount(0)
+
+  // Seeded captures show up in the summary (3 files).
+  await expect(page.getByText(/3 items/)).toBeVisible()
+
+  // Arm the toggle for captures - pressed state flips (pure UI state, no delete).
+  const toggle = page.getByRole('button', { name: 'Back up Screen captures before deleting' })
+  await expect(toggle).toHaveAttribute('aria-pressed', 'false')
+  await toggle.click()
+  await expect(toggle).toHaveAttribute('aria-pressed', 'true')
+
+  await page.screenshot({ path: 'e2e/screenshots/retention-backup-panel.png' })
+})
+
+test('Automatic cleanup arms from Off and reveals folder + Run now', async () => {
+  await expect(page.getByText('Automatic cleanup')).toBeVisible()
+  const off = page.getByRole('button', { name: 'Off', exact: true })
+  await expect(off).toHaveAttribute('aria-pressed', 'true')
+  await expect(page.getByRole('button', { name: /run now/i })).toHaveCount(0)
+
+  await page.getByRole('button', { name: '30 days', exact: true }).click()
+  await expect(page.getByRole('button', { name: '30 days', exact: true })).toHaveAttribute(
+    'aria-pressed',
+    'true'
+  )
+  await expect(page.getByText(/no backup - choose a folder/i)).toBeVisible()
+  await expect(page.getByRole('button', { name: /run now/i })).toBeVisible()
+
+  await page.screenshot({ path: 'e2e/screenshots/retention-auto-cleanup.png' })
+})

src/main/__tests__/data-categories.test.ts

@@ -0,0 +1,30 @@
+import { describe, expect, it } from 'vitest'
+import { ARCHIVABLE_CATEGORIES, CATEGORY_DIRS, type DataCategoryId } from '../data-categories'
+
+describe('the data-category dir map (SSOT for delete + archive)', () => {
+  it('gives every category at least one userData-relative dir name', () => {
+    for (const [id, dirs] of Object.entries(CATEGORY_DIRS)) {
+      expect(dirs.length, `category ${id} has no dirs`).toBeGreaterThan(0)
+      for (const dir of dirs) {
+        // Relative names only - callers resolve against userData. A path separator or
+        // traversal here would silently point delete/archive somewhere else.
+        expect(dir).not.toMatch(/[/\\]|\.\./)
+        expect(dir.trim().length).toBeGreaterThan(0)
+      }
+    }
+  })
+
+  it('archivable categories are a subset of the map', () => {
+    const ids = Object.keys(CATEGORY_DIRS) as DataCategoryId[]
+    for (const id of ARCHIVABLE_CATEGORIES) {
+      expect(ids).toContain(id)
+    }
+  })
+
+  it('keeps the retention-critical mappings stable', () => {
+    // The retention flows (age-based delete + pre-delete archive) are wired to these
+    // two categories; renaming their dirs is a data-loss-shaped change - fail loudly.
+    expect(CATEGORY_DIRS.captures).toEqual(['captures'])
+    expect(CATEGORY_DIRS.meetings).toEqual(['meetings'])
+  })
+})

src/main/backup/__tests__/auto-cleanup.test.ts

@@ -0,0 +1,145 @@
+import fs from 'node:fs'
+import os from 'node:os'
+import path from 'node:path'
+import JSZip from 'jszip'
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import {
+  AUTO_CLEANUP_INTERVAL_MS,
+  cleanupDue,
+  folderDeliver,
+  runAutoCleanup
+} from '../auto-cleanup'
+
+let userData: string
+let temp: string
+let archiveDir: string
+
+const writeOldCapture = (name: string, content = 'png', ageDays = 40): void => {
+  const dir = path.join(userData, 'captures')
+  fs.mkdirSync(dir, { recursive: true })
+  const p = path.join(dir, name)
+  fs.writeFileSync(p, content)
+  const t = new Date(Date.now() - ageDays * 86_400_000)
+  fs.utimesSync(p, t, t)
+}
+
+beforeEach(() => {
+  userData = fs.mkdtempSync(path.join(os.tmpdir(), 'offgrid-auto-ud-'))
+  temp = fs.mkdtempSync(path.join(os.tmpdir(), 'offgrid-auto-tmp-'))
+  archiveDir = fs.mkdtempSync(path.join(os.tmpdir(), 'offgrid-auto-arc-'))
+})
+afterEach(() => {
+  for (const d of [userData, temp, archiveDir]) fs.rmSync(d, { recursive: true, force: true })
+})
+
+describe('cleanupDue', () => {
+  it('is due on first ever run, after the interval, and not before', () => {
+    const now = 1_800_000_000_000
+    expect(cleanupDue(null, now)).toBe(true)
+    expect(cleanupDue(now - AUTO_CLEANUP_INTERVAL_MS - 1, now)).toBe(true)
+    expect(cleanupDue(now - AUTO_CLEANUP_INTERVAL_MS + 60_000, now)).toBe(false)
+  })
+})
+
+describe('folderDeliver', () => {
+  const stage = (content: string): string => {
+    const dir = fs.mkdtempSync(path.join(temp, 'stage-'))
+    const p = path.join(dir, 'a.zip')
+    fs.writeFileSync(p, content)
+    return p
+  }
+
+  it('copies into the folder, verifies the size, and removes staging', async () => {
+    const staged = stage('ZIPBYTES')
+    const result = await folderDeliver(archiveDir)(staged, 'offgrid-captures.zip')
+    expect(result.canceled).toBe(false)
+    expect(fs.readFileSync(result.path!, 'utf8')).toBe('ZIPBYTES')
+    expect(fs.existsSync(staged)).toBe(false)
+  })
+
+  it('never overwrites an earlier archive - collisions get a suffix', async () => {
+    fs.writeFileSync(path.join(archiveDir, 'offgrid-captures.zip'), 'EARLIER')
+    const result = await folderDeliver(archiveDir)(stage('NEWER'), 'offgrid-captures.zip')
+    expect(path.basename(result.path!)).toBe('offgrid-captures-2.zip')
+    expect(fs.readFileSync(path.join(archiveDir, 'offgrid-captures.zip'), 'utf8')).toBe('EARLIER')
+  })
+
+  it('an unwritable destination throws (which the orchestration treats as: do not prune)', async () => {
+    const file = path.join(temp, 'not-a-dir')
+    fs.writeFileSync(file, 'x')
+    await expect(folderDeliver(file)(stage('Z'), 'a.zip')).rejects.toThrow()
+  })
+})
+
+describe('runAutoCleanup', () => {
+  it('does nothing when retention is off', async () => {
+    const clear = vi.fn(async () => ({ success: true }))
+    const result = await runAutoCleanup({
+      config: { retentionDays: 0, archiveDir: null },
+      userDataDir: userData,
+      clear
+    })
+    expect(result.status).toBe('off')
+    expect(clear).not.toHaveBeenCalled()
+  })
+
+  it('with no archive folder it is a plain rolling window - prune, no ZIP', async () => {
+    writeOldCapture('old.png')
+    const clear = vi.fn(async () => ({ success: true }))
+    const result = await runAutoCleanup({
+      config: { retentionDays: 30, archiveDir: null },
+      userDataDir: userData,
+      clear
+    })
+    expect(result).toMatchObject({ status: 'cleared', archivedFiles: 0 })
+    expect(clear).toHaveBeenCalledTimes(1)
+    expect(fs.readdirSync(archiveDir)).toEqual([])
+  })
+
+  it('with a folder it archives the old captures, then prunes', async () => {
+    writeOldCapture('old.png', 'OLDPNG')
+    writeOldCapture('fresh.png', 'FRESH', 1) // inside the window - not archived
+    const clear = vi.fn(async () => ({ success: true }))
+    const result = await runAutoCleanup({
+      config: { retentionDays: 30, archiveDir },
+      userDataDir: userData,
+      tempDir: temp,
+      clear
+    })
+    expect(result.status).toBe('cleared')
+    expect(result.archivedFiles).toBe(1)
+    const zip = await JSZip.loadAsync(fs.readFileSync(result.archivePath!))
+    expect(await zip.file('captures/old.png')!.async('string')).toBe('OLDPNG')
+    expect(zip.file('captures/fresh.png')).toBeNull()
+    expect(clear).toHaveBeenCalledTimes(1)
+  })
+
+  it('a failed archive means nothing is pruned (fail closed)', async () => {
+    writeOldCapture('old.png')
+    const file = path.join(temp, 'blocked')
+    fs.writeFileSync(file, 'x') // archiveDir points at a FILE - copy will fail
+    const clear = vi.fn(async () => ({ success: true }))
+    const result = await runAutoCleanup({
+      config: { retentionDays: 30, archiveDir: file },
+      userDataDir: userData,
+      tempDir: temp,
+      clear
+    })
+    expect(result.status).toBe('failed')
+    expect(clear).not.toHaveBeenCalled()
+  })
+
+  it('nothing older than the window archives nothing and still reports cleared', async () => {
+    writeOldCapture('fresh.png', 'F', 2)
+    const clear = vi.fn(async () => ({ success: true }))
+    const result = await runAutoCleanup({
+      config: { retentionDays: 30, archiveDir },
+      userDataDir: userData,
+      tempDir: temp,
+      clear
+    })
+    expect(result).toMatchObject({ status: 'cleared', archivedFiles: 0 })
+    expect(fs.readdirSync(archiveDir)).toEqual([]) // no empty ZIPs accumulating
+    expect(clear).toHaveBeenCalledTimes(1)
+  })
+})