feat(cleanup): notify and delete idle workspaces on coder.ddev.com

CLAUDE.md

@@ -358,6 +358,7 @@ Additional logs in workspace:
 - `image/scripts/.ddev/global_config.yaml` - DDEV defaults copied into workspaces
 - `scripts/coder-delete-workspace-dir.sh` - Sudo wrapper for workspace host dir cleanup (must be installed on server)
 - `scripts/cleanup-deleted-workspaces.sh` - Manual cleanup for orphaned workspace dirs/volumes
+- `scripts/workspace-lifecycle-cleanup.sh` - Notifies then deletes idle workspaces; runs via systemd timer on coder.ddev.com (production only, see `scripts/workspace-lifecycle-cleanup.service`/`.timer`, must be installed on server)
 - `VERSION` - Image version used by all templates (read automatically by Makefile)
 - `openspec/project.md` - Project conventions and constraints
 - `openspec/AGENTS.md` - OpenSpec workflow instructions

docs/admin/operations-guide.md

@@ -417,6 +417,70 @@ docker system df
 
 ---
 
+### Automated Idle Workspace Cleanup
+
+A systemd timer runs `scripts/workspace-lifecycle-cleanup.sh` daily, directly on coder.ddev.com (production only — see [Server Setup Guide: Step 14](./server-setup.md#step-14-set-up-workspace-lifecycle-cleanup) for why staging doesn't need it), to keep idle workspaces from accumulating and slowly filling `/data` with `*-dind-cache` Docker volumes (each workspace keeps its full Docker-in-Docker cache volume until it's deleted, even while stopped).
+
+Policy per workspace, based on `last_used_at`:
+
+- **7 days idle** — owner gets a notice email (via Mailgun) explaining the workspace will be deleted, and pointing to the auth/access announcement if they can no longer log in.
+- **7 more days idle after the notice** — the workspace is deleted with `coder delete --yes`.
+- If the owner uses the workspace again before deletion, the pending notice is cleared automatically.
+
+State (which workspaces have been notified and when) is tracked in a local JSON file on the server (`/var/lib/workspace-lifecycle-cleanup/state.json` by default) — it never needs to leave the box, so there's no commit-back-to-git step to manage.
+
+```bash
+# Dry run — shows what would be notified/deleted, sends no email, deletes nothing
+./scripts/workspace-lifecycle-cleanup.sh
+
+# Actually send notices, delete workspaces past their grace period, and persist state
+./scripts/workspace-lifecycle-cleanup.sh --force
+```
+
+#### One-off purge (owners already notified out of band)
+
+`--purge-idle-days=N` bypasses the notify/grace state machine entirely and deletes every workspace idle at least `N` days (owners in `EXCLUDE_OWNERS` are still skipped). It sends no email, needs no Mailgun credentials, and neither reads nor writes the state file — the next normal timer run prunes any state entries for workspaces purged this way. Use it when owners have already been warned by other means (e.g. a manual email blast) and you need disk back now:
+
+```bash
+# Preview what a 14-day purge would delete — deletes nothing
+./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14
+
+# Actually delete every workspace idle >= 14 days
+./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 --force
+```
+
+Like the normal flow, purge requires the `coder` CLI authenticated with the `owner` role (see below). Always run the preview first and eyeball the list — purge has no grace period and no second chance.
+
+**Prerequisites:** `coder` CLI on `PATH` and authenticated (either via `coder login` or the `CODER_URL`/`CODER_SESSION_TOKEN` env vars) as a user with the `owner` role (see below); `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` set for `--force` runs — DDEV's existing Mailgun account credentials are in the shared **DDEV** 1Password vault, item **Mailgun** (same account/domain used to send other DDEV mail; no new domain or DNS verification needed). See the script header for all environment overrides (`NOTIFY_DAYS`, `DELETE_AFTER_DAYS`, `EXCLUDE_OWNERS`, `STATE_FILE`, etc.).
+
+On the server, all of this is supplied via `/etc/workspace-lifecycle-cleanup.env`, loaded by the `workspace-lifecycle-cleanup.service` systemd unit — see the install steps linked above. Check `sudo systemctl status workspace-lifecycle-cleanup.timer` and `sudo journalctl -u workspace-lifecycle-cleanup -q -f` to inspect runs.
+
+#### Provisioning the `CODER_SESSION_TOKEN` credential
+
+The script needs to list *every* user's workspaces (`coder list --all`) and delete workspaces it doesn't own. On this deployment (no Premium license, so no custom RBAC roles), `owner` is the only built-in role that can do both — there's no narrower "workspace admin" role available. That makes this token effectively full site-admin, so it's provisioned as a dedicated non-human account rather than a personal token:
+
+```bash
+# 1. Create a machine identity — no GitHub OAuth login required
+coder users create --username workspace-janitor \
+  --email workspace-janitor@ddev.com \
+  --full-name "Workspace Lifecycle Janitor" \
+  --login-type none
+
+# 2. Grant it owner — the only built-in role that covers list-all + delete-any-workspace
+coder users edit-roles workspace-janitor --roles owner --yes
+
+# 3. Mint a long-lived token for it (run as an existing owner/admin, e.g. your own account)
+coder tokens create -u workspace-janitor --name workspace-lifecycle-cleanup --lifetime 8760h
+```
+
+Put the resulting token in `/etc/workspace-lifecycle-cleanup.env` as `CODER_SESSION_TOKEN` (see the install steps). Using a dedicated account rather than a personal token keeps deletions attributable to the bot (not an individual) in the audit log, and means the janitor doesn't break if the admin's own account is later deactivated or re-authenticated.
+
+Check the server's configured max token lifetime before choosing `--lifetime` — if it's capped below a year, use the max allowed and set a reminder to rotate the token before it expires, since an expired token makes the timer fail silently until someone notices (`journalctl -u workspace-lifecycle-cleanup` will show the auth error).
+
+`--login-type none` is deprecated in favor of `--service-account` (Premium-only) but remains functional for this purpose.
+
+---
+
 ### Orphaned Workspace Cleanup
 
 When a workspace is deleted, the destroy provisioner automatically removes the host directory at `/coder-workspaces/<owner>-<workspace>`. Directories can still be orphaned if the provisioner fails or for workspaces deleted before the provisioner was added. Run the cleanup script to reclaim disk space in those cases.

docs/admin/server-setup.md

@@ -1280,6 +1280,81 @@ Update this variable whenever the issue is closed or merged. The current default
 
 ---
 
+## Step 14: Set Up Workspace Lifecycle Cleanup
+
+Stopped workspaces don't free their Docker volumes — each Sysbox workspace keeps a multi-GB `*-dind-cache` volume until it's deleted, so idle workspaces slowly fill `/data`. A systemd timer runs `scripts/workspace-lifecycle-cleanup.sh` daily to email owners of idle workspaces, then delete them if they stay idle after the notice period.
+
+This is set up on **coder.ddev.com (production) only**. staging-coder.ddev.com doesn't need it: its workspaces are almost entirely owned by `ci-bot` (already excluded via the default `EXCLUDE_OWNERS`), it isn't sending real notices to real people, and `./scripts/workspace-lifecycle-cleanup.sh` (no `--force`) already gives a safe dry-run preview against production itself when you need to sanity-check the policy — no separate staging deployment adds coverage. If that changes (e.g. staging starts accumulating real user workspaces), repeat these steps there with its own `workspace-janitor` account/token and a verified Mailgun domain for staging.
+
+### Create the `workspace-janitor` Coder account and token
+
+The script needs `owner` (list-all + delete-any-workspace; this deployment has no Premium license, so there's no narrower role):
+
+```bash
+coder users create --username workspace-janitor \
+  --email workspace-janitor@ddev.com \
+  --full-name "Workspace Lifecycle Janitor" \
+  --login-type none
+
+coder users edit-roles workspace-janitor --roles owner --yes
+
+coder tokens create -u workspace-janitor --name workspace-lifecycle-cleanup --lifetime 8760h
+```
+
+Copy the printed token — you'll put it in the env file below. See [Automated Idle Workspace Cleanup](./operations-guide.md#automated-idle-workspace-cleanup) in the operations guide for why a dedicated account is used instead of a personal token.
+
+### Install the cleanup service
+
+```bash
+REPO=~/workspace/coder-ddev   # adjust if your repo is elsewhere
+
+# Install the script
+sudo install -m 755 $REPO/scripts/workspace-lifecycle-cleanup.sh /usr/local/bin/workspace-lifecycle-cleanup
+
+# Create the state directory
+sudo mkdir -p /var/lib/workspace-lifecycle-cleanup
+sudo chown rfay:rfay /var/lib/workspace-lifecycle-cleanup
+
+# Create the env file — fill in the token from above and the Mailgun API key.
+# (1Password shared "DDEV" vault → "Mailgun" item → field "coder.ddev.com sending
+# api key". The domain is already a verified Mailgun sending domain — no new
+# Mailgun domain setup or DNS/SPF/DKIM records are needed.)
+sudo tee /etc/workspace-lifecycle-cleanup.env > /dev/null <<'EOF'
+CODER_URL=https://coder.ddev.com
+CODER_SESSION_TOKEN=REPLACE_WITH_WORKSPACE_JANITOR_TOKEN
+MAILGUN_API_KEY=REPLACE_WITH_KEY_FROM_1PASSWORD
+MAILGUN_DOMAIN=coder.ddev.com
+STATE_FILE=/var/lib/workspace-lifecycle-cleanup/state.json
+EOF
+sudo chmod 600 /etc/workspace-lifecycle-cleanup.env
+
+# Install and enable the systemd service + timer
+sudo install -m 644 $REPO/scripts/workspace-lifecycle-cleanup.service /etc/systemd/system/
+sudo install -m 644 $REPO/scripts/workspace-lifecycle-cleanup.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now workspace-lifecycle-cleanup.timer
+```
+
+### Test it
+
+```bash
+# Dry run as the service user, without touching the timer
+sudo -u rfay bash -c 'set -a; source /etc/workspace-lifecycle-cleanup.env; set +a; /usr/local/bin/workspace-lifecycle-cleanup'
+
+# Or run the real service unit once, on demand
+sudo systemctl start workspace-lifecycle-cleanup.service
+sudo journalctl -u workspace-lifecycle-cleanup -q -n50
+```
+
+### Notes
+
+- The `[Service]` unit's `User=rfay` only needs to run `coder`, `curl`, and `jq` — no special Linux privileges are required; change it to any account that can read the env file if `rfay` isn't appropriate on a given box.
+- `coder` picks up `CODER_URL`/`CODER_SESSION_TOKEN` directly from the environment — no `coder login` session is needed for the service user.
+- The timer fires daily at 06:17 server time (`RandomizedDelaySec=5m` to avoid a fixed-second thundering herd); check `systemctl list-timers workspace-lifecycle-cleanup.timer` to see the next run.
+- If you rotate the `workspace-janitor` token, update `/etc/workspace-lifecycle-cleanup.env` — no restart needed, it's read fresh on each timer-triggered run.
+
+---
+
 ## Troubleshooting
 
 **Coder service won't start:**

scripts/workspace-lifecycle-cleanup.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Notify and delete idle Coder workspaces
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+User=rfay
+EnvironmentFile=/etc/workspace-lifecycle-cleanup.env
+ExecStart=/usr/local/bin/workspace-lifecycle-cleanup --force