diff --git a/CLAUDE.md b/CLAUDE.md index cf313dc..26f909f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -358,6 +358,7 @@ Additional logs in workspace: - `image/scripts/.ddev/global_config.yaml` - DDEV defaults copied into workspaces - `scripts/coder-delete-workspace-dir.sh` - Sudo wrapper for workspace host dir cleanup (must be installed on server) - `scripts/cleanup-deleted-workspaces.sh` - Manual cleanup for orphaned workspace dirs/volumes +- `scripts/workspace-lifecycle-cleanup.sh` - Notifies then deletes idle workspaces; runs via systemd timer on coder.ddev.com (production only, see `scripts/workspace-lifecycle-cleanup.service`/`.timer`, must be installed on server) - `VERSION` - Image version used by all templates (read automatically by Makefile) - `openspec/project.md` - Project conventions and constraints - `openspec/AGENTS.md` - OpenSpec workflow instructions diff --git a/docs/admin/operations-guide.md b/docs/admin/operations-guide.md index 1b59241..265843c 100644 --- a/docs/admin/operations-guide.md +++ b/docs/admin/operations-guide.md @@ -417,6 +417,70 @@ docker system df --- +### Automated Idle Workspace Cleanup + +A systemd timer runs `scripts/workspace-lifecycle-cleanup.sh` daily, directly on coder.ddev.com (production only — see [Server Setup Guide: Step 14](./server-setup.md#step-14-set-up-workspace-lifecycle-cleanup) for why staging doesn't need it), to keep idle workspaces from accumulating and slowly filling `/data` with `*-dind-cache` Docker volumes (each workspace keeps its full Docker-in-Docker cache volume until it's deleted, even while stopped). + +Policy per workspace, based on `last_used_at`: + +- **7 days idle** — owner gets a notice email (via Mailgun) explaining the workspace will be deleted, and pointing to the auth/access announcement if they can no longer log in. +- **7 more days idle after the notice** — the workspace is deleted with `coder delete --yes`. +- If the owner uses the workspace again before deletion, the pending notice is cleared automatically. + +State (which workspaces have been notified and when) is tracked in a local JSON file on the server (`/var/lib/workspace-lifecycle-cleanup/state.json` by default) — it never needs to leave the box, so there's no commit-back-to-git step to manage. + +```bash +# Dry run — shows what would be notified/deleted, sends no email, deletes nothing +./scripts/workspace-lifecycle-cleanup.sh + +# Actually send notices, delete workspaces past their grace period, and persist state +./scripts/workspace-lifecycle-cleanup.sh --force +``` + +#### One-off purge (owners already notified out of band) + +`--purge-idle-days=N` bypasses the notify/grace state machine entirely and deletes every workspace idle at least `N` days (owners in `EXCLUDE_OWNERS` are still skipped). It sends no email, needs no Mailgun credentials, and neither reads nor writes the state file — the next normal timer run prunes any state entries for workspaces purged this way. Use it when owners have already been warned by other means (e.g. a manual email blast) and you need disk back now: + +```bash +# Preview what a 14-day purge would delete — deletes nothing +./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 + +# Actually delete every workspace idle >= 14 days +./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 --force +``` + +Like the normal flow, purge requires the `coder` CLI authenticated with the `owner` role (see below). Always run the preview first and eyeball the list — purge has no grace period and no second chance. + +**Prerequisites:** `coder` CLI on `PATH` and authenticated (either via `coder login` or the `CODER_URL`/`CODER_SESSION_TOKEN` env vars) as a user with the `owner` role (see below); `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` set for `--force` runs — DDEV's existing Mailgun account credentials are in the shared **DDEV** 1Password vault, item **Mailgun** (same account/domain used to send other DDEV mail; no new domain or DNS verification needed). See the script header for all environment overrides (`NOTIFY_DAYS`, `DELETE_AFTER_DAYS`, `EXCLUDE_OWNERS`, `STATE_FILE`, etc.). + +On the server, all of this is supplied via `/etc/workspace-lifecycle-cleanup.env`, loaded by the `workspace-lifecycle-cleanup.service` systemd unit — see the install steps linked above. Check `sudo systemctl status workspace-lifecycle-cleanup.timer` and `sudo journalctl -u workspace-lifecycle-cleanup -q -f` to inspect runs. + +#### Provisioning the `CODER_SESSION_TOKEN` credential + +The script needs to list *every* user's workspaces (`coder list --all`) and delete workspaces it doesn't own. On this deployment (no Premium license, so no custom RBAC roles), `owner` is the only built-in role that can do both — there's no narrower "workspace admin" role available. That makes this token effectively full site-admin, so it's provisioned as a dedicated non-human account rather than a personal token: + +```bash +# 1. Create a machine identity — no GitHub OAuth login required +coder users create --username workspace-janitor \ + --email workspace-janitor@ddev.com \ + --full-name "Workspace Lifecycle Janitor" \ + --login-type none + +# 2. Grant it owner — the only built-in role that covers list-all + delete-any-workspace +coder users edit-roles workspace-janitor --roles owner --yes + +# 3. Mint a long-lived token for it (run as an existing owner/admin, e.g. your own account) +coder tokens create -u workspace-janitor --name workspace-lifecycle-cleanup --lifetime 8760h +``` + +Put the resulting token in `/etc/workspace-lifecycle-cleanup.env` as `CODER_SESSION_TOKEN` (see the install steps). Using a dedicated account rather than a personal token keeps deletions attributable to the bot (not an individual) in the audit log, and means the janitor doesn't break if the admin's own account is later deactivated or re-authenticated. + +Check the server's configured max token lifetime before choosing `--lifetime` — if it's capped below a year, use the max allowed and set a reminder to rotate the token before it expires, since an expired token makes the timer fail silently until someone notices (`journalctl -u workspace-lifecycle-cleanup` will show the auth error). + +`--login-type none` is deprecated in favor of `--service-account` (Premium-only) but remains functional for this purpose. + +--- + ### Orphaned Workspace Cleanup When a workspace is deleted, the destroy provisioner automatically removes the host directory at `/coder-workspaces/-`. Directories can still be orphaned if the provisioner fails or for workspaces deleted before the provisioner was added. Run the cleanup script to reclaim disk space in those cases. diff --git a/docs/admin/server-setup.md b/docs/admin/server-setup.md index e72c5a4..a402ea8 100644 --- a/docs/admin/server-setup.md +++ b/docs/admin/server-setup.md @@ -1280,6 +1280,81 @@ Update this variable whenever the issue is closed or merged. The current default --- +## Step 14: Set Up Workspace Lifecycle Cleanup + +Stopped workspaces don't free their Docker volumes — each Sysbox workspace keeps a multi-GB `*-dind-cache` volume until it's deleted, so idle workspaces slowly fill `/data`. A systemd timer runs `scripts/workspace-lifecycle-cleanup.sh` daily to email owners of idle workspaces, then delete them if they stay idle after the notice period. + +This is set up on **coder.ddev.com (production) only**. staging-coder.ddev.com doesn't need it: its workspaces are almost entirely owned by `ci-bot` (already excluded via the default `EXCLUDE_OWNERS`), it isn't sending real notices to real people, and `./scripts/workspace-lifecycle-cleanup.sh` (no `--force`) already gives a safe dry-run preview against production itself when you need to sanity-check the policy — no separate staging deployment adds coverage. If that changes (e.g. staging starts accumulating real user workspaces), repeat these steps there with its own `workspace-janitor` account/token and a verified Mailgun domain for staging. + +### Create the `workspace-janitor` Coder account and token + +The script needs `owner` (list-all + delete-any-workspace; this deployment has no Premium license, so there's no narrower role): + +```bash +coder users create --username workspace-janitor \ + --email workspace-janitor@ddev.com \ + --full-name "Workspace Lifecycle Janitor" \ + --login-type none + +coder users edit-roles workspace-janitor --roles owner --yes + +coder tokens create -u workspace-janitor --name workspace-lifecycle-cleanup --lifetime 8760h +``` + +Copy the printed token — you'll put it in the env file below. See [Automated Idle Workspace Cleanup](./operations-guide.md#automated-idle-workspace-cleanup) in the operations guide for why a dedicated account is used instead of a personal token. + +### Install the cleanup service + +```bash +REPO=~/workspace/coder-ddev # adjust if your repo is elsewhere + +# Install the script +sudo install -m 755 $REPO/scripts/workspace-lifecycle-cleanup.sh /usr/local/bin/workspace-lifecycle-cleanup + +# Create the state directory +sudo mkdir -p /var/lib/workspace-lifecycle-cleanup +sudo chown rfay:rfay /var/lib/workspace-lifecycle-cleanup + +# Create the env file — fill in the token from above and the Mailgun API key. +# (1Password shared "DDEV" vault → "Mailgun" item → field "coder.ddev.com sending +# api key". The domain is already a verified Mailgun sending domain — no new +# Mailgun domain setup or DNS/SPF/DKIM records are needed.) +sudo tee /etc/workspace-lifecycle-cleanup.env > /dev/null <<'EOF' +CODER_URL=https://coder.ddev.com +CODER_SESSION_TOKEN=REPLACE_WITH_WORKSPACE_JANITOR_TOKEN +MAILGUN_API_KEY=REPLACE_WITH_KEY_FROM_1PASSWORD +MAILGUN_DOMAIN=coder.ddev.com +STATE_FILE=/var/lib/workspace-lifecycle-cleanup/state.json +EOF +sudo chmod 600 /etc/workspace-lifecycle-cleanup.env + +# Install and enable the systemd service + timer +sudo install -m 644 $REPO/scripts/workspace-lifecycle-cleanup.service /etc/systemd/system/ +sudo install -m 644 $REPO/scripts/workspace-lifecycle-cleanup.timer /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now workspace-lifecycle-cleanup.timer +``` + +### Test it + +```bash +# Dry run as the service user, without touching the timer +sudo -u rfay bash -c 'set -a; source /etc/workspace-lifecycle-cleanup.env; set +a; /usr/local/bin/workspace-lifecycle-cleanup' + +# Or run the real service unit once, on demand +sudo systemctl start workspace-lifecycle-cleanup.service +sudo journalctl -u workspace-lifecycle-cleanup -q -n50 +``` + +### Notes + +- The `[Service]` unit's `User=rfay` only needs to run `coder`, `curl`, and `jq` — no special Linux privileges are required; change it to any account that can read the env file if `rfay` isn't appropriate on a given box. +- `coder` picks up `CODER_URL`/`CODER_SESSION_TOKEN` directly from the environment — no `coder login` session is needed for the service user. +- The timer fires daily at 06:17 server time (`RandomizedDelaySec=5m` to avoid a fixed-second thundering herd); check `systemctl list-timers workspace-lifecycle-cleanup.timer` to see the next run. +- If you rotate the `workspace-janitor` token, update `/etc/workspace-lifecycle-cleanup.env` — no restart needed, it's read fresh on each timer-triggered run. + +--- + ## Troubleshooting **Coder service won't start:** diff --git a/scripts/workspace-lifecycle-cleanup.service b/scripts/workspace-lifecycle-cleanup.service new file mode 100644 index 0000000..a94b85b --- /dev/null +++ b/scripts/workspace-lifecycle-cleanup.service @@ -0,0 +1,10 @@ +[Unit] +Description=Notify and delete idle Coder workspaces +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=rfay +EnvironmentFile=/etc/workspace-lifecycle-cleanup.env +ExecStart=/usr/local/bin/workspace-lifecycle-cleanup --force diff --git a/scripts/workspace-lifecycle-cleanup.sh b/scripts/workspace-lifecycle-cleanup.sh new file mode 100755 index 0000000..2d8d7fa --- /dev/null +++ b/scripts/workspace-lifecycle-cleanup.sh @@ -0,0 +1,427 @@ +#!/usr/bin/env bash +# Notify and delete idle workspaces on coder.ddev.com. +# +# Why this exists: +# Stopping a workspace does not free its Docker volume — each Sysbox +# workspace keeps a multi-GB dind-cache volume until the workspace is +# deleted (the destroy provisioner in each template removes it). With no +# lifecycle policy, stopped workspaces accumulate forever and slowly fill +# /data on the host. This janitor enforces: notify at N days idle, delete +# 7 days after the notice if the workspace is still untouched. +# +# Policy (state machine per workspace, keyed by workspace id): +# - Not yet notified, age(last_used_at) >= NOTIFY_DAYS +# -> email the owner, record notified_at=now in the state file +# - Already notified, but last_used_at advanced past the notice +# -> the owner came back; clear the state entry (no deletion) +# - Already notified, DELETE_AFTER_DAYS have passed since notified_at, +# and last_used_at still hasn't advanced +# -> `coder delete /`, drop the state entry (on failure +# the entry is kept so deletion is retried next run) +# - Already notified, still within the grace window +# -> leave alone +# - State entry exists for a workspace that no longer exists +# -> drop the stale entry +# +# The very first run against a fleet with no history only sends notices — +# nothing is old enough to be "already notified" yet, so nothing gets +# deleted until a full DELETE_AFTER_DAYS grace period has elapsed from the +# first notice. That is the intended one-time "grandfather" behavior; no +# separate rollout mode is needed. +# +# One-off purge mode (--purge-idle-days=N): +# Bypasses the notify/grace state machine and deletes every workspace idle +# >= N days (EXCLUDE_OWNERS still applies). Sends no email and neither +# reads nor writes the state file — the next normal run prunes state +# entries for anything purged here. Use when owners have already been +# notified out of band. Like everything else, it's a dry run unless +# combined with --force. Mailgun credentials are not required. +# +# By default runs in dry-run mode (prints what would happen, does not send +# email, delete workspaces, or write the state file). Pass --force to act. +# +# `coder delete`'s own build-log output is captured rather than streamed — +# it's noise on success. It's only shown (indented) when a delete fails. +# +# Every run ends with a counts line (notified=... deleted=... failed=...) +# followed by a "what and why" breakdown: one section per category listing +# the affected workspace and the reason, e.g. under "Failed" a delete that +# will be retried next run, or a notice email that didn't send. +# +# Requires: coder CLI (authenticated), jq, curl, a Mailgun account. +# +# Environment: +# CODER_URL Coder deployment to operate on (informational; auth +# is whatever the coder CLI is currently logged into) +# NOTIFY_DAYS Idle days before the first notice (default: 7) +# DELETE_AFTER_DAYS Days after notice before deletion (default: 7) +# EXCLUDE_OWNERS Space-separated owner usernames to skip (default: ci-bot) +# STATE_FILE Path to the JSON state file +# (default: scripts/state/workspace-lifecycle-state.json) +# MAILGUN_API_KEY Mailgun private API key (required to send, not for dry-run) +# MAILGUN_DOMAIN Mailgun sending domain, e.g. mg.ddev.com +# MAILGUN_BASE_URL Mailgun API base (default: https://api.mailgun.net/v3; +# use https://api.eu.mailgun.net/v3 for EU-region domains) +# MAILGUN_FROM From header (default: "DDEV Coder ") +# ANNOUNCE_URL Blog post explaining the auth change +# (default: https://ddev.com/blog/coder-ddev-com-announcement/) +# +# Usage: +# ./scripts/workspace-lifecycle-cleanup.sh # dry run +# ./scripts/workspace-lifecycle-cleanup.sh --force # actually notify/delete +# ./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 # preview purge +# ./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 --force # delete idle >=14d now + +set -euo pipefail + +NOTIFY_DAYS="${NOTIFY_DAYS:-7}" +DELETE_AFTER_DAYS="${DELETE_AFTER_DAYS:-7}" +EXCLUDE_OWNERS="${EXCLUDE_OWNERS:-ci-bot}" +STATE_FILE="${STATE_FILE:-scripts/state/workspace-lifecycle-state.json}" +MAILGUN_BASE_URL="${MAILGUN_BASE_URL:-https://api.mailgun.net/v3}" +MAILGUN_FROM="${MAILGUN_FROM:-DDEV Coder }" +ANNOUNCE_URL="${ANNOUNCE_URL:-https://ddev.com/blog/coder-ddev-com-announcement/}" +FORCE=false +PURGE_IDLE_DAYS="" + +for arg in "$@"; do + case "$arg" in + --force) FORCE=true ;; + --purge-idle-days=*) + PURGE_IDLE_DAYS="${arg#*=}" + if ! [[ "$PURGE_IDLE_DAYS" =~ ^[0-9]+$ ]] || [[ "$PURGE_IDLE_DAYS" -eq 0 ]]; then + echo "ERROR: --purge-idle-days requires a whole number of days >= 1, got '${PURGE_IDLE_DAYS}'" >&2 + exit 1 + fi + ;; + --help | -h) + awk 'NR > 1 && !/^#/ { exit } NR > 1 { sub(/^# ?/, ""); print }' "$0" + exit 0 + ;; + *) + echo "Unknown argument: $arg" >&2 + exit 1 + ;; + esac +done + +if [[ "$FORCE" == false ]]; then + echo "DRY RUN — pass --force to actually notify/delete and persist state" + echo +fi + +# Purge mode sends no email, so Mailgun credentials are only needed for +# the normal notify/delete flow. +if [[ "$FORCE" == true && -z "$PURGE_IDLE_DAYS" ]]; then + : "${MAILGUN_API_KEY:?MAILGUN_API_KEY is required with --force}" + : "${MAILGUN_DOMAIN:?MAILGUN_DOMAIN is required with --force}" +fi + +# Portable date handling: prefer GNU date (Linux, or `gdate` from +# coreutils on macOS); fall back to BSD date's `-j -f` / `-r` syntax. +if date -d "1970-01-01T00:00:00Z" +%s >/dev/null 2>&1; then + DATE_BIN=(date) +elif command -v gdate >/dev/null 2>&1; then + DATE_BIN=(gdate) +else + DATE_BIN=() +fi + +iso_to_epoch() { + local iso="$1" + if [[ ${#DATE_BIN[@]} -gt 0 ]]; then + "${DATE_BIN[@]}" -d "$iso" +%s 2>/dev/null && return + fi + date -j -f "%Y-%m-%dT%H:%M:%S" "${iso%%.*}" +%s 2>/dev/null || echo 0 +} + +epoch_to_ymd() { + local epoch="$1" + if [[ ${#DATE_BIN[@]} -gt 0 ]]; then + "${DATE_BIN[@]}" -d "@${epoch}" +%Y-%m-%d 2>/dev/null && return + fi + date -j -r "$epoch" +%Y-%m-%d 2>/dev/null || echo "unknown" +} + +epoch_to_iso() { + local epoch="$1" + if [[ ${#DATE_BIN[@]} -gt 0 ]]; then + "${DATE_BIN[@]}" -u -d "@${epoch}" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null && return + fi + date -u -j -r "$epoch" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null +} + +if ! workspaces_json=$(coder list --all --output json 2>/dev/null); then + echo "ERROR: 'coder list --all --output json' failed. Is the coder CLI authenticated?" >&2 + exit 1 +fi + +workspace_count=$(echo "$workspaces_json" | jq 'length') +if [[ "$workspace_count" -eq 0 ]]; then + echo "ERROR: 'coder list' returned zero workspaces. Refusing to proceed — this looks like an auth or connectivity problem, not an empty fleet." >&2 + exit 1 +fi + +now=$(date +%s) + +# Print a labeled list only if non-empty — used to build the detailed, +# what-and-why breakdown that follows the terse counts summary. +print_details() { + local title="$1" + shift + local -a items=("$@") + if [[ ${#items[@]} -gt 0 ]]; then + echo + echo "${title}:" + printf ' - %s\n' "${items[@]}" + fi +} + +# --- One-off purge mode: delete everything idle >= N days, no notices --- +if [[ -n "$PURGE_IDLE_DAYS" ]]; then + purge_deleted=0 purge_failed=0 purge_kept=0 + purge_deleted_details=() purge_failed_details=() + while IFS=$'\t' read -r id name owner last_used_at; do + [[ -z "$id" ]] && continue + + if grep -qx "$owner" <<<"$(tr ' ' '\n' <<<"$EXCLUDE_OWNERS")"; then + continue + fi + + last_epoch=$(iso_to_epoch "$last_used_at") + age_days=$(((now - last_epoch) / 86400)) + + if [[ "$age_days" -ge "$PURGE_IDLE_DAYS" ]]; then + reason="idle ${age_days}d (>= ${PURGE_IDLE_DAYS}d), last used $(epoch_to_ymd "$last_epoch")" + echo "DELETE $owner/$name — $reason" + if [[ "$FORCE" == true ]]; then + # Capture coder's build-log output instead of letting it stream to + # the terminal/journal — it's noisy and uninteresting on success; + # only show it when the delete actually failed. + if delete_output=$(coder delete "$owner/$name" --yes 2>&1); then + purge_deleted=$((purge_deleted + 1)) + purge_deleted_details+=("$owner/$name — $reason") + else + echo " WARN: delete failed for $owner/$name" >&2 + [[ -n "$delete_output" ]] && echo " ${delete_output//$'\n'/$'\n' }" >&2 + purge_failed=$((purge_failed + 1)) + purge_failed_details+=("$owner/$name — delete failed, $reason") + fi + else + purge_deleted=$((purge_deleted + 1)) + purge_deleted_details+=("$owner/$name — $reason") + fi + else + purge_kept=$((purge_kept + 1)) + fi + done < <(echo "$workspaces_json" | jq -r '.[] | [.id, .name, .owner_name, .last_used_at] | @tsv') + + echo + echo "Purge summary: deleted=$purge_deleted failed=$purge_failed kept=$purge_kept" + print_details "Deleted" "${purge_deleted_details[@]}" + print_details "Failed to delete" "${purge_failed_details[@]}" + if [[ "$FORCE" == false ]]; then + echo + echo "Re-run with --force to actually delete." + fi + exit 0 +fi + +mkdir -p "$(dirname "$STATE_FILE")" +if [[ -f "$STATE_FILE" ]]; then + state_json=$(cat "$STATE_FILE") +else + state_json='{}' +fi + +# Persist immediately after every state transition, not just at the end of +# the run: notice emails are the one side effect that can't be taken back, +# so a mid-run failure must not forget who has already been notified. +persist_state() { + if [[ "$FORCE" == true ]]; then + echo "$state_json" | jq '.' >"$STATE_FILE" + fi +} + +if ! users_json=$(coder users list --output json 2>/dev/null); then + echo "ERROR: 'coder users list --output json' failed." >&2 + exit 1 +fi + +send_notice_email() { + local to_email="$1" owner_name="$2" workspace_name="$3" last_used_human="$4" delete_date_human="$5" + + local subject="Action needed: your coder.ddev.com workspace \"${workspace_name}\" will be deleted on ${delete_date_human}" + # `read -d ''` rather than `body=$(cat </dev/null; then + echo " WARN: Mailgun send to ${to_email} failed" >&2 + return 1 + fi + fi +} + +notified=0 revived=0 deleted=0 pending=0 kept=0 pruned=0 failed=0 +notified_details=() revived_details=() deleted_details=() pending_details=() pruned_details=() failed_details=() + +# --- Prune state entries for workspaces that no longer exist --- +current_ids=$(echo "$workspaces_json" | jq -r '.[].id') +new_state_json="$state_json" +while IFS= read -r stale_id; do + [[ -z "$stale_id" ]] && continue + if ! grep -qx "$stale_id" <<<"$current_ids"; then + echo "PRUNE state entry for deleted workspace $stale_id" + new_state_json=$(echo "$new_state_json" | jq --arg id "$stale_id" 'del(.[$id])') + pruned=$((pruned + 1)) + pruned_details+=("$stale_id — workspace no longer exists") + fi +done < <(echo "$state_json" | jq -r 'keys[]') +state_json="$new_state_json" +persist_state + +# --- Walk workspaces --- +while IFS=$'\t' read -r id name owner last_used_at; do + [[ -z "$id" ]] && continue + + if grep -qx "$owner" <<<"$(tr ' ' '\n' <<<"$EXCLUDE_OWNERS")"; then + continue + fi + + last_epoch=$(iso_to_epoch "$last_used_at") + age_days=$(((now - last_epoch) / 86400)) + + entry=$(echo "$state_json" | jq --arg id "$id" '.[$id]') + + if [[ "$entry" != "null" ]]; then + notified_at=$(echo "$entry" | jq -r '.notified_at') + notified_epoch=$(iso_to_epoch "$notified_at") + + if [[ "$last_epoch" -gt "$notified_epoch" ]]; then + echo "REVIVE $name (owner=$owner) — used again since notice, clearing pending deletion" + revived=$((revived + 1)) + revived_details+=("$owner/$name — used again since notice, deletion canceled") + state_json=$(echo "$state_json" | jq --arg id "$id" 'del(.[$id])') + persist_state + continue + fi + + since_notice_days=$(((now - notified_epoch) / 86400)) + if [[ "$since_notice_days" -ge "$DELETE_AFTER_DAYS" ]]; then + reason="idle ${age_days}d, notified ${since_notice_days}d ago, never used since" + echo "DELETE $owner/$name — $reason" + if [[ "$FORCE" == true ]]; then + # Capture coder's build-log output instead of streaming it — only + # show it (indented, on stderr) when the delete actually failed. + if delete_output=$(coder delete "$owner/$name" --yes 2>&1); then + deleted=$((deleted + 1)) + deleted_details+=("$owner/$name — $reason") + state_json=$(echo "$state_json" | jq --arg id "$id" 'del(.[$id])') + persist_state + else + echo " WARN: delete failed for $owner/$name — keeping state entry to retry next run" >&2 + [[ -n "$delete_output" ]] && echo " ${delete_output//$'\n'/$'\n' }" >&2 + failed=$((failed + 1)) + failed_details+=("$owner/$name — delete failed, will retry next run ($reason)") + fi + else + deleted=$((deleted + 1)) + deleted_details+=("$owner/$name — $reason") + fi + else + echo "PENDING $name (owner=$owner) — notified ${since_notice_days}d ago, deletes in $((DELETE_AFTER_DAYS - since_notice_days))d unless used" + pending=$((pending + 1)) + pending_details+=("$owner/$name — notified ${since_notice_days}d ago, deletes in $((DELETE_AFTER_DAYS - since_notice_days))d unless used") + fi + continue + fi + + if [[ "$age_days" -ge "$NOTIFY_DAYS" ]]; then + owner_email=$(echo "$users_json" | jq -r --arg u "$owner" '.[] | select(.username==$u) | .email // empty') + owner_display=$(echo "$users_json" | jq -r --arg u "$owner" '.[] | select(.username==$u) | .name // empty') + if [[ -z "$owner_email" ]]; then + echo " WARN: no email found for owner $owner (workspace $name), skipping notice" >&2 + failed=$((failed + 1)) + failed_details+=("$owner/$name — no email on file for owner, notice skipped") + continue + fi + + last_used_human=$(epoch_to_ymd "$last_epoch") + delete_epoch=$((now + DELETE_AFTER_DAYS * 86400)) + delete_date_human=$(epoch_to_ymd "$delete_epoch") + + echo "NOTIFY $name (owner=$owner <$owner_email>) — idle ${age_days}d, last used $last_used_human, deletes $delete_date_human unless used" + if send_notice_email "$owner_email" "${owner_display:-$owner}" "$name" "$last_used_human" "$delete_date_human"; then + notified=$((notified + 1)) + notified_details+=("$owner/$name <$owner_email> — idle ${age_days}d, last used $last_used_human, deletes $delete_date_human unless used") + state_json=$(echo "$state_json" | jq --arg id "$id" --arg at "$(epoch_to_iso "$now")" --arg name "$name" --arg owner "$owner" \ + '.[$id] = {notified_at: $at, name: $name, owner: $owner}') + persist_state + else + echo " WARN: notice for $name not recorded; will retry next run" >&2 + failed=$((failed + 1)) + failed_details+=("$owner/$name — notice email failed, will retry next run") + fi + else + kept=$((kept + 1)) + fi +done < <(echo "$workspaces_json" | jq -r '.[] | [.id, .name, .owner_name, .last_used_at] | @tsv') + +echo +echo "Summary: notified=$notified pending=$pending deleted=$deleted revived=$revived kept=$kept pruned=$pruned failed=$failed" +print_details "Deleted" "${deleted_details[@]}" +print_details "Notified" "${notified_details[@]}" +print_details "Revived" "${revived_details[@]}" +print_details "Pending (notified, in grace period)" "${pending_details[@]}" +print_details "Pruned state entries" "${pruned_details[@]}" +print_details "Failed (will retry next run)" "${failed_details[@]}" + +if [[ "$FORCE" == true ]]; then + persist_state + echo "State written to $STATE_FILE" +else + echo + echo "Re-run with --force to send notices, delete workspaces, and persist state." +fi diff --git a/scripts/workspace-lifecycle-cleanup.timer b/scripts/workspace-lifecycle-cleanup.timer new file mode 100644 index 0000000..7b21997 --- /dev/null +++ b/scripts/workspace-lifecycle-cleanup.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Daily idle Coder workspace cleanup + +[Timer] +OnCalendar=*-*-* 06:17:00 +RandomizedDelaySec=5m +Persistent=true + +[Install] +WantedBy=timers.target