feat(cleanup): notify and delete idle workspaces on coder.ddev.com#175
Open
rfay wants to merge 12 commits into
Open
feat(cleanup): notify and delete idle workspaces on coder.ddev.com#175rfay wants to merge 12 commits into
rfay wants to merge 12 commits into
Conversation
|
rfay
force-pushed
the
20260702_delete_old_workspaces
branch
from
July 2, 2026 23:32
07d2363 to
cecce9d
Compare
rfay
added a commit
that referenced
this pull request
Jul 17, 2026
…tate incrementally Code review of #175 found several defects in the janitor: - `coder delete <name>` targets the *calling user's* workspace, and the workspace-janitor account owns nothing — so every deletion of another user's workspace would fail, its state entry would be dropped anyway, and the owner would get a fresh notice (and new grace period) every run, forever. Delete `<owner>/<name>` instead, and keep the state entry when deletion fails so it's retried next run. - A single failed Mailgun send (`curl -sf` under `set -e`) aborted the whole run before the state file was written, forgetting every notice already sent that run and re-emailing those owners the next day. The send now warns and returns nonzero; the caller skips recording state for just that workspace. - State was written once at end-of-run; it's now persisted after every transition (prune/notify/revive/delete) so a mid-run crash can't forget who was already notified. Emails are the one side effect that can't be taken back. - REVIVE checked `age_days < NOTIFY_DAYS` instead of "last_used_at advanced past the notice" as the header documented. If the timer was down for 7+ days while an owner briefly revived a workspace, it could still be deleted. Compare last_used_at to notified_at directly. - jq user lookups now use `// empty` so a JSON null email/name can't leak the literal string "null" into an email. - `--help` used a hardcoded `sed -n '2,45p'` range that had gone stale; print the whole comment header with awk instead. Verified with stubbed coder/curl binaries covering all transitions: notify, notify-with-email-failure, revive, revive-after-notice-while- still-idle-past-threshold, delete, delete-failure-retains-state, prune, pending, and owner exclusion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rfay
marked this pull request as ready for review
July 17, 2026 19:00
Member
Author
|
I used the script manually to clean up coder.ddev.com and staging-coder today; it worked fine. |
Stopped workspaces keep their multi-GB dind-cache Docker volume until deleted, and with no lifecycle policy they've accumulated to ~405GB across 105 volumes on a 512GB disk. Adds a daily janitor that emails owners at 7 days idle and deletes at 7 more days idle unless the workspace is used again, plus the GitHub Actions workflow to run it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PROD_CODER_SESSION_TOKEN needs the owner role (this deployment has no Premium license, so there's no narrower built-in role that can list every user's workspaces and delete ones it doesn't own). Document the dedicated-service-account provisioning steps in the operations guide and cross-reference them from the workflow header.
…imer instead of GitHub Actions Avoids minting an owner-role Coder token that has to leave production and live in GitHub Secrets/1Password-for-CI. The janitor now runs directly on each Coder server (matching the existing coder-discord-relay pattern), with state kept in a local file instead of being committed back to git.
…istent coder@ddev.com
…tate incrementally Code review of #175 found several defects in the janitor: - `coder delete <name>` targets the *calling user's* workspace, and the workspace-janitor account owns nothing — so every deletion of another user's workspace would fail, its state entry would be dropped anyway, and the owner would get a fresh notice (and new grace period) every run, forever. Delete `<owner>/<name>` instead, and keep the state entry when deletion fails so it's retried next run. - A single failed Mailgun send (`curl -sf` under `set -e`) aborted the whole run before the state file was written, forgetting every notice already sent that run and re-emailing those owners the next day. The send now warns and returns nonzero; the caller skips recording state for just that workspace. - State was written once at end-of-run; it's now persisted after every transition (prune/notify/revive/delete) so a mid-run crash can't forget who was already notified. Emails are the one side effect that can't be taken back. - REVIVE checked `age_days < NOTIFY_DAYS` instead of "last_used_at advanced past the notice" as the header documented. If the timer was down for 7+ days while an owner briefly revived a workspace, it could still be deleted. Compare last_used_at to notified_at directly. - jq user lookups now use `// empty` so a JSON null email/name can't leak the literal string "null" into an email. - `--help` used a hardcoded `sed -n '2,45p'` range that had gone stale; print the whole comment header with awk instead. Verified with stubbed coder/curl binaries covering all transitions: notify, notify-with-email-failure, revive, revive-after-notice-while- still-idle-past-threshold, delete, delete-failure-retains-state, prune, pending, and owner exclusion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owners of the current idle fleet were already notified manually a week
ago, so the built-in notify -> 7-day grace -> delete state machine would
make everyone wait another two weeks before any disk is reclaimed. Add a
one-off purge mode that skips the ceremony:
./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 # preview
./scripts/workspace-lifecycle-cleanup.sh --purge-idle-days=14 --force # delete
Purge mode deletes every workspace idle >= N days (EXCLUDE_OWNERS still
applies), sends no email, requires no Mailgun credentials, and neither
reads nor writes the state file — the next normal timer run prunes any
state entries for workspaces purged here. Like the rest of the script it
is a dry run unless --force is given, and it shares the existing
zero-workspaces auth-failure guard. Failed deletions are counted
separately in the summary rather than silently inflating "deleted".
Documented in the operations guide under "Automated Idle Workspace
Cleanup".
Verified with stubbed coder/curl binaries: dry run deletes nothing and
needs no Mailgun vars, force deletes exactly the >=N-day set as
<owner>/<name>, threshold boundary (13d idle vs 14d cutoff) kept,
excluded owners skipped, delete failure counted as failed=1, state file
untouched, non-numeric and zero day counts rejected. The existing
state-machine stub tests still pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The whole script failed to even parse under macOS's default /bin/bash
(3.2.57, the last GPLv2 release Apple ships): `send_notice_email` built
its message body with `body=$(cat <<EOF ... EOF)`. bash 3.2 has a real
parser bug where a heredoc nested inside a `$(...)` command substitution
breaks whenever the heredoc body contains an odd number of apostrophes,
failing with "unexpected EOF while looking for matching `''" for the
entire file, not just that function. The email body's apostrophes
("hasn't", "can't", "that's", ...) summed to 7 - odd - which triggered it
on every invocation, dry run or not.
Confirmed by reproducing the bug in isolation, counting apostrophes in
the real heredoc, and testing the fix against bash 3.2.57 and bash 5.3
side by side. Switched to `read -r -d '' body <<EOF || true`, which puts
no command substitution around the heredoc, so the bug can't trigger.
`read -d ''` always returns non-zero at end-of-input, hence `|| true`
under `set -e`. Verified the rendered email body is byte-identical
between bash 3.2 and 5.3, and the existing stub-binary test suite passes
under both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`coder delete --yes` streams its own build-log to the terminal/journal on
every call. That's fine for one-off interactive use but drowns the daily
timer's journal in noise for every successful deletion. Capture its
combined stdout/stderr into a variable instead and only print it
(indented, on stderr) when the delete actually fails — the case an
operator actually needs to see.
The end-of-run summary was just counts (notified=1 deleted=2 ...), with
no record of which workspaces were affected or why short of scrolling
back through the day's live NOTIFY/DELETE/REVIVE lines. Every workspace
that's notified, deleted, revived, pending, pruned, or failed is now also
appended to a category array, and the summary line is followed by a
"Deleted:", "Notified:", etc. breakdown naming each workspace and the
reason — matching text for the live per-item lines, but grouped once at
the end for a quick post-run read. Same treatment in --purge-idle-days
mode ("Deleted" / "Failed to delete").
This surfaced a latent miscount along the way: a failed `coder delete`
was still incremented into `deleted` (the original code bumped the
counter before knowing whether the delete succeeded, and never backed it
out on failure in the normal notify/delete path, only in purge mode).
Failures are now counted separately in a new `failed=` field and listed
under "Failed (will retry next run)" instead of masquerading as
successful deletes; the "no email on file for owner" case also now
increments `failed` instead of vanishing silently.
Verified with the existing stub-binary test suite (bash 3.2 and 5.3):
updated the expected summary line for the corrected deleted/failed
split, and added checks that each detail section is present and correct,
and that a stubbed successful delete's build-log noise never reaches
stdout while a failed delete's output is still surfaced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rfay
force-pushed
the
20260702_delete_old_workspaces
branch
from
July 17, 2026 19:13
75cfff7 to
b00d79d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/datais at 82% (394GiB/512GiB, 92GiB free); ~405GB across 105*-dind-cacheDocker volumes is the actual disk hog — stopping a workspace doesn't free its volume, onlycoder deletedoes.last_used_at, so there's no lifecycle policy in practice today.scripts/workspace-lifecycle-cleanup.sh: emails the owner (via Mailgun) at 7 days idle, deletes the workspace 7 more days later if still untouched, and clears the pending deletion if the owner comes back. State (who's been notified, when) is tracked in a local JSON file on the server.scripts/workspace-lifecycle-cleanup.service+.timer, matching the existingcoder-discord-relaypattern. staging-coder.ddev.com doesn't get this: its workspaces are almost entirely owned byci-bot(already excluded), it isn't sending real notices to anyone, and a dry-run against production itself (./scripts/workspace-lifecycle-cleanup.sh, no--force) already covers testing the policy without deploying it twice.docs/admin/operations-guide.md, new "Step 14: Set Up Workspace Lifecycle Cleanup" indocs/admin/server-setup.mdwith the prod install steps, entry added toCLAUDE.md.Why on-host, not CI
The script has to list every user's workspaces and delete workspaces it doesn't own, which on this (non-Premium) deployment requires the
ownerrole — there's no narrower built-in role that covers both. An earlier version of this PR ran the job from GitHub Actions, which meant putting thatowner-role token in GitHub Secrets/1Password-for-CI so it could reach production over the network — a real blast-radius increase (a leaked Actions secret would mean full site-admin on prod).Running it as a systemd timer on the Coder server itself avoids that: the token stays on the box (in
/etc/workspace-lifecycle-cleanup.env, mode 600), never travels through GitHub, and state lives in a local file instead of being committed back to git on every run.coderpicks upCODER_URL/CODER_SESSION_TOKENstraight from the environment, so nocoder loginsession is needed either.Before this can actually send/delete anything
workspace-janitorservice account + owner-role token on coder.ddev.com (seedocs/admin/server-setup.md→ Step 14) and put it in/etc/workspace-lifecycle-cleanup.envcoder.ddev.comsending domain is already verified; API key confirmed working via a real test send (1Password shared "DDEV" vault → "Mailgun" item →coder.ddev.com sending api keyfield)MAILGUN_FROMfixed tosupport@ddev.com(the original default,coder@ddev.com, doesn't exist)send_notice_email()in the script before it goes out to ~90 peopleworkspace-lifecycle-cleanup.timeron coder.ddev.com per Step 14Test plan
bash -nsyntax check andshellcheckclean on the scriptcoder/curlbinariescoder.ddev.comdomainsudo -u rfay ... workspace-lifecycle-cleanup, no--force) on coder.ddev.com once the env file is populated--forcerun via the timer (will only send notices — nothing can delete until 7 days later since state starts empty)🤖 Generated with Claude Code