diff --git a/CHANGELOG.md b/CHANGELOG.md index 98c368c..0bffa7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,47 @@ range. Versions before 0.3.0 predate the GitHub releases and are reconstructed f Merge commits, automated `Tagged ` commits and version bumps are omitted, as are bullets that only restate the release note above them. +## Unreleased + +### Added + +- **`roll shutdown` takes down everything for the end of the day.** It runs `roll env down` in + every environment that has a running container, found through the Compose project labels on the + containers, and then `roll svc down`. Volumes are kept. An environment whose `.env.roll` no longer + carries the name it was started under is left running and reported, because `roll env down` in + that directory would target a different Compose project, and so is one whose project directory or + `.env.roll` has gone (recognised by its `dev.roll.environment.name` network label); the command + then exits non-zero, as it does when any step fails. +- **Superseded images are cleaned up, by hand and automatically.** A pull that finds a newer build + of a tag leaves the previous image behind untagged, and nothing ever removed it, so hosts + pulling the rebuilt images filled up. `roll image-cleanup` removes those superseded images and + every tagged image no container uses; an environment that is only down has no containers, so it + downloads its images again on its next `up`. The superseded half also runs automatically after + `roll svc pull|up` and `roll env pull|up` (`ROLL_IMAGE_AUTO_CLEANUP=0` turns that off); unused + images are never removed automatically, so a `roll shutdown` does not cause re-downloads and + parallel builds cannot delete each other's freshly pulled images. A tag a + container was created from counts as used even when the tag has since moved to a newer image, so + a freshly pulled image is not deleted before its environment's next `up`. Only images roll runs + are touched — `ROLL_IMAGE_REPOSITORY` plus the third-party images named in roll's compose files, + ignoring any name in a user override that is only an interpolation — and removal never forces, so + Docker keeps any image a container still uses. On Docker's containerd image store a superseded + image keeps no name, so roll records its image IDs in `~/.roll/tmp/roll-image-ids` before every + pull and recognises superseded images from that ledger on either store. + +### Fixed + +- **`roll db import < dump.sql` executes the dump again.** Since 0.8.0 it exited 0 having run + nothing. The MariaDB 11 client probe (`resolveDbBinary` in `commands/db.cmd`) ran + `roll env exec -T db sh -c 'command -v …'` with the caller's stdin, and `exec -T` forwards stdin + into the container, so the probe drained the dump and the real client read an empty stream. The + exit code gave no sign of it; only the missing rows did. The probe now reads from `/dev/null`. + The same drain hit `roll db connect < file.sql` and `echo … | roll db connect`, which are fixed + by the same change. +- **`roll db connect` no longer requests a TTY when stdin is not one.** It passes `-T` to + `roll env exec` unless stdin is a terminal, so piped and scripted use no longer depends on + `docker compose exec` detecting the missing terminal on its own. An interactive session still + gets a TTY. + ## [0.8.4](https://github.com/epartment/rolldev/releases/tag/0.8.4) — 2026-09-17 ### Added diff --git a/FEATURE-REQUESTS.md b/FEATURE-REQUESTS.md index b213b1d..a47f8cb 100644 --- a/FEATURE-REQUESTS.md +++ b/FEATURE-REQUESTS.md @@ -79,9 +79,9 @@ None outstanding. **L4. `roll config schema` silently omits registered keys** — `commands/config.cmd:93-165` - *What:* The schema display is seven hand-maintained category filters with no catch-all, so any - registered key matching none of them is invisible. `ROLL_ADMIN_AUTOLOGIN` and - `ROLL_MAGENTO_STATIC_CACHING` are two current examples; both are in `initConfigSchema` and neither - appears in the output. Every key added from now on has to be wired into a filter by hand or it + registered key matching none of them is invisible. `ROLL_ADMIN_AUTOLOGIN`, + `ROLL_MAGENTO_STATIC_CACHING`, `ROLL_ADVISORIES` and `ROLL_IMAGE_AUTO_CLEANUP` are current + examples; all are in `initConfigSchema` and none appears in the output. Every key added from now on has to be wired into a filter by hand or it vanishes from the only listing users have. - *Why it matters:* `config schema` is the discovery surface for configuration, and an omitted key reads as "not supported". It also makes the listing quietly wrong rather than visibly incomplete. @@ -89,6 +89,21 @@ None outstanding. column alongside `ROLL_CONFIG_SCHEMA_VALUES`) and render groups from that, so registering a key is the only step needed. Failing that, add a final "Other" group printing whatever matched nothing. +**L5. `roll env up` exits 1 on an environment without `php-fpm` whenever an SSH agent is running** — `commands/env.cmd:245-248` + +- *What:* After a successful `up`/`start`, `env.cmd` runs `roll root chown` and `roll root chmod` on + `/run/host-services/ssh-auth.sock` whenever `SSH_AUTH_SOCK` is set. Both exec into the shell + container (`php-fpm` by default), so on an environment that has none — the `local` type with its + own `.roll/roll-env.yml`, or any project with `ROLL_ENV_SHELL_CONTAINER` pointing at a missing + service — the command ends with `service "php-fpm" is not running` and exit status 1 although + every container started. **Verified** on macOS with a `local` environment running only redis; + the same code is in 0.8.4. +- *Why it matters:* A script, CI job or the build server cannot tell a + failed `up` from a successful one, and a developer sees an error for a step that does not apply. +- *Suggested fix:* Run the two socket commands only when the shell container is part of this + environment (for example `roll env ps -q "${ROLL_ENV_SHELL_CONTAINER}"` returns an id), and do not + let their failure decide the exit status of `up`. + ## Checked, and already covered Recorded so the next reader does not re-propose them: diff --git a/README.md b/README.md index 3e0a6c6..63305d4 100644 --- a/README.md +++ b/README.md @@ -286,7 +286,7 @@ responsibility is still describable. | `bin/roll` | 98 lines | Entry point, dispatch, argument parsing | **Coherent** | | `utils/core.sh` | 184 lines | Messaging helpers, array/version utilities, network peering | **Coherent** — the three box-drawing functions are one-line wrappers around a shared `box` helper | | `utils/config.sh` | 946 lines | Config schema, loading, validation, post-processing, `.env.roll` writes | **Coherent**; sole owner of configuration defaults | -| `utils/images.sh` | 375 lines | Service catalog (version key, toggle and image per service) and image-tag discovery from the registry | **Coherent** | +| `utils/images.sh` | 655 lines | Service catalog (version key, toggle and image per service), image-tag discovery from the registry, and cleanup of the local images roll runs | **Coherent** | | `utils/registry.sh` | 461 lines | Command discovery and priority resolution | **Oversized for what it delivers** — ~200 lines serve `roll registry`'s reporting subcommands; the metadata layer they report on is a stub (**M6**) | | `utils/env.sh` | 108 lines | Env path location, partial precedence, env-type validation | **Coherent** | | `utils/install.sh` | 63 lines | Host install assertion, SSH config | **Coherent** | @@ -425,6 +425,20 @@ roll env down roll restart ``` +Take down every running environment and then the shared services (end of the working day): + +```bash +roll shutdown +``` + +Remove superseded roll images (the untagged previous builds a pull leaves behind) and tagged roll +images no container uses. The superseded half also runs automatically after `svc pull|up` and +`env pull|up` unless `ROLL_IMAGE_AUTO_CLEANUP=0`: + +```bash +roll image-cleanup +``` + Open a shell in the `php-fpm` container as `www-data`: ```bash @@ -529,6 +543,7 @@ exists and where it lives. | `roll config check-pins` / `fix-pins` | `commands/config.cmd` | | `roll config versions [service]` — pinned versions, or a service's published versions one per line on stdout | `commands/config.cmd`, `utils/images.sh` | | `roll config version ` — set a pin without prompting | `commands/config.cmd` | +| `roll image-cleanup [--dry-run]` — remove (or list) superseded and unused images, never prompts | `commands/image-cleanup.cmd`, `utils/images.sh` | | Flag-first prompts | `utils/interact.sh` | Two rules bind anything added here: @@ -678,6 +693,7 @@ line-audited, and neither backup nor restore was executed against a live environ | A command run right after `roll env up` fails to connect to the DB or search engine | No fragment declares a `healthcheck:` and `up` does not pass `--wait`, so it returns when containers *start* | Retry with a wait loop against the service. See `FEATURE-REQUESTS.md` H1 | | `Mutagen sync sessions are not used on "linux" host environments` | `roll sync` is macOS-only | Expected; Linux and WSL bind-mount directly | | `Mutagen configuration does not exist for environment type "…"` | Only `magento1`, `magento2` and `shopware` ship a `.mutagen.yml` | Expected; that type uses a bind mount on macOS too | +| `roll db import < dump.sql` exits 0 but no rows change (RollDev 0.8.0–0.8.4) | The client-binary probe in `commands/db.cmd` inherited stdin and drained the dump before the real client ran | Upgrade. On an affected version pipe straight into `roll env exec -T db sh -c 'mariadb -u"$MYSQL_USER" -p"$MYSQL_PASSWORD" "$MYSQL_DATABASE"'` (`mysql` on MySQL images) and check row counts, not the exit code | | In-container `composer install` reports a missing `.git` in a `vendor/` package | Mutagen's `ignore.vcs: true` strips `.git` at every depth, so source installs have no checkout | Remove the affected `vendor/` directories and re-run `composer install` to force dist installs | | Files written to `var/` or `pub/static/` on the host are invisible in the container | Both paths are excluded from the Mutagen session | Write through the container (`roll cli`), or `docker cp` into the volume | | A `roll` subcommand exits 1 printing only its opening `INFO:` lines, on Linux but not macOS | `set -e` plus a statement returning non-zero — classically a bare `((x++))` from 0 — kills the sourced command, and the `ERR` trap is not inherited by functions | Use `x=$((x + 1))`; see [Dispatch](#dispatch) | diff --git a/bin/roll b/bin/roll index 3fd7804..32d381a 100755 --- a/bin/roll +++ b/bin/roll @@ -43,7 +43,7 @@ declare ROLL_PARAMS=() declare ROLL_CMD_VERB= declare ROLL_CMD_EXEC= declare ROLL_CMD_HELP= -declare ROLL_CMD_ANYARGS=(svc env db redis sync shell debug rootnotty rootshell clinotty root node npm cli copyfromcontainer copytocontainer composer grunt magento magerun mydumper myloader backup restore restore-full duplicate magento2-init epartment-init pull tapsync status describe registry doctor) +declare ROLL_CMD_ANYARGS=(svc env db redis sync shell debug rootnotty rootshell clinotty root node npm cli copyfromcontainer copytocontainer composer grunt magento magerun mydumper myloader backup restore restore-full duplicate magento2-init epartment-init pull tapsync status describe registry doctor image-cleanup) ## parse first argument as command and determine validity if (( "$#" )); then diff --git a/commands/db.cmd b/commands/db.cmd index cee54b3..39644c8 100644 --- a/commands/db.cmd +++ b/commands/db.cmd @@ -33,10 +33,12 @@ eval "$( ## MariaDB 11 images dropped the mysql*/mysqldump compatibility symlinks and only ship ## mariadb/mariadb-dump; probe once per invocation and use whichever pair is present, so both ## MariaDB 11 and MySQL/MariaDB 10.x images keep working. Sets RESOLVED_DB_BIN on success. +## The probe must not inherit stdin: `exec -T` forwards it into the container, so without the +## /dev/null)" || true + bin="$("${ROLL_DIR}/bin/roll" env exec -T db sh -c "command -v ${preferred} || command -v ${fallback}" /dev/null)" || true if [[ ! ${bin} ]]; then local image image="$(docker container inspect "${DB_CONTAINER}" --format '{{.Config.Image}}' 2>/dev/null)" || true @@ -50,7 +52,11 @@ resolveDbBinary() { case "${ROLL_PARAMS[0]}" in connect) resolveDbBinary mariadb mysql - "${ROLL_DIR}/bin/roll" env exec db \ + ## Only ask for a TTY when stdin is one, so `roll db connect < file.sql` and scripted + ## `-e` calls do not depend on docker compose noticing the missing terminal itself. + CONNECT_EXEC_FLAGS=() + [[ -t 0 ]] || CONNECT_EXEC_FLAGS=(-T) + "${ROLL_DIR}/bin/roll" env exec "${CONNECT_EXEC_FLAGS[@]}" db \ "${RESOLVED_DB_BIN}" -u"${MYSQL_USER}" -p"${MYSQL_PASSWORD}" --database="${MYSQL_DATABASE}" "${ROLL_PARAMS[@]:1}" "$@" ;; import) diff --git a/commands/env.cmd b/commands/env.cmd index e28e72d..fc48c71 100644 --- a/commands/env.cmd +++ b/commands/env.cmd @@ -148,6 +148,12 @@ if [[ "${ROLL_PARAMS[0]}" == "sh" ]]; then ROLL_PARAMS=("exec" "-T" "${ROLL_PARAMS[1]}" "sh" "-c" "${ROLL_PARAMS[2]}") fi +## put the images this pull may supersede on record while they still carry their name; the +## containerd image store keeps no name on a superseded image (see utils/images.sh) +if [[ "${ROLL_PARAMS[0]}" == "pull" || "${ROLL_PARAMS[0]}" == "up" ]]; then + recordRollImageIds +fi + ## disconnect peered service containers from environment network if [[ "${ROLL_PARAMS[0]}" == "down" ]]; then disconnectPeeredServices "$(renderEnvNetworkName)" @@ -204,6 +210,11 @@ docker compose \ --env-file "${ROLL_ENV_PATH}/.env.roll" --project-directory "${ROLL_ENV_PATH}" -p "${ROLL_ENV_NAME}" \ "${DOCKER_COMPOSE_ARGS[@]}" "${ROLL_PARAMS[@]}" "$@" +## remove the images this pull (or the up after it) left superseded; see utils/images.sh +if [[ "${ROLL_PARAMS[0]}" == "pull" || "${ROLL_PARAMS[0]}" == "up" ]]; then + autoCleanupRollImages +fi + if [[ ("${ROLL_PARAMS[0]}" == "up" || "${ROLL_PARAMS[0]}" == "start") && -n "${ROLL_EXTRA_PHP_EXT}" ]]; then info "Adding additional PHP extension, This may take a while... (output hidden)" roll add-php-ext "${ROLL_EXTRA_PHP_EXT}" > /dev/null 2>&1 diff --git a/commands/image-cleanup.cmd b/commands/image-cleanup.cmd new file mode 100644 index 0000000..47d0712 --- /dev/null +++ b/commands/image-cleanup.cmd @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +[[ ! ${ROLL_DIR} ]] && >&2 echo -e "\033[31mThis script is not intended to be run directly!\033[0m" && exit 1 + +## `image-cleanup` is on roll's ROLL_CMD_ANYARGS list so its flags reach this script, which means +## roll's own parser stops at the first dash-prefixed argument and leaves ROLL_PARAMS empty. +## Parse flags from "$@", and render help by sourcing usage.cmd - re-invoking +## `roll image-cleanup --help` would land right back here and fork until killed. +CLEANUP_DRY_RUN=0 + +if (( ${#ROLL_PARAMS[@]} > 0 )); then + fatal "Unsupported argument ${ROLL_PARAMS[0]}" +fi + +while (( "$#" )); do + case "$1" in + -h|--help) + source "${ROLL_DIR}/commands/usage.cmd" + ;; + --dry-run) + CLEANUP_DRY_RUN=1 + ;; + *) + fatal "Unsupported argument $1" + ;; + esac + shift +done + +assertDockerRunning + +## ROLL_IMAGE_REPOSITORY decides which images are roll's; this command is host-wide, so only the +## global configuration applies, loaded in the same order loadRollConfig uses +initConfigSchema +for globalConfig in "${ROLL_HOME_DIR}/.env.roll" "${ROLL_HOME_DIR}/.env"; do + if [[ -f "${globalConfig}" ]]; then + loadConfigFromFile "${globalConfig}" "false" "global" + fi +done +ROLL_IMAGE_REPOSITORY="$(getConfig ROLL_IMAGE_REPOSITORY "ghcr.io/epartment/roll")" + +recordRollImageIds + +cleanupReferences=() +cleanupListing=() +supersededCount=0 +unusedCount=0 + +while IFS=$'\t' read -r reference name size created; do + cleanupReferences+=("${reference}") + printf -v listingLine ' %-10s %-60s %-10s %s' "superseded" "${name}" "${size}" "${created}" + cleanupListing+=("${listingLine}") + supersededCount=$((supersededCount + 1)) +done < <(listSupersededRollImages) + +while IFS=$'\t' read -r reference name size created; do + cleanupReferences+=("${reference}") + printf -v listingLine ' %-10s %-60s %-10s %s' "unused" "${name}" "${size}" "${created}" + cleanupListing+=("${listingLine}") + unusedCount=$((unusedCount + 1)) +done < <(listUnusedRollImages) + +if (( ${#cleanupReferences[@]} == 0 )); then + info "No superseded or unused roll images found." + exit 0 +fi + +printf ' %-10s %-60s %-10s %s\n' "KIND" "IMAGE" "SIZE" "CREATED" +printf '%s\n' "${cleanupListing[@]}" +echo "" +info "Sizes include layers images share with each other, so the space freed can be less than their sum." + +if [[ ${CLEANUP_DRY_RUN} -eq 1 ]]; then + info "Dry run: ${supersededCount} superseded and ${unusedCount} unused image(s) would be removed." + exit 0 +fi + +removeImages "${cleanupReferences[@]}" + +success "Removed ${ROLL_IMAGES_REMOVED} of ${#cleanupReferences[@]} image(s). An environment that needs one of them again downloads it on its next 'roll env up'." + +if (( ROLL_IMAGES_GONE > 0 )); then + info "${ROLL_IMAGES_GONE} image(s) were already gone, removed by another roll run in the meantime." +fi + +if (( ${#ROLL_IMAGES_KEPT[@]} > 0 )); then + warning "Docker kept ${#ROLL_IMAGES_KEPT[@]} image(s), usually because a container still uses them:" + for kept in "${ROLL_IMAGES_KEPT[@]}"; do + >&2 printf ' %s\n' "${kept}" + done +fi diff --git a/commands/image-cleanup.help b/commands/image-cleanup.help new file mode 100644 index 0000000..b4c53bf --- /dev/null +++ b/commands/image-cleanup.help @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +[[ ! ${ROLL_DIR} ]] && >&2 echo -e "\033[31mThis script is not intended to be run directly!\033[0m" && exit 1 +## @description: Remove Docker images roll downloaded and no longer needs. +## @category: utility + +ROLL_USAGE=$(cat <<'USAGE' +\033[33mUsage:\033[0m + image-cleanup Remove superseded and unused roll images + image-cleanup --dry-run List what would be removed and change nothing + +\033[33mOptions:\033[0m + --dry-run Only list the images + -h, --help Display this help menu + +\033[33mDescription:\033[0m + Removes two kinds of image, without asking: + + superseded the previous build of a tag, left behind without a tag when a pull found a + newer one. Nothing can use it again. + unused a tagged image no container uses, running or stopped, such as an old PHP + version a project no longer pins. An environment taken down with roll env down + has no containers left, so its images are removed as well and downloaded again + on its next roll env up. + + A tag a container was created from counts as used even after a pull moved it to a newer + build, so an environment that is up or stopped keeps its images. + + Superseded images are also removed automatically after roll svc pull, roll svc up, + roll env pull and roll env up; unused ones only by this command. Set + ROLL_IMAGE_AUTO_CLEANUP=0 in ~/.roll/.env to turn the automatic run off. + + Only images roll itself runs are touched: everything under ROLL_IMAGE_REPOSITORY plus the + third-party images named in the roll compose files (traefik, mailpit, selenium and others). + Docker refuses to remove an image a container still uses, and this command never forces it. + + The containerd image store of newer Docker Desktop installs keeps no name on a superseded + image, so roll records the IDs of its images in ~/.roll/tmp/roll-image-ids before every pull + and recognises them from there. An image superseded before roll recorded it is left alone. +USAGE +) diff --git a/commands/shutdown.cmd b/commands/shutdown.cmd new file mode 100644 index 0000000..3edb623 --- /dev/null +++ b/commands/shutdown.cmd @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +[[ ! ${ROLL_DIR} ]] && >&2 echo -e "\033[31mThis script is not intended to be run directly!\033[0m" && exit 1 + +if (( ${#ROLL_PARAMS[@]} > 0 )); then + fatal "Unsupported argument ${ROLL_PARAMS[0]}" +fi + +assertDockerRunning + +## allow return codes from sub-process to bubble up normally +trap '' ERR + +## Every compose project with a running container, as "\t". Compose stamps +## both labels on each container it creates, and env.cmd passes the project directory explicitly, +## so the working dir is the directory `roll env down` has to run in. +function listRunningComposeProjects() { + docker container ls \ + --filter "label=com.docker.compose.project" \ + --format '{{.Label "com.docker.compose.project"}}{{"\t"}}{{.Label "com.docker.compose.project.working_dir"}}' \ + | sort -u + + return 0 +} + +## The names of every environment roll created a network for, one per line. networks.base.yml labels +## each environment network with its name, which is how a roll environment is recognised after its +## project directory or .env.roll has gone. +function listRollNetworkEnvNames() { + docker network ls \ + --filter "label=dev.roll.environment.name" \ + --format '{{.Label "dev.roll.environment.name"}}' + + return 0 +} + +## The environment name a project's .env.roll declares, or nothing +function envNameInDir() { + local dir="$1" + + grep -m1 '^ROLL_ENV_NAME=' "${dir}/.env.roll" 2>/dev/null | cut -d '=' -f2- | tr -d '\r"'\' + + return 0 +} + +envNames=() +envDirs=() +skipped=0 +rollNetworkEnvNames="$(listRollNetworkEnvNames)" + +while IFS=$'\t' read -r project dir; do + [[ -z "${project}" || "${project}" == "roll" ]] && continue + + if [[ -z "${dir}" || ! -f "${dir}/.env.roll" ]]; then + ## not a roll environment: a compose project roll did not start + [[ $'\n'"${rollNetworkEnvNames}"$'\n' == *$'\n'"${project}"$'\n'* ]] || continue + + warning "Skipping ${project}: ${dir:-its project directory}/.env.roll no longer exists, so roll env down cannot be run for it. Stop it with: docker stop \$(docker ps -q --filter label=com.docker.compose.project=${project})" + skipped=$((skipped + 1)) + continue + fi + + if [[ "$(envNameInDir "${dir}")" != "${project}" ]]; then + warning "Skipping ${project}: ${dir}/.env.roll no longer names it, so roll env down there would target another environment. Stop it with: docker stop \$(docker ps -q --filter label=com.docker.compose.project=${project})" + skipped=$((skipped + 1)) + continue + fi + + envNames+=("${project}") + envDirs+=("${dir}") +done < <(listRunningComposeProjects) + +if (( ${#envNames[@]} == 0 && skipped == 0 )); then + info "No running environments found." +fi + +failed=() +i=0 +while (( i < ${#envNames[@]} )); do + info "Shutting down ${envNames[$i]} (${envDirs[$i]})" + if ! (cd "${envDirs[$i]}" && "${ROLL_DIR}/bin/roll" env down); then + error "roll env down failed for ${envNames[$i]}" + failed+=("${envNames[$i]}") + fi + i=$((i + 1)) +done + +coreServicesDown=0 +if [[ -n "$(docker container ls -aq --filter "label=com.docker.compose.project=roll")" ]]; then + info "Shutting down the RollDev core services" + if "${ROLL_DIR}/bin/roll" svc down; then + coreServicesDown=1 + else + error "roll svc down failed" + failed+=("core services") + fi +else + info "RollDev core services are not running." +fi + +if (( ${#failed[@]} > 0 )); then + fatal "Shutdown incomplete, failed: ${failed[*]}" +fi + +if (( skipped > 0 )); then + fatal "Shut down ${#envNames[@]} environment(s); ${skipped} left running, see above." +fi + +if [[ ${coreServicesDown} -eq 1 ]]; then + success "Shut down ${#envNames[@]} environment(s) and the RollDev core services." +else + success "Shut down ${#envNames[@]} environment(s)." +fi diff --git a/commands/shutdown.help b/commands/shutdown.help new file mode 100644 index 0000000..4ad9487 --- /dev/null +++ b/commands/shutdown.help @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +[[ ! ${ROLL_DIR} ]] && >&2 echo -e "\033[31mThis script is not intended to be run directly!\033[0m" && exit 1 +## @description: Take down every running environment and the RollDev core services. +## @category: environment + +ROLL_USAGE=$(cat <<'USAGE' +\033[33mUsage:\033[0m + shutdown + +\033[33mOptions:\033[0m + -h, --help Display this help menu + +\033[33mDescription:\033[0m + Runs roll env down in every environment that has a running container, then roll svc down + for the core services (traefik, dnsmasq, mailhog, tunnel, portainer, startpage). Meant for + the end of the working day with several environments up. + + Like roll env down it removes containers and networks but keeps volumes, so databases and + search indexes survive and roll env up brings each environment back as it was. On macOS the + Mutagen sync of each environment is terminated as well. + + An environment whose .env.roll no longer carries the name it was started under is left + running and reported, because roll env down in that directory would target another name. + So is one whose project directory or .env.roll has been moved or deleted. + The command exits non-zero when any environment is left running or a step failed. +USAGE +) diff --git a/commands/svc.cmd b/commands/svc.cmd index ff579f5..53c3f44 100644 --- a/commands/svc.cmd +++ b/commands/svc.cmd @@ -82,6 +82,12 @@ if [[ "${ROLL_PARAMS[0]}" == "up" ]]; then fi fi +## put the images this pull may supersede on record while they still carry their name; the +## containerd image store keeps no name on a superseded image (see utils/images.sh) +if [[ "${ROLL_PARAMS[0]}" == "pull" || "${ROLL_PARAMS[0]}" == "up" ]]; then + recordRollImageIds +fi + ROLL_VERSION=$(cat ${ROLL_DIR}/version) ## pass orchestration through to docker-compose @@ -89,6 +95,11 @@ ROLL_VERSION=${ROLL_VERSION:-"in-dev"} docker compose \ --project-directory "${ROLL_HOME_DIR}" -p roll \ "${DOCKER_COMPOSE_ARGS[@]}" "${ROLL_PARAMS[@]}" "$@" +## remove the images this pull (or the up after it) left superseded; see utils/images.sh +if [[ "${ROLL_PARAMS[0]}" == "pull" || "${ROLL_PARAMS[0]}" == "up" ]]; then + autoCleanupRollImages +fi + ## connect peered service containers to environment networks when 'svc up' is run if [[ "${ROLL_PARAMS[0]}" == "up" ]]; then for network in $(docker network ls -f label=dev.roll.environment.name --format {{.Name}}); do diff --git a/commands/usage.help b/commands/usage.help index b8a713e..936f165 100755 --- a/commands/usage.help +++ b/commands/usage.help @@ -40,6 +40,8 @@ RollDev version $(cat ${ROLL_DIR}/version) install Initializes or updates roll configuration on host machine shell Launches into a shell within the current project environment status Display list of all running RollDev project environments + shutdown Take down every running environment and the core services (end of day) + image-cleanup Remove superseded and unused roll images to free disk space describe Display detailed information about the current environment doctor Diagnose whether the current project environment is fit to run debug Launches debug enabled shell within current project environment diff --git a/docs/configuration/unattended-operation.md b/docs/configuration/unattended-operation.md index b172b7e..963eadf 100644 --- a/docs/configuration/unattended-operation.md +++ b/docs/configuration/unattended-operation.md @@ -63,3 +63,23 @@ so they keep their voice. Pass it per invocation, as above, or export it for a whole unattended session. It is deliberately awkward to set project-wide: a committed `.env.roll` that hid this advice would hide it from every other developer on that project, which is the opposite of what it is for. + +## Automatic image cleanup + +After `roll svc pull`, `roll svc up`, `roll env pull` and `roll env up`, RollDev removes the roll +images a pull superseded: the previous build of a tag, left behind without a tag once the newer one +arrived. Docker never removes those on its own, so a host that pulls the nightly-rebuilt images +fills its disk with them. Only images roll runs are considered, and Docker refuses to remove an +image a container — running or stopped — still uses, so an environment built in parallel keeps its +images. + +To keep every image, for example while debugging which build a container came from, pass +`ROLL_IMAGE_AUTO_CLEANUP=0` for the invocation or export it for the session: + +```bash +ROLL_IMAGE_AUTO_CLEANUP=0 roll env up +``` + +It can also be set in `~/.roll/.env`. The automatic run never removes a tagged image, so parallel +builds cannot delete each other's freshly pulled images. `roll image-cleanup` does remove tagged +images no container uses, so run it on a build server only when no build is in progress. diff --git a/docs/usage.md b/docs/usage.md index b420839..0e2764c 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -23,6 +23,33 @@ Starting a stopped environment: roll env start +Take down every running environment and the RollDev core services at the end of the day (volumes +are kept, so `roll env up` brings each project back as it was): + + roll shutdown + +Remove the Docker images roll pulled and no longer needs, without asking: + + roll image-cleanup + roll image-cleanup --dry-run + +It removes superseded images — the previous build of a tag, left behind without a tag when a pull +found a newer one — and tagged roll images no container uses, such as an old PHP version a project +no longer pins. An environment taken down with `roll env down` has no containers, so its images are +removed too and downloaded again on its next `roll env up`. Superseded images are also removed +automatically after `roll svc pull`, `roll svc up`, `roll env pull` and `roll env up`; unused ones +only by this command, so a `roll shutdown` in the evening never causes re-downloads the next day. + +Only images roll itself runs are touched — everything under `ROLL_IMAGE_REPOSITORY` plus the +third-party images named in roll's compose files — and an image a container still uses, running or +stopped, is never removed. Set `ROLL_IMAGE_AUTO_CLEANUP=0` in `~/.roll/.env` to turn the automatic +cleanup off. + +On Docker's containerd image store (the default of newer Docker Desktop installs) a superseded image +keeps no name at all, so RollDev records the IDs of its images in `~/.roll/tmp/roll-image-ids` +before every pull and recognises them from there. An image superseded before RollDev first recorded +it carries no trace of where it came from and is left on disk. + Import a database (if you don't have `pv` installed, use `cat` instead): pv /path/to/dump.sql.gz | gunzip -c | roll db import diff --git a/utils/config.sh b/utils/config.sh index c33ffb9..c01b3b2 100644 --- a/utils/config.sh +++ b/utils/config.sh @@ -92,6 +92,9 @@ function initConfigSchema() { ## once a key has one, which would overwrite the value an unattended caller passed in the ## environment. advisory() in utils/core.sh reads it from there. ROLL_CONFIG_SCHEMA_KEYS+=(ROLL_ADVISORIES); ROLL_CONFIG_SCHEMA_VALUES+=("boolean:optional") + ## Optional for the same reason: autoCleanupRollImages() in utils/images.sh falls back to 1 + ## itself, so an exported schema default cannot override ROLL_IMAGE_AUTO_CLEANUP=0 from a caller. + ROLL_CONFIG_SCHEMA_KEYS+=(ROLL_IMAGE_AUTO_CLEANUP); ROLL_CONFIG_SCHEMA_VALUES+=("boolean:optional") ROLL_CONFIG_SCHEMA_KEYS+=(ROLL_SELENIUM); ROLL_CONFIG_SCHEMA_VALUES+=("boolean:0") ROLL_CONFIG_SCHEMA_KEYS+=(ROLL_SELENIUM_DEBUG); ROLL_CONFIG_SCHEMA_VALUES+=("boolean:0") ROLL_CONFIG_SCHEMA_KEYS+=(ROLL_TEST_DB); ROLL_CONFIG_SCHEMA_VALUES+=("boolean:0") diff --git a/utils/images.sh b/utils/images.sh index ff1d043..28b9490 100644 --- a/utils/images.sh +++ b/utils/images.sh @@ -373,3 +373,283 @@ function showServiceVersions() { return 0 } + +## Image cleanup. +## +## Every `docker compose pull` that finds a newer build of a tag moves the tag onto the new image +## and leaves the old one behind as a dangling image, still taking the full size on disk. Nothing in +## Docker ever removes those, so on a developer machine that pulls the rebuilt roll images they +## accumulate without limit. +## +## Two classes are told apart here, because only one of them is always garbage: +## +## superseded dangling images that were a roll image. Nothing can start a container from them any +## more, so they are removed without asking, and automatically after a pull. +## unused tagged roll images no container uses, e.g. php-fpm:8.1 after a project moved to +## 8.2. Removed by `roll image-cleanup` only, never automatically: `roll env down` +## and `roll shutdown` remove a project's containers, so the images of every project +## that is merely down count as unused too, and removing them after every `up` +## would make each environment download its images again after a shutdown. +## +## Whether a dangling image was roll's depends on the image store. The classic store keeps the +## repository name through the image's digest (`ghcr.io/epartment/roll/php-fpm `); the +## containerd store, the default of newer Docker Desktop installs, keeps nothing at all (` +## `, no tags, no digests). So roll also keeps a ledger of the image IDs it has seen carrying +## a roll name (recordRollImageIds), and a dangling image is roll's when either its name matches or +## its ID is in that ledger. An image that was superseded before roll first recorded it stays +## unrecognised on the containerd store, which leaves it on disk rather than guessing. +## +## Removal never forces. `docker image rm` refuses an image a container still uses, running or +## stopped, and that refusal is what keeps an image a stopped environment needs on disk. + +## Repository patterns, as bash globs, of every image roll itself runs: the ${ROLL_IMAGE_REPOSITORY} +## namespace plus the third-party images named literally in roll's compose fragments (traefik, +## mailpit, selenium, ...). Read from the fragments rather than listed here, so a service added +## later is covered without touching this function. An interpolation at the end of a name becomes +## `*`, so selenium/standalone-chrome${ROLL_SELENIUM_DEBUG:-} matches the plain and the -debug image. +## +## A `*` anywhere else is dropped, pattern and all. These patterns decide what gets deleted, and a +## user override in ~/.roll/environments declaring `image: ${IMAGE}` or `${REGISTRY}/name` would +## otherwise become `*` or `*/name` and claim images that have nothing to do with roll. +function rollImageRepositoryPatterns() { + local repository="${ROLL_IMAGE_REPOSITORY:-ghcr.io/epartment/roll}" + local root="" + + echo "${repository}/*" + + for root in "${ROLL_DIR}/docker" "${ROLL_DIR}/environments" "${ROLL_HOME_DIR:-${HOME}/.roll}/environments"; do + [[ -d "${root}" ]] || continue + + find "${root}" -type f -name '*.yml' -exec grep -h '^[[:space:]]*image:' {} + 2>/dev/null \ + | sed -e 's/^[[:space:]]*image:[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/["'\'']//g' \ + | grep -v '^[$][{]ROLL_IMAGE_REPOSITORY[}]' \ + | sed -e 's/[$][{][^}]*[}]/*/g' -e 's/@.*$//' -e 's/:.*$//' \ + | grep -E '^[^*]+$|^[^*]*[^/*][*]$' || true + done | sort -u + + return 0 +} + +## isRollImageRepository ... +function isRollImageRepository() { + local repository="$1" + shift + local pattern="" + + for pattern in "$@"; do + # shellcheck disable=SC2053 # the pattern is a glob on purpose + [[ "${repository}" == ${pattern} ]] && return 0 + done + + return 1 +} + +function rollImageLedgerFile() { + echo "${ROLL_HOME_DIR:-${HOME}/.roll}/tmp/roll-image-ids" + return 0 +} + +## Record the IDs of every image that currently carries a roll name, plus the image each container +## was created from when that container names a roll image, and drop ledger entries for images that +## no longer exist. Run before a pull, so the build a pull is about to supersede is on record while +## it still has its name; the container half covers an image that was superseded while a container +## held it, which the next `up` frees. +## +## The rewrite goes through a temp file and a rename, so a parallel roll run never reads a half +## written ledger; an entry lost to two runs rewriting at once only means that image is not cleaned +## up automatically, never that a wrong one is. +function recordRollImageIds() { + local patterns=() containers=() line="" id="" repository="" image="" config_image="" reference="" + local ledger="" existing="" seen="" + + ledger="$(rollImageLedgerFile)" + + while IFS= read -r line; do + patterns+=("${line}") + done < <(rollImageRepositoryPatterns) + + existing="$(docker image ls -a -q --no-trunc 2>/dev/null)" || return 0 + + while IFS=$'\t' read -r id repository; do + [[ -z "${id}" || "${repository}" == "" ]] && continue + isRollImageRepository "${repository}" "${patterns[@]}" || continue + seen="${seen}${id}"$'\n' + done < <(docker image ls --filter dangling=false --no-trunc --format '{{.ID}}\t{{.Repository}}' 2>/dev/null) + + while IFS= read -r line; do + [[ -n "${line}" ]] && containers+=("${line}") + done < <(docker container ls -aq --no-trunc 2>/dev/null) + + if (( ${#containers[@]} > 0 )); then + while IFS=$'\t' read -r image config_image; do + normalizeImageReference reference "${config_image}" + if [[ "${reference}" == *@* ]]; then + reference="${reference%%@*}" + else + reference="${reference%:*}" + fi + isRollImageRepository "${reference}" "${patterns[@]}" || continue + seen="${seen}${image}"$'\n' + done < <(docker container inspect --format '{{.Image}}{{"\t"}}{{.Config.Image}}' "${containers[@]}" 2>/dev/null) + fi + + if [[ -f "${ledger}" ]]; then + while IFS= read -r id; do + [[ -n "${id}" ]] || continue + [[ $'\n'"${existing}"$'\n' == *$'\n'"${id}"$'\n'* ]] || continue + seen="${seen}${id}"$'\n' + done < "${ledger}" + fi + + mkdir -p "${ledger%/*}" 2>/dev/null || return 0 + printf '%s' "${seen}" | sort -u > "${ledger}.$$" 2>/dev/null \ + && mv -f "${ledger}.$$" "${ledger}" 2>/dev/null + rm -f "${ledger}.$$" 2>/dev/null + + return 0 +} + +## Superseded roll images, one "\t\t\t" line each: dangling images +## whose kept repository name is roll's, or whose ID the ledger recorded. A dangling image with +## neither carries no trace of where it came from, so it is not roll's to remove. +function listSupersededRollImages() { + local patterns=() line="" id="" repository="" size="" created="" recorded="" + + while IFS= read -r line; do + patterns+=("${line}") + done < <(rollImageRepositoryPatterns) + + if [[ -f "$(rollImageLedgerFile)" ]]; then + recorded="$(cat "$(rollImageLedgerFile)" 2>/dev/null)" || true + fi + + while IFS=$'\t' read -r id repository size created; do + [[ -z "${id}" ]] && continue + if [[ "${repository}" != "" ]] && isRollImageRepository "${repository}" "${patterns[@]}"; then + printf '%s\t%s\t%s\t%s\n' "${id}" "${repository}" "${size}" "${created}" + elif [[ $'\n'"${recorded}"$'\n' == *$'\n'"${id}"$'\n'* ]]; then + printf '%s\t%s\t%s\t%s\n' "${id}" "" "${size}" "${created}" + fi + done < <(docker image ls --filter dangling=true --no-trunc \ + --format '{{.ID}}\t{{.Repository}}\t{{.Size}}\t{{.CreatedSince}}' 2>/dev/null) + + return 0 +} + +## An image reference as `docker image ls` spells it: Docker Hub prefixes dropped and the implicit +## :latest made explicit, so `traefik` and `docker.io/library/traefik:latest` compare equal. +## normalizeImageReference - assigns rather than prints, so a loop over every +## container on the host does not fork a subshell per container. +function normalizeImageReference() { + local var="$1" image_reference="$2" + + image_reference="${image_reference#docker.io/}" + image_reference="${image_reference#library/}" + + if [[ "${image_reference}" != *@* && "${image_reference##*/}" != *:* ]]; then + image_reference="${image_reference}:latest" + fi + + printf -v "${var}" '%s' "${image_reference}" + return 0 +} + +## Tagged roll images no container uses, one "\t\t\t" +## line each. The reference is the tag rather than the image id: one image can carry several tags, +## and removing it by id is refused while more than one tag points at it. +## +## A container counts against a tag both through the image id it runs and through the tag it was +## created from. The second matters after a pull: the tag has moved on to a newer image that no +## container runs yet, while the environment still names that tag and would pull it straight back +## on its next `up`. +function listUnusedRollImages() { + local patterns=() containers=() used="" line="" id="" repository="" tag="" size="" created="" + local image="" config_image="" reference="" + + while IFS= read -r line; do + patterns+=("${line}") + done < <(rollImageRepositoryPatterns) + + ## Stopped containers count: `roll env stop` leaves them in place and `env start` reuses them + while IFS= read -r line; do + [[ -n "${line}" ]] && containers+=("${line}") + done < <(docker container ls -aq --no-trunc 2>/dev/null) + + if (( ${#containers[@]} > 0 )); then + while IFS=$'\t' read -r image config_image; do + normalizeImageReference reference "${config_image}" + used="${used}${image}"$'\n'"${reference}"$'\n' + done < <(docker container inspect --format '{{.Image}}{{"\t"}}{{.Config.Image}}' "${containers[@]}" 2>/dev/null) + fi + + while IFS=$'\t' read -r id repository tag size created; do + [[ -z "${id}" || "${tag}" == "" ]] && continue + isRollImageRepository "${repository}" "${patterns[@]}" || continue + [[ $'\n'"${used}" == *$'\n'"${id}"$'\n'* ]] && continue + [[ $'\n'"${used}" == *$'\n'"${repository}:${tag}"$'\n'* ]] && continue + printf '%s:%s\t%s:%s\t%s\t%s\n' "${repository}" "${tag}" "${repository}" "${tag}" "${size}" "${created}" + done < <(docker image ls --filter dangling=false --no-trunc \ + --format '{{.ID}}\t{{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}' 2>/dev/null) + + return 0 +} + +## removeImages ... +## +## Removes each image without forcing, and leaves the outcome in ROLL_IMAGES_REMOVED (a count), +## ROLL_IMAGES_GONE (a count of images that had disappeared before this run got to them) and +## ROLL_IMAGES_KEPT (one ": " line per image Docker refused, typically +## because a container still uses it). +## +## An image that has disappeared is told apart by asking Docker whether it still exists, not by +## parsing the refusal: parallel `env up`s each run the automatic cleanup over the same list, and +## Docker words "already removed" differently depending on the reference form ("No such image", +## "unrecognized image ID"), so reporting those as kept would blame a container that does not exist. +function removeImages() { + local reference="" output="" + + ROLL_IMAGES_REMOVED=0 + ROLL_IMAGES_GONE=0 + ROLL_IMAGES_KEPT=() + + for reference in "$@"; do + if output="$(docker image rm "${reference}" 2>&1)"; then + ROLL_IMAGES_REMOVED=$((ROLL_IMAGES_REMOVED + 1)) + elif ! docker image inspect "${reference}" >/dev/null 2>&1; then + ROLL_IMAGES_GONE=$((ROLL_IMAGES_GONE + 1)) + else + ROLL_IMAGES_KEPT+=("${reference}: ${output%%$'\n'*}") + fi + done + + return 0 +} + +## The automatic cleanup after `roll svc pull|up` and `roll env pull|up`: superseded images only, +## relying on the caller having run recordRollImageIds before the pull or up, which is the moment +## the builds about to be superseded still carry their name. Silent unless it removed something. `up` is included because a pull cannot remove the image a +## running container still holds; the `up` that recreates that container is the first moment the +## old image is free. +## +## ROLL_IMAGE_AUTO_CLEANUP=0 turns it off, from ~/.roll/.env, a project's .env.roll or the process +## environment. The schema declares the key optional rather than defaulting it, so an exported +## default can never overwrite the value an unattended caller passed in. +function autoCleanupRollImages() { + local references=() line="" + + [[ "$(getConfig ROLL_IMAGE_AUTO_CLEANUP 1)" == "0" ]] && return 0 + + while IFS= read -r line; do + references+=("${line%%$'\t'*}") + done < <(listSupersededRollImages) + + (( ${#references[@]} > 0 )) || return 0 + + removeImages "${references[@]}" + + if (( ROLL_IMAGES_REMOVED > 0 )); then + info "Removed ${ROLL_IMAGES_REMOVED} superseded roll image(s). Run 'roll image-cleanup' to also remove images no container uses." + fi + + return 0 +}