diff --git a/README.md b/README.md index c3f65b1..4fc00cd 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,10 @@ image's native package manager before testing. Package setup is opt-in and requires bridge networking. See [configuration and migration](docs/configuration.md) and [why a shell script works on Ubuntu but fails on Alpine](docs/troubleshooting.md). +Testing an installer? Use the [installation smoke test workflow](docs/installer-smoke-test.md) +to check the installed binary inside each container. A script's zero exit status +alone does not prove it installed successfully. + ## Quickstart ### In GitHub Actions diff --git a/docs/installer-smoke-test.md b/docs/installer-smoke-test.md new file mode 100644 index 0000000..79243c4 --- /dev/null +++ b/docs/installer-smoke-test.md @@ -0,0 +1,55 @@ +# Verify what an installer actually installed + +A shell script can print “Installation finished” and return zero after a failed +download or extraction. OpsScript Gate checks the exit status of the command it +runs; it cannot infer which files an arbitrary installer should have created. +For an installer, make that contract explicit: install the binary, check that it +exists, then execute a harmless smoke command inside the same container. + +## Copy the workflow + +Copy [installer-smoke.yml](../examples/github-actions/installer-smoke.yml) into +`.github/workflows/installer-smoke.yml` in your project. Edit: + +- `install.sh` to your standalone installer path. +- `$HOME/.local/bin/mytool` to the installed binary path. +- `--version` to a noninteractive command that demonstrates the installed binary + can start. It must return nonzero on failure. +- The packages, timeout and distribution matrix to match your supported setup. +- The trigger paths to include release packaging configuration if it affects the + archive name or layout. + +The workflow embeds the current checked-out installer into a temporary script. +It runs the installer in a subshell and checks the binary afterwards. Even an +installer with an explicit `exit 0` cannot skip the check. A nonzero installer +status is preserved. The check and installer run in the **same container**; +checking for the installed binary on the GitHub runner would test a different +filesystem. + +The default example enables network access to download release assets and adds +Bash, curl, CA certificates and tar. Those are test prerequisites, not evidence +that your installer provisions them. If you promise installation on a bare +image, also test that scenario separately. + +## Interpret the result + +- An executable binary and a successful smoke command prove the tested artifact + could install and start in that container during that run. +- A missing binary fails even if the installer returned zero. +- A downloaded binary that cannot start also fails. +- HTTP errors, rate limits and changed release assets can affect the result. + Inspect the saved report before attributing a failure to a distribution. + +This is not a test of your application's complete behavior. Containers share +the host kernel and do not represent a full booted machine. OpsScript Gate mounts +only the target script; installers that need sibling repository files require a +different integration setup. + +## A failure this check caught + +In a [public fork validation of Latitude's CLI installer](https://github.com/Mresyzz/cli/actions/runs/38145896335), +the Ubuntu 22.04 job downloaded content that could not be extracted as gzip. The +installer continued and printed “Installation finished”. An installation check +then failed because `$HOME/.lsh/lsh` was absent. Debian 12, Ubuntu 24.04 and Alpine +3.20 passed in that run. This demonstrates a caught installation failure, not a +proven Ubuntu-specific defect or an endorsement by the upstream maintainers. diff --git a/examples/github-actions/README.md b/examples/github-actions/README.md index 1c08f0b..97b564f 100644 --- a/examples/github-actions/README.md +++ b/examples/github-actions/README.md @@ -3,3 +3,9 @@ This example shows a GitHub Actions workflow for checking a shell script on pull requests. Copy `workflow.yml` into your repository at `.github/workflows/script-gate.yml` and point `script-path` to the script you want to validate. The example uses the published v0.9.4 release. + +For a downloading installer, use [installer-smoke.yml](installer-smoke.yml) and +adapt its installed binary path and smoke command. This checks that installation +actually produced a working binary, even when the installer returns zero after +a failed download. See the [installer smoke test guide](../../docs/installer-smoke-test.md) +for the result boundaries and setup instructions. diff --git a/examples/github-actions/installer-smoke.yml b/examples/github-actions/installer-smoke.yml new file mode 100644 index 0000000..0a852af --- /dev/null +++ b/examples/github-actions/installer-smoke.yml @@ -0,0 +1,53 @@ +name: Installer smoke test + +on: + pull_request: + paths: + - install.sh + - .github/workflows/installer-smoke.yml + workflow_dispatch: + +permissions: + contents: read + +jobs: + installer: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + + # Replace the binary path and smoke command with your installer's contract. + - name: Prepare installer and installation check + shell: bash + run: | + { + printf '#!/bin/bash\n(\n' + cat install.sh + printf '\n)\ninstaller_status=$?\n[ "$installer_status" -eq 0 ] || exit "$installer_status"\n' + cat <<'EOF' + binary="$HOME/.local/bin/mytool" + test -x "$binary" || { echo "Installer did not create $binary" >&2; exit 1; } + "$binary" --version + EOF + } > installer-smoke.sh + + - name: Install and verify in each distribution + uses: Mresyzz/opsscript-gate@v0.9.4 + with: + script-path: installer-smoke.sh + shell: bash + matrix: debian:12-slim,ubuntu:22.04,ubuntu:24.04,alpine:3.20 + network: bridge + packages: bash,curl,ca-certificates,tar + timeout: '120' + output: reports/installer-smoke.json + + - name: Save report, including failed installs + if: always() + uses: actions/upload-artifact@v4 + with: + name: installer-smoke + path: reports/installer-smoke.json + if-no-files-found: ignore