Skip to content

test: add behavioral tests for all automation scripts (#25) - #142

Merged
PAMulligan merged 2 commits into
mainfrom
feature/issue-25-script-tests
Sep 9, 2026
Merged

PAMulligan merged 2 commits into
mainfrom
feature/issue-25-script-tests

Conversation

@PAMulligan

Copy link
Copy Markdown
Contributor

Summary

Closes #25.

Adds behavioral tests for every automation script that did not yet have them, on top of the Vitest harness introduced in #141. Each test shells out to the real script against a throwaway project directory with fake pnpm, npx, or k6 shims on PATH, so nothing touches the network and each file asserts the repo stays clean.

Script Tests What is covered
setup-project.sh 19 Dry-run structure for all five platforms, multi-tenant additions, missing name, unknown flag, existing dir refusal, full generation with a fake pnpm (package.json scripts, pnpm-workspace.yaml, copied templates)
run-tests.sh 16 Flag translation to vitest, config-driven integration timeout, coverage gate above and below threshold, failure propagation
check-types.sh 14 Flag translation to tsc, missing tsconfig and deps, noAnyTypes advisory
security-scan.sh 23 Every detector (secrets, JWT, SQL interpolation, rate limiting, CORS, console.log), test-file exclusion, --json shape, --no-fail and failOnVulnerability, audit level pass-through, lockfile policy
seed-database.sh 17 Env validation, production refusal in non-interactive mode, env-file precedence, URL masking, NODE_ENV pass-through
load-test.sh 12 Baseline generation with config thresholds, flag overrides, --json, k6 missing
generate-openapi-docs.sh 8 Generator creation, spec output, fallback spec on failure, Postman skip path
generate-client.sh 10 Types-only and --runtime generation, install paths, failure handling

Deviation from the issue text: the issue proposed BATS under tests/scripts/. Since #141 already established a Vitest harness under scripts/__tests__/ (the same convention Aurelius uses, and it runs in the existing script-tests CI job), these tests use that instead of adding a second framework.

Bugs the tests found and this PR fixes

Three scripts ended a block with [[ cond ]] && cmd, so a false condition became the exit status:

  • check-types.sh exited 1 whenever the noAnyTypes advisory found 1 to 10 hits.
  • load-test.sh exited 1 on every successful run without --json.
  • security-scan.sh aborted under set -e on any issue without a file path (rate limiting, CORS, logging, lockfile, dependency), skipping the summary and ignoring --no-fail.

Also fixed: security-scan.sh --json with zero issues crashed on macOS bash 3.2 (empty array under set -u), and the console.log count carried wc padding.

Noted, not changed

  • generate-openapi-docs.sh hardwires its Postman output to the framework root's postman/ rather than the API project, so the collection merge is only tested via its skip path.
  • generate-client.sh resolves relative --spec/--output against the framework root.
  • setup-project.sh defines its own colors and ignores NO_COLOR.

Test plan

  • pnpm test: 16 files, 381 tests pass
  • ShellCheck clean on all scripts
  • pnpm verify passes
  • CI script-tests job green

🤖 Generated with Claude Code

Paul Mulligan and others added 2 commits September 9, 2026 17:02
Three scripts ended a block with '[[ cond ]] && cmd', so a false condition
became the exit status: check-types exited 1 when the noAnyTypes advisory
found 1-10 hits, load-test exited 1 on every successful run without --json,
and security-scan aborted under set -e on any issue without a file path.
Also keep an empty JSON issue list from tripping set -u on bash 3.2 and
strip wc padding from the console.log count. Found by the new script tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Cover setup-project (dry-run structure per platform, multi-tenant, refusal
paths, full generation with a fake pnpm), run-tests, check-types,
security-scan detectors and audit policy, seed-database, load-test, and the
OpenAPI and client generators. Tests shell out to the real scripts against
throwaway projects with fake pnpm/npx/k6 shims, so nothing touches the
network. Document how to run them in CONTRIBUTING.md.

Closes #25

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot added the area: scripts Automation scripts label Sep 9, 2026
@PAMulligan
PAMulligan merged commit f114c59 into main Sep 9, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: scripts Automation scripts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add framework-level tests for all bash scripts

1 participant