Skip to content

feat(cli): --repeat <n> to hunt flaky tests before they reach CI #1013

Description

@Chemaclass

Problem

--retry mitigates flakiness once it has already burned a CI run. There is no way to go looking for it: to prove a test is deterministic, you have to run the suite N times by hand and diff the outputs.

Related: #1012 classifies a test that only passed on retry as flaky — that tells you a test was flaky after the fact; --repeat lets you find out before merging.

Proposal

--repeat <n>      Run each selected test n times (default 1); the test fails if any iteration fails
  • Applies per test, inside the runner, so --repeat 50 --filter flaky_candidate is the natural "hammer this one test" invocation.
  • A test is reported once, with the aggregate outcome. On failure, the message names the failing iteration: failed on iteration 7 of 50.
  • Counters count the test once, not n times.
  • Composes with --parallel (each worker repeats its own test) and with --retry (repeat is the outer loop, retry the inner one) — document the interaction explicitly.
  • --repeat 0 is a usage error, not a silent no-op.

Where to change

  • src/runner/exec.sh:312 — the retry while :; loop is the natural place; repeat wraps it.
  • src/main/test.sh, bashunit::main::require_non_negative_int_or_exit (src/main/validate.sh:19) for validation, src/config/env.sh (BASHUNIT_REPEAT), .env.example.

Acceptance criteria

  • --repeat 3 runs each test body three times
  • A test that fails on any iteration is reported failed, with the iteration number in the message
  • The test appears once in the summary and in every report
  • Assertion counts are not multiplied by the repeat factor (or the chosen semantics are documented and tested)
  • --repeat 1 is byte-identical to no flag
  • --repeat abc and --repeat -1 exit non-zero with a usage error (--jobs with a non-integer value hangs on Bash 3.x and is silently ignored on Bash 4.3+ #873 class of bug)
  • Interaction with --retry is documented and covered by a test
  • Works under --parallel
  • Per-test hooks (set_up / tear_down) run once per iteration; set_up_before_script does not
  • Acceptance test in tests/acceptance/

Repo checklist (agent)

  • TDD: RED → GREEN → REFACTOR. Write the failing test first.
  • Bash 3.0+ only: no printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}.
  • A new CLI flag must be wired in all of these or a parity test fails:
  • Gates: make sa, make lint, ./bashunit tests/, ./bashunit --parallel tests/. Never run shfmt -w.
  • Docs: update docs/command-line.md. Editing docs/assertions.md invalidates the bashunit doc acceptance snapshot — regenerate it.
  • CHANGELOG.md: add one line under ## Unreleased.
  • Fixtures under tests/acceptance/fixtures/ must not end in *test.sh.
  • One issue = one PR.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions