Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ BASHUNIT_LOGIN_SHELL= # Default: false (source login shell profile
BASHUNIT_LOG_JUNIT= # JUnit XML report path (e.g., report.xml)
BASHUNIT_LOG_GHA= # GitHub Actions workflow-commands log path (e.g., gha.log)
BASHUNIT_REPORT_HTML= # HTML test report path (e.g., report.html)
BASHUNIT_REPORT_MD= # Markdown summary path (e.g., summary.md); unset: appends to $GITHUB_STEP_SUMMARY

#───────────────────────────────────────────────────────────────────────────────
# Code Coverage
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## Unreleased

### Added
- `--report-md <file>` writes a human-readable Markdown summary: verdict, counts table, failures with fenced messages, coverage percentage when `--coverage` ran and the slowest tests under `--profile`. Inside GitHub Actions the summary is appended to `$GITHUB_STEP_SUMMARY` automatically when no explicit path is given (#1015)
- `--coverage-diff <ref>` restricts the coverage console report to lines changed since a base ref, and `--coverage-min` then gates on that diff percentage (#1032)
- `--gha-annotations <auto|always|never>` controls GitHub Actions annotations on stdout; `auto` turns them on inside GitHub Actions and stays quiet everywhere else (#1014)
- `--repeat <n>` runs each selected test n times so flakiness can be hunted before it reaches CI. The test is reported once with the aggregate outcome, a failure names the iteration it happened on, and repeat wraps `--retry` rather than the other way round (#1013)
Expand Down
4 changes: 2 additions & 2 deletions adrs/adr-011-source-layout-and-build-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ Seventeen, in load order. The order is the dependency layering: leaves first.
| 10 | `cli/` | 5 | 445 | the `doc`/`init`/`upgrade`/`watch` subcommand implementations |
| 11 | `assert/` | 11 | 2326 | every assertion |
| 12 | `doubles/` | 4 | 505 | spies and mocks — **sourced by `assert/index.sh`, not the entrypoint** |
| 13 | `reports/` | 7 | 465 | JUnit, TAP, JSON, GHA and HTML writers |
| 13 | `reports/` | 8 | 693 | JUnit, TAP, JSON, GHA, HTML and Markdown writers |
| 14 | `runner/` | 12 | 2434 | the file loop, per-test execution, retry, result parsing, `--list` |
| 15 | `benchmark/` | 4 | 219 | the bench implementation (`runner/bench.sh` is its loop) |
| 16 | `learn/` | 14 | 1296 | the interactive tutorial (9 of those files are `learn/lessons/`) |
| 17 | `main/` | 8 | 1473 | flag parsing per subcommand and the run lifecycle |

The file counts sum to 118, which is every `.sh` file in `src/` — that is the check to re-run
The file counts sum to 121, which is every `.sh` file in `src/` — that is the check to re-run
when editing this table, because both of its previous errors came from counting the wrong way.
`doubles/` was absent entirely: the table was generated from the entrypoint's `source` lines,
and `doubles/` is the one module the entrypoint does not source, so it fell through while the
Expand Down
1 change: 1 addition & 0 deletions completions/_bashunit
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ _bashunit() {
'(-r --report-html)'{-r,--report-html}'[Write HTML report]:file:_files' \
'--report-tap[Write TAP version 13 report]:file:_files' \
'--report-json[Write machine-readable JSON report]:file:_files' \
'--report-md[Write Markdown summary report]:file:_files' \
'(-s --simple)'{-s,--simple}'[Simple output with dots]' \
'--detailed[Detailed output, the default]' \
'--output[Output format]:format:(tap)' \
Expand Down
2 changes: 1 addition & 1 deletion completions/bashunit.bash
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ _BASHUNIT_COMPLETIONS_TEST_OPTS="--assert --boot --changed --coverage --coverage
--no-coverage-report --no-output --no-output-on-failure --no-parallel \
--no-progress --no-snapshot-create --order-by --output --parallel --profile \
--random-order --repeat --report-html \
--report-json --report-junit --report-tap --rerun-failed --retry --run-all \
--report-json --report-junit --report-md --report-tap --rerun-failed --retry --run-all \
--seed --shard --show-incomplete --show-output --show-skipped --simple \
--skip-env-file --snapshot-report-unused --snapshot-update \
--stop-on-failure --strict --tag \
Expand Down
19 changes: 19 additions & 0 deletions docs/command-line.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ bashunit test tests/ --parallel --simple
| `-r, --report-html <file>` | Write HTML report |
| `--report-tap <file>` | Write TAP version 13 report to a file |
| `--report-json <file>` | Write machine-readable JSON report to a file |
| `--report-md <file>` | Write Markdown summary to a file (automatic on GitHub Actions) |
| `-R, --run-all` | Run all assertions (don't stop on first failure) |
| `-s, --simple` | Simple output (dots) |
| `--detailed` | Detailed output (default) |
Expand Down Expand Up @@ -456,6 +457,9 @@ bashunit test tests/
```bash [JSON]
bashunit test tests/ --report-json report.json
```
```bash [Markdown]
bashunit test tests/ --report-md summary.md
```
:::

### GitHub Actions annotations
Expand Down Expand Up @@ -507,6 +511,21 @@ The `--report-json` flag writes machine-readable results for scripts, dashboards

`status` is one of `passed`, `failed`, `skipped`, `incomplete` (`snapshot` and `risky` are also emitted per test and counted as passed in the summary). Like the other file reporters, per-test rows come from a sequential run; under `--parallel` the file is still valid JSON.

### Markdown summary

> `bashunit test --report-md <file>`

Every other report format targets a machine; `--report-md` targets the place a developer looks first. It writes a human-readable Markdown summary: a one-line verdict with the total duration, a counts table, and each failure's name, `file:line` and message in a fenced code block. The coverage percentage is included when `--coverage` ran, and the slowest tests when `--profile` is on (`BASHUNIT_PROFILE_COUNT` entries, default 10).

Inside GitHub Actions there is nothing to configure: when `$GITHUB_STEP_SUMMARY` is set and no explicit `--report-md` path is given, the summary is appended to the job's step summary automatically, so it renders directly on the workflow run page. Only the outermost bashunit process writes it — nested runs (a script under test that itself calls bashunit) stay quiet.

```bash
bashunit test tests/ --report-md summary.md

# In a workflow: no flag needed, the step summary is filled automatically
bashunit test tests/
```

### Show Output on Failure

> `bashunit test --show-output`
Expand Down
16 changes: 16 additions & 0 deletions src/config/env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ _BASHUNIT_DEFAULT_LOG_GHA=""
_BASHUNIT_DEFAULT_REPORT_HTML=""
_BASHUNIT_DEFAULT_REPORT_TAP=""
_BASHUNIT_DEFAULT_REPORT_JSON=""
_BASHUNIT_DEFAULT_REPORT_MD=""

# Coverage defaults (following kcov, bashcov, SimpleCov conventions)
_BASHUNIT_DEFAULT_COVERAGE="false"
Expand Down Expand Up @@ -191,6 +192,9 @@ _BASHUNIT_DEFAULT_COVERAGE_DIFF=""
: "${BASHUNIT_REPORT_HTML:=${REPORT_HTML:=$_BASHUNIT_DEFAULT_REPORT_HTML}}"
: "${BASHUNIT_REPORT_TAP:=${REPORT_TAP:=$_BASHUNIT_DEFAULT_REPORT_TAP}}"
: "${BASHUNIT_REPORT_JSON:=${REPORT_JSON:=$_BASHUNIT_DEFAULT_REPORT_JSON}}"
# No bare REPORT_MD alias: the unprefixed forms are deprecated, so a new
# setting only ever ships under the BASHUNIT_ prefix.
: "${BASHUNIT_REPORT_MD:=$_BASHUNIT_DEFAULT_REPORT_MD}"

# Watch mode polling interval (seconds) used by the pure-shell fallback
_BASHUNIT_DEFAULT_WATCH_INTERVAL="2"
Expand Down Expand Up @@ -672,6 +676,18 @@ function bashunit::env::should_print_gha_annotations() {
! bashunit::env::is_tap_output_enabled
}

##
# Whether the run should append its Markdown summary to $GITHUB_STEP_SUMMARY.
# Same ownership rule as the annotations above: GITHUB_STEP_SUMMARY is
# inherited by every child process, so only the outermost run may write to it.
# An explicit --report-md path takes over as the destination instead.
##
function bashunit::env::should_write_github_step_summary() {
[ -z "${BASHUNIT_REPORT_MD:-}" ] &&
[ "${_BASHUNIT_IS_OUTERMOST_RUN:-true}" = true ] &&
[ -n "${GITHUB_STEP_SUMMARY:-}" ]
}

function bashunit::env::is_fail_on_flaky_enabled() {
[ "${BASHUNIT_FAIL_ON_FLAKY:-false}" = "true" ]
}
Expand Down
1 change: 1 addition & 0 deletions src/console/header.sh
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Options:
-r, --report-html <file> Write HTML report
--report-tap <file> Write TAP version 13 report
--report-json <file> Write machine-readable JSON report
--report-md <file> Write Markdown summary (auto-appends to \$GITHUB_STEP_SUMMARY without the flag)
-s, --simple Simple output (dots)
--detailed Detailed output (default)
--output <format> Output format: tap (TAP version 13)
Expand Down
8 changes: 8 additions & 0 deletions src/main/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,14 @@ function bashunit::main::exec_tests() {
bashunit::coverage::cleanup
fi

# After the coverage block on purpose: the Markdown summary embeds the
# coverage percentage, which exists only once precompute_file_stats ran.
if [ -n "$BASHUNIT_REPORT_MD" ]; then
bashunit::reports::generate_report_md "$BASHUNIT_REPORT_MD"
elif bashunit::env::should_write_github_step_summary; then
bashunit::reports::append_github_step_summary "$GITHUB_STEP_SUMMARY"
fi

if bashunit::parallel::is_enabled; then
bashunit::parallel::cleanup
fi
Expand Down
5 changes: 5 additions & 0 deletions src/main/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,11 @@ function bashunit::main::cmd_test() {
export -n BASHUNIT_REPORT_JSON
shift
;;
--report-md)
BASHUNIT_REPORT_MD="$2"
export -n BASHUNIT_REPORT_MD
shift
;;
--no-output)
BASHUNIT_NO_OUTPUT=true
export -n BASHUNIT_NO_OUTPUT
Expand Down
2 changes: 1 addition & 1 deletion src/main/validate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ function bashunit::main::validate_config_or_exit() {

local _report_var _report_path
for _report_var in BASHUNIT_LOG_JUNIT BASHUNIT_LOG_GHA BASHUNIT_REPORT_HTML \
BASHUNIT_REPORT_TAP BASHUNIT_REPORT_JSON; do
BASHUNIT_REPORT_TAP BASHUNIT_REPORT_JSON BASHUNIT_REPORT_MD; do
_report_path=${!_report_var:-}
if [ -n "$_report_path" ]; then
bashunit::main::require_writable_path_or_exit "$_report_path" "$_report_var"
Expand Down
4 changes: 3 additions & 1 deletion src/reports/collect.sh
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ function bashunit::reports::is_enabled() {
[ -n "${BASHUNIT_LOG_GHA:-}" ] ||
[ -n "${BASHUNIT_REPORT_TAP:-}" ] ||
[ -n "${BASHUNIT_REPORT_JSON:-}" ] ||
bashunit::env::should_print_gha_annotations
[ -n "${BASHUNIT_REPORT_MD:-}" ] ||
bashunit::env::should_print_gha_annotations ||
bashunit::env::should_write_github_step_summary
}

function bashunit::reports::add_test() {
Expand Down
1 change: 1 addition & 0 deletions src/reports/index.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ source "$BASHUNIT_ROOT_DIR/src/reports/tap.sh"
source "$BASHUNIT_ROOT_DIR/src/reports/json.sh"
source "$BASHUNIT_ROOT_DIR/src/reports/gha.sh"
source "$BASHUNIT_ROOT_DIR/src/reports/html.sh"
source "$BASHUNIT_ROOT_DIR/src/reports/markdown.sh"
128 changes: 128 additions & 0 deletions src/reports/markdown.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#!/usr/bin/env bash

# Markdown summary report writer. Aimed at humans on the pull-request page:
# GitHub renders anything appended to $GITHUB_STEP_SUMMARY, so the same
# document serves --report-md <file> and the automatic step-summary append.

# Escapes a string for Markdown prose and table cells: ANSI stripped first,
# then backslash before the others so the escapes added below are not doubled.
# Pipes must be escaped or a test name would end its table cell early.
function bashunit::reports::__md_escape() {
local text
text=$(bashunit::reports::__strip_ansi "$1")
text="${text//\\/\\\\}"
text="${text//\`/\\\`}"
text="${text//\*/\\*}"
text="${text//_/\\_}"
text="${text//|/\\|}"
printf '%s' "$text"
}

##
# Prints the whole Markdown summary to stdout. The file and step-summary
# entry points below only differ in redirection, so the document itself is
# rendered in exactly one place.
##
function bashunit::reports::print_report_md() {
local passed failed skipped incomplete risky snapshot flaky
passed=$(bashunit::state::get_tests_passed)
failed=$(bashunit::state::get_tests_failed)
skipped=$(bashunit::state::get_tests_skipped)
incomplete=$(bashunit::state::get_tests_incomplete)
risky=$(bashunit::state::get_tests_risky)
snapshot=$(bashunit::state::get_tests_snapshot)
flaky=$(bashunit::state::get_tests_flaky)

local time_ms time_s
time_ms=$(bashunit::clock::total_runtime_in_milliseconds)
# `env` rather than a bare `LC_ALL=C` prefix: C keeps awk's radix a dot, and
# that prefix form segfaults inside `$()` on Bash 5.3 macOS (#912).
time_s=$(env LC_ALL=C awk -v ms="$time_ms" 'BEGIN {printf "%.3f", ms/1000}')

if [ "$failed" -gt 0 ]; then
printf '### ❌ %s failed, %s passed in %ss\n\n' "$failed" "$passed" "$time_s"
else
printf '### ✅ %s passed in %ss\n\n' "$passed" "$time_s"
fi

printf '| Passed | Failed | Skipped | Incomplete | Risky | Snapshot | Flaky |\n'
printf '|---:|---:|---:|---:|---:|---:|---:|\n'
printf '| %s | %s | %s | %s | %s | %s | %s |\n\n' \
"$passed" "$failed" "$skipped" "$incomplete" "$risky" "$snapshot" "$flaky"

if [ "$failed" -gt 0 ]; then
bashunit::reports::__print_md_failures
fi

if bashunit::env::is_coverage_enabled; then
printf '**Coverage:** %s%%\n\n' "$(bashunit::coverage::get_percentage)"
fi

if bashunit::env::is_profile_enabled; then
bashunit::reports::__print_md_slowest
fi
}

# The failures section: name, file:line and the message in a fenced block --
# the part that saves a click into the raw job log.
function bashunit::reports::__print_md_failures() {
printf '#### Failures\n\n'

local i
for i in "${!_BASHUNIT_REPORTS_TEST_NAMES[@]}"; do
[ "${_BASHUNIT_REPORTS_TEST_STATUSES[$i]:-}" = "failed" ] || continue

local name location message
name=$(bashunit::reports::__md_escape "${_BASHUNIT_REPORTS_TEST_NAMES[$i]:-}")
location="${_BASHUNIT_REPORTS_TEST_FILES[$i]:-}"
if [ -n "${_BASHUNIT_REPORTS_TEST_LINES[$i]:-}" ]; then
location="$location:${_BASHUNIT_REPORTS_TEST_LINES[$i]}"
fi
message=$(bashunit::reports::__strip_ansi "${_BASHUNIT_REPORTS_TEST_FAILURES[$i]:-}")

printf -- '- **%s** — %s\n\n' "$name" "$location"
printf '```\n%s\n```\n\n' "$message"
done
}

# The slowest tests recorded in the report rows, duration descending, limited
# to BASHUNIT_PROFILE_COUNT. Reads the same rows every writer reads instead of
# PROFILE_OUTPUT_PATH, which print_profile_and_reset has already consumed.
function bashunit::reports::__print_md_slowest() {
[ "${#_BASHUNIT_REPORTS_TEST_NAMES[@]}" -gt 0 ] || return 0

printf '#### Slowest tests\n\n'
printf '| Duration (ms) | Test | File |\n'
printf '|---:|---|---|\n'

local i
local duration name file
for i in "${!_BASHUNIT_REPORTS_TEST_NAMES[@]}"; do
printf '%s\t%s\t%s\n' \
"${_BASHUNIT_REPORTS_TEST_DURATIONS[$i]:-0}" \
"${_BASHUNIT_REPORTS_TEST_NAMES[$i]:-}" \
"${_BASHUNIT_REPORTS_TEST_FILES[$i]:-}"
done \
| sort -rn \
| head -n "${BASHUNIT_PROFILE_COUNT:-10}" \
| while IFS="$(printf '\t')" read -r duration name file; do
printf '| %s | %s | %s |\n' \
"$duration" "$(bashunit::reports::__md_escape "$name")" "$file"
done

printf '\n'
}

function bashunit::reports::generate_report_md() {
local output_file="$1"

bashunit::reports::print_report_md >"$output_file"
}

# $GITHUB_STEP_SUMMARY aggregates every step of a job, so the summary is
# appended -- truncating would erase what earlier steps wrote.
function bashunit::reports::append_github_step_summary() {
local output_file="$1"

bashunit::reports::print_report_md >>"$output_file"
}
22 changes: 15 additions & 7 deletions tests/acceptance/bashunit_gha_annotations_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
# Clearing _BASHUNIT_GHA_ANNOTATIONS_CLAIMED below is how a nested run says
# "pretend I am the top-level one": this suite is itself a bashunit run and has
# already claimed the job log for its process tree, which is the very pollution
# the marker exists to prevent.
# the marker exists to prevent. A run that claims top-level also claims
# $GITHUB_STEP_SUMMARY, so it is pinned empty alongside or the fixtures'
# summaries would land on the real job page under CI.

function set_up_before_script() {
TEST_ENV_FILE="tests/acceptance/fixtures/.env.default"
Expand All @@ -16,7 +18,8 @@ function set_up_before_script() {

function test_annotations_reach_stdout_inside_github_actions() {
local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --no-parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --no-parallel --no-color \
--env "$TEST_ENV_FILE" "$FIXTURE")" || true

assert_contains "::error file=$FIXTURE" "$output"
Expand All @@ -25,7 +28,8 @@ function test_annotations_reach_stdout_inside_github_actions() {

function test_the_annotation_carries_the_failing_line() {
local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --no-parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --no-parallel --no-color \
--env "$TEST_ENV_FILE" "$FIXTURE")" || true

assert_matches "::error file=[^,]*,line=[0-9]+,title=" "$output"
Expand All @@ -52,7 +56,8 @@ function test_a_nested_run_never_annotates_the_parents_log() {

function test_never_suppresses_annotations_inside_github_actions() {
local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --no-parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --no-parallel --no-color \
--env "$TEST_ENV_FILE" --gha-annotations never "$FIXTURE")" || true

assert_not_contains "::error" "$output"
Expand All @@ -68,7 +73,8 @@ function test_always_emits_annotations_outside_github_actions() {

function test_a_multi_line_message_stays_one_annotation() {
local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --no-parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --no-parallel --no-color \
--env "$TEST_ENV_FILE" "$FIXTURE")" || true

# One failing test, so one ::error line, with the newlines percent-encoded.
Expand All @@ -81,7 +87,8 @@ function test_log_gha_still_writes_the_file_without_duplicating_stdout() {
log_file="$(bashunit::temp_file)"

local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --no-parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --no-parallel --no-color \
--env "$TEST_ENV_FILE" --log-gha "$log_file" "$FIXTURE")" || true

assert_contains "::error" "$(cat "$log_file")"
Expand All @@ -90,7 +97,8 @@ function test_log_gha_still_writes_the_file_without_duplicating_stdout() {

function test_annotations_survive_parallel_aggregation() {
local output
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_ACTIONS=true ./bashunit --parallel --no-color \
output="$(_BASHUNIT_GHA_ANNOTATIONS_CLAIMED='' GITHUB_STEP_SUMMARY='' GITHUB_ACTIONS=true \
./bashunit --parallel --no-color \
--env "$TEST_ENV_FILE" "$FIXTURE")" || true

assert_contains "::error file=$FIXTURE" "$output"
Expand Down
Loading