feat(cli): --list / --dry-run to show which tests would run - #1034
Merged
Conversation
Every selection mechanism — --filter, --tag, --exclude-tag, --shard, --random-order --seed, --rerun-failed, file::fn, file:LINE — was only observable by actually running the suite. Checking shard balance took one full run per shard, and a typo'd --filter was indistinguishable from a genuinely empty suite. --list prints `path::function` per line on stdout with the count on stderr, so it pipes cleanly; --list-format json adds name, line and tags. --dry-run is a shellspec-compatible alias. The listing hooks into load_test_files after every selection step and before render_running_file_header and set_up_before_script, so nothing runs beyond sourcing the files. Ordering is shared with the runner via the extracted runner::order_functions_for_script, so a listed order cannot drift from the order that seed executes. An empty selection exits 0: this is a query, not a run. Closes #1007
…red output The seed-order test parsed result lines with `sed 's/\x1b.../'` and `grep -oE '^. Passed:'`. Both are environment-dependent — \x1b is a GNU sed extension, and the status glyph is multi-byte, so `.` matches a different amount under a non-UTF-8 locale. It passed on macOS and failed on every Linux and locale job. The fixture now records its own execution order to a file, so the test compares orders directly with no rendering in the path. A paired test asserts the seed actually reorders, so the comparison cannot hold trivially.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1007
There was no way to ask bashunit which tests it would run. Every selection mechanism was only observable by executing the suite: checking
--shardbalance took one full run per shard, and a typo'd--filterproduced "No tests found", indistinguishable from a genuinely empty suite.💡 Changes
--list(alias--dry-run) printspath::functionper line on stdout, count on stderr, so the list pipes intogrep/fzf/a CI matrix--list-format jsonaddsname,lineandtags; an unsupported format is rejected rather than silently falling backrunner::order_functions_for_script, so--list --random-order --seed Ncannot drift from what seed N actually executesset_up_before_script, so no test body, no lifecycle hook and no report file — verified by a fixture whose hook and body both write a markerNotes
An empty selection exits 0 — a query, not a run. A
@data_providertest is listed once, by function: that id is what you can pass back to--filter.Both mutation-tested: moving the interception after the hooks fails 11 tests, and making the listing re-derive its own order fails exactly the seed-order test. Green on the Bash 3.2 floor.