Skip to content

fix: stop counting a mutant whose subprocess never ran a test as killed - #45

Open
tiagoabsantos wants to merge 1 commit into
pestphp:5.xfrom
tiagoabsantos:fix/mutant-subprocess-paratest-options-and-php-binary
Open

tiagoabsantos wants to merge 1 commit into
pestphp:5.xfrom
tiagoabsantos:fix/mutant-subprocess-paratest-options-and-php-binary

Conversation

@tiagoabsantos

Copy link
Copy Markdown

Fixes pestphp/pest#1937.

--mutate can report a 100% score for mutants whose subprocess never ran a single test. MutationTest::hasFinished() reads any unsuccessful exit as a kill, so a subprocess that dies before reaching the test suite is indistinguishable from a legitimately killed mutant. Two inherited arguments make that happen.

ParaTest-only options are forwarded to the mutant (all platforms)

A mutant runs one PHPUnit process, never ParaTest, but the original arguments are passed through as they were. --parallel is already dropped by ParallelOption::remove(); --processes and --passthru-php are not, and PHPUnit rejects them:

INFO  Unknown option "--processes". Most similar options are --covers, --colors, --group, --no-progress, --no-results.

Exit code 2 in a fraction of a second, counted as killed. This is the Linux CI case in the issue, and it applies to every ParaTest-only option, so MutantArguments strips them all rather than only the two that were reported. It also takes over the coverage stripping that was inline, so the rules for what a mutant may not receive live in one place.

The options are removed from the mutant's arguments only, not from the configuration, so the original run keeps --processes and --passthru-php for its own parallel execution — dropping --passthru-php there would take pcov with it and leave the run without coverage.

--processes 2 spends two arguments, so the value is dropped along with the option; a stray 2 would otherwise be read as a test path.

The Pest script is used as the executable (Windows)

new Process([...$filteredArguments, ...]) puts $originalArguments[0], the Pest script, in the executable position. That works wherever the shebang is honoured. On Windows cmd.exe reads vendor/bin/pest as the command vendor:

'vendor' is not recognized as an internal or external command, operable program or batch file.

Exit code 1, counted as killed. The mutant now runs through PHP_BINARY, the same way the original run reached the same script.

Verification

Against tests/.tests/Untested, whose snapshot expects 3 untested out of 4, on Windows 11 with PHP 8.4.25:

before after
serial Mutations: 4 tested, Score: 100.00%, 0.17s Mutations: 3 untested, 1 tested, Score: 25.00%, 2.36s
--parallel --processes=2 --passthru-php=... Mutations: 4 tested, Score: 100.00% Mutations: 3 untested, 1 tested, Score: 25.00%

Both new tests in tests/Unit/MutationTestTest.php fail on the current 5.x and pass here. They run a real subprocess against a stand-in for the Pest script that records its own $argv, so they cover the executable and the forwarded arguments the way a mutant actually sees them, on Linux as well as Windows.

pint --test, phpstan, and pest --type-coverage --min=100 are clean. The 7 failures left in pest on my Windows machine are all present on 5.x before this branch: they are the path and line-ending issues already covered by #36 and #44.

Not included

The issue also suggests not counting a run in which no test was executed as tested, as a backstop for any other environment problem in the subprocess. That needs a signal beyond the exit code — exit code 2 is also what a legitimately killed mutant produces when it makes a test error — so it is a separate change, and #42 is already exploring one shape of it. This PR fixes the two causes that are unambiguous.

A mutant's subprocess inherited two things it cannot use, and both made it
exit before running a single test. MutationTest::hasFinished() reads any
unsuccessful exit as a kill, so those mutants were reported as tested and
the score read 100%.

ParaTest-only options such as --processes and --passthru-php were forwarded
verbatim. The mutant runs one PHPUnit process, which aborts with
'Unknown option "--processes"' and exit code 2. They are now stripped in
MutantArguments, alongside the coverage options that were already dropped,
so the original run keeps them for its own parallel execution.

The subprocess also used the Pest script as its executable. That works
wherever the shebang is honoured, but on Windows cmd.exe reads
'vendor/bin/pest' as the command 'vendor' and fails with exit code 1. The
mutant now runs through PHP_BINARY, the same way the original run did.

Closes pestphp/pest#1937
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant