Repository navigation
AsyncAPI 3.x: a black-box run reads no coverage - #1869
Open
LautaroPetaccio wants to merge 1 commit into
Open
LautaroPetaccio wants to merge 1 commit into
LautaroPetaccio wants to merge 1 commit into
Conversation
AsyncAPI needs the driver in black-box mode too, as the only way onto the broker, and AsyncApiFitness has one code path for both modes. That path asked the driver for the coverage of every test regardless of the mode, and Main and Statistics, which gate on EMConfig.usesDriver(), read the units and boot-time targets of the SUT in black-box runs as well. A black-box AsyncAPI run therefore reported line coverage whenever the driver happened to be instrumented, and added those lines to the search targets. REST and GraphQL never read coverage in black-box mode, not even with --bbExperiments. AsyncAPI does the same: - AsyncApiFitness calls updateFitnessAfterEvaluation and handleExtra only in white-box mode. resetSUT is still called in both. - EMConfig.readsSutCoverage(), i.e. !blackBox || bbExperiments, is what Main.logCodeCoverage, Statistics.getData and Statistics.writeCoveredTargets check. For every problem type other than AsyncAPI it is exactly what usesDriver() was, so their output is unchanged. A black-box AsyncAPI run reports its AsyncAPI targets and zero lines and branches, and logs no coverage, as a black-box REST run does. With --bbExperiments it reads the units and boot-time targets, as REST does. FakeAsyncApiDriver answers coverage requests with given test results, and counts them. AsyncApiFitnessTest checks that a white-box run asks for coverage and gets the SUT lines into the fitness, and that a black-box run never asks. StatisticsTest checks the reported totals in both modes with an instrumented driver, and EMConfigTest the new predicate against usesDriver() for REST and AsyncAPI, with and without --bbExperiments. docs/write_driver.md says the driver is not asked for coverage in black-box mode. No change to white-box runs of any problem type, nor to black-box runs of REST, GraphQL or RPC.
LautaroPetaccio
force-pushed
the
feature/asyncapi-black-box-coverage
branch
from
October 12, 2026 02:54
0efa1c1 to
35e8490
Compare
jgaleotti
approved these changes
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second in the AsyncAPI white-box stack, on top of #1868. A black-box AsyncAPI run reads nothing the driver measures of the SUT, as a black-box REST or GraphQL run does.
Why
AsyncAPI needs the driver in black-box mode too, as the only way onto the broker, and
AsyncApiFitnesshas one code path for both modes. That path asked the driver for the coverage of every test regardless of the mode.Main.logCodeCoverageandStatistics, which gate onEMConfig.usesDriver(), also read the units and boot-time targets of the SUT in black-box runs. So a black-box AsyncAPI run reported line coverage whenever the driver happened to be instrumented, added those lines to the search targets, and, with the kill switch on, had the driver stop the service's threads after every test.What it changes
AsyncApiFitnesscallsupdateFitnessAfterEvaluationandhandleExtraonly in white-box mode.resetSUTis still called in both modes. The outcome and reply targets come from the reply DTO, so they are unaffected.EMConfig.readsSutCoverage(), i.e.!blackBox || bbExperiments, is whatMain.logCodeCoverage,Statistics.getDataandStatistics.writeCoveredTargetscheck instead ofusesDriver().What a black-box AsyncAPI run reports after this, which is the same as a black-box REST run: its AsyncAPI targets in
coveredTargets,0fornumberOfLines,numberOfBranches,coveredLines,coveredBranchesand the unit counts, and-1(unavailable) for the boot-time columns. The console shows no coverage lines, as for black-box REST. The covered-targets file lists no boot-time targets. With--bbExperiments, units and boot-time targets are read, as they are for REST.What this changes outside AsyncAPI
MainandStatisticsswitch fromusesDriver()toreadsSutCoverage(). The two differ only for a black-box AsyncAPI run: for every other problem typeusesDriver()is!blackBox || bbExperiments, so REST, GraphQL, RPC and Web report exactly what they did before. Happy to split that into its own PR if you would rather.docs/write_driver.mdsays the driver is not asked for coverage in black-box mode.Testing
AsyncApiFitnessTest, 2 new tests: with aFakeAsyncApiDriverthat reports a SUT line, a white-box run asks for coverage and gets the line into the fitness, and a black-box run never asks, while its outcome and reply targets are still covered. The fake answers coverage requests with given test results, and counts them.StatisticsTest, 2 new tests: with a driver that declares 120 lines and one line covered at boot time, white-box reports both, black-box reports neither.EMConfigTest, 1 new test:readsSutCoverage()againstusesDriver()for REST and AsyncAPI, black-box with and without--bbExperiments, and white-box.StatisticsTest: 287 tests, 0 failures.