Skip to content

AsyncAPI 3.x: a black-box run reads no coverage - #1869

Open
LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-kill-switchfrom
feature/asyncapi-black-box-coverage
Open

LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-kill-switchfrom
feature/asyncapi-black-box-coverage

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Second in the AsyncAPI white-box stack, on top of #1868. A black-box AsyncAPI run reads nothing the driver measures of the SUT, as a black-box REST or GraphQL run does.

Why

AsyncAPI needs the driver in black-box mode too, as the only way onto the broker, and AsyncApiFitness has one code path for both modes. That path asked the driver for the coverage of every test regardless of the mode. Main.logCodeCoverage and Statistics, which gate on EMConfig.usesDriver(), also read the units and boot-time targets of the SUT in black-box runs. So a black-box AsyncAPI run reported line coverage whenever the driver happened to be instrumented, added those lines to the search targets, and, with the kill switch on, had the driver stop the service's threads after every test.

What it changes

  • AsyncApiFitness calls updateFitnessAfterEvaluation and handleExtra only in white-box mode. resetSUT is still called in both modes. The outcome and reply targets come from the reply DTO, so they are unaffected.
  • EMConfig.readsSutCoverage(), i.e. !blackBox || bbExperiments, is what Main.logCodeCoverage, Statistics.getData and Statistics.writeCoveredTargets check instead of usesDriver().

What a black-box AsyncAPI run reports after this, which is the same as a black-box REST run: its AsyncAPI targets in coveredTargets, 0 for numberOfLines, numberOfBranches, coveredLines, coveredBranches and the unit counts, and -1 (unavailable) for the boot-time columns. The console shows no coverage lines, as for black-box REST. The covered-targets file lists no boot-time targets. With --bbExperiments, units and boot-time targets are read, as they are for REST.

What this changes outside AsyncAPI

Main and Statistics switch from usesDriver() to readsSutCoverage(). The two differ only for a black-box AsyncAPI run: for every other problem type usesDriver() is !blackBox || bbExperiments, so REST, GraphQL, RPC and Web report exactly what they did before. Happy to split that into its own PR if you would rather.

docs/write_driver.md says the driver is not asked for coverage in black-box mode.

Testing

  • AsyncApiFitnessTest, 2 new tests: with a FakeAsyncApiDriver that reports a SUT line, a white-box run asks for coverage and gets the line into the fitness, and a black-box run never asks, while its outcome and reply targets are still covered. The fake answers coverage requests with given test results, and counts them.
  • StatisticsTest, 2 new tests: with a driver that declares 120 lines and one line covered at boot time, white-box reports both, black-box reports neither.
  • EMConfigTest, 1 new test: readsSutCoverage() against usesDriver() for REST and AsyncAPI, black-box with and without --bbExperiments, and white-box.
  • All AsyncAPI suites plus the config ones and StatisticsTest: 287 tests, 0 failures.

AsyncAPI needs the driver in black-box mode too, as the only way onto
the broker, and AsyncApiFitness has one code path for both modes. That
path asked the driver for the coverage of every test regardless of the
mode, and Main and Statistics, which gate on EMConfig.usesDriver(), read
the units and boot-time targets of the SUT in black-box runs as well.
A black-box AsyncAPI run therefore reported line coverage whenever the
driver happened to be instrumented, and added those lines to the search
targets.

REST and GraphQL never read coverage in black-box mode, not even with
--bbExperiments. AsyncAPI does the same:

 - AsyncApiFitness calls updateFitnessAfterEvaluation and handleExtra
   only in white-box mode. resetSUT is still called in both.
 - EMConfig.readsSutCoverage(), i.e. !blackBox || bbExperiments, is
   what Main.logCodeCoverage, Statistics.getData and
   Statistics.writeCoveredTargets check. For every problem type other
   than AsyncAPI it is exactly what usesDriver() was, so their
   output is unchanged. A black-box AsyncAPI run reports its AsyncAPI
   targets and zero lines and branches, and logs no coverage, as a
   black-box REST run does. With --bbExperiments it reads the units
   and boot-time targets, as REST does.

FakeAsyncApiDriver answers coverage requests with given test results,
and counts them. AsyncApiFitnessTest checks that a white-box
run asks for coverage and gets the SUT lines into the fitness, and that
a black-box run never asks. StatisticsTest checks the reported totals
in both modes with an instrumented driver, and EMConfigTest the new
predicate against usesDriver() for REST and AsyncAPI, with and without
--bbExperiments. docs/write_driver.md says the driver is not asked for
coverage in black-box mode.

No change to white-box runs of any problem type, nor to black-box runs
of REST, GraphQL or RPC.
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-black-box-coverage branch from 0efa1c1 to 35e8490 Compare October 12, 2026 02:54
@jgaleotti
jgaleotti requested a review from arcuri82 October 13, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants