Skip to content

AsyncAPI 3.x: stable coverage as a completion strategy - #1872

Open
LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-settlefrom
feature/asyncapi-quiescence
Open

LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-settlefrom
feature/asyncapi-quiescence

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fifth in the AsyncAPI white-box stack, on top of #1871.
This is the first of the asynchronous stop heuristics: ways for the settle window from the previous PR to end early, once the service looks done with a message, instead of always waiting for the deadline. This one, COVERAGE_STABLE, ends the window once the SUT's coverage has stayed stable for a set time, reaching nothing new. The other two, HANDLER_RETURN (#1875) and CONSUMER_LAG (#1877), come later in the stack.

Why

A window long enough for the slowest handler is wasted on every message the service handles quickly, and that is most of them. With a 5 s window and a 100 ms stable period, the new E2E runs 40 evaluations in about 30 s. If every wait went to the deadline, that would take several minutes.

What it adds

  • ExecutionTracer.getObjectiveUpdateCount(): how many times, in the current test, an objective was reached for the first time or improved on its best value. It is incremented inside the synchronized block updateObjective already holds, and reset with the rest of the per-test state.
  • SutController.getObjectiveUpdateCount(): returns null by default, so no driver has to implement it. EmbeddedSutController reads it directly, and returns null without the agent, where a count stuck at zero would read as stable coverage. ExternalSutController asks the agent with a new Command.OBJECTIVE_UPDATE_COUNT, which AgentController answers with a Long, the same way BOOT_TIME_INFO is answered with an object (ServerController.getObjectiveUpdateCount()), and with null when it fails to read it, so the controller never stalls on a missing answer.
  • AsyncApiActionDto.Completion {FIXED, COVERAGE_STABLE} and stableMs. AsyncApiReplyDto.SettleEnd {DEADLINE, COVERAGE_STABLE, INTERRUPTED} in settleEndedBy says why the wait ended. All boxed, with null semantics in the Javadoc.
  • AsyncApiSettle reads the count every 25 ms. It stops once the count has not moved for stableMs, or at settleMs, whichever comes first. A count that cannot be read, or stops being readable, including a read that throws (warned about once), means the whole window.
  • --asyncApiCompletion (FIXED by default) and --asyncApiStableMs (200), both experimental. COVERAGE_STABLE without --asyncApiSettleMs is a ConfigProblemException in white-box, since there is no window to end early; a stable period not shorter than the window gets a warning. In black-box both settle options are put back to their defaults, with a warning, as a black-box run never keeps an action open. Why each wait ended is kept on AsyncApiCallResult.

The one decision: count improvements, not executions

A service that polls a broker runs the same instrumented lines forever, so its executions never settle. Once those lines are covered in the test, though, they stop improving anything, and the count stops moving.

The cost: a handler that only re-runs paths already covered earlier in the same test does not move the count either, so its wait ends early. That costs nothing for fitness, but whatever it runs after the window closes can be credited to the next action (taint, SQL info). So this is a heuristic. It is designed independently of the transport, but has only been validated on Kafka, and the option's description says so. A transport-level signal, such as records consumed, can be added later as a second source of activity.

What this changes outside AsyncAPI

  • ExecutionTracer: one AtomicLong, incremented only when objectiveCoverage gets a new or better value, and reset in reset(). It is declared before the static block that calls reset(). Nothing reads it except the new methods.
  • Command: one new value, appended at the end. AgentController/ServerController: one new case and one new method, following the existing request/response pattern.
  • SutController/EmbeddedSutController/ExternalSutController: one new method each. It is not abstract in SutController.

None of this changes behaviour for other problem types. Happy to split the instrumentation part into its own PR if you would rather.

Not here

Other completion strategies (handler return, consumer lag) need transport-level tracking and the Kafka helper module, and come later in the stack.

Testing

  • ExecutionTracerTest (new, client-java/instrumentation), 5 tests: zero at start; first reach counts; re-running the same code does not; only a better value counts; reset starts from zero.
  • ServerControllerTest (new), 3 tests, with a real AgentController connected over a local socket in the same JVM: the count round-trips; NEW_TEST resets it in the agent; it interleaves with other commands without desynchronising replies.
  • AsyncApiSettleTest, 16 tests (10 new), on a fake clock: a SUT with stable coverage let go after stableMs; activity extends the wait to a stable window after the last move; the deadline caps a SUT whose coverage never settles; an unreadable, lost or throwing count means the whole window, or the rest of it; no or non-positive stableMs means a fixed wait; a fixed wait never reads the count; an interrupt during the loop ends it with INTERRUPTED.
  • EMControllerTest, 8 tests (3 new), through newAction with the embedded direct read: a SUT reaching nothing new is let go at stableMs; one that keeps reaching new objectives (simulated through ExecutionTracer) is waited on until the deadline; without the agent the whole window is waited.
  • AsyncApiFitnessTest, 3 new tests: completion and stable window sent in white-box, not in black-box, and why the wait ended is kept on the result. EMConfigTest, 3 new tests: the cross-option check and the defaults, the settle options ignored in black-box, and a stable period no shorter than the window allowed.
  • NcsKafkaEMTest.testRunEMWithCoverageStable, a new E2E with a 5000 ms window and a 100 ms stable period: no reply missed, every result settled, at least one wait ended on COVERAGE_STABLE, none interrupted.
  • Core: AsyncApi*Test, ConfigToMarkdownTest, EMConfigTest: 275 tests, 0 failures. client-java/instrumentation, whole module: 737 tests, 0 failures (1 skipped, as before). client-java/controller: AsyncApiSettleTest, EMControllerTest, AsyncApiProblemTest: 31 tests, 0 failures. NcsKafkaEMTest: 5 tests, 0 failures.

A settle window long enough for the slowest handler is wasted on every
message the service handles quickly, and that is most of them. This
lets the controller end the window early, once the SUT's coverage has
stayed stable for a set time, reaching nothing new.

What counts as "new" is an objective reached for the first time in the
test, or improved on its best value. Executions would not do: a
service polling a broker runs the same instrumented lines forever, so
its executions never settle. Once those lines
are covered, though, they stop improving anything. The cost is worth
flagging for review: a handler that only re-runs paths already covered
earlier in the same test does not move the count either, so its wait
ends early. Fitness loses nothing by that, but what it executes after
the window closes can be credited to the next action. This is a
heuristic, designed independently of the transport and validated on
Kafka.

 - ExecutionTracer counts improvements in updateObjective, under the
   lock it already holds, and resets the count with the rest of the
   per-test state. InstrumentationController passes it through.
 - SutController.getObjectiveUpdateCount, returning null by default
   so that no driver has to implement it. The embedded controller
   reads it directly, and returns null without the agent; the external
   one asks the agent with a new OBJECTIVE_UPDATE_COUNT command,
   answered with a Long the way BOOT_TIME_INFO is answered with an
   object, and with null when the agent fails to read it, so the
   controller never stalls on a missing answer.
 - AsyncApiActionDto.Completion {FIXED, COVERAGE_STABLE} and
   stableMs; AsyncApiReplyDto.SettleEnd {DEADLINE, COVERAGE_STABLE,
   INTERRUPTED} says why the wait ended.
 - AsyncApiSettle reads the count every 25 ms, and stops once it has
   not moved for stableMs, or at settleMs, whichever comes first. A
   count that cannot be read, including a read that throws, means the
   whole window.
 - --asyncApiCompletion (FIXED by default) and --asyncApiStableMs, both
   experimental. COVERAGE_STABLE without --asyncApiSettleMs is a
   configuration error in white-box, as there is no window to end
   early, and a stable period not shorter than the window is warned
   about. In black-box both settle options are put back to their
   defaults with a warning: a black-box run never keeps an action
   open. Why each wait ended is kept on AsyncApiCallResult.

Nothing changes unless --asyncApiCompletion is set. No shared REST code
is touched.
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-quiescence branch from 76a3cc5 to 49dca15 Compare October 12, 2026 02:54
@LautaroPetaccio
LautaroPetaccio marked this pull request as draft October 12, 2026 12:44
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-quiescence branch from 49dca15 to 9cf428a Compare October 17, 2026 23:48
@LautaroPetaccio LautaroPetaccio changed the title AsyncAPI 3.x: coverage quiescence as a completion strategy AsyncAPI 3.x: stable coverage as a completion strategy Oct 17, 2026
@LautaroPetaccio
LautaroPetaccio marked this pull request as ready for review October 18, 2026 01:43
* How the controller decides that the service is done with a message, and the action can
* be closed, within the {@link #settleMs} window.
*/
public enum Completion {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename Completion to a more descriptive name in the context of AsyncAPIs

/**
* Why the controller stopped keeping the action open.
*/
public enum SettleEnd {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename settleEnd to a more descriptive name, perhaps ReasonSettleEnded?

EXTRACT_JVM_DTO,
BOOTING_SUT
BOOTING_SUT,
OBJECTIVE_UPDATE_COUNT

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Javadoc for new OBJECTIVE_UPDATE_COUNT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants