Repository navigation
AsyncAPI 3.x: stable coverage as a completion strategy - #1872
Open
LautaroPetaccio wants to merge 1 commit into
Open
LautaroPetaccio wants to merge 1 commit into
LautaroPetaccio wants to merge 1 commit into
Conversation
LautaroPetaccio
added this pull request to stack #1812
October 11, 2026 19:08
A settle window long enough for the slowest handler is wasted on every
message the service handles quickly, and that is most of them. This
lets the controller end the window early, once the SUT's coverage has
stayed stable for a set time, reaching nothing new.
What counts as "new" is an objective reached for the first time in the
test, or improved on its best value. Executions would not do: a
service polling a broker runs the same instrumented lines forever, so
its executions never settle. Once those lines
are covered, though, they stop improving anything. The cost is worth
flagging for review: a handler that only re-runs paths already covered
earlier in the same test does not move the count either, so its wait
ends early. Fitness loses nothing by that, but what it executes after
the window closes can be credited to the next action. This is a
heuristic, designed independently of the transport and validated on
Kafka.
- ExecutionTracer counts improvements in updateObjective, under the
lock it already holds, and resets the count with the rest of the
per-test state. InstrumentationController passes it through.
- SutController.getObjectiveUpdateCount, returning null by default
so that no driver has to implement it. The embedded controller
reads it directly, and returns null without the agent; the external
one asks the agent with a new OBJECTIVE_UPDATE_COUNT command,
answered with a Long the way BOOT_TIME_INFO is answered with an
object, and with null when the agent fails to read it, so the
controller never stalls on a missing answer.
- AsyncApiActionDto.Completion {FIXED, COVERAGE_STABLE} and
stableMs; AsyncApiReplyDto.SettleEnd {DEADLINE, COVERAGE_STABLE,
INTERRUPTED} says why the wait ended.
- AsyncApiSettle reads the count every 25 ms, and stops once it has
not moved for stableMs, or at settleMs, whichever comes first. A
count that cannot be read, including a read that throws, means the
whole window.
- --asyncApiCompletion (FIXED by default) and --asyncApiStableMs, both
experimental. COVERAGE_STABLE without --asyncApiSettleMs is a
configuration error in white-box, as there is no window to end
early, and a stable period not shorter than the window is warned
about. In black-box both settle options are put back to their
defaults with a warning: a black-box run never keeps an action
open. Why each wait ended is kept on AsyncApiCallResult.
Nothing changes unless --asyncApiCompletion is set. No shared REST code
is touched.
LautaroPetaccio
force-pushed
the
feature/asyncapi-quiescence
branch
from
October 12, 2026 02:54
76a3cc5 to
49dca15
Compare
LautaroPetaccio
marked this pull request as draft
October 12, 2026 12:44
LautaroPetaccio
force-pushed
the
feature/asyncapi-quiescence
branch
from
October 17, 2026 23:48
49dca15 to
9cf428a
Compare
LautaroPetaccio
marked this pull request as ready for review
October 18, 2026 01:43
jgaleotti
requested changes
Oct 19, 2026
| * How the controller decides that the service is done with a message, and the action can | ||
| * be closed, within the {@link #settleMs} window. | ||
| */ | ||
| public enum Completion { |
Collaborator
There was a problem hiding this comment.
rename Completion to a more descriptive name in the context of AsyncAPIs
| /** | ||
| * Why the controller stopped keeping the action open. | ||
| */ | ||
| public enum SettleEnd { |
Collaborator
There was a problem hiding this comment.
rename settleEnd to a more descriptive name, perhaps ReasonSettleEnded?
| EXTRACT_JVM_DTO, | ||
| BOOTING_SUT | ||
| BOOTING_SUT, | ||
| OBJECTIVE_UPDATE_COUNT |
Collaborator
There was a problem hiding this comment.
add Javadoc for new OBJECTIVE_UPDATE_COUNT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth in the AsyncAPI white-box stack, on top of #1871.
This is the first of the asynchronous stop heuristics: ways for the settle window from the previous PR to end early, once the service looks done with a message, instead of always waiting for the deadline. This one,
COVERAGE_STABLE, ends the window once the SUT's coverage has stayed stable for a set time, reaching nothing new. The other two,HANDLER_RETURN(#1875) andCONSUMER_LAG(#1877), come later in the stack.Why
A window long enough for the slowest handler is wasted on every message the service handles quickly, and that is most of them. With a 5 s window and a 100 ms stable period, the new E2E runs 40 evaluations in about 30 s. If every wait went to the deadline, that would take several minutes.
What it adds
ExecutionTracer.getObjectiveUpdateCount(): how many times, in the current test, an objective was reached for the first time or improved on its best value. It is incremented inside thesynchronizedblockupdateObjectivealready holds, and reset with the rest of the per-test state.SutController.getObjectiveUpdateCount(): returnsnullby default, so no driver has to implement it.EmbeddedSutControllerreads it directly, and returnsnullwithout the agent, where a count stuck at zero would read as stable coverage.ExternalSutControllerasks the agent with a newCommand.OBJECTIVE_UPDATE_COUNT, whichAgentControlleranswers with aLong, the same wayBOOT_TIME_INFOis answered with an object (ServerController.getObjectiveUpdateCount()), and withnullwhen it fails to read it, so the controller never stalls on a missing answer.AsyncApiActionDto.Completion{FIXED, COVERAGE_STABLE}andstableMs.AsyncApiReplyDto.SettleEnd{DEADLINE, COVERAGE_STABLE, INTERRUPTED}insettleEndedBysays why the wait ended. All boxed, with null semantics in the Javadoc.AsyncApiSettlereads the count every 25 ms. It stops once the count has not moved forstableMs, or atsettleMs, whichever comes first. A count that cannot be read, or stops being readable, including a read that throws (warned about once), means the whole window.--asyncApiCompletion(FIXEDby default) and--asyncApiStableMs(200), both experimental.COVERAGE_STABLEwithout--asyncApiSettleMsis aConfigProblemExceptionin white-box, since there is no window to end early; a stable period not shorter than the window gets a warning. In black-box both settle options are put back to their defaults, with a warning, as a black-box run never keeps an action open. Why each wait ended is kept onAsyncApiCallResult.The one decision: count improvements, not executions
A service that polls a broker runs the same instrumented lines forever, so its executions never settle. Once those lines are covered in the test, though, they stop improving anything, and the count stops moving.
The cost: a handler that only re-runs paths already covered earlier in the same test does not move the count either, so its wait ends early. That costs nothing for fitness, but whatever it runs after the window closes can be credited to the next action (taint, SQL info). So this is a heuristic. It is designed independently of the transport, but has only been validated on Kafka, and the option's description says so. A transport-level signal, such as records consumed, can be added later as a second source of activity.
What this changes outside AsyncAPI
ExecutionTracer: oneAtomicLong, incremented only whenobjectiveCoveragegets a new or better value, and reset inreset(). It is declared before the static block that callsreset(). Nothing reads it except the new methods.Command: one new value, appended at the end.AgentController/ServerController: one new case and one new method, following the existing request/response pattern.SutController/EmbeddedSutController/ExternalSutController: one new method each. It is not abstract inSutController.None of this changes behaviour for other problem types. Happy to split the instrumentation part into its own PR if you would rather.
Not here
Other completion strategies (handler return, consumer lag) need transport-level tracking and the Kafka helper module, and come later in the stack.
Testing
ExecutionTracerTest(new, client-java/instrumentation), 5 tests: zero at start; first reach counts; re-running the same code does not; only a better value counts; reset starts from zero.ServerControllerTest(new), 3 tests, with a realAgentControllerconnected over a local socket in the same JVM: the count round-trips;NEW_TESTresets it in the agent; it interleaves with other commands without desynchronising replies.AsyncApiSettleTest, 16 tests (10 new), on a fake clock: a SUT with stable coverage let go afterstableMs; activity extends the wait to a stable window after the last move; the deadline caps a SUT whose coverage never settles; an unreadable, lost or throwing count means the whole window, or the rest of it; no or non-positivestableMsmeans a fixed wait; a fixed wait never reads the count; an interrupt during the loop ends it withINTERRUPTED.EMControllerTest, 8 tests (3 new), throughnewActionwith the embedded direct read: a SUT reaching nothing new is let go atstableMs; one that keeps reaching new objectives (simulated throughExecutionTracer) is waited on until the deadline; without the agent the whole window is waited.AsyncApiFitnessTest, 3 new tests: completion and stable window sent in white-box, not in black-box, and why the wait ended is kept on the result.EMConfigTest, 3 new tests: the cross-option check and the defaults, the settle options ignored in black-box, and a stable period no shorter than the window allowed.NcsKafkaEMTest.testRunEMWithCoverageStable, a new E2E with a 5000 ms window and a 100 ms stable period: no reply missed, every result settled, at least one wait ended onCOVERAGE_STABLE, none interrupted.AsyncApi*Test,ConfigToMarkdownTest,EMConfigTest: 275 tests, 0 failures. client-java/instrumentation, whole module: 737 tests, 0 failures (1 skipped, as before). client-java/controller:AsyncApiSettleTest,EMControllerTest,AsyncApiProblemTest: 31 tests, 0 failures.NcsKafkaEMTest: 5 tests, 0 failures.