Repository navigation
AsyncAPI 3.x: settle after a message - #1871
Open
LautaroPetaccio wants to merge 1 commit into
Open
LautaroPetaccio wants to merge 1 commit into
LautaroPetaccio wants to merge 1 commit into
Conversation
LautaroPetaccio
added this pull request to stack #1812
October 11, 2026 19:08
A message-driven service handles a message on a thread of its own, and nothing tells the driver when that handling is over. For a fire-and-forget operation the driver returns as soon as the broker acknowledges the publish; even after a reply, the handler may still be running. The core then moves on to the next action, or collects the test's results, while the service is still at work: the code it reaches is credited to the wrong action, or to none. This adds a settle window: once the driver has returned, the controller keeps the action open for a time the core asks for. The wait lives in EMController.newAction, right after executeAsyncApiAction, so that every driver gets it without implementing anything, and no extra round trip is needed. It is a plain wait for now; how long to wait is the only thing the core decides. - AsyncApiActionDto.settleMs: how long to keep the action open. Null, zero or negative mean no wait. - AsyncApiReplyDto.settledMs: how long the controller actually waited, filled in by the controller, not by the driver. Null when no wait was asked for, or when the message was not published. - AsyncApiSettle: the wait itself, with the clock and the sleep behind two small interfaces so that its tests do not depend on timing. An interrupt ends the wait and is left set for the thread's owner. - --asyncApiSettleMs, experimental, 0 (off) by default. AsyncApiFitness sends it in white-box mode only: a black-box run collects no code coverage to credit. What was waited is kept on AsyncApiCallResult. - Generated tests (Java and Kotlin) sleep the whole window after every published message the search kept open, with or without a reply: where the search ended early it did so on what the instrumentation saw, which a generated test has not got. No sleep after a failed publish. Worth flagging for review: the wait is not inside a noKillSwitch. The switch is turned off for the action by SutController.newAction, and has to stay off for the service's threads during the wait, which a noKillSwitch around the controller's own thread would not change. The waiting thread runs controller code, which is never instrumented, so the kill switch never interrupts it. A message that could not be published is not waited on, whether the driver threw or reported published as false or unset: it is a broken setup, and the test stops there as before. Nothing changes unless --asyncApiSettleMs is set. No shared REST code is touched.
LautaroPetaccio
force-pushed
the
feature/asyncapi-settle
branch
from
October 12, 2026 02:54
d5c9357 to
553534d
Compare
jgaleotti
approved these changes
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth in the AsyncAPI white-box stack, on top of #1870. This lets a white-box run keep each published message's action open for a while after the driver returns, so that the code the service runs to handle the message is credited to that action.
Why
A message-driven service handles a message on a thread of its own, and nothing tells the driver when that handling is over. For a fire-and-forget operation the driver returns as soon as the broker acknowledges the publish; even after a reply, the handler may still be running. The core then moves on to the next action, or collects the test's results, while the service is still at work: what it reaches is credited to the wrong action, or to none.
What it adds
AsyncApiActionDto.settleMs: how long to keep the action open after the driver returns. Null, zero or negative mean no wait.AsyncApiReplyDto.settledMs: how long the controller actually waited. Filled in by the controller, never by the driver. Null when no wait was asked for, or when the message was not published.AsyncApiSettle: the wait itself. The clock and the sleep sit behind two package-private interfaces, so its tests never depend on timing. An interrupt ends the wait and is left set for the thread's owner.--asyncApiSettleMs: experimental,0(off) by default.AsyncApiFitnesssends it in white-box mode only, since a black-box run collects no code coverage to credit. What was waited is kept onAsyncApiCallResult.Where the wait lives
In
EMController.newAction, right afterexecuteAsyncApiActionreturns. That way every driver gets it without implementing anything, and it costs no extra round trip. Waiting in the core instead would have needed one more HTTP call per action, and the next PR needs to poll coverage while it waits, which is cheap on the controller side.The wait is not inside a
noKillSwitch.SutController.newActionturns the switch off for the action, and it has to stay off for the service's threads during the wait. AnoKillSwitcharound the controller's own thread would not change that. The waiting thread runs controller code, which is never instrumented (org.evomaster.is in the skip list), so the kill switch never interrupts it.A message that could not be published is not waited on, whether the driver threw or reported
publishedfalse or unset. That is a broken setup, and the test stops there as before.What this changes outside AsyncAPI
EMControllerhas a new package-private constructor that takes theAsyncApiSettle, so tests can wait on a clock they control. The public constructor is unchanged. TheasyncApiCallbranch ofnewActionis the only code path that changes, and only whensettleMsis set.Not here
The wait is a plain sleep for the whole window. Ending it early, once coverage stops moving, comes in the next PR.
Testing
AsyncApiSettleTest(client-java/controller), 6 tests: waits as long as asked; no wait or record for null/0/negative; an interrupt ends the wait, keeps the flag and records the time actually waited; the driver's fields are left alone; null arguments; the real clock and sleeper record a wait (no assertion on duration).EMControllerTest, 5 tests, callingnewActiondirectly with a fake AsyncAPI driver: the wait starts only after the driver returns; no wait when none is asked for; a failed publish (500) is not waited on, nor one the driver reports as not published or leaves unset; the kill switch is off during the wait.AsyncApiFitnessTest, 5 new tests: no window by default; sent in white-box; not sent in black-box;settledMskept on the result; absent when the controller reports none.AsyncApiTestCaseWriterTest, 3 new tests: the generated test sleeps the whole window after a fire-and-forget message and after one with a reply or none, and compiles in Java and Kotlin; no sleep where the search kept nothing open or the message was not published.NcsKafkaEMTest.testRunEMWithASettleWindow, a new E2E with a 200 ms window: messages are still answered, none go unanswered or unpublished, and every result records a settle.AsyncApi*Test,ConfigToMarkdownTest,EMConfigTest: 269 tests, 0 failures. client-java:AsyncApiSettleTest,EMControllerTest,AsyncApiProblemTest: 18 tests, 0 failures.NcsKafkaEMTest: 4 tests, 0 failures.