Skip to content

AsyncAPI 3.x: settle after a message - #1871

Open
LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-taintfrom
feature/asyncapi-settle
Open

LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-taintfrom
feature/asyncapi-settle

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fourth in the AsyncAPI white-box stack, on top of #1870. This lets a white-box run keep each published message's action open for a while after the driver returns, so that the code the service runs to handle the message is credited to that action.

Why

A message-driven service handles a message on a thread of its own, and nothing tells the driver when that handling is over. For a fire-and-forget operation the driver returns as soon as the broker acknowledges the publish; even after a reply, the handler may still be running. The core then moves on to the next action, or collects the test's results, while the service is still at work: what it reaches is credited to the wrong action, or to none.

What it adds

  • AsyncApiActionDto.settleMs: how long to keep the action open after the driver returns. Null, zero or negative mean no wait.
  • AsyncApiReplyDto.settledMs: how long the controller actually waited. Filled in by the controller, never by the driver. Null when no wait was asked for, or when the message was not published.
  • AsyncApiSettle: the wait itself. The clock and the sleep sit behind two package-private interfaces, so its tests never depend on timing. An interrupt ends the wait and is left set for the thread's owner.
  • --asyncApiSettleMs: experimental, 0 (off) by default. AsyncApiFitness sends it in white-box mode only, since a black-box run collects no code coverage to credit. What was waited is kept on AsyncApiCallResult.
  • Generated tests (Java and Kotlin) sleep the whole window after every published message the search kept open, with or without a reply, since where the search ended early it did so on what the instrumentation saw, which a generated test has not got. No sleep after a failed publish.

Where the wait lives

In EMController.newAction, right after executeAsyncApiAction returns. That way every driver gets it without implementing anything, and it costs no extra round trip. Waiting in the core instead would have needed one more HTTP call per action, and the next PR needs to poll coverage while it waits, which is cheap on the controller side.

The wait is not inside a noKillSwitch. SutController.newAction turns the switch off for the action, and it has to stay off for the service's threads during the wait. A noKillSwitch around the controller's own thread would not change that. The waiting thread runs controller code, which is never instrumented (org.evomaster. is in the skip list), so the kill switch never interrupts it.

A message that could not be published is not waited on, whether the driver threw or reported published false or unset. That is a broken setup, and the test stops there as before.

What this changes outside AsyncAPI

EMController has a new package-private constructor that takes the AsyncApiSettle, so tests can wait on a clock they control. The public constructor is unchanged. The asyncApiCall branch of newAction is the only code path that changes, and only when settleMs is set.

Not here

The wait is a plain sleep for the whole window. Ending it early, once coverage stops moving, comes in the next PR.

Testing

  • AsyncApiSettleTest (client-java/controller), 6 tests: waits as long as asked; no wait or record for null/0/negative; an interrupt ends the wait, keeps the flag and records the time actually waited; the driver's fields are left alone; null arguments; the real clock and sleeper record a wait (no assertion on duration).
  • EMControllerTest, 5 tests, calling newAction directly with a fake AsyncAPI driver: the wait starts only after the driver returns; no wait when none is asked for; a failed publish (500) is not waited on, nor one the driver reports as not published or leaves unset; the kill switch is off during the wait.
  • AsyncApiFitnessTest, 5 new tests: no window by default; sent in white-box; not sent in black-box; settledMs kept on the result; absent when the controller reports none.
  • AsyncApiTestCaseWriterTest, 3 new tests: the generated test sleeps the whole window after a fire-and-forget message and after one with a reply or none, and compiles in Java and Kotlin; no sleep where the search kept nothing open or the message was not published.
  • NcsKafkaEMTest.testRunEMWithASettleWindow, a new E2E with a 200 ms window: messages are still answered, none go unanswered or unpublished, and every result records a settle.
  • Core: AsyncApi*Test, ConfigToMarkdownTest, EMConfigTest: 269 tests, 0 failures. client-java: AsyncApiSettleTest, EMControllerTest, AsyncApiProblemTest: 18 tests, 0 failures. NcsKafkaEMTest: 4 tests, 0 failures.

A message-driven service handles a message on a thread of its own, and
nothing tells the driver when that handling is over. For a
fire-and-forget operation the driver returns as soon as the broker
acknowledges the publish; even after a reply, the handler may still be
running. The core then moves on to the next action, or collects the
test's results, while the service is still at work: the code it reaches
is credited to the wrong action, or to none.

This adds a settle window: once the driver has returned, the controller
keeps the action open for a time the core asks for. The wait lives in
EMController.newAction, right after executeAsyncApiAction, so that
every driver gets it without implementing anything, and no extra round
trip is needed. It is a plain wait for now; how long to wait is the
only thing the core decides.

 - AsyncApiActionDto.settleMs: how long to keep the action open. Null,
   zero or negative mean no wait.
 - AsyncApiReplyDto.settledMs: how long the controller actually waited,
   filled in by the controller, not by the driver. Null when no wait
   was asked for, or when the message was not published.
 - AsyncApiSettle: the wait itself, with the clock and the sleep behind
   two small interfaces so that its tests do not depend on timing. An
   interrupt ends the wait and is left set for the thread's owner.
 - --asyncApiSettleMs, experimental, 0 (off) by default. AsyncApiFitness
   sends it in white-box mode only: a black-box run collects no code
   coverage to credit. What was waited is kept on AsyncApiCallResult.
 - Generated tests (Java and Kotlin) sleep the whole window after every
   published message the search kept open, with or without a reply:
   where the search ended early it did so on what the instrumentation
   saw, which a generated test has not got. No sleep after a failed
   publish.

Worth flagging for review: the wait is not inside a noKillSwitch. The
switch is turned off for the action by SutController.newAction, and has
to stay off for the service's threads during the wait, which a
noKillSwitch around the controller's own thread would not change. The
waiting thread runs controller code, which is never instrumented, so
the kill switch never interrupts it.

A message that could not be published is not waited on, whether the
driver threw or reported published as false or unset: it is a broken
setup, and the test stops there as before.

Nothing changes unless --asyncApiSettleMs is set. No shared REST code
is touched.
@jgaleotti
jgaleotti requested a review from arcuri82 October 13, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants