Skip to content

AsyncAPI 3.x: a fire-and-forget Spring Kafka E2E - #1874

Draft
LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-jackson-bytesfrom
feature/asyncapi-e2e-fire-and-forget
Draft

LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-jackson-bytesfrom
feature/asyncapi-e2e-fire-and-forget

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Seventh in the AsyncAPI white-box stack, on top of #1873. An E2E over a service that never answers: every operation is fire-and-forget, so whether the search got anywhere is only known from the service's coverage, and that is what it asserts on.

This one is in core-tests, plus one Maven property moved to the root pom — no core or client-java code changes.

What it adds

A new module, core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener, with a small shop back end written for the test. It consumes three topics with @KafkaListener, and Spring Kafka's JsonDeserializer reads each record into a typed object. asyncapi/shop-kafka.yaml (AsyncAPI 3.0, marked as artificial) describes it.

Operation What the listener does What it is there for
placeOrder Books a courier when shipping is EXPRESS (equals), pages on-call when priority is URGENT (Priority.valueOf) Both strings are free in the document, so random values do not get there; taint analysis does
requestInvoice Hands the rendering to an executor, which renders the invoice 150 ms after the listener has returned Work that the handler returning, or the offset being committed, says nothing about
adjustStock Keeps the stock of each product in an in-memory H2 database through JdbcTemplate; selling beyond the stock of a product already recorded is its own branch State written by one message and read by the next one in the same test

The code each of these reaches is in a class of its own (ExpressDispatch, UrgentEscalation, InvoiceRenderer, OversoldStock), none of them a Spring bean, so nothing runs it at start-up.

  • ShopKafkaController: creates the topics, waits until every listener has been assigned its topic before the search starts, publishes with no key, as the generated tests do, and returns as soon as the broker has the message. It sits outside com.foo.asyncapi.shop, so its own lines are neither targets nor activity that COVERAGE_STABLE would mistake for the service's.
  • ShopKafkaEMTest: two searches, below.
  • AsyncApiTestBase: initAndDebug, which keeps the injector, and assertClassCovered, which looks a class target up in the IdMapper. Its failure message lists the SUT classes that were covered.

Things that are less obvious than they look

  • Both tests need a settle window. Without one, the results of a test are collected as soon as its last message is on the broker, often before the service has read it. For taint, what the service compared that message's strings with is lost too.
  • The fixed window is what catches the executor. With COVERAGE_STABLE, a stable period shorter than the hand-off lets the action go before the invoice is rendered. That is its known limit. Nothing here asserts on it either way, since that would be a timing assertion.
  • No stale records. Between tests, the driver waits until every listener's group has committed up to the end of the partitions it was assigned, then empties the stock table. A message of the previous test can therefore not be handled after the reset. It also waits for the invoice executor to run everything handed to it, so a rendering of the previous test is not covered, and credited, in the next one. The listeners read from the earliest offset: the topics are created empty right before the service starts, so nothing is skipped however long joining the group takes.
  • Assertions are about classes, not lines. A class target is covered as soon as any of its lines runs. With one class per branch of interest, the test does not depend on how the sources are laid out.
  • Jackson from bytes. Record values are read with ObjectReader.readValue(byte[]), so the replacement from AsyncAPI 3.x: Jackson reads from bytes, for message payloads #1873 runs on every message. Nothing in the core uses the DTOs it registers for AsyncAPI yet, so nothing asserts on them. Taint does not depend on them either, because the comparisons happen in the service's own code.
  • No database is handed to EvoMaster. The instrumentation sees the SQL the service runs, but there are no SQL heuristics here. The database is only there to keep state from one message to the next.

What this changes outside AsyncAPI

  • Root pom.xml: the spring-kafka.version property, 2.7.6, moved from client-java/pom.xml, so this module and the client-java/instrumentation tests share one definition. That is the version Spring Boot 2.5.4 manages, the Boot version of the other e2e modules; spring-kafka 3 needs Spring 6. It runs against the kafka-clients 3.7.2 managed in the root pom: every Consumer method its listener container calls is still in 3.7.2 (checked with javap).
  • core-tests/e2e-tests/spring/pom.xml: the new module is registered; .gitignore: its target folder, like every other module's.
  • The module is separate from spring-asyncapi-kafka because, with spring-kafka on the classpath, Spring Boot's Kafka auto-configuration would apply to NCS too. NCS consumes with a plain KafkaConsumer. Its tests are skipped with skipAsyncApi, so CI runs them in the existing AsyncApiTests job.

Not here

  • The generated tests publish and move on: nothing in them waits for the service, and nothing asserts on what it did. That needs the emitted-message work that comes later in the stack.
  • No assertion that something is not covered without a window.

Testing

  • ShopKafkaEMTest (new), 2 tests:
    • testRunEM: taint plus a COVERAGE_STABLE window (2000 ms, stable 100 ms), 300 messages, tests of at most 3 messages. Asserts ExpressDispatch and UrgentEscalation are covered, no publish failed, at least one wait ended on COVERAGE_STABLE, and EXPRESS/URGENT appear in the generated Kotlin suite, which is compiled and run.
    • testRunEMWithASettleWindow: a FIXED 500 ms window, 150 messages. Asserts InvoiceRenderer and OversoldStock are covered, every message was settled, and some test credits each class to the operation that leads there: InvoiceRenderer to a requestInvoice, OversoldStock to an adjustStock with at most one earlier adjustStock in the test.
    • Three runs of the class: 0 failures each time.
  • NcsKafkaEMTest: 5 tests, 0 failures, unchanged by the two helpers added to the base class.

NCS answers every message, so what it did is visible from outside. Most
message-driven services answer nothing: a listener takes the message,
and whatever it leads to happens inside the service. This adds an E2E
over such a service, where every assertion is about coverage, since
nothing else says whether the search got anywhere.

The SUT is a small shop back end, written for the test, consuming three
topics with @KafkaListener and Spring Kafka's JsonDeserializer into
typed objects, as described by an AsyncAPI 3.0 document written for it:
 - placeOrder: express shipping and an urgent priority are behind
   strings the document leaves free, one compared with equals, the
   other read with Priority.valueOf. Random strings do not get there;
   taint analysis does.
 - requestInvoice: the listener hands the rendering to an executor,
   which renders the invoice 150 ms after the listener has returned.
 - adjustStock: the stock of each product is kept in an in-memory H2
   database through JdbcTemplate. Selling beyond the stock needs an
   earlier message of the same test to have recorded the product.
The code each of these reaches is in a class of its own, so the test
asks whether that class was covered rather than naming lines.

The driver creates the topics, waits until every listener has been
assigned its topic before the search starts, and publishes, with no key
as the generated tests do, without waiting for anything back. Between
tests it waits until every listener's group has committed up to the end
of its partitions, so that no message of the previous test is handled
after the reset, then for the invoice executor to run everything handed
to it, and then empties the stock table. The listeners read from the
earliest offset: the topics are created empty right before the service
starts, so nothing is skipped however long joining the group takes.

ShopKafkaEMTest:
 - testRunEM: with taint (on by default in white-box mode) and a
   stable-coverage settle window, the express and urgent branches
   are covered, some wait ends on stable coverage, and the values that
   got there are in the generated suite, which is compiled and run.
 - testRunEMWithASettleWindow: with a fixed window, the invoice
   rendered on the executor thread and the oversold stock are covered,
   and some test credits each to the operation that leads there (an
   oversold stock to an adjustStock with at most one earlier one).

Worth flagging for review:
 - A new module, spring-asyncapi-kafka-listener, rather than more of
   spring-asyncapi-kafka: with spring-kafka on the classpath, Spring
   Boot's Kafka auto-configuration would apply to NCS too.
 - spring-kafka 2.7.6, the version Spring Boot 2.5.4 manages, runs
   against the kafka-clients 3.7.2 managed in the root pom. Every
   Consumer method its listener container calls is still in 3.7.2.
   Its version property moves from client-java/pom.xml to the root
   pom, so this module and the client-java/instrumentation tests
   share one definition.
 - Both tests need a settle window. Without one, the results of a test
   are collected as soon as its last message is on the broker, often
   before the service has read it.
 - The fixed window is what catches the executor: a stable period
   shorter than the hand-off would let the action go before the
   invoice is rendered. That is the known limit of
   COVERAGE_STABLE, and nothing asserts on it either way.
 - The driver hands EvoMaster no database: the SQL the service runs is
   seen by the instrumentation, but there are no SQL heuristics here.
 - The record values are read through ObjectReader.readValue(byte[]),
   so the ObjectReader replacement runs on every message. Nothing in
   the core uses the DTOs it registers for AsyncAPI yet, so nothing
   asserts on them. Taint does not depend on
   them: the comparisons happen in the service's own code.

AsyncApiTestBase gets initAndDebug and assertClassCovered, and the
module's target folder is ignored like the others. No core or
client-java code is changed.
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-fire-and-forget branch from ee683c0 to d38b99d Compare October 12, 2026 02:54
@LautaroPetaccio
LautaroPetaccio marked this pull request as draft October 12, 2026 12:45
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-e2e-fire-and-forget branch from d38b99d to 0f38839 Compare October 17, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant