Repository navigation
AsyncAPI 3.x: a fire-and-forget Spring Kafka E2E - #1874
Draft
LautaroPetaccio wants to merge 1 commit into
Draft
LautaroPetaccio wants to merge 1 commit into
LautaroPetaccio wants to merge 1 commit into
Conversation
LautaroPetaccio
added this pull request to stack #1812
October 11, 2026 19:08
NCS answers every message, so what it did is visible from outside. Most message-driven services answer nothing: a listener takes the message, and whatever it leads to happens inside the service. This adds an E2E over such a service, where every assertion is about coverage, since nothing else says whether the search got anywhere. The SUT is a small shop back end, written for the test, consuming three topics with @KafkaListener and Spring Kafka's JsonDeserializer into typed objects, as described by an AsyncAPI 3.0 document written for it: - placeOrder: express shipping and an urgent priority are behind strings the document leaves free, one compared with equals, the other read with Priority.valueOf. Random strings do not get there; taint analysis does. - requestInvoice: the listener hands the rendering to an executor, which renders the invoice 150 ms after the listener has returned. - adjustStock: the stock of each product is kept in an in-memory H2 database through JdbcTemplate. Selling beyond the stock needs an earlier message of the same test to have recorded the product. The code each of these reaches is in a class of its own, so the test asks whether that class was covered rather than naming lines. The driver creates the topics, waits until every listener has been assigned its topic before the search starts, and publishes, with no key as the generated tests do, without waiting for anything back. Between tests it waits until every listener's group has committed up to the end of its partitions, so that no message of the previous test is handled after the reset, then for the invoice executor to run everything handed to it, and then empties the stock table. The listeners read from the earliest offset: the topics are created empty right before the service starts, so nothing is skipped however long joining the group takes. ShopKafkaEMTest: - testRunEM: with taint (on by default in white-box mode) and a stable-coverage settle window, the express and urgent branches are covered, some wait ends on stable coverage, and the values that got there are in the generated suite, which is compiled and run. - testRunEMWithASettleWindow: with a fixed window, the invoice rendered on the executor thread and the oversold stock are covered, and some test credits each to the operation that leads there (an oversold stock to an adjustStock with at most one earlier one). Worth flagging for review: - A new module, spring-asyncapi-kafka-listener, rather than more of spring-asyncapi-kafka: with spring-kafka on the classpath, Spring Boot's Kafka auto-configuration would apply to NCS too. - spring-kafka 2.7.6, the version Spring Boot 2.5.4 manages, runs against the kafka-clients 3.7.2 managed in the root pom. Every Consumer method its listener container calls is still in 3.7.2. Its version property moves from client-java/pom.xml to the root pom, so this module and the client-java/instrumentation tests share one definition. - Both tests need a settle window. Without one, the results of a test are collected as soon as its last message is on the broker, often before the service has read it. - The fixed window is what catches the executor: a stable period shorter than the hand-off would let the action go before the invoice is rendered. That is the known limit of COVERAGE_STABLE, and nothing asserts on it either way. - The driver hands EvoMaster no database: the SQL the service runs is seen by the instrumentation, but there are no SQL heuristics here. - The record values are read through ObjectReader.readValue(byte[]), so the ObjectReader replacement runs on every message. Nothing in the core uses the DTOs it registers for AsyncAPI yet, so nothing asserts on them. Taint does not depend on them: the comparisons happen in the service's own code. AsyncApiTestBase gets initAndDebug and assertClassCovered, and the module's target folder is ignored like the others. No core or client-java code is changed.
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-fire-and-forget
branch
from
October 12, 2026 02:54
ee683c0 to
d38b99d
Compare
LautaroPetaccio
marked this pull request as draft
October 12, 2026 12:45
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-fire-and-forget
branch
from
October 17, 2026 23:48
d38b99d to
0f38839
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seventh in the AsyncAPI white-box stack, on top of #1873. An E2E over a service that never answers: every operation is fire-and-forget, so whether the search got anywhere is only known from the service's coverage, and that is what it asserts on.
This one is in
core-tests, plus one Maven property moved to the root pom — no core or client-java code changes.What it adds
A new module,
core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener, with a small shop back end written for the test. It consumes three topics with@KafkaListener, and Spring Kafka'sJsonDeserializerreads each record into a typed object.asyncapi/shop-kafka.yaml(AsyncAPI 3.0, marked as artificial) describes it.placeOrdershippingisEXPRESS(equals), pages on-call whenpriorityisURGENT(Priority.valueOf)requestInvoiceadjustStockJdbcTemplate; selling beyond the stock of a product already recorded is its own branchThe code each of these reaches is in a class of its own (
ExpressDispatch,UrgentEscalation,InvoiceRenderer,OversoldStock), none of them a Spring bean, so nothing runs it at start-up.ShopKafkaController: creates the topics, waits until every listener has been assigned its topic before the search starts, publishes with no key, as the generated tests do, and returns as soon as the broker has the message. It sits outsidecom.foo.asyncapi.shop, so its own lines are neither targets nor activity thatCOVERAGE_STABLEwould mistake for the service's.ShopKafkaEMTest: two searches, below.AsyncApiTestBase:initAndDebug, which keeps the injector, andassertClassCovered, which looks a class target up in theIdMapper. Its failure message lists the SUT classes that were covered.Things that are less obvious than they look
COVERAGE_STABLE, a stable period shorter than the hand-off lets the action go before the invoice is rendered. That is its known limit. Nothing here asserts on it either way, since that would be a timing assertion.ObjectReader.readValue(byte[]), so the replacement from AsyncAPI 3.x: Jackson reads from bytes, for message payloads #1873 runs on every message. Nothing in the core uses the DTOs it registers for AsyncAPI yet, so nothing asserts on them. Taint does not depend on them either, because the comparisons happen in the service's own code.What this changes outside AsyncAPI
pom.xml: thespring-kafka.versionproperty,2.7.6, moved fromclient-java/pom.xml, so this module and the client-java/instrumentation tests share one definition. That is the version Spring Boot 2.5.4 manages, the Boot version of the other e2e modules; spring-kafka 3 needs Spring 6. It runs against thekafka-clients3.7.2 managed in the root pom: everyConsumermethod its listener container calls is still in 3.7.2 (checked withjavap).core-tests/e2e-tests/spring/pom.xml: the new module is registered;.gitignore: itstargetfolder, like every other module's.spring-asyncapi-kafkabecause, with spring-kafka on the classpath, Spring Boot's Kafka auto-configuration would apply to NCS too. NCS consumes with a plainKafkaConsumer. Its tests are skipped withskipAsyncApi, so CI runs them in the existingAsyncApiTestsjob.Not here
Testing
ShopKafkaEMTest(new), 2 tests:testRunEM: taint plus aCOVERAGE_STABLEwindow (2000 ms, stable 100 ms), 300 messages, tests of at most 3 messages. AssertsExpressDispatchandUrgentEscalationare covered, no publish failed, at least one wait ended onCOVERAGE_STABLE, andEXPRESS/URGENTappear in the generated Kotlin suite, which is compiled and run.testRunEMWithASettleWindow: aFIXED500 ms window, 150 messages. AssertsInvoiceRendererandOversoldStockare covered, every message was settled, and some test credits each class to the operation that leads there:InvoiceRendererto arequestInvoice,OversoldStockto anadjustStockwith at most one earlieradjustStockin the test.NcsKafkaEMTest: 5 tests, 0 failures, unchanged by the two helpers added to the base class.