AsyncAPI 3.x: an E2E test, NCS driven through Kafka - #1760
Draft
LautaroPetaccio wants to merge 17 commits into
Draft
LautaroPetaccio wants to merge 17 commits into
LautaroPetaccio wants to merge 17 commits into
Conversation
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 14, 2026 11:59
4b4a66a to
64b5a5a
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-examples
branch
from
September 14, 2026 22:17
5d3566d to
66440f3
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 14, 2026 22:19
64b5a5a to
e69f386
Compare
Everything so far is in core and reads a document. This is the other side: what a driver must provide so that the core can drive an AsyncAPI service without knowing anything about brokers. AsyncApiProblem declares that the SUT is driven by messages. It carries only the document -- a location to fetch it from, or the text itself. The connection to the broker stays in the driver and never crosses, which is what keeps the core free of any broker library. Note it may be declared with the document inline, which REST has no real need for. A REST service usually serves its own contract over HTTP; a service that speaks only Kafka has no endpoint to serve anything from, so its document is far more likely to be a file shipped beside it. SutController.executeAsyncApiAction is where a driver publishes one message and, when a reply is expected, waits for the one that answers it. It is the counterpart of executeAction for RPC. It has a default that throws rather than being abstract, so that adding it does not break every existing driver; the message says what to override. Only publish and await are protocol-specific, and they never leave the driver. The driver is asked to report what happened, not to judge it: AsyncApiReplyDto distinguishes published-with-no-reply-expected, a reply that arrived, silence within the window, and a failure to publish at all. The last of those is a broken setup rather than a finding about the service, which is why it is kept apart from silence. Deciding what an outcome means is the core's job, so that it means the same thing whatever the transport. Also carried in the reply is whether the correlation id came back. Whether correlation works cannot be read off a contract, since echoing the id is the service's own behaviour, so it is established by watching for it. Wiring: a field on SutInfoDto, a field on ActionDto, and two branches in EMController.
Review feedback: CORRELATION_IN_HEADER and CORRELATION_IN_PAYLOAD closed the class, after every instance field. docs/for_developers.md puts constants first.
Review feedback, and docs/for_developers.md: a Map field says what its key and value are. Both header maps go from header name to its text.
A null list failed inside addAll anyway; the check now sits where the method starts, named, as docs/for_developers.md asks of public methods.
Review feedback: replyTimeoutMs is null when no reply is expected, and waitedMs when the driver did not wait at all, which the javadoc left unsaid.
Review feedback: it echoes ActionDto.index, which newAction has already dereferenced by the time the reply is built, so an int says what the Integer could only promise.
LautaroPetaccio
force-pushed
the
feature/asyncapi-examples
branch
from
September 15, 2026 16:29
66440f3 to
179e5d3
Compare
ProblemType.ASYNCAPI, experimental like the other recent types. Main infers it from SutInfoDto.asyncApiProblem, picks its algorithm key, and stops with a clear message where the module will go: the sampler and fitness that complete it are the next changes. The one decision here is EMConfig.usesDriver(). Every other type reaches the driver only in white-box mode, or in black-box experiments. An AsyncAPI service has no universal wire to point at, so the driver holds the broker connection even when the service is a black box. The places in Main and Statistics that asked "is there a driver" now ask the config, instead of each spelling the rule out.
AsyncApiSampler asks the driver where the document is, reads it, builds one action per publishable message, and samples sequences of them. The first individuals it hands out publish one message each, one per operation, so every operation is tried before the search starts combining them. One sampler for both modes. Where REST and GraphQL fetch the schema themselves in black-box mode, here the document comes through the driver either way, as the driver is what holds the connection to the broker. Seeded test cases are not supported yet, and say so.
Review: what the parser skips was reported to the user and kept for the final report, but nothing checked it; nor that the single-message individuals return after a reset, nor the two other ways start-up can fail. Four tests, and the helper that builds the injector is shared.
Four TODOs, where someone will next look: no AsyncAPI-specific smart strategy yet, message examples parsed but unused, no seeding format, and a method shared verbatim with the REST sampler.
RemoteController.executeNewAsyncApiActionAndGetReply PUTs the action to the driver and reads back an AsyncApiReplyDto, as its RPC counterpart does. It has a default that throws, so the controllers that never publish need not say so one by one. EMConfig gains asyncApiReplyTimeoutMs, experimental, and a constraint: there is no test writer for AsyncAPI yet, so a run must say --createTests false rather than search for an hour and fail at the end. The two tests that already parsed --problemType ASYNCAPI say so now. Two experimental fault categories: a promised reply that never arrives, and a reply matching none of the messages the contract declares.
AsyncApiBlackBoxFitness publishes each message of a test through the driver and turns what comes back into targets: for every operation, what publishing to it was seen to do, and, when a reply came back, which of the messages the contract declares for the reply it was. A contract listing a result and an error thus gives the search two things to reach, which is the AsyncAPI analogue of REST's (status x endpoint). AsyncApiReplyClassifier is what recognises a reply: a structural match against each declared payload schema, reading the parts of JSON Schema that tell one message from another and giving the benefit of the doubt on the rest. The most specific match wins. AsyncApiModule binds it all, with the driver bound unconditionally, and Main uses it in place of the message it showed until now. One test runs a whole MIO search against a stand-in for the NCS service and sees both declared replies of an operation covered.
Review: with room for exactly two messages, a one-message test was mutated by removal, leaving a test that publishes nothing; and a test could never grow to the maximum the user allowed, only to one less. Both bounds are now stated as such, and AsyncApiStructureMutatorTest holds them. Seeding test cases is refused at start-up, like writing them, instead of failing inside the sampler. The whole-search suite is named after the class it exercises, AsyncApiModule, and the three suites that build the injector share how they do it.
Review: the ':' joining a target id and the '-' inside a correlation id were spelled out where used; they are constants now, and the two kinds of target id are built in one place each. The remote controller named its queryFromDatabase parameter four times over, once in the new call and three in the ones it mirrors; it is a constant now. The fake driver's one field comes before its companion.
A document's message examples are what its author knows the service accepts. A service that silently drops what it does not recognise may never answer a sampled payload, so with --probAsyncApiExamples the first example of a message is offered as a whole value beside the schema-derived genes, at that probability. Off by default, like every new feature. It reuses what REST already does with a schema's 'example': the example is put on the message's own copy of the payload schema, and the gene builder turns it into the same choice it offers REST. Only the first example is used: the OpenAPI parser the genes go through reads the schemas as 3.0, which drops the plural 'examples'. Scalars are left alone, as the builder would complain about 'example' on them. A headers example loses the field the correlation id is stamped into, as the headers gene did before it.
The first search against a real broker. The SUT is the NCS case study re-expressed as a Kafka service: six request topics answered on six reply topics with a result or an error, as ncs-kafka.yaml describes. It speaks only Kafka; there is no HTTP endpoint. Its routines are ported from EMB's NCS and cited as such. The driver owns the broker, a testcontainers apache/kafka, and is the reference implementation of SutController.executeAsyncApiAction: publish with the correlation id where the document says it goes, wait for the reply that carries it back, report and do not judge. kafka-clients and testcontainers-kafka are used only here, in the test tree, and are managed in the root pom like everything else. The test asserts that every operation answers and that expint and gammq each reach both their declared replies. Those two are the operations whose rejected inputs the schema does not rule out; bessj's and remainder's bounds are in the schema, so their errors need invalid data, which the search does not publish yet. 662 targets in 26 seconds, on two runs.
LautaroPetaccio
force-pushed
the
feature/asyncapi-examples
branch
from
September 15, 2026 23:46
179e5d3 to
2495b28
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-e2e-kafka
branch
from
September 15, 2026 23:46
e69f386 to
8e17ab8
Compare
LautaroPetaccio
force-pushed
the
feature/asyncapi-examples
branch
from
September 16, 2026 01:16
2495b28 to
82d1889
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourteenth in the AsyncAPI stack, on top of #1754. Draft, like the three below it. This is the first search against a real broker, and the E2E test
for_developers.mdasks every feature to have.The SUT — NCS over Kafka
The NCS case study re-expressed as a message-driven service: six request topics (
ncs.<op>.request), each answered on a reply topic with a result message or, for the inputs the REST version answers with a 400, an error message — exactly whatncs-kafka.yaml(already in core's test resources) describes. The routines are ported from EMB'sorg.restncs.impand cited at the top of each file; the rejection rules mirrorNcsRest. A Spring Boot application with no web layer: it speaks only Kafka, which is the point.One deliberate deviation: a result that is
NaNor infinite is answered as an error. JSON has no such numbers, and a string where the contract promises a number would be a false fault.The driver — the reference
executeAsyncApiActionNcsKafkaControllerowns the broker (testcontainers,apache/kafka:3.8.0, KRaft), creates the twelve topics up front, and implements the hook exactly as #1738's javadoc sketches it:replyTimeoutMsruns out; a reply carrying someone else's id is skipped, one carrying none is taken and reported ascorrelationMatched = false;AsyncApiReplyDtoand judge nothing.kafka-clientsandtestcontainers-kafkaappear only in this test module, managed in the rootpom.xmlas the doc requires. Nothing is added to any shipped jar.The test
AsyncApiTestBasejoins the other bases ine2e-tests-utils:initAndRun, and helpers that readAsyncApiCallResultoff the solution.NcsKafkaEMTestruns 300 action evaluations and asserts:checkTriangle,bessjandremainderreached their result reply;expintandgammqreached both their result and their error — the two operations whose rejected inputs (x < 0;a ≤ 0 || x < 0) the schema does not rule out.bessj's order andremainder's operands are bounded in the schema, so their error replies need invalid data, which the search does not publish yet — worth knowing, and a natural next lever;NO_REPLY, noPUBLISH_FAILED.Because the driver is embedded, the SUT is instrumented, so the search covers lines and branches of a message-driven service as well: 662 targets in 26 seconds, identical on two runs.
Not here
The
createTests falseconstraint still stands, sorunTestHandlingFlakyis used rather than the compilation variant. Once the test writer exists, this is where its output gets compiled and run too.