AsyncAPI 3.x: write the test suites - #1809
Draft
LautaroPetaccio wants to merge 7 commits into
Draft
LautaroPetaccio wants to merge 7 commits into
LautaroPetaccio wants to merge 7 commits into
Conversation
A generated test cannot go through the driver: there would be no test without EvoMaster running. It cannot use a client from the core either, since a broker has no universal one and the transport code belongs in drivers. So the driver renders the publish and await as source lines while the search runs, and the writer pastes them, which is what RPC already does for an endpoint invocation. The core asks for them on the action, naming the language and the variable the reply has to land in, so two actions in one test cannot collide. It asks only when a test will be written. Around the pasted lines it adds what the driver cannot know: the operation and its outcome, that a promised reply arrived, and which declared message the classifier recognised it as. A driver that renders nothing produces a test that says so rather than an empty one. With this, --problemType ASYNCAPI no longer refuses to write tests. What replaces that refusal is a narrower one: only Java and Kotlin, because the lines being pasted are the driver's. For the same reason black-box no longer defaults the format to Python for AsyncAPI, which would have made an ordinary run fail at start-up: it has a driver in both modes, so the format comes from the driver as it does in white-box. The suite splitter read a status code off every result, which would have thrown for the first AsyncAPI individual written. It now branches on the result type, counting a message that went out and was answered as a success.
REST writes its own RestAssured calls rather than asking the driver for them, and Kafka can be written the same way: the document names the broker on the server, the topic on the channel or its binding, and the field the correlation id rides in. Nothing else is needed to publish the same message again. So a Kafka test now stands up a producer and a consumer of its own. It seeks to the end of the reply topic before publishing, so a reply older than the test is not taken for an answer to it, mints a fresh id each run and matches it on the way back. Core writes the text and links nothing: the dependency lands on the generated suite, as rest-assured does for REST. The driver's own rendering stays, and wins where it is present. It is what a transport the contract cannot describe needs: one whose correlation id rides inside the service's own message layout, where the document says nothing about where to put it. Neither available, and the test says so rather than being silently empty. What the search resolved is kept on the result, so none of it is worked out a second time here.
The REST writer calls its variables res_0 and body_0, so there is no reason for these to carry an asyncApi prefix on every one. They are now producer_0, consumer_0, record_0, correlationId_0 and reply_0, keyed off the reply variable the core names rather than off the action index, so the locals of one action always agree with the reply it leaves behind. A test name read verb, target, result in REST, and ours was missing the verb: publish_on_<operation>_returns_<message>. The two outcomes that are not a declared reply say what they are rather than naming an enum constant.
…once A REST action is a two-line RestAssured call. Publishing to a broker and correlating a reply needs properties, a producer, a consumer, a seek, a send and a poll loop, and repeating that per action buried the test in scaffolding: eighteen lines to say what a reader wanted in one. It is a method now, written once for the suite, so an action is the call and its assertion. It follows the SSRF assertion helper, which is emitted into the same part of the class and gated the same way, on whether the solution has anything that uses it. A suite that publishes over a transport the contract cannot describe, where the driver renders the lines instead, is not made to carry a Kafka dependency it never calls. The new hook takes the solution, which addExtraStaticVariables does not, because whether a member is needed is a property of the whole suite rather than of one action.
Every other problem type's naming strategy has a unit test asserting the exact strings it produces; this one had none. It does now, including that silence is named the same whether or not the experimental oracles are on. It is named by the outcome when they are off and by the fault's label when they are on, and a test's name must not change with a flag that is only about reporting. Two places deliberately depart from what the other writers do, and now say so where a reader meets them. The reply variable is not minted from the counter that gives REST res_0 and body_0, because the name has to be settled while the search runs, being part of what a driver renders against; the action's index is the only number in hand there. And the emitted Kafka code names its types in full instead of adding imports, because the suite's imports are decided before it is known whether anything will publish over Kafka, and adding them always would make every AsyncAPI suite need the dependency.
What the search published is now kept on the result -- broker, topics, the header the correlation id rode in, the payload -- so the writer has the whole call in hand and never reads the document again. A suite asks the driver where each server is, the way it asks for baseUrlOfSut, and keeps the answer in a variable per server. A broker started on a random port is then reachable from the generated test, which the document's own address would not be. The default keeps the address the document gave, so a black-box suite, which has no driver to ask, is unchanged. The reply is asserted on the way REST asserts on a response: the same enableBasicAssertions, the same fieldsToSkipInAssertions, the same size and collection limits, the same tolerance on doubles. Nothing is predicted from the schema; every assertion is a regression on what came back. Kafka is written in Java and Python as well as Kotlin, so the suite can be generated in any of the three the rest of EvoMaster offers. The fault line of a test's comment block moves up into TestCaseWriter, which is what the TODO beside it asked for, and the AsyncAPI block now lists its calls with their outcome where REST lists them with their status code.
write_driver gains the AsyncAPI section: what a driver has to answer for a message-driven service, and that one is needed even in black-box mode, which is the opposite of every other problem type and so worth saying twice. library_dependencies gains the two a generated Kafka suite compiles against, and options.md is regenerated for outputFormat, which now accepts the languages a Kafka suite can be written in.
This was referenced Sep 30, 2026
LautaroPetaccio
changed the title
LautaroPetaccio
added this pull request to stack #1812
September 30, 2026 17:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fifth of the AsyncAPI 3.x series, on top of #1754. A search over a message-driven service now leaves a test suite behind, as every other problem type does.
What a generated test looks like
The test talks to the broker with an ordinary
kafka-clientsproducer and consumer. It does not go through an EvoMaster driver at run time, the same way a REST test does not: once written, it stands on its own.How the publishing lines are written
Two ways, in this order:
enablePureRPCTestGenerationpath. It is what a transport the document cannot describe needs: one whose correlation id rides inside the service's own message layout, where nothing in core could know where to put it.kafka-clientsout of core: the text is emitted as text, and only the generated suite ever compiles against it.publishAndAwaitReplyis written once per suite, in the companion object, rather than inlined in every test. It seeks to the end of the reply topic before publishing and mints a fresh correlation id per call, so a reply left over from an earlier run is never taken for an answer to this one.Finding the broker
A generated test cannot point at the address in the document: that is where the service was deployed, not where a system started for testing is.
SutHandlergainsgetAsyncApiServerAddress(serverName), defaulting tonull, and the suite fills one variable per server before the tests, exactly asbaseUrlOfSutis filled. When there is no driver, or the driver says nothing, the document's address is used.Assertions
Regression on the observed reply, never a prediction from the schema: the same
enableBasicAssertions,fieldsToSkipInAssertions,maxResponseByteSize,maxAssertionForDataInCollectionand double tolerance the REST writer honours.The same suite as any other
The scaffolding is the shared writer's, and a test asserts that line by line against what the REST writer produces: of the 57 lines an EvoMaster suite opens with, 55 are identical and the other two differ only in the test and target counts. The reply variable is
res_0, as in REST and RPC; the test name comes from the sameLanguageConventionFormatter; the comment block lists calls with their outcome where REST lists them with their status code, and the fault line moved up intoTestCaseWriterso both say it the same way.Java, Kotlin and Python can all be generated for Kafka.