From 369e5b21680b8000869de51d6749e9ae54f36f4d Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Sat, 10 Oct 2026 22:28:40 -0300 Subject: [PATCH] AsyncAPI 3.x: track what the service consumes and emits over Kafka A settle window can end early on coverage staying stable, but that says nothing about the message itself. With the service's Kafka client instrumented, the controller can see the message being taken, and see the thread that took it come back for more, which is what a consumer loop does once its handler has returned. This adds that tracking, the messages the service emits as a by-product, and a completion strategy built on it. - Method replacements for poll(Duration) on KafkaConsumer and on the Consumer interface, and for send(record) and send(record, callback) on KafkaProducer and on the Producer interface. Replacements match the class a call is compiled against, so both are needed: NCS declares the concrete classes, Spring Kafka calls the interfaces. kafka-clients is only a test dependency; records are read by reflection. They are in a new KAFKA category, behind --instrumentMR_KAFKA, experimental and off by default. - poll(long), removed in kafka-clients 4.0 (checked against the 4.0.0 jar), is in classes of its own: every replacement of a class is resolved against the library on first use, and a missing method fails all of them. ReplacementList asserted at most one replacement class per target for constructor lookups; it asserts at most one that replaces constructors, and the visitor takes the consume method from the class of the chosen replacement. - A send made from inside another replaced send on the same thread is not recorded again. That covers Spring's CloseSafeProducer, which sends each record twice through Producer, and KafkaProducer.send (record) calling send(record, null) on itself. A send a callback or a custom producer makes from inside another send, on the same thread, is not recorded either. - ExecutionTracer.setExecutingDriverAction, set by EMController around executeAsyncApiAction, on that thread only. Calls made on that thread while it is set are not recorded. A driver class loaded once the agent is in place and the replacement categories are set gets the replacements too, and a package filter cannot tell it from the service, as they can share a prefix. With NCS's driver loaded that way and the flag not set, its publishes are recorded as messages the service emitted. Calls a driver makes on threads it starts itself are recorded as the service's; docs/write_driver.md says so. - AdditionalInfo records, per action, the messages emitted: address, payload as text, cut at 4096 characters and never inside a surrogate pair, whether it was cut, and the payload's class, 100 per action at most. A String is recorded as given and bytes as UTF-8; any other object, boxed primitives included, is kept as its class name, as its bytes are up to the serializer and calling toString would run SUT code. The text is what goes on the wire only with StringSerializer or ByteArraySerializer. The messages reach the core through AdditionalInfoDto.emittedMessages, and AsyncApiFitness keeps them on each action's AsyncApiCallResult, payload cut at 1024 characters, in white-box mode only. - AsyncMessageTracker, reset with ExecutionTracer, keeps per thread the batches a poll returned; a batch counts as handled once the same thread polls again with a timeout. A message received again is unhandled again, and a batch read into a test that is already over is dropped. - HANDLER_RETURN, appended to --asyncApiCompletion and to AsyncApiActionDto.Completion, with SettleEnd.HANDLER_RETURN: the window ends once something was polled from the published address in this action and all of it was handled. It needs --asyncApiSettleMs and --instrumentMR_KAFKA. External drivers read the tracker through a new MESSAGE_HANDLING_STATE agent command. A state that cannot be read, without the agent or when the agent fails to answer, means the rest of the window. Worth flagging for review, as this is a heuristic with blind spots: - Work handed off to another thread looks done as soon as it is handed off, since the receiving thread polls again at once. - A poll with a zero timeout does not count as the handler having returned. Spring Kafka polls that way to keep the consumer alive while it retries a failed batch. A loop that only polls with a zero timeout therefore never looks done, and waits for the deadline. - A failed record that an error handler seeks back to reads as handled in the moment between the failed attempt and the next poll's return. When it comes back it is unhandled again. - What was polled in the current action counts, so a message left over from an earlier action and consumed now counts too. - A thread that closes its consumer without polling again leaves its last batch unhandled, and the wait goes to the deadline. Two E2Es run with HANDLER_RETURN: NcsKafkaEMTest, where the service polls a KafkaConsumer and answers with a KafkaProducer from its own code, and ShopKafkaHandlerReturnEMTest, where Spring Kafka's listener containers poll through the Consumer interface. Every wait ends on the handler returning, none on the deadline, and the string branches reached by taint in the synchronous listeners are still covered. The Shop listener that hands rendering off to an executor is the blind spot above, and is not asserted on. It is a class of its own because the Kafka replacements have to be asked for when the service is started. No behaviour changes unless --instrumentMR_KAFKA or HANDLER_RETURN is set. No shared REST code is touched. --- .../controller/api/dto/AdditionalInfoDto.java | 6 + .../controller/api/dto/EmittedMessageDto.java | 29 ++ .../problem/asyncapi/AsyncApiActionDto.java | 11 +- .../problem/asyncapi/AsyncApiReplyDto.java | 11 +- .../controller/EmbeddedSutController.java | 9 + .../controller/ExternalSutController.java | 8 + .../controller/internal/AsyncApiSettle.java | 80 ++++- .../controller/internal/EMController.java | 24 +- .../controller/internal/SutController.java | 13 + .../internal/AsyncApiSettleTest.java | 171 ++++++++++ .../controller/internal/EMControllerTest.java | 147 +++++++++ .../shared/ReplacementCategory.java | 7 +- client-java/instrumentation/pom.xml | 6 + .../java/instrumentation/AdditionalInfo.java | 27 ++ .../java/instrumentation/EmittedMessage.java | 92 ++++++ .../InstrumentationController.java | 9 + .../instrumentation/MessageHandlingState.java | 58 ++++ .../methodreplacement/ReplacementList.java | 19 +- .../KafkaConsumerClassReplacement.java | 36 +++ ...afkaConsumerInterfaceClassReplacement.java | 34 ++ ...erInterfaceLegacyPollClassReplacement.java | 31 ++ ...fkaConsumerLegacyPollClassReplacement.java | 34 ++ .../KafkaProducerClassReplacement.java | 46 +++ ...afkaProducerInterfaceClassReplacement.java | 46 +++ .../KafkaReplacementHelper.java | 263 ++++++++++++++++ .../MethodReplacementMethodVisitor.java | 3 +- .../external/AgentController.java | 13 + .../instrumentation/external/Command.java | 3 +- .../external/ServerController.java | 36 +++ .../staticstate/AsyncMessageTracker.java | 188 +++++++++++ .../staticstate/ExecutionTracer.java | 58 ++++ .../examples/kafka/KafkaClientsImpl.java | 116 +++++++ .../instrumentation/EmittedMessageTest.java | 101 ++++++ .../MessageHandlingStateTest.java | 24 ++ .../ReplacementListTest.java | 8 + .../KafkaConsumerClassReplacementTest.java | 92 ++++++ ...ConsumerInterfaceClassReplacementTest.java | 158 ++++++++++ ...terfaceLegacyPollClassReplacementTest.java | 70 +++++ ...onsumerLegacyPollClassReplacementTest.java | 63 ++++ .../KafkaProducerClassReplacementTest.java | 99 ++++++ ...ProducerInterfaceClassReplacementTest.java | 296 ++++++++++++++++++ .../example/kafka/KafkaClients.java | 44 +++ .../example/kafka/KafkaClientsTest.java | 145 +++++++++ .../external/ServerControllerTest.java | 45 +++ .../staticstate/AsyncMessageTrackerTest.java | 236 ++++++++++++++ .../staticstate/ExecutionTracerTest.java | 48 ++- .../e2etests/utils/AsyncApiTestBase.java | 10 + .../ShopKafkaHandlerReturnEMTest.java | 104 ++++++ .../spring/asyncapi/kafka/NcsKafkaEMTest.java | 73 ++++- .../kotlin/org/evomaster/core/EMConfig.kt | 34 +- .../asyncapi/data/AsyncApiCallResult.kt | 65 ++++ .../asyncapi/service/AsyncApiFitness.kt | 22 ++ .../kotlin/org/evomaster/core/EMConfigTest.kt | 42 +++ .../asyncapi/data/AsyncApiCallResultTest.kt | 88 ++++++ .../asyncapi/service/AsyncApiFitnessTest.kt | 84 +++++ docs/options.md | 3 +- docs/write_driver.md | 7 + 57 files changed, 3568 insertions(+), 27 deletions(-) create mode 100644 client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/EmittedMessageDto.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/EmittedMessage.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/MessageHandlingState.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacement.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaReplacementHelper.java create mode 100644 client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTracker.java create mode 100644 client-java/instrumentation/src/test/java/com/foo/somedifferentpackage/examples/kafka/KafkaClientsImpl.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/EmittedMessageTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/MessageHandlingStateTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacementTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClients.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClientsTest.java create mode 100644 client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTrackerTest.java create mode 100644 core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener/src/test/java/org/evomaster/e2etests/spring/asyncapi/listener/ShopKafkaHandlerReturnEMTest.java create mode 100644 core/src/test/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResultTest.kt diff --git a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/AdditionalInfoDto.java b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/AdditionalInfoDto.java index 0e21970046..b13c7d5383 100644 --- a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/AdditionalInfoDto.java +++ b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/AdditionalInfoDto.java @@ -67,4 +67,10 @@ public class AdditionalInfoDto { public List employedDefaultWM = new ArrayList<>(); public List hostnameResolutionInfoDtos = new ArrayList<>(); + + /** + * The messages the SUT handed to a broker client to publish (eg a Kafka producer), in order. + * Only recorded for clients the instrumentation tracks. + */ + public List emittedMessages = new ArrayList<>(); } diff --git a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/EmittedMessageDto.java b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/EmittedMessageDto.java new file mode 100644 index 0000000000..9d39a69b0d --- /dev/null +++ b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/EmittedMessageDto.java @@ -0,0 +1,29 @@ +package org.evomaster.client.java.controller.api.dto; + +/** + * A message the SUT handed to a broker client to publish during an action, eg a record given to + * a Kafka producer. Only the address and the payload, the latter only when it is text, truncated. + */ +public class EmittedMessageDto { + + /** + * Where the message was published to, eg a Kafka topic. Null if the client did not say. + */ + public String address; + + /** + * The payload as text: a String, or bytes read as UTF-8. Null for no payload or any other + * object, whose wire form depends on the client's serializer; see {@link #payloadType}. + */ + public String payload; + + /** + * Whether {@link #payload} was cut because it was too long. Null if not known. + */ + public Boolean payloadTruncated; + + /** + * The name of the class of the payload the SUT gave. Null if it gave none. + */ + public String payloadType; +} diff --git a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiActionDto.java b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiActionDto.java index 0113fca802..e88ce3b325 100644 --- a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiActionDto.java +++ b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiActionDto.java @@ -38,7 +38,16 @@ public enum MessageCompletionStrategy { * objective for the first time, or improving one. A heuristic: a handler that only runs * code already covered earlier in the same test leaves coverage stable too. */ - COVERAGE_STABLE + COVERAGE_STABLE, + + /** + * Stop early once the SUT has received at least one message from {@link #address}, and + * every thread that received one has gone back to its client for more, which is what a + * consumer loop does when its handler has returned. Read off what the instrumentation + * records of the SUT's broker client calls, so only for clients it tracks. A heuristic: + * work handed off to another thread looks done as soon as it is handed off. + */ + HANDLER_RETURN } /** diff --git a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiReplyDto.java b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiReplyDto.java index a1a41a7c63..2df8994d95 100644 --- a/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiReplyDto.java +++ b/client-java/controller-api/src/main/java/org/evomaster/client/java/controller/api/dto/problem/asyncapi/AsyncApiReplyDto.java @@ -22,7 +22,8 @@ public enum SettleEndReason { /** * The whole {@link AsyncApiActionDto#settleMs} window went by. Also what a * {@link AsyncApiActionDto.MessageCompletionStrategy#COVERAGE_STABLE} wait ends with when the SUT's - * coverage cannot be read. + * coverage cannot be read, and a {@link AsyncApiActionDto.MessageCompletionStrategy#HANDLER_RETURN} one + * when what the SUT received cannot be. */ DEADLINE, @@ -35,7 +36,13 @@ public enum SettleEndReason { /** * The waiting thread was interrupted. */ - INTERRUPTED + INTERRUPTED, + + /** + * Every message the SUT received from {@link AsyncApiActionDto#address} in this action + * was handled, see {@link AsyncApiActionDto.MessageCompletionStrategy#HANDLER_RETURN}. + */ + HANDLER_RETURN } /** diff --git a/client-java/controller/src/main/java/org/evomaster/client/java/controller/EmbeddedSutController.java b/client-java/controller/src/main/java/org/evomaster/client/java/controller/EmbeddedSutController.java index 224a4cb066..b511380015 100644 --- a/client-java/controller/src/main/java/org/evomaster/client/java/controller/EmbeddedSutController.java +++ b/client-java/controller/src/main/java/org/evomaster/client/java/controller/EmbeddedSutController.java @@ -156,6 +156,15 @@ public final Long getObjectiveUpdateCount() { return InstrumentationController.getObjectiveUpdateCount(); } + @Override + public final MessageHandlingState getMessageHandlingState(String address) { + //nothing to read without the agent, as for the objective update count + if (!isInstrumentationActivated()) { + return null; + } + return InstrumentationController.getMessageHandlingState(address); + } + @Override public final String getExecutableFullPath(){ return null; //not needed for embedded diff --git a/client-java/controller/src/main/java/org/evomaster/client/java/controller/ExternalSutController.java b/client-java/controller/src/main/java/org/evomaster/client/java/controller/ExternalSutController.java index e1b8af0ce0..be246f9205 100644 --- a/client-java/controller/src/main/java/org/evomaster/client/java/controller/ExternalSutController.java +++ b/client-java/controller/src/main/java/org/evomaster/client/java/controller/ExternalSutController.java @@ -463,6 +463,14 @@ public final Long getObjectiveUpdateCount(long timeoutMs) { return serverController.getObjectiveUpdateCount(timeoutMs); } + @Override + public final MessageHandlingState getMessageHandlingState(String address) { + if (!isInstrumentationActivated()) { + return null; + } + return serverController.getMessageHandlingState(address); + } + @Override public final void newActionSpecificHandler(ActionDto dto) { if (isInstrumentationActivated()) { diff --git a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/AsyncApiSettle.java b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/AsyncApiSettle.java index d0b0eae87d..597ea953ef 100644 --- a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/AsyncApiSettle.java +++ b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/AsyncApiSettle.java @@ -2,8 +2,10 @@ import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiActionDto; import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; +import org.evomaster.client.java.instrumentation.MessageHandlingState; import java.util.Objects; +import java.util.function.Function; import java.util.function.Supplier; import java.util.function.LongFunction; @@ -20,10 +22,13 @@ public class AsyncApiSettle { private static final String OBJECTIVE_UPDATES_FAILURE_WARNING = "Failed to read how many coverage objectives" + " the SUT has reached or improved, so the whole settle window is waited"; + private static final String MESSAGE_HANDLING_FAILURE_WARNING = "Failed to read what the SUT received from the" + + " address a message was published to, so the whole settle window is waited"; + /** - * How often the SUT's coverage is read while waiting for it to become stable, in milliseconds. - * Short next to any useful stable window, and long enough that reading it, a socket round - * trip for an external SUT, costs little. + * How often the SUT's coverage, or what it received, is read while waiting for it to be done, + * in milliseconds. Short next to any useful stable window, and long enough that reading it, a + * socket round trip for an external SUT, costs little. */ static final long POLL_INTERVAL_MS = 25; @@ -75,9 +80,27 @@ public AsyncApiSettle() { * Only read for {@link AsyncApiActionDto.MessageCompletionStrategy#COVERAGE_STABLE} */ public void settle(AsyncApiActionDto dto, AsyncApiReplyDto reply, LongFunction objectiveUpdates) { + settle(dto, reply, objectiveUpdates, address -> null); + } + + /** + * As {@link #settle(AsyncApiActionDto, AsyncApiReplyDto, LongFunction)}, for every completion. + * + * @param messageHandling reads what the SUT received from an address in the current action, + * and how much of it it handled, or null when that cannot be known. + * Only read, with {@link AsyncApiActionDto#address}, for + * {@link AsyncApiActionDto.MessageCompletionStrategy#HANDLER_RETURN} + */ + public void settle( + AsyncApiActionDto dto, + AsyncApiReplyDto reply, + LongFunction objectiveUpdates, + Function messageHandling + ) { Objects.requireNonNull(dto); Objects.requireNonNull(reply); Objects.requireNonNull(objectiveUpdates); + Objects.requireNonNull(messageHandling); Long settleMs = dto.settleMs; if (settleMs == null || settleMs <= 0) { @@ -86,13 +109,18 @@ public void settle(AsyncApiActionDto dto, AsyncApiReplyDto reply, LongFunction 0; + boolean handlerReturn = dto.completion == AsyncApiActionDto.MessageCompletionStrategy.HANDLER_RETURN; long start = clock.nowMs(); AsyncApiReplyDto.SettleEndReason end; try { - end = coverageStable - ? waitForStableCoverage(start, settleMs, dto.stableMs, objectiveUpdates) - : waitUntil(start + settleMs); + if (coverageStable) { + end = waitForStableCoverage(start, settleMs, dto.stableMs, objectiveUpdates); + } else if (handlerReturn) { + end = waitForHandlerReturn(start + settleMs, dto.address, messageHandling); + } else { + end = waitUntil(start + settleMs); + } } catch (InterruptedException e) { Thread.currentThread().interrupt(); end = AsyncApiReplyDto.SettleEndReason.INTERRUPTED; @@ -136,10 +164,42 @@ private AsyncApiReplyDto.SettleEndReason waitUntil(long deadline) throws Interru } /** - * Read the SUT's coverage every {@link #POLL_INTERVAL_MS}, and stop once it has gone - * {@code stableMs} without moving, or at the deadline, whichever comes first. The stable - * period is counted from the start of the wait, so a service that does nothing new after - * the driver returned is let go after {@code stableMs}. + * Stop once the SUT received something from the action's address in this action and is done + * with all of it, or at the deadline. What was received before the driver returned counts + * too, as does a leftover message from an earlier action consumed only now. A missing + * address or unreadable state means the whole window. + */ + private AsyncApiReplyDto.SettleEndReason waitForHandlerReturn( + long deadline, + String address, + Function messageHandling + ) throws InterruptedException { + + if (address == null) { + return waitUntil(deadline); + } + + while (true) { + MessageHandlingState state = readOrNull(() -> messageHandling.apply(address), + MESSAGE_HANDLING_FAILURE_WARNING); + if (state == null) { + return waitUntil(deadline); + } + if (state.isHandled()) { + return AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN; + } + + long now = clock.nowMs(); + if (now >= deadline) { + return AsyncApiReplyDto.SettleEndReason.DEADLINE; + } + sleeper.sleep(Math.min(POLL_INTERVAL_MS, deadline - now)); + } + } + + /** + * Stop once the SUT's coverage has gone {@code stableMs} without moving, counted from the + * start of the wait, or at the deadline. */ private AsyncApiReplyDto.SettleEndReason waitForStableCoverage( long start, diff --git a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/EMController.java b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/EMController.java index 96a8dced57..16fe275627 100644 --- a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/EMController.java +++ b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/EMController.java @@ -21,6 +21,7 @@ import org.evomaster.client.java.controller.redis.RedisCommandExecutor; import org.evomaster.client.java.controller.redis.ReflectionBasedRedisClient; import org.evomaster.client.java.instrumentation.AdditionalInfo; +import org.evomaster.client.java.instrumentation.EmittedMessage; import org.evomaster.client.java.instrumentation.InputProperties; import org.evomaster.client.java.instrumentation.TargetInfo; import org.evomaster.client.java.instrumentation.shared.StringSpecializationInfo; @@ -603,6 +604,15 @@ private void prepareForTestResults(HttpServletRequest httpServletRequest) { assert trackRequestSource(httpServletRequest); } + private static EmittedMessageDto toDto(EmittedMessage message) { + EmittedMessageDto dto = new EmittedMessageDto(); + dto.address = message.getAddress(); + dto.payload = message.getPayload(); + dto.payloadTruncated = message.isPayloadTruncated(); + dto.payloadType = message.getPayloadType(); + return dto; + } + private Response computeTestResults( Set ids, boolean killSwitch, @@ -675,6 +685,9 @@ private Response computeTestResults( des.getRemotePort() ) ).collect(Collectors.toList()); + info.emittedMessages = a.getEmittedMessagesView().stream() + .map(EMController::toDto) + .collect(Collectors.toList()); info.stringSpecializations = new LinkedHashMap<>(); for (Map.Entry> entry : a.getStringSpecializationsView().entrySet()) { @@ -867,6 +880,12 @@ public Response newAction( replyDto.index = index; try { + /* + So that the driver's own broker client calls are not recorded as the SUT's, + when the driver's classes are instrumented. Only this thread: the SUT's + threads keep consuming meanwhile. + */ + ExecutionTracer.setExecutingDriverAction(true); sutController.executeAsyncApiAction(dto.asyncApiCall, replyDto); } catch (Exception e) { /* @@ -879,6 +898,8 @@ public Response newAction( replyDto.published = false; replyDto.errorMessage = msg; return Response.status(500).entity(WrappedResponseDto.withData(replyDto)).build(); + } finally { + ExecutionTracer.setExecutingDriverAction(false); } if (!Boolean.TRUE.equals(replyDto.published)) { @@ -897,7 +918,8 @@ public Response newAction( was already turned off for this action by newAction above, and must stay off for the service's threads, not only this one. */ - asyncApiSettle.settle(dto.asyncApiCall, replyDto, sutController::getObjectiveUpdateCount); + asyncApiSettle.settle(dto.asyncApiCall, replyDto, + sutController::getObjectiveUpdateCount, sutController::getMessageHandlingState); return Response.status(200).entity(WrappedResponseDto.withData(replyDto)).build(); } diff --git a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/SutController.java b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/SutController.java index 0ab5df7026..7e560c3cbc 100644 --- a/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/SutController.java +++ b/client-java/controller/src/main/java/org/evomaster/client/java/controller/internal/SutController.java @@ -60,6 +60,7 @@ import org.evomaster.client.java.controller.redis.ReflectionBasedRedisClient; import org.evomaster.client.java.instrumentation.AdditionalInfo; import org.evomaster.client.java.instrumentation.BootTimeObjectiveInfo; +import org.evomaster.client.java.instrumentation.MessageHandlingState; import org.evomaster.client.java.instrumentation.TargetInfo; import org.evomaster.client.java.instrumentation.staticstate.UnitsInfoRecorder; import org.evomaster.client.java.sql.*; @@ -1883,6 +1884,18 @@ public Long getObjectiveUpdateCount(long timeoutMs) { return getObjectiveUpdateCount(); } + /** + * What the SUT received, in the current action, from a broker address (eg a Kafka topic), + * and how much of it it is done handling, as recorded by the instrumentation. Read while an + * action is kept open. + * + * @param address where the SUT receives messages from + * @return the state, or null when unknown, eg because the SUT is not instrumented + */ + public MessageHandlingState getMessageHandlingState(String address) { + return null; + } + protected BootTimeInfoDto getBootTimeInfoDto(BootTimeObjectiveInfo info){ if (info == null) return null; diff --git a/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/AsyncApiSettleTest.java b/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/AsyncApiSettleTest.java index 3151605949..cc1009e0f6 100644 --- a/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/AsyncApiSettleTest.java +++ b/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/AsyncApiSettleTest.java @@ -3,12 +3,14 @@ import com.fasterxml.jackson.databind.ObjectMapper; import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiActionDto; import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; +import org.evomaster.client.java.instrumentation.MessageHandlingState; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import java.util.ArrayList; import java.util.List; import java.util.function.LongFunction; +import java.util.function.Function; import static org.junit.jupiter.api.Assertions.*; @@ -21,6 +23,17 @@ public class AsyncApiSettleTest { private static final long SETTLE_MS = 1_000; + /** + * Where the action under test publishes to. + */ + private static final String ADDRESS = "requests"; + + private static final MessageHandlingState NOTHING_RECEIVED = new MessageHandlingState(0, 0); + + private static final MessageHandlingState BEING_HANDLED = new MessageHandlingState(1, 1); + + private static final MessageHandlingState HANDLED = new MessageHandlingState(1, 0); + private static final long STABLE_MS = 100; /** @@ -353,4 +366,162 @@ public void testAnInterruptWhileWaitingForStableCoverageEndsTheWait() { assertEquals(AsyncApiReplyDto.SettleEndReason.INTERRUPTED, reply.settleEndedBy); assertEquals(50L, reply.settledMs); } + + private static AsyncApiActionDto untilHandled() { + AsyncApiActionDto dto = asking(SETTLE_MS); + dto.completion = AsyncApiActionDto.MessageCompletionStrategy.HANDLER_RETURN; + dto.address = ADDRESS; + return dto; + } + + private static Function always(MessageHandlingState state) { + return address -> state; + } + + private static Function neverRead() { + return address -> fail("what the SUT received was read for address " + address); + } + + @Test + public void testAMessageAlreadyHandledWhenTheDriverReturnsEndsTheWaitAtOnce() { + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, always(HANDLED)); + + assertEquals(AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN, reply.settleEndedBy); + assertEquals(0L, reply.settledMs); + assertTrue(sleeps.isEmpty(), "slept although the SUT was done: " + sleeps); + } + + @Test + public void testTheActionIsKeptOpenUntilTheHandlerReturns() { + + Function doneAfter100 = + address -> now - START < 100 ? BEING_HANDLED : HANDLED; + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, doneAfter100); + + assertEquals(AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN, reply.settleEndedBy); + assertEquals(100L, reply.settledMs); + assertTrue(sleeps.stream().allMatch(ms -> ms <= AsyncApiSettle.POLL_INTERVAL_MS), + "what the SUT received was not read often enough: " + sleeps); + } + + @Test + public void testNothingReceivedIsWaitedOnUntilTheDeadline() { + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, always(NOTHING_RECEIVED)); + + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testTheDeadlineCapsAHandlerThatNeverReturns() { + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, always(BEING_HANDLED)); + + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testWhatTheSutReceivedThatCannotBeReadMeansTheWholeWindow() { + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, address -> null); + + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testWhatTheSutReceivedThatFailsToBeReadMeansTheRestOfTheWindow() { + + //eg the external SUT's agent going away half-way through the wait + Function failingAfter50 = address -> { + if (now - START >= 50) { + throw new IllegalStateException("agent gone"); + } + return BEING_HANDLED; + }; + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(untilHandled(), reply, NO_COVERAGE, failingAfter50); + + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testWithoutAnAddressTheWholeWindow() { + + AsyncApiActionDto dto = untilHandled(); + dto.address = null; + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(dto, reply, NO_COVERAGE, neverRead()); + + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testWhatIsReadIsWhatCameFromThePublishedAddress() { + + List asked = new ArrayList<>(); + Function recording = address -> { + asked.add(address); + return now - START < 50 ? NOTHING_RECEIVED : HANDLED; + }; + + settleThatSleeps().settle(untilHandled(), new AsyncApiReplyDto(), NO_COVERAGE, recording); + + assertFalse(asked.isEmpty()); + assertTrue(asked.stream().allMatch(ADDRESS::equals), "Asked about: " + asked); + } + + @Test + public void testOtherCompletionsNeverReadWhatTheSutReceived() { + + AsyncApiActionDto fixed = asking(SETTLE_MS); + fixed.address = ADDRESS; + settleThatSleeps().settle(fixed, new AsyncApiReplyDto(), NO_COVERAGE, neverRead()); + + AsyncApiActionDto stable = coverageStable(STABLE_MS); + stable.address = ADDRESS; + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settleThatSleeps().settle(stable, reply, () -> 7L, neverRead()); + + assertEquals(AsyncApiReplyDto.SettleEndReason.COVERAGE_STABLE, reply.settleEndedBy); + } + + @Test + public void testTheHandlingCannotBeNull() { + + assertThrows(NullPointerException.class, + () -> settleThatSleeps().settle(untilHandled(), new AsyncApiReplyDto(), NO_COVERAGE, null)); + } + + @Test + public void testAnInterruptWhileWaitingForTheHandlerEndsTheWait() { + + AsyncApiSettle settle = new AsyncApiSettle(() -> now, ms -> { + now += ms; + if (now - START >= 50) { + throw new InterruptedException(); + } + }); + + AsyncApiReplyDto reply = new AsyncApiReplyDto(); + settle.settle(untilHandled(), reply, NO_COVERAGE, always(BEING_HANDLED)); + + assertTrue(Thread.currentThread().isInterrupted(), "the interrupt was swallowed"); + assertEquals(AsyncApiReplyDto.SettleEndReason.INTERRUPTED, reply.settleEndedBy); + assertEquals(50L, reply.settledMs); + } } diff --git a/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/EMControllerTest.java b/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/EMControllerTest.java index 6cc5052f25..694fe99223 100644 --- a/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/EMControllerTest.java +++ b/client-java/controller/src/test/java/org/evomaster/client/java/controller/internal/EMControllerTest.java @@ -9,6 +9,11 @@ import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; import org.evomaster.client.java.controller.problem.AsyncApiProblem; import org.evomaster.client.java.controller.problem.ProblemInfo; +import org.evomaster.client.java.controller.api.dto.AdditionalInfoDto; +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto; +import org.evomaster.client.java.controller.api.dto.TestResultsDto; +import org.evomaster.client.java.instrumentation.EmittedMessage; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; import org.evomaster.client.java.instrumentation.InstrumentingAgent; import org.evomaster.client.java.sql.DbSpecification; @@ -20,7 +25,13 @@ import javax.ws.rs.core.Response; import java.lang.reflect.Field; import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.mock; @@ -40,6 +51,15 @@ public class EMControllerTest { */ private static final String OBJECTIVE_PREFIX = "EMControllerTest_"; + /** + * Where the actions published here go, and the SUT consumes from. + */ + private static final String ADDRESS = "EMControllerTest_requests"; + + private static final String PAYLOAD = "{\"result\":42}"; + + private static final long CONSUMER_TIMEOUT_SECONDS = 10; + /** * What happened, in order, across the driver and the controller's wait. */ @@ -49,6 +69,12 @@ public class EMControllerTest { private AsyncApiDriver driver; + /** + * Stands in for the SUT's consumer thread: what it receives, and whether it is done with it, + * is tracked per thread. + */ + private ExecutorService consumer; + private EMController controller; /** @@ -81,6 +107,7 @@ public void init() { now = 0; driver = new AsyncApiDriver(); driver.newTest(); + consumer = Executors.newSingleThreadExecutor(); controller = new EMController(driver, new AsyncApiSettle(() -> now, ms -> { events.add("settle " + ms); now += ms; @@ -91,6 +118,26 @@ public void init() { public void tearDown() { //static state, shared with whatever runs next in this JVM ExecutionTracer.reset(); + consumer.shutdownNow(); + } + + /** + * Run on the stand-in for the SUT's consumer thread, and wait for it. + */ + private void onTheConsumerThread(Runnable runnable) { + try { + consumer.submit(runnable).get(CONSUMER_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } catch (InterruptedException | ExecutionException | TimeoutException e) { + throw new IllegalStateException(e); + } + } + + private static AsyncApiActionDto untilHandled() { + AsyncApiActionDto call = new AsyncApiActionDto(); + call.settleMs = SETTLE_MS; + call.completion = AsyncApiActionDto.MessageCompletionStrategy.HANDLER_RETURN; + call.address = ADDRESS; + return call; } private Response publish(Long settleMs) { @@ -261,6 +308,94 @@ public void testASutStillReachingNewCodeIsWaitedOnUntilTheDeadline() { assertEquals(SETTLE_MS, reply.settledMs); } + @Test + public void testTheDriverRunsMarkedAsTheDriver() { + + Response response = publish(null); + + assertEquals(OK, response.getStatus()); + assertEquals(true, driver.markedAsDriver, "the driver published unmarked"); + assertFalse(ExecutionTracer.isExecutingDriverAction(), "the mark outlived the driver"); + } + + @Test + public void testTheDriverMarkIsClearedWhenTheDriverFails() { + + driver.fails = true; + + Response response = publish(null); + + assertEquals(SERVER_ERROR, response.getStatus()); + assertEquals(true, driver.markedAsDriver, "the driver published unmarked"); + assertFalse(ExecutionTracer.isExecutingDriverAction(), "the mark outlived the failed driver"); + } + + @Test + public void testASutDoneWithTheMessageIsLetGo() { + + //the SUT takes the message while the driver is still publishing + driver.onPublish = () -> onTheConsumerThread(() -> AsyncMessageTracker.received(Collections.singletonList(ADDRESS))); + + //and goes back for more while the controller waits + controller = new EMController(driver, new AsyncApiSettle(() -> now, ms -> { + now += ms; + onTheConsumerThread(() -> AsyncMessageTracker.askingForMore(true)); + })); + + Response response = publishCall(untilHandled()); + + AsyncApiReplyDto reply = replyIn(response); + assertEquals(AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN, reply.settleEndedBy); + assertEquals(AsyncApiSettle.POLL_INTERVAL_MS, reply.settledMs); + } + + @Test + public void testWithoutTheAgentHandlerReturnWaitsTheWholeWindow() { + + //what the tracker holds is not the SUT's when nothing is instrumented + setAgentActive(false); + driver.onPublish = () -> onTheConsumerThread(() -> { + AsyncMessageTracker.received(Collections.singletonList(ADDRESS)); + AsyncMessageTracker.askingForMore(true); + }); + + Response response = publishCall(untilHandled()); + + AsyncApiReplyDto reply = replyIn(response); + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testASutThatNeverTakesTheMessageIsWaitedOnUntilTheDeadline() { + + Response response = publishCall(untilHandled()); + + AsyncApiReplyDto reply = replyIn(response); + assertEquals(AsyncApiReplyDto.SettleEndReason.DEADLINE, reply.settleEndedBy); + assertEquals(SETTLE_MS, reply.settledMs); + } + + @Test + public void testWhatTheSutEmittedAndConsumedReachesTheTestResults() { + + ExecutionTracer.addEmittedMessage(new EmittedMessage(ADDRESS, PAYLOAD, String.class.getName())); + + Response response = controller.getTestResultsWithIdsInBody( + Collections.emptyList(), false, false, false, false, mock(HttpServletRequest.class)); + + assertEquals(OK, response.getStatus()); + TestResultsDto results = (TestResultsDto) ((WrappedResponseDto) response.getEntity()).data; + AdditionalInfoDto info = results.additionalInfoList.get(0); + + assertEquals(1, info.emittedMessages.size()); + EmittedMessageDto message = info.emittedMessages.get(0); + assertEquals(ADDRESS, message.address); + assertEquals(PAYLOAD, message.payload); + assertEquals(false, message.payloadTruncated); + assertEquals(String.class.getName(), message.payloadType); + } + /** * A driver that publishes nothing, and says it did unless told to fail. */ @@ -268,9 +403,21 @@ private class AsyncApiDriver extends EmbeddedSutController { private boolean fails = false; + /** + * What the SUT does while the driver publishes. + */ + private Runnable onPublish = () -> {}; + + /** + * Whether the thread publishing was marked as running the driver, when it published. + */ + private Boolean markedAsDriver = null; + @Override public void executeAsyncApiAction(AsyncApiActionDto dto, AsyncApiReplyDto reply) { events.add("publish"); + markedAsDriver = ExecutionTracer.isExecutingDriverAction(); + onPublish.run(); if (fails) { throw new IllegalStateException("no broker"); } diff --git a/client-java/instrumentation-shared/src/main/java/org/evomaster/client/java/instrumentation/shared/ReplacementCategory.java b/client-java/instrumentation-shared/src/main/java/org/evomaster/client/java/instrumentation/shared/ReplacementCategory.java index e7a2468d95..1a8e0d6f53 100644 --- a/client-java/instrumentation-shared/src/main/java/org/evomaster/client/java/instrumentation/shared/ReplacementCategory.java +++ b/client-java/instrumentation-shared/src/main/java/org/evomaster/client/java/instrumentation/shared/ReplacementCategory.java @@ -60,6 +60,11 @@ public enum ReplacementCategory { /** * Replacements to handle DYNAMODB command interceptions */ - DYNAMODB + DYNAMODB, + + /** + * Replacements that track the messages a service consumes from and emits to Kafka + */ + KAFKA } diff --git a/client-java/instrumentation/pom.xml b/client-java/instrumentation/pom.xml index 6b011d9bf8..8abeddb031 100644 --- a/client-java/instrumentation/pom.xml +++ b/client-java/instrumentation/pom.xml @@ -204,6 +204,12 @@ spring-kafka test + + + org.apache.kafka + kafka-clients + test + mysql mysql-connector-java diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/AdditionalInfo.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/AdditionalInfo.java index 72aba7a97e..9effc7321f 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/AdditionalInfo.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/AdditionalInfo.java @@ -8,6 +8,7 @@ import java.io.Serializable; import java.util.*; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.CopyOnWriteArraySet; /** @@ -16,6 +17,11 @@ */ public class AdditionalInfo implements Serializable { + /** + * At most how many of the messages the SUT emits in an action are kept. + */ + public static final int MAX_EMITTED_MESSAGES = 100; + /** * In REST APIs, it can happen that some query parameters do not * appear in the schema if they are indirectly accessed via @@ -121,6 +127,12 @@ public StatementDescription(String line, String method) { private final Set cassandraTableMetadataData = new CopyOnWriteArraySet<>(); + /** + * The messages the SUT handed to a broker client to publish, in order, up to + * {@link #MAX_EMITTED_MESSAGES}. + */ + private final List emittedMessages = new CopyOnWriteArrayList<>(); + public Set getSqlInfoData(){ return Collections.unmodifiableSet(executedSqlCommandData); } @@ -193,6 +205,21 @@ public void addCassandraTableMetadata(CassandraTableMetadata cassandraTableMetad cassandraTableMetadataData.add(cassandraTableMetadata); } + public void addEmittedMessage(EmittedMessage message){ + Objects.requireNonNull(message); + //checked and added separately, so a burst on several threads can go a little over: harmless + if(emittedMessages.size() < MAX_EMITTED_MESSAGES) { + emittedMessages.add(message); + } + } + + /** + * @return the messages the SUT emitted, in the order it emitted them + */ + public List getEmittedMessagesView(){ + return Collections.unmodifiableList(emittedMessages); + } + public Set getParsedDtoNamesView(){ return Collections.unmodifiableSet(parsedDtoNames); } diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/EmittedMessage.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/EmittedMessage.java new file mode 100644 index 0000000000..c69c13b08e --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/EmittedMessage.java @@ -0,0 +1,92 @@ +package org.evomaster.client.java.instrumentation; + +import java.io.Serializable; + +/** + * A message the SUT handed to a broker client to publish during an action, eg a record given to + * a Kafka producer. + * + * Only the address and the payload are kept, the payload only when it is text and up to + * {@link #MAX_TEXT_LENGTH} characters, as this is recorded for every message and sent to the + * core. Keys and headers are not kept, as the core does not read them. Serializable for + * external mode. + */ +public class EmittedMessage implements Serializable { + + private static final long serialVersionUID = 1L; + + /** + * How many characters of the payload are kept, see {@link #isPayloadTruncated()}. + */ + public static final int MAX_TEXT_LENGTH = 4096; + + /** + * Where the message was published to, eg a Kafka topic. + */ + private final String address; + + /** + * The payload as text, or null when there was none or it has no faithful text form, eg an + * object the client's serializer turns into bytes. + */ + private final String payload; + + /** + * Whether {@link #payload} was cut at {@link #MAX_TEXT_LENGTH} characters. + */ + private final boolean payloadTruncated; + + /** + * The name of the class of the payload the SUT gave, or null if it gave none. + */ + private final String payloadType; + + /** + * @param address where the message was published to, eg a Kafka topic + * @param payload the payload as text, null if none or not text. Cut here if longer than + * {@link #MAX_TEXT_LENGTH} + * @param payloadType the class name of the payload object, null if there was none + */ + public EmittedMessage(String address, String payload, String payloadType) { + this.address = address; + this.payload = cut(payload); + this.payloadTruncated = payload != null && payload.length() > MAX_TEXT_LENGTH; + this.payloadType = payloadType; + } + + /** + * Cut the text at {@link #MAX_TEXT_LENGTH} characters, one less if the cut would leave half + * of a surrogate pair at the end. + */ + private static String cut(String text) { + if (text == null || text.length() <= MAX_TEXT_LENGTH) { + return text; + } + int end = MAX_TEXT_LENGTH; + if (Character.isHighSurrogate(text.charAt(end - 1)) && Character.isLowSurrogate(text.charAt(end))) { + end--; + } + return text.substring(0, end); + } + + public String getAddress() { + return address; + } + + public String getPayload() { + return payload; + } + + public boolean isPayloadTruncated() { + return payloadTruncated; + } + + public String getPayloadType() { + return payloadType; + } + + @Override + public String toString() { + return "EmittedMessage{address='" + address + "', payloadType='" + payloadType + "'}"; + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/InstrumentationController.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/InstrumentationController.java index 2809ffb8ac..d105cafc40 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/InstrumentationController.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/InstrumentationController.java @@ -2,6 +2,7 @@ import org.evomaster.client.java.instrumentation.cassandra.CassandraSchemaTracer; import org.evomaster.client.java.instrumentation.object.ClassToSchema; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; import org.evomaster.client.java.instrumentation.staticstate.ObjectiveRecorder; import org.evomaster.client.java.instrumentation.staticstate.UnitsInfoRecorder; @@ -47,6 +48,14 @@ public static long getObjectiveUpdateCount(){ return ExecutionTracer.getObjectiveUpdateCount(); } + /** + * @return what the SUT received from {@code address} in the current action, and how much of + * it it is done with, see {@link AsyncMessageTracker} + */ + public static MessageHandlingState getMessageHandlingState(String address){ + return AsyncMessageTracker.getHandlingState(address); + } + public static void newAction(Action action){ ExecutionTracer.setAction(action); diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/MessageHandlingState.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/MessageHandlingState.java new file mode 100644 index 0000000000..d485d504c3 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/MessageHandlingState.java @@ -0,0 +1,58 @@ +package org.evomaster.client.java.instrumentation; + +import java.io.Serializable; + +/** + * How far the SUT got, in the current action, with the messages it consumed from one address, + * eg a Kafka topic. A message counts as handled once the thread that received it asks the + * client for more, see + * {@link org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker} for the + * limits of this. Serializable for external mode. + */ +public class MessageHandlingState implements Serializable { + + private static final long serialVersionUID = 1L; + + /** + * How many messages from the address the SUT received in the current action. + */ + private final int received; + + /** + * How many of those are not handled yet. + */ + private final int unhandled; + + public MessageHandlingState(int received, int unhandled) { + if (received < 0 || unhandled < 0) { + throw new IllegalArgumentException("Negative counts: " + received + ", " + unhandled); + } + if (unhandled > received) { + throw new IllegalArgumentException("More unhandled (" + unhandled + ") messages than received (" + + received + ")"); + } + this.received = received; + this.unhandled = unhandled; + } + + public int getReceived() { + return received; + } + + public int getUnhandled() { + return unhandled; + } + + /** + * @return whether at least one message was received from the address in the current + * action, and every thread that received one has since come back for more + */ + public boolean isHandled() { + return received > 0 && unhandled == 0; + } + + @Override + public String toString() { + return "MessageHandlingState{received=" + received + ", unhandled=" + unhandled + "}"; + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementList.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementList.java index a51d8c9b9b..0180ccc8c9 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementList.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementList.java @@ -48,6 +48,12 @@ public static List getList() { new IntegerClassReplacement(), new InetAddressClassReplacement(), new InetSocketAddressClassReplacement(), + new KafkaConsumerClassReplacement(), + new KafkaConsumerInterfaceClassReplacement(), + new KafkaConsumerInterfaceLegacyPollClassReplacement(), + new KafkaConsumerLegacyPollClassReplacement(), + new KafkaProducerClassReplacement(), + new KafkaProducerInterfaceClassReplacement(), new KotlinIntrinsicsClassReplacement(), new JacksonObjectMapperClassReplacement(), new JacksonObjectReaderClassReplacement(), @@ -191,8 +197,19 @@ public static List getReplacements(String target, boolea ) .collect(Collectors.toList()); - assert !strict || (list.size() <= 1); //if strict, at most 1 class can match + /* + If strict, at most 1 class can replace the constructors of the target. Other classes can + target the same class for its methods, eg the Kafka consumer's poll(long) is in a class + of its own, as newer versions of the library no longer have it. + */ + assert !strict || list.stream().filter(ReplacementList::replacesConstructors).count() <= 1; return list; } + + private static boolean replacesConstructors(MethodReplacementClass mrc) { + return Arrays.stream(mrc.getClass().getDeclaredMethods()) + .map(m -> m.getAnnotation(Replacement.class)) + .anyMatch(r -> r != null && r.replacingConstructor()); + } } diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacement.java new file mode 100644 index 0000000000..7be5e6fd2d --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacement.java @@ -0,0 +1,36 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +import java.time.Duration; + +/** + * Tracks the records the SUT receives with KafkaConsumer.poll(Duration), see {@link KafkaReplacementHelper}. + * + * Replacements match the class a call is compiled against, so this one applies where the consumer is + * declared as a KafkaConsumer, and {@link KafkaConsumerInterfaceClassReplacement} where it is declared as a + * Consumer, as in Spring Kafka. + */ +public class KafkaConsumerClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaConsumerClassReplacement singleton = new KafkaConsumerClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.KAFKA_CONSUMER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.POLL, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA, + castTo = KafkaReplacementHelper.CONSUMER_RECORDS) + public static Object poll(Object caller, Duration timeout) { + return KafkaReplacementHelper.poll(singleton, caller, KafkaReplacementHelper.waits(timeout), timeout); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacement.java new file mode 100644 index 0000000000..33beea0c76 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacement.java @@ -0,0 +1,34 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +import java.time.Duration; + +/** + * Tracks the records the SUT receives with Consumer.poll(Duration), the interface Spring Kafka's + * listener containers poll through. See {@link KafkaConsumerClassReplacement} for why there is a + * replacement for the interface and one for the class. + */ +public class KafkaConsumerInterfaceClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaConsumerInterfaceClassReplacement singleton = new KafkaConsumerInterfaceClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.CONSUMER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.POLL, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA, + castTo = KafkaReplacementHelper.CONSUMER_RECORDS) + public static Object poll(Object caller, Duration timeout) { + return KafkaReplacementHelper.poll(singleton, caller, KafkaReplacementHelper.waits(timeout), timeout); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacement.java new file mode 100644 index 0000000000..d2d692e328 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacement.java @@ -0,0 +1,31 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +/** + * Tracks the records the SUT receives with Consumer.poll(long). In a class of its own for the same + * reason as {@link KafkaConsumerLegacyPollClassReplacement}. + */ +public class KafkaConsumerInterfaceLegacyPollClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaConsumerInterfaceLegacyPollClassReplacement singleton = new KafkaConsumerInterfaceLegacyPollClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.CONSUMER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.POLL, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA, + castTo = KafkaReplacementHelper.CONSUMER_RECORDS) + public static Object poll(Object caller, long timeoutMs) { + return KafkaReplacementHelper.poll(singleton, caller, KafkaReplacementHelper.waits(timeoutMs), timeoutMs); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacement.java new file mode 100644 index 0000000000..d52720b800 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacement.java @@ -0,0 +1,34 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +/** + * Tracks the records the SUT receives with KafkaConsumer.poll(long), deprecated since + * kafka-clients 2.0 and removed in 4.0. + * + * In a class of its own because all replacements of a class are resolved together, and a missing + * method fails all of them: next to poll(Duration), it would break every poll on kafka-clients 4.x. + */ +public class KafkaConsumerLegacyPollClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaConsumerLegacyPollClassReplacement singleton = new KafkaConsumerLegacyPollClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.KAFKA_CONSUMER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.POLL, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA, + castTo = KafkaReplacementHelper.CONSUMER_RECORDS) + public static Object poll(Object caller, long timeoutMs) { + return KafkaReplacementHelper.poll(singleton, caller, KafkaReplacementHelper.waits(timeoutMs), timeoutMs); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacement.java new file mode 100644 index 0000000000..1e47ff1be3 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacement.java @@ -0,0 +1,46 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyCast; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +import java.util.concurrent.Future; + +/** + * Tracks the records the SUT emits with KafkaProducer.send, see {@link KafkaReplacementHelper}. + * {@link KafkaProducerInterfaceClassReplacement} covers a producer declared as a Producer, see + * {@link KafkaConsumerClassReplacement} for why both are needed. + */ +public class KafkaProducerClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaProducerClassReplacement singleton = new KafkaProducerClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.KAFKA_PRODUCER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.SEND, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA) + public static Future send(Object caller, + @ThirdPartyCast(actualType = KafkaReplacementHelper.PRODUCER_RECORD) Object record) { + return KafkaReplacementHelper.send(singleton, KafkaReplacementHelper.SEND, caller, record); + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.SEND_WITH_CALLBACK, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA) + public static Future send(Object caller, + @ThirdPartyCast(actualType = KafkaReplacementHelper.PRODUCER_RECORD) Object record, + @ThirdPartyCast(actualType = KafkaReplacementHelper.CALLBACK) Object callback) { + return KafkaReplacementHelper.send(singleton, KafkaReplacementHelper.SEND_WITH_CALLBACK, caller, record, callback); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacement.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacement.java new file mode 100644 index 0000000000..850999e770 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacement.java @@ -0,0 +1,46 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.Replacement; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyCast; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.UsageFilter; +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory; +import org.evomaster.client.java.instrumentation.shared.ReplacementType; + +import java.util.concurrent.Future; + +/** + * Tracks the records the SUT emits with Producer.send, the interface Spring Kafka's KafkaTemplate + * sends through. A record that goes through a producer wrapping another one, as Spring's + * CloseSafeProducer does, is recorded once, see {@link KafkaReplacementHelper}. + */ +public class KafkaProducerInterfaceClassReplacement extends ThirdPartyMethodReplacementClass { + + private static final KafkaProducerInterfaceClassReplacement singleton = new KafkaProducerInterfaceClassReplacement(); + + @Override + protected String getNameOfThirdPartyTargetClass() { + return KafkaReplacementHelper.PRODUCER; + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.SEND, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA) + public static Future send(Object caller, + @ThirdPartyCast(actualType = KafkaReplacementHelper.PRODUCER_RECORD) Object record) { + return KafkaReplacementHelper.send(singleton, KafkaReplacementHelper.SEND, caller, record); + } + + @Replacement(replacingStatic = false, + type = ReplacementType.TRACKER, + id = KafkaReplacementHelper.SEND_WITH_CALLBACK, + usageFilter = UsageFilter.ANY, + category = ReplacementCategory.KAFKA) + public static Future send(Object caller, + @ThirdPartyCast(actualType = KafkaReplacementHelper.PRODUCER_RECORD) Object record, + @ThirdPartyCast(actualType = KafkaReplacementHelper.CALLBACK) Object callback) { + return KafkaReplacementHelper.send(singleton, KafkaReplacementHelper.SEND_WITH_CALLBACK, caller, record, callback); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaReplacementHelper.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaReplacementHelper.java new file mode 100644 index 0000000000..07c6b7d6cf --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaReplacementHelper.java @@ -0,0 +1,263 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.evomaster.client.java.instrumentation.EmittedMessage; +import org.evomaster.client.java.instrumentation.coverage.methodreplacement.ThirdPartyMethodReplacementClass; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.evomaster.client.java.utils.SimpleLogger; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Future; + +/** + * What the Kafka replacements share: calling the original, and recording what the SUT received + * and emitted. kafka-clients is not a dependency of EvoMaster, so records are only read by + * reflection. + * + * Two kinds of calls are never recorded: + *
    + *
  • those made on the thread running the driver's executeAsyncApiAction, while it runs, see + * {@link ExecutionTracer#isExecutingDriverAction()};
    • + *
  • those made from inside another replaced call on the same thread. A producer that wraps + * another one, as Spring Kafka's CloseSafeProducer does, sends each record twice through the + * {@code Producer} interface, and KafkaProducer.send(record) calls send(record, null) on + * itself. Both inner calls are replaced as well, since library code is instrumented too, and + * would otherwise record the same message again. The price: a send that the code of a + * callback or of a custom producer makes, on the same thread, from inside another send, is + * not recorded either.
    • + *
+ */ +final class KafkaReplacementHelper { + + static final String CONSUMER = "org.apache.kafka.clients.consumer.Consumer"; + + static final String KAFKA_CONSUMER = "org.apache.kafka.clients.consumer.KafkaConsumer"; + + static final String CONSUMER_RECORDS = "org.apache.kafka.clients.consumer.ConsumerRecords"; + + static final String PRODUCER = "org.apache.kafka.clients.producer.Producer"; + + static final String KAFKA_PRODUCER = "org.apache.kafka.clients.producer.KafkaProducer"; + + static final String PRODUCER_RECORD = "org.apache.kafka.clients.producer.ProducerRecord"; + + static final String CALLBACK = "org.apache.kafka.clients.producer.Callback"; + + /** + * The id of the replacement of poll, in every consumer replacement class. + */ + static final String POLL = "poll"; + + /** + * The id of the replacement of send(record), in every producer replacement class. + */ + static final String SEND = "send"; + + /** + * The id of the replacement of send(record, callback), in every producer replacement class. + */ + static final String SEND_WITH_CALLBACK = "sendWithCallback"; + + /* + Accessors of ConsumerRecord and ProducerRecord + */ + private static final String TOPIC = "topic"; + private static final String VALUE = "value"; + + /** + * Most bytes a code point takes in UTF-8. Decoding this many bytes per char, plus one code + * point, gives more than {@link EmittedMessage#MAX_TEXT_LENGTH} chars for any longer payload, + * so EmittedMessage can tell it was cut. + */ + private static final int UTF8_MAX_BYTES_PER_CODE_POINT = 4; + + /** + * How many bytes of a byte[] payload are decoded at most. + */ + private static final int MAX_DECODED_BYTES = UTF8_MAX_BYTES_PER_CODE_POINT * (EmittedMessage.MAX_TEXT_LENGTH + 1); + + /** + * Class of a record -> accessor name -> the public no-input method with that name. A + * ClassValue, so a cached class does not keep its class loader alive. + */ + private static final ClassValue> accessors = new ClassValue>() { + @Override + protected Map computeValue(Class type) { + return new ConcurrentHashMap<>(); + } + }; + + /** + * How deep the current thread is in replaced poll calls. + */ + private static final ThreadLocal pollDepth = ThreadLocal.withInitial(() -> new int[1]); + + /** + * How deep the current thread is in replaced send calls. A send made while this is above zero + * is inside another send on the same thread, and is not recorded. + */ + private static final ThreadLocal sendDepth = ThreadLocal.withInitial(() -> new int[1]); + + private KafkaReplacementHelper() { + } + + /** + * @return whether a poll with this timeout waits for records to arrive + */ + static boolean waits(Duration timeout) { + return timeout != null && !timeout.isZero() && !timeout.isNegative(); + } + + /** + * @return whether a poll with this timeout waits for records to arrive + */ + static boolean waits(long timeoutMs) { + return timeoutMs > 0; + } + + /** + * Call the original poll, and record what it returned as received by the current thread. + * + * @param waits whether the call waits for records, see {@link AsyncMessageTracker#askingForMore(boolean)} + */ + static Object poll(ThirdPartyMethodReplacementClass singleton, Object consumer, boolean waits, Object... args) { + + Method original = ThirdPartyMethodReplacementClass.getOriginal(singleton, POLL, consumer); + + int[] depth = pollDepth.get(); + if (depth[0] > 0 || ExecutionTracer.isExecutingDriverAction()) { + return invoke(original, consumer, args); + } + + depth[0]++; + try { + //before the call, which can block for the whole timeout + + AsyncMessageTracker.askingForMore(waits); + Object records = invoke(original, consumer, args); + recordReceived(records); + return records; + } finally { + depth[0]--; + } + } + + /** + * Call the original send, and record the record as emitted if the call returned. A future that + * then fails still counts: the SUT did emit it. + */ + static Future send(ThirdPartyMethodReplacementClass singleton, String id, Object producer, Object record, Object... args) { + + Method original = ThirdPartyMethodReplacementClass.getOriginal(singleton, id, producer); + + Object[] allArgs = new Object[args.length + 1]; + allArgs[0] = record; + System.arraycopy(args, 0, allArgs, 1, args.length); + + int[] depth = sendDepth.get(); + if (depth[0] > 0 || ExecutionTracer.isExecutingDriverAction()) { + return (Future) invoke(original, producer, allArgs); + } + + depth[0]++; + try { + Future result = (Future) invoke(original, producer, allArgs); + recordEmitted(record); + return result; + } finally { + depth[0]--; + } + } + + private static Object invoke(Method original, Object target, Object... args) { + try { + return original.invoke(target, args); + } catch (IllegalAccessException e) { + throw new RuntimeException(e); + } catch (InvocationTargetException e) { + Throwable cause = e.getCause(); + if (cause instanceof RuntimeException) { + throw (RuntimeException) cause; + } + if (cause instanceof Error) { + throw (Error) cause; + } + //neither poll nor send declares a checked exception + throw new RuntimeException(cause); + } + } + + private static void recordReceived(Object records) { + if (!(records instanceof Iterable)) { + return; + } + try { + List received = new ArrayList<>(); + for (Object record : (Iterable) records) { + String topic = (String) call(record, TOPIC); + if (topic == null) { + continue; + } + received.add(topic); + } + AsyncMessageTracker.received(received); + } catch (Exception e) { + //what is recorded is extra information, never a reason to break the SUT + SimpleLogger.uniqueWarn("Failed to read the records a Kafka consumer returned: " + e); + } + } + + private static void recordEmitted(Object record) { + try { + String topic = (String) call(record, TOPIC); + Object value = call(record, VALUE); + + ExecutionTracer.addEmittedMessage(new EmittedMessage( + topic, + asText(value), + value == null ? null : value.getClass().getName())); + } catch (Exception e) { + SimpleLogger.uniqueWarn("Failed to read a record given to a Kafka producer: " + e); + } + } + + /** + * @return the payload as text: a String as given, or the first bytes read as UTF-8, enough + * for {@link EmittedMessage} to keep and to tell it was cut. Matches the wire only with + * StringSerializer or ByteArraySerializer (spring-kafka's JsonSerializer quotes a String). + * Null for any other object: its wire form is up to the serializer, and computing a text form + * could run SUT code from inside the replacement. + */ + static String asText(Object value) { + if (value instanceof String) { + return (String) value; + } + if (value instanceof byte[]) { + byte[] bytes = (byte[]) value; + return new String(bytes, 0, Math.min(bytes.length, MAX_DECODED_BYTES), StandardCharsets.UTF_8); + } + return null; + } + + /** + * Call a public method with no inputs, made accessible as the class of the object might not + * be public. + */ + private static Object call(Object target, String methodName) throws ReflectiveOperationException { + Map methods = accessors.get(target.getClass()); + Method m = methods.get(methodName); + if (m == null) { + m = target.getClass().getMethod(methodName); + m.setAccessible(true); + methods.put(methodName, m); + } + return m.invoke(target); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/visitor/methodv/MethodReplacementMethodVisitor.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/visitor/methodv/MethodReplacementMethodVisitor.java index 9d40159fd2..8234bee0e7 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/visitor/methodv/MethodReplacementMethodVisitor.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/coverage/visitor/methodv/MethodReplacementMethodVisitor.java @@ -159,7 +159,8 @@ public void visitMethodInsn(int opcode, String owner, String name, Method m = r.get(); replaceMethod(m); if(isConstructor){ - handleConstruct(m, candidateClasses.get(0).getClass()); + //the class the replacement is in, which is not necessarily the only one targeting the owner + handleConstruct(m, (Class) m.getDeclaringClass()); } //Can be put back if needed for debugging... otherwise it is just expensive overhead diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/AgentController.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/AgentController.java index a2bfe9aa87..690a79065a 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/AgentController.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/AgentController.java @@ -127,6 +127,9 @@ public static void start(int port){ case OBJECTIVE_UPDATE_COUNT: handleObjectiveUpdateCount(); break; + case MESSAGE_HANDLING_STATE: + handleMessageHandlingState(); + break; default: SimpleLogger.error("Unrecognized command: "+command); return; @@ -295,6 +298,16 @@ private static void handleObjectiveUpdateCount(){ } } + private static void handleMessageHandlingState(){ + try { + String address = (String) in.readObject(); + sendObject(InstrumentationController.getMessageHandlingState(address)); + } catch (Exception e) { + SimpleLogger.error("Failure in handling message handling state: "+e.getMessage()); + sendUnknown(); + } + } + private static void handleExtractingSpecifiedDto(){ try { Object msg = in.readObject(); diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/Command.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/Command.java index 56ffa74907..cad6c04262 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/Command.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/Command.java @@ -34,5 +34,6 @@ public enum Command implements Serializable { * The controller polls it to detect stable coverage while waiting for an AsyncAPI action * to finish; it counts updates, not distinct objectives or executions of covered code. */ - OBJECTIVE_UPDATE_COUNT + OBJECTIVE_UPDATE_COUNT, + MESSAGE_HANDLING_STATE } diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/ServerController.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/ServerController.java index f56ed10410..e311fbe058 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/ServerController.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/external/ServerController.java @@ -12,6 +12,7 @@ import java.net.Socket; import java.util.Collection; import java.util.List; +import java.util.Objects; /** * The SutController will start a TCP server, and the Agent in the external @@ -381,6 +382,41 @@ public synchronized Long getObjectiveUpdateCount(long timeoutMs) { } } + /** + * Cheap enough to be polled every few milliseconds, as {@link #getObjectiveUpdateCount()}. + * + * @param address where the SUT receives messages from, eg a Kafka topic + * @return what the SUT received from {@code address} in its current action, and how much of + * it it is done with, or null if the agent could not be asked, or failed to answer + */ + public synchronized MessageHandlingState getMessageHandlingState(String address) { + + Objects.requireNonNull(address); + + boolean sent = sendCommand(Command.MESSAGE_HANDLING_STATE); + if (!sent) { + SimpleLogger.error("Failed to send message"); + return null; + } + + if (!sendObject(address)) { + SimpleLogger.error("Failed to send address"); + return null; + } + + Object response = waitAndGetResponse(); + if (response == null) { + SimpleLogger.error("Failed to read response about message handling state"); + return null; + } + + if (!(response instanceof MessageHandlingState)) { + throw new IllegalStateException(errorMsgExpectingResponse(response, "a MessageHandlingState")); + } + + return (MessageHandlingState) response; + } + public synchronized UnitsInfoRecorder getUnitsInfoRecorder(){ boolean sent = sendCommand(Command.UNITS_INFO); diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTracker.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTracker.java new file mode 100644 index 0000000000..5e228b2dd9 --- /dev/null +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTracker.java @@ -0,0 +1,188 @@ +package org.evomaster.client.java.instrumentation.staticstate; + +import org.evomaster.client.java.instrumentation.MessageHandlingState; + +import java.util.*; + +/** + * Keeps track, per test, of the messages the SUT received from a broker client (eg a Kafka + * consumer's poll) and of whether it is done handling them, so that the controller can tell when + * a message-driven service has finished with the message an action published. + * + * Nothing tells when a handler is done, so a batch counts as handled when the thread that + * received it asks the client for more, as a consumer loop does. Blind spots of this heuristic: + *
    + *
  • Work handed off to another thread (an executor, an async listener) reads as handled + * while still running.
    • + *
  • A thread that never asks for more (it stops, or closes the client) leaves its last + * batch unhandled for good.
    • + *
  • Asking for more without waiting, eg a Kafka poll with a zero timeout, does not count: + * Spring Kafka does that to keep the consumer alive while it retries a failed batch. A loop + * that only ever polls with a zero timeout never looks done.
    • + *
  • A failed message the client is told to redeliver (eg a seek back, as Spring Kafka's + * error handlers do) reads as handled until it is received again.
    • + *
+ * + * The SUT's calls are recorded by method replacements, which do not record the calls the driver + * makes itself while publishing an action, see {@link ExecutionTracer#isExecutingDriverAction()}. + * Reset with {@link ExecutionTracer#reset()}. + */ +public class AsyncMessageTracker { + + private static final Object lock = new Object(); + + /** + * How many times the tracker was reset, ie which test it is in. Only changed under + * {@link #lock}; volatile so that it can be read without it. + */ + private static volatile long generation = 0; + + /** + * Thread id -> the batches that thread received and has not asked for more after. + */ + private static final Map> openBatches = new HashMap<>(); + + /** + * Action index -> address -> received and handled counts. + */ + private static final Map> progress = new HashMap<>(); + + /** + * What one call to the client returned to one thread. + */ + private static final class Batch { + + /** + * The action current when the batch was received, which it is counted for. + */ + final int actionIndex; + + /** + * Address -> how many of the batch's messages came from it. + */ + final Map messagesPerAddress = new LinkedHashMap<>(); + + Batch(int actionIndex) { + this.actionIndex = actionIndex; + } + } + + /** + * Counts for one address within one action. + */ + private static final class Progress { + int received; + int handled; + } + + /** + * Forget everything. Called by {@link ExecutionTracer#reset()} at the start of every test. + */ + public static void reset() { + synchronized (lock) { + openBatches.clear(); + progress.clear(); + generation++; + } + } + + /** + * The current thread is asking the client for more messages. If it waits for them, the ones + * it received before count as handled. + * + * @param waitsForMessages whether the call blocks until messages arrive or a timeout runs + * out, as opposed to returning only what is already there + */ + public static void askingForMore(boolean waitsForMessages) { + if (!waitsForMessages) { + return; + } + long thread = Thread.currentThread().getId(); + synchronized (lock) { + List done = openBatches.remove(thread); + if (done == null) { + return; + } + for (Batch batch : done) { + batch.messagesPerAddress.forEach((address, count) -> + progressOf(batch.actionIndex, address).handled += count); + } + } + } + + /** + * The current thread received these messages, in one call to the client. They count for the + * current action, and stay unhandled until this thread asks for more, see + * {@link #askingForMore(boolean)}. + * + * @param addresses where each message came from, eg its Kafka topic, one entry per message + */ + public static void received(List addresses) { + Objects.requireNonNull(addresses); + if (addresses.isEmpty()) { + return; + } + /* + The action index is read before taking this lock, so it is never held while waiting for + ExecutionTracer's. A reset can then come in between, and the batch would land in the next + test under an index of the previous one. The generation, read before the index, detects + that: if it changed by the time the lock is held, the batch is dropped. + */ + long readInGeneration = generation; + int actionIndex = ExecutionTracer.getActionIndex(); + received(addresses, readInGeneration, actionIndex); + } + + /** + * As {@link #received(List)}, with the generation and the action index already read, so that + * tests can put a reset in between. + */ + static void received(List addresses, long readInGeneration, int actionIndex) { + addresses.forEach(Objects::requireNonNull); + long thread = Thread.currentThread().getId(); + + synchronized (lock) { + if (readInGeneration != generation) { + return; + } + Batch batch = new Batch(actionIndex); + for (String address : addresses) { + batch.messagesPerAddress.merge(address, 1, Integer::sum); + progressOf(actionIndex, address).received++; + } + openBatches.computeIfAbsent(thread, k -> new ArrayList<>()).add(batch); + } + } + + /** + * @return how many times the tracker was reset so far + */ + static long getGeneration() { + return generation; + } + + /** + * @param address where messages come from, eg a Kafka topic + * @return what the SUT received from {@code address} in the current action, and how much of + * it is handled. Never null: nothing received reads as all zeros. + */ + public static MessageHandlingState getHandlingState(String address) { + Objects.requireNonNull(address); + int actionIndex = ExecutionTracer.getActionIndex(); + synchronized (lock) { + Map ofAction = progress.get(actionIndex); + Progress p = ofAction == null ? null : ofAction.get(address); + if (p == null) { + return new MessageHandlingState(0, 0); + } + return new MessageHandlingState(p.received, p.received - p.handled); + } + } + + private static Progress progressOf(int actionIndex, String address) { + assert Thread.holdsLock(lock); + return progress + .computeIfAbsent(actionIndex, k -> new HashMap<>()) + .computeIfAbsent(address, k -> new Progress()); + } +} diff --git a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracer.java b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracer.java index b635211936..593d63b618 100644 --- a/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracer.java +++ b/client-java/instrumentation/src/main/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracer.java @@ -50,6 +50,25 @@ public class ExecutionTracer { */ private static boolean executingAction = false; + /** + * Whether the current thread is running the driver's code that publishes an AsyncAPI action, + * set by the controller around it. Per thread, as the SUT's own threads keep running in the + * meantime, and what they do is exactly what is to be recorded. + * + * This matters when the driver and the SUT share the JVM and the driver's classes get the + * method replacements too. That happens to a driver class loaded once the agent is in place + * and the replacement categories are set, eg in a JVM where an earlier test already started a + * SUT, as can happen with EvoMaster's own E2E tests. The broker client calls the driver makes + * to publish, and to wait for the reply, would otherwise be recorded as messages the SUT + * emitted and consumed. A package filter cannot tell the two apart, as they can share a + * package prefix. Not reset in {@link #reset()}: whoever sets it on a thread clears it. + * + * Only the thread running executeAsyncApiAction is covered, and only while it runs. Broker + * client calls the driver makes on threads it starts itself (eg a consumer it keeps polling in + * the background) are recorded as the SUT's when the driver's classes get the replacements. + */ + private static final ThreadLocal executingDriverAction = ThreadLocal.withInitial(() -> false); + /* Careful if you change the signature of any of the methods in this class, as they are injected in the @@ -172,6 +191,8 @@ public static void reset() { skippedExternalServices = new ArrayList<>(); executingInitDynamoDB = false; } + //outside the lock above, so that the tracker's lock is never taken while holding this one + AsyncMessageTracker.reset(); } @@ -255,6 +276,36 @@ public static void setExecutingAction(boolean executingAction) { ExecutionTracer.executingAction = executingAction; } + /** + * @return whether the current thread is running the driver's code that publishes an AsyncAPI + * action, whose broker client calls are not the SUT's and must not be recorded as such. Calls + * on any other thread are not covered, see {@link #executingDriverAction} + */ + public static boolean isExecutingDriverAction() { + return executingDriverAction.get(); + } + + /** + * Mark the current thread as running, or no longer running, the driver's code that publishes + * an AsyncAPI action. Only that thread is affected. + */ + public static void setExecutingDriverAction(boolean executing) { + if (executing) { + executingDriverAction.set(true); + } else { + executingDriverAction.remove(); + } + } + + /** + * @return the index of the action currently being evaluated in the test + */ + public static int getActionIndex() { + synchronized (lock) { + return actionIndex; + } + } + public static void setAction(Action action) { synchronized (lock) { setKillSwitch(false); @@ -508,6 +559,13 @@ public static void addRedisCommand(RedisCommand info){ getCurrentAdditionalInfo().addRedisCommand(info); } + /** + * Record a message the SUT handed to a broker client to publish, for the current action. + */ + public static void addEmittedMessage(EmittedMessage message) { + getCurrentAdditionalInfo().addEmittedMessage(message); + } + public static void addDynamoDbInfo(DynamoDbCommand info) { if (!executingInitDynamoDB) getCurrentAdditionalInfo().addDynamoDbInfo(info); diff --git a/client-java/instrumentation/src/test/java/com/foo/somedifferentpackage/examples/kafka/KafkaClientsImpl.java b/client-java/instrumentation/src/test/java/com/foo/somedifferentpackage/examples/kafka/KafkaClientsImpl.java new file mode 100644 index 0000000000..93d964d6de --- /dev/null +++ b/client-java/instrumentation/src/test/java/com/foo/somedifferentpackage/examples/kafka/KafkaClientsImpl.java @@ -0,0 +1,116 @@ +package com.foo.somedifferentpackage.examples.kafka; + +import org.apache.kafka.clients.consumer.Consumer; +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.clients.consumer.MockConsumer; +import org.apache.kafka.clients.consumer.OffsetResetStrategy; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.MockProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.TopicPartition; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.evomaster.client.java.instrumentation.example.kafka.KafkaClients; +import org.springframework.kafka.core.DefaultKafkaProducerFactory; +import org.springframework.kafka.core.KafkaTemplate; + +import java.time.Duration; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + +public class KafkaClientsImpl implements KafkaClients { + + private static final String NO_BROKER = "localhost:9"; + + private static final Duration WAIT = Duration.ofMillis(10); + + private static final long WAIT_MS = 10; + + private static final long SEND_TIMEOUT_SECONDS = 10; + + @Override + public int pollAConsumer(String topic, int records) { + TopicPartition partition = new TopicPartition(topic, 0); + MockConsumer mock = new MockConsumer<>(OffsetResetStrategy.EARLIEST); + mock.assign(Collections.singletonList(partition)); + mock.updateBeginningOffsets(Collections.singletonMap(partition, 0L)); + for (int i = 0; i < records; i++) { + mock.addRecord(new ConsumerRecord<>(topic, 0, i, "key" + i, "value" + i)); + } + Consumer consumer = mock; + return consumer.poll(WAIT).count(); + } + + private static KafkaConsumer unsubscribedConsumer() { + Properties props = new Properties(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, NO_BROKER); + //instances rather than class names, which kafka-clients would load outside this class loader + return new KafkaConsumer<>(props, new StringDeserializer(), new StringDeserializer()); + } + + @Override + public void pollAnUnsubscribedKafkaConsumer() { + KafkaConsumer consumer = unsubscribedConsumer(); + try { + consumer.poll(WAIT); + } finally { + consumer.close(Duration.ZERO); + } + } + + @SuppressWarnings("deprecation") + @Override + public void pollAnUnsubscribedKafkaConsumerTheLegacyWay() { + KafkaConsumer consumer = unsubscribedConsumer(); + try { + consumer.poll(WAIT_MS); + } finally { + consumer.close(Duration.ZERO); + } + } + + @Override + public boolean sendWithAKafkaProducer(String topic, String key, String payload) { + Properties props = new Properties(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, NO_BROKER); + props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, "0"); + KafkaProducer producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer()); + try { + Future sent = producer.send(new ProducerRecord<>(topic, key, payload)); + return sent.isDone(); + } finally { + producer.close(Duration.ZERO); + } + } + + @Override + public int sendWithAKafkaTemplate(String topic, String key, String payload) throws Exception { + + MockProducer actual = new MockProducer<>(true, new StringSerializer(), new StringSerializer()); + + Map configs = new HashMap<>(); + configs.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, NO_BROKER); + + DefaultKafkaProducerFactory factory = + new DefaultKafkaProducerFactory(configs, new StringSerializer(), new StringSerializer()) { + @Override + protected Producer createRawProducer(Map rawConfigs) { + return actual; + } + }; + + KafkaTemplate template = new KafkaTemplate<>(factory); + template.send(topic, key, payload).get(SEND_TIMEOUT_SECONDS, TimeUnit.SECONDS); + factory.destroy(); + + return actual.history().size(); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/EmittedMessageTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/EmittedMessageTest.java new file mode 100644 index 0000000000..25dc988700 --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/EmittedMessageTest.java @@ -0,0 +1,101 @@ +package org.evomaster.client.java.instrumentation; + +import org.junit.jupiter.api.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; + +import static org.junit.jupiter.api.Assertions.*; + +public class EmittedMessageTest { + + private static final String TOPIC = "replies"; + + /** + * A character outside the Basic Multilingual Plane, two chars in Java: a surrogate pair. + */ + private static final String SUPPLEMENTARY = new String(Character.toChars(0x1F600)); + + private static String repeat(String s, int times) { + StringBuilder text = new StringBuilder(); + for (int i = 0; i < times; i++) { + text.append(s); + } + return text.toString(); + } + + @Test + public void testALongPayloadIsCut() { + + String tooLong = repeat("y", EmittedMessage.MAX_TEXT_LENGTH + 1); + + EmittedMessage message = new EmittedMessage(TOPIC, tooLong, String.class.getName()); + + assertEquals(EmittedMessage.MAX_TEXT_LENGTH, message.getPayload().length()); + assertTrue(message.isPayloadTruncated()); + } + + @Test + public void testAShortPayloadIsKeptWhole() { + + EmittedMessage message = new EmittedMessage(TOPIC, "{}", String.class.getName()); + + assertEquals("{}", message.getPayload()); + assertFalse(message.isPayloadTruncated()); + } + + @Test + public void testAPayloadOfExactlyTheMaximumIsNotCut() { + + String payload = repeat("y", EmittedMessage.MAX_TEXT_LENGTH); + + EmittedMessage message = new EmittedMessage(TOPIC, payload, String.class.getName()); + + assertEquals(payload, message.getPayload()); + assertFalse(message.isPayloadTruncated()); + } + + @Test + public void testTheCutDoesNotSplitASurrogatePair() { + + //the pair straddles the cut: its high surrogate is the last char kept + String payload = repeat("y", EmittedMessage.MAX_TEXT_LENGTH - 1) + SUPPLEMENTARY; + + EmittedMessage message = new EmittedMessage(TOPIC, payload, String.class.getName()); + + assertEquals(repeat("y", EmittedMessage.MAX_TEXT_LENGTH - 1), message.getPayload()); + assertTrue(message.isPayloadTruncated()); + } + + @Test + public void testAPairBeforeTheCutIsKept() { + + String kept = repeat("y", EmittedMessage.MAX_TEXT_LENGTH - 2) + SUPPLEMENTARY; + + EmittedMessage message = new EmittedMessage(TOPIC, kept + "y", String.class.getName()); + + assertEquals(kept, message.getPayload()); + assertTrue(message.isPayloadTruncated()); + } + + @Test + public void testItSurvivesTheTripToAnExternalController() throws Exception { + + EmittedMessage message = new EmittedMessage(TOPIC, "{}", "java.lang.String"); + + ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + try (ObjectOutputStream out = new ObjectOutputStream(bytes)) { + out.writeObject(message); + } + EmittedMessage copy; + try (ObjectInputStream in = new ObjectInputStream(new ByteArrayInputStream(bytes.toByteArray()))) { + copy = (EmittedMessage) in.readObject(); + } + + assertEquals(TOPIC, copy.getAddress()); + assertEquals("{}", copy.getPayload()); + assertEquals("java.lang.String", copy.getPayloadType()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/MessageHandlingStateTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/MessageHandlingStateTest.java new file mode 100644 index 0000000000..191f22432e --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/MessageHandlingStateTest.java @@ -0,0 +1,24 @@ +package org.evomaster.client.java.instrumentation; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.*; + +public class MessageHandlingStateTest { + + @Test + public void testHandledNeedsSomethingReceivedAndNothingPending() { + + assertFalse(new MessageHandlingState(0, 0).isHandled(), "nothing received"); + assertFalse(new MessageHandlingState(2, 1).isHandled(), "one still pending"); + assertTrue(new MessageHandlingState(2, 0).isHandled()); + } + + @Test + public void testCountsThatCannotBeAreRejected() { + + assertThrows(IllegalArgumentException.class, () -> new MessageHandlingState(-1, 0)); + assertThrows(IllegalArgumentException.class, () -> new MessageHandlingState(1, -1)); + assertThrows(IllegalArgumentException.class, () -> new MessageHandlingState(1, 2)); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementListTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementListTest.java index cbdee4cd56..39acdd7c89 100644 --- a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementListTest.java +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/ReplacementListTest.java @@ -23,6 +23,14 @@ public void testIntegerReplacement(){ assertTrue(list.size() > 0); } + @Test + public void testSeveralClassesCanTargetTheSameClassWhenAtMostOneReplacesItsConstructors(){ + + //poll(Duration) and poll(long) are in separate classes, and neither replaces a constructor + List list = ReplacementList.getReplacements("org/apache/kafka/clients/consumer/KafkaConsumer", true); + assertEquals(2, list.size(), "Replacements: " + list); + } + @Test public void testReplacementMethods() { diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacementTest.java new file mode 100644 index 0000000000..cf6d399984 --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerClassReplacementTest.java @@ -0,0 +1,92 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.consumer.ConsumerConfig; +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.apache.kafka.common.serialization.StringDeserializer; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.time.Duration; +import java.util.Collections; +import java.util.Properties; + +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaConsumerInterfaceClassReplacementTest.*; +import static org.junit.jupiter.api.Assertions.*; + +/** + * KafkaConsumer.poll(Duration). Without a broker a real KafkaConsumer returns no records, but an + * unsubscribed one fails its poll straight away: enough to check that the call reaches the + * original, its exception comes back unchanged, and the thread's previous batch is handled. + */ +public class KafkaConsumerClassReplacementTest { + + /** + * Never contacted: an unsubscribed consumer fails before connecting. + */ + static final String NO_BROKER = "localhost:9"; + + private static final Duration CLOSE_TIMEOUT = Duration.ZERO; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.setExecutingDriverAction(false); + ExecutionTracer.reset(); + } + + static KafkaConsumer unsubscribedConsumer() { + Properties props = new Properties(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, NO_BROKER); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); + return new KafkaConsumer<>(props); + } + + /** + * As if this thread had received a record from {@link KafkaConsumerInterfaceClassReplacementTest#TOPIC} + * and was handling it. + */ + static void handlingARecord() { + AsyncMessageTracker.received(Collections.singletonList( + TOPIC)); + assertEquals(1, state().getUnhandled()); + } + + @Test + public void testThePollReachesTheOriginalAndEndsThePreviousBatch() { + + handlingARecord(); + + KafkaConsumer consumer = unsubscribedConsumer(); + try { + assertThrows(IllegalStateException.class, () -> KafkaConsumerClassReplacement.poll(consumer, WAIT)); + } finally { + consumer.close(CLOSE_TIMEOUT); + } + + assertTrue(state().isHandled(), "back in poll, yet the batch is not handled: " + state()); + } + + @Test + public void testTheDriversPollsDoNotEndTheSutsBatch() { + + handlingARecord(); + ExecutionTracer.setExecutingDriverAction(true); + + KafkaConsumer consumer = unsubscribedConsumer(); + try { + assertThrows(IllegalStateException.class, () -> KafkaConsumerClassReplacement.poll(consumer, WAIT)); + } finally { + consumer.close(CLOSE_TIMEOUT); + } + + assertEquals(1, state().getUnhandled()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacementTest.java new file mode 100644 index 0000000000..91f6720111 --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceClassReplacementTest.java @@ -0,0 +1,158 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.consumer.ConsumerRecord; +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.MockConsumer; +import org.apache.kafka.clients.consumer.OffsetResetStrategy; +import org.apache.kafka.common.KafkaException; +import org.apache.kafka.common.TopicPartition; +import org.evomaster.client.java.instrumentation.MessageHandlingState; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.time.Duration; +import java.util.Collections; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Consumer.poll(Duration), against kafka-clients' own mock consumer, so no broker is needed. + */ +public class KafkaConsumerInterfaceClassReplacementTest { + + static final String TOPIC = "requests"; + + static final Duration WAIT = Duration.ofMillis(10); + + private static final TopicPartition PARTITION = new TopicPartition(TOPIC, 0); + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.setExecutingDriverAction(false); + ExecutionTracer.reset(); + } + + /** + * A consumer assigned to {@link #TOPIC}, with {@code records} records waiting from offset 0. + */ + static MockConsumer consumerWith(int records) { + MockConsumer consumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST); + consumer.assign(Collections.singletonList(PARTITION)); + consumer.updateBeginningOffsets(Collections.singletonMap(PARTITION, 0L)); + for (int i = 0; i < records; i++) { + consumer.addRecord(new ConsumerRecord<>(TOPIC, 0, i, "key" + i, "value" + i)); + } + return consumer; + } + + static MessageHandlingState state() { + return AsyncMessageTracker.getHandlingState(TOPIC); + } + + @Test + public void testWhatThePollReturnsIsRecordedAsConsumed() { + + ConsumerRecords records = (ConsumerRecords) + KafkaConsumerInterfaceClassReplacement.poll(consumerWith(2), WAIT); + + assertEquals(2, records.count(), "the records did not reach the caller"); + + MessageHandlingState state = state(); + assertEquals(2, state.getReceived()); + assertEquals(2, state.getUnhandled()); + } + + @Test + public void testTheNextWaitingPollHandlesTheBatch() { + + MockConsumer consumer = consumerWith(1); + + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + + assertTrue(state().isHandled(), "back in poll, yet the batch is not handled: " + state()); + } + + @Test + public void testAPollThatDoesNotWaitLeavesTheBatchUnhandled() { + + MockConsumer consumer = consumerWith(1); + + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + KafkaConsumerInterfaceClassReplacement.poll(consumer, Duration.ZERO); + + assertEquals(1, state().getUnhandled()); + } + + @Test + public void testAnEmptyPollRecordsNothing() { + + ConsumerRecords records = (ConsumerRecords) + KafkaConsumerInterfaceClassReplacement.poll(consumerWith(0), WAIT); + + assertTrue(records.isEmpty()); + assertEquals(0, state().getReceived()); + } + + @Test + public void testTheDriversPollsAreNotRecorded() { + + ExecutionTracer.setExecutingDriverAction(true); + + ConsumerRecords records = (ConsumerRecords) + KafkaConsumerInterfaceClassReplacement.poll(consumerWith(1), WAIT); + + assertEquals(1, records.count(), "the driver did not get its records"); + assertEquals(0, state().getReceived(), "the driver's records were recorded as the SUT's"); + } + + @Test + public void testTheDriverPollingDoesNotHandleTheSutsBatch() { + + KafkaConsumerInterfaceClassReplacement.poll(consumerWith(1), WAIT); + + ExecutionTracer.setExecutingDriverAction(true); + KafkaConsumerInterfaceClassReplacement.poll(consumerWith(0), WAIT); + + assertEquals(1, state().getUnhandled()); + } + + @Test + public void testAFailingPollThrowsAsTheOriginalAndStillEndsThePreviousBatch() { + + MockConsumer consumer = consumerWith(1); + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + + KafkaException failure = new KafkaException("broker gone"); + consumer.setPollException(failure); + + KafkaException thrown = assertThrows(KafkaException.class, + () -> KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT)); + + assertSame(failure, thrown, "the exception was wrapped"); + assertTrue(state().isHandled(), "the thread came back for more, so it was done with the batch"); + } + + @Test + public void testARecordPolledAgainIsUnhandledAgain() { + + MockConsumer consumer = consumerWith(1); + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + + //as an error handler does after a failed record + consumer.seek(PARTITION, 0); + consumer.addRecord(new ConsumerRecord<>(TOPIC, 0, 0, "key0", "value0")); + KafkaConsumerInterfaceClassReplacement.poll(consumer, WAIT); + + assertEquals(2, state().getReceived()); + assertEquals(1, state().getUnhandled(), "the record is being handled again"); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacementTest.java new file mode 100644 index 0000000000..65ca88937a --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerInterfaceLegacyPollClassReplacementTest.java @@ -0,0 +1,70 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.consumer.ConsumerRecords; +import org.apache.kafka.clients.consumer.MockConsumer; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + + +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaConsumerInterfaceClassReplacementTest.*; +import static org.junit.jupiter.api.Assertions.*; + +/** + * Consumer.poll(long), against kafka-clients' own mock consumer. + */ +public class KafkaConsumerInterfaceLegacyPollClassReplacementTest { + + private static final long WAIT_MS = 10; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.setExecutingDriverAction(false); + ExecutionTracer.reset(); + } + + @Test + public void testWhatThePollReturnsIsRecordedAndHandledOnTheNextPoll() { + + MockConsumer consumer = consumerWith(3); + + ConsumerRecords records = (ConsumerRecords) + KafkaConsumerInterfaceLegacyPollClassReplacement.poll(consumer, WAIT_MS); + + assertEquals(3, records.count()); + assertEquals(3, state().getUnhandled()); + + KafkaConsumerInterfaceLegacyPollClassReplacement.poll(consumer, WAIT_MS); + + assertTrue(state().isHandled()); + } + + @Test + public void testAPollWithoutTimeoutLeavesTheBatchUnhandled() { + + MockConsumer consumer = consumerWith(1); + + KafkaConsumerInterfaceLegacyPollClassReplacement.poll(consumer, WAIT_MS); + KafkaConsumerInterfaceLegacyPollClassReplacement.poll(consumer, 0L); + + assertEquals(1, state().getUnhandled()); + } + + @Test + public void testTheDriversPollsAreNotRecorded() { + + ExecutionTracer.setExecutingDriverAction(true); + + ConsumerRecords records = (ConsumerRecords) + KafkaConsumerInterfaceLegacyPollClassReplacement.poll(consumerWith(1), WAIT_MS); + + assertEquals(1, records.count()); + assertEquals(0, state().getReceived()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacementTest.java new file mode 100644 index 0000000000..d5cb0d4500 --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaConsumerLegacyPollClassReplacementTest.java @@ -0,0 +1,63 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.consumer.KafkaConsumer; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.time.Duration; + +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaConsumerClassReplacementTest.handlingARecord; +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaConsumerClassReplacementTest.unsubscribedConsumer; +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaConsumerInterfaceClassReplacementTest.state; +import static org.junit.jupiter.api.Assertions.*; + +/** + * KafkaConsumer.poll(long), see {@link KafkaConsumerClassReplacementTest} for why an unsubscribed + * consumer is enough. + */ +public class KafkaConsumerLegacyPollClassReplacementTest { + + private static final long WAIT_MS = 10; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.reset(); + } + + @Test + public void testThePollReachesTheOriginalAndEndsThePreviousBatch() { + + handlingARecord(); + + KafkaConsumer consumer = unsubscribedConsumer(); + try { + assertThrows(IllegalStateException.class, () -> KafkaConsumerLegacyPollClassReplacement.poll(consumer, WAIT_MS)); + } finally { + consumer.close(Duration.ZERO); + } + + assertTrue(state().isHandled(), "back in poll, yet the batch is not handled: " + state()); + } + + @Test + public void testAPollWithoutTimeoutLeavesTheBatchUnhandled() { + + handlingARecord(); + + KafkaConsumer consumer = unsubscribedConsumer(); + try { + assertThrows(IllegalStateException.class, () -> KafkaConsumerLegacyPollClassReplacement.poll(consumer, 0L)); + } finally { + consumer.close(Duration.ZERO); + } + + assertEquals(1, state().getUnhandled()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacementTest.java new file mode 100644 index 0000000000..291499de2f --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerClassReplacementTest.java @@ -0,0 +1,99 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerConfig; +import org.apache.kafka.common.serialization.StringSerializer; +import org.evomaster.client.java.instrumentation.EmittedMessage; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.time.Duration; +import java.util.Properties; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicReference; + +import static org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses.KafkaProducerInterfaceClassReplacementTest.*; +import static org.junit.jupiter.api.Assertions.*; + +/** + * KafkaProducer.send, against a real producer with no broker. Told not to wait for one, it fails + * each send through the returned future and the callback rather than by throwing. + */ +public class KafkaProducerClassReplacementTest { + + /** + * Never reached: with no time to wait for metadata, a send fails before connecting. + */ + static final String NO_BROKER = "localhost:9"; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.setExecutingDriverAction(false); + ExecutionTracer.reset(); + } + + static KafkaProducer producerWithNoBroker() { + Properties props = new Properties(); + props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, NO_BROKER); + props.put(ProducerConfig.MAX_BLOCK_MS_CONFIG, "0"); + props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); + props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); + return new KafkaProducer<>(props); + } + + @Test + public void testASendThatFailsLaterIsStillRecorded() { + + KafkaProducer producer = producerWithNoBroker(); + try { + Future sent = KafkaProducerClassReplacement.send(producer, record(PAYLOAD)); + assertThrows(ExecutionException.class, sent::get); + } finally { + producer.close(Duration.ZERO); + } + + EmittedMessage message = onlyEmitted(); + assertEquals(TOPIC, message.getAddress()); + assertEquals(PAYLOAD, message.getPayload()); + } + + @Test + public void testWithACallbackTheCallbackGetsTheFailure() { + + AtomicReference failure = new AtomicReference<>(); + + KafkaProducer producer = producerWithNoBroker(); + try { + KafkaProducerClassReplacement.send(producer, record(PAYLOAD), + (org.apache.kafka.clients.producer.Callback) (metadata, e) -> failure.set(e)); + } finally { + producer.close(Duration.ZERO); + } + + assertNotNull(failure.get(), "the callback was not given to the original"); + assertEquals(1, emitted().size()); + } + + @Test + public void testTheDriversSendsAreNotRecorded() { + + ExecutionTracer.setExecutingDriverAction(true); + + KafkaProducer producer = producerWithNoBroker(); + try { + KafkaProducerClassReplacement.send(producer, record(PAYLOAD)); + } finally { + producer.close(Duration.ZERO); + } + + assertTrue(emitted().isEmpty()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacementTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacementTest.java new file mode 100644 index 0000000000..15792887fe --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/coverage/methodreplacement/thirdpartyclasses/KafkaProducerInterfaceClassReplacementTest.java @@ -0,0 +1,296 @@ +package org.evomaster.client.java.instrumentation.coverage.methodreplacement.thirdpartyclasses; + +import org.apache.kafka.clients.producer.Callback; +import org.apache.kafka.clients.producer.MockProducer; +import org.apache.kafka.clients.producer.Producer; +import org.apache.kafka.clients.producer.ProducerRecord; +import org.apache.kafka.common.serialization.Serializer; +import org.apache.kafka.common.serialization.StringSerializer; +import org.evomaster.client.java.instrumentation.AdditionalInfo; +import org.evomaster.client.java.instrumentation.EmittedMessage; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.lang.reflect.Proxy; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicBoolean; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Producer.send, against kafka-clients' own mock producer, so no broker is needed. + */ +public class KafkaProducerInterfaceClassReplacementTest { + + static final String TOPIC = "replies"; + + static final String KEY = "id-1"; + + static final String PAYLOAD = "{\"result\":42}"; + + static final String HEADER = "correlationId"; + + static final String HEADER_VALUE = "abc"; + + /** + * Not ASCII, to check that bytes are read as UTF-8. + */ + private static final String NON_ASCII = "ñandú €"; + + /** + * Turns any value into bytes, so that payloads of any type can be sent. + */ + private static final Serializer ANYTHING = (topic, data) -> new byte[0]; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.setExecutingDriverAction(false); + ExecutionTracer.reset(); + } + + static MockProducer producer() { + return new MockProducer<>(true, new StringSerializer(), ANYTHING); + } + + static ProducerRecord record(Object payload) { + ProducerRecord record = new ProducerRecord<>(TOPIC, KEY, payload); + record.headers().add(HEADER, HEADER_VALUE.getBytes(StandardCharsets.UTF_8)); + return record; + } + + static List emitted() { + AdditionalInfo info = ExecutionTracer.exposeAdditionalInfoList().get(0); + return info.getEmittedMessagesView(); + } + + static EmittedMessage onlyEmitted() { + List emitted = emitted(); + assertEquals(1, emitted.size(), "Emitted: " + emitted); + return emitted.get(0); + } + + /** + * A payload whose text form must never be computed: that would run SUT code from inside the + * replacement. + */ + private static class Dto { + final AtomicBoolean printed = new AtomicBoolean(false); + + @Override + public String toString() { + printed.set(true); + throw new IllegalStateException("the payload was turned into text"); + } + } + + @Test + public void testASentRecordIsRecordedAsEmitted() throws Exception { + + MockProducer producer = producer(); + + Future sent = KafkaProducerInterfaceClassReplacement.send(producer, record(PAYLOAD)); + + assertTrue(sent.isDone(), "the future of the original did not come back"); + assertNotNull(sent.get()); + assertEquals(1, producer.history().size(), "the record did not reach the producer"); + + EmittedMessage message = onlyEmitted(); + assertEquals(TOPIC, message.getAddress()); + assertEquals(PAYLOAD, message.getPayload()); + assertFalse(message.isPayloadTruncated()); + assertEquals(String.class.getName(), message.getPayloadType()); + } + + @Test + public void testWithACallbackTheCallbackStillRuns() { + + AtomicBoolean called = new AtomicBoolean(false); + Callback callback = (metadata, failure) -> called.set(failure == null && metadata != null); + + KafkaProducerInterfaceClassReplacement.send(producer(), record(PAYLOAD), callback); + + assertTrue(called.get(), "the callback was not given to the original"); + assertEquals(PAYLOAD, onlyEmitted().getPayload()); + } + + @Test + public void testBytesAreReadAsUtf8() { + + KafkaProducerInterfaceClassReplacement.send(producer(), record(NON_ASCII.getBytes(StandardCharsets.UTF_8))); + + EmittedMessage message = onlyEmitted(); + assertEquals(NON_ASCII, message.getPayload()); + assertEquals(byte[].class.getName(), message.getPayloadType()); + } + + @Test + public void testABoxedPrimitiveHasNoText() { + + /* + What goes on the wire for it is up to the serializer: 8 binary bytes with + LongSerializer, not "42". Only its type is kept. + */ + KafkaProducerInterfaceClassReplacement.send(producer(), record(42L)); + + EmittedMessage message = onlyEmitted(); + assertNull(message.getPayload()); + assertEquals(Long.class.getName(), message.getPayloadType()); + } + + @Test + public void testOnlyThePrefixOfLongBytesIsKept() { + + // 3 bytes each in UTF-8, so the bytes are well over the decoded limit + String payload = repeat("\u20ac", 2 * EmittedMessage.MAX_TEXT_LENGTH); + + KafkaProducerInterfaceClassReplacement.send(producer(), record(payload.getBytes(StandardCharsets.UTF_8))); + + EmittedMessage message = onlyEmitted(); + assertEquals(payload.substring(0, EmittedMessage.MAX_TEXT_LENGTH), message.getPayload()); + assertTrue(message.isPayloadTruncated()); + } + + @Test + public void testLongBytesOfAsciiAreCutAndFlagged() { + + String payload = repeat("x", 10 * EmittedMessage.MAX_TEXT_LENGTH); + + KafkaProducerInterfaceClassReplacement.send(producer(), record(payload.getBytes(StandardCharsets.UTF_8))); + + EmittedMessage message = onlyEmitted(); + assertEquals(EmittedMessage.MAX_TEXT_LENGTH, message.getPayload().length()); + assertTrue(message.isPayloadTruncated()); + } + + @Test + public void testBytesOfExactlyTheMaximumAreKeptWhole() { + + String payload = repeat("\u20ac", EmittedMessage.MAX_TEXT_LENGTH); + + KafkaProducerInterfaceClassReplacement.send(producer(), record(payload.getBytes(StandardCharsets.UTF_8))); + + EmittedMessage message = onlyEmitted(); + assertEquals(payload, message.getPayload()); + assertFalse(message.isPayloadTruncated()); + } + + private static String repeat(String s, int times) { + StringBuilder text = new StringBuilder(); + for (int i = 0; i < times; i++) { + text.append(s); + } + return text.toString(); + } + + @Test + public void testAnObjectPayloadIsNotTurnedIntoText() { + + Dto dto = new Dto(); + + KafkaProducerInterfaceClassReplacement.send(producer(), record(dto)); + + EmittedMessage message = onlyEmitted(); + assertNull(message.getPayload()); + assertEquals(Dto.class.getName(), message.getPayloadType()); + assertFalse(dto.printed.get()); + } + + @Test + public void testNoPayload() { + + KafkaProducerInterfaceClassReplacement.send(producer(), new ProducerRecord(TOPIC, null)); + + EmittedMessage message = onlyEmitted(); + assertNull(message.getPayload()); + assertNull(message.getPayloadType()); + } + + @Test + public void testALongPayloadIsCut() { + + StringBuilder payload = new StringBuilder(); + while (payload.length() <= EmittedMessage.MAX_TEXT_LENGTH) { + payload.append("x"); + } + + KafkaProducerInterfaceClassReplacement.send(producer(), record(payload.toString())); + + EmittedMessage message = onlyEmitted(); + assertTrue(message.isPayloadTruncated()); + assertEquals(EmittedMessage.MAX_TEXT_LENGTH, message.getPayload().length()); + } + + @Test + public void testOnlySoManyMessagesAreKeptPerAction() { + + MockProducer producer = producer(); + for (int i = 0; i < AdditionalInfo.MAX_EMITTED_MESSAGES + 1; i++) { + KafkaProducerInterfaceClassReplacement.send(producer, record(PAYLOAD)); + } + + assertEquals(AdditionalInfo.MAX_EMITTED_MESSAGES + 1, producer.history().size(), "a send was dropped"); + assertEquals(AdditionalInfo.MAX_EMITTED_MESSAGES, emitted().size()); + } + + @Test + public void testTheDriversSendsAreNotRecorded() { + + MockProducer producer = producer(); + ExecutionTracer.setExecutingDriverAction(true); + + KafkaProducerInterfaceClassReplacement.send(producer, record(PAYLOAD)); + + assertEquals(1, producer.history().size(), "the driver's record was not sent"); + assertTrue(emitted().isEmpty(), "the driver's record was recorded as the SUT's"); + } + + @Test + public void testASendThatThrowsIsNotRecorded() { + + MockProducer producer = producer(); + producer.close(); + + assertThrows(IllegalStateException.class, + () -> KafkaProducerInterfaceClassReplacement.send(producer, record(PAYLOAD))); + + assertTrue(emitted().isEmpty()); + } + + @Test + public void testARecordSentThroughAWrappingProducerIsRecordedOnce() { + + MockProducer delegate = producer(); + + /* + What Spring's CloseSafeProducer looks like once instrumented: a Producer whose send + calls send on the Producer it wraps, through the replacement. + */ + Producer wrapper = (Producer) Proxy.newProxyInstance( + getClass().getClassLoader(), + new Class[]{Producer.class}, + (proxy, method, args) -> { + if (method.getName().equals("send")) { + return KafkaProducerInterfaceClassReplacement.send(delegate, args[0], args[1]); + } + throw new UnsupportedOperationException(method.getName()); + }); + + KafkaProducerInterfaceClassReplacement.send(wrapper, record(PAYLOAD), (Callback) (metadata, failure) -> {}); + + assertEquals(1, delegate.history().size()); + assertEquals(PAYLOAD, onlyEmitted().getPayload()); + + //and the guard is released afterwards + KafkaProducerInterfaceClassReplacement.send(delegate, record(PAYLOAD)); + assertEquals(2, emitted().size()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClients.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClients.java new file mode 100644 index 0000000000..8f35c418c6 --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClients.java @@ -0,0 +1,44 @@ +package org.evomaster.client.java.instrumentation.example.kafka; + +/** + * A SUT that consumes and emits Kafka records in the ways the Kafka replacements have to catch. + * Only JDK types cross this interface, as the Kafka classes behind it are loaded, and + * instrumented, by another class loader. + */ +public interface KafkaClients { + + /** + * Poll, once, a consumer declared as a Consumer, with {@code records} records waiting on + * {@code topic}. + * + * @return how many records the poll returned + */ + int pollAConsumer(String topic, int records); + + /** + * Poll a KafkaConsumer that was never subscribed, which fails with an IllegalStateException + * before contacting any broker. + */ + void pollAnUnsubscribedKafkaConsumer(); + + /** + * The same, with the poll(long) deprecated since kafka-clients 2.0. + */ + void pollAnUnsubscribedKafkaConsumerTheLegacyWay(); + + /** + * Send a record with a KafkaProducer declared as such, through send(record), which + * KafkaProducer implements by calling send(record, null) on itself. + * + * @return whether the send was over, as it fails at once with no broker to wait for + */ + boolean sendWithAKafkaProducer(String topic, String key, String payload); + + /** + * Send a record with Spring's KafkaTemplate, which sends through the CloseSafeProducer that + * wraps the actual producer: two Producer.send calls for one record. + * + * @return how many records reached the actual producer + */ + int sendWithAKafkaTemplate(String topic, String key, String payload) throws Exception; +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClientsTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClientsTest.java new file mode 100644 index 0000000000..6b0292028b --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/example/kafka/KafkaClientsTest.java @@ -0,0 +1,145 @@ +package org.evomaster.client.java.instrumentation.example.kafka; + +import com.foo.somedifferentpackage.examples.kafka.KafkaClientsImpl; +import org.evomaster.client.java.instrumentation.EmittedMessage; +import org.evomaster.client.java.instrumentation.InputProperties; +import org.evomaster.client.java.instrumentation.InstrumentingClassLoader; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; +import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; +import org.evomaster.client.java.instrumentation.staticstate.ObjectiveRecorder; +import org.evomaster.client.java.instrumentation.staticstate.UnitsInfoRecorder; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * The Kafka replacements applied by the instrumentation, to SUT code declaring the classes or the + * interfaces, and to library code calling replaced methods (KafkaProducer calling itself, + * Spring's CloseSafeProducer calling the producer it wraps), where a record is recorded once. + */ +public class KafkaClientsTest { + + private static final String TOPIC = "replies"; + + private static final String KEY = "id-1"; + + private static final String PAYLOAD = "{\"result\":42}"; + + private static String defaultReplacement; + + @BeforeAll + public static void initClass() { + defaultReplacement = System.getProperty(InputProperties.REPLACEMENT_CATEGORIES); + System.setProperty(InputProperties.REPLACEMENT_CATEGORIES, "BASE,SQL,EXT_0,KAFKA"); + } + + @AfterAll + public static void tearDown() { + if (defaultReplacement == null) { + System.clearProperty(InputProperties.REPLACEMENT_CATEGORIES); + } else { + System.setProperty(InputProperties.REPLACEMENT_CATEGORIES, defaultReplacement); + } + } + + @BeforeEach + public void initTest() { + UnitsInfoRecorder.reset(); + ObjectiveRecorder.reset(true); + ExecutionTracer.reset(); + } + + private static KafkaClients getInstance() throws Exception { + InstrumentingClassLoader cl = new InstrumentingClassLoader("com.foo"); + return (KafkaClients) cl.loadClass(KafkaClientsImpl.class.getName()).newInstance(); + } + + private static List emitted() { + return ExecutionTracer.exposeAdditionalInfoList().get(0).getEmittedMessagesView(); + } + + private static void handlingARecord() { + AsyncMessageTracker.received(Collections.singletonList(TOPIC)); + assertEquals(1, AsyncMessageTracker.getHandlingState(TOPIC).getUnhandled()); + } + + private static void assertBackInPoll() { + assertTrue(AsyncMessageTracker.getHandlingState(TOPIC).isHandled(), + "the poll was not replaced, as the thread's previous batch is still unhandled"); + } + + @Test + public void testAPollThroughTheConsumerInterfaceIsTracked() throws Exception { + + int polled = getInstance().pollAConsumer(TOPIC, 2); + + assertEquals(2, polled); + assertEquals(2, AsyncMessageTracker.getHandlingState(TOPIC).getUnhandled()); + } + + @Test + public void testAPollOnAKafkaConsumerIsTracked() throws Exception { + + KafkaClients sut = getInstance(); + handlingARecord(); + + assertThrows(IllegalStateException.class, sut::pollAnUnsubscribedKafkaConsumer); + + assertBackInPoll(); + } + + @Test + public void testALegacyPollOnAKafkaConsumerIsTracked() throws Exception { + + KafkaClients sut = getInstance(); + handlingARecord(); + + assertThrows(IllegalStateException.class, sut::pollAnUnsubscribedKafkaConsumerTheLegacyWay); + + assertBackInPoll(); + } + + @Test + public void testASendWithAKafkaProducerIsRecordedOnce() throws Exception { + + boolean done = getInstance().sendWithAKafkaProducer(TOPIC, KEY, PAYLOAD); + + assertTrue(done, "with no broker to wait for, the send should be over at once"); + List emitted = emitted(); + assertEquals(1, emitted.size(), "KafkaProducer calling itself recorded the record again: " + emitted); + assertEquals(TOPIC, emitted.get(0).getAddress()); + assertEquals(PAYLOAD, emitted.get(0).getPayload()); + } + + @Test + public void testASendWithSpringsKafkaTemplateIsRecordedOnce() throws Exception { + + int sent = getInstance().sendWithAKafkaTemplate(TOPIC, KEY, PAYLOAD); + + assertEquals(1, sent); + List emitted = emitted(); + assertEquals(1, emitted.size(), "the producer wrapping the actual one recorded the record again: " + emitted); + assertEquals(PAYLOAD, emitted.get(0).getPayload()); + } + + @Test + public void testWithoutTheKafkaCategoryNothingIsTracked() throws Exception { + + System.setProperty(InputProperties.REPLACEMENT_CATEGORIES, "BASE,SQL,EXT_0"); + try { + getInstance().sendWithAKafkaProducer(TOPIC, KEY, PAYLOAD); + getInstance().pollAConsumer(TOPIC, 1); + } finally { + System.setProperty(InputProperties.REPLACEMENT_CATEGORIES, "BASE,SQL,EXT_0,KAFKA"); + } + + assertTrue(emitted().isEmpty()); + assertEquals(0, AsyncMessageTracker.getHandlingState(TOPIC).getReceived()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/external/ServerControllerTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/external/ServerControllerTest.java index 914079c91e..d3ab8bdd89 100644 --- a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/external/ServerControllerTest.java +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/external/ServerControllerTest.java @@ -1,11 +1,14 @@ package org.evomaster.client.java.instrumentation.external; +import org.evomaster.client.java.instrumentation.MessageHandlingState; +import org.evomaster.client.java.instrumentation.staticstate.AsyncMessageTracker; import org.evomaster.client.java.instrumentation.staticstate.ExecutionTracer; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import java.io.IOException; +import java.util.Collections; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; @@ -34,6 +37,10 @@ public class ServerControllerTest { private static final String COMPARISON = "ServerControllerTest_comparison"; + private static final String TOPIC = "ServerControllerTest_requests"; + + private static final Integer NOT_AN_ADDRESS = 42; + /** * A numeric comparison reports three objectives at once: less than, equal and greater than. */ @@ -141,4 +148,42 @@ void testUnresponsiveAgentIsDisconnectedWithinTheReadBudget() throws Exception { executor.shutdownNow(); } } + + @Test + public void testTheMessageHandlingStateComesBackFromTheAgent() { + + MessageHandlingState nothing = server.getMessageHandlingState(TOPIC); + assertEquals(0, nothing.getReceived()); + + AsyncMessageTracker.received(Collections.singletonList(TOPIC)); + + MessageHandlingState pending = server.getMessageHandlingState(TOPIC); + assertEquals(1, pending.getReceived()); + assertEquals(1, pending.getUnhandled()); + + //the agent answers on a thread of its own, so this one is still the one that received + AsyncMessageTracker.askingForMore(true); + + assertTrue(server.getMessageHandlingState(TOPIC).isHandled()); + //and other commands still get their own answers + assertEquals(0L, server.getObjectiveUpdateCount()); + } + + @Test + public void testAQuestionTheAgentFailsToAnswerIsAnsweredWithNull() { + + /* + An address that is not a String makes the agent fail. It still answers, with null, + rather than leaving the controller waiting until its read times out. + */ + Object answer = assertTimeoutPreemptively(Duration.ofMillis(TIMEOUT_MS / 2), () -> { + assertTrue(server.sendCommand(Command.MESSAGE_HANDLING_STATE)); + assertTrue(server.sendObject(NOT_AN_ADDRESS)); + return server.waitAndGetResponse(); + }); + + assertNull(answer); + //and the connection is still in step + assertEquals(0L, server.getObjectiveUpdateCount()); + } } diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTrackerTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTrackerTest.java new file mode 100644 index 0000000000..e9b966728c --- /dev/null +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/AsyncMessageTrackerTest.java @@ -0,0 +1,236 @@ +package org.evomaster.client.java.instrumentation.staticstate; + +import org.evomaster.client.java.instrumentation.Action; +import org.evomaster.client.java.instrumentation.MessageHandlingState; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; + +import static org.junit.jupiter.api.Assertions.*; + +public class AsyncMessageTrackerTest { + + private static final String REQUESTS = "requests"; + + private static final String OTHER = "other"; + + private static final long JOIN_TIMEOUT_MS = 10_000; + + @BeforeEach + public void init() { + ExecutionTracer.reset(); + } + + @AfterEach + public void tearDown() { + ExecutionTracer.reset(); + } + + /** + * As if this thread had received one message from each of these addresses, in one call. + */ + private static void receive(String... addresses) { + AsyncMessageTracker.received(Arrays.asList(addresses)); + } + + private static MessageHandlingState stateOf(String address) { + return AsyncMessageTracker.getHandlingState(address); + } + + private static void startAction(int index) { + ExecutionTracer.setAction(new Action(index, "action" + index, Collections.emptySet(), + new HashMap<>(), new HashMap<>(), new ArrayList<>())); + } + + /** + * Run on a thread of its own, as a second consumer loop would, and wait for it. + */ + private static void onAnotherThread(Runnable runnable) throws InterruptedException { + Thread thread = new Thread(runnable); + thread.start(); + thread.join(JOIN_TIMEOUT_MS); + assertFalse(thread.isAlive(), "the other thread did not finish"); + } + + @Test + public void testNothingReceivedIsNotHandled() { + + MessageHandlingState state = stateOf(REQUESTS); + + assertEquals(0, state.getReceived()); + assertEquals(0, state.getUnhandled()); + assertFalse(state.isHandled(), "nothing was received, so nothing can have been handled"); + } + + @Test + public void testABatchIsHandledWhenTheSameThreadWaitsForMore() { + + receive(REQUESTS, REQUESTS); + + MessageHandlingState received = stateOf(REQUESTS); + assertEquals(2, received.getReceived()); + assertEquals(2, received.getUnhandled()); + assertFalse(received.isHandled()); + + AsyncMessageTracker.askingForMore(true); + + MessageHandlingState handled = stateOf(REQUESTS); + assertEquals(2, handled.getReceived()); + assertEquals(0, handled.getUnhandled()); + assertTrue(handled.isHandled()); + } + + @Test + public void testAskingForMoreWithoutWaitingDoesNotHandleTheBatch() { + + receive(REQUESTS); + + //as Spring Kafka does to keep the consumer alive while it retries a failed batch + AsyncMessageTracker.askingForMore(false); + + assertEquals(1, stateOf(REQUESTS).getUnhandled()); + } + + @Test + public void testAnotherThreadAskingForMoreDoesNotHandleTheBatch() throws InterruptedException { + + receive(REQUESTS); + + onAnotherThread(() -> AsyncMessageTracker.askingForMore(true)); + + assertEquals(1, stateOf(REQUESTS).getUnhandled(), "a thread finished a batch it never received"); + } + + @Test + public void testEachThreadHandlesItsOwnBatch() throws InterruptedException { + + receive(REQUESTS); + onAnotherThread(() -> receive(REQUESTS)); + + AsyncMessageTracker.askingForMore(true); + + MessageHandlingState state = stateOf(REQUESTS); + assertEquals(2, state.getReceived()); + assertEquals(1, state.getUnhandled(), "the other thread's batch was taken as handled"); + } + + @Test + public void testOnlyTheAskedAddressCounts() { + + receive(OTHER); + + assertEquals(0, stateOf(REQUESTS).getReceived()); + assertEquals(1, stateOf(OTHER).getReceived()); + } + + @Test + public void testABatchFromSeveralAddressesIsHandledForAllOfThem() { + + receive(REQUESTS, OTHER); + + assertFalse(stateOf(REQUESTS).isHandled()); + assertFalse(stateOf(OTHER).isHandled()); + + AsyncMessageTracker.askingForMore(true); + + assertTrue(stateOf(REQUESTS).isHandled()); + assertTrue(stateOf(OTHER).isHandled()); + } + + @Test + public void testSeveralBatchesWithoutWaitingInBetweenAreAllHandledAtOnce() { + + receive(REQUESTS); + AsyncMessageTracker.askingForMore(false); + receive(REQUESTS); + + assertEquals(2, stateOf(REQUESTS).getUnhandled()); + + AsyncMessageTracker.askingForMore(true); + + assertEquals(0, stateOf(REQUESTS).getUnhandled()); + } + + @Test + public void testWhatIsReceivedCountsForTheActionItWasReceivedIn() { + + startAction(0); + receive(REQUESTS); + + startAction(1); + assertEquals(0, stateOf(REQUESTS).getReceived(), "what the previous action received was counted for this one"); + + receive(REQUESTS); + AsyncMessageTracker.askingForMore(true); + + MessageHandlingState state = stateOf(REQUESTS); + assertEquals(1, state.getReceived()); + assertTrue(state.isHandled()); + } + + @Test + public void testAMessageReceivedAgainIsUnhandledAgain() { + + receive(REQUESTS); + AsyncMessageTracker.askingForMore(true); + //as after a failed handler, when the consumer is sent back to the message + receive(REQUESTS); + + MessageHandlingState state = stateOf(REQUESTS); + assertEquals(2, state.getReceived()); + assertEquals(1, state.getUnhandled(), "the message delivered again is being handled again"); + } + + @Test + public void testAResetForgetsEverything() { + + receive(REQUESTS); + + ExecutionTracer.reset(); + + assertEquals(0, stateOf(REQUESTS).getReceived()); + + //and the batch left open in the previous test is not handled in this one + AsyncMessageTracker.askingForMore(true); + receive(REQUESTS); + assertEquals(1, stateOf(REQUESTS).getUnhandled()); + } + + @Test + public void testABatchReadIntoAnActionOfATestThatIsOverIsDropped() { + + /* + A consumer thread reads the action index, then a reset starts the next test before + the batch is recorded: the batch belongs to the test that is over, and must not show + up in the next one under that action's index. + */ + startAction(3); + long generation = AsyncMessageTracker.getGeneration(); + int actionIndex = ExecutionTracer.getActionIndex(); + + ExecutionTracer.reset(); + startAction(3); + + AsyncMessageTracker.received(Collections.singletonList(REQUESTS), generation, actionIndex); + + assertEquals(0, stateOf(REQUESTS).getReceived(), "a batch of the previous test was counted in this one"); + + //while one read in this test is counted + AsyncMessageTracker.received(Collections.singletonList(REQUESTS), AsyncMessageTracker.getGeneration(), + ExecutionTracer.getActionIndex()); + assertEquals(1, stateOf(REQUESTS).getReceived()); + } + + @Test + public void testNothingReceivedInACallIsNoBatch() { + + AsyncMessageTracker.received(Collections.emptyList()); + + assertEquals(0, stateOf(REQUESTS).getReceived()); + } +} diff --git a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracerTest.java b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracerTest.java index 62a1700700..64037d0909 100644 --- a/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracerTest.java +++ b/client-java/instrumentation/src/test/java/org/evomaster/client/java/instrumentation/staticstate/ExecutionTracerTest.java @@ -1,16 +1,17 @@ package org.evomaster.client.java.instrumentation.staticstate; import org.evomaster.client.java.instrumentation.Action; +import org.evomaster.client.java.instrumentation.EmittedMessage; import org.evomaster.client.java.instrumentation.shared.TaintInputName; - -import java.nio.charset.StandardCharsets; -import java.util.Collections; - import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; -import static org.junit.jupiter.api.Assertions.assertEquals; +import java.nio.charset.StandardCharsets; +import java.util.Collections; +import java.util.concurrent.atomic.AtomicBoolean; + +import static org.junit.jupiter.api.Assertions.*; public class ExecutionTracerTest { @@ -19,6 +20,10 @@ public class ExecutionTracerTest { */ private static final int OBJECTIVES_PER_COMPARISON = 3; + private static final String TOPIC = "replies"; + + private static final long JOIN_TIMEOUT_MS = 10_000; + private static final String COMPARISON = "ExecutionTracerTest_comparison"; private static final String OTHER_COMPARISON = "ExecutionTracerTest_other"; @@ -145,4 +150,37 @@ public void testEachTestStartsFromZero() { ExecutionTracer.executedNumericComparison(COMPARISON, 0.2, 0.2, 0.2); assertEquals(OBJECTIVES_PER_COMPARISON, ExecutionTracer.getObjectiveUpdateCount()); } + + @Test + public void testTheDriverFlagIsPerThread() throws InterruptedException { + + assertFalse(ExecutionTracer.isExecutingDriverAction()); + + ExecutionTracer.setExecutingDriverAction(true); + try { + assertTrue(ExecutionTracer.isExecutingDriverAction()); + + AtomicBoolean seenElsewhere = new AtomicBoolean(true); + Thread sut = new Thread(() -> seenElsewhere.set(ExecutionTracer.isExecutingDriverAction())); + sut.start(); + sut.join(JOIN_TIMEOUT_MS); + + assertFalse(seenElsewhere.get(), "the SUT's threads were taken for the driver"); + } finally { + ExecutionTracer.setExecutingDriverAction(false); + } + + assertFalse(ExecutionTracer.isExecutingDriverAction()); + } + + @Test + public void testWhatTheSutEmitsIsRecordedForTheCurrentAction() { + + EmittedMessage message = new EmittedMessage(TOPIC, null, null); + + ExecutionTracer.addEmittedMessage(message); + + assertEquals(Collections.singletonList(message), + ExecutionTracer.exposeAdditionalInfoList().get(0).getEmittedMessagesView()); + } } diff --git a/core-tests/e2e-tests/e2e-tests-utils/src/test/java/org/evomaster/e2etests/utils/AsyncApiTestBase.java b/core-tests/e2e-tests/e2e-tests-utils/src/test/java/org/evomaster/e2etests/utils/AsyncApiTestBase.java index a99f893db0..0012e56f7c 100644 --- a/core-tests/e2e-tests/e2e-tests-utils/src/test/java/org/evomaster/e2etests/utils/AsyncApiTestBase.java +++ b/core-tests/e2e-tests/e2e-tests-utils/src/test/java/org/evomaster/e2etests/utils/AsyncApiTestBase.java @@ -3,6 +3,7 @@ import com.google.inject.Injector; import com.webfuzzing.commons.faults.FaultCategory; import kotlin.Pair; +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto; import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; import org.evomaster.client.java.instrumentation.shared.ObjectiveNaming; import org.evomaster.core.Main; @@ -132,6 +133,15 @@ protected long countSettleEnd(Solution solution, AsyncApiRep .count(); } + /** + * Every message the service was recorded emitting, across the whole solution. + */ + protected List emittedMessagesOf(Solution solution) { + return allResultsOf(solution).stream() + .flatMap(r -> r.getEmittedMessages().stream()) + .collect(Collectors.toList()); + } + protected void assertReplied(Solution solution, String operation) { boolean ok = resultsOf(solution, operation).stream().anyMatch(r -> r.getOutcome() == AsyncApiOutcome.REPLIED); assertTrue(ok, "With seed " + defaultSeed + ": no reply to '" + operation + "' was ever received"); diff --git a/core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener/src/test/java/org/evomaster/e2etests/spring/asyncapi/listener/ShopKafkaHandlerReturnEMTest.java b/core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener/src/test/java/org/evomaster/e2etests/spring/asyncapi/listener/ShopKafkaHandlerReturnEMTest.java new file mode 100644 index 0000000000..d88ff2ab1c --- /dev/null +++ b/core-tests/e2e-tests/spring/spring-asyncapi-kafka-listener/src/test/java/org/evomaster/e2etests/spring/asyncapi/listener/ShopKafkaHandlerReturnEMTest.java @@ -0,0 +1,104 @@ +package org.evomaster.e2etests.spring.asyncapi.listener; + +import com.foo.asyncapi.ShopKafkaController; +import com.google.inject.Injector; +import kotlin.Pair; +import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; +import org.evomaster.core.EMConfig; +import org.evomaster.core.problem.asyncapi.data.AsyncApiIndividual; +import org.evomaster.core.problem.asyncapi.data.AsyncApiOutcome; +import org.evomaster.core.search.Solution; +import org.evomaster.e2etests.utils.AsyncApiTestBase; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * The shop service with each message let go once its listener has returned, as told by Spring + * Kafka's listener container polling again for more. The container polls through the Consumer + * interface, so this is what the Kafka instrumentation sees of a {@code @KafkaListener}. + * + * A class of its own, rather than one more test in {@link ShopKafkaEMTest}: the Kafka + * replacements can only be asked for when {@link #initClass()} starts, and instruments, the + * service. Each test class runs in a JVM of its own in this module. + */ +public class ShopKafkaHandlerReturnEMTest extends AsyncApiTestBase { + + private static final String SUT_PACKAGE = "com.foo.asyncapi.shop."; + + private static final String EXPRESS_DISPATCH = SUT_PACKAGE + "ExpressDispatch"; + + private static final String URGENT_ESCALATION = SUT_PACKAGE + "UrgentEscalation"; + + private static final List OPERATIONS = Arrays.asList("placeOrder", "requestInvoice", "adjustStock"); + + @BeforeAll + public static void initClass() throws Exception { + + EMConfig config = new EMConfig(); + config.setInstrumentMR_KAFKA(true); + + AsyncApiTestBase.initClass(new ShopKafkaController(), config); + } + + /** + * Express shipping and an urgent priority are reached as in {@link ShopKafkaEMTest#testRunEM()}, + * by taint analysis: the listener compares the payload's strings before it returns, so what + * it compared them with is recorded within the window. + * + * Not asserted: the invoice. Its listener hands the rendering to an executor and returns at + * once, so the wait ends before the invoice is rendered: the blind spot of this completion. + */ + @Test + public void testRunEMWithHandlerReturn() throws Throwable { + + runTestHandlingFlaky( + "ShopKafkaHandlerReturnEM", + "org.foo.asyncapi.ShopKafkaHandlerReturnEM", + 300, + false, + (args) -> { + + /* + A cap far longer than a listener here ever needs: a wait that went to it + would mean the end of the handler was never seen. + */ + setOption(args, "asyncApiSettleMs", "5000"); + setOption(args, "asyncApiCompletion", "HANDLER_RETURN"); + setOption(args, "instrumentMR_KAFKA", "true"); + + //as in ShopKafkaEMTest: more, shorter tests for the budget + setOption(args, "maxTestSize", "3"); + + Pair> run = initAndDebug(args); + Solution solution = run.getSecond(); + + assertTrue(countOutcome(solution, AsyncApiOutcome.PUBLISHED) > 0, "nothing was published"); + assertEquals(0, countOutcome(solution, AsyncApiOutcome.PUBLISH_FAILED), "a message never left"); + assertEverySettled(solution); + + for (String operation : OPERATIONS) { + long ended = resultsOf(solution, operation).stream() + .filter(r -> r.getSettleEndedBy() == AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN) + .count(); + assertTrue(ended > 0, "no wait on '" + operation + "' ended on its listener returning"); + } + assertEquals(0, countSettleEnd(solution, AsyncApiReplyDto.SettleEndReason.DEADLINE), + "a wait went to the deadline, so a listener was never seen returning"); + assertEquals(0, countSettleEnd(solution, AsyncApiReplyDto.SettleEndReason.INTERRUPTED), + "a wait was interrupted"); + + assertClassCovered(run, EXPRESS_DISPATCH); + assertClassCovered(run, URGENT_ESCALATION); + + //the service publishes nothing, so nothing is recorded as emitted + assertTrue(emittedMessagesOf(solution).isEmpty(), "emitted: " + emittedMessagesOf(solution)); + }, + 5); + } +} diff --git a/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java b/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java index 0638ae88a7..dadcf84c8c 100644 --- a/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java +++ b/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java @@ -1,7 +1,9 @@ package org.evomaster.e2etests.spring.asyncapi.kafka; import com.foo.asyncapi.ncs.NcsKafkaController; +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto; import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto; +import org.evomaster.core.EMConfig; import org.evomaster.core.problem.asyncapi.data.AsyncApiIndividual; import org.evomaster.core.problem.asyncapi.data.AsyncApiOutcome; import org.evomaster.core.problem.enterprise.ExperimentalFaultCategory; @@ -12,8 +14,10 @@ import java.util.Arrays; import java.util.List; +import java.util.stream.Collectors; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; /** @@ -27,9 +31,23 @@ public class NcsKafkaEMTest extends AsyncApiTestBase { private static final NcsKafkaController controller = new NcsKafkaController(); + /** + * The suffix of the topics NCS answers on; the requests go to topics of their own. + */ + private static final String REPLY_TOPIC_SUFFIX = ".reply"; + @BeforeAll public static void initClass() throws Exception { - AsyncApiTestBase.initClass(controller); + + /* + The service is instrumented once, when started here. The Kafka replacements are off + by default and the handler-return run needs them; the other runs ignore what they + record. + */ + EMConfig config = new EMConfig(); + config.setInstrumentMR_KAFKA(true); + + AsyncApiTestBase.initClass(controller, config); } @Test @@ -186,6 +204,59 @@ public void testRunEMWithCoverageStable() throws Throwable { 3); } + /** + * The same, but each action ends once the service's consumer thread goes back to poll. NCS + * polls with KafkaConsumer and answers with KafkaProducer, which the Kafka instrumentation + * tracks, so every wait ends that way, and its replies are recorded on the action answered. + */ + @Test + public void testRunEMWithHandlerReturn() throws Throwable { + + runTestHandlingFlaky( + "NcsKafkaHandlerReturnEM", + "org.foo.asyncapi.NcsKafkaHandlerReturnEM", + 40, + false, + (args) -> { + + setOption(args, "asyncApiReplyTimeoutMs", "3000"); + + /* + A window far longer than NCS ever needs: a wait that went to the deadline + would mean the end of the handler was never seen. + */ + setOption(args, "asyncApiSettleMs", "5000"); + setOption(args, "asyncApiCompletion", "HANDLER_RETURN"); + setOption(args, "instrumentMR_KAFKA", "true"); + + Solution solution = initAndRun(args); + + assertTrue(countOutcome(solution, AsyncApiOutcome.REPLIED) > 0, "nothing was answered"); + assertEquals(0, countOutcome(solution, AsyncApiOutcome.NO_REPLY), "a promised reply never came"); + assertEquals(0, countOutcome(solution, AsyncApiOutcome.PUBLISH_FAILED), "a message never left"); + assertReplyReached(solution, "bessj", "doubleResult"); + assertReplyReached(solution, "checkTriangle", "intResult"); + assertEverySettled(solution); + + assertTrue(countSettleEnd(solution, AsyncApiReplyDto.SettleEndReason.HANDLER_RETURN) > 0, + "no wait ended on the handler returning"); + assertEquals(0, countSettleEnd(solution, AsyncApiReplyDto.SettleEndReason.DEADLINE), + "a wait went to the deadline, so the service was never seen done with a message"); + assertEquals(0, countSettleEnd(solution, AsyncApiReplyDto.SettleEndReason.INTERRUPTED), + "a wait was interrupted"); + + //only its replies: the driver's publishes, made in this JVM, are not the service's + List emitted = emittedMessagesOf(solution); + assertFalse(emitted.isEmpty(), "nothing the service emitted was recorded"); + List notReplies = emitted.stream() + .map(m -> m.address) + .filter(a -> a == null || !a.endsWith(REPLY_TOPIC_SUFFIX)) + .collect(Collectors.toList()); + assertTrue(notReplies.isEmpty(), "recorded as emitted by the service: " + notReplies); + }, + 3); + } + /** * The no-reply oracle, end to end. With fixture replies deliberately disabled, * every message goes unanswered, and that is reported as a fault once experimental oracles diff --git a/core/src/main/kotlin/org/evomaster/core/EMConfig.kt b/core/src/main/kotlin/org/evomaster/core/EMConfig.kt index befb55e79c..733fd1cbae 100644 --- a/core/src/main/kotlin/org/evomaster/core/EMConfig.kt +++ b/core/src/main/kotlin/org/evomaster/core/EMConfig.kt @@ -918,6 +918,17 @@ class EMConfig { " settle window early, so it needs one: set 'asyncApiSettleMs' to the longest wait to allow") } + if (asyncApiCompletion == AsyncApiCompletion.HANDLER_RETURN && asyncApiSettleMs <= 0) { + throw ConfigProblemException("'asyncApiCompletion' ${AsyncApiCompletion.HANDLER_RETURN} ends a" + + " settle window early, so it needs one: set 'asyncApiSettleMs' to the longest wait to allow") + } + + if (asyncApiCompletion == AsyncApiCompletion.HANDLER_RETURN && !instrumentMR_KAFKA) { + throw ConfigProblemException("'asyncApiCompletion' ${AsyncApiCompletion.HANDLER_RETURN} reads what the" + + " instrumentation records of the messages the service consumes, which only Kafka's does so far:" + + " set 'instrumentMR_KAFKA' to true") + } + if (problemType == ProblemType.RPC && createTests && (enablePureRPCTestGeneration || enableRPCAssertionWithInstance) @@ -2465,6 +2476,14 @@ class EMConfig { @Experimental var instrumentMR_REDIS = false + @Cfg("Execute instrumentation for method replace with category KAFKA." + + " It records the messages the SUT consumes from and emits to Kafka, which testing an AsyncAPI service" + + " in white-box mode can use." + + " Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin" + + " on the JVM.") + @Experimental + var instrumentMR_KAFKA = false + @Cfg("Enable to expand the genotype of REST individuals based on runtime information missing from Swagger") var expandRestIndividuals = true @@ -3087,7 +3106,14 @@ class EMConfig { * Stop early once the service has gone `asyncApiStableMs` without reaching any coverage * objective for the first time, or improving one. */ - COVERAGE_STABLE + COVERAGE_STABLE, + + /** + * Stop early once the service has received at least one message from the address the + * action published to, and every thread that received one has gone back to its broker + * client for more. Read off the Kafka instrumentation (`instrumentMR_KAFKA`). + */ + HANDLER_RETURN } @Experimental @@ -3095,7 +3121,10 @@ class EMConfig { " driver's controller decides the service is done with a message. FIXED waits for the whole window." + " COVERAGE_STABLE ends it early once the service has gone asyncApiStableMs without reaching any new" + " code, or improving on what it reached. That is a heuristic: a handler that only runs code already" + - " covered earlier in the same test looks done too.") + " covered earlier in the same test looks done too. HANDLER_RETURN ends it once the service has received" + + " a message from the address published to, and every thread that received one has asked its broker" + + " client for more, as a consumer loop does when its handler returns. It needs instrumentMR_KAFKA, and is" + + " a heuristic too: work a handler hands off to another thread looks done as soon as it is handed off.") var asyncApiCompletion = AsyncApiCompletion.FIXED @Experimental @@ -3764,6 +3793,7 @@ class EMConfig { if (instrumentMR_REDIS) categories.add(ReplacementCategory.REDIS.toString()) if (instrumentMR_NEO4J) categories.add(ReplacementCategory.NEO4J.toString()) if (instrumentMR_DYNAMODB) categories.add(ReplacementCategory.DYNAMODB.toString()) + if (instrumentMR_KAFKA) categories.add(ReplacementCategory.KAFKA.toString()) return categories.joinToString(",") } diff --git a/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResult.kt b/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResult.kt index 8051938ef0..86e7e766f3 100644 --- a/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResult.kt +++ b/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResult.kt @@ -1,5 +1,8 @@ package org.evomaster.core.problem.asyncapi.data +import com.fasterxml.jackson.core.type.TypeReference +import com.fasterxml.jackson.databind.ObjectMapper +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto import org.evomaster.core.problem.enterprise.EnterpriseActionResult import org.evomaster.core.search.action.Action @@ -18,6 +21,7 @@ class AsyncApiCallResult : EnterpriseActionResult { const val SETTLED_MS = "SETTLED_MS" const val SETTLE_ENDED_BY = "SETTLE_ENDED_BY" const val TEST_SCRIPT = "TEST_SCRIPT" + const val EMITTED_MESSAGES = "EMITTED_MESSAGES" /* What was published, kept so that a generated test can publish the same thing without @@ -39,6 +43,21 @@ class AsyncApiCallResult : EnterpriseActionResult { * A result holds strings, and a line of source never contains this. */ private const val SCRIPT_SEPARATOR = "\n" + + /** + * How many characters of the payload of an emitted message a result keeps. Results are + * kept for every action in the archive, and the instrumentation's own cut, at a few + * thousand characters, is more than a generated test needs. A payload cut here is + * flagged as truncated, and is not asserted on. + */ + const val EMITTED_PAYLOAD_MAX_LENGTH = 1024 + + /** + * Writes and reads the emitted messages, which a result can only hold as a string. + */ + private val mapper = ObjectMapper() + + private val emittedMessagesType = object : TypeReference>() {} } constructor(sourceLocalId: String, stopping: Boolean = false) : super(sourceLocalId, stopping) @@ -106,6 +125,52 @@ class AsyncApiCallResult : EnterpriseActionResult { fun getSettleEndedBy(): AsyncApiReplyDto.SettleEndReason? = getResultValue(SETTLE_ENDED_BY)?.let { AsyncApiReplyDto.SettleEndReason.valueOf(it) } + /** + * The messages the service emitted while this one was being handled, in the order it emitted + * them, as the instrumentation of its broker client recorded them. Only in white-box mode. + * + * Only what the search and the generated tests read is kept of each: the address, the + * payload cut at [EMITTED_PAYLOAD_MAX_LENGTH] characters, whether it was cut, and the class + * of the payload. + */ + fun setEmittedMessages(messages: List) { + addResultValue(EMITTED_MESSAGES, mapper.writeValueAsString(messages.map { slim(it) })) + } + + /** + * A copy of [message] with only what a result keeps of it. + */ + private fun slim(message: EmittedMessageDto): EmittedMessageDto { + + val payload = message.payload + val cut = payload != null && payload.length > EMITTED_PAYLOAD_MAX_LENGTH + + return EmittedMessageDto().apply { + address = message.address + payloadType = message.payloadType + if (cut) { + //not between the two halves of a surrogate pair, which would leave half a character + val end = if (Character.isHighSurrogate(payload!![EMITTED_PAYLOAD_MAX_LENGTH - 1])) { + EMITTED_PAYLOAD_MAX_LENGTH - 1 + } else { + EMITTED_PAYLOAD_MAX_LENGTH + } + this.payload = payload.substring(0, end) + payloadTruncated = true + } else { + this.payload = payload + payloadTruncated = message.payloadTruncated + } + } + } + + /** + * @return the messages the service emitted while this one was being handled, or empty when + * none was recorded + */ + fun getEmittedMessages(): List = + getResultValue(EMITTED_MESSAGES)?.let { mapper.readValue(it, emittedMessagesType) } ?: listOf() + /** * The lines the driver rendered for a generated test, or empty when it rendered none. */ diff --git a/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitness.kt b/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitness.kt index f911fcf1bd..97fd8c0d17 100644 --- a/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitness.kt +++ b/core/src/main/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitness.kt @@ -6,6 +6,8 @@ import com.google.inject.Inject import com.webfuzzing.asyncapi.models.AsyncApiChannel import com.webfuzzing.asyncapi.models.AsyncApiCorrelationId import com.webfuzzing.asyncapi.models.AsyncApiReply +import org.evomaster.client.java.controller.api.dto.AdditionalInfoDto +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto import org.evomaster.client.java.controller.api.dto.SutInfoDto import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiActionDto import org.evomaster.client.java.controller.api.dto.problem.asyncapi.AsyncApiReplyDto @@ -154,6 +156,8 @@ class AsyncApiFitness : ApiWsFitness() { ?: return null handleExtra(dto, fv) + recordEmittedMessages(dto.additionalInfoList, actionResults.filterIsInstance()) + if (epc.isInSearch() && config.isEnabledTaintAnalysis()) { /* A tainted string in a payload or a header reaches the service inside the @@ -180,6 +184,23 @@ class AsyncApiFitness : ApiWsFitness() { ) } + /** + * Keep on the result of each action the messages the service emitted while it was being + * handled, as the instrumentation of the service's broker client recorded them. + * + * Entry i of [infos] is for the action with index i, the one [results] has at i too: the + * controller records one entry per action it is told about, in order. A test stopped by a + * message that could not be published has fewer entries than results, never more. + */ + private fun recordEmittedMessages(infos: List, results: List) { + infos.zip(results).forEach { (info, result) -> + val emitted: List? = info.emittedMessages + if (!emitted.isNullOrEmpty()) { + result.setEmittedMessages(emitted) + } + } + } + /** * @return whether the message reached the wire, so that the test can go on */ @@ -442,6 +463,7 @@ class AsyncApiFitness : ApiWsFitness() { dto.stableMs = config.asyncApiStableMs.toLong() AsyncApiActionDto.MessageCompletionStrategy.COVERAGE_STABLE } + EMConfig.AsyncApiCompletion.HANDLER_RETURN -> AsyncApiActionDto.MessageCompletionStrategy.HANDLER_RETURN } } diff --git a/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt b/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt index afd23b1519..e13b061f97 100644 --- a/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt +++ b/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt @@ -1,6 +1,7 @@ package org.evomaster.core import org.evomaster.client.java.controller.api.ControllerConstants +import org.evomaster.client.java.instrumentation.shared.ReplacementCategory import org.evomaster.core.EMConfig.AIResponseClassifierModel import org.evomaster.core.config.ConfigProblemException import org.evomaster.core.output.OutputFormat @@ -978,6 +979,47 @@ internal class EMConfigTest{ assertEquals(EMConfig.AsyncApiCompletion.FIXED, defaults.asyncApiCompletion) } + @Test + fun testHandlerReturnNeedsASettleWindowAndTheKafkaInstrumentation(){ + + //in white-box, as the settle options are ignored in black-box + fun parse(vararg args: String) = + EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse("--blackBox", "false", *args)) } + + val noWindow = assertThrows { + parse("--problemType", "ASYNCAPI", "--asyncApiCompletion", "HANDLER_RETURN", "--instrumentMR_KAFKA", "true") + } + assertTrue(noWindow.message!!.contains("asyncApiSettleMs"), noWindow.message) + + val noKafka = assertThrows { + parse("--problemType", "ASYNCAPI", "--asyncApiCompletion", "HANDLER_RETURN", "--asyncApiSettleMs", "2000") + } + assertTrue(noKafka.message!!.contains("instrumentMR_KAFKA"), noKafka.message) + + val config = parse("--problemType", "ASYNCAPI", "--asyncApiCompletion", "HANDLER_RETURN", + "--asyncApiSettleMs", "2000", "--instrumentMR_KAFKA", "true") + assertEquals(EMConfig.AsyncApiCompletion.HANDLER_RETURN, config.asyncApiCompletion) + + //and in black-box, where there is no settle window to end, neither is needed + val blackBoxRun = EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse( + "--blackBox", "true", "--problemType", "ASYNCAPI", "--createTests", "false", + "--asyncApiCompletion", "HANDLER_RETURN")) } + assertEquals(EMConfig.AsyncApiCompletion.FIXED, blackBoxRun.asyncApiCompletion) + } + + @Test + fun testTheKafkaReplacementsAreOnlyAskedForWhenTurnedOn(){ + + val defaults = EMConfig() + assertFalse(defaults.instrumentMR_KAFKA) + assertFalse(defaults.methodReplacementCategories().split(",").contains(ReplacementCategory.KAFKA.name), + defaults.methodReplacementCategories()) + + val kafka = EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse("--instrumentMR_KAFKA", "true")) } + assertTrue(kafka.methodReplacementCategories().split(",").contains(ReplacementCategory.KAFKA.name), + kafka.methodReplacementCategories()) + } + @Test fun testSetFalseOnDependsOn(){ diff --git a/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResultTest.kt b/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResultTest.kt new file mode 100644 index 0000000000..99fe275d9a --- /dev/null +++ b/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/data/AsyncApiCallResultTest.kt @@ -0,0 +1,88 @@ +package org.evomaster.core.problem.asyncapi.data + +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test + +class AsyncApiCallResultTest { + + companion object { + private const val ADDRESS = "shop.orders.confirmed" + + private const val SOURCE_ID = "action-0" + + private const val FILLER = 'a' + + /** + * A character outside the Basic Multilingual Plane, written as two chars. + */ + private const val SURROGATE_PAIR = "😀" + + private const val MAX = AsyncApiCallResult.EMITTED_PAYLOAD_MAX_LENGTH + } + + private fun message(payload: String?, truncated: Boolean? = false) = EmittedMessageDto().apply { + address = ADDRESS + this.payload = payload + payloadTruncated = truncated + payloadType = String::class.java.name + } + + private fun keep(message: EmittedMessageDto): EmittedMessageDto { + val result = AsyncApiCallResult(SOURCE_ID) + result.setEmittedMessages(listOf(message)) + return result.getEmittedMessages().single() + } + + @Test + fun testAShortPayloadIsKeptAsItIs() { + + val payload = FILLER.toString().repeat(MAX) + + val kept = keep(message(payload)) + + assertEquals(ADDRESS, kept.address) + assertEquals(payload, kept.payload) + assertEquals(false, kept.payloadTruncated) + assertEquals(String::class.java.name, kept.payloadType) + } + + @Test + fun testALongPayloadIsKeptCutAndFlagged() { + + val kept = keep(message(FILLER.toString().repeat(MAX + 1))) + + assertEquals(FILLER.toString().repeat(MAX), kept.payload) + assertEquals(true, kept.payloadTruncated, "a cut payload must say so, or a test would assert on it") + } + + @Test + fun testACutDoesNotSplitASurrogatePair() { + + //the pair takes the last char that fits and the first that does not + val payload = FILLER.toString().repeat(MAX - 1) + SURROGATE_PAIR + + val kept = keep(message(payload)) + + assertEquals(FILLER.toString().repeat(MAX - 1), kept.payload) + assertEquals(true, kept.payloadTruncated) + } + + @Test + fun testWhatTheInstrumentationCutStaysFlagged() { + + val kept = keep(message(FILLER.toString(), truncated = true)) + + assertEquals(true, kept.payloadTruncated) + } + + @Test + fun testAMessageWithoutTextPayloadKeepsItsType() { + + val kept = keep(message(null, truncated = null)) + + assertNull(kept.payload) + assertNull(kept.payloadTruncated) + assertEquals(String::class.java.name, kept.payloadType) + } +} diff --git a/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitnessTest.kt b/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitnessTest.kt index 07dddfe7b9..d57d483d09 100644 --- a/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitnessTest.kt +++ b/core/src/test/kotlin/org/evomaster/core/problem/asyncapi/service/AsyncApiFitnessTest.kt @@ -4,6 +4,7 @@ import com.fasterxml.jackson.databind.ObjectMapper import com.google.inject.Injector import com.webfuzzing.asyncapi.access.AsyncApiAccess import org.evomaster.client.java.controller.api.dto.AdditionalInfoDto +import org.evomaster.client.java.controller.api.dto.EmittedMessageDto import org.evomaster.client.java.controller.api.dto.StringSpecializationInfoDto import org.evomaster.client.java.controller.api.dto.TargetInfoDto import org.evomaster.client.java.controller.api.dto.TestResultsDto @@ -65,6 +66,11 @@ class AsyncApiFitnessTest { */ private const val COMPARED_WITH = "order-created" + /** + * Where NCS answers bessj, and what it emits there, in the tests on emitted messages. + */ + private const val BESSJ_REPLY = "ncs.bessj.reply" + /** * A value taint analysis would recognise as its own once the service reports it back. */ @@ -784,6 +790,84 @@ class AsyncApiFitnessTest { assertEquals(150L, dto.stableMs) } + @Test + fun testHandlerReturnIsAskedFor() { + + startNcs("--asyncApiSettleMs=2000", "--asyncApiCompletion=HANDLER_RETURN", "--instrumentMR_KAFKA=true") { + fireAndForget() + } + + evaluate("bessj") + + val dto = driver.published.single() + assertEquals(2000L, dto.settleMs) + assertEquals(AsyncApiActionDto.MessageCompletionStrategy.HANDLER_RETURN, dto.completion) + assertNull(dto.stableMs) + } + + /** + * What an instrumented driver reports for an action during which the service emitted one + * message to [address]. + */ + private fun emitted(address: String) = AdditionalInfoDto().apply { + emittedMessages.add(EmittedMessageDto().apply { + this.address = address + payload = DOUBLE_RESULT + payloadTruncated = false + payloadType = String::class.java.name + }) + } + + @Test + fun testWhatTheServiceEmittedIsKeptOnTheResultOfItsAction() { + + startNcs("--asyncApiSettleMs=2000", "--asyncApiCompletion=HANDLER_RETURN", "--instrumentMR_KAFKA=true") { + fireAndForget() + } + driver.testResults = { + TestResultsDto().apply { + additionalInfoList.add(emitted(BESSJ_REPLY)) + //the second message made the service emit nothing + additionalInfoList.add(AdditionalInfoDto()) + } + } + + val results = results(evaluate("bessj", "expint")) + + val first = results[0].getEmittedMessages().single() + assertEquals(BESSJ_REPLY, first.address) + assertEquals(DOUBLE_RESULT, first.payload) + assertEquals(false, first.payloadTruncated) + assertEquals(String::class.java.name, first.payloadType) + + assertTrue(results[1].getEmittedMessages().isEmpty()) + //and a copy of the result keeps them + assertEquals(BESSJ_REPLY, results[0].copy().getEmittedMessages().single().address) + } + + @Test + fun testFewerEntriesThanActionsLeaveTheRestWithoutEmittedMessages() { + + startNcs { fireAndForget() } + driver.testResults = { TestResultsDto().apply { additionalInfoList.add(emitted(BESSJ_REPLY)) } } + + val results = results(evaluate("bessj", "expint")) + + assertEquals(1, results[0].getEmittedMessages().size) + assertTrue(results[1].getEmittedMessages().isEmpty()) + } + + @Test + fun testABlackBoxRunKeepsNoEmittedMessages() { + + startNcs("$BLACK_BOX=true") { fireAndForget() } + driver.testResults = { TestResultsDto().apply { additionalInfoList.add(emitted(BESSJ_REPLY)) } } + + val result = results(evaluate("bessj")).single() + + assertTrue(result.getEmittedMessages().isEmpty()) + } + @Test fun testABlackBoxRunAsksForNoStableCoverageEither() { diff --git a/docs/options.md b/docs/options.md index 4a51643082..a0b9797fa2 100644 --- a/docs/options.md +++ b/docs/options.md @@ -283,7 +283,7 @@ There are 3 types of options: |`aiResponseClassifierWarmup`| __Int__. Number of training iterations required to update classifier parameters. For example, in the Gaussian model this affects mean and variance updates. For neural network (NN) models, the warm-up should typically be larger than 1000. *Default value*: `100`.| |`appendToTargetHeuristicsFile`| __Boolean__. Whether should add to an existing target heuristics file, instead of replacing it. It is only used when processFormat is TARGET_HEURISTIC. *Default value*: `false`.| |`arazzoLocation`| __String__. arazzo location on disk. *Default value*: `""`.| -|`asyncApiCompletion`| __Enum__. When testing an AsyncAPI service in white-box mode with a settle window (asyncApiSettleMs), how the driver's controller decides the service is done with a message. FIXED waits for the whole window. COVERAGE_STABLE ends it early once the service has gone asyncApiStableMs without reaching any new code, or improving on what it reached. That is a heuristic: a handler that only runs code already covered earlier in the same test looks done too. *Valid values*: `FIXED, COVERAGE_STABLE`. *Default value*: `FIXED`.| +|`asyncApiCompletion`| __Enum__. When testing an AsyncAPI service in white-box mode with a settle window (asyncApiSettleMs), how the driver's controller decides the service is done with a message. FIXED waits for the whole window. COVERAGE_STABLE ends it early once the service has gone asyncApiStableMs without reaching any new code, or improving on what it reached. That is a heuristic: a handler that only runs code already covered earlier in the same test looks done too. HANDLER_RETURN ends it once the service has received a message from the address published to, and every thread that received one has asked its broker client for more, as a consumer loop does when its handler returns. It needs instrumentMR_KAFKA, and is a heuristic too: work a handler hands off to another thread looks done as soon as it is handed off. *Valid values*: `FIXED, COVERAGE_STABLE, HANDLER_RETURN`. *Default value*: `FIXED`.| |`asyncApiReplyTimeoutMs`| __Int__. When testing an AsyncAPI service, how long to wait for the reply to a published message before treating it as unanswered, in milliseconds. *Constraints*: `min=1.0`. *Default value*: `5000`.| |`asyncApiSettleMs`| __Int__. When testing an AsyncAPI service in white-box mode, how long to keep each published message's action open after the driver has returned, in milliseconds, so that the code the service runs to handle it is still credited to that action. A message-driven service handles a message on a thread of its own, which may still be running once the driver has seen it published, or its reply arrive. The wait happens in the driver's controller. 0 means no wait. *Constraints*: `min=0.0`. *Default value*: `0`.| |`asyncApiStableMs`| __Int__. When testing an AsyncAPI service with asyncApiCompletion COVERAGE_STABLE, how long the service has to go without improving its coverage, in milliseconds, for the settle window to end early. *Constraints*: `min=1.0`. *Default value*: `200`.| @@ -335,6 +335,7 @@ There are 3 types of options: |`initStructureMutationProbability`| __Double__. Probability of applying a mutation that can change the structure of test's initialization if it has. *Constraints*: `probability 0.0-1.0`. *Default value*: `0.0`.| |`instrumentMR_CASSANDRA`| __Boolean__. Execute instrumentation for method replace with category CASSANDRA. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| |`instrumentMR_DYNAMODB`| __Boolean__. Execute instrumentation for method replace with category DYNAMODB. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| +|`instrumentMR_KAFKA`| __Boolean__. Execute instrumentation for method replace with category KAFKA. It records the messages the SUT consumes from and emits to Kafka, which testing an AsyncAPI service in white-box mode can use. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| |`instrumentMR_NEO4J`| __Boolean__. Execute instrumentation for method replace with category NEO4J. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| |`instrumentMR_NET`| __Boolean__. Execute instrumentation for method replace with category NET. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| |`instrumentMR_OPENSEARCH`| __Boolean__. Execute instrumentation for method replace with category OPENSEARCH. Note: this applies only for languages in which instrumentation is applied at runtime, like Java/Kotlin on the JVM. *Default value*: `false`.| diff --git a/docs/write_driver.md b/docs/write_driver.md index 47bffd04ec..bdbacc740c 100644 --- a/docs/write_driver.md +++ b/docs/write_driver.md @@ -393,6 +393,13 @@ Two things are worth knowing when writing one: from `startSut()`. A plain black-box suite has no driver in it, so it publishes to the address the document declares, as it does when this returns `null`, which is the default. +In white-box mode, with `--instrumentMR_KAFKA true`, the instrumentation also records what the service +consumes from and emits to Kafka, through `poll` on a `KafkaConsumer` or `Consumer` and `send` on a +`KafkaProducer` or `Producer`. A driver in the same JVM as the service can have its own calls recorded as +the service's when they are made on threads it starts itself. The calls it makes on the +thread running `executeAsyncApiAction`, while that runs, are never +taken for the service's, even when the driver runs in the same JVM as the service. + Generated tests publish with a client of the transport rather than through the driver, so the suite needs that client library on its classpath: see [library dependencies](library_dependencies.md).