Repository navigation
AsyncAPI 3.x: Jackson reads from bytes, for message payloads - #1873
Draft
LautaroPetaccio wants to merge 1 commit into
Draft
LautaroPetaccio wants to merge 1 commit into
LautaroPetaccio wants to merge 1 commit into
Conversation
LautaroPetaccio
added this pull request to stack #1812
October 11, 2026 19:08
A message payload reaches the service as bytes, not as a String. Spring Kafka's JsonDeserializer reads every record value with ObjectReader.readValue(byte[]), and the message converters of spring-messaging and spring-kafka call ObjectMapper.readValue(byte[], JavaType). None of these had a method replacement, so a Kafka listener that deserializes its payload registered no DTO schema and produced no JSON taint specialization. The same applies to any SUT, REST included, that reads JSON from bytes. - ObjectMapper: readValue(byte[], Class), readValue(byte[], JavaType), readValue(byte[], TypeReference) and readValue(InputStream, TypeReference), next to the String and InputStream ones. - ObjectReader: a new replacement class for readValue(byte[]) and readValue(String). The target type is the one configured in the reader, read with getValueType() (Jackson 2.10+) or the _valueType field before that. A reader without a type is not analyzed, and the call goes through unchanged. Worth flagging for review: - Replacements match the exact owner class written in the bytecode, and third-party code is instrumented too, so the ObjectReader replacement fires inside JsonDeserializer. A test loads a class that only calls JsonDeserializer.deserialize through the instrumenting class loader and checks that the DTO gets registered and the taint specialized; without the replacement those assertions fail. - initMethods resolves every replacement of a class eagerly, so every replaced method must exist in every Jackson 2.x. All of them do (checked with javap from 2.6 to 2.17). getValueType() does not, which is why it is only called by reflection, with the field as fallback. - None of the replaced methods delegates to another replaced one inside Jackson, so a call is analyzed once. In recent 2.x versions (2.11 at the latest), ObjectReader.readValue(byte[]|String, Class) is implemented as forType(type).readValue(content), so those calls are analyzed too, with the type given as input. readValue(byte[], int, int) is left alone. - The schema is always registered. The bytes are decoded as UTF-8 only for the taint check, and only when they could be a tainted value: they start with the taint prefix, or are not longer than the longest input variable once encoded. So a large payload is not decoded on every read. Jackson always gets the original bytes, so its own encoding detection is not affected. - An InputStream is read once, here, and Jackson gets a new stream over the very same bytes whose close() closes the caller's stream, so the source is closed when, and only when, AUTO_CLOSE_SOURCE says so. A failed read closes the source under the same condition, and rethrows. - A type that cannot be resolved by reflection does not break the SUT: the analysis is skipped, and the cause logged once. - Both target names keep the trimmed-string trick, so the shade plugin does not relocate them in the controller jar. spring-kafka 2.7.6 (the version Spring Boot 2.5.4 manages, Spring 5, class version 52) is added in test scope to client-java/instrumentation, with its version in the client-java dependencyManagement. It brings kafka-clients in test scope, which the root pom comment on kafka-clients says. The existing ObjectMapper replacements behave as before: the TypeReference resolution is moved into a helper shared with the new methods, and an array type is still analyzed as a whole, with a schema of type array. ExecutionTracer keeps the length of the longest input variable, reset with the rest of the per-test state, for the could-be-taint check.
LautaroPetaccio
force-pushed
the
feature/asyncapi-jackson-bytes
branch
from
October 12, 2026 02:54
da653e7 to
5f7dfe4
Compare
LautaroPetaccio
marked this pull request as draft
October 12, 2026 12:44
LautaroPetaccio
force-pushed
the
feature/asyncapi-jackson-bytes
branch
from
October 17, 2026 23:48
5f7dfe4 to
4f9a404
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sixth in the AsyncAPI white-box stack, on top of #1872. EvoMaster now notices when a service reads JSON from bytes, not only from text, so it learns the shape of the messages a Kafka listener receives.
This one is entirely in
client-java/instrumentationandinstrumentation-shared— no core changes. It does not depend on anything AsyncAPI-specific: any SUT that reads JSON from bytes benefits, REST included, so it can be reviewed on its own.Why
EvoMaster learns what a request or message should look like by watching how the service reads it. When the service turns JSON into one of its classes with Jackson, the instrumentation reports two things:
Order, which hasshippingandpriority". The schema of that class is registered, so the search knows which fields it can send.This already works for REST: Spring MVC reads a request body from an
InputStream, and code that parses a body by hand usually has it as aString. The Jackson methods for both were already watched.Kafka delivers messages as bytes instead. A typical listener never calls Jackson itself:
Spring Kafka's
JsonDeserializerturns the bytes into anOrderwithObjectReader.readValue(byte[]), and the message converters of spring-messaging and spring-kafka useObjectMapper.readValue(byte[], JavaType). None of these methods was watched. So for a Kafka listener, EvoMaster never learned that the message is anOrder, nor which fields it has. That matters most when the AsyncAPI document does not describe the message body, which is common: EvoMaster then has nothing to tell it what to send.This PR watches those byte-based reads too. #1878 builds on it: when the document leaves a payload undescribed, the payload takes its shape from the class the service reads it into.
What this PR does not change: once the message is an
Order, checks on its fields, such asorder.shipping.equals("EXPRESS"), were already visible to EvoMaster through the existingStringmethod replacements. This PR is about learning the message's shape, not those checks.What it adds
JacksonObjectMapperClassReplacement: new replacements forreadValue(byte[], Class),readValue(byte[], JavaType),readValue(byte[], TypeReference)andreadValue(InputStream, TypeReference).JacksonObjectReaderClassReplacement(new):readValue(byte[])andreadValue(String). The target type is the one configured in the reader (readerFor(...)), read withgetValueType()on Jackson 2.10+ and from the_valueTypefield before that. A reader built without a type is not analyzed, and the call goes through unchanged.JsonUtils:readAllBytes(Java 8 has noInputStream.readAllBytes),decodeUtf8, anddecodeUtf8IfCouldBeTaint, which decodes only bytes thatExecutionTracer.couldBeTaintInput(byte[])says could be a tainted value: they start with the taint prefix (TaintInputName.startsWithTaintPrefix), or are not longer than the longest input variable once encoded (3 bytes per char).Things that are less obvious than they look
ObjectReaderreplacement applies at the call insideJsonDeserializer.deserialize.ReadBytesWithJacksonTestloads a class that only callsJsonDeserializer.deserializethrough the instrumenting class loader, and checks that the DTO is registered and the taint specialized. Without the new replacement those assertions fail.initMethodsresolves all the replacements of a class eagerly, and a missing one breaks every call. All the replaced methods are present from 2.6 to 2.17 (the versions inspected withjavap).getValueType()only appeared in 2.10, which is why it is called by reflection, with the field as fallback, and is not a replacement itself.ObjectReader.readValue(byte[]|String, Class)is implemented asforType(type).readValue(content), so those calls get analyzed too, with the type given as input.readValue(byte[], int, int)is left alone.InputStreamis read once, here. Jackson gets a new stream over the very same bytes, whoseclose()closes the caller's stream, so the source is closed when, and only when,AUTO_CLOSE_SOURCEsays so. A failed read closes the source under the same condition, and rethrows.shaded.prefix).What this changes outside AsyncAPI
client-java/pom.xml: spring-kafka2.7.6independencyManagement, test scope. It is the version Spring Boot 2.5.4 manages (the Boot version the instrumentation tests use), so it runs on Spring 5 and is class version 52; spring-kafka 3 needs Spring 6.client-java/instrumentation/pom.xml: spring-kafka in test scope.pom.xml: spring-kafka bringskafka-clients(3.7.2, from the rootdependencyManagement) in test scope, so the comment there says that client-java/instrumentation uses it.ObjectMapperreplacements behave as before. TheTypeReferencetoJavaTyperesolution moved into a helper shared with the new methods. An array type is still analyzed as a whole, with a schema of type array (core turns that into an array gene), as it already was for aFoo[]class.ExecutionTracerkeeps the length of the longest input variable, reset with the rest of the per-test state, forcouldBeTaintInput.Not here
ObjectReader.readValue(InputStream)andreadValue(Reader). Spring MVC 5.3+ reads request bodies withObjectReader.readValue(InputStream), so replacing it would change what REST SUTs report. Worth doing, but on its own, with the REST E2E suites as the check.ObjectMappersubclass as static type (e.g.JsonMapper) still do not match, as before.Testing
JacksonObjectMapperTest: 22 tests (17 new). Each new replacement registers the DTO; a tainted payload getsJSON_OBJECT, orJSON_ARRAYfor a list, also from a stream; aFoo[]keeps its array schema; non-ASCII content is parsed from the original bytes; null input fails in Jackson, not in the replacement; the caller's stream is closed or left open asAUTO_CLOSE_SOURCEsays, also on a failed read; large bytes register the schema without being decoded; bytes equal to an input variable are checked.ExecutionTracerTest, 4 new tests, andTaintInputNameTest, 1 new test: which bytes could be a tainted value, and that a reset forgets the input variables.JacksonObjectReaderClassReplacementTest(new): 10 tests. Bytes and String, taint and no taint, list and array types, reader without a type (the same Jackson exception goes through, nothing recorded), original bytes passed, null input.ReadBytesWithJacksonTest(new): 6 tests throughInstrumentingClassLoader:readerFor(...).readValue(bytes),readValue(bytes, Class)and spring-kafka'sJsonDeserializer, each with a valid payload and with a taint.client-java/instrumentation: 766 tests, 0 failures, 1 skipped (TestabilityExcInstrumentedTest, already disabled).