Skip to content

AsyncAPI 3.x: Jackson reads from bytes, for message payloads - #1873

Draft
LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-quiescencefrom
feature/asyncapi-jackson-bytes
Draft

LautaroPetaccio wants to merge 1 commit into
feature/asyncapi-quiescencefrom
feature/asyncapi-jackson-bytes

Conversation

@LautaroPetaccio

@LautaroPetaccio LautaroPetaccio commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Sixth in the AsyncAPI white-box stack, on top of #1872. EvoMaster now notices when a service reads JSON from bytes, not only from text, so it learns the shape of the messages a Kafka listener receives.

This one is entirely in client-java/instrumentation and instrumentation-shared — no core changes. It does not depend on anything AsyncAPI-specific: any SUT that reads JSON from bytes benefits, REST included, so it can be reviewed on its own.

Why

EvoMaster learns what a request or message should look like by watching how the service reads it. When the service turns JSON into one of its classes with Jackson, the instrumentation reports two things:

  • The class and its fields. For example, "this JSON is read as an Order, which has shipping and priority". The schema of that class is registered, so the search knows which fields it can send.
  • That a value has to be JSON. If the whole input was a value EvoMaster generated and is tracking (taint), it learns that the value has to be a JSON object of that class, and starts sending objects of the right shape instead of plain strings.

This already works for REST: Spring MVC reads a request body from an InputStream, and code that parses a body by hand usually has it as a String. The Jackson methods for both were already watched.

Kafka delivers messages as bytes instead. A typical listener never calls Jackson itself:

@KafkaListener(topics = "orders")
void onOrder(Order order) { ... }

Spring Kafka's JsonDeserializer turns the bytes into an Order with ObjectReader.readValue(byte[]), and the message converters of spring-messaging and spring-kafka use ObjectMapper.readValue(byte[], JavaType). None of these methods was watched. So for a Kafka listener, EvoMaster never learned that the message is an Order, nor which fields it has. That matters most when the AsyncAPI document does not describe the message body, which is common: EvoMaster then has nothing to tell it what to send.

This PR watches those byte-based reads too. #1878 builds on it: when the document leaves a payload undescribed, the payload takes its shape from the class the service reads it into.

What this PR does not change: once the message is an Order, checks on its fields, such as order.shipping.equals("EXPRESS"), were already visible to EvoMaster through the existing String method replacements. This PR is about learning the message's shape, not those checks.

What it adds

  • JacksonObjectMapperClassReplacement: new replacements for readValue(byte[], Class), readValue(byte[], JavaType), readValue(byte[], TypeReference) and readValue(InputStream, TypeReference).
  • JacksonObjectReaderClassReplacement (new): readValue(byte[]) and readValue(String). The target type is the one configured in the reader (readerFor(...)), read with getValueType() on Jackson 2.10+ and from the _valueType field before that. A reader built without a type is not analyzed, and the call goes through unchanged.
  • JsonUtils: readAllBytes (Java 8 has no InputStream.readAllBytes), decodeUtf8, and decodeUtf8IfCouldBeTaint, which decodes only bytes that ExecutionTracer.couldBeTaintInput(byte[]) says could be a tainted value: they start with the taint prefix (TaintInputName.startsWithTaintPrefix), or are not longer than the longest input variable once encoded (3 bytes per char).

Things that are less obvious than they look

  • It fires inside library code. Replacements match the exact owner class in the bytecode, and third-party classes are instrumented too, so the ObjectReader replacement applies at the call inside JsonDeserializer.deserialize. ReadBytesWithJacksonTest loads a class that only calls JsonDeserializer.deserialize through the instrumenting class loader, and checks that the DTO is registered and the taint specialized. Without the new replacement those assertions fail.
  • Every replaced method must exist in every Jackson 2.x. initMethods resolves all the replacements of a class eagerly, and a missing one breaks every call. All the replaced methods are present from 2.6 to 2.17 (the versions inspected with javap). getValueType() only appeared in 2.10, which is why it is called by reflection, with the field as fallback, and is not a replacement itself.
  • One call, one analysis. None of the replaced methods delegates to another replaced one inside Jackson. In recent 2.x versions (2.11 at the latest), ObjectReader.readValue(byte[]|String, Class) is implemented as forType(type).readValue(content), so those calls get analyzed too, with the type given as input. readValue(byte[], int, int) is left alone.
  • Bytes are decoded only for the taint check, as UTF-8 (the JSON encoding, and the one taint values are written in), and only when they could be a tainted value, so a large payload is not decoded on every read. The schema of the target type is always registered. Jackson always gets the original bytes, so its own encoding detection is not affected.
  • An InputStream is read once, here. Jackson gets a new stream over the very same bytes, whose close() closes the caller's stream, so the source is closed when, and only when, AUTO_CLOSE_SOURCE says so. A failed read closes the source under the same condition, and rethrows.
  • A type that cannot be resolved by reflection does not break the SUT. The analysis is skipped, and the cause logged once.
  • Shading. Both target class names keep the trimmed-string trick, so the shade plugin does not relocate them in the controller jar (checked in the built jar: no shaded. prefix).

What this changes outside AsyncAPI

  • client-java/pom.xml: spring-kafka 2.7.6 in dependencyManagement, test scope. It is the version Spring Boot 2.5.4 manages (the Boot version the instrumentation tests use), so it runs on Spring 5 and is class version 52; spring-kafka 3 needs Spring 6.
  • client-java/instrumentation/pom.xml: spring-kafka in test scope.
  • Root pom.xml: spring-kafka brings kafka-clients (3.7.2, from the root dependencyManagement) in test scope, so the comment there says that client-java/instrumentation uses it.
  • The existing ObjectMapper replacements behave as before. The TypeReference to JavaType resolution moved into a helper shared with the new methods. An array type is still analyzed as a whole, with a schema of type array (core turns that into an array gene), as it already was for a Foo[] class.
  • ExecutionTracer keeps the length of the longest input variable, reset with the rest of the per-test state, for couldBeTaintInput.

Not here

  • ObjectReader.readValue(InputStream) and readValue(Reader). Spring MVC 5.3+ reads request bodies with ObjectReader.readValue(InputStream), so replacing it would change what REST SUTs report. Worth doing, but on its own, with the REST E2E suites as the check.
  • Calls compiled against an ObjectMapper subclass as static type (e.g. JsonMapper) still do not match, as before.

Testing

  • JacksonObjectMapperTest: 22 tests (17 new). Each new replacement registers the DTO; a tainted payload gets JSON_OBJECT, or JSON_ARRAY for a list, also from a stream; a Foo[] keeps its array schema; non-ASCII content is parsed from the original bytes; null input fails in Jackson, not in the replacement; the caller's stream is closed or left open as AUTO_CLOSE_SOURCE says, also on a failed read; large bytes register the schema without being decoded; bytes equal to an input variable are checked.
  • ExecutionTracerTest, 4 new tests, and TaintInputNameTest, 1 new test: which bytes could be a tainted value, and that a reset forgets the input variables.
  • JacksonObjectReaderClassReplacementTest (new): 10 tests. Bytes and String, taint and no taint, list and array types, reader without a type (the same Jackson exception goes through, nothing recorded), original bytes passed, null input.
  • ReadBytesWithJacksonTest (new): 6 tests through InstrumentingClassLoader: readerFor(...).readValue(bytes), readValue(bytes, Class) and spring-kafka's JsonDeserializer, each with a valid payload and with a taint.
  • All of client-java/instrumentation: 766 tests, 0 failures, 1 skipped (TestabilityExcInstrumentedTest, already disabled).

A message payload reaches the service as bytes, not as a String. Spring
Kafka's JsonDeserializer reads every record value with
ObjectReader.readValue(byte[]), and the message converters of
spring-messaging and spring-kafka call ObjectMapper.readValue(byte[],
JavaType). None of these had a method replacement, so a Kafka listener
that deserializes its payload registered no DTO schema and produced no
JSON taint specialization. The same applies to any SUT, REST included,
that reads JSON from bytes.

 - ObjectMapper: readValue(byte[], Class), readValue(byte[], JavaType),
   readValue(byte[], TypeReference) and readValue(InputStream,
   TypeReference), next to the String and InputStream ones.
 - ObjectReader: a new replacement class for readValue(byte[]) and
   readValue(String). The target type is the one configured in the
   reader, read with getValueType() (Jackson 2.10+) or the _valueType
   field before that. A reader without a type is not analyzed, and the
   call goes through unchanged.

Worth flagging for review:
 - Replacements match the exact owner class written in the bytecode,
   and third-party code is instrumented too, so the ObjectReader
   replacement fires inside JsonDeserializer. A test loads a class that
   only calls JsonDeserializer.deserialize through the instrumenting
   class loader and checks that the DTO gets registered and the taint
   specialized; without the replacement those assertions fail.
 - initMethods resolves every replacement of a class eagerly, so every
   replaced method must exist in every Jackson 2.x. All of them do
   (checked with javap from 2.6 to 2.17). getValueType() does not, which
   is why it is only called by reflection, with the field as fallback.
 - None of the replaced methods delegates to another replaced one inside
   Jackson, so a call is analyzed once. In recent 2.x versions (2.11 at
   the latest), ObjectReader.readValue(byte[]|String, Class) is
   implemented as forType(type).readValue(content), so those calls are
   analyzed too, with the type given as input. readValue(byte[], int,
   int) is left alone.
 - The schema is always registered. The bytes are decoded as UTF-8
   only for the taint check, and only when they could be a tainted
   value: they start with the taint prefix, or are not longer than the
   longest input variable once encoded. So a large payload is not
   decoded on every read. Jackson always gets the original bytes, so
   its own encoding detection is not affected.
 - An InputStream is read once, here, and Jackson gets a new stream
   over the very same bytes whose close() closes the caller's stream,
   so the source is closed when, and only when, AUTO_CLOSE_SOURCE says
   so. A failed read closes the source under the same condition, and
   rethrows.
 - A type that cannot be resolved by reflection does not break the
   SUT: the analysis is skipped, and the cause logged once.
 - Both target names keep the trimmed-string trick, so the shade plugin
   does not relocate them in the controller jar.

spring-kafka 2.7.6 (the version Spring Boot 2.5.4 manages, Spring 5,
class version 52) is added in test scope to client-java/instrumentation,
with its version in the client-java dependencyManagement. It brings
kafka-clients in test scope, which the root pom comment on
kafka-clients says.

The existing ObjectMapper replacements behave as before: the TypeReference
resolution is moved into a helper shared with the new methods, and an
array type is still analyzed as a whole, with a schema of type array.
ExecutionTracer keeps the length of the longest input variable, reset
with the rest of the per-test state, for the could-be-taint check.
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-jackson-bytes branch from da653e7 to 5f7dfe4 Compare October 12, 2026 02:54
@LautaroPetaccio
LautaroPetaccio marked this pull request as draft October 12, 2026 12:44
@LautaroPetaccio
LautaroPetaccio force-pushed the feature/asyncapi-jackson-bytes branch from 5f7dfe4 to 4f9a404 Compare October 17, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant