APM agent breaks log.error when using StructuredLogEncoder with ecs format

[michbeck100]

Describe the bug

We are using the version 1.55.5 with experimental Spring Boot 4 support. Since the update we have problems with log.error in SLF4J. This is the logback-spring.xml configuration:

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
  <encoder class="org.springframework.boot.logging.logback.StructuredLogEncoder">
    <format>ecs</format>
  </encoder>
</appender>

When using the apm agent we are seeing this in the logs instead of the real error log:

07:18:54,651 |-ERROR in ch.qos.logback.core.ConsoleAppender[CONSOLE] - Appender [CONSOLE] failed to append. java.lang.IllegalStateException: The name 'error' has already been written

According to my research, this happens if the JSON log already contains a field with the same name, in this case error. Logback uses the field error for a complex object that contains error.message and error.stack_trace. This is also documented in the Elastic Common Schema.

Steps to reproduce

Use the org.springframework.boot.logging.logback.StructuredLogEncoder with ecs format as described and call org.slf4j.Logger#error(java.lang.String, java.lang.Throwable).

Expected behavior

No exception during logging, but a well-formed JSON log containing the error.

[michbeck100]

May be related to #3745

[jackshirazi]

Try with log-error instrumentation disabled, ie disable_instrumentations=log-error and tell us whether it still happens

[michbeck100]

This seems to help, but what do i disable here in particular? This doesn't seem to be documented.

[jackshirazi]

That disables the log-error instrumentation which captures exceptions that are passed to logger error calls, so they become APM errors

You'll still have all errors/exceptions from traces which tend to be the main source of that information