Question regarding multi-format + multi-line

[bobpecor]

Hi,

I have a question rather than an issue.

Does the fluent-plugin-grok-parser support multi-line and multi-format?

I am tailing multiple log4j2 log files. Each of them may have a different format and include stack traces or other multi-line messages.

Is it possible to use the grok parser in this situation? If not then any guidance on how to accomplish this would be greatly appreciated.

Here is a copy of my config's <source> element:

<source>
  @type tail
  @id eap_app_log_tail_source
  path /var/log/applogs/app1.log, /var/log/applogs/app2.log,/var/log/applogs/app3.log
  pos_file /var/log/td-agent/eap-apps.log.pos
  path_key log_file  
  tag jboss.node.eap.app.logs
  <parse>
    @type multiline_grok
    grok_failure_key grokfailure 
    <grok>
        pattern ^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{JAVACLASS:logger-class} %{GREEDYDATA:message}$
        multiline_start_regexp   /\d{4}-\d{1,2}-\d{1,2}/
    </grok>	  
    <grok>
        pattern ^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel} \[(?<thread>[A-Za-z0-9_  \-]+)\] {} - %{GREEDYDATA:message}$
        multiline_start_regexp   /\d{4}-\d{1,2}-\d{1,2}/
    </grok>
    <grok>
        pattern ^%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp} \[(?<thread>[A-Za-z0-9_  \-]+)\] - %{GREEDYDATA:message}$
        multiline_start_regexp   /([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))(?:\s*)/
    </grok>	    
 </parse>
</source>

[okkez]

Yes, this plugin supports multi-line and multi-format.
But this plugin does not support multiple multiline_start_regexp.
You cannot use multiline_start_regexp in <grok> section.

If you have multiple files for log4j2, you can add multiple <source> section to handle them.

[cosmo0920]

This plugin does not work contaminated structured log.
1-by-1 formatted log can be handled like with (Note that this configuration is pseudo configuration not the real one):

<source>
  @type tail
  @id eap_app_log_tail_source_app1
  path /var/log/applogs/app1.log
  pos_file /var/log/td-agent/eap-apps-1.log.pos
  path_key log_file  
  tag jboss.node.eap.app1.logs
  multiline_start_regexp   /\d{4}-\d{1,2}-\d{1,2}/
  <parse>
    @type multiline_grok
    grok_failure_key grokfailure 
    <grok>
        pattern ^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{JAVACLASS:logger-class} %{GREEDYDATA:message}$
    </grok>	  
 </parse>
</source>
<source>
  @type tail
  @id eap_app_log_tail_source_app2
  path /var/log/applogs/app2.log
  pos_file /var/log/td-agent/eap-apps-2.log.pos
  path_key log_file  
  tag jboss.node.eap.app2.logs
  <parse>
    @type multiline_grok
    grok_failure_key grokfailure  
    multiline_start_regexp   /\d{4}-\d{1,2}-\d{1,2}/ 
    <grok>
        pattern ^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel} \[(?<thread>[A-Za-z0-9_  \-]+)\] {} - %{GREEDYDATA:message}$
    </grok>    
 </parse>
</source>
<source>
  @type tail
  @id eap_app_log_tail_source_app3
  path /var/log/applogs/app3.log
  pos_file /var/log/td-agent/eap-apps-3.log.pos
  path_key log_file  
  tag jboss.node.eap.app3.logs
  <parse>
    @type multiline_grok
    grok_failure_key grokfailure 
    multiline_start_regexp   /([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))(?:\s*)/
    <grok>
        pattern ^%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp} \[(?<thread>[A-Za-z0-9_  \-]+)\] - %{GREEDYDATA:message}$
    </grok>	    
 </parse>
</source>