Skip to content

nginx logs are not parsed using grok pattern #83

@jatinder10884

Description

@jatinder10884

I am using below grok_pattern for parsing Nginx logs

<source>

  @type tail
  path /data/nginxlogs/access.log
  pos_file /var/log/td-agent/nginx-access.log.pos
  tag es.nginx.acces
  <parse>
   @type multiline_grok
   <grok>
    pattern Started %{IPORHOST:client_ip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}\" %{NUMBER:server_response} %{NUMBER:bytes} \"%{URI:referrer}\" %{QS:user_agent}
#   grok_pattern %{MY_SUPER_PATTERN}
#   custom_pattern_path /etc/td-agent/MY_SUPER_PATTERN
#   multiline_start_regexp /^[^\s]/
  </grok>
  keep_time_key true
  </parse>
</source>
<filter es.nginx.acces>
  @type record_transformer
  <record>
  hostname ${hostname}
  </record>
</filter>
<match es.**>
  type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
 logstash_prefix "ngrokcontainer"
include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
<endpoint>
    url http://testing-logs-ab-east-1.es.amazonaws.com
    region us-east-1
  </endpoint>
</match>

Kibana :

message 3.231.213.28 - - [18/Apr/2020:22:20:16 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
Its not parsing properly, but when I check this grok debugger its working fine.
Please let me know how I can fix this issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions