UDP packet drops

[alamzeeshan]

I have installed td-agent in RHEL7 machine with 2 CPU and 4 GB RAM

• td-agent 1.2.6
• gem 'fluent-plugin-netflow' version '0.2.8'
• gem 'fluentd' version '1.2.6'

Tested with the packet generator below:

• https://github.com/mshindo/NetFlow-Generator
• ./flowgen -n10000 -i50 -w1 -p5160 localhost

Each time I am sending data using the packet generator and writing to a file I am not getting more than 3000 events. I have tested this by sending 5,000, 10,000 and 50,000 packets. Never got more than 3000 packets written to the file

td-agent.conf

<source>
  @type netflow
  tag netflow.event
  port 5160
  cache_ttl 6000
  versions [5, 9]
</source>
<match netflow.event>
  @type file
  @log_level debug
  path /apps/tddump/dump1
  buffer_type file
  buffer_queue_limit 50000
  buffer_chunk_limit 24m
  flush_interval 30
</match>

td-agent logs:

2018-11-02 09:38:52 -0400 [info]: using configuration file: <ROOT>
  <source>
    @type netflow
    tag "netflow.event"
    port 5160
    cache_ttl 6000
    versions [5,9]
  </source>
  <match netflow.event>
    @type file
    @log_level "debug"
    path "/apps/tddump/dump1"
    buffer_type file
    buffer_queue_limit 50000
    buffer_chunk_limit 24m
    flush_interval 30
    <buffer time>
      @type file
      flush_interval 30
      chunk_limit_size 24m
      queue_limit_length 50000
      timekey 86400
      path /apps/tddump/dump1
    </buffer>
  </match>
</ROOT>
2018-11-02 09:38:52 -0400 [info]: starting fluentd-1.2.6 pid=605 ruby="2.4.4"
2018-11-02 09:38:52 -0400 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-kafka' version '0.7.9'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-netflow' version '0.2.8'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-s3' version '1.1.6'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-11-02 09:38:53 -0400 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-11-02 09:38:53 -0400 [info]: gem 'fluentd' version '1.2.6'
2018-11-02 09:38:53 -0400 [info]: adding match pattern="netflow.event" type="file"
2018-11-02 09:38:53 -0400 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2018-11-02 09:38:53 -0400 [info]: adding source type="netflow"
2018-11-02 09:38:53 -0400 [info]: #0 starting fluentd worker pid=615 ppid=610 worker=0
2018-11-02 09:38:53 -0400 [debug]: #0 buffer started instance=70093046745520 stage_size=0 queue_size=0
2018-11-02 09:38:53 -0400 [debug]: #0 flush_thread actually running
2018-11-02 09:38:53 -0400 [info]: #0 listening netflow socket on 0.0.0.0:5160 with udp
2018-11-02 09:38:53 -0400 [info]: #0 fluentd worker is now running worker=0
2018-11-02 09:38:53 -0400 [debug]: #0 enqueue_thread actually running
2018-11-02 09:41:14 -0400 [debug]: #0 Created new chunk chunk_id="579aeb06b11c1609e89807a451f5f4a0" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1541131200, tag=nil, variables=nil>

I installed Logstash in the same machine and was able to write 10,000, 100,000 and even 3 million events to a file with the following config:

input {
    udp {
        type => "netflow"
        port => 5160
        codec => netflow {
            versions => [5,9]
        }
        workers => 2
        receive_buffer_bytes => 212992
        queue_size => 500000
    }
}

output {
 file {
   path => "/apps/dump/dump2"
 }
}

[alamzeeshan]

I have downloaded and installed td-agent-3.2.1-0.el7.x86_64.rpm from https://td-agent-package-browser.herokuapp.com/3/redhat/7/x86_64

$ yum localinstall td-agent-3.2.1-0.el7.x86_64.rpm
$ td-agent –version

td-agent 1.2.6

I have installed the plugins 'fluent-plugin-netflow' version '1.0.0.rc2' and 'fluent-plugin-flowcounter' version '1.3.0'.

Even now the packets are being dropped if more than 10,00 events/sec are being sent by
$ flowgen –n10000 -i50 -w1 -p5160 localhost

SENT	RECEIVED
5,000	5,000
10,000	9,810
15,000	13,230
20,000	17,280
25,000	20,280
30,000	22,050
40,000	31,020
50,000	38,460
100,000	74,760
500,000	364,080
3,000,000	1298250 + 871590 = 2169840

My LOGS:

2018-11-12 08:18:00 -0500 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2018-11-12 08:18:00 -0500 [info]: using configuration file: <ROOT>
  <source>
    @type netflow
    tag "netflow.event"
    port 5160
    cache_ttl 6000
    versions [5,9]
  </source>
  <match netflow.event>
    @type flowcounter
    unit minute
    count_keys count
    tag "flowcount"
  </match>
  <match flowcount>
    @type stdout
  </match>
</ROOT>
2018-11-12 08:18:00 -0500 [info]: starting fluentd-1.2.6 pid=26943 ruby="2.4.4"
2018-11-12 08:18:00 -0500 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-kafka' version '0.7.9'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-netflow' version '1.0.0.rc2'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-s3' version '1.1.6'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-11-12 08:18:00 -0500 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-11-12 08:18:00 -0500 [info]: gem 'fluentd' version '1.2.6'
2018-11-12 08:18:00 -0500 [info]: adding match pattern="netflow.event" type="flowcounter"
2018-11-12 08:18:00 -0500 [info]: adding match pattern="flowcount" type="stdout"
2018-11-12 08:18:00 -0500 [info]: adding source type="netflow"
2018-11-12 08:18:00 -0500 [info]: #0 starting fluentd worker pid=26953 ppid=26948 worker=0
2018-11-12 08:18:00 -0500 [info]: #0 fluentd worker is now running worker=0
2018-11-12 08:19:00.976684593 -0500 flowcount: {"netflow.event_count":5000,"netflow.event_bytes":0,"netflow.event_count_rate":83.33,"netflow.event_bytes_rate":0.0}
2018-11-12 08:20:00.976428024 -0500 flowcount: {"netflow.event_count":9810,"netflow.event_bytes":0,"netflow.event_count_rate":163.49,"netflow.event_bytes_rate":0.0}
2018-11-12 08:21:01.476624895 -0500 flowcount: {"netflow.event_count":13230,"netflow.event_bytes":0,"netflow.event_count_rate":218.67,"netflow.event_bytes_rate":0.0}
2018-11-12 08:22:01.476744400 -0500 flowcount: {"netflow.event_count":17280,"netflow.event_bytes":0,"netflow.event_count_rate":287.99,"netflow.event_bytes_rate":0.0}
2018-11-12 08:23:01.476621916 -0500 flowcount: {"netflow.event_count":20280,"netflow.event_bytes":0,"netflow.event_count_rate":337.99,"netflow.event_bytes_rate":0.0}
2018-11-12 08:24:01.476481512 -0500 flowcount: {"netflow.event_count":22050,"netflow.event_bytes":0,"netflow.event_count_rate":367.49,"netflow.event_bytes_rate":0.0}
2018-11-12 08:25:01.476072996 -0500 flowcount: {"netflow.event_count":31020,"netflow.event_bytes":0,"netflow.event_count_rate":516.99,"netflow.event_bytes_rate":0.0}
2018-11-12 08:26:01.476596552 -0500 flowcount: {"netflow.event_count":38460,"netflow.event_bytes":0,"netflow.event_count_rate":640.98,"netflow.event_bytes_rate":0.0}
2018-11-12 08:27:01.976063434 -0500 flowcount: {"netflow.event_count":74760,"netflow.event_bytes":0,"netflow.event_count_rate":1235.7,"netflow.event_bytes_rate":0.0}
2018-11-12 08:28:01.976689509 -0500 flowcount: {"netflow.event_count":364080,"netflow.event_bytes":0,"netflow.event_count_rate":6067.89,"netflow.event_bytes_rate":0.0}
2018-11-12 08:29:01.976957052 -0500 flowcount: {"netflow.event_count":1298250,"netflow.event_bytes":0,"netflow.event_count_rate":21637.25,"netflow.event_bytes_rate":0.0}
2018-11-12 08:30:02.476184596 -0500 flowcount: {"netflow.event_count":871590,"netflow.event_bytes":0,"netflow.event_count_rate":14406.53,"netflow.event_bytes_rate":0.0}
2018-11-12 08:31:02.476034916 -0500 flowcount: {"netflow.event_count":0,"netflow.event_bytes":0,"netflow.event_count_rate":0.0,"netflow.event_bytes_rate":0.0}
2018-11-12 08:32:02.476590427 -0500 flowcount: {"netflow.event_count":0,"netflow.event_bytes":0,"netflow.event_count_rate":0.0,"netflow.event_bytes_rate":0.0}

Also I had a old version of Fluentd 0.12.33 installed in a different VM and there also I faced the same issue.

SENT	RECEIVED
5,000	5,000
10,000	8,400
50,000	31,530

2018-11-12 14:06:08 +0000 [info]: reading config file path="fluent.conf"
2018-11-12 14:06:08 +0000 [info]: starting fluentd-0.12.33
2018-11-12 14:06:08 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2018-11-12 14:06:08 +0000 [info]: gem 'fluent-plugin-flowcounter' version '0.4.3'
2018-11-12 14:06:08 +0000 [info]: gem 'fluent-plugin-kafka' version '0.5.3'
2018-11-12 14:06:08 +0000 [info]: gem 'fluent-plugin-netflow' version '0.2.8'
2018-11-12 14:06:08 +0000 [info]: gem 'fluentd' version '0.12.33'
2018-11-12 14:06:08 +0000 [info]: adding match pattern="netflow.event" type="flowcounter"
2018-11-12 14:06:08 +0000 [info]: adding match pattern="flowcount" type="stdout"
2018-11-12 14:06:09 +0000 [info]: adding source type="netflow"
2018-11-12 14:06:09 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type netflow
    tag netflow.event
    port 5160
    cache_ttl 6000
    versions [5,9]
  </source>
  <match netflow.event>
    @type flowcounter
    unit minute
    count_keys count
    tag flowcount
  </match>
  <match flowcount>
    @type stdout
  </match>
</ROOT>
2018-11-12 14:06:09 +0000 [info]: listening netflow socket on 0.0.0.0:5160 with udp
2018-11-12 14:07:09 +0000 flowcount: {"netflow.event_count":5000,"netflow.event_bytes":0,"netflow.event_count_rate":83.33,"netflow.event_bytes_rate":0.0}
2018-11-12 14:08:09 +0000 flowcount: {"netflow.event_count":8400,"netflow.event_bytes":0,"netflow.event_count_rate":140.0,"netflow.event_bytes_rate":0.0}
2018-11-12 14:09:09 +0000 flowcount: {"netflow.event_count":31530,"netflow.event_bytes":0,"netflow.event_count_rate":525.5,"netflow.event_bytes_rate":0.0}
2018-11-12 14:10:09 +0000 flowcount: {"netflow.event_count":0,"netflow.event_bytes":0,"netflow.event_count_rate":0.0,"netflow.event_bytes_rate":0.0}