问题描述:

I have configuration as filebeat with loadbalance true & two logstash hosts i.e. logstash1 & logstash2 which are input to elasticsearch.

The filebeat configurtion for logstash is as below

prospectors:

-

paths:

- /home/testLogs/temp.log

spool_size: 1

publish_async: true

### Logstash as output

logstash:

hosts: ["10.186.187.44:5044", "10.186.187.6:5044"]

worker: 1

loadbalance: true

The logstash input on both logstash I set as below

input {

beats {

port => 5044

congestion_threshold => 100000

}

}

output {

stdout {codec => rubydebug}

elasticsearch {

hosts => "ipaddress:9200"

index => "testidx"

}

}

However when I echo some lines into file temp.log, I see that in the elasticsearch processed lines are duplicated docs, it looks to me that both logstash are processing all the lines. and on stdout of both logstash sometime I see forex line1 & line2 processed by both logstash, or sometimes line1 processed by logstash1 & line1, line2, line2 processed by logstash2

However even making loadbalance=flase I still see the duplicated docs. & in stdout I sed line1 & line2 processed twice either by logstash1 or logstash2.

So just wanted to know is my configuration wrong that the lines are not distributed on logstashes or is it like its behavior or bug.

网友答案:

From the documentation:

If [loadbalance option] set to true and multiple Logstash hosts are configured, the output plugin load balances published events onto all Logstash hosts.

So it's expected behavior from filebeat to have the events duplicated to the two Logstash.


If you want to use two Logstash, you can sent the events to a queue (like redis) and have the Logstash consume from the queue.

网友答案:

Inspired by your post, I tested this use-case (because we need it too) with a minimal configuration, and it works. No duplicated documents in elasticsearch. I tested both, loadbalance true&false.

Filebeat:

logstash:
hosts: ["IP-LS1:8098","IP-LS2:8098"]
loadbalance: true

Logstash

input {
    beats {
        port => 8098
    }
}
filter {...}
output {

    if [type] == "filebeat-test" {
        stdout {codec => rubydebug}
        elasticsearch {
          hosts => ['IP-LS1|IP-LS2:9200']
          index => "filebeat-test-%{+YYYY.MM.dd}"
        }
    }
}
相关阅读:
Top