I have already written different posts on ELK stack ( Elasticsearch, Logstash and Kibana), the super-heroic application log monitoring setup. If you are not familiar with them, please check my posts here. We were using SCP to copy files from the application server to the log server and the logs were then read by the Logstash component. As the application moved to microservices-based architecture, we started having more and more servers running small services and setting up the SCP from each server to ELK server became a pain.
Time for Filebeat
This is where Filebeat came to the rescue. To quote about Filebeat from the official website.
One of major issues faced by modern day enterprise applications are the management and monitoring of application logs. During the phase of development, the application logging is overlooked as a rather unimportant part. But once the application is live in production, we realize that the first level of understanding to what is happening in the system is generated logs. With the modern day architecture , the components are independent and they all have their own logging. It becomes very cumbersome to have all these checked and analyzed when the system is very huge. This is where the ‘ELK’ stack comes to the rescue.
What is ELK ??