One common use case when sending logs to Elasticsearch is to send different lines of the log file to different indexes based on matching patterns. In this article, we will go through the process of setting this up using both Fluentd and Logstash in order to give you more flexibility and ideas on how to approach the topic.
Looking to increase developer productivity and observability at Otter, we noticed that when using one Elasticsearch index for each application, search becomes faster, the queries become easier, and the logs can be parsed using custom regex patterns, and we have full control over the cleanup policy when using Elasticsearch Curator.
Elasticsearch is the engine of choice for many companies looking for a distributed, RESTful search and analytics solution. At CloudHero, we deploy Elasticsearch on Kubernetes and use it quite a lot for storing and analyzing data. Using our hands-on experience, we compiled a cheat sheet containing the top five most helpful commands that you can use to manage your Elasticsearch cluster.
Today, we are going to talk about the EFK stack: Elasticsearch, Fluent, and Kibana. You will learn about the stack and how to configure it to centralize logging for applications deployed on Kubernetes. We will focus on Fluentbit, as it is the lightweight version of Fluentd and more suitable for Kubernetes. Additionally, we will talk about how we reached the final solution and the hurdles we had to overcome. Last but not least, we’ll show you how we handled application logs without actually installing 3rd party clients.