Logs Monitoring Issues – 3 Tools, Logstash, Elasticsearch and Kibana to Fix it!


Problem Statement –

As production environment is getting more distributed and the numbers of servers are continuously increasing, maintaining their performance by continuously monitoring them through logs can be a big tedious job. The main problem comes into the picture because these log files are scattered across different locations on multiple virtual or physical servers and are required to be monitored in near real-time environment.

Introduction –

The motive of this article is to provide the best method for handling huge amount of logs on multiple servers in distributed environment and get insights required to manage them in more efficient manner. The step by step process will guide you – how to avoid opening large amount of log files on different servers to make your life easy with Logs Monitoring. Further it also imparts know how to identify the errors and root issues in the Log files.

Pre – Requisites –

The following tools are require to manage the logs monitoring problem –

a) Logstash-1.5.0 –

Logstash is a data pipeline which provides a centralized area where logs and data from variety of systems are stored, and are monitored if any further subsequent actions are required.

b) ElasticSearch-1.5.2 –

ElasticSearch is a Search Server having capabilities of providing real time data and analytics. It helps in improving data analytics by performing any search operation and is used by Logstash to store incoming parsed logs.

c) Kibana-4.0.1 –

Kibana, a powerful and intelligent tool that gives a streamlined shape to any kind of structured and unstructured data. It easily visualizes data stored in Elasticsearch and gives a better understanding by slicing down large volumes of data.

Getting started:

  • ElasticSearch
  1. Start elasticsearch
  • Log stash
  1. Configuration to track the server log file
  2. Configuration to filter on log level basis
  3. Configuration to push those logs to elasticsearch
  • Kibana
  1. Configuration to connect with Elasticsearch
  2. Give shape for those logs data

Detail:

 Step 1: Download Elasticsearch-1.5.2 from https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.5.2.zip and extract the zip file, move to elasticsearch-1.5.2/bin/ and starts with ./elasticsearch       elasticsearch

Step 2: Download logstash-1.5.0 from https://download.elastic.co/logstash/logstash/logstash-1.5.0.zip and extract the zip file, move to logstash-1.5.0/bin/

  • Create indexer.conf – configuration file for track the logs and push to elasticsearch

indexer

  • Starts with ./logstash agent –f indexer.conf
  • While printing logs into /home/tomcat/logs/catalina.out, logstash catches every log and pushes to elasticsearch

Note: You can repeat this same step with multiple servers log file.

Step 3: Download kibana-4.0.1 from https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz and extract file.

  • Modify host to localhost on config/kibana.yml ( host : “localhost” )

kibana

  • Starts with bin/kibana

startkibana

loginkibana

  • Select Discover to search the corresponding index logs

kibanaindex

  • Search with log message ( below search with build ) and add selected fields to show only required fields

kibanaindexsearch

  • Kibana shows the logs with multiple shapes, clicks on Visualize and select one of the shapes or can create customized shape.

kibanavisualization

Selected Vertical bar chart and searched with log message

kibanaverticalbar

Summary –

In this article, we explained how to deal with huge amount of log files by using 3 simple tools – “Logstash, Elasticsearch and Kibana”. With the introduction of these tools (updated version) it became very easy to manage the logs as compared to previous one.

Hope this article was informative and leaves you with a better understanding of what logs are and how to manage them!

Liked this Article, Want to know more about us? Get in touch with us through our website Walking Tree and know the latest updates and happenings in the IT industry.

 

Posted in Elastic Search

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

We Have Moved Our Blog!

We have moved our blog to our company site. Check out https://walkingtree.tech/index.php/blog for all latest blogs.

Sencha Select Partner Sencha Training Partner
Xamarin Authorized Partner
Recent Publication
%d bloggers like this: