Anyone who has looked at a log file can testify to how cumbersome, difficult to read and unattractive they can be. It is often said that 'knowledge is power', however, this comes with an important caveat: if you don’t know how to interpret the data, you still won’t learn much from it. Visualising data and making it easy to interpret is the key and visual presentation makes that possible.
A log file generally comprises a list of rules whereby each line has a date and timestamp followed by data on particular events. Those entries in themselves are simple enough to read and understand. Consider however the example of an Apache web server, which processes thousands of requests per minute. Any overview is now quickly lost. The trick is to organise the data into a series of "keys" or fields. Which application sent what request and when? How long did it take for that request to be processed? How much data is involved? Where did the request originate, from what IP address, and so forth?
Although there are several log file creation suites that help to make log files more understandable and to analyze (big) data, my personal preference is for ELK Stack. ELK stands for Elasticsearch, Logstash, Kibana, a set of open source tools you can use to search, analyze and visualize data in real time. The data source or format does not matter and it can be deployed as a service in the cloud or on premise. ELK Stack is highly scalable and is used by, among others, LinkedIn and Netflix, to name but a few prominent players with big, BIG data. It has also been used to search through and analyze the Panama Papers.
But what exactly can you do with the various tools? Elasticsearch is a search and analytics engine that lets you filter and search data quickly. Smart queries can be built without having to program. Logstash ensures that all available data is connected and can be processed centrally, and Kibana helps to format and visualise everything nicely. It does so using dashboards, charts, diagrams and other visual aids.
ELK Stack is constantly expanding and an important tool to mention here is Beats, which can be used to easily create links to different applications. Where once you could merely access simple log files, today more and more information is available for monitoring, such as network traffic or system metrics (disk, CPU, memory etc). As further data becomes available, so does the chance of losing track and of everything becoming chaotic. Creating order from chaos is quite difficult, but with Beats this is much easier as generic logs can be turned into much more specific ones.
The beauty here is that a readymade beatlib is also available, an open source framework with building blocks to promote the ELK Stack integration. As a result, you don’t have to invent everything from scratch. In addition, you can still make specific links to the desired applications.
ELK Stack is, in short, a wonderful toolset that enables us at Sentia to keep a keen eye on everything. "Simply measuring is not enough, how you use what you are measuring also counts. By filtering, analyzing and visualizing data using the various tools from the ELK Stack dashboards 'you will truly know!’
Continuity Engineer at Sentia