Log Management a guide for buyers
Stay on top of the IT industry
Subscribe to ITOps Times Weekly News Digest to get the latest news, news analysis and commentary delivered to your inbox. • Reports on the technologies affecting IT Operators — APM, Data Center Optimization, Multi-Cloud, ITSM and Storage
• Insights into the practices and innovations reshaping
IT Ops such as AIOps Automation, Containers, DevOps,
Edge Computing and more
• The latest news from the IT providers, industry consortia, open source projects and research institutions
Subscribe today to keep up with everything happening in the IT Ops industry. www.ITOpsTimes.com
Log Management a guide for buyers
Contents Index-free logging is the way forward page 8
Connecting the data dots through log management page 4
How does Graylog help companies with log management efforts? page 6
Report: SOCs wasting 25% of time chasing false positives page 9
Log analysis and the challenge of processing Big Data page 10
A guide to log management tools page 13
September 2019
3
Connecting the data dots through Log Management By Jenna Sargent
as long as there has been business, there has been data. unlike some of the newer technologies that have popped up in the past century, companies have always been gathering data. it may not have been the same type or amount that companies collect now, but it was always there. but with the boom in big data, a new challenge has arisen. How do we keep track of all of this data? and perhaps even more important, how do we manage this data in such a way that we can gain useful insights from it? “We’ve always had logs and we’ve always had to do something with them and try to analyze them and gain insights from them,” said rich Lane, senior analyst at forrester. “Now what we’re trying to do is connect the dots.” This is where log management comes in. according to Nick Carstensen, technical product evangelist at log management provider Graylog, log management is a way 4 September 2019
of centrally managing and gathering all of the logs an organization is generating. Typically, this involves making them accessible in an easy and simple front end for viewing. Keeping logs is important because it gives you an idea of what is going on in your organization. a few things that a company can learn from logs are what time people are coming in in the morning, if they’re accessing files after hours, and if new software is being installed in the environment. “all of that can be detected and actually recorded through log management,” he said. “essentially, most of the monitoring platforms today want to take in log information, be able to put analytics across that, be that machine learning or other advanced algorithms and with the other data in the enterprise correlate it all together and give you better insights,” said Lane. in the past, organizations found log management to be a hard problem to solve, Carstensen explained.
This was because there was no easy way to gather up the logs an organization generated in a way that was useful. “When logs were gathered, they were never used as there were too many to look through, and were mostly kept for audit reasons,” Carstensen said. but with current log management tools, it’s possible to take in high volumes of logs and then index them for quick viewing and searching. a properly configured log management solution can help ease the workload of iT staff. Most log management solutions will allow for alerting. This means that organizations can proactively monitor their environment. “These tools also allow for alerting on the logs to be more proactive than reactive, and
even can give you trends over time to understand patterns in your organization. The ability to archive years of data and recall it when needed has made centralized log management more attractive than ever,” said Carstensen. Log management solutions can allow iT teams to quickly pinpoint the location of an error or get alerted about upcoming issues so that they can respond faster. “When you do not have a central log management, logging into every device to go through the logs can take many hours, where a quick search could find the same data,” he said. Typically, the responsibility of log management comes down to iT, but sometimes the responsibility can branch out to other groups. depending on the size of the organization,
sometimes log management can become something that the audit team would have a part in. The audit team would typically only be responsible for the configuration of the tool, in terms of specifying the length of time logs are retained, what reports will be generated, and what alerts will be generated. The iT team would oversee the regular use of the tool. “audit will normally ask for data from the log management system through the iT team, but will not manage or maintain the system on a day-to-day basis,” said Carstensen. Carstensen explained that within iT teams, the owners of particular data will have the logs for that data. Networking teams will own the networking data logs, while a Windows infrastructure admin would own the
data relevant to that. This can cause some issues in cases where individuals or teams don’t want to share data with each other. This can be a major challenge when trying to implement log management solutions. in order to be successful, it’s important to break down those siloed groups so that the data can be centrally managed. Having a top-down push for centralized log management can help break those silos. Carstensen explained that often when there is an issue in an organization, groups will pass it off to another team, saying it is not their issue. This results in long delays in actually fixing the issue. by allowing read-only access to all teams, teams will be able to quickly find the root of issues. This continued on page 6 >
September 2019
5
Connecting the data dots throough log management < continued from page 5
How does Graylog help companies with log management efforts?
organizations of any size can create large amounts of logs, leaving
them to require a tool to consolidate the logs in a centralized location. graylog steps up to this challenge and provides an easy to use inter-
face for your daily interactions to your logs.
While you gather your logs, you want a solution to help organize
them, make them human friendly for troubleshooting and provide dif-
ferent retention times to the logs based on your compliance or organi-
zational needs. Having logs enriched with the correct data, like threat intelligence information, speeds up your investigation time for root
cause analysis. Processing logs and putting them into useful visuals
and dashboards make graylog excel in usability.
alerting on your logs as they come in, increases your company’s
awareness and allows for quick response to events which are happening on your network. These alerts can be security related or could be more operational and developmental. Having an alert sent out via an
email or posted to a slack channel will quickly let you keep up with your inflow of logs. graylog’s correlation engine, can put together many different events to create one alarm, helping to reduce event fatigue plaguing many organizations.
once an alert has been generated, use graylog’s workflow features
to help you quickly get to the information in a standardized method, usable by any analyst with access. information can be exported from the
system, or reports can be generated as well on an automated schedule for easy consumption.
With graylog’s ability to ingest any type of logs created and its open
message processing, the options for your logs are endless. 6 September 2019
will enhance the stability and security of the organization as a whole. according to Carstensen, another challenge associated with log management is getting proper buyin from upper management. another thing to keep in mind when looking into log management solution is making sure you have the proper storage. organizations need to keep in mind that big data takes a lot of space. When looking for a log management solution, it’s important to carefully look at what features a vendor is offering. Carstensen believes that the most important feature to consider is the ability to class logs from different sources. Windows generates logs differently than Linux, so if you’re running both operating systems in your environment, you want to make sure you can gather logs from each. The second thing to keep in mind is the retention strategy. organizations need to ask themselves how long they want to keep logs for. “is it 30 days, a year, five years? you want to make sure the solution can support that ability to go longer,” Carstensen said. The third consideration is ease of use. Carstensen explained that a lot of log management systems are powerful, but may take a lot of effort and staff to use. “you want something with minimal amounts of overhead to run with the flexibility to search and find your data very quickly.” and as with any solution that an organization would be implementing, security is a concern that organizations will have to consider. continued on page 9 >
Index-free logging is the way forward By Jenna Sargent
Log management may never go away, but the way it is done may change. While indexes have long been the backbone of most log management solutions, Kresten Krab Thorup, CTo and co-founder of Humio, explains that’s not the only way to manage log data. according to Krab Thorup, indexing has been an accepted approach for collecting and analyzing log data, but it’s time to move on. Humio is not the only company offering log management solutions based on index-free architectures; companies like Loki and scalyr also offer similar solutions. in a talk given at QCon 2019 in New york City, Thorup described what index-free logging is and how it is beneficial. When using an index-based system, organizations pay upfront for disk space and CPu usage. a downside of this is that “if the time and space to build the index grows out of proportion with the real data that you are actually interested in, then you have lost,” said Thorup. “database indexes provide a trade off suitable for systems with low ingest rate and high query frequency. The core activity with logs is to write a lot and only search parts of them in specific time ranges, when an incident occurs. indexes are good for many things but not for logging.” With index-free logging, benefits include lower ingest latency, nearreal time alerts and dashboards, lower disk space requirements, and much lower hardware requirements, 8 September 2019
Thorup explained. “The key interesting thing is just to be able to do 10 times as much with the same hardware,” he said. according to Thorup, there are two major issues with using indexes. first, the “high-cardinality” problem. High cardinality occurs when data contains a large set of keys and values. users often create keys for log data, user-defined events, and traces; but indexing all of that property data tends to result in indexes that end up being larger than the data they want to put into the system. Thus, it takes a long time to compute the index and the data may be irrelevant by the time it is completed. another potential issue is a lack
of coverage. Thorup explained that when a user queries log and metrics data, it returns matches from all time, not just the time frame they are interested in. This is because the index maps a domain of keys that corresponds to the data set, but doesn’t carry other information over. according to Thorup, this issue can be mitigated in sQL by creating a covering index, but that’s not possible with full-text searching, such as with apache Lucene. “Without a covering index that includes the time index of the log message is the only way to find the matches in the time frame of the query is by doing extra work,” said Thorup. Thorup believes that index-free logging solves these issues. With index-free logging, data is stored in buckets, which are then labeled with information that enables to query engine to decide if data could be in that bucket or not. He recognizes that this technique could also be viewed as a sort of index, but that the term “index-free logging” refers to the fact that individual keys and values are not indexed. “index-free Logging provides for a different set of tradeoffs uniquely better suited for logs, events, and traces than solutions based on exact term indexing,” he said. in short, index-free logging gives users more time to understand and debug their solution and not waste time troubleshooting the logging platform itself, Thorup stated. n
Report: SOCs wasting 25% of time chasing false positives
Legacy tools are preventing security operations center (soC) employees from reaching their full potential. according to a joint survey between exabeam and the Ponemon institute, soCs waste 25 percent of their time on false positives because of incorrect security alerts. The companies surveyed 596 iT and iT security practitioners for the survey. The report highlights the need for productivity improvements in soC centers. according to the report, soC teams need to respond to approximately 4,000 security alerts every week. in addition to chasing false positives, which is the biggest time-waster, soC teams also spend about 15 percent of their time on each building incident timelines and cleaning, fixing, or patching networks after an incident. Put together, these inefficiencies can lower response times and leave organizations vulnerable to data and financial losses. “sieMs are central to soC cybersecurity for collecting logs and data from multiple network sources for the evaluation, analysis and correlation of network events used for threat detection. However, modern sieMs are most effective because they leverage machine learning and behavior analytics to identify increasingly sophisticated cyberattacks and highly targeted hack techniques. When used in conjunction with a full arsenal of tools like intelligent incident timeline construction and automated response, modern sieMs provide significantly more context for how attackers think, work or what they are after,” exabeam wrote in a post. —Jenna Sargent
Connecting the data dots ‘The ability to archive years of data and recall it when needed has made centralized log management more attractive than ever.’
—Carstensen
< continued from page 6
Logs contain information about authentication systems, what software versions are in use in your organization, and the types of security tools that are in place. “Having access to all that information will help the attacker know your network before the attack begins,” Carstensen said. Carstensen recommends that organizations limit access to the log management tool in order to keep logs secure. He also recommends that the data be encrypted and an audit trail be kept in order to understand who is accessing the logs and what they did with those logs. He also recommends organizations regularly evaluate access to the system. if during evaluations it is determined that access for certain individuals needs to be different, you can be a bit more proactive about it. one trend that Carstensen has been seeing recently is a desire to enrich data. Log data itself has been pretty stagnant. older router switches and firewalls could have been generating the same data for the past 20 to 30 years. Now the question is: “How do you enrich that data further by taking their iP address and then using a threat
intelligence lookup feed or correlation engines to take that and say [let’s take out this iP that’s part of a known botnet attack] right now, or is this iP known to be malware? so we can enrich your data and make better alerts and decisions based on that.” Carstensen predicts that log management will become easier to use going forward. With the addition of intelligence to data, it will be easier to create a baseline of trends and see when those change over time. according to forrester’s Lane, log management vendors moving forward will need to come up with better ways of storing this data. in particular, they need to move away from storing data in its raw format. organizations need to be able to filter out the noise in order to get good insights, and a lot of log data is just repetitive information. “sort of the trend now that people are starting to try to figure out is how do we handle all that massive amounts of data without having to keep it around … i would say a huge proportion of logs themselves aren’t very meaningful. right now, for most solutions, we have to take all that data in even if it’s something we don’t want or don’t need,” said Lane. n September 20198
9
Log analysis and the challenge of processing To stay competitive, companies who want to run an agile business need log analysis to navigate the complex world of big data in search of actionable insight. However, scouring through the apparently boundless data lakes to find meaningful info means treading troubled waters when appropriate tools are not employed. bwest case scenario, data amounts to terabytes (hence the name “big data”), if not petabytes. if an efficient automated process is not available, it’s virtually and practically impossible to look at only a specific set of information (such as discerning a trend). robust enterprise log management software is rare and can be used to filter that single, useful, data-driven advice out of the immensely vast big data pools simmering in your business cauldron. on the one hand, it will automatically archive and store the lessimportant data you rarely search through. on the other, it will help you audit all your logs in the blink of an eye to avoid dumping highly-valuable information in a roughly unprocessed data lake.
Big Data for business — a bottomless pit of information
Modern enterprises generate an immense volume of data, which presents iT professionals with both an opportunity and a challenge. However, even if big data has largely become one of the most popular buzzwords in the last few years, this technology trend is anything but a novelty. big data has always been there as a wondrous vault full of unreachable treasures. What really has changed lately is that today we possess the instrument and tools to
10 September 2019
Big Data
crack this safe and access it to drive the interests of a given company forward. big data is defined as data possessing some very specific characteristics. in particular, other than its enormous size (volume), big data is characterized by high variety, velocity, and quality (in the form of validity and veracity). Machine-generated logs represent an immensely rich source of information that can be mined for many purposes. from investigating or preventing potentially hazardous activities, from obtaining performance info about the current health of existing networks, data from logs has many uses that can significantly improve the efficiency of a company. all applications, operating systems, and networking devices produce logs full of both useful and useless messages. but without an agile-enough log management system, much of this data is too big and unwieldy to be accessed. Log management, processing, and analysis must deal with a massive flow of extremely granular and diversified information produced in real time. automation is necessary to “skim” all irrelevant data to extract and decrypt
useful insights coming from all kinds of unstructured data sources. The most competitive enterprises know that the self-serving route can be walked with relatively contained efforts. on top of that, there’s no need to explain how expensive it could be to pay a 3rd party analytics company.
Analyzing Big Data with log management software
To manage the unbridled volume of high-velocity incoming data without excess strain on the end user, a log management tool needs to be sophisticated and flexible. The iT environment of even a comparably smaller enterprise generates countless complex logs every day. if these logs are not centralized during the storage process, retrieving and processing simply become impossible tasks. Logs are not just used for troubleshooting anymore, and must be proactively integrated before any data found inside them could be integrated and correlated. event logs provide interesting information about the internal processes and a comprehensive view of the performance of your systems. user logs, on the other hand, are necessary to provide your enterprise with a practical perspective of your technology use cases. They’re an external source of raw information that needs to be integrated with great accuracy with internal information to pinpoint the root causes of an issue or other types of data-driven insights. The overall volume of these logs can be
massive, both because of the large total number of logs, and because the sheer size of individual logs can sometimes be huge. since no typical Notepad editor can manage files as large as tens of gigabytes, log management systems become a necessity. Variety and veracity of the log files must be confirmed. Configuration differences may generate inaccurate information that must be validated before it is indexed, parsed, and analyzed. a reliable log management platform must also be able to collect and store raw log files from different business sources at the same time to identify market and clustering trends. The high speed at which data is collected may make the aggregation and transformation process cumbersome if the logging strategy is not planned to be fluid and agile enough. The speed at which business intelligence is analyzed through saP HaNa and Hadoop doesn’t matter if data is bottlenecked at the log gathering step.
Final thoughts
accessing the world of big data through log analysis can bring an unexpected breath of fresh air to any business. Log file visualization and analysis may improve the performance of apps and servers, and allow customer and business intelligencedriven insights to positively impact the enterprise in a practical way. However, logging management can be a very time-consuming process when it is not optimized with the right tools. n —graylog.org September 20198
11
A guide to log management tools
n Datadog’s log management solution provides collection, processing, live tailing, exploration, graphing, dashboarding, alerting, and archival of logs generated by an organization. its Logging without Limits feature allows teams to select what they want to include or exclude, giving them more control over costs.
n Elasticsearch’s Logstash is a open-source, server-side pipeline for data processing. it ingests data from a variety of sources and then transforms it, regardless of what format the data is in. Logs can be viewed from a single ui, making it easy to orchestrate and manage pipelines. n Humio enables observability of systems through its log management system. it is able to ingest data as soon as it is created, even when there are bursts. This means you can ingest terabytes of data every day and search it in seconds. The solution also works with any log format and features a easy-to-learn query language.
n LogDNA provides real-time log aggregation, monitoring, and analysis. it can be used with any platform and for any volume of data. LogdNa’s log management solution brings together log aggregation, custom parsing, smart alerting, role based access controls, and real-time search, graphs, and log analysis. n Logz.io provides a log man-
agement solution that empowers engineers and maximizes performance. engineers will be able 12 September 2019
FEATURED PROVIDER
n Graylog is a leading centralized log management solution built to open standards for capturing, storing, and enabling real-time analysis of terabytes of machine data. We deliver a better user experience by making analysis ridiculously fast and efficient using a more cost-effective and flexible architecture. Thousands of iT professionals rely on graylog’s scalability, comprehensive access to complete data, and exceptional user experience to solve security, compliance, operational, and devops issues every day.
to spend less time maintaining monitoring systems, and more time on improving applications. The solution provides operational visibility into applications and infrastructure, ensuring maximum performance, availability, and a better customer experience.
n ManageEngine’s Log360 solves the challenges associated with log management and network security challenges. by integrating a number of solutions — including adaudit Plus, eventLog analyzer, o365 Manager Plus, exchange reporter Plus, and Cloud security Plus — organizations will gain complete control over their network. from a single console, Log360 users can audit active directory changes, network device logs, Microsoft exchange servers, Microsoft exchange online, azure active directory, and public cloud infrastructure.
n Netwrix provides a log management solution for Windows server events. Netwrix believes its solution helps organizations gain control over what is going on in their environment, allowing them to stay alert to critical events and reduce
the amount of time it takes to prepare for compliance audits.
n SolarWinds’ Papertrail allows
iT teams to consolidate logs into a single place. This will allow them to quickly diagnose and fix issues. The solution is also easy to implement, use, and understand, solarWinds said. The company offers multiple different plans to suit the needs of every company.
n Splunk consolidates log and machine data. it allows organizations to collect, store, index, search, correlate, visualize, analyze, and report on machine-generated data in order to resolve operation issues faster. supported use cases include log consolidation and retention, security, iT operations troubleshooting, application troubleshooting, and compliance reporting. n Sumo Logic’s platform helps
organizations make better data-driven decisions. it also reduces the time it takes to look into security or operational issues, freeing up resources for other tasks. its unified platform can be used across development, security, operations, and Lob teams. n