New ESG Research Finds Large Organizations Experiencing Explosive Growth in Log Data Collection, Analysis, and Storage

The Enterprise Strategy Group (ESG) recently released a new research report, "Security Management Matures," a unique research study that examines the security management policies, procedures, and technologies at large organizations. ESG undertook this research project to assess how large organizations are managing security threats and to determine how security management practices complement regulatory compliance and IT operations efforts.

The new report is based upon a survey of Information Security professionals responsible for evaluating, purchasing, and operating security management technologies at North American-based public and private organizations ranging in size from 1,000 to over 20,000 employees.

The report uncovers an important trend: To analyze security events, regulatory compliance controls, and IT operations, large organizations are collecting a significant amount of log file data from security appliances, networking devices, and applications. Today's log data capacity will pale in comparison to impending requirements; however, the report finds that large organizations will collect, store, and analyze more log data from more sources in the very near future. For example:

• More than 40% of large organizations collect at least 1TB (terabyte) of log data on a monthly basis while 11% collect more than 10TB of data each month. Additionally, nearly one-fourth of large organizations collect data from 1,000 or more sources (e.g., security, networking, and IT devices and applications).
• More than one-fourth of large organizations expect that the number of sources (e.g., security, networking, and IT devices and applications) from which they collect log file data will "increase substantially" over the next 12 months. The same growth pattern holds true with respect to total log data capacity.
• More than one-fourth of large organizations expect their log file data capacity to "increase substantially" over the next 12 months.
• While log data growth is occurring across all market sectors, it is especially pronounced among the largest organizations. For example, nearly half of organizations with over 20,000 employees expect the number of log file sources to "increase substantially" over the next 12 months.

The explosive growth in log data is a result of several drivers. First, large organizations are collecting more data from more sources to gain a wider purview of security threats and their ramifications on IT infrastructure. Second, log data analysis is used beyond security threat management alone. Business managers, IT operations, compliance administrators, and "C-level" executives are increasingly using log data analysis to monitor numerous business and IT metrics. Finally, firms are archiving more log data for longer periods of time for future analysis or as liability protection in the event of a legal discovery. "Security management needs have significantly increased the value of log data but ESG's research illustrates that analyzing log data provides benefits spanning across IT and the business," said Jon Oltsik, Senior Analyst at ESG and the author of this report. "With more data available, lots of people are using log data for all kinds of analysis. Security managers are doing forensic investigations, compliance officers are monitoring the effectiveness of controls, and IT operations administrators are tracking device configurations and troubleshooting problems. The cycle here is clear: More data leads to more analysis and more analysis leads to further business benefits and ROI."

The report concludes that the uncontrollable volume of log data growth will lead large organizations to build a dedicated log management architecture for log data collection, processing, and storage. Oltsik comments: "The explosive growth in log data for security, compliance, IT management, and business monitoring will lead to an inevitable situation where large organizations want to collect more and more data and analyze it in a variety of ways. This is exactly what is happening now. We are witnessing a paradigm shift where log file collection and processing becomes a discrete service-based architecture and acts as the foundation of a new IT-based data warehousing/business intelligence capability."