Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management
460
Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management
460eBook
Related collections and offers
Overview
- Comprehensive coverage of log management including analysis, visualization, reporting and more
- Includes information on different uses for logs -- from system operations to regulatory compliance
- Features case Studies on syslog-ng and actual real-world situations where logs came in handy in incident response
- Provides practical guidance in the areas of report, log analysis system selection, planning a log analysis system and log data normalization and correlation
Product Details
| ISBN-13: | 9781597496360 |
|---|---|
| Publisher: | Elsevier Science |
| Publication date: | 12/31/2012 |
| Sold by: | Barnes & Noble |
| Format: | eBook |
| Pages: | 460 |
| File size: | 5 MB |
About the Author
Prior to SecureWorks, Kevin worked for Reflex Security where he worked on an IPS engine and anti-virus software. And prior to this he was a lead developer and architect at GuardedNet, Inc.,which built one of the industry’s first SIEM platforms. Kevin is also a commissioned officer in the United States Navy Reserve (USNR).
Kevin has over 19 years of experience in software development and design, 11 of which have been in the network security space. He holds a B.Sc. in computer science.
Christopher Phillips is a manager and senior software developer at Dell SecureWorks, Inc. He is responsible for the design and development of the company's Threat Intelligence service platform. He also has responsibility for a team involved in integrating log and event information from many third party providers for customers to have their information analyzed by the Dell SecureWorks systems and security professionals. Prior to Dell SecureWorks, Christopher has worked for McKesson and Allscripts where he worked with clients on HIPAA compliance and security and integrating healthcare systems. Christopher has over 18 years of experience in software development and design. He holds a Bachelors of Science in Computer Science and an MBA.
Dr. Anton Chuvakin is a recognized security expert in the field of log
management and PCI DSS compliance. He is an author of the books "Security Warrior" and "PCI
Compliance" and has contributed to many others, while also publishing dozens of papers on
log management, correlation, data analysis, PCI DSS, and security management. His blog
(http://www.securitywarrior.org) is one of the most popular in the industry.
Additionaly, Anton teaches classes and presents at many security conferences across the world
and he works on emerging security standards and serves on the advisory boards of
several security start-ups. Currently, Anton is developing his security consulting practice,
focusing on logging and PCI DSS compliance for security vendors and Fortune 500 organizations.
Anton earned his Ph.D. from Stony Brook University.
Read an Excerpt
Logging and Log Management
The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management
By Anton A. Chuvakin, Kevin J. Schmidt, Christopher Phillips, Partricia Moulder
Elsevier Science
Copyright © 2013 Elsevier, Inc.All rights reserved.
ISBN: 978-1-59749-636-0
Excerpt
CHAPTER 1
Logs, Trees, Forest: The Big Picture
CONTENTS
Introduction 1
Log Data Basics 2
What Is Log Data? 2
How is Log Data Transmitted
and Collected? 4
What is a Log Message? 6
The Logging Ecosystem 7
First Things First: Ask
Questions, Have a Plan 8
Log Message Generation 8
Log Message Filtering and
Normalization 9
Log Message Collection 11
Logging in the Cloud 13
Log Analysis 14
Log Message Long-Term
Storage 14
A Look at Things to
Come 15
Logs Are Underrated 16
Logs Can Be Useful 17
Resource Management 17
Intrusion Detection 18
Troubleshooting 21
Forensics 21
Boring Audit, Fun
Discovery 22
People, Process,
Technology 23
Security Information
and Event Management
(SIEM) 24
Summary 27
References 27
INFORMATION IN THIS CHAPTER:
* Log Data Basics
* A Look at Things to Come
* Logs Are Underrated
* Logs Can Be Useful
* People, Process, Technology
* Security Information and Event Management (SIEM)
* Case Studies
INTRODUCTION
This book is about how to get a handle on systems logs. More precisely, it is about how to get useful information out of your logs of all kinds. Logs, while often under-appreciated, are a very useful source of information for computer system resource management (printers, disk systems, battery backup systems, operating systems, etc.), user and application management (login and logout, application access, etc.), and security. It should be noted that sometimes the type of information can be categorized into more than one bucket. User login and logout messages are both relevant for both user management and security. A few examples are now presented to show how useful log data can be.
Various disk storage products will log messages when hardware errors occur. Having access to this information can often times mean small problems are resolved before they become really big nightmares.
As a second example, let's briefly consider how user management and security logs can be used together to shed light on a user activity. When a user logs onto a Windows environment, this action is logged in some place as a logon record. We will call this a user management log data. Anytime this user accesses various parts of the network, a firewall is more than likely in use. This firewall also records network access in the form of whether or not it allowed network packets to flow from the source, a user's workstation, to a particular part of the network. We will call this as security log data. Now, let's say your company is developing some new product and you want to know who attempts to access your R&D server. Of course, you can use firewall access control lists (ACLs) to control this, but you want to take it a step further. The logon data for a user can be matched up with the firewall record showing that the user attempted to access the server. And if this occurred outside of normal business hours, you might have reason to speak with the employee to better understand their intent. While this example is a little bit out there, it does drive home an important point. If you have access to the right information, you are able to do some sophisticated things.
But getting that information takes some time and some work. At first glance (and maybe the second one too) it can seem an overwhelming task—the sheer volume of data can alone be daunting. But we think we can help "de-whelm" you. We'll present an overall strategy for handling your logs. We'll show you some different log types and formats. The point of using different log types and formats is twofold. First, it will get you accustomed to looking at log messages and data so you become more familiar with them. But, second it will help you establish a mindset of understanding basic logging formats so you can more easily identify and deal with new or previously unseen log data in your environment. It's a fact of life that different vendors will implement log messages in different formats, but at the end of the day it's all about how you deal with and manage log data. The faster you can understand and integrate new log data into your overall logging system, the faster you will begin to gain value from it.
The remainder of this chapter is geared toward providing a foundation for the concepts that will be presented throughout the rest of this book. The ideas around log data, people, process, and technology will be explored, with some real-world examples sprinkled in to ensure you see the real value in log data.
LOG DATA BASICS
So far we have been making reference to logging and log data without providing a real concrete description of what these things are. Let's define these now in no uncertain terms the basics around logging and log data.
What Is Log Data?
At the heart of log data are, simply, log messages, or logs. A log message is what a computer system, device, software, etc. generates in response to some sort of stimuli. What exactly the stimuli are greatly depends on the source of the log message. For example, Unix systems will have user login and logout messages, firewalls will have ACL accept and deny messages, disk storage systems will generate log messages when failures occur or, in some cases, when the system perceives an impending failure.
Log data is the intrinsic meaning that a log message has. Or put another way, log data is the information pulled out of a log message to tell you why the log message generated. For example, a Web server will often log whenever someone accesses a resource (image, file, etc.) on a Web page. If the user accessing the page had to authenticate herself, the log message would contain the user's name. This is an example of log data: you can use the username to determine who accessed a resource.
The term logs is really used to indicate a collection of log messages that will be used collectively to paint a picture of some occurrence.
Log messages can be classified into the following general categories:
* Informational: Messages of this type are designed to let users and administrators know that something benign has occurred. For example, Cisco IOS will generate messages when the system is rebooted. Care must be taken, however. If a reboot, for example, occurs out of normal maintenance or business hours, you might have reason to be alarmed. Subsequent chapters in this book will provide you with the skills and techniques to be able to detect when something like this occurs.
* Debug: Debug messages are generally generated from software systems in order to aid software developers troubleshoot and identify problems with running application code.
* Warning: Warning messages are concerned with situations where things may be missing or needed for a system, but the absence of which will not impact system operation. For example, if a program isn't given the proper number of command line arguments, but yet it can run without them, is something the program might log just as a warning to the user or operator.
* Error: Error log messages are used to relay errors that occur at various levels in a computer system. For example, an operating system might generate an error log when it cannot synchronize buffers to disk. Unfortunately, many error messages only give you a starting point as to why they occurred. Further investigation is often required in order to get at the root cause of the error. Chapters 7, 8, 9, 10, 11, 12, 13, 15, and 16 in this book will provide you with ways to deal with this.
* Alert: An alert is meant to indicate that something interesting has happened. Alerts, in general, are the domain of security devices and security-related systems, but this is not a hard and fast rule. An Intrusion Prevention System (IPS) may sit in-line on a computer network, examining all inbound traffic. It will make a determination on whether or not a given network connection is allowed through based on the contents of the packet data. If the IPS encounters a connection that might be malicious it can take any number of pre-configured actions. The determination, along with the action taken, will be logged.
We will now turn to a brief discussion of how log data is transmitted and collected. Then we will discuss what constitutes a log message.
How is Log Data Transmitted and Collected?
Log data transmission and collection is conceptually simple. A computer or device implements a logging subsystem whereby it can generate a message anytime it determines it needs to. The exact way the determination is made depends on the device. For example, you may have the option to configure the device or the device may be hard coded to generate a pre-set list of messages. On the flip side, you have to have a place where the log message is sent and collected. This place is generally referred to as a loghost. A loghost is a computer system, generally a Unix system or Windows server, where log messages are collected in a central location. The advantages to using a central log collector are as follows:
* It's a centralized place to store log messages from multiple locations.
* It's a place to store backup copies of your logs.
* It's a place where analysis can be performed on you log data.
While this is all well and good, how are log messages transmitted in the first place? The most common way is via the Syslog protocol. The Syslog protocol is a standard for log message interchange. It is commonly found on Unix systems, but it exists for Windows and other non-Unix based platforms. But basically there is a client and server component implemented over the User Datagram Protocol (UDP), although many open source and commercial Syslog implementations also support the Transmission Control Protocol (TCP) for guaranteed delivery. The client portion is the actual device or computer system that generates and sends log messages. The server side would typically be found on a log collection server. Its main job is to take receipt of Syslog-based log messages and store them to local disk storage where they can be analyzed, backed up and stored for long-term use.
Syslog is not the only mechanism for log data transmission and collection. For example, Microsoft implements their own logging system for Windows. It is called the Windows Event Log. Things like user login and logoffs, application messages, and so on are stored in a proprietary storage format. There are open source and commercial applications that run on top of the Event Log which will convert event log entries to Syslog, where they are forwarded to a Syslog server. We will discuss the Windows Event Log in a little more detail in Chapters 3 and 16.
The Simple Network Management Protocol (SNMP) is a standards based protocol for managing networked devices. The protocol is based on two concepts: traps and polling. A trap is merely a form of log message that a device or computer system emits whenever something has happened. A trap is sent to a management station, which is analogous to a loghost. A management station is used to manage SNMP-based systems. Polling is where the management station is able use SNMP to query a device for pre-defined variables such as interface statistics, bytes transferred in and out on an interface, etc. A key differentiator between SNMP and Syslog is that SNMP is supposed to be structured with respect to data format. But this not always found in practice. If you would like to learn more about SNMP, see Essential SNMP (Mauro & Schmidt, 2005).
Databases have become a convenient way for applications to store log messages. Instead of generating a Syslog message, an application can write its log messages to a database schema. Or in some cases, the Syslog server itself can write directly a relational database. This has great advantages, especially around providing a structured way to store, analyze and report on log messages.
(Continues...)
Excerpted from Logging and Log Management by Anton A. Chuvakin. Copyright © 2013 by Elsevier, Inc.. Excerpted by permission of Elsevier Science.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Chapter 1 Logs, Trees, Forest: the Big Picture
Chapter 2 What is a Log?
Chapter 3 Generating Logs
Chapter 4 Log Storage Technologies
Chapter 5 Case study: syslog-ng
Chapter 6 Covert logging
Chapter 7 Analysis Goals, Planning and Preparation: What Are We Looking for?
Chapter 8 Simple Analysis Techniques
Chapter 9 Filtering, Matching and Correlation
Chapter 10 Statistical Analysis
Chapter 11 Log Data Mining
Chapter 12 Reporting and Summarization
Chapter 13 Visualizing Log Data
Chapter 14 Logging Laws and Logging Mistakes
Chapter 15 Tools for Log Analysis and Collection
Chapter 16 Log Management Procedures: Escalation, Response
Chapter 17 Attacks against logging systems
Chapter 18 Logging for Programmers
Chapter 19 Logs and Compliance
Chapter 20 Planning Your Own Log Analysis System
Chapter 21 Cloud Logging
Chapter 22 Log Standard and Future Trends