Practical Intrusion Analysis: Prevention and Detection for the Twenty-First Century

by Ryan Trost

Overview

“Practical Intrusion Analysis provides a solid fundamental overview of the art and science of intrusion analysis.”

   –Nate Miller, Cofounder, Stratum Security

 

The Only Definitive Guide to New State-of-the-Art Techniques in Intrusion Detection and Prevention

 

Recently, powerful innovations in intrusion detection and prevention have evolved in response to emerging threats and changing business environments. However, security practitioners have found little reliable, usable information about these new IDS/IPS technologies. In Practical Intrusion Analysis, one of the field’s leading experts brings together these innovations for the first time and demonstrates how they can be used to analyze attacks, mitigate damage, and track attackers.

 

Ryan Trost reviews the fundamental techniques and business drivers of intrusion detection and prevention by analyzing today’s new vulnerabilities and attack vectors. Next, he presents complete explanations of powerful new IDS/IPS methodologies based on Network Behavioral Analysis (NBA), data visualization, geospatial analysis, and more.

 

Writing for security practitioners and managers at all experience levels, Trost introduces new solutions for virtually every environment. Coverage includes

 

• Assessing the strengths and limitations of mainstream monitoring tools and IDS technologies
• Using Attack Graphs to map paths of network vulnerability and becoming more proactive about preventing intrusions
• Analyzing network behavior to immediately detect polymorphic worms, zero-day exploits, and botnet DoS attacks
• Understanding the theory, advantages, and disadvantages of the latest Web Application Firewalls
• Implementing IDS/IPS systems that protect wireless data traffic
• Enhancing your intrusion detection efforts by converging with physical security defenses
• Identifying attackers’ “geographical fingerprints” and using that information to respond more effectively
• Visualizing data traffic to identify suspicious patterns more quickly
• Revisiting intrusion detection ROI in light of new threats, compliance risks, and technical alternatives

 

Includes contributions from these leading network security experts:

 

Jeff Forristal, a.k.a. Rain Forest Puppy, senior security professional and creator of libwhisker

Seth Fogie, CEO, Airscanner USA; leading-edge mobile security researcher; coauthor of Security Warrior

 

Dr. Sushil Jajodia, Director, Center for Secure Information Systems; founding Editor-in-Chief, Journal of Computer Security

 

Dr. Steven Noel, Associate Director and Senior Research Scientist, Center for Secure Information Systems, George Mason University

 

Alex Kirk, Member, Sourcefire Vulnerability Research Team

 

Product Details

ISBN-13:	9780321591883
Publisher:	Pearson Education
Publication date:	06/24/2009
Sold by:	Barnes & Noble
Format:	eBook
Pages:	480
File size:	14 MB
Note:	This product may take a few minutes to download.

About the Author

Ryan Trost is the Director of Security and Data Privacy Officer at Comprehensive Health Services where he oversees all the organization’s security and privacy decisions. He teaches several Information Technology courses, including Ethical Hacking, Intrusion Detection, and Data Visualization at Northern Virginia Community College. This enables him to continue exploring his technical interests among the endless managerial meetings. In his spare time, Ryan works to cross-pollinate network security, GIS, and data visualization. He is considered a leading expert in geospatial intrusion detection techniques and has spoken at several conferences on the topic, most notably DEFCON 16. Ryan participated as a RedTeamer in the first annual Collegiate Cyber Defense Competition (CCDC) and now fields a team of students in the annual event. Ryan has been a senior security consultant for several government agencies before transitioning over to the private sector. In 2005, Ryan received his masters of science degree in computer science from George Washington University where he developed his first geospatial intrusion detection tool.

Read an Excerpt

Practical Intrusion AnalysisPreface

This book was developed to help fill multiple gaps in practical intrusion detection within a single cover-to-cover publication. Traditionally, intrusion detection books concentrate on narrow subject matter that focuses on vendor-specific information, like Snort or Cisco MARS, Intrusion Detection System (IDS) installation, and sensor placement or signature writing. This book incorporates the essential core knowledge to understand the IDS, but it also expands the subject matter to other relevant areas of intrusion interest, such as NetFlow, wireless IDS/Intrusion Prevention System (IPS), physical security, and geospatial intrusion detection. Don’t get me wrongelthe previously mentioned books are the foundation of my security knowledge, but as the industry matures to include various facets of incursion, its books should incorporate those facets into a single publication so security aficionados don’t have to fracture their attention across so many titles.

Who Should Read This Book

This book’s audience is any and all security practitioners; whether you’re an entry-level security analyst, a chief security officer, or even a prospective college student researching a career in network security. Every chapter might not provide a silver-bullet solution that protects your company from every well-versed attacker. But, as you peel the onion layers, you find a combination of included security defenses that help ensure your company’s security posture and out-endure even the most motivated attacker(s).

How to Read This Book

Although, at first glance, the chapters might seem independent, a structure guides you from the first few chapters that provide a fundamental foundation, including Chapter 1 “Network Overview,” and Chapter 2, “Infrastructure Monitoring,” to more advanced chapters. You are introduced to new intrusion detection strategies consisting of wireless IDS/IPS, network behavioral analysis (NBA), converging of physical and logical security, and geospatial intrusion detection. Several traditional chapters explore new approaches, including ones that cover IDSs, vulnerability signature dissection, and Web application firewalls.

I was lucky enough to have several knowledgeable friends that, with some begging and pleading, agreed to include their extensive security insight, experience, and opinions. I avoid duplicating materials presented in other books because I want to fill the gaps of current security initiatives and/or explore the arena of new concepts and strategies.

How This Book Is Organized

This book follows a compartmentalized organization because each chapter focuses on specific technologies. The beginning of this book introduces basic networking terminology, and it transitions into overviewing intrusion detection, which caters to the InfoSec newbies and finally dives into more sophisticated and advanced intrusion defenses. Here is a brief description of each chapter:

Chapter 1, “Network Overview,” focuses on basic network structure and briefly explains the anatomy of TCP/IP and OSI. Most IT-related books must include some introductory chapter to either define the foundation of the technology or refresh readers that might not deal with it in their daily lives; this book is no different. It is not meant to be an in-depth analysis, but it eases you into the more sophisticated work to come.

Chapter 2, “Infrastructure Monitoring,” explores some common network security practices, including vulnerability assessments, packet sniffing, IDS, file integrity checking, password auditing, wireless toolkits, exploitation toolkits, and network reconnaissance tools. Network security heavily relies on the tools used to “see” the traffic. However, as the chapter title indicates, a majority of this chapter concentrates on mainstream monitoring capabilities and the never-ending battle between using a tap or SPAN for monitoring purposes.

Chapter 3, “Intrusion Detection Systems,” provides you with insight into the IDS industry by introducing fundamental concepts and then progressively jumping into more complex topics, including evasion techniques, signature dissection, and a look into the Snort and BRO IDSs, while simultaneously providing as little duplication of previous material as possible. Most IDS books written in the past focus solely on Snort, snort.conf (Snort’s configuration file), and the signature syntax. However, few publications truly clarify the distinction between writing a signature looking for an exploit versus writing a signature identifying a system’s vulnerability. Finally, the chapter ends with an assessment of two open source systems, Snort and Bro, which take different approaches to intrusion detection.

Chapter 4, “Lifecycle of a Vulnerability,” steps you through the natural evolution of a vulnerability, from discovering the vulnerability, to capturing the packet stream, to analyzing the malicious content within the packet, and writing an efficient Snort signature to alert on it. It does all this, while simultaneously exposing you to a small subset of necessary tools to help you in your quest. The examples escalate in complexity and are specifically chosen to reflect relatively recent events, because they were all released within the past few months. For newcomers, the analysis of a packet might appear overwhelming and tedious, but if you segment it and step through the packet capture packet-by-packet, the process starts to fall into place. For the already skilled signature writers, the advanced examples, which use flowbits, PCRE, and newly shared object rules, shed some light on the thought process and technique that the Sourcefire VRT team uses.

Chapter 5, “Proactive Intrusion Prevention and Response via Attack Graphs,” examines proactive methods of attack risk reduction and response through attack graphs. Administrators and security analysts are overwhelmed by constant outside threats, complexity of security measures, and network growth. Today’s status quo for network defense is often reduced to mere triage and post-mortem remediation. The attack graphs map potential paths of vulnerability through a network, showing exactly how attackers might penetrate a network. Attack graph analysis identifies critical vulnerabilities and provides strategies for protecting critical network assets. But, because of operational realities, vulnerability paths often remain. In such cases, attack graphs provide an ideal methodology for planning appropriate attack responses. This includes optimal placement of intrusion detection sensors, correlating intrusion alarms, accounting for missed detections, prioritizing alarms, and predicting the next possible attack steps.

Chapter 6, “Network Flows and Anomaly Detection,” explores the topic of network flow data: its collection for network security analysis and, specifically, an emerging field called Network Behavior Analysis (NBA). First, this chapter explores flow technology and analyzes the different flow formats: their characteristics, respective datasets, and key fields. It discusses how network flow deployments affect device performance and statistical sampling and then introduces possible data flow collection strategies. Although traditional IDS/IPS technologies are still an environment staple, they are blind to specific attacks, whereas NBA fills those gaps and perfectly complements because it excels at immediately detecting polymorphic worms, zero-day exploits, and botnet denial of service (DoS) attacks. Whereas IDS and packet sniffing software are microanalytical tools that examine packet contents, data flow is a macroanalytical mechanism that characterizes large volumes of traffic in real time.

Chapter 7, “Web Application Firewalls,” exposes you to the terms, theories, advantages, and disadvantages of the Web Application Firewall (WAF), which is quickly becoming a solution of choice for companies who operate mission-critical Web sites. With the explosion of the Internet, an entire new family of attack vectors has been created that redefine the traditional concept of a threat. Whether it is the database server, Web server or even the visitors of the targeted site, these threats are often embedded in seemingly innocent traffic that many IDSs do not have the power or capability to detect.

Chapter 8, “Wireless IDS/IPS.” For the most part, intrusion detection focuses on the data passing from point A to point B. However, this is a limited view of data transmission, because it fails to consider the physical properties of the transmission process. Thanks to wireless networking, data no longer has to exist as electronic pulses on a wire, but can now live as radio waves in the air. Unfortunately, this means traditional IDS solutions are no longer qualified to fully protect this information, if only because they cannot interpret RF energy. In this chapter, you gain an understanding of the issues related to wireless security, the shortcomings of the network-based IDS, and the options available to those who want to keep a close eye on their wireless traffic.

Chapter 9, “Physical Intrusion Detection for IT,” gets IT security staffs thinking about how intrusion detection efforts can be bolstered by converging with the physical security team. This chapter includes an overview of physical security technologies to help IT security personnel understand the perspective of the physical security team and familiarize themselves with the physical security technology terrain. A few example scenarios illustrate the possibilities of what converged detection can offer.

Chapter 10 “Geospatial Intrusion Detection.” IDSs/IPSs are becoming more advanced, and geocoding source IP addresses is adding another layer of defensive intelligence. The ultimate goal of geospatial intrusion detection is to maximize situational awareness and threat visualization techniques among security analysts. Most attackers use multiple zombie machines to launch professional attacks, but even a zombie's network reconnaissance leaves geographic fingerprints that are easily picked up by pattern recognition algorithms from the Geographic Information Systems (GIS) industry. This chapter proves how the source IP address is one of the most overlooked and powerful components of an intrusion detection log.

Chapter 11, “Visual Data Communication.” Visualization of security data has become an increasingly discussed topic. As data retention policies (regulatory, federal, and especially state) increasingly capture the compliance spotlight, it is forcing companies to retain audit logs for extended time periods and, in some cases indefinitely, because of the lack of legal normalization. NetFlow is a perfect example of how beneficial visualizing data can be. As it samples the network traffic, an analyst can immediately identify suspicious patterns. Countless possible datapoints can be tracked and visualized within a company’s network. The driving focus is to put into words that visualizing security alerts are left to interpretation because what helps me defend my network might not help you preserve yours. This chapter provides a broad view of the different visualization possibilities.

Chapter 12, “Return on Investment: Business Justification,” involves the nontechnical anomaly as it focuses on management decisions regarding intrusion detection security. Looking back on my career path, I went from system administrator to security analyst to security manager and, finally, to director of security (where were few resources helped me make the psychological transition). The leap from security analyst to security manager changed my responsibilities. Whereas before, I focused on packet analysis and IDS/FW correlation, I now had to focus on shift coverage, the interview process, policy development, and billable hours to the client. Through natural progression, my next career move was to director of security, where my responsibilities expanded to contract review, department budgetary considerations, and keeping the security department aligned with business goals and regulatory compliance. This chapter conveys valuable insight on the compliance landscape, a breakdown on ROI strategies, and introduces cyber liability insurance. This chapter conveys valuable insight for both today’s, and tomorrow’s, security directors. Regardless of what your security tier, you’re always training for the next escalation of privileges.

© Copyright Pearson Education. All rights reserved.

Table of Contents

Preface     xv

Chapter 1: Network Overview     1

Chapter 2: Infrastructure Monitoring     31

Chapter 3: Intrusion Detection Systems     53

Chapter 4: Lifecycle of a Vulnerability     87

Chapter 5: Proactive Intrusion Prevention and Response via Attack Graphs     119

Chapter 6: Network Flows and Anomaly Detection     151

Chapter 7: Web Application Firewalls     185

Chapter 8: Wireless IDS/IPS     209

Chapter 9: Physical Intrusion Detection for IT     235

Chapter 10: Geospatial Intrusion Detection     275

Chapter 11: Visual Data Communications     347

Chapter 12: Return on Investment: Business Justification     391

Appendix: Bro Installation Guide     435

Index     441

Preface

Practical Intrusion Analysis Preface

This book was developed to help fill multiple gaps in practical intrusion detection within a single cover-to-cover publication. Traditionally, intrusion detection books concentrate on narrow subject matter that focuses on vendor-specific information, like Snort or Cisco MARS, Intrusion Detection System (IDS) installation, and sensor placement or signature writing. This book incorporates the essential core knowledge to understand the IDS, but it also expands the subject matter to other relevant areas of intrusion interest, such as NetFlow, wireless IDS/Intrusion Prevention System (IPS), physical security, and geospatial intrusion detection. Don’t get me wrongelthe previously mentioned books are the foundation of my security knowledge, but as the industry matures to include various facets of incursion, its books should incorporate those facets into a single publication so security aficionados don’t have to fracture their attention across so many titles.

Who Should Read This Book

This book’s audience is any and all security practitioners; whether you’re an entry-level security analyst, a chief security officer, or even a prospective college student researching a career in network security. Every chapter might not provide a silver-bullet solution that protects your company from every well-versed attacker. But, as you peel the onion layers, you find a combination of included security defenses that help ensure your company’s security posture and out-endure even the most motivated attacker(s).

How to Read This Book

Although, at first glance, the chapters might seem independent, a structure guides you from the first few chapters that provide a fundamental foundation, including Chapter 1 “Network Overview,” and Chapter 2, “Infrastructure Monitoring,” to more advanced chapters. You are introduced to new intrusion detection strategies consisting of wireless IDS/IPS, network behavioral analysis (NBA), converging of physical and logical security, and geospatial intrusion detection. Several traditional chapters explore new approaches, including ones that cover IDSs, vulnerability signature dissection, and Web application firewalls.

I was lucky enough to have several knowledgeable friends that, with some begging and pleading, agreed to include their extensive security insight, experience, and opinions. I avoid duplicating materials presented in other books because I want to fill the gaps of current security initiatives and/or explore the arena of new concepts and strategies.

How This Book Is Organized

This book follows a compartmentalized organization because each chapter focuses on specific technologies. The beginning of this book introduces basic networking terminology, and it transitions into overviewing intrusion detection, which caters to the InfoSec newbies and finally dives into more sophisticated and advanced intrusion defenses. Here is a brief description of each chapter:

• Chapter 1, “Network Overview,” focuses on basic network structure and briefly explains the anatomy of TCP/IP and OSI. Most IT-related books must include some introductory chapter to either define the foundation of the technology or refresh readers that might not deal with it in their daily lives; this book is no different. It is not meant to be an in-depth analysis, but it eases you into the more sophisticated work to come.
• Chapter 2, “Infrastructure Monitoring,” explores some common network security practices, including vulnerability assessments, packet sniffing, IDS, file integrity checking, password auditing, wireless toolkits, exploitation toolkits, and network reconnaissance tools. Network security heavily relies on the tools used to “see” the traffic. However, as the chapter title indicates, a majority of this chapter concentrates on mainstream monitoring capabilities and the never-ending battle between using a tap or SPAN for monitoring purposes.
• Chapter 3, “Intrusion Detection Systems,” provides you with insight into the IDS industry by introducing fundamental concepts and then progressively jumping into more complex topics, including evasion techniques, signature dissection, and a look into the Snort and BRO IDSs, while simultaneously providing as little duplication of previous material as possible. Most IDS books written in the past focus solely on Snort, snort.conf (Snort’s configuration file), and the signature syntax. However, few publications truly clarify the distinction between writing a signature looking for an exploit versus writing a signature identifying a system’s vulnerability. Finally, the chapter ends with an assessment of two open source systems, Snort and Bro, which take different approaches to intrusion detection.
• Chapter 4, “Lifecycle of a Vulnerability,” steps you through the natural evolution of a vulnerability, from discovering the vulnerability, to capturing the packet stream, to analyzing the malicious content within the packet, and writing an efficient Snort signature to alert on it. It does all this, while simultaneously exposing you to a small subset of necessary tools to help you in your quest. The examples escalate in complexity and are specifically chosen to reflect relatively recent events, because they were all released within the past few months. For newcomers, the analysis of a packet might appear overwhelming and tedious, but if you segment it and step through the packet capture packet-by-packet, the process starts to fall into place. For the already skilled signature writers, the advanced examples, which use flowbits, PCRE, and newly shared object rules, shed some light on the thought process and technique that the Sourcefire VRT team uses.
• Chapter 5, “Proactive Intrusion Prevention and Response via Attack Graphs,” examines proactive methods of attack risk reduction and response through attack graphs. Administrators and security analysts are overwhelmed by constant outside threats, complexity of security measures, and network growth. Today’s status quo for network defense is often reduced to mere triage and post-mortem remediation. The attack graphs map potential paths of vulnerability through a network, showing exactly how attackers might penetrate a network. Attack graph analysis identifies critical vulnerabilities and provides strategies for protecting critical network assets. But, because of operational realities, vulnerability paths often remain. In such cases, attack graphs provide an ideal methodology for planning appropriate attack responses. This includes optimal placement of intrusion detection sensors, correlating intrusion alarms, accounting for missed detections, prioritizing alarms, and predicting the next possible attack steps.
• Chapter 6, “Network Flows and Anomaly Detection,” explores the topic of network flow data: its collection for network security analysis and, specifically, an emerging field called Network Behavior Analysis (NBA). First, this chapter explores flow technology and analyzes the different flow formats: their characteristics, respective datasets, and key fields. It discusses how network flow deployments affect device performance and statistical sampling and then introduces possible data flow collection strategies. Although traditional IDS/IPS technologies are still an environment staple, they are blind to specific attacks, whereas NBA fills those gaps and perfectly complements because it excels at immediately detecting polymorphic worms, zero-day exploits, and botnet denial of service (DoS) attacks. Whereas IDS and packet sniffing software are microanalytical tools that examine packet contents, data flow is a macroanalytical mechanism that characterizes large volumes of traffic in real time.
• Chapter 7, “Web Application Firewalls,” exposes you to the terms, theories, advantages, and disadvantages of the Web Application Firewall (WAF), which is quickly becoming a solution of choice for companies who operate mission-critical Web sites. With the explosion of the Internet, an entire new family of attack vectors has been created that redefine the traditional concept of a threat. Whether it is the database server, Web server or even the visitors of the targeted site, these threats are often embedded in seemingly innocent traffic that many IDSs do not have the power or capability to detect.
• Chapter 8, “Wireless IDS/IPS.” For the most part, intrusion detection focuses on the data passing from point A to point B. However, this is a limited view of data transmission, because it fails to consider the physical properties of the transmission process. Thanks to wireless networking, data no longer has to exist as electronic pulses on a wire, but can now live as radio waves in the air. Unfortunately, this means traditional IDS solutions are no longer qualified to fully protect this information, if only because they cannot interpret RF energy. In this chapter, you gain an understanding of the issues related to wireless security, the shortcomings of the network-based IDS, and the options available to those who want to keep a close eye on their wireless traffic.
• Chapter 9, “Physical Intrusion Detection for IT,” gets IT security staffs thinking about how intrusion detection efforts can be bolstered by converging with the physical security team. This chapter includes an overview of physical security technologies to help IT security personnel understand the perspective of the physical security team and familiarize themselves with the physical security technology terrain. A few example scenarios illustrate the possibilities of what converged detection can offer.
• Chapter 10 “Geospatial Intrusion Detection.” IDSs/IPSs are becoming more advanced, and geocoding source IP addresses is adding another layer of defensive intelligence. The ultimate goal of geospatial intrusion detection is to maximize situational awareness and threat visualization techniques among security analysts. Most attackers use multiple zombie machines to launch professional attacks, but even a zombie's network reconnaissance leaves geographic fingerprints that are easily picked up by pattern recognition algorithms from the Geographic Information Systems (GIS) industry. This chapter proves how the source IP address is one of the most overlooked and powerful components of an intrusion detection log.
• Chapter 11, “Visual Data Communication.” Visualization of security data has become an increasingly discussed topic. As data retention policies (regulatory, federal, and especially state) increasingly capture the compliance spotlight, it is forcing companies to retain audit logs for extended time periods and, in some cases indefinitely, because of the lack of legal normalization. NetFlow is a perfect example of how beneficial visualizing data can be. As it samples the network traffic, an analyst can immediately identify suspicious patterns. Countless possible datapoints can be tracked and visualized within a company’s network. The driving focus is to put into words that visualizing security alerts are left to interpretation because what helps me defend my network might not help you preserve yours. This chapter provides a broad view of the different visualization possibilities.
• Chapter 12, “Return on Investment: Business Justification,” involves the nontechnical anomaly as it focuses on management decisions regarding intrusion detection security. Looking back on my career path, I went from system administrator to security analyst to security manager and, finally, to director of security (where were few resources helped me make the psychological transition). The leap from security analyst to security manager changed my responsibilities. Whereas before, I focused on packet analysis and IDS/FW correlation, I now had to focus on shift coverage, the interview process, policy development, and billable hours to the client. Through natural progression, my next career move was to director of security, where my responsibilities expanded to contract review, department budgetary considerations, and keeping the security department aligned with business goals and regulatory compliance. This chapter conveys valuable insight on the compliance landscape, a breakdown on ROI strategies, and introduces cyber liability insurance. This chapter conveys valuable insight for both today’s, and tomorrow’s, security directors. Regardless of what your security tier, you’re always training for the next escalation of privileges.

© Copyright Pearson Education. All rights reserved.