Reinforcement Learning for Cyber Operations: Applications of Artificial Intelligence for Penetration Testing

by Abdul Rahman, Christopher Redino, Dhruv Nandakumar, Tyler Cody, Sachin Shetty

Overview

A comprehensive and up-to-date application of reinforcement learning concepts to offensive and defensive cybersecurity

In Reinforcement Learning for Cyber Operations: Applications of Artificial Intelligence for Penetration Testing, a team of distinguished researchers delivers an incisive and practical discussion of reinforcement learning (RL) in cybersecurity that combines intelligence preparation for battle (IPB) concepts with multi-agent techniques. The authors explain how to conduct path analyses within networks, how to use sensor placement to increase the visibility of adversarial tactics and increase cyber defender efficacy, and how to improve your organization’s cyber posture with RL and illuminate the most probable adversarial attack paths in your networks.

Containing entirely original research, this book outlines findings and real-world scenarios that have been modeled and tested against custom generated networks, simulated networks, and data. You’ll also find:

• A thorough introduction to modeling actions within post-exploitation cybersecurity events, including Markov Decision Processes employing warm-up phases and penalty scaling
• Comprehensive explorations of penetration testing automation, including how RL is trained and tested over a standard attack graph construct
• Practical discussions of both red and blue team objectives in their efforts to exploit and defend networks, respectively
• Complete treatment of how reinforcement learning can be applied to real-world cybersecurity operational scenarios

Perfect for practitioners working in cybersecurity, including cyber defenders and planners, network administrators, and information security professionals, Reinforcement Learning for Cyber Operations: Applications of Artificial Intelligence for Penetration Testing will also benefit computer science researchers.

Product Details

ISBN-13:	9781394206452
Publisher:	Wiley
Publication date:	01/15/2025
Pages:	288
Product dimensions:	6.50(w) x 1.50(h) x 9.50(d)

About the Author

Dr. Abdul Rahman holds PhDs in physics, math, information technology–cybersecurity and has expertise in cybersecurity, big data, blockchain, and analytics (AI, ML).

Dr. Christopher Redino holds a PhD in theoretical physics and has extensive data science experience in every part of the AI / ML lifecycle.

Mr. Dhruv Nandakumar has extensive data science expertise in deep learning.

Dr. Tyler Cody is an Assistant Research Professor at the Virginia Tech National Security Institute.

Dr. Sachin Shetty is a Professor in the Electrical and Computer Engineering Department at Old Dominion University and the Executive Director of the Center for Secure and Intelligent Critical Systems at the Virginia Modeling, Analysis and Simulation Center.

Mr. Dan Radke is an Information Security professional with extensive experience in both offensive and defensive cybersecurity.

Table of Contents

List of Figures xv

About the Authors xix

Foreword xxi

Preface xxiii

Acknowledgments xxv

Acronyms xxvii

Introduction xxix

1 Motivation 1

1.1 Introduction 1

1.1.1 Cyberattack Campaigns via MITRE ATT&CK 4

1.2 Attack Graphs 4

1.3 Cyber Terrain 5

1.4 Penetration Testing 6

1.5 AI Reinforcement Learning Overview 6

1.6 Organization of the Book 8

2 Overview of Penetration Testing 11

2.1 Penetration Testing 11

2.2 Importance of Data 43

2.3 Conclusion 56

3 Reinforcement Learning: Theory and Application 61

3.1 An Introduction to Reinforcement Learning (RL) 61

3.2 RL and Markov Decision Processes 63

3.3 Learnable Functions for Agents 66

3.4 Enter Deep Learning 69

3.5 Q-Learning and Deep Q-Learning 72

3.6 Advantage Actor-Critic (A2C) 78

3.7 Proximal Policy Optimization 83

3.8 Conclusion 85

4 Motivation for Model-driven Penetration Testing 89

4.1 Introduction 89

4.2 Limits of Modern Attack Graphs 91

4.3 RL for Penetration Testing 93

4.4 Modeling MDPs 95

4.5 Conclusion 98

5 Operationalizing RL for Cyber Operations 105

5.1 A High-Level Architecture 105

5.2 Layered Reference Model 107

5.3 Key Challenges for Operationalizing RL 113

5.4 Conclusions 117

6 Toward Practical RL for Pen-Testing 121

6.1 Current Challenges to Practicality 121

6.2 Practical Scalability in RL 130

6.3 Model Realism 136

6.4 Examples of Applications 144

6.5 Realism and Scale 154

7 Putting it Into Practice: RL for Scalable Penetration Testing 161

7.1 Crown Jewels Analysis 161

7.2 Discovering Exfiltration Paths 165

7.3 Discovering Command and Control Channels 171

7.4 Exposing Surveillance Detection Routes 176

7.5 Enhanced Exfiltration Path Analysis 183

8 Using and Extending These Models 193

8.1 Supplementing Penetration Testing 193

8.2 Risk Scoring 199

8.3 Further Modeling 201

8.4 Generalization 214

9 Model-driven Penetration Testing in Practice 225

9.1 Recap 225

9.2 The Case for Model-driven Cyber Detections 231

References 246

A Appendix 251

Index 253