The Shellcoder's Handbook: Discovering and Exploiting Security Holes

by Chris Anley, John Heasman, Felix Lindner, Gerardo Richarte

Overview

• This much-anticipated revision, written by the ultimate group of top security experts in the world, features 40 percent new content on how to find security holes in any operating system or application
• New material addresses the many new exploitation techniques that have been discovered since the first edition, including attacking "unbreakable" software packages such as McAfee's Entercept, Mac OS X, XP, Office 2003, and Vista
• Also features the first-ever published information on exploiting Cisco's IOS, with content that has never before been explored
• The companion Web site features downloadable code files

Product Details

ISBN-13:	9780470080238
Publisher:	Wiley
Publication date:	08/20/2007
Edition description:	Revised Edition
Pages:	752
Product dimensions:	7.40(w) x 9.20(h) x 1.60(d)

About the Author

Chris Anley is a founder and director of NGSSoftware, a security software, consultancy, and research company based in London, England. He is actively involved in vulnerability research and has discovered security flaws in a wide variety of platforms including Microsoft Windows, Oracle, SQL Server, IBM DB2, Sybase ASE, MySQL, and PGP.

John Heasman is the Director of Research at NGSSoftware. He is a prolific security researcher and has published many security advisories in enterprise level software. He has a particular interest in rootkits and has authored papers on malware persistence via device firmware and the BIOS. He is also a co-author of The Database Hacker’s Handbook: Defending Database Servers (Wiley 2005).

Felix “FX” Linder leads SABRE Labs GmbH, a Berlin-based professional consulting company specializing in security analysis, system design creation, and verification work. Felix looks back at 18 years of programming and over a decade of computer security consulting for enterprise, carrier, and software vendor clients. This experience allows him to rapidly dive into complex systems and evaluate them from a security and robustness point of view, even in atypical scenarios and on arcane platforms. In his spare time, FX works with his friends from the Phenoelit hacking group on different topics, which have included Cisco IOS, SAP, HP printers, and RIM BlackBerry in the past.

Gerardo Richarte has been doing reverse engineering and exploit development for more than 15 years non-stop. In the past 10 years he helped build the technical arm of Core Security Technologies, where he works today. His current duties include developing exploits for Core IMPACT, researching new exploitation techniques and other low-level subjects, helping other exploit writers when things get hairy, and teaching internal and external classes on assembly and exploit writing. As result of his research and as a humble thank you to the community, he has published some technical papers and open source projects, presented in a few conferences, and released part of his training material. He really enjoys solving tough problems and reverse engineering any piece of code that falls in his reach just for the fun of doing it.

Read an Excerpt

Click to read or download

Table of Contents

About the Authors vii

Acknowledgments xi

Introduction to the Second Edition xxiii

Part I Introduction to Exploitation: Linux on X 86

Chapter 1 Before You Begin 3

Basic Concepts 3

Memory Management 4

Assembly 6

Recognizing C and C++ Code Constructs in Assembly 7

Conclusion 10

Chapter 2 Stack Overflows 11

Buffers 12

The Stack 13

Functions and the Stack 15

Overflowing Buffers on the Stack 18

Controlling EIP 22

An Interesting Diversion 23

Using an Exploit to Get Root Privileges 25

The Address Problem 27

The NOP Method 33

Defeating a Non-Executable Stack 35

Return to libc 35

Conclusion 39

Chapter 3 Shellcode 41

Understanding System Calls 42

Writing Shellcode for the exit() Syscall 44

Injectable Shellcode 48

Spawning a Shell 50

Conclusion 59

Chapter 4 Introduction to Format String Bugs 61

Prerequisites 61

What Is a Format String? 61

What Is a Format String Bug? 63

Format String Exploits 68

Crashing Services 69

Information Leakage 70

Controlling Execution for Exploitation 75

Why Did This Happen? 84

Format String Technique Roundup 85

Conclusion 88

Chapter 5 Introduction to Heap Overflows 89

What Is a Heap? 90

How a Heap Works 91

Finding Heap Overflows 91

Basic Heap Overflows 93

Intermediate Heap Overflows 98

Advanced Heap Overflow Exploitation 105

Conclusion 107

Part II other Platforms—windows, Solaris, OS/X, and Cisco

Chapter 6 The Wild World of Windows 111

How Does Windows Differ from Linux? 111

Win32 API and PE-COFF 112

Heaps 114

Threading 115

The Genius and Idiocy of the Distributed Common Object Model and DCE-RPC 116

Recon 118

Exploitation 120

Tokens and Impersonation 120

Exception Handling under Win 32 122

Debugging Windows 124

Bugs in Win 32 124

Writing Windows Shellcode 125

A Hacker’s Guide to the Win32 API 126

A Windows Family Tree from the Hacker’s Perspective 126

Conclusion 127

Chapter 7 Windows Shellcode 129

Syntax and Filters 129

Setting Up 131

Parsing the PEB 132

Heapoverflow.c Analysis 132

Searching with Windows Exception Handling 148

Popping a Shell 153

Why You Should Never Pop a Shell on Windows 153

Conclusion 154

Chapter 8 Windows Overflows 155

Stack-Based Buffer Overflows 156

Frame-Based Exception Handlers 156

Abusing Frame-Based Exception Handling on Windows 2003 Server 161

A Final Note about Frame-Based Handler Overwrites 166

Stack Protection and Windows 2003 Server 166

Heap-Based Buffer Overflows 173

The Process Heap 173

Dynamic Heaps 173

Working with the Heap 173

How the Heap Works 174

Exploiting Heap-Based Overflows 178

Overwrite Pointer to RtlEnterCriticalSection in the PEB 178

Overwrite Pointer to Unhandled Exception Filter 185

Repairing the Heap 191

Other Aspects of Heap-Based Overflows 193

Wrapping Up the Heap 194

Other Overflows 194

.data Section Overflows 194

TEB/PEB Overflows 196

Exploiting Buffer Overflows and Non-Executable Stacks 197

Conclusion 203

Chapter 9 Overcoming Filters 205

Writing Exploits for Use with an Alphanumeric Filter 205

Writing Exploits for Use with a Unicode Filter 209

What Is Unicode? 210

Converting from ASCII to Unicode 210

Exploiting Unicode-Based Vulnerabilities 211

The Available Instruction Set in Unicode Exploits 212

The Venetian Method 213

An ASCII Venetian Implementation 214

Decoder and Decoding 218

The Decoder Code 219

Getting a Fix on the Buffer Address 220

Conclusion 221

Chapter 10 Introduction to Solaris Exploitation 223

Introduction to the SPARC Architecture 224

Registers and Register Windows 224

The Delay Slot 227

Synthetic Instructions 228

Solaris/SPARC Shellcode Basics 228

Self-Location Determination and SPARC Shellcode 228

Simple SPARC exec Shellcode 229

Useful System Calls on Solaris 230

NOP and Padding Instructions 231

Solaris/SPARC Stack Frame Introduction 231

Stack-Based Overflow Methodologies 232

Arbitrary Size Overflow 232

Register Windows and Stack Overflow Complications 233

Other Complicating Factors 233

Possible Solutions 234

Off-By-One Stack Overflow Vulnerabilities 234

Shellcode Locations 235

Stack Overflow Exploitation In Action 236

The Vulnerable Program 236

The Exploit 238

Heap-Based Overflows on Solaris/SPARC 241

Solaris System V Heap Introduction 242

Heap Tree Structure 242

Basic Exploit Methodology (t_delete) 263

Standard Heap Overflow Limitations 266

Targets for Overwrite 267

Other Heap-Related Vulnerabilities 270

Off-by-One Overflows 270

Double Free Vulnerabilities 270

Arbitrary Free Vulnerabilities 271

Heap Overflow Example 271

The Vulnerable Program 272

Other Solaris Exploitation Techniques 276

Static Data Overflows 276

Bypassing the Non-Executable Stack Protection 276

Conclusion 277

Chapter 11 Advanced Solaris Exploitation 279

Single Stepping the Dynamic Linker 281

Various Style Tricks for Solaris SPARC Heap Overflows 296

Advanced Solaris/SPARC Shellcode 299

Conclusion 311

Chapter 12 OS X Shellcode 313

OS X Is Just BSD, Right? 314

Is OS X Open Source? 314

OS X for the Unix-aware 315

Password Cracking 316

OS X PowerPC Shellcode 316

OS X Intel Shellcode 324

Example Shellcode 326

ret2libc 327

ret2str(l)cpy 329

OS X Cross-Platform Shellcode 332

OS X Heap Exploitation 333

Bug Hunting on OS X 335

Some Interesting Bugs 335

Essential Reading for OS X Exploits 337

Conclusion 338

Chapter 13 Cisco IOS Exploitation 339

An Overview of Cisco IOS 339

Hardware Platforms 340

Software Packages 340

IOS System Architecture 343

Vulnerabilities in Cisco IOS 346

Protocol Parsing Code 347

Services on the Router 347

Security Features 348

The Command-Line Interface 348

Reverse Engineering IOS 349

Taking the Images Apart 349

Diffing IOS Images 350

Runtime Analysis 351

Exploiting Cisco IOS 357

Stack Overflows 357

Heap Overflows 359

Shellcodes 364

Conclusion 373

Chapter 14 Protection Mechanisms 375

Protections 375

Non-Executable Stack 376

W^X (Either Writable or Executable) Memory 381

Stack Data Protection 388

AAAS: ASCII Armored Address Space 394

ASLR: Address Space Layout Randomization 396

Heap Protections 399

Windows SEH Protections 407

Other Protections 411

Implementation Differences 413

Windows 413

Linux 417

OpenBSD 421

Mac OS X 422

Solaris 423

Conclusion 425

Part III Vulnerability Discovery

Chapter 15 Establishing a Working Environment 429

What You Need for Reference 430

What You Need for Code 430

gcc 430

gdb 430

NASM 431

WinDbg 431

OllyDbg 431

Visual C++ 431

Python 432

What You Need for Investigation 432

Useful Custom Scripts/Tools 432

All Platforms 434

Unix 434

Windows 435

What You Need to Know 436

Paper Archives 438

Optimizing Shellcode Development 439

Plan the Exploit 439

Write the Shellcode in Inline Assembler 439

Maintain a Shellcode Library 441

Make It Continue Nicely 441

Make the Exploit Stable 442

Make It Steal the Connection 443

Conclusion 443

Chapter 16 Fault Injection 445

Design Overview 447

Input Generation 447

Fault Injection 450

Modification Engines 450

Fault Delivery 455

Nagel Algorithm 455

Timing 455

Heuristics 456

Stateless versus State-Based Protocols 456

Fault Monitoring 456

Using a Debugger 457

FaultMon 457

Putting It Together 458

Conclusion 459

Chapter 17 The Art of Fuzzing 461

General Theory of Fuzzing 461

Static Analysis versus Fuzzing 466

Fuzzing Is Scalable 466

Weaknesses in Fuzzers 468

Modeling Arbitrary Network Protocols 469

Other Fuzzer Possibilities 469

Bit Flipping 469

Modifying Open Source Programs 470

Fuzzing with Dynamic Analysis 470

Spike 471

What Is a Spike? 471

Why Use the SPIKE Data Structure to Model Network Protocols? 472

Other Fuzzers 480

Conclusion 480

Chapter 18 Source Code Auditing: Finding Vulnerabilities in C-Based Languages 481

Tools 482

Cscope 482

Ctags 483

Editors 483

Cbrowser 484

Automated Source Code Analysis Tools 484

Methodology 485

Top-Down (Specific) Approach 485

Bottom-Up Approach 485

Selective Approach 485

Vulnerability Classes 486

Generic Logic Errors 486

(Almost) Extinct Bug Classes 487

Format Strings 487

Generic Incorrect Bounds-Checking 489

Loop Constructs 490

Off-by-One Vulnerabilities 490

Non-Null Termination Issues 492

Skipping Null-Termination Issues 493

Signed Comparison Vulnerabilities 494

Integer-Related Vulnerabilities 495

Different-Sized Integer Conversions 497

Double Free Vulnerabilities 498

Out-of-Scope Memory Usage Vulnerabilities 499

Uninitialized Variable Usage 499

Use After Free Vulnerabilities 500

Multithreaded Issues and Re-Entrant Safe Code 500

Beyond Recognition: A Real Vulnerability versus a Bug 501

Conclusion 501

Chapter 19 Instrumented Investigation: A Manual Approach 503

Philosophy 503

Oracle extproc Overflow 504

Common Architectural Failures 508

Problems Happen at Boundaries 508

Problems Happen When Data Is Translated 509

Problems Cluster in Areas of Asymmetry 511

Problems Occur When Authentication and Authorization Are Confused 512

Problems Occur in the Dumbest Places 512

Bypassing Input Validation and Attack Detection 513

Stripping Bad Data 513

Using Alternate Encodings 514

Using File-Handling Features 515

Evading Attack Signatures 517

Defeating Length Limitations 517

Windows 2000 SNMP DOS 520

Finding DOS Attacks 521

SQL-UDP 522

Conclusion 523

Chapter 20 Tracing for Vulnerabilities 525

Overview 526

A Vulnerable Program 527

Component Design 529

Building VulnTrace 538

Using VulnTrace 543

Advanced Techniques 546

Conclusion 548

Chapter 21 Binary Auditing: Hacking Closed Source Software 549

Binary versus Source-Code Auditing: The Obvious Differences 550

IDA Pro—The Tool of the Trade 550

Features: A Quick Crash Course 551

Debugging Symbols 552

Binary Auditing Introduction 552

Stack Frames 552

Calling Conventions 554

Compiler-Generated Code 556

memcpy-Like Code Constructs 560

strlen-Like Code Constructs 560

C++ Code Constructs 561

The this Pointer 561

Reconstructing Class Definitions 562

vtables 562

Quick but Useful Tidbits 563

Manual Binary Analysis 563

Quick Examination of Library Calls 564

Suspicious Loops and Write Instructions 564

Higher-Level Understanding and Logic Bugs 565

Graphical Analysis of Binaries 566

Manual Decompilation 566

Binary Vulnerability Examples 566

Microsoft SQL Server Bugs 566

LSD’s RPC-DCOM Vulnerability 567

IIS WebDAV Vulnerability 568

Conclusion 570

Part IV Advanced Materials

Chapter 22 Alternative Payload Strategies 573

Modifying the Program 574

The SQL Server 3-Byte Patch 575

The MySQL 1-Bit Patch 578

OpenSSH RSA Authentication Patch 580

Other Runtime Patching Ideas 581

GPG 1.2.2 Randomness Patch 583

Upload and Run (or Proglet Server) 584

Syscall Proxies 584

Problems with Syscall Proxies 587

Conclusion 596

Chapter 23 Writing Exploits that Work in the Wild 597

Factors in Unreliability 597

Magic Numbers 597

Versioning 598

Shellcode Problems 599

Countermeasures 601

Preparation 602

Brute Forcing 602

Local Exploits 603

OS/Application Fingerprinting 603

Information Leaks 605

Conclusion 606

Chapter 24 Attacking Database Software 607

Network Layer Attacks 608

Application Layer Attacks 618

Running Operating System Commands 619

Microsoft SQL Server 619

Oracle 620

IBM DB 2 621

Exploiting Overruns at the SQL Level 623

SQL Functions 623

Conclusion 625

Chapter 25 Unix Kernel Overflows 627

Kernel Vulnerability Types 627

0day Kernel Vulnerabilities 636

OpenBSD exec_ibcs2_coff_prep_zmagic() Stack Overflow 636

The Vulnerability 638

Solaris vfs_getvfssw() Loadable Kernel Module Traversal Vulnerability 642

The sysfs() System Call 644

The mount() System Call 645

Conclusion 646

Chapter 26 Exploiting Unix Kernel Vulnerabilities 647

The exec_ibcs2_coff_prep_zmagic() Vulnerability 647

Calculating Offsets and Breakpoints 652

Overwriting the Return Address and Redirecting Execution 654

Locating the Process Descriptor (or the Proc Structure) 655

Kernel Mode Payload Creation 658

Returning Back from Kernel Payload 659

Getting root (uid=0) 665

Solaris vfs_getvfssw() Loadable Kernel Module Path Traversal Exploit 672

Crafting the Exploit 673

The Kernel Module to Load 674

Getting root (uid=0) 678

Conclusion 678

Chapter 27 Hacking the Windows Kernel 681

Windows Kernel Mode Flaws—An Increasingly Hunted Species 681

Introduction to the Windows Kernel 682

Common Kernel-Mode Programming Flaws 683

Stack Overflows 684

Heap Overflows 688

Insufficient Validation of User-Mode Addresses 688

Repurposing Attacks 689

Shared Object Attacks 689

Windows System Calls 690

Understanding System Calls 690

Attacking System Calls 692

Communicating with Device Drivers 693

I/O Control Code Components 693

Finding Flaws in IOCTL Handlers 694

Kernel-Mode Payloads 695

Elevating a User-Mode Process 696

Running an Arbitrary User-Mode Payload 699

Subverting Kernel Security 701

Installing a Rootkit 703

Essential Reading for Kernel Shellcoders 703

Conclusion 704

Index 705

Editorial Reviews

The Barnes & Noble Review
Every day, someone discovers a critical new security flaw in software you depend on. Quite often, those flaws appear first on Bugtraq, the world’s No. 1 vulnerability tracking maillist. Now, some of Bugtraq’s leading contributors show you exactly how they discover those holes -- and how to exploit them. Along the way, they reveal some never-before-published bugs -- including holes they claim are big enough to “take down the Internet.”

The authors cover platforms ranging from Windows to Solaris. They start with Linux on x86, where it’s easiest to understand the hacks because you can get inside the source code. Here, they introduce memory management and stack buffer overflows, format string overflows, and heap-based overflows. They also introduce basic shellcoding: how to write instructions that directly manipulate registers and a program’s functions. This can’t be done with high-level languages: We’re talking hexadecimal opcodes.

Next, the authors move on to Windows. Here, until recently, shellcoding was a true black art. Win32 doesn’t provide direct access to system calls, so more complex techniques are required. (As an example, the authors show how to bypass the vaunted stack protection Microsoft added to Windows 2003 Server.)

The authors then turn to the tools and techniques of vulnerability discovery: fault injection, fuzzing, source code auditing in C-based languages, instrumented investigation, tracing, and binary auditing of closed-source software. The book concludes with several especially sophisticated techniques, ranging from alternative payload strategies and database attacks to Unix kernel exploits. This stuff’s the real deal. Bill Camarda

Bill Camarda is a consultant, writer, and web/multimedia content developer. His 15 books include Special Edition Using Word 2003 and Upgrading & Fixing Networks for Dummies, Second Edition.

bn.com