Linux Observability with BPF: Advanced Programming for Performance Analysis and Networking

by David Calavera, Lorenzo Fontana

Overview

Build your expertise in the BPF virtual machine in the Linux kernel with this practical guide for systems engineers. You’ll not only dive into the BPF program lifecycle but also learn to write applications that observe and modify the kernel’s behavior; inject code to monitor, trace, and securely observe events in the kernel; and more.

Authors David Calavera and Lorenzo Fontana help you harness the power of BPF to make any computing system more observable. Familiarize yourself with the essential concepts you’ll use on a day-to-day basis and augment your knowledge about performance optimization, networking, and security. Then see how it all comes together with code examples in C, Go, and Python.

• Write applications that use BPF to observe and modify the Linux kernel’s behavior on demand
• Inject code to monitor, trace, and observe events in the kernel in a secure way—no need to recompile the kernel or reboot the system
• Explore code examples in C, Go, and Python
• Gain a more thorough understanding of the BPF program lifecycle

Product Details

ISBN-13:	9781492050209
Publisher:	O'Reilly Media, Incorporated
Publication date:	12/03/2019
Pages:	177
Product dimensions:	6.90(w) x 9.10(h) x 0.50(d)

About the Author

David Calavera works as CTO at Netlify. He's served as the maintainer of Docker and contributor to Runc, Go, and BCC tools as well as other open source projects. He's known for his work on the Docker projects, building and fostering the Docker plugin ecosystem. David has a strong fondness for flame graphs and performance optimizations.

Lorenzo Fontana is on the Open Source Team at Sysdig where he primarily works on Falco, a Cloud Native Computing Foundation project that does container runtime security and anomaly detection through a kernel module and eBPF. He's passionate about distributed systems, software-defined networking, the Linux kernel, and performance analysis.

Table of Contents

Foreword vii

Preface ix

1 Introduction 1

BPF's History 3

Architecture 4

Conclusion 5

2 Running Your First BPF Programs 7

Writing BPF Programs 8

BPF Program Types 10

Socket Filter Programs 11

Kprobe Programs 11

Tracepoint Programs 12

XDP Programs 12

Perf Event Programs 13

Cgroup Socket Programs 13

Cgroup Open Socket Programs 13

Socket Option Programs 13

Socket Map Programs 14

Cgroup Device Programs 14

Socket Message Delivery Programs 15

Raw Tracepoint Programs 15

Cgroup Socket Address Programs 15

Socket Reuseport Programs 15

Flow Dissection Programs 16

Other BPF Programs 16

The BPF Verifier 17

BPF Type Format 19

BPF Tail Calls 19

Conclusion 20

3 BPF Maps 21

Creating BPF Maps 21

ELF Conventions to Create BPF Maps 22

Working with BFP Maps 23

Updating Elements in a BPF Map 24

Reading Elements from a BPF Map 26

Removing an Element from a BPF Map 27

Iterating Over Elements in a BPF Map 28

Looking Up and Deleting Elements 30

Concurrent Access to Map Elements 31

Types of BPF Maps 33

Hash-Table Maps 33

Array Maps 34

Program Array Maps 34

Perf Events Array Maps 36

Per-CPU Hash Maps 37

Per-CPU Array Maps 37

Stack Trace Maps 37

Cgroup Array Maps 37

LRU Hash and Per-CPU Hash Maps 38

LPM Trie Maps 38

Array of Maps and Hash of Maps 39

Device Map Maps 39

CPU Map Maps 39

Open Socket Maps 40

Socket Array and Hash Maps 40

Cgroup Storage and Per-CPU Storage Maps 40

Reuseport Socket Maps 41

Queue Maps 41

Stack Maps 42

The BPF Virtual Filesystem 43

Conclusion 46

4 Tracing with BPF 47

Probes 48

Kernel Probes 49

Tracepoints 51

User-Space Probes 53

User Statically Defined Tracepoints 56

Visualizing Tracing Data 61

Flame Graphs 61

Histograms 67

Perf Events 69

Conclusion 71

5 BPF Utilities 73

BPFTool 73

Installation 74

Feature Display 74

Inspecting BPF Programs 75

Inspecting BPF Maps 78

Inspecting Programs Attached to Specific Interfaces 80

Loading Commands in Batch Mode 81

Displaying BTF Information 82

BPFTrace 83

Installation 83

Language Reference 83

Filtering 85

Dynamic Mapping 85

Kubectl-trace 86

Installation 87

Inspecting Kubernetes Nodes 87

eBPF Exporter 88

Installation 88

Exporting Metrics from BPF 88

Conclusion 90

6 Linux Networking and BPF 91

BPF and Packet Filtering 92

tcpdump and BPF Expressions 92

Packet Filtering for Raw Sockets 97

BPF-Based Traffic Control Classifier 103

Terminology 103

Traffic Control Classifier Program Using cls_bpf 106

Differences Between Traffic Control and XDP 112

Conclusion 112

7 Express Data Path 113

XDP Programs Overview 114

Operation Modes 114

The Packet Processor 116

XDP and iproute2 as a Loader 119

XDP and BCC 124

Testing XDP Programs 127

XDP Testing Using the Python Unit Testing Framework 128

XDP Use Cases 133

Monitoring 133

DDoS Mitigation 133

Load Balancing 133

Firewalling 134

Conclusion 134

8 Linux Kernel Security, Capabilities, and Seccomp 135

Capabilities 135

Seccomp 139

Seccomp Errors 140

Seccomp BPF Filter Example 142

BPF LSM Hooks 146

Conclusion 147

9 Real-World Use Cases 149

Sysdig eBPF God Mode 149

Flowmill 152

Index 155