Monitoring with Graphite: Tracking Dynamic Host and Application Metrics at Scale

by Jason Dixon

Overview

Graphite has become one of the most powerful monitoring tools available today, due to its ease of use, rapid graph prototyping abilities, and a friendly rendering API. With this practical guide, system administrators and engineers will learn how to use this open source tool to track operational data you need to monitor your systems, as well as application-level metrics for profiling your services.

Author Jason Dixon, member of the Graphite project, provides a thorough introduction of Graphite from the basics to the skills and tools you need for troubleshooting and scaling out its software components. If you want to learn more about monitoring systems, services, or applications, this is the book you need.

• Get an introduction to monitoring, including important concepts and terminology
• Examine the features and functionality of key Graphite components, including Carbon and Whisper
• Learn the typical user workflow necessary to create a basic line chart
• Build complex charts with chained functions and multiple axes that interact directly with the rendering API
• Understand how to use the native Graphite dashboard, as well as the more popular third-party dashboards
• Master the art of scaling and troubleshooting high-performance or highly available Graphite clusters

Product Details

ISBN-13:	9781491916438
Publisher:	O'Reilly Media, Incorporated
Publication date:	03/31/2017
Pages:	287
Product dimensions:	7.00(w) x 9.10(h) x 0.70(d)

About the Author

Jason Dixon is a leading thinker on metrics, monitoring, and infrastructure. He is the creator of the worldwide Monitorama conference series and a member of the Graphite project. He works as the Director of
Product Solutions at Librato, specializing in data and service integrations. He has held engineering and managing roles at leading companies including Heroku, GitHub, and Dyn. He lives in Maryland with his family.

Table of Contents

Preface ix

1 What is Graphite? 1

What Is Time-Series Data? 2

Time-Series Databases 3

Storage Considerations 3

Prioritizing Workloads 4

What Is the History of Graphite? 5

What Makes Graphite Unique? 6

Simple Metrics Format 6

Graphing API 7

Rapid Prototyping 8

Rich Statistical Library 9

Chained Functions 9

Case Studies: Who Uses Graphite in Production? 11

Booking.com 11

GitHub 12

Etsy 12

Electronic Arts 13

Why Should I Use Graphite? 13

2 Monitoring Conventions 15

Three Tenets of Monitoring 15

Fault Detection 16

Alerting 17

Capacity Planning 18

Rethinking the Poll/Pull Model 18

Pull Model 19

Push Model 20

Where Does Graphite Fit into the Picture? 21

Composable Monitoring Systems 22

Telemetry 24

Metrics Router 26

Aggregation 26

State Engine 27

Notification Routers 28

Storage Engine 28

Visualization 29

Conclusion 29

3 Graphite Components: The Moving Parts 31

Carbon 31

Carbon-cache 32

Carbon-relay 32

Carbon-aggregator 35

Filtering Metrics 35

Internal Statistics 36

Network Security Considerations 37

Whisper 38

How Do Whisper Files Get Created? 38

Retention Policies and Archives 38

The Laws of Whisper Archives 39

Calculating Whisper File Sizes 40

Deconstructing a Whisper File 41

Which Archive Handles My Query? 41

Aggregation Methods 42

XFilesFactor 44

Planning Your Namespaces 44

Performance Considerations 45

Graphite-Web 46

Django Framework 46

Web Server 47

Database 47

Memcached 48

Events 49

Storage Backends 50

Putting It All Together 52

Basic Setup 52

Vertical Scaling 53

Horizontal Scaling 54

Multisite Replication 56

A Final Thought 58

4 Building Your First Graphite Server 59

Quick Start with Synthesize 60

Where Does Graphite Store All My Files? 61

Are Packages Available for My Distro? 62

What Installation Methods Are Available? 62

Should I Use Virtualeny? 63

Using sudo Effectively 63

Dependencies 64

Installing from Source 65

Preparing Your Web Database 66

Configuring Carbon 68

Carbon.conf 68

Storage-schemas.conf 71

Storage-aggregation.conf 72

Some Final Preparations 73

Starting Your Carbon Daemons 73

Configuring Graphite-Web 74

Local_settiugs.py 74

Setting Up Apache 75

Verifying Your Graphite Installation 76

Carbon Statistics 77

Feeding New Data to Carbon 78

Building Your First Graph 78

5 The Graphite User Interface 83

Finding Metrics 84

Navigating the Tree 84

Using the Search Feature 86

Working Smarter with the Auto-Completer 88

Wildcards 88

The Graphite Composer Window 89

The Embedded Chart 90

The Toolbar 92

Selecting Recent Data 93

Refreshing the Graph 95

Selecting a Date Range 96

Exporting a Short URL 97

Loading a Graph from URL 98

Saving to My Graphs 99

Deleting from My Graphs 101

The Graph Options Menu 102

Adding a Graph Title 103

Overriding the Graph Legend 104

Toggling Axes and the Grid 105

Applying a Graph Template 106

Line Chart Modes 106

Area and Stacked Graphs 110

Tweaking the Y-Axis 114

The Graph Data Dialog 122

What Are Targets Anyway? 122

Building a Carbon Performance Graph 123

Sharing Your Work 130

6 The Render API 133

Working with Functions 134

Starting with the Basics 135

Math and Statistical Transforms 140

Filtering and Sorting 143

Grouping on Wildcards 147

Data Smoothing and Forecasting 148

Adjusting Metric Labels 153

Alternate Output Formats 158

7 Dashboards 163

Why Do I Need a Dashboard? 163

Graphite Dashboard 164

Third-Party Dashboards 166

Grafana 166

Tasseo 169

Dusk 171

Do-It-Yourself 173

Dashing 173

Rickshaw and D3.js 175

Conclusion 181

8 Troubleshooting Graphite Performance 183

First, the Basics 184

The Troubleshooting Toolbelt 185

Generating Metrics and Benchmarking 185

CPU Utilization 187

Disk Performance (I/O) 188

Networking 191

Inspecting Metrics 195

Configuration Settings 199

Carbon 199

Graphite-Web 202

Logging 203

Carbon 204

Graphite-Web 210

Kernel messages 216

Failure Scenarios 216

The Full Disk 217

CPU Saturation 219

Rendering Problems 221

Taking It to the Next Level 224

9 Scaling Graphite 225

What Makes It "Hard" to Scale Graphite? 225

Peter's Graphite Story 226

The Beginning 226

The Pains of Popularity 227

Clearing the Next Hurdle 229

Try, Try Again 230

Maximizing Resources 232

Avoiding Outages 235

Shared Web Database 242

Scaling in Both Directions 249

Some Final Thoughts 257

Summary 257

A Carbon and Graphite-Web Internal Statistics 259

Index 265