Software Telemetry: Reliable logging and monitoring

by Jamie Riedesel

Overview

Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems.

Summary
In Software Telemetry you will learn how to:

Manage toxic telemetry and confidential records
Master multi-tenant techniques and transformation processes
Update to improve the statistical validity of your metrics and dashboards
Make software telemetry emissions easier to parse
Build easily-auditable logging systems
Prevent and handle accidental data leaks
Maintain processes for legal compliance
Justify increased spend on telemetry software

Software Telemetry teaches you best practices for operating and updating telemetry systems. These vital systems trace, log, and monitor infrastructure by observing and analyzing the events generated by the system. This practical guide is filled with techniques you can apply to any size of organization, with troubleshooting techniques for every eventuality, and methods to ensure your compliance with standards like GDPR.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Take advantage of the data generated by your IT infrastructure! Telemetry systems provide feedback on what’s happening inside your data center and applications, so you can efficiently monitor, maintain, and audit them. This practical book guides you through instrumenting your systems, setting up centralized logging, doing distributed tracing, and other invaluable telemetry techniques.

About the book
Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. Manage the pillars of observability—logs, metrics, and traces—in an end-to-end telemetry system that integrates with your existing infrastructure. You’ll discover how software telemetry benefits both small startups and legacy enterprises. And at a time when data audits are increasingly common, you’ll appreciate the thorough coverage of legal compliance processes, so there’s no reason to panic when a discovery request arrives.

What's inside

Multi-tenant techniques and transformation processes
Toxic telemetry and confidential records
Updates to improve the statistical validity of your metrics and dashboards
Revisions that make software telemetry emissions easier to parse

About the reader
For software developers and infrastructure engineers supporting and building telemetry systems.

About the author
Jamie Riedesel is a staff engineer at Dropbox with over twenty years of experience in IT.

Table of Contents
1 Introduction
PART 1 TELEMETRY SYSTEM ARCHITECTURE
2 The Emitting stage: Creating and submitting telemetry
3 The Shipping stage: Moving and storing telemetry
4 The Shipping stage: Unifying diverse telemetry formats
5 The Presentation stage: Displaying telemetry
6 Marking up and enriching telemetry
7 Handling multitenancy
PART 2 USE CASES REVISITED: APPLYING ARCHITECTURE CONCEPTS
8 Growing cloud-based startup
9 Nonsoftware business
10 Long-established business IT
PART 3 TECHNIQUES FOR HANDLING TELEMETRY
11 Optimizing for regular expressions at scale
12 Standardized logging and event formats
13 Using more nonfile emitting techniques
14 Managing cardinality in telemetry
15 Ensuring telemetry integrity
16 Redacting and reprocessing telemetry
17 Building policies for telemetry retention and aggregation
18 Surviving legal processes

Product Details

ISBN-13:	9781617298141
Publisher:	Manning
Publication date:	08/31/2021
Pages:	560
Product dimensions:	7.38(w) x 9.25(h) x 1.20(d)

About the Author

Jamie Riedesel is a staff engineer at Dropbox. She has over twenty years of experience in IT, working in government, education, legacy companies, and startups. She has specialized in DevOps for the past decade, running distributed systems in public clouds, getting over workplace trauma, and designing software telemetry architectures.

Table of Contents

Preface xiii

Acknowledgments xvi

About this book xviii

About the author xxiii

About the cover illustration xxiv

1 Introduction 1

1.1 Defining the styles of telemetry 4

Defining centralized logging 4

Defining metrics 6

Defining distributed tracing 8

Defining STEM 10

1.2 How telemetry is consumed by different teams 11

Telemetry use by Operations, DevOps, and SEE teams 11

Telemetry use by Security and Compliance teams 12

Telemetry use by Software Engineering and SRE teams 13

Telemetry use by Customer Support teams 13

Telemetry use by business intelligence 14

1.3 Challenges facing telemetry systems 15

Chronic underinvestment harms decision-making 15

Diverse needs resist standardization 17

Information spills and cleaning them up to avoid legal problems 18

Court orders break your assumptions 19

1.4 What you will learn 20

Part 1 Telemetry system architecture 23

2 The Emitting stage: Creating and submitting telemetry 27

2.1 Emitting from production code 29

Emitting telemetry into a log file 31

Emitting telemetry into the system log 33

Emitting telemetry into standard output 37

Formatting telemetry for emissions 40

2.2 Emitting from hardware 43

Explaining SNMP 43

Ingesting telemetry from a Cisco ASA firewall 45

2.3 Emitting from as-a-Service systems 47

Emitting events from SaaS systems 47

Emitting events from IaaS systems 49

3 The Shipping stage: Moving and storing telemetry 52

3.1 Emitter/shipper functions, telemetry from production code 54

Shipping directly into storage 54

Shipping through queues and streams 57

Shipping to SaaS systems 67

3.2 Shipping between SaaS systems 69

3.3 Tipping points in Shipping-stage architecture 71

4 The Shipping stage: Unifying diverse telemetry formats 74

4.1 Shipping locally-emitted telemetry 75

Shipping telemetry from a log file 76

Shipping telemetry from the system logger 79

Shipping telemetry from standard output 81

4.2 Unifying diverse emitting formats 83

Encoding telemetry into strings 84

Picking a shipping format 89

Converting Syslog to JSON or other object-encoding formats 100

Designing with cardinality in mind 104

5 The Presentation stage: Displaying telemetry 107

5.1 Displaying telemetry in metrics systems 109

Making pretty pictures with telemetry 110

Feeding the graphs with aggregation functions 112

Using aggregations with pdf_pages 114

5.2 Displaying telemetry in centralized logging systems 118

Selecting needed features in a display system for centralized logging 119

Demonstrating centralized logging display 121

5.3 Displaying telemetry in security systems 127

5.4 Displaying telemetry distributed tracing systems 131

5.5 Displaying telemetry in large organizations 135

6 Marking up and enriching telemetry 138

6.1 Markup in the Emitting stage 141

6.2 Markup and enrichment in the Shipping stage 146

Applying context-related telemetry in the Shipping stage 147

Extracting and enriching telemetry in-flight 150

Converting field types during the Shipping stage 156

6.3 Enrichment in the Presentation stage 162

6.4 How telemetry style affects markup and enrichment 165

Markup and enrichment with centralized logging 166

Markup and enrichment with STEM systems 167

Markup and enrichment with metrics 169

Markup and enrichment with distributed tracing systems 170

7 Handling multitenancy 174

7.1 How multitenant architectures come about 175

Evolving multitenancy in an early-stage startup 175

Evolving multitenancy in a culture of free sharing 176

Evolving multitenancy in a culture of strong separation 178

7.2 Designing multitenant telemetry systems 180

Multitenancy in the Shipping stage 181

Multitenancy in the Presentation stage 189

Part 2 Use cases revisited: Applying Architecture Concepts 193

8 Growing cloud-based startup 195

8.1 Telemetry at the small-company stage 197

Describing the small company's telemetry system 198

Analyzing the small company's telemetry system 199

8.2 Telemetry at the medium-size company stage 201

Describing the medium-size company's telemetry system 201

Analyzing the medium-size company's telemetry system 204

8.3 Telemetty at the large-company stage 206

Describing the large company's telemetry system 209

Analyzing the large company's telemetry system 210

8.4 Telemetry at the enterprise stage 213

8.5 Looking back at all this growth 223

9 Nonsoftware business 226

9.1 Telemetry use in small organizations 227

9.2 Telemetry use in medium-size organizations 230

9.3 Telemetry use in large organizations 233

9.4 Telemetry use in enterprise organizations 239

10 Long-established business IT 248

10.1 Telemetry use in medium-size organizations 250

Telemetry use in office IT 251

Telemetry use in production systems 254

10.2 Telemetry use in large organizations 255

10.3 Telemetry use in global organizations 262

Telemetry use in the Booking and Passenger Manifest department 265

Telemetry use in the Loyalty Programs department 269

Part 3 Techniques for handling telemetry 275

11 Optimizing for regular expressions at scale 277

11.1 Anchoring expressions for speed 279

11.2 Building expressions to fail fast 285

11.3 Digging into the Cisco ASA firewall telemetry 290

11.4 Refining emissions to speed regular-expression performance 297

11.5 Additional regular-expression resources 305

12 Standardized logging and event formats 307

12.1 Implementing structured logging in your code 309

12.2 Implementing standards in your code 314

12.3 Implementing standards in the Shipping stage 325

13 Using more nonfile emitting techniques 335

13.1 Designing for socket-and datagram-based emitters 336

13.2 Emitting and shipping for container- and serverless-based code 344

Emitting and shipping from containerd-based code 345

Emitting and shipping from serverless-based code 347

13.3 Encrypting UDP-based telemetry 350

14 Managing cardinality in telemetry 357

14.1 Identifying cardinality problems 359

Cardinality in time-series databases 360

Cardinality in logging databases 364

14.2 Lowering the cost of cardinality 366

Use logging standards to contain cardinality 366

Using storage-side methods to tame cardinality 373

Make cardinality someone else's problem 379

15 Ensuring telemetry integrity 384

15.1 Getting telemetry out of reach of an attacker 386

Move telemetry too fast to catch 386

Use ACLs to enforce write-only telemetry 389

Durable telemetry when using SaaS providers 393

15.2 Making telemetry harder to mess with 394

Using access control requirements to defend against attacks 395

Ensuring configuration integrity in your telemetry systems 397

Making changes obvious 400

16 Redacting and reprocessing telemetry 411

16.1 Identifying toxic data and where it comes from 412

16.2 Redacting toxic information spills 416

16.3 Reprocessing telemetry to support upgrades 423

16.4 Isolating toxic data to reduce cleanup costs 429

17 Building policies for telemetry retention and aggregation 439

17.1 Creating a retention policy 440

Building a policy for centralized, logging 443

Building a policy for metrics 445

Building a policy for distributed tracing 446

Building a policy for SIEM systems 447

17.2 Creating an aggregation policy 448

17.3 Using sampling to reduce costs and increase retention 457

18 Surviving legal processes 463

18.1 Defining the eDiscovery process 466

18.2 Dealing with records-retention requests 469

Examining an ELK-based centralized logging system 471

Examining a Sumo Logic-based centralized logging system 474

18.3 Dealing with document-production requests 477

Telemetry in the collection phase 478

Telemetry in the review phase 480

Telemetry in the production phase 481

18.4 Working with lawyers 482

Appendix A Telemetry storage systems 487

Appendix B Recommendation checklist reference 499

Appendix C Exercise answers 520

Index 525