Logging in Action: With Fluentd, Kubernetes and more

by Phil Wilkins

Overview

Make log processing a real asset to your organization with powerful and free open source tools.

In Logging in Action you will learn how to:

Deploy Fluentd and Fluent Bit into traditional on-premises, IoT, hybrid, cloud, and multi-cloud environments, both small and hyperscaled
Configure Fluentd and Fluent Bit to solve common log management problems
Use Fluentd within Kubernetes and Docker services
Connect a custom log source or destination with Fluentd’s extensible plugin framework
Logging best practices and common pitfalls

Logging in Action is a guide to optimize and organize logging using the CNCF Fluentd and Fluent Bit projects. You’ll use the powerful log management tool Fluentd to solve common log management, and learn how proper log management can improve performance and make management of software and infrastructure solutions easier. Through useful examples like sending log-driven events to Slack, you’ll get hands-on experience applying structure to your unstructured data.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Don’t fly blind! An effective logging system can help you see and correct problems before they cripple your software. With the Fluentd log management tool, it’s a snap to monitor the behavior and health of your software and infrastructure in real time. Designed to collect and process log data from multiple sources using the industry-standard JSON format, Fluentd delivers a truly unified logging layer across all your systems.

About the book
Logging in Action teaches you to record and analyze application and infrastructure data using Fluentd. Using clear, relevant examples, it shows you exactly how to transform raw system data into a unified stream of actionable information. You’ll discover how logging configuration impacts the way your system functions and set up Fluentd to handle data from legacy IT environments, local data centers, and massive Kubernetes-driven distributed systems. You’ll even learn how to implement complex log parsing with RegEx and output events to MongoDB and Slack.

What's inside

Capture log events from a wide range of systems and software, including Kubernetes and Docker
Connect to custom log sources and destinations
Employ Fluentd’s extensible plugin framework
Create a custom plugin for niche problems

About the reader
For developers, architects, and operations professionals familiar with the basics of monitoring and logging.

About the author
Phil Wilkins has spent over 30 years in the software industry. Has worked for small startups through to international brands.

Table of Contents
PART 1 FROM ZERO TO “HELLO WORLD”
1 Introduction to Fluentd
2 Concepts, architecture, and deployment of Fluentd
PART 2 FLUENTD IN DEPTH
3 Using Fluentd to capture log events
4 Using Fluentd to output log events
5 Routing log events
6 Filtering and extrapolation
PART 3 BEYOND THE BASICS
7 Performance and scaling
8 Driving logs with Docker and Kubernetes
9 Creating custom plugins
PART 4 GOOD LOGGING PRACTICES AND FRAMEWORKS TO MAXIMIZE LOG VALUE
10 Logging best practices
11 Logging frameworks

Product Details

ISBN-13:	9781617298356
Publisher:	Manning
Publication date:	03/29/2022
Pages:	392
Product dimensions:	7.38(w) x 9.25(h) x 0.80(d)

About the Author

Phil Wilkins has spent over 30 years in the software industry. Has worked for small startups through to international brands.

Table of Contents

Foreword xiii

Preface xvi

Acknowledgments xviii

About this book xx

About the author xxv

About the cover illustration xxvi

Part 1 From Zero to "Hello World" 1

1 Introduction to Fluentd 3

1.1 Elevator pitch for Fluentd 4

What is a log event? 4

Fluentd compared to middleware 5

1.2 Why do we produce logs? 6

1.3 Evolving ideas 9

Four golden signals 9

Three pillars of observability 10

1.4 Log unification 12

Unifying logs vs. log analytics 14

1.5 Software stacks 14

ELK stack 15

Comparing Fluentd and Logstash 16

The relationship between Fluentd and Fluent Bit 17

The relationship between Logstash and Beats 18

1.6 Log routing as a vehicle for security 18

1.7 Log event life cycle 19

1.8 Evolution of Fluentd 20

Treasure Data 20

CNCF 22

Relationship to major cloud venders PaaS/IaaS 22

1.9 Where can Fluentd and Fluent Bit be used? 23

Platform constraints 23

1.10 Fluentd UI-based editing 24

1.11 Plugins 26

1.12 How Fluentd can be used to make operational tasks easier 27

Actionable log events 27

Making logs more meaningful 27

Polyglot environments 28

Multiple targets 28

Controlling log data costs 28

Logs to metrics 28

Rapid operational consolidation 29

2 Concepts, architecture, and deployment of Fluentd 30

2.1 Architecture and core concepts 30

The makeup of a log event 31

Handling time 32

Architecture of Fluentd 33

Fluent configuration execution order 35

Directives 36

Putting timing requirements into action 38

2.2 Deployment of Fluentd 38

Deploying Fluentd for the book's examples 39

Deployment considerations for Fluentd 39

Fluentd minimum footprint 40

Simple deployment of Ruby 40

Simple deployment of Fluentd 41

Deploying a log generator 44

Installing Postman 46

2.3 Bringing Fluentd to life with "Hello World" 46

"Hello World" scenario 46

"Hello World" configuration 47

Starting Fluentd 47

2.4 "Hello World" with Fluent Bit 50

Starting Bit 50

Alternate Fluent Bit startup options 52

Fluent Bit configuration file comparison 53

Fluent Bit configuration file in detail 54

Putting the dummy plugin into action 55

2.5 Fluentd deployment with Kubernetes and containers 55

Fluentd Daemon Set 56

Dockerized Fluentd 58

2.6 Using Fluentd UI 59

Installing Fluentd with UI 59

Part 2 Fluentd in Depth 65

3 Using Fluentd to capture log events 67

3.1 Dry running to check a configuration 68

Putting validating Fluentd configuration into action 69

3.2 Reading log files 69

Putting the adaption of a Fluentd configuration to Fluent Bit into action 70

Rereading and resuming reading of log files 71

Configuration considerations for tracking position 71

Wildcards in the path attribute 72

Expressing time 73

Controlling the impact of wildcards in filenames 73

Replacing wildcards with delimited lists in action 75

Handling log rotation 75

3.3 Self-monitoring 78

HTTP interface check 78

3.4 Imposing structure on log events 81

Standard parsers 81

Third-party parsers 86

Applying a Regex parser to a complex log 86

Putting parser configuration into action 91

4 Using Fluentd to output log events 93

4.1 File output plugin 94

Basic file output 94

Basics of buffering 95

Chunks and Controlling Buffering 98

Retry and backoff 102

Putting configuring buffering size settings into action 103

4.2 Output formatting options 103

Out file 104

Json 104

Itsv 104

Csv 104

Msgpack 105

Applying formatters 105

Putting JSON formatter configuration into action 106

4.3 Sending log events to MongoDB 106

Deploying MongoDB Fluentd plugin 107

Configuring the Mon go output plugin for Fluentd 108

Putting MongoDB connection configuration strings into action 111

4.4 Actionable log events 111

Actionable log events through service invocation 112

Actionable through user interaction tools 112

4.5 Slack to demonstrate the social output 113

Handling tokens and credentials more carefully 115

Externalizing Slack configuration attributes in action 116

4.6 The right tool for the right job 117

5 Routing log events 119

5.1 Reaching multiple outputs by copying 120

Copy by reference or by value 122

Handling errors when copying 124

5.2 Configuration reuse and extension through inclusion 126

Place holding with null output 129

Putting inclusions with a MongoDB output into action 129

5.3 Injecting context into log events 130

Extraction of values 131

5.4 Tag-based routing 132

Using exec output plugin 135

Putting tag naming conventions into action 135

Putting dynamic tagging with extract into action 136

5.5 Tag plugins 136

5.6 Labels: Taking tags to a new level 137

Using a stdout filter to see what is happening 137

Illustrating label and tag routing 137

Connecting pipelines 139

Label sequencing 141

Special labels 143

Putting a common pipeline into action 144

6 Filtering and extrapolation 145

6.1 Application of Filters 146

All is well events do not need to be distributed 146

Spotting the needle in a haystack 147

False urgency 147

Releveling 147

Unimplemented housekeeping 148

6.2 Why change log events? 148

Easier to process meaning downstream 148

Add context 148

Record when we have reacted to a log event 148

Data redaction/masking 149

6.3 Applying filters and parsers 150

Filter plugins 150

Applying grep filters 150

Changing log events with the record_transformer plugin 153

Filter parser vs. record transformer 156

6.4 Demonstrating change impact with stdout in action 157

A solution demonstrating change impact with stdout in action 157

6.5 Extract to set key values 157

6.6 Deriving new data values with the record_transformer 159

Putting the incorporation of calculations into a log event transformation into action 160

6.7 Generating simple Fluentd metrics 160

Putting log event counting into action 163

Part 3 Beyond the Basics 165

7 Performance and scaling 167

7.1 Threading and processes to scale with workers 168

Seeing workers in action 169

Worker constraints 171

Controlling output plugin threads 172

Memory management optimization 172

7.2 Scaling and moving workloads 173

Fan-in/log aggregation and consolidation 173

Fan-out and workload distribution 179

High availability 184

Putting a high-availability comparison into action 185

7.3 Fluentd scaling in containers vs. native and virtual environments 185

Kubernetes worker node configuration 186

Per-cluster configuration 186

Container as visualization 187

Sidecar pattern 188

Options comparison 189

7.4 Securing traffic between Fluentd nodes 190

TLS configuration 190

TLS not just for encryption 191

Certificate and private key storage 191

Security is more than certificates 192

7.5 Credentials management 192

Simple credentials use case 193

Putting certification into action 195

8 Driving logs with Docker and Kubernetes 197

8.1 Fluentd out of the box from Docker Hub 198

Official Docker images 198

Docker log drivers 199

Getting set up for Docker log drivers 199

8.2 Using Docker log drivers 200

Docker drivers via the command line 200

A quick check of network connections 201

Running Docker command line 202

Switching to driver configuration through a configuration file 204

8.3 Kubernetes components logging and the use of Fluentd 207

Kubernetes components and structured logging 208

Kubernetes default log retention and log rotation 208

Kubectl with logging 209

8.4 Demonstrating logging with Kubernetes 209

Kubernetes setup 210

Creating logs to capture 212

Understanding how Fluentd DaemonSets are put together 215

8.5 Getting a peek at host logs 219

8.6 Configuring a Kubernetes logging DaemonSet 222

Getting the Fluentd configuration ready to be used 222

Creating our Kubernetes deployment configuration 224

Putting the implementation of a Fluentd for Kubernetes into action 226

Deploying to minikube 227

Tidying up 228

8.7 Kubernetes configuration in action 228

Answer 228

8.8 More Kubernetes monitoring and logging to watch for 228

Node monitoring 229

Termination messages 229

9 Creating custom plugins 231

9.1 Plugin source code 232

9.2 What is Redis, and why build a plugin with the Redis list capability? 232

Redis list over RedisTimeSeries 233

9.3 Illustrating our objective using Redis CLI 233

9.4 Preparing for development 234

9.5 Plugin frameworks 235

Creating the skeleton plugin 235

Plugin life cycle 237

9.6 Implementing the plugin core 238

How configuration attributes work 238

Starting up and shutting down 240

Getting the plugin to work with our Fluentd installation 241

Putting additional configuration validation into action 242

Implementing the Redis output logic 243

Putting the testing of synchronous output into action 244

9.7 Implementing the Redis input plugin 244

Testing input and output plugin execution 246

9.8 Extending output with buffering 247

Improving our scenario by putting maintainability into action 250

9.9 Unit testing 250

9.10 Putting the development of unit tests into action 251

Answer 251

9.11 Package and deployment 252

Documentation 252

Complete metadata aha manifest 253

Building the gem package 253

Rerun without the plugin paths 253

9.12 Extending to be an enterprise-class solution 254

Part 4 Good logging practices and frameworks to maximize log value 257

10 Logging best practices 259

10.1 Audit events vs. log events 260

10.2 Log levels and severities 260

Trace 261

Debug 261

Info(rmation) 262

Warn(ing) 262

Error 263

Fatal 263

Extending or creating your own log levels 263

10.3 Clear language 264

10.4 Human and machine-readable 265

10.5 Context is key 265

Context: What 266

Context: When 266

Context: where 266

Context: Why 267

Context: Who 268

A practical checklist for capturing context 268

10.6 Error codes 269

Using standard errors 271

Codes can be far more than errors 271

10.7 Too little logging or too much? 271

What qualifies as sensitive? 272

GDPR is only the start 274

10.8 Log structure and format 275

Putting making log entries ready for application shipping into action 276

10.9 Use frameworks if you can 277

10.10 Development practices 277

Rethrowing exceptions 278

Using standard exceptions and error structures 278

String construction as a reason not to log 279

11 Logging frameworks 281

11.1 Value of logging frameworks 282

11.2 Typical structure of a logging framework 283

Logger context 283

Appender 284

Logger 284

Filter 285

Formatter 285

Configuration 285

Logger config 285

11.3 Appender structures 285

11.4 Logging framework landscape 286

11.5 Choosing a framework 287

Putting optimizing application logging into action 288

11.6 Fluentd's own logging and appenders 288

11.7 Illustrations of an application logging directly to Fluentd 290

Python with logging framework: Using the Fluentd library 290

Invoking Fluentd appender directly 293

Illustration with only Python's logging 294

Illustration without Python's logging or Fluentd library 295

Porting the Fluentd calls to another language into action 296

Using generic appenders: The takeaways 297

Appendix A Installation of additional tools and services 299

Appendix B Processing times and dates, regular expressions, and other configuration values 315

Appendix C Plugins summary 321

Appendix D Real-world use case 327

Appendix E Useful resources 337

Index 353