Kafka Streams in Action: Real-time apps and microservices with the Kafka Streams API

by Bill Bejeck

Overview

Summary

Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort.

Foreword by Neha Narkhede, Cocreator of Apache Kafka

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the Technology

Not all stream-based applications require a dedicated processing cluster. The lightweight Kafka Streams library provides exactly the power and simplicity you need for message handling in microservices and real-time event processing. With the Kafka Streams API, you filter and transform data streams with just Kafka and your application.

About the Book

Kafka Streams in Action teaches you to implement stream processing within the Kafka platform. In this easy-to-follow book, you'll explore real-world examples to collect, transform, and aggregate data, work with multiple processors, and handle real-time events. You'll even dive into streaming SQL with KSQL! Practical to the very end, it finishes with testing and operational aspects, such as monitoring and debugging.

What's inside

• Using the KStreams API
• Filtering, transforming, and splitting data
• Working with the Processor API
• Integrating with external systems

About the Reader

Assumes some experience with distributed systems. No knowledge of Kafka or streaming applications required.

About the Author

Bill Bejeck is a Kafka Streams contributor and Confluent engineer with over 15 years of software development experience.

Table of Contents

PART 1 - GETTING STARTED WITH KAFKA STREAMS
1. Welcome to Kafka Streams
2. Kafka quicklyPART 2 - KAFKA STREAMS DEVELOPMENT
3. Developing Kafka Streams
4. Streams and state
5. The KTable API
6. The Processor APIPART 3 - ADMINISTERING KAFKA STREAMS
7. Monitoring and performance
8. Testing a Kafka Streams applicationPART 4 - ADVANCED CONCEPTS WITH KAFKA STREAMS
9. Advanced applications with Kafka StreamsAPPENDIXES
10. Appendix A - Additional configuration information
11. Appendix B - Exactly once semantics

Product Details

ISBN-13:	9781638356028
Publisher:	Manning
Publication date:	08/29/2018
Sold by:	SIMON & SCHUSTER
Format:	eBook
Pages:	280
File size:	6 MB

About the Author

Bill Bejeck is a Confluent engineer and a Kafka Streams contributor with over 15 years of software development experience. Bill is also a committer on the Apache KafkaR project.

Table of Contents

Foreword xi

Preface xiii

Acknowledgments xiv

About this book xv

About the author xix

About the cover illustration xx

Part 1 Getting started with Kafka Streams 1

1 Welcome to Kafka Streams 3

1.1 The big data movement, and how it changed the programming landscape 4

The genesis of big data 4

Important concepts from MapReduce 5

Batch processing is not enough 8

1.2 Introducing stream processing 8

When to use stream processing, and when not to use it 9

1.3 Handling a purchase transaction 10

Weighing the stream-processing option 10

Deconstructing the requirements into a graph 11

1.4 Changing perspective on a purchase transaction 12

Source node 12

Credit card masking node 12

Patterns node 13

Rewards node 13

Storage node 13

1.5 Kafka Streams as a graph of processing nodes 15

1.6 Applying Kafka Streams to the purchase transaction flow 16

Defining the source 16

The first processor: masking credit card numbers 17

The second processor: purchase patterns 18

The third processor: customer rewards 19

The fourth processor-writing purchase records 20

2 Kafka quickly 22

2.1 The data problem 23

2.2 Using Kafka to handle data 23

ZMart's original data platform 23

A Kafka sales transaction data hub 24

2.3 Kafka architecture 25

Kafka is a message broker 26

Kafka is a log 27

How logs work in Kafka 27

Kafka and partitions 28

Partitions group data by key 29

Writing a custom partitions 30

Specifying a custom partitioner 31

Determining the correct number of partitions 32

The distributed log 32

ZooKeeper: leaders, followers, and replication 33

Apache ZooKeeper 33

Electing a controller 34

Replication 34

Controller responsibilities 35

Log management 37

Deleting logs 37

Compacting logs 38

2.4 Sending messages with producers 40

Producer properties 42

Specifying partitions and timestamps 42

Specifying a partition 43

Timestamps in Kafka 43

2.5 Reading messages with consumers 44

Managing offsets 44

Automatic offset commits 46

Manual offset commits 46

Creating the consumer 47

Consumers and partitions 47

Rebalancing 47

Finer-grained consumer assignment 48

Consumer-example 48

2.6 Installing and running Kafka 49

Kafka local configuration 49

Running Kafka 50

Sending your first message 52

Part 2 Kafka Streams Development 55

3 Developing Kafka Streams 57

3.1 The Streams Processor API 58

3.2 Hello World for Kafka Streams 58

Creating the topology for the Yelling App 59

Kafka Stream configuration 63

Serde creation 63

3.3 Working with customer data 65

Constructing a topology 66

Creating a custom Serde 72

3.4 Interactive development 74

3.5 Next steps 76

New requirements 76

Writing records outside of Kafka 81

4 Streams and state 84

4.1 Thinking of events 85

Streams need state 86

4.2 Applying stateful operations to Kafka Streams 86

The transform Values processor 87

Stateful customer rewards 88

Initializing the value transformer 90

Mapping the Purchase object to a Reward Accumulator using state 90

Updating the rewards processor 94

4.3 Using state stores for lookups and previously seen data 96

Data locality 96

Failure recovery and fault tolerance 97

Using stale stores in Kafka Streams 98

Additional key/value store suppliers 99

StateStore fault tolerance 99

Configuring changelog topics 99

4.4 Joining streams for added insight 100

Data setup 102

Generating keys containing customer IDs to perform joins 103

Constructing the join 104

Other join options 109

4.5 Timestamps in Kafka Streams 110

Provided TimestampExtractor implementations 112

WallclockTimestampExtractor 113

Custom TimestampExtractor 114

Specifying a TimestampExtractor 115

5 The KTable API 117

5.1 The relationship between streams and tables 118

The record stream 118

Updates to records or the changelog 119

Event streams vs. update streams 122

5.2 Record updates and KTable configuration 123

Setting cache buffering size 124

Setting the commit interval 125

5.3 Aggregations and windowing operations 126

Aggregating share volume by industry 127

Windowing operations 132

Joining KStreams and KTables 139

GlobalKTables 140

Queryable state 143

6 The Processor API 145

6.1 The trade-offs of higher-level abstractions vs. more control 146

6.2 Working with sources, processors, and sinks to create a topology 146

Adding a source node 147

Adding a processor node 148

Adding a sink node 151

6.3 Digging deeper into the Processor API with a stock analysis processor 152

The stock-performance processor application 153

The process() method 157

The punctuator execution 158

6.4 The co-group processor 159

Building the co-grouping processor 161

6.5 Integrating the Processor API and the Kafka Streams API 170

Part 3 Administering Kafka Streams 173

7 Monitoring and performance 175

7.1 Basic Kafka monitoring 176

Measuring consumer and producer performance 176

Checking for consumer lag 178

Intercepting the producer and consumer 179

7.2 Application metrics 182

Metrics configuration 184

How to hook into the collected metrics 185

Using JMX 185

Viewing metrics 189

7.3 More Kafka Streams debugging techniques 191

Viewing a representation of the application 191

Getting notification on various states of the application 192

Using the StateListener 193

State restore listener 195

Uncaught exception handler 198

8 Testing a Kafka Streams application 199

8.1 Testing a topology 201

Building the test 202

Testing a stale store in the topology 204

Testing processors and transformers 205

8.2 Integration testing 208

Building an integration test 209

Part 4 Advanced Concepts with Kafka Streams 215

9 Advanced applications with Kafka Streams 217

9.1 Integrating Kafka with other data sources 218

Using Kafka Connect to integrate data 219

Setting up Kafka Connect 219

Transforming data 222

9.2 Kicking your database to the curb 226

How interactive queries work 228

Distributing state stores 229

Setting up and discovering a distributed state store 230

Coding interactive queries 232

Inside the query server 234

9.3 KSQL 237

KSQL streams and tables 238

KSQL architecture 238

Installing and running KSQL 240

Creating a KSQL stream 241

Writing a KSQL query 242

Creating a KSQL table 243

Configuring KSQL 244

Appendix A Additional configuration information 245

Appendix B Exactly once semantics 251

Index 253