Apache Pulsar in Action

by David Kjerrumgaard

Overview

Deliver lightning fast and reliable messaging for your distributed applications with the flexible and resilient Apache Pulsar platform.

In Apache Pulsar in Action you will learn how to:

Publish from Apache Pulsar into third-party data repositories and platforms
Design and develop Apache Pulsar functions
Perform interactive SQL queries against data stored in Apache Pulsar

Apache Pulsar in Action is a comprehensive and practical guide to building high-traffic applications with Pulsar. You’ll learn to use this mature and battle-tested platform to deliver extreme levels of speed and durability to your messaging. Apache Pulsar committer David Kjerrumgaard teaches you to apply Pulsar’s seamless scalability through hands-on case studies, including IOT analytics applications and a microservices app based on Pulsar functions.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Reliable server-to-server messaging is the heart of a distributed application. Apache Pulsar is a flexible real-time messaging platform built to run on Kubernetes and deliver the scalability and resilience required for cloud-based systems. Pulsar supports both streaming and message queuing, and unlike other solutions, it can communicate over multiple protocols including MQTT, AMQP, and Kafka’s binary protocol.

About the book
Apache Pulsar in Action teaches you to build scalable streaming messaging systems using Pulsar. You’ll start with a rapid introduction to enterprise messaging and discover the unique benefits of Pulsar. Following crystal-clear explanations and engaging examples, you’ll use the Pulsar Functions framework to develop a microservices-based application. Real-world case studies illustrate how to implement the most important messaging design patterns.

What's inside

Publish from Pulsar into third-party data repositories and platforms
Design and develop Apache Pulsar functions
Create an event-driven food delivery application

About the reader
Written for experienced Java developers. No prior knowledge of Pulsar required.

About the author
David Kjerrumgaard is a committer on the Apache Pulsar project. He currently serves as a Developer Advocate for StreamNative, where he develops Pulsar best practices and solutions.

Table of Contents
PART 1 GETTING STARTED WITH APACHE PULSAR
1 Introduction to Apache Pulsar
2 Pulsar concepts and architecture
3 Interacting with Pulsar
PART 2 APACHE PULSAR DEVELOPMENT ESSENTIALS
4 Pulsar functions
5 Pulsar IO connectors
6 Pulsar security
7 Schema registry
PART 3 HANDS-ON APPLICATION DEVELOPMENT WITH APACHE PULSAR
8 Pulsar Functions patterns
9 Resiliency patterns
10 Data access
11 Machine learning in Pulsar
12 Edge analytics

Product Details

ISBN-13:	9781617296888
Publisher:	Manning
Publication date:	12/14/2021
Pages:	400
Product dimensions:	7.38(w) x 9.25(h) x 0.80(d)

About the Author

David Kjerrumgaard is a committer on the Apache Pulsar project and serves as a Developer Advocate for StreamNative with a focus on educating developers about Apache Pulsar. He was formerly the Global Practice Director at Hortonworks, where he was responsible for the development of best practices and solutions for the professional services team, with a focus on Streaming technologies including Kafka, NiFi, and Storm. He has earned both a BS and MS in Computer Science and Mathematics from Kent State University.

Table of Contents

Foreword xiii

Preface xv

Acknowledgments xviii

About this book xx

About the author xxv

About the cover illustration xxvi

Part 1 Getting started with Apache Pulsar 1

1 Introduction to Apache Pulsar 3

1.1 Enterprise messaging systems 4

Key capabilities 6

1.2 Message consumption patterns 7

Publish-subscribe messaging 7

Message queuing 7

1.3 The evolution of messaging systems 8

Generic messaging systems 8

Message-oriented middleware 9

Enterprise service bus 11

Distributed messaging systems 13

1.4 Comparison to Apache Kafka 19

Multilayered architecture 19

Message consumption 21

Data durability 23

Message acknowledgment 26

Message retention 28

1.5 Why do I need Pulsar? 29

Guaranteed message delivery 29

Infinite scalability 30

Resilient to failure 31

Support for millions of topics 32

Geo-replication and active failover 32

1.6 Real-world use cases 34

Unified messaging systems 35

Microservices platforms 35

Connected cars 36

Fraud detection 36

2 Pulsar concepts and architecture 38

2.1 Pulsar's physical architecture 39

Pulsar's layered architecture 39

Stateless serving layer 41

Stream storage layer 44

Metadata storage 48

2.2 Pulsar's logical architecture 50

Tenants, namespaces, and topics 50

Addressing topics in Pulsar 54

Producers, consumers, and subscriptions 54

Subscription types 56

2.3 Message retention and expiration 59

Data retention 60

Backlog quotas 61

Message expiration 62

Message backlog vs. message expiration 63

2.4 Tiered storage 64

3 Interacting with Pulsar 68

3.1 Getting started with Pulsar 69

3.2 Administering Pulsar 70

Creating a tenant, namespace, and topic 71

Java Admin API 72

3.3 Pulsar clients 72

The Pulsar Java client 75

The Pulsar Python client 83

The Pulsar Go client 86

3.4 Advanced administration 91

Persistent topic metrics 91

Message inspection 93

Part 2 Apache Pulsar development essentials 95

4 Pulsar Junctions 97

4.1 Stream processing 97

Traditional batching 98

Micro-batching 98

Stream native processing 99

4.2 What is Pulsar Functions? 100

Programming model 101

4.3 Developing Pulsar functions 102

Language native functions 102

The Pulsar SDK 103

Stateful functions 108

4.4 Testing Pulsar functions 111

Unit testing 112

Integration testing 113

4.5 Deploying Pulsar functions 117

Generating a deployment artifact 118

Function configuration 120

Function deployment 124

The function deployment life cycle 126

Deployment modes 127

Pulsar function data flow 128

5 Pulsar IO connectors 130

5.1 What are Pulsar IO connectors? 131

Sink connectors 131

Source connectors 134

PushSource connectors 135

5.2 Developing Pulsar IO connectors 137

Developing a sink connector 138

Developing a PushSource connector 139

5.3 Testing Pulsar IO connectors 142

Unit testing 142

Integration testing 144

Packaging Pulsar IO connectors 146

5.4 Deploying Pulsar IO connectors 147

Creating and deleting connectors 148

Debugging deployed connectors 149

5.5 Pulsar's built-in connectors 152

Launching the MongoDB cluster 152

Link the Pulsar and MongoDB containers 153

Configure and create the MongoDB sink 154

5.6 Administering Pulsar IO connectors 156

Listing connectors 156

Monitoring connectors 157

6 Pulsar security 161

6.1 Transport encryption 162

6.2 Authentication 169

TLS authentication 169

JSON Web Token authentication 174

6.3 Authorization 179

Roles 179

An example scenario 181

6.4 Message encryption 186

7 Schema registry 191

7.1 Microservice communication 192

Microservice APIs 193

The need for a schema registry 195

7.2 The Pulsar schema registry 195

Architecture 195

Schema versioning 198

Schema compatibility 198

Schema compatibility check strategies 200

7.3 Using the schema registry 205

Modelling the food order event in Avro 207

Producing food order events 209

Consuming the food order events 211

Complete example 212

7.4 Evolving the schema 215

Part 3 Hands-on application development with Apache Pulsar 219

8 Pulsar Functions patterns 221

8.1 Data pipelines 222

Procedural programming 222

DataFlow programming 223

8.2 Message routing patterns 225

Splitter 225

Dynamic router 229

Content-based router 232

8.3 Message transformation patterns 234

Message translator 235

Content enricher 237

Content filter 239

9 Resiliency patterns 241

9.1 Pulsar Functions resiliency 242

Adverse events 242

Fault detection 247

9.2 Resiliency design patterns 248

Retry pattern 249

Circuit breaker 252

Rate limiter 257

Time limiter 260

Cache 262

Fallback pattern 264

Credential refresh pattern 266

9.3 Multiple layers of resiliency 268

10 Data access 271

10.1 Data sources 272

10.2 Data access use cases 273

Device validation 273

Driver location data 283

11 Machine learning in Pulsar 290

11.1 Deploying ML models 291

Batch processing 291

Near real-time 291

11.2 Near real-time model deployment 292

11.3 Feature vectors 294

Feature stores 295

Feature calculation 296

11.4 Delivery time estimation 297

ML model export 297

Feature vector mapping 300

Model deployment 302

11.5 Neural nets 304

Neural net training 304

Neural net deployment in Java 306

12 Edge analytics 308

12.1 IIoT architecture 311

The perception and reaction layer 311

The transportation layer 312

The data processing layer 313

12.2 A Pulsar-based processing layer 313

12.3 Edge analytics 316

Telemetric data 316

Univariate and multivariate 318

12.4 Univariate analysis 319

Noise reduction 319

Statistical analysis 321

Approximation 325

12.5 Multivariate analysis 328

Creating a bidirectional messaging mesh 328

Multivariate dataset construction 332

12.6 Beyond the book 336

Appendix A Running Pulsar on Kubernetes 339

Appendix B Geo-replication 352

Index 363