RabbitMQ in Depth

by Gavin M. Roy, James Titcumb (Editor)

Overview

Summary

RabbitMQ in Depth is a practical guide to building and maintaining message-based applications. This book provides detailed coverage of RabbitMQ with an emphasis on why it works the way it does.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the Technology

At the heart of most modern distributed applications is a queue that buffers, prioritizes, and routes message traffic. RabbitMQ is a high-performance message broker based on the Advanced Message Queueing Protocol. It's battle tested, ultrafast, and powerful enough to handle anything you can throw at it. It requires a few simple setup steps, and you can instantly start using it to manage low-level service communication, application integration, and distributed system message routing.

About the Book

RabbitMQ in Depth is a practical guide to building and maintaining message-based applications. This book provides detailed coverage of RabbitMQ with an emphasis on why it works the way it does. You'll find examples and detailed explanations based in real-world systems ranging from simple networked services to complex distributed designs. You'll also find the insights you need to make core architectural choices and develop procedures for effective operational management.

What's Inside

• AMQP, the Advanced Message Queueing Protocol
• Communicating via MQTT, Stomp, and HTTP
• Valuable troubleshooting techniques
• Database integration

About the Reader

Written for programmers with a basic understanding of messaging-oriented systems.

About the Author

Gavin M. Roy is an active, open source evangelist and advocate who has been working with internet and enterprise technologies since the mid-90s. Technical editor James Titcumb is a freelance developer, trainer, speaker, and active contributor to open source projects.

Table of Contents

PART 1 - RABBITMQ AND APPLICATION ARCHITECTURE
1. Foundational RabbitMQ
2. How to speak Rabbit: the AMQ Protocol
3. An in-depth tour of message properties
4. Performance trade-offs in publishing
5. Don't get messages; consume them
6. Message patterns via exchange routing
PART 2 - MANAGING RABBITMQ IN THE DATA CENTER OR THE CLOUD
7. Scaling RabbitMQ with clusters
8. Cross-cluster message distribution
PART 3 - INTEGRATIONS AND CUSTOMIZATION
9. Using alternative protocols
10. Database integrations

Product Details

ISBN-13:	9781638353225
Publisher:	Manning
Publication date:	09/18/2017
Sold by:	SIMON & SCHUSTER
Format:	eBook
Pages:	264
File size:	5 MB

About the Author

Gavin M. Roy is an active, open source evangelist and advocate who has been working with internet and enterprise technologies since the mid-90s. 

Technical editor James Titcumb is a freelance developer, trainer, speaker, and active contributor to open source projects.

Table of Contents

Preface xiii

Acknowledgments xiv

About this book xv

Part 1 RabbitMQ and Application Architecture 1

1 Foundational RabbitMQ 3

1.1 RabbitMQ's features and benefits 4

RabbitMQ and Erlang 5

RabbitMQ and AMQP 6

1.2 Who's using RabbitMQ, and how? 7

1.3 The advantages of loosely coupled architectures 8

Decoupling your application 10

Decoupling database writes 11

Seamlessly adding new functionality 12

Replication of data and events 12

Multi-master federation of data and events 13

The Advanced Message Queuing model 14

1.4 Summary 16

2 How to speak Rabbit: the AMQ Protocol 18

2.1 AMQP as an RFC transport 19

Kicking off the conversation 20

Tuning in to the right channel 20

2.2 AMQP's RPC frame structure 21

AMQP frame components 21

Types of frames 22

Marshaling messages into frames 23

The anatomy of a method frame 24

The content header frame 26

The body frame 26

2.3 Putting the protocol to use 27

Declaring an exchange 27

Declaring a queue 28

Binding a queue to an exchange 29

Publishing a message to RabbitMQ 29

Consuming messages from RabbitMQ 30

2.4 Writing a message publisher in Python 32

2.5 Getting messages from RabbitMQ 36

2.6 Summary 37

3 An in-depth tour of message properties 38

3.1 Using properties properly 39

3.2 Creating an explicit message contract with content-type 41

3.3 Reducing message size with gzip and content-encoding 43

3.4 Referencing messages with message-id and correlation-id 45

Message-id 45

Correlation-id 45

3.5 Born-on dating: the timestamp properly 46

3.6 Automatically expiring messages 47

3.7 Balancing speed with safety using delivery-mode 48

3.8 Validating message origin with app-id and user-id 49

App-id 50

User-id 51

3.9 Getting specific with the message type property 51

3.10 Using reply-to for dynamic workflows 52

3.11 Custom properties using the headers property 53

3.12 The priority property 54

3.13 A property you can't use: cluster-id/reserved 54

3.14 Summary 55

4 Performance trade-offs in publishing 58

4.1 Balancing delivery speed with guaranteed delivery 59

What to expect with no guarantees 60

RabbitMQ won't accept non-routable messages with mandatory set 62

Publisher Confirms as a lightweight alternative to transactions 64

Using alternate exchanges for unroutable messages 66

Batch processing with transactions 68

Surviving node failures with HA queues 70

HA queues with transactions 72

Persisting messages to disk via delivery-mode 2 72

4.2 When RabbitMQ pushes back 75

Checking the connection status with rabbitpy 77

Using the management APT for connection status 77

4.3 Summary 78

5 Don't get messages; consume them 79

5.1 Basic.Get vs. Basic.Consume 80

Basic.Get 80

Basic.Consume 82

5.2 Performance-tuning consumers 84

Using no-ack mode for faster throughput 85

Controlling consumer prefetching via quality of service settings 86

Using transactions with consumers 89

5.3 Rejecting messages 90

Basic.Reject 90

Basic.Nack 91

Dead letter exchanges 92

5.4 Controlling queues 94

Temporary queues 94

Permanent queues 97

Arbitrary queue settings 99

5.5 Summary 99

6 Message patterns via exchange routing 101

6.1 Simple message routing using the direct exchange 102

Creating the application architecture 103

Creating the BPC worker 107

Writing a simple RPC publisher 110

6.2 Broadcasting messages via the fanout exchange 115

Modifying the facial detection consumer 116

Creating a simple image-hashing consumer 117

6.3 Selectively routing messages with the topic exchange 119

6.4 Selective routing with the headers exchange 122

6.5 Exchange performance benchmarking 124

6.6 Going meta: exchange-to-exchange routing 125

6.7 Routing messages with the consistent-hashing exchange 127

6.8 Summary 131

Part 2 Managing rabbitMQ in the Data Center or the Cloud 133

7 Scaling RabbitMQ with clusters 135

7.1 About clusters 136

Clusters and the management UI 137

Cluster node types 138

Clusters and queue behavior 139

7.2 Cluster setup 142

Virtual machine setup 143

Adding nodes to the cluster 144

7.3 Summary 147

8 Cross-cluster message distribution 148

8.1 Federating exchanges and queues 149

Federated exchanges 149

Federated queues 152

8.2 Creating the RabbitMO virtual machines 153

Creating the first instance 153

Duplicating the EC2 instance 159

8.3 Connecting upstream 162

Defining federation upstreams 162

Defining a policy 164

Leveraging upstream sets 167

Bidirectional federated exchanges 170

Federation for cluster upgrades 171

8.4 Summary 173

Part 3 Integration and Customization 175

9 Using alternative protocols 177

9.1 MQTT and RabbitMQ 178

The MQTT protocol 178

Publishing via MQTT 182

MQTT subscribers 184

MQTT plugin configuration 187

9.2 STOMP and RabbitMQ 189

The STOMP protocol 190

Publishing messages 191

Consuming messages 195

Configuring the STOMP plugin 198

Using STOMP in the web browser 199

9.3 Stateless publishing via HTTP 200

How statelessd came to be 200

Using statelessd 201

Operational architecture 202

Publishing messages via statelessd 203

9.4 Summary 203

10 Database integrations 205

10.1 The pg_amqp PostgreSQL extension 206

Installing the pg_amqp extension 207

Configuringthe pg_amqp extension 209

Publishing a message via pg_amqp 210

10.2 Listening to PostgreSQL notifications 212

Installing the PostgreSQL LISTEN exchange 213

Policy-based configuration 215

Creating the exchange 217

Creating and binding a test queue 217

Publishing via NOTIFY 218

10.3 Storing messages in InfluxDB 219

InfluxDB installation and setup 220

Installing the InfluxDB storage exchange 222

Creating a test exchange 223

Testing the exchange 224

10.4 Summary 227

Appendix: Getting set up 228

Index 237