Bootstrapping Microservices with Docker, Kubernetes, and Terraform: A project-based guide

by Ashley Davis

Overview

Summary
The best way to learn microservices development is to build something! Bootstrapping Microservices with Docker, Kubernetes, and Terraform guides you from zero through to a complete microservices project, including fast prototyping, development, and deployment. You’ll get your feet wet using industry-standard tools as you learn and practice the practical skills you’ll use for every microservices application. Following a true bootstrapping approach, you’ll begin with a simple, familiar application and build up your knowledge and skills as you create and deploy a real microservices project.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Taking microservices from proof of concept to production is a complex, multi-step operation relying on tools like Docker, Terraform, and Kubernetes for packaging and deployment. The best way to learn the process is to build a project from the ground up, and that’s exactly what you’ll do with this book!

About the book
In Bootstrapping Microservices with Docker, Kubernetes, and Terraform, author Ashley Davis lays out a comprehensive approach to building microservices. You’ll start with a simple design and work layer-by-layer until you’ve created your own video streaming application. As you go, you’ll learn to configure cloud infrastructure with Terraform, package microservices using Docker, and deploy your finished project to a Kubernetes cluster.

What's inside

    Developing and testing microservices applications
    Working with cloud providers
    Applying automated testing
    Implementing infrastructure as code and setting up a continuous delivery pipeline
    Monitoring, managing, and troubleshooting

About the reader
Examples are in JavaScript. No experience with microservices, Kubernetes, Terraform, or Docker required.

About the author
Ashley Davis is a software developer, entrepreneur, stock trader, and the author of Manning’s Data Wrangling with JavaScript.

Table of Contents

1 Why microservices?

2 Creating your first microservice

3 Publishing your first microservice

4 Data management for microservices

5 Communication between microservices

6 Creating your production environment

7 Getting to continuous delivery

8 Automated testing for microservices

9 Exploring FlixTube

10 Healthy microservices

11 Pathways to scalability

Product Details

ISBN-13:	9781638350934
Publisher:	Manning
Publication date:	01/23/2021
Sold by:	SIMON & SCHUSTER
Format:	eBook
Pages:	440
File size:	15 MB
Note:	This product may take a few minutes to download.

About the Author

Ashley Davis is a software craftsman, entrepreneur, and author with over 25 years of experience in software development—from coding, to managing teams, to founding companies. He has worked for a range of companies, from the tiniest startups to the largest internationals. Along the way, he has contributed back to the community through his writing and open source coding. He is currently VP of Engineering at Hone, building products on the Algorand blockchain. He is also the creator of Data-Forge Notebook, a desktop application for exploratory coding and data visualization using JavaScript and TypeScript.

Table of Contents

Preface xv

Acknowledgments xvii

About this book xviii

About the author xxii

About the cover illustration xxiii

1 Why microservices? 1

1.1 This book is practical 2

1.2 What will I learn? 4

1.3 What do I need to know? 5

1.4 Managing complexity 6

1.5 What is a microservice? 6

1.6 What is a microservices application? 7

1.7 What's wrong with the monolith? 9

1.8 Why are microservices popular now? 11

1.9 Benefits of microservices 12

1.10 Drawbacks of microservices 14

1.11 Modern tooling for microservices 15

1.12 Designing a microservices application 16

1.13 An example application 18

2 Creating your first microservice 20

2.1 New tools 21

2.2 Getting the code 22

2.3 Why Node.js? 22

2.4 Our philosophy of development 24

2.5 Establishing our single-service development environment 27

Installing Git 28

Cloning the code repo 29

Getting Visual Studio (VS) Code 30

Installing Node.js 30

2.6 Building an HTTP server for video streaming 31

Creating a Node.js project 32

Installing Express 34

Creating the Express boilerplate 37

Running our simple web server 38

Adding streaming video 39

Configuring our microservice 41

Setting up for production 44

Live reloading for fast iteration 45

Running the finished code from this chapter 48

2.7 Node.js review 49

2.8 Continue your learning 50

3 Publishing your first microservice 51

3.1 New tools 52

3.2 Getting the code 52

3.3 What is a container? 53

3.4 What is an image? 54

3.5 Why Docker? 55

3.6 What are we doing with Docker? 55

3.7 Extending our development environment with Docker 56

Installing Docker 57

Checking your Docker installation 57

3.8 Packaging our microservice 58

Creating a Dockerfile 58

Packaging and checking our Docker image 60

Booting our microservice in a container 63

3.9 Publishing our microservice 66

Creating a private container registry 67

Pushing our microservice to the registry 72

Booting our microservice from the registry 76

3.10 Docker review 78

3.11 Continue your learning 80

4 Data management for microservices 82

4.1 New tools 83

4.2 Getting the code 83

4.3 Developing microservices with Docker Compose 84

Why Docker Composed 85

Installing Docker Compose 86

Creating our Docker Compose file 86

Booting our microservices application 89

Working with the application 90

Shutting down the application 91

Can we use Docker Compose for production? 93

4.4 Adding file storage to our application 94

Using Azure Storage 95

Updating the video-streaming microservice 103

Adding our new microservice to the Docker Compose file 104

Testing the updated application 106

Cloud storage vs. cluster storage 108

What did we achieve? 108

4.5 Adding a database to our application 109

Why MongoDB? 110

Adding a database server in development 110

Adding a database server in production 116

Database-per-microservice or database-per-application? 117

What did we achieve? 117

4.6 Docker Compose review 118

4.7 Continue your learning 119

5 Communication between microservices 121

5.1 New and familiar tools 122

5.2 Getting the code 122

5.3 Getting our microservices talking 123

5.4 Introducing the history microservice 124

5.5 Live reload for fast iterations 125

Creating a stub for the history microservice 126

Augmenting the microservice for live reload 127

Splitting our Dockerfile for development and production 129

Updating the Docker Compose file for live reload 131

Trying out live reload 133

Testing production mode in development 134

What have we achieved? 135

5.6 Methods of communication for microservices 135

Direct messaging 136

Indirect messaging 137

5.7 Direct messaging with HTTP 138

Why HTTP? 138

Directly targeting messages at particular microservices 139

Sending a message with HTTP POST 140

Receiving a message with HTTP POST 141

Testing the updated application 142

Sequencing direct messages 142

What have we achieved? 143

5.8 Indirect messaging with RabbitMQ 144

Why RabbitMQ? 144

Indirectly targeting messages to microservices 145

Creating a RabbitMQ server 146

Investigating the RabbitMQ dashboard 147

Connecting our microservice to the message queue 148

Single-recipient indirect messaging 151

Multiple-recipient messages 154

Sequencing indirect messages 157

What have we achieved? 159

5.9 Microservices communication review 159

5.10 Continue your learning 160

6 Creating your production environment 162

6.1 New tools 163

6.2 Getting the code 163

6.3 Getting to production 163

6.4 Infrastructure as code 164

6.5 Hosting microservices on Kubernetes 166

Why Kubernetes? 166

How does Kubernetes work? 167

6.6 Working with the Azure CLI 168

Installing the Azure CLI 168

Authenticating with Azure 169

Which version of Kubernetes? 170

What have we achieved? 171

6.7 Creating infrastructure with Terraform 171

Why Terraform? 172

Installing Terraform 173

Terraform project setup 173

6.8 Creating an Azure resource group for your application 175

Evolutionary architecture with Terraform 175

Scripting infrastructure creation 177

Initializing Terraform 178

By-products of Terraform initialization 178

Fixing provider version numbers 179

Building your infrastructure 180

Understanding Terraform state 181

Destroying and recreating your infrastructure 184

What have we achieved? 186

6.9 Creating your container registry 186

Continuing the evolution of our infrastructure 186

Creating the container registry 186

Terraform outputs 188

What have we achieved? 188

6.10 Refactoring to share configuration data 189

Continuing the evolution of our infrastructure 189

Introducing Terraform variables 189

6.11 Creating our Kubernetes cluster 190

Scripting creation of your Chester 190

Cluster authentication with Azure 192

Building your cluster 194

What have we achieved? 194

6.12 Interacting with Kubernetes 195

Kubernetes authentication 195

The Kubernetes CLI 196

The Kubernetes dashboard 197

What have we achieved? 199

6.13 Terraform review 200

6.14 Continue your learning 200

7 Getting to continuous delivery 202

7.1 New and familiar tools 203

7.2 Getting the code 203

7.3 Continuing to evolve our infrastructure 203

7.4 Continuous delivery (CD) 204

7.5 Deploying containers with Terraform 206

Configuring the Kubernetes provider 206

Deploying our database 207

Preparing for continuous delivery 208

Testing the new database server 209

Deploying and testing RabbitMQ 210

Tightening our security 212

What have we achieved? 212

7.6 Deploying our first microservice with Terraform 212

Using local variables to compute configuration 213

Building and publishing the Docker image 213

Authentication with the container registry 215

Deploying the video-streaming microservice 216

Testing your microservice 218

What have we achieved? 218

7.7 Continuous delivery with Bitbucket Pipelines 218

Why Bitbucket Pipelines? 219

Importing the example code to Bitbucket 219

Creating a deployment shell script 223

Managing Terraform state 224

The Bitbucket Pipelines script 225

Configuring environment variables 226

Testing your deployment pipeline 227

Debugging your deployment pipeline 229

What have we achieved? 230

7.8 Continue your learning 231

8 Automated testing for microservices 233

8.1 New tools 234

8.2 Getting the code 235

8.3 Testing for microservices 235

8.4 Automated testing 235

8.5 Testing with Jest 238

Why Jest? 239

Setting up fest 239

The math library to test 241

Your first Jest test 241

Running your first test 243

Live reload with Jest 243

Interpreting test failures 243

Invoking Jest with npm 245

Populating your test suite 246

Mocking with Jest 247

What have we achieved? 249

8.6 Unit testing 250

The metadata microservice 251

Creating unit tests with Jest 253

Running the tests 257

What have we achieved? 257

8.7 Integration testing 257

The code to test 258

Running a MongoDB database 259

Loading database fixtures 259

Creating an integration test with Jest 260

Running the test 263

What have we achieved1? 263

8.8 End-to-end testing 263

Why Cypress? 265

Installing Cypress 266

Starting the Cypress UI 267

Setting up database fixtures 268

Booting your application 270

Creating an end-to-end test with Cypress 271

Invoking Cypress with npm 273

What have we achieved? 273

8.9 Automated testing in the CD pipeline 274

8.10 Review of testing 276

8.11 Continue your learning 276

9 Exploring FlixTube 278

9.1 No new tools! 279

9.2 Getting the code 279

9.3 Revisiting essential skills 279

9.4 Overview of FlixTube 280

FlixTube microservices 281

Microservice project structure 282

FlixTube project structure 282

9.5 Running FlixTube in development 283

Booting a microservice 284

Booting the application 285

9.6 Testing FlixTube in development 287

Testing a microservice with Jest 287

Testing the application with Cypress 288

9.7 FlixTube deep dive 289

Database fixtures 289

Mocking storage 290

The gateway 293

The user interface (UI) 295

Video streaming 298

Video upload 300

9.8 Manually deploying FlixTube to production with Terraform 304

The Terraform scripts structure 305

Prerequisites 306

Azure authentication 306

Configuring storage 307

Deploying the application 307

Checking that it works 308

Teardown 309

Terraform modules 309

9.9 Continuous delivery to production 313

Prerequisites 313

Setting up your code repository 313

Preparing the backend 314

The deployment shell script 315

FlixTube's CD configuration 315

Testing the continuous delivery (CD) pipeline 316

Adding automated testing 316

9.10 Review 318

9.11 FlixTube in the future 319

9.12 Continue your learning 320

10 Healthy microservices 322

10.1 Maintaining healthy microservices 323

10.2 Monitoring your microservices 324

Logging in development 325

Error handling 326

Logging with Docker Compose 329

Basic logging with Kubernetes 330

Roll your own log aggregation for Kubernetes 333

Enterprise logging, monitoring and alerts 334

Automatic restarts with Kubernetes health checks 336

Tracing across microservices 339

10.3 Debugging microservices 340

The debugging process 341

Debugging production microservices 346

10.4 Reliability and recovery 346

Practice defensive programming 346

Practice defensive testing 347

Protect your data 347

Replication and redundancy 348

Fault isolation and graceful degradation 350

Simple techniques for fault tolerance 351

Advanced techniques for fault tolerance 354

10.5 Continue your learning 357

11 Pathways to scalability 358

11.1 Our future is scalable 359

11.2 Scaling the development process 359

Multiple teams 360

Independent microservices 361

Splitting the code repository 363

Splitting the continuous delivery (CD)pipeline 365

The meta-repo 367

Creating multiple environments 369

Production workflow 371

11.3 Scaling performance 374

Vertically scaling the cluster 375

Horizontally scaling the cluster 376

Horizontally scaling an individual microservice 377

Elastic scaling for the cluster 378

Elastic scaling far an individual microservice 379

Scaling the database 380

Managing changes to infrastructure 382

11.4 Security 384

Trast models 385

Sensitive configuration 386

11.5 Refactoring to microservices 387

11.6 Microservices on a budget 392

11.7 From simple beginnings 393

11.8 Continue your learning 393

Appendix A Creating a development environment with Vagrant 397

Appendix B Bootstrapping Microservices cheat sheet 401

Index 405