Pipeline as Code: Continuous Delivery with Jenkins, Kubernetes, and Terraform

by Mohamed Labouardy

Overview

Start thinking about your development pipeline as a mission-critical application. Discover techniques for implementing code-driven infrastructure and CI/CD workflows using Jenkins, Docker, Terraform, and cloud-native services.

In Pipeline as Code, you will master:

    Building and deploying a Jenkins cluster from scratch
    Writing pipeline as code for cloud-native applications
    Automating the deployment of Dockerized and Serverless applications
    Containerizing applications with Docker and Kubernetes
    Deploying Jenkins on AWS, GCP and Azure
    Managing, securing and monitoring a Jenkins cluster in production
    Key principles for a successful DevOps culture

Pipeline as Code is a practical guide to automating your development pipeline in a cloud-native, service-driven world. You’ll use the latest infrastructure-as-code tools like Packer and Terraform to develop reliable CI/CD pipelines for numerous cloud-native applications. Follow this book's insightful best practices, and you’ll soon be delivering software that’s quicker to market, faster to deploy, and with less last-minute production bugs.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the technology
Treat your CI/CD pipeline like the real application it is. With the Pipeline as Code approach, you create a collection of scripts that replace the tedious web UI wrapped around most CI/CD systems. Code-driven pipelines are easy to use, modify, and maintain, and your entire CI pipeline becomes more efficient because you directly interact with core components like Jenkins, Terraform, and Docker.

About the book
In Pipeline as Code you’ll learn to build reliable CI/CD pipelines for cloud-native applications. With Jenkins as the backbone, you’ll programmatically control all the pieces of your pipeline via modern APIs. Hands-on examples include building CI/CD workflows for distributed Kubernetes applications, and serverless functions. By the time you’re finished, you’ll be able to swap manual UI-based adjustments with a fully automated approach!

What's inside

    Build and deploy a Jenkins cluster on scale
    Write pipeline as code for cloud-native applications
    Automate the deployment of Dockerized and serverless applications
    Deploy Jenkins on AWS, GCP, and Azure
    Grasp key principles of a successful DevOps culture

About the reader
For developers familiar with Jenkins and Docker. Examples in Go.

About the author
Mohamed Labouardy is the CTO and co-founder of Crew.work, a Jenkins contributor, and a DevSecOps evangelist.

Table of Contents

PART 1 GETTING STARTED WITH JENKINS
1 What’s CI/CD?
2 Pipeline as code with Jenkins
PART 2 OPERATING A SELF-HEALING JENKINS CLUSTER
3 Defining Jenkins architecture
4 Baking machine images with Packer
5 Discovering Jenkins as code with Terraform
6 Deploying HA Jenkins on multiple cloud providers
PART 3 HANDS-ON CI/CD PIPELINES
7 Defining a pipeline as code for microservices
8 Running automated tests with Jenkins
9 Building Docker images within a CI pipeline
10 Cloud-native applications on Docker Swarm
11 Dockerized microservices on K8s
12 Lambda-based serverless functions
PART 4 MANAGING, SCALING, AND MONITORING JENKINS
13 Collecting continuous delivery metrics
14 Jenkins administration and best practices

Product Details

ISBN-13:	9781638350378
Publisher:	Manning
Publication date:	11/23/2021
Sold by:	SIMON & SCHUSTER
Format:	eBook
Pages:	528
File size:	54 MB
Note:	This product may take a few minutes to download.

About the Author

Mohamed Labouardy is the CTO and co-founder of Crew.work, and a DevSecOps evangelist. He is the founder of Komiser.io, an author, open-source contributor, and regular conference speaker.

Table of Contents

Preface xiii

Acknowledgments xv

About this book xvi

About the author xix

About the cover illustration xx

Part 1 Getting Started with Jenkins 1

1 What's CI/CD? 3

1.1 Going cloud native 4

Monolithic 4

Microservices 5

Cloud native 8

Serverless 10

1.2 Defining continuous integration 12

1.3 Defining continuous deployment 13

1.4 Defining continuous delivery 14

1.5 Embracing CI/CD practices 15

1.6 Using essential CI/CD tools 16

Choosing a CI/CD tool 17

Introducing Jenkins 18

2 Pipeline as code with Jenkins 21

2.1 Introducing the Jenkinsfile 22

Blue Ocean plugin 26

Scripted pipeline 29

Declarative pipeline 31

2.2 Understanding multibranch pipelines 36

2.3 Exploring the GitFlow branch model 38

2.4 Test-driven development with Jenkins 39

The Jenkins Replay button 40

Command-line pipeline linter 41

IDE integrations 43

Part 2 Operating a Self-Healing Jenkins Cluster 47

3 Defining Jenkins architecture 49

3.1 Understanding master-worker architecture 50

3.2 Managing Jenkins workers 52

SSH 52

Command line 53

JNLP 53

Windows service 54

3.3 Architecting Jenkins for scale in AWS 55

Preparing the AWS environment 64

Configuring the AWS CLI 65

Creating and managing the IAM user 66

4 Baking machine images with Packer 70

4.1 Immutable infrastructure 71

4.2 Introducing Packer 72

How does it work? 73

Installation and configuration 74

Baking a machine image 75

4.3 Baking the Jenkins master AMI 85

Configuring Jenkins upon startup 85

Discovering Jenkins plugins 88

4.4 Baking the Jenkins worker AMI 96

5 Discovering Jenkins as code with Terraform 100

5.1 Introducing infrastructure as code 101

Terraform usage 102

5.2 Provisioning an AWS VPC 103

AWS VPC 104

VPC subnets 108

VPC route tables 111

VPC bastion host 114

5.3 Setting up a self-healing Jenkins master 117

5.4 Running Jenkins with native SSL/HTTPS 124

5.5 Dynamically autoscaling the Jenkins worker pool 128

Launch configuration 128

Auto Scaling group 131

Autoscaling scaling policies 133

Workers CPU utilization load 136

6 Deploying HA Jenkins on multiple cloud providers 140

6.1 Google Cloud Platform 141

Building Jenkins VM images 141

Configuring a GCP network with Terraform 147

Deploying Jenkins on Google Compute Engine 153

Launching auto-managed workers on GCP 157

6.2 Microsoft Azure 162

Building golden Jenkins VM images in Azure 162

Deploying a private virtual network 166

Deploying a Jenkins master virtual machine 171

Applying autoscaling to Jenkins workers 178

6.3 DigitalOcean 183

Creating Jenkins DigitalOcean Snapshots 183

Deploying a Jenkins master Droplet 186

Building Jenkins worker Droplets 190

Part 3 Hands-on CI/CD pipelines 195

7 Defining a pipeline as code for microservices 197

7.1 Introducing micro services-based applications 199

7.2 Defining multibranch pipeline jobs 203

7.3 Git and GitHub integration 205

7.4 Discovering Jenkins jobs' XML configuration 215

7.5 Configuring SSH authentication with Jenkins 219

7.6 Triggering Jenkins builds with GitHub webhooks 222

8 Running automated tests with Jenkins 231

8.1 Running unit tests inside Docker containers 233

8.2 Automating code linter integration with Jenkins 238

8.3 Generating code coverage reports 240

8.4 Injecting security in the CI pipeline 242

8.5 Running parallel tests with Jenkins 244

8.6 Improving quality with code analysis 246

8.7 Running mocked database tests 248

8.8 Generating HTML coverage reports 250

8.9 Automating UI testing with Headless Chrome 254

8.10 Integrating SonarQube Scanner with Jenkins 260

9 Building Docker images within a CI pipeline 271

9.1 Building Docker images 273

Using the Docker DSL 273

Docker build arguments 277

9.2 Deploying a Docker private registry 279

Nexus Repository OSS 279

Amazon Elastic Container Registry 286

Azure Container Registry 288

Google Container Registry 290

9.3 Tagging Docker images the right way 291

9.4 Scanning Docker images for vulnerabilities 296

9.5 Writing a Jenkins declarative pipeline 301

9.6 Managing pull requests with Jenkins 305

10 Cloud-native applications on Docker Swarm 309

10.1 Running a distributed Docker Swarm cluster 310

10.2 Defining a continuous deployment process 321

10.3 Integrating Jenkins with Slack notifications 335

10.4 Handling code promotion with Jenkins 341

10.5 Implementing the Jenkins delivery pipeline 346

11 Dockerized microservices on K8s 355

11.1 Setting up a Kubernetes cluster 356

11.2 Automating continuous deployment flow with Jenkins 360

Migrating Docker Compose to K8s manifests with Kompose 371

11.3 Walking through continuous delivery steps 372

11.4 Packaging Kubernetes applications with Helm 381

11.5 Running post-deployment smoke tests 387

11.6 Discovering Jenkins X 390

12 Lambda-based serverless functions 401

12.1 Deploying a Lambda-based application 402

12.2 Creating deployment packages 407

Mono-repo strategy 407

Multi-repo strategy 413

12.3 Updating Lambda function code 417

12.4 Hosting a static website on S3 420

12.5 Maintaining multiple Lambda environments 423

12.6 Configuring email notification in Jenkins 434

Part 4 Managing, Scaling, and Monitoring Jenkins 439

13 Collecting continuous delivery metrics 441

13.1 Monitoring Jenkins cluster health 442

13.2 Centralized logging for Jenkins logs with ELK 452

Streaming logs with Filebeat 454

Streaming logs with the Logstash plugin 461

13.3 Creating alerts based on metrics 462

14 Jenkins administration and best practices 467

14.1 Exploring Jenkins security and RBAC authorization 468

Matrix authorization strategy 469

Role-based authorization strategy 471

14.2 Configuring GitHub OAuth for Jenkins 472

14.3 Keeping track of Jenkins users' actions 475

14.4 Extending Jenkins with shared libraries 476

14.5 Backing up and restoring Jenkins 480

14.6 Setting up cron jobs with Jenkins 484

14.7 Running Jenkins locally as a Docker container 487

Index 493