Mesos in Action

by Roger Ignazio

Overview

Summary

Mesos in Action introduces readers to the Apache Mesos cluster manager and the concept of application-centric infrastructure. Filled with helpful figures and hands-on instructions, this book guides you from your first steps creating a highly-available Mesos cluster through deploying applications in production and writing native Mesos frameworks.

Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.

About the Technology
Modern datacenters are complex environments, and when you throw Docker and other container-based systems into the mix, there’s a great need to simplify. Mesos is an open source cluster management platform that transforms the whole datacenter into a single pool of compute, memory, and storage resources that you can allocate, automate, and scale as if you’re working with a single supercomputer.
About the Book

Mesos in Action introduces readers to the Apache Mesos cluster manager and the concept of application-centric infrastructure. Filled with helpful figures and hands-on instructions, this book guides you from your first steps creating a highly-available Mesos cluster through deploying applications in production and writing native Mesos frameworks. You’ll learn how to scale to thousands of nodes, while providing resource isolation between processes using Linux and Docker containers. You’ll also learn practical techniques for deploying applications using popular key frameworks.

What’s Inside

• Spinning up your first Mesos cluster
• Scheduling, resource administration, and logging
• Deploying containerized applications with Marathon, Chronos, and Aurora
• Writing Mesos frameworks using Python

About the Reader

Readers need to be familiar with the core ideas of datacenter administration and need a basic knowledge of Python or a similar programming language.

About the Author

Roger Ignazio is an experienced systems engineer with a focus on distributed, fault-tolerant, and scalable infrastructure. He is currently a technical lead at Mesosphere.

Table of Contents

PART 1 HELLO, MESOS
1. Introducing Mesos
2. Managing datacenter resources with Mesos
PART 2 CORE MESOS
3. Setting up Mesos
4. Mesos fundamentals
5. Logging and debugging
6. Mesos in production
PART 3 RUNNING ON MESOS
7. Deploying applications with MarathoN
8. Managing scheduled tasks with Chronos
9. Deploying applications and managing scheduled tasks with Aurora
10. Developing a framework

Product Details

ISBN-13:	9781617292927
Publisher:	Manning
Publication date:	05/23/2016
Edition description:	1st Edition
Pages:	272
Product dimensions:	7.40(w) x 9.20(h) x 1.00(d)

About the Author

Roger Ignazio is an experienced systems engineer with a focus on distributed, fault-tolerant, and scalable infrastructure. He is currently a technical lead at Mesosphere.

Table of Contents

Foreword xv

Preface xvii

Acknowledgements xix

About this book xxi

About the author xxv

About the cover illustration xxvi

Part 1 Hello, Mesos 1

1 Introducing Mesos 3

1.1 Meet Mesos 4

Understanding how it works 5

Comparing virtual machines and containers 7

Knowing when (and why) to use Mesos 9

1.2 Why we need to rethink the datacenter 10

Partitioning of resources 11

Deploying applications 12

1.3 The Mesos distributed architecture 13

Masters 13

Slaves 14

Frameworks 15

1.4 Summary 15

2 Managing datacenter resources urith Mesos 17

2.1 A brief introduction to Spark 18

Spark on a standalone cluster 19

Spark on Mesos 19

2.2 Running a Spark job on Mesos 22

Finding prime numbers in a set 22

Getting and packaging up the code 23

Submitting the job 24

Observing the output 24

2.3 Exploring further 26

Mesos UI 26

Spark UI 26

2.4 Summary 28

Part 2 Core Mesos 31

3 Setting up Mesos 33

3.1 Deploying Mesos 34

Mesos duster components 34

Considerations for a development environment 35

Considerations for a production environment 36

3.2 Installing Mesos and ZooKeeper 38

Installing from packages 39

Compiling and installing from source 40

3.3 Configuring Mesos and ZooKeeper 44

ZooKeeper configuration 44

Mesas configuration 46

3.4 Installing and configuring Docker 52

Installation 52

Configuration 54

Configuring Mesos slaves for Docker 55

3.5 Upgrading Mesos 55

Upgrading Mesos wasters 56

Upgrading Mesos slaves 56

3.6 Summary 57

4 Mesos fundamentals 58

4.1 Scheduling and allocating datacenter resources 58

Understanding resource scheduling 59

Understanding resource allocation 60

Customizing Mesos stave resources and attributes 63

4.2 Isolating resources with containers 64

Isolating and monitoring CPU, memory, and disk 64

Network monitoring and rate limiting 66

4.3 Understanding fault tolerance and high availability 70

Fault tolerance 71

High availability 72

Handling failures and upgrades 72

4.4 Summary 77

5 Logging and debugging 79

5.1 Understanding and configuring Mesos logging 80

Locating and interpreting log files 80

Configuring logging 82

5.2 Debugging a Mesos cluster and its tasks 83

Using the Mesos web interface 85

Using the built-in command-line tools 92

Using Mesospheres mesos-cli tool 93

5.3 Summary 95

6 Mesos in production 97

6.1 Monitoring the Mesos and ZooKeeper clusters 98

Monitoring the Mesos master 98

Monitoring the Mesos slave 100

Monitoring ZooKeeper 101

6.2 Modifying the Mesos master quorum 103

Adding masters 104

Removing masters 105

Replacing masters 105

6.3 Implementing security and access control 106

Slave and framework authentication 106

Authorization and access control lists 109

Framework rate limiting 111

6.4 Summary 114

Part 3 Running on Mesos 117

7 Deploying applications with Marathon 119

7.1 Getting to know Marathon 120

Exploring the web interface and API 121

Service discovery and routing 123

7.2 Deploying Marathon and HAProxy 126

Installing and configuring Marathon 126

Installing and configuring HAProxy 129

7.3 Creating and scaling applications 132

Deploying a simple application 132

Deploying a Docker container 135

Performing health checks and rolling application upgrades 136

7.4 Creating application groups 139

Understanding the anatomy of an application group 139

Deploying an application group 140

7.5 Logging and debugging 142

Configuring logging for Marathon 142

Debugging Marathon applications and tasks 143

7.6 Summary 145

8 Managing scheduled tasks with Chronos 147

8.1 Getting to know Chronos 148

Exploring the web interface and API 149

8.2 Installing and configuring Chronos 151

Reviewing prerequisites 151

Installing Chronos 152

Configuring Chronos 153

8.3 Working with simple jobs 154

Creating a schedule-based job 154

Creating a schedule-based job using Docker 158

8.4 Working with complex jobs 160

Combining schedule-based and dependency-based jobs 160

Visualizing job dependencies 163

8.5 Monitoring the output and status of Chronos jobs 164

Job failure notifications and monitoring 165

Observing standard output and standard error via Mesos 167

8.6 Summary 167

9 Deploying applications and managing scheduled tasks with Aurora 169

9.1 Introducing Aurora 170

The Aurora scheduler 171

The Thermos executor and observer 171

The Aurora user and admin clients 172

The Aurora domain-specific language 173

9.2 Deploying Aurora 174

Trying out Aurora in a development environment 175

Building and installing Aurora 176

Configuring Aurora 178

9.3 Deploying applications 183

Deploying a simple application 184

Deploying a Docker-based application 187

9.4 Managing scheduled tasks 189

Creating a Cron job 189

Creating a Docker-based Cronjob 190

9.5 Administering Aurora 192

Managing users and quota 192

Performing maintenance 194

9.6 Summary 195

10 Developing a framework 196

10.1 Understanding framework basics 197

When and why would you write a framework 199

The scheduler implementation 200

The executor implementation 202

10.2 Developing a scheduler 206

Working with the Scheduler API 207

Working with the Scheduler Driver 209

10.3 Developing an executor 210

Working with the Executor API 210

Working with the executor driver 212

10.4 Running the framework 213

Deploying in development 213

Considerations for a production deployment 215

10.5 Summary 216

Appendix A Case study: Mesosphere DCOS, an enterprise Mesos distribution 217

Appendix B List of Mesos frameworks and tools 229

Index 237