Deep Learning from Scratch: Building with Python from First Principles

by Seth Weidman

Overview

With the resurgence of neural networks in the 2010s, deep learning has become essential for machine learning practitioners and even many software engineers. This book provides a comprehensive introduction for data scientists and software engineers with machine learning experience. You'll start with deep learning basics and move quickly to the details of important advanced architectures, implementing everything from scratch along the way.

Author Seth Weidman shows you how neural networks work using a first principles approach. You'll learn how to apply multilayer neural networks, convolutional neural networks, and recurrent neural networks from the ground up. With a thorough understanding of how neural networks work mathematically, computationally, and conceptually, you'll be set up for success on all future deep learning projects.

This book provides:

• Extremely clear and thorough mental models—accompanied by working code examples and mathematical explanations—for understanding neural networks
• Methods for implementing multilayer neural networks from scratch, using an easy-to-understand object-oriented framework
• Working implementations and clear-cut explanations of convolutional and recurrent neural networks
• Implementation of these neural network concepts using the popular PyTorch framework

Product Details

ISBN-13:	9781492041412
Publisher:	O'Reilly Media, Incorporated
Publication date:	10/01/2019
Pages:	250
Product dimensions:	6.90(w) x 9.00(h) x 0.60(d)

About the Author

Seth Weidman is a data scientist who has applied and taught machine learning concepts for several years. He started out as the first data scientist at Trunk Club, where he built lead scoring models and recommender systems, and currently works at Facebook, where he builds machine learning models for their infrastructure team. In between he taught data science and machine learning for the bootcamps and on the corporate training team at Metis. He is passionate about explaining complex concepts simply, striving to find the simplicity on the other side of complexity.

Table of Contents

Preface vii

1 Foundations 1

Functions 2

Derivatives 7

Nested Functions 9

The Chain Rule 11

A Slightly Longer Example 14

Functions with Multiple Inputs 17

Derivatives of Functions with Multiple Inputs 19

Functions with Multiple Vector Inputs 20

Creating New Features from Existing Features 21

Derivatives of Functions with Multiple Vector Inputs 23

Vector Functions and Their Derivatives: One Step Further 26

Computational Graph with Two 2D Matrix Inputs 29

The Fun Part: The Backward Pass 33

Conclusion 39

2 Fundamentals 41

Supervised Learning Overview 42

Supervised Learning Models 44

Linear Regression 45

Training the Model 50

Assessing Our Model: Training Set Versus Testing Set 54

Assessing Our Model: The Code 55

Neural Networks from Scratch 58

Training and Assessing Our First Neural Network 65

Conclusion 68

3 Deep Learning from Scratch 71

Deep Learning Definition: A First Pass 72

The Building Blocks of Neural Networks: Operations 73

The Building Blocks of Neural Networks: Layers 77

Building Blocks on Building Blocks 79

The NeuralNetwork Class, and Maybe Others 85

Deep Learning from Scratch 88

Trainer and Optimizer 91

Putting Everything Together 96

Conclusion and Next Steps 98

4 Extensions 99

Some Intuition About Neural Networks 100

The Softmax Cross Entropy Loss Function 102

Experiments 110

Momentum 113

Learning Rate Decay 116

Weight Initialization 118

Dropout 122

Conclusion 125

5 Convolutional Neural Networks 127

Neural Networks and Representation Learning 127

Convolutional Layers 132

Implementing the Multichannel Convolution Operation 138

Using This Operation to Train a CNN 153

Conclusion 156

6 Recurrent Neural Networks 159

The Key Limitation: Handling Branching 160

Automatic Differentiation 162

Motivation for Recurrent Neural Networks 167

Introduction to Recurrent Neural Networks 168

RNNs: The Code 175

Conclusion 193

7 PyTorch 195

PyTorch Tensors 195

Deep Learning with PyTorch 197

Convolutional Neural Networks in PyTorch 205

Postscript: Unsupervised Learning via Autoencoders 212

Conclusion 220

A Deep Dives 221

Index 231