Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

by Savaş Yıldırım, Meysam Asgari-Chenaghlu

Overview

Take a problem-solving approach to learning all about transformers and get up and running in no time by implementing methodologies that will build the future of NLP

Key Features

• Explore quick prototyping with up-to-date Python libraries to create effective solutions to industrial problems
• Solve advanced NLP problems such as named-entity recognition, information extraction, language generation, and conversational AI
• Monitor your model's performance with the help of BertViz, exBERT, and TensorBoard

Book Description

Transformer-based language models have dominated natural language processing (NLP) studies and have now become a new paradigm. With this book, you'll learn how to build various transformer-based NLP applications using the Python Transformers library. The book gives you an introduction to Transformers by showing you how to write your first hello-world program. You'll then learn how a tokenizer works and how to train your own tokenizer. As you advance, you'll explore the architecture of autoencoding models, such as BERT, and autoregressive models, such as GPT. You'll see how to train and fine-tune models for a variety of natural language understanding (NLU) and natural language generation (NLG) problems, including text classification, token classification, and text representation. This book also helps you to learn efficient models for challenging problems, such as long-context NLP tasks with limited computational capacity. You'll also work with multilingual and cross-lingual problems, optimize models by monitoring their performance, and discover how to deconstruct these models for interpretability and explainability. Finally, you'll be able to deploy your transformer models in a production environment. By the end of this NLP book, you'll have learned how to use Transformers to solve advanced NLP problems using advanced models.

What you will learn

• Explore state-of-the-art NLP solutions with the Transformers library
• Train a language model in any language with any transformer architecture
• Fine-tune a pre-trained language model to perform several downstream tasks
• Select the right framework for the training, evaluation, and production of an end-to-end solution
• Get hands-on experience in using TensorBoard and Weights & Biases
• Visualize the internal representation of transformer models for interpretability

Who this book is for

This book is for deep learning researchers, hands-on NLP practitioners, as well as ML/NLP educators and students who want to start their journey with Transformers. Beginner-level machine learning knowledge and a good command of Python will help you get the best out of this book.

Product Details

ISBN-13:	9781801077651
Publisher:	Packt Publishing
Publication date:	09/15/2021
Pages:	374
Product dimensions:	7.50(w) x 9.25(h) x 0.77(d)

About the Author

Savaş Yıldırım graduated from the Istanbul Technical University Department of Computer Engineering and holds a Ph.D. degree in Natural Language Processing (NLP). Currently, he is an associate professor at the Istanbul Bilgi University, Turkey, and is a visiting researcher at the Ryerson University, Canada. He is a proactive lecturer and researcher with more than 20 years of experience teaching courses on machine learning, deep learning, and NLP. He has significantly contributed to the Turkish NLP community by developing a lot of open source software and resources. He also provides comprehensive consultancy to AI companies on their R&D projects. In his spare time, he writes and directs short films, and enjoys practicing yoga.

Meysam Asgari-Chenaghlu is an AI manager at Carbon Consulting and is also a Ph.D. candidate at the University of Tabriz. He has been a consultant for Turkey's leading telecommunication and banking companies. He has also worked on various projects, including natural language understanding and semantic search.

Table of Contents

Preface

Section 1 Introduction - Recent Developments in the Field, Installations, and Hello World Applications

1 From Bag-of-Words to the Transformer

Technical requirements 4

Evolution of NLP toward Transformers 5

Understanding distributional semantics 8

BoW implementation 8

Overcoming the dimensionality problem 10

Language modeling and generation 11

Leveraging DL 13

Learning word embeddings 14

A brief overview of RNNs 16

LSTMs and gated recurrent units 18

A brief overview of CNNs 22

Overview of the Transformer architecture 26

Attention mechanism 26

Multi-head attention mechanisms 29

Using TL with Transformers 34

Summary 36

References 37

2 A Hands-On Introduction to the Subject

Technical requirements 41

Installing Transformer with Anaconda 41

Installation on Linux 42

Installation on Windows 43

Installation on macOS 45

Installing TensorFlow, PyTorch, and Transformer 46

Installing using Google Colab 47

Working with language models and tokenizers 48

Working with community-provided models 51

Working with benchmarks and datasets 54

Important benchmarks 54

Accessing the datasets with an Application Programming interface 57

Benchmarking for speed and memory 67

Summary 72

Section 2 Transformer Models - From Autoencoding to Autoregressive Models

3 Autoencoding Language Model

Technical requirements 76

BERT - one of the autoencoding language models 76

BERT language model pretraining tasks 77

A deeper look into the BERT language model 78

Autoencoding language model training for any language 81

Sharing models with the community 93

Understanding other autoencoding models 93

Introducing ALBERT 94

RoBERTa 97

ELECTRA 100

Working with tokenization algorithms 101

Byte pair encoding 103

WordPiece tokenization 104

Sentence piece tokenization 105

The tokenizers library 105

Summary 113

4 Autoregressive and Other Language Models

Technical requirements 116

Working with AR language models 116

Introduction and training models with GPT 117

Transformer-XL 120

XLNet 121

Working with Seq2Seq models 121

T5 122

Introducing BART 123

AR language model training 126

NLG using AR models 131

Summarization and MT fine-tuning using simpletransformers 135

Summary 138

References 138

5 Fine-Tuning Language Models for Text Classification

Technical requirements 142

Introduction to text classification 143

Fine-tuning a BERT model for single-sentence binary classification 144

Training a classification model with native PyTorch 153

Fine-tuning BERT for multi-class classification with custom datasets 159

Fine-tuning the BERT model for sentence-pair regression 166

Utilizing run_glue.py to fine-tune the models 172

Summary 173

6 Fine-Tuning Language Models for Token Classification

Technical requirements 176

Introduction to token classification 176

Understanding NER 177

Understanding POS tagging 178

Understanding QA 179

Fine-tuning language models for NER 180

Question answering using token classification 189

Summary 199

7 Text Representation

Technical requirements 202

Introduction to sentence embeddings 202

Cross-encoder versus bi-encoder 204

Benchmarking sentence similarity models 205

Using BART for zero-shot learning 209

Semantic similarity experiment with FLAIR 213

Average word embeddings 216

RNN-based document embeddings 217

Transformer-based BERT embeddings 217

Sentence-BERT embeddings 218

Text clustering with Sentence-BERT 221

Topic modeling with BERTopic 225

Semantic search with Sentence-BERT 227

Summary 231

Further reading 231

Section 3 Advanced Topics

8 Working with Efficient Transformers

Technical requirements 236

Introduction to efficient, light, and fast transformers 236

Implementation for model size reduction 238

Working with DistilBERT for knowledge distillation 239

Pruning transformers 241

Quantization 244

Working with efficient self-attention 246

Sparse attention with fixed patterns 246

Learnable patterns 262

Low-rank factorization, kernel methods, and other approaches 267

Summary 267

References 268

9 Cross-Lingual and Multilingual Language Modeling

Technical requirements 270

Translation language modeling and cross-lingual knowledge sharing 271

XLM and mBERT 273

mBERT 273

XLM 274

Cross-lingual similarity tasks 278

Cross-lingual text similarity 278

Visualizing cross-lingual textual similarity 282

Cross-lingual classification 286

Cross-lingual zero-shot learning 291

Fundamental limitations of multilingual models 295

Fine-tuning the performance of multilingual models 296

Summary 299

References 299

10 Serving Transformer Models

Technical requirements 302

fastAPI Transformer model serving 303

Dockerizing APIs 306

Faster Transformer model serving using TFX 307

Load testing using Locust 310

Summary 314

References 314

11 Attention Visualization and Experiment Tracking

Technical requirements 316

Interpreting attention heads 316

Visualizing attention heads with exBERT 318

Multiscale visualization of attention heads with BertViz 323

Understanding the inner parts of BERT with probing classifiers 334

Tracking model metrics 334

Tracking model training with TensorBoard 335

Tracking model training live with W&B 339

Summary 344

References 344

Why subscribe? 345

Tracking model metrics 334

Other Books You May Enjoy

Index