How Google Tests Software

by James Whittaker, Jason Arbon, Jeff Carollo

Overview

2012 Jolt Award finalist!

 

Pioneering the Future of Software Test

 

Do you need to get it right, too? Then, learn from Google. Legendary testing expert James Whittaker, until recently a Google testing leader, and two top Google experts reveal exactly how Google tests software, offering brand-new best practices you can use even if you’re not quite Google’s size…yet!

 

Breakthrough Techniques You Can Actually Use

 

Discover 100% practical, amazingly scalable techniques for analyzing risk and planning tests…thinking like real users…implementing exploratory, black box, white box, and acceptance testing…getting usable feedback…tracking issues…choosing and creating tools…testing “Docs & Mocks,” interfaces, classes, modules, libraries, binaries, services, and infrastructure…reviewing code and refactoring…using test hooks, presubmit scripts, queues, continuous builds, and more. With these techniques, you can transform testing from a bottleneck into an accelerator–and make your whole organization more productive!

 

Product Details

ISBN-13:	9780132851558
Publisher:	Pearson Education
Publication date:	03/21/2012
Sold by:	Barnes & Noble
Format:	eBook
Pages:	320
File size:	20 MB
Note:	This product may take a few minutes to download.
Age Range:	18 Years

About the Author

James Whittaker is an engineering director at Google and has been responsible for testing Chrome, maps, and Google web apps. He used to work for Microsoft and was a professor before that. James is one of the best-known names in testing the world over.

Jason Arbon is a test engineer at Google and has been responsible for testing Google Desktop, Chrome, and Chrome OS. He also served as development lead for an array of open-source test tools and personalization experiments. He worked at Microsoft prior to joining Google.

Jeff Carollo is a software engineer in test at Google and has been responsible for testing Google Voice, Toolbar, Chrome, and Chrome OS. He has consulted with dozens of internal Google development teams helping them improve initial code quality. He converted to a software engineer in 2010 and leads development of Google+ APIs. He also worked at Microsoft prior to joining Google."

Table of Contents

Foreword by Alberto Savoia     xiii
Foreword by Patrick Copeland     xvii
Preface     xxiii

Chapter 1: Introduction to Google Software Testing     1
Quality?Test     5
Roles     6
Organizational Structure     8
Crawl, Walk, Run     10
Types of Tests     12

Chapter 2: The Software Engineer in Test     15
The Life of an SET     17
Development and Test Workflow     17
Who Are These SETs Anyway?     22
The Early Phase of a Project     22
Team Structure     24
Design Docs     25
Interfaces and Protocols     27
Automation Planning     28
Testability     29
SET Workflow: An Example     32
Test Execution     40
Test Size Definitions     41
Use of Test Sizes in Shared Infrastructure     44
Benefits of Test Sizes     46
Test Runtime Requirements     48
Case 1: Change in Common Library     52
Test Certified     54
An Interview with the Founders of the Test
Certified Program     57
Interviewing SETs     62
An Interview with Tool Developer Ted Mao     68
An Interview with Web Driver Creator Simon Stewart     70

Chapter 3: The Test Engineer     75
A User-Facing Test Role     75
The Life of a TE     76
Test Planning     79
Risk     97
Life of a Test Case     108
Life of a Bug     113
Recruiting TEs     127
Test Leadership at Google     134
Maintenance Mode Testing     137
Quality Bots Experiment     141
BITE Experiment     153
Google Test Analytics     163
Free Testing Workflow     169
External Vendors     173
An Interview with Google Docs TE Lindsay Webster     175
An Interview with YouTube TE Apple Chow     181

Chapter 4: The Test Engineering Manager     187
The Life of a TEM     187
Getting Projects and People     189
Impact     191
An Interview with Gmail TEM Ankit Mehta     193
An Interview with Android TEM Hung Dang     198
An Interview with Chrome TEM Joel Hynoski     202
The Test Engineering Director     206
An Interview with Search and Geo Test Director
Shelton Mar     207
An Interview with Engineering Tools Director
Ashish Kumar     211
An Interview with Google India Test Director Sujay Sahni     214
An Interview with Engineering Manager Brad Green     219
An Interview with James Whittaker     222

Chapter 5: Improving How Google Tests Software     229
Fatal Flaws in Google’s Process     229
The Future of the SET     231
The Future of the TE     233
The Future of the Test Director and Manager     234
The Future of Test Infrastructure     234
In Conclusion     235

Appendix A: Chrome OS Test Plan     237
Overview of Themes     237
Risk Analysis     238
Per-Build Baseline Testing     239
Per-LKG Day Testing     239
Per-Release Testing     239
Manual Versus Automation     240
Dev Versus Test Quality Focus     240
Release Channels     240
User Input     241
Test Case Repositories     241
Test Dashboarding     241
Virtualization     241
Performance     242
Stress, Long-Running, and Stability     242
Test Execution Framework (Autotest)     242
OEMs     242
Hardware Lab     242
E2E Farm Automation     243
Testing the Browser AppManager     243
Browser Testability     243
Hardware     244
Timeline     244
Primary Test Drivers     246
Relevant Documents     246

Appendix B: Test Tours for Chrome     247
The Shopping Tour     247
The Student Tour     248
Suggested Areas to Test     248
The International Calling Tour     249
Suggested Areas to Test     249
The Landmark Tour     249
Suggested Landmarks in Chrome     249
The All Nighter Tour     250
Suggested Areas to Test     250
The Artisan’s Tour     251
Tools in Chrome     251
The Bad Neighborhood Tour     251
Bad Neighborhoods in Chrome OS     251
The Personalization Tour     252
Ways to Customize Chrome     252

Appendix C: Blog Posts on Tools and Code     253
Take a BITE out of Bugs and Redundant Labor     253
Unleash the QualityBots     255
RPF: Google’s Record Playback Framework     257
Google Test Analytics—Now in Open Source     260
Comprehensive     260
Quick     260
Actionable     260
Sustained Value     260

Index     265