Experimentation Works: The Surprising Power of Business Experiments

by Stefan H. Thomke

Overview

Don't fly blind. See how the power of experiments works for you.

When it comes to improving customer experiences, trying out new business models, or developing new products, even the most experienced managers often get it wrong. They discover that intuition, experience, and big data alone don't work. What does? Running disciplined business experiments. And what if companies roll out new products or introduce new customer experiences without running these experiments? They fly blind.

That's what Harvard Business School professor Stefan Thomke shows in this rigorously researched and eye-opening book. It guides you through best practices in business experimentation, illustrates how these practices work at leading companies, and answers some fundamental questions: What makes a good experiment? How do you test in online and brick-and-mortar businesses? In B2B and B2C? How do you build an experimentation culture? Also, best practice means running many experiments. Indeed, some hugely successful companies, such as Amazon, Booking.com, and Microsoft, run tens of thousands of controlled experiments annually, engaging millions of users. Thomke shows us how these and many other organizations prove that experimentation provides significant competitive advantage.

How can managers create this capability at their own companies? Essential is developing an experimentation organization that prizes the science of testing and puts the discipline of experimentation at the center of its innovation process. While it once took companies years to develop the tools for such large-scale experiments, advances in technology have put these tools at the fingertips of almost any business professional. By combining the power of software and the rigor of controlled experiments, today's managers can make better decisions, create magical customer experiences, and generate big financial returns.

Experimentation Works is your guidebook to a truly new way of thinking and innovating.

Product Details

ISBN-13:	9781633697102
Publisher:	Harvard Business Review Press
Publication date:	02/18/2020
Pages:	288
Sales rank:	659,194
Product dimensions:	6.10(w) x 9.30(h) x 1.00(d)

About the Author

Stefan H. Thomke, a leading authority on the management of innovation, is the William Barclay Harding Professor of Business Administration at Harvard Business School. He is the author of Experimentation Matters: Unlocking the Potential of New Technologies for Innovation (Harvard Business Review Press, 2003), as well as over one hundred articles, cases, and notes published in books and journals such as Harvard Business Review, Management Science, and MIT Sloan Management Review. Thomke has chaired numerous executive education programs, both at Harvard Business School and in companies around the world. He is a frequent conference speaker and adviser to global business leaders.

You can find the author at:
hbs.edu/faculty/Pages/profile.aspx?facId=6566
linkedin.com/in/stefan-thomke-innovation/

Table of Contents

Preface: A Tribute to the Scientific Method ix

Why I wrote the book (and why you should read it)

Understanding the power of thinking and acting scientifically

Entering a new age of large-scale business experimentation

Introduction: The Business Experimentation Imperative 1

The need for business experiments to drive innovation and profitable growth

Deploying large-scale experimentation as a businesswide practice

Brief introduction to the book's ideas and frameworks

1 Why Experimentation Works 15

The role of business experimentation in innovation

Understanding the power of tools and the experimentation process

Leveraging the operational drivers of high-velocity learning

2 What Makes a Good Business Experiment? 51

The elements of a good business experiment

Seven questions that yield better management decisions

Appreciating the limits of experimentation

3 How to Experiment Online 81

The business value of A/B testing

Leveraging the power of incremental innovation for business performance

Learning the best experimentation practices from leading digital companies

4 Can Your Culture Handle Large-Scale Experimentation? 115

The seven attributes of a true experimentation culture

Diagnosing and addressing cultural obstacles

Adopting a new management model for experimentation organizations

5 Inside an Experimentation Organization 153

The operating model of a true experimentation organization

Democratizing testing through process, management, and cultural discipline

Using technology, scale, and velocity for competitive advantage

6 Becoming an Experimentation Organization 187

The steps to becoming a true experimentation organization

Using seven system levers and the ABCDE maturity framework to analyze your situation

Deploying experimentation tools most effectively

7 Seven Myths of Business Experimentation 213

The myths that undermine experimentation and innovation

Realizing that your actions will lead to opposite reactions

Addressing fallacies that slow down progress

Epilogue: A Brief Look at the Future 225

The future of experimentation is already here

Understanding the role of AI

Adding value to automated testing and decision making

Notes 231

Selected Bibliography 249

Index 257

Acknowledgments 267

About the Author 271

Editorial Reviews

12/16/2019

The secret to business success is to test everything, writes Harvard Business School professor Thomke in this helpful debut about using fresh information, not old assumptions, to drive the development of new products and services. Managers often lack the data to make strategic decisions, he finds, so they fall back on personal experience, emotions, and intuition—and, as one of Thomke’s interviewees tells him, “we see evidence every day that people are terrible at guessing.” Companies that employ rigorous experimentation and make good use of the results, on the other hand, gain a competitive edge, illustrated by such examples as 3M, Booking.com, and Amazon. Thomke finds numerous Jeff Bezos quotes on the subject, including on why “failure and invention are inseparable twins,” and gives significant space to explaining how organizations can build experimentation into workflow and foster a culture that nurtures it. He also breaks down what makes for worthwhile experiments (such as whether a testable hypothesis is involved), and how to use those experiments to make better informed decisions. This is a thorough, encouraging guide for leaders looking for ways to encourage their teams to fail early and often in the service of ultimate success. (Feb.)

Publishers Weekly

Named one of the "Top Ten Technology Books of 2020" by Forbes

Named one of the "Best Business Books 2020: Technology & Innovation" and the Top Shelf Pick by strategy+business

Named one of the "10 Best New Business Books of 2020" by Inc. magazine

"[Experiments] are a powerful tool for curious teams…" — Wall Street Journal

"[Offers new] ideas on what works in the digital experimentation world and it is all illuminating." -- Financial Times

"This is a thorough, encouraging guide for leaders looking for ways to encourage their teams to fail early and often in the service of ultimate success." -- Publisher's Weekly

Advance Praise for Experimentation Works:

"Experimentation Works is required reading for all leaders who want to know how companies like Amazon, Google, and Netflix win through fast-cycle scientific experiments—and how it can be done in their organizations. Making business decisions without experimentation is like bungee jumping without testing the cord. Yet far too many organizations still do this. With Thomke's clear explanations and examples, yours won't be one of them." -- Scott Cook, cofounder and Chairman of the Executive Committee, Intuit

"Stefan Thomke has written a marvelous book that's both thought provoking and highly informative. As CEO, I saw how his research on business experimentation can transform innovation cultures for the better. Experimentation does indeed work. I highly recommend the book." -- Marijn Dekkers, Chairman, Unilever

"Experimentation Works is a masterpiece and a must-read. Thomke, a top academic authority with deep practical insights, explains how low-cost business experiments can revolutionize the way firms design everything from business models to customer experiences. The book's ideas are very powerful, rigorous, and will change how you think about the science and practice of innovation management." -- Eric von Hippel, Professor, Massachusetts Institute of Technology; author, Democratizing Innovation

"Stefan Thomke has written the definitive handbook for executives on how to build a high-impact experimentation capability. Experimentation Works is an amazing book with some sage advice: Experiment or perish. It's absolutely the right message at the right time in a world that has gone digital." -- Jay Larson, CEO, Optimizely

"Stefan Thomke has written a rare book, one that combines deep and original insights with great practical advice. Experimentation Works is about revolutionary changes in how innovation is managed, decisions are made, and organizations are led. Thomke shows us how business experiments fuel innovation and why companies must test at large scale to compete. The book is a landmark achievement." -- Anand Mahindra, Chairman, Mahindra Group

"In an increasingly digital world, CEOs need to drive large-scale scientific business experimentation. If they don't, their companies will die. Companies with strong experimentation systems and cultures will win every time. If you want to learn how this is done, you must read Experimentation Works. The book will change how you think about managing and decision making." -- Mark Okerstrom, President and CEO, Expedia Group

"Being an effective leader today means acting with a sense of urgency: making many decisions quickly under great uncertainty. Experimentation Works describes a vital methodology--large-scale business experimentation--that takes the madness out of that equation and sets you and your organization up for success." -- Ajay Banga, President and CEO, Mastercard

"Business breakthroughs are achieved through the combination of great ideas and effective implementation. Stefan Thomke's work, which I applied enthusiastically during my tenure as CEO, demonstrates that a process of disciplined experimentation is the best way to drive innovation and gain sustained competitive advantage." -- Gary Loveman, former Chairman and CEO, Caesars Entertainment Corporation

"Experimentation Works presents a very compelling case for how companies can leverage the scientific method to drive innovation and reinvent business models for sustainable growth. An immensely valuable guide for leaders of both B2C and B2B companies in an era of accelerating change and disruption." -- Loh Chin-Hua, CEO, Keppel Corporation

From the Publisher