Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics
336
Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics
336eBook
Related collections and offers
Overview
Measuring the User Experience provides the first single source of practical information to enable usability professionals and product developers to effectively measure the usability of any product by choosing the right metric, applying it, and effectively using the information it reveals.
Authors Tullis and Albert organize dozens of metrics into six categories: performance, issues-based, self-reported, web navigation, derived, and behavioral/physiological. They explore each metric, considering best methods for collecting, analyzing, and presenting the data. They provide step-by-step guidance for measuring the usability of any type of product using any type of technology.
This book is recommended for usability professionals, developers, programmers, information architects, interaction designers, market researchers, and students in an HCI or HFE program.
• Presents criteria for selecting the most appropriate metric for every case• Takes a product and technology neutral approach • Presents in-depth case studies to show how organizations have successfully used the metrics and the information they revealed
Product Details
| ISBN-13: | 9780080558264 |
|---|---|
| Publisher: | Elsevier Science |
| Publication date: | 07/27/2010 |
| Series: | Interactive Technologies |
| Sold by: | Barnes & Noble |
| Format: | eBook |
| Pages: | 336 |
| File size: | 5 MB |
About the Author
Thomas S. (Tom) Tullis retired as Vice President of User Experience Research at Fidelity Investments in 2017. Tom was also an Adjunct Professor in Human Factors in Information Design at Bentley University since 2004. He joined Fidelity in 1993 and was instrumental in the development of the company’s User Research department, whose facilities include state-of-the-art Usability Labs. Prior to joining Fidelity, he held positions at Canon Information Systems, McDonnell Douglas, Unisys Corporation, and Bell Laboratories. He and Fidelity’s usability team have been featured in a number of publications, including Newsweek, Business 2.0, Money, The Boston Globe, The Wall Street Journal, and The New York Times.
Read an Excerpt
Measuring the User Experience
Collecting, Analyzing, and Presenting Usability MetricsBy Tom Tullis Bill Albert
MORGAN KAUFMANN PUBLISHERS
Copyright © 2008 Elsevier Inc.All right reserved.
ISBN: 978-0-08-055826-4
Chapter One
Introduction
The goal of this book is to show how usability metrics can be a powerful tool for successfully evaluating the user experience for any product. When some people think about usability metrics, they feel overwhelmed by complicated formulas, contradictory research, and advanced statistical methods. We hope to demystify much of the research and focus on the practical application of usability metrics. We'll walk you through a step-by-step approach to collecting, analyzing, and presenting usability metrics. We'll help you choose the right metrics for each situation or application, and show you how to use them to produce reliable, actionable results without breaking your budget. We'll give you guidelines and tips for analyzing a wide range of usability metrics and provide many different examples of how to present usability metrics to others in simple and effective ways.
Our intention is to make this book a practical, how-to guide about measuring the usability of any product. We aren't going to give you a lot of formulas; in fact, there are very few. The statistics will be fairly limited, and the calculations can be done easily in Excel or some other common software package or web application. Our goal is to give you the tools you need to evaluate the usability of any product, without overwhelming you with unnecessary details.
This book is both product- and technology-neutral. The usability metrics we describe can be used for practically any type of product and any type of technology. This is one of the great features of usability metrics: They aren't just for websites or any single technology. For example, task success and satisfaction are equally valid whether you evaluate a website, a treadmill, or a toaster. More advanced technologies, such as websites, mobile phones, software, and consumer electronics, are of special concern because they're generally more complicated, but the basic premise remains the same.
The "half-life" of usability metrics is much greater than any specific design or technology. Despite all the changes in technology, the metrics essentially stay the same. Some metrics may change with the development of new technologies to measure usability, but the underlying phenomena being measured don't change. where exactly a user is looking on the screen. Now, with the latest advances in eye-tracking technology, measurement has become much easier and far more accurate.
So why did we write this book? There's certainly no shortage of books on human factors, statistics, experimental design, and usability methods. Some of those books even cover the more common usability metrics. Does a book that focuses entirely on usability metrics even make sense? Obviously, we think so. In our (humble) opinion, this book makes five unique contributions to the realm of usability publications:
* We take a comprehensive look at usability metrics. No other books review so many different usability metrics. We provide details on collecting, analyzing, and presenting nearly every type of usability metric you could possibly use.
* This book takes a practical approach. We assume you're interested in applying usability metrics as part of your job. We don't waste your time with unnecessary details. We want you to be able to use these metrics easily every day. If you're interested in the theoretical side, we point you to additional resources.
* We provide help in making the right decisions about usability metrics. One of the most difficult aspects of a usability professional's job is deciding whether to collect metrics and, if so, which ones to use. We guide you through the decision process so that you find the right metrics for your situation.
* We provide many examples of how usability metrics have been applied within different organizations and how they have been used to address specific usability questions. We also provide in-depth case studies to help you determine how best to use the information revealed by the usability metrics.
* We present usability metrics that can be used with any product or technology. We take a broad view so that these usability metrics can be helpful throughout your career even as technology and products change.
1.1 ORGANIZATION OF THIS BOOK
This book is organized into three main parts. The first one (Chapters 1–3) provides background information needed to get up to speed on usability metrics. This part is intended for those who are less familiar with usability, data analysis, or statistics.
* Chapter 1 provides an overview of usability and usability metrics. We define user experience, usability, and different types of usability metrics; discuss the value of measuring the user experience; and dispel some of the common myths about usability metrics.
* Chapter 2 includes background information on usability data and some basic statistical concepts. We walk you through a step-by-step process to set up a usability study using different metrics and provide a guide for performing common statistical procedures related to different usability methods.
* Chapter 3 focuses on planning a usability study, including defining participant goals and study goals and choosing the right metrics for a wide variety of situations.
The second part (Chapters 4–9) reviews five general types of usability metrics, as well as some special topics that don't fall neatly into any single type. For each metric, we explain what it is, when to use it, and when not to use it. We show you how to collect the data and different ways to analyze and present it. We provide examples of how it has been used in real-world usability studies.
* Chapter 4 covers various types of performance metrics, including task success, time on task, errors, efficiency, and ease of learning. These metrics are grouped under an "umbrella" of performance because they measure different aspects of the user's behavior.
* Chapter 5 looks at measuring usability issues. Usability issues can easily be quantified by measuring the frequency, severity, and type of issue. We also discuss some of the debates about appropriate sample sizes and how to capture usability issues reliably.
* Chapter 6 focuses on self-reported metrics, such as satisfaction, expectations, ease-of-use ratings, usefulness, and awareness. Self-reported metrics are based on what users share about their experiences, not what the usability specialist measures about their actual behaviors.
* Chapter 7 is devoted to behavioral and physiological metrics. These metrics include eye-tracking, facial expressions, and various measures of stress. All of these metrics capture something about how the body behaves as a result of the experience of working with a user interface.
* Chapter 8 discusses how to combine different types of metrics and derive new metrics. Sometimes it's helpful to get an overall assessment of the usability of any product. This global assessment is achieved by combining different types of metrics into a single usability score, summarizing them in a usability scorecard, or comparing them to expert performance.
* Chapter 9 presents special topics that we believe are important but that don't fit squarely into one of the five general categories. These include A/B testing on a live website, card-sorting data, Six Sigma, accessibility data, and return on investment (ROI).
The third part (Chapters 10–11) shows how usability metrics are put into practice. In this part, we highlight how usability metrics are actually used within different types of organizations and how to promote the use of metrics within an organization.
* Chapter 10 presents six case studies. Each case study reviews how different types of usability metrics were used, how the data were collected and analyzed, and the results. These case studies were drawn from usability practitioners in various types of organizations, including consulting, government, industry, and not-for-profit/education.
* Chapter 11 provides ten steps to help you move forward in using metrics within your organization. We discuss how usability metrics can fit within different types of organizations, practical tips for making metrics work within your organization, and recipes for success.
1.2 WHAT IS USABILITY?
Before we try to measure usability, we should know what it is and what it isn't. There are many definitions of usability—maybe even one for every person involved in the field! We're going to focus on three definitions.
The International Standards Organization (ISO 9241-11) identifies three aspects of usability, defining it as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use."
The Usability Professionals Association (UPA) definition focuses more on the product development process: "Usability is an approach to product development that incorporates direct user feedback throughout the development cycle in order to reduce costs and create products and tools that meet user needs."
In his popular book Don't Make Me Think, Steve Krug (2000) provides a simple perspective: "Usability really just means making sure that something works well: that a person of average (or even below average) ability and experience can use the thing—whether it's a website, a fighter jet, or a revolving door—for its intended purpose without getting hopelessly frustrated."
All three of these definitions, as well as other definitions of usability, share some common themes:
* A user is involved.
* That user is doing something.
* That user is doing something with a product, system, or other thing.
Some people distinguish between the terms usability and user experience. Usability is usually considered the ability of the user to use the thing to carry out a task successfully, whereas user experience takes a broader view, looking at the individual's entire interaction with the thing, as well as the thoughts, feelings, and perceptions that result from that interaction. We take a very broad view of usability and examine the entire user experience. Therefore, when we talk about "measuring usability," we're really looking at the entire user experience.
1.3 WHY DOES USABILITY MATTER?
In any casual conversation about usability, most people would agree that it's good to have something that works well and isn't confusing to use. In our years of evaluating products with thousands of test participants, no one has ever complained that something was too easy to use! Everyone has some favorite stories about how something works remarkably well or really terribly. Even as we write this book, we're challenged in formatting the manuscript to give it to the publisher in an acceptable format. Many other stories demonstrate that the usability of something can actually save lives, bankrupt businesses, and have a tremendous impact on society at large.
Usability can sometimes mean the difference between life and death. For example, the health industry is not immune to poor usability. Usability issues abound in medical devices, procedures, and even diagnostic tools. Jakob Nielsen (2005) cites one study that found 22 separate usability issues that contributed to patients receiving the wrong medicine. Similar situations can arise on a regular basis in the workplace or in the home. Just think of the written instructions for such actions as lighting the pilot light on a furnace or installing a new lighting fixture. An instruction that's misunderstood or misread can easily result in property damage, personal injury, or even death. Similar situations arise outside the home: confusing street signs, complicated or poorly designed automobile dashboards, or distracting devices such as vehicle navigation systems or even cell phones.
Sadly, one of the factors involved in a fatal accident on March 2, 2007, in Atlanta, Georgia, may have been poor usability of the high-occupancy vehicle (HOV) lane and its associated signage (see Figure 1.1). A charter bus was carrying the Bluffton University baseball team to their first playoff game in Sarasota, Florida. The bus was driving south in the predawn hours (when it was still dark) on I-75 in the HOV lane, when the driver, who was not from the area, was faced with the signs in Figure 1.1.
(Continues...)
Excerpted from Measuring the User Experience by Tom Tullis Bill Albert Copyright © 2008 by Elsevier Inc.. Excerpted by permission of MORGAN KAUFMANN PUBLISHERS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Introduction; Background: Data Types; Sampling Size; Experimental Design; Data Analysis. Overview of Usability Metrics: Types of Metrics; Methods and Metrics; Summative vs. Formative; Choosing Appropriate Metrics. Performance Metrics: Task Success; Completion Time; Errors; Efficiency (clicks, pages, steps, etc.). Issues-Based Metrics: What is a Usability Issue; Severity Ratings; Test Biases; Reporting Positive Issues. Peferential-Based Metrics: Satisfaction; Ease of Use, Usefulness; Expectations; Standard Questionnaires. Web Navigation Metrics: Web-page Click-through Rates; Web page Abandonment Rates. Derived Metrics: Task-based; Aggregate. Observational Metrics: Eye Movements; Stress; Facial Expressions; Other Observational Metrics. Case Studies. Special Topics: Six Sigma and Usability; Automated Methods; Discount Techniques; Server Log Analysis; A/B Testing. Conclusion: Communication to Management; Cost Justification; Industry Trends.What People are Saying About This
Find practical, detailed instructions on how to measure the user experience for web sites, applications, and products of all kinds