Read an Excerpt
Moderating Usability Tests
Principles and Practices for Interacting
By Joseph S. Dumas Beth A. Loring
MORGAN KAUFMANN PUBLISHERS
Copyright © 2008 Elsevier Inc.
All right reserved.
ISBN: 978-0-08-055827-1
Chapter One
Introduction
This book grew out of a tutorial that we presented at several Usability Professionals' Association (UPA) conferences starting in 2004. The success of the tutorial and the positive reaction of the participants led us to expand the scope and publish it as a book. It is intended for both newcomers to usability testing and experienced practitioners who want to reflect on their own practices.
1.1 WHY THIS BOOK?
Usability testing is now accepted as the evaluation method that influences product design the most (Rosenbaum, Rohm, & Homburg, 2000). To a large extent, successful usability testing depends on the skills of the person moderating the test. Most practitioners, however, learn how to moderate with little formal training and almost no feedback on their technique. They watch a few test sessions, moderate a few under supervision, and then proceed without further evaluation. It's not surprising that researchers have found that moderators, even in the same organization, don't follow the same practices (Boren & Ramey, 2000).
With the exception of short sections in Rubin (1994) and Snyder (2003), the literature on usability testing provides only general advice about interacting with test participants. In addition, there are no sources about either general principles or specific guidelines that provide a rationale for what moderators should or shouldn't do. Maybe this is because many people consider it part of the "art" of testing—a skill that is difficult to teach. The lack of relevant literature became a problem as we tried to teach people how to moderate. Therefore, in this book we go into depth about what to do (and what not to do) while interacting with test participants, and we present a set of rules of good practice.
Although one paper discusses situations that happen infrequently (that is, Loring & Patel, 2001), this book focuses on the common situations you will encounter in typical usability test sessions.
Over time, as we taught people how to moderate test sessions, we recognized the need to address the difficulties that moderators have with the following issues:
* learning how to overcome the initial anxiety that all new moderators experience * staying in control of the test session * being friendly but resisting becoming friends with participants * dealing with personal—rather than the participant's—anxiety while struggling with usability problems * balancing the trade-off between respecting participants' rights and pushing them to keep working * knowing when and how to provide assistance to participants * knowing how to probe for more information in an unbiased way
We hope that by addressing these and other issues in this book, we are making a valuable contribution to the field.
1.2 WHAT IS USABILITY TESTING?
Usability testing is a systematic way of observing actual and potential users of a product as they work with it under controlled conditions. It differs from other evaluation methods (such as quality assurance testing and product demonstrations) in that users try to complete tasks with a product on their own, with little help. Usability testing can be conducted in a laboratory, in a conference room, in the participant's environment, or remotely. Companies use this method to evaluate software, hardware, documentation, web sites, or any product with a user interface.
People are recruited, and usually compensated, to participate in the sessions. The goal of the study may be to uncover as many usability "bugs" as possible or to compare the usability of two products. A typical test session involves one participant and one moderator, lasts one to two hours, and includes these tasks:
* greeting the participant * explaining the participant's rights and having him or her sign an informed consent * explaining how the session will proceed * guiding participants through a set of carefully selected tasks using the product, usually while thinking aloud * recording data in one or more ways * asking participants to summarize their experience
1.3 THE IMPORTANCE OF MODERATING SKILLS
Usability testing is like many things—easy to do, but hard to do well, and improved only through practice. The moderator's manner of interaction with participants from first recruiting them to thanking them at the end of the session is critical to the success of the test, the validity of the data, and the reputation of the moderator's organization. A test moderator must be unbiased and neutral regarding the product while being open and approachable to participants. This mixture of neutrality and approachability can be difficult to accomplish.
1.4 THE GOLDEN RULES OF MODERATING
Because most practitioners have learned only by example, they have seldom thought about the underlying principles of interacting with test participants. This led us to create the golden rules of interacting for moderators of usability tests. Our rules attempt to capture the best practices of interacting and provide a rationale for the practices described in the rules. The golden rules are:
1. Decide how to interact based on the purpose of the test.
2. Respect the participants' rights.
3. Remember your responsibility to future users.
4. Respect the participants as experts, but remain in charge.
5. Be professional, which includes being genuine.
6. Let the participants speak!
7. Remember that your intuition can hurt and help you.
8. Be unbiased.
9. Don't give away information inadvertently.
10. Watch yourself to keep sharp.
These rules will help you deal with new situations that you may encounter. We discuss these rules in detail in chapters 3 and 4.
1.5 CULTURAL POINTS OF VIEW
As we developed the material for this book, it became clear that we have the perspective and assumptions of usability specialists from the United States. In the book, we don't pretend to be otherwise. Instead, we state our biases up front and, where possible, discuss the need to be sensitive to cultural differences. In some cases, we cite literature about ways people from various cultures might react, but we don't attempt to guess how moderators and participants from other cultures might react. While we cite the few studies that describe ways that people from other cultures react to testing situations, we don't extrapolate beyond those studies by guessing how people from other cultures may react to our golden rules. The role of culture in usability testing is a new and interesting topic that is just beginning to be explored. It would be wonderful if practitioners from other cultures would create localized sets of golden rules!
1.6 ABOUT THE SIDEBARS IN THIS BOOK
Throughout the book, we offer sidebars to supplement the text. There are two types of sidebar. The first type presents short descriptions of relevant research studies that shed light on the topic at hand. Entitled "What the Research Says," each focuses on the practical implications of one published study.
The second type presents interviews with a variety of moderators at different organizations to allow them to express their views about a topic. Some of the moderators are new to moderating and others are quite experienced. Hence, these are entitled "Interviews with a New [or Experienced] Moderator." We found that stating the principles in moderators' own words can be more informative and colorful than if we described them ourselves.
1.7 ABOUT THE VIDEOS THAT ACCOMPANY THIS BOOK
Perhaps one reason that interacting with test participants has not been adequately addressed in the literature is that it doesn't lend itself well to description only on paper. To fully grasp both the art and the craft of test moderation, it helps to observe the interactions—including facial expressions, body language, tone of voice, and numerous other factors—that can have a direct impact on the test outcome but are difficult to convey in words.
To provide another dimension, we created a number of short videos to accompany this book. The role-playing videos show both good and not-so-good moderating practices. The filmed discussion by a panel of usability experts gives their opinions about what they see in the videos and discusses trade-offs that you might consider in each situation.
The videos, available on the book's web site were filmed in the Bentley College Design and Usability Center laboratory and were carefully scripted to illustrate the points we want to make.
Chapter 11 describes the type and content of the videos and suggests how to get the most value from them.
1.8 ABOUT THE COMPANION WEB SITE
The companion web site for this book is www.mkp.com/ moderatingtests. In addition to the videos, you will find downloadable versions of the sample test materials in the book, such as the checklists, forms, and recruiting scripts.
Chapter Two
Getting started as a test moderator
This chapter provides some guidance to moderators who are just starting out, but our comments may also be useful for experienced moderators who need to refresh their memories or who are training others. We discuss the attributes of a great moderator, types of testing, moderator (sometimes conflicting) roles, the basics of running a test, and finally some ways to get started quickly as a moderator. If you're familiar with usability testing, most of this material will be a review.
2.1 WHAT MAKES A GREAT MODERATOR?
Moderating usability tests is not as easy as it looks. If you're reading this book, you have probably seen a number of test sessions and may even have moderated some (or many) yourself. Some moderators make it look easy and others are so nervous that you feel nervous just watching them. Some moderators are able to elicit wonderful insights from test participants; others simply go through the motions. Some remain neutral and others seem to lead or bias the participants. Why is there so much variation?
Becoming a great test moderator takes four things:
1. Understanding the basics of usability testing
2. Interacting well with test participants (our golden rules)
3. Ability to establish and maintain rapport with participants
4. Lots of practice
2.1.1 Understanding usability testing
To be an effective moderator, you need a firm understanding of what usability testing is all about. You need to know the following:
* the purpose of the test that you are moderating. The purpose determines how much and when to interact with participants. * how usability testing differs from other evaluation methods. The emphasis here is on obtaining valid data from a small sample of typical users, and letting the users speak. * how tests are designed. For example, design might be important in presenting the product and related tasks in the context of their intended use. * how tests are planned. Every task has a purpose and the order of tasks may be important to the test's validity. Sometimes it's important to probe for an understanding of the concept behind a user interface component. Stopping a task before the participant has completed it may be required in some tests but not in others. * how and why data is being collected. The emphasis may be on the quantitative measures or the qualitative measures, depending on the test goals.
Fundamental knowledge of usability testing is vital to understanding the material in this book, so if you need more information, we suggest that you read a book on the basics of usability tests (e.g., Dumas & Redish, 1999; Rubin, 1994) or take one of the many courses available.
2.1.2 The basics of interacting
The ways that practitioners interact with test participants has a huge effect on test results. Moderators (or their team mates) interact with participants throughout the testing process, including
* selecting them for the study.
* greeting them when they arrive.
* providing an introduction to the test.
* guiding them as they are performing tasks.
* interacting during the post-test debriefing.
* thanking them and giving them their compensation.
(Continues...)
Excerpted from Moderating Usability Tests by Joseph S. Dumas Beth A. Loring Copyright © 2008 by Elsevier Inc.. Excerpted by permission of MORGAN KAUFMANN PUBLISHERS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
