Table of Contents
Foreword xxiii
Reader Support of This Book xxvii
Part I Introduction 1
Chapter 1 Why Another Book About Testing? 3
Many Cultures of Testing 3
Why Us? 6
Why Testing? 7
Why Testers? 9
By The Way, Now Everyone is a Tester. 11
Why Not “Traditional Testing?” 11
Part II Rapid Software Testing Methodology 15
Chapter 2 Foundation 17
The Meaning of Testing 17
Testing vs. Checking 18
Testing vs. Performing a Test 20
Deep vs. Shallow Testing 20
Narrow vs. Broad Testing 21
Our Vision for RST 22
Design Features 25
Foundational Ideas 26
Who We Are 27
Practitioners and Clients 28
Students and Experts 30
What We Study 32
Systems and Models 32
Products and Users 35
Quality and Risk 36
How We Manage 39
Context and Mission 39
Process and Methodology 41
Roles and Trading Zones 43
Tacit and Explicit Knowledge 45
Freedom and Formality 48
Spontaneity and Deliberation 50
Iteration and Emergence 52
Control and Legibility 54
Stories and Strategy 56
Heuristics and Skills 57
Activities and Enoughness 59
How We See 61
Stance and Distance 61
Focusing and Defocusing 64
Experience and Instrumentation 65
Sensemaking and Unconscious Bias 66
Experiments and Demonstrations 70
Assessment and Measurement 71
Chapter 3 How to Do a Test 75
The Process of Testing 75
Our View of the Scientific Method 76
The Fundamental Method of Testing 77
Why Do We Say “Explore” Instead of “Record” or “Measure”? 80
How Is the Reality of the Product Explored? 81
The Anatomy of a Check 84
The Meaning of a Check 86
Fail Is Not an Option (for Good Reporting) 88
The Anatomy of a Test 89
Coverage 90
Intentional Coverage 92
Incidental Coverage 93
To Cover Is to Sample 94
How Not to Think About Equivalence Classes 95
A Better Way to Think About Equivalence Classes 97
A Better Way to Think About Boundaries, Too 98
Pushing Limits 100
Oracles 103
Oracles May Take Many Forms 105
Procedures 108
Designing a Test Procedure 109
Anatomy of “One Step” of a Procedure for a Test 112
How To Go Wrong When Performing a Test 116
The Art of Bug Reporting 118
Normal Bug or Enhancement Request? 118
How to Investigate a Bug 119
How to Investigate Intermittent Problems 120
Be Comforted: The Cause Is Probably Not Evil Spirits 121
General Suggestions for Investigating Intermittent Problems 122
Considering the Causes of Intermittent Problems 123
How to Report a Bug 128
Formal vs. Informal Bug Reporting 130
Elements of a Basic Formal Bug Report 131
Give the Bug Report a Good Focus 133
Assessing the Significance of a Bug 133
The Bug Pipeline 134
Chapter 4 How to Do a Test Strategy 137
The Structure of Test Strategy 138
Welcome to Your New “Normal” 142
Pleasing vs. Disturbing 143
Easy vs. Stressful 143
Acceptable vs. Disallowed 144
Natural vs. Contrived 144
Us vs. Them 145
Typical vs. Unusual 145
Standard vs. Special 146
“This Is Fine.” 146
First of All Things: Center Yourself 148
Developing a Strategy Is an Ongoing Exploration 149
Twelve Test Strategy Entry Points 150
What Are You Here For? 150
What Do You Need to Learn? 150
What Is Happening Right Now? 150
How Is Your Testing Constrained? 151
What Testing Has Been Done Already? 151
How Is the Product Being Built? 152
What Is the Product? 152
How Important Is Your Testing? 153
How Will People Most Likely Use the Product? 154
What Testing Is Easy to Do? 154
What Do People Expect You to Do? 155
What’s Fun to Do? 155
Seasons of Testing: Strategy Throughout the Project Cycle 156
Regression Testing May Not Be What You Think 158
The Challenge of Alignment: Strategy Is Different for Testers and Builders 159
Testability Enables Test Strategy 163
Good Test Strategy Is More Than Just Following Behind Development 164
Practices Worth Practicing 167
Make a Product Coverage Outline 167
Apply the Heuristic Test Strategy Model 169
Organize Strategy According to Risk 171
Risk Analysis Is Mostly Conversation and Consensus 173
Brainstorming Using the Four-Part Risk Story 175
Putting It All Together: Lenses of Testing 179
Be a Problem-Solver, Not a Task-Doer 181
A Strategy Example: Bibliography Generator 181
Context 182
Strategy 182
Strategy Example #2: A More Normalized Version 184
Activity Types Mentioned in the Table 186
What About Oracles? 187
Final Thought 187
Chapter 5 How to Account for Your Work 189
Telling a Compelling Story 190
Note-taking Is a Core Testing Skill 193
Jon Bach’s PROOF Heuristic 194
Safety Language 195
Telescoping Reports 195
James’s Low-Tech Testing Dashboard 196
Stories Mediate Software Projects 198
High Alignment vs. Low Alignment 200
The Temptation to Coerce 201
Artifact-based vs. Activity-based Management 204
Artifact-Based Management 204
Beware of Counting Incommensurable Artifacts 205
The Basic Problem of Artifact-Based Management 207
Activity-based Management 208
Session-Based Test Management 209
Thread-Based Test Management 212
Please Don’t Fake Your Testing 213
Chapter 6 How to Apply Automation in Testing 215
Robots! Help! 216
The Trouble with “Automation” 217
In RST, we don’t call it test automation 219
yet we explore many ways to use tools 220
Consider Augmented Experiential Testing 222
Beware of Oversimplified Output Checking 223
we incorporate tools incrementally and opportunistically 229
Consider Applying a Blink Oracle 231
Notice the Hidden Costs of Automation 232
Consider Recruiting a Dedicated Toolsmith 235
How Testers and Toolsmiths Can Work Together 236
.and we promote testability, so tools work better 237
Classic Traps of Automation in Testing 237
Traps of Ignorance 237
Scripting Trap 238
Trusting Trap 238
Atrophy Trap 238
Obscurity Trap 239
Traps of Economy 239
Shallowness Trap 239
Testability Trap 240
Maintenance Trap 240
Sunk Cost Trap 241
Learning Curve Trap 241
Traps of Alignment 241
Rathole Trap 242
Legibility Trap 242
Harmony Trap 242
Considerations for GUI-level Automation 243
First Things First 243
Programmatic Access to the GUI Can be Difficult to Achieve 244
Variation in the GUI Multiplies the Cost and Trouble 245
GUI automation means you are simulating users. How good is that simulation? 247
Considerations for Adopting a Tool 249
Capabilities and Power 249
Effort to Operate 250
Feasibility of Adoption 251
Learning, Troubleshooting, and Support 251
The Golden Rule of Tool Adoption 252
Chapter 7 How to Approach AI and Testing 255
What If We Could Test by Magic? 255
How Is Modern AI Special? 259
Problems to Look For with AI in Testing 262
Transpection: A Basic Skill for Collaborating with AI 264
Beware of the Productivity Paradox 267
Are People Magic Boxes? Can Be 267
Part III Application and Customization 269
Chapter 8 Prospective Testing 271
Notice Something Before We Say Anything More 275
But Why Bother Doing This at All? 277
How to Do Prospective Testing 278
Who Is Involved in Prospective Testing? 278
What Does Prospective Testing Look Like? 280
When Does Prospective Testing Happen? 282
How Can a BA or Developer Prepare for Prospective Testing? 282
How Can You (the Tester) Deal with Resistance to Questions during Meetings? 284
“Analysis Paralysis” or “Go Fast and Break Things?” 286
A Cheat Sheet for Prospective Testing 288
What exactly are we talking about? 288
Is this worth discussing here and now? 288
What exactly are we trying to achieve? 288
What influences must we consider? 289
What other features or requirements will be affected? 289
What specific data or conditions must this feature be able to process or work with? 289
What are the merits of different ways of designing or implementing this feature? 290
How will the new feature handle errors or recover from failure? 290
How will we test the new feature once it exists? 291
Chapter 9 Test Reporting (Without Pretentious Metrics) 293
Metrics Are Nothing Without a Story 293
Bad Metrics 295
Elements of Bad Metrics 295
Coverage 297
A Simple Story for Coverage 299
Release Coverage Outline 300
Better Test Reports 300
Sample Test Report 302
Chapter 10 Working with Quality Characteristics 305
Developing the Model 306
Quality Characteristics 308
Internal Quality Characteristics 312
Using the Model 314
Epilogue 315
Chapter 11 Adventures in Testability 317
Testability on the High Seas 318
Advocating for Testability 320
Step 1: Identify Things That Make It Difficult to Test 320
Step 2: Determine Who Can Help You Solve the Problem 321
Step 3: Sell It (“What’s In It for Them?”) 322
My Job Is Testability 322
Chapter 12 Mindopsy: Dissecting My Thinking as a Tester 325
What Is a Mindopsy? 325
Conversation with the Business Analyst 326
Analysis of the Conversation 329
Tree-Type Notes for This Conversation 334
Summary 337
Chapter 13 Rapid Usability Testing 339
Why Do We Fail to Test for Usability? 340
Understanding Users and Goals 341
Creating Personas and User Goals 341
The Electricians and the Rental Company 342
“Eric,” the Electrician 342
“Rob,” the Rental Guy 343
Living the Persona 344
How to Document a Useful Persona 345
Rapid Usability Testing Basics 345
Recruit Users 346
Plan and Prepare 346
Run the Test 347
I Am Testing an Off-the-Shelf Standard System—What Should I Do? 348
Checking Usability 349
General Checklist of Usability (Nielsen-Norman) 350
Web Content Accessibility Guidelines (WCAG) 350
Chapter 14 RST Meets Signals-Based Testing 353
Signals for Testing 353
How This Looked at Microsoft 354
Self-Taught AI-Generated Workload 355
Analyzing the Signals and Looking for Failure 356
Feature Coverage 356
Crashes and Hangs 356
Assertion Failures 357
Performance Markers 358
Problematic Patterns in Events 358
Getting Started 359
How Does Signals-Based Testing Fit with RST? 361
Does Signals-Based Testing Relate to the HTSM? 363
Chapter 15 Deliberate Practice of Testing Skills 365
The Assignment 366
Creating a Coverage Outline 366
Drafting the Test Strategy 367
Drafting the Test Charters 369
Conducting the Testing 370
Testing the Main Flow 370
Debriefing 371
Testing the Handling of Date and Time 372
Debriefing 372
So What Happened Here? 373
Differing Approaches 373
Part IV Context and Culture 377
Chapter 16 Lessons of the Post Office Horizon Scandal 379
The Post Office Horizon Scandal 380
Dysfunction #1: Management Held a Narrow and Defensive View of the Purposes, Stakeholders, and Quality Standards for Horizon 382
Dysfunction #2: Management Allowed a Terrible Development Process to Overwhelm the Testers 386
Dysfunction #3: Management Systematically Discouraged Collaborative Critical Thinking That Could Have Saved the Project 388
How Bad Was Horizon? 390
The Significance for Testers of the Horizon Issues 393
Why RST Appeals to Me 395
Chapter 17 Sociology and Software Testing 401
Chapter 18 How Testing Looks to Management 415
Introduction 415
Interview 416
Chapter 19 Technical Work Is All About Learning 433
Learning through Play 433
Learning the Importance of the Business Side 435
Learning to Be Helpful in More Than One Way as a Tester 436
Learning That a “Quality” Process Can Interfere with Software Testing 438
Learning to Speak and Persuade 441
Learning Forever 443
Chapter 20 From RST to AST 445
The Start of My Testing Career 445
How Experiencing RST Led to My Passion for Testing 446
Becoming Involved in the CDT Community 448
Building a Close Working Relationship with the AST 449
Chapter 21 Notes on a Seeker’s Journey into Testing 451
“I Trust I Make Myself Obscure” 452
Praxis 452
Core Ideas 453
Words 453
Data, Information, and Knowledge 454
Mindset 456
Focus and Roles 459
Emergence 460
Answers 461
What should programmers know about RST? How is that different from what they think they know? 461
Is there an important difference between a programmer and tester? 462
Why do managers push back against the testers trying to be careful and responsible about their work? 464
What about the trendy idea that “we do not need testers; programmers will do all the testing”? 465
Chapter 22 Becoming an RST Instructor 467
From Development to Management to Training 467
Years of Trying to Be Normal 468
I Found Context-Driven Testing 470
Instructor in the Making 472
The Key to Mastery Is Deliberate Practice 473
Why I Prefer the Context-Driven Approach to Testing 475
Part V Appendices 477
Appendix A Roadmap of Testing 479
Appendix B The Heuristic Test Strategy Model 481
General Test Techniques 482
Project Environment 484
Product Elements 486
Quality Criteria Categories 489
Appendix c Heuristics of Testability 493
Interesting Testability Dynamics 494
Guidewords for Analyzing Testability 495
Appendix D Elements of Excellent Testing 499
Evolving Work Products 501
Testing Skills and Tactics 502
Self-Management Skills and Tactics 502
Collaboration Skills and Tactics 504
Learning Skills and Tactics 504
Test Performance Skills and Tactics 505
Knowledge That Helps 506
Helpful Skills Some Testers Have 507
Exploratory Polarities 508
About the Contributors 509
Index 513
