Beautiful Testing: Leading Professionals Reveal How They Improve Software [NOOK Book]


Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed ...

See more details below
Beautiful Testing: Leading Professionals Reveal How They Improve Software

Available on NOOK devices and apps  
  • NOOK Devices
  • Samsung Galaxy Tab 4 NOOK 7.0
  • Samsung Galaxy Tab 4 NOOK 10.1
  • NOOK HD Tablet
  • NOOK HD+ Tablet
  • NOOK eReaders
  • NOOK Color
  • NOOK Tablet
  • Tablet/Phone
  • NOOK for Windows 8 Tablet
  • NOOK for iOS
  • NOOK for Android
  • NOOK Kids for iPad
  • PC/Mac
  • NOOK for Windows 8
  • NOOK for PC
  • NOOK for Mac
  • NOOK for Web

Want a NOOK? Explore Now

NOOK Book (eBook)
$22.99 price
(Save 42%)$39.99 List Price


Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed beautiful ways of testing a wide range of products -- valuable knowledge that you can apply to your own projects.

Here's a sample of what you'll find inside:

  • Microsoft's Alan Page knows a lot about large-scale test automation, and shares some of his secrets on how to make it beautiful
  • Scott Barber explains why performance testing needs to be a collaborative process, rather than simply an exercise in measuring speed
  • Karen Johnson describes how her professional experience intersected her personal life while testing medical software
  • Rex Black reveals how satisfying stakeholders for 25 years is a beautiful thing
  • Mathematician John D. Cook applies a classic definition of beauty, based on complexity and unity, to testing random number generators

All author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year.

This book includes contributions from:

  • Adam Goucher
  • Linda Wilkinson
  • Rex Black
  • Martin Schröder
  • Clint Talbert
  • Scott Barber
  • Kamran Khan
  • Emily Chen
  • Brian Nitz
  • Remko Tronçon
  • Alan Page
  • Neal Norwitz
  • Michelle Levesque
  • Jeffrey Yasskin
  • John D. Cook
  • Murali Nandigama
  • Karen N. Johnson
  • Chris McMahon
  • Jennitta Andrea
  • Lisa Crispin
  • Matt Heusser
  • Andreas Zeller
  • David Schuler
  • Tomasz Kojm
  • Adam Christian
  • Tim Riley
  • Isaac Clerencia

Read More Show Less

Product Details

  • ISBN-13: 9781449388683
  • Publisher: O'Reilly Media, Incorporated
  • Publication date: 10/14/2009
  • Series: Theory in Practice
  • Sold by: Barnes & Noble
  • Format: eBook
  • Edition number: 1
  • Pages: 350
  • File size: 7 MB

Meet the Author

Tim Riley is the Director of Quality Assurance at Mozilla. He has tested software for 18 years including everything from spacecraft simulators, ground control systems, high security operating systems, language platforms, application servers, hosted services and open source web applications. He has managed software testing teams in startups to large corporations consisting of 3 to 120 people in size and in up to 6 countries. He has a software patent for a testing execution framework which matches test suites to available test systems. He enjoys being a breeder caretaker for Canine Companions for Independence ( along with live and studio sound engineering.

Adam Goucher has been testing software professionally for over ten years. In that time he has worked with start-ups, large multi-nationals and ones in between in both traditional and agile testing environments. A believer in the communication of ideas big and small, he writes frequently at and teaches testing skills at a Toronto area technical college. In his off hours he can be found either playing or coaching box lacrosse - and then promptly applying lessons learned to testing. He is also an active member of the Association for Software Testing.

Read More Show Less

Table of Contents

Preface Adam Goucher xiii

Part 1 Beautiful Testers

1 Was It Good For You Linda Wilkinson 3

2 Beautiful Testing Satifies Stakeholders Rex Black 15

For Whom Do We Test? 16

What Satisfies? 18

What Beauty Is External? 20

What Beauty Is Internal? 23

Conclusions 25

3 Building Open Source Qa Communities Martin Schröder Clint Talbert 27

Communication 27

Volunteers 28

Coordination 29

Events 32

Conclusions 35

4 Collaboration Is The Cornerstone of Beautiful Performance Testing Scott Barber 37

Settingthe Stage 38

100%?!? Fail 38

The Memory Leak Thai Wasn't 45

Can't Handle the Load? Change the UI 46

It Can't Be the Network 48

Wrap-Up 51

Part 3 Beautiful Process

5 Just Peachy: Making Office Software More Reliable With Fuzz Testing Kamran Khan 55

User Expectations 55

What Is Fuzzing? 57

Why Fuzz Test? 57

Fuzz Testing 60

Future Considerations 65

6 Bug Management and Test Case Effectiveness Emily Chen Brian Nitz 67

Bug Management 68

The First Step in Managing a Defect Is Defining It 70

Test Case Effectiveness 77

Case Study of the OpenSolaris Desktop Team 79

Conclusions 83

Acknowledgments 83

References 84

7 Beautiful XMMP Testing Remko Troncon 85

Introduction 85

XMPP 101 86

Testing XMPP Protocols 88

Unit Testing Simple Request-Response Protocols 89

Unit Testing Multistage Protocols 94

Testing Session initialization 97

Automated Interoperability Testing 99

Diamond in the Rough: Testing XML Validity 101

Conclusions 101

References 102

8 Beautiful Large-Scale Test Automation Alan Page 103

Before We Start 104

What Is Large-Scale Test Automation? 104

The First Steps 106

Automated Tests and TestCase Management 107

The Automated Test Lab 111

Test Distribution 112

Failure Analysis 114

Reporting 114

Putting It All Together 116

9 Beautiful Is Better Then Ugly Meal Norwltz Michelle Levesque Jeffrey Yasskin 119

The Value of Stability 120

Ensuring Correctness 121

Conclusions 127

10 Testing A Random Number Generator John D. Cook 125

What Makes Random Number Generators Subtle to Test? 130

Uniform Random Number Generators 131

Nonuniform Random Number Generators 132

A Progression of Tests 134

Conclusions 141

11 Change-Centric Testing Murali Nandigama 143

How to set Up the Document-Driven, Change-Centric Testing Framework? 145

Change-Centric Testing for Complex Code Development Models 146

What Have We Learned So Far? 152

Conclusions 154

12 Software In Use Karen N.Johnson 155

A Connection to My Work 156

From the Inside 157

Adding Different Perspectives 159

Exploratory, Ad-Hoc, and Scripted Testing 161

Multiuser Testing 163

The Science Lab 165

Simulating Real Use 166

Testing in the Regulated World 168

At the End 169

13 Software Development is a Creative Process Chris McMahon 171

Agile Development As Performance 172

Practice, Rehearse, Perform 173

Evaluating the Ineffable 174

Two Critical Tools 174

Software Testing Movements 176

The Beauty of Agile Testing 177

QA Is Not Evil 178

Beauty Is the Nature of This Work 179

References 179

14 Test-Driven Development: Driving New Standards of Beauty Jennitta Andrea 181

Beauty As Proportion and Balance 181

Agile: A New Proportion and Balance 182

Test-Driven Development 182

Examples Versus Tests 184

Readable Examples 185

Permanent Requirement Artifacts 186

Testable Designs 187

Tool Support 189

Team Collaboration 192

Experience the Beauty of TDD 193

References 194

15 Beautiful Testing as the Cornerstone of Business Success Lisa Crispin 195

The Whole-Team Approach 197

Automating Tests 199

Driving Development with Tests 202

Delivering Value 206

A Success Story 208

PostScript 208

16 Pealing The Glass Onion At Socialtext Matthew Heusser 209

It's Not Business...It's Personal 209

Tester Remains On-Stage; Enter Beauty, Stage Right 210

Come Walk with Me, The Best Is Yet to Be 213

Automated Testing Isn't 211

Into Socialtext 215

A Balanced Breakfast Approach 227

Regression and Process Improvement 231

The Last Pieces of the Puzzle 231

Acknowledgments 233

17 Beautiful Testing Is Efficient Testing Adam Coucher 235


Scripting 239

Discovering Developer Notes 210

Oracles and Test Data Generation 241

Mindmaps 242

Efficiency Achieved 244

Part 3 Beautiful Tools

18 Seeding Bugs to Find Bugs: Beautiful Mutation Testing Andreas Zellea David Schuler 247

Assessing Test Suite Quality 247

Watching the Watchmen 249

An AspectJ Example 252

Equivalent Mutants 253

Focusing on Impact 254

The Javalanche Framework 255

Odds and Ends 255

Acknowledgments 256

References 256

19 Reference Testing as Beautiful Testing Clint Talbert 257

Reference Test Structure 258

Reference Test Extensibility 261

Building Community 266

20 Clam Anti-Virus: Testing Open Source with Open Tools Tomasz Kojm 269

The Clam Anti-Virus Project 270

Testing Methods 270

Summary 283

Credits 283

21 Web Application Testing With Windmill Adam Christian 285

Introduction 285

Overview 286

Writing Tests 286

The Project 292

Comparison 293

Conclusions 293

References 291

22 Testing One Million Web Pages Tim Riley 295

In the Beginning... 296

The Tools Merge and Evolve 297

The Nitty-Gritty 299

Summary 301

Acknowledgments 301

23 Testing Network Services in Multimachine Scenarios Isaac Clerencia 303

The Need for an Advanced Testing Tool in eBox 303

Development of ANSTE lo Improve the eBox QA Process 304

How eBox Uses ANSTE 307

How Other Projects Can Benefit from ANSTE 315

A Contributors 317

Index 323

Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star


4 Star


3 Star


2 Star


1 Star


Your Rating:

Your Name: Create a Pen Name or

Barnes & Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation


  • - By submitting a review, you grant to Barnes & and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Terms of Use.
  • - Barnes & reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously
Sort by: Showing 1 Customer Reviews
  • Posted July 27, 2010

    Beautiful case studies and essays

    As a software developer, I see the importance of good testing on a day to day basis. It is the essential piece of the puzzle that ensures that your efforts bare fruit in terms of business value. However, I also see it as the most neglected and poorly implemented aspect of producing software - at least with some of the projects and environments I worked on (and, sadly, I certainly haven't always been an exception).

    Having read this book my drive to improve the testing practices, procedures, and tools has been dramatically invigorated. Of particular interest to me, because they tie more directly with the work I do on a day to day basis, were the chapters on agile development, continuous integration, automated testing, and the associated tools. Chapter 14, "Test-Driven Development: Driving New Standards of Beauty" and "Beautiful Testing as the Cornerstone of Business Success" were truly awe-inspiring. They drew a picture in my mind of the perfect development environment, and even give somewhat of a road-map to get there.

    Some of the chapters were excellent for their high level breadth. Chapter 2, "Beautiful Testing Satisfies Stakeholders" and chapter 6, "Bug Management and Test Case Effectiveness" gives you a good sense of the value proposition of a good testing foundation.

    Others were a real joy to read (well...I guess if you're a geek). Chapter 1 was an entertaining, yet insightful, view into the mind of a tester. Chapter 12 was dramatic and really made you feel the weight of the pressure to make sure the testing was impeccable. One of my favorite chapters was "Software Development is a Creative Process", chapter 13. This chapter distilled how I feel about development.

    There were a few chapter that didn't give me much value. Many of the chapters were very narrowly focused on a specific technology or project, and didn't do enough, in my opinion, to relate the chapter to the larger scope. Chapter 21, "Web Application Testing With Windmill" read almost like marketing material for the product. I also didn't get much out of "Beautiful XMPP Testing" (chapter 6) and "Testing a Random Number Generator" (chapter 10). These chapter, however, really didn't detract from the overall superb quality of the writing or high value of the information contained in this book - not to mention that it is probably a matter of perspective.

    If you are involved in any aspect of software development, you should consider this required reading. The knowledge and insight taken out of this book will certainly make me more effective professionally.

    Was this review helpful? Yes  No   Report this review
Sort by: Showing 1 Customer Reviews

If you find inappropriate content, please report it to Barnes & Noble
Why is this product inappropriate?
Comments (optional)