Read an Excerpt
Chapter 1: You'll Never Catch 'em all!
Bugs, that is.
I'll begin with a rash-sounding claim. In even the smallest program (that has input, output, and processing), it's impossible to construct tests that will locate all bugs!
Now, if that's true, is perfection but a dream? Should we abandon hope and resign ourselves to producing poor-to-mediocre code? God forbid! We who have learned one of the hottest programming languages in history must now strive to create programs so clean that the chances of anyone finding a bug are nil. That's why this book exists. Testing is one of the programmer's best tools for writing master-level code, but as you shall see shortly, testing is insufficient. For example, just before Christmas, 1999, astronauts upgraded the Hubble telescope's computers from 386 to 486 technology, while Pentium-Ill computers were flying off the shelves. NASA couldn't use newer CPUs because they needed four years to test the chips' imbedded firmware. After four years of testing, NASA still didn't claim their tests were exhaustive. Rigorous testing is not enough. I'll try to prove that statement.
You may recall Heron's formula from high school trigonometry. Give Heron's formula the three sides of a triangle and it reveals the area. This formula is just complex enough to challenge early computer students, but we'll use it for a different purpose.
Pretend your task is to use input and output to test a computer program that implements Heron's formula. You are to build test cases. What kinds of data would you input and what would you expect for output? For example, youmight input
- Integers 3, 4, and 5, and expect an integer area of 6.
- Integers other than 3, 4, and 5, expecting a floating-point answer.
- Various floating-point numbers as input.
Neither Debugging nor Testing Finds All Bugs
Teaching university computer classes, I would give the students a ten-minute, ungraded quiz, asking them to devise all the kinds of test data they could for the hypothetical Heron's formula problem. Then we'd tabulate the results. Most classes found 15-18 examples. Occasionally, succumbing to the ultimate bug, a student would write a computer program to solve Heron's formula! That always got a laugh. After ten minutes, I'd propose a few clever ideas found by previous classes and challenge the students to find more. In the ensuing fun, they would invariably expand the list to 30 or more, and I'd add a couple of items to my personal list. I did this as much for research as for teaching.
One conclusion is that testing is a highly creative and intuitive process. It is quite difficult to think of an exhaustive set of tests. None of my students or colleagues ever did, nor have I. Without an exhaustive set of tests, how can testing find all bugs?
Even if a tester could write an exhaustive set of test classes, such a set is insufficient. For example, one of my best students discovered something amazing about the Pascal compiler. At random, he input 16, 20, and 12.50613300415 into his Heron's formula program, and his program printed 101.001. So far, so good. The second part of his assignment was to round the results to two decimal places. He expected 101.00 and got an erroneous 101.01 instead. The Pascal compiler rounded incorrectly! However, 101.002 rounded correctly. So did 101.0001, 101.0011, and 101.005 .
He proved it was the compiler at fault, and then wrote a new program to test the compiler for further errors. His new program compared the compiler's rounding function with the standard algorithm that rounds numbers. If the comparison revealed a discrepancy, his program printed it.
The first design was to step through all possible floating-point numbers allowed by the quad-precision VAX compiler. When it became apparent this program would consume the VAX for most of the semester, he restricted the program's range. In four evenings, checking billions of possibilities, 101.001 was the only erroneous example it found. We wrote a letter to Digital Equipment Corp. (DEC) and received a kind response. Later, we received a note that DEC had fixed the problem...