Testing more than the features

One issue you can never discuss too much is testing. How do you test a software application before turning users loose on it? Dr Dobbs, one of my favorite software magazines for almost two decades, had a nice piece by Scott Ambler on test in the December edition. Scott has emerged as one of the principal evangelists of the agile development movement, and in this issue he discusses testing from an agile perspective. Like much current writing on testing in the Extreme Programming and Agile universe, he talks a lot about unit testing and the test-first strategy. This approach stresses writing automated tests for every bite-sized chunk of program logic. Automated unit testing can more than double the amount of code that needs to be written, but provides a mechanism for detecting when new code breaks old features. And don't you hate when that happens! Ambler also stresses a new concept: what he terms investigative testing. Most of today's testing literature emphasizes testing the specified functionality. Investigative testing takes a different tack: The investigative test team's goal should be to ask, "What could go wrong," and to explore potential scenarios that neither the development team nor business stakeholders may have considered. They're attempting to address the question, "Is this system any good?" and not, "Does this system fulfill the written specification?" A blogger I've been quoting a good bit lately, Jeff Atwood, similarly advises us this week not to limit testing to confirmation of the specifications. His post on Lo-Fi Usability Testing takes off from a book he's recommending to all software designers, Don't Make Me Think, by Steve Krug. His point: the users' agreement that you've met their requirements has very little to do with their ultimate satisfaction with the application. The users, like the programmers, are too close to the project to be the best testers. The best bet is to rope in a few people who know next to nothing about your project. Usability testing doesn't have to be complicated. If you really want to know if what you're building works, ask someone to use it while you watch. If nothing else, grab Joe from accounting, Sue from marketing, grab anyone nearby who isn't directly involved with the project, and have them try it. Don't tell them what to do. Give them a task, and remind them to think out loud while they do it. Then quietly sit back and watch what happens. I can tell you from personal experience that the results are often eye-opening.