Testing more than the features

One issue you can never discuss too much is testing. How do you test a software application before turning users loose on it? Dr Dobbs, one of my favorite software magazines for almost two decades, had a nice piece by Scott Ambler on test in the December edition. Scott has emerged as one of the principal evangelists of the agile development movement, and in this issue he discusses testing from an agile perspective. Like much current writing on testing in the Extreme Programming and Agile universe, he talks a lot about unit testing and the test-first strategy. This approach stresses writing automated tests for every bite-sized chunk of program logic. Automated unit testing can more than double the amount of code that needs to be written, but provides a mechanism for detecting when new code breaks old features. And don't you hate when that happens! Ambler also stresses a new concept: what he terms investigative testing. Most of today's testing literature emphasizes testing the specified functionality. Investigative testing takes a different tack: The investigative test team's goal should be to ask, "What could go wrong," and to explore potential scenarios that neither the development team nor business stakeholders may have considered. They're attempting to address the question, "Is this system any good?" and not, "Does this system fulfill the written specification?" A blogger I've been quoting a good bit lately, Jeff Atwood, similarly advises us this week not to limit testing to confirmation of the specifications. His post on Lo-Fi Usability Testing takes off from a book he's recommending to all software designers, Don't Make Me Think, by Steve Krug. His point: the users' agreement that you've met their requirements has very little to do with their ultimate satisfaction with the application. The users, like the programmers, are too close to the project to be the best testers. The best bet is to rope in a few people who know next to nothing about your project. Usability testing doesn't have to be complicated. If you really want to know if what you're building works, ask someone to use it while you watch. If nothing else, grab Joe from accounting, Sue from marketing, grab anyone nearby who isn't directly involved with the project, and have them try it. Don't tell them what to do. Give them a task, and remind them to think out loud while they do it. Then quietly sit back and watch what happens. I can tell you from personal experience that the results are often eye-opening.

The permanent Beta?

Browsing through some old posts in the always worthwhile Creating Passionate Users blog, I came across this interesting post from last March: Ultra-fast release cycles and the new plane. The post is about the growing popularity of extremely short turn arounds between versions that has become characteristic of web2.0-type development - what O'Reilly called the permanent beta. This is an issue we discuss in our office quite a bit. We have some users who demand rapid release. They feel that if they find a small bug on Monday, dream up a new piece of functionality they'd like on Wednesday, and want the menus reorganized on Friday, they should certainly be allowed three versions that week. After all, they're paying. What about the increased risk of defects? Well, they don't mind... they'll report them and they will be fixed in the next rapid release. On the other hand, we have users who despise the rapid release. These clients expect us to organize their requests, implement them all, test them all, and give them an upgrade once or twice a year at most. Installing any more often, they've told us, complicates user training and increases the risk that a new release will break something that was already working. Far from demanding rapid release, they see it as a lack of professionalism. How do we understand this difference? In the Creating Passionate Users piece, Kathy Sierra talks about the short-release trend as a "cultural" one. She's focusing on the youthful subscribers to the Myspace website, her model of the rapid release project. So she focuses on factors relating to youth culture. What she does not dwell on is that Myspace is fundementally different from business software - it is not critical to most users' bottom lines. Let's look at this "cultural" difference among the staff of non-profit organizations using our Members Only software. It seems to me the difference here hinges on the perception of real costs and real risks associated with installing a new version. These costs and risks actually differ significantly between organizations, so the evolution of different cultures around software installation is not at all irrational. It has to do with the organization's size (in number of users, and in number of locations) and the level of activity (How many transactions go through the software in an average day.) Large organization using software for mission critical purposes are aware of the costs of training and of the bottom-line impact of a significant defect, and therefore want to minimize the number of training and testing cycles they expose themselves to. Where does you organization fit on this spectrum? Are you happy to use a permanent beta? Are new features on a regular basis worth encountering bugs you will need to report? Are you the kind of user who is always pushing for the next update, or are you loathe to install them? What factors have shaped your attitudes?