Categories: Software Part 2: Usability Gaffes: Things That Make You Go Arrrgh! Part 3: Talkback: Tell us about usability gaffes you've found Related Stories: 1999 Software Sneak Preview Next: Real-World Usability Results

Continued from Types of Usability Testing

To research this article, I visited the Usability Testing facilities at the Microsoft campus in Redmond, Washington. Microsoft's Usability Research Group celebrated its tenth anniversary in May. In the past, I have worked with usability labs at IBM in Toronto and Northern Telecom in Ottawa, and I ran a small usability lab at the University of Waterloo in Canada.

What I discovered at Microsoft was that not much had changed in concepts or facilities, except that usability testing is now a recognized area of research in academic disciplines ranging from computer science and human/computer interaction through psychology and technical communication. It has also become an expected feature of the design process, rather than a hoped-for addition. One important perceptual difference is that testing personnel, who are drawn from a variety of academic disciplines, are now called usability engineers, at least at Microsoft. That kind of recognition is obviously important in a high-tech firm. Information about Microsoft's usability facilities is available at www.microsoft.com/usability.

But the basic lab itself has remained much the same over the past decade. The principle is to have two adjacent rooms, the observation room and the testing room, or in Microsoft parlance the observer side and the participant side. The two rooms are separated by a wall (soundproof, ideally) and a one-way mirror, so that the observers can see the participant but not vice versa.

Figure 1: An example layout of a typical usability-testing laboratory.

The testing room (see Figure 2) is typically equipped with one or more computers, a desk and chair (or several), and at least one camera and microphone. The participant sits at the computer while the camera and microphone record all activity. In many cases, the testing room contains two cameras or even three, depending on the subject of the test. One camera captures what's on-screen, unless a scan converter is available to record full-screen motion onto a videotape. A second camera captures the participant's facial expressions and body movements, including mouse and keyboard movements. If the object of the test is a document, such as a user manual, another camera will focus on it. The testing room also contains speakers through which the participant can hear questions or instructions from the observation room.

Figure 2: A usability-testing laboratory is equipped with cameras, microphones, and a one-way mirror (back) to observe participants.

The observation room is even more fully equipped, with at least one computer, a microphone, a VCR with time-stamping capabilities, equipment for controlling cameras, and a video mixer. The microphone lets the observer communicate with the participant, an activity that should happen infrequently. The computer runs event-logging software, synchronized to the time stamp on the VCR. The logging software runs throughout the test, and the observer records when the participant starts and finishes a specific procedure and when any significant event occurs.

Significant events include a participant's comments, questions, facial expressions, technical errors, or anything else that reflects the attempt to use the software. Only if the participant needed prompting or if the participant asked a question would the observer use the microphone. Prompting would be necessary, for example, if the participant were so far off-track that he or she couldn't proceed or if frustration were clearly setting in.

In some cases, a second observation room is available, separated from the main observation room by a window. Here, members of the development team can watch the test and discuss the participant's actions as they happen.

Participants are typically drawn from local businesses, sometimes from organizations on the company's client list. Microsoft maintains a database of 25,000 potential participants near its headquarters and uses this database to get the best possible match between product and user. Testing Microsoft Money, for example, requires participants with different levels of experience from those testing Visual C++, so the match is important. There's also a difference between testing basic interface concepts and advanced maintenance options in an operating system.

The number of tests for a specific product depends on how much detail the development team requires. Once the testing is complete, the videotape is edited to show only the highlights of the tests (usability observation isn't a great spectator sport), and the tape and a written report are given to the product manager. At Microsoft, each product or product group has its own usability manager, whose tasks are to determine which of a product's features require testing, to report the results of the testing to the development team, and to work with them on possible alterations.

Next: Real-World Usability Results

Published as Tutor in the 10/6/98 issue of PC Magazine.