Choose your usability approach wisely
Editor's note: Lon Taylor is senior usability consultant, Nick Gould is CEO, and Peter Hughes is principal UI designer, at Catalyst Group Design, New York.
With the increasing maturity of the Internet as a medium, most Internet marketers would agree that the usability of a Web site's user interface is a crucial driver of business success. In fact, over the past several years, many major research reports have been published by organizations such as Forrester, Jupiter, and others that attribute a staggering loss of revenue opportunities due to poor usability. From significant shopping cart abandonment to negative brand perception, nowadays it's clear that a site with usability problems ultimately leads to a low return on investment. What is less clear at the moment may be how to best identify the right testing methodology for effectively (and cost-efficiently) uncovering and resolving potentially disastrous usability issues.
The traditional testing method, we'll call it lab-based usability testing, involves a series of one-on-one moderated site walkthroughs with no more than 20 total participants drawn from the site's target audience. Lab-based testing focuses on a close, in-person, observation of the test participant as they attempt to complete the tasks presented to them. An experienced moderator actively engages in a dialogue with the participant that is aimed at revealing the core reasons for any usability problems they have encountered.
More recently, new technologies - generally referred to as online usability testing - have emerged that promise to deliver equally valuable usability feedback with the additional benefit of a much larger sample size (200 or more participants). At a high level, online usability testing is an online survey (without a moderator) that enables participants to walk through a series of sites, pages, and/or site tasks. Test participants are queried along the way for their feedback regarding the material they have seen. Online usability tests can be designed to elicit a variety of different types of feedback, including reactions to design and branding options and other more traditional demographic and behavior oriented questions. Another interesting aspect of online usability testing technology is that the systems provide a detailed click-through tracking of every step a user takes, and this data can be combined with the participant's survey responses in the final analysis.
Lab-based testing long predates the advent of the Internet and is understood and accepted in most design and technology circles. However, traditional research and marketing professionals have only recently begun to incorporate this methodology into their Internet/Intranet product development plans. And now, just as it seems that this method of usability testing is gaining a broader acceptance, the new online usability testing technologies are gaining a following amongst people who feel that the ability to test with a larger sample size is of paramount importance. In fact, with the different options available to today's Internet marketers, the very nature of the term usability testing seems to be in question. Is usability testing by definition a purely qualitative exercise, or can these new technologies be used to obtain feedback on a site's usability that is quantitatively significant?
Not surprisingly, the online usability companies (such as Vividence, Relevant View, and NetRaker) tout the quantitative aspects of their service. Most promote that they routinely conduct tests with 200-1,000 participants. Another selling point that is frequently mentioned is the "real-world" context of the testing (i.e., the fact that tests are conducted at home or at work, as opposed to in a lab setting). Alternatively, practitioners of traditional lab-based usability testing focus on the quality and depth of the feedback they gather in the one-on-one experience of a controlled environment, as opposed to the size of the testing sample. These practitioners believe that the new research technology is useful, but that it's no substitute for the type of qualitative results that a polished moderator can glean when working to identify issues and learn why they exist in the first place.
Real-world scenarios
This methodology debate is crucial since selecting the right approach is just as important as making the decision to test in the first place. Lab-based usability interviews and online usability testing both offer the promise of rich feedback, quick turnaround times, a reasonable price tag ($20K-$60K) and the occasional "aha" that can uncover a huge glitch that everyone seemed to overlook (see comparison chart). So, in order to better understand the benefits and drawbacks of both methodologies, let's examine their respective pros and cons based on some potential real-world scenarios.
 |
A financial services company has recently merged with a firm in another state, so it is interested in redeveloping its corporate Web site to reflect this change and it is also working to select a new company name. The company is interested in conducting some research to:
- gather usability data about its current site;
- evaluate some competitive sites; and
- obtain responses to several possible names for the new entity.
Is this a job for lab-based testing? Although one-on-one methods would certainly reveal the usability issues relating to the current site, an online usability evaluation might offer several additional advantages for this type of engagement. For issues that need to be backed up by a statistically significant sample size, such as feedback on a new company name, the survey based methodology works well.
Since the financial institution is also looking to unearth usability issues from its existing site and from some competitive sites, the online evaluation might provide a more efficient mechanism for gauging the time it takes to complete specific tasks across all the sites of interest. An example might be to ask participants to locate a mutual fund and its rate of return. This exercise and other similar tasks will offer plenty of insight on navigation pathways and how easily participants can use the sites they visit. The test could also obtain measurable user responses regarding demographics, behavioral issues and the various branding or visual design styles adopted by each competitor. Finally, the online evaluation methodology is ideal for this situation in that participants from all geographic regions served by the newly merged company can be recruited via e-mail from or via a pop-up window on the site itself.
Now let's imagine that a brick-and-mortar retailer has just committed to going online. It has defined its site requirements and has prepared a features matrix outlining how items on the site should be weighted based on importance (e.g., should the "shop for a shirt" button be the same size as the "about us" button). Its Internet professional services agency has just completed the first round of design ideas and wants to conduct usability testing on an HTML wireframe/prototype with 75 percent of the links being active. The company is interested in conducting research to:
- gain insights on the usability of its wireframe/prototype;
- develop recommendations to address the issues uncovered; and
- observe how effectively users accomplish core site tasks.
Which usability testing method would best serve these objectives? Lab-based usability testing is likely to elicit the best feedback on ease-of-use issues in this scenario even though the site is not yet fully functional. When a Web site or interface design is at this early stage, it is very important that the usability research reveal whether the value proposition is clearly understood by participants. A well-trained moderator will be able to ask a participant relevant follow-up questions that not only draw out usability issues on a site at the wireframe/prototype stage, but also allow them to explain why they could not complete a task. If a participant fails to complete a task as intended, he or she can be queried in depth about whether the problem was related to nomenclature, functionality, navigation, or readability. The type of dynamic dialogue between a participant and moderator is something that works extremely well in a controlled usability lab over the course of a 60-to-90-minute session and really can't be done as well online.
The lab-based testing methodology also accounts for situations where the line of questioning or the rotation of questions may change based on client input. If an issue is resolved during the first day of testing, time can be spent on other items during the next day of testing as two days are typically scheduled to conduct testing. In addition, there's no substitute for having clients view the sessions behind a one-way mirror. Seeing a participant's non-verbal communication and videotaping sessions for colleagues offers feedback "in living color" directly from the target audience. Although the findings and recommendations using this type of methodology are not projectable, and don't offer geographic diversity without added cost, lab-based testing would serve all stakeholders needs given the scenario provided for the retailer.
Interesting and actionable
Clearly, both ways of implementing usability testing will yield interesting findings and actionable recommendations. If you are looking to obtain "think aloud" feedback on an interface that's more along the early stages of development, then lab-based usability will likely work best. If you are trying to demonstrate a preference for ideas and need measurable opinions, go with an online usability scenario. Debating whether a usability test with 800 people is inherently better than a test with eight people misses the point. Online usability testing technology is not going away and neither is lab-based testing. Each method can provide powerful and actionable results to the Internet marketer who knows when to use it.