What works when?

Editor’s note: Tema Frank is president of Web Mystery Shoppers International Inc., Edmonton, Alberta.

First, the good news: companies are increasingly recognizing that Web sites are too important to be left strictly in the hands of programmers and also that market research is essential to effective Web site design. The bad news: there is a bewildering array of techniques being touted to assess a company’s Web site effectiveness. How do you know which to use?

This article discusses different approaches to getting feedback on your Web site. In an ideal world, you would use each of the techniques discussed in this article at some point during your Web site’s life cycle. But this isn’t an ideal world, and odds are you don’t have the time or the budget to do them all. So how do you decide?

The type of testing you do should depends on a combination of your organization’s stage in the Web site life cycle and your goals for testing. Realistically, it will also be influenced by cost and time pressures.

This article examines six Web site testing methods:

1. Focus groups (in-person or online)

2. Lab-based usability testing

3. Web-based (pop-up) surveys

4. Metrics/analytics software

5. Unsolicited customer feedback

6. Remote usability testing

Focus groups

Most market researchers are already familiar with focus groups, and so they naturally gravitate to using them to get qualitative feedback on Web sites. Such focus groups typically involve six to 12 people, who may be chosen from your existing customer base or recruited externally. They are brought into a meeting room where they have a group discussion, led by a moderator, about the Web site. If the site does not yet exist, they may be asked what they would like to see in such a site. They may be shown page mock-ups, drawings, or led through an card-sorting exercise which is intended to help the site designers figure out how users would interpret various topic labels and how they would organize the information.

As with typical focus groups, the client may be behind a one-way mirror and the facility may be set up in such a way that the client can ask the moderator (who is wearing a headset) to ask specific follow-up questions. Costs are based on a combination of recruitment costs, participant expenses and consulting-related fees that may be charged by the company running the focus group. Typical focus groups cost between $3,000-$8,000 per group, which includes a fee of typically $50 to $100 per participant.

Advantages of focus groups

  • Client gets to watch participant reactions, and may be able to have moderator ask follow-up questions.
  • Participants can play with the content - for example, rearranging cards in an order that makes sense to them.
  • Can be done early - before money has been spent on programming.

Drawbacks of focus groups

  • User’s guesses as to how they would use the site are very different from reality.
  • Will miss most navigational problems.
  • Risk of group-think. What people say is influenced by the others in the room.
  • Risk of poor moderators. Need a good moderator - must not bias the conversation, but ensure no one dominates.
  • Participants limited by geography, time and willingness to travel to the testing location.
  • Often either no transcript available, or significant cost or delay to get a transcript.
  • Poorly constructed participant group can negate the value (i.e., respondents must be like your target customers, and not so familiar with your site or company that they are unable to view the concepts with fresh eyes).

Focus groups are useful when you are in the very early stages of making a Web site plan to get rough idea of desired content. They can help with the conceptual, brainstorming work. They can also help when you want to assess emotional reactions to colors or visual treatments and you don’t have the time or funding for individual interviews.

Online focus groups

A variant of traditional focus groups, online focus groups allow participants to log in from their own computers at a pre-arranged time, and the discussion uses a chat room approach. In the more sophisticated set-ups, participants can view materials that the moderator displays on the screen and possibly go back and forth between the chat room and the viewing screen, or see both at once.

Advantages of online focus groups

  • They overcome geographic limitations of traditional focus groups.
  • Real-time transcript is available for analysis.

Drawbacks of online focus groups

  • Cannot do physical activities such as card-sorting.
  • Hard to follow the flow of conversation.
  • No facial expressions or tone of voice.

Online focus groups are useful when you are doing early-stage testing for a teen-oriented site, for example. Teens tend to be the group that is most comfortable with online chat, and they may prefer the semi-anonymity of this format to a traditional focus group. For most other users, this is not an ideal format.

Lab-based usability testing

Users are brought into a testing lab, one at a time. A typical session lasts one to two hours with each user. The lab may be configured to resemble an office or a home. The participants work on the lab’s computer (there may be a choice between a Mac-based OS and a Windows-based system), and are asked to vocalize their reactions (“think aloud”).

The client can usually watch behind a one-way mirror. As with a focus group, there may be a moderator present in the lab with the user, who can prod for further insight, ask follow up questions, or, in extreme cases, give permission for the user to give up and try something else.

In some labs there is also a video camera taping the participant’s facial reactions, and some form of clickstream monitoring (tracking of where the person clicks or scrolls on a page).

Given the intensity of this type of testing, there are usually only a handful of people tested. The traditionally accepted minimum was five testers, but current research suggests that this is far too few for today’s complex Web sites. The problems identified by the first few users may not reflect the most important problems in need of repair.

A typical set of lab-based tests with five to 10 users will cost in the range of $20,000-$100,000, depending on the degree of consulting that is included in the project and the market clout of the testing company.

Advantages of lab-based usability testing

  • Users are interacting with the real site, so you can learn about navigational problems.
  • Can ask the users questions as they interact with the site, so they can explain surprising actions.
  • Because you see the user’s facial expressions and hesitations, you get a sense of the frustration level and uncertainties experienced by the user.

Drawbacks of lab-based usability testing

  • It is a stressful, unnatural environment. Research clearly demonstrates that stress affects performance.
  • Respondents must use an unfamiliar computer, which will cause frustrations that the user would not encounter on his or her own computer.
  • Most lab-based computers are more finely-tuned and using more current software than most users’ real computers, so many errors users encounter in real life will be missed with this approach.
  • Users need to be willing and available to come to the lab. This causes a skew in who is willing to participate.
  • Geographic limitations. Most labs are in major cities. But are all your users?
  • Small number of users tested. Are the problems the first five to 10 people encounter typical?
  • Often expensive, particularly if a transcript is prepared and analyzed after-the-fact.

Lab-based usability testing is useful when you want to see users’ facial reactions and hear their vocalized thoughts. One strength is that you can intervene to ask questions about why they are doing what they are doing if they have not explained something to your satisfaction. (But there’s a risk of changing how they behave if such questions are overdone.) Obviously you need to have at least a “draft” site available for on-screen use. They work best if your anticipated site users are a homogeneous group, so that it is possible to recruit a small but representative sample.

Web site (pop-up) surveys

There are many companies that offer surveys which can either be mailed to site visitors after the fact or that pop up on site while the visitor is using it. Mailed surveys are typically too long after the fact to yield any meaningful results for Web site assessment. At most you will get vague, general feelings about the overall experience (“It left me feeling frustrated” or “It was OK”).

Pop-up surveys are programmed to either pop up for every visitor, or for every nth visitor, or for visitors to specific pages. Visitors are asked if they will participate in a survey and are sometimes offered a chance to win a prize as an incentive for participation. The survey asks a few questions about the site experience. Generally, a maximum of about five minutes of questions and answers is the longest you can expect people to stick with such a survey.

Costs vary from small, free surveys offered by companies such as Zoomerang to sophisticated surveys that change the questions asked depending on previous answers and have a lot of built-in analytic tools. The latter type of survey can easily cost in the $10,000-$20,000 range. Because a heavily trafficked site will usually be able to collect a large number of responses very quickly in this way, pricing is often quoted on a per-respondent basis.

Advantages of Web site (pop-up) surveys

  • Large numbers can be surveyed.
  • Fast turnaround.
  • Low cost/respondent.
  • Actual site visitors responding.
  • Questions can be changed often.
  • No need to pay respondents.

Drawbacks of Web site (pop-up) surveys

  • Survey fatigue: people are less willing than they used to be to spend time answering such surveys.
  • Bias in who replies. Respondents are more likely to be new Internet users, as they are less likely to have become accustomed to being asked their opinion on Web sites.
  •  Intrusive; it may annoy site users. (Note: One common mistake is to ask visitors for their opinion before they’ve had a chance to explore site!)
  • Increasing prevalence of pop-up stoppers is making it harder to get results.
  • Very limited number of questions feasible, so you cannot get in-depth responses.
  • No tracking of the path taken and the customer’s reactions as they moved along that path.
  • Requires special coding be placed on your site. This places demands on your IT department.

Web site (pop-up) surveys are useful once the site is already live and getting large numbers of visitors. They’ll give you feedback quickly, but not in-depth. You’ll get general reactions to the site. Our research suggests that most of those who are willing to fill them out will be fairly new Internet users, so these surveys won’t be as helpful if you are targeting a market of experienced or heavy Internet users.

Metrics/analytics software

Your Web site hosting company should be able to provide you with some basic statistics about who is visiting your site: how many unique visitors, which pages they are visiting, where they came to the site from, etc. Metrics or analytics software consists of sophisticated versions of these server logs. Some can track clickstream details (what pages people clicked on and in what order) and analyze patterns and drop-off points. They can usually track which pages were viewed and the average length of time spent per page.

They are useful in conjunction with other methods, but you are left to infer customer motivations. They may tell you, for instance, that 63 percent of your site visitors abandon your site at the first page of your order form. You have no way of knowing whether they are leaving at that point because the page isn’t displaying properly on the screens of Windows XP users, or if they couldn’t find information about shipping costs and were unwilling to continue without knowing that, or if they pressed the wrong button and then decided it wasn’t worth the hassle to go back and re-enter the information, etc.

Costs for such packages range from the free basic statistics provided by many site-hosting companies to sophisticated programs in the tens or hundreds of thousands of dollars.

Advantages of site metrics/analytics programs

  • Can provide detailed statistical information on your actual site visitors.
  • Can help you identify problem pages.

Drawbacks of site metrics/analytics programs

  • The volume of statistical data can be overwhelming. Many site owners find it impossible to interpret, and end up ignoring most of it.
  • Does not explain why visitors are behaving as they are.

Site metrics or analytics programs are useful when you have the staff and budget to do sophisticated analysis of the data and can combine it with qualitative information gained from other approaches. At a minimum, you should analyze whatever data you can get free from your Internet hosting company.

Unsolicited customer feedback

Almost everybody gets some feedback on their site. From family and friends to site visitors who feel strongly enough that they’ll take the time to send you an e-mail or fill in a comment form, people will give you their opinions. If you have a call center, you can also learn a lot about peoples’ reactions to your site from analyzing the help requests that relate to the Web site.

Since you get it, use it. You may get some gems from this sort of feedback. The big mistake site owners often make, though, is assuming that what they hear from such channels is typical, and that they thus know all they need to know about what people think of their site. Wrong! The people who take the time to comment are not typical of the general public. They are a vocal minority who tend to perceive things in more extreme terms than most, and who have enough time on their hands that they consider it worth the time to comment.

Advantages of unsolicited customer feedback

  • Minimal cost.
  • Real users.
  • Chance to repair relationship with dissatisfied customers who take the time to comment.

Drawbacks of unsolicited customer feedback

  • Biased sample.
  • Miss many issues.
  • Analysis can be costly in large organizations.
  • You must respond quickly and well or your lack of correct response will make matters worse.
  • Often you will not learn exactly where the problem occurred. The comments are frequently too vague to be useful.

Unsolicited customer feedback is useful when making minor adjustments to the site as you go along, and as part of the input for redesign. If tracked, it also helps ensure a good linkage between customer service staff and site promises.

Remote usability testing

A newer approach is what’s called remote usability testing. Some people mistakenly call online focus groups remote usability testing, but, as discussed earlier, focus groups help you develop concepts but do not help learn about the actual usability of a Web site.

One remote usability approach is to work one-on-one, as in a lab-based usability test, but with the testers working from their own computers and either talking by phone or through online chat with the interviewer while they work through the site. Typically this uses technology that allows the interviewer to see the same screen as the person doing the testing. Let’s call this one-on-one remote usability testing. The advantages and drawbacks of this approach are similar to lab-based usability testing, except that you can use testers from any geographic region. You may or may not be able to see their facial expressions, depending on whether they have also set up a camera to record the user’s face. Clearly, setting that up requires a level of complexity that will put off most users, unless the market researcher is there to set up the equipment for the user. But having to send the market researcher to each user’s home or office negates the cost savings of such an approach, and the intrusiveness of the market researcher and/or camera and recording equipment detracts from the benefits of testing in the user’s natural environment.

Another approach is to have large numbers of people (typically 30-200) work independently on their own computers to try to complete tasks at the site, and report on their progress by answering questions as they go through the site. Since they are expected to devote much more time to the process than is expected in a pop-up survey, they are usually paid $10-$20 for their effort. We’ll call this “large sample-size remote usability testing” (LSSRUT).

Advantages of large sample-size remote usability testing

  • No geographic or time-based limitations.
  • Collect lots of detail as they go.
  • Includes both qualitative and quantitative information.
  • Realistic setting; low stress.
  • Wide range of computer systems; uncover problems faced using buggy personal computers.
  • No demands on IT department. (Depending on system used to collect the information.)
  • If a system that does not involve the IT department is used, it allows benchmarking - both against competitors and to compare present and future performance. The large sample size, if properly recruited, provides statistical validity.

Drawbacks of large sample-size remote usability testing

  • May involve imperfect click-stream analysis. In some systems, the testers must report on where they clicked rather than having it tracked automatically. Those that do track it automatically require that the users download special software to do so. This skews the sample, since many people are now reluctant to download software.
  • Cannot see facial expressions.
  • Needs reasonably literate testers. They have to be able to express their thoughts in writing.
  • Site needs to already be accessible online (even if only on a test server).

LSSRUT is useful when a current site or a draft version of a new site is accessible online and you are preparing for a redesign or double-checking and fine-tuning a design before or after launch. If it is set up in such a way that no special scripts need to be inserted into the Web site being tested, it can be used for research on competitive Web sites and for benchmarking. It can also be used to monitor and reward customer service staff, if site testers also check on e-mail handling and/or call-center replies to questions about the Web site. It is particularly valuable if you need either a wide range of testers (e.g., people from across North America ) or a specific type of testers (e.g., rural New Mexicans aged 50+).

Which are best?

Which techniques are best at what stages? In a nutshell:

  • Concept development: use focus groups, plus feedback from clients and current customers.
  • Once online (test or live server): use lab-based testing if you need to see facial expressions; otherwise consider alternatives that take place in a more natural setting, such as remote usability testing.
  • Ongoing: pop-up surveys if and when needed; Web analytics/metrics, collect and analyze unsolicited feedback; use LSSRUT to ensure no slipping and for competitive analysis.
  • Redesign: use LSSRUT, lab tests and data from ongoing metrics before redesigning so your organization doesn’t throw out the baby with the bathwater!