On the front line of online
For some real-world examples of on-line research, I talked to researchers at three companies about some of their recent on-line projects. Two of the companies, Bay Networks and Informix, serve the computer industry. The other, Nickelodeon, provides a range of TV programming through its Nickelodeon and TV Land networks.
Bay Networks and Informix are users of SurveyBuilder.com, a service offered by Virtual Architechs, Sausalito, Calif., in which respondents are recruited to visit the SurveyBuilder.com Web site to participate in surveys.
Menlo Park, Calif.-based database technology firm Informix turned to research to obtain benchmark data for its Web site, including information on where visitors went in the site, what they liked/disliked, what information they wanted. "Prior to the research, we had no benchmarks about what people thought of our site," says Sandra Bateman, director of corporate Web marketing, Informix Software. "Because the Web is new - I always say that WWW stands for wild, wild west - there is a lot of maverick behavior, a lot of claims are being made, so trying to get actual user data is important. We’re spending all this time and money making these Web sites, but are people using them? You can measure the number of hits, but that doesn’t tell you why people are coming to your site."
In Informix’s case, a script was written so that every fourth person who visited the Informix site would be asked if they wanted to take the survey. Those who said yes were taken directly to the SurveyBuilder Web site. As an incentive, SurveyBuilder.com lets respondents select a charity to which to donate $2. "It’s a win-win," Bateman says. "We get the information and the charities benefit."
The survey was kept brief so that respondents could complete it in 10-15 minutes. "I like the fact there is a lot of flexibility in designing the questionnaires," Bateman says. "We have full control over the kinds of questions and the number of questions. The people at SurveyBuilder.com have good advice about length and usability and how to attract people to take the survey."
Bateman says the information from the initial survey will serve as a benchmark for a new site that’s under development. Once that site is up, more research will follow. "I think it’s a very cost-effective way to go. Focus groups are a very handy tool, but they take time and money to set up. This kind of research is very quick and easy. You can get a good amount of data in a short time. The limitation is that you’re going after a selective audience, so there’s no way for me to find people who aren’t going to the site and find out why. That’s where a focus group would be helpful. But for our intents and purposes it’s a good tool and we plan to use it on a regular basis."
Bay Networks, a Santa Clara, Calif.-based hardware networking company, has also used SurveyBuilder.com to research visitors to its Web site, www.baynetworks.com. Network managers, corporate management and others use the site to get product information. The site also features service and support areas and a knowledge database.
Bay Networks conducts a quarterly user survey to obtain information about Web site visitors’ Internet architecture, the browsers they use, which platform(s) they use, if they use Java, and their overall satisfaction with the Bay Networks site.
Surveys are also run in the different sections of the site, to see how the various audiences (resellers, network managers, etc.) feel about the sections designed for them. A Java applet randomly invites visitors to take the survey.
Quick turnaround of results is one of on-line research’s selling points, says Connie Sanchez, Bay Networks’ Web marketing manager. "It depends on how much traffic your site generates, but we have no problem turning around 500-response surveys in a few days. Recently, for example, we were designing a new section on the site and we wanted to find out what size screen users have, so that we can design content to take advantage of that. We were surprised to find out that 80 percent were on larger screen sizes. We needed that information right away to move ahead with the design work."
Sanchez says that the ability to embed graphics in the survey form aids respondent recall. "For a technology survey in which we asked what plug-ins [like Acrobat or Shockwave] respondents had, we embedded the download icons into the survey so they would click on the ones they had on their system. That helped people remember, because they might not recall the name of the product but they recognize the logo."
That capability will help Bay Networks as it looks at conducting on-line usability testing. "We’re exploring that as a way to test different concepts and mock ups of user interfaces. We would like to actually load the graphical interface; it would be embedded within the survey so people can view it and make comments and answer questions."
For some of its on-line research, Nickelodeon uses Digital Marketing Services (DMS), a Dallas, Texas, firm that gives clients access to America Online’s (AOL) 12 million users through AOL’s Opinion Place, a member area devoted to research. Visitors to Opinion Place provide screening information about themselves which enables them to be considered for a wide range of surveys.
For Nickelodeon’s surveys, parents of nearly 1,500 7-to-14-year-old kids were recruited to help their kids complete a survey on their on-line habits, including their favorite on-line locations (both Web sites and AOL areas), their impressions of Nickelodeon, etc. "Our aim was to segment kids in terms of how they use on-line and find out what they like or don’t like about given areas in sites," says Dave Charmatz, director of Nick at Night and TV Land research for Nickelodeon. "Kids are hard enough to talk to anyway, and with our strong presence on AOL it made sense to do it on AOL and the Opinion Place seemed like the perfect place to do it.
"All the controls that you have on a regular survey are there. If you want to make sure you don’t oversample in a certain age group, you just put those requirements in up front. You can fill out many questionnaires in a short period of time; we can do 1,500 kids in a week."
Involving the parents was key, Charmatz says. "AOL overall has relatively strict guidelines on talking to kids on their service. Our standards are even more rigid than theirs are. But any time you’re talking to kids, especially the young ones, you need to involve the parents because of their ability to type and help answer the questions. As far as parent supervision, we want to make sure they’re comfortable with what we’re asking the kids, and make sure that anything we’re asking isn’t too confusing. We’re very cognizant of on-line security and we don’t ask anything very personal. And since it is done in a secured area on AOL, they’re more relaxed about the process."
Like Bay Networks’ Sanchez, Charmatz cites the ability to incorporate graphics into surveys as a welcome aspect of on-line research. "We used miniaturized front screen captures of the Web sites and on-line areas we were asking the kids about, which was a big plus for us because we felt that it limited some of the confusion between complementary Web sites and AOL areas."
"We were able to use images and logos as reference points to help the kids identify if they had visited the site, as a point of reference when we were asking them about it," says Shelly Bracken, senior director, business development, DMS.
Caveats
On-line research has a great future, Charmatz says, as long as people keep some important issues in mind. "For certain types of products it’s ideal. But overall, on-line has a ton of caveats that have to be assumed going in, especially when you’re looking at other people’s research. If you know what those caveats are, that’s fine. Any research has its natural biases; on-line has more than just one or two.
"We use on-line because we want to talk about on-line to kids who are on-line and we use AOL because it’s such a major presence; two-thirds of kids on-line are using AOL to some extent. It’s the best place for us to do this kind of research. That’s not to say we won’t do research on the Web using the nth visitor or something like that. But then there are other issues with security on the Web where it’s a lot harder to get parents to sign off on letting their kids take surveys. In this environment we know the parents are more willing to give us that kind of permission."
In one-PC, one-phone line households, many kids are allowed only a limited time on-line. So they may not want to take valuable time to conduct a survey, Charmatz says. "There are some issues with self-selection, but DMS tries to go back and sample off-line as well as on-line to see that their sampling procedures match the overall AOL universe and how it projects to the national universe."
The capabilities of on-line research are growing (to include more advanced graphics presentations and audio and video streaming), but unless home computer technology keeps up, the advances won’t mean much. "If the respondent can’t handle all the graphics and data because they are using a 14.4 modem and a 386 machine, it’s not that useful," Charmatz says.
Respondent control
As these examples show, though on-line research isn’t yet suitable for surveys of the general population, it is a great way to reach specific audiences. And in many cases, it allows respondents to control when they participate in the research process. "Over my years in research, I’ve found that most consumers truly do want to give their opinion. Most are interested in companies providing better things for them, but they want to give us that information at their convenience, not when we want them to," Bracken says.
In addition to controlling when they take the survey, on-line respondents can often control how long they spend completing it. They aren’t at the mercy of an interviewer on the other end of the phone line. "That’s important because one of the things that has hurt our industry the most is getting people on the phone and saying, ‘I just have a few questions for you’ and keeping them on the phone for an hour," Bracken says. "They decide they aren’t going to do that again. We’re up-front with respondents about the length of the survey and they are there because they have the time to take the survey. When they are ready to give their opinion, the survey is available for them 24 hours a day, seven days a week.
"We know how difficult it is to interview anybody, especially kids. People are not answering the phones, they’re not in the malls stopping for surveys. As an industry we do research to find out what consumers want to tell our clients. With the rising non-contact and refusal rates, consumers are clearly telling us that [on-line research] is what they want and yet we are the last to take our own advice. We ask our clients to start listening to the consumer. I think we in the research industry need to start listening to the consumer as well."