Editor’s note: David Bradford is vice president of international sales and marketing of Itracks International, a Canada-based developer of online research software.
It would be difficult to find another research methodology that has overcome the challenges and objections encountered by online qualitative research. When the first groups were conducted (around 1994), they met the typical resistance and skepticism any new, unproven research methodology must overcome. In addition, potential researchers were faced with a number of obstacles: a limited sample, a myriad of technological complexities, criticism over the lack of non-verbal clues, and few ways to learn how to properly conduct the research. However, the potential of faster turnaround, lower costs, and greater convenience were too powerful to ignore, and better research software tools, along with improvements in Internet expansion and connectivity, fueled growth. Recent academic research findings and conference case studies provide compelling evidence that results comparable to traditional methods can be achieved with proper online qualitative methods in appropriate applications.
Comparing online and traditional methods
Miller and Dickson (2001) refer to important academic research conducted by the AC Nielsen Center at the University of Wisconsin that provides the most complete analysis and comparison of traditional and online qualitative methods to date. Nine groups in each of three modalities (in-person focus groups, online focus groups, and telephone focus groups) were conducted and the findings compared. According to Neli Esipova (the lead author of the study, now with the Gallup Organization), three main research findings stood out and dispelled certain misconceptions regarding online groups. First, although there were observed differences across modalities in the proportion of speech segments in which participants interacted with each other, these differences were not statistically significant. Second, the proportion of strong (positive or negative) words was significantly higher for the online groups. Finally, there were no significant differences in responses to the specific sensitive questions presented to these groups. “The interaction that occurs does not depend on the modality as much as the people who are involved in the discussion,” Esipova observed. “The online dynamics differ because there is less influence from others’ opinions and more equal opportunity to respond rather than waiting for their turn to speak.”
These findings echo a number of presentations delivered at recent professional conferences that document comparisons of online and traditional methods. According to Tim Stehle, senior director of research at KnightRidder.com, there is virtually no difference when it comes to the integral components of a study, regardless of the methodology. His experiences with traditional focus groups were compared to a series of 25 online groups that dealt with branding, consumer benefits, and other marketing research issues. According to Stehle, “Whether real or virtual, a study needs to be conducted in an appropriate research environment with a qualified moderator to manage the project and deliver a quality report.
“The recruiting matters a lot,” he adds, “and it takes time to be effective. Just like traditional focus groups, you are going to have some respondents and sessions that are duds, but at least you don’t waste a day getting there and another day getting home.”
Maritza DiSciullo, director of market intelligence for AT&T Broadband, has presented the results of parallel research studies conducted within AT&T Customer Sciences. Their purpose was to compare several traditional data collection methods with online data collection methods, and examine any differences in responses based on the type of study (customer satisfaction, advertising, product concept evaluations, etc.). Out of six customer segments where traditional focus groups were conducted, three were identified as appropriate for a comparison and online focus groups conducted with them.
The study concluded that face-to-face and online qualitative research methods lead to nearly identical study results. While it was felt that slightly more depth could be gained through face-to-face groups via interpersonal communication, online research still derives the same main perceptions, opinions, and concerns from participants. Participants in both types of discussions bonded with one another and felt open to express their thoughts and opinions and had nearly the same reactions to the concepts and stimuli. Recently completed sets of parallel online groups resulted in mirroring these findings with small business owners.
“We all need to realize that we are in the beginning stages of online qualitative research and still finding out some of the things that we need to do,” DiSciullo says. “When used with the right audience and with an experienced Internet moderator, online groups are just as reliable as in-person groups. The onus is on the researcher to make sure it is an appropriate application, recruiting is properly conducted, and the moderator has the right skill sets and experience for online groups.”
Moderation in all things?
According to Sweet and Walkowski (2000), very few QRCA moderators they surveyed had any experience moderating online groups, and even fewer had done more than a handful. The fact that so few had successfully made the transition to online moderating may be due to either technical or philosophical reasons. Some might be resistant to learn the new technical skills, or alternatively, feel they are unable to provide meaningful analysis without the non-verbal communication clues they been trained to rely on.
Most moderators will find it easier to conduct bulletin-board focus groups, as they are more technically forgiving and don’t require the fast typing skills essential for a live chat-based group. According to Naomi R. Henderson, president and founder of RIVA, a firm that provides moderator training, new online moderating courses were added to accommodate researchers wanting to increase their skills and develop good practices in this area. “A truth about online moderating is that the moderator who does this well is one who has shifted their paradigm about what moderating is,” says Henderson. “A good online moderator needs to be a good face-to-face moderator first and then can be coached to fine-tune their skills, prevent the adoption of poor techniques, build confidence and be able to offer a more diverse set of skills to their clients.”
Jeff Walkowski of Qualcore Inc. teaches classes based on his experience with both chat and bulletin-board groups and conference workshops he has conducted on the subject. “There are still some limitations on interventions that can be used during an online focus group,” he says. “Some of these interventions, projection techniques for example, can be modified for the online environment if they are simplified and used in a more basic form. Studies that require tactile reactions to stimuli would not yield the same feedback that a traditional setting provides. Other interventions such as picture sorts and collages may eventually become technically possible but are, at best, very cumbersome to execute. It is possible to send stimuli or homework assignments to respondents before the groups, but it adds complexity to the process.”
Raw recruits or well done?
Successful recruiting for online or traditional qualitative studies requires additional steps to locate the best possible respondents to provide meaningful insights. In a traditional focus group, participants who are unable to articulate their thoughts and feelings or unable to speak the language would be considered unacceptable. The same holds true in the online version, particularly if respondents cannot type well or are ineffective in expressing their emotions, thoughts and feelings while using a keyboard. Screening questions for online qualitative studies must address their Internet comfort level, chat room or e-mail experience, and include a typewritten response to an open-ended question. Asking these questions before the session begins allows you to avoid having unqualified participants in your study. Other criteria when recruiting for online qualitative include determining if they have adequate computer and Internet connectivity, and screening for any bandwidth or plug-in requirements.
Online opt-in recruiting sources may not be cost-effective for smaller projects or those requiring rapid turnaround. Client-supplied sources including existing customer databases or Web-based random intercepts are fast and very cost-effective for many studies. For smaller online qualitative projects some electronic panels are effective, as is traditional telephone recruiting (which also provides an important validation component). Open-ended questions in Web screeners or sent by e-mail will help to confirm both a potential panelist’s Internet capability and their ability to articulate a meaningful response.
“Response and show rates will vary widely depending on the recruiting source, incentives, and the motivations of the respondents,” says Ira Goodman, vice president at Custom Research. “Over-recruiting, reminder e-mails and phone calls, and additional screening may add slightly to the costs and preparation involved in an online study, but are preferable to a disappointing turnout and inadequate findings.”
Right tools for the job
Online qualitative is continuing to evolve into an array of new methodologies that should leverage the uniqueness and power of the Internet in appropriate research situations, rather than virtually imitating traditional methods. Any new online methodology used must still be the best match for the desired research objectives. One such application is online one-on-one interviews while presenting stimuli. “Favorable comparisons of online and traditional focus groups with college students resulted in our applying the technology to another segment of our Internet-based customers,” says David Vázquez, who conducts qualitative research for CDNOW. “In a single day, we’ve conducted individual interviews with key international customers that would have been impossible using traditional means within the available budget and timeframe.”
At the other end of the size spectrum, the growth of large participant bulletin-board focus groups is taking online qualitative research into entirely new directions. In recent concept testing for the college market, a 60-person bulletin board focus group was conducted over a five-day period. According to Juan Garcia, vice president of the youth market research consultancy U-30, “We asked six segmented groups of college students a series of 40-50 questions which needed detailed verbatims that we could probe further. We were impressed with the amount of quality data we collected and the extremely high participation rates of the target market.”
The beginning of online qualitative studies presented early researchers with formidable challenges, many of which have now been overcome or minimized. Today’s researcher has better technology, a more diverse online population, numerous training opportunities, and sophisticated online research tools developed specifically for their needs and creative uses. We are just beginning to realize the potential for online qualitative research, and our understanding will increase as more online qualitative studies are conducted and the findings analyzed.
References
Miller, Thomas W. and Peter R. Dickson, “On-line Market Research”, International Journal of Electronic Commerce, Spring 2001, Vol. 5, No. 3, pp. 139-167.
Sweet, Casey and Jeff Walkowski “Online Qualitative Research Task Force: Report of Findings”, Quirk’s Marketing Research Review, December, 2000 (accessible online using the Quirk’s Article Archive at www.quirks.com).