Guideposts for a fresh path

Editor's note: Nicole Freund, consumer research manager at The Coleman Company, Wichita, Kan., will be leaving the firm this month to pursue a Ph.D. in community psychology.

Sometimes an attendee at a conference does not need the heights of a 20th-floor ballroom or an open bar to feel dizzy. The many fine researcher conferences that pepper the calendar every year are so chock-full of presentations on new qualitative methods and cases that extol their virtues that making sense of it all can feel like a Sisyphean task. MROCs, mobile qual, immersive technology, virtual engagement ... it can be overwhelming and mind-altering. The aim of this article is to peek behind the curtain and ground some of these techniques in reality so users know what is useful and when.

As in all qualitative work, context is important and the birth of these approaches comes from an important demand. It is not simply about the evolution of the craft, though that certainly is a part. The cry for innovation in general rings loudly and insistently and, depending on where someone sits, it is alternately a banshee, a siren or a choir of angels.

As the public increasingly demands product innovation, the researchers who support those product developers feel the pressure just as acutely. They need to provide the data to support that innovation or even to create it – and with declining resources, generally speaking. Demand for knowledge that costs less but delivers more and even delights has sparked a deluge of new ways to gain access to the hearts and minds of consumers.

Enter innovation in data collection and, more specifically, in qualitative techniques, which for many are the preferred vehicles for driving into the garage of desires owned by modern consumers. The pressure to perform to standards that change daily and are difficult to define makes new techniques attractive. But at what cost to the data collection? What are the methodologies available? How do they compare to traditional methods? Can they deliver on the same objectives or must those objectives be modified?

The scope of this article is not to try and answer every question that may possibly come up in the course of a discussion of new qualitative techniques. What I do hope to accomplish, however, is to provide some guideposts for how these techniques stand up against each other and against the traditional qualitative methods available.

The most productive way to go about evaluating these methods is to classify them: those that mimic traditional methods but use non-traditional ways to access the participants; and those that really have no direct comparison to traditional methods.

Classifying the methods in this way allows those trying to evaluate the utility of a method to use their own internal benchmarks and experience.

Measurable with a yardstick

First, a quick note of definition and discussion regarding traditional qualitative methods like in-person focus groups and IDIs. What makes these valuable are the connections that moderators, researchers and clients form with consumers on a given topic. Having a conversation (or witnessing one) where the distance is measurable with a yardstick offers an intimacy and ability to react that is unique to traditional qualitative. There is truly no substitute for those connections and in some cases, no way to supplant an in-person interaction with a virtual one. And sometimes logistics actually compel traditional interactions; if the client has only one prototype or it’s a product that must be played with or if the target is children, traditional methods will generally win out.

For these reasons, it is important to see new qualitative methods as supplements or additions to the toolkit, not as wholesale substitutes – at least not as wholesale substitutes that come without significant trade-offs. Comparisons made here of what a new method does well will generally be benchmarked against the objectives that traditional qualitative delivers well and the challenges with the same (see Table 1).

These are not new observations but serve to set any evaluation on even terms through the course of this discussion.

Traditionally-inspired methods in unconventional venues

Newer methodologies that build on the successes of traditional qualitative techniques typically exploit efficiencies in technology to deliver on similar objectives. In this category are the following: marketing research online communities (MROCs), bulletin boards and video/Web groups.

In each case, surveys and discussions leverage the Internet to expand geographically the reach of what could otherwise be done locally. In a lot of cases, the work done in these techniques is not totally new but the reach and speed are. That said, there are distinct differences between them to note and it is a worthy exercise to explore them individually.

MROCs are proprietary communities of advocates (either brand- or activity-based) to which companies can send surveys, initiate discussions and crowdsource decision-making. There are really two kinds of MROCs: 1) ongoing, branded communities that are built for long-term use and 2) instant communities that are built for a specific project and then disbanded. Each will be discussed in turn.

Ongoing, branded MROCs

What they do well: Brand-building, crowdsourced innovation, quick-turn reactions to specific questions.

Objectives that work well: Testing messages with a target market, understanding attitudes and usage within a target market, gauging interest from a target market, connecting with a target market (notice the theme?).

Caveats: There is a temptation to mistake survey results as quantitative data – the population of this community is not representative of anything beyond the target market criteria outlined at the recruitment stage. Results of any surveys are qualitative in nature even if there are hundreds of responses. That’s not necessarily an issue depending on how bulletproof the organization needs the information to be but it is an important distinction. Also, unless the organization can afford full service, these MROCs are extremely resource-intensive to build and maintain the right way. Making sure that there is enough stimuli but not too much; that any gamification is maintained; and that the health of the community stays at a productive level takes staff and budget.

Best for: Objectives that include longitudinal goals with a continuous stream of stimuli; organizations that have dedicated personnel who can perform community management.

Bottom line: Branded MROCs do a really good job of reacting quickly, efficiently and cost-effectively to “daisy” issues (those that pop up unexpectedly) and can be a great resource for tapping into the stream of consciousness of a defined target. However, undertaking their creation should not be a snap decision based on a desire for efficiency.

Instant MROCs

What they do well: Create a fun and interesting forum for discussing an issue; quickly and cost-effectively convene a group of consumers to discuss and react to stimuli.

Objectives that work well: Almost any objective that would work in a focus group works well in this format as well: reactions to concepts, open-ended responses to stimuli.

Caveats: The stimuli have to make sense for the delivery system. If the stimuli are physical items that truly require 3-D perception, this is the wrong venue. Video and images work well but don’t necessarily convey everything needed in all cases. Additionally, there is no way to react to body language. While not necessarily asynchronous like some other methods, with instant MROCs researchers must still trust the words being typed without the benefit of body language or tone of voice to confirm or deny the sentiment.

Best for: Issues that require feedback quickly and don’t have complicated stimuli.

Bottom line: Instant MROCs are effective for those projects where traditional qualitative would typically be used but in-person techniques are too time-consuming or expensive for the current issue.

Bulletin boards

Bulletin boards (BBs) are similar to both MROCs and online focus groups in that they provide a common place for people to discuss an issue in a convenient, electronic format. Typically, BBs are much less complicated and therefore easier to set up and manage; they function as asynchronous chat rooms and do not require the same degree of involvement though they do require an experienced moderator to keep the conversations moving and on track.

What they do well: Inexpensively provide a place for people to discuss an issue.

Objectives that work well: Concept development where strong opinions may be present. For example, reactions to potential changes in political policy or in how a product solution might work in a given situation.

Caveats: Recruitment for BBs is tricky. Participants really need to be able to express themselves exceptionally well since there’s typically little in the way of imagery or other nonverbal communication. Anyone who has tried to find the answer to a service question on a BB can attest to this. If the participants are difficult to understand or they use only short, nondescript answers, the usefulness of the data is decreased.

Best for: Issues where there are strong opinions and an established population from which to recruit.

Bottom line: Instant communities and other asynchronous methods have really overtaken this method because of their flexibility in stimuli and reduced need for verbal acuity. That said, this is still a useful tool for established groups to exchange ideas, especially when opinions about the solution to an issue are diverse and passionate.

Video/Web groups

The most closely related to traditional qualitative techniques, video and/or Web groups tend to offer some of the best of both worlds. The groups can be geographically diverse yet still let clients see physical reactions and tonal cues. Additionally, these tend to be more cost-effective than in-person techniques due to reductions in travel and other costs associated with traditional methods. However, some of the efficiencies of time that come with community or BB work are not available with Web groups because the time to recruit, execute and analyze the groups is similar to what it would be in person.

What they do well: Offer a cost-effective way to execute a traditional focus group.

Objectives that work well: Any objective for which the organization would typically initiate a traditional in-person methodology.

Caveats: Similar to instant MROCs, the stimuli used need to coincide with how they will be delivered. Depending on how the groups are set up, video or images may or may not be sufficient. If the group is a videostream for the client but the participants are all in the same place, then shelf setups or prototypes may be used as they would in a traditional setting. Care simply needs to be taken in considering how, where and when the groups will convene.

Best for: Projects that really require a more traditional setup but need the additional flexibility the Internet can provide either in recruitment, client needs or stimuli.

Bottom line: If an organization is comfortable with qualitative techniques in general, there’s no reason not to seek out suppliers of video or Web groups.

Non-traditional techniques

Some of the newer innovative techniques in online qualitative work are simply not comparable to traditional methods because they rely so heavily on the technology or platforms that are used. They may also be the techniques that are most confusing or risky to some researchers. The techniques most often associated with this classification are mobile qualitative, immersive online platforms and online journaling.

There are a couple characteristics that are common to these three and contribute to their effectiveness relative to traditional means of data collection. The first is the asynchronous nature of the collection. Convenience is valued very highly by modern consumers and the ability to participate when time permits (rather than at a specific time determined by the researcher) can lead to greater engagement and more candid responses. Additionally, being able to respond to stimuli in situ rather than in a contrived environment can add considerable value to the study.

The second characteristic is the limited nature of the available pool of respondents. This affects online journaling to a lesser degree than mobile qual or immersive platforms but researchers should understand that the respondents who feel comfortable enough to participate in these types of studies are still relatively unique. Some of these consumers would likely be included in traditional methods anyway but not all those who would participate in a focus group would be able (or willing) to participate in a mobile study. It’s not that this means those methods are less valuable; it just means that researchers need to be aware of the potential differences in respondent pools.

Mobile qualitative

What they do well: Cost-effectively capture thoughts, opinions, images and other data at the point of experience. Consumers have to rely less on recall and can more accurately relate how long it took to find a display, how often they heard a particular phrase or share what they particularly liked about a given experience.

Objectives that work well: Testing concepts, products or messages that are already on the market. These may lead to new ideas or concepts but the experience needs to be readily available in order for mobile to make sense.

Caveats: It’s easy with some platforms to get hundreds of responses and there is the temptation, much like with MROCs, to count those responses as quantitative data – this would be a mistake. Also, some platforms make analysis easier than others, so it’s worth exploring with a supplier how results will be delivered. If it is too time-consuming to digest or use the information, then it may not be worth the investment.

Best for: Getting real-time feedback on issues, environments and concepts that are already available and might suffer from deterioration if recall is too far removed from the experience itself.

Bottom line: While in no way traditional, the ability of researchers to access thoughts or opinions closer to the actual experience make them a valuable addition to many research arsenals.

Immersive online platforms and online journaling

Immersive online and journaling tend to go somewhat hand-in-hand. Neither is dependent on the other and researchers can absolutely use them independently. However, the value they each provide feeds the other. Many immersive platforms utilize or offer journaling and many journals feed into other activities supported by online immersive platforms. The end result is typically a kind of ethnographic peek into the lives of consumers that facilitates more intimate levels of sharing.

What they do well: These techniques also cost-effectively capture thoughts, opinions, images and other data at the point of experience as well as within the context of consumers’ lives in general.

Objectives that work well: Studies that might traditionally look to ethnography for in-depth representation of consumer lives within certain contexts like health and wellness, outdoor experiences or shopping excursions.

Caveats: While there are aspects of ethnographic research that are a part of how this data is collected, it is not ethnography and shouldn’t be considered as such. If what the organization truly needs is ethnography, this technique can inform and potentially direct what that looks like but it should not be considered a replacement for a trained ethnographer experiencing the same things as the consumer. Additionally, these techniques tend to generate a lot of data, so having a plan for how to attack the analysis is essential.

Best for: Projects that need a lot of depth and the ability to dig past surface impressions without going to the expense of full ethnography.

Bottom line: If an organization has to choose one type of new methodology to try, this one should be top of the list. It is a very cost-effective and valuable technique to add understanding, dimension and context to a business issue.

Supplier-client partnership

One thing to keep in mind is that a lot of these newer techniques are billed as DIY opportunities for organizations to do research without the cost of going to full-service suppliers. That is certainly true and can be very effective, but there is really no substitute for third-party objectivity.

The best scenario is one where there is a partnership between a supplier and the client so that the cost-effectiveness of these techniques can be fully realized while at the same time the supplier maintains the helpful objectivity that comes from not being entrenched in the day-to-day operations of the client. This partnership might mean that the client does everything but analyze the results and write the report or perhaps the client hires a professional moderator. Whatever the right balance is, having that balance will inherently boost the value of any of these techniques.

There is power and utility in the emerging world of qualitative data collection; seizing that power is simply a matter of making sure the technique fits the objective.