Overcome these common hazards to seize their potential

No more mixed reviews on your mixed-method studies

Editor’s note: Ellen Cabacungan is a qualitative manager and moderator at CMI, an Atlanta research company. Hannah Baker Hitzhusen and Janet Patterson, also of CMI, contributed content to this article.

Many marketers and market researchers have come to embrace the benefits of an integrated mixed-method approach - multi-phase studies that interweave several different types of research - as it can provide a more holistic, robust perspective than a single approach. For example, conducting qualitative research to guide the development of a quantitative survey can help ensure that all the real issues are examined, the right factors are evaluated, the right terms are used and the sampling strategy is well-constructed. Similarly, an early quantitative phase can help determine the key issues and targets for subsequent qualitative research to probe deeper into outstanding questions, provide a greater understanding of what ambiguous data mean or bring a key audience to life.

Ideally, each phase of a mixed-methods study elegantly builds on insights from other phases. Too often in practice, however, rather than truly integrating all research components, studies are often treated as series of discrete projects strung together, with too little anticipation of the complexity of aligning each phase and effectively leveraging all insights upon conclusion. It is this complexity that makes mixed-method studies rife with potential pitfalls; there are more opportunities for things to go awry and a greater chance that issues will compound.

Among the potential hazards are:

Budget overages and missed deadlines. “The report from the first phase didn’t include the input we needed for the second phase so we had to go back into the data, revise the report and delay the next round of fieldwork - at an added cost.”

Failure to meet the study’s objectives. “It wasn’t clear whether certain objectives would be addressed by the qual or the quant. In the end, some key questions remained unanswered.”

Redundancy. “Too much time was spent in the qual phase addressing questions we already knew the answers to from our survey, so we didn’t really get much out of it.”

Inconsistency. “The terminology, concepts and models used in the qual report were totally different than those used in the quant report, making it difficult and time-consuming to summarize the whole study and understand all of the implications.”

Confusion. “The findings from one phase appeared to totally contradict what we learned in the other phase because each was developed in a vacuum, so it was impossible to take action without understanding the discrepancies.”

Frustration. “Even though we worked with a single research supplier, I felt like I was the only link between the qual and quant teams and had to spend too much of my time bringing different contacts up to speed to ensure consistency throughout the study.”

Conscious and observant

Many of the solutions to issues with mixed-method studies described below may seem obvious, but simply being more conscious and observant of prosaic, sound research practices can prevent hours of additional work, panicked eleventh-hour revisions or awkward sessions with stakeholders.

1. Solidify key study elements and objectives. Researchers may be tempted to breeze through a study’s objectives, parameters and deliverables without careful deliberation because the exercise can seem rote and formulaic. However, these areas are especially critical to ensure all parties have a clear understanding of how the many moving parts must work together through multiple phases.

Furthermore, because a number of stakeholders with different priorities are often involved in mixed-method studies, it is critical to get their buy-in and agreement on the objectives from the outset. Begin mixed-method studies by clearly articulating the business objective the research will ultimately address, then determine the requisite research objectives, parameters (recruiting specifications, sample sizes, etc.) and deliverables. It is crucial that the research objectives and deliverables be specified for each phase of the research to avoid confusion about what each phase must accomplish.

Established processes must be in place to ensure all involved parties are consistently mindful of the key elements of the study. For example, at the outset of a study we create a project map - a brief document that includes the methodology, timeline, and key deliverables - along with a chart outlining the key questions, tools and techniques to achieve each research objective and focus the research on addressing the ultimate business objective. Once we have finalized the project map with our clients, we reference it throughout the study and encourage our clients to do so as well when reviewing screeners, guides, questionnaires or reports. We have found that this practice ensures the study’s objectives are satisfied and provides a framework to manage stakeholder expectations and prevent scope creep.

2. Select and use methods appropriately. Some stakeholders may find mixed-method studies attractive because they sound new and intriguing. For example, they may advocate an in-store shop-along and discrete-choice modeling hybrid methodology when a short customer survey would suffice. In contrast, some may be uncomfortable with certain methods, for example, embracing quantitative methods for “scientific analysis” with large sample sizes over an ethnography of a smaller sample of representative respondents. In other instances, stakeholders may try to loosely mix methods, using tactics like adding many open-ended questions to a survey “to get some qualitative responses” instead of conducting in-depth interviews.

To avoid these scenarios, urge stakeholders to be open but cautious when considering new approaches and to carefully evaluate a potential methodology based on how sufficiently and efficiently it addresses the business questions. Educate stakeholders who are hesitant about new approaches using analogies and layman’s terms to explain complex techniques and by offering case studies and examples of output. For stakeholders who are eager to try an interesting but unsuitable method, discuss potential limitations and propose creative modifications to engage them in the optimal approach.

3. Carefully determine method sequence. Often, the order in which methods are employed is just as critical as selecting the right methods, as sequence can dramatically affect outcomes, timing and cost. For example, when assessing the opportunity for a new product concept, conducting a survey to determine who to target followed by focus groups with those consumers to refine the concept would likely lead to very different conclusions than if the study began with focus groups. Participants in the survey sample who would have rejected the original, unrefined version of the concept might have embraced the optimized version, and vice versa. Similarly, reactions to the concept among focus group participants would likely be very different if the groups were composed of a broad sample of consumers, versus a highly targeted sample of the category’s early adopters.

So, when designing a mixed-method study, it is important to identify the specific function of each method to be employed, as functions can indicate the proper sequence. In their 1989 Educational Evaluation and Policy Analysis article titled, “Toward a conceptual framework for mixed-method evaluation designs,” Jennifer C. Greene et al. articulated five functions for conducting mixed-method research:

Complementarity - Elaborating on, enhancing or clarifying results from another method.

Development - Using results from one method to inform the development of another.

Initiation - Discovering contradictions in results or new constructs with which to interpret results from another method.

Triangulation - Validating results from one method with another.

Expansion - Extending the breadth and range of understanding by using different methods to study different components of an issue.

We recently conducted a mixed-method study that illustrates the influence of function on the study design. A pharmaceutical client needed a new-patient segmentation in a particular treatment category that requires patients to modify their behavior to improve their conditions. Although our client’s previous quantitative segmentation had been predicated on focus groups, it failed to include important factors that distinguished real-world patient types, provide a means for envisioning or prioritizing the segments or help accurately identify patients who were truly ready to modify their behavior and try the client’s prescription drug.

In order to develop a more meaningful and actionable segmentation for our client, we combined several phases of qualitative and quantitative research (Figure 1).

In an example of initiation, we screened participants in all phases using an algorithm from the previous segmentation so we could later demonstrate where and why the new segmentation overlapped with or diverged from the old one.

Because focus groups conducted to inform the last segmentation survey had not identified actual indicators of patients’ readiness to modify their behavior or take a prescription drug, we sought expansion, employing additional qualitative methods to thoroughly understand factors that contribute to readiness. After participating in an initial mini-group, patients in the sample who said they intended to modify their behavior in the next month completed journals for a number of weeks before reconvening to debrief on what occurred in the period between discussions. Adding this longitudinal element allowed us to identify factors that indicated whether a patient would actually take action to modify his or her behavior in the short term.

We then leveraged the qualitative findings for development of the quantitative instrument and selection of the new segmentation scheme.

The quantitative served the purpose of triangulation, validating the qualitative findings with a larger more representative sample.

Finally, to achieve complementarity, we re-contacted participants from the qualitative sample and gave them an algorithm-based typing tool to determine each individual’s segment. This allowed us to create rich profiles of each of the new segments by augmenting the quantitative data with qualitative artifacts from earlier phases.

4. Carefully plan the timeline. Mixed-method studies can be more efficient than conducting each phase as a discrete study, particularly if recruiting or reporting for several phases can be combined. However, certain steps usually require more time than is typically allotted for a single method because more parties and components are involved. In the diagram representing the mixed-method segmentation study described above (Figure 1), it is evident that a number of touchpoints are required for the research and client teams to successfully coordinate a study of that scope. Clearly, without proper planning, issues in one phase could have delayed later phases or compromised the quality of the outcomes.

It is often possible to start one phase before the preceding phase is complete to compress the timeline. However, not allowing enough time to understand the implications from one phase and modify a later phase accordingly can negate the purpose of conducting the earlier phase - and at a much higher cost than if only the later phase had been conducted.

A colleague had a recent experience with a client who was facing tremendous pressure to have statistical data in hand for an upcoming meeting. Therefore, the launch of a survey to evaluate product claims had to be moved up, without the benefit of focus group results that would have refined the claims and reduced the number under consideration. Fielding the survey prematurely meant evaluating over 40 unrefined claims, a longer field period and higher sampling and incentive costs due to high dropout rates from respondent fatigue. Had the survey launched a week later as scheduled, it would have been possible to leverage the qualitative findings, which revealed that many of the “best performing” claims were table stakes in the category and that only a handful were resonant and differentiating enough to evaluate further. As such, the survey could have been much shorter and the results less confounding.

Even when different phases can be conducted concurrently, it is critical to ensure enough time during analysis and reporting to thoroughly integrate results from all phases. To avoid problems when developing a timeline:

• Include time upfront for coordination with the full research team and client.
• Build in reasonable time for information exchange at key points.
• Allow for contingency between phases for unforeseen delays.
• Specify the nature of and timeline for all deliverables.
• Ensure stakeholders are aware of and agree to delivery dates.

5. Maintain consistency but change course if necessary. Because mixed-method studies often involve multiple stakeholders and extend across several months, there can be many opportunities for the research to lose focus.

As new business questions emerge over the course of the study, there is often a temptation to use later phases to address topics beyond the original scope. It is sometimes possible to accommodate requests to include other topics, saving the client the time and cost of fielding additional studies. However, doing so can be detrimental to meeting the study’s original objectives if the inclusion of additional topics leaves less time for data capture and analysis fundamental to the original objectives or diverts focus from the purpose of the study.

Additionally, different team members may engage in the study on a limited basis, or new researchers or stakeholders may join a team after a study is underway. Without sufficient knowledge transfer regarding the study’s history and purpose, they may inadvertently direct the study off course.

Furthermore, parties involved in one phase of the research may use different models or terminology than used in other phases, making findings impossible to align and interpret.

To ensure greater consistency throughout a mixed-method study:

• Use a single supplier for all phases of research, or collectively engage all suppliers from the outset and provide ongoing updates.
• Identify at least one individual from the supplier and one client contact who will be heavily engaged in all phases of the research to promote continuity.
• Ensure all team members and stakeholders are familiar with and agree to the project map or research plan and revisit it frequently.
• Consider including a phase’s objectives and deliverables on the respective discussion guides or draft questionnaires.

Hazards can be avoided

By nature, mixed-method studies present a number of common hazards with significant consequences. These hazards can be avoided with careful consideration and planning, yielding a study that is greater than the sum of its parts. At its best, mixed-method research can provide a richer, more comprehensive understanding of consumers and markets to guide sound business decisions.

Choosing a mixed-method supplier

Suppliers who successfully execute mixed-method studies generally:

• Specify how and in which phase they will address each of the study’s objectives.
• Indicate how they will leverage the deliverables from each phase in subsequent phases (e.g., “Phase 1 deliverables will include a comprehensive list of desired product attributes in consumer-friendly terms for use in the Phase 2 survey…”).
• Appoint at least one team member who will be engaged in all phases of the project to ensure continuity.
• Involve members from both their qualitative and quantitative teams throughout the research, from planning to fieldwork to final reporting.

Suppliers should show that they can plan and allow sufficient time for transfer of knowledge between teams at various points in the timeline. They should also be willing to express concern about an aggressive final delivery deadline and propose alternative solutions to prevent a potential delay in one phase from jeopardizing later phases.