‘You can’t do that, can you?’

Editor's note: Michael Heasley is partner and chief research officer at Evolution Marketing Research, Blue Bell, Pa.

As marketing researchers, many of us have found ourselves in the position of wanting to conduct a conjoint exercise in the qualitative setting. The reasons vary. Perhaps we have a limited budget and need to meet multiple goals (a very common problem in our current economic environment). Or maybe the results of a quantitative conjoint study require further explanation. Or, it could simply be that the best way to meet the team’s objectives would be to conduct a conjoint exercise in the qualitative setting. Whatever our reason, many researchers are met with a very familiar response from marketers and senior management alike: “Huh? You can‘t do that, can you?”

This type of objection has led us to find other compromises as opposed to convincing others that it can, in fact, be done. These solutions usually take the form of completing a quantitative and a qualitative study in sequence (a sometimes very costly alternative that can lead to contradictory and sometimes confusing results) or choosing a qualitative or a quantitative solution to answer the major questions where neither is completely satisfactory.

However, it is completely possible to accomplish this kind of research as long as the appropriate parameters are put into place to ensure its success.

But how can you garner an appropriate result? How can you conduct what is most decidedly a quantitative exercise in what is very much a qualitative setting? The answers to these questions may seem a bit out-of-the-box but when you stop and think about it, they come from lessons learned in basic marketing research/statistics 101.

In my time in marketing research (both on the client and vendor sides), I have heard a number of objections to conducting this type of research. The two most common have been:

  1. You can’t possibly conduct an interview that takes you through, roughly, 12 cards (market scenarios, product profiles, etc.), plus initial interview setup, probing and final questions, in an hour AND do it right.
  2. How can you collect quantitative data in a qualitative setting? Obviously the results aren’t collected in a quantitative manner and the sample is surely too small.

Let’s examine and debunk each of these objections.

Timing. Of course, evaluating approximately 12 conjoint cards (market scenarios, product profiles, etc.) takes time on the parts of the respondent and the moderator. If the moderator is not prepared appropriately, the interview could take more than the usual 45 to 60 allotted minutes or, more embarrassingly, go off the tracks completely. However, if the moderator is well versed in both the methodology and the market of concern, then this exercise should be no more difficult than the average message-testing study. As it is, then, this objection is easily rebutted as long as the right person is on the other side of the glass.

Setting and sample size. The second general objection is not as clear in terms of its solution and focuses on two main points: collecting quantitative data in a qualitative setting; and sample size.

Let’s first discuss the idea of data collection. In order for data to be “quantitative” in nature, it must be collected in a structured manner so that the results may be tabulated and reported in terms of statistical parameters or nonparametric measures (mean responses, percentage of respondents, medians, etc.). We do something like this regularly in qualitative research in the form of self-administered questionnaires (SAQs) and report the resultant data to our clients. In fact, SAQs more than meet the criterion to be quantitative. All that is required to meet the “quantitative bar” is to have the same stimuli (such as a questionnaire) shown to all respondents so that they respond in the same exact manner (a number or a binary choice, etc.) and, therefore, an overall parameter estimate can be computed from the sample.

In the question of a conjoint, we know that the stimuli are formalized and structured so that respondents are reacting to a statistical design. Therefore, the structure required to collect quantitative data is provided via the design inherent in any conjoint study. This inherently structured nature of a conjoint provides the rigor required to collect quantitative data. In other words, all of the respondents are reacting to the same combinations of attributes and providing responses that can be collected and reported quantitatively. As long as the moderator introduces the respondents to the conjoint exercise in the same way every interview and collects the data in a responsible manner, the data collected are “quantitative.”

The more, dare I say, controversial matter with regard to this methodology regards sample size. I want to make this one point regarding sample size as strongly and as clearly as possible before proceeding into this discussion: In no way, whatsoever, do I suggest replacing a large-scale, multi-sample conjoint with a small-sample conjoint executed in the qualitative setting. That notion simply makes no sense. However, the argument can be made that we can conduct this type of research with the smaller samples associated with qualitative research.

First off, to say that a small, qualitative sample (n = 24-to-30) isn’t valid is, on the face of it, nonsense. What then would be the utility of conducting qualitative research in the first place? What’s the use of determining the whys of the market if those whys aren’t representative? As a group, we have become so ingrained in rules of thumb that we have begun ignoring certain basic concepts tied to marketing research and statistics.

A common misconception is that a sample is not “quantitative” or worse, representative, unless it is greater than or equal to 30 respondents. At first, this limitation on sample size may seem reasonable. However, the notion that a sample is more or less statistically valid or “quantitative” depending on its size is inherently flawed. For the purposes of this article, it is sufficient to note that the idea of a sample minimum of 30 stems from a concept found in high school and college statistics textbooks, that the student’s t distribution approximates the normal (or z) distribution when the sample size is 31. This notion has pervaded our industry in a rather reckless and indiscriminately applied manner.

However, discussion of sample size basically revolves around the idea of detecting population differences as well as determining representation. We are mostly concerned with representation when it comes to conjoint analysis. These ideas are more concerned with the variability of the population along certain parameters than anything else. In certain populations (in my case, the physician population), variability is VERY low. As such, representation can be achieved with small sample sizes.

In focusing on my particular area of expertise – pharmaceuticals – physicians are often (and I believe, rightfully so) self-limiting when it comes to the options of treating patients. After all, we don’t want our doctors diverging from the norm when it comes to treating us. In terms of marketing research, the notion of doctors staying near to the mean when it comes to their treatment decisions is excellent. In short, there is low variability with regard to physicians’ decision processes and, as such, a large sample really is not needed to determine physician behaviors on the large scale, as they generally behave the same when treating patients for particular conditions.

Furthermore, when the physicians become more specialized, their deviation from the norm is greatly reduced and the variability in their responses is also greatly reduced. The effect of reduced variability should be applicable to any population where there are subsamples of greater specialization, such as physicians.

We can, therefore, see that the sample size is not hugely important as long as you meet basic representation standards, randomize your sample (very important) and are representative of your target. The sample size calculation is really dependent on the researcher’s knowledge of the market and, as long as variability is generally low, a conjoint is acceptable in the qualitative setting to analyze the market.

Discuss the logistics

Now that we have (hopefully) dispelled any myths regarding the validity of a qualitative conjoint study, let’s discuss the logistics of such a study.

First, we need to make sure that the study is appropriate for the situation. The following are situations in which a qualitative conjoint may be the appropriate route:

  1. There is not enough project budget to conduct a conjoint exercise followed by a qualitative study to explain the results.
  2. The marketing team wants to know the whys behind customer decisions based on product attributes.
  3. The population of concern isn’t large enough to conduct a large quantitative conjoint.

If the resources are available to run a large study then, by all means, do so. However, if you find yourself in a situation where you need to kill two birds with one stone due to budget limitations or small physician populations, then this may be an optimal methodology.

Second, you need to sit down with your marketing team and settle on a number of attributes and levels. This discussion must keep the number of attributes and levels manageable so that there aren’t too many cards for the respondents to evaluate. This is more easily said than done. However, you cannot successfully run one of these studies without using the appropriate number of stimuli. If there are too many cards for the respondents to review, you a) will not have enough time to discuss them, and b) will wear out the respondents.

You also want to limit the number of attributes and levels to avoid blocking your design. Basically, the design represents all possible combinations of attributes (and levels of those attributes) and, if there are too many cards in the design to represent the set, you may have to block (or split) the design to appropriately manage the fatigue on the respondent. This situation is not ideal as you would have to double the sample to accommodate even one blocking scenario and simply does not work well with a qualitative study.

Finally, a market researcher needs to find the right moderator to conduct the research. A moderator must be able to maintain the discipline and rigor required of a survey while probing appropriately on the dependent variable (such as ranking, rating, etc.). The value of this component to the research cannot be underestimated.

To summarize, the necessary considerations in order to appropriately conduct this kind of research boil down to a simple list:

  1. A respondent population that is low in variability with regard to their opinions and perceptions of the subject matter.
  2. A reasonable number of attributes/levels to feed into the study.
  3. A marketing team that has bought into the process.
  4. An excellent moderator. 

The qualitative conjoint is both a feasible and valuable approach as long as all parties involved are realistic in their expectations and the study is conducted in the appropriate context.