Editor's note: Steve Herman is vice president of Bretton-Clark, a New York-based research software company. This article was written in response to Robert Roy's October, 1990 Data Use article "Conjoint evolves into discrete choice modeling." A reply by Robert Roy follows.
A recent article in this journal announced the demise of conjoint analysis ("Conjoint evolves into discrete choice modeling") - We believe this obituary is premature and that the article contained a number of significant omissions and errors.
The author first states that the designs employed in conjoint analysis impose special constraints which often lead to unrealistic product descriptions, whereas no such constraints are involved in the designs used in discrete choice analysis.
This is simply incorrect. Both conjoint analysis and discrete choice analysis use fractional factorial designs to generate product descriptions. To the extent that both techniques use the same basic designs, they cannot differ in terms of unrealistic product descriptions.
Moreover, detecting unrealistic product descriptions is actually a quick and relatively trivial matter. Respondents in a conjoint task generally require less than fifteen minutes to complete the entire conjoint task. Researchers and clients can screen the product descriptions in considerably less time.
However, this entire discussion assumes that realistic product descriptions are of major importance. In fact, a number of research studies indicate that product realism has little impact on the validity of conjoint studies. For example, one study included unrealistic combinations of engine size and gas mileage. Despite the fact that some combinations far exceeded the performance in today's market, the authors found that this had no deleterious effects on the results of the study. Although it is widely believed that unrealistic stimuli are detrimental, published research studies consistently demonstrate that this is not the case.
In fact, by modifying fractional factorial designs to increase the realism of product descriptions, you sacrifice some of the beneficial properties of these designs. For this reason, changing these product descriptions can actually lead to a loss of predictive validity.
However, we believe that product descriptions which involve "impossible" or "highly implausible" products, as opposed to simply unrealistic ones, can degrade the results of conjoint and discrete choice studies. To date, however, there is no published evidence to support even this modest hypothesis, and further research is warranted.
We would now like to discuss a difference between conjoint analysis and discrete choice analysis which Mr. Roy acknowledges but glosses over. In conjoint analysis, a respondent typically evaluates all the product profiles required by the experimental design. Therefore, the researcher can measure each respondent's utility function. (The utility function quantifies the respondent's degree of preference for each of the product features being studied, as well as the levels associated with each of these features.)
In discrete choice analysis, on the other hand, it is generally impossible to have a respondent evaluate all the scenarios required. There are two reasons for this. First, discrete choice studies often employ compromise designs, which require many more product descriptions to be evaluated. Second, and more importantly, the number of product descriptions is only one determinant of the size of the discrete choice task. Unlike conjoint studies, where product descriptions are evaluated one at a time, in discrete choice analysis the respondents evaluate scenarios consisting of sets of product descriptions.
For example, if a discrete choice study employed a simple orthogonal array requiring only 16 product descriptions, but had respondents evaluate scenarios involving four products - i.e., select which four products they most prefer - 1820 scenarios would normally be required. While techniques exist to reduce this number, these cannot reduce the number to a manageable level for a single respondent.
Therefore, even if a discrete choice study involves a relatively small number of product descriptions, the number of scenarios that must be evaluated is generally much larger than a single respondent can handle.
For this reason, discrete choice studies generally cannot evaluate each respondent's utility function, but must rely on a group-level or aggregate utility function. This has very important consequences.
If a market is segmented - that is, if consumerpreferences are not identical - use of aggregate level analysis can create serious problems. This is a matter of great practical importance, since virtually all real world markets are segmented. Statisticians and econometricians refer to this problem as aggregations error. In layman's terms, this is what happens when you add apples and oranges.
We'll use a simple example to illustrate the effects of aggregation error. Suppose we are studying a product that can be described by two features - Brand and Price - and that there are two equal sized market segments, A and B. The "true" utility functions for each segment are shown below:
Segment A Segment B
Brand A: 2.0 Brand A: -2.0
Brand B: -2.0 Brand B: 2.0
Low Price: 1.0 Low Price: 1.0
High Price: -1.0 High Price: -1.0
That is, both segments feel that Brand is twice as important as Price, but Segment A prefers Brand A while Segment B prefers Brand B. While this is an artificial example, it is similar to the situation for "mood items" such as various beverages and perfumes.
An individual-level analysis will yield the correct results, as each respondent's utilities for Brand and Price are recovered. On the other hand, an aggregate-level analysis will result in serious errors, as shown below:
Aggregate-Level Analysis
Brand A: 0.0
Brand B: 0.0
Low Price 1.0 "
High Price -1.0
The aggregate-level analysis averages the results of all respondents, implicitly assuming they all have identical preferences and utility functions. Based on this aggregate analysis, the researcher would conclude that consumers are indifferent to Brand, and are only concerned with Price. Decisions based on such analyses would prove quite costly, since Brand is actually twice as important as Price.
In fact, aggregate-level methods are inconsistent with basic marketing concepts - that the marketer should identify the needs of individual consumers and attempt to fill them. Aggregate-level methods assume that the market is totally unsegmented—a situation which rarely, if ever, exists. Thus to the extent that discrete choice analysis is conducted at the aggregate level, it can result in serious errors.
Individual-level conjoint analysis on the other hand, provides a powerful tool for market segmentation. By quantifying the preferences or utilities for each consumer, it allows the researcher to discover "benefit segments" - groups of consumers with similar preferences. Armed with such information, the marketer can position the product or develop a product line to better satisfy consumer needs.
Furthermore, benefit segments generally have a very weak relation to traditional demographic segmentation variables. Thus, traditional segmentation techniques do a poor job of capturing these segments. In addition, even if researchers conduct aggregate-level analyses for each demographic segment, serious aggregation errors can still occur.
For these reasons, we feel researchers should exercise caution in applying aggregate-level methodologies.
Until research has determined under what conditions they are preferable to individual-level methods, the dangers of aggregation error are too great to be ignored. Furthermore, they offer no advantage with respect to realistic product descriptions. While it is no panacea, we believe that individual-level conjoint analysis is still the standard for multi-attribute research methods.
Robert Roy responds:
Steve seems a tad upset. I wonder whose ox has been gored? Steve, you made the assertion that conjoint and discrete choice, "...cannot differ in terms of unrealistic product descriptions." Oh really? "Cannot" is a big word. The Earth cannot revolve around the sun. The Copernican theory cannot be correct. And Galileo cannot preach such heresy. Amen.
"Aggregate-level methods assume that the market is totally unsegmented..." Oh really? Perhaps Steve is correct when he says that traditional conjoint analysis, "...is no panacea." Perhaps he is incorrect in his assertion that discrete choice analysis is not directed towards market segments.