Editor's note: Michael Lieberman is founder and president of Multivariate Solutions, a New York based research consulting firm. He can be reached at 646-257-3794 or at michael@mvsolution.com.

Recently on Yahoo Finance, Rick Newman wrote that there is “one vital skill that transcends many jobs and may be every worker’s best shot at financial security.” The skill is conceptualization: the ability to see how elements of an abstract fit together.

Lumosity is an online cognitive training site at which you can exercise your memory and attention by playing neuroscientist-designed games. I particularly enjoy the exercise called, “Train leaving the station.” The task of this game is to use strategy so that the variously-colored trains arrive at the correct stations. As the complexity of the game increases, planning further ahead becomes necessary. In essence, this is a practical application of conceptualization.

Marketing research has a mantra: begin with the end in mind. Conceptualize what the client wants to see at the end of the project. This article will showcase the capacity of conceptualization. In our example, a fictional client, GreenPlanet LLC, makes a challenging request that is solved by creative conceptualization.

GreenPlanet is a startup public relations company that communicates to the public an organization's corporate social responsibility or environmentally-friendly practices. The goal is to broaden brand awareness and improve the organization's reputation. Now, GreenPlanet is seeking to create its own GreenPlanet Environmental Score. It hopes to market these scores to major corporations, in the manner of the JD Power automobile company ratings or Millward Brown’s ratings for brand health.

GreenPlanet established 12 general criteria for this score. These are shown in Figure 1.

 

The primary goal of the GreenPlanet pilot study is to establish which of the corporate behaviors is most important to the average consumer. We proposed a straightforward TURF analysis or a rotated pair-comparison analysis with attribute comparisons rotated. GreenPlanet agreed that this was a good first step.

Here comes the twist: our client wanted to know which of the two claims work best together. This might sound like a simple question but answering it is actually rather difficult. While TURF or pair-comparison might determine the top two claims independently, neither analysis addresses the issue of claim interaction. That is, the top two or three independent claims may not constitute the best mix.

We considered running a designed experiment – a conjoint-like exercise called repeated measure ANOVA. In this exercise, the respondent sees a list of, say, three claims, and is asked to rate the degree to which the claims impact his or her thinking. The output for the ANOVA model are beta regressions scores, which look very much like conjoint utilities.

The problem with the ANOVA model is that interactions – the f-statistic that tests combinations of claims – are rarely, if ever, significant. This means, in short, that such an analysis could yield a deliverable list of statistically insignificant maybes.

Conceptualizing the design

At this point, we came up with a twist on the paired-comparisons design that managed to meet GreenPlanet’s request. Why not show respondents two pairs of claims and query which pair they like better? This would be unusual in survey design but creativity was called for.

There were some challenges.

Combinatorics is a branch of mathematics concerning the study of finite or countable discrete structures. Combinatorial problems arise in many areas of pure mathematics, notably in algebra, probability theory, topology and computer science. An example of such a problem would be: How many different combinations of three letters can be formed from a group of six letters? (Answer: 20).

Combinatorics is often used to compute odds. A famous example is the New York State lottery. The chance of winning the New York State lottery with one ticket is one in a number of possible combinations of six numbers out of 42. There are 5,245,786 number combinations, assuming that no number can be chosen twice. This translates into one in 5,245,786, or 0.00002 percent chance of winning the lottery. A 50 percent chance of winning the jackpot would require the purchase of 2,622,893 individual tickets.

How many possible individual combinations exist for our paired-paired design? That is, how many two-attribute combinations are possible with 12 attributes? The answer is 66. Given that each respondent would see two at a time, a full model would require 33 choice scenarios. Respondent fatigue ruled out this method.

Conceptualizing the study

Attempting to put into practice the principle of conceptualization, we came up with the following solution. Using our random choice scenario generator (an Excel macro designed in-house), each respondent would see each of the 12 attributes twice. In other words, she would be presented with six choice scenarios. The random choice scenario generator is able to produce a large number of choice scenarios – we’ll refer to them as splits – in a short period of time. Thus, though each respondent would see all choices the same number of times, pairs and rotation would differ. Figure 2 illustrates an example of the design. Statements are indicated by numbers, which are found to the left of the text on the graph above. When programmed, respondents will see the text.

Paired set makes up the individual choice scenarios each respondent views. Respondents compare statements in black (Item 1 and Item 2) to statements in red (Item 3 and Item 4). In order to ensure that all possible combinations are shown, we employed a random choice scenario generator to generate 100 splits.

For this method to work, GreenPlanet would have to survey a fairly large sample. GreenPlanet planned to draw from its panel a sample size of 5,000, so each split would be shown 50 times.

Output

Forming the output is a two-step process. Within SPSS, we structure the data so that each pair is positioned in such a way that its choice is calculated only when it was shown.

The second step is to calculate the percentage of times a pair was chosen among the set when it was shown. The order of attribute or what it was shown against are already randomized, so no further work has to be done by the field house. At issue are simply which scenario number they saw and what they chose.

Figure 3 shows the top output among three key GreenPlanet consumer groups.


Here we show the top 10 paired pairs among the 66 possible ones. There are differences among the samples. In fact, GreenPlanet ran these tables for about 20 different subgroups. Once the programing is done, it is relatively simple to filter by subgroup and produce the summary table.

The creative application of common techniques is a powerful way to conceptualize outputs and deliver the answer to the client.

GreenPlanet’s paired-paired analysis, of course, had defined finite boundaries. Unlike max-diff, which can easily accommodate 30-40 statements, paired-paired is quite limited with regard to the number of statements it can accommodate.

Nonetheless, the results are clear and statistically viable. Moreover, paired-paired analysis is far less expensive than more difficult multivariate methods and is easy to apply.