Data Use: Back to basics: remember to rotate

Editor's note: Gary M. Mullet is president of Gary Mullet Associates, Inc., a suburban Atlanta-based consulting and data processing firm concentrating in statistical analyses for marketing research.

We all know that when putting together a good questionnaire it is necessary to rotate the order of scales, the order of presentation of items to be tested and, perhaps, even the order of major sections of the instrument itself. We've known this forever and even if the ubiquitous "they" didn't tell us to do so, common sense so dictates.

If we go to the textbooks to find out why we need to change the order of presentation of items from respondent-to-respondent, we find, for example in Aaker & Day (1990) that, "The nature of the preceding questions definitely establishes the frame of reference to be used by the respondent." We also find (Schiffman & Kanuk, 1987) that, ". . . politicians and other professional communicators often jockey for position when they address an audience sequentially; they are aware that the first and last speeches are more likely to be retained in the audience's memory . . ." Those of us involved with laundry lists of attribute ratings (see Sudman & Bradburn, 1982) know that, ". . . order may effect responses as a result of fatigue." And these three citations are just the tip of the iceberg.

Thus, most of us are convinced that rotation is necessary and conscientiously see that rotation codes are written into all relevant questionnaires. While it's easier to do on studies involving CRT data collection, we generally try to do so with other instruments as well.

The following sections will demonstrate that it is (still) necessary to rotate the order in which a series of rating scales are presented to respondents. Also, we'll see how many of us may be overlooking significant differences when we carefully rotate products or concepts in our tests.

Is rotation really necessary?

The obvious way is to field a questionnaire with a large number of scales. In half of them, carefully rotate the order of the scales and in the other half keep a fixed ordering. Then compare the answers item-by-item. No one will or should do this since, if the answers do disagree, you won't know which half to believe. Thus, we rotate scales as a matter of routine. Should we?

Recently a study was conducted (and the results below are disguised) in which 200 respondents were asked to rate a product on 10 attributes on 10-point scales - the type of thing market researchers request people to do every day. The results looked something like this:

Now, in a vacuum, there's nothing at all unsettling about these results. But this particular test was actually a pretest of a new product evaluation system that was under consideration. In fact, since these scales were a small portion of a much larger, more elaborate questionnaire and the pretest was of an entirely new methodology, the scales weren't rotated - intentionally so, not as an indication of bad research design. When the order of asking is tacked on to the above results, we see something a little different.

These results were, as you might imagine, quite unnerving. With one exception, the means decrease uniformly by question order. So, it seems to me, these data illustrate yet again the necessity of rotating items to control for position bias.

Further, if you're of a statistical bent and are faced with data such as that above, i.e., not rotated and with a definite pattern in the mean responses, you can test whether or not the order effect is statistically significant (which, of course, says nothing about it being something of substance that you need to worry about) by using the Page test for ordered responses (see, for instance, Marascuilo and McSweeney, 1977). As far as I know, this test is not included in the standard statistical software packages but you can easily use the results of the Friedman test, which is included in many, to test whether or not such results are significant.

Analysis of results when stimuli are rotated

As the above example shows, there seems to be a response bias by position of the scale. This has been observed many times by many researchers in many studies. So we do the obvious and rotate the order of presentation of items/scales. Then what do we do? Most of us merely "derotate" the items/ scales and then perform whatever type of statistical analysis we think is appropriate.

Let's be concrete and assume that we are comparing two food items on a series of attributes, such as taste, sweetness and so on. What do we do? We always make sure that (around) half of the sample sees product A first and the other half sees product B first. Each half of the sample sees the other product second. We generally rotate the scales within the products as well. Then in order to see which product is perceived as sweeter we run a dependent or matched group test for means after lining up the data by derotating (either explicitly or by software instructions). If the computed statistic reaches the appropriate value, we deem the products to be significantly different in sweetness at the confidence level we're using. If the t-statistic doesn't make it to the criterion value, we say that we can't see any real difference in sweetness.

There's nothing wrong with this approach except that it's very conservative. You'll fail to find as many significant differences as you would if you took advantage of the fact that the items were presented in rotated order. Since you've gone to the trouble and expense to rotate the two products (the same comments would also apply to three or more products) you get an analytical bonus. At the same time you look for mean differences in product sweetness, you can also look for mean differences in product position. This has the effect of making the denominator in your t-test for mean differences smaller. It's easy to see what the result of that is - the t-statistic itself will generally be larger, sometimes enough to yield a significant result where there would not be one if we don't consider the order effect.

All of these gyrations go under the name of "cross-over analysis" (Cochran and Cox, 1957). This is another analysis that doesn't seem to be included in most statistical software packages but actually is. It is very easy to run a crossover analysis using the everyday analysis of variance programs included in these packages, even if they don't show the words "cross-over" in their indices. We don't need to pay extra for either programming or analysis using a special cross-over program.

Conclusions

There are two conclusions to be drawn from the discussion above. First, it is still necessary to rotate items/scales in marketing research. The Page test for ordered responses can be used to test data that were not rotated to see whether there is significant order bias. (It can be used in other situations when data are collected in a series and you feel there might be a trend in the mean responses, such as decreased consumption of a particular product on a month-to-month basis using diary data from a household panel. Here, obviously, the order of presentation of the months cannot be rotated.)

Secondly, since data are frequently gathered using questionnaires with rotation patterns built into them, we are not using the full power of the rotation patterns if we are not doing cross-over analyses. If we don't, we may be erring on the conservative side (which is certainly all right in many studies).

References

Aaker, David A. and George S. Day (1990), Marketing Research, Fourth Edition, New York: John Wiley & Sons, Inc.

Cochran, William G. and Gertrude M. Cox (1957), Experimental Design, Second Edition (11th printing), New York: John Wiley & Sons, Inc. ,

Marascuilo, Leonard A. and Maryellen McSweeney (1977), Nonparametric and Distribution- Free Methods for the Social Sciences, Monterey, CA: Brooks/Cole.

Schiffman, Leon G. and Leslie Lazar Kanuk (1987), Consumer Behavior, Third Edition, Englewood Cliffs, NJ: Prentice-Hall, Inc.

Sudman, Seymour and Norman M. Bradburn (1982), Asking Questions, San Francisco, CA: Jossey-Bass Publishers.