What is order bias?

Bias is all around us – it’s an innate part of the human experience, regardless of how much sway it holds over one’s daily life. This holds true for research as well. Both the researcher and the respondent are prone to introducing bias at multiple points during the research process – making it critical to understand the kinds of bias that can affect research and enact strategies to avoid letting bias distort the data.

Bias in marketing research

Broadly defined, bias is “an inclination of temperament or outlook” – in the research world, it is used more specifically to describe how an individual’s personal perceptions and ideas may distort the way they interact with a moderator, survey or prompt. “Bias” finds its roots in French and Greek words meaning “oblique,” an adequate description of how bias affects research results. It has the ability to quickly cause data to veer off course from the actual values.

So, how does bias distort the data? A whole host of potential biases can find their way into research, though a few notable examples include:

  • Confirmation bias, in which a researcher forms a hypothesis and uses research results to simply confirm the hypothesis rather than test it.
  • Acquiescence bias, in which respondents are inclined to agree with whatever idea is posed to them.
  • Order bias, in which the order of questions in a survey influences how a respondent answers each prompt.

A closer look at order bias

In this article, we’ll focus specifically on order bias – both how it presents in research and how it can be mitigated. It may be that order bias can never truly be eradicated from survey results, but it’s possible to account for and minimize its effects so as to still collect valuable and accurate data.

Most often, order bias presents when a respondent forms their answer to one question based on their answer to the previous question. For example, if a survey asks respondents about major varieties of breakfast cereal, the ranking of cereals noted in later questions will likely be based on how a respondent ranked earlier cereals. On a scale of one to 10, if I give Cocoa Puffs a six, then the way I next rank Captain Crunch will depend on how I compare it to Cocoa Puffs, rather than how I view it separately on a 10-point scale.

Mitigating the effects of order bias

One potential solution to this problem lies in consistently rotating the order in which questions are asked, so that the final data set will not show a trend based on order. In a 1993 Quirk’s article, “Data Use: Back to basics: remember to rotate,” Gary M. Mullet, president of Gary Mullet Associates Inc., recommends using this strategy to avoid establishing a frame of reference in survey design. However, Mullet points out that simply rotating questions is a conservative approach to collecting survey data – in order to fully understand the data set, crossover analyses of multiple rotated question sets should be undertaken.

Rebecca Sarniak, moderating services specialist at iModerate, offered another approach to mitigating order bias in a 2018 Quirk’s article titled, “9 types of research bias and how to avoid them.” When writing a survey, Sarniak suggests, arrange questions so that general questions come before specific, unaided before aided, and positive before negative.

Ultimately, bias will always find its way into the research process. By acknowledging this hurdle and taking steps to minimize the way bias affects the research process, researchers can ensure their data is both accurate and actionable.