Editor’s note: Lilah Koski is CEO of marketing research firm Koski Research, San Francisco.

When it comes to questionnaire development, it is important to regularly test new, more gamified approaches against more traditional designs. The goal is to evolve question types for the digital age while making sure that we have not lost any data integrity.

Our team recently conducted a test that included modern question design vs. traditional question design for a ranking question. For this test, the survey was programmed with a traditional drop down ranking question and a more gamified drag and drop question.

We tested both designs across generations, including the oldest members of Gen Z (16-to-18-year-olds). Taking question design and respondent age into account, we found that more modern programming can improve the respondent experience at no cost to data integrity or quality.

In this test, we had respondents rank the most common way they expect they will pay for things five years from now. The results were comparable for both techniques, with no significant differences.

In another test, we compared a 5-point rating question to a “Tinder” binary, gamified question. We tested a Tinder-like survey question and found that a heart generally equated to the positive end of a 5-point scale. What we found is that the heart tended to match a 3-to-5 rating (within 5 to 10 percentage points), while the X was very comparable to a 1-to-2 rating.

The data suggest that when the scores are more spread out in the 5-point scale, as is the case with Apple and Starbucks below, the gamified version with the binary X and heart will lead to more polarized differences.

The table below summarizes the scores for overall opinion.

In these tests, we have found that the data from modernized or gamified survey programming are similar to data from traditional programming. We also have seen that survey length, response rate and ease of taking the survey are comparable.

We found the more game-oriented survey to be comparable to the traditional survey in terms of:

  • interview length (seven minutes for the gamified and six minutes for the traditional); and
  • respondents' ease of taking the survey (74 percent rated the gamified survey as easy to take and 73 percent in the standard survey also rated the survey as easy to take).

What does this mean for you?

Like other decisions in your research design, deciding how you craft questions means weighing the options against the objectives you want to achieve. Consider more gamified, modern question types for:

  • shorter, more engaging surveys;
  • surveys that are primarily taken on mobile devices;
  • populations known to be tech savvy or smartphone centric;
  • when the feedback should mirror the online or social media experience (for example, binary results or star ratings vs. a 10-point more sensitive scale);
  • when an immediate gut (or heart) reaction to a brand or product is needed;
  • for topics that lend themselves to visuals (consumer products and brands); and
  • when you don’t need a large sensitive scale for traditional analytics.

This is good news for researchers who are seeking to construct survey questions that are modernized for today’s survey takers and mobile devices.