Editor’s note: Jim Lane is chief marketing officer at Directions Research, Cincinnati, Ohio.

As recently as a decade ago our industry was analyzing the comparability between telephone and Web-based surveys and using that insight to transition to online data collection. Since then, the pace of technological change has continued to accelerate at an exponential rate and the shift of consumers to mobile devices for an ever increasing proportion of their online needs has researchers recalibrating once again.

Current estimates place 5 percent of custom marketing research being taken on mobile devices in the United States, and an even greater percentage internationally where mobile is the only online access tool available to consumers. In the future, as consumers rely even more heavily on their mobile devices and become comfortable completing longer surveys via this medium, industry analysts predict that a much higher percentage of consumer research will be conducted on mobile devices. Some industry analysts go as far as predicting the complete demise of lengthy, online surveys. However, little public information exists on how this transition may occur and what will be lost in the process.

In the summer of 2014 a foundational coffee brand equity study, conducted by Directions Research on behalf of one of its restaurant clients, provided an opportunity to conduct a companion mobile study to assess whether the key study findings could be replicated on a mobile platform. Together we were interested in evaluating the feasibility of a mobile study replacing the foundational research and/or if such a study could be useful for future updating of the foundational findings.

The first step: online research

The Quick-Service Restaurant (QSR) “coffee wars” are a hot topic in the industry, with a broad range of competitors fighting it out in the growing $30 billion QSR breakfast market. Traditional lunch and dinner restaurants have joined convenience stores and other coffee shops on the breakfast bandwagon. Competitors are creating more breakfast offerings, developing more mobile loyalty and payment programs and exploring additional distribution channels to bring their brands to consumers more often.

The research focus was designed to help better understand our client’s position in the away from home coffee market as they navigate this increasingly crowded battlefield. The study included 2000 surveys among a national sample of coffee drinkers, balanced by age and gender across all nine census regions.

The online survey used a proprietary unique drag and drop technology for the ratings questions that Directions routinely employs on its client’s brand image/equity trackers. This technique provides good disparity of data for actionable, predictive results, without the respondent fatigue that results from more traditional ratings questions. In addition to delivering more insightful results, this method incorporates gamification which has proven to be quite popular with respondents due to its more engaging nature.

The results were modeled to create a coffee equity score for each brand, normalized between 0 and 100 and the resulting brand equity score accurately reflects market, category and demographic sub-group unique characteristics.

The strategic guidance gained from this work included:

  • clarification of the chain’s overall position in the QSR coffee market, relative to market leaders and secondary players on both national and regional levels;
  • quantification of tradeoffs that consumers make in away-from-home coffee purchase decisions and how these trade-offs impact brand image;
  • determination of which competitors could be targeted to most effectively improve overall market position nationally and by region; and
  • identification of tactical maneuvers to improve image and strengthen performance relative to specific competitors.

With the foundational research and modeling completed, the question became: could the key results be replicated on a mobile platform?

Conversion from online to mobile survey

While the foundational Web-based survey was technically device agnostic, it was designed for online administration and the length of the survey virtually guaranteed that very few respondents would elect to use a mobile device for survey completion.

The mobile survey, however, was designed specifically for mobile devices and executed via a mobile app among a mobile-only panel. The technology inherent in today’s mobile research apps required several key design changes.

Survey length

To attain reasonable completion rates, the survey was scaled back from 27 to 10 minutes in length, by:

  • including only the key component parts of the equity model and eliminating questions that would be non-essential to the test at hand;
  • restricting the number of brands rated to three from the original five focusing on only the top competitors of interest; and
  • reducing the number of brand and functional attributes from 50 to 16 by using the attributes that were the largest coffee equity score drivers and those that most differentiated among brands.


Technique changes

The unique drag and drop technology used in the online survey was not available in the mobile app, due to screen size and programming limitations. Instead, the functional attribute ratings were gathered using a 100-point slider scale.

Concerns about variation in sample composition between the two methods were nullified by rigid screening and quota control to ensure that the 1000 mobile surveys among coffee drinkers were balanced in exactly the same manner as the online surveys in terms of age and gender by region.

Comparing the results

We were pleased to see that awareness and usage of the key brands were similar, regardless of method, and well within the survey research data error range.

However, while brand performance across the two studies was fairly consistent, the strengths of one key competitor were amplified on the mobile platform, likely as a result of the brand’s strong mobile identity. This brand was an early entrant into the mobile app space and customers routinely engage with it on a mobile basis; therefore it was not unexpected that our mobile respondents were more prone to have a positive affinity for this competitor.

The mobile platform also produced a slightly different respondent profile in terms of their attitudes toward coffee as compared to online respondents. While not a dramatic difference, mobile users were slightly less likely to say they “need my coffee every day.” Relative to the online sample, mobile respondents were also more likely to:

  • put a premium on convenience,
  • believe their chosen brand of coffee defines who they are,
  • prefer medium roast coffee, and
  • be slightly less affluent.

The most significant differences in results between the two methods were on the functional coffee attributes. On all 16 attributes, for all three key brands, mobile scores were lower, with the disparity ranging from a low of 3 percentage points to a high of 25 points. Two design changes mandated by the limitations of the mobile app surely led to this variation – the use of slider technology vs. the drag and drop, and the limitation to three brands vs. five which necessitated a different brand selection protocol.

Not surprisingly, the lower functional attribute ratings created a difference in the final Coffee Equity scores. The model developed using the online survey resulted in brand equity scores ranging between 63 and 75, a fairly tight range. While the relative brand positions remained the same, running the same model on mobile data resulted in lower scores for all three brands with a wider range of scores (50-68).

Insights

Although the relative position of the key brands is similar between the two methods, there are fundamental differences in results.

  • The shorter survey required for a mobile interview allows for less in-depth analysis and richness of data, limiting the ability to identify tactical moves likely to enhance brand perceptions and drive share.
  • The need to limit the number of competitors included in the mobile survey means the full market landscape can’t be profiled, which has implications for strategic direction.

Clearly a mobile-only study would not meet the overall information needs of the client in this case as it could not replace the foundational research completely through accurately replicating the results.
At the same time our research suggests that mobile research can be used in this manner once the foundational research is conducted, and consistent brand decisions and respondent tasks are identified and incorporated between the foundational online execution and ongoing mobile updates.

Implications for MR

Mobile surely has its place in the mix of methods to consider and we must continue to work to overcome the shortcomings that exist today. Right now, mobile may be best used to supplement traditional foundational research or to reach key groups of respondents that may be more, or even exclusively, mobile.

It is important to design the foundational research to the current capabilities of mobile apps. Researchers should begin with the mobile capabilities and think backwards to be sure all data collection techniques are as comparable as possible between the two methods.

Obviously, as mobile capabilities advance, some technological challenges will be resolved. However, survey length will continue to be an important consideration. How will custom research adapt to the shorter interview length required in mobile surveys? To what extent will consumers adapt to longer interviews as mobile devices become their primary means of interacting with the world? These are questions which require further research and analysis.