A harmonious pairing

Editor's note: Kathryn Winland is vice president at research firm Mozaic Group Inc.

When my clients from a major national retailer wanted to test new technology integrated into their stores and mobile app, they knew they needed to observe customers interacting with the technology firsthand. Conducting shop-alongs at a nearby location was a convenient and effective approach for examining the in-store technology but gaining a deep understanding of mobile app usage behaviors, as well as feedback on the new features, required a longer period of observation and a larger group of respondents.

A combination of mobile and in-person shop-alongs worked well to satisfy both the in-store and mobile app needs and provided a unique opportunity to compare the two methods – from fieldwork experiences to analysis and reporting. This side-by-side comparison in the context of a single study highlights the advantages, special considerations and precautions associated with each approach. Drawing from these experiences helps to understand when mobile research may have the advantage and how to gain buy-in from client-side stakeholders.

Once it was decided to combine mobile and in-person research, we made the most of the blended approach by beginning with the mobile component. This allowed us to explore some foundational topics and establish important context for the results. We designed the mobile research phase to include some initial discussion that could be completed via computer or mobile, followed by a mobile shop-along in which respondents downloaded the research app to their smartphones and completed activities and questions while they were in the store, including photo uploads and screen shots from the retailer’s mobile app.

The mobile research took place over a three-day period and respondents were invited to participate at their own pace whenever it was convenient for them. Because we were requiring a visit to the retail store for the mobile shop-along, we designed the fieldwork timeline to include a weekend so that our respondent base could be as diverse as possible, including those who have little or no time to complete such an exercise on weekdays. We knew the mobile phase results would likely impact the in-person shop-alongs to a certain degree, so we designed the research accordingly, planning to build upon learnings from the mobile phase to optimize our in-person time with respondents. As a result, the in-person shop-alongs could be more efficient and more focused.

Following the mobile phase, the team collaborated on a guide for the in-person shop-alongs. Thanks to our mobile respondents, we identified a handful of topics we wanted to explore further, including in-person observation of certain mobile app features to learn more about some usability gaps that were revealed during the mobile phase. The in-person sessions began with a quick chat before entering the store and then focused primarily on the shop-along. We gently steered respondents toward the vicinity of the in-store technology to gauge discoverability but tried to be careful not to interfere too much lest we miss out on new insights that could inform how technology could improve shopper experiences in the future and we did, in fact, collect some key takeaways with clear and direct opportunities to tie in technology.

Uninhibited input

While the in-person shop-alongs were especially useful for collecting unaided reactions to in-store technology, the mobile phase provided the benefit of extended fieldwork and uninhibited input on both general and very specific topics and features. I also felt, as I’ve seen in other online and mobile qualitative research, that mobile respondents more willingly or more easily provided unfettered responses and that some of the in-person respondents felt self-conscious and needed consistent coaching and encouragement to be as natural as possible.

From a fieldwork perspective, advantages of each approach and tips for making the most of it are shown in Table 1.

During analysis and reporting, having the wealth of data provided by the mobile phase was a major plus. Quotes, photos uploaded by respondents and the ability to easily track themes throughout the typed responses factored heavily into creating a rich and insightful report. In-person sessions were recorded but the clear advantage went to mobile for the instantaneous and exact transcription of all the data collected in that phase. This isn’t to say that the in-person sessions didn’t deliver new insights and new verbatims; of course they did and some of those insights may have been overlooked by mobile research alone. Additionally, the client’s preference to observe at least a portion of the shop-alongs was an important consideration that should not be discounted. Ultimately, I’ve decided, it comes down to what type of input you want to capture and the collective goals of the team.

Make unaided observations

During fieldwork and reporting, in-person shop-alongs proved their value for more exploratory or unanticipated feedback and gave us an opportunity to make unaided observations about specific in-store components. That last piece, in particular, would have been more difficult – perhaps significantly – without being there in person so that we could observe unaided and then inquire about the item in question and dive deeper into reactions.

On the other hand, the mobile phase had some clear advantages for more targeted, activity-based feedback, such as testing new features. Mobile was also useful for establishing a foundational understanding of shopper behaviors as they relate to technology and helped refine the discussions going forward. (Table 2 further explores the comparison of the two approaches for analysis and reporting.)

The benefits of mobile research – for shop-alongs or otherwise – are not superior or inferior to in-person research; rather, they are simply different. Comparing the two in the context of a single study helps to identify several factors one should consider when determining which approach will be best.

In-person research may be best if:

  • objectives are more exploratory in nature;
  • research needs to take place in specific locations that have a certain layout or feature (as with the integrated in-store technology in this study);
  • stakeholders have a strong preference to observe in person;
  • discussion will evolve over the course of fieldwork.

Mobile has the advantage when:

  • a diverse or hard-to-find audience is required (suggesting a need for larger geographic scope, such as nation-wide recruitment);
  • objectives center on collecting input on specific hypotheses, topics or features, leading to activity-based de-sign;
  • there is a need for foundational or in-depth understanding of behaviors, attitudes or general context, in addition to other types of observation (e.g., a shop-along);
  • stakeholders need quick-turn reporting while maintaining detail and incorporating rich data, such as voice-of-customer and research artifacts (e.g., respondent photos).

A few key concerns

Another important factor to consider with mobile research is gaining buy-in from stakeholders who are likely to be more familiar and more comfortable with traditional in-person research. In my experience, people who are hesitant to go mobile have a few key concerns: Will respondents be sufficiently engaged? How do we ensure that we get quality responses? Will I be able to ask follow-up questions? What do we do if we want to dig deeper on something we learn along the way?

Addressing these concerns is critical, of course, and this is where experience executing mobile qualitative provides a researcher with the knowledge and skills to foster success and anticipate challenges. Though there is no substitute for experience in this regard, there are some basic tools that can be effective selling points for mobile qualitative.

  • All mobile research platforms allow moderators to reply to respondent posts and, if desired, notify them by e-mail that there is a reply that requires their attention. Using this feature, moderators can: engage respondents right away; greet them after their first post and establish rapport so that they feel committed to the effort; carefully monitor responses and actively moderate; coach participants along the way by asking them to elaborate on insufficient responses and praise them for thoughtful posts that provide the level of detail and interaction you desire.
  • In most platforms, stakeholders can add comments either in a virtual backroom or directly in line with a respondent’s post (or both). Only the moderator and other stakeholders can see these comments. This is a perfect way to request follow-ups, draw attention to important responses and make notes for analysis.
  • Follow-up questions for all respondents can be added at any time. In a recent study, we added a handful of questions toward the end of fieldwork. Because we anticipated this need, and because fieldwork spanned a three-day period, we were able to allow some extra space for these follow-up questions without asking respondents to give us more time than originally planned or extending fieldwork beyond the original time frame. In this particular study, the follow-up questions provided important clarifications and validations and, because we could pose these questions to our entire sample and not just a specific individual or group, mobile research probably gave us a better view on these follow-ups than traditional focus groups would have done.

There are, of course, many other meaningful benefits of mobile research, some of which are highlighted here. If my experiences are any indicator, researchers should plan for a continued need to explain these benefits; although there has been a significant increase in interest and openness to mobile qualitative, many client-side researchers still have to work at getting their stakeholders on board.

Similar trends

Though this article focuses on a single study for the purposes of illustration, I and other researchers at our firm have observed very similar trends in other studies and certainly beyond shop-alongs. We have conducted many studies that included a mix of in-person and mobile fieldwork and much of this holds true.

One important factor, in particular, centers on adapting the design to the client’s own preferences and learning style. While I absolutely encourage promoting new options and helping clients to stretch, researchers must give thought to how well their stakeholder audience will interact with the information and the process for getting there. In fact, in many instances we use a mix of in-person and mobile to strike a balance among the advantages of each approach and to acclimate clients to mobile research. This study is a perfect example of how mobile and in-person research can come together to deliver a collection of insights that may not have been possible with either approach on its own.

Finally, in an increasingly mobile population, there is more overlap than ever before in online research and mobile research and this is important to remember in creating an effective design and planning for successful fieldwork. It is wise to assume that all online research is effectively mobile research and to approach design accordingly.