Maximizing those mobile moments

If your goal is to conduct research with on-the-go people, what better way than with mobile research? That was part of the thinking behind Marriott International’s decision to conduct a mobile-based study with users of Marriott Mobile, the mobile version of the Marriott Web site.

Marriott Mobile debuted in August 2008 and was a smash hit, generating $2 million in gross revenues by the end of that year. Wanting to keep pace with technology and with travelers’ evolving needs, Marriott sought to add new features in 2009 and so turned to research firm Vision Critical for help with developing a mobile survey. “We wanted a quick way to determine what our mobile customers are looking for, which is precisely what the [mobile survey] allowed us to do,” says Gina Villavicencio, senior manager, user research, Marriott International - eCommerce. “The survey enabled us to speak to 700 mobile users in roughly 30 days and very quickly get insight into today’s mobile customers.”

The brief questionnaire included some general-impression questions, some usage pattern questions and also queried about a city guide feature which offered information on restaurants and other attractions in cities with Marriott hotels. “We wanted to understand how our mobile users would respond to the launch,” says Villavicencio. “The feedback was overwhelmingly positive. In fact, we also benefited from a halo effect where those who clicked on the city guide link rated the mobile experience higher.”

Marriott had a wealth of data from other research approaches on the mobile site - online surveys, secondary data, etc. - but to get feedback on the mobile site it made sense to use a mobile-based survey approach, Villavicencio says. There was some prior experience with a mobile-based survey, thanks to a previous ad-effectiveness test, so the researchers knew at the outset what they could and couldn’t do, and what kind of information they could expect to garner, using the mobile survey approach. “We knew there were some limitations, including survey length and question type. We had one open-ended question but we knew it wouldn’t yield much information.”

While mobile surveying has a number of applications, it is obviously well-suited to researching the frequent business travelers, says Chris Ferneyhough, senior vice president of Vision Critical’s technology practice. “When you think about a scenario when the respondent is the mobile professional, someone who is constantly at the airport and staying in hotels, their down time sitting at the airport gate is often spent on their BlackBerry or their iPhone and there is an opportunity to hit them with surveys to measure their satisfaction with their most recent flight or hotel stay.

“In a lot of these cases, you are able to catch the respondent in the moment, once they get off the plane or when they leave a store, as opposed to having to contact them two weeks later, and ask them about that flight or that experience in the store or that hotel stay,” he says.

That immediacy makes up for any concerns about the need to be brief, Villavicencio says. “One of the benefits of [the mobile survey approach] is immediacy. Because you get the respondent right there, when they are using it, while they are going through the experience, the quality of the data is richer and more accurate. The drawback is the inability to ask several questions or probe. However, if you have a specific topic or objective, the way that we did, a mobile survey is ideal.”

Response rates were very good

While no incentive was offered to respondents, response rates were very good, which was likely a result of customers’ loyalty to Marriott, Villavicencio says. “They tend to respond to our surveys very well. Plus, taking a mobile survey is still a novelty and the invitation to do so probably piqued their curiosity.”

Beyond incentives, it is crucial to make sure that prospective respondents have an unlimited data plan through their mobile service and therefore won’t have to pay to respond to your survey. Paying to do something they expect to be compensated for in some way, even if that compensation only takes the form of a chance in a prize draw, not only damages the respondent’s view of the research process but also makes it more likely they will choose not to respond. “Cost to the respondent has been a point of contention,” says Gordon McNeill, vice president of product innovation at Vision Critical. “In North America we are getting better at having unlimited data plans, but it is something to be aware of and understand that you are more likely to get a response from someone with an unlimited data plan than someone who has to pay to respond to your survey.”

Villavicencio says that Marriott expected most of the respondents would complete the survey while they were on the road or on the go but a surprising one-third took the survey while at home. “That speaks to the adoption of mobile. It was interesting to learn that a person who is at home would choose to take a survey on their mobile device,” she says.

The location factor is one that Vision Critical has been examining in some of its research on research, says McNeill. “One of the questions we have been adding to our mobile projects is, where are you right now? Are you in a car? At the airport? We want to start to combine those kinds of numbers from across our surveys to see if we are capturing respondents at different times than we are capturing them when they complete a survey on their computer.”

Going forward, those kinds of questions may be moot, as GPS and other location-pinpointing technologies become standard in smartphones, McNeill says. “It’s getting to the point where you don’t have to ask someone where they are located - you can capture that from the device if they give you permission. Thus you can shorten up the survey because you don’t have to ask those questions but you get that rich data.”

Paint a clearer picture

One of the main reasons to do the Marriott Mobile survey was to be able to link it to Web analytics data to paint a clearer picture, Villavicencio says. “In the past, we measured improvements to Marriott.com using behavioral or Web analytics data and attitudinal or survey data. The mobile survey allowed us to do the same for Marriott Mobile.”

For Marriott, it also helped that the various internal teams had discussed their shared expectations and objectives going into the research, so there were no surprises or disappointments as the data came in, Villavicencio says. “I think that when trying new tools, or new approaches, it’s always good for the client teams to go in completely in synch.”

Grow rapidly

Ferneyhough feels that use of mobile research will continue to grow rapidly. “I think it’s an important area and I’ll be surprised if five years from now at least a third of the data that is being collected isn’t being done this way. The phone makers themselves are doing a great job of driving people to these devices and getting them to change their Web behavior. More and more companies are recognizing the importance of the platform and getting a better understanding of the platform and definitely it is the most exciting aspect of where research is going.”

While mobile research shows a lot of promise as a new and exciting technique, that very newness means that things won’t always go smoothly, Ferneyhough says. “We are still so early in this methodology that there will be a lot of hiccups along the way. We as researchers are going to have to learn as we go, and we will unfortunately learn things the hard way. I would almost make a parallel to the early days of the Web research in the late ‘90s, where [with mobile research] you are getting the opinions of a select group of respondents who are comfortable using their technology and spend a lot of time using it and as a result are not terribly representative in all likelihood of the target market. The good thing is, many of those issues are going to go away as penetration levels keep rising, just as the same types of arguments related to the Internet have by and large gone away.”

While mobile research is new and different, it has to be used properly to avoid making its use more than just a reaction to the need to try the latest thing. “It really comes down to, what is the type of person that you need to collect this data from, how representative are smartphone users among this type of respondent and what is the category?” says Ferneyhough.

Complements rather than replaces

For Marriott, Villavicencio says that while the mobile research experience was satisfying and helpful, the method will be viewed as one that complements rather than replaces its existing research approaches. “We have a host of different tools such as Web analytics and the online research panel hosted by Vision Critical. The mobile approach is definitely another tool we plan to use in the future.”

Quick suggestions for those considering going mobile:

  • Keep it brief. Short and sweet is the strategy for both the questionnaire in total and the questions themselves. “In these mobile moments, as we call them, you don’t have a lot of time so you want to make sure you get them through the survey quite quickly,” Vision Critical’s Gordon McNeill says.
  • Keep it simple. Remember that there are thousands of models of smartphones on the market, and to make a survey appear in roughly the same way on all of those phones, things need to be kept simple. Colors can display differently on different phones, questions can wrap differently depending on screen size, etc.

Not only does this maintain respondent satisfaction by showing respect for their time but it also compensates for the effects of mobile phones’ small screens and the myriad types of phones that the survey must appear on. For example, overlong questions can break in odd ways and you want to keep respondents’ scrolling to a minimum. “The way that a longer question would show up on an iPhone will be dramatically different than on a BlackBerry Pearl. The more scrolling a respondent has to do, the more frustrating the experience. So the shorter you can keep the questions, and the fewer response options you offer, the better,” Vision Critical’s Chris Ferneyhough says.

  • When possible, know your respondent. “We have profiled our panelists to make sure they are willing to complete surveys and to find out what type of device they are using. You have to respect the fact that there may be respondents who don’t want to complete surveys on their mobile device or might be intimidated by doing so,” Ferneyhough says.
  • Make sure respondents are completing mobile surveys on their mobile phones. “There is a chance that they could get the invite via e-mail and then they go and complete it on their computer and of course things look very different,” Ferneyhough says, which could affect their impression of the survey process.