A moving target

Editor's note: Adam Berman is president
of CatalystMR, Oakland, Calif.

The proliferation of smart, Web-ready mobile devices continues unabated and for many consumers these machines are replacing their desktop and laptop computers, which, until recently, were the primary means to collect online survey data. In 2011 the percentage of surveys being taken on mobile devices was less than 2 percent. However, within the last year, the U.S. and other nations have experienced a major shift to a more mobile-centric culture.

With this prolific adoption of mobile devices, our firm is now seeing 25-30 percent of its online surveys being completed on mobile devices vs. desktop and laptop computers. Coupled with the accelerated use of mobile devices to access e-mail, with 36 percent of all e-mails in the U.S. being opened on mobile devices (Knotice, Sept. 2012), a mobile data collection strategy must be at the forefront of consideration when conducting online research.

For data collection within the U.S., consideration for feature phones (or non-smart Web-enabled devices) is not crucial because of dropping market share and consumer behavior differences. In the U.S., those with non-smart Internet-accessible phones don’t tend to surf the Web or make purchases on their feature phones as compared to modern smartphones.

Because international smartphone adoption over feature phones has not been as rapid in many parts of the world due to cultural (Internet-accessible feature phones are popular) and economic differences, surveys being conducted abroad require special consideration and programming to allow feature phones to take online mobile surveys.

In other words, it’s more important to make the survey accessible to a feature phone in other parts of the world compared to North America and Europe. While there is a continued trend toward smartphone adoption internationally, in regions of the world such as Asia-Pacific and Latin America for example, the ownership and usage including Internet surfing and online purchasing of feature phones exceed that of smartphones and tablets. In these regions and most parts of the world with the exception of Europe and North America, consumers surf the Internet on full-featured phones.

Mobile devices are continuing to evolve rapidly, as is the technology available for conducting surveys on these devices and the behavior of those using them. As a result, any overview of best practices is a constantly moving target. Nevertheless, assembling a list of best practices is still a worthwhile endeavor.

Number of questions and survey length

A couple of years ago, CatalystMR was advising its clients that five to seven minutes was a pretty firm ceiling for mobile survey length and should not be exceeded. However, tolerance of longer surveys is increasing as devices continue to improve and respondents use these devices more frequently. In fact, for many people, mobile is a preferred platform to communicate rather than a desktop. Additionally, respondents recognize and accept that mobile devices have less screen space to work with than desktop and laptop computers. Our current recommendation for top-end survey length on mobile devices is roughly the same as our recommendation for desktop-only surveys: 20 minutes or less, with an ideal of 10-15 minutes or less for mobile devices, especially smartphones.

With survey lengths exceeding 20 minutes, we see a drop-off of respondents completing the survey, regardless of device type. With that said, always remember: The shorter the better. You need to take into consideration the respondent’s relationship with the brand, the customer’s expectations, the marketing effect of the survey, the incentive amount, panel vs. non panel and other factors that might lead you to conduct a very brief survey, perhaps only five to 15 simple questions.

Respondents using tablets are more tolerant of longer surveys than those taking surveys on smartphones, primarily due to the need to scroll more on a smaller screen and general fatigue. Those taking a survey on a feature phone are less tolerant than smartphone users for similar reasons. For the purposes of feature-phone users, these respondents should be given a scaled-down version of the survey to lessen respondent fatigue and to mitigate the dropout rate.

With all this in mind, it should still be expected that a mobile survey will experience a higher dropout or abandonment rate than that of a desktop survey. However, survey results trend the same whether online or via mobile. With few exceptions, such as surveys that require a large viewing screen (i.e., conjoint studies, large or complex grid presentations, etc.), we have found that survey results are not affected by the platform used to take the survey.

When to scale down the questions asked in mobile vs. desktop

If your survey exceeds 15 minutes, consider limiting the number of questions asked on mobile devices. To avoid respondent fatigue and higher than necessary dropout rates on these devices, think about asking up to 10 minutes of questions as the core of your research. For example, if you have a battery of questions asking respondents to rate their satisfaction on 15 items, consider asking the most important 10 items to smartphone users. Or perhaps rotate blocks of attributes so that all attributes are rated by a subset of respondents.

Survey design: question types, setup and logic

Scale questions should be limited to five points or seven points at most. Ten-point scale questions take up too much screen real estate and therefore require scrolling left/right or pinching a mobile screen’s viewable area so small that the screen presentation causes the question to be difficult to manage. With that said, if you have established a 10-point scale as a scale norm previously, not to worry. Ten-point scales do fit to screen ultimately; the scale presentation is just smaller if the respondent pinches the screen to see the entire scale question.

More on scale questions: Try to limit the amount of text on each attribute. Overly wordy attribute text takes up more valuable screen real estate and increases the need for scrolling.

Ranking questions are fully functional on smartphones and tablet devices, however on feature phones, ranking questions require a different programming setup. Once again, just keep in mind that overly wordy text or too many items to rank can make the page too complex visually for smaller screen sizes. On smartphones, always consider that longer questions will require vertical scrolling, which makes those questions more difficult to navigate.

Other question types including numeric, radio (single-response), checkbox (multiple response), text and pull-down work perfectly on smartphones and tablets.

To minimize scrolling, the best practice is to program one question per screen, except in some rare cases where wording and a short response list allow for two questions.

Survey logic: Any survey logic available in an online survey is similarly available in a mobile survey whether the logic is complex skip or rotate patterns; quota or segmentation algorithms; data piping or rich content presentation, etc.

One survey engine – platform-agnostic: The same survey that runs on a desktop runs on a mobile Web-enabled mobile device because the survey engine is the same. In our firm’s case, because the survey engine is the same and CatalystMR tracks user agent data such as device type, user agent data can drive survey logic. This means that you have the ability to customize survey logic based on the device accessing the survey.

Device information

Our survey system, for example, captures many data points automatically from the user’s device. We can report this data back in an easily understood format.

Here is the information we collect from device capture recording: device type – desktop, mobile (smartphone/tablet v. feature phone); device make – Apple, Motorola, Samsung, etc.; device model – iPhone, BlackBerry and other models (though currently in about 3-4 percent of the cases, devices don’t pass this information along); browser – Chrome, Internet Explorer, Safari, etc.; browser version.

Survey introduction (including pop-up invitations)

Special consideration needs to be paid to the introduction and invitation text for your survey. Consider the following to improve the traffic to your survey:

  • Make the invitation engaging. Keep in mind where the respondent will be when they receive the invitation. Are they sent an invite via e-mail or is the invitation presented to them on-screen while viewing a Web page? Will they be invited via store signage, a receipt, a postcard, a proprietary mobile app or short-code text messaging? For pop-up invitations coming from a Web site, tie the look and feel of that site into the invite (use similar colors, font, etc.) so it feels like a part of the site they’re visiting. Always try to reinforce the brand when possible.
  • Graphically-rich e-mails are not considered spam anymore. There was a time when we recommended to our clients that we send a text-based e-mail so that the invitation doesn’t appear as spam to the prospective respondent or to e-mail spam filters. Since spam fatigue has largely disappeared due to spam filtering technology, participation rates increase with graphically-rich branded e-mail invitations. 
  • Keep the invitation text short, clear and compelling. The invitation will only have the viewer’s attention for a few seconds, so make the words you use count. Be sure to convey the sense that customer or site visitors’ opinions matter and that the feedback they provide can influence the development of new and existing products and services. Be informative and honest about the survey length and let the respondents know about any incentives being offered, if applicable. If it’s a 10-minute survey and the invitation tries to draw respondents in by stating a five-minute survey length, respondents are more likely to drop out or give more negative responses because they will feel misled.
  • If the survey invitations are only being sent to a specified group, let the respondent know this. This can convey the idea that the respondent has a special opportunity to give feedback.

Other information/considerations

  • Web site pop-up invitations: Utilize site-wide pop-up survey invitation code across a given site. Pop-up invitations have proven to be very effective in generating traffic for Web site evaluation surveys, etc.
  • Media campaign-generated traffic: Are you or your client doing a media campaign to drive traffic to you or your client’s site? Pre- and post- surveys are a great way to measure media/ad campaign effectiveness. If there’s a media campaign running or upcoming, help establish a baseline prior to its launch. Then run the survey again to measure increased traffic, satisfaction, purchases, etc., depending on what the research is designed to measure.
  • Text-back messaging (for example, Text “my survey” to 41411 on your mobile phone and you’ll receive an invitation via text to a CatalystMR demo mobile survey): This can be especially effective for invitations where you’re looking to draw in a large audience at an event (sporting event, conference, etc.) or store location, point-of-purchase, etc.
  • QR codes: Custom QR codes that link to a mobile survey help drive traffic and require little effort of the respondent. QR codes can carry data within them that can aid the research, drive survey logic and customize the survey experience. For example, QR codes can contain data like a store number and purchase information to track where respondents are coming from. QR codes can also reinforce your client’s brand by putting a logo or other image within the QR code itself.

What to ask when conducting Web site intercept surveys

Understanding the research objective is critical to guiding your mobile survey data collection. Here are some things to understand:

  1. What are the site traffic statistics? 
  2. What does the traffic to the site look like? How many site visits to the site per month? What percentage of them are new? How much time do visitors spend on the site and on what specific pages of the site? What pages are the most visited pages? 
  3. How many unique visitors per month vs. return visitors?
  4. What reason(s) do visitors have for visiting the mobile site? Are they looking for general or specific information, to make a purchase or because they’re excited about a new product offering?

As mobile continues its rapid growth, the list of best practices will no doubt evolve to meet changing technological capabilities and respondent habits and preferences. Please feel free to reach out to us with your own ideas or insights on how the industry can work together to advance this very important research methodology.