Be prepared
Editor’s note: Al Fitzgerald is president and founder of Answers Research, Inc., Solana Beach, Calif.
With well over 600 million Internet users worldwide, there is no questioning the viability of online surveying to collect market research data. A steady general growth along with huge usage percentage leaps in the U.S., Europe and the Asia-Pacific region have allowed marketers to maximize scant research budgets and include otherwise cost-prohibitive samples in a study’s scope.
This online expansion has occurred despite a common economic slowing, and as consumer and business users continue to recognize the value online communication provides - with sure-sell features including real-time data transaction and the elimination of time and space consuming paper trails - we can only expect a continued rise. To ensure an online study delivers actionable, timely and cost-saving results, marketers must be mindful of the vast number of pitfalls that can trip up even the most seasoned researcher. These snares can occur at any phase in the research process but are easily circumvented by careful planning by project managers prior to beginning any online study.
Underestimating complexity
Because large phone rooms are not needed for online data collection, companies have been tempted to “in-source” studies for which they would typically hire an outside data collection firm. In theory, this provides huge research budget savings by having an in-house IT department program and host the survey. In practice, it can stretch an internal research department too thin.
Even an online study completed strictly via e-mail can be a time-consuming event. In this case, recruitment is conducted via e-mail with the survey included in the invitation. Respondents need only e-mail back their answers and the internal market research team can keypunch results to slice the numbers and conduct analysis. Most e-mail studies, however, involve sample procurement and cleaning, keypunch programming to ensure logical skip patterns, fielding management (being mindful of any quotas), data download and cleaning, and final analysis and reporting. With the internal research team providing critical strategy and objectives, the implementation and analysis processes are often best completed by securing the help of experienced outside researchers assigned to ensure each phase is completed without a hitch.
Panel overuse
There is no denying the rewards of using a panel to recruit for an online study, but it is imperative to utilize this tool without compromising data quality. Since contact and profiling data (provided in the initial membership survey and subsequent screeners) for each member is included in a database, a list of targeted sample can be created in a few keystrokes.
While panel usage is a key factor in being able to minimize fielding costs, careful database management must be taken to prevent respondent overuse. At a minimum, survey participation must be limited to avoid any bias, and utilizing the same panel to complete multiple simultaneous studies should be avoided or scrupulously monitored.
Recognizing the limitations
As global online penetration grows, using an Internet methodology is fast mirroring a traditional telephone random-digit dial methodology. Even when online data collection closely matches the projectability of a telephone survey, there may still be benefits to telephone or in-person approaches that trump online surveying.
Conducting questionnaires via phone or in-person enables probing to better understand answers. In a case where it is crucial to ask open-ended questions, having a trained interviewer is an advantage. For instance, on written surveys (such as paper or Web-based surveys) a respondent may say “speed” was the reason for selecting a brand without clarifying what they mean by speed. An in-person or phone interviewer can ask for this clarification.
Telephone or in-person data collection methodologies are necessary if no e-mail address or name is available, although these methods are often relatively unproductive and expensive when compared with online data collection. For example, with no e-mail address - or even the name of a person - respondents can be recruited by calling a company switchboard and asking for a referral (in the case of a business interview) or using a telephone book or in-person intercept methods (to find consumers).
Programming
Aside from finding the actual respondents to complete the survey, the most important factor in the success of an online study is the survey design. It is critical for researchers to have a lucid survey (including logical skip patterns, etc.) prepared before the questionnaire is programmed. The sponsoring marketers should be involved in survey content development by providing a list of clear objectives, including the business decisions to be made using the collected data.
Once the survey is programmed, it must be tested to ensure a flawless presentation on all browsers and monitor sizes. This is the final chance for mistakes to be corrected and any modifications to be added. Re-programming costs should be clearly discussed so that unexpected additional costs do not erode the relationship between a company and contracted research firm.
Professional respondents
Typically, using a Web-based approach lowers non-respondent bias due to higher response rates. Since respondents can take the survey at a time convenient for them, a higher percentage of respondents participate. Otherwise, respondents who are less busy or easier to reach disproportionately participate in surveys. This can be seen in telephone surveys where education and government respondents often disproportionately complete more surveys than they represent in the overall telephone list.
Marketers require thoughtful answers to their questionnaires, not hurried responses to get to the incentive at the finish line. To minimize these incentive-interested respondents, it is imperative to conduct careful sample management. If panels are utilized, limitations must be included in the membership contract that specify the maximum number of surveys that can be completed annually. Enforcing strict screening requirements and disclosing the study’s sponsor can aid in getting the most honest answers out a respondent. If participants stand to benefit from their answers, they are more likely to carefully consider their responses.
Overestimating the convenience factor
A common misconception when conducting online research is that the data collection is “simple” and little can go wrong. While the pressure of not having to complete surveys via phone lessens the visible urgency, the fielding process requires constant monitoring. Respondents may require two or more e-mail reminders or even a phone call to spur them to complete the questionnaire. If quota groups are included, respondents fitting that group’s profile should not be allowed to complete the survey once that quota is reached. Regardless of whether the data is needed for analysis, the respondent must be paid the incentive for completion.
Hard-to-fill quotas may require additional recruitment efforts. Daily or even hourly monitoring is required to gauge how many completed surveys are still needed, and whether additional incentives or sample is required. Research project managers are wise to scour the recruitment specifications prior to the start of fielding to ensure the study’s sponsor knows the difficulty of reaching their targets.
Phone vs. Web
Notwithstanding cost-savings being a key selling point of utilizing Web-based surveying, marketers must not assume that Internet surveys are always less expensive than a phone approach. In general, when conducting surveys among a group of less than 100, programming costs outweigh the benefit of completing a survey online.
In cases where there are small sample sizes with a small list of qualified candidates from which to recruit, it is most effective to call each respondent on the list to complete the questionnaire over the phone. Additionally, researchers must be aware of the situations in which a pure random sample of a population is required. In this case, a telephone random-digit dial method is often employed.
Projectability
The growth in the percentage of people who have access to the Internet either at home, work, or school results not only a rise in online access rates but a decrease in any data bias that may occur via an online methodology. Still, until Internet usage rates match those of the phone (with over 95 percent U.S. penetration), measures must be taken to ensure results are projectable to the whole of the population the study represents.
Selecting samples that accurately match the overall population or weighting the final data can fuel a study’s projectability. Additionally, a dual telephone/Web methodology can be implemented. In this case, respondents are recruited via phone and given the option to complete the survey immediately on the telephone, or at a later time on the Web.
Poorly-written invitation letter
Even if a survey is designed according to the client’s exact specifications and programmed with logic in place, if the invitation to complete the survey is not enticing, the study may never get off the ground. The invitation should clearly outline the rewards the respondent will receive in exchange for participation, state how long the survey will take to complete, and the reason why the research is being conducted.
Unfortunately, the explosive rise of Internet usage has been closely shadowed by the proliferation of spam. In the continuing battle of e-mail user vs. spammer, people tired of sifting through messages from unknown senders often program their e-mail mailboxes to filter messages of unknown origin to a junk or trash folder. Whether the e-mail invitation makes it to the inbox or junk mail file, it stands a better chance of being opened if as much as possible is revealed in the heading.
Internet connections
As is true for most studies, the simpler the survey the better. However, the Internet does offer the benefit of supporting images, and if including these will give respondents a better idea of the product, service or concept they are answering questions about, marketers are wise to incorporate them. While simple graphic images (such as jpeg or gif graphic files) can be accommodated by respondents regardless of their connection speed, more sophisticated images require greater bandwidth.
Incorporating real-time video into a survey may work fine if the respondent has a broadband connection - Internet users in a business environment fit this bill. However, expecting the average consumer to be able to complete the same survey would lead a marketer back to the drawing board. No consumer with a dial-up modem connection will wait the many minutes required for a video download. Researchers must have a thorough understanding of their audiences’ technical capabilities - as well as their patience levels - prior to adding large image files.
Make the most of it
It stands to reason that as Internet usage increases, the implementation of studies using a Web-based fielding methodology will also grow. Despite its association with speed and savings, marketers need only get burned once to never again underestimate the complexities of online implementation. By beginning a Web-based research project with an awareness of the factors for success and the consequences of poor planning, marketers are best prepared to make the most of this viable methodology.