A comparison of phone and online research

By Michael Hesser | January 1, 2008

Reading time: 7 minutes

Abstract

Telephone surveys have, in the past, been the method most used by research companies based on positive response rates and broad access to most U.S. households. With the continuing evolution of online surveys and the depth of Internet penetration, marketers and researchers must consider the pros and cons of each method carefully prior to commencing a research project. This overview discusses a variety of issues and considerations related to telephone and online survey research with the aim of providing direction for firms in the initial stages of developing or reviewing their approach to this area of research.

Research Topics:: Online Research | Quantitative Research | Telephone Interviewing/CATI
Content Type: Research Methodology

Share Print

Each has its strengths and weaknesses

Editor’s note: Michael Hesser is president of the Praxi Group, a Kittredge, Colo., research firm.

One of the most important issues for researchers when developing a study is the selection of an appropriate data collection technique. Key questions faced by researchers at this point in the project lifecycle include:

What is the most direct and cost effective means of reaching the targeted sample segment?
What methodology will yield the most representative and projectable sample for the research project?

One of the historic strengths of telephone surveys has been the ability to be representative and projectable to the greater U.S. population. With more than 90 percent of households having a landline telephone, these surveys have provided researchers with a direct and socially acceptable means of contacting and recruiting survey participants. In recent years, however, several trends have emerged that have diminished the advantages of telephone surveys and compelled researchers to seek out new alternatives for data collection. These include:

declining participation rates for telephone surveys, which result in higher costs;
a growing dependence on wireless telephones (some estimates put cellular-only households at more than 10 percent of the U.S. population); and
a decreased receptiveness to unsolicited contacts at home (e.g., the growth of the Do Not Call list). Growing concerns with the reliability and validity of telephone survey research have coincided with the rampant growth of online or Internet research. Online surveys typically utilize groups of previously-recruited consumers - or panels - as a means of creating an accessible sample for fielding the surveys. Given the nature of the medium, online surveys can offer certain advantages over more traditional approaches, including cost-effectiveness, time-in-field and questionnaire format and complexity. At the same time, the publicly-held view of Internet research as cheaper and faster brings with it a unique set of challenges that must be taken into consideration, such as:
How appropriate is an online survey given the characteristics of our target segment? For example, do they have access to the Internet to an extent that will support survey research?
What is the nature of the available Internet survey panels, in terms of geographic coverage and demographic characteristics?
How is the Internet panel managed in terms of recruitment, turnover, incentives and limits on ongoing participation over time?

Projectable to the universe

A key consideration of any primary research study has to do with the degree to which the results of the sample survey are projectable to the population of constituents that sample is designed to represent. Toward this end, a truly random - and therefore projectable - sample must meet two key criteria:

all universe population members must have an equal chance of being contacted for the survey; and
all universe population members must have a known chance of being contacted for the survey.

Certainly, any primary research project likely falls somewhere short of the ideal relative to these definitions. Still, the respective strengths and weaknesses of telephone versus online research must be taken into account when selecting the appropriate methodology for any given study.

One of the greatest concerns with online research is whether it is truly representative of the U.S. population and can be administered in a truly random fashion. In the instance of Internet panels, online studies are reflective only of the panel members and not necessarily of the entire population of online consumers in the U.S.

A study by MRWeb.com (March 2007) reported that 30 percent of consumers in the U.S. are not online and of those, 44 percent are not interested in anything on the Internet. This calls into question the ability of Internet research to be projectable to the U.S. population, when 30 percent of that population is not reachable via the medium.

Caution must be exercised when considering the validity of Internet panels that claim high levels of representation simply because 70 percent of the U.S. population has access to the Internet. With panel sizes typically ranging from three million to five million members, panel samples are representative only of the panels from which they are selected.

That is not to say that online research is not valid; rather, caution must be taken when conducting online studies. If the sample is to be representative of the U.S. population, then measures need to be taken to reflect the offline population such as weighting on the back end or balancing the sample before fieldwork.

Additionally, to be truly random as stated above, each panel member must have an equal and known chance of participating in any given study. This requires the careful eye of the online sample management team.

The amount of time spent in data collection is inversely related to the reliability of a sample. Projects that complete thousands of interviews in a single weekend suffer from greater non-response bias than any telephone research survey.

As a means of managing these issues, an online study must be left open long enough to include those who are not online as often the chance to receive the invitation and participate in the study. “Professional respondents” also need to be carefully controlled as people become adept at knowing how to qualify for studies and once in the survey they may fabricate answers.

Again, managing the online sample to ensure panel members are invited randomly is important, as is controlling the number of surveys panel members can participate in during a certain period of time.

Fit the objectives

Like all research studies, the approach and methodology taken must fit the objectives of the research. Neither telephone surveys nor online surveys are appropriate for all research projects; therefore, it is important for researchers to consider the strengths and weaknesses of each approach and choose the method that best suits the study objectives.

Telephone

Until recently, telephone surveys have been the most prevalent means in which to collect consumer data in the U.S. Telephone surveys surpassed the earlier methods of time-consuming mail surveys or costly face-to-face surveys by providing faster and more effective data collection without sacrificing validity.

Several key advantages remain for conducting telephone surveys, including:

Random-digit dialing techniques ensure a randomly chosen sample.
The sample can be representative of the U.S. population, with 90 percent of households having a home phone.
Interviewing difficult-to-reach respondents. Telephone interviews can be more productive and yield a better response rate.
Capturing open-ended responses. A live interviewer can thoroughly probe the respondent and gain a better understanding behind why the respondent answered the way they did.

There are, however, several factors that have risen in recent years that impact telephone interviewing, including:

The creation of the do-not-call list. Although this does not directly affect market research firms, it does indicate the direction in which consumers are moving to restrict unsolicited calls.
More consumers are compelled to have an unlisted number or cell-only number, which further restricts undesired calls.
Costs can be high when specific populations are targeted or the incidence is low - like finding a needle in a haystack.
Time-in-field can also be longer, in that fieldwork cannot be conducted as quickly when the rules of randomness are applied (such as a minimum number of callbacks made to reach each respondent).

Online

Online research has been used in the U.S. for more than 10 years, with continuous progress made in terms of both sample availability and survey technology.

Some advantages to online research include the following:

More sophisticated questionnaires. 3-D images or videos can be shown online that are realistic and complex rotations can be programmed for difficult-to-administer tasks such as choice modeling exercises.
Speed. Data collection can be shortened as thousands of e-mail invitations can be delivered at one time.
Costs. Costs can be less than telephone interviews especially if the targeted samples are of general online users rather than the general consumer population.
Reduction of incidence. Because many panels include specific demographic and behavioral characteristics of their members, the concept of random incidence is avoided based on the ability to target specific panel members who are known to possess the required characteristics.

Again, there are also disadvantages to using online surveys including:

Truly random and representative samples are difficult to achieve, as noted previously.
Speed of data collection is inversely related to the quality of the data - again, as noted previously. If sample management teams are allowed to send out as many e-mail invitations as desired to fill the quotas, this also allows only those online at that time to complete the survey. Slow starts need to be employed to ensure some randomness.
Costs can also be more than traditional telephone interviews if the incidence of the targeted population is low. For some groups, there is little chance the targeted population has e-mail addresses, thus requiring a hybrid approach such as a combined telephone recruit to a Web survey. In other instances, a significant incentive is required to encourage specialized populations to participate (such as physicians).

A point of comparison was made using a hypothetical study that could be conducted using either an online panel or by random-digit dialing on the telephone. The study was outlined as being a general-population study of respondents aged 18-64 years of age, with a mix of gender and a qualified incidence rate of 85 percent. The questionnaire was assumed to be 15 minutes in length, with a targeted sample size of 500 respondents.

The two different methodologies produced the comparison points as shown in Table 1.

Diligent consideration

In summary, any primary research project will benefit from diligent consideration of the advantages and disadvantages associated with various alternative data collection methodologies. While the rampant growth of Internet research continues to offer marketers new and different options, the disadvantages of Internet research from the standpoint of validity and reliability must be considered in relation to the potential and relative cost and timing benefits.