Does speed kill the data?

Editor’s note: Paul Curran is senior project director at Gongos and Associates, Inc., a Bloomfield Hills, Mich., research firm.  Nicole Cicogna is vice president of sales at FGI Research, Norwalk, Conn. Formerly she was a senior account executive at Survey Sampling International, Fairfield, Conn.

A key benefit of using an online consumer panel is the ability to shorten field length. Obtaining consumer input faster can give marketers a competitive advantage, especially for quick-hit issues that require timely decisions.

Although sampling is the foundation of any research project, researchers have struggled to find a sound source from which to draw an online research sample. Online consumer panels have become one of the most popular methods of sampling for an Internet survey. Mail and telephone consumer panels have been used for years, so many researchers see Internet-based panels as simply a logical extension of an established sampling technique.

Researchers can collect hundreds of surveys in one evening using an online consumer panel. To compare, it would take at least three weeks to collect the same amount of surveys using a mail methodology. If cost is not an issue, telephone interviewing can sometimes be used to complete fieldwork in one night. Unfortunately, experience tells us that completing a telephone study in one night does not always produce the most representative or reliable sample (i.e., different types of people tend to be available at different times on different nights).

Some researchers have wondered if the rapid pace at which online surveys are completed may introduce similar problems associated with overnight telephone studies. Past research suggests that between 70 percent and 80 percent of responses to online survey invitations are received during the first day of fielding. If the quota for a given project or quota cell is reached in that first night, then 20 percent to 30 percent of your sample elements did not have an equal opportunity to take part in the study.

In this article, we explore the use of replicate sampling in an online application. Replicate sampling is a procedure often used by government agencies, academic institutions and research organizations that require strict adherence to probability sampling. Instead of using one large sample frame, the sample frame is divided into small mini-samples. Each of these mini-samples has identical characteristics of the full sample frame. Researchers thoroughly work one replicate at a time, not releasing a new replicate until the previous one is exhausted. This technique ensures full use of the sample frame and prevents a certain type of consumer from being disproportionately represented in the final set of data.

Pure replicate sampling requires a lengthy, systematic process that, unfortunately, isn’t always possible or realistic in the realm of commercial market research. Market researchers have, however, adapted the procedure in such a way that is now commonly used in offline methodologies such as telephone and door-to-door interviewing. It is unclear if applying a similar technique to Internet research would prove beneficial, particularly when the research involves the use of an online research panel.

This is the main focus of our research. While faster fieldwork is a key benefit of doing online research, we need to understand if completing the fieldwork quickly impacts the quality of the data we collect. By expediting the fieldwork, do we bias our sample with those who are more eager to participate in surveys or by those who may check their e-mail more frequently? Are those more eager to participate in surveys reading diligently and answering in a coherent way, or are they rushing through just to get the incentive?

To answer these questions, we analyzed the results of a typical online survey taken by members of an online research panel. Panelists were randomly placed into two groups — One-Time and Staggered. All One-Time invitations were sent to panelists on a Thursday evening (Thursdays tend to produce high response rates). Invitations for the second group, Staggered, were divided into replicates of equal size and released periodically over an eight-day period. Responses for the Staggered group were collected for 10 days to ensure all records were available on at least one Thursday evening.

The panel and survey instrument

The sample for this project was donated by Survey Sampling International (SSI). The sample was drawn from SSI’s online panel, SurveySpot, which consists of over one million panelist households. Panel size is monitored to prevent over-surveying as well as under-surveying in an effort to maintain panelists’ interest in participating. Panelists are also offered rewards in the form of entries into a monthly cash prize drawing with each survey invitation, increasing their likelihood of participation.

The sample was national in scope and balanced to the most recent U.S. Census demographics. In this experiment, the same sample sizes were selected for both groups, using an nth selection technique. A total of 2,340 were selected for each group.

The survey instrument focused on consumer preferences for Hollywood movies and had four main sections: demographic profile questions, general movie preferences, attitudinal rating and semantic differential scales, and a discrete choice exercise. Each of the four sections was used in our assessment of the data.

All data collection was done using Gongos and Associates’ online research tools. Likewise, Gongos and Associates did all the data analysis.

Field diagnostics

A review of field diagnostics between the two groups revealed a couple of interesting findings.

First, we found replicate sampling in an online application to be extremely inefficient. The One-Time sample yielded 216 completions in the first 24 hours compared to the Staggered approach’s 143 completions over a 10-day fielding period. As noted previously, we knew that Thursdays tend to be a very productive night for online panel sampling. Even though all the Staggered records were active for at least one Thursday evening, its response rate was dismal compared to that of the One-Time group.

From this observation, we hypothesize that mailbox clutter may impact response rates as much or more than the day of the week invitations are sent. The more full one’s e-mailbox becomes, the less likely one is to respond to a survey invitation. The time and day survey invitations are sent becomes an important consideration when executing a successful online research project.

The other finding worth noting was the rate at which qualified respondents abandoned the survey. Those responding to the One-Time invitations were much more likely to stick with the survey than were those in the Staggered group. Response and incidence rates are shown in Table 1.

Experiment results

Our first challenge was to measure the effect of the sampling procedure on the ultimate survey results. We wanted to establish whether or not the two groups were composed of the same type of respondents and if the two groups answered questions in the same way. To determine this, we looked at the demographic profile of the respondents and the extent to which questions were answered differently by the two groups.

Overall, there were no statistically significant differences observed between the two groups in terms of demographic profile. Likewise, there were very few differences observed in survey responses (about what you would expect to see using a 90 percent confidence level).

Our second challenge was to assess the quality of the data. To do this, we used three different types of assessments:

Respondent attentiveness (appropriateness of interview length):

Companies experienced in conducting online research know the importance of monitoring the time respondents spend going through the survey and even the time respondents spend on each page of the survey. Regardless of the sample source, there is always a small proportion of respondents who complete the survey in an incredibly short period of time. At Gongos and Associates, this proportion is referred to as the “race-through rate.”

Consistency in response choices:

When doing a discrete choice or choice-based conjoint survey, researchers will often include “hold-out tasks.” Hold-out tasks appear just like any other choice task except they are held out of the analysis when building the choice model (hence the name). While the main purpose of the hold-out tasks is to test the predictive ability of the model, they can also be used to test the reliability of the data being collected. This is accomplished by placing two identical hold-out tasks at different parts of the choice experiment. The percent that respond differently to identical choice tasks is referred to as the “test-retest failure rate.”

Data reliability:

Our survey instrument includes a “Hollywood animosity scale.” This is an additive scale based on five attitudinal statements related to common complaints of Hollywood movies. As part of our evaluation of data quality, we used a test statistic called Cronbach’s alpha to measure the reliability of the scale. A higher alpha score indicates a more reliable scale. We would expect to see no differences in the alpha statistic between the two groups of respondents.

With respect to data quality, the results were somewhat mixed:

  • There were no statistically significant differences between the two groups in terms of average length of interview. What’s more, both groups had similar race-through rates (well under 10 percent).
  • The test-retest failure rate was comparable for both groups. There were, however, some directional indications that suggest those in the Staggered group answered in a more consistent fashion.
  • The alpha measure for the Staggered group was slightly higher than that of the One-Time group, suggesting respondents in the Staggered group answered in a more cogent manner.
  • As one might expect, data quality measures increased considerably after removing those who were thought to have raced through the survey.

Conclusions

Overall, researchers would draw the same general conclusions from either dataset. Although our research shows that the Staggered approach may produce slightly higher-quality data, the negatives outweigh the positives in that the Staggered approach generates an extremely poor response rate. This experiment, along with other research conducted by Gongos and Associates and Survey Sampling International, allows us to offer the following recommendations to online researchers:

  • In most situations, it is best to release all the survey invitations at one time. Care needs to be taken, however, when a project involves filling multiple quota cells at an even pace (for example, when recruiting for an in-home product test or product clinic).
  • Encourage clients to field projects over a four- to five-day period. When an overnight sample is absolutely necessary, work with your research sample provider to determine the most strategic day/time to release the survey invitations.
  • Use quota sampling to ensure that your final sample matches the demographic profile of your population. Set quota cells up in a matrix format, as opposed to standalone quotas.
  • Dynamically monitor the length of time respondents move through the survey, terminating those who appear to be racing through without reading the questions. When working with a company that hosts a survey for you, make sure it can accommodate this request and insist it include real-time monitoring of race-through rates.

Sound practices

The early days of online research were marked with the fervor of gung-ho advocates and the cautious uncertainty of old-school skeptics. Over the years, top research companies and sample providers have worked hard to cut through the hype and hyperbole and carve out a set of sound online research practices. Many of these best practices have been refined through careful analysis of case studies and experiments. We hope this article contributes to that body of research and that it may encourage additional research focused on online research sampling.