E-interviewers add human touch to Web-based research

Editor’s note: Karl G. Feld manages quantitative research for clients of Western Wats Center, a Provo, Utah, research firm. Steven Wygant works in the office of assessment at Brigham Young University in Provo.

Despite the recent swings in the tech-heavy NASDAQ, e-commerce remains a hot topic. Depending on who you talk to, the horizon beckons with seemingly limitless opportunity. Is it any surprise then that Web-based research intrigues the research industry with its possibilities? Eyeing the rapid growth of the Internet, some research insiders predict phenomenal growth and business opportunities in Web-based research -- the gold rush of 1849 all over again. Some research companies have openly embraced the Web methodology; others race to catch up, while still others warily consider it. All of which leads to some very important questions. Is a high-tech Web-based survey really any better than an old-fashioned, plain vanilla mail survey? Can we move beyond current Web technology to increase the quality of the data by reintroducing people into the process? Can the e-commerce concept also work for e-interviewing?

A confluence of need and curiosity created an opportunity to provide preliminary answers to some of these questions. Brigham Young University (BYU) and Western Wats Center (WWC) joined forces to explore some of these issues. BYU’s Office of Assessment hired WWC to interview university alumni about BYU publications. WWC proposed testing a new concept: live interviewers interacting with Web-based respondents (e-interviewers). WWC proposed a data gathering approach that would facilitate comparisons between three data collection methods: traditional CATI, self-administered Web surveys, and interviewer-assisted Web surveys. WWC used its data collection staff and facilities to conduct the study for BYU.

Our approach

To meet the customized needs of this client’s study, WWC’s Internet technicians wrote the software to program the Web survey. The software incorporated capabilities long available in standard CATI programs. It facilitated list rotation, error traps, forced responses, and complicated skip patterns. It also prevented the respondents from reading or answering questions out of order. To facilitate legitimate and defensible comparisons, the WWC programmers incorporated these features into the CATI program, the self-administered program, and the e-interviewer-assisted Web program.

Like most Web-based research today, the standard Web questionnaire filled the entire screen. The respondents clicked to answer the questions. The e-interviewer version of the Web survey contained additional features designed to introduce a live interviewer into the process. The actual questionnaire filled the right two-thirds of the respondent’s screen. An e-interviewer chat box occupied the remaining one-third of their screen. The chat box contained separate fields and “submit” buttons. This allowed the e-interviewer and respondent to carry on real-time chat independent of the questionnaire fields. The intent was to approximate the positive benefits of a real respondent interacting with a live interviewer over the phone.

Sampling procedures

WWC used the entire universe of BYU alumni with an e-mail address (11,892 individuals) to create the Web sample frame. The study design excluded these individuals from the CATI sample frame. The remaining known universe of BYU alumni (141,337 names) comprised the sample frame for the CATI portion of the research. The goal was 300 completed interviews in each of the three groups (900 total interviews). Projecting one completed interview per four sample points, WWC selected 1,200 sample points for each of the three interviewing methods (3,600 total sample points). WWC selected every nth name from the two files. Using the gender, census region, and age information contained in the BYU alumni database, WWC selected the three samples proportional to gender, census region and age of the U.S. online population as defined by Iconoclast in its Winter 2000 edition of “Internet At A Glance.”

The study design assumed a response rate of 25 percent. The Internet versions of the study achieved lower than anticipated response rates. Consequently, we adjusted the final sample sizes to 100 for the two Internet methods. The sample demographics match the U.S. Internet population. Consequently, the actual final sample sizes exceed 100.

Data collection procedures

CATI
From March 28 to April 19 WWC’s professional interviewers initially dialed the telephone sample between 5 p.m. and 8 p.m. Monday through Saturday. WWC controlled for the effects of gender bias by assigning equal numbers of hours to male and female interviewers. To limit interviewer bias no interviewer worked two consecutive days. The study design included a maximum daily quota of 35 completed interviews to minimize any potential day of the week effects. The interviewers dialed each number in the sample frame a maximum of three times over the course of the study. No more than two attempts occurred per day. The interviewers subsequently attempted to contact individuals between 9 a.m. and 8 p.m. to reach the full-time housewives, students and retirees who made up part of the sample. In an attempt to achieve a balanced sample, supervisors called any individuals who initially refused to participate.

Internet
WWC used e-mail to invite individuals in the two Internet samples to participate in the survey. The body of the e-mail contained a URL directing them to the questionnaire and a unique PIN number that allowed them access to the questionnaire. Each day over a two-week period, 100 individuals from the self-administered sample and 75 individuals from the e-interviewer-assisted sample received an e-mail seeking their participation in the study. Each sample point subsequently received a reminder e-mail message two days later. The difference in the number of invitations resulted from a purposeful over-sample of the self-administered respondents. Respondents could exit the survey and return to it at any time. The software automatically resumed the survey at the correct question.

The individuals in the self-administered sample completed the interviewer unassisted and at their convenience. In today’s jargon, WWC provided 24/7 staffing for the e-interviewer-assisted version of the questionnaire. As individuals in the assisted sample entered their PIN number and birth date, the system notified the interviewers of impending respondents. The interviewers communicated with each respondent using the live chat function. Initially they confirmed that the appropriate respondent completed the questionnaire. Then they monitored and assisted the progress of the respondent.

Questionnaire length
WWC and BYU used identical questionnaires for the three methods. The CATI surveys lasted an average of 17 minutes. The self-administered Internet questionnaire lasted an average of 22 minutes. The interviewer-assisted Internet questionnaire lasted an average of 37 minutes. Technical issues and the interaction of the interviewer accounted for the difference in the average length of the interviewer-assisted and self-administered Internet questionnaires. A fundamental premise underlying the e-interviewer concept is that the interaction and oversight of a human will improve the quality and completeness of responses -- in essence, a human touch in an otherwise high-tech wilderness. By design, the Internet questionnaire was divided into 10 “pages.” After the respondents completed and submitted a page, the e-interviewers reviewed every response for completeness before sending the next page. If they discovered incomplete or inconsistent answers, they clarified or probed the respondent using the chat feature.

Findings

Cooperation and refusal rates ¹
Despite the fact that it took more time, on average, than the self-administered questionnaire to complete, the e-interviewer version also achieved a higher cooperation rate than the self-administered Internet questionnaire. The use of live interviewers helped retain reluctant respondents. By using the chat function, the interviewers encouraged reluctant respondents to complete the survey, much as live interviewers keep respondents on the telephone during CATI-based studies. The CATI method also attained the highest cooperation rates.

CATI = 59.5%
Interviewer-assisted Internet = 29.3%
Self-administered Internet = 26.4%

Finally, the e-interviewer-assisted method had a lower refusal rate than the self-administered Internet study. Evidently a human is still more effective in convincing a respondent to participate in a survey than e-mail and the Internet. The unintended length of the e-interviewer version directly contributed to a refusal rate higher than it otherwise might have been. Using the chat feature, we learned that a sizeable number of respondents terminated the interview because of the length. The CATI method had a notably lower refusal rate than the two Internet methods.

CATI = 26.8%
Interviewer-assisted Internet = 63.3%
Self-administered Internet = 67.7%

Data quality: open-end question responses

Obtaining rich and meaningful responses to open-end questions remains problematic for any research method, but particularly for self-administered studies. In our study, each version of the questionnaire included the same three open-end questions. All three versions of the questionnaire prompted the interviewer and/or the respondent for more complete answers to the open-end questions. The CATI program initially prompted the interviewer to ask the question. It then prompted the interviewer to probe for a more complete or in-depth response. Similarly, the Internet version of the questionnaire presented the question on the screen. After the respondent submitted an answer, the program displayed the answer and requested additional information. All three versions used identical language for the questions and the subsequent probes. The CATI and e-interviewer versions enjoyed the advantage of live interviewer interaction. The CATI interviewers (using the telephone) and the e-interviewers (using the chat function) probed answers from reluctant respondents who either initially refused to answer the question or who provided short or incomplete responses.

On average, the e-interviewer-assisted version gathered more words per response than either the CATI or self-administered Internet versions on all three open-ended questions and their written follow up probes. Admittedly, live interviewers interacting with a respondent and the CATI program don’t typically record every idle word uttered by the respondents. Nevertheless, the e-interviewer-assisted version collected from one to four words more than the self-administered Internet version. It collected from less than one to as many as 11 words more than the CATI version (see Table 1).

Table 1
Average Number of Words Per Open-end Question Response

While verbosity does not automatically translate into quality, a non-response is a non-response. An analysis of the open-end responses revealed that the typed responses on the Internet version generally yielded richer data than the verbal responses given on the CATI version. This coincides with findings from previous analysis of responses to mail surveys.

Eligibility control

Curiously, a few respondents who don’t meet the eligibility requirements of a given study often attempt to participate in the study, occasionally circumventing fairly detailed security measures in the process.² We observed this phenomenon in our study. Two individuals who did not meet the eligibility requirements attempted to participate in the e-interviewer-assisted version of the questionnaire. Using the chat function, the e-interviewers discovered both attempts and terminated the interviews. The CATI interviewers terminated five interviews when they discovered that an individual not named in the sample was attempting to complete the questionnaire. By definition, self-administered surveys cannot screen for the correct respondent. They depend totally on the honor of the respondent. The nature of increasing non-response rates and the documented problems with professional respondents who avoid screeners suggests that some of the respondents to the self-administered survey were certainly not eligible.

Conclusion

It is clear that interviewer-assisted Internet surveys produce data with greater reliability, lower refusal rates and richer open-ends than unassisted Web surveys. Accounting for the constraints of the pilot technology used, it is quite probable that an improved survey engine will achieve even greater reductions in response rates while preserving the gains in data quality outlined above. It is clear that the use of e-interviewers in survey research warrants further examination.

Notes

1 These rates have been calculated using AAPOR’s 1998 Standard Definitions for Final Dispositions of Case Codes and Outcome Rates for RDD Telephone Surveys. These definitions are built on CASRO’s 1982 standards.

2 See “A statistical approach to security/past participation problems” in the March 2000 issue of Quirk’s Marketing Research Review for a discussion of this phenomenon.